Instruction:
Implement a Python 3 function for
authenticating a user with a TMDb username and password and return a dictionary that contains the JSON response from the API.

Response:
def token_validate_with_login(self, **kwargs):
        """
        Authenticate a user with a TMDb username and password.  The user
        must have a verified email address and be registered on TMDb.

        Args:
            request_token: The token you generated for the user to approve.
            username: The user's username on TMDb.
            password: The user's password on TMDb.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('token_validate_with_login')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Can you create a Python 3 function that
generates a new session id for user based authentication.

Response:
def session_new(self, **kwargs):
        """
        Generate a session id for user based authentication.

        A session id is required in order to use any of the write methods.

        Args:
            request_token: The token you generated for the user to approve.
                           The token needs to be approved before being
                           used here.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('session_new')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
How would you code a function in Python 3 to
generate a guest session id.

Response:
def guest_session_new(self, **kwargs):
        """
        Generate a guest session id.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('guest_session_new')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
How would you code a function in Python 3 to
get a list of rated moviews for a specific guest session id.

Response:
def rated_movies(self, **kwargs):
        """
        Get a list of rated moview for a specific guest session id.

        Args:
            page: (optional) Minimum 1, maximum 1000.
            sort_by: (optional) 'created_at.asc' | 'created_at.desc'
            language: (optional) ISO 639-1 code.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_guest_session_id_path('rated_movies')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Write a Python 3 script to
check to see if a movie id is already added to a list.

Response:
def item_status(self, **kwargs):
        """
        Check to see if a movie id is already added to a list.

        Args:
            movie_id: The id of the movie.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_id_path('item_status')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Can you generate the documentation for the following Python 3 function
def create_list(self, **kwargs):
        path = self._get_path('create_list')
        kwargs.update({'session_id': self.session_id})

        payload = {
            'name': kwargs.pop('name', None),
            'description': kwargs.pop('description', None),
        }
        if 'language' in kwargs:
            payload['language'] = kwargs['language']

        response = self._POST(path, kwargs, payload)
        self._set_attrs_to_values(response)
        return response

Response: Create a new list.


Instruction:
How would you implement a function in Python 3 that
removes a movie from a list that the user created.

Response:
def remove_item(self, **kwargs):
        """
        Delete movies from a list that the user created.

        A valid session id is required.

        Args:
            media_id: A movie id.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_id_path('remove_item')
        kwargs.update({'session_id': self.session_id})

        payload = {
            'media_id': kwargs.pop('media_id', None),
        }

        response = self._POST(path, kwargs, payload)
        self._set_attrs_to_values(response)
        return response


Instruction:
Here you have a function in Python 3, explain what it does
def clear_list(self, **kwargs):
        path = self._get_id_path('clear')
        kwargs.update({'session_id': self.session_id})

        payload = {}

        response = self._POST(path, kwargs, payload)
        self._set_attrs_to_values(response)
        return response

Response: Clears all of the items within a list.


Instruction:
How would you implement a function in Python 3 that
gets the content ratings for a TV Series.

Response:
def content_ratings(self, **kwargs):
        """
        Get the content ratings for a TV Series.

        Args:
            language: (optional) ISO 639 code.
            append_to_response: (optional) Comma separated, any collection
                                method.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_id_path('content_ratings')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Write a Python 3 script for
getting the similar TV series for a specific TV series id.

Response:
def similar(self, **kwargs):
        """
        Get the similar TV series for a specific TV series id.

        Args:
            page: (optional) Minimum value of 1.  Expected value is an integer.
            language: (optional) ISO 639-1 code.
            append_to_response: (optional) Comma separated, any TV method.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_id_path('similar')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Can you tell what is the following Python 3 function doing
def on_the_air(self, **kwargs):
        path = self._get_path('on_the_air')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Query the list of TV shows that are currently on the air. This is a simple API call.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def airing_today(self, **kwargs):
        path = self._get_path('airing_today')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the list of TV shows that air today.


Instruction:
Explain what the following Python 3 code does
def info(self, **kwargs):
        path = self._get_series_id_season_number_path('info')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the primary information about a TV season by its season number.


Instruction:
Write a Python 3 function for
getting the cast & crew credits for a TV season by season number.

Response:
def credits(self, **kwargs):
        """
        Get the cast & crew credits for a TV season by season number.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_series_id_season_number_path('credits')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
How would you explain what the following Python 3 function does
def external_ids(self, **kwargs):
        path = self._get_series_id_season_number_path('external_ids')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response:
Get the external ids that we have stored for a TV season by season
        number.


Instruction:
How would you explain what the following Python 3 function does
def images(self, **kwargs):
        path = self._get_series_id_season_number_path('images')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response:
Get the images that we have stored for a TV season by season
        number.


Instruction:
Write a Python 3 function that can
get the videos that have been added to a TV season.

Response:
def videos(self, **kwargs):
        """
        Get the videos that have been added to a TV season (trailers, teasers,
        etc...).

        Args:
            language: (optional) ISO 639 code.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_series_id_season_number_path('videos')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Here you have a function in Python 3, explain what it does
def info(self, **kwargs):
        path = self._get_series_id_season_number_episode_number_path('info')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the primary information about a TV episode by combination of season and episode number.


Instruction:
How would you explain what the following Python 3 function does
def credits(self, **kwargs):
        path = self._get_series_id_season_number_episode_number_path('credits')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the TV episode credits by combination of season and episode number.


Instruction:
Explain what the following Python 3 code does
def external_ids(self, **kwargs):
        path = self._get_series_id_season_number_episode_number_path(
            'external_ids')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the external ids for a TV episode by combination of a season and episode number.


Instruction:
Explain what the following Python 3 code does
def images(self, **kwargs):
        path = self._get_series_id_season_number_episode_number_path('images')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the images for a TV episode by combination of season number and episode number.


Instruction:
Can you generate the documentation for the following Python 3 function
def videos(self, **kwargs):
        path = self._get_series_id_season_number_episode_number_path('videos')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Get the videos that have been added to a TV episode.


Instruction:
Implement a function in Python 3 to
set attributes to values.

Response:
def _set_attrs_to_values(self, response={}):
        """
        Set attributes to dictionary values.

        - e.g.
        >>> import tmdbsimple as tmdb
        >>> movie = tmdb.Movies(103332)
        >>> response = movie.info()
        >>> movie.title  # instead of response['title']
        """
        if isinstance(response, dict):
            for key in response.keys():
                if not hasattr(self, key) or not callable(getattr(self, key)):
                    setattr(self, key, response[key])


Instruction:
Can you write a function in Python 3 where it
searches for movies by title.

Response:
def movie(self, **kwargs):
        """
        Search for movies by title.

        Args:
            query: CGI escpaed string.
            page: (optional) Minimum value of 1. Expected value is an integer.
            language: (optional) ISO 639-1 code.
            include_adult: (optional) Toggle the inclusion of adult titles.
                           Expected value is True or False.
            year: (optional) Filter the results release dates to matches that
                  include this value.
            primary_release_year: (optional) Filter the results so that only
                                  the primary release dates have this value.
            search_type: (optional) By default, the search type is 'phrase'.
                         This is almost guaranteed the option you will want.
                         It's a great all purpose search type and by far the
                         most tuned for every day querying. For those wanting
                         more of an "autocomplete" type search, set this
                         option to 'ngram'.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('movie')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Implement a function in Python 3 to
search for collections by name.

Response:
def collection(self, **kwargs):
        """
        Search for collections by name.

        Args:
            query: CGI escpaed string.
            page: (optional) Minimum value of 1. Expected value is an integer.
            language: (optional) ISO 639-1 code.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('collection')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Create a Python 3 function for
searching for TV shows by title.

Response:
def tv(self, **kwargs):
        """
        Search for TV shows by title.

        Args:
            query: CGI escpaed string.
            page: (optional) Minimum value of 1. Expected value is an integer.
            language: (optional) ISO 639-1 code.
            first_air_date_year: (optional) Filter the results to only match
                                 shows that have a air date with with value.
            search_type: (optional) By default, the search type is 'phrase'.
                         This is almost guaranteed the option you will want.
                         It's a great all purpose search type and by far the
                         most tuned for every day querying. For those wanting
                         more of an "autocomplete" type search, set this
                         option to 'ngram'.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('tv')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Create a Python 3 function for
searching for people by name.

Response:
def person(self, **kwargs):
        """
        Search for people by name.

        Args:
            query: CGI escpaed string.
            page: (optional) Minimum value of 1. Expected value is an integer.
            include_adult: (optional) Toggle the inclusion of adult titles.
                           Expected value is True or False.
            search_type: (optional) By default, the search type is 'phrase'.
                         This is almost guaranteed the option you will want.
                         It's a great all purpose search type and by far the
                         most tuned for every day querying. For those wanting
                         more of an "autocomplete" type search, set this
                         option to 'ngram'.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('person')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Can you generate the documentation for the following Python 3 function
def company(self, **kwargs):
        path = self._get_path('company')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response

Response: Search for companies by name.


Instruction:
Can you create a Python 3 function that
searches for keywords by name.

Response:
def keyword(self, **kwargs):
        """
        Search for keywords by name.

        Args:
            query: CGI escpaed string.
            page: (optional) Minimum value of 1. Expected value is an integer.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('keyword')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Create a Python 3 function to
search the movie tv show person collections with a single query.

Response:
def multi(self, **kwargs):
        """
        Search the movie, tv show and person collections with a single query.

        Args:
            query: CGI escpaed string.
            page: (optional) Minimum value of 1. Expected value is an integer.
            language: (optional) ISO 639-1 code.
            include_adult: (optional) Toggle the inclusion of adult titles.
                           Expected value is True or False.

        Returns:
            A dict respresentation of the JSON returned from the API.
        """
        path = self._get_path('multi')

        response = self._GET(path, kwargs)
        self._set_attrs_to_values(response)
        return response


Instruction:
Can you create a Python 3 function that
normalizes and tokenize text. This is lifted from NIST mteval - v11a. pl.

Response:
def normalize(s):
    '''Normalize and tokenize text. This is lifted from NIST mteval-v11a.pl.'''
    # Added to bypass NIST-style pre-processing of hyp and ref files -- wade
    if (nonorm):
        return s.split()
    try:
        s.split()
    except:
        s = " ".join(s)
    # language-independent part:
    for (pattern, replace) in normalize1:
        s = re.sub(pattern, replace, s)
    s = xml.sax.saxutils.unescape(s, {'"':'"'})
    # language-dependent part (assuming Western languages):
    s = " %s " % s
    if not preserve_case:
        s = s.lower()         # this might not be identical to the original
    return [tok for tok in normalize3.split(s) if tok and tok != ' ']


Instruction:
Can you implement a function in Python 3 that
takes a list of reference sentences for a single segment and returns an object that encapsulates everything that BLEU needs to know about them.

Response:
def cook_refs(refs, n=4):
    '''Takes a list of reference sentences for a single segment
    and returns an object that encapsulates everything that BLEU
    needs to know about them.'''

    refs = [normalize(ref) for ref in refs]
    maxcounts = {}
    for ref in refs:
        counts = count_ngrams(ref, n)
        for (ngram,count) in list(counts.items()):
            maxcounts[ngram] = max(maxcounts.get(ngram,0), count)
    return ([len(ref) for ref in refs], maxcounts)


Instruction:
Write a Python 3 function that can
take a list of reference sentences and returns a set that encapsulates everything that bleualign wants.

Response:
def cook_ref_set(ref, n=4):
    '''Takes a reference sentences for a single segment
    and returns an object that encapsulates everything that BLEU
    needs to know about them.  Also provides a set cause bleualign wants it'''
    ref = normalize(ref)
    counts = count_ngrams(ref, n)
    return (len(ref), counts, frozenset(counts))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def align_probability(i, j, source_sentences, target_sentences, alignment, params):
    l_s = sum(source_sentences[i - offset] for offset in range(alignment[0]))
    l_t = sum(target_sentences[j - offset] for offset in range(alignment[1]))
    try:
        # actually, the paper says l_s * params.VARIANCE_CHARACTERS, this is based on the C
        # reference implementation. With l_s in the denominator, insertions are impossible.
        m = (l_s + l_t / params.AVERAGE_CHARACTERS) / 2
        delta = (l_t - l_s * params.AVERAGE_CHARACTERS) / math.sqrt(m * params.VARIANCE_CHARACTERS)
    except ZeroDivisionError:
        delta = infinity

    return 2 * (1 - norm_cdf(abs(delta))) * params.PRIORS[alignment]

Response: Returns the probability of the two sentences C{source_sentences[i], C{target_sentences[j] } being aligned with a specific alignment.


Instruction:
Can you tell what is the following Python 3 function doing
def align_blocks(source_sentences, target_sentences, params = LanguageIndependent):
    alignment_types = list(params.PRIORS.keys())

    # there are always three rows in the history (with the last of them being filled)
    # and the rows are always |target_text| + 2, so that we never have to do
    # boundary checks
    D = [(len(target_sentences) + 2) * [0] for x in range(2)]

    # for the first sentence, only substitution, insertion or deletion are
    # allowed, and they are all equally likely ( == 1)

    D.append([0, 1])
    try:
      D[-2][1] = 1
      D[-2][2] = 1
    except:
      pass

    backlinks = {}

    for i in range(len(source_sentences)):
        for j in range(len(target_sentences)):
            m = []
            for a in alignment_types:
                k = D[-(1 + a[0])][j + 2 - a[1]]
                if k > 0:
                    p = k * \
                      align_probability(i, j, source_sentences, target_sentences, a, params)
                    m.append((p, a))

            if len(m) > 0:
                v = max(m)
                backlinks[(i, j)] = v[1]
                D[-1].append(v[0])
            else:
                backlinks[(i, j)] = (1, 1)
                D[-1].append(0)

        D.pop(0)
        D.append([0, 0])

    return trace(backlinks, source_sentences, target_sentences)

Response: Creates the sentence alignment of two blocks of texts.


Instruction:
Create a Python 3 function to
create a list of sentences that aligns two texts.

Response:
def align_texts(source_blocks, target_blocks, params = LanguageIndependent):
    """Creates the sentence alignment of two texts.

    Texts can consist of several blocks. Block boundaries cannot be crossed by sentence
    alignment links.

    Each block consists of a list that contains the lengths (in characters) of the sentences
    in this block.

    @param source_blocks: The list of blocks in the source text.
    @param target_blocks: The list of blocks in the target text.
    @param params: the sentence alignment parameters.

    @returns: A list of sentence alignment lists
    """
    if len(source_blocks) != len(target_blocks):
        raise ValueError("Source and target texts do not have the same number of blocks.")

    return [align_blocks(source_block, target_block, params)
            for source_block, target_block in zip(source_blocks, target_blocks)]


Instruction:
Given the following Python 3 function, write the documentation
def split_at(it, split_value):
    def _chunk_iterator(first):
        v = first
        while v != split_value:
            yield v
            v = next(it)

    while True:
        yield _chunk_iterator(next(it))

Response: Splits an iterator C { it } at values of split_value.


Instruction:
Can you write a function in Python 3 where it
parses a stream of tokens and splits it into sentences and blocks.

Response:
def parse_token_stream(stream, soft_delimiter, hard_delimiter):
    """Parses a stream of tokens and splits it into sentences (using C{soft_delimiter} tokens)
    and blocks (using C{hard_delimiter} tokens) for use with the L{align_texts} function.
    """
    return [
        [sum(len(token) for token in sentence_it)
         for sentence_it in split_at(block_it, soft_delimiter)]
        for block_it in split_at(stream, hard_delimiter)]


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_descriptors_from_module(mdl, submodule=False):
    warnings.warn("use get_descriptors_in_module", DeprecationWarning)
    __all__ = getattr(mdl, "__all__", None)
    if __all__ is None:
        __all__ = dir(mdl)

    all_functions = (getattr(mdl, name) for name in __all__ if name[:1] != "_")

    if submodule:
        descs = [
            d
            for fn in all_functions
            if is_descriptor_class(fn) or isinstance(fn, ModuleType)
            for d in (
                [fn] if is_descriptor_class(fn)
                else get_descriptors_from_module(fn, submodule=True)
            )
        ]
    else:
        descs = [
            fn
            for fn in all_functions
            if is_descriptor_class(fn)
        ]

    return descs

Response: r Get all the descriptors from a module.


Instruction:
How would you explain what the following Python 3 function does
def get_descriptors_in_module(mdl, submodule=True):
    __all__ = getattr(mdl, "__all__", None)
    if __all__ is None:
        __all__ = dir(mdl)

    all_values = (getattr(mdl, name) for name in __all__ if name[:1] != "_")

    if submodule:
        for v in all_values:
            if is_descriptor_class(v):
                yield v
            if isinstance(v, ModuleType):
                for v in get_descriptors_in_module(v, submodule=True):
                    yield v

    else:
        for v in all_values:
            if is_descriptor_class(v):
                yield v

Response: r Returns an iterator over all the descriptors in a module.


Instruction:
How would you implement a function in Python 3 that
registers Descriptors from json object objects.

Response:
def register_json(self, obj):
        """Register Descriptors from json descriptor objects.

        Parameters:
            obj(list or dict): descriptors to register

        """
        if not isinstance(obj, list):
            obj = [obj]

        self.register(Descriptor.from_json(j) for j in obj)


Instruction:
Explain what the following Python 3 code does
def register(self, desc, version=None, ignore_3D=False):
        if version is None:
            version = __version__

        version = StrictVersion(version)
        return self._register(desc, version, ignore_3D)

Response: r Register a new object in the cache.


Instruction:
Here you have a function in Python 3, explain what it does
def echo(self, s, file=sys.stdout, end="\n"):
        p = getattr(self, "_progress_bar", None)
        if p is not None:
            p.write(s, file=file, end="\n")
            return

        print(s, file=file, end="\n")

Response: Output a string to file


Instruction:
Given the following Python 3 function, write the documentation
def pandas(self, mols, nproc=None, nmols=None, quiet=False, ipynb=False, id=-1):
        from .pandas_module import MordredDataFrame, Series

        if isinstance(mols, Series):
            index = mols.index
        else:
            index = None

        return MordredDataFrame(
            (list(r) for r in self.map(mols, nproc, nmols, quiet, ipynb, id)),
            columns=[str(d) for d in self.descriptors],
            index=index,
        )

Response: r Returns a pandas DataFrame with the descriptors of the given MOCs.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def is_descriptor_class(desc, include_abstract=False):
    return (
        isinstance(desc, type)
        and issubclass(desc, Descriptor)
        and (True if include_abstract else not inspect.isabstract(desc))
    )

Response: r Check calculatable descriptor class or not.


Instruction:
Can you create a Python 3 function that
converts to json serializable dictionary.

Response:
def to_json(self):
        """Convert to json serializable dictionary.

        Returns:
            dict: dictionary of descriptor

        """
        d, ps = self._to_json()
        if len(ps) == 0:
            return {"name": d}
        else:
            return {"name": d, "args": ps}


Instruction:
Can you generate the documentation for the following Python 3 function
def coord(self):
        if not self.require_3D:
            self.fail(AttributeError("use 3D coordinate in 2D descriptor"))

        return self._context.get_coord(self)

Response:
Get 3D coordinate.

        Returns:
            numpy.array[3, N]: coordinate matrix


Instruction:
How would you explain what the following Python 3 function does
def rethrow_zerodiv(self):
        with np.errstate(divide="raise", invalid="raise"):
            try:
                yield
            except (FloatingPointError, ZeroDivisionError) as e:
                self.fail(ZeroDivisionError(*e.args))

Response: Raises ZeroDivisionError and raises a Failure.


Instruction:
Make a summary of the following Python 3 code
def atomic_sa(self, i):
        sa = 4.0 * np.pi * self.rads2[i]

        neighbors = self.neighbors.get(i)

        if neighbors is None:
            return sa

        XYZi = self.xyzs[i, np.newaxis].T

        sphere = self.sphere * self.rads[i] + XYZi
        N = sphere.shape[1]

        for j, _ in neighbors:
            XYZj = self.xyzs[j, np.newaxis].T

            d2 = (sphere - XYZj) ** 2
            mask = (d2[0] + d2[1] + d2[2]) > self.rads2[j]
            sphere = np.compress(mask, sphere, axis=1)

        return sa * sphere.shape[1] / N

Response: r Returns the atomic surface area of the atom with the given index.


Instruction:
Here you have a function in Python 3, explain what it does
def surface_area(self):
        return [self.atomic_sa(i) for i in range(len(self.rads))]

Response: rCalculate all atomic surface area.


Instruction:
Here you have a function in Python 3, explain what it does
def from_mol(cls, mol, conformer=-1, solvent_radius=1.4, level=4):
        rs = atoms_to_numpy(lambda a: vdw_radii[a.GetAtomicNum()] + solvent_radius, mol)

        conf = mol.GetConformer(conformer)

        ps = np.array([list(conf.GetAtomPosition(i)) for i in range(mol.GetNumAtoms())])

        return cls(rs, ps, level)

Response: rConstruct a SurfaceArea from rdkit Mol type.


Instruction:
Can you create a Python 3 function that
creates a Descriptor instance from a json dict.

Response:
def _Descriptor_from_json(self, obj):
    """Create Descriptor instance from json dict.

    Parameters:
        obj(dict): descriptor dict

    Returns:
        Descriptor: descriptor

    """
    descs = getattr(self, "_all_descriptors", None)

    if descs is None:
        from mordred import descriptors
        descs = {
            cls.__name__: cls
            for cls in get_descriptors_in_module(descriptors)
        }
        descs[ConstDescriptor.__name__] = ConstDescriptor
        self._all_descriptors = descs

    return _from_json(obj, descs)


Instruction:
Can you generate the documentation for the following Python 3 function
def fill_missing(self, value=np.nan):
        return self.__class__(
            self.mol,
            [(value if is_missing(v) else v) for v in self.values()],
            self.keys(),
        )

Response: Returns a new object with the missing values replaced.


Instruction:
Given the following Python 3 function, write the documentation
def drop_missing(self):
        newvalues = []
        newdescs = []
        for d, v in self.items():
            if not is_missing(v):
                newvalues.append(v)
                newdescs.append(d)

        return self.__class__(self.mol, newvalues, newdescs)

Response: r Delete missing value.


Instruction:
Write a Python 3 function for
getting items. Returns a Iterable of tuples.

Response:
def items(self):
        r"""Get items.

        Returns:
            Iterable[(Descriptor, value)]

        """
        return ((k, v) for k, v in zip(self.keys(), self.values()))


Instruction:
How would you code a function in Python 3 to
convert Result to dict.

Response:
def asdict(self, rawkey=False):
        r"""Convert Result to dict.

        Parameters:
            rawkey(bool):
                * True: dict key is Descriptor instance
                * False: dict key is str

        Returns:
            dict

        """
        if rawkey:
            return dict(self.items())
        else:
            return {
                str(k): v
                for k, v in self.items()
            }


Instruction:
Given the following Python 3 function, write the documentation
def log_calls(func):
  '''Decorator to log function calls.'''
  def wrapper(*args, **kargs):
    callStr = "%s(%s)" % (func.__name__, ", ".join([repr(p) for p in args] + ["%s=%s" % (k, repr(v)) for (k, v) in list(kargs.items())]))
    debug(">> %s", callStr)
    ret = func(*args, **kargs)
    debug("<< %s: %s", callStr, repr(ret))
    return ret
  return wrapper

Response: Decorator to log function calls.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def synchronized(func):
  '''Decorator to synchronize function.'''
  func.__lock__ = threading.Lock()
  def synced_func(*args, **kargs):
    with func.__lock__:
      return func(*args, **kargs)
  return synced_func

Response: Decorator to synchronize function.


Instruction:
How would you code a function in Python 3 to
show current progress message to stderr.

Response:
def progress(msg, *args):
  '''Show current progress message to stderr.
     This function will remember the previous message so that next time,
     it will clear the previous message before showing next one.
  '''
  # Don't show any progress if the output is directed to a file.
  if not (sys.stdout.isatty() and sys.stderr.isatty()):
    return

  text = (msg % args)
  if progress.prev_message:
    sys.stderr.write(' ' * len(progress.prev_message) + '\r')
  sys.stderr.write(text + '\r')
  progress.prev_message = text


Instruction:
How would you explain what the following Python 3 function does
def fail(message, exc_info=None, status=1, stacktrace=False):
  '''Utility function to handle runtime failures gracefully.
     Show concise information if possible, then terminate program.
  '''
  text = message
  if exc_info:
    text += str(exc_info)
  error(text)
  if stacktrace:
    error(traceback.format_exc())
  clean_tempfiles()
  if __name__ == '__main__':
    sys.exit(status)
  else:
    raise RuntimeError(status)

Response: Utility function to handle runtime failures gracefully.


Instruction:
Make a summary of the following Python 3 code
def tempfile_get(target):
  '''Get a temp filename for atomic download.'''
  fn = '%s-%s.tmp' % (target, ''.join(random.Random().sample("0123456789abcdefghijklmnopqrstuvwxyz", 15)))
  TEMP_FILES.add(fn)
  return fn

Response: Get a temp filename for atomic download.


Instruction:
Given the following Python 3 function, write the documentation
def clean_tempfiles():
  '''Clean up temp files'''
  for fn in TEMP_FILES:
    if os.path.exists(fn):
      os.unlink(fn)

Response: Clean up temp files


Instruction:
Here you have a function in Python 3, explain what it does
def get_loggers(self):
    '''Return a list of the logger methods: (debug, info, warn, error)'''

    return self.log.debug, self.log.info, self.log.warn, self.log.error

Response: Return a list of the logger methods


Instruction:
Can you write a function in Python 3 where it
gets the fixed part of the path without wildcard

Response:
def get_fixed_path(self):
    '''Get the fixed part of the path without wildcard'''
    pi = self.path.split(PATH_SEP)
    fi = []
    for p in pi:
      if '*' in p or '?' in p:
        break
      fi.append(p)
    return PATH_SEP.join(fi)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_legal_params(self, method):
    '''Given a API name, list all legal parameters using boto3 service model.'''
    if method not in self.client.meta.method_to_api_mapping:
      # Injected methods. Ignore.
      return []
    api = self.client.meta.method_to_api_mapping[method]
    shape = self.client.meta.service_model.operation_model(api).input_shape
    if shape is None:
      # No params needed for this API.
      return []
    return shape.members.keys()

Response: Given a API name list all legal parameters using boto3 service model.


Instruction:
Implement a function in Python 3 to
combine existing parameters with extra options supplied from command line options. Carefully merge special type of parameter if needed.

Response:
def merge_opt_params(self, method, kargs):
    '''Combine existing parameters with extra options supplied from command line
       options. Carefully merge special type of parameter if needed.
    '''
    for key in self.legal_params[method]:
      if not hasattr(self.opt, key) or getattr(self.opt, key) is None:
        continue
      if key in kargs and type(kargs[key]) == dict:
        assert(type(getattr(self.opt, key)) == dict)
        # Merge two dictionaries.
        for k, v in getattr(self.opt, key).iteritems():
          kargs[key][k] = v
      else:
        # Overwrite values.
        kargs[key] = getattr(self.opt, key)

    return kargs


Instruction:
Here you have a function in Python 3, explain what it does
def add_options(parser):
    '''Add the whole list of API parameters into optparse.'''
    for param, param_type, param_doc in BotoClient.EXTRA_CLIENT_PARAMS:
      parser.add_option('--API-' + param, help=param_doc, type=param_type, dest=param)

Response: Add the whole list of API parameters into optparse.


Instruction:
How would you explain what the following Python 3 function does
def join(self):
    '''Override original join() with a timeout and handle keyboard interrupt.'''
    self.all_tasks_done.acquire()
    try:
      while self.unfinished_tasks:
        self.all_tasks_done.wait(1000)

        # Child thread has exceptions, fail main thread too.
        if self.exc_info:
          fail('[Thread Failure] ', exc_info=self.exc_info)
    except KeyboardInterrupt:
      raise Failure('Interrupted by user')
    finally:
      self.all_tasks_done.release()

Response: Override original join with a timeout and handle keyboard interrupt.


Instruction:
Make a summary of the following Python 3 code
def terminate(self, exc_info=None):
    '''Terminate all threads by deleting the queue and forcing the child threads
       to quit.
    '''
    if exc_info:
      self.exc_info = exc_info
    try:
      while self.get_nowait():
        self.task_done()
    except Queue.Empty:
      pass

Response:
Terminate all threads by deleting the queue and forcing the child threads
       to quit.


Instruction:
Make a summary of the following Python 3 code
def add_task(self, func_name, *args, **kargs):
    '''Utility function to add a single task into task queue'''
    self.tasks.put((func_name, 0, args, kargs))

Response: Utility function to add a single task into task queue


Instruction:
Can you generate a brief explanation for the following Python 3 code
def join(self):
    '''Utility function to wait all tasks to complete'''
    self.tasks.join()

    # Force each thread to break loop.
    for worker in self.workers:
      self.tasks.put(None)

    # Wait for all thread to terminate.
    for worker in self.workers:
      worker.join()
      worker.s3 = None

Response: Utility function to wait all tasks to complete


Instruction:
Can you generate the documentation for the following Python 3 function
def processed(self):
    '''Increase the processed task counter and show progress message'''
    self.processed_tasks += 1
    qsize = self.tasks.qsize()
    if qsize > 0:
      progress('[%d task(s) completed, %d remaining, %d thread(s)]', self.processed_tasks, qsize, len(self.workers))
    else:
      progress('[%d task(s) completed, %d thread(s)]', self.processed_tasks, len(self.workers))

Response: Increase the processed task counter and show progress message


Instruction:
Given the following Python 3 function, write the documentation
def s3_keys_from_env():
    '''Retrieve S3 access keys from the environment, or None if not present.'''
    env = os.environ
    if S3_ACCESS_KEY_NAME in env and S3_SECRET_KEY_NAME in env:
      keys = (env[S3_ACCESS_KEY_NAME], env[S3_SECRET_KEY_NAME])
      debug("read S3 keys from environment")
      return keys
    else:
      return None

Response: Retrieve S3 access keys from the environment or None if not present.


Instruction:
Write a Python 3 script to
retrieve S3 access keys from the commandline or None if not present.

Response:
def s3_keys_from_cmdline(opt):
    '''Retrieve S3 access keys from the command line, or None if not present.'''
    if opt.access_key != None and opt.secret_key != None:
      keys = (opt.access_key, opt.secret_key)
      debug("read S3 keys from commandline")
      return keys
    else:
      return None


Instruction:
How would you explain what the following Python 3 function does
def s3_keys_from_s3cfg(opt):
    '''Retrieve S3 access key settings from s3cmd's config file, if present; otherwise return None.'''
    try:
      if opt.s3cfg != None:
        s3cfg_path = "%s" % opt.s3cfg
      else:
        s3cfg_path = "%s/.s3cfg" % os.environ["HOME"]
      if not os.path.exists(s3cfg_path):
        return None
      config = ConfigParser.ConfigParser()
      config.read(s3cfg_path)
      keys = config.get("default", "access_key"), config.get("default", "secret_key")
      debug("read S3 keys from %s file", s3cfg_path)
      return keys
    except Exception as e:
      info("could not read S3 keys from %s file; skipping (%s)", s3cfg_path, e)
      return None

Response: Retrieve S3 access key settings from s3cmd s config file if present ; otherwise return None.


Instruction:
Write a Python 3 script for
initializing s3 access keys from command line or config file.

Response:
def init_s3_keys(opt):
    '''Initialize s3 access keys from environment variable or s3cfg config file.'''
    S3Handler.S3_KEYS = S3Handler.s3_keys_from_cmdline(opt) or S3Handler.s3_keys_from_env() \
                        or S3Handler.s3_keys_from_s3cfg(opt)


Instruction:
Can you generate the documentation for the following Python 3 function
def connect(self):
    '''Connect to S3 storage'''
    try:
      if S3Handler.S3_KEYS:
        self.s3 = BotoClient(self.opt, S3Handler.S3_KEYS[0], S3Handler.S3_KEYS[1])
      else:
        self.s3 = BotoClient(self.opt)
    except Exception as e:
      raise RetryFailure('Unable to connect to s3: %s' % e)

Response: Connect to S3 storage


Instruction:
Make a summary of the following Python 3 code
def s3walk(self, basedir, show_dir=None):
    '''Walk through a S3 directory. This function initiate a walk with a basedir.
       It also supports multiple wildcards.
    '''
    # Provide the default value from command line if no override.
    if not show_dir:
      show_dir = self.opt.show_dir

    # trailing slash normalization, this is for the reason that we want
    # ls 's3://foo/bar/' has the same result as 's3://foo/bar'. Since we
    # call partial_match() to check wildcards, we need to ensure the number
    # of slashes stays the same when we do this.
    if basedir[-1] == PATH_SEP:
      basedir = basedir[0:-1]

    s3url = S3URL(basedir)
    result = []

    pool = ThreadPool(ThreadUtil, self.opt)
    pool.s3walk(s3url, s3url.get_fixed_path(), s3url.path, result)
    pool.join()

    # automatic directory detection
    if not show_dir and len(result) == 1 and result[0]['is_dir']:
      path = result[0]['name']
      s3url = S3URL(path)
      result = []
      pool = ThreadPool(ThreadUtil, self.opt)
      pool.s3walk(s3url, s3url.get_fixed_path(), s3url.path, result)
      pool.join()

    def compare(x, y):
      '''Comparator for ls output'''
      result = -cmp(x['is_dir'], y['is_dir'])
      if result != 0:
        return result
      return cmp(x['name'], y['name'])
    return sorted(result, key=cmp_to_key(compare))

Response: This function initiate a walk with a basedir. It also supports multiple wildcards.


Instruction:
How would you implement a function in Python 3 that
walks through local directories from root basedir and return a list of all local files.

Response:
def local_walk(self, basedir):
    '''Walk through local directories from root basedir'''
    result = []

    for root, dirs, files in os.walk(basedir):
      for f in files:
        result.append(os.path.join(root, f))
    return result


Instruction:
Can you generate a brief explanation for the following Python 3 code
def source_expand(self, source):
    '''Expand the wildcards for an S3 path. This emulates the shall expansion
       for wildcards if the input is local path.
    '''
    result = []

    if not isinstance(source, list):
      source = [source]

    for src in source:
      # XXX Hacky: We need to disable recursive when we expand the input
      #            parameters, need to pass this as an override parameter if
      #            provided.
      tmp = self.opt.recursive
      self.opt.recursive = False
      result += [f['name'] for f in self.s3walk(src, True)]
      self.opt.recursive = tmp

    if (len(result) == 0) and (not self.opt.ignore_empty_source):
      fail("[Runtime Failure] Source doesn't exist.")

    return result

Response:
Expand the wildcards for an S3 path. This emulates the shall expansion
       for wildcards if the input is local path.


Instruction:
Explain what the following Python 3 code does
def put_single_file(self, pool, source, target):
    '''Upload a single file or a directory by adding a task into queue'''
    if os.path.isdir(source):
      if self.opt.recursive:
        for f in (f for f in self.local_walk(source) if not os.path.isdir(f)):
          target_url = S3URL(target)
          # deal with ./ or ../ here by normalizing the path.
          joined_path = os.path.normpath(os.path.join(target_url.path, os.path.relpath(f, source)))
          pool.upload(f, S3URL.combine('s3', target_url.bucket, joined_path))
      else:
        message('omitting directory "%s".' % source)
    else:
      pool.upload(source, target)

Response: Upload a single file or a directory by adding a task into queue


Instruction:
Write a Python 3 function for
uploading files to S3.

Response:
def put_files(self, source, target):
    '''Upload files to S3.
       This function can handle multiple file upload if source is a list.
       Also, it works for recursive mode which copy all files and keep the
       directory structure under the given source directory.
    '''
    pool = ThreadPool(ThreadUtil, self.opt)
    if not isinstance(source, list):
      source = [source]

    if target[-1] == PATH_SEP:
      for src in source:
        self.put_single_file(pool, src, os.path.join(target, self.get_basename(src)))
    else:
      if len(source) == 1:
        self.put_single_file(pool, source[0], target)
      else:
        raise Failure('Target "%s" is not a directory (with a trailing slash).' % target)

    pool.join()


Instruction:
Can you tell what is the following Python 3 function doing
def create_bucket(self, source):
    '''Use the create_bucket API to create a new bucket'''
    s3url = S3URL(source)

    message('Creating %s', source)
    if not self.opt.dry_run:
      resp = self.s3.create_bucket(Bucket=s3url.bucket)
      if resp['ResponseMetadata']["HTTPStatusCode"] == 200:
        message('Done.')
      else:
        raise Failure('Unable to create bucket %s' % source)

Response: Use the create_bucket API to create a new bucket


Instruction:
Given the following Python 3 function, write the documentation
def update_privilege(self, obj, target):
    '''Get privileges from metadata of the source in s3, and apply them to target'''
    if 'privilege' in obj['Metadata']:
      os.chmod(target, int(obj['Metadata']['privilege'], 8))

Response: Update the privileges of the source in s3 to target


Instruction:
Can you generate the documentation for the following Python 3 function
def print_files(self, source):
    '''Print out a series of files'''
    sources = self.source_expand(source)

    for source in sources:
      s3url = S3URL(source)
      response = self.s3.get_object(Bucket=s3url.bucket, Key=s3url.path)
      message('%s', response['Body'].read())

Response: Print out a series of files


Instruction:
Write a Python 3 script for
downloading a single file or a directory by adding a task into queue

Response:
def get_single_file(self, pool, source, target):
    '''Download a single file or a directory by adding a task into queue'''
    if source[-1] == PATH_SEP:
      if self.opt.recursive:
        basepath = S3URL(source).path
        for f in (f for f in self.s3walk(source) if not f['is_dir']):
          pool.download(f['name'], os.path.join(target, os.path.relpath(S3URL(f['name']).path, basepath)))
      else:
        message('omitting directory "%s".' % source)
    else:
      pool.download(source, target)


Instruction:
How would you code a function in Python 3 to
download files. This function can handle multiple files if source S3 URL has wildcard characters. It also handles recursive mode by download all files and keep the directory structure.

Response:
def get_files(self, source, target):
    '''Download files.
       This function can handle multiple files if source S3 URL has wildcard
       characters. It also handles recursive mode by download all files and
       keep the directory structure.
    '''
    pool = ThreadPool(ThreadUtil, self.opt)
    source = self.source_expand(source)

    if os.path.isdir(target):
      for src in source:
        self.get_single_file(pool, src, os.path.join(target, self.get_basename(S3URL(src).path)))
    else:
      if len(source) > 1:
        raise Failure('Target "%s" is not a directory.' % target)
        # Get file if it exists on s3 otherwise do nothing
      elif len(source) == 1:
        self.get_single_file(pool, source[0], target)
      else:
        #Source expand may return empty list only if ignore-empty-source is set to true
        pass

    pool.join()


Instruction:
Implement a Python 3 function for
removing remote files that are not present in the local source.

Response:
def delete_removed_files(self, source, target):
    '''Remove remote files that are not present in the local source.
       (Obsolete) It is used for old sync command now.
    '''
    message("Deleting files found in %s and not in %s", source, target)
    if os.path.isdir(source):
      unecessary = []
      basepath = S3URL(target).path
      for f in [f for f in self.s3walk(target) if not f['is_dir']]:
        local_name = os.path.join(source, os.path.relpath(S3URL(f['name']).path, basepath))
        if not os.path.isfile(local_name):
          message("%s not found locally, adding to delete queue", local_name)
          unecessary.append(f['name'])
      if len(unecessary) > 0:
        pool = ThreadPool(ThreadUtil, self.opt)
        for del_file in unecessary:
          pool.delete(del_file)
        pool.join()
    else:
      raise Failure('Source "%s" is not a directory.' % target)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def cp_single_file(self, pool, source, target, delete_source):
    '''Copy a single file or a directory by adding a task into queue'''
    if source[-1] == PATH_SEP:
      if self.opt.recursive:
        basepath = S3URL(source).path
        for f in (f for f in self.s3walk(source) if not f['is_dir']):
          pool.copy(f['name'], os.path.join(target, os.path.relpath(S3URL(f['name']).path, basepath)), delete_source=delete_source)
      else:
        message('omitting directory "%s".' % source)
    else:
      pool.copy(source, target, delete_source=delete_source)

Response: Copy a single file or a directory by adding a task into queue


Instruction:
Can you generate the documentation for the following Python 3 function
def cp_files(self, source, target, delete_source=False):
    '''Copy files
       This function can handle multiple files if source S3 URL has wildcard
       characters. It also handles recursive mode by copying all files and
       keep the directory structure.
    '''
    pool = ThreadPool(ThreadUtil, self.opt)
    source = self.source_expand(source)

    if target[-1] == PATH_SEP:
      for src in source:
        self.cp_single_file(pool, src, os.path.join(target, self.get_basename(S3URL(src).path)), delete_source)
    else:
      if len(source) > 1:
        raise Failure('Target "%s" is not a directory (with a trailing slash).' % target)
        # Copy file if it exists otherwise do nothing
      elif len(source) == 1:
        self.cp_single_file(pool, source[0], target, delete_source)
      else:
        # Source expand may return empty list only if ignore-empty-source is set to true
        pass

    pool.join()

Response: Copy files from source to target.


Instruction:
Can you generate the documentation for the following Python 3 function
def del_files(self, source):
    '''Delete files on S3'''
    src_files = []
    for obj in self.s3walk(source):
      if not obj['is_dir']: # ignore directories
        src_files.append(obj['name'])

    pool = ThreadPool(ThreadUtil, self.opt)
    pool.batch_delete(src_files)
    pool.join()

Response: Delete files on S3


Instruction:
Implement a function in Python 3 to
get the size component of the given s3url.

Response:
def size(self, source):
    '''Get the size component of the given s3url. If it is a
       directory, combine the sizes of all the files under
       that directory. Subdirectories will not be counted unless
       --recursive option is set.
    '''
    result = []
    for src in self.source_expand(source):
      size = 0
      for f in self.s3walk(src):
        size += f['size']
      result.append((src, size))

    return result


Instruction:
How would you code a function in Python 3 to
calculate MD5 hash code for a local file

Response:
def file_hash(self, filename, block_size=2**20):
    '''Calculate MD5 hash code for a local file'''
    m = hashlib.md5()
    with open(filename, 'rb') as f:
      while True:
        data = f.read(block_size)
        if not data:
          break
        m.update(data)
    return m.hexdigest()


Instruction:
How would you explain what the following Python 3 function does
def get_md5(self):
    '''Get or calculate MD5 value of the local file.'''
    if self.md5 is None:
      self.md5 = self.file_hash(self.filename)
    return self.md5

Response: Get or calculate MD5 value of the local file.


Instruction:
Implement a Python 3 function for
ensuring all directories are created for a given target file.

Response:
def mkdirs(self, target):
    '''Ensure all directories are created for a given target file.'''
    path = os.path.dirname(target)
    if path and path != PATH_SEP and not os.path.isdir(path):
      # Multi-threading means there will be intervleaved execution
      # between the check and creation of the directory.
      try:
        os.makedirs(path)
      except OSError as ose:
        if ose.errno != errno.EEXIST:
          raise Failure('Unable to create directory (%s)' % (path,))


Instruction:
Implement a function in Python 3 to
check MD5 for a local file and a remote file.

Response:
def sync_check(self, md5cache, remoteKey):
    '''Check MD5 for a local file and a remote file.
       Return True if they have the same md5 hash, otherwise False.
    '''
    if not remoteKey:
      return False
    if not os.path.exists(md5cache.filename):
      return False
    localmd5 = md5cache.get_md5()

    # check multiple md5 locations
    return ('ETag' in remoteKey and remoteKey['ETag'] == '"%s"' % localmd5) or \
           ('md5' in remoteKey and remoteKey['md5'] == localmd5) or \
           ('md5' in remoteKey['Metadata'] and remoteKey['Metadata']['md5'] == localmd5)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def s3walk(self, s3url, s3dir, filter_path, result):
    '''Thread worker for s3walk.
       Recursively walk into all subdirectories if they still match the filter
       path partially.
    '''

    paginator = self.s3.get_paginator('list_objects')
    filter_path_level = filter_path.count(PATH_SEP)

    for page in paginator.paginate(Bucket=s3url.bucket, Prefix=s3dir, Delimiter=PATH_SEP, PaginationConfig={'PageSize': 1000}):
      # Get subdirectories first.
      for obj in page.get('CommonPrefixes') or []:
        obj_name = obj['Prefix']

        if not self.partial_match(obj_name, filter_path):
          continue

        if self.opt.recursive or (obj_name.count(PATH_SEP) != filter_path_level + 1):
          self.pool.s3walk(s3url, obj_name, filter_path, result)
        else:
          self.conditional(result, {
            'name': S3URL.combine(s3url.proto, s3url.bucket, obj_name),
            'is_dir': True,
            'size': 0,
            'last_modified': None
          })

      # Then get all items in this folder.
      for obj in page.get('Contents') or []:
        obj_name = obj['Key']
        if not self.partial_match(obj_name, filter_path):
          continue

        if self.opt.recursive or obj_name.count(PATH_SEP) == filter_path_level:
          self.conditional(result, {
            'name': S3URL.combine(s3url.proto, s3url.bucket, obj_name),
            'is_dir': False,
            'size': obj['Size'],
            'last_modified': obj['LastModified']
          })

Response: Thread worker for s3walk


Instruction:
Can you create a Python 3 function that
checks all file item with given conditions.

Response:
def conditional(self, result, obj):
    '''Check all file item with given conditions.'''
    fileonly = (self.opt.last_modified_before is not None) or (self.opt.last_modified_after is not None)

    if obj['is_dir']:
      if not fileonly:
        result.append(obj)
      return

    if (self.opt.last_modified_before is not None) and obj['last_modified'] >= self.opt.last_modified_before:
      return

    if (self.opt.last_modified_after is not None) and obj['last_modified'] <= self.opt.last_modified_after:
      return

    result.append(obj)


Instruction:
Can you generate the documentation for the following Python 3 function
def get_file_splits(self, id, source, target, fsize, splitsize):
    '''Get file splits for upload/download/copy operation.'''
    pos = 0
    part = 1 # S3 part id starts from 1
    mpi = ThreadUtil.MultipartItem(id)
    splits = []

    while pos < fsize:
      chunk = min(splitsize, fsize - pos)
      assert(chunk > 0)
      splits.append((source, target, mpi, pos, chunk, part))
      part += 1
      pos += chunk
    mpi.total = len(splits)

    return splits

Response: Get file splits for upload or download operation.


Instruction:
Write a Python 3 function for
getting privileges of a local file

Response:
def get_file_privilege(self, source):
    '''Get privileges of a local file'''
    try:
      return str(oct(os.stat(source).st_mode)[-3:])
    except Exception as e:
      raise Failure('Could not get stat for %s, error_message = %s', source, e)


Instruction:
Can you generate the documentation for the following Python 3 function
def lookup(self, s3url):
    '''Get the s3 object with the S3 URL. Return None if not exist.'''
    try:
      return self.s3.head_object(Bucket=s3url.bucket, Key=s3url.path)
    except BotoClient.ClientError as e:
      if e.response['ResponseMetadata']['HTTPStatusCode'] == 404:
        return None
      else:
        raise e

Response: Get the s3 object with the S3 URL. Return None if not exist.


Instruction:
Given the following Python 3 function, write the documentation
def read_file_chunk(self, source, pos, chunk):
    '''Read local file chunk'''
    if chunk==0:
        return StringIO()
    data = None
    with open(source, 'rb') as f:
      f.seek(pos)
      data = f.read(chunk)
    if not data:
      raise Failure('Unable to read data from source: %s' % source)
    return StringIO(data)

Response: Read a chunk of data from a local file


Instruction:
How would you explain what the following Python 3 function does
def upload(self, source, target, mpi=None, pos=0, chunk=0, part=0):
    '''Thread worker for upload operation.'''
    s3url = S3URL(target)
    obj = self.lookup(s3url)

    # Initialization: Set up multithreaded uploads.
    if not mpi:
      fsize = os.path.getsize(source)
      md5cache = LocalMD5Cache(source)

      # optional checks
      if self.opt.dry_run:
        message('%s => %s', source, target)
        return
      elif self.opt.sync_check and self.sync_check(md5cache, obj):
        message('%s => %s (synced)', source, target)
        return
      elif not self.opt.force and obj:
        raise Failure('File already exists: %s' % target)

      if fsize < self.opt.max_singlepart_upload_size:
        data = self.read_file_chunk(source, 0, fsize)
        self.s3.put_object(Bucket=s3url.bucket,
                           Key=s3url.path,
                           Body=data,
                           Metadata={'md5': md5cache.get_md5(),
                                     'privilege': self.get_file_privilege(source)})
        message('%s => %s', source, target)
        return

      # Here we need to have our own md5 value because multipart upload calculates
      # different md5 values.
      response = self.s3.create_multipart_upload(Bucket=s3url.bucket,
                                                 Key=s3url.path,
                                                 Metadata={'md5': md5cache.get_md5(),
                                                           'privilege': self.get_file_privilege(source)})
      upload_id = response['UploadId']

      for args in self.get_file_splits(upload_id, source, target, fsize, self.opt.multipart_split_size):
        self.pool.upload(*args)
      return

    data = self.read_file_chunk(source, pos, chunk)
    response = self.s3.upload_part(Bucket=s3url.bucket, Key=s3url.path, UploadId=mpi.id, Body=data, PartNumber=part)

    # Finalize
    if mpi.complete({'ETag': response['ETag'], 'PartNumber': part}):
      try:
        self.s3.complete_multipart_upload(Bucket=s3url.bucket, Key=s3url.path, UploadId=mpi.id, MultipartUpload={'Parts': mpi.sorted_parts()})
        message('%s => %s', source, target)
      except Exception as e:
        message('Unable to complete upload: %s', str(e))
        self.s3.abort_multipart_upload(Bucket=s3url.bucket, Key=s3url.path, UploadId=mpi.id)
        raise RetryFailure('Upload failed: Unable to complete upload %s.' % source)

Response: Thread worker for uploading a file.


Instruction:
Explain what the following Python 3 code does
def _verify_file_size(self, obj, downloaded_file):
    '''Verify the file size of the downloaded file.'''
    file_size = os.path.getsize(downloaded_file)
    if int(obj['ContentLength']) != file_size:
      raise RetryFailure('Downloaded file size inconsistent: %s' % (repr(obj)))

Response: Verify the file size of the downloaded file.


Instruction:
Create a Python 3 function for
writing a chunk of data to a local file.

Response:
def write_file_chunk(self, target, pos, chunk, body):
    '''Write local file chunk'''
    fd = os.open(target, os.O_CREAT | os.O_WRONLY)
    try:
      os.lseek(fd, pos, os.SEEK_SET)
      data = body.read(chunk)
      num_bytes_written = os.write(fd, data)
      if(num_bytes_written != len(data)):
        raise RetryFailure('Number of bytes written inconsistent: %s != %s' % (num_bytes_written, sys.getsizeof(data)))
    finally:
      os.close(fd)


Instruction:
Here you have a function in Python 3, explain what it does
def download(self, source, target, mpi=None, pos=0, chunk=0, part=0):
    '''Thread worker for download operation.'''
    s3url = S3URL(source)
    obj = self.lookup(s3url)
    if obj is None:
      raise Failure('The obj "%s" does not exists.' % (s3url.path,))

    # Initialization: Set up multithreaded downloads.
    if not mpi:
      # optional checks
      if self.opt.dry_run:
        message('%s => %s', source, target)
        return
      elif self.opt.sync_check and self.sync_check(LocalMD5Cache(target), obj):
        message('%s => %s (synced)', source, target)
        return
      elif not self.opt.force and os.path.exists(target):
        raise Failure('File already exists: %s' % target)

      fsize = int(obj['ContentLength'])

      # Small file optimization.
      if fsize < self.opt.max_singlepart_download_size:
        # Create a single part to chain back main download operation.
        mpi = ThreadUtil.MultipartItem(tempfile_get(target))
        mpi.total = 1
        pos = 0
        chunk = fsize
        # Continue as one part download.
      else:
        # Here we use temp filename as the id of mpi.
        for args in self.get_file_splits(tempfile_get(target), source, target, fsize, self.opt.multipart_split_size):
          self.pool.download(*args)
        return

    tempfile = mpi.id
    if self.opt.recursive:
      self.mkdirs(tempfile)

    # Download part of the file, range is inclusive.
    response = self.s3.get_object(Bucket=s3url.bucket, Key=s3url.path, Range='bytes=%d-%d' % (pos, pos + chunk - 1))
    self.write_file_chunk(tempfile, pos, chunk, response['Body'])

    # Finalize
    if mpi.complete({'PartNumber': part}):
      try:
        self.update_privilege(obj, tempfile)
        self._verify_file_size(obj, tempfile)
        tempfile_set(tempfile, target)
        message('%s => %s', source, target)
      except Exception as e:
        # Note that we don't retry in this case, because
        # We are going to remove the temp file, and if we
        # retry here with original parameters (wrapped in
        # the task item), it would fail anyway
        tempfile_set(tempfile, None)
        raise Failure('Download Failure: %s, Source: %s.' % (e.message, source))

Response: Thread worker for download operation.


Instruction:
Explain what the following Python 3 code does
def copy(self, source, target, mpi=None, pos=0, chunk=0, part=0, delete_source=False):
    '''Copy a single file from source to target using boto S3 library.'''

    if self.opt.dry_run:
      message('%s => %s' % (source, target))
      return

    source_url = S3URL(source)
    target_url = S3URL(target)

    if not mpi:
      obj = self.lookup(source_url)
      fsize = int(obj['ContentLength'])

      if fsize < self.opt.max_singlepart_copy_size:
        self.s3.copy_object(Bucket=target_url.bucket, Key=target_url.path,
                            CopySource={'Bucket': source_url.bucket, 'Key': source_url.path})

        message('%s => %s' % (source, target))
        if delete_source:
          self.delete(source)

        return

      response = self.s3.create_multipart_upload(Bucket=target_url.bucket,
                                                 Key=target_url.path,
                                                 Metadata=obj['Metadata'])
      upload_id = response['UploadId']

      for args in self.get_file_splits(upload_id, source, target, fsize, self.opt.multipart_split_size):
        self.pool.copy(*args, delete_source=delete_source)
      return

    response = self.s3.upload_part_copy(Bucket=target_url.bucket,
                                        Key=target_url.path,
                                        CopySource={'Bucket': source_url.bucket, 'Key': source_url.path},
                                        CopySourceRange='bytes=%d-%d' % (pos, pos + chunk - 1),
                                        UploadId=mpi.id,
                                        PartNumber=part)

    if mpi.complete({'ETag': response['CopyPartResult']['ETag'], 'PartNumber': part}):
      try:
        # Finalize copy operation.
        self.s3.complete_multipart_upload(Bucket=target_url.bucket, Key=target_url.path, UploadId=mpi.id, MultipartUpload={'Parts': mpi.sorted_parts()})

        if delete_source:
          self.delete(source)

        message('%s => %s' % (source, target))
      except Exception as e:
        message('Unable to complete upload: %s', str(e))
        self.s3.abort_multipart_upload(Bucket=source_url.bucket, Key=source_url.path, UploadId=mpi.id)
        raise RetryFailure('Copy failed: Unable to complete copy %s.' % source)

Response: Copy a single file from source to target using boto S3 library.


Instruction:
How would you code a function in Python 3 to
thread worker for download operation.

Response:
def delete(self, source):
    '''Thread worker for download operation.'''
    s3url = S3URL(source)

    message('Delete %s', source)
    if not self.opt.dry_run:
      self.s3.delete_object(Bucket=s3url.bucket, Key=s3url.path)


Instruction:
Write a Python 3 function that can
delete a list of files in batch of batch_delete_size ( default = 1000.

Response:
def batch_delete(self, sources):
    '''Delete a list of files in batch of batch_delete_size (default=1000).'''
    assert(type(sources) == list)

    if len(sources) == 0:
      return
    elif len(sources) == 1:
      self.delete(sources[0])
    elif len(sources) > self.opt.batch_delete_size:
      for i in range(0, len(sources), self.opt.batch_delete_size):
        self.pool.batch_delete(sources[i:i+self.opt.batch_delete_size])
    else:
      bucket = S3URL(sources[0]).bucket
      deletes = []
      for source in sources:
        s3url = S3URL(source)
        if s3url.bucket != bucket:
          raise Failure('Unable to delete keys in different bucket %s and %s.' % (s3url.bucket, bucket))
        deletes.append({'Key': s3url.path})

      response = self.s3.delete_objects(Bucket=bucket, Delete={'Objects': deletes})

      # Output result of deletion.
      for res in response.get('Deleted') or []:
        message('Delete %s', S3URL.combine('s3', bucket, res['Key']))

      for err in response.get('Errors') or []:
        message('Error deleting %s, code(%s) %s', S3URL.combine('s3', bucket, res['Key']), err['Code'], err['Message'])

      if response.get('Errors') is not None:
        raise RetryFailure('Unable to complete deleting %d files.' % len(response.get('Errors')))


Instruction:
How would you explain what the following Python 3 function does
def run(self, args):
    '''Main entry to handle commands. Dispatch to individual command handler.'''
    if len(args) == 0:
      raise InvalidArgument('No command provided')
    cmd = args[0]
    if cmd + '_handler' in CommandHandler.__dict__:
      CommandHandler.__dict__[cmd + '_handler'](self, args)
    else:
      raise InvalidArgument('Unknown command %s' % cmd)

Response: Main entry to handle commands. Dispatch to individual command handler.


Instruction:
Here you have a function in Python 3, explain what it does
def validate(self, format, args):
    '''Validate input parameters with given format.
       This function also checks for wildcards for recursive mode.
    '''
    fmtMap = {
      'cmd': 'Command',
      's3': 's3 path',
      'local': 'local path'
    }
    fmts = format.split('|')
    if len(fmts) != len(args):
      raise InvalidArgument('Invalid number of parameters')

    for i, fmt in enumerate(fmts):
      valid = False
      for f in fmt.split(','):
        if f == 'cmd' and args[i] + '_handler' in CommandHandler.__dict__:
          valid = True
        if f == 's3' and S3URL.is_valid(args[i]):
          valid = True
        if f == 'local' and not S3URL.is_valid(args[i]):
          valid = True
      if not valid:
        raise InvalidArgument('Invalid parameter: %s, %s expected' % (args[i], fmtMap[fmt.split(',')[0]]))

Response: Validate input parameters with given format.


Instruction:
Given the following Python 3 function, write the documentation
def pretty_print(self, objlist):
    '''Pretty print the result of s3walk. Here we calculate the maximum width
       of each column and align them.
    '''

    def normalize_time(timestamp):
      '''Normalize the timestamp format for pretty print.'''
      if timestamp is None:
        return ' ' * 16

      return TIMESTAMP_FORMAT % (timestamp.year, timestamp.month, timestamp.day, timestamp.hour, timestamp.minute)

    cwidth = [0, 0, 0]
    format = '%%%ds %%%ds %%-%ds'

    # Calculate maximum width for each column.
    result = []
    for obj in objlist:
      last_modified = normalize_time(obj['last_modified'])
      size = str(obj['size']) if not obj['is_dir'] else 'DIR'
      name = obj['name']
      item = (last_modified, size, name)
      for i, value in enumerate(item):
        if cwidth[i] < len(value):
          cwidth[i] = len(value)
      result.append(item)

    # Format output.
    for item in result:
      text = (format % tuple(cwidth)) % item
      message('%s', text.rstrip())

Response: Pretty print the result of s3walk. Here we calculate the maximum width of each column and align them.


Instruction:
Here you have a function in Python 3, explain what it does
def mb_handler(self, args):
    '''Handler for mb command'''
    if len(args) == 1:
      raise InvalidArgument('No s3 bucketname provided')

    self.validate('cmd|s3', args)
    self.s3handler().create_bucket(args[1])

Response: Handler for mb command


Instruction:
How would you explain what the following Python 3 function does
def put_handler(self, args):
    '''Handler for put command'''

    # Special check for shell expansion
    if len(args) < 3:
      raise InvalidArgument('Invalid number of parameters')
    self.validate('|'.join(['cmd'] + ['local'] * (len(args) - 2) + ['s3']), args)

    source = args[1:-1] # shell expansion
    target = args[-1]

    self.s3handler().put_files(source, target)

Response: Handler for put command.


Instruction:
Make a summary of the following Python 3 code
def cat_handler(self, args):
    '''Handler for cat command'''

    self.validate('cmd|s3', args)
    source = args[1]

    self.s3handler().print_files(source)

Response: Handler for cat command


Instruction:
How would you explain what the following Python 3 function does
def dsync_handler(self, args):
    '''Handler for dsync command.'''
    self.opt.recursive = True
    self.opt.sync_check = True
    self.opt.force = True

    self.validate('cmd|s3,local|s3,local', args)
    source = args[1]
    target = args[2]

    self.s3handler().dsync_files(source, target)

Response: Handler for dsync command.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def cp_handler(self, args):
    '''Handler for cp command'''

    self.validate('cmd|s3|s3', args)
    source = args[1]
    target = args[2]
    self.s3handler().cp_files(source, target)

Response: Handler for cp command


Instruction:
Can you tell what is the following Python 3 function doing
def mv_handler(self, args):
    '''Handler for mv command'''

    self.validate('cmd|s3|s3', args)
    source = args[1]
    target = args[2]
    self.s3handler().cp_files(source, target, delete_source=True)

Response: Handler for mv command


Instruction:
Explain what the following Python 3 code does
def del_handler(self, args):
    '''Handler for del command'''
    self.validate('cmd|s3', args)
    source = args[1]
    self.s3handler().del_files(source)

Response: Handler for del command


Instruction:
Here you have a function in Python 3, explain what it does
def du_handler(self, args):
    '''Handler for size command'''
    for src, size in self.s3handler().size(args[1:]):
      message('%s\t%s' % (size, src))

Response: Handler for size command


Instruction:
Here you have a function in Python 3, explain what it does
def _totalsize_handler(self, args):
    '''Handler of total_size command'''
    total_size = 0
    for src, size in self.s3handler().size(args[1:]):
      total_size += size
    message(str(total_size))

Response: Handler of total_size command


Instruction:
Can you create a Python 3 function that
searches for date information in the string

Response:
def match_date(self, value):
    '''Search for date information in the string'''
    m = self.REGEX_DATE.search(value)
    date = datetime.datetime.utcnow().date()
    if m:
      date = datetime.date(int(m.group(1)), int(m.group(2)), int(m.group(3)))
      value = self.REGEX_DATE.sub('', value)
    return (date, value)


Instruction:
Can you write a function in Python 3 where it
searches for time information in the string

Response:
def match_time(self, value):
    '''Search for time information in the string'''
    m = self.REGEX_TIME.search(value)
    time = datetime.datetime.utcnow().time()
    if m:
      time = datetime.time(int(m.group(1)), int(m.group(2)))
      value = self.REGEX_TIME.sub('', value)
    return (time, value)


Instruction:
Can you generate the documentation for the following Python 3 function
def match_delta(self, value):
    '''Search for timedelta information in the string'''
    m = self.REGEX_DELTA.search(value)
    delta = datetime.timedelta(days=0)
    if m:
      d = int(m.group(1))
      if m.group(3) == 'ago' or m.group(3) == 'before':
        d = -d

      if m.group(2) == 'minute':
        delta = datetime.timedelta(minutes=d)
      elif m.group(2) == 'hour':
        delta = datetime.timedelta(hours=d)
      elif m.group(2) == 'day':
        delta = datetime.timedelta(days=d)
      elif m.group(2) == 'week':
        delta = datetime.timedelta(weeks=d)
      value = self.REGEX_DELTA.sub('', value)
    return (delta, value)

Response: Search for timedelta information in the string


Instruction:
Implement a function in Python 3 to
take json as dictionary parameter

Response:
def check_dict(self, opt, value):
    '''Take json as dictionary parameter'''
    try:
      return json.loads(value)
    except:
      raise optparse.OptionValueError("Option %s: invalid dict value: %r" % (opt, value))


Instruction:
Implement a Python 3 function for
discovering gateways using multicast.

Response:
def discover_gateways(self):
        """Discover gateways using multicast"""

        _socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        _socket.settimeout(5.0)
        if self._interface != 'any':
            _socket.bind((self._interface, 0))

        for gateway in self._gateways_config:
            host = gateway.get('host')
            port = gateway.get('port')
            sid = gateway.get('sid')

            if not (host and port and sid):
                continue
            try:
                ip_address = socket.gethostbyname(host)
                if gateway.get('disable'):
                    _LOGGER.info(
                        'Xiaomi Gateway %s is disabled by configuration', sid)
                    self.disabled_gateways.append(ip_address)
                    continue
                _LOGGER.info(
                    'Xiaomi Gateway %s configured at IP %s:%s',
                    sid, ip_address, port)

                self.gateways[ip_address] = XiaomiGateway(
                    ip_address, port, sid,
                    gateway.get('key'), self._device_discovery_retries,
                    self._interface, gateway.get('proto'))
            except OSError as error:
                _LOGGER.error(
                    "Could not resolve %s: %s", host, error)

        try:
            _socket.sendto('{"cmd":"whois"}'.encode(),
                           (self.MULTICAST_ADDRESS, self.GATEWAY_DISCOVERY_PORT))

            while True:
                data, (ip_add, _) = _socket.recvfrom(1024)
                if len(data) is None or ip_add in self.gateways:
                    continue

                if ip_add in self.gateways.keys() or ip_add in self.disabled_gateways:
                    continue

                resp = json.loads(data.decode())
                if resp["cmd"] != 'iam':
                    _LOGGER.error("Response does not match return cmd")
                    continue

                if resp["model"] not in GATEWAY_MODELS:
                    _LOGGER.error("Response must be gateway model")
                    continue

                disabled = False
                gateway_key = None
                for gateway in self._gateways_config:
                    sid = gateway.get('sid')
                    if sid is None or sid == resp["sid"]:
                        gateway_key = gateway.get('key')
                    if sid and sid == resp['sid'] and gateway.get('disable'):
                        disabled = True

                sid = resp["sid"]
                if disabled:
                    _LOGGER.info("Xiaomi Gateway %s is disabled by configuration",
                                 sid)
                    self.disabled_gateways.append(ip_add)
                else:
                    _LOGGER.info('Xiaomi Gateway %s found at IP %s', sid, ip_add)
                    self.gateways[ip_add] = XiaomiGateway(
                        ip_add, resp["port"], sid, gateway_key,
                        self._device_discovery_retries, self._interface,
                        resp["proto_version"] if "proto_version" in resp else None)

        except socket.timeout:
            _LOGGER.info("Gateway discovery finished in 5 seconds")
            _socket.close()


Instruction:
Can you tell what is the following Python 3 function doing
def listen(self):

        _LOGGER.info('Creating Multicast Socket')
        self._mcastsocket = self._create_mcast_socket()
        self._listening = True
        thread = Thread(target=self._listen_to_msg, args=())
        self._threads.append(thread)
        thread.daemon = True
        thread.start()

Response: Start listening to messages.


Instruction:
Create a Python 3 function to
send data to the hub.

Response:
def write_to_hub(self, sid, **kwargs):
        """Send data to gateway to turn on / off device"""
        if self.key is None:
            _LOGGER.error('Gateway Key is not provided. Can not send commands to the gateway.')
            return False
        data = {}
        for key in kwargs:
            data[key] = kwargs[key]
        if not self.token:
            _LOGGER.debug('Gateway Token was not obtained yet. Cannot send commands to the gateway.')
            return False

        cmd = dict()
        cmd['cmd'] = 'write'
        cmd['sid'] = sid
        if int(self.proto[0:1]) == 1:
            data['key'] = self._get_key()
            cmd['data'] = data
        else:
            cmd['key'] = self._get_key()
            cmd['params'] = [data]
        resp = self._send_cmd(json.dumps(cmd), "write_ack") if int(self.proto[0:1]) == 1 \
            else self._send_cmd(json.dumps(cmd), "write_rsp")
        _LOGGER.debug("write_ack << %s", resp)
        if _validate_data(resp):
            return True
        if not _validate_keyerror(resp):
            return False

        # If 'invalid key' message we ask for a new token
        resp = self._send_cmd('{"cmd" : "get_id_list"}', "get_id_list_ack") if int(self.proto[0:1]) == 1 \
            else self._send_cmd('{"cmd" : "discovery"}', "discovery_rsp")
        _LOGGER.debug("get_id_list << %s", resp)

        if resp is None or "token" not in resp:
            _LOGGER.error('No new token from gateway. Can not send commands to the gateway.')
            return False
        self.token = resp['token']
        if int(self.proto[0:1]) == 1:
            data['key'] = self._get_key()
            cmd['data'] = data
        else:
            cmd['key'] = self._get_key()
            cmd['params'] = [data]
        resp = self._send_cmd(json.dumps(cmd), "write_ack") if int(self.proto[0:1]) == 1 \
            else self._send_cmd(json.dumps(cmd), "write_rsp")
        _LOGGER.debug("write_ack << %s", resp)
        return _validate_data(resp)


Instruction:
Write a Python 3 script for
getting data from hub.

Response:
def get_from_hub(self, sid):
        """Get data from gateway"""
        cmd = '{ "cmd":"read","sid":"' + sid + '"}'
        resp = self._send_cmd(cmd, "read_ack") if int(self.proto[0:1]) == 1 else self._send_cmd(cmd, "read_rsp")
        _LOGGER.debug("read_ack << %s", resp)
        return self.push_data(resp)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def push_data(self, data):
        if not _validate_data(data):
            return False
        jdata = json.loads(data['data']) if int(self.proto[0:1]) == 1 else _list2map(data['params'])
        if jdata is None:
            return False
        sid = data['sid']
        for func in self.callbacks[sid]:
            func(jdata, data)
        return True

Response: Push data broadcasted from gateway to device


Instruction:
Can you generate the documentation for the following Python 3 function
def _get_key(self):
        init_vector = bytes(bytearray.fromhex('17996d093d28ddb3ba695a2e6f58562e'))
        encryptor = Cipher(algorithms.AES(self.key.encode()), modes.CBC(init_vector),
                           backend=default_backend()).encryptor()
        ciphertext = encryptor.update(self.token.encode()) + encryptor.finalize()
        if isinstance(ciphertext, str):  # For Python 2 compatibility
            return ''.join('{:02x}'.format(ord(x)) for x in ciphertext)
        return ''.join('{:02x}'.format(x) for x in ciphertext)

Response: Get key using token from gateway


Instruction:
Can you tell what is the following Python 3 function doing
def train(hparams, *args):
    # Initialize experiments and track all the hyperparameters
    exp = Experiment(
        name=hparams.test_tube_exp_name,
        # Location to save the metrics.
        save_dir=hparams.log_path,
        autosave=False,
    )
    exp.argparse(hparams)

    # Pretend to train.
    x = torch.rand((1, hparams.x_val))
    for train_step in range(0, 100):
        y = torch.rand((hparams.x_val, 1))
        out = x.mm(y)
        exp.log({'fake_err': out.item()})

    # Save exp when .
    exp.save()

Response: Train your awesome model with the given hyperparameters.


Instruction:
How would you explain what the following Python 3 function does
def train(hparams, *args):
    # Initialize experiments and track all the hyperparameters
    exp = Experiment(
        name=hparams.test_tube_exp_name,
        # Location to save the metrics.
        save_dir=hparams.log_path,
        # The experiment version is optional, but using the one
        # from SLURM means the exp will not collide with other
        # versions if SLURM runs multiple at once.
        version=hparams.hpc_exp_number,
        autosave=False,
    )
    exp.argparse(hparams)

    # Pretend to train.
    x = hparams.x_val
    for train_step in range(0, 100):
        y = hparams.y_val
        out = x * y
        exp.log({'fake_err': out.item()})  # Log metrics.

    # Save exp when done.
    exp.save()

Response: Train your awesome model with hyperparameters.


Instruction:
How would you code a function in Python 3 to
call by RQ when there is a failure in a worker process.

Response:
def exception_handler(job, *exc_info):
    """
    Called by RQ when there is a failure in a worker.

    NOTE: Make sure that in your RQ worker process, rollbar.init() has been called with
    handler='blocking'. The default handler, 'thread', does not work from inside an RQ worker.
    """
    # Report data about the job with the exception.
    job_info = job.to_dict()
    # job_info['data'] is the pickled representation of the job, and doesn't json-serialize well.
    # repr() works nicely.
    job_info['data'] = repr(job_info['data'])

    extra_data = {'job': job_info}
    payload_data = {'framework': 'rq'}

    rollbar.report_exc_info(exc_info, extra_data=extra_data, payload_data=payload_data)

    # continue to the next handler
    return True


Instruction:
Explain what the following Python 3 code does
def patch_debugtoolbar(settings):
    try:
        from pyramid_debugtoolbar import tbtools
    except ImportError:
        return

    rollbar_web_base = settings.get('rollbar.web_base', DEFAULT_WEB_BASE)
    if rollbar_web_base.endswith('/'):
        rollbar_web_base = rollbar_web_base[:-1]

    def insert_rollbar_console(request, html):
        # insert after the closing

        item_uuid = request.environ.get('rollbar.uuid')
        if not item_uuid:
            return html

        url = '%s/item/uuid/?uuid=%s' % (rollbar_web_base, item_uuid)
        link = 'View in Rollbar' % url
        new_data = "
Rollbar: %s
" % link
        insertion_marker = "
"
        replacement = insertion_marker + new_data
        return html.replace(insertion_marker, replacement, 1)

    # patch tbtools.Traceback.render_full
    old_render_full = tbtools.Traceback.render_full

    def new_render_full(self, request, *args, **kw):
        html = old_render_full(self, request, *args, **kw)
        return insert_rollbar_console(request, html)

    tbtools.Traceback.render_full = new_render_full

Response: Patches pyramid_debugtoolbar to display a link to the related rollbar item.


Instruction:
Can you create a Python 3 function that
includes pyramid entry point in rollbar.

Response:
def includeme(config):
    """
    Pyramid entry point
    """
    settings = config.registry.settings

    config.add_tween('rollbar.contrib.pyramid.rollbar_tween_factory', over=EXCVIEW)

    # run patch_debugtoolbar, unless they disabled it
    if asbool(settings.get('rollbar.patch_debugtoolbar', True)):
        patch_debugtoolbar(settings)

    def hook(request, data):
        data['framework'] = 'pyramid'

        if request:
            request.environ['rollbar.uuid'] = data['uuid']

            if request.matched_route:
                data['context'] = request.matched_route.name

    rollbar.BASE_DATA_HOOK = hook

    kw = parse_settings(settings)

    access_token = kw.pop('access_token')
    environment = kw.pop('environment', 'production')

    if kw.get('scrub_fields'):
        kw['scrub_fields'] = set([str.strip(x) for x in kw.get('scrub_fields').split('\n') if x])

    if kw.get('exception_level_filters'):
        r = DottedNameResolver()
        exception_level_filters = []
        for line in kw.get('exception_level_filters').split('\n'):
            if line:
                dotted_path, level = line.split()

                try:
                    cls = r.resolve(dotted_path)
                    exception_level_filters.append((cls, level))
                except ImportError:
                    log.error('Could not import %r' % dotted_path)

        kw['exception_level_filters'] = exception_level_filters

    kw['enabled'] = asbool(kw.get('enabled', True))

    rollbar.init(access_token, environment, **kw)


Instruction:
Here you have a function in Python 3, explain what it does
def _ensure_log_handler(self):
        if log.handlers:
            return
        handler = logging.StreamHandler()
        formatter = logging.Formatter(
            '%(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s')
        handler.setFormatter(formatter)
        log.addHandler(handler)

Response: Ensure that the log handler is set up.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_request():

    # TODO(cory): add in a generic _get_locals_request() which
    # will iterate up through the call stack and look for a variable
    # that appears to be valid request object.
    for fn in (_get_bottle_request,
               _get_flask_request,
               _get_pyramid_request,
               _get_pylons_request):
        try:
            req = fn()
            if req is not None:
                return req
        except:
            pass

    return None

Response: Get the current request object.


Instruction:
Implement a function in Python 3 to
initialize the Rollbar module with the given access token and environment.

Response:
def init(access_token, environment='production', scrub_fields=None, url_fields=None, **kw):
    """
    Saves configuration variables in this module's SETTINGS.

    access_token: project access token. Get this from the Rollbar UI:
                  - click "Settings" in the top nav
                  - click "Projects" in the left nav
                  - copy-paste the appropriate token.
    environment: environment name. Can be any string; suggestions: 'production', 'development',
                 'staging', 'yourname'
    **kw: provided keyword arguments will override keys in SETTINGS.
    """
    global SETTINGS, agent_log, _initialized, _transforms, _serialize_transform, _threads

    if scrub_fields is not None:
       SETTINGS['scrub_fields'] = list(scrub_fields)
    if url_fields is not None:
       SETTINGS['url_fields'] = list(url_fields)

    # Merge the extra config settings into SETTINGS
    SETTINGS = dict_merge(SETTINGS, kw)
    if _initialized:
        # NOTE: Temp solution to not being able to re-init.
        # New versions of pyrollbar will support re-initialization
        # via the (not-yet-implemented) configure() method.
        if not SETTINGS.get('suppress_reinit_warning'):
            log.warning('Rollbar already initialized. Ignoring re-init.')
        return

    SETTINGS['access_token'] = access_token
    SETTINGS['environment'] = environment

    if SETTINGS.get('allow_logging_basic_config'):
        logging.basicConfig()

    if SETTINGS.get('handler') == 'agent':
        agent_log = _create_agent_log()

    # We will perform these transforms in order:
    # 1. Serialize the payload to be all python built-in objects
    # 2. Scrub the payloads based on the key suffixes in SETTINGS['scrub_fields']
    # 3. Scrub URLs in the payload for keys that end with 'url'
    # 4. Optional - If local variable gathering is enabled, transform the
    #       trace frame values using the ShortReprTransform.
    _serialize_transform = SerializableTransform(safe_repr=SETTINGS['locals']['safe_repr'],
                                                 whitelist_types=SETTINGS['locals']['whitelisted_types'])
    _transforms = [
        ScrubRedactTransform(),
        _serialize_transform,
        ScrubTransform(suffixes=[(field,) for field in SETTINGS['scrub_fields']], redact_char='*'),
        ScrubUrlTransform(suffixes=[(field,) for field in SETTINGS['url_fields']], params_to_scrub=SETTINGS['scrub_fields'])
    ]

    # A list of key prefixes to apply our shortener transform to. The request
    # being included in the body key is old behavior and is being retained for
    # backwards compatibility.
    shortener_keys = [
        ('request', 'POST'),
        ('request', 'json'),
        ('body', 'request', 'POST'),
        ('body', 'request', 'json'),
    ]

    if SETTINGS['locals']['enabled']:
        shortener_keys.append(('body', 'trace', 'frames', '*', 'code'))
        shortener_keys.append(('body', 'trace', 'frames', '*', 'args', '*'))
        shortener_keys.append(('body', 'trace', 'frames', '*', 'kwargs', '*'))
        shortener_keys.append(('body', 'trace', 'frames', '*', 'locals', '*'))

    shortener_keys.extend(SETTINGS['shortener_keys'])

    shortener = ShortenerTransform(safe_repr=SETTINGS['locals']['safe_repr'],
                                   keys=shortener_keys,
                                   **SETTINGS['locals']['sizes'])
    _transforms.append(shortener)
    _threads = queue.Queue()
    events.reset()
    filters.add_builtin_filters(SETTINGS)

    _initialized = True


Instruction:
Write a Python 3 function that can
report an exception to Rollbar.

Response:
def report_exc_info(exc_info=None, request=None, extra_data=None, payload_data=None, level=None, **kw):
    """
    Reports an exception to Rollbar, using exc_info (from calling sys.exc_info())

    exc_info: optional, should be the result of calling sys.exc_info(). If omitted, sys.exc_info() will be called here.
    request: optional, a WebOb, Werkzeug-based or Sanic request object.
    extra_data: optional, will be included in the 'custom' section of the payload
    payload_data: optional, dict that will override values in the final payload
                  (e.g. 'level' or 'fingerprint')
    kw: provided for legacy purposes; unused.

    Example usage:

    rollbar.init(access_token='YOUR_PROJECT_ACCESS_TOKEN')
    try:
        do_something()
    except:
        rollbar.report_exc_info(sys.exc_info(), request, {'foo': 'bar'}, {'level': 'warning'})
    """
    if exc_info is None:
        exc_info = sys.exc_info()

    try:
        return _report_exc_info(exc_info, request, extra_data, payload_data, level=level)
    except Exception as e:
        log.exception("Exception while reporting exc_info to Rollbar. %r", e)


Instruction:
Implement a function in Python 3 to
report a message to Rollbar.

Response:
def report_message(message, level='error', request=None, extra_data=None, payload_data=None):
    """
    Reports an arbitrary string message to Rollbar.

    message: the string body of the message
    level: level to report at. One of: 'critical', 'error', 'warning', 'info', 'debug'
    request: the request object for the context of the message
    extra_data: dictionary of params to include with the message. 'body' is reserved.
    payload_data: param names to pass in the 'data' level of the payload; overrides defaults.
    """
    try:
        return _report_message(message, level, request, extra_data, payload_data)
    except Exception as e:
        log.exception("Exception while reporting message to Rollbar. %r", e)


Instruction:
Explain what the following Python 3 code does
def send_payload(payload, access_token):
    payload = events.on_payload(payload)
    if payload is False:
        return

    payload_str = _serialize_payload(payload)

    handler = SETTINGS.get('handler')
    if handler == 'blocking':
        _send_payload(payload_str, access_token)
    elif handler == 'agent':
        agent_log.error(payload_str)
    elif handler == 'tornado':
        if TornadoAsyncHTTPClient is None:
            log.error('Unable to find tornado')
            return
        _send_payload_tornado(payload_str, access_token)
    elif handler == 'gae':
        if AppEngineFetch is None:
            log.error('Unable to find AppEngine URLFetch module')
            return
        _send_payload_appengine(payload_str, access_token)
    elif handler == 'twisted':
        if treq is None:
            log.error('Unable to find Treq')
            return
        _send_payload_twisted(payload_str, access_token)
    else:
        # default to 'thread'
        thread = threading.Thread(target=_send_payload, args=(payload_str, access_token))
        _threads.put(thread)
        thread.start()

Response: Send a payload to the current thread.


Instruction:
Write a Python 3 function for
searching a project for items that match the input criteria.

Response:
def search_items(title, return_fields=None, access_token=None, endpoint=None, **search_fields):
    """
    Searches a project for items that match the input criteria.

    title: all or part of the item's title to search for.
    return_fields: the fields that should be returned for each item.
            e.g. ['id', 'project_id', 'status'] will return a dict containing
                 only those fields for each item.
    access_token: a project access token. If this is not provided,
                  the one provided to init() will be used instead.
    search_fields: additional fields to include in the search.
            currently supported: status, level, environment
    """
    if not title:
        return []

    if return_fields is not None:
        return_fields = ','.join(return_fields)

    return _get_api('search/',
                    title=title,
                    fields=return_fields,
                    access_token=access_token,
                    endpoint=endpoint,
                    **search_fields)