Instruction:
Write a Python 3 script to
return Matrices u b v with self = ubv and self = ubv and v is in bidiagonal form.

Response:
def householder(self):
        """Return Matrices u,b,v with self = ubv and b is in bidiagonal form

        The algorithm uses householder transformations.

        :return tuple (u,b,v): A tuple with the Matrix u, b and v.
                and self = ubv (except some rounding errors)
                u is a unitary matrix
                b is a bidiagonal matrix.
                v is a unitary matrix.
        :note: Currently the algorithm only works for squared matrices

        :todo: Make sure, that the bidiagonal matrix is 0.0 except for the bidiagonal.
            Due to rounding errors, this is currently not ensured
        """
        # copy instance to transform it to bidiagonal form.
        bidiagMatrix = Matrix.from_two_dim_array(self.get_width(), self.get_height(), self.matrix)

        # build identity matrix, which is used to calculate householder transformations
        identityMatrixRow = Matrix(self.get_height(), self.get_height())
        for i in xrange(self.get_height()):
            identityMatrixRow.set_value(i, i, 1.0)

        identityMatrixCol = Matrix(self.get_width(), self.get_width())
        for i in xrange(self.get_width()):
            identityMatrixCol.set_value(i, i, 1.0)

        # zero out the k'th column and row
        for k in xrange(self.get_width() - 1):
            # vector with the values of the k'th column (first k-1 rows are 0)
            x = Vector(self.get_height())
            y = Vector(self.get_height())
            if k > 0:
                x.set_value(0, k - 1, bidiagMatrix.get_value(k, k - 1))
                y.set_value(0, k - 1, bidiagMatrix.get_value(k, k - 1))
            s = 0.0
            for i in xrange(k, self.get_height()):
                val = bidiagMatrix.get_value(k, i)
                x.set_value(0, i, val)
                s += (val ** 2)
            s = sqrt(s)
            # y must have same length as x
            y.set_value(0, k, s)
            tmp = x - y
            norm = sqrt(sum(i[0] ** 2 for i in tmp.get_array()))
            # calculate w = (x-y)/(|x-y|)
            w = tmp / norm
            # uk is the k'th householder matrix for the column
            uk = identityMatrixRow - 2 * (w * w.transform())

            bidiagMatrix = uk * bidiagMatrix

            if k == 0:
                # set u in first iteration.
                u = uk
            else:
                u = u * uk

            # zero out the the row
            if k < self.get_width() - 2:
                x = Vector(self.get_width())
                y = Vector(self.get_width())
                x.set_value(0, k, bidiagMatrix.get_value(k, k))
                y.set_value(0, k, bidiagMatrix.get_value(k, k))

                s = 0.0
                for i in xrange(k + 1, bidiagMatrix.get_width()):
                    val = bidiagMatrix.get_value(i, k)
                    x.set_value(0, i, val)
                    s += (val ** 2)
                # length of vector x ignoring the k'th value
                s = sqrt(s)
                # y must have same length as x, since k'th value is equal
                # set k+1 value to s
                y.set_value(0, k + 1, s)
                tmp = x - y
                norm = sqrt(sum(i[0] ** 2 for i in tmp.get_array()))
                w = tmp / norm
                # vk is the k'th householder matrix for the row
                vk = identityMatrixCol - (2 * (w * w.transform()))
                bidiagMatrix = bidiagMatrix * vk
                if k == 0:
                    # set v in first iteration
                    v = vk
                else:
                    v = vk * v

        return (u, bidiagMatrix, v)


Instruction:
Write a Python 3 function for
returning the singular value decomposition of the Matrix object.

Response:
def svd(self, maxIteration=50):
        """Return the singular value decomposition of the Matrix instance

        :param integer maxIteration: The maximmum number of iterations,
            which are executed in the qr decomposition

        :return: A tuple with Matrices u, sigma, v with
                    so that u * sigma * v^T = self
        :rtype: tuple

        :raise: Raises a :py:exc:`ValueError` if the Matrix object has
                    more columns than rows

        :note:  Translation of the FORTRAN implementation if the SVD given
                in the NUMERICAL RECIPES IN FORTRAN 77. THE ART OF SCIENTIFIC
                COMPUTING.
                The algorithm is not yet numerical stable, so the results may
                not be in all cases as expected.
        """
        if(self.get_width() > self.get_height()):
            raise ValueError("Matrix has more columns than rows.")
        eps = 1.e-15
        tol = 1.e-64 / eps
        a = self.get_array(False)
        m = len(a[0])
        n = len(a)

        v = []
        for k in xrange(n):
            v.append([0.0] * n)
        # output diagonal
        w = [0.0] * n
        # upper diagonal (for bidiagonal form)
        rv1 = [0.0] * n

        # Householder Reduction to bidiagional form
        g = 0.0
        anorm = 0.0

        for i in xrange(n):
            l = i + 1
            rv1[i] = g

            s = 0.0
            # calculate length of relevant row vector in matrix (part of i'th column)
            s = sum(a[i][k] ** 2 for k in xrange(i, m))
            if s <= tol:
                g = 0.0
            else:
                f = a[i][i]
                # square root to get actual length of vector
                g = sqrt(s) if f < 0 else -sqrt(s)
                h = f * g - s
                a[i][i] = f - g
                for j in xrange(l, n):
                    s = sum(a[i][k] * a[j][k] for k in xrange(i, m))
                    f = s / h
                    for k in xrange(i, m):
                        a[j][k] += (f * a[i][k])
            w[i] = g
            # calculate length of relevant column vector in matrix (part of i'th row)
            s = 0.0
            s = sum(a[k][i] ** 2 for k in xrange(l, n))
            if s <= tol:
                g = 0.0
            else:
                f = a[l][i]
                g = sqrt(s) if f < 0 else -sqrt(s)
                h = f * g - s
                a[l][i] = f - g
                for k in xrange(l, n):
                    rv1[k] = a[k][i] / h
                for j in xrange(l, m):
                    s = sum(a[k][j] * a[k][i] for k in xrange(l, n))
                    for k in xrange(l, n):
                        a[k][j] += (s * rv1[k])
            anorm = max(anorm, (abs(w[i]) + abs(rv1[i])))

        # Accumulation of right hand transformations
        for i in xrange(n - 1, -1, -1):
            if g != 0.0:
                for j in xrange(l, n):
                    v[i][j] = a[j][i] / (g * a[i + 1][i])
                for j in xrange(l, n):
                    s = sum(a[k][i] * v[j][k] for k in xrange(l, n))
                    for k in xrange(l, n):
                        v[j][k] += (s * v[i][k])
            for j in xrange(l, n):
                v[j][i] = 0.0
                v[i][j] = 0.0
            v[i][i] = 1.0
            g = rv1[i]
            l = i

        # Accumulation of left hand transformations
        for i in xrange(n - 1, -1, -1):
            l = i + 1
            g = w[i]
            for j in xrange(l, n):
                a[j][i] = 0.0
            if g != 0.0:
                for j in xrange(l, n):
                    s = sum(a[i][k] * a[j][k] for k in xrange(l, m))
                    f = s / (a[i][i] * g)
                    for k in xrange(i, m):
                        a[j][k] += f * a[i][k]
                for j in xrange(i, m):
                    a[i][j] /= g
            else:
                for j in xrange(i, m):
                    a[i][j] = 0.0
            a[i][i] += 1.0

        eps *= anorm

        # Diagonalization of the bidiagonal form.
        # Loop over singular values and over allowed iterations
        for k in xrange(n - 1, -1, -1):
            for dummy in xrange(maxIteration):
                for l in xrange(k, -1, -1):
                    convergenceTest = False
                    if abs(rv1[l]) <= eps:
                        convergenceTest = True
                        break
                    if abs(w[l - 1]) <= eps:
                        # convergenceTest = False (already default)
                        break
                if not convergenceTest:
                    c = 0.0
                    s = 1.0
                    nm = l - 1
                    for i in xrange(l, k + 1):
                        f = s * rv1[i]
                        rv1[i] = c * rv1[i]
                        if abs(f) <= eps:
                            break
                        g = w[i]
                        h = pythag(f, g)
                        w[i] = h
                        c = g / h
                        s = -f / h
                        for j in xrange(m):
                            y = a[nm][j]
                            z = a[i][j]
                            a[nm][j] = (y * c) + (z * s)
                            a[i][j] = -(y * s) + (z * c)
                z = w[k]
                if l == k:
                    # convergence
                    if z < 0.0:
                        w[k] = -z
                        for j in xrange(n):
                            v[k][j] = -v[k][j]
                    break

                x = w[l]
                y = w[k - 1]
                g = rv1[k - 1]
                h = rv1[k]
                f = ((y - z) * (y + z) + (g - h) * (g + h)) / (2.0 * h * y)
                g = pythag(f, 1.0)
                f = ((x - z) * (x + z) + h * ((y / (f + sign(g, f))) - h)) / x
                c = 1.0
                s = 1.0
                for i in xrange(l + 1, k + 1):
                    g = rv1[i]
                    y = w[i]
                    h = s * g
                    g = c * g
                    z = pythag(f, h)
                    rv1[i - 1] = z
                    c = f / z
                    s = h / z
                    f = (x * c) + (g * s)
                    g = -x * s + g * c
                    h = y * s
                    y = y * c
                    for jj in xrange(n):
                        x = v[i - 1][jj]
                        z = v[i][jj]
                        v[i - 1][jj] = (x * c) + (z * s)
                        v[i][jj] = -(x * s) + (z * c)
                    z = pythag(f, h)
                    w[i - 1] = z
                    if z != 0.0:
                        z = 1.0 / z
                        c = f * z
                        s = h * z
                    f = (c * g) + (s * y)
                    x = -s * g + c * y
                    for jj in xrange(m):
                        y = a[i - 1][jj]
                        z = a[i][jj]
                        a[i - 1][jj] = (y * c) + (z * s)
                        a[i][jj] = -(y * s) + (z * c)
                rv1[l] = 0.0
                rv1[k] = f
                w[k] = x
        # Build Matrix instances for the result
        uM = Matrix.from_two_dim_array(len(a), len(a[0]), a)

        diagMatrix = Matrix(len(w), len(w))
        for i in xrange(len(w)):
            diagMatrix.set_value(i, i, w[i])

        vM = Matrix.from_two_dim_array(len(v), len(v[0]), v)

        return uM, diagMatrix, vM


Instruction:
How would you explain what the following Python 3 function does
def pseudoinverse(self):
        transform = False
        if self.get_width() > self.get_height():
            transform = True
            u, sigma, v = self.transform().svd()
        else:
            u, sigma, v = self.svd()
        # calculate inverse of sigma
        for i in xrange(min(sigma.get_height(), sigma.get_width())):
            val = sigma.get_value(i, i)
            # divide only if the value is not 0 or close to zero (rounding errors)
            eps = 1.e-15
            if eps < val or val < -eps:
                sigma.set_value(i, i, 1 / val)
        if transform:
            return (v * sigma * u.transform()).transform()
        else:
            return v * sigma * u.transform()

Response: Return the pseudoinverse of the entry.


Instruction:
Write a Python 3 script for
creating vector from matrix.

Response:
def initialize_from_matrix(cls, matrix, column):
        """Create vector from matrix

        :param Matrix matrix: The Matrix, which should be used to create the vector.
        :param integer column: The column of the matrix, which should be used
                                to create the new vector.
        :raise: Raises an :py:exc:`IndexError` if the matrix does not have the specified column.
        """
        vec = Vector(matrix.get_height())
        for row in xrange(matrix.get_height()):
            vec.set_value(0, row, matrix.get_value(column, row))
        return vec


Instruction:
How would you code a function in Python 3 to
unify the vector. The length of the vector will be 1. :return: Return the instance itself :rtype: Vector

Response:
def unify(self):
        """Unifies the vector. The length of the vector will be 1.

        :return: Return the instance itself
        :rtype: Vector
        """
        length = float(self.norm())
        for row in xrange(self.get_height()):
            self.set_value(0, row, self.get_value(0, row) / length)
        return self


Instruction:
How would you explain what the following Python 3 function does
def text(self):
        divisions = list(self.divisions)
        if len(divisions) == 0:
            return ''
        elif len(divisions) == 1:
            return divisions[0].text.strip()
        else:
            return super().text

Response: Get the entire text content as str


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _calculate(self, startingPercentage, endPercentage, startDate, endDate):
        # get the defined subset of error values
        errorValues = self._get_error_values(startingPercentage, endPercentage, startDate, endDate)
        errorValues = filter(lambda item: item is not None, errorValues)

        return sorted(errorValues)[len(errorValues)//2]

Response: This method is used to calculate the error value for a specific resource.


Instruction:
Can you tell what is the following Python 3 function doing
def moving_frequency(self, data_frame):
        f = []
        for i in range(0, (data_frame.td[-1].astype('int') - self.window)):
            f.append(sum(data_frame.action_type[(data_frame.td >= i) & (data_frame.td < (i + self.window))] == 1) /
                     float(self.window))

        diff_mov_freq = (np.array(f[1:-1]) - np.array(f[0:-2])) / np.array(f[0:-2])

        duration = math.ceil(data_frame.td[-1])

        return diff_mov_freq, duration

Response: This method returns the moving frequency of the assessment.


Instruction:
Can you tell what is the following Python 3 function doing
def incoordination_score(self, data_frame):
        diff = data_frame.td[1:-1].values - data_frame.td[0:-2].values
        inc_s = np.var(diff[np.arange(1, len(diff), 2)], dtype=np.float64) * 1000.0
        duration = math.ceil(data_frame.td[-1])
        return inc_s, duration

Response: This method calculates the variance of the time interval in msec between taps


Instruction:
How would you explain what the following Python 3 function does
def mean_alnt_target_distance(self, data_frame):
        dist = np.sqrt((data_frame.x[1:-1].values-data_frame.x[0:-2].values)**2+
                       (data_frame.y[1:-1].values-data_frame.y[0:-2].values)**2)
        matd = np.mean(dist[np.arange(1,len(dist),2)])
        duration = math.ceil(data_frame.td[-1])
        return matd, duration

Response: This method calculates the mean target distance between the alternate tapping and the alternate target distance in pixels


Instruction:
How would you explain what the following Python 3 function does
def __train(self, n_neighbors=3):

        # m = self.labels.drop(['id','MDS_UPDRSIII'], axis=1).values
        # print(itemfreq(m))
        #
        # for i, row in enumerate(self.labels.drop(['id','MDS_UPDRSIII'], axis=1).values):
        #     print(np.bincount(row))

        try:
            for obs in self.observations:
                features, ids = self.__get_features_for_observation(observation=obs, skip_id=3497,
                                                                    last_column_is_id=True)
                normalised_data = whiten(features)

                x = pd.DataFrame(normalised_data)
                y = self.labels[obs].values
                # x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=42)

                knn = KNeighborsClassifier(n_neighbors=n_neighbors, weights='distance')
                # knn.fit(x_train, y_train)
                knn.fit(x, y)

                # print('Accuracy of K-NN classifier: {:.2f}'.format(knn.score(x, y)))
                # print('Accuracy of K-NN classifier on training set: {:.2f}'.format(knn.score(x_train, y_train)))
                # print('Accuracy of K-NN classifier on test set: {:.2f}'.format(knn.score(x_test, y_test)))
                # print('------')

                if not self.knns:
                    self.knns = [[obs, knn]]
                else:
                    self.knns.append([obs, knn])
        except IOError as e:
            ierr = "({}): {}".format(e.errno, e.strerror)
            logging.error("Error training Clinical UPDRS, file not found, I/O error %s", ierr)

        except ValueError as verr:
            logging.error("Error training Clinical UPDRS ValueError ->%s", verr.message)

        except:
            logging.error("Unexpected error on training Clinical UPDRS init: %s", sys.exc_info()[0])

Response: Train the K - nearest neighbors vote for the current class.


Instruction:
Make a summary of the following Python 3 code
def __get_features_for_observation(self, data_frame=None, observation='LA-LL',
                                       skip_id=None, last_column_is_id=False):
        """
            Extract the features for a given observation from a data frame

            :param data_frame: data frame to get features from
            :type data_frame: pandas.DataFrame
            :param observation: observation name
            :type observation: string
            :param skip_id: skip any test with a given id (optional)
            :type skip_id: int
            :param last_column_is_id: skip the last column of the data frame (useful when id is last column - optional)
            :type last_column_is_id: bool
            :return features: the features
            :rtype features: np.array
        """
        try:
            features = np.array([])

            if data_frame is None:
                data_frame = self.data_frame

            for index, row in data_frame.iterrows():
                if not skip_id == row['id']:
                    features_row = np.nan_to_num(row[row.keys().str.contains(observation)].values)
                    features_row = np.append(features_row, row['id'])
                    features = np.vstack([features, features_row]) if features.size else features_row

            # not the same when getting a single point
            if last_column_is_id:
                if np.ndim(features) > 1:
                    to_return = features[:,:-1]
                else:
                    to_return = features[:-1]
            else:
                to_return = features

            return to_return, data_frame['id'].values
        except:
            logging.error(" observation not found in data frame")

Response: Extract the features for a given observation from a data frame.


Instruction:
Here you have a function in Python 3, explain what it does
def _namify_arguments(mapping):
    result = []
    for name, parameter in mapping.iteritems():
        parameter.name = name
        result.append(parameter)
    return result

Response: Ensure that a mapping of names to parameters has the correct name.


Instruction:
Given the following Python 3 function, write the documentation
def _merge_associative_list(alist, path, value):
    for key in path[:-1]:
        for item in alist:
            if item[0] == key:
                alist = item[1]
                break
        else:
            subalist = []
            alist.append((key, subalist))
            alist = subalist
    alist.append((path[-1], value))

Response:
Merge a value into an associative list at the given path maintaining
    insertion order.


Instruction:
Write a Python 3 function for
coercing a single value according to this parameter s settings.

Response:
def coerce(self, value):
        """Coerce a single value according to this parameter's settings.

        @param value: A L{str}, or L{None}. If L{None} is passed - meaning no
            value is avalable at all, not even the empty string - and this
            parameter is optional, L{self.default} will be returned.
        """
        if value is None:
            if self.optional:
                return self.default
            else:
                value = ""
        if value == "":
            if not self.allow_none:
                raise MissingParameterError(self.name, kind=self.kind)
            return self.default
        try:
            self._check_range(value)
            parsed = self.parse(value)
            if self.validator and not self.validator(parsed):
                raise ValueError(value)
            return parsed
        except ValueError:
            try:
                value = value.decode("utf-8")
                message = "Invalid %s value %s" % (self.kind, value)
            except UnicodeDecodeError:
                message = "Invalid %s value" % self.kind
            raise InvalidParameterValueError(message)


Instruction:
Create a Python 3 function to
check that the given value is in the expected range.

Response:
def _check_range(self, value):
        """Check that the given C{value} is in the expected range."""
        if self.min is None and self.max is None:
            return

        measure = self.measure(value)
        prefix = "Value (%s) for parameter %s is invalid.  %s"

        if self.min is not None and measure < self.min:
            message = prefix % (value, self.name,
                                self.lower_than_min_template % self.min)
            raise InvalidParameterValueError(message)

        if self.max is not None and measure > self.max:
            message = prefix % (value, self.name,
                                self.greater_than_max_template % self.max)
            raise InvalidParameterValueError(message)


Instruction:
Create a Python 3 function to
convert a dictionary of relative index to a list of parsed s.

Response:
def parse(self, value):
        """
        Convert a dictionary of {relative index: value} to a list of parsed
        C{value}s.
        """
        indices = []
        if not isinstance(value, dict):
            # We interpret non-list inputs as a list of one element, for
            # compatibility with certain EC2 APIs.
            return [self.item.coerce(value)]
        for index in value.keys():
            try:
                indices.append(int(index))
            except ValueError:
                raise UnknownParameterError(index)
        result = [None] * len(value)
        for index_index, index in enumerate(sorted(indices)):
            v = value[str(index)]
            if index < 0:
                raise UnknownParameterError(index)
            result[index_index] = self.item.coerce(v)
        return result


Instruction:
Can you tell what is the following Python 3 function doing
def format(self, value):
        if isinstance(value, Arguments):
            return dict((str(i), self.item.format(v)) for i, v in value)
        return dict((str(i + 1), self.item.format(v))
                    for i, v in enumerate(value))

Response:
Convert a list like ::

            to a dictionary like ::

            1 2 3.


Instruction:
Write a Python 3 function that can
convert a dictionary of raw values to a dictionary of processed values.

Response:
def parse(self, value):
        """
        Convert a dictionary of raw values to a dictionary of processed values.
        """
        result = {}
        rest = {}
        for k, v in value.iteritems():
            if k in self.fields:
                if (isinstance(v, dict)
                        and not self.fields[k].supports_multiple):
                    if len(v) == 1:
                        # We support "foo.1" as "foo" as long as there is only
                        # one "foo.#" parameter provided.... -_-
                        v = v.values()[0]
                    else:
                        raise InvalidParameterCombinationError(k)
                result[k] = self.fields[k].coerce(v)
            else:
                rest[k] = v
        for k, v in self.fields.iteritems():
            if k not in result:
                result[k] = v.coerce(None)
        if rest:
            raise UnknownParametersError(result, rest)
        return result


Instruction:
Explain what the following Python 3 code does
def format(self, value):
        if not isinstance(value, Arguments):
            value = value.iteritems()
        return dict((k, self.fields[k].format(v)) for k, v in value)

Response: Convert a dictionary of processed values to a dictionary of raw values.


Instruction:
Explain what the following Python 3 code does
def _wrap(self, value):
        if isinstance(value, dict):
            if any(isinstance(name, int) for name in value.keys()):
                if not all(isinstance(name, int) for name in value.keys()):
                    raise RuntimeError("Integer and non-integer keys: %r"
                                       % value.keys())
                items = sorted(value.iteritems(), key=itemgetter(0))
                return [self._wrap(val) for _, val in items]
            else:
                return Arguments(value)
        elif isinstance(value, list):
            return [self._wrap(x) for x in value]
        else:
            return value

Response: Wrap the given L { tree } with L { Arguments } as necessary.


Instruction:
Make a summary of the following Python 3 code
def extract(self, params):
        structure = Structure(fields=dict([(p.name, p)
                                           for p in self._parameters]))
        try:
            tree = structure.coerce(self._convert_flat_to_nest(params))
            rest = {}
        except UnknownParametersError, error:
            tree = error.result
            rest = self._convert_nest_to_flat(error.unknown)
        return Arguments(tree), rest

Response: Extract parameters from a raw C { dict according to this schema.


Instruction:
Given the following Python 3 function, write the documentation
def bundle(self, *arguments, **extra):
        params = {}

        for argument in arguments:
            params.update(argument)

        params.update(extra)
        result = {}
        for name, value in params.iteritems():
            if value is None:
                continue
            segments = name.split('.')
            first = segments[0]
            parameter = self.get_parameter(first)
            if parameter is None:
                raise RuntimeError("Parameter '%s' not in schema" % name)
            else:
                if value is None:
                    result[name] = ""
                else:
                    result[name] = parameter.format(value)

        return self._convert_nest_to_flat(result)

Response: Bundle the given arguments in a C { dict with EC2 - style format.


Instruction:
Explain what the following Python 3 code does
def get_parameter(self, name):
        for parameter in self._parameters:
            if parameter.name == name:
                return parameter

Response: Get the parameter with the given name.


Instruction:
Make a summary of the following Python 3 code
def _convert_flat_to_nest(self, params):
        result = {}
        for k, v in params.iteritems():
            last = result
            segments = k.split('.')
            for index, item in enumerate(segments):
                if index == len(segments) - 1:
                    newd = v
                else:
                    newd = {}
                if not isinstance(last, dict):
                    raise InconsistentParameterError(k)
                if type(last.get(item)) is dict and type(newd) is not dict:
                    raise InconsistentParameterError(k)
                last = last.setdefault(item, newd)
        return result

Response: Convert a flat structure to a nested structure.


Instruction:
Can you generate the documentation for the following Python 3 function
def _convert_nest_to_flat(self, params, _result=None, _prefix=None):
        if _result is None:
            _result = {}
        for k, v in params.iteritems():
            if _prefix is None:
                path = k
            else:
                path = _prefix + '.' + k
            if isinstance(v, dict):
                self._convert_nest_to_flat(v, _result=_result, _prefix=path)
            else:
                _result[path] = v
        return _result

Response: Convert a nested dict to a flat dict.


Instruction:
Implement a function in Python 3 to
extend this schema with any number of schema items.

Response:
def extend(self, *schema_items, **kwargs):
        """
        Add any number of schema items to a new schema.

        Takes the same arguments as the constructor, and returns a new
        L{Schema} instance.

        If parameters, result, or errors is specified, they will be merged with
        the existing parameters, result, or errors.
        """
        new_kwargs = {
            'name': self.name,
            'doc': self.doc,
            'parameters': self._parameters[:],
            'result': self.result.copy() if self.result else {},
            'errors': self.errors.copy() if self.errors else set()}
        if 'parameters' in kwargs:
            new_params = kwargs.pop('parameters')
            new_kwargs['parameters'].extend(new_params)
        new_kwargs['result'].update(kwargs.pop('result', {}))
        new_kwargs['errors'].update(kwargs.pop('errors', set()))
        new_kwargs.update(kwargs)

        if schema_items:
            parameters = self._convert_old_schema(schema_items)
            new_kwargs['parameters'].extend(parameters)
        return Schema(**new_kwargs)


Instruction:
Can you implement a function in Python 3 that
converts an ugly old schema using dotted names to the hot new schema.

Response:
def _convert_old_schema(self, parameters):
        """
        Convert an ugly old schema, using dotted names, to the hot new schema,
        using List and Structure.

        The old schema assumes that every other dot implies an array. So a list
        of two parameters,

            [Integer("foo.bar.baz.quux"), Integer("foo.bar.shimmy")]

        becomes::

            [List(
                "foo",
                item=Structure(
                    fields={"baz": List(item=Integer()),
                            "shimmy": Integer()}))]

        By design, the old schema syntax ignored the names "bar" and "quux".
        """
        # 'merged' here is an associative list that maps parameter names to
        # Parameter instances, OR sub-associative lists which represent nested
        # lists and structures.
        # e.g.,
        #    [Integer("foo")]
        # becomes
        #    [("foo", Integer("foo"))]
        # and
        #    [Integer("foo.bar")]
        # (which represents a list of integers called "foo" with a meaningless
        # index name of "bar") becomes
        #     [("foo", [("bar", Integer("foo.bar"))])].
        merged = []
        for parameter in parameters:
            segments = parameter.name.split('.')
            _merge_associative_list(merged, segments, parameter)
        result = [self._inner_convert_old_schema(node, 1) for node in merged]
        return result


Instruction:
Can you generate a brief explanation for the following Python 3 code
def wait_for_processes(processes, size, progress_queue, watcher, item):
    while size > 0:
        progress_type, value = progress_queue.get()
        if progress_type == ProgressQueue.PROCESSED:
            chunk_size = value
            watcher.transferring_item(item, increment_amt=chunk_size)
            size -= chunk_size
        elif progress_type == ProgressQueue.START_WAITING:
            watcher.start_waiting()
        elif progress_type == ProgressQueue.DONE_WAITING:
            watcher.done_waiting()
        else:
            error_message = value
            for process in processes:
                process.terminate()
            raise ValueError(error_message)
    for process in processes:
        process.join()

Response: Wait for processes to finish downloading a file.


Instruction:
Here you have a function in Python 3, explain what it does
def verify_file_private(filename):
    if platform.system().upper() != 'WINDOWS':
        filename = os.path.expanduser(filename)
        if os.path.exists(filename):
            file_stat = os.stat(filename)
            if mode_allows_group_or_other(file_stat.st_mode):
                raise ValueError(CONFIG_FILE_PERMISSIONS_ERROR)

Response: Verify that the file is private.


Instruction:
Create a Python 3 function to
update progress that the item is about to be transferred.

Response:
def transferring_item(self, item, increment_amt=1):
        """
        Update progress that item is about to be transferred.
        :param item: LocalFile, LocalFolder, or LocalContent(project) that is about to be sent.
        :param increment_amt: int amount to increase our count(how much progress have we made)
        """
        self.cnt += increment_amt
        percent_done = int(float(self.cnt) / float(self.total) * 100.0)
        if KindType.is_project(item):
            details = 'project'
        else:
            details = os.path.basename(item.path)
        self.progress_bar.update(percent_done, '{} {}'.format(self.msg_verb, details))
        self.progress_bar.show()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def finished(self):
        self.progress_bar.set_state(ProgressBar.STATE_DONE)
        self.progress_bar.show()

Response: Print final progress bar.


Instruction:
Can you write a function in Python 3 where it
starts waiting progress bar until done_waiting is called.

Response:
def start_waiting(self):
        """
        Show waiting progress bar until done_waiting is called.
        Only has an effect if we are in waiting state.
        """
        if not self.waiting:
            self.waiting = True
            wait_msg = "Waiting for project to become ready for {}".format(self.msg_verb)
            self.progress_bar.show_waiting(wait_msg)


Instruction:
Make a summary of the following Python 3 code
def done_waiting(self):
        if self.waiting:
            self.waiting = False
            self.progress_bar.show_running()

Response: Show running progress bar if we are in waiting state.


Instruction:
Write a Python 3 script for
showing waiting progress bar until done_waiting is called.

Response:
def show_waiting(self, wait_msg):
        """
        Show waiting progress bar until done_waiting is called.
        Only has an effect if we are in waiting state.
        :param wait_msg: str: message describing what we are waiting for
        """
        self.wait_msg = wait_msg
        self.set_state(ProgressBar.STATE_WAITING)
        self.show()


Instruction:
Make a summary of the following Python 3 code
def _visit_content(item, parent, visitor):
        if KindType.is_project(item):
            visitor.visit_project(item)
        elif KindType.is_folder(item):
            visitor.visit_folder(item, parent)
        else:
            visitor.visit_file(item, parent)
        if not KindType.is_file(item):
            for child in item.children:
                ProjectWalker._visit_content(child, item, visitor)

Response: Recursively visit the content of the item.


Instruction:
Can you implement a function in Python 3 that
creates a URL based on the given service endpoint bucket and object name.

Response:
def s3_url_context(service_endpoint, bucket=None, object_name=None):
    """
    Create a URL based on the given service endpoint and suitable for
    the given bucket or object.

    @param service_endpoint: The service endpoint on which to base the
        resulting URL.
    @type service_endpoint: L{AWSServiceEndpoint}

    @param bucket: If given, the name of a bucket to reference.
    @type bucket: L{unicode}

    @param object_name: If given, the name of an object or object
        subresource to reference.
    @type object_name: L{unicode}
    """

    # Define our own query parser which can handle the consequences of
    # `?acl` and such (subresources).  At its best, parse_qsl doesn't
    # let us differentiate between these and empty values (such as
    # `?acl=`).
    def p(s):
        results = []
        args = s.split(u"&")
        for a in args:
            pieces = a.split(u"=")
            if len(pieces) == 1:
                results.append((unquote(pieces[0]),))
            elif len(pieces) == 2:
                results.append(tuple(map(unquote, pieces)))
            else:
                raise Exception("oh no")
        return results

    query = []
    path = []
    if bucket is None:
        path.append(u"")
    else:
        if isinstance(bucket, bytes):
            bucket = bucket.decode("utf-8")
        path.append(bucket)
        if object_name is None:
            path.append(u"")
        else:
            if isinstance(object_name, bytes):
                object_name = object_name.decode("utf-8")
            if u"?" in object_name:
                object_name, query = object_name.split(u"?", 1)
                query = p(query)
            object_name_components = object_name.split(u"/")
            if object_name_components[0] == u"":
                object_name_components.pop(0)
            if object_name_components:
                path.extend(object_name_components)
            else:
                path.append(u"")
    return _S3URLContext(
        scheme=service_endpoint.scheme.decode("utf-8"),
        host=service_endpoint.get_host().decode("utf-8"),
        port=service_endpoint.port,
        path=path,
        query=query,
    )


Instruction:
Write a Python 3 script to
list all buckets owned by the authenticated sender of the request.

Response:
def list_buckets(self):
        """
        List all buckets.

        Returns a list of all the buckets owned by the authenticated sender of
        the request.
        """
        details = self._details(
            method=b"GET",
            url_context=self._url_context(),
        )
        query = self._query_factory(details)
        d = self._submit(query)
        d.addCallback(self._parse_list_buckets)
        return d


Instruction:
Implement a Python 3 function for
parsing XML bucket list response.

Response:
def _parse_list_buckets(self, (response, xml_bytes)):
        """
        Parse XML bucket list response.
        """
        root = XML(xml_bytes)
        buckets = []
        for bucket_data in root.find("Buckets"):
            name = bucket_data.findtext("Name")
            date_text = bucket_data.findtext("CreationDate")
            date_time = parseTime(date_text)
            bucket = Bucket(name, date_time)
            buckets.append(bucket)
        return buckets


Instruction:
Can you write a function in Python 3 where it
creates a new bucket.

Response:
def create_bucket(self, bucket):
        """
        Create a new bucket.
        """
        details = self._details(
            method=b"PUT",
            url_context=self._url_context(bucket=bucket),
        )
        query = self._query_factory(details)
        return self._submit(query)


Instruction:
How would you implement a function in Python 3 that
deletes a bucket. The bucket must be empty before it can be deleted.

Response:
def delete_bucket(self, bucket):
        """
        Delete a bucket.

        The bucket must be empty before it can be deleted.
        """
        details = self._details(
            method=b"DELETE",
            url_context=self._url_context(bucket=bucket),
        )
        query = self._query_factory(details)
        return self._submit(query)


Instruction:
Explain what the following Python 3 code does
def get_bucket(self, bucket, marker=None, max_keys=None, prefix=None):
        args = []
        if marker is not None:
            args.append(("marker", marker))
        if max_keys is not None:
            args.append(("max-keys", "%d" % (max_keys,)))
        if prefix is not None:
            args.append(("prefix", prefix))
        if args:
            object_name = "?" + urlencode(args)
        else:
            object_name = None
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name=object_name),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_get_bucket)
        return d

Response: Get all the objects in a bucket.


Instruction:
Can you implement a function in Python 3 that
gets the location of a bucket.

Response:
def get_bucket_location(self, bucket):
        """
        Get the location (region) of a bucket.

        @param bucket: The name of the bucket.
        @return: A C{Deferred} that will fire with the bucket's region.
        """
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name="?location"),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_bucket_location)
        return d


Instruction:
Here you have a function in Python 3, explain what it does
def get_bucket_lifecycle(self, bucket):
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name="?lifecycle"),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_lifecycle_config)
        return d

Response: Get the lifecycle configuration of a bucket.


Instruction:
Write a Python 3 function for
parsing a C{LifecycleConfiguration XML document.

Response:
def _parse_lifecycle_config(self, (response, xml_bytes)):
        """Parse a C{LifecycleConfiguration} XML document."""
        root = XML(xml_bytes)
        rules = []

        for content_data in root.findall("Rule"):
            id = content_data.findtext("ID")
            prefix = content_data.findtext("Prefix")
            status = content_data.findtext("Status")
            expiration = int(content_data.findtext("Expiration/Days"))
            rules.append(
                LifecycleConfigurationRule(id, prefix, status, expiration))

        return LifecycleConfiguration(rules)


Instruction:
How would you explain what the following Python 3 function does
def get_bucket_website_config(self, bucket):
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name='?website'),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_website_config)
        return d

Response: Get the website configuration of a bucket.


Instruction:
Write a Python 3 function that can
parse a C{WebsiteConfiguration XML document.

Response:
def _parse_website_config(self, (response, xml_bytes)):
        """Parse a C{WebsiteConfiguration} XML document."""
        root = XML(xml_bytes)
        index_suffix = root.findtext("IndexDocument/Suffix")
        error_key = root.findtext("ErrorDocument/Key")

        return WebsiteConfiguration(index_suffix, error_key)


Instruction:
Can you create a Python 3 function that
gets the notification configuration of a bucket.

Response:
def get_bucket_notification_config(self, bucket):
        """
        Get the notification configuration of a bucket.

        @param bucket: The name of the bucket.
        @return: A C{Deferred} that will request the bucket's notification
        configuration.
        """
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name="?notification"),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_notification_config)
        return d


Instruction:
Create a Python 3 function for
parsing a C{NotificationConfiguration XML document.

Response:
def _parse_notification_config(self, (response, xml_bytes)):
        """Parse a C{NotificationConfiguration} XML document."""
        root = XML(xml_bytes)
        topic = root.findtext("TopicConfiguration/Topic")
        event = root.findtext("TopicConfiguration/Event")

        return NotificationConfiguration(topic, event)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_bucket_versioning_config(self, bucket):
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name="?versioning"),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_versioning_config)
        return d

Response: Get the versioning configuration of a bucket.


Instruction:
Create a Python 3 function to
parse a C{VersioningConfiguration XML document.

Response:
def _parse_versioning_config(self, (response, xml_bytes)):
        """Parse a C{VersioningConfiguration} XML document."""
        root = XML(xml_bytes)
        mfa_delete = root.findtext("MfaDelete")
        status = root.findtext("Status")

        return VersioningConfiguration(mfa_delete=mfa_delete, status=status)


Instruction:
Explain what the following Python 3 code does
def get_bucket_acl(self, bucket):
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name="?acl"),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_acl)
        return d

Response: Get the access control policy for a bucket.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def put_object(self, bucket, object_name, data=None, content_type=None,
                   metadata={}, amz_headers={}, body_producer=None):
        """
        Put an object in a bucket.

        An existing object with the same name will be replaced.

        @param bucket: The name of the bucket.
        @param object_name: The name of the object.
        @type object_name: L{unicode}
        @param data: The data to write.
        @param content_type: The type of data being written.
        @param metadata: A C{dict} used to build C{x-amz-meta-*} headers.
        @param amz_headers: A C{dict} used to build C{x-amz-*} headers.
        @return: A C{Deferred} that will fire with the result of request.
        """
        details = self._details(
            method=b"PUT",
            url_context=self._url_context(bucket=bucket, object_name=object_name),
            headers=self._headers(content_type),
            metadata=metadata,
            amz_headers=amz_headers,
            body=data,
            body_producer=body_producer,
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(itemgetter(1))
        return d

Response: Put an object in a bucket.


Instruction:
Explain what the following Python 3 code does
def copy_object(self, source_bucket, source_object_name, dest_bucket=None,
                    dest_object_name=None, metadata={}, amz_headers={}):
        """
        Copy an object stored in S3 from a source bucket to a destination
        bucket.

        @param source_bucket: The S3 bucket to copy the object from.
        @param source_object_name: The name of the object to copy.
        @param dest_bucket: Optionally, the S3 bucket to copy the object to.
            Defaults to C{source_bucket}.
        @param dest_object_name: Optionally, the name of the new object.
            Defaults to C{source_object_name}.
        @param metadata: A C{dict} used to build C{x-amz-meta-*} headers.
        @param amz_headers: A C{dict} used to build C{x-amz-*} headers.
        @return: A C{Deferred} that will fire with the result of request.
        """
        dest_bucket = dest_bucket or source_bucket
        dest_object_name = dest_object_name or source_object_name
        amz_headers["copy-source"] = "/%s/%s" % (source_bucket,
                                                 source_object_name)
        details = self._details(
            method=b"PUT",
            url_context=self._url_context(
                bucket=dest_bucket, object_name=dest_object_name,
            ),
            metadata=metadata,
            amz_headers=amz_headers,
        )
        d = self._submit(self._query_factory(details))
        return d

Response: Copy an object stored in S3 from a source bucket to a destination bucket.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_object(self, bucket, object_name):
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name=object_name),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(itemgetter(1))
        return d

Response: Get an object from a bucket.


Instruction:
Here you have a function in Python 3, explain what it does
def head_object(self, bucket, object_name):
        details = self._details(
            method=b"HEAD",
            url_context=self._url_context(bucket=bucket, object_name=object_name),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(lambda (response, body): _to_dict(response.responseHeaders))
        return d

Response: Retrieve object metadata only.


Instruction:
Can you create a Python 3 function that
deletes an object from a bucket.

Response:
def delete_object(self, bucket, object_name):
        """
        Delete an object from a bucket.

        Once deleted, there is no method to restore or undelete an object.
        """
        details = self._details(
            method=b"DELETE",
            url_context=self._url_context(bucket=bucket, object_name=object_name),
        )
        d = self._submit(self._query_factory(details))
        return d


Instruction:
Make a summary of the following Python 3 code
def put_object_acl(self, bucket, object_name, access_control_policy):
        data = access_control_policy.to_xml()
        details = self._details(
            method=b"PUT",
            url_context=self._url_context(
                bucket=bucket, object_name='%s?acl' % (object_name,),
            ),
            body=data,
        )
        query = self._query_factory(details)
        d = self._submit(query)
        d.addCallback(self._parse_acl)
        return d

Response: Set access control policy on an object.


Instruction:
How would you implement a function in Python 3 that
sets request payment configuration on bucket to payer.

Response:
def put_request_payment(self, bucket, payer):
        """
        Set request payment configuration on bucket to payer.

        @param bucket: The name of the bucket.
        @param payer: The name of the payer.
        @return: A C{Deferred} that will fire with the result of the request.
        """
        data = RequestPayment(payer).to_xml()
        details = self._details(
            method=b"PUT",
            url_context=self._url_context(bucket=bucket, object_name="?requestPayment"),
            body=data,
        )
        d = self._submit(self._query_factory(details))
        return d


Instruction:
How would you code a function in Python 3 to
get the request payment configuration on a bucket.

Response:
def get_request_payment(self, bucket):
        """
        Get the request payment configuration on a bucket.

        @param bucket: The name of the bucket.
        @return: A C{Deferred} that will fire with the name of the payer.
        """
        details = self._details(
            method=b"GET",
            url_context=self._url_context(bucket=bucket, object_name="?requestPayment"),
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(self._parse_get_request_payment)
        return d


Instruction:
Can you implement a function in Python 3 that
initiates a multipart upload to a bucket.

Response:
def init_multipart_upload(self, bucket, object_name, content_type=None,
                              amz_headers={}, metadata={}):
        """
        Initiate a multipart upload to a bucket.

        @param bucket: The name of the bucket
        @param object_name: The object name
        @param content_type: The Content-Type for the object
        @param metadata: C{dict} containing additional metadata
        @param amz_headers: A C{dict} used to build C{x-amz-*} headers.
        @return: C{str} upload_id
        """
        objectname_plus = '%s?uploads' % object_name
        details = self._details(
            method=b"POST",
            url_context=self._url_context(bucket=bucket, object_name=objectname_plus),
            headers=self._headers(content_type),
            metadata=metadata,
            amz_headers=amz_headers,
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(
            lambda (response, body): MultipartInitiationResponse.from_xml(body)
        )
        return d


Instruction:
Explain what the following Python 3 code does
def upload_part(self, bucket, object_name, upload_id, part_number,
                    data=None, content_type=None, metadata={},
                    body_producer=None):
        """
        Upload a part of data corresponding to a multipart upload.

        @param bucket: The bucket name
        @param object_name: The object name
        @param upload_id: The multipart upload id
        @param part_number: The part number
        @param data: Data (optional, requires body_producer if not specified)
        @param content_type: The Content-Type
        @param metadata: Additional metadata
        @param body_producer: an C{IBodyProducer} (optional, requires data if
            not specified)
        @return: the C{Deferred} from underlying query.submit() call
        """
        parms = 'partNumber=%s&uploadId=%s' % (str(part_number), upload_id)
        objectname_plus = '%s?%s' % (object_name, parms)
        details = self._details(
            method=b"PUT",
            url_context=self._url_context(bucket=bucket, object_name=objectname_plus),
            headers=self._headers(content_type),
            metadata=metadata,
            body=data,
        )
        d = self._submit(self._query_factory(details))
        d.addCallback(lambda (response, data): _to_dict(response.responseHeaders))
        return d

Response: Upload a part of data corresponding to a multipart upload.


Instruction:
Here you have a function in Python 3, explain what it does
def complete_multipart_upload(self, bucket, object_name, upload_id,
                                  parts_list, content_type=None, metadata={}):
        """
        Complete a multipart upload.

        N.B. This can be possibly be a slow operation.

        @param bucket: The bucket name
        @param object_name: The object name
        @param upload_id: The multipart upload id
        @param parts_list: A List of all the parts
            (2-tuples of part sequence number and etag)
        @param content_type: The Content-Type of the object
        @param metadata: C{dict} containing additional metadata
        @return: a C{Deferred} that fires after request is complete
        """
        data = self._build_complete_multipart_upload_xml(parts_list)
        objectname_plus = '%s?uploadId=%s' % (object_name, upload_id)
        details = self._details(
            method=b"POST",
            url_context=self._url_context(bucket=bucket, object_name=objectname_plus),
            headers=self._headers(content_type),
            metadata=metadata,
            body=data,
        )
        d = self._submit(self._query_factory(details))
        # TODO - handle error responses
        d.addCallback(
            lambda (response, body): MultipartCompletionResponse.from_xml(body)
        )
        return d

Response: Complete a multipart upload.


Instruction:
Write a Python 3 function for
setting the content type based on the file extension used in the object.

Response:
def set_content_type(self):
        """
        Set the content type based on the file extension used in the object
        name.
        """
        if self.object_name and not self.content_type:
            # XXX nothing is currently done with the encoding... we may
            # need to in the future
            self.content_type, encoding = mimetypes.guess_type(
                self.object_name, strict=False)


Instruction:
How would you implement a function in Python 3 that
builds the list of headers needed in order to perform S3 operations.

Response:
def get_headers(self, instant):
        """
        Build the list of headers needed in order to perform S3 operations.
        """
        headers = {'x-amz-date': _auth_v4.makeAMZDate(instant)}
        if self.body_producer is None:
            data = self.data
            if data is None:
                data = b""
            headers["x-amz-content-sha256"] = hashlib.sha256(data).hexdigest()
        else:
            data = None
            headers["x-amz-content-sha256"] = b"UNSIGNED-PAYLOAD"
        for key, value in self.metadata.iteritems():
            headers["x-amz-meta-" + key] = value
        for key, value in self.amz_headers.iteritems():
            headers["x-amz-" + key] = value

        # Before we check if the content type is set, let's see if we can set
        # it by guessing the the mimetype.
        self.set_content_type()
        if self.content_type is not None:
            headers["Content-Type"] = self.content_type
        if self.creds is not None:
            headers["Authorization"] = self.sign(
                headers,
                data,
                s3_url_context(self.endpoint, self.bucket, self.object_name),
                instant,
                method=self.action)
        return headers


Instruction:
Can you tell what is the following Python 3 function doing
def sign(self, headers, data, url_context, instant, method,
             region=REGION_US_EAST_1):
        """Sign this query using its built in credentials."""
        headers["host"] = url_context.get_encoded_host()

        if data is None:
            request = _auth_v4._CanonicalRequest.from_request_components(
                method=method,
                url=url_context.get_encoded_path(),
                headers=headers,
                headers_to_sign=('host', 'x-amz-date'),
                payload_hash=None,
            )
        else:
            request = _auth_v4._CanonicalRequest.from_request_components_and_payload(
                method=method,
                url=url_context.get_encoded_path(),
                headers=headers,
                headers_to_sign=('host', 'x-amz-date'),
                payload=data,
            )

        return _auth_v4._make_authorization_header(
            region=region,
            service="s3",
            canonical_request=request,
            credentials=self.creds,
            instant=instant)

Response: Sign this request using the built in credentials.


Instruction:
Explain what the following Python 3 code does
def submit(self, url_context=None, utcnow=datetime.datetime.utcnow):
        if not url_context:
            url_context = s3_url_context(
                self.endpoint, self.bucket, self.object_name)
        d = self.get_page(
            url_context.get_encoded_url(),
            method=self.action,
            postdata=self.data or b"",
            headers=self.get_headers(utcnow()),
        )

        return d.addErrback(s3_error_wrapper)

Response:
Submit this query.

        @return: A deferred from get_page


Instruction:
Make a summary of the following Python 3 code
def attributes(self):
        if 'id' in self.node.attrib:
            yield PlaceholderAttribute('id', self.node.attrib['id'])

        if 'tei-tag' in self.node.attrib:
            yield PlaceholderAttribute('tei-tag', self.node.attrib['tei-tag'])

        """Contain attributes applicable to this element"""
        for attributes in self.node.iterchildren('attributes'):
            for attribute in self.__iter_attributes__(attributes):
                yield attribute

Response: Iterate over the attributes applicable to this element.


Instruction:
How would you implement a function in Python 3 that
yields all the text divisions of this element.

Response:
def divisions(self):
        """
        Recursively get all the text divisions directly part of this element. If an element contains parts or text without tag. Those will be returned in order and wrapped with a TextDivision.
        """

        from .placeholder_division import PlaceholderDivision

        placeholder = None
        for item in self.__parts_and_divisions:
            if item.tag == 'part':
                if not placeholder:
                    placeholder = PlaceholderDivision()
                placeholder.parts.append(item)
            else:
                if placeholder:
                    yield placeholder
                    placeholder = None
                yield item
        if placeholder:
            yield placeholder


Instruction:
Implement a Python 3 function for
getting the parts below this element.

Response:
def parts(self):
        """
        Get the parts directly below this element.
        """

        for item in self.__parts_and_divisions:
            if item.tag == 'part':
                yield item
            else:
                # Divisions shouldn't be beneath a part, but here's a fallback
                # for if this does happen
                for part in item.parts:
                    yield part


Instruction:
Write a Python 3 script to
yield the parts and divisions of this element.

Response:
def __parts_and_divisions(self):
        """
        The parts and divisions directly part of this element.
        """

        from .division import Division
        from .part import Part
        from .placeholder_part import PlaceholderPart

        text = self.node.text
        if text:
            stripped_text = text.replace('\n', '')
            if stripped_text.strip():
                yield PlaceholderPart(stripped_text)

        for item in self.node:
            if item.tag == 'part':
                yield Part(item)
            elif item.tag == 'div':
                yield Division(item)

            if item.tail:
                stripped_tail = item.tail.replace('\n', '')
                if stripped_tail.strip():
                    yield PlaceholderPart(stripped_tail)


Instruction:
Write a Python 3 function for
converting an element to a single string and allow the passed inject method to place content before any element.

Response:
def tostring(self, inject):
        """
        Convert an element to a single string and allow the passed inject method to place content before any
        element.
        """

        return inject(self, '\n'.join(f'{division.tostring(inject)}' for division in self.divisions))


Instruction:
How would you explain what the following Python 3 function does
def RCL(input_shape,
        rec_conv_layers,
        dense_layers,
        output_layer=[1, 'sigmoid'],
        padding='same',
        optimizer='adam',
        loss='binary_crossentropy'):
    """Summary

    Args:
        input_shape (tuple): The shape of the input layer.
        output_nodes (int): Number of nodes in the output layer. It depends on the loss function used.
        rec_conv_layers (list): RCL descriptor
                                [
                                  [
                                    [(filter, kernel), (pool_size, stride), leak, drop],
                                    [(filter, kernel), (pool_size, stride), leak, drop],
                                    [(filter, kernel), (pool_size, stride), leak, drop, timesteps],
                                  ],
                                  ...
                                  [
                                    [],[],[]
                                  ]
                                ]
        dense_layers (TYPE): Dense layer descriptor [[fully_connected, leak, drop], ... []]
        padding (str, optional): Type of padding for conv and pooling layers
        optimizer (str or object optional): Keras optimizer as string or keras optimizer

    Returns:
        model: The compiled Kears model, ready for training.
    """

    inputs = Input(shape=input_shape)

    for i, c in enumerate(rec_conv_layers):

        conv = Conv1D(c[0][0][0], c[0][0][1], padding=padding)(inputs)
        batch = BatchNormalization()(conv)
        act = LeakyReLU(alpha=c[0][2])(batch)
        pool = MaxPooling1D(pool_size=c[0][1][0], strides=c[0][1][1], padding=padding)(act)
        d1 = Dropout(c[0][3])(pool)

        inner = time_steps( input=d1,
                            conv_layer=c[1],
                            time_conv_layer=c[2],
                            padding=padding)

    drop = Flatten()(inner)

    for i, d in enumerate(dense_layers):
        dense = Dense(d[0], activation='relu')(drop)
        bn = BatchNormalization()(dense)
        act = LeakyReLU(alpha=d[1])(bn)
        drop = Dropout(d[2])(act)

    output = Dense(output_layer[0], activation=output_layer[1])(drop)

    model = Model(inputs=inputs, outputs=output)

    model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

    return model

Response: Summary function for training a new RCL model.


Instruction:
Can you generate the documentation for the following Python 3 function
def VOICE(input_shape,
           conv_layers,
           dense_layers,
           output_layer=[1, 'sigmoid'],
           padding='same',
           optimizer='adam',
           loss='binary_crossentropy'):
    """Conv1D CNN used primarily for voice data.

    Args:
        input_shape (tuple): The shape of the input layer
        targets (int): Number of targets
        conv_layers (list): Conv layer descriptor [[(filter, kernel), (pool_size, stride), leak, drop], ... []]
        dense_layers (TYPE): Dense layer descriptor [[fully_connected, leak, drop]]
        padding (str, optional): Type of padding for conv and pooling layers
        optimizer (str or object optional): Keras optimizer as string or keras optimizer

    Returns:
        TYPE: model, build_arguments
    """

    inputs = Input(shape=input_shape)

    for i, c in enumerate(conv_layers):
        if i == 0:
            conv = Conv1D(c[0][0], c[0][1], padding=padding)(inputs)
        else:
            conv = Conv1D(c[0][0], c[0][1], padding=padding)(drop)
        bn = BatchNormalization()(conv)
        act = LeakyReLU(alpha=c[2])(bn)
        pool = MaxPooling1D(pool_size=c[1][0], strides=c[1][1], padding=padding)(act)
        drop = Dropout(c[3])(pool)

    drop = Flatten()(drop)

    for i, d in enumerate(dense_layers):
        dense = Dense(d[0], activation='relu')(drop)
        bn = BatchNormalization()(dense)
        act = LeakyReLU(alpha=d[1])(bn)
        drop = Dropout(d[2])(act)

    output = Dense(output_layer[0], activation=output_layer[1])(drop)

    model = Model(inputs=inputs, outputs=output)
    model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

    return model

Response: Function to build a single voice data base on the input_shape.


Instruction:
Write a Python 3 function that can
initialize the ErrorMeasure. During initialization, all :py:meth:`BaseErrorMeasure.local_errors` are calculated. :param TimeSeries originalTimeSeries: TimeSeries containing the original data. :param TimeSeries calculatedTimeSeries: TimeSeries containing calculated data. Calculated data is smoothed or forecasted data. :return: Return :py:const:`True` if the error could be calculated, :py:const:`False` otherwise based on the minimalErrorCalculationPercentage. :rtype: boolean :raise: Raises a :py:exc:`StandardError` if the error measure is initialized multiple times.

Response:
def initialize(self, originalTimeSeries, calculatedTimeSeries):
        """Initializes the ErrorMeasure.

        During initialization, all :py:meth:`BaseErrorMeasure.local_errors` are calculated.

        :param TimeSeries originalTimeSeries:    TimeSeries containing the original data.
        :param TimeSeries calculatedTimeSeries:    TimeSeries containing calculated data.
            Calculated data is smoothed or forecasted data.

        :return:    Return :py:const:`True` if the error could be calculated, :py:const:`False`
            otherwise based on the minimalErrorCalculationPercentage.
        :rtype: boolean

        :raise:    Raises a :py:exc:`StandardError` if the error measure is initialized multiple times.
        """
        # ErrorMeasure was already initialized.
        if 0 < len(self._errorValues):
            raise StandardError("An ErrorMeasure can only be initialized once.")

        # sort the TimeSeries to reduce the required comparison operations
        originalTimeSeries.sort_timeseries()
        calculatedTimeSeries.sort_timeseries()

        # Performance optimization
        append      = self._errorValues.append
        appendDate  = self._errorDates.append
        local_error = self.local_error

        minCalcIdx  = 0

        # calculate all valid local errors
        for orgPair in originalTimeSeries:
            for calcIdx in xrange(minCalcIdx, len(calculatedTimeSeries)):
                calcPair = calculatedTimeSeries[calcIdx]

                # Skip values that can not be compared
                if calcPair[0] != orgPair[0]:
                    continue

                append(local_error(orgPair[1:], calcPair[1:]))
                appendDate(orgPair[0])

        # return False, if the error cannot be calculated
        calculatedErrors    = len(filter(lambda item: item is not None, self._errorValues))
        minCalculatedErrors = self._minimalErrorCalculationPercentage * len(originalTimeSeries)

        if  calculatedErrors < minCalculatedErrors:
            self._errorValues = []
            self._errorDates = []
            return False

        return True


Instruction:
Create a Python 3 function to
return the defined subset of self. _errorValues.

Response:
def _get_error_values(self, startingPercentage, endPercentage, startDate, endDate):
        """Gets the defined subset of self._errorValues.

        Both parameters will be correct at this time.

        :param float startingPercentage: Defines the start of the interval. This has to be a value in [0.0, 100.0].
            It represents the value, where the error calculation should be started.
            25.0 for example means that the first 25% of all calculated errors will be ignored.
        :param float endPercentage:    Defines the end of the interval. This has to be a value in [0.0, 100.0].
            It represents the value, after which all error values will be ignored. 90.0 for example means that
            the last 10% of all local errors will be ignored.
        :param float startDate: Epoch representing the start date used for error calculation.
        :param float endDate: Epoch representing the end date used in the error calculation.

        :return:    Returns a list with the defined error values.
        :rtype: list

        :raise:    Raises a ValueError if startDate or endDate do not represent correct boundaries for error calculation.
        """
        if startDate is not None:
            possibleDates = filter(lambda date: date >= startDate, self._errorDates)
            if 0 == len(possibleDates):
                raise ValueError("%s does not represent a valid startDate." % startDate)

            startIdx = self._errorDates.index(min(possibleDates))
        else:
            startIdx = int((startingPercentage * len(self._errorValues)) / 100.0)

        if endDate is not None:
            possibleDates = filter(lambda date: date <= endDate, self._errorDates)
            if 0 == len(possibleDates):
                raise ValueError("%s does not represent a valid endDate." % endDate)

            endIdx = self._errorDates.index(max(possibleDates)) + 1
        else:
            endIdx = int((endPercentage * len(self._errorValues)) / 100.0)

        return self._errorValues[startIdx:endIdx]


Instruction:
Write a Python 3 function that can
calculate the error for the given interval startingPercentage and endPercentage.

Response:
def get_error(self, startingPercentage=0.0, endPercentage=100.0, startDate=None, endDate=None):
        """Calculates the error for the given interval (startingPercentage, endPercentage) between the TimeSeries
        given during :py:meth:`BaseErrorMeasure.initialize`.

        :param float startingPercentage: Defines the start of the interval. This has to be a value in [0.0, 100.0].
            It represents the value, where the error calculation should be started.
            25.0 for example means that the first 25% of all calculated errors will be ignored.
        :param float endPercentage:    Defines the end of the interval. This has to be a value in [0.0, 100.0].
            It represents the value, after which all error values will be ignored. 90.0 for example means that
            the last 10% of all local errors will be ignored.
        :param float startDate: Epoch representing the start date used for error calculation.
        :param float endDate: Epoch representing the end date used in the error calculation.

        :return:    Returns a float representing the error.
        :rtype: float

        :raise:    Raises a :py:exc:`ValueError` in one of the following cases:

            - startingPercentage not in [0.0, 100.0]
            - endPercentage      not in [0.0, 100.0]
            - endPercentage < startingPercentage

        :raise:    Raises a :py:exc:`StandardError` if :py:meth:`BaseErrorMeasure.initialize` was not successfull before.
        """
        # not initialized:
        if len(self._errorValues) == 0:
            raise StandardError("The last call of initialize(...) was not successfull.")

        # check for wrong parameters
        if not (0.0 <= startingPercentage <= 100.0):
            raise ValueError("startingPercentage has to be in [0.0, 100.0].")
        if not (0.0 <= endPercentage <= 100.0):
            raise ValueError("endPercentage has to be in [0.0, 100.0].")
        if endPercentage < startingPercentage:
            raise ValueError("endPercentage has to be greater or equal than startingPercentage.")

        return self._calculate(startingPercentage, endPercentage, startDate, endDate)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def confidence_interval(self, confidenceLevel):

        if not (confidenceLevel >= 0 and confidenceLevel <= 1):
            raise ValueError("Parameter percentage has to be in [0,1]")

        underestimations = []
        overestimations = []
        for error in self._errorValues:
            if error is None:
                # None was in the lists causing some confidenceLevels not be calculated, not sure if that was intended, I suggested ignoring None values
                continue
            #Want 0 errors in both lists!
            if error >= 0:
                overestimations.append(error)
            if error <= 0:
                underestimations.append(error)

        #sort and cut off at confidence level.
        overestimations.sort()
        underestimations.sort(reverse=True)

        overIdx  = int(len(overestimations) * confidenceLevel) - 1
        underIdx = int(len(underestimations) * confidenceLevel) - 1

        overestimation  = 0.0
        underestimation = 0.0

        if overIdx >= 0:
            overestimation = overestimations[overIdx]
        else:
            print len(overestimations), confidenceLevel

        if underIdx >= 0:
            underestimation = underestimations[underIdx]

        return underestimation, overestimation

Response: Calculates the interval of the underestimations and overestimations for the given error level.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load_segmented_data(filename):
    data = pd.read_csv(filename, index_col=0)
    data.index = data.index.astype(np.datetime64)

    return data

Response: Helper function to load segmented gait time series data.


Instruction:
Can you tell what is the following Python 3 function doing
def load_finger_tapping_cloudupdrs_data(filename, convert_times=1000.0):
    data_m = np.genfromtxt(filename, delimiter=',', invalid_raise=False, skip_footer=1)
    date_times = pd.to_datetime((data_m[:, 0] - data_m[0, 0]))
    time_difference = (data_m[:, 0] - data_m[0, 0]) / convert_times
    data = {'td': time_difference, 'action_type': data_m[:, 2],'x': data_m[:, 3], 'y': data_m[:, 4],
            'x_target': data_m[:, 7], 'y_target': data_m[:, 8]}
    data_frame = pd.DataFrame(data, index=date_times, columns=['td', 'action_type','x', 'y', 'x_target', 'y_target'])
    return data_frame

Response: This method loads the data from a CSV file and returns a DataFrame with the data in the cloudupdrs format.


Instruction:
Can you generate the documentation for the following Python 3 function
def numerical_integration(signal, sampling_frequency):

    integrate = sum(signal[1:]) / sampling_frequency + sum(signal[:-1])
    integrate /= sampling_frequency * 2

    return np.array(integrate)

Response: Numerically integrates a signal with it s sampling frequency.


Instruction:
Can you write a function in Python 3 where it
finds the local maxima and minima (peaks) in a 1-dimensional signal. Converted from MATLAB script <http://billauer.co.il/peakdet.html> :param array signal: A 1-dimensional array or list (the signal). :type signal: array :param delta: The peak threashold. A point is considered a maximum peak if it has the maximal value, and was preceded (to the left) by a value lower by delta. :type delta: float :param x: Indices in local maxima and minima are replaced with the corresponding values in x (None default). :type x: array :return maxtab: The highest peaks. :rtype maxtab: numpy.ndarray :return mintab: The lowest peaks. :rtype mintab: numpy.ndarray

Response:
def peakdet(signal, delta, x=None):
    """
        Find the local maxima and minima (peaks) in a 1-dimensional signal.
        Converted from MATLAB script <http://billauer.co.il/peakdet.html>

        :param array signal: A 1-dimensional array or list (the signal).
        :type signal: array
        :param delta: The peak threashold. A point is considered a maximum peak if it has the maximal value, and was preceded (to the left) by a value lower by delta.
        :type delta: float
        :param x: Indices in local maxima and minima are replaced with the corresponding values in x (None default).
        :type x: array
        :return maxtab: The highest peaks.
        :rtype maxtab: numpy.ndarray
        :return mintab: The lowest peaks.
        :rtype mintab: numpy.ndarray
    """

    maxtab = []
    mintab = []

    if x is None:
        x = np.arange(len(signal))

    v = np.asarray(signal)

    if len(v) != len(x):
        sys.exit('Input vectors v and x must have same length')

    if not np.isscalar(delta):
        sys.exit('Input argument delta must be a scalar')

    if delta <= 0:
        sys.exit('Input argument delta must be positive')

    mn, mx = np.inf, -np.inf
    mnpos, mxpos = np.nan, np.nan

    lookformax = True

    for i in np.arange(len(v)):
        this = v[i]
        if this > mx:
            mx = this
            mxpos = x[i]
        if this < mn:
            mn = this
            mnpos = x[i]

        if lookformax:
            if this < mx - delta:
                maxtab.append((mxpos, mx))
                mn = this
                mnpos = x[i]
                lookformax = False
        else:
            if this > mn + delta:
                mintab.append((mnpos, mn))
                mx = this
                mxpos = x[i]
                lookformax = True

    return np.array(maxtab), np.array(mintab)


Instruction:
Can you implement a function in Python 3 that
computes number of samples between signal peaks using the real part of FFT.

Response:
def compute_interpeak(data, sample_rate):
    """
        Compute number of samples between signal peaks using the real part of FFT.

        :param data: 1-dimensional time series data.
        :type data: array
        :param sample_rate: Sample rate of accelerometer reading (Hz)
        :type sample_rate: float
        :return interpeak: Number of samples between peaks
        :rtype interpeak: int

        :Examples:

        >>> import numpy as np
        >>> from mhealthx.signals import compute_interpeak
        >>> data = np.random.random(10000)
        >>> sample_rate = 100
        >>> interpeak = compute_interpeak(data, sample_rate)
    """

    # Real part of FFT:
    freqs = fftfreq(data.size, d=1.0/sample_rate)
    f_signal = rfft(data)

    # Maximum non-zero frequency:
    imax_freq = np.argsort(f_signal)[-2]
    freq = np.abs(freqs[imax_freq])

    # Inter-peak samples:
    interpeak = np.int(np.round(sample_rate / freq))

    return interpeak


Instruction:
Can you generate a brief explanation for the following Python 3 code
def butter_lowpass_filter(data, sample_rate, cutoff=10, order=4, plot=False):

    nyquist = 0.5 * sample_rate
    normal_cutoff = cutoff / nyquist
    b, a = butter(order, normal_cutoff, btype='low', analog=False)

    if plot:
        w, h = freqz(b, a, worN=8000)
        plt.subplot(2, 1, 1)
        plt.plot(0.5*sample_rate*w/np.pi, np.abs(h), 'b')
        plt.plot(cutoff, 0.5*np.sqrt(2), 'ko')
        plt.axvline(cutoff, color='k')
        plt.xlim(0, 0.5*sample_rate)
        plt.title("Lowpass Filter Frequency Response")
        plt.xlabel('Frequency [Hz]')
        plt.grid()
        plt.show()

    y = lfilter(b, a, data)

    return y

Response: Low - pass filter for the given sample rate and cutoff.


Instruction:
Write a Python 3 script for
finding indices of zero crossings from positive to negative values <http://stackoverflow. com/questions / 338248 / efficiently - detect - sign - changes - in - python > _

Response:
def crossings_nonzero_pos2neg(data):
    """
        Find `indices of zero crossings from positive to negative values <http://stackoverflow.com/questions/3843017/efficiently-detect-sign-changes-in-python>`_.

        :param data: numpy array of floats
        :type data: numpy array of floats
        :return crossings: crossing indices to data
        :rtype crossings: numpy array of integers

        :Examples:

        >>> import numpy as np
        >>> from mhealthx.signals import crossings_nonzero_pos2neg
        >>> data = np.random.random(100)
        >>> crossings = crossings_nonzero_pos2neg(data)

    """
    import numpy as np

    if isinstance(data, np.ndarray):
        pass
    elif isinstance(data, list):
        data = np.asarray(data)
    else:
        raise IOError('data should be a numpy array')

    pos = data > 0

    crossings = (pos[:-1] & ~pos[1:]).nonzero()[0]

    return crossings


Instruction:
Can you generate the documentation for the following Python 3 function
def autocorrelate(data, unbias=2, normalize=2):

    # Autocorrelation:
    coefficients = correlate(data, data, 'full')
    size = np.int(coefficients.size/2)
    coefficients = coefficients[size:]
    N = coefficients.size

    # Unbiased:
    if unbias:
        if unbias == 1:
            coefficients /= (N - np.arange(N))
        elif unbias == 2:
            coefficient_ratio = coefficients[0]/coefficients[-1]
            coefficients /= np.linspace(coefficient_ratio, 1, N)
        else:
            raise IOError("unbias should be set to 1, 2, or None")

    # Normalize:
    if normalize:
        if normalize == 1:
            coefficients /= np.abs(coefficients[0])
        elif normalize == 2:
            coefficients /= np.max(np.abs(coefficients))
        else:
            raise IOError("normalize should be set to 1, 2, or None")

    return coefficients, N

Response: Compute the autocorrelation coefficients for a time series data.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_signal_peaks_and_prominences(data):
    peaks, _ = sig.find_peaks(data)
    prominences = sig.peak_prominences(data, peaks)[0]

    return peaks, prominences

Response: Get the signal peaks and peak prominences.


Instruction:
Can you tell what is the following Python 3 function doing
def smoothing_window(data, window=[1, 1, 1]):

    for i in range(len(data) - sum(window)):

        start_window_from = i
        start_window_to = i+window[0]

        end_window_from = start_window_to + window[1]
        end_window_to = end_window_from + window[2]

        if np.all(data[start_window_from: start_window_to] == data[end_window_from: end_window_to]):
            data[start_window_from: end_window_to] = data[start_window_from]

    return data

Response: This function is a smoother function so we can fix misclassifications. It will take a sliding window of the data and apply the border to the middle of the original data.


Instruction:
Explain what the following Python 3 code does
def plot_segmentation(data, peaks, segment_indexes, figsize=(10, 5)):
    fig, ax = plt.subplots(figsize=figsize)
    plt.plot(data);

    for segment in np.unique(segment_indexes):
        plt.plot(peaks[np.where(segment_indexes == segment)[0]], data[peaks][np.where(segment_indexes == segment)[0]], 'o')

    plt.show()

Response: Plots the data and segmentation based on the peaks and segment indexes.


Instruction:
Given the following Python 3 function, write the documentation
def centroid_sort(centroids):

    dimensions = len(centroids[0])
    negative_base_point = array(dimensions*[-100])

    decorated = [
        (euclidean(centroid, negative_base_point), centroid)
        for centroid in centroids
    ]
    decorated.sort()

    return array([centroid for dist, centroid in decorated])

Response: Sort the centroids of the current set of cluster centroids.


Instruction:
Here you have a function in Python 3, explain what it does
def non_zero_index(arr):
    for index, row in enumerate(arr):
        if non_zero_row(row):
            return index
    raise ValueError('No non-zero values')

Response: Returns the index of the first non - zero entry in an array.


Instruction:
Implement a function in Python 3 to
return True if the row is completely free of zeros.

Response:
def non_zero_row(arr):
    """
        0.  Empty row returns False.

            >>> arr = array([])
            >>> non_zero_row(arr)

            False

        1.  Row with a zero returns False.

            >>> arr = array([1, 4, 3, 0, 5, -1, -2])
            >>> non_zero_row(arr)

            False
        2.  Row with no zeros returns True.

            >>> arr = array([-1, -0.1, 0.001, 2])
            >>> non_zero_row(arr)

            True

        :param arr: array
        :type arr: numpy array
        :return empty: If row is completely free of zeros
        :rtype empty: bool
    """

    if len(arr) == 0:
        return False

    for item in arr:
        if item == 0:
            return False

    return True


Instruction:
Implement a Python 3 function for
generating indexes for sliding windows with overlap

Response:
def window_features(idx, window_size=100, overlap=10):
    """ Generate indexes for a sliding window with overlap

        :param array idx: The indexes that need to be windowed.
        :param int window_size: The size of the window.
        :param int overlap: How much should each window overlap.

        :return array view: The indexes for the windows with overlap.
    """
    overlap = window_size - overlap
    sh = (idx.size - window_size + 1, window_size)
    st = idx.strides * 2
    view = np.lib.stride_tricks.as_strided(idx, strides=st, shape=sh)[0::overlap]

    return view


Instruction:
Create a Python 3 function to
return the version info from pypi for this app.

Response:
def get_pypi_version():
    """
    Returns the version info from pypi for this app.
    """
    try:
        response = requests.get(PYPI_URL, timeout=HALF_SECOND_TIMEOUT)
        response.raise_for_status()
        data = response.json()
        version_str = data["info"]["version"]
        return _parse_version_str(version_str)
    except requests.exceptions.ConnectionError:
        raise VersionException(UNABLE_TO_ACCESS_PYPI + " Failed to connect.")
    except requests.exceptions.Timeout:
        raise VersionException(UNABLE_TO_ACCESS_PYPI + " Timeout")


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_project_by_id(self, project_id):
        return self._create_item_response(
            self.data_service.get_project_by_id(project_id),
            Project)

Response: Get details about project with the specified uuid.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def create_project(self, name, description):
        return self._create_item_response(
            self.data_service.create_project(name, description),
            Project)

Response: Create a new project with the specified name and description.


Instruction:
Given the following Python 3 function, write the documentation
def create_folder(self, folder_name, parent_kind_str, parent_uuid):
        return self._create_item_response(
            self.data_service.create_folder(folder_name, parent_kind_str, parent_uuid),
            Folder
        )

Response: Create a folder under a particular parent


Instruction:
Write a Python 3 script for
getting direct files and folders of a project.

Response:
def get_project_children(self, project_id, name_contains=None):
        """
        Get direct files and folders of a project.
        :param project_id: str: uuid of the project to list contents
        :param name_contains: str: filter children based on a pattern
        :return: [File|Folder]: list of Files/Folders contained by the project
        """
        return self._create_array_response(
            self.data_service.get_project_children(
                project_id, name_contains
            ),
            DDSConnection._folder_or_file_constructor
        )


Instruction:
Can you tell what is the following Python 3 function doing
def get_folder_children(self, folder_id, name_contains=None):
        return self._create_array_response(
            self.data_service.get_folder_children(
                folder_id, name_contains
            ),
            DDSConnection._folder_or_file_constructor
        )

Response: Get direct files and folders of a folder.


Instruction:
How would you code a function in Python 3 to
get a FileDownload object that contains temporary url settings needed to download the contents of a file.

Response:
def get_file_download(self, file_id):
        """
        Get a file download object that contains temporary url settings needed to download the contents of a file.
        :param file_id: str: uuid of the file
        :return: FileDownload
        """
        return self._create_item_response(
            self.data_service.get_file_url(file_id),
            FileDownload
        )


Instruction:
Can you tell what is the following Python 3 function doing
def upload_file(self, local_path, project_id, parent_data, existing_file_id=None, remote_filename=None):
        path_data = PathData(local_path)
        hash_data = path_data.get_hash()
        file_upload_operations = FileUploadOperations(self.data_service, None)
        upload_id = file_upload_operations.create_upload(project_id, path_data, hash_data,
                                                         remote_filename=remote_filename,
                                                         storage_provider=self.config.storage_provider_id)
        context = UploadContext(self.config, self.data_service, upload_id, path_data)
        ParallelChunkProcessor(context).run()
        remote_file_data = file_upload_operations.finish_upload(upload_id, hash_data, parent_data, existing_file_id)
        return File(self, remote_file_data)

Response: Uploads a file under a specific location in DDSConnection possibly replacing an existing file.


Instruction:
Given the following Python 3 function, write the documentation
def _folder_or_file_constructor(dds_connection, data_dict):
        kind = data_dict['kind']
        if kind == KindType.folder_str:
            return Folder(dds_connection, data_dict)
        elif data_dict['kind'] == KindType.file_str:
            return File(dds_connection, data_dict)

Response: Create a File or Folder based on the kind value in data_dict.


Instruction:
Write a Python 3 script to
get details for a folder id.

Response:
def get_folder_by_id(self, folder_id):
        """
        Get folder details for a folder id.
        :param folder_id: str: uuid of the folder
        :return: Folder
        """
        return self._create_item_response(
            self.data_service.get_folder(folder_id),
            Folder
        )


Instruction:
Can you generate the documentation for the following Python 3 function
def get_file_by_id(self, file_id):
        return self._create_item_response(
            self.data_service.get_file(file_id),
            File
        )

Response: Get file details by id.


Instruction:
Implement a Python 3 function for
saving the contents of the remote file to a local file.

Response:
def save_to_path(self, file_path, chunk_size=DOWNLOAD_FILE_CHUNK_SIZE):
        """
        Save the contents of the remote file to a local path.
        :param file_path: str: file path
        :param chunk_size: chunk size used to write local file
        """
        response = self._get_download_response()
        with open(file_path, 'wb') as f:
            for chunk in response.iter_content(chunk_size=chunk_size):
                if chunk:  # filter out keep-alive new chunks
                    f.write(chunk)


Instruction:
Here you have a function in Python 3, explain what it does
def get_child(self):
        path_parts = self.remote_path.split(os.sep)
        return self._get_child_recurse(path_parts, self.node)

Response: Find file or folder at the remote_path


Instruction:
Can you write a function in Python 3 where it
adds this task to the lookup based on it s wait_for_task_id property.

Response:
def add(self, task):
        """
        Add this task to the lookup based on it's wait_for_task_id property.
        :param task: Task: task to add to the list
        """
        wait_id = task.wait_for_task_id
        task_list = self.wait_id_to_task.get(wait_id, [])
        task_list.append(task)
        self.wait_id_to_task[wait_id] = task_list


Instruction:
How would you code a function in Python 3 to
create a task for the command that will wait for parent_task_id before starting.

Response:
def add(self, parent_task_id, command):
        """
        Create a task for the command that will wait for parent_task_id before starting.
        :param parent_task_id: int: id of task to wait for or None if it can start immediately
        :param command: TaskCommand: contains data function to run
        :return: int: task id we created for this command
        """
        task_id = self._claim_next_id()
        self.waiting_task_list.add(Task(task_id, parent_task_id, command))
        return task_id


Instruction:
Can you generate the documentation for the following Python 3 function
def run(self):
        for task in self.get_next_tasks(None):
            self.executor.add_task(task, None)
        while not self.executor.is_done():
            done_task_and_result = self.executor.wait_for_tasks()
            for task, task_result in done_task_and_result:
                self._add_sub_tasks_to_executor(task, task_result)

Response: Runs all tasks in this runner on the executor.


Instruction:
Create a Python 3 function to
add all subtasks for parent_task to the executor.

Response:
def _add_sub_tasks_to_executor(self, parent_task, parent_task_result):
        """
        Add all subtasks for parent_task to the executor.
        :param parent_task: Task: task that has just finished
        :param parent_task_result: object: result of task that is finished
        """
        for sub_task in self.waiting_task_list.get_next_tasks(parent_task.id):
            self.executor.add_task(sub_task, parent_task_result)


Instruction:
Explain what the following Python 3 code does
def add_task(self, task, parent_task_result):
        self.tasks.append((task, parent_task_result))
        self.task_id_to_task[task.id] = task

Response: Add a task to run with the specified result from this tasks parent


Instruction:
Can you tell what is the following Python 3 function doing
def wait_for_tasks(self):
        finished_tasks_and_results = []
        while len(finished_tasks_and_results) == 0:
            if self.is_done():
                break
            self.start_tasks()
            self.process_all_messages_in_queue()
            finished_tasks_and_results = self.get_finished_results()
        return finished_tasks_and_results

Response: Wait for one or more tasks to finish or return empty list.


Instruction:
Can you tell what is the following Python 3 function doing
def start_tasks(self):
        while self.tasks_at_once > len(self.pending_results) and self._has_more_tasks():
            task, parent_result = self.tasks.popleft()
            self.execute_task(task, parent_result)

Response: Start tasks until we have enough results.


Instruction:
Can you create a Python 3 function that
executes a single task in another process saving the result to our pending_results list.

Response:
def execute_task(self, task, parent_result):
        """
        Run a single task in another process saving the result to our list of pending results.
        :param task: Task: function and data we can run in another process
        :param parent_result: object: result from our parent task
        """
        task.before_run(parent_result)
        context = task.create_context(self.message_queue)
        pending_result = self.pool.apply_async(execute_task_async, (task.func, task.id, context))
        self.pending_results.append(pending_result)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def process_single_message_from_queue(self):
        try:
            message = self.message_queue.get_nowait()
            task_id, data = message
            task = self.task_id_to_task[task_id]
            task.on_message(data)
            return True
        except queue.Empty:
            return False

Response: Tries to read a single message from the queue and let the associated task process it.