Instruction:
Implement a function in Python 3 to
aggregate the values of each key using given combine functions and a neutral zero value.

Response:
def aggregateByKey(self, zeroValue, seqFunc, combFunc, numPartitions=None,
                       partitionFunc=portable_hash):
        """
        Aggregate the values of each key, using given combine functions and a neutral
        "zero value". This function can return a different result type, U, than the type
        of the values in this RDD, V. Thus, we need one operation for merging a V into
        a U and one operation for merging two U's, The former operation is used for merging
        values within a partition, and the latter is used for merging values between
        partitions. To avoid memory allocation, both of these functions are
        allowed to modify and return their first argument instead of creating a new U.
        """
        def createZero():
            return copy.deepcopy(zeroValue)

        return self.combineByKey(
            lambda v: seqFunc(createZero(), v), seqFunc, combFunc, numPartitions, partitionFunc)


Instruction:
Write a Python 3 script for
returning a new table with the values for each key in the table grouped by func.

Response:
def foldByKey(self, zeroValue, func, numPartitions=None, partitionFunc=portable_hash):
        """
        Merge the values for each key using an associative function "func"
        and a neutral "zeroValue" which may be added to the result an
        arbitrary number of times, and must not change the result
        (e.g., 0 for addition, or 1 for multiplication.).

        >>> rdd = sc.parallelize([("a", 1), ("b", 1), ("a", 1)])
        >>> from operator import add
        >>> sorted(rdd.foldByKey(0, add).collect())
        [('a', 2), ('b', 1)]
        """
        def createZero():
            return copy.deepcopy(zeroValue)

        return self.combineByKey(lambda v: func(createZero(), v), func, func, numPartitions,
                                 partitionFunc)


Instruction:
How would you explain what the following Python 3 function does
def groupByKey(self, numPartitions=None, partitionFunc=portable_hash):
        def createCombiner(x):
            return [x]

        def mergeValue(xs, x):
            xs.append(x)
            return xs

        def mergeCombiners(a, b):
            a.extend(b)
            return a

        memory = self._memory_limit()
        serializer = self._jrdd_deserializer
        agg = Aggregator(createCombiner, mergeValue, mergeCombiners)

        def combine(iterator):
            merger = ExternalMerger(agg, memory * 0.9, serializer)
            merger.mergeValues(iterator)
            return merger.items()

        locally_combined = self.mapPartitions(combine, preservesPartitioning=True)
        shuffled = locally_combined.partitionBy(numPartitions, partitionFunc)

        def groupByKey(it):
            merger = ExternalGroupBy(agg, memory, serializer)
            merger.mergeCombiners(it)
            return merger.items()

        return shuffled.mapPartitions(groupByKey, True).mapValues(ResultIterable)

Response: Return an RDD of all the values for each key in the RDD.


Instruction:
Explain what the following Python 3 code does
def flatMapValues(self, f):
        flat_map_fn = lambda kv: ((kv[0], x) for x in f(kv[1]))
        return self.flatMap(flat_map_fn, preservesPartitioning=True)

Response: Return a new RDD with the elements of each key - value pair obtained by applying a function to each value in the key - value pair.


Instruction:
Given the following Python 3 function, write the documentation
def mapValues(self, f):
        map_values_fn = lambda kv: (kv[0], f(kv[1]))
        return self.map(map_values_fn, preservesPartitioning=True)

Response: Return an RDD with the values of each key - value pair mapped through a map function.


Instruction:
Here you have a function in Python 3, explain what it does
def sampleByKey(self, withReplacement, fractions, seed=None):
        for fraction in fractions.values():
            assert fraction >= 0.0, "Negative fraction value: %s" % fraction
        return self.mapPartitionsWithIndex(
            RDDStratifiedSampler(withReplacement, fractions, seed).func, True)

Response: Return a subset of this RDD with the values of the keys specified by fractions.


Instruction:
Can you implement a function in Python 3 that
returns each key value pair in self that has no pair with matching key in other.

Response:
def subtractByKey(self, other, numPartitions=None):
        """
        Return each (key, value) pair in C{self} that has no pair with matching
        key in C{other}.

        >>> x = sc.parallelize([("a", 1), ("b", 4), ("b", 5), ("a", 2)])
        >>> y = sc.parallelize([("a", 3), ("c", None)])
        >>> sorted(x.subtractByKey(y).collect())
        [('b', 4), ('b', 5)]
        """
        def filter_func(pair):
            key, (val1, val2) = pair
            return val1 and not val2
        return self.cogroup(other, numPartitions).filter(filter_func).flatMapValues(lambda x: x[0])


Instruction:
Given the following Python 3 function, write the documentation
def subtract(self, other, numPartitions=None):
        # note: here 'True' is just a placeholder
        rdd = other.map(lambda x: (x, True))
        return self.map(lambda x: (x, True)).subtractByKey(rdd, numPartitions).keys()

Response: Return each value in self that is not contained in other.


Instruction:
Can you tell what is the following Python 3 function doing
def coalesce(self, numPartitions, shuffle=False):
        if shuffle:
            # Decrease the batch size in order to distribute evenly the elements across output
            # partitions. Otherwise, repartition will possibly produce highly skewed partitions.
            batchSize = min(10, self.ctx._batchSize or 1024)
            ser = BatchedSerializer(PickleSerializer(), batchSize)
            selfCopy = self._reserialize(ser)
            jrdd_deserializer = selfCopy._jrdd_deserializer
            jrdd = selfCopy._jrdd.coalesce(numPartitions, shuffle)
        else:
            jrdd_deserializer = self._jrdd_deserializer
            jrdd = self._jrdd.coalesce(numPartitions, shuffle)
        return RDD(jrdd, self.ctx, jrdd_deserializer)

Response: Return a new RDD that is reduced into numPartitions partitions.


Instruction:
Can you tell what is the following Python 3 function doing
def zip(self, other):
        def get_batch_size(ser):
            if isinstance(ser, BatchedSerializer):
                return ser.batchSize
            return 1  # not batched

        def batch_as(rdd, batchSize):
            return rdd._reserialize(BatchedSerializer(PickleSerializer(), batchSize))

        my_batch = get_batch_size(self._jrdd_deserializer)
        other_batch = get_batch_size(other._jrdd_deserializer)
        if my_batch != other_batch or not my_batch:
            # use the smallest batchSize for both of them
            batchSize = min(my_batch, other_batch)
            if batchSize <= 0:
                # auto batched or unlimited
                batchSize = 100
            other = batch_as(other, batchSize)
            self = batch_as(self, batchSize)

        if self.getNumPartitions() != other.getNumPartitions():
            raise ValueError("Can only zip with RDD which has the same number of partitions")

        # There will be an Exception in JVM if there are different number
        # of items in each partitions.
        pairRDD = self._jrdd.zip(other._jrdd)
        deserializer = PairDeserializer(self._jrdd_deserializer,
                                        other._jrdd_deserializer)
        return RDD(pairRDD, self.ctx, deserializer)

Response: Returns an RDD of key - value pairs with the same entry - level as this one.


Instruction:
Can you create a Python 3 function that
returns an RDD with its element indices.

Response:
def zipWithIndex(self):
        """
        Zips this RDD with its element indices.

        The ordering is first based on the partition index and then the
        ordering of items within each partition. So the first item in
        the first partition gets index 0, and the last item in the last
        partition receives the largest index.

        This method needs to trigger a spark job when this RDD contains
        more than one partitions.

        >>> sc.parallelize(["a", "b", "c", "d"], 3).zipWithIndex().collect()
        [('a', 0), ('b', 1), ('c', 2), ('d', 3)]
        """
        starts = [0]
        if self.getNumPartitions() > 1:
            nums = self.mapPartitions(lambda it: [sum(1 for i in it)]).collect()
            for i in range(len(nums) - 1):
                starts.append(starts[-1] + nums[i])

        def func(k, it):
            for i, v in enumerate(it, starts[k]):
                yield v, i

        return self.mapPartitionsWithIndex(func)


Instruction:
How would you explain what the following Python 3 function does
def zipWithUniqueId(self):
        n = self.getNumPartitions()

        def func(k, it):
            for i, v in enumerate(it):
                yield v, i * n + k

        return self.mapPartitionsWithIndex(func)

Response: Returns an RDD with generated unique Long ids.


Instruction:
Write a Python 3 function that can
get the current storage level of the RDD.

Response:
def getStorageLevel(self):
        """
        Get the RDD's current storage level.

        >>> rdd1 = sc.parallelize([1,2])
        >>> rdd1.getStorageLevel()
        StorageLevel(False, False, False, False, 1)
        >>> print(rdd1.getStorageLevel())
        Serialized 1x Replicated
        """
        java_storage_level = self._jrdd.getStorageLevel()
        storage_level = StorageLevel(java_storage_level.useDisk(),
                                     java_storage_level.useMemory(),
                                     java_storage_level.useOffHeap(),
                                     java_storage_level.deserialized(),
                                     java_storage_level.replication())
        return storage_level


Instruction:
How would you code a function in Python 3 to
return the default number of partitions to use during reduce tasks.

Response:
def _defaultReducePartitions(self):
        """
        Returns the default number of partitions to use during reduce tasks (e.g., groupBy).
        If spark.default.parallelism is set, then we'll use the value from SparkContext
        defaultParallelism, otherwise we'll use the number of partitions in this RDD.

        This mirrors the behavior of the Scala Partitioner#defaultPartitioner, intended to reduce
        the likelihood of OOMs. Once PySpark adopts Partitioner-based APIs, this behavior will
        be inherent.
        """
        if self.ctx._conf.contains("spark.default.parallelism"):
            return self.ctx.defaultParallelism
        else:
            return self.getNumPartitions()


Instruction:
Can you tell what is the following Python 3 function doing
def lookup(self, key):
        values = self.filter(lambda kv: kv[0] == key).values()

        if self.partitioner is not None:
            return self.ctx.runJob(values, lambda x: x, [self.partitioner(key)])

        return values.collect()

Response:
Return the list of values in the RDD for key key. This operation is efficiently by searching the partitioner by only
       .


Instruction:
Write a Python 3 script to
return a JavaRDD of Object by unpickling

Response:
def _to_java_object_rdd(self):
        """ Return a JavaRDD of Object by unpickling

        It will convert each Python object into Java object by Pyrolite, whenever the
        RDD is serialized in batch or not.
        """
        rdd = self._pickled()
        return self.ctx._jvm.SerDeUtil.pythonToJava(rdd._jrdd, True)


Instruction:
Can you implement a function in Python 3 that
counts the number of incomplete entries within a given timeout.

Response:
def countApprox(self, timeout, confidence=0.95):
        """
        .. note:: Experimental

        Approximate version of count() that returns a potentially incomplete
        result within a timeout, even if not all tasks have finished.

        >>> rdd = sc.parallelize(range(1000), 10)
        >>> rdd.countApprox(1000, 1.0)
        1000
        """
        drdd = self.mapPartitions(lambda it: [float(sum(1 for i in it))])
        return int(drdd.sumApprox(timeout, confidence))


Instruction:
How would you explain what the following Python 3 function does
def sumApprox(self, timeout, confidence=0.95):
        jrdd = self.mapPartitions(lambda it: [float(sum(it))])._to_java_object_rdd()
        jdrdd = self.ctx._jvm.JavaDoubleRDD.fromRDD(jrdd.rdd())
        r = jdrdd.sumApprox(timeout, confidence).getFinalValue()
        return BoundedFloat(r.mean(), r.confidence(), r.low(), r.high())

Response: Return the sum of the elements within a given timeout.


Instruction:
Can you tell what is the following Python 3 function doing
def meanApprox(self, timeout, confidence=0.95):
        jrdd = self.map(float)._to_java_object_rdd()
        jdrdd = self.ctx._jvm.JavaDoubleRDD.fromRDD(jrdd.rdd())
        r = jdrdd.meanApprox(timeout, confidence).getFinalValue()
        return BoundedFloat(r.mean(), r.confidence(), r.low(), r.high())

Response: Return the mean of the set of entries within a given timeout.


Instruction:
Explain what the following Python 3 code does
def countApproxDistinct(self, relativeSD=0.05):
        if relativeSD < 0.000017:
            raise ValueError("relativeSD should be greater than 0.000017")
        # the hash space in Java is 2^32
        hashRDD = self.map(lambda x: portable_hash(x) & 0xFFFFFFFF)
        return hashRDD._to_java_object_rdd().countApproxDistinct(relativeSD)

Response: Return approximate number of distinct elements in the RDD.


Instruction:
Write a Python 3 function for
returning an iterator that contains all of the elements in this RDD.

Response:
def toLocalIterator(self):
        """
        Return an iterator that contains all of the elements in this RDD.
        The iterator will consume as much memory as the largest partition in this RDD.

        >>> rdd = sc.parallelize(range(10))
        >>> [x for x in rdd.toLocalIterator()]
        [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
        """
        with SCCallSiteSync(self.context) as css:
            sock_info = self.ctx._jvm.PythonRDD.toLocalIteratorAndServe(self._jrdd.rdd())
        return _load_from_socket(sock_info, self._jrdd_deserializer)


Instruction:
Can you tell what is the following Python 3 function doing
def mapPartitions(self, f, preservesPartitioning=False):
        def func(s, iterator):
            return f(iterator)
        return PipelinedRDD(self.rdd, func, preservesPartitioning, isFromBarrier=True)

Response: Returns a new RDD by applying a function to each partition of the wrapped RDD.


Instruction:
Can you generate the documentation for the following Python 3 function
def _to_seq(sc, cols, converter=None):
    if converter:
        cols = [converter(c) for c in cols]
    return sc._jvm.PythonUtils.toSeq(cols)

Response: Convert a list of Column names or names into a JVM Seq of Column objects.


Instruction:
Write a Python 3 function that can
convert a list of Column or names into a JVM ( Scala ) List of Column objects.

Response:
def _to_list(sc, cols, converter=None):
    """
    Convert a list of Column (or names) into a JVM (Scala) List of Column.

    An optional `converter` could be used to convert items in `cols`
    into JVM Column objects.
    """
    if converter:
        cols = [converter(c) for c in cols]
    return sc._jvm.PythonUtils.toList(cols)


Instruction:
Write a Python 3 script for
creating a method for given unary operator

Response:
def _unary_op(name, doc="unary operator"):
    """ Create a method for given unary operator """
    def _(self):
        jc = getattr(self._jc, name)()
        return Column(jc)
    _.__doc__ = doc
    return _


Instruction:
Can you create a Python 3 function that
creates a method that returns a new object for the given binary operator.

Response:
def _bin_op(name, doc="binary operator"):
    """ Create a method for given binary operator
    """
    def _(self, other):
        jc = other._jc if isinstance(other, Column) else other
        njc = getattr(self._jc, name)(jc)
        return Column(njc)
    _.__doc__ = doc
    return _


Instruction:
Make a summary of the following Python 3 code
def _reverse_op(name, doc="binary operator"):
    def _(self, other):
        jother = _create_column_from_literal(other)
        jc = getattr(jother, name)(self._jc)
        return Column(jc)
    _.__doc__ = doc
    return _

Response: Create a method for binary operator


Instruction:
Create a Python 3 function for
returning a Column which is a substring of the column.

Response:
def substr(self, startPos, length):
        """
        Return a :class:`Column` which is a substring of the column.

        :param startPos: start position (int or Column)
        :param length:  length of the substring (int or Column)

        >>> df.select(df.name.substr(1, 3).alias("col")).collect()
        [Row(col=u'Ali'), Row(col=u'Bob')]
        """
        if type(startPos) != type(length):
            raise TypeError(
                "startPos and length must be the same type. "
                "Got {startPos_t} and {length_t}, respectively."
                .format(
                    startPos_t=type(startPos),
                    length_t=type(length),
                ))
        if isinstance(startPos, int):
            jc = self._jc.substr(startPos, length)
        elif isinstance(startPos, Column):
            jc = self._jc.substr(startPos._jc, length._jc)
        else:
            raise TypeError("Unexpected type: %s" % type(startPos))
        return Column(jc)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def isin(self, *cols):
        if len(cols) == 1 and isinstance(cols[0], (list, set)):
            cols = cols[0]
        cols = [c._jc if isinstance(c, Column) else _create_column_from_literal(c) for c in cols]
        sc = SparkContext._active_spark_context
        jc = getattr(self._jc, "isin")(_to_seq(sc, cols))
        return Column(jc)

Response:
A boolean expression that is evaluated to true if the value of this
        expression is contained by the evaluated values of the arguments.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def alias(self, *alias, **kwargs):

        metadata = kwargs.pop('metadata', None)
        assert not kwargs, 'Unexpected kwargs where passed: %s' % kwargs

        sc = SparkContext._active_spark_context
        if len(alias) == 1:
            if metadata:
                jmeta = sc._jvm.org.apache.spark.sql.types.Metadata.fromJson(
                    json.dumps(metadata))
                return Column(getattr(self._jc, "as")(alias[0], jmeta))
            else:
                return Column(getattr(self._jc, "as")(alias[0]))
        else:
            if metadata:
                raise ValueError('metadata can only be provided for a single column')
            return Column(getattr(self._jc, "as")(_to_seq(sc, list(alias))))

Response: Returns this column aliased with a new name or names.


Instruction:
Implement a function in Python 3 to
convert the column into type dataType.

Response:
def cast(self, dataType):
        """ Convert the column into type ``dataType``.

        >>> df.select(df.age.cast("string").alias('ages')).collect()
        [Row(ages=u'2'), Row(ages=u'5')]
        >>> df.select(df.age.cast(StringType()).alias('ages')).collect()
        [Row(ages=u'2'), Row(ages=u'5')]
        """
        if isinstance(dataType, basestring):
            jc = self._jc.cast(dataType)
        elif isinstance(dataType, DataType):
            from pyspark.sql import SparkSession
            spark = SparkSession.builder.getOrCreate()
            jdt = spark._jsparkSession.parseDataType(dataType.json())
            jc = self._jc.cast(jdt)
        else:
            raise TypeError("unexpected type: %s" % type(dataType))
        return Column(jc)


Instruction:
How would you code a function in Python 3 to
evaluate a list of conditions and returns one of multiple possible result expressions.

Response:
def when(self, condition, value):
        """
        Evaluates a list of conditions and returns one of multiple possible result expressions.
        If :func:`Column.otherwise` is not invoked, None is returned for unmatched conditions.

        See :func:`pyspark.sql.functions.when` for example usage.

        :param condition: a boolean :class:`Column` expression.
        :param value: a literal value, or a :class:`Column` expression.

        >>> from pyspark.sql import functions as F
        >>> df.select(df.name, F.when(df.age > 4, 1).when(df.age < 3, -1).otherwise(0)).show()
        +-----+------------------------------------------------------------+
        | name|CASE WHEN (age > 4) THEN 1 WHEN (age < 3) THEN -1 ELSE 0 END|
        +-----+------------------------------------------------------------+
        |Alice|                                                          -1|
        |  Bob|                                                           1|
        +-----+------------------------------------------------------------+
        """
        if not isinstance(condition, Column):
            raise TypeError("condition should be a Column")
        v = value._jc if isinstance(value, Column) else value
        jc = self._jc.when(condition._jc, v)
        return Column(jc)


Instruction:
Implement a function in Python 3 to
evaluate a list of conditions and returns one of multiple possible result expressions.

Response:
def otherwise(self, value):
        """
        Evaluates a list of conditions and returns one of multiple possible result expressions.
        If :func:`Column.otherwise` is not invoked, None is returned for unmatched conditions.

        See :func:`pyspark.sql.functions.when` for example usage.

        :param value: a literal value, or a :class:`Column` expression.

        >>> from pyspark.sql import functions as F
        >>> df.select(df.name, F.when(df.age > 3, 1).otherwise(0)).show()
        +-----+-------------------------------------+
        | name|CASE WHEN (age > 3) THEN 1 ELSE 0 END|
        +-----+-------------------------------------+
        |Alice|                                    0|
        |  Bob|                                    1|
        +-----+-------------------------------------+
        """
        v = value._jc if isinstance(value, Column) else value
        jc = self._jc.otherwise(v)
        return Column(jc)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def over(self, window):
        from pyspark.sql.window import WindowSpec
        if not isinstance(window, WindowSpec):
            raise TypeError("window should be WindowSpec")
        jc = self._jc.over(window._jspec)
        return Column(jc)

Response: Define a windowing column.


Instruction:
How would you code a function in Python 3 to
apply transformation on a vector or an RDD of Vectors.

Response:
def transform(self, vector):
        """
        Applies transformation on a vector or an RDD[Vector].

        .. note:: In Python, transform cannot currently be used within
            an RDD transformation or action.
            Call transform directly on the RDD instead.

        :param vector: Vector or RDD of Vector to be transformed.
        """
        if isinstance(vector, RDD):
            vector = vector.map(_convert_to_vector)
        else:
            vector = _convert_to_vector(vector)
        return self.call("transform", vector)


Instruction:
Implement a function in Python 3 to
fit the model to the given dataset.

Response:
def fit(self, dataset):
        """
        Computes the mean and variance and stores as a model to be used
        for later scaling.

        :param dataset: The data used to compute the mean and variance
                     to build the transformation model.
        :return: a StandardScalarModel
        """
        dataset = dataset.map(_convert_to_vector)
        jmodel = callMLlibFunc("fitStandardScaler", self.withMean, self.withStd, dataset)
        return StandardScalerModel(jmodel)


Instruction:
Implement a Python 3 function for
fitting a ChiSquared feature selector.

Response:
def fit(self, data):
        """
        Returns a ChiSquared feature selector.

        :param data: an `RDD[LabeledPoint]` containing the labeled dataset
                     with categorical features. Real-valued features will be
                     treated as categorical for each distinct value.
                     Apply feature discretizer before using this function.
        """
        jmodel = callMLlibFunc("fitChiSqSelector", self.selectorType, self.numTopFeatures,
                               self.percentile, self.fpr, self.fdr, self.fwe, data)
        return ChiSqSelectorModel(jmodel)


Instruction:
Can you tell what is the following Python 3 function doing
def fit(self, data):
        jmodel = callMLlibFunc("fitPCA", self.k, data)
        return PCAModel(jmodel)

Response: Computes a PCAModel that contains the principal components of the input vectors.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def transform(self, document):
        if isinstance(document, RDD):
            return document.map(self.transform)

        freq = {}
        for term in document:
            i = self.indexOf(term)
            freq[i] = 1.0 if self.binary else freq.get(i, 0) + 1.0
        return Vectors.sparse(self.numFeatures, freq.items())

Response:
Transforms the input document to term frequency
        vectors or transform the RDD of document to RDD of term frequency
        vectors or transform the input document to RDD of term frequency
        vectors or transform the RDD of document to RDD of term frequency
        vectors.


Instruction:
Can you generate the documentation for the following Python 3 function
def fit(self, dataset):
        if not isinstance(dataset, RDD):
            raise TypeError("dataset should be an RDD of term frequency vectors")
        jmodel = callMLlibFunc("fitIDF", self.minDocFreq, dataset.map(_convert_to_vector))
        return IDFModel(jmodel)

Response: Computes the inverse document frequency of the given term frequency vectors.


Instruction:
Write a Python 3 function that can
find synonyms of a word.

Response:
def findSynonyms(self, word, num):
        """
        Find synonyms of a word

        :param word: a word or a vector representation of word
        :param num: number of synonyms to find
        :return: array of (word, cosineSimilarity)

        .. note:: Local use only
        """
        if not isinstance(word, basestring):
            word = _convert_to_vector(word)
        words, similarity = self.call("findSynonyms", word, num)
        return zip(words, similarity)


Instruction:
How would you explain what the following Python 3 function does
def load(cls, sc, path):
        jmodel = sc._jvm.org.apache.spark.mllib.feature \
            .Word2VecModel.load(sc._jsc.sc(), path)
        model = sc._jvm.org.apache.spark.mllib.api.python.Word2VecModelWrapper(jmodel)
        return Word2VecModel(model)

Response: Loads a word2vec model from the given path.


Instruction:
Write a Python 3 function that can
compute the Hadamard product of the vector.

Response:
def transform(self, vector):
        """
        Computes the Hadamard product of the vector.
        """
        if isinstance(vector, RDD):
            vector = vector.map(_convert_to_vector)

        else:
            vector = _convert_to_vector(vector)
        return callMLlibFunc("elementwiseProductVector", self.scalingVector, vector)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def predict(self, x):
        if isinstance(x, RDD):
            return self.call("predict", x.map(_convert_to_vector))

        else:
            return self.call("predict", _convert_to_vector(x))

Response: Predict values for a single data point or an RDD of points.


Instruction:
Write a Python 3 script to
train a decision tree model for classification.

Response:
def trainClassifier(cls, data, numClasses, categoricalFeaturesInfo,
                        impurity="gini", maxDepth=5, maxBins=32, minInstancesPerNode=1,
                        minInfoGain=0.0):
        """
        Train a decision tree model for classification.

        :param data:
          Training data: RDD of LabeledPoint. Labels should take values
          {0, 1, ..., numClasses-1}.
        :param numClasses:
          Number of classes for classification.
        :param categoricalFeaturesInfo:
          Map storing arity of categorical features. An entry (n -> k)
          indicates that feature n is categorical with k categories
          indexed from 0: {0, 1, ..., k-1}.
        :param impurity:
          Criterion used for information gain calculation.
          Supported values: "gini" or "entropy".
          (default: "gini")
        :param maxDepth:
          Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1
          means 1 internal node + 2 leaf nodes).
          (default: 5)
        :param maxBins:
          Number of bins used for finding splits at each node.
          (default: 32)
        :param minInstancesPerNode:
          Minimum number of instances required at child nodes to create
          the parent split.
          (default: 1)
        :param minInfoGain:
          Minimum info gain required to create a split.
          (default: 0.0)
        :return:
          DecisionTreeModel.

        Example usage:

        >>> from numpy import array
        >>> from pyspark.mllib.regression import LabeledPoint
        >>> from pyspark.mllib.tree import DecisionTree
        >>>
        >>> data = [
        ...     LabeledPoint(0.0, [0.0]),
        ...     LabeledPoint(1.0, [1.0]),
        ...     LabeledPoint(1.0, [2.0]),
        ...     LabeledPoint(1.0, [3.0])
        ... ]
        >>> model = DecisionTree.trainClassifier(sc.parallelize(data), 2, {})
        >>> print(model)
        DecisionTreeModel classifier of depth 1 with 3 nodes

        >>> print(model.toDebugString())
        DecisionTreeModel classifier of depth 1 with 3 nodes
          If (feature 0 <= 0.5)
           Predict: 0.0
          Else (feature 0 > 0.5)
           Predict: 1.0
        <BLANKLINE>
        >>> model.predict(array([1.0]))
        1.0
        >>> model.predict(array([0.0]))
        0.0
        >>> rdd = sc.parallelize([[1.0], [0.0]])
        >>> model.predict(rdd).collect()
        [1.0, 0.0]
        """
        return cls._train(data, "classification", numClasses, categoricalFeaturesInfo,
                          impurity, maxDepth, maxBins, minInstancesPerNode, minInfoGain)


Instruction:
Can you create a Python 3 function that
trains a Decision Tree model for regression.

Response:
def trainRegressor(cls, data, categoricalFeaturesInfo,
                       impurity="variance", maxDepth=5, maxBins=32, minInstancesPerNode=1,
                       minInfoGain=0.0):
        """
        Train a decision tree model for regression.

        :param data:
          Training data: RDD of LabeledPoint. Labels are real numbers.
        :param categoricalFeaturesInfo:
          Map storing arity of categorical features. An entry (n -> k)
          indicates that feature n is categorical with k categories
          indexed from 0: {0, 1, ..., k-1}.
        :param impurity:
          Criterion used for information gain calculation.
          The only supported value for regression is "variance".
          (default: "variance")
        :param maxDepth:
          Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1
          means 1 internal node + 2 leaf nodes).
          (default: 5)
        :param maxBins:
          Number of bins used for finding splits at each node.
          (default: 32)
        :param minInstancesPerNode:
          Minimum number of instances required at child nodes to create
          the parent split.
          (default: 1)
        :param minInfoGain:
          Minimum info gain required to create a split.
          (default: 0.0)
        :return:
          DecisionTreeModel.

        Example usage:

        >>> from pyspark.mllib.regression import LabeledPoint
        >>> from pyspark.mllib.tree import DecisionTree
        >>> from pyspark.mllib.linalg import SparseVector
        >>>
        >>> sparse_data = [
        ...     LabeledPoint(0.0, SparseVector(2, {0: 0.0})),
        ...     LabeledPoint(1.0, SparseVector(2, {1: 1.0})),
        ...     LabeledPoint(0.0, SparseVector(2, {0: 0.0})),
        ...     LabeledPoint(1.0, SparseVector(2, {1: 2.0}))
        ... ]
        >>>
        >>> model = DecisionTree.trainRegressor(sc.parallelize(sparse_data), {})
        >>> model.predict(SparseVector(2, {1: 1.0}))
        1.0
        >>> model.predict(SparseVector(2, {1: 0.0}))
        0.0
        >>> rdd = sc.parallelize([[0.0, 1.0], [0.0, 0.0]])
        >>> model.predict(rdd).collect()
        [1.0, 0.0]
        """
        return cls._train(data, "regression", 0, categoricalFeaturesInfo,
                          impurity, maxDepth, maxBins, minInstancesPerNode, minInfoGain)


Instruction:
Can you tell what is the following Python 3 function doing
def trainClassifier(cls, data, numClasses, categoricalFeaturesInfo, numTrees,
                        featureSubsetStrategy="auto", impurity="gini", maxDepth=4, maxBins=32,
                        seed=None):
        """
        Train a random forest model for binary or multiclass
        classification.

        :param data:
          Training dataset: RDD of LabeledPoint. Labels should take values
          {0, 1, ..., numClasses-1}.
        :param numClasses:
          Number of classes for classification.
        :param categoricalFeaturesInfo:
          Map storing arity of categorical features. An entry (n -> k)
          indicates that feature n is categorical with k categories
          indexed from 0: {0, 1, ..., k-1}.
        :param numTrees:
          Number of trees in the random forest.
        :param featureSubsetStrategy:
          Number of features to consider for splits at each node.
          Supported values: "auto", "all", "sqrt", "log2", "onethird".
          If "auto" is set, this parameter is set based on numTrees:
          if numTrees == 1, set to "all";
          if numTrees > 1 (forest) set to "sqrt".
          (default: "auto")
        :param impurity:
          Criterion used for information gain calculation.
          Supported values: "gini" or "entropy".
          (default: "gini")
        :param maxDepth:
          Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1
          means 1 internal node + 2 leaf nodes).
          (default: 4)
        :param maxBins:
          Maximum number of bins used for splitting features.
          (default: 32)
        :param seed:
          Random seed for bootstrapping and choosing feature subsets.
          Set as None to generate seed based on system time.
          (default: None)
        :return:
          RandomForestModel that can be used for prediction.

        Example usage:

        >>> from pyspark.mllib.regression import LabeledPoint
        >>> from pyspark.mllib.tree import RandomForest
        >>>
        >>> data = [
        ...     LabeledPoint(0.0, [0.0]),
        ...     LabeledPoint(0.0, [1.0]),
        ...     LabeledPoint(1.0, [2.0]),
        ...     LabeledPoint(1.0, [3.0])
        ... ]
        >>> model = RandomForest.trainClassifier(sc.parallelize(data), 2, {}, 3, seed=42)
        >>> model.numTrees()
        3
        >>> model.totalNumNodes()
        7
        >>> print(model)
        TreeEnsembleModel classifier with 3 trees
        <BLANKLINE>
        >>> print(model.toDebugString())
        TreeEnsembleModel classifier with 3 trees
        <BLANKLINE>
          Tree 0:
            Predict: 1.0
          Tree 1:
            If (feature 0 <= 1.5)
             Predict: 0.0
            Else (feature 0 > 1.5)
             Predict: 1.0
          Tree 2:
            If (feature 0 <= 1.5)
             Predict: 0.0
            Else (feature 0 > 1.5)
             Predict: 1.0
        <BLANKLINE>
        >>> model.predict([2.0])
        1.0
        >>> model.predict([0.0])
        0.0
        >>> rdd = sc.parallelize([[3.0], [1.0]])
        >>> model.predict(rdd).collect()
        [1.0, 0.0]
        """
        return cls._train(data, "classification", numClasses,
                          categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity,
                          maxDepth, maxBins, seed)

Response: Train a RandomForestModel for binary or multiclass classification.


Instruction:
How would you implement a function in Python 3 that
trains a random forest model for regression.

Response:
def trainRegressor(cls, data, categoricalFeaturesInfo, numTrees, featureSubsetStrategy="auto",
                       impurity="variance", maxDepth=4, maxBins=32, seed=None):
        """
        Train a random forest model for regression.

        :param data:
          Training dataset: RDD of LabeledPoint. Labels are real numbers.
        :param categoricalFeaturesInfo:
          Map storing arity of categorical features. An entry (n -> k)
          indicates that feature n is categorical with k categories
          indexed from 0: {0, 1, ..., k-1}.
        :param numTrees:
          Number of trees in the random forest.
        :param featureSubsetStrategy:
          Number of features to consider for splits at each node.
          Supported values: "auto", "all", "sqrt", "log2", "onethird".
          If "auto" is set, this parameter is set based on numTrees:
          if numTrees == 1, set to "all";
          if numTrees > 1 (forest) set to "onethird" for regression.
          (default: "auto")
        :param impurity:
          Criterion used for information gain calculation.
          The only supported value for regression is "variance".
          (default: "variance")
        :param maxDepth:
          Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1
          means 1 internal node + 2 leaf nodes).
          (default: 4)
        :param maxBins:
          Maximum number of bins used for splitting features.
          (default: 32)
        :param seed:
          Random seed for bootstrapping and choosing feature subsets.
          Set as None to generate seed based on system time.
          (default: None)
        :return:
          RandomForestModel that can be used for prediction.

        Example usage:

        >>> from pyspark.mllib.regression import LabeledPoint
        >>> from pyspark.mllib.tree import RandomForest
        >>> from pyspark.mllib.linalg import SparseVector
        >>>
        >>> sparse_data = [
        ...     LabeledPoint(0.0, SparseVector(2, {0: 1.0})),
        ...     LabeledPoint(1.0, SparseVector(2, {1: 1.0})),
        ...     LabeledPoint(0.0, SparseVector(2, {0: 1.0})),
        ...     LabeledPoint(1.0, SparseVector(2, {1: 2.0}))
        ... ]
        >>>
        >>> model = RandomForest.trainRegressor(sc.parallelize(sparse_data), {}, 2, seed=42)
        >>> model.numTrees()
        2
        >>> model.totalNumNodes()
        4
        >>> model.predict(SparseVector(2, {1: 1.0}))
        1.0
        >>> model.predict(SparseVector(2, {0: 1.0}))
        0.5
        >>> rdd = sc.parallelize([[0.0, 1.0], [1.0, 0.0]])
        >>> model.predict(rdd).collect()
        [1.0, 0.5]
        """
        return cls._train(data, "regression", 0, categoricalFeaturesInfo, numTrees,
                          featureSubsetStrategy, impurity, maxDepth, maxBins, seed)


Instruction:
Make a summary of the following Python 3 code
def trainClassifier(cls, data, categoricalFeaturesInfo,
                        loss="logLoss", numIterations=100, learningRate=0.1, maxDepth=3,
                        maxBins=32):
        """
        Train a gradient-boosted trees model for classification.

        :param data:
          Training dataset: RDD of LabeledPoint. Labels should take values
          {0, 1}.
        :param categoricalFeaturesInfo:
          Map storing arity of categorical features. An entry (n -> k)
          indicates that feature n is categorical with k categories
          indexed from 0: {0, 1, ..., k-1}.
        :param loss:
          Loss function used for minimization during gradient boosting.
          Supported values: "logLoss", "leastSquaresError",
          "leastAbsoluteError".
          (default: "logLoss")
        :param numIterations:
          Number of iterations of boosting.
          (default: 100)
        :param learningRate:
          Learning rate for shrinking the contribution of each estimator.
          The learning rate should be between in the interval (0, 1].
          (default: 0.1)
        :param maxDepth:
          Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1
          means 1 internal node + 2 leaf nodes).
          (default: 3)
        :param maxBins:
          Maximum number of bins used for splitting features. DecisionTree
          requires maxBins >= max categories.
          (default: 32)
        :return:
          GradientBoostedTreesModel that can be used for prediction.

        Example usage:

        >>> from pyspark.mllib.regression import LabeledPoint
        >>> from pyspark.mllib.tree import GradientBoostedTrees
        >>>
        >>> data = [
        ...     LabeledPoint(0.0, [0.0]),
        ...     LabeledPoint(0.0, [1.0]),
        ...     LabeledPoint(1.0, [2.0]),
        ...     LabeledPoint(1.0, [3.0])
        ... ]
        >>>
        >>> model = GradientBoostedTrees.trainClassifier(sc.parallelize(data), {}, numIterations=10)
        >>> model.numTrees()
        10
        >>> model.totalNumNodes()
        30
        >>> print(model)  # it already has newline
        TreeEnsembleModel classifier with 10 trees
        <BLANKLINE>
        >>> model.predict([2.0])
        1.0
        >>> model.predict([0.0])
        0.0
        >>> rdd = sc.parallelize([[2.0], [0.0]])
        >>> model.predict(rdd).collect()
        [1.0, 0.0]
        """
        return cls._train(data, "classification", categoricalFeaturesInfo,
                          loss, numIterations, learningRate, maxDepth, maxBins)

Response: Train a gradient - boosted trees model for classification.


Instruction:
Create a Python 3 function for
setting a configuration property.

Response:
def set(self, key, value):
        """Set a configuration property."""
        # Try to set self._jconf first if JVM is created, set self._conf if JVM is not created yet.
        if self._jconf is not None:
            self._jconf.set(key, unicode(value))
        else:
            self._conf[key] = unicode(value)
        return self


Instruction:
Write a Python 3 function for
setting a configuration property if not already set.

Response:
def setIfMissing(self, key, value):
        """Set a configuration property, if not already set."""
        if self.get(key) is None:
            self.set(key, value)
        return self


Instruction:
Explain what the following Python 3 code does
def setExecutorEnv(self, key=None, value=None, pairs=None):
        if (key is not None and pairs is not None) or (key is None and pairs is None):
            raise Exception("Either pass one key-value pair or a list of pairs")
        elif key is not None:
            self.set("spark.executorEnv." + key, value)
        elif pairs is not None:
            for (k, v) in pairs:
                self.set("spark.executorEnv." + k, v)
        return self

Response: Set an environment variable to be passed to executors.


Instruction:
Can you generate the documentation for the following Python 3 function
def setAll(self, pairs):
        for (k, v) in pairs:
            self.set(k, v)
        return self

Response: Set multiple parameters passed as a list of key - value pairs.


Instruction:
Here you have a function in Python 3, explain what it does
def get(self, key, defaultValue=None):
        if defaultValue is None:   # Py4J doesn't call the right get() if we pass None
            if self._jconf is not None:
                if not self._jconf.contains(key):
                    return None
                return self._jconf.get(key)
            else:
                if key not in self._conf:
                    return None
                return self._conf[key]
        else:
            if self._jconf is not None:
                return self._jconf.get(key, defaultValue)
            else:
                return self._conf.get(key, defaultValue)

Response: Get the configured value for some key or return a default otherwise.


Instruction:
How would you explain what the following Python 3 function does
def getAll(self):
        if self._jconf is not None:
            return [(elem._1(), elem._2()) for elem in self._jconf.getAll()]
        else:
            return self._conf.items()

Response: Get all values as a list of key - value pairs.


Instruction:
Can you create a Python 3 function that
does this configuration contain a given key?

Response:
def contains(self, key):
        """Does this configuration contain a given key?"""
        if self._jconf is not None:
            return self._jconf.contains(key)
        else:
            return key in self._conf


Instruction:
Create a Python 3 function to
return a printable version of the configuration as a list of key = value pairs one per line.

Response:
def toDebugString(self):
        """
        Returns a printable version of the configuration, as a list of
        key=value pairs, one per line.
        """
        if self._jconf is not None:
            return self._jconf.toDebugString()
        else:
            return '\n'.join('%s=%s' % (k, v) for k, v in self._conf.items())


Instruction:
Implement a function in Python 3 to
return a list of databases available across all sessions.

Response:
def listDatabases(self):
        """Returns a list of databases available across all sessions."""
        iter = self._jcatalog.listDatabases().toLocalIterator()
        databases = []
        while iter.hasNext():
            jdb = iter.next()
            databases.append(Database(
                name=jdb.name(),
                description=jdb.description(),
                locationUri=jdb.locationUri()))
        return databases


Instruction:
Write a Python 3 script for
returning a list of tables in the specified database.

Response:
def listTables(self, dbName=None):
        """Returns a list of tables/views in the specified database.

        If no database is specified, the current database is used.
        This includes all temporary views.
        """
        if dbName is None:
            dbName = self.currentDatabase()
        iter = self._jcatalog.listTables(dbName).toLocalIterator()
        tables = []
        while iter.hasNext():
            jtable = iter.next()
            tables.append(Table(
                name=jtable.name(),
                database=jtable.database(),
                description=jtable.description(),
                tableType=jtable.tableType(),
                isTemporary=jtable.isTemporary()))
        return tables


Instruction:
Implement a Python 3 function for
returning a list of functions registered in the specified database.

Response:
def listFunctions(self, dbName=None):
        """Returns a list of functions registered in the specified database.

        If no database is specified, the current database is used.
        This includes all temporary functions.
        """
        if dbName is None:
            dbName = self.currentDatabase()
        iter = self._jcatalog.listFunctions(dbName).toLocalIterator()
        functions = []
        while iter.hasNext():
            jfunction = iter.next()
            functions.append(Function(
                name=jfunction.name(),
                description=jfunction.description(),
                className=jfunction.className(),
                isTemporary=jfunction.isTemporary()))
        return functions


Instruction:
Create a Python 3 function for
returning a list of columns for the given table or view in the specified database.

Response:
def listColumns(self, tableName, dbName=None):
        """Returns a list of columns for the given table/view in the specified database.

        If no database is specified, the current database is used.

        Note: the order of arguments here is different from that of its JVM counterpart
        because Python does not support method overloading.
        """
        if dbName is None:
            dbName = self.currentDatabase()
        iter = self._jcatalog.listColumns(dbName, tableName).toLocalIterator()
        columns = []
        while iter.hasNext():
            jcolumn = iter.next()
            columns.append(Column(
                name=jcolumn.name(),
                description=jcolumn.description(),
                dataType=jcolumn.dataType(),
                nullable=jcolumn.nullable(),
                isPartition=jcolumn.isPartition(),
                isBucket=jcolumn.isBucket()))
        return columns


Instruction:
Write a Python 3 script for
creating an external table based on the dataset in a data source.

Response:
def createExternalTable(self, tableName, path=None, source=None, schema=None, **options):
        """Creates a table based on the dataset in a data source.

        It returns the DataFrame associated with the external table.

        The data source is specified by the ``source`` and a set of ``options``.
        If ``source`` is not specified, the default data source configured by
        ``spark.sql.sources.default`` will be used.

        Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
        created external table.

        :return: :class:`DataFrame`
        """
        warnings.warn(
            "createExternalTable is deprecated since Spark 2.2, please use createTable instead.",
            DeprecationWarning)
        return self.createTable(tableName, path, source, schema, **options)


Instruction:
How would you explain what the following Python 3 function does
def createTable(self, tableName, path=None, source=None, schema=None, **options):
        if path is not None:
            options["path"] = path
        if source is None:
            source = self._sparkSession._wrapped._conf.defaultDataSourceName()
        if schema is None:
            df = self._jcatalog.createTable(tableName, source, options)
        else:
            if not isinstance(schema, StructType):
                raise TypeError("schema should be StructType")
            scala_datatype = self._jsparkSession.parseDataType(schema.json())
            df = self._jcatalog.createTable(tableName, source, scala_datatype, options)
        return DataFrame(df, self._sparkSession._wrapped)

Response: Creates a table based on the dataset in a data source.


Instruction:
Explain what the following Python 3 code does
def _load_from_socket(port, auth_secret):
    (sockfile, sock) = local_connect_and_auth(port, auth_secret)
    # The barrier() call may block forever, so no timeout
    sock.settimeout(None)
    # Make a barrier() function call.
    write_int(BARRIER_FUNCTION, sockfile)
    sockfile.flush()

    # Collect result.
    res = UTF8Deserializer().loads(sockfile)

    # Release resources.
    sockfile.close()
    sock.close()

    return res

Response: Load data from a given socket.


Instruction:
How would you explain what the following Python 3 function does
def _getOrCreate(cls):
        if not isinstance(cls._taskContext, BarrierTaskContext):
            cls._taskContext = object.__new__(cls)
        return cls._taskContext

Response: Internal function to get or create BarrierTaskContext object.


Instruction:
Write a Python 3 script to
initialize BarrierTaskContext, other methods within BarrierTaskContext can only be called after BarrierTaskContext is initialized.

Response:
def _initialize(cls, port, secret):
        """
        Initialize BarrierTaskContext, other methods within BarrierTaskContext can only be called
        after BarrierTaskContext is initialized.
        """
        cls._port = port
        cls._secret = secret


Instruction:
How would you implement a function in Python 3 that
sets a global barrier and waits until all tasks in this stage hit this barrier.

Response:
def barrier(self):
        """
        .. note:: Experimental

        Sets a global barrier and waits until all tasks in this stage hit this barrier.
        Similar to `MPI_Barrier` function in MPI, this function blocks until all tasks
        in the same stage have reached this routine.

        .. warning:: In a barrier stage, each task much have the same number of `barrier()`
            calls, in all possible code branches.
            Otherwise, you may get the job hanging or a SparkException after timeout.

        .. versionadded:: 2.4.0
        """
        if self._port is None or self._secret is None:
            raise Exception("Not supported to call barrier() before initialize " +
                            "BarrierTaskContext.")
        else:
            _load_from_socket(self._port, self._secret)


Instruction:
Make a summary of the following Python 3 code
def getTaskInfos(self):
        if self._port is None or self._secret is None:
            raise Exception("Not supported to call getTaskInfos() before initialize " +
                            "BarrierTaskContext.")
        else:
            addresses = self._localProperties.get("addresses", "")
            return [BarrierTaskInfo(h.strip()) for h in addresses.split(",")]

Response: Returns a list of BarrierTaskInfo objects for all tasks in this barrier stage ordered by partition ID.


Instruction:
Can you tell what is the following Python 3 function doing
def since(version):
    import re
    indent_p = re.compile(r'\n( +)')

    def deco(f):
        indents = indent_p.findall(f.__doc__)
        indent = ' ' * (min(len(m) for m in indents) if indents else 0)
        f.__doc__ = f.__doc__.rstrip() + "\n\n%s.. versionadded:: %s" % (indent, version)
        return f
    return deco

Response: A decorator that adds the version of Spark the function was added.


Instruction:
Explain what the following Python 3 code does
def copy_func(f, name=None, sinceversion=None, doc=None):
    # See
    # http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
    fn = types.FunctionType(f.__code__, f.__globals__, name or f.__name__, f.__defaults__,
                            f.__closure__)
    # in case f was given attrs (note this dict is a shallow copy):
    fn.__dict__.update(f.__dict__)
    if doc is not None:
        fn.__doc__ = doc
    if sinceversion is not None:
        fn = since(sinceversion)(fn)
    return fn

Response: Returns a copy of a function f.


Instruction:
Implement a function in Python 3 to
generate the header part for a param.

Response:
def _gen_param_header(name, doc, defaultValueStr, typeConverter):
    """
    Generates the header part for shared variables

    :param name: param name
    :param doc: param doc
    """
    template = '''class Has$Name(Params):
    """
    Mixin for param $name: $doc
    """

    $name = Param(Params._dummy(), "$name", "$doc", typeConverter=$typeConverter)

    def __init__(self):
        super(Has$Name, self).__init__()'''

    if defaultValueStr is not None:
        template += '''
        self._setDefault($name=$defaultValueStr)'''

    Name = name[0].upper() + name[1:]
    if typeConverter is None:
        typeConverter = str(None)
    return template \
        .replace("$name", name) \
        .replace("$Name", Name) \
        .replace("$doc", doc) \
        .replace("$defaultValueStr", str(defaultValueStr)) \
        .replace("$typeConverter", typeConverter)


Instruction:
How would you explain what the following Python 3 function does
def _gen_param_code(name, doc, defaultValueStr):
    # TODO: How to correctly inherit instance attributes?
    template = '''
    def set$Name(self, value):
        """
        Sets the value of :py:attr:`$name`.
        """
        return self._set($name=value)

    def get$Name(self):
        """
        Gets the value of $name or its default value.
        """
        return self.getOrDefault(self.$name)'''

    Name = name[0].upper() + name[1:]
    return template \
        .replace("$name", name) \
        .replace("$Name", Name) \
        .replace("$doc", doc) \
        .replace("$defaultValueStr", str(defaultValueStr))

Response: Generates code for a shared param class.


Instruction:
Write a Python 3 script to
run the bisecting k - means algorithm and returns the model.

Response:
def train(self, rdd, k=4, maxIterations=20, minDivisibleClusterSize=1.0, seed=-1888008604):
        """
        Runs the bisecting k-means algorithm return the model.

        :param rdd:
          Training points as an `RDD` of `Vector` or convertible
          sequence types.
        :param k:
          The desired number of leaf clusters. The actual number could
          be smaller if there are no divisible leaf clusters.
          (default: 4)
        :param maxIterations:
          Maximum number of iterations allowed to split clusters.
          (default: 20)
        :param minDivisibleClusterSize:
          Minimum number of points (if >= 1.0) or the minimum proportion
          of points (if < 1.0) of a divisible cluster.
          (default: 1)
        :param seed:
          Random seed value for cluster initialization.
          (default: -1888008604 from classOf[BisectingKMeans].getName.##)
        """
        java_model = callMLlibFunc(
            "trainBisectingKMeans", rdd.map(_convert_to_vector),
            k, maxIterations, minDivisibleClusterSize, seed)
        return BisectingKMeansModel(java_model)


Instruction:
Create a Python 3 function to
train a k - means clustering model on the given data set.

Response:
def train(cls, rdd, k, maxIterations=100, runs=1, initializationMode="k-means||",
              seed=None, initializationSteps=2, epsilon=1e-4, initialModel=None):
        """
        Train a k-means clustering model.

        :param rdd:
          Training points as an `RDD` of `Vector` or convertible
          sequence types.
        :param k:
          Number of clusters to create.
        :param maxIterations:
          Maximum number of iterations allowed.
          (default: 100)
        :param runs:
          This param has no effect since Spark 2.0.0.
        :param initializationMode:
          The initialization algorithm. This can be either "random" or
          "k-means||".
          (default: "k-means||")
        :param seed:
          Random seed value for cluster initialization. Set as None to
          generate seed based on system time.
          (default: None)
        :param initializationSteps:
          Number of steps for the k-means|| initialization mode.
          This is an advanced setting -- the default of 2 is almost
          always enough.
          (default: 2)
        :param epsilon:
          Distance threshold within which a center will be considered to
          have converged. If all centers move less than this Euclidean
          distance, iterations are stopped.
          (default: 1e-4)
        :param initialModel:
          Initial cluster centers can be provided as a KMeansModel object
          rather than using the random or k-means|| initializationModel.
          (default: None)
        """
        if runs != 1:
            warnings.warn("The param `runs` has no effect since Spark 2.0.0.")
        clusterInitialModel = []
        if initialModel is not None:
            if not isinstance(initialModel, KMeansModel):
                raise Exception("initialModel is of "+str(type(initialModel))+". It needs "
                                "to be of <type 'KMeansModel'>")
            clusterInitialModel = [_convert_to_vector(c) for c in initialModel.clusterCenters]
        model = callMLlibFunc("trainKMeansModel", rdd.map(_convert_to_vector), k, maxIterations,
                              runs, initializationMode, seed, initializationSteps, epsilon,
                              clusterInitialModel)
        centers = callJavaFunc(rdd.context, model.clusterCenters)
        return KMeansModel([c.toArray() for c in centers])


Instruction:
Can you create a Python 3 function that
trains a Gaussian Mixture clustering model.

Response:
def train(cls, rdd, k, convergenceTol=1e-3, maxIterations=100, seed=None, initialModel=None):
        """
        Train a Gaussian Mixture clustering model.

        :param rdd:
          Training points as an `RDD` of `Vector` or convertible
          sequence types.
        :param k:
          Number of independent Gaussians in the mixture model.
        :param convergenceTol:
          Maximum change in log-likelihood at which convergence is
          considered to have occurred.
          (default: 1e-3)
        :param maxIterations:
          Maximum number of iterations allowed.
          (default: 100)
        :param seed:
          Random seed for initial Gaussian distribution. Set as None to
          generate seed based on system time.
          (default: None)
        :param initialModel:
          Initial GMM starting point, bypassing the random
          initialization.
          (default: None)
        """
        initialModelWeights = None
        initialModelMu = None
        initialModelSigma = None
        if initialModel is not None:
            if initialModel.k != k:
                raise Exception("Mismatched cluster count, initialModel.k = %s, however k = %s"
                                % (initialModel.k, k))
            initialModelWeights = list(initialModel.weights)
            initialModelMu = [initialModel.gaussians[i].mu for i in range(initialModel.k)]
            initialModelSigma = [initialModel.gaussians[i].sigma for i in range(initialModel.k)]
        java_model = callMLlibFunc("trainGaussianMixtureModel", rdd.map(_convert_to_vector),
                                   k, convergenceTol, maxIterations, seed,
                                   initialModelWeights, initialModelMu, initialModelSigma)
        return GaussianMixtureModel(java_model)


Instruction:
Implement a Python 3 function for
loading a PowerIterationClusteringModel from the given path.

Response:
def load(cls, sc, path):
        """
        Load a model from the given path.
        """
        model = cls._load_java(sc, path)
        wrapper =\
            sc._jvm.org.apache.spark.mllib.api.python.PowerIterationClusteringModelWrapper(model)
        return PowerIterationClusteringModel(wrapper)


Instruction:
Explain what the following Python 3 code does
def train(cls, rdd, k, maxIterations=100, initMode="random"):
        model = callMLlibFunc("trainPowerIterationClusteringModel",
                              rdd.map(_convert_to_vector), int(k), int(maxIterations), initMode)
        return PowerIterationClusteringModel(model)

Response: r Trains a power iteration clustering model on the given RDD.


Instruction:
How would you explain what the following Python 3 function does
def update(self, data, decayFactor, timeUnit):
        if not isinstance(data, RDD):
            raise TypeError("Data should be of an RDD, got %s." % type(data))
        data = data.map(_convert_to_vector)
        decayFactor = float(decayFactor)
        if timeUnit not in ["batches", "points"]:
            raise ValueError(
                "timeUnit should be 'batches' or 'points', got %s." % timeUnit)
        vectorCenters = [_convert_to_vector(center) for center in self.centers]
        updatedModel = callMLlibFunc(
            "updateStreamingKMeansModel", vectorCenters, self._clusterWeights,
            data, decayFactor, timeUnit)
        self.centers = array(updatedModel[0])
        self._clusterWeights = list(updatedModel[1])
        return self

Response: Update the k - MEAN model with new data.


Instruction:
Can you tell what is the following Python 3 function doing
def setHalfLife(self, halfLife, timeUnit):
        self._timeUnit = timeUnit
        self._decayFactor = exp(log(0.5) / halfLife)
        return self

Response: Sets the number of batches after which the centroids of that set have half the weightage.


Instruction:
Implement a function in Python 3 to
set initial centers for this store.

Response:
def setInitialCenters(self, centers, weights):
        """
        Set initial centers. Should be set before calling trainOn.
        """
        self._model = StreamingKMeansModel(centers, weights)
        return self


Instruction:
Write a Python 3 script for
setting the initial centres to be random samples from the gaussian population with constant weights.

Response:
def setRandomCenters(self, dim, weight, seed):
        """
        Set the initial centres to be random samples from
        a gaussian population with constant weights.
        """
        rng = random.RandomState(seed)
        clusterCenters = rng.randn(self._k, dim)
        clusterWeights = tile(weight, self._k)
        self._model = StreamingKMeansModel(clusterCenters, clusterWeights)
        return self


Instruction:
Create a Python 3 function to
train the model on the incoming dstream.

Response:
def trainOn(self, dstream):
        """Train the model on the incoming dstream."""
        self._validate(dstream)

        def update(rdd):
            self._model.update(rdd, self._decayFactor, self._timeUnit)

        dstream.foreachRDD(update)


Instruction:
Here you have a function in Python 3, explain what it does
def predictOn(self, dstream):
        self._validate(dstream)
        return dstream.map(lambda x: self._model.predict(x))

Response: Make predictions on a dstream.


Instruction:
Create a Python 3 function to
make predictions on a keyed dstream.

Response:
def predictOnValues(self, dstream):
        """
        Make predictions on a keyed dstream.
        Returns a transformed dstream object.
        """
        self._validate(dstream)
        return dstream.mapValues(lambda x: self._model.predict(x))


Instruction:
Implement a Python 3 function for
returning the topics described by weighted terms.

Response:
def describeTopics(self, maxTermsPerTopic=None):
        """Return the topics described by weighted terms.

        WARNING: If vocabSize and k are large, this can return a large object!

        :param maxTermsPerTopic:
          Maximum number of terms to collect for each topic.
          (default: vocabulary size)
        :return:
          Array over topics. Each topic is represented as a pair of
          matching arrays: (term indices, term weights in topic).
          Each topic's terms are sorted in order of decreasing weight.
        """
        if maxTermsPerTopic is None:
            topics = self.call("describeTopics")
        else:
            topics = self.call("describeTopics", maxTermsPerTopic)
        return topics


Instruction:
Here you have a function in Python 3, explain what it does
def load(cls, sc, path):
        if not isinstance(sc, SparkContext):
            raise TypeError("sc should be a SparkContext, got type %s" % type(sc))
        if not isinstance(path, basestring):
            raise TypeError("path should be a basestring, got type %s" % type(path))
        model = callMLlibFunc("loadLDAModel", sc, path)
        return LDAModel(model)

Response: Load the LDAModel from disk.


Instruction:
Make a summary of the following Python 3 code
def train(cls, rdd, k=10, maxIterations=20, docConcentration=-1.0,
              topicConcentration=-1.0, seed=None, checkpointInterval=10, optimizer="em"):
        """Train a LDA model.

        :param rdd:
          RDD of documents, which are tuples of document IDs and term
          (word) count vectors. The term count vectors are "bags of
          words" with a fixed-size vocabulary (where the vocabulary size
          is the length of the vector). Document IDs must be unique
          and >= 0.
        :param k:
          Number of topics to infer, i.e., the number of soft cluster
          centers.
          (default: 10)
        :param maxIterations:
          Maximum number of iterations allowed.
          (default: 20)
        :param docConcentration:
          Concentration parameter (commonly named "alpha") for the prior
          placed on documents' distributions over topics ("theta").
          (default: -1.0)
        :param topicConcentration:
          Concentration parameter (commonly named "beta" or "eta") for
          the prior placed on topics' distributions over terms.
          (default: -1.0)
        :param seed:
          Random seed for cluster initialization. Set as None to generate
          seed based on system time.
          (default: None)
        :param checkpointInterval:
          Period (in iterations) between checkpoints.
          (default: 10)
        :param optimizer:
          LDAOptimizer used to perform the actual calculation. Currently
          "em", "online" are supported.
          (default: "em")
        """
        model = callMLlibFunc("trainLDAModel", rdd, k, maxIterations,
                              docConcentration, topicConcentration, seed,
                              checkpointInterval, optimizer)
        return LDAModel(model)

Response: Train a LDA model.


Instruction:
Explain what the following Python 3 code does
def _to_java_object_rdd(rdd):
    rdd = rdd._reserialize(AutoBatchedSerializer(PickleSerializer()))
    return rdd.ctx._jvm.org.apache.spark.mllib.api.python.SerDe.pythonToJava(rdd._jrdd, True)

Response: Return a JavaRDD of Object by unpickling


Instruction:
Here you have a function in Python 3, explain what it does
def _py2java(sc, obj):
    if isinstance(obj, RDD):
        obj = _to_java_object_rdd(obj)
    elif isinstance(obj, DataFrame):
        obj = obj._jdf
    elif isinstance(obj, SparkContext):
        obj = obj._jsc
    elif isinstance(obj, list):
        obj = [_py2java(sc, x) for x in obj]
    elif isinstance(obj, JavaObject):
        pass
    elif isinstance(obj, (int, long, float, bool, bytes, unicode)):
        pass
    else:
        data = bytearray(PickleSerializer().dumps(obj))
        obj = sc._jvm.org.apache.spark.mllib.api.python.SerDe.loads(data)
    return obj

Response: Convert Python object into Java object.


Instruction:
Explain what the following Python 3 code does
def callMLlibFunc(name, *args):
    sc = SparkContext.getOrCreate()
    api = getattr(sc._jvm.PythonMLLibAPI(), name)
    return callJavaFunc(sc, api, *args)

Response: Call the specified PythonMLLib API with the given arguments.


Instruction:
Implement a function in Python 3 to
call method of java_model

Response:
def call(self, name, *a):
        """Call method of java_model"""
        return callJavaFunc(self._sc, getattr(self._java_model, name), *a)


Instruction:
How would you implement a function in Python 3 that
returns a new DStream in which each RDD has a single element generated by counting each element of this DStream.

Response:
def count(self):
        """
        Return a new DStream in which each RDD has a single element
        generated by counting each RDD of this DStream.
        """
        return self.mapPartitions(lambda i: [sum(1 for _ in i)]).reduce(operator.add)


Instruction:
Can you tell what is the following Python 3 function doing
def filter(self, f):
        def func(iterator):
            return filter(f, iterator)
        return self.mapPartitions(func, True)

Response: Return a new DStream containing only the elements that satisfy the predicate.


Instruction:
Can you implement a function in Python 3 that
returns a new DStream by applying a function to each element of the DStream.

Response:
def map(self, f, preservesPartitioning=False):
        """
        Return a new DStream by applying a function to each element of DStream.
        """
        def func(iterator):
            return map(f, iterator)
        return self.mapPartitions(func, preservesPartitioning)


Instruction:
How would you code a function in Python 3 to
return a new DStream by applying a function to each RDD of each RDD in which each RDD is generated by applying mapPartitionsWithIndex.

Response:
def mapPartitionsWithIndex(self, f, preservesPartitioning=False):
        """
        Return a new DStream in which each RDD is generated by applying
        mapPartitionsWithIndex() to each RDDs of this DStream.
        """
        return self.transform(lambda rdd: rdd.mapPartitionsWithIndex(f, preservesPartitioning))


Instruction:
Given the following Python 3 function, write the documentation
def reduce(self, func):
        return self.map(lambda x: (None, x)).reduceByKey(func, 1).map(lambda x: x[1])

Response:
Return a new DStream in which each RDD has a single element
        generated by reducing each element of each RDD.


Instruction:
Explain what the following Python 3 code does
def reduceByKey(self, func, numPartitions=None):
        if numPartitions is None:
            numPartitions = self._sc.defaultParallelism
        return self.combineByKey(lambda x: x, func, func, numPartitions)

Response: Return a new DStream by applying reduceByKey to each RDD.


Instruction:
Write a Python 3 script to
return a new DStream by applying combineByKey to each RDD.

Response:
def combineByKey(self, createCombiner, mergeValue, mergeCombiners,
                     numPartitions=None):
        """
        Return a new DStream by applying combineByKey to each RDD.
        """
        if numPartitions is None:
            numPartitions = self._sc.defaultParallelism

        def func(rdd):
            return rdd.combineByKey(createCombiner, mergeValue, mergeCombiners, numPartitions)
        return self.transform(func)


Instruction:
Make a summary of the following Python 3 code
def partitionBy(self, numPartitions, partitionFunc=portable_hash):
        return self.transform(lambda rdd: rdd.partitionBy(numPartitions, partitionFunc))

Response: Return a new DStream in which each RDD is partitioned by numPartitions partitions.


Instruction:
Given the following Python 3 function, write the documentation
def foreachRDD(self, func):
        if func.__code__.co_argcount == 1:
            old_func = func
            func = lambda t, rdd: old_func(rdd)
        jfunc = TransformFunction(self._sc, func, self._jrdd_deserializer)
        api = self._ssc._jvm.PythonDStream
        api.callForeachRDD(self._jdstream, jfunc)

Response: Apply a function to each RDD in this DStream.


Instruction:
Create a Python 3 function to
print the first num elements of each RDD generated in this DStream.

Response:
def pprint(self, num=10):
        """
        Print the first num elements of each RDD generated in this DStream.

        @param num: the number of elements from the first will be printed.
        """
        def takeAndPrint(time, rdd):
            taken = rdd.take(num + 1)
            print("-------------------------------------------")
            print("Time: %s" % time)
            print("-------------------------------------------")
            for record in taken[:num]:
                print(record)
            if len(taken) > num:
                print("...")
            print("")

        self.foreachRDD(takeAndPrint)


Instruction:
How would you implement a function in Python 3 that
persists the RDDs of this DStream with the given storage level.

Response:
def persist(self, storageLevel):
        """
        Persist the RDDs of this DStream with the given storage level
        """
        self.is_cached = True
        javaStorageLevel = self._sc._getJavaStorageLevel(storageLevel)
        self._jdstream.persist(javaStorageLevel)
        return self


Instruction:
Here you have a function in Python 3, explain what it does
def checkpoint(self, interval):
        self.is_checkpointed = True
        self._jdstream.checkpoint(self._ssc._jduration(interval))
        return self

Response:
Enable periodic checkpointing of generated
                         RDDs of this DStream.


Instruction:
Make a summary of the following Python 3 code
def groupByKey(self, numPartitions=None):
        if numPartitions is None:
            numPartitions = self._sc.defaultParallelism
        return self.transform(lambda rdd: rdd.groupByKey(numPartitions))

Response: Return a new DStream by applying groupByKey on each RDD.


Instruction:
Can you tell what is the following Python 3 function doing
def countByValue(self):
        return self.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x+y)

Response: Return a new DStream with the counts of each distinct value in each RDD.


Instruction:
How would you code a function in Python 3 to
save each RDD in this DStream as text files using string representation of elements.

Response:
def saveAsTextFiles(self, prefix, suffix=None):
        """
        Save each RDD in this DStream as at text file, using string
        representation of elements.
        """
        def saveAsTextFile(t, rdd):
            path = rddToFileName(prefix, suffix, t)
            try:
                rdd.saveAsTextFile(path)
            except Py4JJavaError as e:
                # after recovered from checkpointing, the foreachRDD may
                # be called twice
                if 'FileAlreadyExistsException' not in str(e):
                    raise
        return self.foreachRDD(saveAsTextFile)


Instruction:
Make a summary of the following Python 3 code
def transform(self, func):
        if func.__code__.co_argcount == 1:
            oldfunc = func
            func = lambda t, rdd: oldfunc(rdd)
        assert func.__code__.co_argcount == 2, "func should take one or two arguments"
        return TransformedDStream(self, func)

Response:
Return a new DStream in which each RDD is generated by applying a function
            on each RDD of this DStream.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def transformWith(self, func, other, keepSerializer=False):
        if func.__code__.co_argcount == 2:
            oldfunc = func
            func = lambda t, a, b: oldfunc(a, b)
        assert func.__code__.co_argcount == 3, "func should take two or three arguments"
        jfunc = TransformFunction(self._sc, func, self._jrdd_deserializer, other._jrdd_deserializer)
        dstream = self._sc._jvm.PythonTransformed2DStream(self._jdstream.dstream(),
                                                          other._jdstream.dstream(), jfunc)
        jrdd_serializer = self._jrdd_deserializer if keepSerializer else self._sc.serializer
        return DStream(dstream.asJavaDStream(), self._ssc, jrdd_serializer)

Response: Return a new DStream in which each RDD is generated by applying a functionon each RDD of this DStream and other DStreams.


Instruction:
Implement a Python 3 function for
returning a new DStream by unifying data of another DStream with this DStream.

Response:
def union(self, other):
        """
        Return a new DStream by unifying data of another DStream with this DStream.

        @param other: Another DStream having the same interval (i.e., slideDuration)
                     as this DStream.
        """
        if self._slideDuration != other._slideDuration:
            raise ValueError("the two DStream should have same slide duration")
        return self.transformWith(lambda a, b: a.union(b), other, True)


Instruction:
Write a Python 3 function for
returning a new DStream by applying cogroup between this DStream and other.

Response:
def cogroup(self, other, numPartitions=None):
        """
        Return a new DStream by applying 'cogroup' between RDDs of this
        DStream and `other` DStream.

        Hash partitioning is used to generate the RDDs with `numPartitions` partitions.
        """
        if numPartitions is None:
            numPartitions = self._sc.defaultParallelism
        return self.transformWith(lambda a, b: a.cogroup(b, numPartitions), other)


Instruction:
How would you implement a function in Python 3 that
converts datetime or unix_timestamp into Time

Response:
def _jtime(self, timestamp):
        """ Convert datetime or unix_timestamp into Time
        """
        if isinstance(timestamp, datetime):
            timestamp = time.mktime(timestamp.timetuple())
        return self._sc._jvm.Time(long(timestamp * 1000))


Instruction:
Write a Python 3 function for
returning all the RDDs between begin and end.

Response:
def slice(self, begin, end):
        """
        Return all the RDDs between 'begin' to 'end' (both included)

        `begin`, `end` could be datetime.datetime() or unix_timestamp
        """
        jrdds = self._jdstream.slice(self._jtime(begin), self._jtime(end))
        return [RDD(jrdd, self._sc, self._jrdd_deserializer) for jrdd in jrdds]


Instruction:
Create a Python 3 function for
returning a new DStream in which each element in seen in this DStream contains all the elements in seen in a sliding window of time over this DStream.

Response:
def window(self, windowDuration, slideDuration=None):
        """
        Return a new DStream in which each RDD contains all the elements in seen in a
        sliding window of time over this DStream.

        @param windowDuration: width of the window; must be a multiple of this DStream's
                              batching interval
        @param slideDuration:  sliding interval of the window (i.e., the interval after which
                              the new DStream will generate RDDs); must be a multiple of this
                              DStream's batching interval
        """
        self._validate_window_param(windowDuration, slideDuration)
        d = self._ssc._jduration(windowDuration)
        if slideDuration is None:
            return DStream(self._jdstream.window(d), self._ssc, self._jrdd_deserializer)
        s = self._ssc._jduration(slideDuration)
        return DStream(self._jdstream.window(d, s), self._ssc, self._jrdd_deserializer)


Instruction:
Make a summary of the following Python 3 code
def reduceByWindow(self, reduceFunc, invReduceFunc, windowDuration, slideDuration):
        keyed = self.map(lambda x: (1, x))
        reduced = keyed.reduceByKeyAndWindow(reduceFunc, invReduceFunc,
                                             windowDuration, slideDuration, 1)
        return reduced.map(lambda kv: kv[1])

Response: Return a new DStream with a single element generated by reducing all the entries in a sliding window over this DStream.


Instruction:
Can you create a Python 3 function that
returns a new DStream with a count of the number of elements in a given window over this DStream.

Response:
def countByWindow(self, windowDuration, slideDuration):
        """
        Return a new DStream in which each RDD has a single element generated
        by counting the number of elements in a window over this DStream.
        windowDuration and slideDuration are as defined in the window() operation.

        This is equivalent to window(windowDuration, slideDuration).count(),
        but will be more efficient if window is large.
        """
        return self.map(lambda x: 1).reduceByWindow(operator.add, operator.sub,
                                                    windowDuration, slideDuration)


Instruction:
Can you tell what is the following Python 3 function doing
def countByValueAndWindow(self, windowDuration, slideDuration, numPartitions=None):
        keyed = self.map(lambda x: (x, 1))
        counted = keyed.reduceByKeyAndWindow(operator.add, operator.sub,
                                             windowDuration, slideDuration, numPartitions)
        return counted.filter(lambda kv: kv[1] > 0)

Response: Return a new DStream with the count of distinct elements in the given sliding window over this DStream.


Instruction:
Can you write a function in Python 3 where it
returns a new DStream by applying groupByKeyAndWindow over a sliding window.

Response:
def groupByKeyAndWindow(self, windowDuration, slideDuration, numPartitions=None):
        """
        Return a new DStream by applying `groupByKey` over a sliding window.
        Similar to `DStream.groupByKey()`, but applies it over a sliding window.

        @param windowDuration: width of the window; must be a multiple of this DStream's
                              batching interval
        @param slideDuration:  sliding interval of the window (i.e., the interval after which
                              the new DStream will generate RDDs); must be a multiple of this
                              DStream's batching interval
        @param numPartitions:  Number of partitions of each RDD in the new DStream.
        """
        ls = self.mapValues(lambda x: [x])
        grouped = ls.reduceByKeyAndWindow(lambda a, b: a.extend(b) or a, lambda a, b: a[len(b):],
                                          windowDuration, slideDuration, numPartitions)
        return grouped.mapValues(ResultIterable)


Instruction:
Can you create a Python 3 function that
returns a new DStream by applying incremental reduceByKey over a sliding window.

Response:
def reduceByKeyAndWindow(self, func, invFunc, windowDuration, slideDuration=None,
                             numPartitions=None, filterFunc=None):
        """
        Return a new DStream by applying incremental `reduceByKey` over a sliding window.

        The reduced value of over a new window is calculated using the old window's reduce value :
         1. reduce the new values that entered the window (e.g., adding new counts)
         2. "inverse reduce" the old values that left the window (e.g., subtracting old counts)

        `invFunc` can be None, then it will reduce all the RDDs in window, could be slower
        than having `invFunc`.

        @param func:           associative and commutative reduce function
        @param invFunc:        inverse function of `reduceFunc`
        @param windowDuration: width of the window; must be a multiple of this DStream's
                              batching interval
        @param slideDuration:  sliding interval of the window (i.e., the interval after which
                              the new DStream will generate RDDs); must be a multiple of this
                              DStream's batching interval
        @param numPartitions:  number of partitions of each RDD in the new DStream.
        @param filterFunc:     function to filter expired key-value pairs;
                              only pairs that satisfy the function are retained
                              set this to null if you do not want to filter
        """
        self._validate_window_param(windowDuration, slideDuration)
        if numPartitions is None:
            numPartitions = self._sc.defaultParallelism

        reduced = self.reduceByKey(func, numPartitions)

        if invFunc:
            def reduceFunc(t, a, b):
                b = b.reduceByKey(func, numPartitions)
                r = a.union(b).reduceByKey(func, numPartitions) if a else b
                if filterFunc:
                    r = r.filter(filterFunc)
                return r

            def invReduceFunc(t, a, b):
                b = b.reduceByKey(func, numPartitions)
                joined = a.leftOuterJoin(b, numPartitions)
                return joined.mapValues(lambda kv: invFunc(kv[0], kv[1])
                                        if kv[1] is not None else kv[0])

            jreduceFunc = TransformFunction(self._sc, reduceFunc, reduced._jrdd_deserializer)
            jinvReduceFunc = TransformFunction(self._sc, invReduceFunc, reduced._jrdd_deserializer)
            if slideDuration is None:
                slideDuration = self._slideDuration
            dstream = self._sc._jvm.PythonReducedWindowedDStream(
                reduced._jdstream.dstream(),
                jreduceFunc, jinvReduceFunc,
                self._ssc._jduration(windowDuration),
                self._ssc._jduration(slideDuration))
            return DStream(dstream.asJavaDStream(), self._ssc, self._sc.serializer)
        else:
            return reduced.window(windowDuration, slideDuration).reduceByKey(func, numPartitions)


Instruction:
Write a Python 3 script for
returning a new DStream where the state for each key is updated by applying updateFunc on the previous state of the key and the new values of the key.

Response:
def updateStateByKey(self, updateFunc, numPartitions=None, initialRDD=None):
        """
        Return a new "state" DStream where the state for each key is updated by applying
        the given function on the previous state of the key and the new values of the key.

        @param updateFunc: State update function. If this function returns None, then
                           corresponding state key-value pair will be eliminated.
        """
        if numPartitions is None:
            numPartitions = self._sc.defaultParallelism

        if initialRDD and not isinstance(initialRDD, RDD):
            initialRDD = self._sc.parallelize(initialRDD)

        def reduceFunc(t, a, b):
            if a is None:
                g = b.groupByKey(numPartitions).mapValues(lambda vs: (list(vs), None))
            else:
                g = a.cogroup(b.partitionBy(numPartitions), numPartitions)
                g = g.mapValues(lambda ab: (list(ab[1]), list(ab[0])[0] if len(ab[0]) else None))
            state = g.mapValues(lambda vs_s: updateFunc(vs_s[0], vs_s[1]))
            return state.filter(lambda k_v: k_v[1] is not None)

        jreduceFunc = TransformFunction(self._sc, reduceFunc,
                                        self._sc.serializer, self._jrdd_deserializer)
        if initialRDD:
            initialRDD = initialRDD._reserialize(self._jrdd_deserializer)
            dstream = self._sc._jvm.PythonStateDStream(self._jdstream.dstream(), jreduceFunc,
                                                       initialRDD._jrdd)
        else:
            dstream = self._sc._jvm.PythonStateDStream(self._jdstream.dstream(), jreduceFunc)

        return DStream(dstream.asJavaDStream(), self._ssc, self._sc.serializer)


Instruction:
Can you generate the documentation for the following Python 3 function
def setParams(self, minSupport=0.3, minConfidence=0.8, itemsCol="items",
                  predictionCol="prediction", numPartitions=None):
        """
        setParams(self, minSupport=0.3, minConfidence=0.8, itemsCol="items", \
                  predictionCol="prediction", numPartitions=None)
        """
        kwargs = self._input_kwargs
        return self._set(**kwargs)

Response: Sets the parameters of the current object to match the input parameters.


Instruction:
Write a Python 3 function for
setting the parameters of the object to be used for the log record.

Response:
def setParams(self, minSupport=0.1, maxPatternLength=10, maxLocalProjDBSize=32000000,
                  sequenceCol="sequence"):
        """
        setParams(self, minSupport=0.1, maxPatternLength=10, maxLocalProjDBSize=32000000, \
                  sequenceCol="sequence")
        """
        kwargs = self._input_kwargs
        return self._set(**kwargs)


Instruction:
Implement a function in Python 3 to
find the complete set of frequent sequential patterns in the input sequences of itemsets.

Response:
def findFrequentSequentialPatterns(self, dataset):
        """
        .. note:: Experimental

        Finds the complete set of frequent sequential patterns in the input sequences of itemsets.

        :param dataset: A dataframe containing a sequence column which is
                        `ArrayType(ArrayType(T))` type, T is the item type for the input dataset.
        :return: A `DataFrame` that contains columns of sequence and corresponding frequency.
                 The schema of it will be:
                 - `sequence: ArrayType(ArrayType(T))` (T is the item type)
                 - `freq: Long`

        >>> from pyspark.ml.fpm import PrefixSpan
        >>> from pyspark.sql import Row
        >>> df = sc.parallelize([Row(sequence=[[1, 2], [3]]),
        ...                      Row(sequence=[[1], [3, 2], [1, 2]]),
        ...                      Row(sequence=[[1, 2], [5]]),
        ...                      Row(sequence=[[6]])]).toDF()
        >>> prefixSpan = PrefixSpan(minSupport=0.5, maxPatternLength=5)
        >>> prefixSpan.findFrequentSequentialPatterns(df).sort("sequence").show(truncate=False)
        +----------+----+
        |sequence  |freq|
        +----------+----+
        |[[1]]     |3   |
        |[[1], [3]]|2   |
        |[[1, 2]]  |3   |
        |[[2]]     |3   |
        |[[3]]     |2   |
        +----------+----+

        .. versionadded:: 2.4.0
        """
        self._transfer_params_to_java()
        jdf = self._java_obj.findFrequentSequentialPatterns(dataset._jdf)
        return DataFrame(jdf, dataset.sql_ctx)


Instruction:
Write a Python 3 function that can
return a CallSite representing the first Spark call in the current call stack.

Response:
def first_spark_call():
    """
    Return a CallSite representing the first Spark call in the current call stack.
    """
    tb = traceback.extract_stack()
    if len(tb) == 0:
        return None
    file, line, module, what = tb[len(tb) - 1]
    sparkpath = os.path.dirname(file)
    first_spark_frame = len(tb) - 1
    for i in range(0, len(tb)):
        file, line, fun, what = tb[i]
        if file.startswith(sparkpath):
            first_spark_frame = i
            break
    if first_spark_frame == 0:
        file, line, fun, what = tb[0]
        return CallSite(function=fun, file=file, linenum=line)
    sfile, sline, sfun, swhat = tb[first_spark_frame]
    ufile, uline, ufun, uwhat = tb[first_spark_frame - 1]
    return CallSite(function=sfun, file=ufile, linenum=uline)


Instruction:
Explain what the following Python 3 code does
def parsePoint(line):
    values = [float(s) for s in line.split(' ')]
    if values[0] == -1:   # Convert -1 labels to 0 for MLlib
        values[0] = 0
    return LabeledPoint(values[0], values[1:])

Response: Parse a line of text into an MLlib LabeledPoint object.


Instruction:
How would you explain what the following Python 3 function does
def fMeasure(self, label, beta=None):
        if beta is None:
            return self.call("fMeasure", label)
        else:
            return self.call("fMeasure", label, beta)

Response: Returns f - measure.


Instruction:
Make a summary of the following Python 3 code
def precision(self, label=None):
        if label is None:
            return self.call("precision")
        else:
            return self.call("precision", float(label))

Response: Returns precision for a given label


Instruction:
Here you have a function in Python 3, explain what it does
def recall(self, label=None):
        if label is None:
            return self.call("recall")
        else:
            return self.call("recall", float(label))

Response: Returns the recall of a given label.


Instruction:
How would you explain what the following Python 3 function does
def f1Measure(self, label=None):
        if label is None:
            return self.call("f1Measure")
        else:
            return self.call("f1Measure", float(label))

Response: Returns the f1Measure of the current user.


Instruction:
Can you create a Python 3 function that
returns the content as an RDD of Row.

Response:
def rdd(self):
        """Returns the content as an :class:`pyspark.RDD` of :class:`Row`.
        """
        if self._lazy_rdd is None:
            jrdd = self._jdf.javaToPython()
            self._lazy_rdd = RDD(jrdd, self.sql_ctx._sc, BatchedSerializer(PickleSerializer()))
        return self._lazy_rdd


Instruction:
Can you write a function in Python 3 where it
converts a DataFrame into a : class : RDD of string.

Response:
def toJSON(self, use_unicode=True):
        """Converts a :class:`DataFrame` into a :class:`RDD` of string.

        Each row is turned into a JSON document as one element in the returned RDD.

        >>> df.toJSON().first()
        u'{"age":2,"name":"Alice"}'
        """
        rdd = self._jdf.toJSON()
        return RDD(rdd.toJavaRDD(), self._sc, UTF8Deserializer(use_unicode))


Instruction:
Given the following Python 3 function, write the documentation
def schema(self):
        if self._schema is None:
            try:
                self._schema = _parse_datatype_json_string(self._jdf.schema().json())
            except AttributeError as e:
                raise Exception(
                    "Unable to parse datatype from schema. %s" % e)
        return self._schema

Response: Returns the schema of this DataFrame as a StructType.


Instruction:
Can you write a function in Python 3 where it
prints the logical and physical plan for debugging purpose.

Response:
def explain(self, extended=False):
        """Prints the (logical and physical) plans to the console for debugging purpose.

        :param extended: boolean, default ``False``. If ``False``, prints only the physical plan.

        >>> df.explain()
        == Physical Plan ==
        *(1) Scan ExistingRDD[age#0,name#1]

        >>> df.explain(True)
        == Parsed Logical Plan ==
        ...
        == Analyzed Logical Plan ==
        ...
        == Optimized Logical Plan ==
        ...
        == Physical Plan ==
        ...
        """
        if extended:
            print(self._jdf.queryExecution().toString())
        else:
            print(self._jdf.queryExecution().simpleString())


Instruction:
Write a Python 3 script to
return a new DataFrame containing rows in this DataFrame but not in another.

Response:
def exceptAll(self, other):
        """Return a new :class:`DataFrame` containing rows in this :class:`DataFrame` but
        not in another :class:`DataFrame` while preserving duplicates.

        This is equivalent to `EXCEPT ALL` in SQL.

        >>> df1 = spark.createDataFrame(
        ...         [("a", 1), ("a", 1), ("a", 1), ("a", 2), ("b",  3), ("c", 4)], ["C1", "C2"])
        >>> df2 = spark.createDataFrame([("a", 1), ("b", 3)], ["C1", "C2"])

        >>> df1.exceptAll(df2).show()
        +---+---+
        | C1| C2|
        +---+---+
        |  a|  1|
        |  a|  1|
        |  a|  2|
        |  c|  4|
        +---+---+

        Also as standard in SQL, this function resolves columns by position (not by name).
        """
        return DataFrame(self._jdf.exceptAll(other._jdf), self.sql_ctx)