Instruction:
Can you create a Python 3 function that
saves the contents of the DataFrame in Parquet format at the specified path.

Response:
def parquet(self, path, mode=None, partitionBy=None, compression=None):
        """Saves the content of the :class:`DataFrame` in Parquet format at the specified path.

        :param path: the path in any Hadoop supported file system
        :param mode: specifies the behavior of the save operation when data already exists.

            * ``append``: Append contents of this :class:`DataFrame` to existing data.
            * ``overwrite``: Overwrite existing data.
            * ``ignore``: Silently ignore this operation if data already exists.
            * ``error`` or ``errorifexists`` (default case): Throw an exception if data already \
                exists.
        :param partitionBy: names of partitioning columns
        :param compression: compression codec to use when saving to file. This can be one of the
                            known case-insensitive shorten names (none, uncompressed, snappy, gzip,
                            lzo, brotli, lz4, and zstd). This will override
                            ``spark.sql.parquet.compression.codec``. If None is set, it uses the
                            value specified in ``spark.sql.parquet.compression.codec``.

        >>> df.write.parquet(os.path.join(tempfile.mkdtemp(), 'data'))
        """
        self.mode(mode)
        if partitionBy is not None:
            self.partitionBy(partitionBy)
        self._set_opts(compression=compression)
        self._jwrite.parquet(path)


Instruction:
Implement a Python 3 function for
saving the content of the DataFrame in a text file at the specified path.

Response:
def text(self, path, compression=None, lineSep=None):
        """Saves the content of the DataFrame in a text file at the specified path.
        The text files will be encoded as UTF-8.

        :param path: the path in any Hadoop supported file system
        :param compression: compression codec to use when saving to file. This can be one of the
                            known case-insensitive shorten names (none, bzip2, gzip, lz4,
                            snappy and deflate).
        :param lineSep: defines the line separator that should be used for writing. If None is
                        set, it uses the default value, ``\\n``.

        The DataFrame must have only one column that is of string type.
        Each row becomes a new line in the output file.
        """
        self._set_opts(compression=compression, lineSep=lineSep)
        self._jwrite.text(path)


Instruction:
Write a Python 3 script for
saving the contents of the DataFrame in ORC format at the specified path.

Response:
def orc(self, path, mode=None, partitionBy=None, compression=None):
        """Saves the content of the :class:`DataFrame` in ORC format at the specified path.

        :param path: the path in any Hadoop supported file system
        :param mode: specifies the behavior of the save operation when data already exists.

            * ``append``: Append contents of this :class:`DataFrame` to existing data.
            * ``overwrite``: Overwrite existing data.
            * ``ignore``: Silently ignore this operation if data already exists.
            * ``error`` or ``errorifexists`` (default case): Throw an exception if data already \
                exists.
        :param partitionBy: names of partitioning columns
        :param compression: compression codec to use when saving to file. This can be one of the
                            known case-insensitive shorten names (none, snappy, zlib, and lzo).
                            This will override ``orc.compress`` and
                            ``spark.sql.orc.compression.codec``. If None is set, it uses the value
                            specified in ``spark.sql.orc.compression.codec``.

        >>> orc_df = spark.read.orc('python/test_support/sql/orc_partitioned')
        >>> orc_df.write.orc(os.path.join(tempfile.mkdtemp(), 'data'))
        """
        self.mode(mode)
        if partitionBy is not None:
            self.partitionBy(partitionBy)
        self._set_opts(compression=compression)
        self._jwrite.orc(path)


Instruction:
Here you have a function in Python 3, explain what it does
def jdbc(self, url, table, mode=None, properties=None):
        if properties is None:
            properties = dict()
        jprop = JavaClass("java.util.Properties", self._spark._sc._gateway._gateway_client)()
        for k in properties:
            jprop.setProperty(k, properties[k])
        self.mode(mode)._jwrite.jdbc(url, table, jprop)

Response: Saves the contents of the DataFrame to an external database table via JDBC.


Instruction:
Implement a function in Python 3 to
create an input stream that pulls messages from a Kinesis stream.

Response:
def createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName,
                     initialPositionInStream, checkpointInterval,
                     storageLevel=StorageLevel.MEMORY_AND_DISK_2,
                     awsAccessKeyId=None, awsSecretKey=None, decoder=utf8_decoder,
                     stsAssumeRoleArn=None, stsSessionName=None, stsExternalId=None):
        """
        Create an input stream that pulls messages from a Kinesis stream. This uses the
        Kinesis Client Library (KCL) to pull messages from Kinesis.

        .. note:: The given AWS credentials will get saved in DStream checkpoints if checkpointing
            is enabled. Make sure that your checkpoint directory is secure.

        :param ssc:  StreamingContext object
        :param kinesisAppName:  Kinesis application name used by the Kinesis Client Library (KCL) to
                                update DynamoDB
        :param streamName:  Kinesis stream name
        :param endpointUrl:  Url of Kinesis service (e.g., https://kinesis.us-east-1.amazonaws.com)
        :param regionName:  Name of region used by the Kinesis Client Library (KCL) to update
                            DynamoDB (lease coordination and checkpointing) and CloudWatch (metrics)
        :param initialPositionInStream:  In the absence of Kinesis checkpoint info, this is the
                                         worker's initial starting position in the stream. The
                                         values are either the beginning of the stream per Kinesis'
                                         limit of 24 hours (InitialPositionInStream.TRIM_HORIZON) or
                                         the tip of the stream (InitialPositionInStream.LATEST).
        :param checkpointInterval:  Checkpoint interval for Kinesis checkpointing. See the Kinesis
                                    Spark Streaming documentation for more details on the different
                                    types of checkpoints.
        :param storageLevel:  Storage level to use for storing the received objects (default is
                              StorageLevel.MEMORY_AND_DISK_2)
        :param awsAccessKeyId:  AWS AccessKeyId (default is None. If None, will use
                                DefaultAWSCredentialsProviderChain)
        :param awsSecretKey:  AWS SecretKey (default is None. If None, will use
                              DefaultAWSCredentialsProviderChain)
        :param decoder:  A function used to decode value (default is utf8_decoder)
        :param stsAssumeRoleArn: ARN of IAM role to assume when using STS sessions to read from
                                 the Kinesis stream (default is None).
        :param stsSessionName: Name to uniquely identify STS sessions used to read from Kinesis
                               stream, if STS is being used (default is None).
        :param stsExternalId: External ID that can be used to validate against the assumed IAM
                              role's trust policy, if STS is being used (default is None).
        :return: A DStream object
        """
        jlevel = ssc._sc._getJavaStorageLevel(storageLevel)
        jduration = ssc._jduration(checkpointInterval)

        try:
            # Use KinesisUtilsPythonHelper to access Scala's KinesisUtils
            helper = ssc._jvm.org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper()
        except TypeError as e:
            if str(e) == "'JavaPackage' object is not callable":
                _print_missing_jar(
                    "Streaming's Kinesis",
                    "streaming-kinesis-asl",
                    "streaming-kinesis-asl-assembly",
                    ssc.sparkContext.version)
            raise
        jstream = helper.createStream(ssc._jssc, kinesisAppName, streamName, endpointUrl,
                                      regionName, initialPositionInStream, jduration, jlevel,
                                      awsAccessKeyId, awsSecretKey, stsAssumeRoleArn,
                                      stsSessionName, stsExternalId)
        stream = DStream(jstream, ssc, NoOpSerializer())
        return stream.map(lambda v: decoder(v))


Instruction:
Write a Python 3 script to
prompt the user to assign the issue to in jira.

Response:
def choose_jira_assignee(issue, asf_jira):
    """
    Prompt the user to choose who to assign the issue to in jira, given a list of candidates,
    including the original reporter and all commentors
    """
    while True:
        try:
            reporter = issue.fields.reporter
            commentors = map(lambda x: x.author, issue.fields.comment.comments)
            candidates = set(commentors)
            candidates.add(reporter)
            candidates = list(candidates)
            print("JIRA is unassigned, choose assignee")
            for idx, author in enumerate(candidates):
                if author.key == "apachespark":
                    continue
                annotations = ["Reporter"] if author == reporter else []
                if author in commentors:
                    annotations.append("Commentor")
                print("[%d] %s (%s)" % (idx, author.displayName, ",".join(annotations)))
            raw_assignee = input(
                "Enter number of user, or userid, to assign to (blank to leave unassigned):")
            if raw_assignee == "":
                return None
            else:
                try:
                    id = int(raw_assignee)
                    assignee = candidates[id]
                except:
                    # assume it's a user id, and try to assign (might fail, we just prompt again)
                    assignee = asf_jira.user(raw_assignee)
                asf_jira.assign_issue(issue.key, assignee.key)
                return assignee
        except KeyboardInterrupt:
            raise
        except:
            traceback.print_exc()
            print("Error assigning JIRA, try again (or leave blank and fix manually)")


Instruction:
How would you code a function in Python 3 to
standardize the JIRA ref.

Response:
def standardize_jira_ref(text):
    """
    Standardize the [SPARK-XXXXX] [MODULE] prefix
    Converts "[SPARK-XXX][mllib] Issue", "[MLLib] SPARK-XXX. Issue" or "SPARK XXX [MLLIB]: Issue" to
    "[SPARK-XXX][MLLIB] Issue"

    >>> standardize_jira_ref(
    ...     "[SPARK-5821] [SQL] ParquetRelation2 CTAS should check if delete is successful")
    '[SPARK-5821][SQL] ParquetRelation2 CTAS should check if delete is successful'
    >>> standardize_jira_ref(
    ...     "[SPARK-4123][Project Infra][WIP]: Show new dependencies added in pull requests")
    '[SPARK-4123][PROJECT INFRA][WIP] Show new dependencies added in pull requests'
    >>> standardize_jira_ref("[MLlib] Spark  5954: Top by key")
    '[SPARK-5954][MLLIB] Top by key'
    >>> standardize_jira_ref("[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl")
    '[SPARK-979] a LRU scheduler for load balancing in TaskSchedulerImpl'
    >>> standardize_jira_ref(
    ...     "SPARK-1094 Support MiMa for reporting binary compatibility across versions.")
    '[SPARK-1094] Support MiMa for reporting binary compatibility across versions.'
    >>> standardize_jira_ref("[WIP]  [SPARK-1146] Vagrant support for Spark")
    '[SPARK-1146][WIP] Vagrant support for Spark'
    >>> standardize_jira_ref(
    ...     "SPARK-1032. If Yarn app fails before registering, app master stays aroun...")
    '[SPARK-1032] If Yarn app fails before registering, app master stays aroun...'
    >>> standardize_jira_ref(
    ...     "[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.")
    '[SPARK-6250][SPARK-6146][SPARK-5911][SQL] Types are now reserved words in DDL parser.'
    >>> standardize_jira_ref("Additional information for users building from source code")
    'Additional information for users building from source code'
    """
    jira_refs = []
    components = []

    # If the string is compliant, no need to process any further
    if (re.search(r'^\[SPARK-[0-9]{3,6}\](\[[A-Z0-9_\s,]+\] )+\S+', text)):
        return text

    # Extract JIRA ref(s):
    pattern = re.compile(r'(SPARK[-\s]*[0-9]{3,6})+', re.IGNORECASE)
    for ref in pattern.findall(text):
        # Add brackets, replace spaces with a dash, & convert to uppercase
        jira_refs.append('[' + re.sub(r'\s+', '-', ref.upper()) + ']')
        text = text.replace(ref, '')

    # Extract spark component(s):
    # Look for alphanumeric chars, spaces, dashes, periods, and/or commas
    pattern = re.compile(r'(\[[\w\s,.-]+\])', re.IGNORECASE)
    for component in pattern.findall(text):
        components.append(component.upper())
        text = text.replace(component, '')

    # Cleanup any remaining symbols:
    pattern = re.compile(r'^\W+(.*)', re.IGNORECASE)
    if (pattern.search(text) is not None):
        text = pattern.search(text).groups()[0]

    # Assemble full text (JIRA ref(s), module(s), remaining text)
    clean_text = ''.join(jira_refs).strip() + ''.join(components).strip() + " " + text.strip()

    # Replace multiple spaces with a single space, e.g. if no jira refs and/or components were
    # included
    clean_text = re.sub(r'\s+', ' ', clean_text.strip())

    return clean_text


Instruction:
Can you write a function in Python 3 where it
parses a LIBSVM line into label indices and values.

Response:
def _parse_libsvm_line(line):
        """
        Parses a line in LIBSVM format into (label, indices, values).
        """
        items = line.split(None)
        label = float(items[0])
        nnz = len(items) - 1
        indices = np.zeros(nnz, dtype=np.int32)
        values = np.zeros(nnz)
        for i in xrange(nnz):
            index, value = items[1 + i].split(":")
            indices[i] = int(index) - 1
            values[i] = float(value)
        return label, indices, values


Instruction:
Make a summary of the following Python 3 code
def _convert_labeled_point_to_libsvm(p):
        from pyspark.mllib.regression import LabeledPoint
        assert isinstance(p, LabeledPoint)
        items = [str(p.label)]
        v = _convert_to_vector(p.features)
        if isinstance(v, SparseVector):
            nnz = len(v.indices)
            for i in xrange(nnz):
                items.append(str(v.indices[i] + 1) + ":" + str(v.values[i]))
        else:
            for i in xrange(len(v)):
                items.append(str(i + 1) + ":" + str(v[i]))
        return " ".join(items)

Response: Converts a LabeledPoint to a string in LIBSVM format.


Instruction:
How would you explain what the following Python 3 function does
def loadLibSVMFile(sc, path, numFeatures=-1, minPartitions=None):
        from pyspark.mllib.regression import LabeledPoint

        lines = sc.textFile(path, minPartitions)
        parsed = lines.map(lambda l: MLUtils._parse_libsvm_line(l))
        if numFeatures <= 0:
            parsed.cache()
            numFeatures = parsed.map(lambda x: -1 if x[1].size == 0 else x[1][-1]).reduce(max) + 1
        return parsed.map(lambda x: LabeledPoint(x[0], Vectors.sparse(numFeatures, x[1], x[2])))

Response: Loads labeled data from a LIBSVM file.


Instruction:
Explain what the following Python 3 code does
def saveAsLibSVMFile(data, dir):
        lines = data.map(lambda p: MLUtils._convert_labeled_point_to_libsvm(p))
        lines.saveAsTextFile(dir)

Response: Save labeled data in LIBSVM format.


Instruction:
Write a Python 3 script for
loading labeled points from a file or directory path.

Response:
def loadLabeledPoints(sc, path, minPartitions=None):
        """
        Load labeled points saved using RDD.saveAsTextFile.

        :param sc: Spark context
        :param path: file or directory path in any Hadoop-supported file
                     system URI
        :param minPartitions: min number of partitions
        @return: labeled data stored as an RDD of LabeledPoint

        >>> from tempfile import NamedTemporaryFile
        >>> from pyspark.mllib.util import MLUtils
        >>> from pyspark.mllib.regression import LabeledPoint
        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])),
        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
        >>> tempFile = NamedTemporaryFile(delete=True)
        >>> tempFile.close()
        >>> sc.parallelize(examples, 1).saveAsTextFile(tempFile.name)
        >>> MLUtils.loadLabeledPoints(sc, tempFile.name).collect()
        [LabeledPoint(1.1, (3,[0,2],[-1.23,4.56e-07])), LabeledPoint(0.0, [1.01,2.02,3.03])]
        """
        minPartitions = minPartitions or min(sc.defaultParallelism, 2)
        return callMLlibFunc("loadLabeledPoints", sc, path, minPartitions)


Instruction:
Implement a function in Python 3 to
return a new vector with 1. 0 ( bias ) appended to the end of the input vector.

Response:
def appendBias(data):
        """
        Returns a new vector with `1.0` (bias) appended to
        the end of the input vector.
        """
        vec = _convert_to_vector(data)
        if isinstance(vec, SparseVector):
            newIndices = np.append(vec.indices, len(vec))
            newValues = np.append(vec.values, 1.0)
            return SparseVector(len(vec) + 1, newIndices, newValues)
        else:
            return _convert_to_vector(np.append(vec.toArray(), 1.0))


Instruction:
Can you generate the documentation for the following Python 3 function
def convertVectorColumnsToML(dataset, *cols):
        if not isinstance(dataset, DataFrame):
            raise TypeError("Input dataset must be a DataFrame but got {}.".format(type(dataset)))
        return callMLlibFunc("convertVectorColumnsToML", dataset, list(cols))

Response: Converts a list of vector columns in an input DataFrame from the pyspark. ml. linalg. Vector type to the newml.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def generateLinearInput(intercept, weights, xMean, xVariance,
                            nPoints, seed, eps):
        """
        :param: intercept bias factor, the term c in X'w + c
        :param: weights   feature vector, the term w in X'w + c
        :param: xMean     Point around which the data X is centered.
        :param: xVariance Variance of the given data
        :param: nPoints   Number of points to be generated
        :param: seed      Random Seed
        :param: eps       Used to scale the noise. If eps is set high,
                          the amount of gaussian noise added is more.

        Returns a list of LabeledPoints of length nPoints
        """
        weights = [float(weight) for weight in weights]
        xMean = [float(mean) for mean in xMean]
        xVariance = [float(var) for var in xVariance]
        return list(callMLlibFunc(
            "generateLinearInputWrapper", float(intercept), weights, xMean,
            xVariance, int(nPoints), int(seed), float(eps)))

Response: This function generates a linear input for the given data.


Instruction:
Can you write a function in Python 3 where it
generates an RDD of LabeledPoints.

Response:
def generateLinearRDD(sc, nexamples, nfeatures, eps,
                          nParts=2, intercept=0.0):
        """
        Generate an RDD of LabeledPoints.
        """
        return callMLlibFunc(
            "generateLinearRDDWrapper", sc, int(nexamples), int(nfeatures),
            float(eps), int(nParts), float(intercept))


Instruction:
Can you tell what is the following Python 3 function doing
def train(cls, data, iterations=100, step=1.0, miniBatchFraction=1.0,
              initialWeights=None, regParam=0.0, regType=None, intercept=False,
              validateData=True, convergenceTol=0.001):
        """
        Train a linear regression model using Stochastic Gradient
        Descent (SGD). This solves the least squares regression
        formulation

            f(weights) = 1/(2n) ||A weights - y||^2

        which is the mean squared error. Here the data matrix has n rows,
        and the input RDD holds the set of rows of A, each with its
        corresponding right hand side label y.
        See also the documentation for the precise formulation.

        :param data:
          The training data, an RDD of LabeledPoint.
        :param iterations:
          The number of iterations.
          (default: 100)
        :param step:
          The step parameter used in SGD.
          (default: 1.0)
        :param miniBatchFraction:
          Fraction of data to be used for each SGD iteration.
          (default: 1.0)
        :param initialWeights:
          The initial weights.
          (default: None)
        :param regParam:
          The regularizer parameter.
          (default: 0.0)
        :param regType:
          The type of regularizer used for training our model.
          Supported values:

            - "l1" for using L1 regularization
            - "l2" for using L2 regularization
            - None for no regularization (default)
        :param intercept:
          Boolean parameter which indicates the use or not of the
          augmented representation for training data (i.e., whether bias
          features are activated or not).
          (default: False)
        :param validateData:
          Boolean parameter which indicates if the algorithm should
          validate data before training.
          (default: True)
        :param convergenceTol:
          A condition which decides iteration termination.
          (default: 0.001)
        """
        warnings.warn(
            "Deprecated in 2.0.0. Use ml.regression.LinearRegression.", DeprecationWarning)

        def train(rdd, i):
            return callMLlibFunc("trainLinearRegressionModelWithSGD", rdd, int(iterations),
                                 float(step), float(miniBatchFraction), i, float(regParam),
                                 regType, bool(intercept), bool(validateData),
                                 float(convergenceTol))

        return _regression_train_wrapper(train, LinearRegressionModel, data, initialWeights)

Response:
Train a linear regression model using Stochastic Gradient
            Descent SGD.


Instruction:
Can you tell what is the following Python 3 function doing
def predict(self, x):
        if isinstance(x, RDD):
            return x.map(lambda v: self.predict(v))
        return np.interp(x, self.boundaries, self.predictions)

Response: Predict labels for provided features.


Instruction:
Given the following Python 3 function, write the documentation
def save(self, sc, path):
        java_boundaries = _py2java(sc, self.boundaries.tolist())
        java_predictions = _py2java(sc, self.predictions.tolist())
        java_model = sc._jvm.org.apache.spark.mllib.regression.IsotonicRegressionModel(
            java_boundaries, java_predictions, self.isotonic)
        java_model.save(sc._jsc.sc(), path)

Response: Save an IsotonicRegressionModel to a file.


Instruction:
Can you generate the documentation for the following Python 3 function
def load(cls, sc, path):
        java_model = sc._jvm.org.apache.spark.mllib.regression.IsotonicRegressionModel.load(
            sc._jsc.sc(), path)
        py_boundaries = _java2py(sc, java_model.boundaryVector()).toArray()
        py_predictions = _java2py(sc, java_model.predictionVector()).toArray()
        return IsotonicRegressionModel(py_boundaries, py_predictions, java_model.isotonic)

Response: Load an IsotonicRegressionModel from a file.


Instruction:
Can you tell what is the following Python 3 function doing
def train(cls, data, isotonic=True):
        boundaries, predictions = callMLlibFunc("trainIsotonicRegressionModel",
                                                data.map(_convert_to_vector), bool(isotonic))
        return IsotonicRegressionModel(boundaries.toArray(), predictions.toArray(), isotonic)

Response: Train an isotonic regression model on the given data.


Instruction:
Create a Python 3 function for
computing the column similarities between the given column.

Response:
def columnSimilarities(self, threshold=0.0):
        """
        Compute similarities between columns of this matrix.

        The threshold parameter is a trade-off knob between estimate
        quality and computational cost.

        The default threshold setting of 0 guarantees deterministically
        correct results, but uses the brute-force approach of computing
        normalized dot products.

        Setting the threshold to positive values uses a sampling
        approach and incurs strictly less computational cost than the
        brute-force approach. However the similarities computed will
        be estimates.

        The sampling guarantees relative-error correctness for those
        pairs of columns that have similarity greater than the given
        similarity threshold.

        To describe the guarantee, we set some notation:
            * Let A be the smallest in magnitude non-zero element of
              this matrix.
            * Let B be the largest in magnitude non-zero element of
              this matrix.
            * Let L be the maximum number of non-zeros per row.

        For example, for {0,1} matrices: A=B=1.
        Another example, for the Netflix matrix: A=1, B=5

        For those column pairs that are above the threshold, the
        computed similarity is correct to within 20% relative error
        with probability at least 1 - (0.981)^10/B^

        The shuffle size is bounded by the *smaller* of the following
        two expressions:

            * O(n log(n) L / (threshold * A))
            * O(m L^2^)

        The latter is the cost of the brute-force approach, so for
        non-zero thresholds, the cost is always cheaper than the
        brute-force approach.

        :param: threshold: Set to 0 for deterministic guaranteed
                           correctness. Similarities above this
                           threshold are estimated with the cost vs
                           estimate quality trade-off described above.
        :return: An n x n sparse upper-triangular CoordinateMatrix of
                 cosine similarities between columns of this matrix.

        >>> rows = sc.parallelize([[1, 2], [1, 5]])
        >>> mat = RowMatrix(rows)

        >>> sims = mat.columnSimilarities()
        >>> sims.entries.first().value
        0.91914503...
        """
        java_sims_mat = self._java_matrix_wrapper.call("columnSimilarities", float(threshold))
        return CoordinateMatrix(java_sims_mat)


Instruction:
Create a Python 3 function to
compute the QR decomposition of this RowMatrix.

Response:
def tallSkinnyQR(self, computeQ=False):
        """
        Compute the QR decomposition of this RowMatrix.

        The implementation is designed to optimize the QR decomposition
        (factorization) for the RowMatrix of a tall and skinny shape.

        Reference:
         Paul G. Constantine, David F. Gleich. "Tall and skinny QR
         factorizations in MapReduce architectures"
         ([[https://doi.org/10.1145/1996092.1996103]])

        :param: computeQ: whether to computeQ
        :return: QRDecomposition(Q: RowMatrix, R: Matrix), where
                 Q = None if computeQ = false.

        >>> rows = sc.parallelize([[3, -6], [4, -8], [0, 1]])
        >>> mat = RowMatrix(rows)
        >>> decomp = mat.tallSkinnyQR(True)
        >>> Q = decomp.Q
        >>> R = decomp.R

        >>> # Test with absolute values
        >>> absQRows = Q.rows.map(lambda row: abs(row.toArray()).tolist())
        >>> absQRows.collect()
        [[0.6..., 0.0], [0.8..., 0.0], [0.0, 1.0]]

        >>> # Test with absolute values
        >>> abs(R.toArray()).tolist()
        [[5.0, 10.0], [0.0, 1.0]]
        """
        decomp = JavaModelWrapper(self._java_matrix_wrapper.call("tallSkinnyQR", computeQ))
        if computeQ:
            java_Q = decomp.call("Q")
            Q = RowMatrix(java_Q)
        else:
            Q = None
        R = decomp.call("R")
        return QRDecomposition(Q, R)


Instruction:
Given the following Python 3 function, write the documentation
def computeSVD(self, k, computeU=False, rCond=1e-9):
        j_model = self._java_matrix_wrapper.call(
            "computeSVD", int(k), bool(computeU), float(rCond))
        return SingularValueDecomposition(j_model)

Response: Computes the singular value decomposition of the row matrix A.


Instruction:
Given the following Python 3 function, write the documentation
def multiply(self, matrix):
        if not isinstance(matrix, DenseMatrix):
            raise ValueError("Only multiplication with DenseMatrix "
                             "is supported.")
        j_model = self._java_matrix_wrapper.call("multiply", matrix)
        return RowMatrix(j_model)

Response: Multiply this matrix by a dense matrix on the right.


Instruction:
Can you write a function in Python 3 where it
returns a distributed matrix whose columns are the leftometry of the SingularValueDecomposition.

Response:
def U(self):
        """
        Returns a distributed matrix whose columns are the left
        singular vectors of the SingularValueDecomposition if computeU was set to be True.
        """
        u = self.call("U")
        if u is not None:
            mat_name = u.getClass().getSimpleName()
            if mat_name == "RowMatrix":
                return RowMatrix(u)
            elif mat_name == "IndexedRowMatrix":
                return IndexedRowMatrix(u)
            else:
                raise TypeError("Expected RowMatrix/IndexedRowMatrix got %s" % mat_name)


Instruction:
Implement a Python 3 function for
returning an RDD of IndexedRows.

Response:
def rows(self):
        """
        Rows of the IndexedRowMatrix stored as an RDD of IndexedRows.

        >>> mat = IndexedRowMatrix(sc.parallelize([IndexedRow(0, [1, 2, 3]),
        ...                                        IndexedRow(1, [4, 5, 6])]))
        >>> rows = mat.rows
        >>> rows.first()
        IndexedRow(0, [1.0,2.0,3.0])
        """
        # We use DataFrames for serialization of IndexedRows from
        # Java, so we first convert the RDD of rows to a DataFrame
        # on the Scala/Java side. Then we map each Row in the
        # DataFrame back to an IndexedRow on this side.
        rows_df = callMLlibFunc("getIndexedRows", self._java_matrix_wrapper._java_model)
        rows = rows_df.rdd.map(lambda row: IndexedRow(row[0], row[1]))
        return rows


Instruction:
Explain what the following Python 3 code does
def toBlockMatrix(self, rowsPerBlock=1024, colsPerBlock=1024):
        java_block_matrix = self._java_matrix_wrapper.call("toBlockMatrix",
                                                           rowsPerBlock,
                                                           colsPerBlock)
        return BlockMatrix(java_block_matrix, rowsPerBlock, colsPerBlock)

Response: Convert this matrix to a BlockMatrix.


Instruction:
Write a Python 3 script to
multiply this matrix by a dense matrix on the right.

Response:
def multiply(self, matrix):
        """
        Multiply this matrix by a local dense matrix on the right.

        :param matrix: a local dense matrix whose number of rows must match the number of columns
                       of this matrix
        :returns: :py:class:`IndexedRowMatrix`

        >>> mat = IndexedRowMatrix(sc.parallelize([(0, (0, 1)), (1, (2, 3))]))
        >>> mat.multiply(DenseMatrix(2, 2, [0, 2, 1, 3])).rows.collect()
        [IndexedRow(0, [2.0,3.0]), IndexedRow(1, [6.0,11.0])]
        """
        if not isinstance(matrix, DenseMatrix):
            raise ValueError("Only multiplication with DenseMatrix "
                             "is supported.")
        return IndexedRowMatrix(self._java_matrix_wrapper.call("multiply", matrix))


Instruction:
How would you explain what the following Python 3 function does
def entries(self):
        # We use DataFrames for serialization of MatrixEntry entries
        # from Java, so we first convert the RDD of entries to a
        # DataFrame on the Scala/Java side. Then we map each Row in
        # the DataFrame back to a MatrixEntry on this side.
        entries_df = callMLlibFunc("getMatrixEntries", self._java_matrix_wrapper._java_model)
        entries = entries_df.rdd.map(lambda row: MatrixEntry(row[0], row[1], row[2]))
        return entries

Response: Return the entries of the CoordinateMatrix stored as an RDD of MatrixEntries.


Instruction:
Here you have a function in Python 3, explain what it does
def blocks(self):
        # We use DataFrames for serialization of sub-matrix blocks
        # from Java, so we first convert the RDD of blocks to a
        # DataFrame on the Scala/Java side. Then we map each Row in
        # the DataFrame back to a sub-matrix block on this side.
        blocks_df = callMLlibFunc("getMatrixBlocks", self._java_matrix_wrapper._java_model)
        blocks = blocks_df.rdd.map(lambda row: ((row[0][0], row[0][1]), row[1]))
        return blocks

Response: Returns the RDD of blocks in this Scala matrix.


Instruction:
Here you have a function in Python 3, explain what it does
def persist(self, storageLevel):
        if not isinstance(storageLevel, StorageLevel):
            raise TypeError("`storageLevel` should be a StorageLevel, got %s" % type(storageLevel))
        javaStorageLevel = self._java_matrix_wrapper._sc._getJavaStorageLevel(storageLevel)
        self._java_matrix_wrapper.call("persist", javaStorageLevel)
        return self

Response: Persist the underlying RDD with the specified storage level.


Instruction:
Create a Python 3 function for
adding two block matrices together and returns a new matrix with the result.

Response:
def add(self, other):
        """
        Adds two block matrices together. The matrices must have the
        same size and matching `rowsPerBlock` and `colsPerBlock` values.
        If one of the sub matrix blocks that are being added is a
        SparseMatrix, the resulting sub matrix block will also be a
        SparseMatrix, even if it is being added to a DenseMatrix. If
        two dense sub matrix blocks are added, the output block will
        also be a DenseMatrix.

        >>> dm1 = Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])
        >>> dm2 = Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12])
        >>> sm = Matrices.sparse(3, 2, [0, 1, 3], [0, 1, 2], [7, 11, 12])
        >>> blocks1 = sc.parallelize([((0, 0), dm1), ((1, 0), dm2)])
        >>> blocks2 = sc.parallelize([((0, 0), dm1), ((1, 0), dm2)])
        >>> blocks3 = sc.parallelize([((0, 0), sm), ((1, 0), dm2)])
        >>> mat1 = BlockMatrix(blocks1, 3, 2)
        >>> mat2 = BlockMatrix(blocks2, 3, 2)
        >>> mat3 = BlockMatrix(blocks3, 3, 2)

        >>> mat1.add(mat2).toLocalMatrix()
        DenseMatrix(6, 2, [2.0, 4.0, 6.0, 14.0, 16.0, 18.0, 8.0, 10.0, 12.0, 20.0, 22.0, 24.0], 0)

        >>> mat1.add(mat3).toLocalMatrix()
        DenseMatrix(6, 2, [8.0, 2.0, 3.0, 14.0, 16.0, 18.0, 4.0, 16.0, 18.0, 20.0, 22.0, 24.0], 0)
        """
        if not isinstance(other, BlockMatrix):
            raise TypeError("Other should be a BlockMatrix, got %s" % type(other))

        other_java_block_matrix = other._java_matrix_wrapper._java_model
        java_block_matrix = self._java_matrix_wrapper.call("add", other_java_block_matrix)
        return BlockMatrix(java_block_matrix, self.rowsPerBlock, self.colsPerBlock)


Instruction:
Given the following Python 3 function, write the documentation
def transpose(self):
        java_transposed_matrix = self._java_matrix_wrapper.call("transpose")
        return BlockMatrix(java_transposed_matrix, self.colsPerBlock, self.rowsPerBlock)

Response: Returns a new BlockMatrix with the same underlying data.


Instruction:
Can you generate the documentation for the following Python 3 function
def _vector_size(v):
    if isinstance(v, Vector):
        return len(v)
    elif type(v) in (array.array, list, tuple, xrange):
        return len(v)
    elif type(v) == np.ndarray:
        if v.ndim == 1 or (v.ndim == 2 and v.shape[1] == 1):
            return len(v)
        else:
            raise ValueError("Cannot treat an ndarray of shape %s as a vector" % str(v.shape))
    elif _have_scipy and scipy.sparse.issparse(v):
        assert v.shape[1] == 1, "Expected column vector"
        return v.shape[0]
    else:
        raise TypeError("Cannot treat type %s as a vector" % type(v))

Response: Returns the size of the vector.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def parse(s):
        start = s.find('[')
        if start == -1:
            raise ValueError("Array should start with '['.")
        end = s.find(']')
        if end == -1:
            raise ValueError("Array should end with ']'.")
        s = s[start + 1: end]

        try:
            values = [float(val) for val in s.split(',') if val]
        except ValueError:
            raise ValueError("Unable to parse values from %s" % s)
        return DenseVector(values)

Response: Parse string representation back into the DenseVector.


Instruction:
Given the following Python 3 function, write the documentation
def dot(self, other):
        if type(other) == np.ndarray:
            if other.ndim > 1:
                assert len(self) == other.shape[0], "dimension mismatch"
            return np.dot(self.array, other)
        elif _have_scipy and scipy.sparse.issparse(other):
            assert len(self) == other.shape[0], "dimension mismatch"
            return other.transpose().dot(self.toArray())
        else:
            assert len(self) == _vector_size(other), "dimension mismatch"
            if isinstance(other, SparseVector):
                return other.dot(self)
            elif isinstance(other, Vector):
                return np.dot(self.toArray(), other.toArray())
            else:
                return np.dot(self.toArray(), other)

Response: Compute the dot product of two Vectors.


Instruction:
Can you implement a function in Python 3 that
returns the squared distance between two Vectors.

Response:
def squared_distance(self, other):
        """
        Squared distance of two Vectors.

        >>> dense1 = DenseVector(array.array('d', [1., 2.]))
        >>> dense1.squared_distance(dense1)
        0.0
        >>> dense2 = np.array([2., 1.])
        >>> dense1.squared_distance(dense2)
        2.0
        >>> dense3 = [2., 1.]
        >>> dense1.squared_distance(dense3)
        2.0
        >>> sparse1 = SparseVector(2, [0, 1], [2., 1.])
        >>> dense1.squared_distance(sparse1)
        2.0
        >>> dense1.squared_distance([1.,])
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        >>> dense1.squared_distance(SparseVector(1, [0,], [1.,]))
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        """
        assert len(self) == _vector_size(other), "dimension mismatch"
        if isinstance(other, SparseVector):
            return other.squared_distance(self)
        elif _have_scipy and scipy.sparse.issparse(other):
            return _convert_to_vector(other).squared_distance(self)

        if isinstance(other, Vector):
            other = other.toArray()
        elif not isinstance(other, np.ndarray):
            other = np.array(other)
        diff = self.toArray() - other
        return np.dot(diff, diff)


Instruction:
Implement a function in Python 3 to
parse string representation back into the SparseVector.

Response:
def parse(s):
        """
        Parse string representation back into the SparseVector.

        >>> SparseVector.parse(' (4, [0,1 ],[ 4.0,5.0] )')
        SparseVector(4, {0: 4.0, 1: 5.0})
        """
        start = s.find('(')
        if start == -1:
            raise ValueError("Tuple should start with '('")
        end = s.find(')')
        if end == -1:
            raise ValueError("Tuple should end with ')'")
        s = s[start + 1: end].strip()

        size = s[: s.find(',')]
        try:
            size = int(size)
        except ValueError:
            raise ValueError("Cannot parse size %s." % size)

        ind_start = s.find('[')
        if ind_start == -1:
            raise ValueError("Indices array should start with '['.")
        ind_end = s.find(']')
        if ind_end == -1:
            raise ValueError("Indices array should end with ']'")
        new_s = s[ind_start + 1: ind_end]
        ind_list = new_s.split(',')
        try:
            indices = [int(ind) for ind in ind_list if ind]
        except ValueError:
            raise ValueError("Unable to parse indices from %s." % new_s)
        s = s[ind_end + 1:].strip()

        val_start = s.find('[')
        if val_start == -1:
            raise ValueError("Values array should start with '['.")
        val_end = s.find(']')
        if val_end == -1:
            raise ValueError("Values array should end with ']'.")
        val_list = s[val_start + 1: val_end].split(',')
        try:
            values = [float(val) for val in val_list if val]
        except ValueError:
            raise ValueError("Unable to parse values from %s." % s)
        return SparseVector(size, indices, values)


Instruction:
Can you implement a function in Python 3 that
dots product of two SparseVector objects.

Response:
def dot(self, other):
        """
        Dot product with a SparseVector or 1- or 2-dimensional Numpy array.

        >>> a = SparseVector(4, [1, 3], [3.0, 4.0])
        >>> a.dot(a)
        25.0
        >>> a.dot(array.array('d', [1., 2., 3., 4.]))
        22.0
        >>> b = SparseVector(4, [2], [1.0])
        >>> a.dot(b)
        0.0
        >>> a.dot(np.array([[1, 1], [2, 2], [3, 3], [4, 4]]))
        array([ 22.,  22.])
        >>> a.dot([1., 2., 3.])
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        >>> a.dot(np.array([1., 2.]))
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        >>> a.dot(DenseVector([1., 2.]))
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        >>> a.dot(np.zeros((3, 2)))
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        """

        if isinstance(other, np.ndarray):
            if other.ndim not in [2, 1]:
                raise ValueError("Cannot call dot with %d-dimensional array" % other.ndim)
            assert len(self) == other.shape[0], "dimension mismatch"
            return np.dot(self.values, other[self.indices])

        assert len(self) == _vector_size(other), "dimension mismatch"

        if isinstance(other, DenseVector):
            return np.dot(other.array[self.indices], self.values)

        elif isinstance(other, SparseVector):
            # Find out common indices.
            self_cmind = np.in1d(self.indices, other.indices, assume_unique=True)
            self_values = self.values[self_cmind]
            if self_values.size == 0:
                return 0.0
            else:
                other_cmind = np.in1d(other.indices, self.indices, assume_unique=True)
                return np.dot(self_values, other.values[other_cmind])

        else:
            return self.dot(_convert_to_vector(other))


Instruction:
Implement a function in Python 3 to
return the squared distance between two SparseVectors or 1 - dimensional NumPy arrays.

Response:
def squared_distance(self, other):
        """
        Squared distance from a SparseVector or 1-dimensional NumPy array.

        >>> a = SparseVector(4, [1, 3], [3.0, 4.0])
        >>> a.squared_distance(a)
        0.0
        >>> a.squared_distance(array.array('d', [1., 2., 3., 4.]))
        11.0
        >>> a.squared_distance(np.array([1., 2., 3., 4.]))
        11.0
        >>> b = SparseVector(4, [2], [1.0])
        >>> a.squared_distance(b)
        26.0
        >>> b.squared_distance(a)
        26.0
        >>> b.squared_distance([1., 2.])
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        >>> b.squared_distance(SparseVector(3, [1,], [1.0,]))
        Traceback (most recent call last):
            ...
        AssertionError: dimension mismatch
        """
        assert len(self) == _vector_size(other), "dimension mismatch"

        if isinstance(other, np.ndarray) or isinstance(other, DenseVector):
            if isinstance(other, np.ndarray) and other.ndim != 1:
                raise Exception("Cannot call squared_distance with %d-dimensional array" %
                                other.ndim)
            if isinstance(other, DenseVector):
                other = other.array
            sparse_ind = np.zeros(other.size, dtype=bool)
            sparse_ind[self.indices] = True
            dist = other[sparse_ind] - self.values
            result = np.dot(dist, dist)

            other_ind = other[~sparse_ind]
            result += np.dot(other_ind, other_ind)
            return result

        elif isinstance(other, SparseVector):
            result = 0.0
            i, j = 0, 0
            while i < len(self.indices) and j < len(other.indices):
                if self.indices[i] == other.indices[j]:
                    diff = self.values[i] - other.values[j]
                    result += diff * diff
                    i += 1
                    j += 1
                elif self.indices[i] < other.indices[j]:
                    result += self.values[i] * self.values[i]
                    i += 1
                else:
                    result += other.values[j] * other.values[j]
                    j += 1
            while i < len(self.indices):
                result += self.values[i] * self.values[i]
                i += 1
            while j < len(other.indices):
                result += other.values[j] * other.values[j]
                j += 1
            return result
        else:
            return self.squared_distance(_convert_to_vector(other))


Instruction:
Can you write a function in Python 3 where it
returns a copy of this SparseVector as a 1 - dimensional NumPy array.

Response:
def toArray(self):
        """
        Returns a copy of this SparseVector as a 1-dimensional NumPy array.
        """
        arr = np.zeros((self.size,), dtype=np.float64)
        arr[self.indices] = self.values
        return arr


Instruction:
Implement a function in Python 3 to
convert this vector to the new mllib - local representation.

Response:
def asML(self):
        """
        Convert this vector to the new mllib-local representation.
        This does NOT copy the data; it copies references.

        :return: :py:class:`pyspark.ml.linalg.SparseVector`

        .. versionadded:: 2.0.0
        """
        return newlinalg.SparseVector(self.size, self.indices, self.values)


Instruction:
Can you implement a function in Python 3 that
creates a dense vector of 64 - bit floats from a Python list or numbers.

Response:
def dense(*elements):
        """
        Create a dense vector of 64-bit floats from a Python list or numbers.

        >>> Vectors.dense([1, 2, 3])
        DenseVector([1.0, 2.0, 3.0])
        >>> Vectors.dense(1.0, 2.0)
        DenseVector([1.0, 2.0])
        """
        if len(elements) == 1 and not isinstance(elements[0], (float, int, long)):
            # it's list, numpy.array or other iterable object.
            elements = elements[0]
        return DenseVector(elements)


Instruction:
Here you have a function in Python 3, explain what it does
def fromML(vec):
        if isinstance(vec, newlinalg.DenseVector):
            return DenseVector(vec.array)
        elif isinstance(vec, newlinalg.SparseVector):
            return SparseVector(vec.size, vec.indices, vec.values)
        else:
            raise TypeError("Unsupported vector type %s" % type(vec))

Response: Converts a vector from the new mllib - local representation.


Instruction:
Create a Python 3 function for
squaring distance between two sets of vectors.

Response:
def squared_distance(v1, v2):
        """
        Squared distance between two vectors.
        a and b can be of type SparseVector, DenseVector, np.ndarray
        or array.array.

        >>> a = Vectors.sparse(4, [(0, 1), (3, 4)])
        >>> b = Vectors.dense([2, 5, 4, 1])
        >>> a.squared_distance(b)
        51.0
        """
        v1, v2 = _convert_to_vector(v1), _convert_to_vector(v2)
        return v1.squared_distance(v2)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def parse(s):
        if s.find('(') == -1 and s.find('[') != -1:
            return DenseVector.parse(s)
        elif s.find('(') != -1:
            return SparseVector.parse(s)
        else:
            raise ValueError(
                "Cannot find tokens '[' or '(' from the input string.")

Response: Parse a string representation back into the Vector.


Instruction:
Explain what the following Python 3 code does
def _equals(v1_indices, v1_values, v2_indices, v2_values):
        v1_size = len(v1_values)
        v2_size = len(v2_values)
        k1 = 0
        k2 = 0
        all_equal = True
        while all_equal:
            while k1 < v1_size and v1_values[k1] == 0:
                k1 += 1
            while k2 < v2_size and v2_values[k2] == 0:
                k2 += 1

            if k1 >= v1_size or k2 >= v2_size:
                return k1 >= v1_size and k2 >= v2_size

            all_equal = v1_indices[k1] == v2_indices[k2] and v1_values[k1] == v2_values[k2]
            k1 += 1
            k2 += 1
        return all_equal

Response: Check equality between sparse vectors v1_indices and v2_indices assume to be strictly increasing.


Instruction:
How would you explain what the following Python 3 function does
def _convert_to_array(array_like, dtype):
        if isinstance(array_like, bytes):
            return np.frombuffer(array_like, dtype=dtype)
        return np.asarray(array_like, dtype=dtype)

Response: Convert matrix attributes which are array - like or buffer to array.


Instruction:
Can you write a function in Python 3 where it
returns an array of the ISO - 8601 related data.

Response:
def toArray(self):
        """
        Return an numpy.ndarray

        >>> m = DenseMatrix(2, 2, range(4))
        >>> m.toArray()
        array([[ 0.,  2.],
               [ 1.,  3.]])
        """
        if self.isTransposed:
            return np.asfortranarray(
                self.values.reshape((self.numRows, self.numCols)))
        else:
            return self.values.reshape((self.numRows, self.numCols), order='F')


Instruction:
Can you create a Python 3 function that
converts this matrix to the new mllib - local representation.

Response:
def asML(self):
        """
        Convert this matrix to the new mllib-local representation.
        This does NOT copy the data; it copies references.

        :return: :py:class:`pyspark.ml.linalg.DenseMatrix`

        .. versionadded:: 2.0.0
        """
        return newlinalg.DenseMatrix(self.numRows, self.numCols, self.values, self.isTransposed)


Instruction:
Can you tell what is the following Python 3 function doing
def toArray(self):
        A = np.zeros((self.numRows, self.numCols), dtype=np.float64, order='F')
        for k in xrange(self.colPtrs.size - 1):
            startptr = self.colPtrs[k]
            endptr = self.colPtrs[k + 1]
            if self.isTransposed:
                A[k, self.rowIndices[startptr:endptr]] = self.values[startptr:endptr]
            else:
                A[self.rowIndices[startptr:endptr], k] = self.values[startptr:endptr]
        return A

Response: Return an array of the ISO - 8601 entries.


Instruction:
Can you tell what is the following Python 3 function doing
def asML(self):
        return newlinalg.SparseMatrix(self.numRows, self.numCols, self.colPtrs, self.rowIndices,
                                      self.values, self.isTransposed)

Response: Convert this matrix to the new mllib - local representation.


Instruction:
Given the following Python 3 function, write the documentation
def fromML(mat):
        if isinstance(mat, newlinalg.DenseMatrix):
            return DenseMatrix(mat.numRows, mat.numCols, mat.values, mat.isTransposed)
        elif isinstance(mat, newlinalg.SparseMatrix):
            return SparseMatrix(mat.numRows, mat.numCols, mat.colPtrs, mat.rowIndices,
                                mat.values, mat.isTransposed)
        else:
            raise TypeError("Unsupported matrix type %s" % type(mat))

Response: Converts a matrix from the new mllib - local representation.


Instruction:
How would you code a function in Python 3 to
give a large dataset and an item approximately find at most k items which have the nearest distance to the item.

Response:
def approxNearestNeighbors(self, dataset, key, numNearestNeighbors, distCol="distCol"):
        """
        Given a large dataset and an item, approximately find at most k items which have the
        closest distance to the item. If the :py:attr:`outputCol` is missing, the method will
        transform the data; if the :py:attr:`outputCol` exists, it will use that. This allows
        caching of the transformed data when necessary.

        .. note:: This method is experimental and will likely change behavior in the next release.

        :param dataset: The dataset to search for nearest neighbors of the key.
        :param key: Feature vector representing the item to search for.
        :param numNearestNeighbors: The maximum number of nearest neighbors.
        :param distCol: Output column for storing the distance between each result row and the key.
                        Use "distCol" as default value if it's not specified.
        :return: A dataset containing at most k items closest to the key. A column "distCol" is
                 added to show the distance between each row and the key.
        """
        return self._call_java("approxNearestNeighbors", dataset, key, numNearestNeighbors,
                               distCol)


Instruction:
Given the following Python 3 function, write the documentation
def approxSimilarityJoin(self, datasetA, datasetB, threshold, distCol="distCol"):
        threshold = TypeConverters.toFloat(threshold)
        return self._call_java("approxSimilarityJoin", datasetA, datasetB, threshold, distCol)

Response: This method joins two datasets to approximately find all pairs of rows whose distance is smaller than threshold.


Instruction:
Given the following Python 3 function, write the documentation
def from_labels(cls, labels, inputCol, outputCol=None, handleInvalid=None):
        sc = SparkContext._active_spark_context
        java_class = sc._gateway.jvm.java.lang.String
        jlabels = StringIndexerModel._new_java_array(labels, java_class)
        model = StringIndexerModel._create_from_java_class(
            "org.apache.spark.ml.feature.StringIndexerModel", jlabels)
        model.setInputCol(inputCol)
        if outputCol is not None:
            model.setOutputCol(outputCol)
        if handleInvalid is not None:
            model.setHandleInvalid(handleInvalid)
        return model

Response: Construct the model directly from an array of label strings.


Instruction:
Can you generate the documentation for the following Python 3 function
def from_arrays_of_labels(cls, arrayOfLabels, inputCols, outputCols=None,
                              handleInvalid=None):
        """
        Construct the model directly from an array of array of label strings,
        requires an active SparkContext.
        """
        sc = SparkContext._active_spark_context
        java_class = sc._gateway.jvm.java.lang.String
        jlabels = StringIndexerModel._new_java_array(arrayOfLabels, java_class)
        model = StringIndexerModel._create_from_java_class(
            "org.apache.spark.ml.feature.StringIndexerModel", jlabels)
        model.setInputCols(inputCols)
        if outputCols is not None:
            model.setOutputCols(outputCols)
        if handleInvalid is not None:
            model.setHandleInvalid(handleInvalid)
        return model

Response: Construct a model directly from an array of array of label strings.


Instruction:
Here you have a function in Python 3, explain what it does
def setParams(self, inputCol=None, outputCol=None, stopWords=None, caseSensitive=False,
                  locale=None):
        """
        setParams(self, inputCol=None, outputCol=None, stopWords=None, caseSensitive=false, \
        locale=None)
        Sets params for this StopWordRemover.
        """
        kwargs = self._input_kwargs
        return self._set(**kwargs)

Response: Sets the parameters for this StopWordRemover.


Instruction:
Write a Python 3 script for
loading the default stop words for the given language.

Response:
def loadDefaultStopWords(language):
        """
        Loads the default stop words for the given language.
        Supported languages: danish, dutch, english, finnish, french, german, hungarian,
        italian, norwegian, portuguese, russian, spanish, swedish, turkish
        """
        stopWordsObj = _jvm().org.apache.spark.ml.feature.StopWordsRemover
        return list(stopWordsObj.loadDefaultStopWords(language))


Instruction:
How would you explain what the following Python 3 function does
def findSynonyms(self, word, num):
        if not isinstance(word, basestring):
            word = _convert_to_vector(word)
        return self._call_java("findSynonyms", word, num)

Response: Find num number of words closest in similarity to word.


Instruction:
How would you implement a function in Python 3 that
finds num number of words closest in similarity to word.

Response:
def findSynonymsArray(self, word, num):
        """
        Find "num" number of words closest in similarity to "word".
        word can be a string or vector representation.
        Returns an array with two fields word and similarity (which
        gives the cosine similarity).
        """
        if not isinstance(word, basestring):
            word = _convert_to_vector(word)
        tuples = self._java_obj.findSynonymsArray(word, num)
        return list(map(lambda st: (st._1(), st._2()), list(tuples)))


Instruction:
Explain what the following Python 3 code does
def install_exception_handler():
    original = py4j.protocol.get_return_value
    # The original `get_return_value` is not patched, it's idempotent.
    patched = capture_sql_exception(original)
    # only patch the one used in py4j.java_gateway (call Java API)
    py4j.java_gateway.get_return_value = patched

Response: Installs an exception handler into the Py4j environment.


Instruction:
Make a summary of the following Python 3 code
def toJArray(gateway, jtype, arr):
    jarr = gateway.new_array(jtype, len(arr))
    for i in range(0, len(arr)):
        jarr[i] = arr[i]
    return jarr

Response: Convert python list to java type array


Instruction:
Create a Python 3 function for
raising ImportError if minimum version of Pandas is not installed.

Response:
def require_minimum_pandas_version():
    """ Raise ImportError if minimum version of Pandas is not installed
    """
    # TODO(HyukjinKwon): Relocate and deduplicate the version specification.
    minimum_pandas_version = "0.19.2"

    from distutils.version import LooseVersion
    try:
        import pandas
        have_pandas = True
    except ImportError:
        have_pandas = False
    if not have_pandas:
        raise ImportError("Pandas >= %s must be installed; however, "
                          "it was not found." % minimum_pandas_version)
    if LooseVersion(pandas.__version__) < LooseVersion(minimum_pandas_version):
        raise ImportError("Pandas >= %s must be installed; however, "
                          "your version was %s." % (minimum_pandas_version, pandas.__version__))


Instruction:
How would you code a function in Python 3 to
raise ImportError if minimum version of pyarrow is not installed.

Response:
def require_minimum_pyarrow_version():
    """ Raise ImportError if minimum version of pyarrow is not installed
    """
    # TODO(HyukjinKwon): Relocate and deduplicate the version specification.
    minimum_pyarrow_version = "0.12.1"

    from distutils.version import LooseVersion
    try:
        import pyarrow
        have_arrow = True
    except ImportError:
        have_arrow = False
    if not have_arrow:
        raise ImportError("PyArrow >= %s must be installed; however, "
                          "it was not found." % minimum_pyarrow_version)
    if LooseVersion(pyarrow.__version__) < LooseVersion(minimum_pyarrow_version):
        raise ImportError("PyArrow >= %s must be installed; however, "
                          "your version was %s." % (minimum_pyarrow_version, pyarrow.__version__))


Instruction:
Explain what the following Python 3 code does
def launch_gateway(conf=None, popen_kwargs=None):
    if "PYSPARK_GATEWAY_PORT" in os.environ:
        gateway_port = int(os.environ["PYSPARK_GATEWAY_PORT"])
        gateway_secret = os.environ["PYSPARK_GATEWAY_SECRET"]
        # Process already exists
        proc = None
    else:
        SPARK_HOME = _find_spark_home()
        # Launch the Py4j gateway using Spark's run command so that we pick up the
        # proper classpath and settings from spark-env.sh
        on_windows = platform.system() == "Windows"
        script = "./bin/spark-submit.cmd" if on_windows else "./bin/spark-submit"
        command = [os.path.join(SPARK_HOME, script)]
        if conf:
            for k, v in conf.getAll():
                command += ['--conf', '%s=%s' % (k, v)]
        submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "pyspark-shell")
        if os.environ.get("SPARK_TESTING"):
            submit_args = ' '.join([
                "--conf spark.ui.enabled=false",
                submit_args
            ])
        command = command + shlex.split(submit_args)

        # Create a temporary directory where the gateway server should write the connection
        # information.
        conn_info_dir = tempfile.mkdtemp()
        try:
            fd, conn_info_file = tempfile.mkstemp(dir=conn_info_dir)
            os.close(fd)
            os.unlink(conn_info_file)

            env = dict(os.environ)
            env["_PYSPARK_DRIVER_CONN_INFO_PATH"] = conn_info_file

            # Launch the Java gateway.
            popen_kwargs = {} if popen_kwargs is None else popen_kwargs
            # We open a pipe to stdin so that the Java gateway can die when the pipe is broken
            popen_kwargs['stdin'] = PIPE
            # We always set the necessary environment variables.
            popen_kwargs['env'] = env
            if not on_windows:
                # Don't send ctrl-c / SIGINT to the Java gateway:
                def preexec_func():
                    signal.signal(signal.SIGINT, signal.SIG_IGN)
                popen_kwargs['preexec_fn'] = preexec_func
                proc = Popen(command, **popen_kwargs)
            else:
                # preexec_fn not supported on Windows
                proc = Popen(command, **popen_kwargs)

            # Wait for the file to appear, or for the process to exit, whichever happens first.
            while not proc.poll() and not os.path.isfile(conn_info_file):
                time.sleep(0.1)

            if not os.path.isfile(conn_info_file):
                raise Exception("Java gateway process exited before sending its port number")

            with open(conn_info_file, "rb") as info:
                gateway_port = read_int(info)
                gateway_secret = UTF8Deserializer().loads(info)
        finally:
            shutil.rmtree(conn_info_dir)

        # In Windows, ensure the Java child processes do not linger after Python has exited.
        # In UNIX-based systems, the child process can kill itself on broken pipe (i.e. when
        # the parent process' stdin sends an EOF). In Windows, however, this is not possible
        # because java.lang.Process reads directly from the parent process' stdin, contending
        # with any opportunity to read an EOF from the parent. Note that this is only best
        # effort and will not take effect if the python process is violently terminated.
        if on_windows:
            # In Windows, the child process here is "spark-submit.cmd", not the JVM itself
            # (because the UNIX "exec" command is not available). This means we cannot simply
            # call proc.kill(), which kills only the "spark-submit.cmd" process but not the
            # JVMs. Instead, we use "taskkill" with the tree-kill option "/t" to terminate all
            # child processes in the tree (http://technet.microsoft.com/en-us/library/bb491009.aspx)
            def killChild():
                Popen(["cmd", "/c", "taskkill", "/f", "/t", "/pid", str(proc.pid)])
            atexit.register(killChild)

    # Connect to the gateway
    gateway = JavaGateway(
        gateway_parameters=GatewayParameters(port=gateway_port, auth_token=gateway_secret,
                                             auto_convert=True))
    # Store a reference to the Popen object for use by the caller (e.g., in reading stdout/stderr)
    gateway.proc = proc

    # Import the classes used by PySpark
    java_import(gateway.jvm, "org.apache.spark.SparkConf")
    java_import(gateway.jvm, "org.apache.spark.api.java.*")
    java_import(gateway.jvm, "org.apache.spark.api.python.*")
    java_import(gateway.jvm, "org.apache.spark.ml.python.*")
    java_import(gateway.jvm, "org.apache.spark.mllib.api.python.*")
    # TODO(davies): move into sql
    java_import(gateway.jvm, "org.apache.spark.sql.*")
    java_import(gateway.jvm, "org.apache.spark.sql.api.python.*")
    java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
    java_import(gateway.jvm, "scala.Tuple2")

    return gateway

Response: Launch the jvm gateway using Spark s submit command.


Instruction:
Here you have a function in Python 3, explain what it does
def _do_server_auth(conn, auth_secret):
    write_with_length(auth_secret.encode("utf-8"), conn)
    conn.flush()
    reply = UTF8Deserializer().loads(conn)
    if reply != "ok":
        conn.close()
        raise Exception("Unexpected reply from iterator server.")

Response: Performs the server authentication protocol on the given conn.


Instruction:
Here you have a function in Python 3, explain what it does
def local_connect_and_auth(port, auth_secret):
    sock = None
    errors = []
    # Support for both IPv4 and IPv6.
    # On most of IPv6-ready systems, IPv6 will take precedence.
    for res in socket.getaddrinfo("127.0.0.1", port, socket.AF_UNSPEC, socket.SOCK_STREAM):
        af, socktype, proto, _, sa = res
        try:
            sock = socket.socket(af, socktype, proto)
            sock.settimeout(15)
            sock.connect(sa)
            sockfile = sock.makefile("rwb", 65536)
            _do_server_auth(sockfile, auth_secret)
            return (sockfile, sock)
        except socket.error as e:
            emsg = _exception_message(e)
            errors.append("tried to connect to %s, but an error occured: %s" % (sa, emsg))
            sock.close()
            sock = None
    raise Exception("could not open socket: %s" % errors)

Response: Connect to local host authenticate with it and return a tuple with sockfile sock and auth_secret.


Instruction:
Can you tell what is the following Python 3 function doing
def ensure_callback_server_started(gw):

    # getattr will fallback to JVM, so we cannot test by hasattr()
    if "_callback_server" not in gw.__dict__ or gw._callback_server is None:
        gw.callback_server_parameters.eager_load = True
        gw.callback_server_parameters.daemonize = True
        gw.callback_server_parameters.daemonize_connections = True
        gw.callback_server_parameters.port = 0
        gw.start_callback_server(gw.callback_server_parameters)
        cbport = gw._callback_server.server_socket.getsockname()[1]
        gw._callback_server.port = cbport
        # gateway with real port
        gw._python_proxy_port = gw._callback_server.port
        # get the GatewayServer object in JVM by ID
        jgws = JavaObject("GATEWAY_SERVER", gw._gateway_client)
        # update the port of CallbackClient with real port
        jgws.resetCallbackClient(jgws.getCallbackClient().getAddress(), gw._python_proxy_port)

Response: Ensure that callback server is started.


Instruction:
Write a Python 3 function for
calculating URL contributions to the rank of other URLs.

Response:
def computeContribs(urls, rank):
    """Calculates URL contributions to the rank of other URLs."""
    num_urls = len(urls)
    for url in urls:
        yield (url, rank / num_urls)


Instruction:
How would you code a function in Python 3 to
return the summary of the current model.

Response:
def summary(self):
        """
        Gets summary (e.g. cluster assignments, cluster sizes) of the model trained on the
        training set. An exception is thrown if no summary exists.
        """
        if self.hasSummary:
            return GaussianMixtureSummary(super(GaussianMixtureModel, self).summary)
        else:
            raise RuntimeError("No training summary available for this %s" %
                               self.__class__.__name__)


Instruction:
Given the following Python 3 function, write the documentation
def summary(self):
        if self.hasSummary:
            return KMeansSummary(super(KMeansModel, self).summary)
        else:
            raise RuntimeError("No training summary available for this %s" %
                               self.__class__.__name__)

Response: Returns the training set summary.


Instruction:
Can you implement a function in Python 3 that
returns the training set summary.

Response:
def summary(self):
        """
        Gets summary (e.g. cluster assignments, cluster sizes) of the model trained on the
        training set. An exception is thrown if no summary exists.
        """
        if self.hasSummary:
            return BisectingKMeansSummary(super(BisectingKMeansModel, self).summary)
        else:
            raise RuntimeError("No training summary available for this %s" %
                               self.__class__.__name__)


Instruction:
Here you have a function in Python 3, explain what it does
def imageSchema(self):

        if self._imageSchema is None:
            ctx = SparkContext._active_spark_context
            jschema = ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema()
            self._imageSchema = _parse_datatype_json_string(jschema.json())
        return self._imageSchema

Response: Returns the image schema.


Instruction:
Can you write a function in Python 3 where it
returns the OpenCV type mapping supported.

Response:
def ocvTypes(self):
        """
        Returns the OpenCV type mapping supported.

        :return: a dictionary containing the OpenCV type mapping supported.

        .. versionadded:: 2.3.0
        """

        if self._ocvTypes is None:
            ctx = SparkContext._active_spark_context
            self._ocvTypes = dict(ctx._jvm.org.apache.spark.ml.image.ImageSchema.javaOcvTypes())
        return self._ocvTypes


Instruction:
Given the following Python 3 function, write the documentation
def columnSchema(self):

        if self._columnSchema is None:
            ctx = SparkContext._active_spark_context
            jschema = ctx._jvm.org.apache.spark.ml.image.ImageSchema.columnSchema()
            self._columnSchema = _parse_datatype_json_string(jschema.json())
        return self._columnSchema

Response: Returns the schema for the image column.


Instruction:
Can you tell what is the following Python 3 function doing
def imageFields(self):

        if self._imageFields is None:
            ctx = SparkContext._active_spark_context
            self._imageFields = list(ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageFields())
        return self._imageFields

Response: Returns a list of field names of image columns.


Instruction:
Make a summary of the following Python 3 code
def undefinedImageType(self):

        if self._undefinedImageType is None:
            ctx = SparkContext._active_spark_context
            self._undefinedImageType = \
                ctx._jvm.org.apache.spark.ml.image.ImageSchema.undefinedImageType()
        return self._undefinedImageType

Response: Returns the name of undefined image type for the invalid image.


Instruction:
Explain what the following Python 3 code does
def toNDArray(self, image):

        if not isinstance(image, Row):
            raise TypeError(
                "image argument should be pyspark.sql.types.Row; however, "
                "it got [%s]." % type(image))

        if any(not hasattr(image, f) for f in self.imageFields):
            raise ValueError(
                "image argument should have attributes specified in "
                "ImageSchema.imageSchema [%s]." % ", ".join(self.imageFields))

        height = image.height
        width = image.width
        nChannels = image.nChannels
        return np.ndarray(
            shape=(height, width, nChannels),
            dtype=np.uint8,
            buffer=image.data,
            strides=(width * nChannels, nChannels, 1))

Response: Converts an image to an array with metadata.


Instruction:
Can you write a function in Python 3 where it
converts an array with metadata to a two - dimensional image.

Response:
def toImage(self, array, origin=""):
        """
        Converts an array with metadata to a two-dimensional image.

        :param `numpy.ndarray` array: The array to convert to image.
        :param str origin: Path to the image, optional.
        :return: a :class:`Row` that is a two dimensional image.

        .. versionadded:: 2.3.0
        """

        if not isinstance(array, np.ndarray):
            raise TypeError(
                "array argument should be numpy.ndarray; however, it got [%s]." % type(array))

        if array.ndim != 3:
            raise ValueError("Invalid array shape")

        height, width, nChannels = array.shape
        ocvTypes = ImageSchema.ocvTypes
        if nChannels == 1:
            mode = ocvTypes["CV_8UC1"]
        elif nChannels == 3:
            mode = ocvTypes["CV_8UC3"]
        elif nChannels == 4:
            mode = ocvTypes["CV_8UC4"]
        else:
            raise ValueError("Invalid number of channels")

        # Running `bytearray(numpy.array([1]))` fails in specific Python versions
        # with a specific Numpy version, for example in Python 3.6.0 and NumPy 1.13.3.
        # Here, it avoids it by converting it to bytes.
        if LooseVersion(np.__version__) >= LooseVersion('1.9'):
            data = bytearray(array.astype(dtype=np.uint8).ravel().tobytes())
        else:
            # Numpy prior to 1.9 don't have `tobytes` method.
            data = bytearray(array.astype(dtype=np.uint8).ravel())

        # Creating new Row with _create_row(), because Row(name = value, ... )
        # orders fields by name, which conflicts with expected schema order
        # when the new DataFrame is created by UDF
        return _create_row(self.imageFields,
                           [origin, height, width, nChannels, mode, data])


Instruction:
Can you implement a function in Python 3 that
reads the images from the local or remote source directory.

Response:
def readImages(self, path, recursive=False, numPartitions=-1,
                   dropImageFailures=False, sampleRatio=1.0, seed=0):
        """
        Reads the directory of images from the local or remote source.

        .. note:: If multiple jobs are run in parallel with different sampleRatio or recursive flag,
            there may be a race condition where one job overwrites the hadoop configs of another.

        .. note:: If sample ratio is less than 1, sampling uses a PathFilter that is efficient but
            potentially non-deterministic.

        .. note:: Deprecated in 2.4.0. Use `spark.read.format("image").load(path)` instead and
            this `readImages` will be removed in 3.0.0.

        :param str path: Path to the image directory.
        :param bool recursive: Recursive search flag.
        :param int numPartitions: Number of DataFrame partitions.
        :param bool dropImageFailures: Drop the files that are not valid images.
        :param float sampleRatio: Fraction of the images loaded.
        :param int seed: Random number seed.
        :return: a :class:`DataFrame` with a single column of "images",
               see ImageSchema for details.

        >>> df = ImageSchema.readImages('data/mllib/images/origin/kittens', recursive=True)
        >>> df.count()
        5

        .. versionadded:: 2.3.0
        """
        warnings.warn("`ImageSchema.readImage` is deprecated. " +
                      "Use `spark.read.format(\"image\").load(path)` instead.", DeprecationWarning)
        spark = SparkSession.builder.getOrCreate()
        image_schema = spark._jvm.org.apache.spark.ml.image.ImageSchema
        jsession = spark._jsparkSession
        jresult = image_schema.readImages(path, jsession, recursive, numPartitions,
                                          dropImageFailures, float(sampleRatio), seed)
        return DataFrame(jresult, spark._wrapped)


Instruction:
How would you explain what the following Python 3 function does
def _create_from_java_class(cls, java_class, *args):
        java_obj = JavaWrapper._new_java_obj(java_class, *args)
        return cls(java_obj)

Response: Construct this object from given Java classname and arguments.


Instruction:
Can you tell what is the following Python 3 function doing
def _new_java_array(pylist, java_class):
        sc = SparkContext._active_spark_context
        java_array = None
        if len(pylist) > 0 and isinstance(pylist[0], list):
            # If pylist is a 2D array, then a 2D java array will be created.
            # The 2D array is a square, non-jagged 2D array that is big enough for all elements.
            inner_array_length = 0
            for i in xrange(len(pylist)):
                inner_array_length = max(inner_array_length, len(pylist[i]))
            java_array = sc._gateway.new_array(java_class, len(pylist), inner_array_length)
            for i in xrange(len(pylist)):
                for j in xrange(len(pylist[i])):
                    java_array[i][j] = pylist[i][j]
        else:
            java_array = sc._gateway.new_array(java_class, len(pylist))
            for i in xrange(len(pylist)):
                java_array[i] = pylist[i]
        return java_array

Response: Create a Java array of given java_class type from a Python list.


Instruction:
Write a Python 3 function for
converting EPITEXT to EPITEXT.

Response:
def _convert_epytext(line):
    """
    >>> _convert_epytext("L{A}")
    :class:`A`
    """
    line = line.replace('@', ':')
    for p, sub in RULES:
        line = re.sub(p, sub, line)
    return line


Instruction:
Explain what the following Python 3 code does
def rddToFileName(prefix, suffix, timestamp):
    if isinstance(timestamp, datetime):
        seconds = time.mktime(timestamp.timetuple())
        timestamp = int(seconds * 1000) + timestamp.microsecond // 1000
    if suffix is None:
        return prefix + "-" + str(timestamp)
    else:
        return prefix + "-" + str(timestamp) + "." + suffix

Response: Convert a RDD to a file name.


Instruction:
Write a Python 3 function for
adding a profiler for RDD id

Response:
def add_profiler(self, id, profiler):
        """ Add a profiler for RDD `id` """
        if not self.profilers:
            if self.profile_dump_path:
                atexit.register(self.dump_profiles, self.profile_dump_path)
            else:
                atexit.register(self.show_profiles)

        self.profilers.append([id, profiler, False])


Instruction:
Can you write a function in Python 3 where it
dumps the profile stats into a directory path.

Response:
def dump_profiles(self, path):
        """ Dump the profile stats into directory `path` """
        for id, profiler, _ in self.profilers:
            profiler.dump(id, path)
        self.profilers = []


Instruction:
Make a summary of the following Python 3 code
def show_profiles(self):
        for i, (id, profiler, showed) in enumerate(self.profilers):
            if not showed and profiler:
                profiler.show(id)
                # mark it as showed
                self.profilers[i][2] = True

Response: Print the profile stats to stdout


Instruction:
Create a Python 3 function for
printing the profile stats to stdout

Response:
def show(self, id):
        """ Print the profile stats to stdout, id is the RDD id """
        stats = self.stats()
        if stats:
            print("=" * 60)
            print("Profile of RDD<id=%d>" % id)
            print("=" * 60)
            stats.sort_stats("time", "cumulative").print_stats()


Instruction:
Given the following Python 3 function, write the documentation
def dump(self, id, path):
        if not os.path.exists(path):
            os.makedirs(path)
        stats = self.stats()
        if stats:
            p = os.path.join(path, "rdd_%d.pstats" % id)
            stats.dump_stats(p)

Response: Dump the profile into path


Instruction:
Here you have a function in Python 3, explain what it does
def profile(self, func):
        pr = cProfile.Profile()
        pr.runcall(func)
        st = pstats.Stats(pr)
        st.stream = None  # make it picklable
        st.strip_dirs()

        # Adds a new profile to the existing accumulated value
        self._accumulator.add(st)

Response: Runs and profiles the method passed in. A profile object is returned.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def getOrCreate(cls, sc):
        if cls._instantiatedContext is None:
            jsqlContext = sc._jvm.SQLContext.getOrCreate(sc._jsc.sc())
            sparkSession = SparkSession(sc, jsqlContext.sparkSession())
            cls(sc, sparkSession, jsqlContext)
        return cls._instantiatedContext

Response: Get the existing SQLContext or create a new one with given SparkContext.


Instruction:
Can you implement a function in Python 3 that
sets the given Spark SQL configuration property.

Response:
def setConf(self, key, value):
        """Sets the given Spark SQL configuration property.
        """
        self.sparkSession.conf.set(key, value)


Instruction:
Can you tell what is the following Python 3 function doing
def getConf(self, key, defaultValue=_NoValue):
        return self.sparkSession.conf.get(key, defaultValue)

Response: Returns the value of Spark SQL configuration property for the given key.


Instruction:
Implement a Python 3 function for
creating a DataFrame with all the elements in a range from start to end.

Response:
def range(self, start, end=None, step=1, numPartitions=None):
        """
        Create a :class:`DataFrame` with single :class:`pyspark.sql.types.LongType` column named
        ``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
        step value ``step``.

        :param start: the start value
        :param end: the end value (exclusive)
        :param step: the incremental step (default: 1)
        :param numPartitions: the number of partitions of the DataFrame
        :return: :class:`DataFrame`

        >>> sqlContext.range(1, 7, 2).collect()
        [Row(id=1), Row(id=3), Row(id=5)]

        If only one argument is specified, it will be used as the end value.

        >>> sqlContext.range(3).collect()
        [Row(id=0), Row(id=1), Row(id=2)]
        """
        return self.sparkSession.range(start, end, step, numPartitions)


Instruction:
How would you explain what the following Python 3 function does
def registerFunction(self, name, f, returnType=None):
        warnings.warn(
            "Deprecated in 2.3.0. Use spark.udf.register instead.",
            DeprecationWarning)
        return self.sparkSession.udf.register(name, f, returnType)

Response: Register a function f with the specified name.


Instruction:
How would you implement a function in Python 3 that
registers a Java function.

Response:
def registerJavaFunction(self, name, javaClassName, returnType=None):
        """An alias for :func:`spark.udf.registerJavaFunction`.
        See :meth:`pyspark.sql.UDFRegistration.registerJavaFunction`.

        .. note:: Deprecated in 2.3.0. Use :func:`spark.udf.registerJavaFunction` instead.
        """
        warnings.warn(
            "Deprecated in 2.3.0. Use spark.udf.registerJavaFunction instead.",
            DeprecationWarning)
        return self.sparkSession.udf.registerJavaFunction(name, javaClassName, returnType)


Instruction:
Given the following Python 3 function, write the documentation
def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=True):
        return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)

Response: Creates a DataFrame from an RDD of data.


Instruction:
Create a Python 3 function for
creating an external table based on the dataset in a data source.

Response:
def createExternalTable(self, tableName, path=None, source=None, schema=None, **options):
        """Creates an external table based on the dataset in a data source.

        It returns the DataFrame associated with the external table.

        The data source is specified by the ``source`` and a set of ``options``.
        If ``source`` is not specified, the default data source configured by
        ``spark.sql.sources.default`` will be used.

        Optionally, a schema can be provided as the schema of the returned :class:`DataFrame` and
        created external table.

        :return: :class:`DataFrame`
        """
        return self.sparkSession.catalog.createExternalTable(
            tableName, path, source, schema, **options)


Instruction:
Write a Python 3 script for
returning a DataFrame containing names of tables in the given database.

Response:
def tables(self, dbName=None):
        """Returns a :class:`DataFrame` containing names of tables in the given database.

        If ``dbName`` is not specified, the current database will be used.

        The returned DataFrame has two columns: ``tableName`` and ``isTemporary``
        (a column with :class:`BooleanType` indicating if a table is a temporary one or not).

        :param dbName: string, name of the database to use.
        :return: :class:`DataFrame`

        >>> sqlContext.registerDataFrameAsTable(df, "table1")
        >>> df2 = sqlContext.tables()
        >>> df2.filter("tableName = 'table1'").first()
        Row(database=u'', tableName=u'table1', isTemporary=True)
        """
        if dbName is None:
            return DataFrame(self._ssql_ctx.tables(), self)
        else:
            return DataFrame(self._ssql_ctx.tables(dbName), self)


Instruction:
Implement a Python 3 function for
returning a list of names of tables in the database dbName.

Response:
def tableNames(self, dbName=None):
        """Returns a list of names of tables in the database ``dbName``.

        :param dbName: string, name of the database to use. Default to the current database.
        :return: list of table names, in string

        >>> sqlContext.registerDataFrameAsTable(df, "table1")
        >>> "table1" in sqlContext.tableNames()
        True
        >>> "table1" in sqlContext.tableNames("default")
        True
        """
        if dbName is None:
            return [name for name in self._ssql_ctx.tableNames()]
        else:
            return [name for name in self._ssql_ctx.tableNames(dbName)]


Instruction:
Here you have a function in Python 3, explain what it does
def streams(self):
        from pyspark.sql.streaming import StreamingQueryManager
        return StreamingQueryManager(self._ssql_ctx.streams())

Response: Returns a new StreamingQueryManager that allows managing all the


Instruction:
Implement a Python 3 function for
converting a binary column of avro format into its corresponding catalyst value.

Response:
def from_avro(data, jsonFormatSchema, options={}):
    """
    Converts a binary column of avro format into its corresponding catalyst value. The specified
    schema must match the read data, otherwise the behavior is undefined: it may fail or return
    arbitrary result.

    Note: Avro is built-in but external data source module since Spark 2.4. Please deploy the
    application as per the deployment section of "Apache Avro Data Source Guide".

    :param data: the binary column.
    :param jsonFormatSchema: the avro schema in JSON string format.
    :param options: options to control how the Avro record is parsed.

    >>> from pyspark.sql import Row
    >>> from pyspark.sql.avro.functions import from_avro, to_avro
    >>> data = [(1, Row(name='Alice', age=2))]
    >>> df = spark.createDataFrame(data, ("key", "value"))
    >>> avroDf = df.select(to_avro(df.value).alias("avro"))
    >>> avroDf.collect()
    [Row(avro=bytearray(b'\\x00\\x00\\x04\\x00\\nAlice'))]
    >>> jsonFormatSchema = '''{"type":"record","name":"topLevelRecord","fields":
    ...     [{"name":"avro","type":[{"type":"record","name":"value","namespace":"topLevelRecord",
    ...     "fields":[{"name":"age","type":["long","null"]},
    ...     {"name":"name","type":["string","null"]}]},"null"]}]}'''
    >>> avroDf.select(from_avro(avroDf.avro, jsonFormatSchema).alias("value")).collect()
    [Row(value=Row(avro=Row(age=2, name=u'Alice')))]
    """

    sc = SparkContext._active_spark_context
    try:
        jc = sc._jvm.org.apache.spark.sql.avro.functions.from_avro(
            _to_java_column(data), jsonFormatSchema, options)
    except TypeError as e:
        if str(e) == "'JavaPackage' object is not callable":
            _print_missing_jar("Avro", "avro", "avro", sc.version)
        raise
    return Column(jc)


Instruction:
Write a Python 3 script to
get the absolute path of a file added through SparkContext. addFile.

Response:
def get(cls, filename):
        """
        Get the absolute path of a file added through C{SparkContext.addFile()}.
        """
        path = os.path.join(SparkFiles.getRootDirectory(), filename)
        return os.path.abspath(path)


Instruction:
How would you code a function in Python 3 to
get the root directory that contains files added through SparkContext. addFile.

Response:
def getRootDirectory(cls):
        """
        Get the root directory that contains files added through
        C{SparkContext.addFile()}.
        """
        if cls._is_running_on_worker:
            return cls._root_directory
        else:
            # This will have to change if we support multiple SparkContexts:
            return cls._sc._jvm.org.apache.spark.SparkFiles.getRootDirectory()


Instruction:
How would you code a function in Python 3 to
return the training summary of the current model.

Response:
def summary(self):
        """
        Gets summary (e.g. accuracy/precision/recall, objective history, total iterations) of model
        trained on the training set. An exception is thrown if `trainingSummary is None`.
        """
        if self.hasSummary:
            if self.numClasses <= 2:
                return BinaryLogisticRegressionTrainingSummary(super(LogisticRegressionModel,
                                                                     self).summary)
            else:
                return LogisticRegressionTrainingSummary(super(LogisticRegressionModel,
                                                               self).summary)
        else:
            raise RuntimeError("No training summary available for this %s" %
                               self.__class__.__name__)


Instruction:
How would you code a function in Python 3 to
evaluate the model on a test dataset.

Response:
def evaluate(self, dataset):
        """
        Evaluates the model on a test dataset.

        :param dataset:
          Test dataset to evaluate model on, where dataset is an
          instance of :py:class:`pyspark.sql.DataFrame`
        """
        if not isinstance(dataset, DataFrame):
            raise ValueError("dataset must be a DataFrame but got %s." % type(dataset))
        java_blr_summary = self._call_java("evaluate", dataset)
        return BinaryLogisticRegressionSummary(java_blr_summary)


Instruction:
Create a Python 3 function for
creating a deep copy of this instance with a randomly generated uid and some extra parameters.

Response:
def copy(self, extra=None):
        """
        Creates a copy of this instance with a randomly generated uid
        and some extra params. This creates a deep copy of the embedded paramMap,
        and copies the embedded and extra parameters over.

        :param extra: Extra parameters to copy to the new instance
        :return: Copy of this instance
        """
        if extra is None:
            extra = dict()
        newModel = Params.copy(self, extra)
        newModel.models = [model.copy(extra) for model in self.models]
        return newModel


Instruction:
Write a Python 3 function that can
give a Java OneVsRestModel create and return a Python wrapper of it.

Response:
def _from_java(cls, java_stage):
        """
        Given a Java OneVsRestModel, create and return a Python wrapper of it.
        Used for ML persistence.
        """
        featuresCol = java_stage.getFeaturesCol()
        labelCol = java_stage.getLabelCol()
        predictionCol = java_stage.getPredictionCol()
        classifier = JavaParams._from_java(java_stage.getClassifier())
        models = [JavaParams._from_java(model) for model in java_stage.models()]
        py_stage = cls(models=models).setPredictionCol(predictionCol).setLabelCol(labelCol)\
            .setFeaturesCol(featuresCol).setClassifier(classifier)
        py_stage._resetUid(java_stage.uid())
        return py_stage


Instruction:
How would you explain what the following Python 3 function does
def _to_java(self):
        sc = SparkContext._active_spark_context
        java_models = [model._to_java() for model in self.models]
        java_models_array = JavaWrapper._new_java_array(
            java_models, sc._gateway.jvm.org.apache.spark.ml.classification.ClassificationModel)
        metadata = JavaParams._new_java_obj("org.apache.spark.sql.types.Metadata")
        _java_obj = JavaParams._new_java_obj("org.apache.spark.ml.classification.OneVsRestModel",
                                             self.uid, metadata.empty(), java_models_array)
        _java_obj.set("classifier", self.getClassifier()._to_java())
        _java_obj.set("featuresCol", self.getFeaturesCol())
        _java_obj.set("labelCol", self.getLabelCol())
        _java_obj.set("predictionCol", self.getPredictionCol())
        return _java_obj

Response: Transfer this instance to a Java OneVsRestModel. Used for ML persistence.


Instruction:
Make a summary of the following Python 3 code
def _exception_message(excp):
    if isinstance(excp, Py4JJavaError):
        # 'Py4JJavaError' doesn't contain the stack trace available on the Java side in 'message'
        # attribute in Python 2. We should call 'str' function on this exception in general but
        # 'Py4JJavaError' has an issue about addressing non-ascii strings. So, here we work
        # around by the direct call, '__str__()'. Please see SPARK-23517.
        return excp.__str__()
    if hasattr(excp, "message"):
        return excp.message
    return str(excp)

Response: Return the message from an exception as either a str or unicode object.


Instruction:
Write a Python 3 script to
get argspec of a function. Supports both Python 2 and Python 3.

Response:
def _get_argspec(f):
    """
    Get argspec of a function. Supports both Python 2 and Python 3.
    """
    if sys.version_info[0] < 3:
        argspec = inspect.getargspec(f)
    else:
        # `getargspec` is deprecated since python3.0 (incompatible with function annotations).
        # See SPARK-23569.
        argspec = inspect.getfullargspec(f)
    return argspec


Instruction:
Can you tell what is the following Python 3 function doing
def fail_on_stopiteration(f):
    def wrapper(*args, **kwargs):
        try:
            return f(*args, **kwargs)
        except StopIteration as exc:
            raise RuntimeError(
                "Caught StopIteration thrown from user's code; failing the task",
                exc
            )

    return wrapper

Response: A decorator that can be used to fail on a StopIteration raised from Spark code.


Instruction:
Can you tell what is the following Python 3 function doing
def majorMinorVersion(sparkVersion):
        m = re.search(r'^(\d+)\.(\d+)(\..*)?$', sparkVersion)
        if m is not None:
            return (int(m.group(1)), int(m.group(2)))
        else:
            raise ValueError("Spark tried to parse '%s' as a Spark" % sparkVersion +
                             " version string, but it could not find the major and minor" +
                             " version numbers.")

Response: Given a Spark version string return the major and minor version numbers.


Instruction:
Make a summary of the following Python 3 code
def _ensure_initialized(cls, instance=None, gateway=None, conf=None):
        with SparkContext._lock:
            if not SparkContext._gateway:
                SparkContext._gateway = gateway or launch_gateway(conf)
                SparkContext._jvm = SparkContext._gateway.jvm

            if instance:
                if (SparkContext._active_spark_context and
                        SparkContext._active_spark_context != instance):
                    currentMaster = SparkContext._active_spark_context.master
                    currentAppName = SparkContext._active_spark_context.appName
                    callsite = SparkContext._active_spark_context._callsite

                    # Raise error if there is already a running Spark context
                    raise ValueError(
                        "Cannot run multiple SparkContexts at once; "
                        "existing SparkContext(app=%s, master=%s)"
                        " created by %s at %s:%s "
                        % (currentAppName, currentMaster,
                            callsite.function, callsite.file, callsite.linenum))
                else:
                    SparkContext._active_spark_context = instance

Response: Checks whether a SparkContext is initialized or not.


Instruction:
Here you have a function in Python 3, explain what it does
def getOrCreate(cls, conf=None):
        with SparkContext._lock:
            if SparkContext._active_spark_context is None:
                SparkContext(conf=conf or SparkConf())
            return SparkContext._active_spark_context

Response: Get or instantiate a SparkContext and register it as a singleton object.