Instruction:
Can you tell what is the following Python 3 function doing
def full_opt_run(cls, vasp_cmd, vol_change_tol=0.02,
                     max_steps=10, ediffg=-0.05, half_kpts_first_relax=False,
                     **vasp_job_kwargs):
        """
        Returns a generator of jobs for a full optimization run. Basically,
        this runs an infinite series of geometry optimization jobs until the
        % vol change in a particular optimization is less than vol_change_tol.

        Args:
            vasp_cmd (str): Command to run vasp as a list of args. For example,
                if you are using mpirun, it can be something like
                ["mpirun", "pvasp.5.2.11"]
            vol_change_tol (float): The tolerance at which to stop a run.
                Defaults to 0.05, i.e., 5%.
            max_steps (int): The maximum number of runs. Defaults to 10 (
                highly unlikely that this limit is ever reached).
            ediffg (float): Force convergence criteria for subsequent runs (
                ignored for the initial run.)
            half_kpts_first_relax (bool): Whether to halve the kpoint grid
                for the first relaxation. Speeds up difficult convergence
                considerably. Defaults to False.
            \*\*vasp_job_kwargs: Passthrough kwargs to VaspJob. See
                :class:`custodian.vasp.jobs.VaspJob`.

        Returns:
            Generator of jobs.
        """
        for i in range(max_steps):
            if i == 0:
                settings = None
                backup = True
                if half_kpts_first_relax and os.path.exists("KPOINTS") and \
                        os.path.exists("POSCAR"):
                    kpts = Kpoints.from_file("KPOINTS")
                    orig_kpts_dict = kpts.as_dict()
                    kpts.kpts = np.maximum(np.array(kpts.kpts) / 2, 1).tolist()
                    low_kpts_dict = kpts.as_dict()
                    settings = [
                        {"dict": "KPOINTS",
                         "action": {"_set": low_kpts_dict}}
                    ]
            else:
                backup = False
                initial = Poscar.from_file("POSCAR").structure
                final = Poscar.from_file("CONTCAR").structure
                vol_change = (final.volume - initial.volume) / initial.volume

                logger.info("Vol change = %.1f %%!" % (vol_change * 100))
                if abs(vol_change) < vol_change_tol:
                    logger.info("Stopping optimization!")
                    break
                else:
                    incar_update = {"ISTART": 1}
                    if ediffg:
                        incar_update["EDIFFG"] = ediffg
                    settings = [
                        {"dict": "INCAR",
                         "action": {"_set": incar_update}},
                        {"file": "CONTCAR",
                         "action": {"_file_copy": {"dest": "POSCAR"}}}]
                    if i == 1 and half_kpts_first_relax:
                        settings.append({"dict": "KPOINTS",
                                         "action": {"_set": orig_kpts_dict}})
            logger.info("Generating job = %d!" % (i+1))
            yield VaspJob(vasp_cmd, final=False, backup=backup,
                          suffix=".relax%d" % (i+1), settings_override=settings,
                          **vasp_job_kwargs)

Response: This function runs a full optimization run of a specific object.


Instruction:
How would you code a function in Python 3 to
return a generator of jobs to run a constrained optimization run on a set of species species.

Response:
def constrained_opt_run(cls, vasp_cmd, lattice_direction, initial_strain,
                            atom_relax=True, max_steps=20, algo="bfgs",
                            **vasp_job_kwargs):
        """
        Returns a generator of jobs for a constrained optimization run. Typical
        use case is when you want to approximate a biaxial strain situation,
        e.g., you apply a defined strain to a and b directions of the lattice,
        but allows the c-direction to relax.

        Some guidelines on the use of this method:
        i.  It is recommended you do not use the Auto kpoint generation. The
            grid generated via Auto may fluctuate with changes in lattice
            param, resulting in numerical noise.
        ii. Make sure your EDIFF/EDIFFG is properly set in your INCAR. The
            optimization relies on these values to determine convergence.

        Args:
            vasp_cmd (str): Command to run vasp as a list of args. For example,
                if you are using mpirun, it can be something like
                ["mpirun", "pvasp.5.2.11"]
            lattice_direction (str): Which direction to relax. Valid values are
                "a", "b" or "c".
            initial_strain (float): An initial strain to be applied to the
                lattice_direction. This can usually be estimated as the
                negative of the strain applied in the other two directions.
                E.g., if you apply a tensile strain of 0.05 to the a and b
                directions, you can use -0.05 as a reasonable first guess for
                initial strain.
            atom_relax (bool): Whether to relax atomic positions.
            max_steps (int): The maximum number of runs. Defaults to 20 (
                highly unlikely that this limit is ever reached).
            algo (str): Algorithm to use to find minimum. Default is "bfgs",
                which is fast, but can be sensitive to numerical noise
                in energy calculations. The alternative is "bisection",
                which is more robust but can be a bit slow. The code does fall
                back on the bisection when bfgs gives a non-sensical result,
                e.g., negative lattice params.
            \*\*vasp_job_kwargs: Passthrough kwargs to VaspJob. See
                :class:`custodian.vasp.jobs.VaspJob`.

        Returns:
            Generator of jobs. At the end of the run, an "EOS.txt" is written
            which provides a quick look at the E vs lattice parameter.
        """
        nsw = 99 if atom_relax else 0

        incar = Incar.from_file("INCAR")

        # Set the energy convergence criteria as the EDIFFG (if present) or
        # 10 x EDIFF (which itself defaults to 1e-4 if not present).
        if incar.get("EDIFFG") and incar.get("EDIFFG") > 0:
            etol = incar["EDIFFG"]
        else:
            etol = incar.get("EDIFF", 1e-4) * 10

        if lattice_direction == "a":
            lattice_index = 0
        elif lattice_direction == "b":
            lattice_index = 1
        else:
            lattice_index = 2

        energies = {}

        for i in range(max_steps):
            if i == 0:
                settings = [
                        {"dict": "INCAR",
                         "action": {"_set": {"ISIF": 2, "NSW": nsw}}}]
                structure = Poscar.from_file("POSCAR").structure
                x = structure.lattice.abc[lattice_index]
                backup = True
            else:
                backup = False
                v = Vasprun("vasprun.xml")
                structure = v.final_structure
                energy = v.final_energy
                lattice = structure.lattice

                x = lattice.abc[lattice_index]

                energies[x] = energy

                if i == 1:
                    x *= (1 + initial_strain)
                else:
                    # Sort the lattice parameter by energies.
                    min_x = min(energies.keys(), key=lambda e: energies[e])
                    sorted_x = sorted(energies.keys())
                    ind = sorted_x.index(min_x)
                    if ind == 0:
                        other = ind + 1
                    elif ind == len(sorted_x) - 1:
                        other = ind - 1
                    else:
                        other = ind + 1 \
                            if energies[sorted_x[ind + 1]] \
                            < energies[sorted_x[ind - 1]] \
                            else ind - 1
                    if abs(energies[min_x]
                           - energies[sorted_x[other]]) < etol:
                        logger.info("Stopping optimization! Final %s = %f"
                                    % (lattice_direction, min_x))
                        break

                    if ind == 0 and len(sorted_x) > 2:
                        # Lowest energy lies outside of range of lowest value.
                        # we decrease the lattice parameter in the next
                        # iteration to find a minimum. This applies only when
                        # there are at least 3 values.
                        x = sorted_x[0] - abs(sorted_x[1] - sorted_x[0])
                        logger.info("Lowest energy lies below bounds. "
                                    "Setting %s = %f." % (lattice_direction, x))
                    elif ind == len(sorted_x) - 1 and len(sorted_x) > 2:
                        # Lowest energy lies outside of range of highest value.
                        # we increase the lattice parameter in the next
                        # iteration to find a minimum. This applies only when
                        # there are at least 3 values.
                        x = sorted_x[-1] + abs(sorted_x[-1] - sorted_x[-2])
                        logger.info("Lowest energy lies above bounds. "
                                    "Setting %s = %f." % (lattice_direction, x))
                    else:
                        if algo.lower() == "bfgs" and len(sorted_x) >= 4:
                            try:
                                # If there are more than 4 data points, we will
                                # do a quadratic fit to accelerate convergence.
                                x1 = list(energies.keys())
                                y1 = [energies[j] for j in x1]
                                z1 = np.polyfit(x1, y1, 2)
                                pp = np.poly1d(z1)
                                from scipy.optimize import minimize
                                result = minimize(
                                    pp, min_x,
                                    bounds=[(sorted_x[0], sorted_x[-1])])
                                if (not result.success) or result.x[0] < 0:
                                    raise ValueError(
                                        "Negative lattice constant!")
                                x = result.x[0]
                                logger.info("BFGS minimized %s = %f."
                                            % (lattice_direction, x))
                            except ValueError as ex:
                                # Fall back on bisection algo if the bfgs fails.
                                logger.info(str(ex))
                                x = (min_x + sorted_x[other]) / 2
                                logger.info("Falling back on bisection %s = %f."
                                            % (lattice_direction, x))
                        else:
                            x = (min_x + sorted_x[other]) / 2
                            logger.info("Bisection %s = %f."
                                        % (lattice_direction, x))

                lattice = lattice.matrix
                lattice[lattice_index] = lattice[lattice_index] / \
                    np.linalg.norm(lattice[lattice_index]) * x

                s = Structure(lattice, structure.species, structure.frac_coords)
                fname = "POSCAR.%f" % x
                s.to(filename=fname)

                incar_update = {"ISTART": 1, "NSW": nsw, "ISIF": 2}

                settings = [
                    {"dict": "INCAR",
                     "action": {"_set": incar_update}},
                    {"file": fname,
                     "action": {"_file_copy": {"dest": "POSCAR"}}}]

            logger.info("Generating job = %d with parameter %f!" % (i + 1, x))
            yield VaspJob(vasp_cmd, final=False, backup=backup,
                          suffix=".static.%f" % x,
                          settings_override=settings, **vasp_job_kwargs)

        with open("EOS.txt", "wt") as f:
            f.write("# %s energy\n" % lattice_direction)
            for k in sorted(energies.keys()):
                f.write("%f %f\n" % (k, energies[k]))


Instruction:
Create a Python 3 function to
set up the necessary data structures for the VASPNEBJob.

Response:
def setup(self):
        """
        Performs initial setup for VaspNEBJob, including overriding any settings
        and backing up.
        """
        neb_dirs = self.neb_dirs

        if self.backup:
            # Back up KPOINTS, INCAR, POTCAR
            for f in VASP_NEB_INPUT_FILES:
                shutil.copy(f, "{}.orig".format(f))
            # Back up POSCARs
            for path in neb_dirs:
                poscar = os.path.join(path, "POSCAR")
                shutil.copy(poscar, "{}.orig".format(poscar))

        if self.half_kpts and os.path.exists("KPOINTS"):
            kpts = Kpoints.from_file("KPOINTS")
            kpts.kpts = np.maximum(np.array(kpts.kpts) / 2, 1)
            kpts.kpts = kpts.kpts.astype(int).tolist()
            if tuple(kpts.kpts[0]) == (1, 1, 1):
                kpt_dic = kpts.as_dict()
                kpt_dic["generation_style"] = 'Gamma'
                kpts = Kpoints.from_dict(kpt_dic)
            kpts.write_file("KPOINTS")

        if self.auto_npar:
            try:
                incar = Incar.from_file("INCAR")
                import multiprocessing
                # Try sge environment variable first
                # (since multiprocessing counts cores on the current
                # machine only)
                ncores = os.environ.get('NSLOTS') or multiprocessing.cpu_count()
                ncores = int(ncores)
                for npar in range(int(math.sqrt(ncores)),
                                  ncores):
                    if ncores % npar == 0:
                        incar["NPAR"] = npar
                        break
                incar.write_file("INCAR")
            except:
                pass

        if self.auto_continue and \
                os.path.exists("STOPCAR") and \
                not os.access("STOPCAR", os.W_OK):
            # Remove STOPCAR
            os.chmod("STOPCAR", 0o644)
            os.remove("STOPCAR")

            # Copy CONTCAR to POSCAR
            for path in self.neb_sub:
                contcar = os.path.join(path, "CONTCAR")
                poscar = os.path.join(path, "POSCAR")
                shutil.copy(contcar, poscar)

        if self.settings_override is not None:
            VaspModder().apply_actions(self.settings_override)


Instruction:
Write a Python 3 script to
set up the backup file if necessary.

Response:
def setup(self):
        """
        Performs backup if necessary.
        """
        if self.backup:
            shutil.copy(self.input_file, "{}.orig".format(self.input_file))


Instruction:
Can you tell what is the following Python 3 function doing
def run(self):
        with zopen(self.output_file, 'w') as fout:
            return subprocess.Popen(self.nwchem_cmd + [self.input_file],
                                    stdout=fout)

Response: Performs actual nwchem run.


Instruction:
How would you explain what the following Python 3 function does
def valid_GC(x):
    x = float(x)
    if x < 0.0 or x > 1.0:
        raise ArgumentTypeError("{} not in range [0.0, 1.0]".format(x))
    return x

Response: type function for argparse to check GC values.


Instruction:
Here you have a function in Python 3, explain what it does
def filter_stream(fq, args):
    if args.quality:
        quality_check = ave_qual
    else:
        quality_check = silent_quality_check
    minlen = args.length + int(args.headcrop or 0) - (int(args.tailcrop or 0))
    for rec in SeqIO.parse(fq, "fastq"):
        if args.GC_filter:
            gc = (rec.seq.upper().count("C") + rec.seq.upper().count("G")) / len(rec)
        else:
            gc = 0.50  # dummy variable
        if quality_check(rec.letter_annotations["phred_quality"]) > args.quality \
                and minlen <= len(rec) <= args.maxlength \
                and args.minGC <= gc <= args.maxGC:
            print(rec[args.headcrop:args.tailcrop].format("fastq"), end="")

Response: Filter a fastq file on stdin and print out the filtered fastq records.


Instruction:
Can you tell what is the following Python 3 function doing
def filter_using_summary(fq, args):
    data = {entry[0]: entry[1] for entry in process_summary(
        summaryfile=args.summary,
        threads="NA",
        readtype=args.readtype,
        barcoded=False)[
        ["readIDs", "quals"]].itertuples(index=False)}
    try:
        for record in SeqIO.parse(fq, "fastq"):
            if data[record.id] > args.quality \
                    and args.length <= len(record) <= args.maxlength:
                print(record[args.headcrop:args.tailcrop].format("fastq"), end="")
    except KeyError:
        logging.error("mismatch between summary and fastq: \
                       {} was not found in the summary file.".format(record.id))
        sys.exit('\nERROR: mismatch between sequencing_summary and fastq file: \
                 {} was not found in the summary file.\nQuitting.'.format(record.id))

Response: Filter the sequence using the albacore summary file for more accurate quality estimate


Instruction:
Given the following Python 3 function, write the documentation
def master_key_required(func):
    '''decorator describing methods that require the master key'''
    def ret(obj, *args, **kw):
        conn = ACCESS_KEYS
        if not (conn and conn.get('master_key')):
            message = '%s requires the master key' % func.__name__
            raise core.ParseError(message)
        func(obj, *args, **kw)
    return ret

Response: decorator describing methods that require the master key


Instruction:
Given the following Python 3 function, write the documentation
def execute(cls, uri, http_verb, extra_headers=None, batch=False, _body=None, **kw):
        if batch:
            urlsplitter = urlparse(API_ROOT).netloc
            ret = {"method": http_verb, "path": uri.split(urlsplitter, 1)[1]}
            if kw:
                ret["body"] = kw
            return ret

        if not ('app_id' in ACCESS_KEYS and 'rest_key' in ACCESS_KEYS):
            raise core.ParseError('Missing connection credentials')

        app_id = ACCESS_KEYS.get('app_id')
        rest_key = ACCESS_KEYS.get('rest_key')
        master_key = ACCESS_KEYS.get('master_key')

        url = uri if uri.startswith(API_ROOT) else cls.ENDPOINT_ROOT + uri
        if _body is None:
            data = kw and json.dumps(kw, default=date_handler) or "{}"
        else:
            data = _body
        if http_verb == 'GET' and data:
            url += '?%s' % urlencode(kw)
            data = None
        else:
            if cls.__name__ == 'File':
                data = data
            else:
                data = data.encode('utf-8')

        headers = {
            'Content-type': 'application/json',
            'X-Parse-Application-Id': app_id,
            'X-Parse-REST-API-Key': rest_key
        }
        headers.update(extra_headers or {})

        if cls.__name__ == 'File':
            request = Request(url.encode('utf-8'), data, headers)
        else:
            request = Request(url, data, headers)

        if ACCESS_KEYS.get('session_token'):
            request.add_header('X-Parse-Session-Token', ACCESS_KEYS.get('session_token'))
        elif master_key:
            request.add_header('X-Parse-Master-Key', master_key)

        request.get_method = lambda: http_verb

        try:
            response = urlopen(request, timeout=CONNECTION_TIMEOUT)
        except HTTPError as e:
            exc = {
                400: core.ResourceRequestBadRequest,
                401: core.ResourceRequestLoginRequired,
                403: core.ResourceRequestForbidden,
                404: core.ResourceRequestNotFound
                }.get(e.code, core.ParseError)
            raise exc(e.read())

        return json.loads(response.read().decode('utf-8'))

Response: Execute a command with the given parameters and return the response JSON.


Instruction:
Can you tell what is the following Python 3 function doing
def batch(self, methods):
        methods = list(methods) # methods can be iterator
        if not methods:
            #accepts also empty list (or generator) - it allows call batch directly with query result (eventually empty)
            return
        queries, callbacks = list(zip(*[m(batch=True) for m in methods]))
        # perform all the operations in one batch
        responses = self.execute("", "POST", requests=queries)
        # perform the callbacks with the response data (updating the existing
        # objets, etc)

        batched_errors = []
        for callback, response in zip(callbacks, responses):
            if "success" in response:
                callback(response["success"])
            else:
                batched_errors.append(response["error"])

        if batched_errors:
            raise core.ParseBatchError(batched_errors)

Response: Calls a list of create update or delete methods in a single batch operation.


Instruction:
Explain what the following Python 3 code does
def update_channels(cls, installation_id, channels_to_add=set(),
                        channels_to_remove=set(), **kw):
        """
        Allow an application to manually subscribe or unsubscribe an
        installation to a certain push channel in a unified operation.

        this is based on:
        https://www.parse.com/docs/rest#installations-updating

        installation_id: the installation id you'd like to add a channel to
        channels_to_add: the name of the channel you'd like to subscribe the user to
        channels_to_remove: the name of the channel you'd like to unsubscribe the user from

        """
        installation_url = cls._get_installation_url(installation_id)
        current_config = cls.GET(installation_url)

        new_channels = list(set(current_config['channels']).union(channels_to_add).difference(channels_to_remove))

        cls.PUT(installation_url, channels=new_channels)

Response: Update the push channels of an anonymization.


Instruction:
How would you code a function in Python 3 to
fetch the objects matching the query.

Response:
def _fetch(self, count=False):
        if self._result_cache is not None:
            return len(self._result_cache) if count else self._result_cache
        """
        Return a list of objects matching query, or if count == True return
        only the number of objects matching.
        """
        options = dict(self._options)  # make a local copy
        if self._where:
            # JSON encode WHERE values
            options['where'] = json.dumps(self._where)
        if self._select_related:
            options['include'] = ','.join(self._select_related)
        if count:
            return self._manager._count(**options)

        self._result_cache = self._manager._fetch(**options)
        return self._result_cache


Instruction:
Can you write a function in Python 3 where it
creates a proper object subclass matching class_name

Response:
def factory(cls, class_name):
        """find proper Object subclass matching class_name
        system types like _User are mapped to types without underscore (parse_resr.user.User)
        If user don't declare matching type, class is created on the fly
        """
        class_name = str(class_name.lstrip('_'))
        types = ParseResource.__subclasses__()
        while types:
            t = types.pop()
            if t.__name__ == class_name:
                return t
            types.extend(t.__subclasses__())
        else:
            return type(class_name, (Object,), {})


Instruction:
Here you have a function in Python 3, explain what it does
def schema(cls):
        root = '/'.join([API_ROOT, 'schemas', cls.__name__])
        schema = cls.GET(root)
        return schema

Response: Retrieves the class s schema.


Instruction:
Implement a Python 3 function for
incrementing one value in the object.

Response:
def increment(self, key, amount=1):
        """
        Increment one value in the object. Note that this happens immediately:
        it does not wait for save() to be called
        """
        payload = {
            key: {
                '__op': 'Increment',
                'amount': amount
                }
            }
        self.__class__.PUT(self._absolute_url, **payload)
        self.__dict__[key] += amount


Instruction:
Make a summary of the following Python 3 code
def remove(self, key):
        payload = {
            key: {
                '__op': 'Delete'
                }
            }
        self.__class__.PUT(self._absolute_url, **payload)
        del self.__dict__[key]

Response: Remove a column value in the object.


Instruction:
Explain what the following Python 3 code does
def login_required(func):
    '''decorator describing User methods that need to be logged in'''
    def ret(obj, *args, **kw):
        if not hasattr(obj, 'sessionToken'):
            message = '%s requires a logged-in session' % func.__name__
            raise ResourceRequestLoginRequired(message)
        return func(obj, *args, **kw)
    return ret

Response: decorator describing User methods that need to be logged in


Instruction:
How would you implement a function in Python 3 that
triggers Parse s Password Process. Return True / False indicate success on the request

Response:
def request_password_reset(email):
        '''Trigger Parse\'s Password Process. Return True/False
        indicate success/failure on the request'''

        url = '/'.join([API_ROOT, 'requestPasswordReset'])
        try:
            User.POST(url, email=email)
            return True
        except ParseError:
            return False


Instruction:
Explain what the following Python 3 code does
def parse(d):
    res = {}
    for k, v in iteritems(d):
        if isinstance(v, string_types) and DATETIME_ISO_FORMAT.match(v):
            v = dateutil.parser.parse(v)
        res[k] = v
    return res

Response: Convert iso formatted timestamps found as values in the dict d to datetime objects.


Instruction:
Can you generate the documentation for the following Python 3 function
def dump(obj, path, **kw):
    open_kw = {'mode': 'w'}
    if PY3:  # pragma: no cover
        open_kw['encoding'] = 'utf-8'

    # avoid indented lines ending with ", " on PY2
    if kw.get('indent') and kw.get('separators') is None:
        kw['separators'] = (',', ': ')

    with open(str(path), **open_kw) as fp:
        return json.dump(obj, fp, **kw)

Response: Python 2 + 3 compatible version of json. dump.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load(path, **kw):
    _kw = {}
    if PY3:  # pragma: no cover
        _kw['encoding'] = 'utf-8'
    with open(str(path), **_kw) as fp:
        return json.load(fp, **kw)

Response: Python 2 + 3 compatible version of json. load.


Instruction:
Create a Python 3 function to
strip brackets and what is inside brackets from text.

Response:
def strip_brackets(text, brackets=None):
    """Strip brackets and what is inside brackets from text.

    .. note::
        If the text contains only one opening bracket, the rest of the text
        will be ignored. This is a feature, not a bug, as we want to avoid that
        this function raises errors too easily.
    """
    res = []
    for c, type_ in _tokens(text, brackets=brackets):
        if type_ == TextType.text:
            res.append(c)
    return ''.join(res).strip()


Instruction:
Implement a Python 3 function for
splitting text at separators outside of brackets.

Response:
def split_text_with_context(text, separators=WHITESPACE, brackets=None):
    """Splits text at separators outside of brackets.

    :param text:
    :param separators: An iterable of single character tokens.
    :param brackets:
    :return: A `list` of non-empty chunks.

    .. note:: This function leaves content in brackets in the chunks.
    """
    res, chunk = [], []
    for c, type_ in _tokens(text, brackets=brackets):
        if type_ == TextType.text and c in separators:
            res.append(''.join(chunk).strip())
            chunk = []
        else:
            chunk.append(c)
    res.append(''.join(chunk).strip())
    return nfilter(res)


Instruction:
Can you write a function in Python 3 where it
splits text along the separators unless they appear within brackets.

Response:
def split_text(text, separators=re.compile('\s'), brackets=None, strip=False):
    """Split text along the separators unless they appear within brackets.

    :param separators: An iterable single characters or a compiled regex pattern.
    :param brackets: `dict` mapping start tokens to end tokens of what is to be \
    recognized as brackets.

    .. note:: This function will also strip content within brackets.
    """
    if not isinstance(separators, PATTERN_TYPE):
        separators = re.compile(
            '[{0}]'.format(''.join('\{0}'.format(c) for c in separators)))

    return nfilter(
        s.strip() if strip else s for s in
        separators.split(strip_brackets(text, brackets=brackets)))


Instruction:
Write a Python 3 function for
yielding marker value pairs from a text block.

Response:
def marker_split(block):
    """Yield marker, value pairs from a text block (i.e. a list of lines).

    :param block: text block consisting of \n separated lines as it will be the case for \
    files read using "rU" mode.
    :return: generator of (marker, value) pairs.
    """
    marker = None
    value = []

    for line in block.split('\n'):
        line = line.strip()
        if line.startswith('\\_'):
            continue  # we simply ignore SFM header fields
        match = MARKER_PATTERN.match(line)
        if match:
            if marker:
                yield marker, '\n'.join(value)
            marker = match.group('marker')
            value = [line[match.end():]]
        else:
            value.append(line)
    if marker:
        yield marker, ('\n'.join(value)).strip()


Instruction:
Write a Python 3 script to
retrieve the first value for a marker or None.

Response:
def get(self, key, default=None):
        """Retrieve the first value for a marker or None."""
        for k, v in self:
            if k == key:
                return v
        return default


Instruction:
Given the following Python 3 function, write the documentation
def read(self,
             filename,
             encoding='utf-8',
             marker_map=None,
             entry_impl=Entry,
             entry_sep='\n\n',
             entry_prefix=None,
             keep_empty=False):
        """Extend the list by parsing new entries from a file.

        :param filename:
        :param encoding:
        :param marker_map: A dict used to map marker names.
        :param entry_impl: Subclass of Entry or None
        :param entry_sep:
        :param entry_prefix:
        """
        marker_map = marker_map or {}
        for entry in parse(
                filename,
                encoding,
                entry_sep,
                entry_prefix or entry_sep,
                keep_empty=keep_empty):
            if entry:
                self.append(entry_impl([(marker_map.get(k, k), v) for k, v in entry]))

Response: Extend the list by parsing new entries from a file.


Instruction:
Can you tell what is the following Python 3 function doing
def write(self, filename, encoding='utf-8'):
        with io.open(str(filename), 'w', encoding=encoding) as fp:
            for entry in self:
                fp.write(entry.__unicode__())
                fp.write('\n\n')

Response: Write the list of entries to a file.


Instruction:
Create a Python 3 function for
asking a yes or no question interactively.

Response:
def confirm(question, default=True):
    """Ask a yes/no question interactively.

    :param question: The text of the question to ask.
    :returns: True if the answer was "yes", False otherwise.
    """
    valid = {"": default, "yes": True, "y": True, "no": False, "n": False}
    while 1:
        choice = input(question + (" [Y/n] " if default else " [y/N] ")).lower()
        if choice in valid:
            return valid[choice]
        print("Please respond with 'y' or 'n' ")


Instruction:
Given the following Python 3 function, write the documentation
def gettext(self, section, option, whitespace_preserving_prefix='.'):
        lines = []
        for line in self.get(section, option, fallback='').splitlines():
            if re.match(re.escape(whitespace_preserving_prefix) + '\s+', line):
                line = line[len(whitespace_preserving_prefix):]
            lines.append(line)
        return '\n'.join(lines)

Response: Return the gettext string for the specified option in section.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def data_url(content, mimetype=None):
    if isinstance(content, pathlib.Path):
        if not mimetype:
            mimetype = guess_type(content.name)[0]
        with content.open('rb') as fp:
            content = fp.read()
    else:
        if isinstance(content, text_type):
            content = content.encode('utf8')
    return "data:{0};base64,{1}".format(
        mimetype or 'application/octet-stream', b64encode(content).decode())

Response: Returns content encoded as base64 Data URI.


Instruction:
Make a summary of the following Python 3 code
def dict_merged(d, _filter=None, **kw):
    def f(s):
        if _filter:
            return _filter(s)
        return s is not None
    d = d or {}
    for k, v in iteritems(kw):
        if f(v):
            d[k] = v
    return d

Response: Update dictionary d with the items passed as kw if the value passes _filter.


Instruction:
Implement a Python 3 function for
removing illegal characters from text.

Response:
def xmlchars(text):
    """Not all of UTF-8 is considered valid character data in XML ...

    Thus, this function can be used to remove illegal characters from ``text``.
    """
    invalid = list(range(0x9))
    invalid.extend([0xb, 0xc])
    invalid.extend(range(0xe, 0x20))
    return re.sub('|'.join('\\x%0.2X' % i for i in invalid), '', text)


Instruction:
Given the following Python 3 function, write the documentation
def slug(s, remove_whitespace=True, lowercase=True):
    res = ''.join(c for c in unicodedata.normalize('NFD', s)
                  if unicodedata.category(c) != 'Mn')
    if lowercase:
        res = res.lower()
    for c in string.punctuation:
        res = res.replace(c, '')
    res = re.sub('\s+', '' if remove_whitespace else ' ', res)
    res = res.encode('ascii', 'ignore').decode('ascii')
    assert re.match('[ A-Za-z0-9]*$', res)
    return res

Response: Condensed version of s containing only lowercase alphanumeric characters.


Instruction:
How would you implement a function in Python 3 that
casts string to binary_type.

Response:
def encoded(string, encoding='utf-8'):
    """Cast string to binary_type.

    :param string: six.binary_type or six.text_type
    :param encoding: encoding which the object is forced to
    :return: six.binary_type
    """
    assert isinstance(string, string_types) or isinstance(string, binary_type)
    if isinstance(string, text_type):
        return string.encode(encoding)
    try:
        # make sure the string can be decoded in the specified encoding ...
        string.decode(encoding)
        return string
    except UnicodeDecodeError:
        # ... if not use latin1 as best guess to decode the string before encoding as
        # specified.
        return string.decode('latin1').encode(encoding)


Instruction:
Given the following Python 3 function, write the documentation
def readlines(p,
              encoding=None,
              strip=False,
              comment=None,
              normalize=None,
              linenumbers=False):
    """
    Read a `list` of lines from a text file.

    :param p: File path (or `list` or `tuple` of text)
    :param encoding: Registered codec.
    :param strip: If `True`, strip leading and trailing whitespace.
    :param comment: String used as syntax to mark comment lines. When not `None`, \
    commented lines will be stripped. This implies `strip=True`.
    :param normalize: 'NFC', 'NFKC', 'NFD', 'NFKD'
    :param linenumbers: return also line numbers.
    :return: `list` of text lines or pairs (`int`, text or `None`).
    """
    if comment:
        strip = True
    if isinstance(p, (list, tuple)):
        res = [l.decode(encoding) if encoding else l for l in p]
    else:
        with Path(p).open(encoding=encoding or 'utf-8') as fp:
            res = fp.readlines()
    if strip:
        res = [l.strip() or None for l in res]
    if comment:
        res = [None if l and l.startswith(comment) else l for l in res]
    if normalize:
        res = [unicodedata.normalize(normalize, l) if l else l for l in res]
    if linenumbers:
        return [(n, l) for n, l in enumerate(res, 1)]
    return [l for l in res if l is not None]

Response: Read a list of lines from a text file.


Instruction:
How would you implement a function in Python 3 that
wraps for os. walk yielding Path objects.

Response:
def walk(p, mode='all', **kw):
    """Wrapper for `os.walk`, yielding `Path` objects.

    :param p: root of the directory tree to walk.
    :param mode: 'all|dirs|files', defaulting to 'all'.
    :param kw: Keyword arguments are passed to `os.walk`.
    :return: Generator for the requested Path objects.
    """
    for dirpath, dirnames, filenames in os.walk(as_posix(p), **kw):
        if mode in ('all', 'dirs'):
            for dirname in dirnames:
                yield Path(dirpath).joinpath(dirname)
        if mode in ('all', 'files'):
            for fname in filenames:
                yield Path(dirpath).joinpath(fname)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def bibtex(self):
        m = max(itertools.chain(map(len, self), [0]))
        fields = ("  %s = {%s}" % (k.ljust(m), self[k]) for k in self)
        return "@%s{%s,\n%s\n}" % (
            getattr(self.genre, 'value', self.genre), self.id, ",\n".join(fields))

Response: Represent the source in BibTeX format.


Instruction:
Can you generate the documentation for the following Python 3 function
def text(self):
        genre = getattr(self.genre, 'value', self.genre)
        pages_at_end = genre in (
            'book',
            'phdthesis',
            'mastersthesis',
            'misc',
            'techreport')
        thesis = genre in ('phdthesis', 'mastersthesis')

        if self.get('editor'):
            editors = self['editor']
            affix = 'eds' if ' and ' in editors or '&' in editors else 'ed'
            editors = " %s (%s.)" % (editors, affix)
        else:
            editors = None

        res = [self.get('author', editors), self.get('year', 'n.d')]
        if genre == 'book':
            res.append(self.get_with_translation('booktitle') or
                       self.get_with_translation('title'))
            series = ', '.join(filter(None, [self.get('series'), self.get('volume')]))
            if series:
                res.append('(%s.)' % series)
        elif genre == 'misc':
            # in case of misc records, we use the note field in case a title is missing.
            res.append(self.get_with_translation('title') or self.get('note'))
        else:
            res.append(self.get_with_translation('title'))

        if genre == 'article':
            atom = ' '.join(filter(None, [self.get('journal'), self.get('volume')]))
            if self.get('issue'):
                atom += '(%s)' % self['issue']
            res.append(atom)
            res.append(self.get('pages'))
        elif genre == 'incollection' or genre == 'inproceedings':
            prefix = 'In'
            atom = ''
            if editors:
                atom += editors
            if self.get('booktitle'):
                if atom:
                    atom += ','
                atom += " %s" % self.get_with_translation('booktitle')
            if self.get('pages'):
                atom += ", %s" % self['pages']
            res.append(prefix + atom)
        else:
            # check for author to make sure we haven't included the editors yet.
            if editors and self.get('author'):
                res.append("In %s" % editors)

            for attr in [
                'journal',
                'volume' if genre != 'book' else None,
            ]:
                if attr and self.get(attr):
                    res.append(self.get(attr))

            if self.get('issue'):
                res.append("(%s)" % self['issue'])

            if not pages_at_end and self.get('pages'):
                res.append(self['pages'])

        if self.get('publisher'):
            res.append(": ".join(filter(None, [self.get('address'), self['publisher']])))
        else:
            if genre == 'misc' and self.get('howpublished'):
                res.append(self.get('howpublished'))

        if not thesis and pages_at_end and self.get('pages'):
            res.append(self['pages'] + 'pp')

        note = self.get('note') or self._genre_note.get(genre)
        if note and note not in res:
            if thesis:
                joiner = ','
                if self.get('school'):
                    note += '{0} {1}'.format(joiner, self.get('school'))
                    joiner = ';'
                if self.get('pages'):
                    note += '{0} {1}pp.'.format(joiner, self.get('pages'))
            res.append('(%s)' % note)

        return ' '.join(
            x if x.endswith(('.', '.)')) else '%s.' % x for x in res if x)

Response: Linearize the bib source according to the rules of the unified style.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def request(self, path, method='GET', params=None, type=REST_TYPE):
        response_text = self._get_http_client(type).request(path, method, params)

        if not response_text:
            return response_text

        response_json = json.loads(response_text)

        if 'errors' in response_json:
            raise (ErrorException([Error().load(e) for e in response_json['errors']]))

        return response_json

Response: Builds a request gets a response and decodes it.


Instruction:
Given the following Python 3 function, write the documentation
def hlr_create(self, msisdn, reference):
        return HLR().load(self.request('hlr', 'POST', {'msisdn': msisdn, 'reference': reference}))

Response: Perform a new HLR lookup.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def message_create(self, originator, recipients, body, params=None):
        if params is None: params = {}
        if type(recipients) == list:
            recipients = ','.join(recipients)

        params.update({'originator': originator, 'body': body, 'recipients': recipients})
        return Message().load(self.request('messages', 'POST', params))

Response: Create a new message.


Instruction:
Write a Python 3 function for
creating a new voice message.

Response:
def voice_message_create(self, recipients, body, params=None):
        """Create a new voice message."""
        if params is None: params = {}
        if type(recipients) == list:
            recipients = ','.join(recipients)

        params.update({'recipients': recipients, 'body': body})
        return VoiceMessage().load(self.request('voicemessages', 'POST', params))


Instruction:
Explain what the following Python 3 code does
def lookup(self, phonenumber, params=None):
        if params is None: params = {}
        return Lookup().load(self.request('lookup/' + str(phonenumber), 'GET', params))

Response: Do a new lookup.


Instruction:
Explain what the following Python 3 code does
def lookup_hlr(self, phonenumber, params=None):
        if params is None: params = {}
        return HLR().load(self.request('lookup/' + str(phonenumber) + '/hlr', 'GET', params))

Response: Retrieve the information of a specific HLR lookup.


Instruction:
How would you code a function in Python 3 to
perform a new HLR lookup.

Response:
def lookup_hlr_create(self, phonenumber, params=None):
        """Perform a new HLR lookup."""
        if params is None: params = {}
        return HLR().load(self.request('lookup/' + str(phonenumber) + '/hlr', 'POST', params))


Instruction:
Can you tell what is the following Python 3 function doing
def verify_create(self, recipient, params=None):
        if params is None: params = {}
        params.update({'recipient': recipient})
        return Verify().load(self.request('verify', 'POST', params))

Response: Create a new verification.


Instruction:
Implement a Python 3 function for
verifying the token of a specific verification.

Response:
def verify_verify(self, id, token):
        """Verify the token of a specific verification."""
        return Verify().load(self.request('verify/' + str(id), params={'token': token}))


Instruction:
Write a Python 3 function that can
create typed objects from the dicts.

Response:
def items(self, value):
        """Create typed objects from the dicts."""
        items = []
        for item in value:
            items.append(self.itemType().load(item))

        self._items = items


Instruction:
Can you create a Python 3 function that
builds a request and gets a response.

Response:
def request(self, path, method='GET', params=None):
        """Builds a request and gets a response."""
        if params is None: params = {}
        url = urljoin(self.endpoint, path)

        headers = {
            'Accept': 'application/json',
            'Authorization': 'AccessKey ' + self.access_key,
            'User-Agent': self.user_agent,
            'Content-Type': 'application/json'
        }

        if method == 'DELETE':
            response = requests.delete(url, verify=True, headers=headers, data=json.dumps(params))
        elif method == 'GET':
            response = requests.get(url, verify=True, headers=headers, params=params)
        elif method == 'PATCH':
            response = requests.patch(url, verify=True, headers=headers, data=json.dumps(params))
        elif method == 'POST':
            response = requests.post(url, verify=True, headers=headers, data=json.dumps(params))
        elif method == 'PUT':
            response = requests.put(url, verify=True, headers=headers, data=json.dumps(params))
        else:
            raise ValueError(str(method) + ' is not a supported HTTP method')

        if response.status_code in self.__supported_status_codes:
            response_text = response.text
        else:
            response.raise_for_status()

        return response_text


Instruction:
Explain what the following Python 3 code does
def run(self):
        config_h = opj("src", "cysignals", "cysignals_config.h")
        if not os.path.isfile(config_h):
            import subprocess
            subprocess.check_call(["make", "configure"])
            subprocess.check_call(["sh", "configure"])

        dist = self.distribution
        ext_modules = dist.ext_modules
        if ext_modules:
            dist.ext_modules[:] = self.cythonize(ext_modules)

        _build.run(self)

Response: Run ``./configure`` and Cython first.


Instruction:
Write a Python 3 function for
returning a list of all cython extra debug information files.

Response:
def cython_debug_files():
    """
    Cython extra debug information files
    """
    # Search all subdirectories of sys.path directories for a
    # "cython_debug" directory. Note that sys_path is a variable set by
    # cysignals-CSI. It may differ from sys.path if GDB is run with a
    # different Python interpreter.
    files = []
    for path in sys_path:  # noqa
        pattern = os.path.join(path, '*', 'cython_debug', 'cython_debug_info_*')
        files.extend(glob.glob(pattern))
    return files


Instruction:
How would you implement a function in Python 3 that
retrieves a new frame from the PhoXi and convert it to a ColorImage DepthImage and IrImage.

Response:
def frames(self):
        """Retrieve a new frame from the PhoXi and convert it to a ColorImage,
        a DepthImage, and an IrImage.

        Returns
        -------
        :obj:`tuple` of :obj:`ColorImage`, :obj:`DepthImage`, :obj:`IrImage`, :obj:`numpy.ndarray`
            The ColorImage, DepthImage, and IrImage of the current frame.
        """
        _, phoxi_depth_im, _ = self._phoxi.frames()
        webcam_color_im, _, _ = self._webcam.frames(most_recent=True)

        # Colorize PhoXi Image
        phoxi_color_im = self._colorize(phoxi_depth_im, webcam_color_im)
        return phoxi_color_im, phoxi_depth_im, None


Instruction:
Implement a Python 3 function for
collecting a series of depth images and return the median of the set.

Response:
def median_depth_img(self, num_img=1, fill_depth=0.0):
        """Collect a series of depth images and return the median of the set.

        Parameters
        ----------
        num_img : int
            The number of consecutive frames to process.

        Returns
        -------
        DepthImage
            The median DepthImage collected from the frames.
        """
        depths = []

        for _ in range(num_img):
            _, depth, _ = self.frames()
            depths.append(depth)

        median_depth = Image.median_images(depths)
        median_depth.data[median_depth.data == 0.0] = fill_depth
        return median_depth


Instruction:
How would you explain what the following Python 3 function does
def sensor(sensor_type, cfg):
        sensor_type = sensor_type.lower()
        if sensor_type == 'kinect2':
            s = Kinect2Sensor(packet_pipeline_mode=cfg['pipeline_mode'],
                              device_num=cfg['device_num'],
                              frame=cfg['frame'])
        elif sensor_type == 'bridged_kinect2':
            s = KinectSensorBridged(quality=cfg['quality'], frame=cfg['frame'])
        elif sensor_type == 'primesense':
            flip_images = True
            if 'flip_images' in cfg.keys():
                flip_images = cfg['flip_images']
            s = PrimesenseSensor(auto_white_balance=cfg['auto_white_balance'],
                                 flip_images=flip_images,
                                 frame=cfg['frame'])
        elif sensor_type == 'virtual':
            s = VirtualSensor(cfg['image_dir'],
                              frame=cfg['frame'])
        elif sensor_type == 'tensor_dataset':
            s = TensorDatasetVirtualSensor(cfg['dataset_dir'],
                                           frame=cfg['frame'])
        elif sensor_type == 'primesense_ros':
            s = PrimesenseSensor_ROS(frame=cfg['frame'])
        elif sensor_type == 'ensenso':
            s = EnsensoSensor(frame=cfg['frame'])
        elif sensor_type == 'phoxi':
            s = PhoXiSensor(frame=cfg['frame'],
                            device_name=cfg['device_name'],
                            size=cfg['size'])
        elif sensor_type == 'webcam':
            s = WebcamSensor(frame=cfg['frame'],
                             device_id=cfg['device_id'])
        elif sensor_type == 'colorized_phoxi':
            s = ColorizedPhoXiSensor(frame=cfg['frame'], phoxi_config=cfg['phoxi_config'],
                                     webcam_config=cfg['webcam_config'], calib_dir=cfg['calib_dir'])
        elif sensor_type == 'realsense':
            s = RealSenseSensor(
                cam_id=cfg['cam_id'],
                filter_depth=cfg['filter_depth'],
                frame=cfg['frame'],
            )
        else:
            raise ValueError('RGBD sensor type %s not supported' %(sensor_type))
        return s

Response: Creates a new sensor object of the specified type.


Instruction:
Create a Python 3 function for
getting the index of a point in an array

Response:
def get_point_index(point, all_points, eps = 1e-4):
        """ Get the index of a point in an array """
        inds = np.where(np.linalg.norm(point - all_points, axis=1) < eps)
        if inds[0].shape[0] == 0:
            return -1
        return inds[0][0]


Instruction:
Write a Python 3 script for
matching features between two graspable objects based on a full distance matrix.

Response:
def match(self, source_obj_features, target_obj_features):
        """
        Matches features between two graspable objects based on a full distance matrix.

        Parameters
        ----------
        source_obj_features : :obj:`BagOfFeatures`
            bag of the source objects features
        target_obj_features : :obj:`BagOfFeatures`
            bag of the target objects features

        Returns
        -------
        corrs : :obj:`Correspondences`
            the correspondences between source and target
        """
        if not isinstance(source_obj_features, f.BagOfFeatures):
            raise ValueError('Must supply source bag of object features')
        if not isinstance(target_obj_features, f.BagOfFeatures):
            raise ValueError('Must supply target bag of object features')

        # source feature descriptors and keypoints
        source_descriptors = source_obj_features.descriptors
        target_descriptors = target_obj_features.descriptors
        source_keypoints = source_obj_features.keypoints
        target_keypoints = target_obj_features.keypoints

        #calculate distance between this model's descriptors and each of the other_model's descriptors
        dists = spatial.distance.cdist(source_descriptors, target_descriptors)

        #calculate the indices of the target_model that minimize the distance to the descriptors in this model
        source_closest_descriptors = dists.argmin(axis=1)
        target_closest_descriptors = dists.argmin(axis=0)
        match_indices = []
        source_matched_points = np.zeros((0,3))
        target_matched_points = np.zeros((0,3))

        #calculate which points/indices the closest descriptors correspond to
        for i, j in enumerate(source_closest_descriptors):
            # for now, only keep correspondences that are a 2-way match
            if target_closest_descriptors[j] == i:
                match_indices.append(j)
                source_matched_points = np.r_[source_matched_points, source_keypoints[i:i+1, :]]
                target_matched_points = np.r_[target_matched_points, target_keypoints[j:j+1, :]]
            else:
                match_indices.append(-1)

        return Correspondences(match_indices, source_matched_points, target_matched_points)


Instruction:
Explain what the following Python 3 code does
def match(self, source_points, target_points, source_normals, target_normals):
        # compute the distances and inner products between the point sets
        dists = ssd.cdist(source_points, target_points, 'euclidean')
        ip = source_normals.dot(target_normals.T) # abs because we don't have correct orientations
        source_ip = source_points.dot(target_normals.T)
        target_ip = target_points.dot(target_normals.T)
        target_ip = np.diag(target_ip)
        target_ip = np.tile(target_ip, [source_points.shape[0], 1])
        abs_diff = np.abs(source_ip - target_ip) # difference in inner products

        # mark invalid correspondences
        invalid_dists = np.where(dists > self.dist_thresh_)
        abs_diff[invalid_dists[0], invalid_dists[1]] = np.inf
        invalid_norms = np.where(ip < self.norm_thresh_)
        abs_diff[invalid_norms[0], invalid_norms[1]] = np.inf

        # choose the closest matches
        match_indices = np.argmin(abs_diff, axis=1)
        match_vals = np.min(abs_diff, axis=1)
        invalid_matches = np.where(match_vals == np.inf)
        match_indices[invalid_matches[0]] = -1

        return NormalCorrespondences(match_indices, source_points, target_points, source_normals, target_normals)

Response: Matches points between two point - normal sets. Uses the closest ip to choose matches with thresholding only.


Instruction:
Here you have a function in Python 3, explain what it does
def _config_pipe(self):
        self._cfg.enable_device(self.id)

        # configure the color stream
        self._cfg.enable_stream(
            rs.stream.color,
            RealSenseSensor.COLOR_IM_WIDTH,
            RealSenseSensor.COLOR_IM_HEIGHT,
            rs.format.bgr8,
            RealSenseSensor.FPS
        )

        # configure the depth stream
        self._cfg.enable_stream(
            rs.stream.depth,
            RealSenseSensor.DEPTH_IM_WIDTH,
            360 if self._depth_align else RealSenseSensor.DEPTH_IM_HEIGHT,
            rs.format.z16,
            RealSenseSensor.FPS
        )

Response: Configures the pipeline to stream color and depth.


Instruction:
Explain what the following Python 3 code does
def _set_depth_scale(self):
        sensor = self._profile.get_device().first_depth_sensor()
        self._depth_scale = sensor.get_depth_scale()

Response: Retrieve the scale of the depth sensor.


Instruction:
Make a summary of the following Python 3 code
def _set_intrinsics(self):
        strm = self._profile.get_stream(rs.stream.color)
        obj = strm.as_video_stream_profile().get_intrinsics()
        self._intrinsics[0, 0] = obj.fx
        self._intrinsics[1, 1] = obj.fy
        self._intrinsics[0, 2] = obj.ppx
        self._intrinsics[1, 2] = obj.ppy

Response: Read the intrinsics matrix from the stream.


Instruction:
Can you generate the documentation for the following Python 3 function
def color_intrinsics(self):
        return CameraIntrinsics(
            self._frame,
            self._intrinsics[0, 0],
            self._intrinsics[1, 1],
            self._intrinsics[0, 2],
            self._intrinsics[1, 2],
            height=RealSenseSensor.COLOR_IM_HEIGHT,
            width=RealSenseSensor.COLOR_IM_WIDTH,
        )

Response: Return a new CameraIntrinsics object for the RealSense color camera.


Instruction:
Can you generate the documentation for the following Python 3 function
def _read_color_and_depth_image(self):
        frames = self._pipe.wait_for_frames()
        if self._depth_align:
            frames = self._align.process(frames)

        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()

        if not depth_frame or not color_frame:
            logging.warning('Could not retrieve frames.')
            return None, None

        if self._filter_depth:
            depth_frame = self._filter_depth_frame(depth_frame)

        # convert to numpy arrays
        depth_image = self._to_numpy(depth_frame, np.float32)
        color_image = self._to_numpy(color_frame, np.uint8)

        # convert depth to meters
        depth_image *= self._depth_scale

        # bgr to rgb
        color_image = color_image[..., ::-1]

        depth = DepthImage(depth_image, frame=self._frame)
        color = ColorImage(color_image, frame=self._frame)
        return color, depth

Response: Read a color and depth image from the device.


Instruction:
Write a Python 3 function that can
start the webcam sensor.

Response:
def start(self):
        """Start the sensor.
        """
        self._cap = cv2.VideoCapture(self._device_id + cv2.CAP_V4L2)
        if not self._cap.isOpened():
            self._running = False
            self._cap.release()
            self._cap = None
            return False

        self._cap.set(cv2.CAP_PROP_FRAME_WIDTH, self._camera_intr.width)
        self._cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self._camera_intr.height)
        self._running = True

        # Capture 5 frames to flush webcam sensor
        for _ in range(5):
            _ = self.frames()

        return True


Instruction:
Write a Python 3 script to
retrieve a new frame from the PhoXi and convert it to a ColorImage DepthImage and IrImage.

Response:
def frames(self, most_recent=False):
        """Retrieve a new frame from the PhoXi and convert it to a ColorImage,
        a DepthImage, and an IrImage.

        Parameters
        ----------
        most_recent: bool
            If true, the OpenCV buffer is emptied for the webcam before reading the most recent frame.

        Returns
        -------
        :obj:`tuple` of :obj:`ColorImage`, :obj:`DepthImage`, :obj:`IrImage`, :obj:`numpy.ndarray`
            The ColorImage, DepthImage, and IrImage of the current frame.
        """
        if most_recent:
            for i in xrange(4):
                self._cap.grab()
        for i in range(1):
            if self._adjust_exposure:
                try:
                    command = 'v4l2-ctl -d /dev/video{} -c exposure_auto=1 -c exposure_auto_priority=0 -c exposure_absolute=100 -c saturation=60 -c gain=140'.format(self._device_id)
                    FNULL = open(os.devnull, 'w')
                    subprocess.call(shlex.split(command), stdout=FNULL, stderr=subprocess.STDOUT)
                except:
                    pass
            ret, frame = self._cap.read()
        rgb_data = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        return ColorImage(rgb_data, frame=self._frame), None, None


Instruction:
How would you code a function in Python 3 to
set the buffer formatting.

Response:
def _set_format(self, msg):
        """ Set the buffer formatting. """
        num_points = msg.height * msg.width
        self._format = '<' + num_points * 'ffff'


Instruction:
Write a Python 3 function that can
set the camera intrinsics from an info msg.

Response:
def _set_camera_properties(self, msg):
        """ Set the camera intrinsics from an info msg. """
        focal_x = msg.K[0]
        focal_y = msg.K[4]
        center_x = msg.K[2]
        center_y = msg.K[5]
        im_height = msg.height
        im_width = msg.width
        self._camera_intr = CameraIntrinsics(self._frame, focal_x, focal_y,
                                             center_x, center_y,
                                             height=im_height,
                                             width=im_width)


Instruction:
Can you generate the documentation for the following Python 3 function
def _depth_im_from_pointcloud(self, msg):
        # set format
        if self._format is None:
            self._set_format(msg)

        # rescale camera intr in case binning is turned on
        if msg.height != self._camera_intr.height:
            rescale_factor = float(msg.height) / self._camera_intr.height
            self._camera_intr = self._camera_intr.resize(rescale_factor)

        # read num points
        num_points = msg.height * msg.width

        # read buffer
        raw_tup = struct.Struct(self._format).unpack_from(msg.data, 0)
        raw_arr = np.array(raw_tup)

        # subsample depth values and reshape
        depth_ind = 2 + 4 * np.arange(num_points)
        depth_buf = raw_arr[depth_ind]
        depth_arr = depth_buf.reshape(msg.height, msg.width)
        depth_im = DepthImage(depth_arr, frame=self._frame)

        return depth_im

Response: Convert a pointcloud2 message to a depth image.


Instruction:
Make a summary of the following Python 3 code
def frames(self):
        # wait for a new image
        while self._cur_depth_im is None:
            time.sleep(0.01)

        # read next image
        depth_im = self._cur_depth_im
        color_im = ColorImage(np.zeros([depth_im.height,
                                        depth_im.width,
                                        3]).astype(np.uint8), frame=self._frame)
        self._cur_depth_im = None
        return color_im, depth_im, None

Response: Retrieve a new frame from the Ensenso and convert it to a ColorImage DepthImage and IrImage.


Instruction:
Given the following Python 3 function, write the documentation
def register(self, source_point_cloud, target_point_cloud,
                 source_normal_cloud, target_normal_cloud, matcher,
                 num_iterations=1, compute_total_cost=True, match_centroids=False,
                 vis=False):
        """
        Iteratively register objects to one another using a modified version of point to plane ICP.
        The cost func is PointToPlane_COST + gamma * PointToPoint_COST.
        Uses a `stochastic Gauss-Newton step` where on each iteration a smaller number of points is sampled.

        Parameters
        ----------
        source_point_cloud : :obj:`autolab_core.PointCloud`
            source object points
        target_point_cloud : :obj`autolab_core.PointCloud`
            target object points
        source_normal_cloud : :obj:`autolab_core.NormalCloud`
            source object outward-pointing normals
        target_normal_cloud : :obj:`autolab_core.NormalCloud`
            target object outward-pointing normals
        matcher : :obj:`PointToPlaneFeatureMatcher`
            object to match the point sets
        num_iterations : int
            the number of iterations to run
        compute_total_cost : bool
            whether or not to compute the total cost upon termination.
        match_centroids : bool
            whether or not to match the centroids of the point clouds

        Returns
        -------
        :obj`RegistrationResult`
            results containing source to target transformation and cost
        """
        # check valid data
        if not isinstance(source_point_cloud, PointCloud) or not isinstance(target_point_cloud, PointCloud):
            raise ValueError('Source and target point clouds must be PointCloud objects')
        if not isinstance(source_normal_cloud, NormalCloud) or not isinstance(target_normal_cloud, NormalCloud):
            raise ValueError('Source and target normal clouds must be NormalCloud objects')
        if not isinstance(matcher, PointToPlaneFeatureMatcher):
            raise ValueError('Feature matcher must be a PointToPlaneFeatureMatcher object')
        if source_point_cloud.num_points != source_normal_cloud.num_points or target_point_cloud.num_points != target_normal_cloud.num_points:
            raise ValueError('Input point clouds must have the same number of points as corresponding normal cloud')

        # extract source and target point and normal data arrays
        orig_source_points = source_point_cloud.data.T
        orig_target_points = target_point_cloud.data.T
        orig_source_normals = source_normal_cloud.data.T
        orig_target_normals = target_normal_cloud.data.T

        # setup the problem
        normal_norms = np.linalg.norm(orig_target_normals, axis=1)
        valid_inds = np.nonzero(normal_norms)
        orig_target_points = orig_target_points[valid_inds[0],:]
        orig_target_normals = orig_target_normals[valid_inds[0],:]

        normal_norms = np.linalg.norm(orig_source_normals, axis=1)
        valid_inds = np.nonzero(normal_norms)
        orig_source_points = orig_source_points[valid_inds[0],:]
        orig_source_normals = orig_source_normals[valid_inds[0],:]

        # alloc buffers for solutions
        source_mean_point = np.mean(orig_source_points, axis=0)
        target_mean_point = np.mean(orig_target_points, axis=0)
        R_sol = np.eye(3)
        t_sol = np.zeros([3, 1]) #init with diff between means
        if match_centroids:
            t_sol[:,0] = target_mean_point - source_mean_point

        # iterate through
        for i in range(num_iterations):
            logging.info('Point to plane ICP iteration %d' %(i))

            # subsample points
            source_subsample_inds = np.random.choice(orig_source_points.shape[0], size=self.sample_size_)
            source_points = orig_source_points[source_subsample_inds,:]
            source_normals = orig_source_normals[source_subsample_inds,:]
            target_subsample_inds = np.random.choice(orig_target_points.shape[0], size=self.sample_size_)
            target_points = orig_target_points[target_subsample_inds,:]
            target_normals = orig_target_normals[target_subsample_inds,:]

            # transform source points
            source_points = (R_sol.dot(source_points.T) + np.tile(t_sol, [1, source_points.shape[0]])).T
            source_normals = (R_sol.dot(source_normals.T)).T

            # closest points
            corrs = matcher.match(source_points, target_points, source_normals, target_normals)

            # solve optimal rotation + translation
            valid_corrs = np.where(corrs.index_map != -1)[0]
            source_corr_points = corrs.source_points[valid_corrs,:]
            target_corr_points = corrs.target_points[corrs.index_map[valid_corrs], :]
            target_corr_normals = corrs.target_normals[corrs.index_map[valid_corrs], :]

            num_corrs = valid_corrs.shape[0]
            if num_corrs == 0:
                logging.warning('No correspondences found')
                break

            # create A and b matrices for Gauss-Newton step on joint cost function
            A = np.zeros([6,6])
            b = np.zeros([6,1])
            Ap = np.zeros([6,6])
            bp = np.zeros([6,1])
            G = np.zeros([3,6])
            G[:,3:] = np.eye(3)

            for i in range(num_corrs):
                s = source_corr_points[i:i+1,:].T
                t = target_corr_points[i:i+1,:].T
                n = target_corr_normals[i:i+1,:].T
                G[:,:3] = skew(s).T
                A += G.T.dot(n).dot(n.T).dot(G)
                b += G.T.dot(n).dot(n.T).dot(t - s)

                Ap += G.T.dot(G)
                bp += G.T.dot(t - s)
            v = np.linalg.solve(A + self.gamma_*Ap + self.mu_*np.eye(6),
                                b + self.gamma_*bp)

            # create pose values from the solution
            R = np.eye(3)
            R = R + skew(v[:3])
            U, S, V = np.linalg.svd(R)
            R = U.dot(V)
            t = v[3:]

            # incrementally update the final transform
            R_sol = R.dot(R_sol)
            t_sol = R.dot(t_sol) + t

        T_source_target = RigidTransform(R_sol, t_sol, from_frame=source_point_cloud.frame, to_frame=target_point_cloud.frame)

        total_cost = 0
        source_points = (R_sol.dot(orig_source_points.T) + np.tile(t_sol, [1, orig_source_points.shape[0]])).T
        source_normals = (R_sol.dot(orig_source_normals.T)).T

        if compute_total_cost:
            # rematch all points to get the final cost
            corrs = matcher.match(source_points, orig_target_points, source_normals, orig_target_normals)
            valid_corrs = np.where(corrs.index_map != -1)[0]
            num_corrs = valid_corrs.shape[0]
            if num_corrs == 0:
                return RegistrationResult(T_source_target, np.inf)

            # get the corresponding points
            source_corr_points = corrs.source_points[valid_corrs,:]
            target_corr_points = corrs.target_points[corrs.index_map[valid_corrs], :]
            target_corr_normals = corrs.target_normals[corrs.index_map[valid_corrs], :]

            # determine total cost
            source_target_alignment = np.diag((source_corr_points - target_corr_points).dot(target_corr_normals.T))
            point_plane_cost = (1.0 / num_corrs) * np.sum(source_target_alignment * source_target_alignment)
            point_dist_cost = (1.0 / num_corrs) * np.sum(np.linalg.norm(source_corr_points - target_corr_points, axis=1)**2)
            total_cost = point_plane_cost + self.gamma_ * point_dist_cost

        return RegistrationResult(T_source_target, total_cost)

Response: This method registers the source and target point clouds with the target point clouds.


Instruction:
Can you create a Python 3 function that
retrieves the next frame from the image directory and convert it to a ColorImage DepthImage and IrImage.

Response:
def frames(self):
        """Retrieve the next frame from the image directory and convert it to a ColorImage,
        a DepthImage, and an IrImage.

        Parameters
        ----------
        skip_registration : bool
            If True, the registration step is skipped.

        Returns
        -------
        :obj:`tuple` of :obj:`ColorImage`, :obj:`DepthImage`, :obj:`IrImage`, :obj:`numpy.ndarray`
            The ColorImage, DepthImage, and IrImage of the current frame.

        Raises
        ------
        RuntimeError
            If the stream is not running or if all images in the
            directory have been used.
        """
        if not self._running:
            raise RuntimeError('Device pointing to %s not runnning. Cannot read frames' %(self._path_to_images))

        if self._im_index >= self._num_images:
            raise RuntimeError('Device is out of images')

        # read images
        color_filename = os.path.join(self._path_to_images, 'color_%d%s' %(self._im_index, self._color_ext))
        color_im = ColorImage.open(color_filename, frame=self._frame)
        depth_filename = os.path.join(self._path_to_images, 'depth_%d.npy' %(self._im_index))
        depth_im = DepthImage.open(depth_filename, frame=self._frame)
        self._im_index = (self._im_index + 1) % self._num_images
        return color_im, depth_im, None


Instruction:
Create a Python 3 function to
collect a series of depth images and return the median of the set.

Response:
def median_depth_img(self, num_img=1):
        """Collect a series of depth images and return the median of the set.

        Parameters
        ----------
        num_img : int
            The number of consecutive frames to process.

        Returns
        -------
        :obj:`DepthImage`
            The median DepthImage collected from the frames.
        """
        depths = []

        for _ in range(num_img):
            _, depth, _ = self.frames()
            depths.append(depth)

        return Image.median_images(depths)


Instruction:
Can you implement a function in Python 3 that
retrieves the next frame from the tensor dataset and convert it to a ColorImage DepthImage and IrImage.

Response:
def frames(self):
        """Retrieve the next frame from the tensor dataset and convert it to a ColorImage,
        a DepthImage, and an IrImage.

        Parameters
        ----------
        skip_registration : bool
            If True, the registration step is skipped.

        Returns
        -------
        :obj:`tuple` of :obj:`ColorImage`, :obj:`DepthImage`, :obj:`IrImage`, :obj:`numpy.ndarray`
            The ColorImage, DepthImage, and IrImage of the current frame.

        Raises
        ------
        RuntimeError
            If the stream is not running or if all images in the
            directory have been used.
        """
        if not self._running:
            raise RuntimeError('Device pointing to %s not runnning. Cannot read frames' %(self._path_to_images))

        if self._im_index >= self._num_images:
            raise RuntimeError('Device is out of images')

        # read images
        datapoint = self._dataset.datapoint(self._im_index,
                                            TensorDatasetVirtualSensor.IMAGE_FIELDS)
        color_im = ColorImage(datapoint[TensorDatasetVirtualSensor.COLOR_IM_FIELD],
                              frame=self._frame)
        depth_im = DepthImage(datapoint[TensorDatasetVirtualSensor.DEPTH_IM_FIELD],
                              frame=self._frame)
        if self._image_rescale_factor != 1.0:
            color_im = color_im.resize(self._image_rescale_factor)
            depth_im = depth_im.resize(self._image_rescale_factor, interp='nearest')
        self._im_index = (self._im_index + 1) % self._num_images
        return color_im, depth_im, None


Instruction:
Can you write a function in Python 3 where it
returns a new CameraIntrinsics object for the primesense color camera.

Response:
def color_intrinsics(self):
        """:obj:`CameraIntrinsics` : The camera intrinsics for the primesense color camera.
        """
        return CameraIntrinsics(self._ir_frame, PrimesenseSensor.FOCAL_X, PrimesenseSensor.FOCAL_Y,
                                PrimesenseSensor.CENTER_X, PrimesenseSensor.CENTER_Y,
                                height=PrimesenseSensor.DEPTH_IM_HEIGHT,
                                width=PrimesenseSensor.DEPTH_IM_WIDTH)


Instruction:
Can you create a Python 3 function that
reads a depth image from the device

Response:
def _read_depth_image(self):
        """ Reads a depth image from the device """
        # read raw uint16 buffer
        im_arr = self._depth_stream.read_frame()
        raw_buf = im_arr.get_buffer_as_uint16()
        buf_array = np.array([raw_buf[i] for i in range(PrimesenseSensor.DEPTH_IM_WIDTH * PrimesenseSensor.DEPTH_IM_HEIGHT)])

        # convert to image in meters
        depth_image = buf_array.reshape(PrimesenseSensor.DEPTH_IM_HEIGHT,
                                        PrimesenseSensor.DEPTH_IM_WIDTH)
        depth_image = depth_image * MM_TO_METERS # convert to meters
        if self._flip_images:
            depth_image = np.flipud(depth_image)
        else:
            depth_image = np.fliplr(depth_image)
        return DepthImage(depth_image, frame=self._frame)


Instruction:
Here you have a function in Python 3, explain what it does
def _read_color_image(self):
        # read raw buffer
        im_arr = self._color_stream.read_frame()
        raw_buf = im_arr.get_buffer_as_triplet()
        r_array = np.array([raw_buf[i][0] for i in range(PrimesenseSensor.COLOR_IM_WIDTH * PrimesenseSensor.COLOR_IM_HEIGHT)])
        g_array = np.array([raw_buf[i][1] for i in range(PrimesenseSensor.COLOR_IM_WIDTH * PrimesenseSensor.COLOR_IM_HEIGHT)])
        b_array = np.array([raw_buf[i][2] for i in range(PrimesenseSensor.COLOR_IM_WIDTH * PrimesenseSensor.COLOR_IM_HEIGHT)])

        # convert to uint8 image
        color_image = np.zeros([PrimesenseSensor.COLOR_IM_HEIGHT, PrimesenseSensor.COLOR_IM_WIDTH, 3])
        color_image[:,:,0] = r_array.reshape(PrimesenseSensor.COLOR_IM_HEIGHT,
                                             PrimesenseSensor.COLOR_IM_WIDTH)
        color_image[:,:,1] = g_array.reshape(PrimesenseSensor.COLOR_IM_HEIGHT,
                                             PrimesenseSensor.COLOR_IM_WIDTH)
        color_image[:,:,2] = b_array.reshape(PrimesenseSensor.COLOR_IM_HEIGHT,
                                             PrimesenseSensor.COLOR_IM_WIDTH)
        if self._flip_images:
            color_image = np.flipud(color_image.astype(np.uint8))
        else:
            color_image = np.fliplr(color_image.astype(np.uint8))
        return ColorImage(color_image, frame=self._frame)

Response: Reads a color image from the device


Instruction:
Make a summary of the following Python 3 code
def frames(self):
        color_im = self._read_color_image()
        depth_im = self._read_depth_image()
        return color_im, depth_im, None

Response: Retrieve a new frame from the Kinect and convert it to a ColorImage DepthImage and IrImage.


Instruction:
Can you create a Python 3 function that
collects a series of depth images and return the min of the set.

Response:
def min_depth_img(self, num_img=1):
        """Collect a series of depth images and return the min of the set.

        Parameters
        ----------
        num_img : int
            The number of consecutive frames to process.

        Returns
        -------
        :obj:`DepthImage`
            The min DepthImage collected from the frames.
        """
        depths = []

        for _ in range(num_img):
            _, depth, _ = self.frames()
            depths.append(depth)

        return Image.min_images(depths)


Instruction:
Make a summary of the following Python 3 code
def _ros_read_images(self, stream_buffer, number, staleness_limit = 10.):

        rospy.wait_for_service(stream_buffer, timeout = self.timeout)
        ros_image_buffer = rospy.ServiceProxy(stream_buffer, ImageBuffer)
        ret = ros_image_buffer(number, 1)
        if not staleness_limit == None:
            if ret.timestamps[-1] > staleness_limit:
                raise RuntimeError("Got data {0} seconds old, more than allowed {1} seconds"
                                   .format(ret.timestamps[-1], staleness_limit))

        data = ret.data.reshape(ret.data_dim1, ret.data_dim2, ret.data_dim3).astype(ret.dtype)

        # Special handling for 1 element, since dstack's behavior is different
        if number == 1:
            return [data]
        return np.dsplit(data, number)

Response: Reads images from a stream buffer.


Instruction:
How would you code a function in Python 3 to
read the depth images from the device

Response:
def _read_depth_images(self, num_images):
        """ Reads depth images from the device """
        depth_images = self._ros_read_images(self._depth_image_buffer, num_images, self.staleness_limit)
        for i in range(0, num_images):
            depth_images[i] = depth_images[i] * MM_TO_METERS # convert to meters
            if self._flip_images:
                depth_images[i] = np.flipud(depth_images[i])
                depth_images[i] = np.fliplr(depth_images[i])
            depth_images[i] = DepthImage(depth_images[i], frame=self._frame)
        return depth_images


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _read_color_images(self, num_images):
        color_images = self._ros_read_images(self._color_image_buffer, num_images, self.staleness_limit)
        for i in range(0, num_images):
            if self._flip_images:
                color_images[i] = np.flipud(color_images[i].astype(np.uint8))
                color_images[i] = np.fliplr(color_images[i].astype(np.uint8))
            color_images[i] = ColorImage(color_images[i], frame=self._frame)
        return color_images

Response: Reads color images from the device


Instruction:
Can you generate a brief explanation for the following Python 3 code
def median_depth_img(self, num_img=1, fill_depth=0.0):
        depths = self._read_depth_images(num_img)

        median_depth = Image.median_images(depths)
        median_depth.data[median_depth.data == 0.0] = fill_depth
        return median_depth

Response: Collect a series of depth images and return the median of the set.


Instruction:
Can you create a Python 3 function that
collects a series of depth images and return the min of the set.

Response:
def min_depth_img(self, num_img=1):
        """Collect a series of depth images and return the min of the set.

        Parameters
        ----------
        num_img : int
            The number of consecutive frames to process.

        Returns
        -------
        :obj:`DepthImage`
            The min DepthImage collected from the frames.
        """
        depths = self._read_depth_images(num_img)

        return Image.min_images(depths)


Instruction:
Can you tell what is the following Python 3 function doing
def T_obj_camera(self):
        if self.stable_pose is None:
            T_obj_world = RigidTransform(from_frame='obj', to_frame='world')
        else:
            T_obj_world = self.stable_pose.T_obj_table.as_frames('obj', 'world')
        T_camera_obj = T_obj_world.inverse() * self.T_camera_world
        return T_camera_obj

Response: Returns the transformation from camera to object when the object is in the given stable pose.


Instruction:
How would you explain what the following Python 3 function does
def image(self, render_mode):
        if render_mode == RenderMode.COLOR:
            return self.color_im
        elif render_mode == RenderMode.DEPTH:
            return self.depth_im
        elif render_mode == RenderMode.SEGMASK:
            return self.binary_im
        else:
            return None

Response: Returns an image generated with a particular render mode.


Instruction:
Can you implement a function in Python 3 that
registers a camera to a chessboard.

Response:
def register(sensor, config):
        """
        Registers a camera to a chessboard.

        Parameters
        ----------
        sensor : :obj:`perception.RgbdSensor`
            the sensor to register
        config : :obj:`autolab_core.YamlConfig` or :obj:`dict`
            configuration file for registration

        Returns
        -------
        :obj:`ChessboardRegistrationResult`
            the result of registration

        Notes
        -----
        The config must have the parameters specified in the Other Parameters section.

        Other Parameters
        ----------------
        num_transform_avg : int
            the number of independent registrations to average together
        num_images : int
            the number of images to read for each independent registration
        corners_x : int
            the number of chessboard corners in the x-direction
        corners_y : int
            the number of chessboard corners in the y-direction
        color_image_rescale_factor : float
            amount to rescale the color image for detection (numbers around 4-8 are useful)
        vis : bool
            whether or not to visualize the registration
        """
        # read config
        num_transform_avg = config['num_transform_avg']
        num_images = config['num_images']
        sx = config['corners_x']
        sy = config['corners_y']
        point_order = config['point_order']
        color_image_rescale_factor = config['color_image_rescale_factor']
        flip_normal = config['flip_normal']
        y_points_left = False
        if 'y_points_left' in config.keys() and sx == sy:
            y_points_left = config['y_points_left']
            num_images = 1
        vis = config['vis']

        # read params from sensor
        logging.info('Registering camera %s' %(sensor.frame))
        ir_intrinsics = sensor.ir_intrinsics

        # repeat registration multiple times and average results
        R = np.zeros([3,3])
        t = np.zeros([3,1])
        points_3d_plane = PointCloud(np.zeros([3, sx*sy]), frame=sensor.ir_frame)

        k = 0
        while k < num_transform_avg:
            # average a bunch of depth images together
            depth_ims = None
            for i in range(num_images):
                start = time.time()
                small_color_im, new_depth_im, _ = sensor.frames()
                end = time.time()
                logging.info('Frames Runtime: %.3f' %(end-start))
                if depth_ims is None:
                    depth_ims = np.zeros([new_depth_im.height,
                                          new_depth_im.width,
                                          num_images])
                depth_ims[:,:,i] = new_depth_im.data

            med_depth_im = np.median(depth_ims, axis=2)
            depth_im = DepthImage(med_depth_im, sensor.ir_frame)

            # find the corner pixels in an upsampled version of the color image
            big_color_im = small_color_im.resize(color_image_rescale_factor)
            corner_px = big_color_im.find_chessboard(sx=sx, sy=sy)

            if vis:
                plt.figure()
                plt.imshow(big_color_im.data)
                for i in range(sx):
                    plt.scatter(corner_px[i,0], corner_px[i,1], s=25, c='b')
                plt.show()

            if corner_px is None:
                logging.error('No chessboard detected! Check camera exposure settings')
                continue

            # convert back to original image
            small_corner_px = corner_px / color_image_rescale_factor

            if vis:
                plt.figure()
                plt.imshow(small_color_im.data)
                for i in range(sx):
                    plt.scatter(small_corner_px[i,0], small_corner_px[i,1], s=25, c='b')
                plt.axis('off')
                plt.show()

            # project points into 3D
            camera_intr = sensor.ir_intrinsics
            points_3d = camera_intr.deproject(depth_im)

            # get round chessboard ind
            corner_px_round = np.round(small_corner_px).astype(np.uint16)
            corner_ind = depth_im.ij_to_linear(corner_px_round[:,0], corner_px_round[:,1])
            if corner_ind.shape[0] != sx*sy:
                logging.warning('Did not find all corners. Discarding...')
                continue

            # average 3d points
            points_3d_plane = (k * points_3d_plane + points_3d[corner_ind]) / (k + 1)
            logging.info('Registration iteration %d of %d' %(k+1, config['num_transform_avg']))
            k += 1

        # fit a plane to the chessboard corners
        X = np.c_[points_3d_plane.x_coords, points_3d_plane.y_coords, np.ones(points_3d_plane.num_points)]
        y = points_3d_plane.z_coords
        A = X.T.dot(X)
        b = X.T.dot(y)
        w = np.linalg.inv(A).dot(b)
        n = np.array([w[0], w[1], -1])
        n = n / np.linalg.norm(n)
        if flip_normal:
            n = -n
        mean_point_plane = points_3d_plane.mean()

        # find x-axis of the chessboard coordinates on the fitted plane
        T_camera_table = RigidTransform(translation = -points_3d_plane.mean().data,
                                    from_frame=points_3d_plane.frame,
                                    to_frame='table')
        points_3d_centered = T_camera_table * points_3d_plane

        # get points along y
        if point_order == 'row_major':
            coord_pos_x = int(math.floor(sx*sy/2.0))
            coord_neg_x = int(math.ceil(sx*sy/2.0))

            points_pos_x = points_3d_centered[coord_pos_x:]
            points_neg_x = points_3d_centered[:coord_neg_x]
            x_axis = np.mean(points_pos_x.data, axis=1) - np.mean(points_neg_x.data, axis=1)
            x_axis = x_axis - np.vdot(x_axis, n)*n
            x_axis = x_axis / np.linalg.norm(x_axis)
            y_axis = np.cross(n, x_axis)
        else:
            coord_pos_y = int(math.floor(sx*(sy-1)/2.0))
            coord_neg_y = int(math.ceil(sx*(sy+1)/2.0))
            points_pos_y = points_3d_centered[:coord_pos_y]
            points_neg_y = points_3d_centered[coord_neg_y:]
            y_axis = np.mean(points_pos_y.data, axis=1) - np.mean(points_neg_y.data, axis=1)
            y_axis = y_axis - np.vdot(y_axis, n)*n
            y_axis = y_axis / np.linalg.norm(y_axis)
            x_axis = np.cross(-n, y_axis)

        # produce translation and rotation from plane center and chessboard basis
        rotation_cb_camera = RigidTransform.rotation_from_axes(x_axis, y_axis, n)
        translation_cb_camera = mean_point_plane.data
        T_cb_camera = RigidTransform(rotation=rotation_cb_camera,
                                     translation=translation_cb_camera,
                                     from_frame='cb',
                                     to_frame=sensor.frame)

        if y_points_left and np.abs(T_cb_camera.y_axis[1]) > 0.1:
            if T_cb_camera.x_axis[0] > 0:
                T_cb_camera.rotation = T_cb_camera.rotation.dot(RigidTransform.z_axis_rotation(-np.pi/2).T)
            else:
                T_cb_camera.rotation = T_cb_camera.rotation.dot(RigidTransform.z_axis_rotation(np.pi/2).T)
        T_camera_cb = T_cb_camera.inverse()

        # optionally display cb corners with detected pose in 3d space
        if config['debug']:
            # display image with axes overlayed
            cb_center_im = camera_intr.project(Point(T_cb_camera.translation, frame=sensor.ir_frame))
            cb_x_im = camera_intr.project(Point(T_cb_camera.translation + T_cb_camera.x_axis * config['scale_amt'], frame=sensor.ir_frame))
            cb_y_im = camera_intr.project(Point(T_cb_camera.translation + T_cb_camera.y_axis * config['scale_amt'], frame=sensor.ir_frame))
            cb_z_im = camera_intr.project(Point(T_cb_camera.translation + T_cb_camera.z_axis * config['scale_amt'], frame=sensor.ir_frame))
            x_line = np.array([cb_center_im.data, cb_x_im.data])
            y_line = np.array([cb_center_im.data, cb_y_im.data])
            z_line = np.array([cb_center_im.data, cb_z_im.data])

            plt.figure()
            plt.imshow(small_color_im.data)
            plt.scatter(cb_center_im.data[0], cb_center_im.data[1])
            plt.plot(x_line[:,0], x_line[:,1], c='r', linewidth=3)
            plt.plot(y_line[:,0], y_line[:,1], c='g', linewidth=3)
            plt.plot(z_line[:,0], z_line[:,1], c='b', linewidth=3)
            plt.axis('off')
            plt.title('Chessboard frame in camera %s' %(sensor.frame))
            plt.show()

        return ChessboardRegistrationResult(T_camera_cb, points_3d_plane)


Instruction:
Can you write a function in Python 3 where it
adds a new feature to the bag.

Response:
def add(self, feature):
        """ Add a new feature to the bag.

        Parameters
        ----------
        feature : :obj:`Feature`
            feature to add
        """
        self.features_.append(feature)
        self.num_features_ = len(self.features_)


Instruction:
Write a Python 3 function that can
add a list of features to the bag.

Response:
def extend(self, features):
        """ Add a list of features to the bag.

        Parameters
        ----------
        feature : :obj:`list` of :obj:`Feature`
            features to add
        """
        self.features_.extend(features)
        self.num_features_ = len(self.features_)


Instruction:
How would you code a function in Python 3 to
return a feature in the list at the specified index.

Response:
def feature(self, index):
        """ Returns a feature.

        Parameters
        ----------
        index : int
            index of feature in list

        Returns
        -------
        :obj:`Feature`
        """
        if index < 0 or index >= self.num_features_:
            raise ValueError('Index %d out of range' %(index))
        return self.features_[index]


Instruction:
Create a Python 3 function to
return some subset of the features.

Response:
def feature_subset(self, indices):
        """ Returns some subset of the features.

        Parameters
        ----------
        indices : :obj:`list` of :obj:`int`
            indices of the features in the list

        Returns
        -------
        :obj:`list` of :obj:`Feature`
        """
        if isinstance(indices, np.ndarray):
            indices = indices.tolist()
        if not isinstance(indices, list):
            raise ValueError('Can only index with lists')
        return [self.features_[i] for i in indices]


Instruction:
How would you explain what the following Python 3 function does
def total_weight(self):
        weights = self._raw_weights()
        if weights.shape[1] == 0:
            return 0.0
        elif weights.shape[1] < self._ntaps:
            return np.sum(np.mean(weights, axis=1))
        else:
            return self._filter_coeffs.dot(np.sum(weights, axis=0))

Response: Read a weight from the sensor in grams.


Instruction:
Can you create a Python 3 function that
reads individual weights from the load cells in grams.

Response:
def individual_weights(self):
        """Read individual weights from the load cells in grams.

        Returns
        -------
        weight : float
            The sensor weight in grams.
        """
        weights = self._raw_weights()
        if weights.shape[1] == 0:
            return np.zeros(weights.shape[0])
        elif weights.shape[1] < self._ntaps:
            return np.mean(weights, axis=1)
        else:
            return weights.dot(self._filter_coeffs)


Instruction:
Can you write a function in Python 3 where it
creates a numpy array containing the raw sensor weights.

Response:
def _raw_weights(self):
        """Create a numpy array containing the raw sensor weights.
        """
        if self._debug:
            return np.array([[],[],[],[]])

        if not self._running:
            raise ValueError('Weight sensor is not running!')
        if len(self._weight_buffers) == 0:
            time.sleep(0.3)
            if len(self._weight_buffers) == 0:
                raise ValueError('Weight sensor is not retrieving data!')
        weights = np.array(self._weight_buffers)
        return weights


Instruction:
Given the following Python 3 function, write the documentation
def _weights_callback(self, msg):
        # Read weights
        weights = np.array(msg.data)

        # If needed, initialize indiv_weight_buffers
        if len(self._weight_buffers) == 0:
            self._weight_buffers = [[] for i in range(len(weights))]

        # Record individual weights
        for i, w in enumerate(weights):
            if len(self._weight_buffers[i]) == self._ntaps:
                self._weight_buffers[i].pop(0)
            self._weight_buffers[i].append(w)

Response: Callback for recording weights from sensor.


Instruction:
Explain what the following Python 3 code does
def vec(self):
        return np.r_[self.fx, self.fy, self.cx, self.cy, self.skew, self.height, self.width]

Response: Vector representation of this camera.


Instruction:
Can you write a function in Python 3 where it
returns the ROS CamerInfo object for the current message.

Response:
def rosmsg(self):
        """:obj:`sensor_msgs.CamerInfo` : Returns ROS CamerInfo msg
        """
        from sensor_msgs.msg import CameraInfo, RegionOfInterest
        from std_msgs.msg import Header

        msg_header = Header()
        msg_header.frame_id = self._frame

        msg_roi = RegionOfInterest()
        msg_roi.x_offset = 0
        msg_roi.y_offset = 0
        msg_roi.height = 0
        msg_roi.width = 0
        msg_roi.do_rectify = 0

        msg = CameraInfo()
        msg.header = msg_header
        msg.height = self._height
        msg.width = self._width
        msg.distortion_model = 'plumb_bob'
        msg.D = [0.0, 0.0, 0.0, 0.0, 0.0]
        msg.K = [self._fx, 0.0, self._cx, 0.0, self._fy, self._cy, 0.0, 0.0, 1.0]
        msg.R = [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
        msg.P = [self._fx, 0.0, self._cx, 0.0, 0.0, self._fx, self._cy, 0.0, 0.0, 0.0, 1.0, 0.0]
        msg.binning_x = 0
        msg.binning_y = 0
        msg.roi = msg_roi

        return msg


Instruction:
How would you explain what the following Python 3 function does
def crop(self, height, width, crop_ci, crop_cj):
        cx = self.cx + float(width-1)/2 - crop_cj
        cy = self.cy + float(height-1)/2 - crop_ci
        cropped_intrinsics = CameraIntrinsics(frame=self.frame,
                                              fx=self.fx,
                                              fy=self.fy,
                                              skew=self.skew,
                                              cx=cx, cy=cy,
                                              height=height,
                                              width=width)
        return cropped_intrinsics

Response: Convert to new camera intrinsics for crop of image from original camera.