Instruction:
How would you implement a function in Python 3 that
writes a new value of some number of PVs.

Response:
def put(self, name, values, request=None, timeout=5.0, throw=True,
            process=None, wait=None, get=True):
        """Write a new value of some number of PVs.

        :param name: A single name string or list of name strings
        :param values: A single value, a list of values, a dict, a `Value`.  May be modified by the constructor nt= argument.
        :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default.
        :param float timeout: Operation timeout in seconds
        :param bool throw: When true, operation error throws an exception.
                     If False then the Exception is returned instead of the Value
        :param str process: Control remote processing.  May be 'true', 'false', 'passive', or None.
        :param bool wait: Wait for all server processing to complete.
        :param bool get: Whether to do a Get before the Put.  If True then the value passed to the builder callable
                         will be initialized with recent PV values.  eg. use this with NTEnum to find the enumeration list.

        :returns: A None or Exception, or list of same

        When invoked with a single name then returns is a single value.
        When invoked with a list of name, then returns a list of values

        If 'wait' or 'process' is specified, then 'request' must be omitted or None.

        >>> ctxt = Context('pva')
        >>> ctxt.put('pv:name', 5.0)
        >>> ctxt.put(['pv:1', 'pv:2'], [1.0, 2.0])
        >>> ctxt.put('pv:name', {'value':5})
        >>>

        The provided value(s) will be automatically coerced to the target type.
        If this is not possible then an Exception is raised/returned.

        Unless the provided value is a dict, it is assumed to be a plain value
        and an attempt is made to store it in '.value' field.
        """
        if request and (process or wait is not None):
            raise ValueError("request= is mutually exclusive to process= or wait=")
        elif process or wait is not None:
            request = 'field()record[block=%s,process=%s]' % ('true' if wait else 'false', process or 'passive')

        singlepv = isinstance(name, (bytes, unicode))
        if singlepv:
            name = [name]
            values = [values]
            request = [request]

        elif request is None:
            request = [None] * len(name)

        assert len(name) == len(request), (name, request)
        assert len(name) == len(values), (name, values)

        # use Queue instead of Event to allow KeyboardInterrupt
        done = Queue()
        result = [TimeoutError()] * len(name)
        ops = [None] * len(name)

        raw_put = super(Context, self).put

        try:
            for i, (n, value, req) in enumerate(izip(name, values, request)):
                if isinstance(value, (bytes, unicode)) and value[:1] == '{':
                    try:
                        value = json.loads(value)
                    except ValueError:
                        raise ValueError("Unable to interpret '%s' as json" % value)

                # completion callback
                def cb(value, i=i):
                    try:
                        done.put_nowait((value, i))
                    except:
                        _log.exception("Error queuing put result %s", LazyRepr(value))

                ops[i] = raw_put(n, cb, builder=value, request=req, get=get)

            for _n in range(len(name)):
                try:
                    value, i = done.get(timeout=timeout)
                except Empty:
                    if throw:
                        raise TimeoutError()
                    break
                if throw and isinstance(value, Exception):
                    raise value
                result[i] = value

            if singlepv:
                return result[0]
            else:
                return result
        finally:
            [op and op.close() for op in ops]


Instruction:
Here you have a function in Python 3, explain what it does
def rpc(self, name, value, request=None, timeout=5.0, throw=True):
        done = Queue()

        op = super(Context, self).rpc(name, done.put_nowait, value, request=request)

        try:
            try:
                result = done.get(timeout=timeout)
            except Empty:
                result = TimeoutError()
            if throw and isinstance(result, Exception):
                raise result

            return result
        except:
            op.close()
            raise

Response: Perform a Remote Procedure Call to get a specific value from the cache.


Instruction:
Create a Python 3 function for
creating a subscription. :param str name: PV name string :param callable cb: Processing callback :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default. :param bool notify_disconnect: In additional to Values, the callback may also be call with instances of Exception. Specifically: Disconnected , RemoteError, or Cancelled :param WorkQueue queue: A work queue through which monitor callbacks are dispatched. :returns: a :py:class:`Subscription` instance The callable will be invoked with one argument which is either. * A p4p.Value (Subject to :py:ref:`unwrap`) * A sub-class of Exception (Disconnected , RemoteError, or Cancelled)

Response:
def monitor(self, name, cb, request=None, notify_disconnect=False, queue=None):
        """Create a subscription.

        :param str name: PV name string
        :param callable cb: Processing callback
        :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default.
        :param bool notify_disconnect: In additional to Values, the callback may also be call with instances of Exception.
                                       Specifically: Disconnected , RemoteError, or Cancelled
        :param WorkQueue queue: A work queue through which monitor callbacks are dispatched.
        :returns: a :py:class:`Subscription` instance

        The callable will be invoked with one argument which is either.

        * A p4p.Value (Subject to :py:ref:`unwrap`)
        * A sub-class of Exception (Disconnected , RemoteError, or Cancelled)
        """
        R = Subscription(self, name, cb, notify_disconnect=notify_disconnect, queue=queue)

        R._S = super(Context, self).monitor(name, R._event, request)
        return R


Instruction:
Write a Python 3 function for
marking the PV as opened and update its initial value.

Response:
def open(self, value, nt=None, wrap=None, unwrap=None):
        """Mark the PV as opened an provide its initial value.
        This initial value is later updated with post().

        :param value:  A Value, or appropriate object (see nt= and wrap= of the constructor).

        Any clients which have begun connecting which began connecting while
        this PV was in the close'd state will complete connecting.

        Only those fields of the value which are marked as changed will be stored.
        """

        self._wrap = wrap or (nt and nt.wrap) or self._wrap
        self._unwrap = unwrap or (nt and nt.unwrap) or self._unwrap

        _SharedPV.open(self, self._wrap(value))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def rpc(rtype=None):
    wrap = None
    if rtype is None or isinstance(rtype, Type):
        pass
    elif isinstance(type, (list, tuple)):
        rtype = Type(rtype)
    elif hasattr(rtype, 'type'):  # eg. one of the NT* helper classes
        wrap = rtype.wrap
        rtype = rtype.type
    else:
        raise TypeError("Not supported")

    def wrapper(fn):
        if wrap is not None:
            orig = fn

            @wraps(orig)
            def wrapper2(*args, **kws):
                return wrap(orig(*args, **kws))
            fn = wrapper2

        fn._reply_Type = rtype
        return fn
    return wrapper

Response: Decorator marks a method for export.


Instruction:
How would you explain what the following Python 3 function does
def quickRPCServer(provider, prefix, target,
                   maxsize=20,
                   workers=1,
                   useenv=True, conf=None, isolate=False):
    """Run an RPC server in the current thread

    Calls are handled sequentially, and always in the current thread, if workers=1 (the default).
    If workers>1 then calls are handled concurrently by a pool of worker threads.
    Requires NTURI style argument encoding.

    :param str provider: A provider name.  Must be unique in this process.
    :param str prefix: PV name prefix.  Along with method names, must be globally unique.
    :param target: The object which is exporting methods.  (use the :func:`rpc` decorator)
    :param int maxsize: Number of pending RPC calls to be queued.
    :param int workers: Number of worker threads (default 1)
    :param useenv: Passed to :class:`~p4p.server.Server`
    :param conf: Passed to :class:`~p4p.server.Server`
    :param isolate: Passed to :class:`~p4p.server.Server`
    """
    from p4p.server import Server
    import time
    queue = ThreadedWorkQueue(maxsize=maxsize, workers=workers)
    provider = NTURIDispatcher(queue, target=target, prefix=prefix, name=provider)
    threads = []
    server = Server(providers=[provider], useenv=useenv, conf=conf, isolate=isolate)
    with server, queue:
        while True:
            time.sleep(10.0)

Response: Run an RPC server in the current thread.


Instruction:
Explain what the following Python 3 code does
def rpcproxy(spec):
    # inject our ctor first so we don't have to worry about super() non-sense.

    def _proxyinit(self, context=None, format={}, **kws):
        assert context is not None, context
        self.context = context
        self.format = format
        spec.__init__(self, **kws)
    obj = {'__init__': _proxyinit}

    for K, V in inspect.getmembers(spec, lambda M: hasattr(M, '_call_PV')):
        obj[K] = _wrapMethod(K, V)

    return type(spec.__name__, (RPCProxyBase, spec), obj)

Response: Decorator to enable this class to proxy RPC client calls to the given PV names.


Instruction:
Can you tell what is the following Python 3 function doing
def unwrap(self, val):
        if val.getID()!=self.id:
            self._update(val)
        return self._unwrap(val)

Response: Unpack a Value as some other python type returning a new object.


Instruction:
Make a summary of the following Python 3 code
def buildType(valtype, extra=[]):
        assert valtype[:1] == 'a', 'valtype must be an array'
        return Type(id="epics:nt/NTMultiChannel:1.0",
                    spec=[
                    ('value', valtype),
                    ('channelName', 'as'),
                    ('descriptor', 's'),
                    ('alarm', alarm),
                    ('timeStamp', timeStamp),
                    ('severity', 'ai'),
                    ('status', 'ai'),
                    ('message', 'as'),
                    ('secondsPastEpoch', 'al'),
                    ('nanoseconds', 'ai'),
                    ('userTag', 'ai'),
                    ('isConnected', 'a?'),
                    ] + extra)

Response: Build a Type object from a type code.


Instruction:
Can you implement a function in Python 3 that
builds a table object.

Response:
def buildType(columns=[], extra=[]):
        """Build a table

        :param list columns: List of column names and types. eg [('colA', 'd')]
        :param list extra: A list of tuples describing additional non-standard fields
        :returns: A :py:class:`Type`
        """
        return Type(id="epics:nt/NTTable:1.0",
                    spec=[
                    ('labels', 'as'),
                    ('value', ('S', None, columns)),
                    ('descriptor', 's'),
                    ('alarm', alarm),
                    ('timeStamp', timeStamp),
                    ] + extra)


Instruction:
Can you write a function in Python 3 where it
packs an iterable of dict into a Value object

Response:
def wrap(self, values):
        """Pack an iterable of dict into a Value

        >>> T=NTTable([('A', 'ai'), ('B', 'as')])
        >>> V = T.wrap([
            {'A':42, 'B':'one'},
            {'A':43, 'B':'two'},
        ])
        """
        if isinstance(values, Value):
            return values
        cols = dict([(L, []) for L in self.labels])
        try:
            # unzip list of dict
            for V in values:
                for L in self.labels:
                    try:
                        cols[L].append(V[L])
                    except (IndexError, KeyError):
                        pass
            # allow omit empty columns
            for L in self.labels:
                V = cols[L]
                if len(V) == 0:
                    del cols[L]

            try:
                return self.Value(self.type, {
                    'labels': self.labels,
                    'value': cols,
                })
            except:
                _log.error("Failed to encode '%s' with %s", cols, self.labels)
                raise
        except:
            _log.exception("Failed to wrap: %s", values)
            raise


Instruction:
Explain what the following Python 3 code does
def unwrap(value):
        ret = []

        # build lists of column names, and value
        lbl, cols = [], []
        for cname, cval in value.value.items():
            lbl.append(cname)
            cols.append(cval)

        # zip together column arrays to iterate over rows
        for rval in izip(*cols):
            # zip together column names and row values
            ret.append(OrderedDict(zip(lbl, rval)))

        return ret

Response:
Iterate an NTTable

        :returns: An iterator yielding an OrderedDict for each column


Instruction:
Explain what the following Python 3 code does
def buildType(args):
        try:
            return Type(id="epics:nt/NTURI:1.0", spec=[
                ('scheme', 's'),
                ('authority', 's'),
                ('path', 's'),
                ('query', ('S', None, args)),
            ])
        except Exception as e:
            raise ValueError('Unable to build NTURI compatible type from %s' % args)

Response: Builds a NTURI compatible type from a list of query argument name and PVD type code.


Instruction:
Write a Python 3 function that can
wrap argument values ( tuple or list with optional dict

Response:
def wrap(self, path, args=(), kws={}, scheme='', authority=''):
        """Wrap argument values (tuple/list with optional dict) into Value

        :param str path: The PV name to which this call is made
        :param tuple args: Ordered arguments
        :param dict kws: Keyword arguments
        :rtype: Value
        """
        # build dict of argument name+value
        AV = {}
        AV.update([A for A in kws.items() if A[1] is not None])
        AV.update([(N, V) for (N, _T), V in zip(self._args, args)])

        # list of argument name+type tuples for which a value was provided
        AT = [A for A in self._args if A[0] in AV]

        T = self.buildType(AT)
        try:
            return Value(T, {
                'scheme': scheme,
                'authority': authority,
                'path': path,
                'query': AV,
            })
        except Exception as e:
            raise ValueError('Unable to initialize NTURI %s from %s using %s' % (AT, AV, self._args))


Instruction:
Implement a Python 3 function for
packing python value into Value

Response:
def wrap(self, value, timestamp=None):
        """Pack python value into Value
        """
        V = self.type()
        S, NS = divmod(float(timestamp or time.time()), 1.0)
        V.timeStamp = {
            'secondsPastEpoch': S,
            'nanoseconds': NS * 1e9,
        }
        if isinstance(value, dict):
            # assume dict of index and choices list
            V.value = value
            self._choices = V['value.choices']
        else:
            # index or string
            self.assign(V, value)
        return V


Instruction:
Given the following Python 3 function, write the documentation
def unwrap(self, value):
        if value.changed('value.choices'):
            self._choices = value['value.choices']

        idx = value['value.index']
        ret = ntenum(idx)._store(value)
        try:
            ret.choice = self._choices[idx]
        except IndexError:
            pass # leave it as None
        return ret

Response: Unpack a Value into an augmented python type


Instruction:
Can you create a Python 3 function that
stores python value in Value

Response:
def assign(self, V, py):
        """Store python value in Value
        """
        if isinstance(py, (bytes, unicode)):
            for i,C in enumerate(V['value.choices'] or self._choices):
                if py==C:
                    V['value.index'] = i
                    return

        # attempt to parse as integer
        V['value.index'] = py


Instruction:
Can you create a Python 3 function that
closes the PV and disconnect any workers.

Response:
def close(self, destroy=False, sync=False, timeout=None):
        """Close PV, disconnecting any clients.

        :param bool destroy: Indicate "permanent" closure.  Current clients will not see subsequent open().
        :param bool sync: When block until any pending onLastDisconnect() is delivered (timeout applies).
        :param float timeout: Applies only when sync=True.  None for no timeout, otherwise a non-negative floating point value.

        close() with destory=True or sync=True will not prevent clients from re-connecting.
        New clients may prevent sync=True from succeeding.
        Prevent reconnection by __first__ stopping the Server, removing with :py:meth:`StaticProvider.remove()`,
        or preventing a :py:class:`DynamicProvider` from making new channels to this SharedPV.
        """
        _SharedPV.close(self, destroy)
        if sync:
            # TODO: still not syncing PVA workers...
            self._queue.sync()
            self._disconnected.wait()


Instruction:
Can you create a Python 3 function that
counts the number of instances of each type and class of each object in the list

Response:
def gcstats():
    """Count the number of instances of each type/class

    :returns: A dict() mapping type (as a string) to an integer number of references
    """
    all = gc.get_objects()
    _stats = {}

    for obj in all:
        K = type(obj)
        if K is StatsDelta:
            continue  # avoid counting ourselves

        elif K is InstanceType:  # instance of an old-style class
            K = getattr(obj, '__class__', K)

        # Track types as strings to avoid holding references
        K = str(K)

        try:
            _stats[K] += 1
        except KeyError:
            _stats[K] = 1

    # explicitly break the reference loop between the list and this frame,
    # which is contained in the list
    # This would otherwise prevent the list from being free'd
    del all

    return _stats


Instruction:
Make a summary of the following Python 3 code
def periodic(period=60.0, file=sys.stderr):
    import threading
    import time
    S = _StatsThread(period=period, file=file)
    T = threading.Thread(target=S)
    T.daemon = True
    T.start()

Response: Start a daemon thread which will periodically print GC stats for the


Instruction:
Write a Python 3 script for
collecting stats and print results to file

Response:
def collect(self, file=sys.stderr):
        """Collect stats and print results to file

        :param file: A writable file-like object
        """
        cur = gcstats()
        Ncur = len(cur)
        if self.stats is not None and file is not None:
            prev = self.stats
            Nprev = self.ntypes  # may be less than len(prev)

            if Ncur != Nprev:
                print("# Types %d -> %d" % (Nprev, Ncur), file=file)

            Scur, Sprev, first = set(cur), set(prev), True
            for T in Scur - Sprev:  # new types
                if first:
                    print('New Types', file=file)
                    first = False
                print(' ', T, cur[T], file=file)

            first = True
            for T in Sprev - Scur:  # collected types
                if first:
                    print('Cleaned Types', file=file)
                    first = False
                print(' ', T, -prev[T], file=file)

            first = True
            for T in Scur & Sprev:
                if cur[T] == prev[T]:
                    continue
                if first:
                    print('Known Types', file=file)
                    first = False
                print(' ', T, cur[T], 'delta', cur[T] - prev[T], file=file)

        else:  # first call
            print("All Types", file=file)
            for T, C in cur.items():
                print(' ', T, C, file=file)

        self.stats, self.ntypes = cur, len(cur)


Instruction:
Can you create a Python 3 function that
fetches the current value of some number of PVs.

Response:
def get(self, name, request=None, timeout=5.0, throw=True):
        """Fetch current value of some number of PVs.

        :param name: A single name string or list of name strings
        :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default.
        :param float timeout: Operation timeout in seconds
        :param bool throw: When true, operation error throws an exception.  If False then the Exception is returned instead of the Value

        :returns: A p4p.Value or Exception, or list of same.  Subject to :py:ref:`unwrap`.

        When invoked with a single name then returns is a single value.
        When invoked with a list of name, then returns a list of values

        >>> ctxt = Context('pva')
        >>> V = ctxt.get('pv:name')
        >>> A, B = ctxt.get(['pv:1', 'pv:2'])
        >>>
        """
        singlepv = isinstance(name, (bytes, unicode))
        if singlepv:
            return self._get_one(name, request=request, timeout=timeout, throw=throw)

        elif request is None:
            request = [None] * len(name)

        assert len(name) == len(request), (name, request)

        return cothread.WaitForAll(
            [cothread.Spawn(self._get_one, N, request=R, timeout=timeout, throw=throw,
                            raise_on_wait=True)
             for N, R in zip(name, request)]
        )


Instruction:
Implement a Python 3 function for
writing a new value of some number of PVs.

Response:
def put(self, name, values, request=None, process=None, wait=None, timeout=5.0, get=True, throw=True):
        """Write a new value of some number of PVs.

        :param name: A single name string or list of name strings
        :param values: A single value, a list of values, a dict, a `Value`.  May be modified by the constructor nt= argument.
        :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default.
        :param float timeout: Operation timeout in seconds
        :param bool throw: When true, operation error throws an exception.
                     If False then the Exception is returned instead of the Value
        :param str process: Control remote processing.  May be 'true', 'false', 'passive', or None.
        :param bool wait: Wait for all server processing to complete.
        :param bool get: Whether to do a Get before the Put.  If True then the value passed to the builder callable
                         will be initialized with recent PV values.  eg. use this with NTEnum to find the enumeration list.

        :returns: A None or Exception, or list of same

        When invoked with a single name then returns is a single value.
        When invoked with a list of name, then returns a list of values

        If 'wait' or 'process' is specified, then 'request' must be omitted or None.

        >>> ctxt = Context('pva')
        >>> ctxt.put('pv:name', 5.0)
        >>> ctxt.put(['pv:1', 'pv:2'], [1.0, 2.0])
        >>> ctxt.put('pv:name', {'value':5})
        >>>

        The provided value(s) will be automatically coerced to the target type.
        If this is not possible then an Exception is raised/returned.

        Unless the provided value is a dict, it is assumed to be a plain value
        and an attempt is made to store it in '.value' field.
        """
        if request and (process or wait is not None):
            raise ValueError("request= is mutually exclusive to process= or wait=")
        elif process or wait is not None:
            request = 'field()record[block=%s,process=%s]' % ('true' if wait else 'false', process or 'passive')

        singlepv = isinstance(name, (bytes, unicode))
        if singlepv:
            return self._put_one(name, values, request=request, timeout=timeout, throw=throw, get=get)

        elif request is None:
            request = [None] * len(name)

        assert len(name) == len(request), (name, request)
        assert len(name) == len(values), (name, values)

        return cothread.WaitForAll(
            [cothread.Spawn(self._put_one, N, V, request=R, timeout=timeout, throw=throw, get=get,
                            raise_on_wait=True)
             for N, V, R in zip(name, values, request)]
        )


Instruction:
How would you implement a function in Python 3 that
creates a subscription. :param str name: PV name string :param callable cb: Processing callback :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default. :param bool notify_disconnect: In additional to Values, the callback may also be call with instances of Exception. Specifically: Disconnected , RemoteError, or Cancelled :returns: a :py:class:`Subscription` instance The callable will be invoked with one argument which is either. * A p4p.Value (Subject to :py:ref:`unwrap`) * A sub-class of Exception (Disconnected , RemoteError, or Cancelled)

Response:
def monitor(self, name, cb, request=None, notify_disconnect=False):
        """Create a subscription.

        :param str name: PV name string
        :param callable cb: Processing callback
        :param request: A :py:class:`p4p.Value` or string to qualify this request, or None to use a default.
        :param bool notify_disconnect: In additional to Values, the callback may also be call with instances of Exception.
                                       Specifically: Disconnected , RemoteError, or Cancelled
        :returns: a :py:class:`Subscription` instance

        The callable will be invoked with one argument which is either.

        * A p4p.Value (Subject to :py:ref:`unwrap`)
        * A sub-class of Exception (Disconnected , RemoteError, or Cancelled)
        """
        R = Subscription(name, cb, notify_disconnect=notify_disconnect)
        cb = partial(cothread.Callback, R._event)

        R._S = super(Context, self).monitor(name, cb, request)
        return R


Instruction:
Create a Python 3 function for
closing the current subscription.

Response:
def close(self):
        """Close subscription.
        """
        if self._S is not None:
            # after .close() self._event should never be called
            self._S.close()
            self._S = None
            self._Q.Signal(None)
            self._T.Wait()


Instruction:
Write a Python 3 function for
returning a list of members of a specific chamber and congress.

Response:
def filter(self, chamber, congress=CURRENT_CONGRESS, **kwargs):
        """
        Takes a chamber and Congress,
        OR state and district, returning a list of members
        """
        check_chamber(chamber)

        kwargs.update(chamber=chamber, congress=congress)

        if 'state' in kwargs and 'district' in kwargs:
            path = ("members/{chamber}/{state}/{district}/"
                    "current.json").format(**kwargs)

        elif 'state' in kwargs:
            path = ("members/{chamber}/{state}/"
                    "current.json").format(**kwargs)

        else:
            path = ("{congress}/{chamber}/"
                    "members.json").format(**kwargs)

        return self.fetch(path, parse=lambda r: r['results'])


Instruction:
How would you explain what the following Python 3 function does
def bills(self, member_id, type='introduced'):
        "Same as BillsClient.by_member"
        path = "members/{0}/bills/{1}.json".format(member_id, type)
        return self.fetch(path)

Response: Same as BillsClient. by_member


Instruction:
Given the following Python 3 function, write the documentation
def compare(self, first, second, chamber, type='votes', congress=CURRENT_CONGRESS):
        check_chamber(chamber)
        path = "members/{first}/{type}/{second}/{congress}/{chamber}.json"
        path = path.format(first=first, second=second, type=type,
                           congress=congress, chamber=chamber)
        return self.fetch(path)

Response:
See how often two members voted together in a given Congress.
        Takes two member IDs, a chamber and a Congress number.


Instruction:
Explain what the following Python 3 code does
def by_member(self, member_id, type='introduced'):
        path = "members/{member_id}/bills/{type}.json".format(
            member_id=member_id, type=type)
        return self.fetch(path)

Response: Returns a list of recent bills by a member ID and a type


Instruction:
Explain what the following Python 3 code does
def upcoming(self, chamber, congress=CURRENT_CONGRESS):
        "Shortcut for upcoming bills"
        path = "bills/upcoming/{chamber}.json".format(chamber=chamber)
        return self.fetch(path)

Response: Shortcut for upcoming bills


Instruction:
Can you generate the documentation for the following Python 3 function
def by_month(self, chamber, year=None, month=None):
        check_chamber(chamber)

        now = datetime.datetime.now()
        year = year or now.year
        month = month or now.month

        path = "{chamber}/votes/{year}/{month}.json".format(
            chamber=chamber, year=year, month=month)
        return self.fetch(path, parse=lambda r: r['results'])

Response: Return a list of all the votes for a single month.


Instruction:
Can you generate the documentation for the following Python 3 function
def by_range(self, chamber, start, end):
        check_chamber(chamber)

        start, end = parse_date(start), parse_date(end)
        if start > end:
            start, end = end, start

        path = "{chamber}/votes/{start:%Y-%m-%d}/{end:%Y-%m-%d}.json".format(
            chamber=chamber, start=start, end=end)
        return self.fetch(path, parse=lambda r: r['results'])

Response: Return a list of all the votes cast in a chamber between two dates.


Instruction:
Can you tell what is the following Python 3 function doing
def by_date(self, chamber, date):
        "Return votes cast in a chamber on a single day"
        date = parse_date(date)
        return self.by_range(chamber, date, date)

Response: Return votes cast in a chamber on a single day


Instruction:
Can you tell what is the following Python 3 function doing
def today(self, chamber):
        "Return today's votes in a given chamber"
        now = datetime.date.today()
        return self.by_range(chamber, now, now)

Response: Return today s votes in a given chamber


Instruction:
Can you tell what is the following Python 3 function doing
def by_type(self, chamber, type, congress=CURRENT_CONGRESS):
        "Return votes by type: missed, party, lone no, perfect"
        check_chamber(chamber)

        path = "{congress}/{chamber}/votes/{type}.json".format(
            congress=congress, chamber=chamber, type=type)
        return self.fetch(path)

Response: Return votes by type : missed party lone no perfect


Instruction:
Can you implement a function in Python 3 that
returns votes on nominations from a given Congress

Response:
def nominations(self, congress=CURRENT_CONGRESS):
        "Return votes on nominations from a given Congress"
        path = "{congress}/nominations.json".format(congress=congress)
        return self.fetch(path)


Instruction:
Write a Python 3 script for
fetching a specific resource from the Congress API.

Response:
def fetch(self, path, parse=lambda r: r['results'][0]):
        """
        Make an API request, with authentication.

        This method can be used directly to fetch new endpoints
        or customize parsing.

        ::

            >>> from congress import Congress
            >>> client = Congress()
            >>> senate = client.fetch('115/senate/members.json')
            >>> print(senate['num_results'])
            101

        """
        url = self.BASE_URI + path
        headers = {'X-API-Key': self.apikey}

        log.debug(url)

        resp, content = self.http.request(url, headers=headers)
        content = u(content)
        content = json.loads(content)

        # handle errors
        if not content.get('status') == 'OK':

            if "errors" in content and content['errors'][0]['error'] == "Record not found":
                raise NotFound(path)

            if content.get('status') == '404':
                raise NotFound(path)

            raise CongressError(content, resp, url)

        if callable(parse):
            content = parse(content)

        return content


Instruction:
How would you explain what the following Python 3 function does
def parse_date(s):
    if isinstance(s, (datetime.datetime, datetime.date)):
        return s
    try:
        from dateutil.parser import parse
    except ImportError:
        parse = lambda d: datetime.datetime.strptime(d, "%Y-%m-%d")
    return parse(s)

Response: Parse a date using dateutil. parser. parse falling back to datetime. strptime


Instruction:
Make a summary of the following Python 3 code
def append(self, v):
        '''
        >>> d = DiskVarArray('/tmp/test3', dtype='uint32')
        >>> d.append([1, 2, 3, 4])
        >>> d.__getitem__(0)
        memmap([1, 2, 3, 4], dtype=uint32)
        >>> d.append([5, 6, 7, 8])
        >>> d.__getitem__(1)
        memmap([5, 6, 7, 8], dtype=uint32)
        >>> shutil.rmtree('/tmp/test3', ignore_errors=True)
        '''
        self.index.append(len(self.data))
        self.data.extend(v)

Response: Append a new entry to the array.


Instruction:
Given the following Python 3 function, write the documentation
def destroy(self):
        '''
        >>> import numpy as np
        >>> d = DiskVarArray('/tmp/test4', dtype='uint32')
        >>> d.append([1, 2, 3, 4])
        >>> d.destroy # doctest:+ELLIPSIS
        <bound method DiskVarArray.destroy of <diskarray.vararray.DiskVarArray object at 0x...>>
        >>> shutil.rmtree('/tmp/test4', ignore_errors=True)
        '''

        self.data.destroy()
        self.data = None

        self.index.destroy()
        self.index = None

Response:
>>> import numpy as np
        >>> d = DiskVarArray('/tmp/test4', dtype='uint32')
        >>> d.append([1, 2, 3, 4])
        >>> d.destroy # doctest:+ELLIPSIS
        <bound method DiskVarArray.destroy of <diskarray.vararray.DiskVarArray object at 0x...>>
        >>> shutil.rmtree('/tmp/test4', ignore_errors=True)


Instruction:
Can you generate the documentation for the following Python 3 function
def append(self, v):
        '''
        >>> import numpy as np
        >>> da = DiskArray('/tmp/test.array', shape=(0, 3), growby=3, dtype=np.float32)
        >>> print(da[:])
        []
        >>> data = np.array([[2,3,4], [1, 2, 3]])
        >>> da.append(data[0])
        >>> print(da[:])
	[[2. 3. 4.]
	 [0. 0. 0.]
	 [0. 0. 0.]]
        '''

        # FIXME: for now we only support
        # append along axis 0 and only
        # for 1d and 2d arrays

        # FIXME: for now we only support
        # appending one item at a time

        nrows = self._shape[0]
        nrows_capacity = self._capacity_shape[0]

        if nrows == nrows_capacity:
            self._capacity_shape = self._incr_shape(self._capacity_shape, self._growby)
            self._update_ndarray()

        shapelen = len(self._shape)

        if shapelen not in (1, 2):
            raise AppendNotSupported(shapelen)

        self.data[nrows] = v
        self._shape = self._incr_shape(self._shape, 1)

Response: Append a new entry to the array.


Instruction:
Can you generate the documentation for the following Python 3 function
def extend(self, v):
        '''
        >>> import numpy as np
        >>> da = DiskArray('/tmp/test.array', shape=(0, 3), capacity=(10, 3), dtype=np.float32)
        >>> print(da[:])
	[[2. 3. 4.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]
	 [0. 0. 0.]]
        >>> data = np.array([[2,3,4], [1, 2, 3]])
        >>> da.extend(data)
        >>> print(da[:])
        [[2. 3. 4.]
         [1. 2. 3.]
         [0. 0. 0.]
         [0. 0. 0.]
         [0. 0. 0.]
         [0. 0. 0.]
         [0. 0. 0.]
         [0. 0. 0.]
         [0. 0. 0.]
         [0. 0. 0.]]
        >>> os.remove('/tmp/test.array')
        '''

        nrows = self._shape[0]
        nrows_capacity = self._capacity_shape[0]
        remaining_capacity = nrows_capacity - nrows

        if remaining_capacity < len(v):
            diff = len(v) - remaining_capacity
            self._capacity_shape = self._incr_shape(self._capacity_shape, diff)
            self._update_ndarray()

        self.data[nrows:nrows+len(v)] = v
        self._shape = self._incr_shape(self._shape, len(v))

Response: Extend the array with the elements in v.


Instruction:
Here you have a function in Python 3, explain what it does
def _prep_spark_sql_groupby(self):
        # Strip the index info
        non_index_columns = filter(lambda x: x not in self._prdd._index_names,
                                   self._prdd._column_names())
        self._grouped_spark_sql = (self._prdd.to_spark_sql()
                                   .select(non_index_columns)
                                   .groupBy(self._by))
        self._columns = filter(lambda x: x != self._by,
                               non_index_columns)

Response: Used Spark SQL group approach


Instruction:
Can you tell what is the following Python 3 function doing
def _prep_pandas_groupby(self):
        myargs = self._myargs
        mykwargs = self._mykwargs

        def extract_keys(groupedFrame):
            for key, group in groupedFrame:
                yield (key, group)

        def group_and_extract(frame):
            return extract_keys(frame.groupby(*myargs, **mykwargs))

        self._baseRDD = self._prdd._rdd()
        self._distributedRDD = self._baseRDD.flatMap(group_and_extract)
        self._mergedRDD = self._sortIfNeeded(
            self._group(self._distributedRDD))

Response: Prepare the old school pandas group by based approach.


Instruction:
Write a Python 3 script to
group together the values with the same key.

Response:
def _group(self, rdd):
        """Group together the values with the same key."""
        return rdd.reduceByKey(lambda x, y: x.append(y))


Instruction:
How would you explain what the following Python 3 function does
def groups(self):
        self._prep_pandas_groupby()

        def extract_group_labels(frame):
            return (frame[0], frame[1].index.values)

        return self._mergedRDD.map(extract_group_labels).collectAsMap()

Response: Returns a dict mapping group name to group labels.


Instruction:
How would you code a function in Python 3 to
return a dict mapping group name to group indices.

Response:
def indices(self):
        """Returns dict {group name -> group indices}."""
        self._prep_pandas_groupby()

        def extract_group_indices(frame):
            return (frame[0], frame[1].index)

        return self._mergedRDD.map(extract_group_indices).collectAsMap()


Instruction:
Here you have a function in Python 3, explain what it does
def median(self):
        self._prep_pandas_groupby()
        return DataFrame.fromDataFrameRDD(
            self._regroup_mergedRDD().values().map(
                lambda x: x.median()), self.sql_ctx)

Response: Compute the median of the items in the group by the entry s keys.


Instruction:
Explain what the following Python 3 code does
def mean(self):
        if self._can_use_new_school():
            self._prep_spark_sql_groupby()
            import pyspark.sql.functions as func
            return self._use_aggregation(func.mean)
        self._prep_pandas_groupby()
        return DataFrame.fromDataFrameRDD(
            self._regroup_mergedRDD().values().map(
                lambda x: x.mean()), self.sql_ctx)

Response: Compute the mean of the items in the current DataFrame.


Instruction:
Given the following Python 3 function, write the documentation
def var(self, ddof=1):
        self._prep_pandas_groupby()
        return DataFrame.fromDataFrameRDD(
            self._regroup_mergedRDD().values().map(
                lambda x: x.var(ddof=ddof)), self.sql_ctx)

Response: Compute the standard deviation of the items in the log.


Instruction:
Can you generate the documentation for the following Python 3 function
def sum(self):
        if self._can_use_new_school():
            self._prep_spark_sql_groupby()
            import pyspark.sql.functions as func
            return self._use_aggregation(func.sum)
        self._prep_pandas_groupby()
        myargs = self._myargs
        mykwargs = self._mykwargs

        def create_combiner(x):
            return x.groupby(*myargs, **mykwargs).sum()

        def merge_value(x, y):
            return pd.concat([x, create_combiner(y)])

        def merge_combiner(x, y):
            return x + y

        rddOfSum = self._sortIfNeeded(self._distributedRDD.combineByKey(
            create_combiner,
            merge_value,
            merge_combiner)).values()
        return DataFrame.fromDataFrameRDD(rddOfSum, self.sql_ctx)

Response: Compute the sum for each group.


Instruction:
Write a Python 3 function for
creating aggregate expressions using the provided function f with the result coming back as the original column name.

Response:
def _create_exprs_using_func(self, f, columns):
        """Create aggregate expressions using the provided function
        with the result coming back as the original column name."""
        expressions = map(lambda c: f(c).alias(c),
                          self._columns)
        return expressions


Instruction:
Given the following Python 3 function, write the documentation
def _use_aggregation(self, agg, columns=None):
        if not columns:
            columns = self._columns
        from pyspark.sql import functions as F
        aggs = map(lambda column: agg(column).alias(column), self._columns)
        aggRdd = self._grouped_spark_sql.agg(*aggs)
        df = DataFrame.from_schema_rdd(aggRdd, self._by)
        return df

Response: Compute the result using the aggregation function provided.


Instruction:
Given the following Python 3 function, write the documentation
def _regroup_mergedRDD(self):
        myargs = self._myargs
        mykwargs = self._mykwargs
        self._prep_pandas_groupby()

        def regroup(df):
            return df.groupby(*myargs, **mykwargs)

        return self._mergedRDD.mapValues(regroup)

Response: Return a new RDD with the grouped elements.


Instruction:
Create a Python 3 function for
taking the nth element of each grouby.

Response:
def nth(self, n, *args, **kwargs):
        """Take the nth element of each grouby."""
        # TODO: Stop collecting the entire frame for each key.
        self._prep_pandas_groupby()
        myargs = self._myargs
        mykwargs = self._mykwargs
        nthRDD = self._regroup_mergedRDD().mapValues(
            lambda r: r.nth(
                n, *args, **kwargs)).values()
        return DataFrame.fromDataFrameRDD(nthRDD, self.sql_ctx)


Instruction:
How would you explain what the following Python 3 function does
def aggregate(self, f):
        if self._can_use_new_school() and f == pd.Series.kurtosis:
            self._prep_spark_sql_groupby()
            import custom_functions as CF
            return self._use_aggregation(CF.kurtosis)
        else:
            self._prep_pandas_groupby()
            return DataFrame.fromDataFrameRDD(
                self._regroup_mergedRDD().values().map(
                    lambda g: g.aggregate(f)), self.sql_ctx)

Response: Apply the aggregation function f to the items in the current DataFrame.


Instruction:
Create a Python 3 function to
apply the provided function and combine the results together in the same way as apply from groupby in pandas.

Response:
def apply(self, func, *args, **kwargs):
        """Apply the provided function and combine the results together in the
        same way as apply from groupby in pandas.

        This returns a DataFrame.
        """
        self._prep_pandas_groupby()

        def key_by_index(data):
            """Key each row by its index.
            """
            # TODO: Is there a better way to do this?
            for key, row in data.iterrows():
                yield (key, pd.DataFrame.from_dict(
                    dict([(key, row)]), orient='index'))

        myargs = self._myargs
        mykwargs = self._mykwargs
        regroupedRDD = self._distributedRDD.mapValues(
            lambda data: data.groupby(*myargs, **mykwargs))
        appliedRDD = regroupedRDD.map(
            lambda key_data: key_data[1].apply(func, *args, **kwargs))
        reKeyedRDD = appliedRDD.flatMap(key_by_index)
        dataframe = self._sortIfNeeded(reKeyedRDD).values()
        return DataFrame.fromDataFrameRDD(dataframe, self.sql_ctx)


Instruction:
Make a summary of the following Python 3 code
def _create_function(name, doc=""):
    def _(col):
        spark_ctx = SparkContext._active_spark_context
        java_ctx = (getattr(spark_ctx._jvm.com.sparklingpandas.functions,
                            name)
                    (col._java_ctx if isinstance(col, Column) else col))
        return Column(java_ctx)
    _.__name__ = name
    _.__doc__ = doc
    return _

Response: Create a function for aggregator by name


Instruction:
Can you implement a function in Python 3 that
adds another DataFrame to the PStatCounter.

Response:
def merge(self, frame):
        """
        Add another DataFrame to the PStatCounter.
        """
        for column, values in frame.iteritems():
            # Temporary hack, fix later
            counter = self._counters.get(column)
            for value in values:
                if counter is not None:
                    counter.merge(value)


Instruction:
Can you tell what is the following Python 3 function doing
def merge_pstats(self, other):
        if not isinstance(other, PStatCounter):
            raise Exception("Can only merge PStatcounters!")

        for column, counter in self._counters.items():
            other_counter = other._counters.get(column)
            self._counters[column] = counter.mergeStats(other_counter)

        return self

Response: Merge all of the stats counters of this PStatCounter with the other PStatCounter.


Instruction:
Write a Python 3 function for
adding another DataFrame to our accumulated stats for each column.

Response:
def merge(self, frame):
        """
        Add another DataFrame to the accumulated stats for each column.
        Parameters
        ----------
        frame: pandas DataFrame we will update our stats counter with.
        """
        for column_name, _ in self._column_stats.items():
            data_arr = frame[[column_name]].values
            count, min_max_tup, mean, _, _, _ = \
                scistats.describe(data_arr)
            stats_counter = StatCounter()
            stats_counter.n = count
            stats_counter.mu = mean
            stats_counter.m2 = np.sum((data_arr - mean) ** 2)
            stats_counter.minValue, stats_counter.maxValue = min_max_tup
            self._column_stats[column_name] = self._column_stats[
                column_name].mergeStats(stats_counter)
        return self


Instruction:
Can you tell what is the following Python 3 function doing
def merge_stats(self, other_col_counters):
        for column_name, _ in self._column_stats.items():
            self._column_stats[column_name] = self._column_stats[column_name] \
                .mergeStats(other_col_counters._column_stats[column_name])
        return self

Response: Merge statistics from a different column stats counter in to this one.


Instruction:
Implement a function in Python 3 to
return an RDD of Panda DataFrame objects. This can be expensive especially if we don t do narrow transformation after and get it back to Spark SQL land quickly.

Response:
def _rdd(self):
        """Return an RDD of Panda DataFrame objects. This can be expensive
        especially if we don't do a narrow transformation after and get it back
        to Spark SQL land quickly."""
        columns = self._schema_rdd.columns
        index_names = self._index_names

        def fromRecords(records):
            if not records:
                return []
            else:
                loaded_df = pd.DataFrame.from_records([records],
                                                      columns=columns)
                indexed_df = _update_index_on_df(loaded_df, index_names)
                return [indexed_df]

        return self._schema_rdd.rdd.flatMap(fromRecords)


Instruction:
How would you code a function in Python 3 to
return the column names of the current resource table.

Response:
def _column_names(self):
        """Return the column names"""
        index_names = set(_normalize_index_names(self._index_names))
        column_names = [col_name for col_name in self._schema_rdd.columns if
                        col_name not in index_names]
        return column_names


Instruction:
Write a Python 3 script to
convert the underlying SchmeaRDD to an RDD of DataFrames. This is hella slow.

Response:
def _evil_apply_with_dataframes(self, func, preserves_cols=False):
        """Convert the underlying SchmeaRDD to an RDD of DataFrames.
        apply the provide function and convert the result back.
        This is hella slow."""
        source_rdd = self._rdd()
        result_rdd = func(source_rdd)
        # By default we don't know what the columns & indexes are so we let
        # from_rdd_of_dataframes look at the first partition to determine them.
        column_idxs = None
        if preserves_cols:
            index_names = self._index_names
            # Remove indexes from the columns
            columns = self._schema_rdd.columns[len(self._index_names):]
            column_idxs = (columns, index_names)
        return self.from_rdd_of_dataframes(
            result_rdd, column_idxs=column_idxs)


Instruction:
Here you have a function in Python 3, explain what it does
def _first_as_df(self):
        columns = self._schema_rdd.columns
        df = pd.DataFrame.from_records(
            [self._schema_rdd.first()],
            columns=self._schema_rdd.columns)
        df = _update_index_on_df(df, self._index_names)
        return df

Response:
Gets the first row as a Panda s DataFrame. Useful for functions like
            dtypes & ftypes


Instruction:
Can you implement a function in Python 3 that
takes an RDD of Panda s DataFrames and return a Dataframe.

Response:
def from_rdd_of_dataframes(self, rdd, column_idxs=None):
        """Take an RDD of Panda's DataFrames and return a Dataframe.
        If the columns and indexes are already known (e.g. applyMap)
        then supplying them with columnsIndexes will skip eveluating
        the first partition to determine index info."""
        def frame_to_spark_sql(frame):
            """Convert a Panda's DataFrame into Spark SQL Rows"""
            return [r.tolist() for r in frame.to_records()]

        def frame_to_schema_and_idx_names(frames):
            """Returns the schema and index names of the frames. Useful
            if the frame is large and we wish to avoid transfering
            the entire frame. Only bothers to apply once per partiton"""
            try:
                frame = frames.next()
                return [(list(frame.columns), list(frame.index.names))]
            except StopIteration:
                return []

        # Store if the RDD was persisted so we don't uncache an
        # explicitly cached input.
        was_persisted = rdd.is_cached
        # If we haven't been supplied with the schema info cache the RDD
        # since we are going to eveluate the first partition and then eveluate
        # the entire RDD as part of creating a Spark DataFrame.
        (schema, index_names) = ([], [])
        if not column_idxs:
            rdd.cache()
            (schema, index_names) = rdd.mapPartitions(
                frame_to_schema_and_idx_names).first()
        else:
            (schema, index_names) = column_idxs
        # Add the index_names to the schema.
        index_names = _normalize_index_names(index_names)
        schema = index_names + schema
        ddf = DataFrame.from_schema_rdd(
            self.sql_ctx.createDataFrame(rdd.flatMap(frame_to_spark_sql),
                                         schema=schema))
        ddf._index_names = index_names
        if not was_persisted:
            rdd.unpersist()
        return ddf


Instruction:
How would you explain what the following Python 3 function does
def fromDataFrameRDD(cls, rdd, sql_ctx):
        result = DataFrame(None, sql_ctx)
        return result.from_rdd_of_dataframes(rdd)

Response: Construct a DataFrame from an RDD of DataFrames.


Instruction:
Create a Python 3 function for
returning a new DataFrame by applying a function to each element of each Panda DataFrame.

Response:
def applymap(self, f, **kwargs):
        """Return a new DataFrame by applying a function to each element of each
        Panda DataFrame."""
        def transform_rdd(rdd):
            return rdd.map(lambda data: data.applymap(f), **kwargs)
        return self._evil_apply_with_dataframes(transform_rdd,
                                                preserves_cols=True)


Instruction:
Write a Python 3 function for
returning a groupby on the schema rdd. This returns a GroupBy object.

Response:
def groupby(self, by=None, axis=0, level=None, as_index=True, sort=True,
                group_keys=True, squeeze=False):
        """Returns a groupby on the schema rdd. This returns a GroupBy object.
        Note that grouping by a column name will be faster than most other
        options due to implementation."""
        from sparklingpandas.groupby import GroupBy
        return GroupBy(self, by=by, axis=axis, level=level, as_index=as_index,
                       sort=sort, group_keys=group_keys, squeeze=squeeze)


Instruction:
Implement a Python 3 function for
collecting the elements in an DataFrame and concatenate the partition.

Response:
def collect(self):
        """Collect the elements in an DataFrame
        and concatenate the partition."""
        local_df = self._schema_rdd.toPandas()
        correct_idx_df = _update_index_on_df(local_df, self._index_names)
        return correct_idx_df


Instruction:
Can you generate the documentation for the following Python 3 function
def stats(self, columns):
        assert (not isinstance(columns, basestring)), "columns should be a " \
                                                      "list of strs,  " \
                                                      "not a str!"
        assert isinstance(columns, list), "columns should be a list!"

        from pyspark.sql import functions as F
        functions = [F.min, F.max, F.avg, F.count]
        aggs = list(
            self._flatmap(lambda column: map(lambda f: f(column), functions),
                          columns))
        return PStats(self.from_schema_rdd(self._schema_rdd.agg(*aggs)))

Response: Compute the stats for each column provided in columns.


Instruction:
Can you implement a function in Python 3 that
returns a new PRDD by applying a function to each element of each pandas DataFrame.

Response:
def applymap(self, func, **kwargs):
        """Return a new PRDD by applying a function to each element of each
        pandas DataFrame."""
        return self.from_rdd(
            self._rdd.map(lambda data: data.applymap(func), **kwargs))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def groupby(self, *args, **kwargs):
        from sparklingpandas.groupby import GroupBy
        return GroupBy(self._rdd, *args, **kwargs)

Response:
Takes the same parameters as groupby on DataFrame.
        This returns a Sparkling Pandas
        object which supports many of the same operations as regular
        GroupBy but not all.


Instruction:
How would you code a function in Python 3 to
collect the elements in an PRDD and concatenate the partition.

Response:
def collect(self):
        """Collect the elements in an PRDD and concatenate the partition."""
        # The order of the frame order appends is based on the implementation
        # of reduce which calls our function with
        # f(valueToBeAdded, accumulator) so we do our reduce implementation.
        def append_frames(frame_a, frame_b):
            return frame_a.append(frame_b)
        return self._custom_rdd_reduce(append_frames)


Instruction:
Write a Python 3 function for
providing a custom RDD reduce which preserves ordering if the RDD has been sorted.

Response:
def _custom_rdd_reduce(self, reduce_func):
        """Provides a custom RDD reduce which preserves ordering if the RDD has
        been sorted. This is useful for us because we need this functionality
        as many pandas operations support sorting the results. The standard
        reduce in PySpark does not have this property.  Note that when PySpark
        no longer does partition reduces locally this code will also need to
        be updated."""
        def accumulating_iter(iterator):
            acc = None
            for obj in iterator:
                if acc is None:
                    acc = obj
                else:
                    acc = reduce_func(acc, obj)
            if acc is not None:
                yield acc
        vals = self._rdd.mapPartitions(accumulating_iter).collect()
        return reduce(accumulating_iter, vals)


Instruction:
Can you tell what is the following Python 3 function doing
def stats(self, columns):
        def reduce_func(sc1, sc2):
            return sc1.merge_pstats(sc2)

        return self._rdd.mapPartitions(lambda partition: [
            PStatCounter(dataframes=partition, columns=columns)])\
            .reduce(reduce_func)

Response: Compute the stats for each column provided in columns.


Instruction:
Given the following Python 3 function, write the documentation
def read_csv(self, file_path, use_whole_file=False, names=None, skiprows=0,
                 *args, **kwargs):
        """Read a CSV file in and parse it into Pandas DataFrames. By default,
        the first row from the first partition of that data is parsed and used
        as the column names for the data from. If no 'names' param is
        provided we parse the first row of the first partition of data and
        use it for column names.

        Parameters
        ----------
        file_path: string
            Path to input. Any valid file path in Spark works here, eg:
            'file:///my/path/in/local/file/system' or 'hdfs:/user/juliet/'
        use_whole_file: boolean
            Whether of not to use the whole file.
        names: list of strings, optional
        skiprows: integer, optional
            indicates how many rows of input to skip. This will
            only be applied to the first partition of the data (so if
            #skiprows > #row in first partition this will not work). Generally
            this shouldn't be an issue for small values of skiprows.
        No other value of header is supported.
        All additional parameters available in pandas.read_csv() are usable
        here.

        Returns
        -------
        A SparklingPandas DataFrame that contains the data from the
        specified file.
        """
        def csv_file(partition_number, files):
            # pylint: disable=unexpected-keyword-arg
            file_count = 0
            for _, contents in files:
                # Only skip lines on the first file
                if partition_number == 0 and file_count == 0 and _skiprows > 0:
                    yield pandas.read_csv(
                        sio(contents), *args,
                        header=None,
                        names=mynames,
                        skiprows=_skiprows,
                        **kwargs)
                else:
                    file_count += 1
                    yield pandas.read_csv(
                        sio(contents), *args,
                        header=None,
                        names=mynames,
                        **kwargs)

        def csv_rows(partition_number, rows):
            # pylint: disable=unexpected-keyword-arg
            in_str = "\n".join(rows)
            if partition_number == 0:
                return iter([
                    pandas.read_csv(
                        sio(in_str), *args, header=None,
                        names=mynames,
                        skiprows=_skiprows,
                        **kwargs)])
            else:
                # could use .iterows instead?
                return iter([pandas.read_csv(sio(in_str), *args, header=None,
                                             names=mynames, **kwargs)])

        # If we need to peak at the first partition and determine the column
        # names
        mynames = None
        _skiprows = skiprows
        if names:
            mynames = names
        else:
            # In the future we could avoid this expensive call.
            first_line = self.spark_ctx.textFile(file_path).first()
            frame = pandas.read_csv(sio(first_line), **kwargs)
            # pylint sees frame as a tuple despite it being a DataFrame
            mynames = list(frame.columns)
            _skiprows += 1

        # Do the actual load
        if use_whole_file:
            return self.from_pandas_rdd(
                self.spark_ctx.wholeTextFiles(file_path)
                .mapPartitionsWithIndex(csv_file))
        else:
            return self.from_pandas_rdd(
                self.spark_ctx.textFile(file_path)
                    .mapPartitionsWithIndex(csv_rows))

Response: Read a CSV file in and parse it into Pandas DataFrames.


Instruction:
Create a Python 3 function to
load a text file storing one JSON object per line as a DataFrame.

Response:
def jsonFile(self, path, schema=None, sampling_ratio=1.0):
        """Loads a text file storing one JSON object per line as a
        L{DataFrame}.
        Parameters
        ----------
        path: string
             The path of the json files to load. Should be Hadoop style
             paths (e.g. hdfs://..., file://... etc.).
        schema: StructType, optional
             If you know the schema of your input data you can specify it. The
             schema is specified using Spark SQL's schema format. If not
             specified will sample the json records to determine the schema.
             Spark SQL's schema format is documented (somewhat) in the
             "Programmatically Specifying the Schema" of the Spark SQL
             programming guide at: http://bit.ly/sparkSQLprogrammingGuide
        sampling_ratio: int, default=1.0
             Percentage of the records to sample when infering schema.
             Defaults to all records for safety, but you may be able to set to
             a lower ratio if the same fields are present accross records or
             your input is of sufficient size.
        Returns
        -------
        A L{DataFrame} of the contents of the json files.
        """
        schema_rdd = self.sql_ctx.jsonFile(path, schema, sampling_ratio)
        return self.from_spark_rdd(schema_rdd)


Instruction:
Create a Python 3 function to
make a Sparkling Pandas dataframe from a local Pandas DataFrame.

Response:
def from_pd_data_frame(self, local_df):
        """Make a Sparkling Pandas dataframe from a local Pandas DataFrame.
        The intend use is for testing or joining distributed data with local
        data.
        The types are re-infered, so they may not match.
        Parameters
        ----------
        local_df: Pandas DataFrame
            The data to turn into a distributed Sparkling Pandas DataFrame.
            See http://bit.ly/pandasDataFrame for docs.
        Returns
        -------
        A Sparkling Pandas DataFrame.
        """
        def frame_to_rows(frame):
            """Convert a Pandas DataFrame into a list of Spark SQL Rows"""
            # TODO: Convert to row objects directly?
            return [r.tolist() for r in frame.to_records()]
        schema = list(local_df.columns)
        index_names = list(local_df.index.names)
        index_names = _normalize_index_names(index_names)
        schema = index_names + schema
        rows = self.spark_ctx.parallelize(frame_to_rows(local_df))
        sp_df = DataFrame.from_schema_rdd(
            self.sql_ctx.createDataFrame(
                rows,
                schema=schema,
                # Look at all the rows, should be ok since coming from
                # a local dataset
                samplingRatio=1))
        sp_df._index_names = index_names
        return sp_df


Instruction:
Here you have a function in Python 3, explain what it does
def sql(self, query):
        return DataFrame.from_spark_rdd(self.sql_ctx.sql(query), self.sql_ctx)

Response: Perform a SQL query and create a Pandas DataFrame of the result.


Instruction:
Explain what the following Python 3 code does
def table(self, table):
        return DataFrame.from_spark_rdd(self.sql_ctx.table(table),
                                        self.sql_ctx)

Response: Returns the provided Spark SQL table as a Pandas DataFrame.


Instruction:
How would you explain what the following Python 3 function does
def DataFrame(self, elements, *args, **kwargs):
        return self.from_pd_data_frame(pandas.DataFrame(
            elements,
            *args,
            **kwargs))

Response: Create a Sparkling Pandas DataFrame for the provided elements.


Instruction:
Explain what the following Python 3 code does
def read_json(self, file_path,
                  *args, **kwargs):
        """Read a json file in and parse it into Pandas DataFrames.
        If no names is provided we use the first row for the names.
        Currently, it is not possible to skip the first n rows of a file.
        Headers are provided in the json file and not specified separately.

        Parameters
        ----------
        file_path: string
            Path to input. Any valid file path in Spark works here, eg:
            'my/path/in/local/file/system' or 'hdfs:/user/juliet/'
        Other than skipRows, all additional parameters available in
        pandas.read_csv() are usable here.

        Returns
        -------
            A SparklingPandas DataFrame that contains the data from the
        specified file.
        """
        def json_file_to_df(files):
            """ Transforms a JSON file into a list of data"""
            for _, contents in files:
                yield pandas.read_json(sio(contents), *args, **kwargs)

        return self.from_pandas_rdd(self.spark_ctx.wholeTextFiles(file_path)
                                    .mapPartitions(json_file_to_df))

Response: Read a json file in and parse it into Pandas DataFrames.


Instruction:
Can you tell what is the following Python 3 function doing
def do_first(self):
        # Load generator module and create instance if we haven't already
        if (not self.srcfile):
            raise IIIFError(text=("No generator specified"))
        if (not self.gen):
            try:
                (name, ext) = os.path.splitext(self.srcfile)
                (pack, mod) = os.path.split(name)
                module_name = 'iiif.generators.' + mod
                try:
                    module = sys.modules[module_name]
                except KeyError:
                    self.logger.debug(
                        "Loading generator module %s" % (module_name))
                    # Would be nice to use importlib but this is available only
                    # in python 2.7 and higher
                    pack = __import__(module_name)  # returns iiif package
                    module = getattr(pack.generators, mod)
                self.gen = module.PixelGen()
            except ImportError:
                raise IIIFError(
                    text=("Failed to load generator %s" % (str(self.srcfile))))
        (self.width, self.height) = self.gen.size

Response: Load generator set size.


Instruction:
Write a Python 3 script for
parsing arguments instantiate IIIFStatic run.

Response:
def main():
    """Parse arguments, instantiate IIIFStatic, run."""
    if (sys.version_info < (2, 7)):
        sys.exit("This program requires python version 2.7 or later")

    # Options and arguments
    p = optparse.OptionParser(description='IIIF Image API static file generator',
                              usage='usage: %prog [options] file [[file2..]] (-h for help)',
                              version='%prog ' + __version__)

    p.add_option('--dst', '-d', action='store', default='/tmp',
                 help="Destination directory for images [default '%default']")
    p.add_option('--tilesize', '-t', action='store', type='int', default=512,
                 help="Tilesize in pixels [default %default]")
    p.add_option('--api-version', '--api', '-a', action='store', default='2.1',
                 help="API version, may be 1.1, 2.0 or 2.1 [default %default]")
    p.add_option('--prefix', '-p', action='store', default=None,
                 help="URI prefix for where the images will be served from (default '%default'). "
                      "An empty prefix may be OK if the HTML page including the image shares the "
                      "the same root on the same server as the images, otherwise a full URL should "
                      "be specified. This is used to construct the @id in the info.json")
    p.add_option('--identifier', '-i', action='store', default=None,
                 help="Identifier for the image that will be used in place of the filename "
                      "(minus extension). Notes that this option cannot be used if more than "
                      "one image file is to be processed")
    p.add_option('--extra', '-e', action='append', default=[],
                 help="Extra request parameters to be used to generate static files, may be "
                      "repeated (e.g. '/full/90,/0/default.jpg' for a 90 wide thumnail)")
    p.add_option('--write-html', action='store', default=None,
                 help="Write HTML page to the specified directory using the 'identifier.html' "
                      "as the filename. HTML will launch OpenSeadragon for this image and to "
                      "display some of information about info.json and tile locations. HTML will "
                      "assume OpenSeadragon at relative path openseadragonVVV/openseadragon.min.js "
                      "and user-interface icons in openseadragonVVV/images, where VVV are the "
                      "three parts of the version number. The --include-osd flag is also specified "
                      "then OpenSeadragon will be copied to these locations")
    p.add_option('--include-osd', action='store_true',
                 help="Include OpenSeadragon files with --write-html flag")
    p.add_option('--osd-version', action='store', default='2.0.0',
                 help="Generate static images for older versions of OpenSeadragon. Use of versions "
                      "prior to 1.2.1 will force use of /w,h/ for size parameter instead of /w,/. "
                      "Likely useful only in combination with --api-version=1.1 "
                      "[default %default]")
    p.add_option('--osd-width', action='store', type='int', default='500',
                 help="Width of OpenSeadragon pane in pixels. Applies only with "
                      "--write-html [default %default]")
    p.add_option('--osd-height', action='store', type='int', default='500',
                 help="Height of OpenSeadragon pane in pixels. Applies only with "
                      "--write-html [default %default]")
    p.add_option('--generator', action='store_true', default=False,
                 help="Use named generator modules in iiif.generators package instead "
                      "of a starting image [default %default]")
    p.add_option('--max-image-pixels', action='store', type='int', default=0,
                 help="Set the maximum number of pixels in an image. A non-zero value "
                      "will set a hard limit on the image size. If left unset then the "
                      "default configuration of the Python Image Libary (PIL) will give "
                      "a DecompressionBombWarning if the image size exceeds a default "
                      "maximum, but otherwise continue as normal")
    p.add_option('--dryrun', '-n', action='store_true',
                 help="Do not write anything, say what would be done")
    p.add_option('--quiet', '-q', action='store_true',
                 help="Quite (no output unless there is a warning/error)")
    p.add_option('--verbose', '-v', action='store_true',
                 help="Verbose")

    (opt, sources) = p.parse_args()

    level = logging.DEBUG if (opt.verbose) else \
        logging.WARNING if (opt.quiet) else logging.INFO
    logging.basicConfig(format='%(name)s: %(message)s',
                        level=level)
    logger = logging.getLogger(os.path.basename(__file__))

    if (not opt.write_html and opt.include_osd):
        logger.warn(
            "--include-osd has no effect without --write-html, ignoring")
    if (len(sources) == 0):
        logger.warn("No sources specified, nothing to do, bye! (-h for help)")
    elif (len(sources) > 1 and opt.identifier):
        logger.error(
            "Cannot use --identifier/-i option with multiple sources, aborting.")
    else:
        try:
            sg = IIIFStatic(dst=opt.dst, tilesize=opt.tilesize,
                            api_version=opt.api_version, dryrun=opt.dryrun,
                            prefix=opt.prefix, osd_version=opt.osd_version,
                            generator=opt.generator,
                            max_image_pixels=opt.max_image_pixels,
                            extras=opt.extra)
            for source in sources:
                # File or directory (or neither)?
                if (os.path.isfile(source) or opt.generator):
                    logger.info("source file: %s" % (source))
                    sg.generate(source, identifier=opt.identifier)
                    if (opt.write_html):
                        sg.write_html(html_dir=opt.write_html, include_osd=opt.include_osd,
                                      osd_width=opt.osd_width, osd_height=opt.osd_height)
                elif (os.path.isdir(source)):
                    logger.warn(
                        "Ignoring source '%s': directory coversion not supported" % (source))
                else:
                    logger.warn(
                        "Ignoring source '%s': neither file nor path" % (source))
        except (IIIFStaticError, IIIFError) as e:
            # catch known errors and report nicely...
            logger.error("Error: " + str(e))


Instruction:
Explain what the following Python 3 code does
def clear(self):
        # API parameters
        self.identifier = None
        self.region = None
        self.size = None
        self.rotation = None
        self.quality = None
        self.format = None
        self.info = None
        # Derived data and flags
        self.region_full = False
        self.region_square = False
        self.region_pct = False
        self.region_xywh = None  # (x,y,w,h)
        self.size_full = False
        self.size_max = False  # new in 2.1
        self.size_pct = None
        self.size_bang = None
        self.size_wh = None     # (w,h)
        self.rotation_mirror = False
        self.rotation_deg = 0.0

Response: Clear all data that might pertain to an IIIF URL.


Instruction:
Write a Python 3 script to
set the api_version and associated configurations.

Response:
def api_version(self, v):
        """Set the api_version and associated configurations."""
        self._api_version = v
        if (self._api_version >= '2.0'):
            self.default_quality = 'default'
            self.allowed_qualities = ['default', 'color', 'bitonal', 'gray']
        else:  # versions 1.0 and 1.1
            self.default_quality = 'native'
            self.allowed_qualities = ['native', 'color', 'bitonal', 'grey']


Instruction:
Can you write a function in Python 3 where it
builds a URL path for an IIIF Image or Information request.

Response:
def url(self, **params):
        """Build a URL path for image or info request.

        An IIIF Image request with parameterized form is assumed unless
        the info parameter is specified, in which case an Image Information
        request URI is constructred.
        """
        self._setattrs(**params)
        path = self.baseurl + self.quote(self.identifier) + "/"
        if (self.info):
            # info request
            path += "info"
            format = self.format if self.format else "json"
        else:
            # region
            if self.region:
                region = self.region
            elif self.region_xywh:
                region = "%d,%d,%d,%d" % tuple(self.region_xywh)
            else:
                region = "full"
            # size
            if self.size:
                size = self.size
            elif self.size_wh:
                if (self.size_wh[0] is None):
                    size = ",%d" % (self.size_wh[1])
                elif (self.size_wh[1] is None):
                    size = "%d," % (self.size_wh[0])
                else:
                    size = "%d,%d" % (self.size_wh[0], self.size_wh[1])
            elif (self.size_max and self.api_version >= '2.1'):
                size = 'max'
            else:
                size = "full"
            # rotation and quality
            rotation = self.rotation if self.rotation else "0"
            quality = self.quality if self.quality else self.default_quality
            # parameterized form
            path += self.quote(region) + "/" +\
                self.quote(size) + "/" +\
                self.quote(rotation) + "/" +\
                self.quote(quality)
            format = self.format
        if (format):
            path += "." + format
        return(path)


Instruction:
Create a Python 3 function to
parse an IIIF API URL path and each component.

Response:
def parse_url(self, url):
        """Parse an IIIF API URL path and each component.

        Will parse a URL or URL path that accords with either the
        parametrized or info request forms. Will raise an
        IIIFRequestError on failure. A wrapper for the split_url()
        and parse_parameters() methods.

        Note that behavior of split_url() depends on whether
        self.identifier is set.
        """
        self.split_url(url)
        if (not self.info):
            self.parse_parameters()
        return(self)


Instruction:
Make a summary of the following Python 3 code
def split_url(self, url):
        # clear data first
        identifier = self.identifier
        self.clear()
        # url must start with baseurl if set (including slash)
        if (self.baseurl is not None):
            (path, num) = re.subn('^' + self.baseurl, '', url, 1)
            if (num != 1):
                raise IIIFRequestError(
                    text="Request URL does not start with base URL")
            url = path
        # Break up by path segments, count to decide format
        segs = url.split('/')
        if (identifier is not None):
            segs.insert(0, identifier)
        elif (self.allow_slashes_in_identifier):
            segs = self._allow_slashes_in_identifier_munger(segs)
        # Now have segments with identifier as first
        if (len(segs) > 5):
            raise IIIFRequestPathError(
                text="Request URL (%s) has too many path segments" % url)
        elif (len(segs) == 5):
            self.identifier = urlunquote(segs[0])
            self.region = urlunquote(segs[1])
            self.size = urlunquote(segs[2])
            self.rotation = urlunquote(segs[3])
            self.quality = self.strip_format(urlunquote(segs[4]))
            self.info = False
        elif (len(segs) == 2):
            self.identifier = urlunquote(segs[0])
            info_name = self.strip_format(urlunquote(segs[1]))
            if (info_name != "info"):
                raise IIIFRequestError(
                    text="Bad name for Image Information")
            if (self.api_version == '1.0'):
                if (self.format not in ['json', 'xml']):
                    raise IIIFRequestError(
                        text="Invalid format for Image Information (json and xml allowed)")
            elif (self.format != 'json'):
                raise IIIFRequestError(
                    text="Invalid format for Image Information (only json allowed)")
            self.info = True
        elif (len(segs) == 1):
            self.identifier = urlunquote(segs[0])
            raise IIIFRequestBaseURI()
        else:
            raise IIIFRequestPathError(
                text="Bad number of path segments in request")
        return(self)

Response: Parse an IIIF API URL path into components.


Instruction:
Can you generate the documentation for the following Python 3 function
def strip_format(self, str_and_format):
        m = re.match("(.+)\.([a-zA-Z]\w*)$", str_and_format)
        if (m):
            # There is a format string at end, chop off and store
            str_and_format = m.group(1)
            self.format = (m.group(2) if (m.group(2) is not None) else '')
        return(str_and_format)

Response:
Look for optional. fmt at end of URI.
            and store it in self. format


Instruction:
How would you implement a function in Python 3 that
parses the parameters of an Image Information request.

Response:
def parse_parameters(self):
        """Parse the parameters of an Image Information request.

        Will throw an IIIFRequestError on failure, set attributes on
        success. Care is taken not to change any of the artibutes
        which store path components. All parsed values are stored
        in new attributes.
        """
        self.parse_region()
        self.parse_size()
        self.parse_rotation()
        self.parse_quality()
        self.parse_format()


Instruction:
Implement a Python 3 function for
parsing the region component of the path and set self. region_full self. region_square and self. region_pct to True.

Response:
def parse_region(self):
        """Parse the region component of the path.

        /full/ -> self.region_full = True (test this first)
        /square/ -> self.region_square = True (test this second)
        /x,y,w,h/ -> self.region_xywh = (x,y,w,h)
        /pct:x,y,w,h/ -> self.region_xywh and self.region_pct = True

        Will throw errors if the parameters are illegal according to the
        specification but does not know about and thus cannot do any tests
        against any image being manipulated.
        """
        self.region_full = False
        self.region_square = False
        self.region_pct = False
        if (self.region is None or self.region == 'full'):
            self.region_full = True
            return
        if (self.api_version >= '2.1' and self.region == 'square'):
            self.region_square = True
            return
        xywh = self.region
        pct_match = re.match('pct:(.*)$', self.region)
        if (pct_match):
            xywh = pct_match.group(1)
            self.region_pct = True
        # Now whether this was pct: or now, we expect 4 values...
        str_values = xywh.split(',', 5)
        if (len(str_values) != 4):
            raise IIIFRequestError(
                code=400, parameter="region",
                text="Bad number of values in region specification, "
                     "must be x,y,w,h but got %d value(s) from '%s'" %
                     (len(str_values), xywh))
        values = []
        for str_value in str_values:
            # Must be either integer (not pct) or interger/float (pct)
            if (pct_match):
                try:
                    # This is rather more permissive that the iiif spec
                    value = float(str_value)
                except ValueError:
                    raise IIIFRequestError(
                        parameter="region",
                        text="Bad floating point value for percentage in "
                             "region (%s)." % str_value)
                if (value > 100.0):
                    raise IIIFRequestError(
                        parameter="region",
                        text="Percentage over value over 100.0 in region "
                             "(%s)." % str_value)
            else:
                try:
                    value = int(str_value)
                except ValueError:
                    raise IIIFRequestError(
                        parameter="region",
                        text="Bad integer value in region (%s)." % str_value)
            if (value < 0):
                raise IIIFRequestError(
                    parameter="region",
                    text="Negative values not allowed in region (%s)." %
                    str_value)
            values.append(value)
        # Zero size region is w or h are zero (careful that they may be float)
        if (values[2] == 0.0 or values[3] == 0.0):
            raise IIIFZeroSizeError(
                code=400, parameter="region",
                text="Zero size region specified (%s))." % xywh)
        self.region_xywh = values


Instruction:
Explain what the following Python 3 code does
def parse_size(self, size=None):
        if (size is not None):
            self.size = size
        self.size_pct = None
        self.size_bang = False
        self.size_full = False
        self.size_wh = (None, None)
        if (self.size is None or self.size == 'full'):
            self.size_full = True
            return
        elif (self.size == 'max' and self.api_version >= '2.1'):
            self.size_max = True
            return
        pct_match = re.match('pct:(.*)$', self.size)
        if (pct_match is not None):
            pct_str = pct_match.group(1)
            try:
                self.size_pct = float(pct_str)
            except ValueError:
                raise IIIFRequestError(
                    code=400, parameter="size",
                    text="Percentage size value must be a number, got "
                         "'%s'." % (pct_str))
            # Note that Image API specificaton places no upper limit on
            # size so none is implemented here.
            if (self.size_pct < 0.0):
                raise IIIFRequestError(
                    code=400, parameter="size",
                    text="Base size percentage, must be > 0.0, got %f." %
                    (self.size_pct))
        else:
            if (self.size[0] == '!'):
                # Have "!w,h" form
                size_no_bang = self.size[1:]
                (mw, mh) = self._parse_w_comma_h(size_no_bang, 'size')
                if (mw is None or mh is None):
                    raise IIIFRequestError(
                        code=400, parameter="size",
                        text="Illegal size requested: both w,h must be "
                             "specified in !w,h requests.")
                self.size_wh = (mw, mh)
                self.size_bang = True
            else:
                # Must now be "w,h", "w," or ",h"
                self.size_wh = self._parse_w_comma_h(self.size, 'size')
            # Sanity check w,h
            (w, h) = self.size_wh
            if ((w is not None and w <= 0) or
                    (h is not None and h <= 0)):
                raise IIIFZeroSizeError(
                    code=400, parameter='size',
                    text="Size parameters request zero size result image.")

Response: Parse the size component of the path.


Instruction:
Write a Python 3 function for
parsing non negative integer from string istr.

Response:
def _parse_non_negative_int(self, istr, name):
        """Parse integer from string (istr).

        The (name) parameter is used just for IIIFRequestError message
        generation to indicate what the error is in.
        """
        if (istr == ''):
            return(None)
        try:
            i = int(istr)
        except ValueError:
            raise ValueError("Failed to extract integer value for %s" % (name))
        if (i < 0):
            raise ValueError("Illegal negative value for %s" % (name))
        return(i)


Instruction:
Create a Python 3 function for
checking and interprets rotation value and sets self. rotation_deg and self. rotation_mirror to True.

Response:
def parse_rotation(self, rotation=None):
        """Check and interpret rotation.

        Uses value of self.rotation as starting point unless rotation
        parameter is specified in the call. Sets self.rotation_deg to a
        floating point number 0 <= angle < 360. Includes translation of
        360 to 0. If there is a prefix bang (!) then self.rotation_mirror
        will be set True, otherwise it will be False.
        """
        if (rotation is not None):
            self.rotation = rotation
        self.rotation_deg = 0.0
        self.rotation_mirror = False
        if (self.rotation is None):
            return
        # Look for ! prefix first
        if (self.rotation[0] == '!'):
            self.rotation_mirror = True
            self.rotation = self.rotation[1:]
        # Interpret value now
        try:
            self.rotation_deg = float(self.rotation)
        except ValueError:
            raise IIIFRequestError(
                code=400, parameter="rotation",
                text="Bad rotation value, must be a number, got '%s'." %
                (self.rotation))
        if (self.rotation_deg < 0.0 or self.rotation_deg > 360.0):
            raise IIIFRequestError(
                code=400, parameter="rotation",
                text="Illegal rotation value, must be 0 <= rotation "
                     "<= 360, got %f." % (self.rotation_deg))
        elif (self.rotation_deg == 360.0):
            # The spec admits 360 as valid, but change to 0
            self.rotation_deg = 0.0


Instruction:
Given the following Python 3 function, write the documentation
def parse_quality(self):
        if (self.quality is None):
            self.quality_val = self.default_quality
        elif (self.quality not in self.allowed_qualities):
            raise IIIFRequestError(
                code=400, parameter="quality",
                text="The quality parameter must be '%s', got '%s'." %
                ("', '".join(self.allowed_qualities), self.quality))
        else:
            self.quality_val = self.quality

Response:
Checks the quality parameterater. Sets self. quality_val based on simple substitution of
        native for default. Checks for three valid values
            is None and throws an IIIFRequestError if the quality parameter is not valid.


Instruction:
Implement a function in Python 3 to
check format parameter. All formats values listed in the specification are lowercase alphanumeric value commonly used as file extensions. To leave opportunity for extension here just do a limited sanity check on characters and length.

Response:
def parse_format(self):
        """Check format parameter.

        All formats values listed in the specification are lowercase
        alphanumeric value commonly used as file extensions. To leave
        opportunity for extension here just do a limited sanity check
        on characters and length.
        """
        if (self.format is not None and
                not re.match(r'''\w{1,20}$''', self.format)):
            raise IIIFRequestError(
                parameter='format',
                text='Bad format parameter')


Instruction:
Explain what the following Python 3 code does
def is_scaled_full_image(self):
        return(self.region_full and
               self.size_wh[0] is not None and
               self.size_wh[1] is not None and
               not self.size_bang and
               self.rotation_deg == 0.0 and
               self.quality == self.default_quality and
               self.format == 'jpg')

Response: True if this request is for a scaled full image.


Instruction:
Write a Python 3 script for
getting config from defaults config file and parse arguments.

Response:
def get_config(base_dir=''):
    """Get config from defaults, config file and/or parse arguments.

    Uses configargparse to allow argments to be set from a config file
    or via command line arguments.

      base_dir - set a specific base directory for file/path defaults.
    """
    p = configargparse.ArgParser(description='IIIF Image API Reference Service',
                                 default_config_files=[os.path.join(base_dir, 'iiif_reference_server.cfg')],
                                 formatter_class=configargparse.ArgumentDefaultsHelpFormatter)
    add_shared_configs(p, base_dir)
    p.add('--scale-factors', default='auto',
          help="Set of tile scale factors or 'auto' to calculate for each image "
               "such that there are tiles up to the full image")
    p.add('--api-versions', default='1.0,1.1,2.0,2.1',
          help="Set of API versions to support")
    args = p.parse_args()

    if (args.debug):
        args.verbose = True
    elif (args.verbose):
        args.quiet = False

    # Split list arguments
    args.scale_factors = split_comma_argument(args.scale_factors)
    args.api_versions = split_comma_argument(args.api_versions)

    logging_level = logging.WARNING
    if args.verbose:
        logging_level = logging.INFO
    elif args.quiet:
        logging_level = logging.ERROR
    logging.basicConfig(format='%(name)s: %(message)s', level=logging_level)

    return(args)


Instruction:
Create a Python 3 function for
creating referece server Flask application with one or more IIIF handlers.

Response:
def create_reference_server_flask_app(cfg):
    """Create referece server Flask application with one or more IIIF handlers."""
    # Create Flask app
    app = Flask(__name__)
    Flask.secret_key = "SECRET_HERE"
    app.debug = cfg.debug
    # Install request handlers
    client_prefixes = dict()
    for api_version in cfg.api_versions:
        handler_config = Config(cfg)
        handler_config.api_version = api_version
        handler_config.klass_name = 'pil'
        handler_config.auth_type = 'none'
        # Set same prefix on local server as expected on iiif.io
        handler_config.prefix = "api/image/%s/example/reference" % (api_version)
        handler_config.client_prefix = handler_config.prefix
        add_handler(app, handler_config)
    return app


Instruction:
Explain what the following Python 3 code does
def pixel(self, x, y, size=None):
        if (size is None):
            size = self.sz
        # Have we go to the smallest element?
        if (size <= 3):
            if (_middle(x, y)):
                return (None)
            else:
                return (0, 0, 0)
        divisor = size // 3
        if (_middle(x // divisor, y // divisor)):
            return None
        return self.pixel(x % divisor, y % divisor, divisor)

Response: Return color for a pixel.


Instruction:
Can you generate the documentation for the following Python 3 function
def id(self, value):
        i = value.rfind('/')
        if (i > 0):
            self.server_and_prefix = value[:i]
            self.identifier = value[(i + 1):]
        elif (i == 0):
            self.server_and_prefix = ''
            self.identifier = value[(i + 1):]
        else:
            self.server_and_prefix = ''
            self.identifier = value

Response: Split into server_and_prefix and identifier.


Instruction:
Write a Python 3 script to
set up normal values for given api_version.

Response:
def set_version_info(self, api_version=None):
        """Set up normal values for given api_version.

        Will use current value of self.api_version if a version number
        is not specified in the call. Will raise an IIIFInfoError
        """
        if (api_version is None):
            api_version = self.api_version
        if (api_version not in CONF):
            raise IIIFInfoError("Unknown API version %s" % (api_version))
        self.params = CONF[api_version]['params']
        self.array_params = CONF[api_version]['array_params']
        self.complex_params = CONF[api_version]['complex_params']
        for a in ('context', 'compliance_prefix', 'compliance_suffix',
                  'protocol', 'required_params'):
            if (a in CONF[api_version]):
                self._setattr(a, CONF[api_version][a])


Instruction:
How would you explain what the following Python 3 function does
def compliance(self, value):
        if (self.api_version < '2.0'):
            self.profile = value
        else:
            try:
                self.profile[0] = value
            except AttributeError:
                # handle case where profile not initialized as array
                self.profile = [value]

Response: Set the compliance profile URI.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def level(self):
        m = re.match(
            self.compliance_prefix +
            r'(\d)' +
            self.compliance_suffix +
            r'$',
            self.compliance)
        if (m):
            return int(m.group(1))
        raise IIIFInfoError(
            "Bad compliance profile URI, failed to extract level number")

Response: Extract level number from compliance profile URI.


Instruction:
How would you explain what the following Python 3 function does
def level(self, value):
        self.compliance = self.compliance_prefix + \
            ("%d" % value) + self.compliance_suffix

Response: Build profile URI from level.


Instruction:
Explain what the following Python 3 code does
def add_service(self, service):
        if (self.service is None):
            self.service = service
        elif (isinstance(self.service, dict)):
            self.service = [self.service, service]
        else:
            self.service.append(service)

Response: Add a service description. Handles transition from self. service = None and self. service = dict for single service and then self. service = list for multiple services.


Instruction:
Given the following Python 3 function, write the documentation
def validate(self):
        errors = []
        for param in self.required_params:
            if (not hasattr(self, param) or getattr(self, param) is None):
                errors.append("missing %s parameter" % (param))
        if (len(errors) > 0):
            raise IIIFInfoError("Bad data for info.json: " + ", ".join(errors))
        return True

Response: Validate this object as Image API data. Raise IIIFInfoError with helpful message if not valid.