Instruction:
How would you explain what the following Python 3 function does
def resplit(prev, pattern, *args, **kw):
    maxsplit = 0 if 'maxsplit' not in kw else kw.pop('maxsplit')
    pattern_obj = re.compile(pattern, *args, **kw)
    for s in prev:
        yield pattern_obj.split(s, maxsplit=maxsplit)

Response: The resplit pipe split previous string by regular expression.


Instruction:
Make a summary of the following Python 3 code
def sub(prev, pattern, repl, *args, **kw):
    count = 0 if 'count' not in kw else kw.pop('count')
    pattern_obj = re.compile(pattern, *args, **kw)
    for s in prev:
        yield pattern_obj.sub(repl, s, count=count)

Response: A wrapper around re. sub method.


Instruction:
How would you implement a function in Python 3 that
yields all data that matches pattern in the previous generator of natalatalatalels

Response:
def wildcard(prev, pattern, *args, **kw):
    """wildcard pipe greps data passed from previous generator
    according to given regular expression.

    :param prev: The previous iterator of pipe.
    :type prev: Pipe
    :param pattern: The wildcard string which used to filter data.
    :type pattern: str|unicode|re pattern object
    :param inv: If true, invert the match condition.
    :type inv: boolean
    :returns: generator
    """
    import fnmatch

    inv = 'inv' in kw and kw.pop('inv')
    pattern_obj = re.compile(fnmatch.translate(pattern), *args, **kw)

    if not inv:
        for data in prev:
            if pattern_obj.match(data):
                yield data
    else:
        for data in prev:
            if not pattern_obj.match(data):
                yield data


Instruction:
How would you explain what the following Python 3 function does
def stdout(prev, endl='\n', thru=False):
    for i in prev:
        sys.stdout.write(str(i) + endl)
        if thru:
            yield i

Response: This pipe writes data from previous iterator to stdout.


Instruction:
How would you implement a function in Python 3 that
executes a shell command.

Response:
def sh(prev, *args, **kw):
    """sh pipe execute shell command specified by args. If previous pipe exists,
    read data from it and write it to stdin of shell process. The stdout of
    shell process will be passed to next pipe object line by line.

    A optional keyword argument 'trim' can pass a function into sh pipe. It is
    used to trim the output from shell process. The default trim function is
    str.rstrip. Therefore, any space characters in tail of
    shell process output line will be removed.

    For example:

    py_files = result(sh('ls') | strip | wildcard('*.py'))

    :param prev: The previous iterator of pipe.
    :type prev: Pipe
    :param args: The command line arguments. It will be joined by space character.
    :type args: list of string.
    :param kw: arguments for subprocess.Popen.
    :type kw: dictionary of options.
    :returns: generator
    """
    endl = '\n' if 'endl' not in kw else kw.pop('endl')
    trim = None if 'trim' not in kw else kw.pop('trim')
    if trim is None:
        trim = bytes.rstrip if is_py3 else str.rstrip

    cmdline = ' '.join(args)
    if not cmdline:
        if prev is not None:
            for i in prev:
                yield i
        else:
            while True:
                yield None

    process = subprocess.Popen(cmdline, shell=True,
        stdin=subprocess.PIPE, stdout=subprocess.PIPE,
        **kw)
    if prev is not None:
        stdin_buffer = StringIO()
        for i in prev:
            stdin_buffer.write(i)
            if endl:
                stdin_buffer.write(endl)
        if is_py3:
            process.stdin.write(stdin_buffer.getvalue().encode('utf-8'))
        else:
            process.stdin.write(stdin_buffer.getvalue())
        process.stdin.flush()
        process.stdin.close()
        stdin_buffer.close()

    for line in process.stdout:
        yield trim(line)

    process.wait()


Instruction:
Given the following Python 3 function, write the documentation
def join(prev, sep, *args, **kw):
    '''alias of str.join'''
    yield sep.join(prev, *args, **kw)

Response: alias of str. join


Instruction:
Can you implement a function in Python 3 that
aliases of string. Template. substitute

Response:
def substitute(prev, *args, **kw):
    '''alias of string.Template.substitute'''
    template_obj = string.Template(*args, **kw)
    for data in prev:
        yield template_obj.substitute(data)


Instruction:
How would you implement a function in Python 3 that
aliases of string. Template. safe_substitute

Response:
def safe_substitute(prev, *args, **kw):
    '''alias of string.Template.safe_substitute'''
    template_obj = string.Template(*args, **kw)
    for data in prev:
        yield template_obj.safe_substitute(data)


Instruction:
Can you tell what is the following Python 3 function doing
def to_str(prev, encoding=None):
    first = next(prev)
    if isinstance(first, str):
        if encoding is None:
            yield first
            for s in prev:
                yield s
        else:
            yield first.encode(encoding)
            for s in prev:
                yield s.encode(encoding)
    else:
        if encoding is None:
            encoding = sys.stdout.encoding or 'utf-8'
        yield first.decode(encoding)
        for s in prev:
            yield s.decode(encoding)

Response: Convert data from previous pipe with specified encoding.


Instruction:
How would you explain what the following Python 3 function does
def register_default_types():
    register_type(type, pipe.map)
    register_type(types.FunctionType, pipe.map)
    register_type(types.MethodType, pipe.map)
    register_type(tuple, seq)
    register_type(list, seq)
    register_type(types.GeneratorType, seq)
    register_type(string_type, sh)
    register_type(unicode_type, sh)
    register_type(file_type, fileobj)

    if is_py3:
        register_type(range, seq)
        register_type(map, seq)

Response: Regiser all default type - to - pipe convertors.


Instruction:
Write a Python 3 script to
convert Paginator instance to dict

Response:
def get_dict(self):
        '''
        Convert Paginator instance to dict

        :return: Paging data
        :rtype: dict
        '''

        return dict(
            current_page=self.current_page,
            total_page_count=self.total_page_count,
            items=self.items,
            total_item_count=self.total_item_count,
            page_size=self.page_size
        )


Instruction:
Make a summary of the following Python 3 code
def write_log(log_directory, log_filename, header, logline, debug,
              require_latest, latest_directory, latest_filename):
    """This function logs a line of data to both a 'log' file, and a 'latest'
    file The 'latest' file is optional, and is sent to this function as a
    boolean value via the variable 'require_latest'.
    So the 2 log directories and filenames are:
            a. (REQUIRED): log_directory + log_filename
            b. (OPTIONAL): latest_directory + latest_filename

    The 'latest' directory and filename is provided so as to have a consistent
    file of the latest events This is usually the latest day of events.
    The way this function works with the 'latest' log_dir is as follows:
            a. It checks for the existance of log_directory + log_filename
            b. If (a) doesn't exist, then any 'latest' file is removed and a new one created
            c. If (a) already exists, logs are written to any existing 'latest' file
                    If one doesn't exist, it will be created

    For both the 'log' and 'latest' files, a header line will be written if a new
    file is created Please note that a header must start with the '#' symbol, so
    the Ardexa agent can interpret this line as a header , and will not send it to
    the cloud
    """
    create_new_file = False

    # Make sure the logging directory exists. The following will create all the necessary subdirs,
    # if the subdirs exist in part or in full
    if not os.path.exists(log_directory):
        os.makedirs(log_directory)
    full_path_log = os.path.join(log_directory, log_filename)
    if debug > 1:
        print("Full path of log directory: ", full_path_log)
    # If the file doesn't exist, annotate that a new 'latest' file is to be created
    # and that a header is to be created
    if not os.path.isfile(full_path_log):
        if debug > 1:
            print("Log file doesn't exist: ", full_path_log)
        create_new_file = True

    # Repeat for the 'latest', if it doesn't exist
    if require_latest:
        if not os.path.exists(latest_directory):
            os.makedirs(latest_directory)
        full_path_latest = os.path.join(latest_directory, latest_filename)
        if debug > 1:
            print("Full path of latest directory: ", full_path_latest)
        # If the 'create_new_file' tag is set AND the file exists, then remove it
        if create_new_file and os.path.isfile(full_path_latest):
            # then remove the file
            os.remove(full_path_latest)

    # Now create both (or open both) and write to them
    if debug > 1:
        print("##########################################")
        print("Writing the line to", full_path_latest)
        print(logline)
        print("##########################################")

    # Write the logline to the log file
    output_file = open(full_path_log, "a")
    if create_new_file:
        output_file.write(header)
    output_file.write(logline)
    output_file.close()

    # And write it to the 'latest' if required
    if require_latest:
        write_latest = open(full_path_latest, "a")
        if create_new_file:
            write_latest.write(header)
        write_latest.write(logline)
        write_latest.close()

Response: This function writes a line of data to a log file.


Instruction:
Here you have a function in Python 3, explain what it does
def check_pidfile(pidfile, debug):
    # Check PID exists and see if the PID is running
    if os.path.isfile(pidfile):
        pidfile_handle = open(pidfile, 'r')
        # try and read the PID file. If no luck, remove it
        try:
            pid = int(pidfile_handle.read())
            pidfile_handle.close()
            if check_pid(pid, debug):
                return True
        except:
            pass

        # PID is not active, remove the PID file
        os.unlink(pidfile)

    # Create a PID file, to ensure this is script is only run once (at a time)
    pid = str(os.getpid())
    open(pidfile, 'w').write(pid)
    return False

Response: Check that a process is not running more than once using PIDFILE


Instruction:
Can you generate the documentation for the following Python 3 function
def check_pid(pid, debug):
    try:
        # A Kill of 0 is to check if the PID is active. It won't kill the process
        os.kill(pid, 0)
        if debug > 1:
            print("Script has a PIDFILE where the process is still running")
        return True
    except OSError:
        if debug > 1:
            print("Script does not appear to be running")
        return False

Response: This function will check whether a process is currently running.


Instruction:
Given the following Python 3 function, write the documentation
def convert_words_to_uint(high_word, low_word):
    try:
        low_num = int(low_word)
        # low_word might arrive as a signed number. Convert to unsigned
        if low_num < 0:
            low_num = abs(low_num) + 2**15
        number = (int(high_word) << 16) | low_num
        return number, True
    except:
        return 0, False

Response: Convert two words to a floating point number


Instruction:
Can you implement a function in Python 3 that
converts two words to a floating point number

Response:
def convert_words_to_float(high_word, low_word):
    """Convert two words to a floating point"""
    number, retval = convert_words_to_uint(high_word, low_word)
    if not retval:
        return 0.0, False

    try:
        packed_float = struct.pack('>l', number)
        return struct.unpack('>f', packed_float)[0], True
    except:
        return 0.0, False


Instruction:
Make a summary of the following Python 3 code
def disown(debug):
    # Get the current PID
    pid = os.getpid()
    cgroup_file = "/proc/" + str(pid) + "/cgroup"
    try:
        infile = open(cgroup_file, "r")
    except IOError:
        print("Could not open cgroup file: ", cgroup_file)
        return False

    # Read each line
    for line in infile:
        # Check if the line contains "ardexa.service"
        if line.find("ardexa.service") == -1:
            continue

        # if the lines contains "name=", replace it with nothing
        line = line.replace("name=", "")
        # Split  the line by commas
        items_list = line.split(':')
        accounts = items_list[1]
        dir_str = accounts + "/ardexa.disown"
        # If accounts is empty, continue
        if not accounts:
            continue

        # Create the dir and all subdirs
        full_dir = "/sys/fs/cgroup/" + dir_str
        if not os.path.exists(full_dir):
            os.makedirs(full_dir)
            if debug >= 1:
                print("Making directory: ", full_dir)
        else:
            if debug >= 1:
                print("Directory already exists: ", full_dir)

        # Add the PID to the file
        full_path = full_dir + "/cgroup.procs"
        prog_list = ["echo", str(pid), ">", full_path]
        run_program(prog_list, debug, True)

        # If this item contains a comma, then separate it, and reverse
        # some OSes will need cpuacct,cpu reversed to actually work
        if accounts.find(",") != -1:
            acct_list = accounts.split(',')
            accounts = acct_list[1] + "," + acct_list[0]
            dir_str = accounts + "/ardexa.disown"
            # Create the dir and all subdirs. But it may not work. So use a TRY
            full_dir = "/sys/fs/cgroup/" + dir_str
            try:
                if not os.path.exists(full_dir):
                    os.makedirs(full_dir)
            except:
                continue

            # Add the PID to the file
            full_path = full_dir + "/cgroup.procs"
            prog_list = ["echo", str(pid), ">", full_path]
            run_program(prog_list, debug, True)

    infile.close()

    # For debug purposes only
    if debug >= 1:
        prog_list = ["cat", cgroup_file]
        run_program(prog_list, debug, False)

    # If there are any "ardexa.service" in the proc file. If so, exit with error
    prog_list = ["grep", "-q", "ardexa.service", cgroup_file]
    if run_program(prog_list, debug, False):
        # There are entries still left in the file
        return False

    return True

Response: This function will disown the Ardexa service and all of its sub - directories and the Ardexa service.


Instruction:
Make a summary of the following Python 3 code
def run_program(prog_list, debug, shell):
    try:
        if not shell:
            process = Popen(prog_list, stdout=PIPE, stderr=PIPE)
            stdout, stderr = process.communicate()
            retcode = process.returncode
            if debug >= 1:
                print("Program : ", " ".join(prog_list))
                print("Return Code: ", retcode)
                print("Stdout: ", stdout)
                print("Stderr: ", stderr)
            return bool(retcode)
        else:
            command = " ".join(prog_list)
            os.system(command)
            return True
    except:
        return False

Response: Run a program and check return code


Instruction:
Make a summary of the following Python 3 code
def parse_address_list(addrs):
    for addr in addrs.split(','):
        elem = addr.split('-')
        if len(elem) == 1: # a number
            yield int(elem[0])
        elif len(elem) == 2: # a range inclusive
            start, end = list(map(int, elem))
            for i in range(start, end+1):
                yield i
        else: # more than one hyphen
            raise ValueError('format error in %s' % addr)

Response: Yield each integer from a complex range string like 1 - 9 12 15 - 20 23


Instruction:
Can you write a function in Python 3 where it
does url - encode resource ids

Response:
def _encode_ids(*args):
    """
    Do url-encode resource ids
    """

    ids = []
    for v in args:
        if isinstance(v, basestring):
            qv = v.encode('utf-8') if isinstance(v, unicode) else v
            ids.append(urllib.quote(qv))
        else:
            qv = str(v)
            ids.append(urllib.quote(qv))

    return ';'.join(ids)


Instruction:
Explain what the following Python 3 code does
def random_string(length):
    '''
    Generate random string with parameter length.
    Example:

        >>> from eggit.egg_string import random_string
        >>> random_string(8)
        'q4f2eaT4'
        >>>

    '''

    str_list = [random.choice(string.digits + string.ascii_letters) for i in range(length)]
    return ''.join(str_list)

Response: Generate random string with parameter length.


Instruction:
Can you tell what is the following Python 3 function doing
def get_item_creator(item_type):
    if item_type not in Pipe.pipe_item_types:
        for registered_type in Pipe.pipe_item_types:
            if issubclass(item_type, registered_type):
                return Pipe.pipe_item_types[registered_type]
        return None
    else:
        return Pipe.pipe_item_types[item_type]

Response: Get item creator function according registered item type.


Instruction:
Given the following Python 3 function, write the documentation
def clone(self):
        new_object = copy.copy(self)
        if new_object.next:
            new_object.next = new_object.next.clone()
        return new_object

Response: Self - cloning. All its next Pipe objects are cloned too.


Instruction:
Write a Python 3 script for
appending next object to the end of the pipe.

Response:
def append(self, next):
        """Append next object to pipe tail.

        :param next: The Pipe object to be appended to tail.
        :type next: Pipe object.
        """
        next.chained = True
        if self.next:
            self.next.append(next)
        else:
            self.next = next


Instruction:
Can you generate the documentation for the following Python 3 function
def iter(self, prev=None):

        if self.next:
            generator = self.next.iter(self.func(prev, *self.args, **self.kw))
        else:
            generator = self.func(prev, *self.args, **self.kw)
        return generator

Response: Return an iterator object which yields the data from the next pipe.


Instruction:
Can you generate the documentation for the following Python 3 function
def reduce(func):
        def wrapper(prev, *argv, **kw):
            accum_value = None if 'init' not in kw else kw.pop('init')
            if prev is None:
                raise TypeError('A reducer must have input.')
            for i in prev:
                accum_value = func(accum_value, i, *argv, **kw)
            yield accum_value
        return Pipe(wrapper)

Response: Wrap a function to reduce the data from a list of entries.


Instruction:
Make a summary of the following Python 3 code
def stopper(func):
        def wrapper(prev, *argv, **kw):
            if prev is None:
                raise TypeError('A stopper must have input.')
            for i in prev:
                if func(i, *argv, **kw):
                    break
                yield i
        return Pipe(wrapper)

Response: Wrap a conditoinal function to a pipe object.


Instruction:
Create a Python 3 function for
returning a dictionary of network name to active status bools.

Response:
def _list_networks():
    """Return a dictionary of network name to active status bools.

        Sample virsh net-list output::

    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    juju-test            inactive   no
    foobar               inactive   no

    Parsing the above would return::
    {"default": True, "juju-test": False, "foobar": False}

    See: http://goo.gl/kXwfC
    """
    output = core.run("virsh net-list --all")
    networks = {}

    # Take the header off and normalize whitespace.
    net_lines = [n.strip() for n in output.splitlines()[2:]]
    for line in net_lines:
        if not line:
            continue
        name, state, auto = line.split()
        networks[name] = state == "active"
    return networks


Instruction:
Can you tell what is the following Python 3 function doing
def flush(self, line):
        # TODO -- maybe use echo?
        sys.stdout.write(line)
        sys.stdout.flush()

Response: flush the line to stdout


Instruction:
Create a Python 3 function to
run the passed in arguments and returns an iterator on the output of the running command

Response:
def execute(self, arg_str='', **kwargs):
        """runs the passed in arguments and returns an iterator on the output of
        running command"""
        cmd = "{} {} {}".format(self.cmd_prefix, self.script, arg_str)
        expected_ret_code = kwargs.pop('code', 0)

        # any kwargs with all capital letters should be considered environment
        # variables
        environ = self.environ
        for k in list(kwargs.keys()):
            if k.isupper():
                environ[k] = kwargs.pop(k)

        # we will allow overriding of these values
        kwargs.setdefault("stderr", subprocess.STDOUT)

        # we will not allow these to be overridden via kwargs
        kwargs["shell"] = True
        kwargs["stdout"] = subprocess.PIPE
        kwargs["cwd"] = self.cwd
        kwargs["env"] = environ

        process = None
        self.buf = deque(maxlen=self.bufsize)

        try:
            process = subprocess.Popen(
                cmd,
                **kwargs
            )

            # another round of links
            # http://stackoverflow.com/a/17413045/5006 (what I used)
            # http://stackoverflow.com/questions/2715847/
            for line in iter(process.stdout.readline, b""):
                line = line.decode(self.encoding)
                self.buf.append(line.rstrip())
                yield line

            process.wait()
            if process.returncode != expected_ret_code:
                if process.returncode > 0:
                    raise RuntimeError("{} returned {} with output: {}".format(
                        cmd,
                        process.returncode,
                        self.output
                    ))

        except subprocess.CalledProcessError as e:
            if e.returncode != expected_ret_code:
                raise RuntimeError("{} returned {} with output: {}".format(
                    cmd,
                    e.returncode,
                    self.output
                ))

        finally:
            if process:
                process.stdout.close()


Instruction:
How would you implement a function in Python 3 that
builds a basic 035 subfield with basic information from the OAI - PMH request.

Response:
def get_request_subfields(root):
    """Build a basic 035 subfield with basic information from the OAI-PMH request.

    :param root: ElementTree root node

    :return: list of subfield tuples [(..),(..)]
    """
    request = root.find('request')
    responsedate = root.find('responseDate')

    subs = [("9", request.text),
            ("h", responsedate.text),
            ("m", request.attrib["metadataPrefix"])]
    return subs


Instruction:
Explain what the following Python 3 code does
def strip_xml_namespace(root):
    try:
        root.tag = root.tag.split('}')[1]
    except IndexError:
        pass

    for element in root.getchildren():
        strip_xml_namespace(element)

Response: This function removes out namespace data from an ElementTree.


Instruction:
Can you generate the documentation for the following Python 3 function
def element_tree_collection_to_records(tree):
    from .bibrecord import create_record

    records = []
    collection = tree.getroot()
    for record_element in collection.getchildren():
        marcxml = ET.tostring(record_element, encoding="utf-8")
        record, status, errors = create_record(marcxml)
        if errors:
            print(str(status))
        records.append(record)
    return records

Response: Take an ElementTree and converts the nodes into BibRecord records.


Instruction:
Implement a Python 3 function for
taking an ElementTree object and converts the nodes into BibRecord records.

Response:
def element_tree_oai_records(tree, header_subs=None):
    """Take an ElementTree and converts the nodes into BibRecord records.

    This expects a clean OAI response with the tree root as ListRecords
    or GetRecord and record as the subtag like so:
    <ListRecords|GetRecord>
        <record>
            <header>
                <!-- Record Information -->
            </header>
            <metadata>
                <record>
                    <!-- MARCXML -->
                </record>
            </metadata>
        </record>
        <record> ... </record>
    </ListRecords|GetRecord>

    :param tree: ElementTree object corresponding to GetRecord node from
                 OAI request
    :param header_subs: OAI header subfields, if any

    :yield: (record, is_deleted) A tuple, with first a BibRecord found and
             second a boolean value saying if this is a deleted record or not.
    """
    from .bibrecord import record_add_field, create_record

    if not header_subs:
        header_subs = []
    # Make it a tuple, this information should not be changed
    header_subs = tuple(header_subs)

    oai_records = tree.getroot()
    for record_element in oai_records.getchildren():
        header = record_element.find('header')

        # Add to OAI subfield
        datestamp = header.find('datestamp')
        identifier = header.find('identifier')
        identifier = identifier.text

        # The record's subfield is based on header information
        subs = list(header_subs)
        subs.append(("a", identifier))
        subs.append(("d", datestamp.text))

        if "status" in header.attrib and header.attrib["status"] == "deleted":
            # Record was deleted - create delete record
            deleted_record = {}
            record_add_field(deleted_record, "037", subfields=subs)
            yield deleted_record, True
        else:
            marc_root = record_element.find('metadata').find('record')
            marcxml = ET.tostring(marc_root, encoding="utf-8")
            record, status, errors = create_record(marcxml)
            if status == 1:
                # Add OAI request information
                record_add_field(record, "035", subfields=subs)
                yield record, False


Instruction:
Write a Python 3 function for
starting a server instance.

Response:
def run(app=None,
        server='wsgiref',
        host='127.0.0.1',
        port=8080,
        interval=1,
        reloader=False,
        quiet=False,
        plugins=None,
        debug=None, **kargs):
    """ Start a server instance. This method blocks until the server terminates.

        :param app: WSGI application or target string supported by
               :func:`load_app`. (default: :func:`default_app`)
        :param server: Server adapter to use. See :data:`server_names` keys
               for valid names or pass a :class:`ServerAdapter` subclass.
               (default: `wsgiref`)
        :param host: Server address to bind to. Pass ``0.0.0.0`` to listens on
               all interfaces including the external one. (default: 127.0.0.1)
        :param port: Server port to bind to. Values below 1024 require root
               privileges. (default: 8080)
        :param reloader: Start auto-reloading server? (default: False)
        :param interval: Auto-reloader interval in seconds (default: 1)
        :param quiet: Suppress output to stdout and stderr? (default: False)
        :param options: Options passed to the server adapter.
     """
    if NORUN: return
    if reloader and not os.environ.get('BOTTLE_CHILD'):
        import subprocess
        lockfile = None
        try:
            fd, lockfile = tempfile.mkstemp(prefix='bottle.', suffix='.lock')
            os.close(fd)  # We only need this file to exist. We never write to it
            while os.path.exists(lockfile):
                args = [sys.executable] + sys.argv
                environ = os.environ.copy()
                environ['BOTTLE_CHILD'] = 'true'
                environ['BOTTLE_LOCKFILE'] = lockfile
                p = subprocess.Popen(args, env=environ)
                while p.poll() is None:  # Busy wait...
                    os.utime(lockfile, None)  # I am alive!
                    time.sleep(interval)
                if p.poll() != 3:
                    if os.path.exists(lockfile): os.unlink(lockfile)
                    sys.exit(p.poll())
        except KeyboardInterrupt:
            pass
        finally:
            if os.path.exists(lockfile):
                os.unlink(lockfile)
        return

    try:
        if debug is not None: _debug(debug)
        app = app or default_app()
        if isinstance(app, basestring):
            app = load_app(app)
        if not callable(app):
            raise ValueError("Application is not callable: %r" % app)

        for plugin in plugins or []:
            if isinstance(plugin, basestring):
                plugin = load(plugin)
            app.install(plugin)

        if server in server_names:
            server = server_names.get(server)
        if isinstance(server, basestring):
            server = load(server)
        if isinstance(server, type):
            server = server(host=host, port=port, **kargs)
        if not isinstance(server, ServerAdapter):
            raise ValueError("Unknown or unsupported server: %r" % server)

        server.quiet = server.quiet or quiet
        if not server.quiet:
            _stderr("Bottle v%s server starting up (using %s)...\n" %
                    (__version__, repr(server)))
            _stderr("Listening on http://%s:%d/\n" %
                    (server.host, server.port))
            _stderr("Hit Ctrl-C to quit.\n\n")

        if reloader:
            lockfile = os.environ.get('BOTTLE_LOCKFILE')
            bgcheck = FileCheckerThread(lockfile, interval)
            with bgcheck:
                server.run(app)
            if bgcheck.status == 'reload':
                sys.exit(3)
        else:
            server.run(app)
    except KeyboardInterrupt:
        pass
    except (SystemExit, MemoryError):
        raise
    except:
        if not reloader: raise
        if not getattr(server, 'quiet', quiet):
            print_exc()
        time.sleep(interval)
        sys.exit(3)


Instruction:
Write a Python 3 script for
loading values from a dictionary structure.

Response:
def load_dict(self, source, namespace=''):
        """ Load values from a dictionary structure. Nesting can be used to
            represent namespaces.

            >>> c = ConfigDict()
            >>> c.load_dict({'some': {'namespace': {'key': 'value'} } })
            {'some.namespace.key': 'value'}
        """
        for key, value in source.items():
            if isinstance(key, str):
                nskey = (namespace + '.' + key).strip('.')
                if isinstance(value, dict):
                    self.load_dict(value, namespace=nskey)
                else:
                    self[nskey] = value
            else:
                raise TypeError('Key has type %r (not a string)' % type(key))
        return self


Instruction:
Explain what the following Python 3 code does
def json(request, *args, **kwargs):
    # coerce to dictionary
    params = dict(request.GET.items())

    callback = params.pop('callback', None)
    url = params.pop('url', None)

    if not url:
        return HttpResponseBadRequest('Required parameter missing: URL')

    try:
        provider = oembed.site.provider_for_url(url)
        if not provider.provides:
            raise OEmbedMissingEndpoint()
    except OEmbedMissingEndpoint:
        raise Http404('No provider found for %s' % url)

    query = dict([(smart_str(k), smart_str(v)) for k, v in params.items() if v])

    try:
        resource = oembed.site.embed(url, **query)
    except OEmbedException, e:
        raise Http404('Error embedding %s: %s' % (url, str(e)))

    response = HttpResponse(mimetype='application/json')
    json = resource.json

    if callback:
        response.write('%s(%s)' % (defaultfilters.force_escape(callback), json))
    else:
        response.write(json)

    return response

Response: Return a JSON representation of the site s metadata.


Instruction:
Create a Python 3 function for
extracting and return oembed content for given urls

Response:
def consume_json(request):
    """
    Extract and return oembed content for given urls.

    Required GET params:
        urls - list of urls to consume

    Optional GET params:
        width - maxwidth attribute for oembed content
        height - maxheight attribute for oembed content
        template_dir - template_dir to use when rendering oembed

    Returns:
        list of dictionaries with oembed metadata and renderings, json encoded
    """
    client = OEmbedConsumer()

    urls = request.GET.getlist('urls')
    width = request.GET.get('width')
    height = request.GET.get('height')
    template_dir = request.GET.get('template_dir')

    output = {}
    ctx = RequestContext(request)

    for url in urls:
        try:
            provider = oembed.site.provider_for_url(url)
        except OEmbedMissingEndpoint:
            oembeds = None
            rendered = None
        else:
            oembeds = url
            rendered = client.parse_text(url, width, height, context=ctx, template_dir=template_dir)

        output[url] = {
            'oembeds': oembeds,
            'rendered': rendered,
        }

    return HttpResponse(simplejson.dumps(output), mimetype='application/json')


Instruction:
Here you have a function in Python 3, explain what it does
def oembed_schema(request):
    current_domain = Site.objects.get_current().domain
    url_schemes = [] # a list of dictionaries for all the urls we can match
    endpoint = reverse('oembed_json') # the public endpoint for our oembeds
    providers = oembed.site.get_providers()

    for provider in providers:
        # first make sure this provider class is exposed at the public endpoint
        if not provider.provides:
            continue

        match = None
        if isinstance(provider, DjangoProvider):
            # django providers define their regex_list by using urlreversing
            url_pattern = resolver.reverse_dict.get(provider._meta.named_view)

            # this regex replacement is set to be non-greedy, which results
            # in things like /news/*/*/*/*/ -- this is more explicit
            if url_pattern:
                regex = re.sub(r'%\(.+?\)s', '*', url_pattern[0][0][0])
                match = 'http://%s/%s' % (current_domain, regex)
        elif isinstance(provider, HTTPProvider):
            match = provider.url_scheme
        else:
            match = provider.regex

        if match:
            url_schemes.append({
                'type': provider.resource_type,
                'matches': match,
                'endpoint': endpoint
            })

    url_schemes.sort(key=lambda item: item['matches'])

    response = HttpResponse(mimetype='application/json')
    response.write(simplejson.dumps(url_schemes))
    return response

Response: A basic schema for OEmbed - able content.


Instruction:
Explain what the following Python 3 code does
def find_meta(*meta_file_parts, meta_key):
    meta_file = read(*meta_file_parts)
    meta_match = re.search(r"^__{}__ = ['\"]([^'\"]*)['\"]".format(meta_key),
                           meta_file, re.M)
    if meta_match:
        return meta_match.group(1)
    raise RuntimeError("Unable to find __{}__ string.".format(meta_key))

Response: Extract __*meta__ from meta_file.


Instruction:
Make a summary of the following Python 3 code
def main(path):
    '''scan path directory and any subdirectories for valid captain scripts'''
    basepath = os.path.abspath(os.path.expanduser(str(path)))

    echo.h2("Available scripts in {}".format(basepath))
    echo.br()
    for root_dir, dirs, files in os.walk(basepath, topdown=True):
        for f in fnmatch.filter(files, '*.py'):
            try:
                filepath = os.path.join(root_dir, f)

                # super edge case, this makes sure the python script won't start
                # an interactive console session which would cause the session
                # to start and not allow the for loop to complete
                with open(filepath, encoding="UTF-8") as fp:
                    body = fp.read()
                    is_console = "InteractiveConsole" in body
                    is_console = is_console or "code" in body
                    is_console = is_console and "interact(" in body
                    if is_console:
                        continue

                s = captain.Script(filepath)
                if s.can_run_from_cli():
                    rel_filepath = s.call_path(basepath)
                    p = s.parser

                    echo.h3(rel_filepath)

                    desc = p.description
                    if desc:
                        echo.indent(desc, indent=(" " * 4))

                    subcommands = s.subcommands
                    if subcommands:
                        echo.br()
                        echo.indent("Subcommands:", indent=(" " * 4))
                        for sc in subcommands.keys():
                            echo.indent(sc, indent=(" " * 6))

                    echo.br()

            except captain.ParseError:
                pass

            except Exception as e:
                #echo.exception(e)
                #echo.err("Failed to parse {} because {}", f, e.message)
                echo.err("Failed to parse {}", f)
                echo.verbose(e.message)
                echo.br()

Response: scan path directory and any subdirectories for valid captain scripts


Instruction:
Write a Python 3 function that can
return Decimal of the tax rate for given info.

Response:
def get_rate(self, zipcode, city=None, state=None, multiple_rates=False):
        """
        Finds sales tax for given info.
        Returns Decimal of the tax rate, e.g. 8.750.
        """
        data = self.make_request_data(zipcode, city, state)

        r = requests.get(self.url, params=data)
        resp = r.json()

        return self.process_response(resp, multiple_rates)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def make_request_data(self, zipcode, city, state):
        data = {'key': self.api_key,
                'postalcode': str(zipcode),
                'city': city,
                'state': state
        }
        data = ZipTaxClient._clean_request_data(data)
        return data

Response: Make the request params given location data


Instruction:
Can you implement a function in Python 3 that
processes the response from the ZipTax API and return the tax rates

Response:
def process_response(self, resp, multiple_rates):
        """ Get the tax rate from the ZipTax response """
        self._check_for_exceptions(resp, multiple_rates)

        rates = {}
        for result in resp['results']:
            rate = ZipTaxClient._cast_tax_rate(result['taxSales'])
            rates[result['geoCity']] = rate
        if not multiple_rates:
            return rates[list(rates.keys())[0]]
        return rates


Instruction:
Can you write a function in Python 3 where it
checks if there are exceptions that should be raised

Response:
def _check_for_exceptions(self, resp, multiple_rates):
        """ Check if there are exceptions that should be raised """
        if resp['rCode'] != 100:
            raise exceptions.get_exception_for_code(resp['rCode'])(resp)

        results = resp['results']
        if len(results) == 0:
            raise exceptions.ZipTaxNoResults('No results found')
        if len(results) > 1 and not multiple_rates:
            # It's fine if all the taxes are the same
            rates = [result['taxSales'] for result in results]
            if len(set(rates)) != 1:
                raise exceptions.ZipTaxMultipleResults('Multiple results found but requested only one')


Instruction:
Here you have a function in Python 3, explain what it does
def get_all_text(node):
    if node.nodeType == node.TEXT_NODE:
        return node.data
    else:
        text_string = ""
        for child_node in node.childNodes:
            text_string += get_all_text(child_node)
        return text_string

Response: Recursively extract all text from node.


Instruction:
Write a Python 3 function that can
extract a package in a new temporary directory.

Response:
def _extract_packages(self):
        """
        Extract a package in a new temporary directory.
        """
        self.path_unpacked = mkdtemp(prefix="scoap3_package_",
                                     dir=CFG_TMPSHAREDDIR)
        for path in self.retrieved_packages_unpacked:
            scoap3utils_extract_package(path, self.path_unpacked, self.logger)

        return self.path_unpacked


Instruction:
Can you tell what is the following Python 3 function doing
def register(self, provider_class):
        if not issubclass(provider_class, BaseProvider):
            raise TypeError('%s is not a subclass of BaseProvider' % provider_class.__name__)

        if provider_class in self._registered_providers:
            raise AlreadyRegistered('%s is already registered' % provider_class.__name__)

        if issubclass(provider_class, DjangoProvider):
            # set up signal handler for cache invalidation
            signals.post_save.connect(
                self.invalidate_stored_oembeds,
                sender=provider_class._meta.model
            )

        # don't build the regex yet - if not all urlconfs have been loaded
        # and processed at this point, the DjangoProvider instances will fail
        # when attempting to reverse urlpatterns that haven't been created.
        # Rather, the regex-list will be populated once, on-demand.
        self._registered_providers.append(provider_class)

        # flag for re-population
        self.invalidate_providers()

Response: Registers a provider with the site.


Instruction:
Can you tell what is the following Python 3 function doing
def unregister(self, provider_class):
        if not issubclass(provider_class, BaseProvider):
            raise TypeError('%s must be a subclass of BaseProvider' % provider_class.__name__)

        if provider_class not in self._registered_providers:
            raise NotRegistered('%s is not registered' % provider_class.__name__)

        self._registered_providers.remove(provider_class)

        # flag for repopulation
        self.invalidate_providers()

Response: Unregisters a provider from the site.


Instruction:
Given the following Python 3 function, write the documentation
def populate(self):
        self._registry = {}

        for provider_class in self._registered_providers:
            instance = provider_class()
            self._registry[instance] = instance.regex

        for stored_provider in StoredProvider.objects.active():
            self._registry[stored_provider] = stored_provider.regex

        self._populated = True

Response:
Populate the internal registry with the regexes for each
        provider instance


Instruction:
Write a Python 3 function for
finding the right provider for a URL

Response:
def provider_for_url(self, url):
        """
        Find the right provider for a URL
        """
        for provider, regex in self.get_registry().items():
            if re.match(regex, url) is not None:
                return provider

        raise OEmbedMissingEndpoint('No endpoint matches URL: %s' % url)


Instruction:
Can you generate the documentation for the following Python 3 function
def embed(self, url, **kwargs):
        try:
            # first figure out the provider
            provider = self.provider_for_url(url)
        except OEmbedMissingEndpoint:
            raise
        else:
            try:
                # check the database for a cached response, because of certain
                # race conditions that exist with get_or_create(), do a filter
                # lookup and just grab the first item
                stored_match = StoredOEmbed.objects.filter(
                    match=url,
                    maxwidth=kwargs.get('maxwidth', None),
                    maxheight=kwargs.get('maxheight', None),
                    date_expires__gte=datetime.datetime.now())[0]
                return OEmbedResource.create_json(stored_match.response_json)
            except IndexError:
                # query the endpoint and cache response in db
                # prevent None from being passed in as a GET param
                params = dict([(k, v) for k, v in kwargs.items() if v])

                # request an oembed resource for the url
                resource = provider.request_resource(url, **params)

                try:
                    cache_age = int(resource.cache_age)
                    if cache_age < MIN_OEMBED_TTL:
                        cache_age = MIN_OEMBED_TTL
                except:
                    cache_age = DEFAULT_OEMBED_TTL

                date_expires = datetime.datetime.now() + datetime.timedelta(seconds=cache_age)

                stored_oembed, created = StoredOEmbed.objects.get_or_create(
                    match=url,
                    maxwidth=kwargs.get('maxwidth', None),
                    maxheight=kwargs.get('maxheight', None))

                stored_oembed.response_json = resource.json
                stored_oembed.resource_type = resource.type
                stored_oembed.date_expires = date_expires

                if resource.content_object:
                    stored_oembed.content_object = resource.content_object

                stored_oembed.save()
                return resource

Response: Get a resource from the database and return it.


Instruction:
Can you implement a function in Python 3 that
loads up StoredProviders from url if it is an oembed scheme

Response:
def autodiscover(self, url):
        """
        Load up StoredProviders from url if it is an oembed scheme
        """
        headers, response = fetch_url(url)
        if headers['content-type'].split(';')[0] in ('application/json', 'text/javascript'):
            provider_data = json.loads(response)
            return self.store_providers(provider_data)


Instruction:
Create a Python 3 function for
storing all providers in the database.

Response:
def store_providers(self, provider_data):
        """
        Iterate over the returned json and try to sort out any new providers
        """
        if not hasattr(provider_data, '__iter__'):
            raise OEmbedException('Autodiscovered response not iterable')

        provider_pks = []

        for provider in provider_data:
            if 'endpoint' not in provider or \
               'matches' not in provider:
                continue

            resource_type = provider.get('type')
            if resource_type not in RESOURCE_TYPES:
                continue

            stored_provider, created = StoredProvider.objects.get_or_create(
                wildcard_regex=provider['matches']
            )

            if created:
                stored_provider.endpoint_url = relative_to_full(
                    provider['endpoint'],
                    provider['matches']
                )
                stored_provider.resource_type = resource_type
                stored_provider.save()

            provider_pks.append(stored_provider.pk)

        return StoredProvider.objects.filter(pk__in=provider_pks)


Instruction:
How would you implement a function in Python 3 that
requests an OEmbedResource for a given url.

Response:
def request_resource(self, url, **kwargs):
        """
        Request an OEmbedResource for a given url.  Some valid keyword args:
        - format
        - maxwidth
        - maxheight
        """
        params = kwargs

        params['url'] = url
        params['format'] = 'json'

        if '?' in self.endpoint_url:
            url_with_qs = '%s&%s' % (self.endpoint_url.rstrip('&'), urlencode(params))
        else:
            url_with_qs = "%s?%s" % (self.endpoint_url, urlencode(params))

        headers, raw_response = self._fetch(url_with_qs)
        resource = self.convert_to_resource(headers, raw_response, params)

        return resource


Instruction:
Given the following Python 3 function, write the documentation
def _image_field(self):
        for field in self.model._meta.fields:
            if isinstance(field, ImageField):
                return field.name

Response: Try to automatically detect an image field


Instruction:
How would you explain what the following Python 3 function does
def _date_field(self):
        for field in self.model._meta.fields:
            if isinstance(field, (DateTimeField, DateField)):
                return field.name

Response: Try to automatically detect an image field


Instruction:
How would you implement a function in Python 3 that
builds a regex that matches the object_id of the object.

Response:
def _build_regex(self):
        """
        Performs a reverse lookup on a named view and generates
        a list of regexes that match that object.  It generates
        regexes with the domain name included, using sites provided
        by the get_sites() method.

        >>> regex = provider.regex
        >>> regex.pattern
        'http://(www2.kusports.com|www2.ljworld.com|www.lawrence.com)/photos/(?P<year>\\d{4})/(?P<month>\\w{3})/(?P<day>\\d{1,2})/(?P<object_id>\\d+)/$'
        """
        # get the regexes from the urlconf
        url_patterns = resolver.reverse_dict.get(self._meta.named_view)

        try:
            regex = url_patterns[1]
        except TypeError:
            raise OEmbedException('Error looking up %s' % self._meta.named_view)

        # get a list of normalized domains
        cleaned_sites = self.get_cleaned_sites()

        site_regexes = []

        for site in self.get_sites():
            site_regexes.append(cleaned_sites[site.pk][0])

        # join the sites together with the regex 'or'
        sites = '|'.join(site_regexes)

        # create URL-matching regexes for sites
        regex = re.compile('(%s)/%s' % (sites, regex))

        return regex


Instruction:
Can you create a Python 3 function that
gives a URL for any of our sites try and match it to one and return the domain & name of the match. If no match is found return the current. nacute_name.

Response:
def provider_from_url(self, url):
        """
        Given a URL for any of our sites, try and match it to one, returning
        the domain & name of the match.  If no match is found, return current.

        Returns a tuple of domain, site name -- used to determine 'provider'
        """
        domain = get_domain(url)
        site_tuples = self.get_cleaned_sites().values()
        for domain_re, name, normalized_domain in site_tuples:
            if re.match(domain_re, domain):
                return normalized_domain, name
        site = Site.objects.get_current()
        return site.domain, site.name


Instruction:
Create a Python 3 function to
extract the named parameters from a url regex.

Response:
def get_params(self, url):
        """
        Extract the named parameters from a url regex.  If the url regex does not contain
        named parameters, they will be keyed _0, _1, ...

        * Named parameters
        Regex:
        /photos/^(?P<year>\d{4})/(?P<month>\w{3})/(?P<day>\d{1,2})/(?P<object_id>\d+)/

        URL:
        http://www2.ljworld.com/photos/2009/oct/11/12345/

        Return Value:
        {u'day': '11', u'month': 'oct', u'object_id': '12345', u'year': '2009'}

        * Unnamed parameters
        Regex:
        /blah/([\w-]+)/(\d+)/

        URL:
        http://www.example.com/blah/hello/123/

        Return Value:
        {u'_0': 'hello', u'_1': '123'}
        """
        match = re.match(self.regex, url)
        if match is not None:
            params = match.groupdict()
            if not params:
                params = {}
                for i, group in enumerate(match.groups()[1:]):
                    params['_%s' % i] = group
            return params

        raise OEmbedException('No regex matched the url %s' % (url))


Instruction:
Write a Python 3 function that can
get an object from the database

Response:
def get_object(self, url):
        """
        Fields to match is a mapping of url params to fields, so for
        the photos example above, it would be:

        fields_to_match = { 'object_id': 'id' }

        This procedure parses out named params from a URL and then uses
        the fields_to_match dictionary to generate a query.
        """
        params = self.get_params(url)
        query = {}
        for key, value in self._meta.fields_to_match.iteritems():
            try:
                query[value] = params[key]
            except KeyError:
                raise OEmbedException('%s was not found in the urlpattern parameters.  Valid names are: %s' % (key, ', '.join(params.keys())))

        try:
            obj = self.get_queryset().get(**query)
        except self._meta.model.DoesNotExist:
            raise OEmbedException('Requested object not found')

        return obj


Instruction:
Can you implement a function in Python 3 that
generates the html attribute of an oembed resource using a template.

Response:
def render_html(self, obj, context=None):
        """
        Generate the 'html' attribute of an oembed resource using a template.
        Sort of a corollary to the parser's render_oembed method.  By default,
        the current mapping will be passed in as the context.

        OEmbed templates are stored in:

        oembed/provider/[app_label]_[model].html

        -- or --

        oembed/provider/media_video.html
        """
        provided_context = context or Context()
        context = RequestContext(mock_request())
        context.update(provided_context)

        context.push()
        context[self._meta.context_varname] = obj
        rendered = render_to_string(self._meta.template_name, context)
        context.pop()
        return rendered


Instruction:
Implement a Python 3 function for
mapping an attribute to the object.

Response:
def map_attr(self, mapping, attr, obj):
        """
        A kind of cheesy method that allows for callables or attributes to
        be used interchangably
        """
        if attr not in mapping and hasattr(self, attr):
            if not callable(getattr(self, attr)):
                mapping[attr] = getattr(self, attr)
            else:
                mapping[attr] = getattr(self, attr)(obj)


Instruction:
Can you implement a function in Python 3 that
returns an ImageFileField instance

Response:
def get_image(self, obj):
        """
        Return an ImageFileField instance
        """
        if self._meta.image_field:
            return getattr(obj, self._meta.image_field)


Instruction:
Given the following Python 3 function, write the documentation
def map_to_dictionary(self, url, obj, **kwargs):
        maxwidth = kwargs.get('maxwidth', None)
        maxheight = kwargs.get('maxheight', None)

        provider_url, provider_name = self.provider_from_url(url)

        mapping = {
            'version': '1.0',
            'url': url,
            'provider_name': provider_name,
            'provider_url': provider_url,
            'type': self.resource_type
        }

        # a hook
        self.preprocess(obj, mapping, **kwargs)

        # resize image if we have a photo, otherwise use the given maximums
        if self.resource_type == 'photo' and self.get_image(obj):
            self.resize_photo(obj, mapping, maxwidth, maxheight)
        elif self.resource_type in ('video', 'rich', 'photo'):
            width, height = size_to_nearest(
                maxwidth,
                maxheight,
                self._meta.valid_sizes,
                self._meta.force_fit
            )
            mapping.update(width=width, height=height)

        # create a thumbnail
        if self.get_image(obj):
            self.thumbnail(obj, mapping)

        # map attributes to the mapping dictionary.  if the attribute is
        # a callable, it must have an argument signature of
        # (self, obj)
        for attr in ('title', 'author_name', 'author_url', 'html'):
            self.map_attr(mapping, attr, obj)

        # fix any urls
        if 'url' in mapping:
            mapping['url'] = relative_to_full(mapping['url'], url)

        if 'thumbnail_url' in mapping:
            mapping['thumbnail_url'] = relative_to_full(mapping['thumbnail_url'], url)

        if 'html' not in mapping and mapping['type'] in ('video', 'rich'):
            mapping['html'] = self.render_html(obj, context=Context(mapping))

        # a hook
        self.postprocess(obj, mapping, **kwargs)

        return mapping

Response: Map the object to a dictionary.


Instruction:
Can you write a function in Python 3 where it
requests an OEmbedResource for a given url.

Response:
def request_resource(self, url, **kwargs):
        """
        Request an OEmbedResource for a given url.  Some valid keyword args:
        - format
        - maxwidth
        - maxheight
        """
        obj = self.get_object(url)

        mapping = self.map_to_dictionary(url, obj, **kwargs)

        resource = OEmbedResource.create(mapping)
        resource.content_object = obj

        return resource


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_object(self, url, month_format='%b', day_format='%d'):
        params = self.get_params(url)
        try:
            year = params[self._meta.year_part]
            month = params[self._meta.month_part]
            day = params[self._meta.day_part]
        except KeyError:
            try:
                # named lookups failed, so try to get the date using the first
                # three parameters
                year, month, day = params['_0'], params['_1'], params['_2']
            except KeyError:
                raise OEmbedException('Error extracting date from url parameters')

        try:
            tt = time.strptime('%s-%s-%s' % (year, month, day),
                               '%s-%s-%s' % ('%Y', month_format, day_format))
            date = datetime.date(*tt[:3])
        except ValueError:
            raise OEmbedException('Error parsing date from: %s' % url)

        # apply the date-specific lookups
        if isinstance(self._meta.model._meta.get_field(self._meta.date_field), DateTimeField):
            min_date = datetime.datetime.combine(date, datetime.time.min)
            max_date = datetime.datetime.combine(date, datetime.time.max)
            query = {'%s__range' % self._meta.date_field: (min_date, max_date)}
        else:
            query = {self._meta.date_field: date}

        # apply the regular search lookups
        for key, value in self._meta.fields_to_match.iteritems():
            try:
                query[value] = params[key]
            except KeyError:
                raise OEmbedException('%s was not found in the urlpattern parameters.  Valid names are: %s' % (key, ', '.join(params.keys())))

        try:
            obj = self.get_queryset().get(**query)
        except self._meta.model.DoesNotExist:
            raise OEmbedException('Requested object not found')

        return obj

Response: Parses the date from a url and uses it in the query.


Instruction:
Explain what the following Python 3 code does
def get_record(self):
        self.recid = self.get_recid()
        self.remove_controlfields()
        self.update_system_numbers()
        self.add_systemnumber("Inspire", recid=self.recid)
        self.add_control_number("003", "SzGeCERN")
        self.update_collections()
        self.update_languages()
        self.update_reportnumbers()
        self.update_authors()
        self.update_journals()
        self.update_subject_categories("INSPIRE", "SzGeCERN", "categories_cds")
        self.update_pagenumber()
        self.update_notes()
        self.update_experiments()
        self.update_isbn()
        self.update_dois()
        self.update_links_and_ffts()
        self.update_date()
        self.update_date_year()
        self.update_hidden_notes()
        self.update_oai_info()
        self.update_cnum()
        self.update_conference_info()

        self.fields_list = [
            "909", "541", "961",
            "970", "690", "695",
            "981",
        ]
        self.strip_fields()

        if "ANNOUNCEMENT" in self.collections:
            self.update_conference_111()
            self.update_conference_links()
            record_add_field(self.record, "690", ind1="C", subfields=[("a", "CONFERENCE")])

        if "THESIS" in self.collections:
            self.update_thesis_information()
            self.update_thesis_supervisors()

        if "PROCEEDINGS" in self.collections:
            # Special proceeding syntax
            self.update_title_to_proceeding()
            self.update_author_to_proceeding()
            record_add_field(self.record, "690", ind1="C", subfields=[("a", "CONFERENCE")])

        # 690 tags
        if self.tag_as_cern:
            record_add_field(self.record, "690", ind1="C", subfields=[("a", "CERN")])

        return self.record

Response: Override the base method to get the record.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def update_oai_info(self):
        for field in record_get_field_instances(self.record, '909', ind1="C", ind2="O"):
            new_subs = []
            for tag, value in field[0]:
                if tag == "o":
                    new_subs.append(("a", value))
                else:
                    new_subs.append((tag, value))
                if value in ["CERN", "CDS", "ForCDS"]:
                    self.tag_as_cern = True
            record_add_field(self.record, '024', ind1="8", subfields=new_subs)
        record_delete_fields(self.record, '909')

Response: Update the 909 OAI info to 035.


Instruction:
Can you tell what is the following Python 3 function doing
def update_cnum(self):
        if "ConferencePaper" not in self.collections:
            cnums = record_get_field_values(self.record, '773', code="w")
            for cnum in cnums:
                cnum_subs = [
                    ("9", "INSPIRE-CNUM"),
                    ("a", cnum)
                ]
                record_add_field(self.record, "035", subfields=cnum_subs)

Response: Check if we shall add cnum in 035.


Instruction:
Given the following Python 3 function, write the documentation
def update_hidden_notes(self):
        if not self.tag_as_cern:
            notes = record_get_field_instances(self.record,
                                               tag="595")
            for field in notes:
                for dummy, value in field[0]:
                    if value == "CDS":
                        self.tag_as_cern = True
        record_delete_fields(self.record, tag="595")

Response: Remove hidden notes and tag a CERN if detected.


Instruction:
Can you tell what is the following Python 3 function doing
def update_system_numbers(self):
        scn_035_fields = record_get_field_instances(self.record, '035')
        new_fields = []
        for field in scn_035_fields:
            subs = field_get_subfields(field)
            if '9' in subs:
                if subs['9'][0].lower() == "cds" and subs.get('a'):
                    self.add_control_number("001", subs.get('a')[0])
                if subs['9'][0].lower() in ["inspire", "spirestex", "inspiretex", "desy", "cds"]:
                    continue
            new_fields.append(field_get_subfield_instances(field))
        record_delete_fields(self.record, tag="035")
        for field in new_fields:
            record_add_field(self.record, tag="035", subfields=field)

Response: Update the system numbers of the 3035 records.


Instruction:
Create a Python 3 function to
update the list of collections that this record belongs to.

Response:
def update_collections(self):
        """Try to determine which collections this record should belong to."""
        for value in record_get_field_values(self.record, '980', code='a'):
            if 'NOTE' in value.upper():
                self.collections.add('NOTE')
            if 'THESIS' in value.upper():
                self.collections.add('THESIS')

            if 'PUBLISHED' in value.upper():
                self.collections.add('ARTICLE')

            if 'CONFERENCES' in value.upper():
                self.collections.add('ANNOUNCEMENT')

            if 'PROCEEDINGS' in value.upper():
                self.collections.add('PROCEEDINGS')
            elif 'CONFERENCEPAPER' in value.upper() and \
                 "ConferencePaper" not in self.collections:
                self.collections.add('ConferencePaper')
                if self.is_published() and "ARTICLE" not in self.collections:
                    self.collections.add('ARTICLE')
                else:
                    self.collections.add('PREPRINT')

            if "HIDDEN" in value.upper():
                self.hidden = True

        # Clear out any existing ones.
        record_delete_fields(self.record, "980")

        if not self.collections:
            self.collections.add('PREPRINT')

        for collection in self.collections:
            record_add_field(self.record,
                             tag='980',
                             subfields=[('a', collection)])
            if collection in self.collection_base:
                subs = [('a', self.collection_base[collection])]
                record_add_field(self.record,
                                 tag='960',
                                 subfields=subs)


Instruction:
Implement a function in Python 3 to
remove INSPIRE specific notes.

Response:
def update_notes(self):
        """Remove INSPIRE specific notes."""
        fields = record_get_field_instances(self.record, '500')
        for field in fields:
            subs = field_get_subfields(field)
            for sub in subs.get('a', []):
                sub = sub.strip()  # remove any spaces before/after
                if sub.startswith("*") and sub.endswith("*"):
                    record_delete_field(self.record, tag="500",
                                        field_position_global=field[4])


Instruction:
Implement a function in Python 3 to
move title info from 245 to 111 proceeding style.

Response:
def update_title_to_proceeding(self):
        """Move title info from 245 to 111 proceeding style."""
        titles = record_get_field_instances(self.record,
                                            tag="245")
        for title in titles:
            subs = field_get_subfields(title)
            new_subs = []
            if "a" in subs:
                new_subs.append(("a", subs['a'][0]))
            if "b" in subs:
                new_subs.append(("c", subs['b'][0]))
            record_add_field(self.record,
                             tag="111",
                             subfields=new_subs)
        record_delete_fields(self.record, tag="245")
        record_delete_fields(self.record, tag="246")


Instruction:
Explain what the following Python 3 code does
def update_experiments(self):
        # 693 Remove if 'not applicable'
        for field in record_get_field_instances(self.record, '693'):
            subs = field_get_subfields(field)
            acc_experiment = subs.get("e", [])
            if not acc_experiment:
                acc_experiment = subs.get("a", [])
                if not acc_experiment:
                    continue
            experiment = acc_experiment[-1]

            # Handle special case of leading experiments numbers NA-050 -> NA 50
            e_suffix = ""
            if "-NA-" in experiment or \
               "-RD-" in experiment or \
               "-WA-" in experiment:
                splitted_experiment = experiment.split("-")
                e_suffix = "-".join(splitted_experiment[2:])
                if e_suffix.startswith("0"):
                    e_suffix = e_suffix[1:]
                experiment = "-".join(splitted_experiment[:2])  # only CERN-NA

            translated_experiment = self.get_config_item(experiment,
                                                         "experiments")
            if not translated_experiment:
                continue
            new_subs = []
            if "---" in translated_experiment:
                experiment_a, experiment_e = translated_experiment.split("---")
                new_subs.append(("a", experiment_a.replace("-", " ")))
            else:
                experiment_e = translated_experiment
            new_subs.append(("e", experiment_e.replace("-", " ") + e_suffix))
            record_delete_field(self.record, tag="693",
                                field_position_global=field[4])
            record_add_field(self.record, "693", subfields=new_subs)

Response: Update the experiments of the CERN record.


Instruction:
Explain what the following Python 3 code does
def update_authors(self):
        author_names = record_get_field_instances(self.record, '100')
        author_names.extend(record_get_field_instances(self.record, '700'))
        for field in author_names:
            subs = field_get_subfields(field)
            for idx, (key, value) in enumerate(field[0]):
                if key == 'a':
                    field[0][idx] = ('a', value.replace(".", " ").strip())
                elif key == 'v':
                    del field[0][idx]
            if subs.get("u", None) == "CERN":
                self.tag_as_cern = True

Response: Update the author names of the archive.


Instruction:
Make a summary of the following Python 3 code
def update_isbn(self):
        isbns = record_get_field_instances(self.record, '020')
        for field in isbns:
            for idx, (key, value) in enumerate(field[0]):
                if key == 'a':
                    field[0][idx] = ('a', value.replace("-", "").strip())

Response: Remove dashes from ISBN.


Instruction:
How would you explain what the following Python 3 function does
def update_dois(self):
        dois = record_get_field_instances(self.record, '024', ind1="7")
        all_dois = {}
        for field in dois:
            subs = field_get_subfield_instances(field)
            subs_dict = dict(subs)
            if subs_dict.get('a'):
                if subs_dict['a'] in all_dois:
                    record_delete_field(self.record, tag='024', ind1='7', field_position_global=field[4])
                    continue
                all_dois[subs_dict['a']] = field

Response: Update all BibMatch DOIs.


Instruction:
Can you tell what is the following Python 3 function doing
def update_journals(self):
        for field in record_get_field_instances(self.record, '773'):
            subs = field_get_subfield_instances(field)
            new_subs = []
            volume_letter = ""
            journal_name = ""
            for idx, (key, value) in enumerate(subs):
                if key == 'p':
                    journal_name = self.get_config_item(value, "journals", allow_substring=False)
                    # Make sure journal names have the form (dot)(space) (I know it's horrible)
                    journal_name = journal_name.replace('. ', '.').replace('.', '. ').replace('. ,', '.,').strip()
                elif key == 'v':
                    volume_letter = value
                else:
                    new_subs.append((key, value))

            if not journal_name == "PoS":
                # Special handling of journal name and volumes, except PoS
                letter = return_letters_from_string(volume_letter)
                if letter:
                    journal_name = "{0} {1}".format(journal_name, letter)
                    volume_letter = volume_letter.strip(letter)

            if journal_name:
                new_subs.append(("p", journal_name))
            if volume_letter:
                new_subs.append(("v", volume_letter))
            record_delete_field(self.record, tag="773",
                                field_position_global=field[4])
            record_add_field(self.record, "773", subfields=new_subs)

Response: Update the journals field in the record.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def update_thesis_supervisors(self):
        for field in record_get_field_instances(self.record, '701'):
            subs = list(field[0])
            subs.append(("e", "dir."))
            record_add_field(self.record, '700', subfields=subs)
        record_delete_fields(self.record, '701')

Response: 700 -> 701 Thesis supervisors.


Instruction:
Can you write a function in Python 3 where it
updates the page number of the records.

Response:
def update_pagenumber(self):
        """300 page number."""
        pages = record_get_field_instances(self.record, '300')
        for field in pages:
            for idx, (key, value) in enumerate(field[0]):
                if key == 'a':
                    field[0][idx] = ('a', "{0} p".format(value))


Instruction:
Can you implement a function in Python 3 that
checks if the record has already been published.

Response:
def is_published(self):
        """Check fields 980 and 773 to see if the record has already been published.

        :return: True is published, else False
        """
        field773 = record_get_field_instances(self.record, '773')
        for f773 in field773:
            if 'c' in field_get_subfields(f773):
                return True
        return False


Instruction:
Write a Python 3 script to
update links and FFTs for the current ISO.

Response:
def update_links_and_ffts(self):
        """FFT (856) Dealing with files."""
        for field in record_get_field_instances(self.record,
                                                tag='856',
                                                ind1='4'):
            subs = field_get_subfields(field)
            newsubs = []
            url = subs.get("u", [])

            if not url:
                record_delete_field(self.record, '856', ind1='4',
                                    field_position_global=field[4])
                continue
            url = url[0]
            if "inspirehep.net/record" in url and url.endswith("pdf"):
                # We have an FFT from INSPIRE
                newsubs.append(('a', url))
                description = subs.get("y", [])
                if description:
                    newsubs.append(('d', description[0]))
                if newsubs:
                    record_add_field(self.record, 'FFT', subfields=newsubs)
                    record_delete_field(self.record, '856', ind1='4',
                                        field_position_global=field[4])
            else:
                # Remove $w
                for idx, (key, value) in enumerate(field[0]):
                    if key == 'w':
                        del field[0][idx]


Instruction:
Can you generate the documentation for the following Python 3 function
def update_languages(self):
        language_fields = record_get_field_instances(self.record, '041')
        language = "eng"
        record_delete_fields(self.record, "041")
        for field in language_fields:
            subs = field_get_subfields(field)
            if 'a' in subs:
                language = self.get_config_item(subs['a'][0], "languages")
                break
        new_subs = [('a', language)]
        record_add_field(self.record, "041", subfields=new_subs)

Response: Update the languages field in the record.


Instruction:
Make a summary of the following Python 3 code
def pathjoin(*args, **kwargs):
    log.debug('pathjoin: %r' % list(args))

    def _pathjoin(*args, **kwargs):
        len_ = len(args) - 1
        if len_ < 0:
            raise Exception('no args specified')
        elif len_ == 0:
            if not isinstance(args, basestring):
                if hasattr(args, '__iter__'):
                    _args = args
                    _args
                    args = args[0]
        for i, arg in enumerate(args):
            if not i:
                yield arg.rstrip('/')
            elif i == len_:
                yield arg.lstrip('/')
            else:
                yield arg.strip('/')
    joined_path = u'/'.join(_pathjoin(*args))
    return sanitize_path(joined_path)

Response: Join the paths of a single object with the key - value pairs.


Instruction:
Write a Python 3 function that can
generate the HTML directory listing for the given directory.

Response:
def generate_dirlist_html(FS, filepath):
    """
    Generate directory listing HTML

    Arguments:
        FS (FS): filesystem object to read files from
        filepath (str): path to generate directory listings for

    Keyword Arguments:
        list_dir (callable: list[str]): list file names in a directory
        isdir (callable: bool): os.path.isdir

    Yields:
        str: lines of an HTML table
    """
    yield '
'
    if filepath == '/':
        filepath = ''
    for name in FS.listdir(filepath):
        full_path = pathjoin(filepath, name)
        if FS.isdir(full_path):
            full_path = full_path + '/'
        yield u'
{0}
'.format(
            cgi.escape(full_path))  # TODO XXX
    yield '
'


Instruction:
Create a Python 3 function to
check if files are not being uploaded to server.

Response:
def check_pkgs_integrity(filelist, logger, ftp_connector,
                         timeout=120, sleep_time=10):
    """
    Checks if files are not being uploaded to server.
    @timeout - time after which the script will register an error.
    """
    ref_1 = []
    ref_2 = []
    i = 1
    print >> sys.stdout, "\nChecking packages integrity."
    for filename in filelist:
        # ref_1.append(self.get_remote_file_size(filename))
        get_remote_file_size(ftp_connector, filename, ref_1)
    print >> sys.stdout, "\nGoing to sleep for %i sec." % (sleep_time,)
    time.sleep(sleep_time)

    while sleep_time*i < timeout:
        for filename in filelist:
            # ref_2.append(self.get_remote_file_size(filename))
            get_remote_file_size(ftp_connector, filename, ref_2)
        if ref_1 == ref_2:
            print >> sys.stdout, "\nIntegrity OK:)"
            logger.info("Packages integrity OK.")
            break
        else:
            print >> sys.stdout, "\nWaiting %d time for itegrity..." % (i,)
            logger.info("\nWaiting %d time for itegrity..." % (i,))
            i += 1
            ref_1, ref_2 = ref_2, []
            time.sleep(sleep_time)
    else:
        not_finished_files = []
        for count, val1 in enumerate(ref_1):
            if val1 != ref_2[count]:
                not_finished_files.append(filelist[count])

        print >> sys.stdout, "\nOMG, OMG something wrong with integrity."
        logger.error("Integrity check faild for files %s"
                     % (not_finished_files,))


Instruction:
How would you code a function in Python 3 to
normalize a query string.

Response:
def normalize_query(query_string, terms=TERMS, norm_space=NORM_SPACE):
    """
    Example:
    >>> normalize_query('  some random  words "with   quotes  " and   spaces')
    ['some', 'random', 'words', 'with quotes', 'and', 'spaces']
    """
    return [
        norm_space(' ', (t[0] or t[1]).strip()) for t in terms(query_string)]


Instruction:
Here you have a function in Python 3, explain what it does
def collapse_initials(name):
    if len(name.split()) > 1:
        name = re.sub(r'([A-Z]\.) +(?=[A-Z]\.)', r'\1', name)
    return name

Response:
Removes the space between initials.
        eg T. A. --> T. A.


Instruction:
Make a summary of the following Python 3 code
def fix_name_capitalization(lastname, givennames):
    lastnames = lastname.split()
    if len(lastnames) == 1:
        if '-' in lastname:
            names = lastname.split('-')
            names = map(lambda a: a[0] + a[1:].lower(), names)
            lastname = '-'.join(names)
        else:
            lastname = lastname[0] + lastname[1:].lower()
    else:
        names = []
        for name in lastnames:
            if re.search(r'[A-Z]\.', name):
                names.append(name)
            else:
                names.append(name[0] + name[1:].lower())
        lastname = ' '.join(names)
        lastname = collapse_initials(lastname)
    names = []
    for name in givennames:
        if re.search(r'[A-Z]\.', name):
            names.append(name)
        else:
            names.append(name[0] + name[1:].lower())
    givennames = ' '.join(names)
    return lastname, givennames

Response: Converts capital letters to lower keeps first letter capital.


Instruction:
Write a Python 3 function that can
scan a block of text and extracts oembed data on any urls and returns a list of dictionaries containing the oembed data.

Response:
def extract_oembeds(self, text, maxwidth=None, maxheight=None, resource_type=None):
        """
        Scans a block of text and extracts oembed data on any urls,
        returning it in a list of dictionaries
        """
        parser = text_parser()
        urls = parser.extract_urls(text)
        return self.handle_extracted_urls(urls, maxwidth, maxheight, resource_type)


Instruction:
Can you tell what is the following Python 3 function doing
def strip(self, text, *args, **kwargs):
        if OEMBED_DEFAULT_PARSE_HTML:
            extracted = self.extract_oembeds_html(text, *args, **kwargs)
        else:
            extracted = self.extract_oembeds(text, *args, **kwargs)

        matches = [r['original_url'] for r in extracted]
        match_handler = lambda m: m.group() not in matches and m.group() or ''

        return re.sub(URL_RE, match_handler, text)

Response: Strip the OEMBED tags from the given text.


Instruction:
How would you explain what the following Python 3 function does
def autodiscover():
    import imp
    from django.conf import settings

    for app in settings.INSTALLED_APPS:
        try:
            app_path = __import__(app, {}, {}, [app.split('.')[-1]]).__path__
        except AttributeError:
            continue

        try:
            imp.find_module('oembed_providers', app_path)
        except ImportError:
            continue

        __import__("%s.oembed_providers" % app)

Response: Autodiscover the provider index.


Instruction:
Write a Python 3 script to
pass in a list of options, promt the user to select one, and return the selected option or None

Response:
def select(options=None):
    """ pass in a list of options, promt the user to select one, and return the selected option or None """
    if not options:
        return None
    width = len(str(len(options)))
    for x,option in enumerate(options):
        sys.stdout.write('{:{width}}) {}\n'.format(x+1,option, width=width))

    sys.stdout.write('{:>{width}} '.format('#?', width=width+1))
    sys.stdout.flush()
    if sys.stdin.isatty():
        # regular prompt
        try:
            response = raw_input().strip()
        except (EOFError, KeyboardInterrupt):
            # handle ctrl-d, ctrl-c
            response = ''
    else:
        # try connecting to current tty, when using pipes
        sys.stdin = open("/dev/tty")
        try:
            response = ''
            while True:
                response += sys.stdin.read(1)
                if response.endswith('\n'):
                    break
        except (EOFError, KeyboardInterrupt):
            sys.stdout.flush()
            pass
    try:
        response = int(response) - 1
    except ValueError:
        return None
    if response < 0 or response >= len(options):
        return None
    return options[response]


Instruction:
How would you implement a function in Python 3 that
transforms the argparse arguments from Namespace to dict and then to Bunch Therefore it is not necessary to access the arguments using the dict syntax The settings can be called like regular vars on the settings object

Response:
def main():
    argparser = ArgumentParser()

    subparsers = argparser.add_subparsers(dest='selected_subparser')

    all_parser = subparsers.add_parser('all')
    elsevier_parser = subparsers.add_parser('elsevier')
    oxford_parser = subparsers.add_parser('oxford')
    springer_parser = subparsers.add_parser('springer')

    all_parser.add_argument('--update-credentials', action='store_true')

    elsevier_parser.add_argument('--run-locally', action='store_true')
    elsevier_parser.add_argument('--package-name')
    elsevier_parser.add_argument('--path')
    elsevier_parser.add_argument('--CONSYN', action='store_true')
    elsevier_parser.add_argument('--update-credentials', action='store_true')
    elsevier_parser.add_argument('--extract-nations', action='store_true')

    oxford_parser.add_argument('--dont-empty-ftp', action='store_true')
    oxford_parser.add_argument('--package-name')
    oxford_parser.add_argument('--path')
    oxford_parser.add_argument('--update-credentials', action='store_true')
    oxford_parser.add_argument('--extract-nations', action='store_true')

    springer_parser.add_argument('--package-name')
    springer_parser.add_argument('--path')
    springer_parser.add_argument('--update-credentials', action='store_true')
    springer_parser.add_argument('--extract-nations', action='store_true')

    '''
    Transforms the argparse arguments from Namespace to dict and then to Bunch
    Therefore it is not necessary to access the arguments using the dict syntax
    The settings can be called like regular vars on the settings object
    '''

    settings = Bunch(vars(argparser.parse_args()))

    call_package(settings)


Instruction:
Implement a function in Python 3 to
read a dom xml element in oaidc format and returns the bibrecord object

Response:
def get_record(self, record):
        """ Reads a dom xml element in oaidc format and
            returns the bibrecord object """
        self.document = record
        rec = create_record()
        language = self._get_language()
        if language and language != 'en':
            record_add_field(rec, '041', subfields=[('a', language)])
        publisher = self._get_publisher()
        date = self._get_date()
        if publisher and date:
            record_add_field(rec, '260', subfields=[('b', publisher),
                                                    ('c', date)])
        elif publisher:
            record_add_field(rec, '260', subfields=[('b', publisher)])
        elif date:
            record_add_field(rec, '260', subfields=[('c', date)])
        title = self._get_title()
        if title:
            record_add_field(rec, '245', subfields=[('a', title)])
        record_copyright = self._get_copyright()
        if record_copyright:
            record_add_field(rec, '540', subfields=[('a', record_copyright)])
        subject = self._get_subject()
        if subject:
            record_add_field(rec, '650', ind1='1', ind2='7', subfields=[('a', subject),
                                                                        ('2', 'PoS')])
        authors = self._get_authors()
        first_author = True
        for author in authors:
            subfields = [('a', author[0])]
            for affiliation in author[1]:
                subfields.append(('v', affiliation))
            if first_author:
                record_add_field(rec, '100', subfields=subfields)
                first_author = False
            else:
                record_add_field(rec, '700', subfields=subfields)
        identifier = self.get_identifier()
        conference = identifier.split(':')[2]
        conference = conference.split('/')[0]
        contribution = identifier.split(':')[2]
        contribution = contribution.split('/')[1]
        record_add_field(rec, '773', subfields=[('p', 'PoS'),
                                                ('v', conference.replace(' ', '')),
                                                ('c', contribution),
                                                ('y', date[:4])])
        record_add_field(rec, '980', subfields=[('a', 'ConferencePaper')])
        record_add_field(rec, '980', subfields=[('a', 'HEP')])
        return rec


Instruction:
How would you implement a function in Python 3 that
displays a progress that can update in place

Response:
def progress(length, **kwargs):
    """display a progress that can update in place

    example --
        total_length = 1000
        with echo.progress(total_length) as p:
            for x in range(total_length):
                # do something crazy
                p.update(x)

    length -- int -- the total size of what you will be updating progress on
    """
    quiet = False
    progress_class = kwargs.pop("progress_class", Progress)
    kwargs["write_method"] = istdout.info
    kwargs["width"] = kwargs.get("width", globals()["WIDTH"])
    kwargs["length"] = length
    pbar = progress_class(**kwargs)
    pbar.update(0)
    yield pbar
    pbar.update(length)
    br()


Instruction:
Given the following Python 3 function, write the documentation
def err(format_msg, *args, **kwargs):
    '''print format_msg to stderr'''
    exc_info = kwargs.pop("exc_info", False)
    stderr.warning(str(format_msg).format(*args, **kwargs), exc_info=exc_info)

Response: print format_msg to stderr


Instruction:
How would you code a function in Python 3 to
print format_msg to stdout taking into account -- quiet setting

Response:
def out(format_msg="", *args, **kwargs):
    '''print format_msg to stdout, taking into account --quiet setting'''
    logmethod = kwargs.get("logmethod", stdout.info)

    if format_msg != "":
        if Prefix.has():
            if isinstance(format_msg, basestring):
                format_msg = Prefix.get() + format_msg
            else:
                format_msg = Prefix.get() + str(format_msg)

        if isinstance(format_msg, basestring):
            if args or kwargs:
                s = format_msg.format(*args, **kwargs)
            else:
                s = format_msg
            logmethod(s)
#             width = globals()["width"]
#             s = textwrap.fill(s, width=width)
#             stdout.info(s)

        else:
            logmethod(str(format_msg))

    else:
        logmethod("")


Instruction:
Given the following Python 3 function, write the documentation
def verbose(format_msg="", *args, **kwargs):
    '''print format_msg to stdout, taking into account --verbose flag'''
    kwargs["logmethod"] = stdout.debug
    out(format_msg, *args, **kwargs)

Response: print format_msg to stdout taking into account verbose flag


Instruction:
How would you implement a function in Python 3 that
prints a banner for the available language files

Response:
def banner(*lines, **kwargs):
    """prints a banner

    sep -- string -- the character that will be on the line on the top and bottom
        and before any of the lines, defaults to *
    count -- integer -- the line width, defaults to 80
    """
    sep = kwargs.get("sep", "*")
    count = kwargs.get("width", globals()["WIDTH"])

    out(sep * count)
    if lines:
        out(sep)

        for line in lines:
            out("{} {}".format(sep, line))

        out(sep)
        out(sep * count)


Instruction:
Make a summary of the following Python 3 code
def table(*columns, **kwargs):
    ret = []
    prefix = kwargs.get('prefix', '')
    buf_count = kwargs.get('buf_count', 2)
    if len(columns) == 1:
        columns = list(columns[0])
    else:
        # without the list the zip iterator gets spent, I'm sure I can make this
        # better
        columns = list(zip(*columns))

    headers = kwargs.get("headers", [])
    if headers:
        columns.insert(0, headers)

    # we have to go through all the rows and calculate the length of each
    # column of each row
    widths = kwargs.get("widths", [])
    row_counts = Counter()
    for i in range(len(widths)):
        row_counts[i] = int(widths[i])

    width = int(kwargs.get("width", 0))
    for row in columns:
        for i, c in enumerate(row):
            if isinstance(c, basestring):
                cl = len(c)
            else:
                cl = len(str(c))
            if cl > row_counts[i]:
                row_counts[i] = cl

    width = int(kwargs.get("width", 0))
    if width:
        for i in row_counts:
            if row_counts[i] < width:
                row_counts[i] = width

    # actually go through and format each row
    def colstr(c):
        if isinstance(c, basestring): return c
        return str(c)

    def rowstr(row, prefix, row_counts):
        row_format = prefix
        cols = list(map(colstr, row))
        for i in range(len(row_counts)):
            c = cols[i]
            # build the format string for each row, we use the row_counts found
            # above to decide how much padding each column should get
            # https://stackoverflow.com/a/9536084/5006
            if re.match(r"^\d+(?:\.\d+)?$", c):
                if i == 0:
                    row_format += "{:>" + str(row_counts[i]) + "}"
                else:
                    row_format += "{:>" + str(row_counts[i] + buf_count) + "}"
            else:
                row_format += "{:<" + str(row_counts[i] + buf_count) + "}"

        return row_format.format(*cols)

    for row in columns:
        ret.append(rowstr(row, prefix, row_counts))

    out(os.linesep.join(ret))

Response: format columned data so we can easily print it out on a console


Instruction:
Can you tell what is the following Python 3 function doing
def prompt(question, choices=None):

    if not re.match("\s$", question):
        question = "{}: ".format(question)

    while True:
        if sys.version_info[0] > 2:
            answer = input(question)

        else:
            answer = raw_input(question)

        if not choices or answer in choices:
            break

    return answer

Response: echo a prompt to the user and wait for an answer


Instruction:
How would you explain what the following Python 3 function does
def get_records(self, url):
        page = urllib2.urlopen(url)
        pages = [BeautifulSoup(page)]
        #content spread over several pages?
        numpag = pages[0].body.findAll('span', attrs={'class': 'number-of-pages'})
        if len(numpag) > 0:
            if re.search('^\d+$', numpag[0].string):
                for i in range(int(numpag[0].string)-1):
                    page = urllib2.urlopen('%s/page/%i' % (url, i+2))
                    pages.append(BeautifulSoup(page))
            else:
                print("number of pages %s not an integer" % (numpag[0].string))
        impl = getDOMImplementation()
        doc = impl.createDocument(None, "collection", None)
        links = []
        for page in pages:
            links += page.body.findAll('p', attrs={'class': 'title'})
            links += page.body.findAll('h3', attrs={'class': 'title'})
        for link in links:
            record = self._get_record(link)
            doc.firstChild.appendChild(record)
        return doc.toprettyxml()

Response:
Returns the records listed in the webpage given as
        parameter as a xml String.


Instruction:
Can you tell what is the following Python 3 function doing
def inject_quiet(levels):
    loggers = list(Logger.manager.loggerDict.items())
    loggers.append(("root", getLogger()))
    level_filter = LevelFilter(levels)

    for logger_name, logger in loggers:
        for handler in getattr(logger, "handlers", []):
            handler.addFilter(level_filter)

Response: injects a level filter to all loggers and handlers


Instruction:
Can you create a Python 3 function that
connects to the specified ftp server and returns the connector.

Response:
def connect(self):
        """Logs into the specified ftp server and returns connector."""
        for tried_connection_count in range(CFG_FTP_CONNECTION_ATTEMPTS):
            try:
                self.ftp = FtpHandler(self.config.OXFORD.URL,
                                      self.config.OXFORD.LOGIN,
                                      self.config.OXFORD.PASSWORD)
                self.logger.debug(("Successful connection to the "
                                   "Oxford University Press server"))
                return
            except socket_timeout_exception as err:
                self.logger.error(('Failed to connect %d of %d times. '
                                   'Will sleep for %d seconds and try again.')
                                  % (tried_connection_count+1,
                                     CFG_FTP_CONNECTION_ATTEMPTS,
                                     CFG_FTP_TIMEOUT_SLEEP_DURATION))
                time.sleep(CFG_FTP_TIMEOUT_SLEEP_DURATION)
            except Exception as err:
                self.logger.error(('Failed to connect to the Oxford '
                                   'University Press server. %s') % (err,))
                break

        raise LoginException(err)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _extract_packages(self):
        if not hasattr(self, "retrieved_packages_unpacked"):
            self.retrieved_packages_unpacked = [self.package_name]
        for path in self.retrieved_packages_unpacked:
            package_name = basename(path)
            self.path_unpacked = join(CFG_UNPACKED_FILES,
                                      package_name.split('.')[0])
            self.logger.debug("Extracting package: %s"
                              % (path.split("/")[-1],))
            try:
                if "_archival_pdf" in self.path_unpacked:
                    self.path_unpacked = (self.path_unpacked
                                          .rstrip("_archival_pdf"))
                    ZipFile(path).extractall(join(self.path_unpacked,
                                                  "archival_pdfs"))
                else:
                    ZipFile(path).extractall(self.path_unpacked)
                #TarFile.open(path).extractall(self.path_unpacked)
            except Exception:
                register_exception(alert_admin=True,
                                   prefix="OUP error extracting package.")
                self.logger.error("Error extraction package file: %s"
                                  % (path,))

        if hasattr(self, "path_unpacked"):
            return self.path_unpacked

Response: Extracts a package in a new directory.