Instruction:
Can you tell what is the following Python 3 function doing
def phrase_to_filename(self, phrase):
        # remove non-word characters
        name = re.sub(r"[^\w\s\.]", '', phrase.strip().lower())
        # replace whitespace with underscores
        name = re.sub(r"\s+", '_', name)

        return name + '.png'

Response: Convert a phrase to a normilized file name.


Instruction:
Given the following Python 3 function, write the documentation
def open(self):
        if self.seed_url:
            self.driver_adapter.open(self.seed_url)
            self.wait_for_page_to_load()
            return self
        raise UsageError("Set a base URL or URL_TEMPLATE to open this page.")

Response:
Open the page.

        Navigates to :py:attr:`seed_url` and calls :py:func:`wait_for_page_to_load`.

        :return: The current page object.
        :rtype: :py:class:`Page`
        :raises: UsageError


Instruction:
Can you generate a brief explanation for the following Python 3 code
def wait_for_page_to_load(self):
        self.wait.until(lambda _: self.loaded)
        self.pm.hook.pypom_after_wait_for_page_to_load(page=self)
        return self

Response: Wait for the page to load.


Instruction:
Can you generate the documentation for the following Python 3 function
def register():
    registerDriver(
        ISelenium,
        Selenium,
        class_implements=[
            Firefox,
            Chrome,
            Ie,
            Edge,
            Opera,
            Safari,
            BlackBerry,
            PhantomJS,
            Android,
            Remote,
            EventFiringWebDriver,
        ],
    )

Response: Register the Selenium specific driver implementation.


Instruction:
How would you explain what the following Python 3 function does
def root(self):
        if self._root is None and self._root_locator is not None:
            return self.page.find_element(*self._root_locator)
        return self._root

Response: Return the root element for the page region.


Instruction:
Make a summary of the following Python 3 code
def wait_for_region_to_load(self):
        self.wait.until(lambda _: self.loaded)
        self.pm.hook.pypom_after_wait_for_region_to_load(region=self)
        return self

Response: Wait for the page region to load.


Instruction:
Make a summary of the following Python 3 code
def find_element(self, strategy, locator):
        return self.driver_adapter.find_element(strategy, locator, root=self.root)

Response: Finds an element on the page.


Instruction:
Can you tell what is the following Python 3 function doing
def find_elements(self, strategy, locator):
        return self.driver_adapter.find_elements(strategy, locator, root=self.root)

Response: Finds elements on the page.


Instruction:
Write a Python 3 function for
checking whether an element is present.

Response:
def is_element_present(self, strategy, locator):
        """Checks whether an element is present.

        :param strategy: Location strategy to use. See :py:class:`~selenium.webdriver.common.by.By` or :py:attr:`~pypom.splinter_driver.ALLOWED_STRATEGIES`.
        :param locator: Location of target element.
        :type strategy: str
        :type locator: str
        :return: ``True`` if element is present, else ``False``.
        :rtype: bool

        """
        return self.driver_adapter.is_element_present(strategy, locator, root=self.root)


Instruction:
Can you create a Python 3 function that
checks whether an element is displayed.

Response:
def is_element_displayed(self, strategy, locator):
        """Checks whether an element is displayed.

        :param strategy: Location strategy to use. See :py:class:`~selenium.webdriver.common.by.By` or :py:attr:`~pypom.splinter_driver.ALLOWED_STRATEGIES`.
        :param locator: Location of target element.
        :type strategy: str
        :type locator: str
        :return: ``True`` if element is displayed, else ``False``.
        :rtype: bool

        """
        return self.driver_adapter.is_element_displayed(
            strategy, locator, root=self.root
        )


Instruction:
Create a Python 3 function to
register driver adapter used by page object

Response:
def registerDriver(iface, driver, class_implements=[]):
    """ Register driver adapter used by page object"""
    for class_item in class_implements:
        classImplements(class_item, iface)

    component.provideAdapter(factory=driver, adapts=[iface], provides=IDriver)


Instruction:
Make a summary of the following Python 3 code
def isHcl(s):
    '''
        Detects whether a string is JSON or HCL

        :param s: String that may contain HCL or JSON

        :returns: True if HCL, False if JSON, raises ValueError
                  if neither
    '''
    for c in s:
        if c.isspace():
            continue

        if c == '{':
            return False
        else:
            return True

    raise ValueError("No HCL object could be decoded")

Response: Detects whether a string contains HCL or JSON or HCL.


Instruction:
Here you have a function in Python 3, explain what it does
def t_string(self, t):
        # Start of a string
        r'\"'
        # abs_start is the absolute start of the string. We use this at the end
        # to know how many new lines we've consumed
        t.lexer.abs_start = t.lexer.lexpos
        # rel_pos is the begining of the unconsumed part of the string. It will
        # get modified when consuming escaped characters
        t.lexer.rel_pos = t.lexer.lexpos
        # The value of the consumed part of the string
        t.lexer.string_value = u''
        t.lexer.begin('string')

Response: Tokenizes the string of the log entries.


Instruction:
Can you implement a function in Python 3 that
handles the escaped character in the string.

Response:
def t_string_escapedchar(self, t):
        # If a quote or backslash is escaped, build up the string by ignoring
        # the escape character. Should this be done for other characters?
        r'(?<=\\)(\"|\\)'
        t.lexer.string_value += (
            t.lexer.lexdata[t.lexer.rel_pos : t.lexer.lexpos - 2] + t.value
        )
        t.lexer.rel_pos = t.lexer.lexpos
        pass


Instruction:
Here you have a function in Python 3, explain what it does
def t_stringdollar_rbrace(self, t):
        r'\}'
        t.lexer.braces -= 1

        if t.lexer.braces == 0:
            # End of the dollar brace, back to the rest of the string
            t.lexer.begin('string')

Response: Tokenize a dollar brace.


Instruction:
Can you generate the documentation for the following Python 3 function
def t_tabbedheredoc(self, t):
        r'<<-\S+\r?\n'
        t.lexer.is_tabbed = True
        self._init_heredoc(t)
        t.lexer.begin('tabbedheredoc')

Response: r << - S+\ r?\n


Instruction:
Can you tell what is the following Python 3 function doing
def objectlist_flat(self, lt, replace):
        '''
            Similar to the dict constructor, but handles dups

            HCL is unclear on what one should do when duplicate keys are
            encountered. These comments aren't clear either:

            from decoder.go: if we're at the root or we're directly within
                             a list, decode into dicts, otherwise lists

            from object.go: there's a flattened list structure
        '''
        d = {}

        for k, v in lt:
            if k in d.keys() and not replace:
                if type(d[k]) is list:
                    d[k].append(v)
                else:
                    d[k] = [d[k], v]
            else:
                if isinstance(v, dict):
                    dd = d.setdefault(k, {})
                    for kk, vv in iteritems(v):
                        if type(dd) == list:
                            dd.append({kk: vv})
                        elif kk in dd.keys():
                            if hasattr(vv, 'items'):
                                for k2, v2 in iteritems(vv):
                                    dd[kk][k2] = v2
                            else:
                                d[k] = [dd, {kk: vv}]
                        else:
                            dd[kk] = vv
                else:
                    d[k] = v

        return d

Response: This function takes a list of objects and returns a flattened list of objects.


Instruction:
Can you generate the documentation for the following Python 3 function
def p_objectlist_2(self, p):
        "objectlist : objectlist COMMA objectitem"
        if DEBUG:
            self.print_p(p)
        p[0] = p[1] + [p[3]]

Response: objectlist : objectlist COMMA objectitem


Instruction:
Write a Python 3 script to
object : LEFTBRACE objectlist RIGHTBRACE

Response:
def p_object_0(self, p):
        "object : LEFTBRACE objectlist RIGHTBRACE"
        if DEBUG:
            self.print_p(p)
        p[0] = self.objectlist_flat(p[2], False)


Instruction:
Create a Python 3 function for
objecting : LEFTBRACE objectlist COMMA RIGHTBRACE

Response:
def p_object_1(self, p):
        "object : LEFTBRACE objectlist COMMA RIGHTBRACE"
        if DEBUG:
            self.print_p(p)
        p[0] = self.objectlist_flat(p[2], False)


Instruction:
How would you explain what the following Python 3 function does
def p_objectitem_0(self, p):
        '''
        objectitem : objectkey EQUAL number
                   | objectkey EQUAL BOOL
                   | objectkey EQUAL STRING
                   | objectkey EQUAL object
                   | objectkey EQUAL list
        '''
        if DEBUG:
            self.print_p(p)
        p[0] = (p[1], p[3])

Response:
objectitem : objectkey EQUAL number
                   | objectkey EQUAL BOOL
                   | objectkey EQUAL STRING
                   | objectkey EQUAL object
                   | objectkey EQUAL list


Instruction:
Write a Python 3 function for
blocking : blockId object

Response:
def p_block_0(self, p):
        "block : blockId object"
        if DEBUG:
            self.print_p(p)
        p[0] = (p[1], p[2])


Instruction:
Make a summary of the following Python 3 code
def p_block_1(self, p):
        "block : blockId block"
        if DEBUG:
            self.print_p(p)
        p[0] = (p[1], {p[2][0]: p[2][1]})

Response: block : blockId block


Instruction:
Make a summary of the following Python 3 code
def p_listitems_1(self, p):
        "listitems : listitems COMMA listitem"
        if DEBUG:
            self.print_p(p)
        p[0] = p[1] + [p[3]]

Response: listitems : listitems COMMA listitem


Instruction:
Implement a function in Python 3 to
number : int exp

Response:
def p_number_2(self, p):
        "number : int exp"
        if DEBUG:
            self.print_p(p)
        p[0] = float("{0}{1}".format(p[1], p[2]))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def p_number_3(self, p):
        "number : float exp"
        if DEBUG:
            self.print_p(p)
        p[0] = float("{0}{1}".format(p[1], p[2]))

Response: number : float exp


Instruction:
Can you write a function in Python 3 where it
initializes the parse table at install time

Response:
def _pre_install():
    '''Initialize the parse table at install time'''

    # Generate the parsetab.dat file at setup time
    dat = join(setup_dir, 'src', 'hcl', 'parsetab.dat')
    if exists(dat):
        os.unlink(dat)

    sys.path.insert(0, join(setup_dir, 'src'))

    import hcl
    from hcl.parser import HclParser

    parser = HclParser()


Instruction:
Can you tell what is the following Python 3 function doing
def append(self, linenumber, raw_text, cells):
        self.rows.append(Row(linenumber, raw_text, cells))

Response: Add a new row of data from a test suite


Instruction:
Create a Python 3 function for
returning a list of statements that are not settings or comments.

Response:
def steps(self):
        """Return a list of steps (statements that are not settings or comments)"""
        steps = []
        for statement in self.statements:
            if ((not statement.is_comment()) and
                (not statement.is_setting())):
                steps.append(statement)
        return steps


Instruction:
Can you tell what is the following Python 3 function doing
def is_comment(self):
        '''Return True if the first non-empty cell starts with "#"'''

        for cell in self[:]:
            if cell == "":
                continue

            # this is the first non-empty cell. Check whether it is
            # a comment or not.
            if cell.lstrip().startswith("#"):
                return True
            else:
                return False
        return False

Response: Return True if the first non - empty cell starts with "#"'


Instruction:
Implement a Python 3 function for
returning an instance of SuiteFile ResourceFile SuiteFolder SuiteFolder Exactly which is returned depends on whether it s a file or a folder.

Response:
def RobotFactory(path, parent=None):
    '''Return an instance of SuiteFile, ResourceFile, SuiteFolder

    Exactly which is returned depends on whether it's a file or
    folder, and if a file, the contents of the file. If there is a
    testcase table, this will return an instance of SuiteFile,
    otherwise it will return an instance of ResourceFile.
    '''

    if os.path.isdir(path):
        return SuiteFolder(path, parent)

    else:
        rf = RobotFile(path, parent)

        for table in rf.tables:
            if isinstance(table, TestcaseTable):
                rf.__class__ = SuiteFile
                return rf

        rf.__class__ = ResourceFile
        return rf


Instruction:
Given the following Python 3 function, write the documentation
def walk(self, *types):
        '''
        Iterator which visits all suites and suite files,
        yielding test cases and keywords
        '''
        requested = types if len(types) > 0 else [SuiteFile, ResourceFile, SuiteFolder, Testcase, Keyword]

        for thing in self.robot_files:
            if thing.__class__ in requested:
                yield thing
            if isinstance(thing, SuiteFolder):
                for child in thing.walk():
                    if child.__class__ in requested:
                        yield child
            else:
                for child in thing.walk(*types):
                    yield child

Response: Iterator which visits all suites and suite files and suite folders and test cases and keywords and yield test cases and keywords.


Instruction:
Can you create a Python 3 function that
returns a list of all folders and test suite files.

Response:
def robot_files(self):
        '''Return a list of all folders, and test suite files (.txt, .robot)
        '''
        result = []
        for name in os.listdir(self.path):
            fullpath = os.path.join(self.path, name)
            if os.path.isdir(fullpath):
                result.append(RobotFactory(fullpath, parent=self))
            else:
                if ((name.endswith(".txt") or name.endswith(".robot")) and
                    (name not in ("__init__.txt", "__init__.robot"))):

                    result.append(RobotFactory(fullpath, parent=self))
        return result


Instruction:
Create a Python 3 function to
load the contents of the file at the given path and creates a list of tables and rows for each of the tables.

Response:
def _load(self, path):

        '''
        The general idea is to do a quick parse, creating a list of
        tables. Each table is nothing more than a list of rows, with
        each row being a list of cells. Additional parsing such as
        combining rows into statements is done on demand. This first
        pass is solely to read in the plain text and organize it by table.
        '''

        self.tables = []
        current_table = DefaultTable(self)

        with Utf8Reader(path) as f:
            # N.B. the caller should be catching errors
            self.raw_text = f.read()

            f._file.seek(0) # bleh; wish this wasn't a private property
            matcher = Matcher(re.IGNORECASE)
            for linenumber, raw_text in enumerate(f.readlines()):
                linenumber += 1; # start counting at 1 rather than zero

                # this mimics what the robot TSV reader does --
                # it replaces non-breaking spaces with regular spaces,
                # and then strips trailing whitespace
                raw_text = raw_text.replace(u'\xA0', ' ')
                raw_text = raw_text.rstrip()

                # FIXME: I'm keeping line numbers but throwing away
                # where each cell starts. I should be preserving that
                # (though to be fair, robot is throwing that away so
                # I'll have to write my own splitter if I want to save
                # the character position)
                cells = TxtReader.split_row(raw_text)
                _heading_regex = r'^\s*\*+\s*(.*?)[ *]*$'

                if matcher(_heading_regex, cells[0]):
                    # we've found the start of a new table
                    table_name = matcher.group(1)
                    current_table = tableFactory(self, linenumber, table_name, raw_text)
                    self.tables.append(current_table)
                else:
                    current_table.append(Row(linenumber, raw_text, cells))


Instruction:
Can you create a Python 3 function that
returns suite or resource if any of the tables are not UnknownTable

Response:
def type(self):
        '''Return 'suite' or 'resource' or None

        This will return 'suite' if a testcase table is found;
        It will return 'resource' if at least one robot table
        is found. If no tables are found it will return None
        '''

        robot_tables = [table for table in self.tables if not isinstance(table, UnknownTable)]
        if len(robot_tables) == 0:
            return None

        for table in self.tables:
            if isinstance(table, TestcaseTable):
                return "suite"

        return "resource"


Instruction:
How would you explain what the following Python 3 function does
def keywords(self):
        '''Generator which returns all keywords in the suite'''
        for table in self.tables:
            if isinstance(table, KeywordTable):
                for keyword in table.keywords:
                    yield keyword

Response: Generator which returns all keywords in the suite


Instruction:
How would you code a function in Python 3 to
dump the tables and rows

Response:
def dump(self):
        '''Regurgitate the tables and rows'''
        for table in self.tables:
            print("*** %s ***" % table.name)
            table.dump()


Instruction:
Can you generate the documentation for the following Python 3 function
def statements(self):
        '''Return a list of statements

        This is done by joining together any rows that
        have continuations
        '''
        # FIXME: no need to do this every time; we should cache the
        # result
        if len(self.rows) == 0:
            return []

        current_statement = Statement(self.rows[0])
        current_statement.startline = self.rows[0].linenumber
        current_statement.endline = self.rows[0].linenumber
        statements = []
        for row in self.rows[1:]:
            if len(row) > 0 and row[0] == "...":
                # we found a continuation
                current_statement += row[1:]
                current_statement.endline = row.linenumber
            else:
                if len(current_statement) > 0:
                    # append current statement to the list of statements...
                    statements.append(current_statement)
                # start a new statement
                current_statement = Statement(row)
                current_statement.startline = row.linenumber
                current_statement.endline = row.linenumber

        if len(current_statement) > 0:
            statements.append(current_statement)

        # trim trailing blank statements
        while (len(statements[-1]) == 0 or
               ((len(statements[-1]) == 1) and len(statements[-1][0]) == 0)):
            statements.pop()
        return statements

Response: Return a list of statements that are part of the table. This is done by joining together any rows that have continuations


Instruction:
Can you create a Python 3 function that
appends a new entry to the list of testcases.

Response:
def append(self, row):
        '''
        The idea is, we recognize when we have a new testcase by
        checking the first cell. If it's not empty and not a comment,
        we have a new test case.

        '''
        if len(row) == 0:
            # blank line. Should we throw it away, or append a BlankLine object?
            return

        if (row[0] != "" and
            (not row[0].lstrip().startswith("#"))):
            # we have a new child table
            self._children.append(self._childClass(self.parent, row.linenumber, row[0]))
            if len(row.cells) > 1:
                # It appears the first row -- which contains the test case or
                # keyword name -- also has the first logical row of cells.
                # We'll create a Row, but we'll make the first cell empty instead
                # of leaving the name in it, since other code always assumes the
                # first cell is empty.
                #
                # To be honest, I'm not sure this is the Right Thing To Do, but
                # I'm too lazy to audit the code to see if it matters if we keep
                # the first cell intact. Sorry if this ends up causing you grief
                # some day...
                row[0] = ""
                self._children[-1].append(row.linenumber, row.raw_text, row.cells)

        elif len(self._children) == 0:
            # something before the first test case
            # For now, append it to self.comments; eventually we should flag
            # an error if it's NOT a comment
            self.comments.append(row)

        else:
            # another row for the testcase
            if len(row.cells) > 0:
                self._children[-1].append(row.linenumber, row.raw_text, row.cells)


Instruction:
Write a Python 3 function for
reporting an error or warning

Response:
def report(self, obj, message, linenum, char_offset=0):
        """Report an error or warning"""
        self.controller.report(linenumber=linenum, filename=obj.path,
                               severity=self.severity, message=message,
                               rulename = self.__class__.__name__,
                               char=char_offset)


Instruction:
Create a Python 3 function for
returning the docstrings of the entry.

Response:
def doc(self):
        '''Algorithm from https://www.python.org/dev/peps/pep-0257/'''
        if not self.__doc__:
            return ""

        lines = self.__doc__.expandtabs().splitlines()

        # Determine minimum indentation (first line doesn't count):
        indent = sys.maxsize
        for line in lines[1:]:
            stripped = line.lstrip()
            if stripped:
                indent = min(indent, len(line) - len(stripped))

        # Remove indentation (first line is special):
        trimmed = [lines[0].strip()]
        if indent < sys.maxsize:
            for line in lines[1:]:
                trimmed.append(line[indent:].rstrip())

        # Strip off trailing and leading blank lines:
        while trimmed and not trimmed[-1]:
            trimmed.pop()
        while trimmed and not trimmed[0]:
            trimmed.pop(0)

        # Return a single string:
        return '\n'.join(trimmed)


Instruction:
Explain what the following Python 3 code does
def run(self, args):

        self.args = self.parse_and_process_args(args)

        if self.args.version:
            print(__version__)
            return 0

        if self.args.rulefile:
            for filename in self.args.rulefile:
                self._load_rule_file(filename)

        if self.args.list:
            self.list_rules()
            return 0

        if self.args.describe:
            self._describe_rules(self.args.args)
            return 0

        self.counts = { ERROR: 0, WARNING: 0, "other": 0}

        for filename in self.args.args:
            if not (os.path.exists(filename)):
                sys.stderr.write("rflint: %s: No such file or directory\n" % filename)
                continue
            if os.path.isdir(filename):
                self._process_folder(filename)
            else:
                self._process_file(filename)

        if self.counts[ERROR] > 0:
            return self.counts[ERROR] if self.counts[ERROR] < 254 else 255

        return 0

Response: Parse command line arguments and run rflint


Instruction:
Make a summary of the following Python 3 code
def list_rules(self):
        for rule in sorted(self.all_rules, key=lambda rule: rule.name):
            print(rule)
            if self.args.verbose:
                for line in rule.doc.split("\n"):
                    print("    ", line)

Response: Print a list of all rules


Instruction:
Can you tell what is the following Python 3 function doing
def report(self, linenumber, filename, severity, message, rulename, char):

        if self._print_filename is not None:
            # we print the filename only once. self._print_filename
            # will get reset each time a new file is processed.
            print("+ " + self._print_filename)
            self._print_filename = None

        if severity in (WARNING, ERROR):
            self.counts[severity] += 1
        else:
            self.counts["other"] += 1

        print(self.args.format.format(linenumber=linenumber, filename=filename,
                                      severity=severity, message=message.encode('utf-8'),
                                      rulename=rulename, char=char))

Response: Report a rule violation


Instruction:
Write a Python 3 script for
returning a list of rules of a given class

Response:
def _get_rules(self, cls):
        """Returns a list of rules of a given class

        Rules are treated as singletons - we only instantiate each
        rule once.
        """

        result = []
        for rule_class in cls.__subclasses__():
            rule_name = rule_class.__name__.lower()
            if rule_name not in self._rules:
                rule = rule_class(self)
                self._rules[rule_name] = rule
            result.append(self._rules[rule_name])
        return result


Instruction:
How would you explain what the following Python 3 function does
def _load_rule_file(self, filename):
        if not (os.path.exists(filename)):
            sys.stderr.write("rflint: %s: No such file or directory\n" % filename)
            return
        try:
            basename = os.path.basename(filename)
            (name, ext) = os.path.splitext(basename)
            imp.load_source(name, filename)
        except Exception as e:
            sys.stderr.write("rflint: %s: exception while loading: %s\n" % (filename, str(e)))

Response: Import the given rule file


Instruction:
How would you code a function in Python 3 to
handle the parsing of command line arguments.

Response:
def parse_and_process_args(self, args):
        """Handle the parsing of command line arguments."""

        parser = argparse.ArgumentParser(
            prog="python -m rflint",
            description="A style checker for robot framework plain text files.",
            formatter_class=argparse.RawDescriptionHelpFormatter,
            epilog = (
                "You can use 'all' in place of RULENAME to refer to all rules. \n"
                "\n"
                "For example: '--ignore all --warn DuplicateTestNames' will ignore all\n"
                "rules except DuplicateTestNames.\n"
                "\n"
                "FORMAT is a string that performs a substitution on the following \n"
                "patterns: {severity}, {linenumber}, {char}, {message}, and {rulename}.\n"
                "\n"
                "For example: --format 'line: {linenumber}: message: {message}'. \n"
                "\n"
                "ARGUMENTFILE is a filename with contents that match the format of \n"
                "standard robot framework argument files\n"
                "\n"
                "If you give a directory as an argument, all files in the directory\n"
                "with the suffix .txt, .robot or .tsv will be processed. With the \n"
                "--recursive option, subfolders within the directory will also be\n"
                "processed."
                )
            )
        parser.add_argument("--error", "-e", metavar="RULENAME", action=SetErrorAction,
                            help="Assign a severity of ERROR to the given RULENAME")
        parser.add_argument("--ignore", "-i", metavar="RULENAME", action=SetIgnoreAction,
                            help="Ignore the given RULENAME")
        parser.add_argument("--warning", "-w", metavar="RULENAME", action=SetWarningAction,
                            help="Assign a severity of WARNING for the given RULENAME")
        parser.add_argument("--list", "-l", action="store_true",
                            help="show a list of known rules and exit")
        parser.add_argument("--describe", "-d", action="store_true",
                            help="describe the given rules")
        parser.add_argument("--no-filenames", action="store_false", dest="print_filenames",
                            default=True,
                            help="suppress the printing of filenames")
        parser.add_argument("--format", "-f",
                            help="Define the output format",
                            default='{severity}: {linenumber}, {char}: {message} ({rulename})')
        parser.add_argument("--version", action="store_true", default=False,
                            help="Display version number and exit")
        parser.add_argument("--verbose", "-v", action="store_true", default=False,
                            help="Give verbose output")
        parser.add_argument("--configure", "-c", action=ConfigureAction,
                            help="Configure a rule")
        parser.add_argument("--recursive", "-r", action="store_true", default=False,
                            help="Recursively scan subfolders in a directory")
        parser.add_argument("--rulefile", "-R", action=RulefileAction,
                            help="import additional rules from the given RULEFILE")
        parser.add_argument("--argumentfile", "-A", action=ArgfileLoader,
                            help="read arguments from the given file")
        parser.add_argument('args', metavar="file", nargs=argparse.REMAINDER)

        # create a custom namespace, in which we can store a reference to
        # our rules. This lets the custom argument actions access the list
        # of rules
        ns = argparse.Namespace()
        setattr(ns, "app", self)
        args = parser.parse_args(args, ns)

        Rule.output_format = args.format

        return args


Instruction:
Make a summary of the following Python 3 code
def from_resolver(cls, spec_resolver):
        spec_validators = cls._get_spec_validators(spec_resolver)
        return validators.extend(Draft4Validator, spec_validators)

Response: Creates a customized Draft4ExtendedValidator.


Instruction:
Write a Python 3 script for
creating json documents validator from spec resolver.

Response:
def create(self, spec_resolver):
        """Creates json documents validator from spec resolver.
        :param spec_resolver: reference resolver.

        :return: RefResolver for spec with cached remote $refs used during
            validation.
        :rtype: :class:`jsonschema.RefResolver`
        """
        validator_cls = self.spec_validator_factory.from_resolver(
            spec_resolver)

        return validator_cls(
            self.schema, resolver=self.schema_resolver)


Instruction:
Given the following Python 3 function, write the documentation
def construct_mapping(self, node, deep=False):
        mapping = super(ExtendedSafeConstructor, self).construct_mapping(
            node, deep)

        return {
            (str(key) if isinstance(key, int) else key): mapping[key]
            for key in mapping
        }

Response: This method coerces all keys in the node to strings.


Instruction:
Create a Python 3 function to
open a file and read it and return its contents.

Response:
def read_yaml_file(path, loader=ExtendedSafeLoader):
    """Open a file, read it and return its contents."""
    with open(path) as fh:
        return load(fh, loader)


Instruction:
Write a Python 3 script for
creating validators generator for the spec resolver.

Response:
def from_spec_resolver(cls, spec_resolver):
        """Creates validators generator for the spec resolver.

        :param spec_resolver: resolver for the spec
        :type instance_resolver: :class:`jsonschema.RefResolver`
        """
        deref = DerefValidatorDecorator(spec_resolver)
        for key, validator_callable in iteritems(cls.validators):
            yield key, deref(validator_callable)


Instruction:
Can you generate the documentation for the following Python 3 function
def grade(grader_data,submission):

    #Initialize result dictionary
    results = {'errors': [],'tests': [],'score': 0, 'feedback' : "", 'success' : False, 'confidence' : 0}
    has_error=False

    grader_set=EssaySet(essaytype="test")
    feedback = {}

    model, extractor = get_classifier_and_ext(grader_data)

    #This is to preserve legacy functionality
    if 'algorithm' not in grader_data:
        grader_data['algorithm'] = util_functions.AlgorithmTypes.classification

    try:
        #Try to add essay to essay set object
        grader_set.add_essay(str(submission),0)
        grader_set.update_prompt(str(grader_data['prompt']))
    except Exception:
        error_message = "Essay could not be added to essay set:{0}".format(submission)
        log.exception(error_message)
        results['errors'].append(error_message)
        has_error=True

    #Try to extract features from submission and assign score via the model
    try:
        grader_feats=extractor.gen_feats(grader_set)
        feedback=extractor.gen_feedback(grader_set,grader_feats)[0]
        results['score']=int(model.predict(grader_feats)[0])
    except Exception:
        error_message = "Could not extract features and score essay."
        log.exception(error_message)
        results['errors'].append(error_message)
        has_error=True

    #Try to determine confidence level
    try:
        results['confidence'] = get_confidence_value(grader_data['algorithm'], model, grader_feats, results['score'], grader_data['score'])
    except Exception:
        #If there is an error getting confidence, it is not a show-stopper, so just log
        log.exception("Problem generating confidence value")

    if not has_error:

        #If the essay is just a copy of the prompt, return a 0 as the score
        if( 'too_similar_to_prompt' in feedback and feedback['too_similar_to_prompt']):
            results['score']=0
            results['correct']=False

        results['success']=True

        #Generate short form output--number of problem areas identified in feedback

        #Add feedback to results if available
        results['feedback'] = {}
        if 'topicality' in feedback and 'prompt_overlap' in feedback:
            results['feedback'].update({
                'topicality' : feedback['topicality'],
                'prompt-overlap' : feedback['prompt_overlap'],
                })

        results['feedback'].update(
            {
                'spelling' : feedback['spelling'],
                'grammar' : feedback['grammar'],
                'markup-text' : feedback['markup_text'],
                }
        )

    else:
        #If error, success is False.
        results['success']=False

    return results

Response: Grades a specified submission using the specified models


Instruction:
Can you implement a function in Python 3 that
grades a set of numeric and textual features using a generic model

Response:
def grade_generic(grader_data, numeric_features, textual_features):
    """
    Grades a set of numeric and textual features using a generic model
    grader_data -- dictionary containing:
    {
        'algorithm' - Type of algorithm to use to score
    }
    numeric_features - list of numeric features to predict on
    textual_features - list of textual feature to predict on

    """
    results = {'errors': [],'tests': [],'score': 0, 'success' : False, 'confidence' : 0}

    has_error=False

    #Try to find and load the model file

    grader_set=predictor_set.PredictorSet(essaytype="test")

    model, extractor = get_classifier_and_ext(grader_data)

    #Try to add essays to essay set object
    try:
        grader_set.add_row(numeric_features, textual_features,0)
    except Exception:
        error_msg = "Row could not be added to predictor set:{0} {1}".format(numeric_features, textual_features)
        log.exception(error_msg)
        results['errors'].append(error_msg)
        has_error=True

    #Try to extract features from submission and assign score via the model
    try:
        grader_feats=extractor.gen_feats(grader_set)
        results['score']=model.predict(grader_feats)[0]
    except Exception:
        error_msg = "Could not extract features and score essay."
        log.exception(error_msg)
        results['errors'].append(error_msg)
        has_error=True

    #Try to determine confidence level
    try:
        results['confidence'] = get_confidence_value(grader_data['algorithm'],model, grader_feats, results['score'])
    except Exception:
        #If there is an error getting confidence, it is not a show-stopper, so just log
        log.exception("Problem generating confidence value")

    if not has_error:
        results['success'] = True

    return results


Instruction:
Here you have a function in Python 3, explain what it does
def get_confidence_value(algorithm,model,grader_feats,score, scores):
    min_score=min(numpy.asarray(scores))
    max_score=max(numpy.asarray(scores))
    if algorithm == util_functions.AlgorithmTypes.classification and hasattr(model, "predict_proba"):
        #If classification, predict with probability, which gives you a matrix of confidences per score point
        raw_confidence=model.predict_proba(grader_feats)[0,(float(score)-float(min_score))]
        #TODO: Normalize confidence somehow here
        confidence=raw_confidence
    elif hasattr(model, "predict"):
        raw_confidence = model.predict(grader_feats)[0]
        confidence = max(float(raw_confidence) - math.floor(float(raw_confidence)), math.ceil(float(raw_confidence)) - float(raw_confidence))
    else:
        confidence = 0

    return confidence

Response: Determines a confidence value given a proper input parameters


Instruction:
Make a summary of the following Python 3 code
def create_model_path(model_path):
    if not model_path.startswith("/") and not model_path.startswith("models/"):
        model_path="/" + model_path
    if not model_path.startswith("models"):
        model_path = "models" + model_path
    if not model_path.endswith(".p"):
        model_path+=".p"

    return model_path

Response: Creates a path to model files based on the model path.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def sub_chars(string):
    #Define replacement patterns
    sub_pat = r"[^A-Za-z\.\?!,';:]"
    char_pat = r"\."
    com_pat = r","
    ques_pat = r"\?"
    excl_pat = r"!"
    sem_pat = r";"
    col_pat = r":"
    whitespace_pat = r"\s{1,}"

    #Replace text.  Ordering is very important!
    nstring = re.sub(sub_pat, " ", string)
    nstring = re.sub(char_pat," .", nstring)
    nstring = re.sub(com_pat, " ,", nstring)
    nstring = re.sub(ques_pat, " ?", nstring)
    nstring = re.sub(excl_pat, " !", nstring)
    nstring = re.sub(sem_pat, " ;", nstring)
    nstring = re.sub(col_pat, " :", nstring)
    nstring = re.sub(whitespace_pat, " ", nstring)

    return nstring

Response: Strips illegal characters from a string. Used to sanitize input essays.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def spell_correct(string):

    # Create a temp file so that aspell could be used
    # By default, tempfile will delete this file when the file handle is closed.
    f = tempfile.NamedTemporaryFile(mode='w')
    f.write(string)
    f.flush()
    f_path = os.path.abspath(f.name)
    try:
        p = os.popen(aspell_path + " -a < " + f_path + " --sug-mode=ultra")

        # Aspell returns a list of incorrect words with the above flags
        incorrect = p.readlines()
        p.close()

    except Exception:
        log.exception("aspell process failed; could not spell check")
        # Return original string if aspell fails
        return string,0, string

    finally:
        f.close()

    incorrect_words = list()
    correct_spelling = list()
    for i in range(1, len(incorrect)):
        if(len(incorrect[i]) > 10):
            #Reformat aspell output to make sense
            match = re.search(":", incorrect[i])
            if hasattr(match, "start"):
                begstring = incorrect[i][2:match.start()]
                begmatch = re.search(" ", begstring)
                begword = begstring[0:begmatch.start()]

                sugstring = incorrect[i][match.start() + 2:]
                sugmatch = re.search(",", sugstring)
                if hasattr(sugmatch, "start"):
                    sug = sugstring[0:sugmatch.start()]

                    incorrect_words.append(begword)
                    correct_spelling.append(sug)

    #Create markup based on spelling errors
    newstring = string
    markup_string = string
    already_subbed=[]
    for i in range(0, len(incorrect_words)):
        sub_pat = r"\b" + incorrect_words[i] + r"\b"
        sub_comp = re.compile(sub_pat)
        newstring = re.sub(sub_comp, correct_spelling[i], newstring)
        if incorrect_words[i] not in already_subbed:
            markup_string=re.sub(sub_comp,'<bs>' + incorrect_words[i] + "</bs>", markup_string)
            already_subbed.append(incorrect_words[i])

    return newstring,len(incorrect_words),markup_string

Response: This function will spell correct an input string.


Instruction:
Write a Python 3 script to
generate ngrams from a list of words.

Response:
def ngrams(tokens, min_n, max_n):
    """
    Generates ngrams(word sequences of fixed length) from an input token sequence.
    tokens is a list of words.
    min_n is the minimum length of an ngram to return.
    max_n is the maximum length of an ngram to return.
    returns a list of ngrams (words separated by a space)
    """
    all_ngrams = list()
    n_tokens = len(tokens)
    for i in xrange(n_tokens):
        for j in xrange(i + min_n, min(n_tokens, i + max_n) + 1):
            all_ngrams.append(" ".join(tokens[i:j]))
    return all_ngrams


Instruction:
Can you generate a brief explanation for the following Python 3 code
def f7(seq):
    seen = set()
    seen_add = seen.add
    return [x for x in seq if x not in seen and not seen_add(x)]

Response:
Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique
    Makes a list unique


Instruction:
Write a Python 3 function for
generating a count of the number of times each unique item appears in a list

Response:
def count_list(the_list):
    """
    Generates a count of the number of times each unique item appears in a list
    """
    count = the_list.count
    result = [(item, count(item)) for item in set(the_list)]
    result.sort()
    return result


Instruction:
Create a Python 3 function for
generating a list of ngrams that appear in the string.

Response:
def regenerate_good_tokens(string):
    """
    Given an input string, part of speech tags the string, then generates a list of
    ngrams that appear in the string.
    Used to define grammatically correct part of speech tag sequences.
    Returns a list of part of speech tag sequences.
    """
    toks = nltk.word_tokenize(string)
    pos_string = nltk.pos_tag(toks)
    pos_seq = [tag[1] for tag in pos_string]
    pos_ngrams = ngrams(pos_seq, 2, 4)
    sel_pos_ngrams = f7(pos_ngrams)
    return sel_pos_ngrams


Instruction:
Implement a Python 3 function for
using a fisher test to find words that are significant in that they separate high scoring essays from low scoring essays. text is a list of input essays. score is a list of scores, with score[n] corresponding to text[n] max_feats is the maximum number of features to consider in the first pass max_feats2 is the maximum number of features to consider in the second (final) pass Returns a list of words that constitute the significant vocabulary

Response:
def get_vocab(text, score, max_feats=750, max_feats2=200):
    """
    Uses a fisher test to find words that are significant in that they separate
    high scoring essays from low scoring essays.
    text is a list of input essays.
    score is a list of scores, with score[n] corresponding to text[n]
    max_feats is the maximum number of features to consider in the first pass
    max_feats2 is the maximum number of features to consider in the second (final) pass
    Returns a list of words that constitute the significant vocabulary
    """
    dict = CountVectorizer(ngram_range=(1,2), max_features=max_feats)
    dict_mat = dict.fit_transform(text)
    set_score = numpy.asarray(score, dtype=numpy.int)
    med_score = numpy.median(set_score)
    new_score = set_score
    if(med_score == 0):
        med_score = 1
    new_score[set_score < med_score] = 0
    new_score[set_score >= med_score] = 1

    fish_vals = []
    for col_num in range(0, dict_mat.shape[1]):
        loop_vec = dict_mat.getcol(col_num).toarray()
        good_loop_vec = loop_vec[new_score == 1]
        bad_loop_vec = loop_vec[new_score == 0]
        good_loop_present = len(good_loop_vec[good_loop_vec > 0])
        good_loop_missing = len(good_loop_vec[good_loop_vec == 0])
        bad_loop_present = len(bad_loop_vec[bad_loop_vec > 0])
        bad_loop_missing = len(bad_loop_vec[bad_loop_vec == 0])
        fish_val = pvalue(good_loop_present, bad_loop_present, good_loop_missing, bad_loop_missing).two_tail
        fish_vals.append(fish_val)

    cutoff = 1
    if(len(fish_vals) > max_feats2):
        cutoff = sorted(fish_vals)[max_feats2]
    good_cols = numpy.asarray([num for num in range(0, dict_mat.shape[1]) if fish_vals[num] <= cutoff])

    getVar = lambda searchList, ind: [searchList[i] for i in ind]
    vocab = getVar(dict.get_feature_names(), good_cols)

    return vocab


Instruction:
Implement a function in Python 3 to
calculate edit distance between strings 1 and string 2.

Response:
def edit_distance(s1, s2):
    """
    Calculates string edit distance between string 1 and string 2.
    Deletion, insertion, substitution, and transposition all increase edit distance.
    """
    d = {}
    lenstr1 = len(s1)
    lenstr2 = len(s2)
    for i in xrange(-1, lenstr1 + 1):
        d[(i, -1)] = i + 1
    for j in xrange(-1, lenstr2 + 1):
        d[(-1, j)] = j + 1

    for i in xrange(lenstr1):
        for j in xrange(lenstr2):
            if s1[i] == s2[j]:
                cost = 0
            else:
                cost = 1
            d[(i, j)] = min(
                d[(i - 1, j)] + 1, # deletion
                d[(i, j - 1)] + 1, # insertion
                d[(i - 1, j - 1)] + cost, # substitution
            )
            if i and j and s1[i] == s2[j - 1] and s1[i - 1] == s2[j]:
                d[(i, j)] = min(d[(i, j)], d[i - 2, j - 2] + cost) # transposition

    return d[lenstr1 - 1, lenstr2 - 1]


Instruction:
Implement a Python 3 function for
generating cross validated predictions using an input classifier and data.

Response:
def gen_cv_preds(clf, arr, sel_score, num_chunks=3):
    """
    Generates cross validated predictions using an input classifier and data.
    clf is a classifier that implements that implements the fit and predict methods.
    arr is the input data array (X)
    sel_score is the target list (y).  y[n] corresponds to X[n,:]
    num_chunks is the number of cross validation folds to use
    Returns an array of the predictions where prediction[n] corresponds to X[n,:]
    """
    cv_len = int(math.floor(len(sel_score) / num_chunks))
    chunks = []
    for i in range(0, num_chunks):
        range_min = i * cv_len
        range_max = ((i + 1) * cv_len)
        if i == num_chunks - 1:
            range_max = len(sel_score)
        chunks.append(range(range_min, range_max))
    preds = []
    set_score = numpy.asarray(sel_score, dtype=numpy.int)
    chunk_vec = numpy.asarray(range(0, len(chunks)))
    for i in xrange(0, len(chunks)):
        loop_inds = list(
            chain.from_iterable([chunks[int(z)] for z, m in enumerate(range(0, len(chunks))) if int(z) != i]))
        sim_fit = clf.fit(arr[loop_inds], set_score[loop_inds])
        preds.append(list(sim_fit.predict(arr[chunks[i]])))
    all_preds = list(chain(*preds))
    return(all_preds)


Instruction:
Explain what the following Python 3 code does
def gen_model(clf, arr, sel_score):
    set_score = numpy.asarray(sel_score, dtype=numpy.int)
    sim_fit = clf.fit(arr, set_score)
    return(sim_fit)

Response: Generates a model from a classifier to data and a target score.


Instruction:
Explain what the following Python 3 code does
def gen_preds(clf, arr):
    if(hasattr(clf, "predict_proba")):
        ret = clf.predict(arr)
        # pred_score=preds.argmax(1)+min(x._score)
    else:
        ret = clf.predict(arr)
    return ret

Response: Generates predictions on a novel data array using a fit classifier clf and arr


Instruction:
Write a Python 3 function for
calculating the average value of a list of numbers Returns a float

Response:
def calc_list_average(l):
    """
    Calculates the average value of a list of numbers
    Returns a float
    """
    total = 0.0
    for value in l:
        total += value
    return total / len(l)


Instruction:
How would you explain what the following Python 3 function does
def quadratic_weighted_kappa(rater_a, rater_b, min_rating=None, max_rating=None):
    assert(len(rater_a) == len(rater_b))
    rater_a = [int(a) for a in rater_a]
    rater_b = [int(b) for b in rater_b]
    if min_rating is None:
        min_rating = min(rater_a + rater_b)
    if max_rating is None:
        max_rating = max(rater_a + rater_b)
    conf_mat = confusion_matrix(rater_a, rater_b,
        min_rating, max_rating)
    num_ratings = len(conf_mat)
    num_scored_items = float(len(rater_a))

    hist_rater_a = histogram(rater_a, min_rating, max_rating)
    hist_rater_b = histogram(rater_b, min_rating, max_rating)

    numerator = 0.0
    denominator = 0.0

    if(num_ratings > 1):
        for i in range(num_ratings):
            for j in range(num_ratings):
                expected_count = (hist_rater_a[i] * hist_rater_b[j]
                                  / num_scored_items)
                d = pow(i - j, 2.0) / pow(num_ratings - 1, 2.0)
                numerator += d * conf_mat[i][j] / num_scored_items
                denominator += d * expected_count / num_scored_items

        return 1.0 - numerator / denominator
    else:
        return 1.0

Response: Calculates the kappa correlation between rater_a and rater_b.


Instruction:
Can you generate the documentation for the following Python 3 function
def confusion_matrix(rater_a, rater_b, min_rating=None, max_rating=None):
    assert(len(rater_a) == len(rater_b))
    rater_a = [int(a) for a in rater_a]
    rater_b = [int(b) for b in rater_b]
    min_rating = int(min_rating)
    max_rating = int(max_rating)
    if min_rating is None:
        min_rating = min(rater_a)
    if max_rating is None:
        max_rating = max(rater_a)
    num_ratings = int(max_rating - min_rating + 1)
    conf_mat = [[0 for i in range(num_ratings)]
                for j in range(num_ratings)]
    for a, b in zip(rater_a, rater_b):
        conf_mat[int(a - min_rating)][int(b - min_rating)] += 1
    return conf_mat

Response: Generates a confusion matrix between rater_a and rater_b


Instruction:
Given the following Python 3 function, write the documentation
def histogram(ratings, min_rating=None, max_rating=None):
    ratings = [int(r) for r in ratings]
    if min_rating is None:
        min_rating = min(ratings)
    if max_rating is None:
        max_rating = max(ratings)
    num_ratings = int(max_rating - min_rating + 1)
    hist_ratings = [0 for x in range(num_ratings)]
    for r in ratings:
        hist_ratings[r - min_rating] += 1
    return hist_ratings

Response:
Generates a frequency count of each rating on the scale
    ratings is a list of scores


Instruction:
Create a Python 3 function to
utilize wordnet to get synonyms for words word is the input word returns a list of unique synonyms

Response:
def get_wordnet_syns(word):
    """
    Utilize wordnet (installed with nltk) to get synonyms for words
    word is the input word
    returns a list of unique synonyms
    """
    synonyms = []
    regex = r"_"
    pat = re.compile(regex)
    synset = nltk.wordnet.wordnet.synsets(word)
    for ss in synset:
        for swords in ss.lemma_names:
            synonyms.append(pat.sub(" ", swords.lower()))
    synonyms = f7(synonyms)
    return synonyms


Instruction:
Can you tell what is the following Python 3 function doing
def get_separator_words(toks1):
    tab_toks1 = nltk.FreqDist(word.lower() for word in toks1)
    if(os.path.isfile(ESSAY_COR_TOKENS_PATH)):
        toks2 = pickle.load(open(ESSAY_COR_TOKENS_PATH, 'rb'))
    else:
        essay_corpus = open(ESSAY_CORPUS_PATH).read()
        essay_corpus = sub_chars(essay_corpus)
        toks2 = nltk.FreqDist(word.lower() for word in nltk.word_tokenize(essay_corpus))
        pickle.dump(toks2, open(ESSAY_COR_TOKENS_PATH, 'wb'))
    sep_words = []
    for word in tab_toks1.keys():
        tok1_present = tab_toks1[word]
        if(tok1_present > 2):
            tok1_total = tab_toks1._N
            tok2_present = toks2[word]
            tok2_total = toks2._N
            fish_val = pvalue(tok1_present, tok2_present, tok1_total, tok2_total).two_tail
            if(fish_val < .001 and tok1_present / float(tok1_total) > (tok2_present / float(tok2_total)) * 2):
                sep_words.append(word)
    sep_words = [w for w in sep_words if not w in nltk.corpus.stopwords.words("english") and len(w) > 5]
    return sep_words

Response:
Finds the words that separate a list of tokens from a background corpus
    Returns a list of separator words


Instruction:
Can you generate a brief explanation for the following Python 3 code
def getMedian(numericValues):
    theValues = sorted(numericValues)

    if len(theValues) % 2 == 1:
        return theValues[(len(theValues) + 1) / 2 - 1]
    else:
        lower = theValues[len(theValues) / 2 - 1]
        upper = theValues[len(theValues) / 2]

        return (float(lower + upper)) / 2

Response: Gets the median of a list of values Returns a float


Instruction:
Create a Python 3 function to
initialize the dictionary objects for the current object based on the input essay set e_set.

Response:
def initialize_dictionaries(self, e_set, max_feats2 = 200):
        """
        Initializes dictionaries from an essay set object
        Dictionaries must be initialized prior to using this to extract features
        e_set is an input essay set
        returns a confirmation of initialization
        """
        if(hasattr(e_set, '_type')):
            if(e_set._type == "train"):
                #normal text (unstemmed) useful words/bigrams
                nvocab = util_functions.get_vocab(e_set._text, e_set._score, max_feats2 = max_feats2)
                #stemmed and spell corrected vocab useful words/ngrams
                svocab = util_functions.get_vocab(e_set._clean_stem_text, e_set._score, max_feats2 = max_feats2)
                #dictionary trained on proper vocab
                self._normal_dict = CountVectorizer(ngram_range=(1,2), vocabulary=nvocab)
                #dictionary trained on proper vocab
                self._stem_dict = CountVectorizer(ngram_range=(1,2), vocabulary=svocab)
                self.dict_initialized = True
                #Average spelling errors in set. needed later for spelling detection
                self._mean_spelling_errors=sum(e_set._spelling_errors)/float(len(e_set._spelling_errors))
                self._spell_errors_per_character=sum(e_set._spelling_errors)/float(sum([len(t) for t in e_set._text]))
                #Gets the number and positions of grammar errors
                good_pos_tags,bad_pos_positions=self._get_grammar_errors(e_set._pos,e_set._text,e_set._tokens)
                self._grammar_errors_per_character=(sum(good_pos_tags)/float(sum([len(t) for t in e_set._text])))
                #Generate bag of words features
                bag_feats=self.gen_bag_feats(e_set)
                #Sum of a row of bag of words features (topical words in an essay)
                f_row_sum=numpy.sum(bag_feats[:,:])
                #Average index of how "topical" essays are
                self._mean_f_prop=f_row_sum/float(sum([len(t) for t in e_set._text]))
                ret = "ok"
            else:
                raise util_functions.InputError(e_set, "needs to be an essay set of the train type.")
        else:
            raise util_functions.InputError(e_set, "wrong input. need an essay set object")
        return ret


Instruction:
Can you generate the documentation for the following Python 3 function
def get_good_pos_ngrams(self):
        if(os.path.isfile(NGRAM_PATH)):
            good_pos_ngrams = pickle.load(open(NGRAM_PATH, 'rb'))
        elif os.path.isfile(ESSAY_CORPUS_PATH):
            essay_corpus = open(ESSAY_CORPUS_PATH).read()
            essay_corpus = util_functions.sub_chars(essay_corpus)
            good_pos_ngrams = util_functions.regenerate_good_tokens(essay_corpus)
            pickle.dump(good_pos_ngrams, open(NGRAM_PATH, 'wb'))
        else:
            #Hard coded list in case the needed files cannot be found
            good_pos_ngrams=['NN PRP', 'NN PRP .', 'NN PRP . DT', 'PRP .', 'PRP . DT', 'PRP . DT NNP', '. DT',
             '. DT NNP', '. DT NNP NNP', 'DT NNP', 'DT NNP NNP', 'DT NNP NNP NNP', 'NNP NNP',
             'NNP NNP NNP', 'NNP NNP NNP NNP', 'NNP NNP NNP .', 'NNP NNP .', 'NNP NNP . TO',
             'NNP .', 'NNP . TO', 'NNP . TO NNP', '. TO', '. TO NNP', '. TO NNP NNP',
             'TO NNP', 'TO NNP NNP']

        return set(good_pos_ngrams)

Response: Gets a set of gramatically correct part of speech sequences from an input file called essaycorpus. txt


Instruction:
Can you generate the documentation for the following Python 3 function
def _get_grammar_errors(self,pos,text,tokens):
        word_counts = [max(len(t),1) for t in tokens]
        good_pos_tags = []
        min_pos_seq=2
        max_pos_seq=4
        bad_pos_positions=[]
        for i in xrange(0, len(text)):
            pos_seq = [tag[1] for tag in pos[i]]
            pos_ngrams = util_functions.ngrams(pos_seq, min_pos_seq, max_pos_seq)
            long_pos_ngrams=[z for z in pos_ngrams if z.count(' ')==(max_pos_seq-1)]
            bad_pos_tuples=[[z,z+max_pos_seq] for z in xrange(0,len(long_pos_ngrams)) if long_pos_ngrams[z] not in self._good_pos_ngrams]
            bad_pos_tuples.sort(key=operator.itemgetter(1))
            to_delete=[]
            for m in reversed(xrange(len(bad_pos_tuples)-1)):
                start, end = bad_pos_tuples[m]
                for j in xrange(m+1, len(bad_pos_tuples)):
                    lstart, lend = bad_pos_tuples[j]
                    if lstart >= start and lstart <= end:
                        bad_pos_tuples[m][1]=bad_pos_tuples[j][1]
                        to_delete.append(j)

            fixed_bad_pos_tuples=[bad_pos_tuples[z] for z in xrange(0,len(bad_pos_tuples)) if z not in to_delete]
            bad_pos_positions.append(fixed_bad_pos_tuples)
            overlap_ngrams = [z for z in pos_ngrams if z in self._good_pos_ngrams]
            if (len(pos_ngrams)-len(overlap_ngrams))>0:
                divisor=len(pos_ngrams)/len(pos_seq)
            else:
                divisor=1
            if divisor == 0:
                divisor=1
            good_grammar_ratio = (len(pos_ngrams)-len(overlap_ngrams))/divisor
            good_pos_tags.append(good_grammar_ratio)
        return good_pos_tags,bad_pos_positions

Response: Internal function to get the number of grammar errors in given text.


Instruction:
Implement a function in Python 3 to
generate length based features from an essay set containing only the words in the object.

Response:
def gen_length_feats(self, e_set):
        """
        Generates length based features from an essay set
        Generally an internal function called by gen_feats
        Returns an array of length features
        e_set - EssaySet object
        """
        text = e_set._text
        lengths = [len(e) for e in text]
        word_counts = [max(len(t),1) for t in e_set._tokens]
        comma_count = [e.count(",") for e in text]
        ap_count = [e.count("'") for e in text]
        punc_count = [e.count(".") + e.count("?") + e.count("!") for e in text]
        chars_per_word = [lengths[m] / float(word_counts[m]) for m in xrange(0, len(text))]

        good_pos_tags,bad_pos_positions= self._get_grammar_errors(e_set._pos,e_set._text,e_set._tokens)
        good_pos_tag_prop = [good_pos_tags[m] / float(word_counts[m]) for m in xrange(0, len(text))]

        length_arr = numpy.array((
        lengths, word_counts, comma_count, ap_count, punc_count, chars_per_word, good_pos_tags,
        good_pos_tag_prop)).transpose()

        return length_arr.copy()


Instruction:
Implement a Python 3 function for
generating bag of words features from an input essay set and trained FeatureExtractor object.

Response:
def gen_bag_feats(self, e_set):
        """
        Generates bag of words features from an input essay set and trained FeatureExtractor
        Generally called by gen_feats
        Returns an array of features
        e_set - EssaySet object
        """
        if(hasattr(self, '_stem_dict')):
            sfeats = self._stem_dict.transform(e_set._clean_stem_text)
            nfeats = self._normal_dict.transform(e_set._text)
            bag_feats = numpy.concatenate((sfeats.toarray(), nfeats.toarray()), axis=1)
        else:
            raise util_functions.InputError(self, "Dictionaries must be initialized prior to generating bag features.")
        return bag_feats.copy()


Instruction:
Make a summary of the following Python 3 code
def gen_feats(self, e_set):
        bag_feats = self.gen_bag_feats(e_set)
        length_feats = self.gen_length_feats(e_set)
        prompt_feats = self.gen_prompt_feats(e_set)
        overall_feats = numpy.concatenate((length_feats, prompt_feats, bag_feats), axis=1)
        overall_feats = overall_feats.copy()

        return overall_feats

Response: Generates bag of words length and prompt features from an essay set object


Instruction:
Explain what the following Python 3 code does
def gen_prompt_feats(self, e_set):
        prompt_toks = nltk.word_tokenize(e_set._prompt)
        expand_syns = []
        for word in prompt_toks:
            synonyms = util_functions.get_wordnet_syns(word)
            expand_syns.append(synonyms)
        expand_syns = list(chain.from_iterable(expand_syns))
        prompt_overlap = []
        prompt_overlap_prop = []
        for j in e_set._tokens:
            tok_length=len(j)
            if(tok_length==0):
                tok_length=1
            prompt_overlap.append(len([i for i in j if i in prompt_toks]))
            prompt_overlap_prop.append(prompt_overlap[len(prompt_overlap) - 1] / float(tok_length))
        expand_overlap = []
        expand_overlap_prop = []
        for j in e_set._tokens:
            tok_length=len(j)
            if(tok_length==0):
                tok_length=1
            expand_overlap.append(len([i for i in j if i in expand_syns]))
            expand_overlap_prop.append(expand_overlap[len(expand_overlap) - 1] / float(tok_length))

        prompt_arr = numpy.array((prompt_overlap, prompt_overlap_prop, expand_overlap, expand_overlap_prop)).transpose()

        return prompt_arr.copy()

Response: Generates prompt based features from an essay set object and internal prompt variable.


Instruction:
How would you implement a function in Python 3 that
generates a list of lists of feedback for a given set of essays.

Response:
def gen_feedback(self, e_set, features=None):
        """
        Generate feedback for a given set of essays
        e_set - EssaySet object
        features - optionally, pass in a matrix of features extracted from e_set using FeatureExtractor
        in order to get off topic feedback.
        Returns a list of lists (one list per essay in e_set)
        e_set - EssaySet object
        """

        #Set ratio to modify thresholds for grammar/spelling errors
        modifier_ratio=1.05

        #Calc number of grammar and spelling errors per character
        set_grammar,bad_pos_positions=self._get_grammar_errors(e_set._pos,e_set._text,e_set._tokens)
        set_grammar_per_character=[set_grammar[m]/float(len(e_set._text[m])+.1) for m in xrange(0,len(e_set._text))]
        set_spell_errors_per_character=[e_set._spelling_errors[m]/float(len(e_set._text[m])+.1) for m in xrange(0,len(e_set._text))]

        #Iterate through essays and create a feedback dict for each
        all_feedback=[]
        for m in xrange(0,len(e_set._text)):
            #Be very careful about changing these messages!
            individual_feedback={'grammar' : "Grammar: Ok.",
                                 'spelling' : "Spelling: Ok.",
                                 'markup_text' : "",
                                 'grammar_per_char' : set_grammar_per_character[m],
                                 'spelling_per_char' : set_spell_errors_per_character[m],
                                 'too_similar_to_prompt' : False,
                                 }
            markup_tokens=e_set._markup_text[m].split(" ")

            #This loop ensures that sequences of bad grammar get put together into one sequence instead of staying
            #disjointed
            bad_pos_starts=[z[0] for z in bad_pos_positions[m]]
            bad_pos_ends=[z[1]-1 for z in bad_pos_positions[m]]
            for z in xrange(0,len(markup_tokens)):
                if z in bad_pos_starts:
                    markup_tokens[z]='<bg>' + markup_tokens[z]
                elif z in bad_pos_ends:
                    markup_tokens[z]=markup_tokens[z] + "</bg>"
            if(len(bad_pos_ends)>0 and len(bad_pos_starts)>0 and len(markup_tokens)>1):
                if max(bad_pos_ends)>(len(markup_tokens)-1) and max(bad_pos_starts)<(len(markup_tokens)-1):
                    markup_tokens[len(markup_tokens)-1]+="</bg>"

            #Display messages if grammar/spelling errors greater than average in training set
            if set_grammar_per_character[m]>(self._grammar_errors_per_character*modifier_ratio):
                individual_feedback['grammar']="Grammar: More grammar errors than average."
            if set_spell_errors_per_character[m]>(self._spell_errors_per_character*modifier_ratio):
                individual_feedback['spelling']="Spelling: More spelling errors than average."

            #Test topicality by calculating # of on topic words per character and comparing to the training set
            #mean.  Requires features to be passed in
            if features is not None:
                f_row_sum=numpy.sum(features[m,12:])
                f_row_prop=f_row_sum/len(e_set._text[m])
                if f_row_prop<(self._mean_f_prop/1.5) or len(e_set._text[m])<20:
                    individual_feedback['topicality']="Topicality: Essay may be off topic."

                if(features[m,9]>.6):
                    individual_feedback['prompt_overlap']="Prompt Overlap: Too much overlap with prompt."
                    individual_feedback['too_similar_to_prompt']=True
                    log.debug(features[m,9])

            #Create string representation of markup text
            markup_string=" ".join(markup_tokens)
            individual_feedback['markup_text']=markup_string
            all_feedback.append(individual_feedback)

        return all_feedback


Instruction:
Can you implement a function in Python 3 that
adds new essay to the set.

Response:
def add_essay(self, essay_text, essay_score, essay_generated=0):
        """
        Add new (essay_text,essay_score) pair to the essay set.
        essay_text must be a string.
        essay_score must be an int.
        essay_generated should not be changed by the user.
        Returns a confirmation that essay was added.
        """
        # Get maximum current essay id, or set to 0 if this is the first essay added
        if(len(self._id) > 0):
            max_id = max(self._id)
        else:
            max_id = 0
            # Verify that essay_score is an int, essay_text is a string, and essay_generated equals 0 or 1

        try:
            essay_text = essay_text.encode('ascii', 'ignore')
            if len(essay_text) < 5:
                essay_text = "Invalid essay."
        except:
            log.exception("Could not parse essay into ascii.")

        try:
            # Try conversion of types
            essay_score = int(essay_score)
            essay_text = str(essay_text)
        except:
            # Nothing needed here, will return error in any case.
            log.exception("Invalid type for essay score : {0} or essay text : {1}".format(type(essay_score), type(essay_text)))

        if isinstance(essay_score, int) and isinstance(essay_text, basestring)\
                and (essay_generated == 0 or essay_generated == 1):
            self._id.append(max_id + 1)
            self._score.append(essay_score)
            # Clean text by removing non digit/work/punctuation characters
            try:
                essay_text = str(essay_text.encode('ascii', 'ignore'))
            except:
                essay_text = (essay_text.decode('utf-8', 'replace')).encode('ascii', 'ignore')
            cleaned_essay = util_functions.sub_chars(essay_text).lower()
            if(len(cleaned_essay) > MAXIMUM_ESSAY_LENGTH):
                cleaned_essay = cleaned_essay[0:MAXIMUM_ESSAY_LENGTH]
            self._text.append(cleaned_essay)
            # Spell correct text using aspell
            cleaned_text, spell_errors, markup_text = util_functions.spell_correct(self._text[len(self._text) - 1])
            self._clean_text.append(cleaned_text)
            self._spelling_errors.append(spell_errors)
            self._markup_text.append(markup_text)
            # Tokenize text
            self._tokens.append(nltk.word_tokenize(self._clean_text[len(self._clean_text) - 1]))
            # Part of speech tag text
            self._pos.append(nltk.pos_tag(self._clean_text[len(self._clean_text) - 1].split(" ")))
            self._generated.append(essay_generated)
            # Stem spell corrected text
            porter = nltk.PorterStemmer()
            por_toks = " ".join([porter.stem(w) for w in self._tokens[len(self._tokens) - 1]])
            self._clean_stem_text.append(por_toks)

            ret = "text: " + self._text[len(self._text) - 1] + " score: " + str(essay_score)
        else:
            raise util_functions.InputError(essay_text, "arguments need to be in format "
                                                        "(text,score). text needs to be string,"
                                                        " score needs to be int.")


Instruction:
Can you generate a brief explanation for the following Python 3 code
def update_prompt(self, prompt_text):
        if(isinstance(prompt_text, basestring)):
            self._prompt = util_functions.sub_chars(prompt_text)
            ret = self._prompt
        else:
            raise util_functions.InputError(prompt_text, "Invalid prompt. Need to enter a string value.")
        return ret

Response: Update the prompt string which is a string.


Instruction:
Can you create a Python 3 function that
substitutes synonyms to generate extra essays from existing ones. This is done to increase the amount of training data. Should only be used with lowest scoring essays. e_text is the text of the original essay. e_score is the score of the original essay. dictionary is a fixed dictionary (list) of words to replace. max_syns defines the maximum number of additional essays to generate. Do not set too high.

Response:
def generate_additional_essays(self, e_text, e_score, dictionary=None, max_syns=3):
        """
        Substitute synonyms to generate extra essays from existing ones.
        This is done to increase the amount of training data.
        Should only be used with lowest scoring essays.
        e_text is the text of the original essay.
        e_score is the score of the original essay.
        dictionary is a fixed dictionary (list) of words to replace.
        max_syns defines the maximum number of additional essays to generate.  Do not set too high.
        """
        e_toks = nltk.word_tokenize(e_text)
        all_syns = []
        for word in e_toks:
            synonyms = util_functions.get_wordnet_syns(word)
            if(len(synonyms) > max_syns):
                synonyms = random.sample(synonyms, max_syns)
            all_syns.append(synonyms)
        new_essays = []
        for i in range(0, max_syns):
            syn_toks = e_toks
            for z in range(0, len(e_toks)):
                if len(all_syns[z]) > i and (dictionary == None or e_toks[z] in dictionary):
                    syn_toks[z] = all_syns[z][i]
            new_essays.append(" ".join(syn_toks))
        for z in xrange(0, len(new_essays)):
            self.add_essay(new_essays[z], e_score, 1)


Instruction:
How would you code a function in Python 3 to
create a new code block from input text associated scores a prompt and a path to the model.

Response:
def create(text,score,prompt_string, dump_data=False):
    """
    Creates a machine learning model from input text, associated scores, a prompt, and a path to the model
    TODO: Remove model path argument, it is needed for now to support legacy code
    text - A list of strings containing the text of the essays
    score - a list of integers containing score values
    prompt_string - the common prompt for the set of essays
    """

    if dump_data:
        dump_input_data(text, score)

    algorithm = select_algorithm(score)
    #Initialize a results dictionary to return
    results = {'errors': [],'success' : False, 'cv_kappa' : 0, 'cv_mean_absolute_error': 0,
               'feature_ext' : "", 'classifier' : "", 'algorithm' : algorithm,
               'score' : score, 'text' : text, 'prompt' : prompt_string}

    if len(text)!=len(score):
        msg = "Target and text lists must be same length."
        results['errors'].append(msg)
        log.exception(msg)
        return results

    try:
        #Create an essay set object that encapsulates all the essays and alternate representations (tokens, etc)
        e_set = model_creator.create_essay_set(text, score, prompt_string)
    except:
        msg = "essay set creation failed."
        results['errors'].append(msg)
        log.exception(msg)
    try:
        #Gets features from the essay set and computes error
        feature_ext, classifier, cv_error_results = model_creator.extract_features_and_generate_model(e_set, algorithm = algorithm)
        results['cv_kappa']=cv_error_results['kappa']
        results['cv_mean_absolute_error']=cv_error_results['mae']
        results['feature_ext']=feature_ext
        results['classifier']=classifier
        results['algorithm'] = algorithm
        results['success']=True
    except:
        msg = "feature extraction and model creation failed."
        results['errors'].append(msg)
        log.exception(msg)

    return results


Instruction:
Write a Python 3 script for
creating a generic list of numeric values and text values and a generic list of target variable.

Response:
def create_generic(numeric_values, textual_values, target, algorithm = util_functions.AlgorithmTypes.regression):
    """
    Creates a model from a generic list numeric values and text values
    numeric_values - A list of lists that are the predictors
    textual_values - A list of lists that are the predictors
    (each item in textual_values corresponds to the similarly indexed counterpart in numeric_values)
    target - The variable that we are trying to predict.  A list of integers.
    algorithm - the type of algorithm that will be used
    """

    algorithm = select_algorithm(target)
    #Initialize a result dictionary to return.
    results = {'errors': [],'success' : False, 'cv_kappa' : 0, 'cv_mean_absolute_error': 0,
               'feature_ext' : "", 'classifier' : "", 'algorithm' : algorithm}

    if len(numeric_values)!=len(textual_values) or len(numeric_values)!=len(target):
        msg = "Target, numeric features, and text features must all be the same length."
        results['errors'].append(msg)
        log.exception(msg)
        return results

    try:
        #Initialize a predictor set object that encapsulates all of the text and numeric predictors
        pset = predictor_set.PredictorSet(essaytype="train")
        for i in xrange(0, len(numeric_values)):
            pset.add_row(numeric_values[i], textual_values[i], target[i])
    except:
        msg = "predictor set creation failed."
        results['errors'].append(msg)
        log.exception(msg)

    try:
        #Extract all features and then train a classifier with the features
        feature_ext, classifier, cv_error_results = model_creator.extract_features_and_generate_model_predictors(pset, algorithm)
        results['cv_kappa']=cv_error_results['kappa']
        results['cv_mean_absolute_error']=cv_error_results['mae']
        results['feature_ext']=feature_ext
        results['classifier']=classifier
        results['success']=True
    except:
        msg = "feature extraction and model creation failed."
        results['errors'].append(msg)
        log.exception(msg)

    return results


Instruction:
Can you write a function in Python 3 where it
creates an essay set from given data.

Response:
def create_essay_set(text, score, prompt_string, generate_additional=True):
    """
    Creates an essay set from given data.
    Text should be a list of strings corresponding to essay text.
    Score should be a list of scores where score[n] corresponds to text[n]
    Prompt string is just a string containing the essay prompt.
    Generate_additional indicates whether to generate additional essays at the minimum score point or not.
    """
    x = EssaySet()
    for i in xrange(0, len(text)):
        x.add_essay(text[i], score[i])
        if score[i] == min(score) and generate_additional == True:
            x.generate_additional_essays(x._clean_text[len(x._clean_text) - 1], score[i])

    x.update_prompt(prompt_string)

    return x


Instruction:
Explain what the following Python 3 code does
def get_cv_error(clf,feats,scores):
    results={'success' : False, 'kappa' : 0, 'mae' : 0}
    try:
        cv_preds=util_functions.gen_cv_preds(clf,feats,scores)
        err=numpy.mean(numpy.abs(numpy.array(cv_preds)-scores))
        kappa=util_functions.quadratic_weighted_kappa(list(cv_preds),scores)
        results['mae']=err
        results['kappa']=kappa
        results['success']=True
    except ValueError as ex:
        # If this is hit, everything is fine.  It is hard to explain why the error occurs, but it isn't a big deal.
        msg = u"Not enough classes (0,1,etc) in each cross validation fold: {ex}".format(ex=ex)
        log.debug(msg)
    except:
        log.exception("Error getting cv error estimates.")

    return results

Response: Gets the cross validated error for a given classifier set of features and scores


Instruction:
How would you code a function in Python 3 to
get two classifiers for each type of algorithm and returns them.

Response:
def get_algorithms(algorithm):
    """
    Gets two classifiers for each type of algorithm, and returns them.  First for predicting, second for cv error.
    type - one of util_functions.AlgorithmTypes
    """
    if algorithm == util_functions.AlgorithmTypes.classification:
        clf = sklearn.ensemble.GradientBoostingClassifier(n_estimators=100, learn_rate=.05,
            max_depth=4, random_state=1,min_samples_leaf=3)
        clf2=sklearn.ensemble.GradientBoostingClassifier(n_estimators=100, learn_rate=.05,
            max_depth=4, random_state=1,min_samples_leaf=3)
    else:
        clf = sklearn.ensemble.GradientBoostingRegressor(n_estimators=100, learn_rate=.05,
            max_depth=4, random_state=1,min_samples_leaf=3)
        clf2=sklearn.ensemble.GradientBoostingRegressor(n_estimators=100, learn_rate=.05,
            max_depth=4, random_state=1,min_samples_leaf=3)
    return clf, clf2


Instruction:
Can you write a function in Python 3 where it
extracts features and generates predictors based on a given predictor set.

Response:
def extract_features_and_generate_model_predictors(predictor_set, algorithm=util_functions.AlgorithmTypes.regression):
    """
    Extracts features and generates predictors based on a given predictor set
    predictor_set - a PredictorSet object that has been initialized with data
    type - one of util_functions.AlgorithmType
    """
    if(algorithm not in [util_functions.AlgorithmTypes.regression, util_functions.AlgorithmTypes.classification]):
        algorithm = util_functions.AlgorithmTypes.regression

    f = predictor_extractor.PredictorExtractor()
    f.initialize_dictionaries(predictor_set)

    train_feats = f.gen_feats(predictor_set)

    clf,clf2 = get_algorithms(algorithm)
    cv_error_results=get_cv_error(clf2,train_feats,predictor_set._target)

    try:
        set_score = numpy.asarray(predictor_set._target, dtype=numpy.int)
        clf.fit(train_feats, set_score)
    except ValueError:
        log.exception("Not enough classes (0,1,etc) in sample.")
        set_score = predictor_set._target
        set_score[0]=1
        set_score[1]=0
        clf.fit(train_feats, set_score)

    return f, clf, cv_error_results


Instruction:
Can you write a function in Python 3 where it
extracts features and generate a classifier based on the training set.

Response:
def extract_features_and_generate_model(essays, algorithm=util_functions.AlgorithmTypes.regression):
    """
    Feed in an essay set to get feature vector and classifier
    essays must be an essay set object
    additional array is an optional argument that can specify
    a numpy array of values to add in
    returns a trained FeatureExtractor object and a trained classifier
    """
    f = feature_extractor.FeatureExtractor()
    f.initialize_dictionaries(essays)

    train_feats = f.gen_feats(essays)

    set_score = numpy.asarray(essays._score, dtype=numpy.int)
    if len(util_functions.f7(list(set_score)))>5:
        algorithm = util_functions.AlgorithmTypes.regression
    else:
        algorithm = util_functions.AlgorithmTypes.classification

    clf,clf2 = get_algorithms(algorithm)

    cv_error_results=get_cv_error(clf2,train_feats,essays._score)

    try:
        clf.fit(train_feats, set_score)
    except ValueError:
        log.exception("Not enough classes (0,1,etc) in sample.")
        set_score[0]=1
        set_score[1]=0
        clf.fit(train_feats, set_score)

    return f, clf, cv_error_results


Instruction:
How would you explain what the following Python 3 function does
def dump_model_to_file(prompt_string, feature_ext, classifier, text, score, model_path):
    model_file = {'prompt': prompt_string, 'extractor': feature_ext, 'model': classifier, 'text' : text, 'score' : score}
    pickle.dump(model_file, file=open(model_path, "w"))

Response: Writes out a model to a file.


Instruction:
Make a summary of the following Python 3 code
def create_essay_set_and_dump_model(text,score,prompt,model_path,additional_array=None):
    essay_set=create_essay_set(text,score,prompt)
    feature_ext,clf=extract_features_and_generate_model(essay_set,additional_array)
    dump_model_to_file(prompt,feature_ext,clf,model_path)

Response: Function that creates essay set extracts features and writes out model to file


Instruction:
Can you generate a brief explanation for the following Python 3 code
def initialize_dictionaries(self, p_set):
        success = False
        if not (hasattr(p_set, '_type')):
            error_message = "needs to be an essay set of the train type."
            log.exception(error_message)
            raise util_functions.InputError(p_set, error_message)

        if not (p_set._type == "train"):
            error_message = "needs to be an essay set of the train type."
            log.exception(error_message)
            raise util_functions.InputError(p_set, error_message)

        div_length=len(p_set._essay_sets)
        if div_length==0:
            div_length=1

        #Ensures that even with a large amount of input textual features, training time stays reasonable
        max_feats2 = int(math.floor(200/div_length))
        for i in xrange(0,len(p_set._essay_sets)):
            self._extractors.append(FeatureExtractor())
            self._extractors[i].initialize_dictionaries(p_set._essay_sets[i], max_feats2=max_feats2)
            self._initialized = True
            success = True
        return success

Response: Initialize the dictionary with the textual inputs in the PredictorSet object that has been fed in in.


Instruction:
Implement a function in Python 3 to
generate features based on an iput p_set

Response:
def gen_feats(self, p_set):
        """
        Generates features based on an iput p_set
        p_set - PredictorSet
        """
        if self._initialized!=True:
            error_message = "Dictionaries have not been initialized."
            log.exception(error_message)
            raise util_functions.InputError(p_set, error_message)

        textual_features = []
        for i in xrange(0,len(p_set._essay_sets)):
            textual_features.append(self._extractors[i].gen_feats(p_set._essay_sets[i]))

        textual_matrix = numpy.concatenate(textual_features, axis=1)
        predictor_matrix = numpy.array(p_set._numeric_features)

        print textual_matrix.shape
        print predictor_matrix.shape

        overall_matrix = numpy.concatenate((textual_matrix, predictor_matrix), axis=1)

        return overall_matrix.copy()


Instruction:
How would you explain what the following Python 3 function does
def t_text(self, t):
        r':\s*<text>'
        t.lexer.text_start = t.lexer.lexpos - len('<text>')
        t.lexer.begin('text')

Response: r \ s*text \ s *


Instruction:
Here you have a function in Python 3, explain what it does
def t_text_end(self, t):
        r'</text>\s*'
        t.type = 'TEXT'
        t.value = t.lexer.lexdata[
            t.lexer.text_start:t.lexer.lexpos]
        t.lexer.lineno += t.value.count('\n')
        t.value = t.value.strip()
        t.lexer.begin('INITIAL')
        return t

Response: r End of text


Instruction:
Can you generate the documentation for the following Python 3 function
def t_KEYWORD_AS_TAG(self, t):
        r'[a-zA-Z]+'
        t.type = self.reserved.get(t.value, 'UNKNOWN_TAG')
        t.value = t.value.strip()
        return t

Response: r Keyword as tag


Instruction:
Explain what the following Python 3 code does
def t_LINE_OR_KEYWORD_VALUE(self, t):
        r':.+'
        t.value = t.value[1:].strip()
        if t.value in self.reserved.keys():
            t.type = self.reserved[t.value]
        else:
            t.type = 'LINE'
        return t

Response: t is a KeywordType instance


Instruction:
How would you explain what the following Python 3 function does
def tv_to_rdf(infile_name, outfile_name):
    parser = Parser(Builder(), StandardLogger())
    parser.build()
    with open(infile_name) as infile:
        data = infile.read()
        document, error = parser.parse(data)
        if not error:
            with open(outfile_name, mode='w') as outfile:
                write_document(document, outfile)
            return True
        else:
            print('Errors encountered while parsing RDF file.')
            messages = []
            document.validate(messages)
            print('\n'.join(messages))
            return False

Response: Convert a SPDX file from tag value format to RDF format.


Instruction:
Can you generate the documentation for the following Python 3 function
def order_error(self, first_tag, second_tag, line):
        self.error = True
        msg = ERROR_MESSAGES['A_BEFORE_B'].format(first_tag, second_tag, line)
        self.logger.log(msg)

Response:
Reports an OrderError. Error message will state that
        first_tag came before second_tag.


Instruction:
How would you explain what the following Python 3 function does
def p_lic_xref_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.add_lic_xref(self.document, value)
        except OrderError:
            self.order_error('LicenseCrossReference', 'LicenseName', p.lineno(1))

Response: \ x1b [ 1mNAME \ x1b [ 0m lic_xref - Adds a LICS_CRS_REF LINE to the document.


Instruction:
Explain what the following Python 3 code does
def p_extr_lic_name_1(self, p):
        try:
            self.builder.set_lic_name(self.document, p[2])
        except OrderError:
            self.order_error('LicenseName', 'LicenseID', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('LicenseName', p.lineno(1))

Response: Parse the LICS_NAME section.


Instruction:
How would you explain what the following Python 3 function does
def p_prj_home_art_1(self, p):
        try:
            self.builder.set_file_atrificat_of_project(self.document, 'home', p[2])
        except OrderError:
            self.order_error('ArtificatOfProjectHomePage', 'FileName', p.lineno(1))

Response: prj_home_art : ART_PRJ_HOME LINE


Instruction:
Given the following Python 3 function, write the documentation
def p_file_notice_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_file_notice(self.document, value)
        except OrderError:
            self.order_error('FileNotice', 'FileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('FileNotice', p.lineno(1))

Response: \ x1b [ 1mNAME \ x1b [ 0m file_notice - Set the file_notice field to the value of the file_notice field.


Instruction:
Here you have a function in Python 3, explain what it does
def p_file_cr_text_1(self, p):
        try:
            self.builder.set_file_copyright(self.document, p[2])
        except OrderError:
            self.order_error('FileCopyrightText', 'FileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('FileCopyrightText', p.lineno(1))

Response: file_cr_text : FILE_CR_TEXT file_cr_value


Instruction:
Can you generate a brief explanation for the following Python 3 code
def p_file_lics_info_1(self, p):
        try:
            self.builder.set_file_license_in_file(self.document, p[2])
        except OrderError:
            self.order_error('LicenseInfoInFile', 'FileName', p.lineno(1))
        except SPDXValueError:
            self.error = True
            msg = ERROR_MESSAGES['FILE_LICS_INFO_VALUE'].format(p.lineno(1))
            self.logger.log(msg)

Response: File License Info In File Method


Instruction:
Explain what the following Python 3 code does
def p_file_name_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_file_name(self.document, value)
        except OrderError:
            self.order_error('FileName', 'PackageName', p.lineno(1))

Response: \ x1b [ 1mNAME \ x1b [ 0m file_name - Set the file name of the user.


Instruction:
How would you explain what the following Python 3 function does
def p_file_comment_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_file_comment(self.document, value)
        except OrderError:
            self.order_error('FileComment', 'FileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('FileComment', p.lineno(1))

Response: \ x1b [ 1mNAME \ x1b [ 0m file_comment - Set the file comment.


Instruction:
Here you have a function in Python 3, explain what it does
def p_file_type_1(self, p):
        try:
            self.builder.set_file_type(self.document, p[2])
        except OrderError:
            self.order_error('FileType', 'FileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('FileType', p.lineno(1))

Response: \ x1b [ 1mNAME \ x1b [ 0m file_type - Set the file type of the class.


Instruction:
Can you tell what is the following Python 3 function doing
def p_file_conc_1(self, p):
        try:
            self.builder.set_concluded_license(self.document, p[2])
        except SPDXValueError:
            self.error = True
            msg = ERROR_MESSAGES['FILE_LICS_CONC_VALUE'].format(p.lineno(1))
            self.logger.log(msg)
        except OrderError:
            self.order_error('LicenseConcluded', 'FileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('LicenseConcluded', p.lineno(1))

Response: File Concretization Entry point.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def p_file_type_value(self, p):
        if six.PY2:
            p[0] = p[1].decode(encoding='utf-8')
        else:
            p[0] = p[1]

Response: \ x1b [ 1mNAME \ x1b [ 0m file_type_value - Gets the file type value of the log entry.


Instruction:
Explain what the following Python 3 code does
def p_pkg_desc_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_pkg_desc(self.document, value)
        except CardinalityError:
            self.more_than_one_error('PackageDescription', p.lineno(1))
        except OrderError:
            self.order_error('PackageDescription', 'PackageFileName', p.lineno(1))

Response: pkg_desc : PKG_DESC TEXT


Instruction:
Can you generate a brief explanation for the following Python 3 code
def p_pkg_summary_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_pkg_summary(self.document, value)
        except OrderError:
            self.order_error('PackageSummary', 'PackageFileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('PackageSummary', p.lineno(1))

Response: pkg_summary : PKG_SUM TEXT


Instruction:
Explain what the following Python 3 code does
def p_pkg_cr_text_1(self, p):
        try:
            self.builder.set_pkg_cr_text(self.document, p[2])
        except OrderError:
            self.order_error('PackageCopyrightText', 'PackageFileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('PackageCopyrightText', p.lineno(1))

Response: PackageCopyrightText | PackageFileName | PackageCopyrightText| | |


Instruction:
Can you generate the documentation for the following Python 3 function
def p_pkg_lic_ff_1(self, p):
        try:
            self.builder.set_pkg_license_from_file(self.document, p[2])
        except OrderError:
            self.order_error('PackageLicenseInfoFromFiles', 'PackageName', p.lineno(1))
        except SPDXValueError:
            self.error = True
            msg = ERROR_MESSAGES['PKG_LIC_FFILE_VALUE'].format(p.lineno(1))
            self.logger.log(msg)

Response: Package LicenseInfoFromFiles entry point.


Instruction:
Here you have a function in Python 3, explain what it does
def p_pkg_lic_ff_value_3(self, p):
        if six.PY2:
            value = p[1].decode(encoding='utf-8')
        else:
            value = p[1]
        p[0] = document.License.from_identifier(value)

Response: pkg_lic_ff_value is a line of code that is used in the pkg_lic_ff_value_2 function.


Instruction:
Given the following Python 3 function, write the documentation
def p_pkg_chksum_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_pkg_chk_sum(self.document, value)
        except OrderError:
            self.order_error('PackageChecksum', 'PackageFileName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('PackageChecksum', p.lineno(1))

Response: PackageChecksum | PKG_CHKSUM | PackageFileName | |


Instruction:
Can you generate a brief explanation for the following Python 3 code
def p_pkg_home_2(self, p):
        self.error = True
        msg = ERROR_MESSAGES['PKG_HOME_VALUE']
        self.logger.log(msg)

Response: pkg_home : PKG_HOME error


Instruction:
Explain what the following Python 3 code does
def p_package_version_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.set_pkg_vers(self.document, value)
        except OrderError:
            self.order_error('PackageVersion', 'PackageName', p.lineno(1))
        except CardinalityError:
            self.more_than_one_error('PackageVersion', p.lineno(1))

Response: package_version : PKG_VERSION LINE


Instruction:
Make a summary of the following Python 3 code
def p_package_name(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.create_package(self.document, value)
        except CardinalityError:
            self.more_than_one_error('PackageName', p.lineno(1))

Response: \ x1b [ 1mNAME \ x1b [ 0m package_name - Creates a new package.


Instruction:
Explain what the following Python 3 code does
def p_annotation_date_1(self, p):
        try:
            if six.PY2:
                value = p[2].decode(encoding='utf-8')
            else:
                value = p[2]
            self.builder.add_annotation_date(self.document, value)
        except CardinalityError:
            self.more_than_one_error('AnnotationDate', p.lineno(1))
        except OrderError:
            self.order_error('AnnotationDate', 'Annotator', p.lineno(1))

Response: Annotation date | Annotation date | Date |