Instruction:
Can you generate a brief explanation for the following Python 3 code
def _roman(data, scheme_map, **kw):
  vowels = scheme_map.vowels
  marks = scheme_map.marks
  virama = scheme_map.virama
  consonants = scheme_map.consonants
  non_marks_viraama = scheme_map.non_marks_viraama
  max_key_length_from_scheme = scheme_map.max_key_length_from_scheme
  to_roman = scheme_map.to_scheme.is_roman

  togglers = kw.pop('togglers', set())
  suspend_on = kw.pop('suspend_on', set())
  suspend_off = kw.pop('suspend_off', set())
  if kw:
    raise TypeError('Unexpected keyword argument %s' % list(kw.keys())[0])

  buf = []
  i = 0
  had_consonant = found = False
  len_data = len(data)
  append = buf.append

  # If true, don't transliterate. The toggle token is discarded.
  toggled = False
  # If true, don't transliterate. The suspend token is retained.
  # `suspended` overrides `toggled`.
  suspended = False

  while i <= len_data:
    # The longest token in the source scheme has length `max_key_length_from_scheme`. Iterate
    # over `data` while taking `max_key_length_from_scheme` characters at a time. If we don`t
    # find the character group in our scheme map, lop off a character and
    # try again.
    #
    # If we've finished reading through `data`, then `token` will be empty
    # and the loop below will be skipped.
    token = data[i:i + max_key_length_from_scheme]

    while token:
      if token in togglers:
        toggled = not toggled
        i += 2  # skip over the token
        found = True  # force the token to fill up again
        break

      if token in suspend_on:
        suspended = True
      elif token in suspend_off:
        suspended = False

      if toggled or suspended:
        token = token[:-1]
        continue

      # Catch the pattern CV, where C is a consonant and V is a vowel.
      # V should be rendered as a vowel mark, a.k.a. a "dependent"
      # vowel. But due to the nature of Brahmic scripts, 'a' is implicit
      # and has no vowel mark. If we see 'a', add nothing.
      if had_consonant and token in vowels:
        mark = marks.get(token, '')
        if mark:
          append(mark)
        elif to_roman:
          append(vowels[token])
        found = True

      # Catch any non_marks_viraama character, including consonants, punctuation,
      # and regular vowels. Due to the implicit 'a', we must explicitly
      # end any lingering consonants before we can handle the current
      # token.
      elif token in non_marks_viraama:
        if had_consonant:
          append(virama[''])
        append(non_marks_viraama[token])
        found = True

      if found:
        had_consonant = token in consonants
        i += len(token)
        break
      else:
        token = token[:-1]

    # We've exhausted the token; this must be some other character. Due to
    # the implicit 'a', we must explicitly end any lingering consonants
    # before we can handle the current token.
    if not found:
      if had_consonant:
        append(virama[''])
      if i < len_data:
        append(data[i])
        had_consonant = False
      i += 1

    found = False

  return ''.join(buf)

Response: Transliterate the given data with the given scheme_map.


Instruction:
Implement a Python 3 function for
giving some devanAgarI sanskrit text, this function produces a "key" so that 1] The key should be the same for different observed orthographical forms of the same text. For example: :: - "dharmma" vs "dharma" - "rAmaM gacChati" vs "rAma~N gacChati" vs "rAma~N gacChati" - "kurvan eva" vs "kurvanneva" 2] The key should be different for different for different texts. - "stamba" vs "stambha" This function attempts to succeed at [1] and [2] almostall the time. Longer the text, probability of failing at [2] decreases, while probability of failing at [1] increases (albeit very slightly). Sources of orthographically divergent forms: - Phonetically sensible grammar rules - Neglect of sandhi while writing - Punctuation, spaces, avagraha-s. - Regional-language-influenced mistakes (La instead of la.) Some example applications of this function: - Create a database of quotes or words with minimal duplication. - Search a database of quotes or words while being robust to optional forms. Also see equivalent function in the scala indic-transliteration package.

Response:
def get_approx_deduplicating_key(text, encoding_scheme=sanscript.DEVANAGARI):
    """
    Given some devanAgarI sanskrit text, this function produces a "key" so
    that

    1] The key should be the same for different observed orthographical
    forms of the same text. For example:

    ::

        - "dharmma" vs "dharma"
        - "rAmaM gacChati" vs "rAma~N gacChati" vs "rAma~N gacChati"
        - "kurvan eva" vs "kurvanneva"

    2] The key should be different for different for different texts.

    -  "stamba" vs "stambha"

    This function attempts to succeed at [1] and [2] almostall the time.
    Longer the text, probability of failing at [2] decreases, while
    probability of failing at [1] increases (albeit very slightly).

    Sources of orthographically divergent forms:

    -  Phonetically sensible grammar rules
    -  Neglect of sandhi while writing
    -  Punctuation, spaces, avagraha-s.
    -  Regional-language-influenced mistakes (La instead of la.)

    Some example applications of this function:

    -  Create a database of quotes or words with minimal duplication.
    -  Search a database of quotes or words while being robust to optional
       forms.

    Also see equivalent function in the scala indic-transliteration package.
    """
    if encoding_scheme == sanscript.DEVANAGARI:
        key = text
        key = regex.sub("\\P{IsDevanagari}", "", key)
        # Remove spaces
        key = regex.sub("\\s", "", key)
        # Remove punctuations
        key = regex.sub("\\p{P}", "", key)
        # Remove digits, abbreviation sign, svara-s, avagraha
        key = regex.sub("[०-९।॥॰ऽ]|[॑-॔]", "", key)
        # Collapse semi-vowel-anunAsika-s संलग्नम् सल्ँलग्नम् into m
        key = regex.sub("[यरल]्ँ", "म्", key)
        # Collapse all panchama-s into m
        key = regex.sub("[ङञणन]", "म", key)
        # Collapse anusvAra into m
        key = regex.sub("ँ|ं", "म्", key)
        key = regex.sub("ॐ", "ओम्", key)
        key = regex.sub("[ळऴ]", "ल", key)
        # Deal with optional forms where consonants are duplicated - like dharmma
        # Details in https://docs.google.com/spreadsheets/d/1GP8Ps_hmgCGLZPWKIVBCfQB9ZmPQOaCwTrH9OybaWaQ/edit#gid=21
        key = regex.sub("([क-हक़-य़])्\\1+", "\\1", key)
        key = regex.sub("[कग]्ख्", "ख्", key)
        key = regex.sub("[कग]्घ्", "घ्", key)
        key = regex.sub("च्छ्", "छ्", key)
        key = regex.sub("ज्झ्", "झ्", key)
        key = regex.sub("त्थ्", "थ्", key)
        key = regex.sub("द्ध्", "ध्", key)
        key = regex.sub("ड्ढ्", "ढ्", key)
        key = regex.sub("प्फ्", "फ्", key)
        key = regex.sub("ब्भ्", "भ्", key)
        return key
    else:
        logging.warning("got script {} for '{}'".format(encoding_scheme, text))
        return regex.sub("\\s", "", text)


Instruction:
Implement a function in Python 3 to
transliterate the given data with the given scheme_map.

Response:
def _brahmic(data, scheme_map, **kw):
  """Transliterate `data` with the given `scheme_map`. This function is used
  when the source scheme is a Brahmic scheme.

  :param data: the data to transliterate
  :param scheme_map: a dict that maps between characters in the old scheme
                     and characters in the new scheme
  """
  if scheme_map.from_scheme.name == northern.GURMUKHI:
    data = northern.GurmukhiScheme.replace_tippi(text=data)
  marks = scheme_map.marks
  virama = scheme_map.virama
  consonants = scheme_map.consonants
  non_marks_viraama = scheme_map.non_marks_viraama
  to_roman = scheme_map.to_scheme.is_roman
  max_key_length_from_scheme = scheme_map.max_key_length_from_scheme

  buf = []
  i = 0
  to_roman_had_consonant = found = False
  append = buf.append
  # logging.debug(pprint.pformat(scheme_map.consonants))

  # We dont just translate each brAhmic character one after another in order to prefer concise transliterations when possible - for example ज्ञ -> jn in optitrans rather than j~n.
  while i <= len(data):
    # The longest token in the source scheme has length `max_key_length_from_scheme`. Iterate
    # over `data` while taking `max_key_length_from_scheme` characters at a time. If we don`t
    # find the character group in our scheme map, lop off a character and
    # try again.
    #
    # If we've finished reading through `data`, then `token` will be empty
    # and the loop below will be skipped.
    token = data[i:i + max_key_length_from_scheme]

    while token:
      if len(token) == 1:
        if token in marks:
          append(marks[token])
          found = True
        elif token in virama:
          append(virama[token])
          found = True
        else:
          if to_roman_had_consonant:
            append('a')
          append(non_marks_viraama.get(token, token))
          found = True
      else:
        if token in non_marks_viraama:
          if to_roman_had_consonant:
            append('a')
          append(non_marks_viraama.get(token))
          found = True

      if found:
        to_roman_had_consonant = to_roman and token in consonants
        i += len(token)
        break
      else:
        token = token[:-1]

    # Continuing the outer while loop.
    # We've exhausted the token; this must be some other character. Due to
    # the implicit 'a', we must explicitly end any lingering consonants
    # before we can handle the current token.
    if not found:
      if to_roman_had_consonant:
        append(next(iter(virama.values())))
      if i < len(data):
        append(data[i])
        to_roman_had_consonant = False
      i += 1

    found = False

  if to_roman_had_consonant:
    append('a')
  return ''.join(buf)


Instruction:
Implement a function in Python 3 to
transliterate data with the given parameters.

Response:
def transliterate(data, _from=None, _to=None, scheme_map=None, **kw):
  """Transliterate `data` with the given parameters::

      output = transliterate('idam adbhutam', HK, DEVANAGARI)

  Each time the function is called, a new :class:`SchemeMap` is created
  to map the input scheme to the output scheme. This operation is fast
  enough for most use cases. But for higher performance, you can pass a
  pre-computed :class:`SchemeMap` instead::

      scheme_map = SchemeMap(SCHEMES[HK], SCHEMES[DEVANAGARI])
      output = transliterate('idam adbhutam', scheme_map=scheme_map)

  :param data: the data to transliterate
  :param scheme_map: the :class:`SchemeMap` to use. If specified, ignore
                     `_from` and `_to`. If unspecified, create a
                     :class:`SchemeMap` from `_from` to `_to`.
  """
  if scheme_map is None:
    scheme_map = _get_scheme_map(_from, _to)
  options = {
    'togglers': {'##'},
    'suspend_on': set('<'),
    'suspend_off': set('>')
  }
  options.update(kw)

  from indic_transliteration.sanscript.brahmic_mapper import _brahmic
  from indic_transliteration.sanscript.roman_mapper import _roman
  func = _roman if scheme_map.from_scheme.is_roman else _brahmic
  return func(data, scheme_map, **options)


Instruction:
Here you have a function in Python 3, explain what it does
def detect(text):
  if sys.version_info < (3, 0):
    # Verify encoding
    try:
      text = text.decode('utf-8')
    except UnicodeError:
      pass

  # Brahmic schemes are all within a specific range of code points.
  for L in text:
    code = ord(L)
    if code >= BRAHMIC_FIRST_CODE_POINT:
      for name, start_code in BLOCKS:
        if start_code <= code <= BRAHMIC_LAST_CODE_POINT:
          return name

  # Romanizations
  if Regex.IAST_OR_KOLKATA_ONLY.search(text):
    if Regex.KOLKATA_ONLY.search(text):
      return Scheme.Kolkata
    else:
      return Scheme.IAST

  if Regex.ITRANS_ONLY.search(text):
    return Scheme.ITRANS

  if Regex.SLP1_ONLY.search(text):
    return Scheme.SLP1

  if Regex.VELTHUIS_ONLY.search(text):
    return Scheme.Velthuis

  if Regex.ITRANS_OR_VELTHUIS_ONLY.search(text):
    return Scheme.ITRANS

  return Scheme.HK

Response: Detects the transliteration scheme of the input.


Instruction:
Create a Python 3 function to
transliterate data with the given parameters.

Response:
def transliterate(data, _from=None, _to=None, scheme_map=None, **kw):
    """Transliterate `data` with the given parameters::

      output = transliterate('idam adbhutam', HK, DEVANAGARI)

  Each time the function is called, a new :class:`SchemeMap` is created
  to map the input scheme to the output scheme. This operation is fast
  enough for most use cases. But for higher performance, you can pass a
  pre-computed :class:`SchemeMap` instead::

      scheme_map = SchemeMap(SCHEMES[HK], SCHEMES[DEVANAGARI])
      output = transliterate('idam adbhutam', scheme_map=scheme_map)

  :param data: the data to transliterate
  :param _from: the name of a source scheme
  :param _to: the name of a destination scheme
  :param scheme_map: the :class:`SchemeMap` to use. If specified, ignore
                     `_from` and `_to`. If unspecified, create a
                     :class:`SchemeMap` from `_from` to `_to`.
  """
    if scheme_map is None:
        from_scheme = SCHEMES[_from]
        to_scheme = SCHEMES[_to]
        scheme_map = sanscript.SchemeMap(from_scheme, to_scheme)
    return sanscript.transliterate(data=data, scheme_map=scheme_map)


Instruction:
Make a summary of the following Python 3 code
def _setup():
    s = str.split
    if sys.version_info < (3, 0):
        # noinspection PyUnresolvedReferences
        s = unicode.split

    def pop_all(some_dict, some_list):
        for scheme in some_list:
            some_dict.pop(scheme)
    global SCHEMES
    SCHEMES = copy.deepcopy(sanscript.SCHEMES)
    pop_all(SCHEMES, [sanscript.ORIYA, sanscript.BENGALI, sanscript.GUJARATI])
    SCHEMES[HK].update({
        'vowels': s("""a A i I u U R RR lR lRR E ai O au""") + s("""e o"""),
        'marks': s("""A i I u U R RR lR lRR E ai O au""") + s("""e o"""),
        'consonants': sanscript.SCHEMES[HK]['consonants'] + s("""n2 r2 zh""")
    })
    SCHEMES[ITRANS].update({
        'vowels': s("""a A i I u U R RR LLi LLI E ai O au""") + s("""e o"""),
        'marks': s("""A i I u U R RR LLi LLI E ai O au""") + s("""e o"""),
        'consonants': sanscript.SCHEMES[ITRANS]['consonants'] + s("""n2 r2 zh""")
    })
    pop_all(SCHEMES[ITRANS].synonym_map, s("""e o"""))
    SCHEMES[OPTITRANS].update({
        'vowels': s("""a A i I u U R RR LLi LLI E ai O au""") + s("""e o"""),
        'marks': s("""A i I u U R RR LLi LLI E ai O au""") + s("""e o"""),
        'consonants': sanscript.SCHEMES[OPTITRANS]['consonants'] + s("""n2 r2 zh""")
    })
    pop_all(SCHEMES[OPTITRANS].synonym_map, s("""e o"""))

Response: Add a variety of default schemes.


Instruction:
How would you code a function in Python 3 to
convert an array of integers to utf8 string

Response:
def to_utf8(y):
    """
    converts an array of integers to utf8 string
    """

    out = []

    for x in y:

        if x < 0x080:
            out.append(x)
        elif x < 0x0800:
            out.append((x >> 6) | 0xC0)
            out.append((x & 0x3F) | 0x80)
        elif x < 0x10000:
            out.append((x >> 12) | 0xE0)
            out.append(((x >> 6) & 0x3F) | 0x80)
            out.append((x & 0x3F) | 0x80)
        else:
            out.append((x >> 18) | 0xF0)
            out.append((x >> 12) & 0x3F)
            out.append(((x >> 6) & 0x3F) | 0x80)
            out.append((x & 0x3F) | 0x80)

    return ''.join(map(chr, out))


Instruction:
Implement a Python 3 function for
setting the value of delta to reflect the current codepage s script

Response:
def set_script(self, i):
        """
        set the value of delta to reflect the current codepage

        """

        if i in range(1, 10):
            n = i - 1
        else:
            raise IllegalInput("Invalid Value for ATR %s" % (hex(i)))

        if n > -1: # n = -1 is the default script ..
            self.curr_script = n
            self.delta = n * DELTA

        return


Instruction:
Given the following Python 3 function, write the documentation
def main(argv=None):
    print (transliterate('jaya gaNeza! zrIrAmajayam', 'harvardkyoto', 'devanagari'))
    if argv is None:
        argv = sys.argv
    try:
        text, inFormat, outFormat = argv[1:4]
    except ValueError:
        print (main.__doc__)
        return 2
    inFormat = inFormat.upper()
    outFormat = outFormat.upper()
    # try assuming "text" is a filename
    try:
        f = open(text)
    except IOError:
        # it wasn't, so it must be the actual text
        print (transliterate(text, inFormat, outFormat))
        return 0
    else:
        i = 1
        for text in f.readlines():
            if len(text) > 0 and not text.startswith('#'):
                print (transliterate(text, inFormat, outFormat).strip('\n'))
            i = i + 1
        f.close()
        return 0

Response: This function is the main function for transliterator. py.


Instruction:
How would you code a function in Python 3 to
transliterate a text into the target format.

Response:
def _transliterate(self, text, outFormat):
        """ Transliterate a devanagari text into the target format.

        Transliterating a character to or from Devanagari is not a simple
        lookup: it depends on the preceding and following characters.
        """
        def getResult():
            if curMatch.isspace():
                result.append(curMatch)
                return
            if prevMatch in self:
                prev = self[prevMatch]
            else:
                prev = None
            if nextMatch in self:
                next = self[nextMatch]
            else:
                next = None
            try:
                equiv = outFormat._equivalent(self[curMatch],
                                                            prev, #self.get(prevMatch, None),
                                                            next, #self.get(nextMatch, None),
                                                            self._implicitA)
            except KeyError:
                equiv = _unrecognised(curMatch)
            for e in equiv:
                result.append(e)

        def incr(c):
            if self._longestEntry == 1:
                return 1
            return len(c)

        result = []
        text = self._preprocess(text)
        i = 0
        prevMatch = None
        nextMatch = None
        curMatch = self._getNextChar(text, i)
        i = i + len(curMatch)
        while i < len(text):
            nextMatch = self._getNextChar(text, i)
            getResult()
            i = i + len(nextMatch)
            prevMatch = curMatch
            curMatch = nextMatch
            nextMatch = None
        getResult()
        return result


Instruction:
Here you have a function in Python 3, explain what it does
def _equivalent(self, char, prev, next, implicitA):
        result = []
        if char.isVowel == False:
            result.append(char.chr)
            if char.isConsonant \
            and ((next is not None and next.isConsonant) \
            or next is None):
                result.append(DevanagariCharacter._VIRAMA)
        else:
            if prev is None or prev.isConsonant == False:
                result.append(char.chr)
            else:
                if char._dependentVowel is not None:
                    result.append(char._dependentVowel)
        return result

Response: Translate a Latin character equivalent to Devanagari.


Instruction:
Can you tell what is the following Python 3 function doing
def from_devanagari(self, data):
        from indic_transliteration import sanscript
        return sanscript.transliterate(data=data, _from=sanscript.DEVANAGARI, _to=self.name)

Response: A convenience method that transliterate a string from the DEVANAGARI to the name of the object.


Instruction:
Write a Python 3 function for
loading and generate num top - level rules from the specified grammar file and write the result to the specified output stream.

Response:
def generate(grammar=None, num=1, output=sys.stdout, max_recursion=10, seed=None):
    """Load and generate ``num`` number of top-level rules from the specified grammar.

    :param list grammar: The grammar file to load and generate data from
    :param int num: The number of times to generate data
    :param output: The output destination (an open, writable stream-type object. default=``sys.stdout``)
    :param int max_recursion: The maximum reference-recursion when generating data (default=``10``)
    :param int seed: The seed to initialize the PRNG with. If None, will not initialize it.
    """
    if seed is not None:
        gramfuzz.rand.seed(seed)

    fuzzer = gramfuzz.GramFuzzer()
    fuzzer.load_grammar(grammar)

    cat_group = os.path.basename(grammar).replace(".py", "")

    results = fuzzer.gen(cat_group=cat_group, num=num, max_recursion=max_recursion)
    for res in results:
        output.write(res)


Instruction:
How would you implement a function in Python 3 that
builds the quote instance of the object.

Response:
def build(self, pre=None, shortest=False):
        """Build the ``Quote`` instance

        :param list pre: The prerequisites list
        :param bool shortest: Whether or not the shortest reference-chain (most minimal) version of the field should be generated.
        """
        res = super(Q, self).build(pre, shortest=shortest)

        if self.escape:
            return repr(res)
        elif self.html_js_escape:
            return ("'" + res.encode("string_escape").replace("<", "\\x3c").replace(">", "\\x3e") + "'")
        else:
            return "".join([self.quote, res, self.quote])


Instruction:
Can you write a function in Python 3 where it
makes the list of verbs into present participles

Response:
def make_present_participles(verbs):
    """Make the list of verbs into present participles

    E.g.:

        empower -> empowering
        drive -> driving
    """
    res = []
    for verb in verbs:
        parts = verb.split()
        if parts[0].endswith("e"):
            parts[0] = parts[0][:-1] + "ing"
        else:
            parts[0] = parts[0] + "ing"
        res.append(" ".join(parts))
    return res


Instruction:
Make a summary of the following Python 3 code
def clear_sent_messages(self, offset=None):
        if offset is None:
            offset = getattr(settings, 'MAILQUEUE_CLEAR_OFFSET', defaults.MAILQUEUE_CLEAR_OFFSET)

        if type(offset) is int:
            offset = datetime.timedelta(hours=offset)

        delete_before = timezone.now() - offset
        self.filter(sent=True, last_attempt__lte=delete_before).delete()

Response: Deletes all sent MailerMessage records


Instruction:
Can you implement a function in Python 3 that
returns string with unicode values

Response:
def unicodevalues_asstring(values):
    """ Return string with unicodenames (unless that is disabled) """
    if not os.environ.get('DISABLE_UNAMES'):
        return map(lambda x: '%s' % format(x).strip(), values)
    return map(lambda x: u'U+%04x %s' % (x, unichr(x)), sorted(values))


Instruction:
Can you create a Python 3 function that
loads the includes of an encoding Namelist file.

Response:
def _loadNamelistIncludes(item, unique_glyphs, cache):
  """Load the includes of an encoding Namelist files.

  This is an implementation detail of readNamelist.
  """
  includes = item["includes"] = []
  charset = item["charset"] = set() | item["ownCharset"]

  noCharcode = item["noCharcode"] = set() | item["ownNoCharcode"]

  dirname =  os.path.dirname(item["fileName"])
  for include in item["header"]["includes"]:
    includeFile = os.path.join(dirname, include)
    try:
      includedItem = readNamelist(includeFile, unique_glyphs, cache)
    except NamelistRecursionError:
      continue
    if includedItem in includes:
      continue
    includes.append(includedItem)
    charset |= includedItem["charset"]
    noCharcode |= includedItem["ownNoCharcode"]
  return item


Instruction:
Can you tell what is the following Python 3 function doing
def __readNamelist(cache, filename, unique_glyphs):
  if filename in cache:
    item = cache[filename]
  else:
    cps, header, noncodes = parseNamelist(filename)
    item = {
      "fileName": filename
    , "ownCharset": cps
    , "header": header
    , "ownNoCharcode": noncodes
    , "includes": None # placeholder
    , "charset": None # placeholder
    , "noCharcode": None
    }
    cache[filename] = item

  if unique_glyphs or item["charset"] is not None:
    return item

  # full-charset/includes are requested and not cached yet
  _loadNamelistIncludes(item, unique_glyphs, cache)
  return item

Response: Return a dict with the data of an encoding Namelist file.


Instruction:
Write a Python 3 function for
detecting infinite recursion and prevent it. This is an implementation detail of readNamelist. Raises NamelistRecursionError if namFilename is in the process of being included

Response:
def _readNamelist(currentlyIncluding, cache, namFilename, unique_glyphs):
  """ Detect infinite recursion and prevent it.

  This is an implementation detail of readNamelist.

  Raises NamelistRecursionError if namFilename is in the process of being included
  """
  # normalize
  filename = os.path.abspath(os.path.normcase(namFilename))
  if filename in currentlyIncluding:
    raise NamelistRecursionError(filename)
  currentlyIncluding.add(filename)
  try:
    result = __readNamelist(cache, filename, unique_glyphs)
  finally:
    currentlyIncluding.remove(filename)
  return result


Instruction:
Make a summary of the following Python 3 code
def readNamelist(namFilename, unique_glyphs=False, cache=None):
  currentlyIncluding = set()
  if not cache:
    cache = {}
  return _readNamelist(currentlyIncluding, cache, namFilename, unique_glyphs)

Response: Reads a Namelist file and returns a dict of the contents.


Instruction:
Explain what the following Python 3 code does
def codepointsInNamelist(namFilename, unique_glyphs=False, cache=None):
  key = 'charset' if not unique_glyphs else 'ownCharset'

  internals_dir = os.path.dirname(os.path.abspath(__file__))
  target = os.path.join(internals_dir, namFilename)
  result = readNamelist(target, unique_glyphs, cache)
  return result[key]

Response: Returns the set of codepoints contained in a given Namelist file.


Instruction:
Given the following Python 3 function, write the documentation
def get_orthographies(self, _library=library):
        ''' Returns list of CharsetInfo about supported orthographies '''
        results = []
        for charset in _library.charsets:
            if self._charsets:
                cn = getattr(charset, 'common_name', False)
                abbr = getattr(charset, 'abbreviation', False)
                nn = getattr(charset, 'short_name', False)
                naive = getattr(charset, 'native_name', False)

                if cn and cn.lower() in self._charsets:
                    results.append(charset)

                elif nn and nn.lower() in self._charsets:
                    results.append(charset)

                elif naive and naive.lower() in self._charsets:
                    results.append(charset)

                elif abbr and abbr.lower() in self._charsets:
                    results.append(charset)
            else:
                results.append(charset)

        for result in results:
            yield CharsetInfo(self, result)

Response: Returns list of CharsetInfo about supported orthographies


Instruction:
Here you have a function in Python 3, explain what it does
def get_codepoints():
        # response = requests.get(EXTENSIS_LANG_XML)
        # if response.status_code != 200:
        #     return []

        path = get_from_cache('languages.xml', EXTENSIS_LANG_XML)

        try:
            xml_content = open(path, 'r').read()
        except IOError:
            logging.error('Could not read languages.xml from cache')
            xml_content = ''

        content = re.sub('<!--.[^>]*-->', '', xml_content)
        doc = etree.fromstring(content.lstrip('`'))

        return doc.findall('.//scanning-codepoints')

Response: Return all XML <scanning - codepoints > in received XML


Instruction:
Implement a function in Python 3 to
return list of unicode codepoints for the given codepoint.

Response:
def get_unicodes(codepoint):
        """ Return list of unicodes for <scanning-codepoints> """
        result = re.sub('\s', '', codepoint.text)
        return Extension.convert_to_list_of_unicodes(result)


Instruction:
Write a Python 3 function that can
handle the OAuth2 flow.

Response:
def handler(self,):
        """* get request token if OAuth1
            * Get user authorization
            * Get access token
        """

        if self.oauth_version == 'oauth1':
            request_token, request_token_secret = self.oauth.get_request_token(params={'oauth_callback': self.callback_uri})
            logger.debug("REQUEST_TOKEN = {0}\n REQUEST_TOKEN_SECRET = {1}\n".format(request_token, request_token_secret))
            authorize_url = self.oauth.get_authorize_url(request_token)
        else:
            authorize_url = self.oauth.get_authorize_url(client_secret=self.consumer_secret, redirect_uri=self.callback_uri, response_type='code')

        logger.debug("AUTHORISATION URL : {0}".format(authorize_url))
        # Open authorize_url
        webbrowser.open(authorize_url)
        self.verifier = input("Enter verifier : ")

        self.token_time = time.time()

        credentials = {'token_time': self.token_time}

        if self.oauth_version == 'oauth1':
            raw_access = self.oauth.get_raw_access_token(request_token, request_token_secret, params={"oauth_verifier": self.verifier})
            parsed_access = parse_utf8_qsl(raw_access.content)
            self.access_token = parsed_access['oauth_token']
            self.access_token_secret = parsed_access['oauth_token_secret']
            self.session_handle = parsed_access['oauth_session_handle']
            self.guid = parsed_access['xoauth_yahoo_guid']

            # Updating credentials
            credentials.update({
                'access_token': self.access_token,
                'access_token_secret': self.access_token_secret,
                'session_handle': self.session_handle,
                'guid': self.guid
            })
        else:
            # Building headers
            headers = self.generate_oauth2_headers()
            # Getting access token
            raw_access = self.oauth.get_raw_access_token(data={"code": self.verifier, 'redirect_uri': self.callback_uri,'grant_type':'authorization_code'}, headers=headers)
            #parsed_access = parse_utf8_qsl(raw_access.content.decode('utf-8'))
            credentials.update(self.oauth2_access_parser(raw_access))

        return credentials


Instruction:
Can you generate a brief explanation for the following Python 3 code
def generate_oauth2_headers(self):
        encoded_credentials = base64.b64encode(('{0}:{1}'.format(self.consumer_key,self.consumer_secret)).encode('utf-8'))
        headers={
            'Authorization':'Basic {0}'.format(encoded_credentials.decode('utf-8')),
            'Content-Type': 'application/x-www-form-urlencoded'
        }

        return headers

Response: Generates header for oauth2


Instruction:
How would you explain what the following Python 3 function does
def oauth2_access_parser(self, raw_access):
        parsed_access = json.loads(raw_access.content.decode('utf-8'))
        self.access_token = parsed_access['access_token']
        self.token_type = parsed_access['token_type']
        self.refresh_token = parsed_access['refresh_token']
        self.guid = parsed_access['xoauth_yahoo_guid']

        credentials = {
            'access_token': self.access_token,
            'token_type': self.token_type,
            'refresh_token': self.refresh_token,
            'guid': self.guid
        }

        return credentials

Response: Parse the oauth2 access token and return a dictionary of credentials.


Instruction:
Make a summary of the following Python 3 code
def refresh_access_token(self,):
        logger.debug("REFRESHING TOKEN")
        self.token_time = time.time()
        credentials = {
            'token_time': self.token_time
        }

        if self.oauth_version == 'oauth1':
            self.access_token, self.access_token_secret = self.oauth.get_access_token(self.access_token, self.access_token_secret, params={"oauth_session_handle": self.session_handle})
            credentials.update({
                'access_token': self.access_token,
                'access_token_secret': self.access_token_secret,
                'session_handle': self.session_handle,
                'token_time': self.token_time
            })
        else:
            headers = self.generate_oauth2_headers()

            raw_access = self.oauth.get_raw_access_token(data={"refresh_token": self.refresh_token, 'redirect_uri': self.callback_uri,'grant_type':'refresh_token'}, headers=headers)
            credentials.update(self.oauth2_access_parser(raw_access))

        return credentials

Response: Refresh the access token and return a dictionary with the new token and the new token_time.


Instruction:
Create a Python 3 function for
checking the validity of the token

Response:
def token_is_valid(self,):
        """Check the validity of the token :3600s
        """
        elapsed_time = time.time() - self.token_time
        logger.debug("ELAPSED TIME : {0}".format(elapsed_time))
        if elapsed_time > 3540: # 1 minute before it expires
            logger.debug("TOKEN HAS EXPIRED")
            return False

        logger.debug("TOKEN IS STILL VALID")
        return True


Instruction:
Create a Python 3 function for
calling right function according to file extension

Response:
def get_data(filename):
    """Calls right function according to file extension
    """
    name, ext = get_file_extension(filename)
    func = json_get_data if ext == '.json' else yaml_get_data
    return func(filename)


Instruction:
Explain what the following Python 3 code does
def write_data(data, filename):
    name, ext = get_file_extension(filename)
    func = json_write_data if ext == '.json' else yaml_write_data
    return func(data, filename)

Response: Call right func to save data according to file extension


Instruction:
Can you tell what is the following Python 3 function doing
def json_write_data(json_data, filename):
    with open(filename, 'w') as fp:
        json.dump(json_data, fp, indent=4, sort_keys=True, ensure_ascii=False)
        return True
    return False

Response: Write json data into a file


Instruction:
Can you tell what is the following Python 3 function doing
def json_get_data(filename):
    with open(filename) as fp:
        json_data = json.load(fp)
        return json_data

    return False

Response: Get data from json file


Instruction:
Given the following Python 3 function, write the documentation
def yaml_get_data(filename):
    with open(filename, 'rb') as fd:
        yaml_data = yaml.load(fd)
        return yaml_data
    return False

Response: Get data from. yml file


Instruction:
Implement a Python 3 function for
writing data into a. yml file

Response:
def yaml_write_data(yaml_data, filename):
    """Write data into a .yml file
    """
    with open(filename, 'w') as fd:
        yaml.dump(yaml_data, fd, default_flow_style=False)
        return True

    return False


Instruction:
Implement a function in Python 3 to
fit the class to the raw pairwise distances.

Response:
def fit(self, X, y=None):
        '''
        If scale_by_median, find :attr:`median_`; otherwise, do nothing.

        Parameters
        ----------
        X : array
            The raw pairwise distances.
        '''

        X = check_array(X)
        if self.scale_by_median:
            self.median_ = np.median(X[np.triu_indices_from(X, k=1)],
                                     overwrite_input=True)
        elif hasattr(self, 'median_'):
            del self.median_
        return self


Instruction:
Explain what the following Python 3 code does
def transform(self, X):
        '''
        Turns distances into RBF values.

        Parameters
        ----------
        X : array
            The raw pairwise distances.

        Returns
        -------
        X_rbf : array of same shape as X
            The distances in X passed through the RBF kernel.
        '''
        X = check_array(X)
        X_rbf = np.empty_like(X) if self.copy else X

        X_in = X
        if not self.squared:
            np.power(X_in, 2, out=X_rbf)
            X_in = X_rbf

        if self.scale_by_median:
            scale = self.median_ if self.squared else self.median_ ** 2
            gamma = self.gamma * scale
        else:
            gamma = self.gamma
        np.multiply(X_in, -gamma, out=X_rbf)

        np.exp(X_rbf, out=X_rbf)
        return X_rbf

Response: Turns distances into RBF values.


Instruction:
How would you explain what the following Python 3 function does
def fit(self, X, y=None):
        '''
        Learn the linear transformation to clipped eigenvalues.

        Note that if min_eig isn't zero and any of the original eigenvalues
        were exactly zero, this will leave those eigenvalues as zero.

        Parameters
        ----------
        X : array, shape [n, n]
            The *symmetric* input similarities. If X is asymmetric, it will be
            treated as if it were symmetric based on its lower-triangular part.
        '''
        n = X.shape[0]
        if X.shape != (n, n):
            raise TypeError("Input must be a square matrix.")

        # TODO: only get negative eigs somehow?
        memory = get_memory(self.memory)
        vals, vecs = memory.cache(scipy.linalg.eigh, ignore=['overwrite_a'])(
            X, overwrite_a=not self.copy)
        vals = vals.reshape(-1, 1)

        if self.min_eig == 0:
            inner = vals > self.min_eig
        else:
            with np.errstate(divide='ignore'):
                inner = np.where(vals >= self.min_eig, 1,
                                 np.where(vals == 0, 0, self.min_eig / vals))

        self.clip_ = np.dot(vecs, inner * vecs.T)
        return self

Response: Fit the Linear transformation to clipped eigenvalues.


Instruction:
Make a summary of the following Python 3 code
def fit(self, X, y=None):
        '''
        Learn the linear transformation to flipped eigenvalues.

        Parameters
        ----------
        X : array, shape [n, n]
            The *symmetric* input similarities. If X is asymmetric, it will be
            treated as if it were symmetric based on its lower-triangular part.
        '''
        n = X.shape[0]
        if X.shape != (n, n):
            raise TypeError("Input must be a square matrix.")

        # TODO: only get negative eigs somehow?
        memory = get_memory(self.memory)
        vals, vecs = memory.cache(scipy.linalg.eigh, ignore=['overwrite_a'])(
            X, overwrite_a=not self.copy)
        vals = vals[:, None]

        self.flip_ = np.dot(vecs, np.sign(vals) * vecs.T)
        return self

Response: Fits the classical entry point to the input similarities.


Instruction:
How would you explain what the following Python 3 function does
def transform(self, X):
        '''
        Transforms X according to the linear transformation corresponding to
        flipping the input eigenvalues.

        Parameters
        ----------
        X : array, shape [n_test, n]
            The test similarities to training points.

        Returns
        -------
        Xt : array, shape [n_test, n]
            The transformed test similarites to training points.
        '''
        n = self.flip_.shape[0]
        if X.ndim != 2 or X.shape[1] != n:
            msg = "X should have {} columns, the number of samples at fit time"
            raise TypeError(msg.format(self.flip_.shape[0]))
        return np.dot(X, self.flip_)

Response:
Transforms X according to the linear transformation corresponding to
            flipping the input eigenvalues.


Instruction:
Write a Python 3 script for
flipping the negative eigenvalues of X and transforms the training similarities to the original training similarities.

Response:
def fit_transform(self, X, y=None):
        '''
        Flips the negative eigenvalues of X.

        Parameters
        ----------
        X : array, shape [n, n]
            The *symmetric* input similarities. If X is asymmetric, it will be
            treated as if it were symmetric based on its lower-triangular part.

        Returns
        -------
        Xt : array, shape [n, n]
            The transformed training similarities.
        '''
        n = X.shape[0]
        if X.shape != (n, n):
            raise TypeError("Input must be a square matrix.")

        memory = get_memory(self.memory)
        discard_X = not self.copy and self.negatives_likely
        vals, vecs = memory.cache(scipy.linalg.eigh, ignore=['overwrite_a'])(
            X, overwrite_a=discard_X)
        vals = vals[:, None]

        self.clip_ = np.dot(vecs, np.sign(vals) * vecs.T)

        if discard_X or vals[0, 0] < 0:
            del X
            np.abs(vals, out=vals)
            X = np.dot(vecs, vals * vecs.T)
            del vals, vecs

            # should be symmetric, but make sure because floats
            X = Symmetrize(copy=False).fit_transform(X)
        return X


Instruction:
Here you have a function in Python 3, explain what it does
def fit(self, X, y=None):
        '''
        Learn the transformation to shifted eigenvalues. Only depends
        on the input dimension.

        Parameters
        ----------
        X : array, shape [n, n]
            The *symmetric* input similarities.
        '''
        n = X.shape[0]
        if X.shape != (n, n):
            raise TypeError("Input must be a square matrix.")

        self.train_ = X

        memory = get_memory(self.memory)
        lo, = memory.cache(scipy.linalg.eigvalsh)(X, eigvals=(0, 0))
        self.shift_ = max(self.min_eig - lo, 0)

        return self

Response: Learn the transformation to shifted eigenvalues. Only dependsOnTheInputDimension on the input dimension. Only dependsOnTheInputDimension on the input dimension. Only dependsOnTheInputDimension on the input dimension. Only dependsOnTheInputDimension on the input dimension.


Instruction:
Can you tell what is the following Python 3 function doing
def transform(self, X):
        '''
        Transforms X according to the linear transformation corresponding to
        shifting the input eigenvalues to all be at least ``self.min_eig``.

        Parameters
        ----------
        X : array, shape [n_test, n]
            The test similarities to training points.

        Returns
        -------
        Xt : array, shape [n_test, n]
            The transformed test similarites to training points. Only different
            from X if X is the training data.
        '''
        n = self.train_.shape[0]
        if X.ndim != 2 or X.shape[1] != n:
            msg = "X should have {} columns, the number of samples at fit time"
            raise TypeError(msg.format(n))

        if self.copy:
            X = X.copy()

        if self.shift_ != 0 and X is self.train_ or (
                X.shape == self.train_.shape and np.allclose(X, self.train_)):
            X[xrange(n), xrange(n)] += self.shift_
        return X

Response:
Transforms X according to the linear transformation corresponding to
           .


Instruction:
Can you generate the documentation for the following Python 3 function
def fit(self, X, y=None):
        '''
        Picks the elements of the basis to use for the given data.

        Only depends on the dimension of X. If it's more convenient, you can
        pass a single integer for X, which is the dimension to use.

        Parameters
        ----------
        X : an integer, a :class:`Features` instance, or a list of bag features
            The input data, or just its dimension, since only the dimension is
            needed here.
        '''
        if is_integer(X):
            dim = X
        else:
            X = as_features(X)
            dim = X.dim
        M = self.smoothness

        # figure out the smooth-enough elements of our basis
        inds = np.mgrid[(slice(M + 1),) * dim].reshape(dim, (M + 1) ** dim).T
        self.inds_ = inds[(inds ** 2).sum(axis=1) <= M ** 2]
        return self

Response: Fits the object to the given data.


Instruction:
Create a Python 3 function for
transforming a list of bag features into a list of projection series series representation.

Response:
def transform(self, X):
        '''
        Transform a list of bag features into its projection series
        representation.

        Parameters
        ----------
        X : :class:`skl_groups.features.Features` or list of bag feature arrays
            New data to transform. The data should all lie in [0, 1];
            use :class:`skl_groups.preprocessing.BagMinMaxScaler` if not.

        Returns
        -------
        X_new : integer array, shape ``[len(X), dim_]``
            X transformed into the new space.
        '''
        self._check_fitted()
        M = self.smoothness
        dim = self.dim_
        inds = self.inds_
        do_check = self.do_bounds_check

        X = as_features(X)
        if X.dim != dim:
            msg = "model fit for dimension {} but got dim {}"
            raise ValueError(msg.format(dim, X.dim))

        Xt = np.empty((len(X), self.inds_.shape[0]))
        Xt.fill(np.nan)

        if self.basis == 'cosine':  # TODO: put this in a C extension?
            coefs = (np.pi * np.arange(M + 1))[..., :]
            for i, bag in enumerate(X):
                if do_check:
                    if np.min(bag) < 0 or np.max(bag) > 1:
                        raise ValueError("Bag {} not in [0, 1]".format(i))

                # apply each phi func to each dataset point: n x dim x M
                phi = coefs * bag[..., np.newaxis]
                np.cos(phi, out=phi)
                phi[:, :, 1:] *= np.sqrt(2)

                # B is the evaluation of each tensor-prodded basis func
                # at each point: n x inds.shape[0]
                B = reduce(op.mul, (phi[:, i, inds[:, i]] for i in xrange(dim)))

                Xt[i, :] = np.mean(B, axis=0)
        else:
            raise ValueError("unknown basis '{}'".format(self.basis))

        return Xt


Instruction:
Can you tell what is the following Python 3 function doing
def get_version(self):
        if (self.name is not None and self.version is not None
            and self.version.startswith(":versiontools:")):
            return (self.__get_live_version() or self.__get_frozen_version()
                    or self.__fail_to_get_any_version())
        else:
            return self.__base.get_version(self)

Response: Get the version of the current version of the object.


Instruction:
Create a Python 3 function to
get a live version string using versiontools. Version. from_expression

Response:
def __get_live_version(self):
        """
        Get a live version string using versiontools
        """
        try:
            import versiontools
        except ImportError:
            return None
        else:
            return str(versiontools.Version.from_expression(self.name))


Instruction:
Implement a function in Python 3 to
fit the transformer on the stacked points.

Response:
def fit(self, X, y=None, **params):
        '''
        Fit the transformer on the stacked points.

        Parameters
        ----------
        X : :class:`Features` or list of arrays of shape ``[n_samples[i], n_features]``
            Training set. If a Features object, it will be stacked.

        any other keyword argument :
            Passed on as keyword arguments to the transformer's ``fit()``.
        '''
        X = as_features(X, stack=True)
        self.transformer.fit(X.stacked_features, y, **params)
        return self


Instruction:
Can you generate the documentation for the following Python 3 function
def transform(self, X, **params):
        '''
        Transform the stacked points.

        Parameters
        ----------
        X : :class:`Features` or list of bag feature arrays
            New data to transform.

        any other keyword argument :
            Passed on as keyword arguments to the transformer's ``transform()``.

        Returns
        -------
        X_new : :class:`Features`
            Transformed features.
        '''
        X = as_features(X, stack=True)
        X_new = self.transformer.transform(X.stacked_features, **params)
        return self._gather_outputs(X, X_new)

Response: Transform the stacked points X.


Instruction:
Can you tell what is the following Python 3 function doing
def fit_transform(self, X, y=None, **params):
        '''
        Fit and transform the stacked points.

        Parameters
        ----------
        X : :class:`Features` or list of bag feature arrays
            Data to train on and transform.

        any other keyword argument :
            Passed on as keyword arguments to the transformer's ``transform()``.

        Returns
        -------
        X_new : :class:`Features`
            Transformed features.
        '''
        X = as_features(X, stack=True)
        X_new = self.transformer.fit_transform(X.stacked_features, y, **params)
        return self._gather_outputs(X, X_new)

Response: Fit and transform the stacked points.


Instruction:
Write a Python 3 function for
transforming data back to its original space i. e. return an input X_original whose transform would be X.

Response:
def inverse_transform(self, X, **params):
        '''
        Transform data back to its original space, i.e., return an input
        X_original whose transform would (maybe approximately) be X.

        Parameters
        ----------
        X : :class:`Features` or list of bag feature arrays
            Data to train on and transform.

        any other keyword argument :
            Passed on as keyword arguments to the transformer's
            ``inverse_transform()``.

        Returns
        -------
        X_original : :class:`Features`
        '''
        X = as_features(X, stack=True)
        Xo = self.transformer.inverse_transform(X.stacked_features, **params)
        return self._gather_outputs(X, Xo)


Instruction:
Can you generate the documentation for the following Python 3 function
def fit(self, X, y=None):
        X = check_array(X, copy=self.copy,
                        dtype=[np.float64, np.float32, np.float16, np.float128])

        feature_range = self.feature_range
        if feature_range[0] >= feature_range[1]:
            raise ValueError("Minimum of desired feature range must be smaller"
                             " than maximum. Got %s." % str(feature_range))
        if self.fit_feature_range is not None:
            fit_feature_range = self.fit_feature_range
            if fit_feature_range[0] >= fit_feature_range[1]:
                raise ValueError("Minimum of desired (fit) feature range must "
                                 "be smaller than maximum. Got %s."
                                 % str(feature_range))
            if (fit_feature_range[0] < feature_range[0] or
                    fit_feature_range[1] > feature_range[1]):
                raise ValueError("fit_feature_range must be a subset of "
                                 "feature_range. Got %s, fit %s."
                                 % (str(feature_range),
                                    str(fit_feature_range)))
            feature_range = fit_feature_range

        data_min = np.min(X, axis=0)
        data_range = np.max(X, axis=0) - data_min
        # Do not scale constant features
        data_range[data_range == 0.0] = 1.0
        self.scale_ = (feature_range[1] - feature_range[0]) / data_range
        self.min_ = feature_range[0] - data_min * self.scale_
        self.data_range = data_range
        self.data_min = data_min
        return self

Response: Fit the minimum and maximum of the current object to the data.


Instruction:
Write a Python 3 script to
scale features of X according to feature_range.

Response:
def transform(self, X):
        """Scaling features of X according to feature_range.

        Parameters
        ----------
        X : array-like with shape [n_samples, n_features]
            Input data that will be transformed.
        """
        X = check_array(X, copy=self.copy)
        X *= self.scale_
        X += self.min_
        if self.truncate:
            np.maximum(self.feature_range[0], X, out=X)
            np.minimum(self.feature_range[1], X, out=X)
        return X


Instruction:
Create a Python 3 function for
undoing the scaling of X according to feature_range.

Response:
def inverse_transform(self, X):
        """Undo the scaling of X according to feature_range.

        Note that if truncate is true, any truncated points will not
        be restored exactly.

        Parameters
        ----------
        X : array-like with shape [n_samples, n_features]
            Input data that will be transformed.
        """
        X = check_array(X, copy=self.copy)
        X -= self.min_
        X /= self.scale_
        return X


Instruction:
How would you implement a function in Python 3 that
fits the KMeans model on a training set.

Response:
def fit(self, X, y=None):
        '''
        Choose the codewords based on a training set.

        Parameters
        ----------
        X : :class:`skl_groups.features.Features` or list of arrays of shape ``[n_samples[i], n_features]``
            Training set. If a Features object, it will be stacked.
        '''
        self.kmeans_fit_ = copy(self.kmeans)
        X = as_features(X, stack=True)
        self.kmeans_fit_.fit(X.stacked_features)
        return self


Instruction:
How would you explain what the following Python 3 function does
def transform(self, X):
        '''
        Transform a list of bag features into its bag-of-words representation.

        Parameters
        ----------
        X : :class:`skl_groups.features.Features` or list of bag feature arrays
            New data to transform.

        Returns
        -------
        X_new : integer array, shape [len(X), kmeans.n_clusters]
            X transformed into the new space.
        '''
        self._check_fitted()
        X = as_features(X, stack=True)
        assignments = self.kmeans_fit_.predict(X.stacked_features)
        return self._group_assignments(X, assignments)

Response: Transform a list of bag features into its bag - of - words representation.


Instruction:
How would you code a function in Python 3 to
compute clustering and transform a list of bag features into its bag - of - words representation. Like calling fit and then transform ( X ).

Response:
def fit_transform(self, X):
        '''
        Compute clustering and transform a list of bag features into its
        bag-of-words representation. Like calling fit(X) and then transform(X),
        but more efficient.

        Parameters
        ----------
        X : :class:`skl_groups.features.Features` or list of bag feature arrays
            New data to transform.

        Returns
        -------
        X_new : integer array, shape [len(X), kmeans.n_clusters]
            X transformed into the new space.
        '''
        X = as_features(X, stack=True)
        self.kmeans_fit_ = copy(self.kmeans)
        assignments = self.kmeans_fit_.fit_predict(X.stacked_features)
        return self._group_assignments(X, assignments)


Instruction:
Can you implement a function in Python 3 that
checks whether the array is either integral or boolean.

Response:
def is_categorical_type(ary):
    "Checks whether the array is either integral or boolean."
    ary = np.asanyarray(ary)
    return is_integer_type(ary) or ary.dtype.kind == 'b'


Instruction:
Can you create a Python 3 function that
returns argument as an integer array converting floats if convertable.

Response:
def as_integer_type(ary):
    '''
    Returns argument as an integer array, converting floats if convertable.
    Raises ValueError if it's a float array with nonintegral values.
    '''
    ary = np.asanyarray(ary)
    if is_integer_type(ary):
        return ary
    rounded = np.rint(ary)
    if np.any(rounded != ary):
        raise ValueError("argument array must contain only integers")
    return rounded.astype(int)


Instruction:
Implement a function in Python 3 to
display the progress logs for a given module.

Response:
def show_progress(name, **kwargs):
    '''
    Sets up a :class:`ProgressBarHandler` to handle progess logs for
    a given module.

    Parameters
    ----------
    name : string
        The module name of the progress logger to use. For example,
        :class:`skl_groups.divergences.KNNDivergenceEstimator`
        uses ``'skl_groups.divergences.knn.progress'``.

    * : anything
        Other keyword arguments are passed to the :class:`ProgressBarHandler`.
    '''
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)
    logger.addHandler(ProgressBarHandler(**kwargs))


Instruction:
Write a Python 3 script to
signal the start of the process.

Response:
def start(self, total):
        '''
        Signal the start of the process.

        Parameters
        ----------
        total : int
            The total number of steps in the process, or None if unknown.
        '''
        self.logger.info(json.dumps(['START', self.name, total]))


Instruction:
Make a summary of the following Python 3 code
def _build_indices(X, flann_args):
    "Builds FLANN indices for each bag."
    # TODO: should probably multithread this
    logger.info("Building indices...")
    indices = [None] * len(X)
    for i, bag in enumerate(plog(X, name="index building")):
        indices[i] = idx = FLANNIndex(**flann_args)
        idx.build_index(bag)
    return indices

Response: Builds FLANN indices for each bag.


Instruction:
How would you implement a function in Python 3 that
gets within - bag distances for each bag.

Response:
def _get_rhos(X, indices, Ks, max_K, save_all_Ks, min_dist):
    "Gets within-bag distances for each bag."
    logger.info("Getting within-bag distances...")

    if max_K >= X.n_pts.min():
        msg = "asked for K = {}, but there's a bag with only {} points"
        raise ValueError(msg.format(max_K, X.n_pts.min()))

    # need to throw away the closest neighbor, which will always be self
    # thus K=1 corresponds to column 1 in the result array
    which_Ks = slice(1, None) if save_all_Ks else Ks

    indices = plog(indices, name="within-bag distances")
    rhos = [None] * len(X)
    for i, (idx, bag) in enumerate(zip(indices, X)):
        r = np.sqrt(idx.nn_index(bag, max_K + 1)[1][:, which_Ks])
        np.maximum(min_dist, r, out=r)
        rhos[i] = r
    return rhos


Instruction:
Can you tell what is the following Python 3 function doing
def linear(Ks, dim, num_q, rhos, nus):
    r'''
    Estimates the linear inner product \int p q between two distributions,
    based on kNN distances.
    '''
    return _get_linear(Ks, dim)(num_q, rhos, nus)

Response: r Estimates the linear inner product between two distributions.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def renyi(alphas, Ks, dim, required, min_val=np.spacing(1),
          clamp=True, to_self=False):
    r'''
    Estimate the Renyi-alpha divergence between distributions, based on kNN
    distances:  1/(\alpha-1) \log \int p^alpha q^(1-\alpha)

    If the inner integral is less than min_val (default ``np.spacing(1)``),
    uses the log of min_val instead.

    If clamp (the default), enforces that the estimates are nonnegative by
    replacing any negative estimates with 0.

    Returns an array of shape (num_alphas, num_Ks).
    '''
    alphas = np.reshape(alphas, (-1, 1))
    est = required

    est = np.maximum(est, min_val)  # TODO: can we modify in-place?
    np.log(est, out=est)
    est /= alphas - 1
    if clamp:
        np.maximum(est, 0, out=est)
    return est

Response: r Estimates the Renyi - alpha divergence between distributions based on kNN alphabetical distances.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def tsallis(alphas, Ks, dim, required, clamp=True, to_self=False):
    r'''
    Estimate the Tsallis-alpha divergence between distributions, based on kNN
    distances:  (\int p^alpha q^(1-\alpha) - 1) / (\alpha - 1)

    If clamp (the default), enforces the estimate is nonnegative.

    Returns an array of shape (num_alphas, num_Ks).
    '''
    alphas = np.reshape(alphas, (-1, 1))
    alpha_est = required

    est = alpha_est - 1
    est /= alphas - 1
    if clamp:
        np.maximum(est, 0, out=est)
    return est

Response: r Estimates the Tsallis - alpha divergence between distributions based on kNN - alpha distance.


Instruction:
Make a summary of the following Python 3 code
def quadratic(Ks, dim, rhos, required=None):
    r'''
    Estimates \int p^2 based on kNN distances.

    In here because it's used in the l2 distance, above.

    Returns array of shape (num_Ks,).
    '''
    # Estimated with alpha=1, beta=0:
    #   B_{k,d,1,0} is the same as B_{k,d,0,1} in linear()
    # and the full estimator is
    #   B / (n - 1) * mean(rho ^ -dim)
    N = rhos.shape[0]
    Ks = np.asarray(Ks)
    Bs = (Ks - 1) / np.pi ** (dim / 2) * gamma(dim / 2 + 1)  # shape (num_Ks,)
    est = Bs / (N - 1) * np.mean(rhos ** (-dim), axis=0)
    return est

Response: r Estimates \ int p^2 based on kNN distances rhos.


Instruction:
Here you have a function in Python 3, explain what it does
def jensen_shannon(Ks, dim, X_rhos, Y_rhos, required,
                   clamp=True, to_self=False):
    r'''
    Estimate the difference between the Shannon entropy of an equally-weighted
    mixture between X and Y and the mixture of the Shannon entropies:

        JS(X, Y) = H[ (X + Y) / 2 ] - (H[X] + H[Y]) / 2

    We use a special case of the Hino-Murata weighted information estimator with
    a fixed M = n \alpha, about equivalent to the K-nearest-neighbor approach
    used for the other estimators:

        Hideitsu Hino and Noboru Murata (2013).
        Information estimators for weighted observations. Neural Networks.
        http://linkinghub.elsevier.com/retrieve/pii/S0893608013001676

    The estimator for JS(X, Y) is:

        log volume of the unit ball - log M + log(n + m - 1) + digamma(M)
        + 1/2 mean_X( d * log radius of largest ball in X+Y around X_i
                                with no more than M/(n+m-1) weight
                                where X points have weight 1 / (2 n - 1)
                                  and Y points have weight n / (m (2 n - 1))
                      - digamma(# of neighbors in that ball) )
        + 1/2 mean_Y( d * log radius of largest ball in X+Y around Y_i
                                with no more than M/(n+m-1) weight
                                where X points have weight m / (n (2 m - 1))
                                  and Y points have weight 1 / (2 m - 1)
                      - digamma(# of neighbors in that ball) )

        - 1/2 (log volume of the unit ball - log M + log(n - 1) + digamma(M))
        - 1/2 mean_X( d * log radius of the largest ball in X around X_i
                                with no more than M/(n-1) weight
                                where X points have weight 1 / (n - 1))
                      - digamma(# of neighbors in that ball) )

        - 1/2 (log volume of the unit ball - log M + log(m - 1) + digamma(M))
        - 1/2 mean_Y( d * log radius of the largest ball in Y around Y_i
                                with no more than M/(n-1) weight
                                where X points have weight 1 / (m - 1))
                      - digamma(# of neighbors in that ball) )

        =

        log(n + m - 1) + digamma(M)
        + 1/2 mean_X( d * log radius of largest ball in X+Y around X_i
                                with no more than M/(n+m-1) weight
                                where X points have weight 1 / (2 n - 1)
                                  and Y points have weight n / (m (2 n - 1))
                      - digamma(# of neighbors in that ball) )
        + 1/2 mean_Y( d * log radius of largest ball in X+Y around Y_i
                                with no more than M/(n+m-1) weight
                                where X points have weight m / (n (2 m - 1))
                                  and Y points have weight 1 / (2 m - 1)
                      - digamma(# of neighbors in that ball) )
        - 1/2 [log(n-1) + mean_X( d * log rho_M(X_i) )]
        - 1/2 [log(m-1) + mean_Y( d * log rho_M(Y_i) )]
    '''

    X_ns = np.array([rho.shape[0] for rho in X_rhos])
    Y_ns = np.array([rho.shape[0] for rho in Y_rhos])
    n_X = X_ns.size
    n_Y = Y_ns.size

    # cores[0, k, i, j, 0] is mean_X(d * ... - psi(...)) for X[i], Y[j], M=Ks[k]
    # cores[0, k, i, j, 1] is mean_Y(d * ... - psi(...)) for X[i], Y[j], M=Ks[k]
    cores = required
    assert cores.shape == (1, Ks.size, n_X, n_Y, 2)

    # X_bits[k, i] is log(n-1) + mean_X( d * log rho_M(X_i) )  for X[i], M=Ks[k]
    X_bits = np.empty((Ks.size, n_X), dtype=np.float32)
    for i, rho in enumerate(X_rhos):
        X_bits[:, i] = dim * np.mean(np.log(rho), axis=0)
    X_bits += np.log(X_ns - 1)[np.newaxis, :]

    # Y_bits[k, j] is log(n-1) + mean_Y( d * log rho_M(Y_i) )  for Y[j], M=Ks[k]
    Y_bits = np.empty((Ks.size, n_Y), dtype=np.float32)
    for j, rho in enumerate(Y_rhos):
        Y_bits[:, j] = dim * np.mean(np.log(rho), axis=0)
    Y_bits += np.log(Y_ns - 1)[np.newaxis, :]

    est = cores.sum(axis=4)
    est -= X_bits.reshape(1, Ks.size, n_X, 1)
    est -= Y_bits.reshape(1, Ks.size, 1, n_Y)
    est /= 2
    est += np.log(-1 + X_ns[None, None, :, None] + Y_ns[None, None, None, :])
    est += psi(Ks)[None, :, None, None]

    # diagonal is zero
    if to_self:
        est[:, :, xrange(n_X), xrange(n_Y)] = 0

    if clamp:  # know that 0 <= JS <= ln(2)
        np.maximum(0, est, out=est)
        np.minimum(np.log(2), est, out=est)
    return est[:, :, :, :, None]

Response: r Estimates the difference between the Shannon entropy between X and Y and the mixture of the Shannon entropies between X and Y.


Instruction:
How would you explain what the following Python 3 function does
def topological_sort(deps):
    '''
    Topologically sort a DAG, represented by a dict of child => set of parents.
    The dependency dict is destroyed during operation.

    Uses the Kahn algorithm: http://en.wikipedia.org/wiki/Topological_sorting
    Not a particularly good implementation, but we're just running it on tiny
    graphs.
    '''
    order = []
    available = set()

    def _move_available():
        to_delete = []
        for n, parents in iteritems(deps):
            if not parents:
                available.add(n)
                to_delete.append(n)
        for n in to_delete:
            del deps[n]

    _move_available()
    while available:
        n = available.pop()
        order.append(n)
        for parents in itervalues(deps):
            parents.discard(n)
        _move_available()

    if available:
        raise ValueError("dependency cycle found")
    return order

Response: Topologically sort a DAG by a dict of child => set of parents.


Instruction:
How would you explain what the following Python 3 function does
def _parse_specs(specs, Ks):
    '''
    Set up the different functions we need to call.

    Returns:
        - a dict mapping base estimator functions to _FuncInfo objects.
          If the function needs_alpha, then the alphas attribute is an array
          of alpha values and pos is a corresponding array of indices.
          Otherwise, alphas is None and pos is a list containing a single index.
          Indices are >= 0 if they correspond to something in a spec,
          and negative if they're just used for a meta estimator but not
          directly requested.
        - an OrderedDict mapping functions to _MetaFuncInfo objects.
          alphas and pos are like for _FuncInfo; deps is a list of indices
          which should be passed to the estimator. Note that these might be
          other meta functions; this list is guaranteed to be in an order
          such that all dependencies are resolved before calling that function.
          If no such order is possible, raise ValueError.
        - the number of meta-only results

    # TODO: update doctests for _parse_specs

    >>> _parse_specs(['renyi:.8', 'hellinger', 'renyi:.9'])
    ({<function alpha_div at 0x10954f848>:
            _FuncInfo(alphas=[0.8, 0.5, 0.9], pos=[-1, -2, -3])},
     OrderedDict([
        (<function hellinger at 0x10954fc80>,
            _MetaFuncInfo(alphas=None, pos=[1], deps=[array(-2)])),
        (<function renyi at 0x10954fcf8>,
            _MetaFuncInfo(alphas=[0.8, 0.9], pos=[0, 2], deps=[-1, -3]))
     ]), 3)

    >>> _parse_specs(['renyi:.8', 'hellinger', 'renyi:.9', 'l2'])
    ({<function alpha_div at 0x10954f848>:
        _FuncInfo(alphas=[0.8, 0.5, 0.9], pos=[-1, -2, -3]),
      <function linear at 0x10954f758>: _FuncInfo(alphas=None, pos=[-4])
     }, OrderedDict([
        (<function hellinger at 0x10954fc80>,
            _MetaFuncInfo(alphas=None, pos=[1], deps=[array(-2)])),
        (<function l2 at 0x10954fde8>,
            _MetaFuncInfo(alphas=None, pos=[3], deps=[-4])),
        (<function renyi at 0x10954fcf8>,
            _MetaFuncInfo(alphas=[0.8, 0.9], pos=[0, 2], deps=[-1, -3]))
     ]), 4)

    >>> _parse_specs(['renyi:.8', 'hellinger', 'renyi:.9', 'l2', 'linear'])
    ({<function alpha_div at 0x10954f848>:
        _FuncInfo(alphas=[0.8, 0.5, 0.9], pos=[-1, -2, -3]),
      <function linear at 0x10954f758>: _FuncInfo(alphas=None, pos=[4])
     }, OrderedDict([
        (<function hellinger at 0x10954fc80>,
            _MetaFuncInfo(alphas=None, pos=[1], deps=[array(-2)])),
        (<function l2 at 0x10954fde8>,
            _MetaFuncInfo(alphas=None, pos=[3], deps=[4])),
        (<function renyi at 0x10954fcf8>,
            _MetaFuncInfo(alphas=[0.8, 0.9], pos=[0, 2], deps=[-1, -3]))
     ]), 3)
    '''
    funcs = {}
    metas = {}
    meta_deps = defaultdict(set)

    def add_func(func, alpha=None, pos=None):
        needs_alpha = getattr(func, 'needs_alpha', False)
        is_meta = hasattr(func, 'needs_results')

        d = metas if is_meta else funcs
        if func not in d:
            if needs_alpha:
                args = {'alphas': [alpha], 'pos': [pos]}
            else:
                args = {'alphas': None, 'pos': [pos]}

            if not is_meta:
                d[func] = _FuncInfo(**args)
            else:
                d[func] = _MetaFuncInfo(deps=[], **args)
                for req in func.needs_results:
                    if callable(req.alpha):
                        req_alpha = req.alpha(alpha)
                    else:
                        req_alpha = req.alpha
                    add_func(req.func, alpha=req_alpha)
                    meta_deps[func].add(req.func)
                    meta_deps[req.func]  # make sure required func is in there

        else:
            # already have an entry for the func
            # need to give it this pos, if it's not None
            # and also make sure that the alpha is present
            info = d[func]
            if not needs_alpha:
                if pos is not None:
                    if info.pos != [None]:
                        msg = "{} passed more than once"
                        raise ValueError(msg.format(func_name))

                    info.pos[0] = pos
            else:  # needs alpha
                try:
                    idx = info.alphas.index(alpha)
                except ValueError:
                    # this is a new alpha value we haven't seen yet
                    info.alphas.append(alpha)
                    info.pos.append(pos)
                    if is_meta:
                        for req in func.needs_results:
                            if callable(req.alpha):
                                req_alpha = req.alpha(alpha)
                            else:
                                req_alpha = req.alpha
                            add_func(req.func, alpha=req_alpha)
                else:
                    # repeated alpha value
                    if pos is not None:
                        if info.pos[idx] is not None:
                            msg = "{} with alpha {} passed more than once"
                            raise ValueError(msg.format(func_name, alpha))
                        info.pos[idx] = pos

    # add functions for each spec
    for i, spec in enumerate(specs):
        func_name, alpha = (spec.split(':', 1) + [None])[:2]
        if alpha is not None:
            alpha = float(alpha)

        try:
            func = func_mapping[func_name]
        except KeyError:
            msg = "'{}' is not a known function type"
            raise ValueError(msg.format(func_name))

        needs_alpha = getattr(func, 'needs_alpha', False)
        if needs_alpha and alpha is None:
            msg = "{} needs alpha but not passed in spec '{}'"
            raise ValueError(msg.format(func_name, spec))
        elif not needs_alpha and alpha is not None:
            msg = "{} doesn't need alpha but is passed in spec '{}'"
            raise ValueError(msg.format(func_name, spec))

        add_func(func, alpha, i)

    # number things that are dependencies only
    meta_counter = itertools.count(-1, step=-1)
    for info in itertools.chain(itervalues(funcs), itervalues(metas)):
        for i, pos in enumerate(info.pos):
            if pos is None:
                info.pos[i] = next(meta_counter)

    # fill in the dependencies for metas
    for func, info in iteritems(metas):
        deps = info.deps
        assert deps == []

        for req in func.needs_results:
            f = req.func
            req_info = (metas if hasattr(f, 'needs_results') else funcs)[f]
            if req.alpha is not None:
                if callable(req.alpha):
                    req_alpha = req.alpha(info.alphas)
                else:
                    req_alpha = req.alpha

                find_alpha = np.vectorize(req_info.alphas.index, otypes=[int])
                pos = np.asarray(req_info.pos)[find_alpha(req_alpha)]
                if np.isscalar(pos):
                    deps.append(pos[()])
                else:
                    deps.extend(pos)
            else:
                pos, = req_info.pos
                deps.append(pos)

    # topological sort of metas
    meta_order = topological_sort(meta_deps)
    metas_ordered = OrderedDict(
        (f, metas[f]) for f in meta_order if hasattr(f, 'needs_results'))

    return funcs, metas_ordered, -next(meta_counter) - 1

Response: Parse the spec and return a new base estimator that can be used to generate the new ones.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _flann_args(self, X=None):
        "The dictionary of arguments to give to FLANN."
        args = {'cores': self._n_jobs}
        if self.flann_algorithm == 'auto':
            if X is None or X.dim > 5:
                args['algorithm'] = 'linear'
            else:
                args['algorithm'] = 'kdtree_single'
        else:
            args['algorithm'] = self.flann_algorithm
        if self.flann_args:
            args.update(self.flann_args)

        # check that arguments are correct
        try:
            FLANNParameters().update(args)
        except AttributeError as e:
            msg = "flann_args contains an invalid argument:\n  {}"
            raise TypeError(msg.format(e))

        return args

Response: The dictionary of arguments to give to FLANN.


Instruction:
Explain what the following Python 3 code does
def fit(self, X, y=None, get_rhos=False):
        '''
        Sets up for divergence estimation "from" new data "to" X.
        Builds FLANN indices for each bag, and maybe gets within-bag distances.

        Parameters
        ----------
        X : list of arrays or :class:`skl_groups.features.Features`
            The bags to search "to".

        get_rhos : boolean, optional, default False
            Compute within-bag distances :attr:`rhos_`. These are only needed
            for some divergence functions or if do_sym is passed, and they'll
            be computed (and saved) during :meth:`transform` if they're not
            computed here.

            If you're using Jensen-Shannon divergence, a higher max_K may
            be needed once it sees the number of points in the transformed bags,
            so the computation here might be wasted.
        '''
        self.features_ = X = as_features(X, stack=True, bare=True)

        # if we're using a function that needs to pick its K vals itself,
        # then we need to set max_K here. when we transform(), might have to
        # re-do this :|
        Ks = self._get_Ks()
        _, _, _, max_K, save_all_Ks, _ = _choose_funcs(
            self.div_funcs, Ks, X.dim, X.n_pts, None, self.version)

        if max_K >= X.n_pts.min():
            msg = "asked for K = {}, but there's a bag with only {} points"
            raise ValueError(msg.format(max_K, X.n_pts.min()))

        memory = self.memory
        if isinstance(memory, string_types):
            memory = Memory(cachedir=memory, verbose=0)

        self.indices_ = id = memory.cache(_build_indices)(X, self._flann_args())
        if get_rhos:
            self.rhos_ = _get_rhos(X, id, Ks, max_K, save_all_Ks, self.min_dist)
        elif hasattr(self, 'rhos_'):
            del self.rhos_
        return self

Response: Fits the divergence estimator to the new data X.


Instruction:
Given the following Python 3 function, write the documentation
def transform(self, X):
        r'''
        Computes the divergences from X to :attr:`features_`.

        Parameters
        ----------
        X : list of bag feature arrays or :class:`skl_groups.features.Features`
            The bags to search "from".

        Returns
        -------
        divs : array of shape ``[len(div_funcs), len(Ks), len(X), len(features_)] + ([2] if do_sym else [])``
            The divergences from X to :attr:`features_`.
            ``divs[d, k, i, j]`` is the ``div_funcs[d]`` divergence
            from ``X[i]`` to ``fetaures_[j]`` using a K of ``Ks[k]``.
            If ``do_sym``, ``divs[d, k, i, j, 0]`` is
            :math:`D_{d,k}( X_i \| \texttt{features_}_j)` and
            ``divs[d, k, i, j, 1]`` is :math:`D_{d,k}(\texttt{features_}_j \| X_i)`.
        '''
        X = as_features(X, stack=True, bare=True)
        Y = self.features_

        Ks = np.asarray(self.Ks)

        if X.dim != Y.dim:
            msg = "incompatible dimensions: fit with {}, transform with {}"
            raise ValueError(msg.format(Y.dim, X.dim))

        memory = self.memory
        if isinstance(memory, string_types):
            memory = Memory(cachedir=memory, verbose=0)

        # ignore Y_indices to avoid slow pickling of them
        # NOTE: if the indices are approximate, then might not get the same
        #       results!
        est = memory.cache(_est_divs, ignore=['n_jobs', 'Y_indices', 'Y_rhos'])
        output, self.rhos_ = est(
            X, Y, self.indices_, getattr(self, 'rhos_', None),
            self.div_funcs, Ks,
            self.do_sym, self.clamp, self.version, self.min_dist,
            self._flann_args(), self._n_jobs)
        return output

Response: r Returns the divergences from X to Y.


Instruction:
Can you implement a function in Python 3 that
returns a version of X as a features object.

Response:
def as_features(X, stack=False, bare=False):
    '''
    Returns a version of X as a :class:`Features` object.

    Parameters
    ----------
    stack : boolean, default False
        Make a stacked version of X. Note that if X is a features object,
        this will stack it in-place, since that's usually what you want.
        (If not, just use the :class:`Features` constructor instead.)

    bare : boolean, default False
        Return a bare version of X (no metadata).

    Returns
    -------
    feats : :class:`Features`
        A version of X. If X is already a :class:`Features` object, the original
        X may be returned, depending on the arguments.
    '''
    if isinstance(X, Features):
        if stack:
            X.make_stacked()
        return X.bare() if bare else X
    return Features(X, stack=stack, bare=bare)


Instruction:
Create a Python 3 function to
copy the Feature object. Makes a copy of the features array.

Response:
def copy(self, stack=False, copy_meta=False, memo=None):
        '''
        Copies the Feature object. Makes a copy of the features array.

        Parameters
        ----------
        stack : boolean, optional, default False
            Whether to stack the copy if this one is unstacked.

        copy_meta : boolean, optional, default False
            Also copy the metadata. If False, metadata in both points to the
            same object.
        '''
        if self.stacked:
            fs = deepcopy(self.stacked_features, memo)
            n_pts = self.n_pts.copy()
        elif stack:
            fs = np.vstack(self.features)
            n_pts = self.n_pts.copy()
        else:
            fs = deepcopy(self.features, memo)
            n_pts = None

        meta = deepcopy(self.meta, memo) if copy_meta else self.meta
        return Features(fs, n_pts, copy=False, **meta)


Instruction:
How would you code a function in Python 3 to
make a Features object with no metadata ; points to the same features.

Response:
def bare(self):
        "Make a Features object with no metadata; points to the same features."
        if not self.meta:
            return self
        elif self.stacked:
            return Features(self.stacked_features, self.n_pts, copy=False)
        else:
            return Features(self.features, copy=False)


Instruction:
Given the following Python 3 function, write the documentation
def kl(Ks, dim, num_q, rhos, nus, clamp=True):
    r'''
    Estimate the KL divergence between distributions:
        \int p(x) \log (p(x) / q(x))
    using the kNN-based estimator (5) of
        Qing Wang, Sanjeev R Kulkarni, and Sergio Verdu (2009).
        Divergence Estimation for Multidimensional Densities Via
        k-Nearest-Neighbor Distances.
        IEEE Transactions on Information Theory.
        http://www.ee.princeton.edu/~verdu/reprints/WanKulVer.May2009.pdf
    which is:
        d * 1/n \sum \log (nu_k(i) / rho_k(i)) + log(m / (n - 1))

    If clamp, enforces KL >= 0.

    Returns an array of shape (num_Ks,).
    '''
    est = dim * np.mean(np.log(nus) - np.log(rhos), axis=0)
    est += np.log(num_q / (rhos.shape[0] - 1))
    if clamp:
        np.maximum(est, 0, out=est)
    return est

Response: r Estimates the KL divergence between a set of multidimensional Densities and a set of kernels.


Instruction:
Write a Python 3 script for
fitting the kernel to the data X.

Response:
def fit(self, X, y=None):
        '''
        Specify the data to which kernel values should be computed.

        Parameters
        ----------
        X : list of arrays or :class:`skl_groups.features.Features`
            The bags to compute "to".
        '''
        self.features_ = as_features(X, stack=True, bare=True)
        # TODO: could precompute things like squared norms if kernel == "rbf".
        # Probably should add support to sklearn instead of hacking it here.
        return self


Instruction:
Write a Python 3 script for
computing kernels from X to Y.

Response:
def transform(self, X):
        '''
        Compute kernels from X to :attr:`features_`.

        Parameters
        ----------
        X : list of arrays or :class:`skl_groups.features.Features`
            The bags to compute "from". Must have same dimension as
            :attr:`features_`.

        Returns
        -------
        K : array of shape ``[len(X), len(features_)]``
            The kernel evaluations from X to :attr:`features_`.
        '''

        X = as_features(X, stack=True, bare=True)
        Y = self.features_

        if X.dim != Y.dim:
            raise ValueError("MMK transform got dimension {} but had {} at fit"
                             .format(X.dim, Y.dim))

        pointwise = pairwise_kernels(X.stacked_features, Y.stacked_features,
                                     metric=self.kernel,
                                     filter_params=True,
                                     **self._get_kernel_params())

        # TODO: is there a way to do this without a Python loop?
        K = np.empty((len(X), len(Y)))
        for i in range(len(X)):
            for j in range(len(Y)):
                K[i, j] = pointwise[X._boundaries[i]:X._boundaries[i+1],
                                    Y._boundaries[j]:Y._boundaries[j+1]].mean()

        return K


Instruction:
Can you tell what is the following Python 3 function doing
def transform(self, X):
        '''
        Transform a list of bag features into a matrix of its mean features.

        Parameters
        ----------
        X : :class:`skl_groups.features.Features` or list of bag feature arrays
            Data to transform.

        Returns
        -------
        X_new : array, shape ``[len(X), X.dim]``
            X transformed into its means.
        '''
        X = as_features(X)
        return np.vstack([np.mean(bag, axis=0) for bag in X])

Response: Transform a list of bag features into a matrix of its mean features.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def from_shypo(cls, xml, encoding='utf-8'):
        score = float(xml.get('SCORE'))
        words = [Word.from_whypo(w_xml, encoding) for w_xml in xml.findall('WHYPO') if w_xml.get('WORD') not in ['<s>', '</s>']]
        return cls(words, score)

Response: Constructor from xml element


Instruction:
Here you have a function in Python 3, explain what it does
def from_whypo(cls, xml, encoding='utf-8'):
        word = unicode(xml.get('WORD'), encoding)
        confidence = float(xml.get('CM'))
        return cls(word, confidence)

Response: Constructor from xml element *WHYPO*


Instruction:
Here you have a function in Python 3, explain what it does
def run(self):
        logger.info(u'Started listening')
        while not self._stop:
            xml = self._readxml()

            # Exit on invalid XML
            if xml is None:
                break

            # Raw xml only
            if not self.modelize:
                logger.info(u'Raw xml: %s' % xml)
                self.results.put(xml)
                continue

            # Model objects + raw xml as fallback
            if xml.tag == 'RECOGOUT':
                sentence = Sentence.from_shypo(xml.find('SHYPO'), self.encoding)
                logger.info(u'Modelized recognition: %r' % sentence)
                self.results.put(sentence)
            else:
                logger.info(u'Unmodelized xml: %s' % xml)
                self.results.put(xml)

        logger.info(u'Stopped listening')

Response: Read XML and store results in self. results.


Instruction:
Explain what the following Python 3 code does
def connect(self):
        try:
            logger.info(u'Connecting %s:%d' % (self.host, self.port))
            self.sock.connect((self.host, self.port))
        except socket.error:
            raise ConnectionError()
        self.state = CONNECTED

Response: Connect to the server and update the internal state of the object.


Instruction:
How would you explain what the following Python 3 function does
def disconnect(self):
        logger.info(u'Disconnecting')
        self.sock.shutdown(socket.SHUT_RDWR)
        self.sock.close()
        self.state = DISCONNECTED

Response: Disconnect from the server


Instruction:
Can you tell what is the following Python 3 function doing
def send(self, command, timeout=5):
        logger.info(u'Sending %s' % command)
        _, writable, __ = select.select([], [self.sock], [], timeout)
        if not writable:
            raise SendTimeoutError()
        writable[0].sendall(command + '\n')

Response: Send a command to the server.


Instruction:
Here you have a function in Python 3, explain what it does
def _readline(self):
        line = ''
        while 1:
            readable, _, __ = select.select([self.sock], [], [], 0.5)
            if self._stop:
                break
            if not readable:
                continue
            data = readable[0].recv(1)
            if data == '\n':
                break
            line += unicode(data, self.encoding)
        return line

Response: Read a line from the server.


Instruction:
Write a Python 3 script for
reading a block from the server. Lines are read until a character. is found.

Response:
def _readblock(self):
        """Read a block from the server. Lines are read until a character ``.`` is found

        :return: the read block
        :rtype: string

        """
        block = ''
        while not self._stop:
            line = self._readline()
            if line == '.':
                break
            block += line
        return block


Instruction:
Given the following Python 3 function, write the documentation
def _readxml(self):
        block = re.sub(r'<(/?)s>', r'<\1s>', self._readblock())
        try:
            xml = XML(block)
        except ParseError:
            xml = None
        return xml

Response: Read a block and return the result as XML


Instruction:
Can you generate the documentation for the following Python 3 function
def cli(id):
    ch = Analyse(id)
    ch.full_analysis()
    click.echo(
        'Created: %s. Modified: %s. Deleted: %s' % (ch.create, ch.modify, ch.delete)
        )
    if ch.is_suspect:
        click.echo('The changeset {} is suspect! Reasons: {}'.format(
            id,
            ', '.join(ch.suspicion_reasons)
            ))
    else:
        click.echo('The changeset %s is not suspect!' % id)

Response: Analyse an OpenStreetMap changeset.


Instruction:
Given the following Python 3 function, write the documentation
def get_user_details(user_id):
    reasons = []
    try:
        url = OSM_USERS_API.format(user_id=requests.compat.quote(user_id))
        user_request = requests.get(url)
        if user_request.status_code == 200:
            user_data = user_request.content
            xml_data = ET.fromstring(user_data).getchildren()[0].getchildren()
            changesets = [i for i in xml_data if i.tag == 'changesets'][0]
            blocks = [i for i in xml_data if i.tag == 'blocks'][0]
            if int(changesets.get('count')) <= 5:
                reasons.append('New mapper')
            elif int(changesets.get('count')) <= 30:
                url = MAPBOX_USERS_API.format(
                    user_id=requests.compat.quote(user_id)
                    )
                user_request = requests.get(url)
                if user_request.status_code == 200:
                    mapping_days = int(
                        user_request.json().get('extra').get('mapping_days')
                        )
                    if mapping_days <= 5:
                        reasons.append('New mapper')
            if int(blocks.getchildren()[0].get('count')) > 1:
                reasons.append('User has multiple blocks')
    except Exception as e:
        message = 'Could not verify user of the changeset: {}, {}'
        print(message.format(user_id, str(e)))
    return reasons

Response: Get information about number of changesets blocks and mapping days of a user.


Instruction:
How would you explain what the following Python 3 function does
def changeset_info(changeset):
    keys = [tag.attrib.get('k') for tag in changeset.getchildren()]
    keys += ['id', 'user', 'uid', 'bbox', 'created_at']
    values = [tag.attrib.get('v') for tag in changeset.getchildren()]
    values += [
        changeset.get('id'), changeset.get('user'), changeset.get('uid'),
        get_bounds(changeset), changeset.get('created_at')
        ]

    return dict(zip(keys, values))

Response:
Return a dictionary with id user user_id bounds date of creation
    and all the tags of the changeset.


Instruction:
Explain what the following Python 3 code does
def get_changeset(changeset):
    url = 'https://www.openstreetmap.org/api/0.6/changeset/{}/download'.format(
        changeset
        )
    return ET.fromstring(requests.get(url).content)

Response: Get the changeset using the OSM API and return the content as a XML


Instruction:
Write a Python 3 script for
getting the metadata of a changeset using the OSM API and return it as a XML .

Response:
def get_metadata(changeset):
    """Get the metadata of a changeset using the OSM API and return it as a XML
    ElementTree.

    Args:
        changeset: the id of the changeset.
    """
    url = 'https://www.openstreetmap.org/api/0.6/changeset/{}'.format(changeset)
    return ET.fromstring(requests.get(url).content).getchildren()[0]


Instruction:
Create a Python 3 function to
get the bounds of the changeset and return it as a Polygon object.

Response:
def get_bounds(changeset):
    """Get the bounds of the changeset and return it as a Polygon object. If
    the changeset has not coordinates (case of the changesets that deal only
    with relations), it returns an empty Polygon.

    Args:
        changeset: the XML string of the changeset.
    """
    try:
        return Polygon([
            (float(changeset.get('min_lon')), float(changeset.get('min_lat'))),
            (float(changeset.get('max_lon')), float(changeset.get('min_lat'))),
            (float(changeset.get('max_lon')), float(changeset.get('max_lat'))),
            (float(changeset.get('min_lon')), float(changeset.get('max_lat'))),
            (float(changeset.get('min_lon')), float(changeset.get('min_lat'))),
            ])
    except TypeError:
        return Polygon()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def find_words(text, suspect_words, excluded_words=[]):
    text = text.lower()
    suspect_found = [i for i in re.finditer(make_regex(suspect_words), text)]
    if len(excluded_words) > 0:
        excluded_found = [i for i in re.finditer(make_regex(excluded_words), text)]
        if len(suspect_found) > len(excluded_found):
            return True
        else:
            return False
    else:
        if len(suspect_found) > 0:
            return True
        else:
            return False

Response: Check if a text has some of the suspect words or if it starts with a non - whitelisted word.


Instruction:
Write a Python 3 function for
downloading the replication changeset file or read it directly from the changeset file system.

Response:
def read_file(self, changeset_file):
        """Download the replication changeset file or read it directly from the
        filesystem (to test purposes).
        """
        if isfile(changeset_file):
            self.filename = changeset_file
        else:
            self.path = mkdtemp()
            self.filename = join(self.path, basename(changeset_file))
            download(changeset_file, self.path)

        self.xml = ET.fromstring(gzip.open(self.filename).read())

        # delete folder created to download the file
        if not isfile(changeset_file):
            rmtree(self.path)


Instruction:
Here you have a function in Python 3, explain what it does
def get_area(self, geojson):
        geojson = json.load(open(geojson, 'r'))
        self.area = Polygon(geojson['features'][0]['geometry']['coordinates'][0])

Response:
Read the first feature from the geojson and return it as a Polygon
        object.


Instruction:
Write a Python 3 script for
filtering the changesets that intersect with the geojson geometry.

Response:
def filter(self):
        """Filter the changesets that intersects with the geojson geometry."""
        self.content = [
            ch
            for ch in self.xml.getchildren()
            if get_bounds(ch).intersects(self.area)
            ]


Instruction:
Create a Python 3 function for
setting the fields of this object based on the changeset.

Response:
def set_fields(self, changeset):
        """Set the fields of this class with the metadata of the analysed
        changeset.
        """
        self.id = int(changeset.get('id'))
        self.user = changeset.get('user')
        self.uid = changeset.get('uid')
        self.editor = changeset.get('created_by', None)
        self.review_requested = changeset.get('review_requested', False)
        self.host = changeset.get('host', 'Not reported')
        self.bbox = changeset.get('bbox').wkt
        self.comment = changeset.get('comment', 'Not reported')
        self.source = changeset.get('source', 'Not reported')
        self.imagery_used = changeset.get('imagery_used', 'Not reported')
        self.date = datetime.strptime(
            changeset.get('created_at'),
            '%Y-%m-%dT%H:%M:%SZ'
            )
        self.suspicion_reasons = []
        self.is_suspect = False
        self.powerfull_editor = False


Instruction:
Can you generate a brief explanation for the following Python 3 code
def label_suspicious(self, reason):
        self.suspicion_reasons.append(reason)
        self.is_suspect = True

Response: Add suspicion reason and set the suspect flag.


Instruction:
Explain what the following Python 3 code does
def full_analysis(self):
        self.count()
        self.verify_words()
        self.verify_user()

        if self.review_requested == 'yes':
            self.label_suspicious('Review requested')

Response: Execute the count and verify_words methods.


Instruction:
Write a Python 3 function that can
verify if the changeset was made by a user that was blocked more than once.

Response:
def verify_user(self):
        """Verify if the changeset was made by a inexperienced mapper (anyone
        with less than 5 edits) or by a user that was blocked more than once.
        """
        user_reasons = get_user_details(self.uid)
        [self.label_suspicious(reason) for reason in user_reasons]


Instruction:
Here you have a function in Python 3, explain what it does
def verify_words(self):
        if self.comment:
            if find_words(self.comment, self.suspect_words, self.excluded_words):
                self.label_suspicious('suspect_word')

        if self.source:
            for word in self.illegal_sources:
                if word in self.source.lower():
                    self.label_suspicious('suspect_word')
                    break

        if self.imagery_used:
            for word in self.illegal_sources:
                if word in self.imagery_used.lower():
                    self.label_suspicious('suspect_word')
                    break

        self.suspicion_reasons = list(set(self.suspicion_reasons))

Response:
Verify the fields source imagery_used and comment of the changeset
        for some suspect words.