Instruction:
Can you implement a function in Python 3 that
disambiguates the specified list of documents in the input document collection docs.

Response:
def disambiguate(self, docs, **kwargs):
        """ Performs morphological analysis along with different morphological
            disambiguation steps (pre-disambiguation, vabamorf's disambiguation
            and post-disambiguation) in the input document collection `docs`.

        Note
        ----
             It is assumed that the documents in the input document collection `docs`
            have some similarities, e.g. they are parts of the same story, they are
            on the same topic etc., so that morphologically ambiguous words (for
            example: proper names) reoccur in different parts of the collection.
            The information about reoccurring ambiguous words is then used in
            pre-disambiguation and post-disambiguation steps for improving the overall
            quality of morphological disambiguation.

             Additionally, the input collection `docs` can have two levels: it can be
            list of list of estnltk.text.Text . For example, if we have a corpus of
            daily newspaper issues from one month, and each issue consists of articles
            (published on a single day), we can place the issues to the outer list,
            and the articles of the issues to the inner lists.

        Parameters
        ----------
        docs: list of estnltk.text.Text
              List of texts (documents) in which the morphological disambiguation
              is performed. Additionally, the list can have two levels: it can be
              list of list of estnltk.text.Text (this can improve quality of the
              post-disambiguation);
        post_disambiguate : boolean, optional
              Applies the lemma-based post-disambiguation on the collection.
              Default: True;
        disambiguate : boolean, optional
              Applies vabamorf's statistical disambiguation on the collection.
              Default: True;
              Note: this step shouldn't be turned off, unless for testing purposes;
        pre_disambiguate : boolean, optional
              Applies the pre-disambiguation of proper names on the collection.
              Default: True;
        vabamorf : boolean, optional
              Applies vabamorf's morphological analyzer on the collection.
              Default: True;
              Note: this step shouldn't be turned off, unless for testing purposes.

        Returns
        -------
        list of estnltk.text.Text
          List of morphologically disambiguated texts (documents). Preserves the
          structure, if the input was  list of list of estnltk.text.Text;

        """
        # For testing purposes, morph analysis and morph disambiguation can both
        # be switched off:
        use_vabamorf              = kwargs.get('vabamorf', True)
        use_vabamorf_disambiguate = kwargs.get('disambiguate', True)
        # Configuration for pre- and post disambiguation:
        use_pre_disambiguation    = kwargs.get('pre_disambiguate', True)
        use_post_disambiguation   = kwargs.get('post_disambiguate', True)

        kwargs = kwargs
        # Inner/default configuration for text objects:
        kwargs['disambiguate'] = False # do not use vabamorf disambiguation at first place
        kwargs['guess']        = True  # should be set for the morph analyzer
        kwargs['propername']   = True  # should be set for the morph analyzer

        # Check, whether the input is a list of lists of docs, or just a list of docs
        if not self.__isListOfLists( docs ):
            if not self.__isListOfTexts( docs ):
                raise Exception("Unexpected input argument 'docs': should be a list of strings or Text-s;")
            collections = [ docs ]
        else:
            collections = docs

        #  I. perform morphological analysis, pre_disambiguation, and
        #     statistical (vabamorf) disambiguation with-in a single
        #     document collection;
        for i in range(len(collections)):
            docs = [Text(doc, **kwargs) for doc in collections[i]]

            # morf.analysis without disambiguation
            if use_vabamorf:
                docs = [doc.tag_analysis() for doc in docs]

            if use_pre_disambiguation:
                docs = self.pre_disambiguate(docs)

            if use_vabamorf_disambiguate:
                docs = self.__vabamorf_disambiguate(docs)

            collections[i] = docs

        #
        #  II. perform post disambiguation over all document collections;
        #
        if use_post_disambiguation:
           collections = self.post_disambiguate( collections )
        return collections if len(collections)>1 else collections[0]


Instruction:
Here you have a function in Python 3, explain what it does
def __isListOfTexts(self, docs):
        return isinstance(docs, list) and \
               all(isinstance(d, (basestring, Text)) for d in docs)

Response: Checks whether the input is a list of strings or Text - s ;


Instruction:
Can you tell what is the following Python 3 function doing
def __isListOfLists(self, docs):
        return isinstance(docs, list) and \
               all(self.__isListOfTexts(ds) for ds in docs)

Response: Checks whether the input is a list of lists of strings - s ;


Instruction:
Implement a Python 3 function for
creating proper names lexicon for each word in the list of documents.

Response:
def __create_proper_names_lexicon(self, docs):
        """ Moodustab dokumendikollektsiooni põhjal pärisnimede sagedussõnastiku
            (mis kirjeldab, mitu korda iga pärisnimelemma esines);
        """
        lemmaFreq = dict()
        for doc in docs:
            for word in doc[WORDS]:
                # 1) Leiame k6ik s6naga seotud unikaalsed pärisnimelemmad
                #    (kui neid on)
                uniqLemmas = set()
                for analysis in word[ANALYSIS]:
                    if analysis[POSTAG] == 'H':
                        uniqLemmas.add( analysis[ROOT] )
                # 2) Jäädvustame lemmade sagedused
                for lemma in uniqLemmas:
                    if lemma not in lemmaFreq:
                        lemmaFreq[lemma] = 1
                    else:
                        lemmaFreq[lemma] += 1
        return lemmaFreq


Instruction:
Make a summary of the following Python 3 code
def __disambiguate_proper_names_1(self, docs, lexicon):
        for doc in docs:
            for word in doc[WORDS]:
                # Vaatame vaid s6nu, millele on pakutud rohkem kui yks analyys:
                if len(word[ANALYSIS]) > 1:
                    # 1) Leiame kõigi pärisnimede sagedused sagedusleksikonist
                    highestFreq = 0
                    properNameAnalyses = []
                    for analysis in word[ANALYSIS]:
                        if analysis[POSTAG] == 'H':
                            if analysis[ROOT] in lexicon:
                                properNameAnalyses.append( analysis )
                                if lexicon[analysis[ROOT]] > highestFreq:
                                    highestFreq = lexicon[analysis[ROOT]]
                            else:
                                raise Exception(' Unable to find lemma ',analysis[ROOT], \
                                      ' from the lexicon. ')
                    # 2) J2tame alles vaid suurima lemmasagedusega pärisnimeanalyysid,
                    #    ylejaanud kustutame maha
                    if highestFreq > 0:
                        toDelete = []
                        for analysis in properNameAnalyses:
                            if lexicon[analysis[ROOT]] < highestFreq:
                                toDelete.append(analysis)
                        for analysis in toDelete:
                            word[ANALYSIS].remove(analysis)

Response: Disambiguate proper names for 1 - word document.


Instruction:
Can you create a Python 3 function that
finds certain proper names for the given set of words.

Response:
def __find_certain_proper_names(self, docs):
        """ Moodustame kindlate pärisnimede loendi: vaatame sõnu, millel ongi
            ainult pärisnimeanalüüsid ning võtame sealt loendisse  unikaalsed
            pärisnimed;
        """
        certainNames = set()
        for doc in docs:
            for word in doc[WORDS]:
                # Vaatame vaid pärisnimeanalüüsidest koosnevaid sõnu
                if all([ a[POSTAG] == 'H' for a in word[ANALYSIS] ]):
                    # Jäädvustame kõik unikaalsed lemmad kui kindlad pärisnimed
                    for analysis in word[ANALYSIS]:
                        certainNames.add( analysis[ROOT] )
        return certainNames


Instruction:
Given the following Python 3 function, write the documentation
def __find_sentence_initial_proper_names(self, docs):
        sentInitialNames = set()
        for doc in docs:
            for sentence in doc.divide( layer=WORDS, by=SENTENCES ):
                sentencePos = 0 # Tavaline lausealgus
                for i in range(len(sentence)):
                    word = sentence[i]
                    #  Täiendavad heuristikud lausealguspositsioonide leidmiseks:
                    #  1) kirjavahemärk, mis pole koma ega semikoolon, on lausealgus:
                    if all([ a[POSTAG] == 'Z' for a in word[ANALYSIS] ]) and \
                       not re.match('^[,;]+$', word[TEXT]):
                        sentencePos = 0
                        #self.__debug_print_word_in_sentence_str(sentence, word)
                        continue
                    # 2) potentsiaalne loendi algus (arv, millele järgneb punkt või
                    #    sulg ja mis ei ole kuupäev);
                    if not re.match('^[1234567890]*$', word[TEXT] ) and \
                       not re.match('^[1234567890]{1,2}.[1234567890]{1,2}.[1234567890]{4}$', word[TEXT] ) and \
                       re.match("^[1234567890.()]*$", word[TEXT]):
                        sentencePos = 0
                        #self.__debug_print_word_in_sentence_str(sentence, word)
                        continue
                    if sentencePos == 0:
                        # Vaatame lausealgulisi sõnu, millel on nii pärisnimeanalüüs(e)
                        # kui ka mitte-pärisnimeanalüüs(e)
                        h_postags = [ a[POSTAG] == 'H' for a in word[ANALYSIS] ]
                        if any( h_postags )  and  not all( h_postags ):
                            for analysis in word[ANALYSIS]:
                                # Jätame meelde kõik unikaalsed pärisnimelemmad
                                if analysis[POSTAG] == 'H':
                                    sentInitialNames.add( analysis[ROOT] )
                    sentencePos += 1
        return sentInitialNames

Response: Find initial proper names for words in a list of sentences.


Instruction:
Write a Python 3 script to
remove redundant proper names from the document list.

Response:
def __remove_redundant_proper_names(self, docs, lemma_set):
        """ Eemaldame yleliigsed pärisnimeanalüüsid etteantud sõnalemmade
            loendi (hulga) põhjal;
        """
        for doc in docs:
            for word in doc[WORDS]:
                # Vaatame vaid s6nu, millele on pakutud rohkem kui yks analyys:
                if len(word[ANALYSIS]) > 1:
                    # 1) Leiame analyysid, mis tuleks loendi järgi eemaldada
                    toDelete = []
                    for analysis in word[ANALYSIS]:
                        if analysis[POSTAG] == 'H' and analysis[ROOT] in lemma_set:
                            toDelete.append( analysis )
                    # 2) Eemaldame yleliigsed analyysid
                    if toDelete:
                        for analysis in toDelete:
                            word[ANALYSIS].remove(analysis)


Instruction:
How would you code a function in Python 3 to
disambiguate the given list of words and sets of words to their respective values.

Response:
def pre_disambiguate(self, docs):
        """  Teostab pärisnimede eelühestamine. Üldiseks eesmärgiks on vähendada mitmesust
             suurtähega algavate sonade morf analüüsil, nt eemaldada pärisnime analüüs, kui
             suurtäht tähistab tõenäoliselt lausealgust.
        """
        # 1) Leiame pärisnimelemmade sagedusleksikoni
        lexicon = self.__create_proper_names_lexicon(docs)
        # 2) Teeme esialgse kustutamise: kui sõnal on mitu erineva korpuse-
        #    sagedusega pärisnimeanalüüsi, siis jätame alles vaid kõige
        #    sagedasema analyysi ...
        self.__disambiguate_proper_names_1(docs, lexicon)

        # 3) Eemaldame yleliigsed lause alguse pärisnimeanalüüsid;
        #  Kõigepealt leiame: kindlad pärisnimed, lause alguses esinevad
        #  p2risnimed ja lause keskel esinevad pärisnimed
        certainNames     = self.__find_certain_proper_names(docs)
        sentInitialNames = self.__find_sentence_initial_proper_names(docs)
        sentCentralNames = self.__find_sentence_central_proper_names(docs)

        # 3.1) Võrdleme lause alguses ja keskel esinevaid lemmasid: leiame
        #      lemmad, mis esinesid ainult lause alguses ...
        onlySentenceInitial = sentInitialNames.difference(sentCentralNames)
        # 3.2) Võrdleme ainult lause alguses esinevaid ning kindlaid pärisnime-
        #      lemmasid: kui sõna esines vaid lause alguses ega ole kindel
        #      pärisnimelemma, pole tõenäoliselt tegu pärisnimega ...
        notProperNames = onlySentenceInitial.difference(certainNames)
        # 3.3) Eemaldame yleliigsed p2risnimeanalyysid (kui selliseid leidus)
        if len(notProperNames) > 0:
            self.__remove_redundant_proper_names(docs, notProperNames)

        # 4) Leiame uue pärisnimelemmade sagedusleksikoni (sagedused on
        #    tõenäoliselt vahepeal muutunud);
        lexicon = self.__create_proper_names_lexicon(docs)

        # 5) Teeme üleliigsete mittepärisnimeanalüüside kustutamise sõnadelt,
        #    millel on lisaks pärisnimeanalüüsidele ka teisi analüüse:
        #    lausealgusesse jätame alles vaid pärisnimeanalüüsid, kui neid
        #    esineb korpuses ka mujal;
        #    lause keskele jätame igal juhul alles vaid pärisnimeanalüüsid;
        self.__disambiguate_proper_names_2(docs, lexicon)
        return docs


Instruction:
Make a summary of the following Python 3 code
def __analyses_match(self, analysisA, analysisB):
        return POSTAG in analysisA and POSTAG in analysisB and \
               analysisA[POSTAG]==analysisB[POSTAG] and \
               ROOT in analysisA and ROOT in analysisB and \
               analysisA[ROOT]==analysisB[ROOT] and \
               FORM in analysisA and FORM in analysisB and \
               analysisA[FORM]==analysisB[FORM] and \
               CLITIC in analysisA and CLITIC in analysisB and \
               analysisA[CLITIC]==analysisB[CLITIC] and \
               ENDING in analysisA and ENDING in analysisB and \
               analysisA[ENDING]==analysisB[ENDING]

Response: Returns True if the analysisA and analysisB are the same.


Instruction:
Make a summary of the following Python 3 code
def __remove_duplicate_and_problematic_analyses(self, docs):
        for doc in docs:
            for word in doc[WORDS]:
                # 1) Leiame k6ik analyysi-duplikaadid (kui neid on)
                toDelete = []
                for i in range(len(word[ANALYSIS])):
                    if i+1 < len(word[ANALYSIS]):
                        for j in range(i+1, len(word[ANALYSIS])):
                            analysisI = word[ANALYSIS][i]
                            analysisJ = word[ANALYSIS][j]
                            if self.__analyses_match(analysisI, analysisJ):
                                if j not in toDelete:
                                    toDelete.append(j)
                # 2) Kustutame yleliigsed analyysid
                if toDelete:
                    for a in sorted(toDelete, reverse=True):
                        del word[ANALYSIS][a]
                #
                # *) Kui verbi analüüside puhul on olemas nii '-tama' kui ka '-ma'
                #    lõpp, siis jätta alles vaid -ma, ülejäänud kustutada;
                #    Nt   lõpetama:    lõp+tama,   lõppe+tama,  lõpeta+ma
                #         teatama:     tead+tama,  teata+ma
                #
                if any([ a[POSTAG]=='V' and a[ENDING]=='tama' for a in word[ANALYSIS] ]) and \
                   any([ a[POSTAG]=='V' and a[ENDING]=='ma' for a in word[ANALYSIS] ]):
                    toDelete = []
                    for a in range( len(word[ANALYSIS]) ):
                        if word[ANALYSIS][a][POSTAG]=='V' and \
                           word[ANALYSIS][a][ENDING]=='tama':
                            toDelete.append(a)
                    if toDelete:
                        for a in sorted(toDelete, reverse=True):
                            del word[ANALYSIS][a]

Response: Remove duplicate and problematic analyses from a list of documents.


Instruction:
Make a summary of the following Python 3 code
def __find_hidden_analyses(self, docs):
        hidden = dict()
        nudTudLopud = re.compile('^.*[ntd]ud$')
        for d in range(len(docs)):
            for w in range(len(docs[d][WORDS])):
                word = docs[d][WORDS][w]
                if ANALYSIS in word and len(word[ANALYSIS]) > 1:
                    #
                    # 1) Kui enamus analüüse on nud/tud/dud analüüsid, peida mitmesus:
                    #    kõla+nud //_V_ nud, //    kõla=nud+0 //_A_ //    kõla=nud+0 //_A_ sg n, //    kõla=nud+d //_A_ pl n, //
                    nudTud = [ nudTudLopud.match(a[ROOT]) != None or \
                               nudTudLopud.match(a[ENDING]) != None \
                               for a in word[ANALYSIS] ]
                    if nudTud.count( True ) > 1:
                        hidden[(d, w)] = 1
                    #
                    # 2) Kui analyysidel on sama lemma ja puudub vormitunnus, siis peida mitmesused ära:
                    #    Nt    kui+0 //_D_ //    kui+0 //_J_ //
                    #          nagu+0 //_D_ //    nagu+0 //_J_ //
                    lemmas = set([ a[ROOT] for a in word[ANALYSIS] ])
                    forms  = set([ a[FORM] for a in word[ANALYSIS] ])
                    if len(lemmas) == 1 and len(forms) == 1 and (list(forms))[0] == '':
                        hidden[(d, w)] = 1
                    #
                    # 3) Kui 'olema'-analyysidel on sama lemma ning sama l6pp, peida mitmesused:
                    #    Nt  'nad on' vs 'ta on' saavad sama olema-analyysi, mis jääb mitmeseks;
                    endings  = set([ a[ENDING] for a in word[ANALYSIS] ])
                    if len(lemmas) == 1 and (list(lemmas))[0] == 'ole' and len(endings) == 1 \
                       and (list(endings))[0] == '0':
                        hidden[(d, w)] = 1
                    #
                    # 4) Kui asesõnadel on sama lemma ja lõpp, peida ainsuse/mitmuse mitmesus:
                    #    Nt     kõik+0 //_P_ sg n //    kõik+0 //_P_ pl n //
                    #           kes+0 //_P_ sg n //    kes+0 //_P_ pl n //
                    postags  = set([ a[POSTAG] for a in word[ANALYSIS] ])
                    if len(lemmas) == 1 and len(postags) == 1 and 'P' in postags and \
                       len(endings) == 1:
                        hidden[(d, w)] = 1
                    #
                    # 5) Kui on sama lemma ja lõpp, peida arv- ja asesõnadevaheline mitmesus:
                    #    Nt     teine+0 //_O_ pl n, //    teine+0 //_P_ pl n, //
                    #           üks+l //_N_ sg ad, //    üks+l //_P_ sg ad, //
                    if len(lemmas) == 1 and 'P' in postags and ('O' in postags or \
                       'N' in postags) and len(endings) == 1:
                        hidden[(d, w)] = 1
        return hidden

Response: Find hidden analyses in the given list of sentences.


Instruction:
How would you explain what the following Python 3 function does
def __supplement_lemma_frequency_lexicon(self, docs, hiddenWords, lexicon, amb_lexicon):
        for d in range(len(docs)):
            for w in range(len(docs[d][WORDS])):
                word = docs[d][WORDS][w]
                # Jätame vahele nn peidetud sõnad
                if (d, w) in hiddenWords:
                    continue
                isAmbiguous = (len(word[ANALYSIS])>1)
                # Jäädvustame sagedused, verbide omad eraldiseisva märkega:
                for a in word[ANALYSIS]:
                    lemma = a[ROOT]+'ma' if a[POSTAG]=='V' else a[ROOT]
                    # 1) Jäädvustame üldise sageduse
                    if lemma not in lexicon:
                        lexicon[lemma] = 1
                    else:
                        lexicon[lemma] += 1
                    # 2) Jäädvustame mitmeste sõnade esinemise
                    if isAmbiguous:
                        amb_lexicon[lemma] = 1
        #  Kanname yldisest sagedusleksikonist sagedused yle mitmeste lemmade
        # sagedusleksikoni
        for lemma in amb_lexicon.keys():
            amb_lexicon[lemma] = lexicon[lemma]

Response: Supplement the lemma frequency of the lemmas in the lexicon.


Instruction:
Write a Python 3 function that can
disambiguate a list of words in a list of words with a given lexicon.

Response:
def __disambiguate_with_lexicon(self, docs, lexicon, hiddenWords):
        """ Teostab lemmade leksikoni järgi mitmeste morf analüüside
            ühestamise - eemaldab üleliigsed analüüsid;
            Toetub ideele "üks tähendus teksti kohta": kui mitmeseks jäänud
            lemma esineb tekstis/korpuses ka mujal ning lõppkokkuvõttes
            esineb sagedamini kui alternatiivsed analüüsid, siis tõenäoliselt
            see ongi õige lemma/analüüs;
        """
        for d in range(len(docs)):
            for w in range(len(docs[d][WORDS])):
                word = docs[d][WORDS][w]
                # Jätame vahele nn peidetud sõnad
                if (d, w) in hiddenWords:
                    continue
                # Vaatame vaid mitmeseks jäänud analüüsidega sõnu
                if len(word[ANALYSIS]) > 1:
                    #  1) Leiame suurima esinemissageduse mitmeste lemmade seas
                    highestFreq = 0
                    for analysis in word[ANALYSIS]:
                        lemma = analysis[ROOT]+'ma' if analysis[POSTAG]=='V' else analysis[ROOT]
                        if lemma in lexicon and lexicon[lemma] > highestFreq:
                            highestFreq = lexicon[lemma]
                    if highestFreq > 0:
                        #  2) Jätame välja kõik analüüsid, mille lemma esinemissagedus
                        #     on väiksem kui suurim esinemissagedus;
                        toDelete = []
                        for analysis in word[ANALYSIS]:
                            lemma = analysis[ROOT]+'ma' if analysis[POSTAG]=='V' else analysis[ROOT]
                            freq = lexicon[lemma] if lemma in lexicon else 0
                            if freq < highestFreq:
                                toDelete.append(analysis)
                        for analysis in toDelete:
                            word[ANALYSIS].remove(analysis)


Instruction:
Can you tell what is the following Python 3 function doing
def post_disambiguate(self, collections):
        #
        #  I etapp: ühestame ühe dokumendikollektsiooni piires
        #           (nt üle kõigi samal päeval ilmunud ajaleheartiklite);
        #
        for docs in collections:
            # 1) Eemaldame analüüside seast duplikaadid ja probleemsed
            self.__remove_duplicate_and_problematic_analyses(docs)
            # 2) Leiame sõnad, mis sisaldavad nn ignoreeritavaid mitmesusi
            #    (selliseid mitmesusi, mida me ühestamisel ei arvesta);
            hiddenWords = self.__find_hidden_analyses(docs)
            # 3) Leiame kaks lemmade sagedusleksikoni: üldise lemmade sagedus-
            #    leksikoni ja mitmeseks jäänud sonade lemmade sagedusleksikoni;
            #    Mitmeste lemmade leksikoni läheb kirja vastavate lemmade yldine
            #    sagedus korpuses (kuhu arvatud ka sagedus ühestatud sõnades);
            genLemmaLex = dict()
            ambLemmaLex = dict()
            self.__supplement_lemma_frequency_lexicon(docs, hiddenWords, ambLemmaLex, genLemmaLex)
            # 4) Teostame lemmade-p6hise yhestamise: mitmeseks j22nud analyyside
            #    puhul j2tame alles analyysid, mille lemma esinemisagedus on suurim
            #    (ja kui k6igi esinemissagedus on v6rdne, siis ei tee midagi)
            self.__disambiguate_with_lexicon(docs, ambLemmaLex, hiddenWords)
        #
        #  II etapp: ühestame üle kõikide dokumendikollektsioonide
        #            (nt üle kõigi ühe aasta ajalehenumbrite, kus
        #             üks ajalehenumber sisaldab kõiki sama päeva artikleid);
        #
        if len(collections) > 1:
            # Genereerime mitmeste sagedusleksikoni
            genLemmaLex = dict()
            ambLemmaLex = dict()
            for docs in collections:
                # *) Leiame sõnad, mis sisaldavad nn ignoreeritavaid mitmesusi
                hiddenWords = self.__find_hidden_analyses(docs)
                # *) Täiendame üldist lemmade sagedusleksikoni ja mitmeseks jäänud
                #    lemmade sagedusleksikoni;
                self.__supplement_lemma_frequency_lexicon(docs, hiddenWords, ambLemmaLex, genLemmaLex)
            # Teostame järelühestamise
            for docs in collections:
                # *) Leiame sõnad, mis sisaldavad nn ignoreeritavaid mitmesusi
                hiddenWords = self.__find_hidden_analyses(docs)
                # *) Teostame lemmade-p6hise yhestamise;
                self.__disambiguate_with_lexicon(docs, ambLemmaLex, hiddenWords)
        return collections

Response: Post disambiguation of the words in the list of collections.


Instruction:
Explain what the following Python 3 code does
def tag_text(self, text, **kwargs):

        for analysis_match in text.analysis:
            for candidate in analysis_match:
                if candidate['partofspeech'] in PYVABAMORF_TO_WORDNET_POS_MAP:
                    # Wordnet contains data about the given lemma and pos combination - will annotate.
                    wordnet_obj = {}
                    tag_synsets(wordnet_obj, candidate, **kwargs)
        return text

Response: Annotates the given text with the word - level analysis.


Instruction:
Can you implement a function in Python 3 that
gives a sentence chunk extract original texts and labels.

Response:
def get_texts_and_labels(sentence_chunk):
    """Given a sentence chunk, extract original texts and labels."""
    words = sentence_chunk.split('\n')
    texts = []
    labels = []
    for word in words:
        word = word.strip()
        if len(word) > 0:
            toks = word.split('\t')
            texts.append(toks[0].strip())
            labels.append(toks[-1].strip())
    return texts, labels


Instruction:
Can you generate the documentation for the following Python 3 function
def parse_doc(doc):
    word_spans = []
    sentence_spans = []
    sentence_chunks = doc.split('\n\n')
    sentences = []
    for chunk in sentence_chunks:
        sent_texts, sent_labels = get_texts_and_labels(chunk.strip())
        sentences.append(list(zip(sent_texts, sent_labels)))
    return sentences

Response: Exract list of sentences containing text and label pairs.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def convert(document):
    raw_tokens = []
    curpos = 0
    text_spans = []
    all_labels = []
    sent_spans = []
    word_texts = []
    for sentence in document:
        startpos = curpos
        for idx, (text, label) in enumerate(sentence):
            raw_tokens.append(text)
            word_texts.append(text)
            all_labels.append(label)
            text_spans.append((curpos, curpos+len(text)))
            curpos += len(text)
            if idx < len(sentence) - 1:
                raw_tokens.append(' ')
                curpos += 1
        sent_spans.append((startpos, curpos))
        raw_tokens.append('\n')
        curpos += 1
    return {
        TEXT: ''.join(raw_tokens),
        WORDS: [{TEXT: text, START: start, END: end, LABEL: label} for text, (start, end), label in zip(word_texts, text_spans, all_labels)],
        SENTENCES: [{START: start, END:end} for start, end in sent_spans]
    }

Response: Convert a document to a Text object


Instruction:
Implement a Python 3 function for
selecting the current user from the database.

Response:
def select(self, cb=None):
        '''
        Set this channel to use transactions.
        '''
        if not self._enabled:
            self._enabled = True
            self.send_frame(MethodFrame(self.channel_id, 90, 10))
            self._select_cb.append(cb)
            self.channel.add_synchronous_cb(self._recv_select_ok)


Instruction:
How would you implement a function in Python 3 that
commits the current transaction.

Response:
def commit(self, cb=None):
        '''
        Commit the current transaction.  Caller can specify a callback to use
        when the transaction is committed.
        '''
        # Could call select() but spec 1.9.2.3 says to raise an exception
        if not self.enabled:
            raise self.TransactionsNotEnabled()

        self.send_frame(MethodFrame(self.channel_id, 90, 20))
        self._commit_cb.append(cb)
        self.channel.add_synchronous_cb(self._recv_commit_ok)


Instruction:
How would you explain what the following Python 3 function does
def rollback(self, cb=None):
        '''
        Abandon all message publications and acks in the current transaction.
        Caller can specify a callback to use when the transaction has been
        aborted.
        '''
        # Could call select() but spec 1.9.2.5 says to raise an exception
        if not self.enabled:
            raise self.TransactionsNotEnabled()

        self.send_frame(MethodFrame(self.channel_id, 90, 30))
        self._rollback_cb.append(cb)
        self.channel.add_synchronous_cb(self._recv_rollback_ok)

Response: Rolls back all messages in the current transaction.


Instruction:
Can you generate the documentation for the following Python 3 function
def synchronous(self):
        '''
        True if transport is synchronous or the connection has been forced
        into synchronous mode, False otherwise.
        '''
        if self._transport is None:
            if self._close_info and len(self._close_info['reply_text']) > 0:
                raise ConnectionClosed("connection is closed: %s : %s" %
                                       (self._close_info['reply_code'],
                                        self._close_info['reply_text']))
            raise ConnectionClosed("connection is closed")
        return self.transport.synchronous or self._synchronous

Response: Returns True if the transport is synchronous False otherwise.


Instruction:
Write a Python 3 function for
connecting to a host and port.

Response:
def connect(self, host, port):
        '''
        Connect to a host and port.
        '''
        # Clear the connect state immediately since we're no longer connected
        # at this point.
        self._connected = False

        # Only after the socket has connected do we clear this state; closed
        # must be False so that writes can be buffered in writePacket(). The
        # closed state might have been set to True due to a socket error or a
        # redirect.
        self._host = "%s:%d" % (host, port)
        self._closed = False
        self._close_info = {
            'reply_code': 0,
            'reply_text': 'failed to connect to %s' % (self._host),
            'class_id': 0,
            'method_id': 0
        }

        self._transport.connect((host, port))
        self._transport.write(PROTOCOL_HEADER)

        self._last_octet_time = time.time()

        if self._synchronous_connect:
            # Have to queue this callback just after connect, it can't go
            # into the constructor because the channel needs to be
            # "always there" for frame processing, but the synchronous
            # callback can't be added until after the protocol header has
            # been written. This SHOULD be registered before the protocol
            # header is written, in the case where the header bytes are
            # written, but this thread/greenlet/context does not return until
            # after another thread/greenlet/context has read and processed the
            # recv_start frame. Without more re-write to add_sync_cb though,
            # it will block on reading responses that will never arrive
            # because the protocol header isn't written yet. TBD if needs
            # refactoring. Could encapsulate entirely here, wherein
            # read_frames exits if protocol header not yet written. Like other
            # synchronous behaviors, adding this callback will result in a
            # blocking frame read and process loop until _recv_start and any
            # subsequent synchronous callbacks have been processed. In the
            # event that this is /not/ a synchronous transport, but the
            # caller wants the connect to be synchronous so as to ensure that
            # the connection is ready, then do a read frame loop here.
            self._channels[0].add_synchronous_cb(self._channels[0]._recv_start)
            while not self._connected:
                self.read_frames()


Instruction:
Write a Python 3 script for
disconnecting from the current host.

Response:
def disconnect(self):
        '''
        Disconnect from the current host, but do not update the closed state.
        After the transport is disconnected, the closed state will be True if
        this is called after a protocol shutdown, or False if the disconnect
        was in error.

        TODO: do we really need closed vs. connected states? this only adds
        complication and the whole reconnect process has been scrapped anyway.

        '''
        self._connected = False
        if self._transport is not None:
            try:
                self._transport.disconnect()
            except Exception:
                self.logger.error(
                    "Failed to disconnect from %s", self._host, exc_info=True)
                raise
            finally:
                self._transport = None


Instruction:
Make a summary of the following Python 3 code
def transport_closed(self, **kwargs):
        msg = 'unknown cause'
        self.logger.warning('transport to %s closed : %s' %
                            (self._host, kwargs.get('msg', msg)))
        self._close_info = {
            'reply_code': kwargs.get('reply_code', 0),
            'reply_text': kwargs.get('msg', msg),
            'class_id': kwargs.get('class_id', 0),
            'method_id': kwargs.get('method_id', 0)
        }

        # We're not connected any more, but we're not closed without an
        # explicit close call.
        self._connected = False
        self._transport = None

        # Call back to a user-provided close function
        self._callback_close()

Response: Called by Transports when they close unexpectedly.


Instruction:
Make a summary of the following Python 3 code
def _next_channel_id(self):
        '''Return the next possible channel id.  Is a circular enumeration.'''
        self._channel_counter += 1
        if self._channel_counter >= self._channel_max:
            self._channel_counter = 1
        return self._channel_counter

Response: Return the next possible channel id. Is a circular enumeration.


Instruction:
Here you have a function in Python 3, explain what it does
def channel(self, channel_id=None, synchronous=False):
        if channel_id is None:
            # adjust for channel 0
            if len(self._channels) - 1 >= self._channel_max:
                raise Connection.TooManyChannels(
                    "%d channels already open, max %d",
                    len(self._channels) - 1,
                    self._channel_max)
            channel_id = self._next_channel_id()
            while channel_id in self._channels:
                channel_id = self._next_channel_id()
        elif channel_id in self._channels:
            return self._channels[channel_id]
        else:
            raise Connection.InvalidChannel(
                "%s is not a valid channel id", channel_id)

        # Call open() here so that ConnectionChannel doesn't have it called.
        # Could also solve this other ways, but it's a HACK regardless.
        rval = Channel(
            self, channel_id, self._class_map, synchronous=synchronous)
        self._channels[channel_id] = rval
        rval.add_close_listener(self._channel_closed)
        rval.open()
        return rval

Response: Fetch a Channel object identified by the numeric channel_id or create a new one if it doesn t already exist.


Instruction:
How would you explain what the following Python 3 function does
def read_frames(self):
        '''
        Read frames from the transport and process them. Some transports may
        choose to do this in the background, in several threads, and so on.
        '''
        # It's possible in a concurrent environment that our transport handle
        # has gone away, so handle that cleanly.
        # TODO: Consider moving this block into Translator base class. In many
        # ways it belongs there. One of the problems though is that this is
        # essentially the read loop. Each Transport has different rules for
        # how to kick this off, and in the case of gevent, this is how a
        # blocking call to read from the socket is kicked off.
        if self._transport is None:
            return

        # Send a heartbeat (if needed)
        self._channels[0].send_heartbeat()

        data = self._transport.read(self._heartbeat)
        current_time = time.time()

        if data is None:
            # Wait for 2 heartbeat intervals before giving up. See AMQP 4.2.7:
            # "If a peer detects no incoming traffic (i.e. received octets) for two heartbeat intervals or longer,
            # it should close the connection"
            if self._heartbeat and (current_time-self._last_octet_time > 2*self._heartbeat):
                msg = 'Heartbeats not received from %s for %d seconds' % (self._host, 2*self._heartbeat)
                self.transport_closed(msg=msg)
                raise ConnectionClosed('Connection is closed: ' + msg)
            return
        self._last_octet_time = current_time
        reader = Reader(data)
        p_channels = set()

        try:
            for frame in Frame.read_frames(reader):
                if self._debug > 1:
                    self.logger.debug("READ: %s", frame)
                self._frames_read += 1
                ch = self.channel(frame.channel_id)
                ch.buffer_frame(frame)
                p_channels.add(ch)
        except Frame.FrameError as e:
            # Frame error in the peer, disconnect
            self.close(reply_code=501,
                       reply_text='frame error from %s : %s' % (
                           self._host, str(e)),
                       class_id=0, method_id=0, disconnect=True)
            raise ConnectionClosed("connection is closed: %s : %s" %
                                   (self._close_info['reply_code'],
                                    self._close_info['reply_text']))

        # NOTE: we process channels after buffering unused data in order to
        # preserve the integrity of the input stream in case a channel needs to
        # read input, such as when a channel framing error necessitates the use
        # of the synchronous channel.close method. See `Channel.process_frames`.
        #
        # HACK: read the buffer contents and re-buffer.  Would prefer to pass
        # buffer back, but there's no good way of asking the total size of the
        # buffer, comparing to tell(), and then re-buffering.  There's also no
        # ability to clear the buffer up to the current position. It would be
        # awesome if we could free that memory without a new allocation.
        if reader.tell() < len(data):
            self._transport.buffer(data[reader.tell():])

        self._transport.process_channels(p_channels)

Response: Read frames from the transport and process them.


Instruction:
Make a summary of the following Python 3 code
def _flush_buffered_frames(self):
        '''
        Callback when protocol has been initialized on channel 0 and we're
        ready to send out frames to set up any channels that have been
        created.
        '''
        # In the rare case (a bug) where this is called but send_frame thinks
        # they should be buffered, don't clobber.
        frames = self._output_frame_buffer
        self._output_frame_buffer = []
        for frame in frames:
            self.send_frame(frame)

Response: Callback when protocol has been initialized on channel 0 and we re ready to send out any buffered frames.


Instruction:
Can you generate the documentation for the following Python 3 function
def send_frame(self, frame):
        '''
        Send a single frame. If there is no transport or we're not connected
        yet, append to the output buffer, else send immediately to the socket.
        This is called from within the MethodFrames.
        '''
        if self._closed:
            if self._close_info and len(self._close_info['reply_text']) > 0:
                raise ConnectionClosed("connection is closed: %s : %s" %
                                       (self._close_info['reply_code'],
                                        self._close_info['reply_text']))
            raise ConnectionClosed("connection is closed")

        if self._transport is None or \
                (not self._connected and frame.channel_id != 0):
            self._output_frame_buffer.append(frame)
            return

        if self._debug > 1:
            self.logger.debug("WRITE: %s", frame)

        buf = bytearray()
        frame.write_frame(buf)
        if len(buf) > self._frame_max:
            self.close(
                reply_code=501,
                reply_text='attempted to send frame of %d bytes, frame max %d' % (
                    len(buf), self._frame_max),
                class_id=0, method_id=0, disconnect=True)
            raise ConnectionClosed(
                "connection is closed: %s : %s" %
                (self._close_info['reply_code'],
                 self._close_info['reply_text']))
        self._transport.write(buf)

        self._frames_written += 1

Response: Send a single frame to the remote socket.


Instruction:
Can you tell what is the following Python 3 function doing
def dispatch(self, frame):
        '''
        Override the default dispatch since we don't need the rest of
        the stack.
        '''
        if frame.type() == HeartbeatFrame.type():
            self.send_heartbeat()

        elif frame.type() == MethodFrame.type():
            if frame.class_id == 10:
                cb = self._method_map.get(frame.method_id)
                if cb:
                    method = self.clear_synchronous_cb(cb)
                    method(frame)
                else:
                    raise Channel.InvalidMethod(
                        "unsupported method %d on channel %d",
                        frame.method_id, self.channel_id)
            else:
                raise Channel.InvalidClass(
                    "class %d is not supported on channel %d",
                    frame.class_id, self.channel_id)

        else:
            raise Frame.InvalidFrameType(
                "frame type %d is not supported on channel %d",
                frame.type(), self.channel_id)

Response: Dispatches the given frame to the appropriate method.


Instruction:
Write a Python 3 script for
sending a heartbeat if needed. Tracks last heartbeat send time.

Response:
def send_heartbeat(self):
        '''
        Send a heartbeat if needed. Tracks last heartbeat send time.
        '''
        # Note that this does not take into account the time that we last
        # sent a frame. Hearbeats are so small the effect should be quite
        # limited. Also note that we're looking for something near to our
        # scheduled interval, because if this is exact, then we'll likely
        # actually send a heartbeat at twice the period, which could cause
        # a broker to kill the connection if the period is large enough. The
        # 90% bound is arbitrary but seems a sensible enough default.
        if self.connection._heartbeat:
            if time.time() >= (self._last_heartbeat_send + 0.9 *
                               self.connection._heartbeat):
                self.send_frame(HeartbeatFrame(self.channel_id))
                self._last_heartbeat_send = time.time()


Instruction:
Implement a function in Python 3 to
send the start_ok message.

Response:
def _send_start_ok(self):
        '''Send the start_ok message.'''
        args = Writer()
        args.write_table(self.connection._properties)
        args.write_shortstr(self.connection._login_method)
        args.write_longstr(self.connection._login_response)
        args.write_shortstr(self.connection._locale)
        self.send_frame(MethodFrame(self.channel_id, 10, 11, args))

        self.add_synchronous_cb(self._recv_tune)


Instruction:
How would you explain what the following Python 3 function does
def declare(self, exchange, type, passive=False, durable=False,
                nowait=True, arguments=None, ticket=None, cb=None):
        """
        Declare the exchange.

        exchange - The name of the exchange to declare
        type - One of
        """
        nowait = nowait and self.allow_nowait() and not cb

        args = Writer()
        args.write_short(ticket or self.default_ticket).\
            write_shortstr(exchange).\
            write_shortstr(type).\
            write_bits(passive, durable, False, False, nowait).\
            write_table(arguments or {})
        self.send_frame(MethodFrame(self.channel_id, 40, 10, args))

        if not nowait:
            self._declare_cb.append(cb)
            self.channel.add_synchronous_cb(self._recv_declare_ok)

Response:
Declare the exchange.

        exchange - The name of the exchange to declare
        type - One of


Instruction:
Write a Python 3 script to
connect assuming a host and port tuple.

Response:
def connect(self, (host, port), klass=socket.socket):
        '''Connect assuming a host and port tuple.

        :param tuple: A tuple containing host and port for a connection.
        :param klass: A implementation of socket.socket.
        :raises socket.gaierror: If no address can be resolved.
        :raises socket.error: If no connection can be made.
        '''
        self._host = "%s:%s" % (host, port)

        for info in socket.getaddrinfo(host, port, 0, 0, socket.IPPROTO_TCP):

            family, socktype, proto, _, sockaddr = info
            self._sock = klass(family, socktype, proto)
            self._sock.settimeout(self.connection._connect_timeout)
            if self.connection._sock_opts:
                _sock_opts = self.connection._sock_opts
                for (level, optname), value in _sock_opts.iteritems():
                    self._sock.setsockopt(level, optname, value)
            try:

                self._sock.connect(sockaddr)

            except socket.error:

                self.connection.logger.exception(
                    "Failed to connect to %s:",
                    sockaddr,
                )
                continue

            # After connecting, switch to full-blocking mode.
            self._sock.settimeout(None)
            break

        else:

            raise


Instruction:
Create a Python 3 function for
reading from the transport. If timeout is 0 will block for timeout seconds. If timeout is None will block for timeout seconds.

Response:
def read(self, timeout=None):
        '''
        Read from the transport. If timeout>0, will only block for `timeout`
        seconds.
        '''
        e = None
        if not hasattr(self, '_sock'):
            return None

        try:
            # Note that we ignore both None and 0, i.e. we either block with a
            # timeout or block completely and let gevent sort it out.
            if timeout:
                self._sock.settimeout(timeout)
            else:
                self._sock.settimeout(None)
            data = self._sock.recv(
                self._sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF))

            if len(data):
                if self.connection.debug > 1:
                    self.connection.logger.debug(
                        'read %d bytes from %s' % (len(data), self._host))
                if len(self._buffer):
                    self._buffer.extend(data)
                    data = self._buffer
                    self._buffer = bytearray()
                return data

            # Note that no data means the socket is closed and we'll mark that
            # below

        except socket.timeout as e:
            # Note that this is implemented differently and though it would be
            # caught as an EnvironmentError, it has no errno. Not sure whose
            # fault that is.
            return None

        except EnvironmentError as e:
            # thrown if we have a timeout and no data
            if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK, errno.EINTR):
                return None

            self.connection.logger.exception(
                'error reading from %s' % (self._host))

        self.connection.transport_closed(
            msg='error reading from %s' % (self._host))
        if e:
            raise


Instruction:
Can you generate a brief explanation for the following Python 3 code
def buffer(self, data):
        '''
        Buffer unused bytes from the input stream.
        '''
        if not hasattr(self, '_sock'):
            return None

        # data will always be a byte array
        if len(self._buffer):
            self._buffer.extend(data)
        else:
            self._buffer = bytearray(data)

Response: Buffer unused bytes from the input stream.


Instruction:
Make a summary of the following Python 3 code
def write(self, data):
        '''
        Write some bytes to the transport.
        '''
        if not hasattr(self, '_sock'):
            return None

        try:
            self._sock.sendall(data)

            if self.connection.debug > 1:
                self.connection.logger.debug(
                    'sent %d bytes to %s' % (len(data), self._host))

            return
        except EnvironmentError:
            # sockets raise this type of error, and since if sendall() fails
            # we're left in an indeterminate state, assume that any error we
            # catch means that the connection is dead. Note that this
            # assumption requires this to be a blocking socket; if we ever
            # support non-blocking in this class then this whole method has
            # to change a lot.
            self.connection.logger.exception(
                'error writing to %s' % (self._host))

        self.connection.transport_closed(
            msg='error writing to %s' % (self._host))

Response: Write some bytes to the transport.


Instruction:
Explain what the following Python 3 code does
def _cleanup(self):
        '''
        Cleanup all the local data.
        '''
        self._pending_consumers = None
        self._consumer_cb = None
        self._get_cb = None
        self._recover_cb = None
        self._cancel_cb = None
        self._return_listener = None
        super(BasicClass, self)._cleanup()

Response: Cleanup all the local data.


Instruction:
Here you have a function in Python 3, explain what it does
def set_return_listener(self, cb):
        '''
        Set a callback for basic.return listening. Will be called with a single
        Message argument.

        The return_info attribute of the Message will have the following
        properties:
            'channel': Channel instance
            'reply_code': reply code (int)
            'reply_text': reply text
            'exchange': exchange name
            'routing_key': routing key

        RabbitMQ NOTE: if the channel was in confirmation mode when the message
        was published, then basic.return will still be followed by basic.ack
        later.

        :param cb: callable cb(Message); pass None to reset
        '''
        if cb is not None and not callable(cb):
            raise ValueError('return_listener callback must either be None or '
                             'a callable, but got: %r' % (cb,))
        self._return_listener = cb

Response:
Set a callback for basic. return listening. Will be called with a single
            Message argument.


Instruction:
Can you write a function in Python 3 where it
sets QoS on this channel.

Response:
def qos(self, prefetch_size=0, prefetch_count=0, is_global=False):
        '''
        Set QoS on this channel.
        '''
        args = Writer()
        args.write_long(prefetch_size).\
            write_short(prefetch_count).\
            write_bit(is_global)
        self.send_frame(MethodFrame(self.channel_id, 60, 10, args))

        self.channel.add_synchronous_cb(self._recv_qos_ok)


Instruction:
Make a summary of the following Python 3 code
def consume(self, queue, consumer, consumer_tag='', no_local=False,
                no_ack=True, exclusive=False, nowait=True, ticket=None,
                cb=None):
        '''
        Start a queue consumer. If `cb` is supplied, will be called when
        broker confirms that consumer is registered.
        '''
        nowait = nowait and self.allow_nowait() and not cb

        if nowait and consumer_tag == '':
            consumer_tag = self._generate_consumer_tag()

        args = Writer()
        args.write_short(ticket or self.default_ticket).\
            write_shortstr(queue).\
            write_shortstr(consumer_tag).\
            write_bits(no_local, no_ack, exclusive, nowait).\
            write_table({})  # unused according to spec
        self.send_frame(MethodFrame(self.channel_id, 60, 20, args))

        if not nowait:
            self._pending_consumers.append((consumer, cb))
            self.channel.add_synchronous_cb(self._recv_consume_ok)
        else:
            self._consumer_cb[consumer_tag] = consumer

Response: Start a queue consumer.


Instruction:
Can you generate the documentation for the following Python 3 function
def cancel(self, consumer_tag='', nowait=True, consumer=None, cb=None):
        '''
        Cancel a consumer. Can choose to delete based on a consumer tag or
        the function which is consuming.  If deleting by function, take care
        to only use a consumer once per channel.
        '''
        if consumer:
            tag = self._lookup_consumer_tag_by_consumer(consumer)
            if tag:
                consumer_tag = tag

        nowait = nowait and self.allow_nowait() and not cb

        args = Writer()
        args.write_shortstr(consumer_tag).\
            write_bit(nowait)
        self.send_frame(MethodFrame(self.channel_id, 60, 30, args))

        if not nowait:
            self._cancel_cb.append(cb)
            self.channel.add_synchronous_cb(self._recv_cancel_ok)
        else:
            self._purge_consumer_by_tag(consumer_tag)

Response: Cancel a consumer. Can only delete based on a consumer tag.


Instruction:
Can you create a Python 3 function that
looks up consumer tag given its consumer function

Response:
def _lookup_consumer_tag_by_consumer(self, consumer):
        '''Look up consumer tag given its consumer function

        NOTE: this protected method may be called by derived classes

        :param callable consumer: consumer function

        :returns: matching consumer tag or None
        :rtype: str or None
        '''
        for (tag, func) in self._consumer_cb.iteritems():
            if func == consumer:
                return tag


Instruction:
Implement a function in Python 3 to
purge consumer entry by consumer tag

Response:
def _purge_consumer_by_tag(self, consumer_tag):
        '''Purge consumer entry from this basic instance

        NOTE: this protected method may be called by derived classes

        :param str consumer_tag:
        '''
        try:
            del self._consumer_cb[consumer_tag]
        except KeyError:
            self.logger.warning(
                'no callback registered for consumer tag " %s "', consumer_tag)
        else:
            self.logger.info('purged consumer with tag " %s "', consumer_tag)


Instruction:
Can you write a function in Python 3 where it
returns a failed message.

Response:
def return_msg(self, reply_code, reply_text, exchange, routing_key):
        '''
        Return a failed message.  Not named "return" because python interpreter
        can't deal with that.
        '''
        args = Writer()
        args.write_short(reply_code).\
            write_shortstr(reply_text).\
            write_shortstr(exchange).\
            write_shortstr(routing_key)

        self.send_frame(MethodFrame(self.channel_id, 60, 50, args))


Instruction:
How would you explain what the following Python 3 function does
def _recv_return(self, method_frame):
        '''
        Handle basic.return method. If we have a complete message, will call the
        user's return listener callabck (if any). If there are not enough
        frames, will re-queue current frames and raise a FrameUnderflow

        NOTE: if the channel was in confirmation mode when the message was
        published, then this will still be followed by basic.ack later
        '''
        msg = self._read_returned_msg(method_frame)

        if callable(self._return_listener):
            self._return_listener(msg)
        else:
            self.logger.error(
                "Published message returned by broker: info=%s, properties=%s",
                msg.return_info, msg.properties)

Response: Handle a basic. return message.


Instruction:
How would you code a function in Python 3 to
ask to fetch a single message from a queue.

Response:
def get(self, queue, consumer=None, no_ack=True, ticket=None):
        '''
        Ask to fetch a single message from a queue.  If a consumer is supplied,
        the consumer will be called with either a Message argument, or None if
        there is no message in queue. If a synchronous transport, Message or
        None is returned.
        '''
        args = Writer()
        args.write_short(ticket or self.default_ticket).\
            write_shortstr(queue).\
            write_bit(no_ack)

        self._get_cb.append(consumer)
        self.send_frame(MethodFrame(self.channel_id, 60, 70, args))
        return self.channel.add_synchronous_cb(self._recv_get_response)


Instruction:
Can you tell what is the following Python 3 function doing
def _recv_get_response(self, method_frame):
        '''
        Handle either get_ok or get_empty. This is a hack because the
        synchronous callback stack is expecting one method to satisfy the
        expectation. To keep that loop as tight as possible, work within
        those constraints. Use of get is not recommended anyway.
        '''
        if method_frame.method_id == 71:
            return self._recv_get_ok(method_frame)
        elif method_frame.method_id == 72:
            return self._recv_get_empty(method_frame)

Response: Handle either get_ok or get_empty.


Instruction:
Here you have a function in Python 3, explain what it does
def ack(self, delivery_tag, multiple=False):
        '''
        Acknowledge delivery of a message.  If multiple=True, acknowledge up-to
        and including delivery_tag.
        '''
        args = Writer()
        args.write_longlong(delivery_tag).\
            write_bit(multiple)

        self.send_frame(MethodFrame(self.channel_id, 60, 80, args))

Response: Acknowledge delivery of a message.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def recover(self, requeue=False, cb=None):
        '''
        Ask server to redeliver all unacknowledged messages.
        '''
        args = Writer()
        args.write_bit(requeue)

        # The XML spec is incorrect; this method is always synchronous
        #  http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010738.html
        self._recover_cb.append(cb)
        self.send_frame(MethodFrame(self.channel_id, 60, 110, args))
        self.channel.add_synchronous_cb(self._recv_recover_ok)

Response: Ask server to redeliver all unacknowledged messages.


Instruction:
Can you implement a function in Python 3 that
reads a Message from the current frame buffer.

Response:
def _read_msg(self, method_frame, with_consumer_tag=False,
                  with_message_count=False):
        '''
        Support method to read a Message from the current frame buffer.
        Will return a Message, or re-queue current frames and raise a
        FrameUnderflow. Takes an optional argument on whether to read the
        consumer tag so it can be used for both deliver and get-ok.
        '''
        header_frame, body = self._reap_msg_frames(method_frame)

        if with_consumer_tag:
            consumer_tag = method_frame.args.read_shortstr()
        delivery_tag = method_frame.args.read_longlong()
        redelivered = method_frame.args.read_bit()
        exchange = method_frame.args.read_shortstr()
        routing_key = method_frame.args.read_shortstr()
        if with_message_count:
            message_count = method_frame.args.read_long()

        delivery_info = {
            'channel': self.channel,
            'delivery_tag': delivery_tag,
            'redelivered': redelivered,
            'exchange': exchange,
            'routing_key': routing_key,
        }
        if with_consumer_tag:
            delivery_info['consumer_tag'] = consumer_tag
        if with_message_count:
            delivery_info['message_count'] = message_count

        return Message(body=body, delivery_info=delivery_info,
                       **header_frame.properties)


Instruction:
Can you create a Python 3 function that
supports method to read a returned (basic.return) Message from the current frame buffer. Will return a Message with return_info, or re-queue current frames and raise a FrameUnderflow. :returns: Message with the return_info attribute set, where return_info is a dict with the following properties: 'channel': Channel instance 'reply_code': reply code (int) 'reply_text': reply text 'exchange': exchange name 'routing_key': routing key

Response:
def _read_returned_msg(self, method_frame):
        '''
        Support method to read a returned (basic.return) Message from the
        current frame buffer. Will return a Message with return_info, or
        re-queue current frames and raise a FrameUnderflow.

        :returns: Message with the return_info attribute set, where return_info
          is a dict with the following properties:
            'channel': Channel instance
            'reply_code': reply code (int)
            'reply_text': reply text
            'exchange': exchange name
            'routing_key': routing key
        '''
        header_frame, body = self._reap_msg_frames(method_frame)

        return_info = {
            'channel': self.channel,
            'reply_code': method_frame.args.read_short(),
            'reply_text': method_frame.args.read_shortstr(),
            'exchange': method_frame.args.read_shortstr(),
            'routing_key': method_frame.args.read_shortstr()
        }

        return Message(body=body, return_info=return_info,
                       **header_frame.properties)


Instruction:
Here you have a function in Python 3, explain what it does
def _reap_msg_frames(self, method_frame):
        '''
        Support method to reap header frame and body from current frame buffer.
        Used in processing of basic.return, basic.deliver, and basic.get_ok.
        Will return a pair (<header frame>, <body>), or re-queue current frames
        and raise a FrameUnderflow.

        :returns: pair (<header frame>, <body>)
        :rtype: tuple of (HeaderFrame, bytearray)
        '''
        # No need to assert that is instance of Header or Content frames
        # because failure to access as such will result in exception that
        # channel will pick up and handle accordingly.
        header_frame = self.channel.next_frame()
        if header_frame:
            size = header_frame.size
            body = bytearray()
            rbuf_frames = deque([header_frame, method_frame])

            while len(body) < size:
                content_frame = self.channel.next_frame()
                if content_frame:
                    rbuf_frames.appendleft(content_frame)
                    body.extend(content_frame.payload.buffer())
                else:
                    self.channel.requeue_frames(rbuf_frames)
                    raise self.FrameUnderflow()
        else:
            self.channel.requeue_frames([method_frame])
            raise self.FrameUnderflow()

        return (header_frame, body)

Response: Support method to reap header frame and body from current frame buffer.


Instruction:
Implement a Python 3 function for
publishing a message. Caller can supply an optional callback which will be fired when the transaction is committed. Tries very hard to avoid closed and inactive channels, but a ChannelError or ConnectionError may still be raised.

Response:
def publish(self, *args, **kwargs):
        '''
        Publish a message. Caller can supply an optional callback which will
        be fired when the transaction is committed. Tries very hard to avoid
        closed and inactive channels, but a ChannelError or ConnectionError
        may still be raised.
        '''
        user_cb = kwargs.pop('cb', None)

        # If the first channel we grab is inactive, continue fetching until
        # we get an active channel, then put the inactive channels back in
        # the pool. Try to keep the overhead to a minimum.
        channel = self._get_channel()

        if channel and not channel.active:
            inactive_channels = set()
            while channel and not channel.active:
                inactive_channels.add(channel)
                channel = self._get_channel()
            self._free_channels.update(inactive_channels)

        # When the transaction is committed, add the channel back to the pool
        # and call any user-defined callbacks. If there is anything in queue,
        # pop it and call back to publish(). Only do so if the channel is
        # still active though, because otherwise the message will end up at
        # the back of the queue, breaking the original order.
        def committed():
            self._free_channels.add(channel)
            if channel.active and not channel.closed:
                self._process_queue()
            if user_cb is not None:
                user_cb()

        if channel:
            channel.publish_synchronous(*args, cb=committed, **kwargs)
        else:
            kwargs['cb'] = user_cb
            self._queue.append((args, kwargs))


Instruction:
Can you generate the documentation for the following Python 3 function
def _process_queue(self):
        '''
        If there are any message in the queue, process one of them.
        '''
        if len(self._queue):
            args, kwargs = self._queue.popleft()
            self.publish(*args, **kwargs)

Response: Process the queue and publish the next message.


Instruction:
Implement a function in Python 3 to
fetch a channel from the free pool. Will return a new channel if necessary. Will return None if there is no free channel.

Response:
def _get_channel(self):
        '''
        Fetch a channel from the pool. Will return a new one if necessary. If
        a channel in the free pool is closed, will remove it. Will return None
        if we hit the cap. Will clean up any channels that were published to
        but closed due to error.
        '''
        while len(self._free_channels):
            rval = self._free_channels.pop()
            if not rval.closed:
                return rval
            # don't adjust _channels value because the callback will do that
            # and we don't want to double count it.

        if not self._size or self._channels < self._size:
            rval = self._connection.channel()
            self._channels += 1
            rval.add_close_listener(self._channel_closed_cb)
            return rval


Instruction:
How would you explain what the following Python 3 function does
def write_frame(self, buf):
        '''
        Write the frame into an existing buffer.
        '''
        writer = Writer(buf)

        writer.write_octet(self.type()).\
            write_short(self.channel_id).\
            write_long(len(self._payload)).\
            write(self._payload).\
            write_octet(0xce)

Response: Write the frame into a buffer.


Instruction:
How would you explain what the following Python 3 function does
def connect(self, (host, port)):
        '''
        Connect assuming a host and port tuple. Implemented as non-blocking,
        and will close the transport if there's an error
        '''
        self._host = "%s:%s" % (host, port)
        self._sock = EventSocket(
            read_cb=self._sock_read_cb,
            close_cb=self._sock_close_cb,
            error_cb=self._sock_error_cb,
            debug=self.connection.debug,
            logger=self.connection.logger)
        if self.connection._sock_opts:
            for k, v in self.connection._sock_opts.iteritems():
                family, type = k
                self._sock.setsockopt(family, type, v)
        self._sock.setblocking(False)
        self._sock.connect(
            (host, port), timeout=self.connection._connect_timeout)
        self._heartbeat_timeout = None

Response: Connects to a specific host and port tuple.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def read(self, timeout=None):
        '''
        Read from the transport. If no data is available, should return None.
        The timeout is ignored as this returns only data that has already
        been buffered locally.
        '''
        # NOTE: copying over this comment from Connection, because there is
        # knowledge captured here, even if the details are stale
        # Because of the timer callback to dataRead when we re-buffered,
        # there's a chance that in between we've lost the socket. If that's
        # the case, just silently return as some code elsewhere would have
        # already notified us. That bug could be fixed by improving the
        # message reading so that we consume all possible messages and ensure
        # that only a partial message was rebuffered, so that we can rely on
        # the next read event to read the subsequent message.
        if not hasattr(self, '_sock'):
            return None

        # This is sort of a hack because we're faking that data is ready, but
        # it works for purposes of supporting timeouts
        if timeout:
            if self._heartbeat_timeout:
                self._heartbeat_timeout.delete()
            self._heartbeat_timeout = \
                event.timeout(timeout, self._sock_read_cb, self._sock)
        elif self._heartbeat_timeout:
            self._heartbeat_timeout.delete()
            self._heartbeat_timeout = None

        return self._sock.read()

Response: Read from the transport.


Instruction:
Can you implement a function in Python 3 that
disconnects from the transport. Typically socket. close.

Response:
def disconnect(self):
        '''
        Disconnect from the transport. Typically socket.close(). This call is
        welcome to raise exceptions, which the Connection will catch.

        The transport is encouraged to allow for any pending writes to complete
        before closing the socket.
        '''
        if not hasattr(self, '_sock'):
            return

        # TODO: If there are bytes left on the output, queue the close for
        # later.
        self._sock.close_cb = None
        self._sock.close()


Instruction:
How would you code a function in Python 3 to
parse a header frame for a channel given a Reader payload.

Response:
def parse(self, channel_id, payload):
        '''
        Parse a header frame for a channel given a Reader payload.
        '''
        class_id = payload.read_short()
        weight = payload.read_short()
        size = payload.read_longlong()
        properties = {}

        # The AMQP spec is overly-complex when it comes to handling header
        # frames. The spec says that in addition to the first 16bit field,
        # additional ones can follow which /may/ then be in the property list
        # (because bit flags aren't in the list). Properly implementing custom
        # values requires the ability change the properties and their types,
        # which someone is welcome to do, but seriously, what's the point?
        # Because the complexity of parsing and writing this frame directly
        # impacts the speed at which messages can be processed, there are two
        # branches for both a fast parse which assumes no changes to the
        # properties and a slow parse. For now it's up to someone using custom
        # headers to flip the flag.
        if self.DEFAULT_PROPERTIES:
            flag_bits = payload.read_short()
            for key, proptype, rfunc, wfunc, mask in self.PROPERTIES:
                if flag_bits & mask:
                    properties[key] = rfunc(payload)
        else:
            flags = []
            while True:
                flag_bits = payload.read_short()
                flags.append(flag_bits)
                if flag_bits & 1 == 0:
                    break

            shift = 0
            for key, proptype, rfunc, wfunc, mask in self.PROPERTIES:
                if shift == 0:
                    if not flags:
                        break
                    flag_bits, flags = flags[0], flags[1:]
                    shift = 15
                if flag_bits & (1 << shift):
                    properties[key] = rfunc(payload)
                shift -= 1

        return HeaderFrame(channel_id, class_id, weight, size, properties)


Instruction:
Can you tell what is the following Python 3 function doing
def write_frame(self, buf):
        '''
        Write the frame into an existing buffer.
        '''
        writer = Writer(buf)
        writer.write_octet(self.type())
        writer.write_short(self.channel_id)

        # Track the position where we're going to write the total length
        # of the frame arguments.
        stream_args_len_pos = len(buf)
        writer.write_long(0)

        stream_method_pos = len(buf)

        writer.write_short(self._class_id)
        writer.write_short(self._weight)
        writer.write_longlong(self._size)

        # Like frame parsing, branch to faster code for default properties
        if self.DEFAULT_PROPERTIES:
            # Track the position where we're going to write the flags.
            flags_pos = len(buf)
            writer.write_short(0)
            flag_bits = 0
            for key, proptype, rfunc, wfunc, mask in self.PROPERTIES:
                val = self._properties.get(key, None)
                if val is not None:
                    flag_bits |= mask
                    wfunc(writer, val)
            writer.write_short_at(flag_bits, flags_pos)
        else:
            shift = 15
            flag_bits = 0
            flags = []
            stack = deque()
            for key, proptype, rfunc, wfunc, mask in self.PROPERTIES:
                val = self._properties.get(key, None)
                if val is not None:
                    if shift == 0:
                        flags.append(flag_bits)
                        flag_bits = 0
                        shift = 15

                    flag_bits |= (1 << shift)
                    stack.append((wfunc, val))

                shift -= 1

            flags.append(flag_bits)
            for flag_bits in flags:
                writer.write_short(flag_bits)
            for method, val in stack:
                method(writer, val)

        # Write the total length back at the beginning of the frame
        stream_len = len(buf) - stream_method_pos
        writer.write_long_at(stream_len, stream_args_len_pos)

        writer.write_octet(0xce)

Response: Write the current state of the object into a buffer.


Instruction:
Can you implement a function in Python 3 that
reads one or more frames from an IO stream.

Response:
def read_frames(cls, reader):
        '''
        Read one or more frames from an IO stream.  Buffer must support file
        object interface.

        After reading, caller will need to check if there are bytes remaining
        in the stream. If there are, then that implies that there is one or
        more incomplete frames and more data needs to be read.  The position
        of the cursor in the frame stream will mark the point at which the
        last good frame was read. If the caller is expecting a sequence of
        frames and only received a part of that sequence, they are responsible
        for buffering those frames until the rest of the frames in the sequence
        have arrived.
        '''
        rval = deque()

        while True:
            frame_start_pos = reader.tell()
            try:
                frame = Frame._read_frame(reader)
            except Reader.BufferUnderflow:
                # No more data in the stream
                frame = None
            except Reader.ReaderError as e:
                # Some other format error
                raise Frame.FormatError, str(e), sys.exc_info()[-1]
            except struct.error as e:
                raise Frame.FormatError, str(e), sys.exc_info()[-1]

            if frame is None:
                reader.seek(frame_start_pos)
                break

            rval.append(frame)

        return rval


Instruction:
Can you implement a function in Python 3 that
unbinds an exchange from another.

Response:
def unbind(self, exchange, source, routing_key='', nowait=True,
               arguments={}, ticket=None, cb=None):
        '''
        Unbind an exchange from another.
        '''
        nowait = nowait and self.allow_nowait() and not cb

        args = Writer()
        args.write_short(ticket or self.default_ticket).\
            write_shortstr(exchange).\
            write_shortstr(source).\
            write_shortstr(routing_key).\
            write_bit(nowait).\
            write_table(arguments or {})
        self.send_frame(MethodFrame(self.channel_id, 40, 40, args))

        if not nowait:
            self._unbind_cb.append(cb)
            self.channel.add_synchronous_cb(self._recv_unbind_ok)


Instruction:
Create a Python 3 function for
publishing a message. Will return the id of the message.

Response:
def publish(self, *args, **kwargs):
        '''
        Publish a message. Will return the id of the message if publisher
        confirmations are enabled, else will return 0.
        '''
        if self.channel.confirm._enabled:
            self._msg_id += 1
        super(RabbitBasicClass, self).publish(*args, **kwargs)
        return self._msg_id


Instruction:
How would you explain what the following Python 3 function does
def _recv_ack(self, method_frame):
        '''Receive an ack from the broker.'''
        if self._ack_listener:
            delivery_tag = method_frame.args.read_longlong()
            multiple = method_frame.args.read_bit()
            if multiple:
                while self._last_ack_id < delivery_tag:
                    self._last_ack_id += 1
                    self._ack_listener(self._last_ack_id)
            else:
                self._last_ack_id = delivery_tag
                self._ack_listener(self._last_ack_id)

Response: Receive an ack from the broker.


Instruction:
Can you write a function in Python 3 where it
sends a nack to the broker.

Response:
def nack(self, delivery_tag, multiple=False, requeue=False):
        '''Send a nack to the broker.'''
        args = Writer()
        args.write_longlong(delivery_tag).\
            write_bits(multiple, requeue)

        self.send_frame(MethodFrame(self.channel_id, 60, 120, args))


Instruction:
How would you explain what the following Python 3 function does
def _recv_nack(self, method_frame):
        '''Receive a nack from the broker.'''
        if self._nack_listener:
            delivery_tag = method_frame.args.read_longlong()
            multiple, requeue = method_frame.args.read_bits(2)
            if multiple:
                while self._last_ack_id < delivery_tag:
                    self._last_ack_id += 1
                    self._nack_listener(self._last_ack_id, requeue)
            else:
                self._last_ack_id = delivery_tag
                self._nack_listener(self._last_ack_id, requeue)

Response: Receive a nack from the broker.


Instruction:
Make a summary of the following Python 3 code
def consume(self, queue, consumer, consumer_tag='', no_local=False,
                no_ack=True, exclusive=False, nowait=True, ticket=None,
                cb=None, cancel_cb=None):
        '''Start a queue consumer.

        Accepts the following optional arg in addition to those of
        `BasicClass.consume()`:

        :param cancel_cb: a callable to be called when the broker cancels the
          consumer; e.g., when the consumer's queue is deleted. See
          www.rabbitmq.com/consumer-cancel.html.
        :type cancel_cb: None or callable with signature cancel_cb(consumer_tag)
        '''
        # Register the consumer's broker-cancel callback entry
        if cancel_cb is not None:
            if not callable(cancel_cb):
                raise ValueError('cancel_cb is not callable: %r' % (cancel_cb,))

        if not consumer_tag:
            consumer_tag = self._generate_consumer_tag()

        self._broker_cancel_cb_map[consumer_tag] = cancel_cb

        # Start consumer
        super(RabbitBasicClass, self).consume(queue, consumer, consumer_tag,
                                              no_local, no_ack, exclusive,
                                              nowait, ticket, cb)

Response: Start a queue consumer.


Instruction:
How would you code a function in Python 3 to
cancel a consumer. Can only delete by function.

Response:
def cancel(self, consumer_tag='', nowait=True, consumer=None, cb=None):
        '''
        Cancel a consumer. Can choose to delete based on a consumer tag or
        the function which is consuming.  If deleting by function, take care
        to only use a consumer once per channel.
        '''
        # Remove the consumer's broker-cancel callback entry
        if consumer:
            tag = self._lookup_consumer_tag_by_consumer(consumer)
            if tag:
                consumer_tag = tag

        try:
            del self._broker_cancel_cb_map[consumer_tag]
        except KeyError:
            self.logger.warning(
                'cancel: no broker-cancel-cb entry for consumer tag %r '
                '(consumer %r)', consumer_tag, consumer)

        # Cancel consumer
        super(RabbitBasicClass, self).cancel(consumer_tag, nowait, consumer, cb)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _recv_cancel(self, method_frame):
        '''Handle Basic.Cancel from broker

        :param MethodFrame method_frame: Basic.Cancel method frame from broker
        '''
        self.logger.warning("consumer cancelled by broker: %r", method_frame)

        consumer_tag = method_frame.args.read_shortstr()

        # NOTE: per RabbitMQ spec, no-wait is always true in Basic.Cancel from
        # broker

        # Remove consumer from this basic instance
        try:
            cancel_cb = self._broker_cancel_cb_map.pop(consumer_tag)
        except KeyError:
            # Must be a race condition between user's cancel and broker's cancel
            self.logger.warning(
                '_recv_cancel: no broker-cancel-cb entry for consumer tag %r',
                consumer_tag)
        else:
            if callable(cancel_cb):
                # Purge from base class only when user supplies cancel_cb
                self._purge_consumer_by_tag(consumer_tag)

                # Notify user
                cancel_cb(consumer_tag)

Response: Handle Basic. Cancel from broker.


Instruction:
Implement a function in Python 3 to
select a new entry from the list of available entries.

Response:
def select(self, nowait=True, cb=None):
        '''
        Set this channel to use publisher confirmations.
        '''
        nowait = nowait and self.allow_nowait() and not cb

        if not self._enabled:
            self._enabled = True
            self.channel.basic._msg_id = 0
            self.channel.basic._last_ack_id = 0
            args = Writer()
            args.write_bit(nowait)

            self.send_frame(MethodFrame(self.channel_id, 85, 10, args))

            if not nowait:
                self._select_cb.append(cb)
                self.channel.add_synchronous_cb(self._recv_select_ok)


Instruction:
Here you have a function in Python 3, explain what it does
def close(self, reply_code=0, reply_text='', class_id=0, method_id=0):
        '''
        Close this channel.  Routes to channel.close.
        '''
        # In the off chance that we call this twice. A good example is if
        # there's an error in close listeners and so we're still inside a
        # single call to process_frames, which will try to close this channel
        # if there's an exception.
        if hasattr(self, 'channel'):
            self.channel.close(reply_code, reply_text, class_id, method_id)

Response: Closes this channel. Routes to channel. close.


Instruction:
Here you have a function in Python 3, explain what it does
def process_frames(self):
        '''
        Process the input buffer.
        '''
        while len(self._frame_buffer):
            # It would make sense to call next_frame, but it's
            # technically faster to repeat the code here.
            frame = self._frame_buffer.popleft()

            if self._emergency_close_pending:
                # Implement stability rule from AMQP 0.9.1 section 1.5.2.5.
                # Method channel.close: "After sending this method, any
                # received methods except Close and Close-OK MUST be discarded."
                #
                # NOTE: presently, we limit our implementation of the rule to
                # the "emergency close" scenario to avoid potential adverse
                # side-effect during normal user-initiated close
                if (not isinstance(frame, MethodFrame) or
                      frame.class_id != self.channel.CLASS_ID or
                      frame.method_id not in (self.channel.CLOSE_METHOD_ID,
                                              self.channel.CLOSE_OK_METHOD_ID)):
                    self.logger.warn("Emergency channel close: dropping input "
                                     "frame %.255s", frame)
                    continue
            try:
                self.dispatch(frame)
            except ProtocolClass.FrameUnderflow:
                return
            except (ConnectionClosed, ChannelClosed):
                # Immediately raise if connection or channel is closed
                raise
            except Exception:
                self.logger.exception(
                    "Closing on failed dispatch of frame %.255s", frame)

                # Spec says that channel should be closed if there's a framing
                # error. Unsure if we can send close if the current exception
                # is transport level (e.g. gevent.GreenletExit)
                self._emergency_close_pending = True

                # Preserve the original exception and traceback during cleanup,
                # only allowing system-exiting exceptions (e.g., SystemExit,
                # KeyboardInterrupt) to override it
                try:
                    raise
                finally:
                    try:
                        self.close(500, "Failed to dispatch %s" % (str(frame)))
                    except Exception:
                        # Suppress secondary non-system-exiting exception in
                        # favor of the original exception
                        self.logger.exception("Channel close failed")
                        pass

Response: Process the input buffer and dispatch the next frame.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def send_frame(self, frame):
        '''
        Queue a frame for sending.  Will send immediately if there are no
        pending synchronous transactions on this connection.
        '''
        if self.closed:
            if self.close_info and len(self.close_info['reply_text']) > 0:
                raise ChannelClosed(
                    "channel %d is closed: %s : %s",
                    self.channel_id,
                    self.close_info['reply_code'],
                    self.close_info['reply_text'])
            raise ChannelClosed()

        # If there's any pending event at all, then it means that when the
        # current dispatch loop started, all possible frames were flushed
        # and the remaining item(s) starts with a sync callback. After careful
        # consideration, it seems that it's safe to assume the len>0 means to
        # buffer the frame. The other advantage here is
        if not len(self._pending_events):
            if not self._active and \
                    isinstance(frame, (ContentFrame, HeaderFrame)):
                raise Channel.Inactive(
                    "Channel %d flow control activated", self.channel_id)
            self._connection.send_frame(frame)
        else:
            self._pending_events.append(frame)

Response: Queue a frame for sending. Will send immediately if there are pending synchronous transactions on this connection. Will send immediately if there are pending synchronous transactions on this connection. Will send immediately if there are pending synchronous transactions on this connection.


Instruction:
Can you implement a function in Python 3 that
adds an expectation of a callback to release a synchronous transaction.

Response:
def add_synchronous_cb(self, cb):
        '''
        Add an expectation of a callback to release a synchronous transaction.
        '''
        if self.connection.synchronous or self._synchronous:
            wrapper = SyncWrapper(cb)
            self._pending_events.append(wrapper)
            while wrapper._read:
                # Don't check that the channel has been closed until after
                # reading frames, in the case that this is processing a clean
                # channel closed. If there's a protocol error during
                # read_frames, this will loop back around and result in a
                # channel closed exception.
                if self.closed:
                    if self.close_info and \
                            len(self.close_info['reply_text']) > 0:
                        raise ChannelClosed(
                            "channel %d is closed: %s : %s",
                            self.channel_id,
                            self.close_info['reply_code'],
                            self.close_info['reply_text'])
                    raise ChannelClosed()
                self.connection.read_frames()

            return wrapper._result
        else:
            self._pending_events.append(cb)


Instruction:
Given the following Python 3 function, write the documentation
def clear_synchronous_cb(self, cb):
        '''
        If the callback is the current expected callback, will clear it off the
        stack.  Else will raise in exception if there's an expectation but this
        doesn't satisfy it.
        '''
        if len(self._pending_events):
            ev = self._pending_events[0]

            # We can't have a strict check using this simple mechanism,
            # because we could be waiting for a synch response while messages
            # are being published. So for now, if it's not in the list, do a
            # check to see if the callback is in the pending list, and if so,
            # then raise, because it means we received stuff out of order.
            # Else just pass it through. Note that this situation could happen
            # on any broker-initiated message.
            if ev == cb:
                self._pending_events.popleft()
                self._flush_pending_events()
                return ev

            elif cb in self._pending_events:
                raise ChannelError(
                    "Expected synchronous callback %s, got %s", ev, cb)
        # Return the passed-in callback by default
        return cb

Response: Clear the synchronous callback for the current node.


Instruction:
Can you generate the documentation for the following Python 3 function
def _flush_pending_events(self):
        '''
        Send pending frames that are in the event queue.
        '''
        while len(self._pending_events) and \
                isinstance(self._pending_events[0], Frame):
            self._connection.send_frame(self._pending_events.popleft())

Response: Send pending frames that are in the event queue.


Instruction:
Here you have a function in Python 3, explain what it does
def _closed_cb(self, final_frame=None):
        '''
        "Private" callback from the ChannelClass when a channel is closed. Only
        called after broker initiated close, or we receive a close_ok. Caller
        has the option to send a final frame, to be used to bypass any
        synchronous or otherwise-pending frames so that the channel can be
        cleanly closed.
        '''
        # delete all pending data and send final frame if thre is one. note
        # that it bypasses send_frame so that even if the closed state is set,
        # the frame is published.
        if final_frame:
            self._connection.send_frame(final_frame)

        try:
            self._notify_close_listeners()
        finally:
            self._pending_events = deque()
            self._frame_buffer = deque()

            # clear out other references for faster cleanup
            for protocol_class in self._class_map.values():
                protocol_class._cleanup()
                delattr(self, protocol_class.name)
            self._connection = None
            self._class_map = None
            self._close_listeners = set()

Response: Private callback from the ChannelClass when a channel is closed.


Instruction:
Write a Python 3 function for
connecting using a host and port tuple

Response:
def connect(self, (host, port)):
        '''
        Connect using a host,port tuple
        '''
        super(GeventTransport, self).connect((host, port), klass=socket.socket)


Instruction:
Implement a Python 3 function for
reading from the transport.

Response:
def read(self, timeout=None):
        '''
        Read from the transport. If no data is available, should return None.
        If timeout>0, will only block for `timeout` seconds.
        '''
        # If currently locked, another greenlet is trying to read, so yield
        # control and then return none. Required if a Connection is configured
        # to be synchronous, a sync callback is trying to read, and there's
        # another read loop running read_frames. Without it, the run loop will
        # release the lock but then immediately acquire it again. Yielding
        # control in the reading thread after bytes are read won't fix
        # anything, because it's quite possible the bytes read resulted in a
        # frame that satisfied the synchronous callback, and so this needs to
        # return immediately to first check the current status of synchronous
        # callbacks before attempting to read again.
        if self._read_lock.locked():
            self._read_wait.wait(timeout)
            return None

        self._read_lock.acquire()
        try:
            return super(GeventTransport, self).read(timeout=timeout)
        finally:
            self._read_lock.release()
            self._read_wait.set()
            self._read_wait.clear()


Instruction:
How would you code a function in Python 3 to
buffer unused bytes from the input stream.

Response:
def buffer(self, data):
        '''
        Buffer unused bytes from the input stream.
        '''
        self._read_lock.acquire()
        try:
            return super(GeventTransport, self).buffer(data)
        finally:
            self._read_lock.release()


Instruction:
How would you explain what the following Python 3 function does
def write(self, data):
        '''
        Write some bytes to the transport.
        '''
        # MUST use a lock here else gevent could raise an exception if 2
        # greenlets try to write at the same time. I was hoping that
        # sendall() would do that blocking for me, but I guess not. May
        # require an eventsocket-like buffer to speed up under high load.
        self._write_lock.acquire()
        try:
            return super(GeventTransport, self).write(data)
        finally:
            self._write_lock.release()

Response: Write some bytes to the transport.


Instruction:
Explain what the following Python 3 code does
def process_channels(self, channels):
        '''
        Process a set of channels by calling Channel.process_frames() on each.
        Some transports may choose to do this in unique ways, such as through
        a pool of threads.

        The default implementation will simply iterate over them and call
        process_frames() on each.
        '''
        for channel in channels:
            self._pool.spawn(channel.process_frames)

Response: Process a set of channels by calling Channel. process_frames on each.


Instruction:
How would you code a function in Python 3 to
dispatch a method for this protocol.

Response:
def dispatch(self, method_frame):
        '''
        Dispatch a method for this protocol.
        '''
        method = self.dispatch_map.get(method_frame.method_id)
        if method:
            callback = self.channel.clear_synchronous_cb(method)
            callback(method_frame)
        else:
            raise self.InvalidMethod(
                "no method is registered with id: %d" % method_frame.method_id)


Instruction:
Can you tell what is the following Python 3 function doing
def seek(self, offset, whence=0):
        '''
        Simple seek. Follows standard interface.
        '''
        if whence == 0:
            self._pos = self._start_pos + offset
        elif whence == 1:
            self._pos += offset
        else:
            self._pos = (self._end_pos - 1) + offset

Response: Simple seek. Follows standard interface.


Instruction:
Create a Python 3 function for
raising BufferUnderflow if there s not enough bytes to satisfy the request.

Response:
def _check_underflow(self, n):
        '''
        Raise BufferUnderflow if there's not enough bytes to satisfy
        the request.
        '''
        if self._pos + n > self._end_pos:
            raise self.BufferUnderflow()