Instruction:
Can you implement a function in Python 3 that
creates dataset objects for splits of the WikiText - 2 dataset.

Response:
def splits(cls, text_field, root='.data', train='wiki.train.tokens',
               validation='wiki.valid.tokens', test='wiki.test.tokens',
               **kwargs):
        """Create dataset objects for splits of the WikiText-2 dataset.

        This is the most flexible way to use the dataset.

        Arguments:
            text_field: The field that will be used for text data.
            root: The root directory that the dataset's zip archive will be
                expanded into; therefore the directory in whose wikitext-2
                subdirectory the data files will be stored.
            train: The filename of the train data. Default: 'wiki.train.tokens'.
            validation: The filename of the validation data, or None to not
                load the validation set. Default: 'wiki.valid.tokens'.
            test: The filename of the test data, or None to not load the test
                set. Default: 'wiki.test.tokens'.
        """
        return super(WikiText2, cls).splits(
            root=root, train=train, validation=validation, test=test,
            text_field=text_field, **kwargs)


Instruction:
Implement a Python 3 function for
creating an iterator object for splits of the WikiText - 2 dataset.

Response:
def iters(cls, batch_size=32, bptt_len=35, device=0, root='.data',
              vectors=None, **kwargs):
        """Create iterator objects for splits of the WikiText-2 dataset.

        This is the simplest way to use the dataset, and assumes common
        defaults for field, vocabulary, and iterator parameters.

        Arguments:
            batch_size: Batch size.
            bptt_len: Length of sequences for backpropagation through time.
            device: Device to create batches on. Use -1 for CPU and None for
                the currently active GPU device.
            root: The root directory that the dataset's zip archive will be
                expanded into; therefore the directory in whose wikitext-2
                subdirectory the data files will be stored.
            wv_dir, wv_type, wv_dim: Passed to the Vocab constructor for the
                text field. The word vectors are accessible as
                train.dataset.fields['text'].vocab.vectors.
            Remaining keyword arguments: Passed to the splits method.
        """
        TEXT = data.Field()

        train, val, test = cls.splits(TEXT, root=root, **kwargs)

        TEXT.build_vocab(train, vectors=vectors)

        return data.BPTTIterator.splits(
            (train, val, test), batch_size=batch_size, bptt_len=bptt_len,
            device=device)


Instruction:
Explain what the following Python 3 code does
def call(self, x, *args):
        if isinstance(x, list):
            return [self.convert_token(tok, *args) for tok in x]
        return self.convert_token(x, *args)

Response:
Applies the convert_token function of the current pipeline
alid to the input.


Instruction:
Write a Python 3 function for
adding a Pipeline to be applied before this processing pipeline.

Response:
def add_before(self, pipeline):
        """Add a Pipeline to be applied before this processing pipeline.

        Arguments:
            pipeline: The Pipeline or callable to apply before this
                Pipeline.
        """
        if not isinstance(pipeline, Pipeline):
            pipeline = Pipeline(pipeline)
        self.pipes = pipeline.pipes[:] + self.pipes[:]
        return self


Instruction:
How would you implement a function in Python 3 that
adds a Pipeline to be applied after this processing pipeline.

Response:
def add_after(self, pipeline):
        """Add a Pipeline to be applied after this processing pipeline.

        Arguments:
            pipeline: The Pipeline or callable to apply after this
                Pipeline.
        """
        if not isinstance(pipeline, Pipeline):
            pipeline = Pipeline(pipeline)
        self.pipes = self.pipes[:] + pipeline.pipes[:]
        return self


Instruction:
Create a Python 3 function for
checking that the split ratio argument is not malformed

Response:
def check_split_ratio(split_ratio):
    """Check that the split ratio argument is not malformed"""
    valid_ratio = 0.
    if isinstance(split_ratio, float):
        # Only the train set relative ratio is provided
        # Assert in bounds, validation size is zero
        assert 0. < split_ratio < 1., (
            "Split ratio {} not between 0 and 1".format(split_ratio))

        test_ratio = 1. - split_ratio
        return (split_ratio, test_ratio, valid_ratio)
    elif isinstance(split_ratio, list):
        # A list of relative ratios is provided
        length = len(split_ratio)
        assert length == 2 or length == 3, (
            "Length of split ratio list should be 2 or 3, got {}".format(split_ratio))

        # Normalize if necessary
        ratio_sum = sum(split_ratio)
        if not ratio_sum == 1.:
            split_ratio = [float(ratio) / ratio_sum for ratio in split_ratio]

        if length == 2:
            return tuple(split_ratio + [valid_ratio])
        return tuple(split_ratio)
    else:
        raise ValueError('Split ratio must be float or a list, got {}'
                         .format(type(split_ratio)))


Instruction:
Write a Python 3 function that can
create train - test - valid splits from the instance s examples.

Response:
def split(self, split_ratio=0.7, stratified=False, strata_field='label',
              random_state=None):
        """Create train-test(-valid?) splits from the instance's examples.

        Arguments:
            split_ratio (float or List of floats): a number [0, 1] denoting the amount
                of data to be used for the training split (rest is used for validation),
                or a list of numbers denoting the relative sizes of train, test and valid
                splits respectively. If the relative size for valid is missing, only the
                train-test split is returned. Default is 0.7 (for the train set).
            stratified (bool): whether the sampling should be stratified.
                Default is False.
            strata_field (str): name of the examples Field stratified over.
                Default is 'label' for the conventional label field.
            random_state (tuple): the random seed used for shuffling.
                A return value of `random.getstate()`.

        Returns:
            Tuple[Dataset]: Datasets for train, validation, and
            test splits in that order, if the splits are provided.
        """
        train_ratio, test_ratio, val_ratio = check_split_ratio(split_ratio)

        # For the permutations
        rnd = RandomShuffler(random_state)
        if not stratified:
            train_data, test_data, val_data = rationed_split(self.examples, train_ratio,
                                                             test_ratio, val_ratio, rnd)
        else:
            if strata_field not in self.fields:
                raise ValueError("Invalid field name for strata_field {}"
                                 .format(strata_field))
            strata = stratify(self.examples, strata_field)
            train_data, test_data, val_data = [], [], []
            for group in strata:
                # Stratify each group and add together the indices.
                group_train, group_test, group_val = rationed_split(group, train_ratio,
                                                                    test_ratio, val_ratio,
                                                                    rnd)
                train_data += group_train
                test_data += group_test
                val_data += group_val

        splits = tuple(Dataset(d, self.fields)
                       for d in (train_data, val_data, test_data) if d)

        # In case the parent sort key isn't none
        if self.sort_key:
            for subset in splits:
                subset.sort_key = self.sort_key
        return splits


Instruction:
Implement a Python 3 function for
downloading and unzip an online archive.

Response:
def download(cls, root, check=None):
        """Download and unzip an online archive (.zip, .gz, or .tgz).

        Arguments:
            root (str): Folder to download data to.
            check (str or None): Folder whose existence indicates
                that the dataset has already been downloaded, or
                None to check the existence of root/{cls.name}.

        Returns:
            str: Path to extracted dataset.
        """
        path = os.path.join(root, cls.name)
        check = path if check is None else check
        if not os.path.isdir(check):
            for url in cls.urls:
                if isinstance(url, tuple):
                    url, filename = url
                else:
                    filename = os.path.basename(url)
                zpath = os.path.join(path, filename)
                if not os.path.isfile(zpath):
                    if not os.path.exists(os.path.dirname(zpath)):
                        os.makedirs(os.path.dirname(zpath))
                    print('downloading {}'.format(filename))
                    download_from_url(url, zpath)
                zroot, ext = os.path.splitext(zpath)
                _, ext_inner = os.path.splitext(zroot)
                if ext == '.zip':
                    with zipfile.ZipFile(zpath, 'r') as zfile:
                        print('extracting')
                        zfile.extractall(path)
                # tarfile cannot handle bare .gz files
                elif ext == '.tgz' or ext == '.gz' and ext_inner == '.tar':
                    with tarfile.open(zpath, 'r:gz') as tar:
                        dirs = [member for member in tar.getmembers()]
                        tar.extractall(path=path, members=dirs)
                elif ext == '.gz':
                    with gzip.open(zpath, 'rb') as gz:
                        with open(zroot, 'wb') as uncompressed:
                            shutil.copyfileobj(gz, uncompressed)

        return os.path.join(path, cls.dirname)


Instruction:
Can you tell what is the following Python 3 function doing
def filter_examples(self, field_names):
        for i, example in enumerate(self.examples):
            for field_name in field_names:
                vocab = set(self.fields[field_name].vocab.stoi)
                text = getattr(example, field_name)
                example_part = [word for word in text if word in vocab]
                setattr(example, field_name, example_part)
            self.examples[i] = example

Response: Remove unknown words from dataset examples with respect to given field.


Instruction:
Here you have a function in Python 3, explain what it does
def splits(cls, text_field, label_field, parse_field=None,
               extra_fields={}, root='.data', train='train.jsonl',
               validation='val.jsonl', test='test.jsonl'):
        """Create dataset objects for splits of the SNLI dataset.

        This is the most flexible way to use the dataset.

        Arguments:
            text_field: The field that will be used for premise and hypothesis
                data.
            label_field: The field that will be used for label data.
            parse_field: The field that will be used for shift-reduce parser
                transitions, or None to not include them.
            extra_fields: A dict[json_key: Tuple(field_name, Field)]
            root: The root directory that the dataset's zip archive will be
                expanded into.
            train: The filename of the train data. Default: 'train.jsonl'.
            validation: The filename of the validation data, or None to not
                load the validation set. Default: 'dev.jsonl'.
            test: The filename of the test data, or None to not load the test
                set. Default: 'test.jsonl'.
        """
        path = cls.download(root)

        if parse_field is None:
            fields = {'sentence1': ('premise', text_field),
                      'sentence2': ('hypothesis', text_field),
                      'gold_label': ('label', label_field)}
        else:
            fields = {'sentence1_binary_parse': [('premise', text_field),
                                                 ('premise_transitions', parse_field)],
                      'sentence2_binary_parse': [('hypothesis', text_field),
                                                 ('hypothesis_transitions', parse_field)],
                      'gold_label': ('label', label_field)}

        for key in extra_fields:
            if key not in fields.keys():
                fields[key] = extra_fields[key]

        return super(NLIDataset, cls).splits(
            path, root, train, validation, test,
            format='json', fields=fields,
            filter_pred=lambda ex: ex.label != '-')

Response: Create a new dataset object for splits of the SNLI dataset.


Instruction:
Make a summary of the following Python 3 code
def iters(cls, batch_size=32, device=0, root='.data',
              vectors=None, trees=False, **kwargs):
        """Create iterator objects for splits of the SNLI dataset.

        This is the simplest way to use the dataset, and assumes common
        defaults for field, vocabulary, and iterator parameters.

        Arguments:
            batch_size: Batch size.
            device: Device to create batches on. Use -1 for CPU and None for
                the currently active GPU device.
            root: The root directory that the dataset's zip archive will be
                expanded into; therefore the directory in whose wikitext-2
                subdirectory the data files will be stored.
            vectors: one of the available pretrained vectors or a list with each
                element one of the available pretrained vectors (see Vocab.load_vectors)
            trees: Whether to include shift-reduce parser transitions.
                Default: False.
            Remaining keyword arguments: Passed to the splits method.
        """
        if trees:
            TEXT = ParsedTextField()
            TRANSITIONS = ShiftReduceField()
        else:
            TEXT = data.Field(tokenize='spacy')
            TRANSITIONS = None
        LABEL = data.Field(sequential=False)

        train, val, test = cls.splits(
            TEXT, LABEL, TRANSITIONS, root=root, **kwargs)

        TEXT.build_vocab(train, vectors=vectors)
        LABEL.build_vocab(train)

        return data.BucketIterator.splits(
            (train, val, test), batch_size=batch_size, device=device)

Response: Create iterator objects for splits of the SNLI dataset.


Instruction:
How would you explain what the following Python 3 function does
def bytes_to_unicode(byte_data, escape, skip_printable=False):
    if isnumber(byte_data):
        if skip_printable:
            return num2unistr(byte_data)
        else:
            byte_data = num2bytes(byte_data)
    else:
        assert (isinstance(byte_data, type(b'')))
        if skip_printable and all(0x20 <= bval(ch) <= 0x7E for ch in byte_data):
            escape = STRING_ESCAPE_RAW

    if escape == STRING_ESCAPE_RAW:
        return byte_data.decode('latin-1')
    elif escape == STRING_ESCAPE_PRINT:
        return escape_ascii(byte_data)
    elif escape == STRING_ESCAPE_UTF8:
        return escape_utf8(byte_data)
    elif escape == STRING_ESCAPE_BASE64:
        return codecs.decode(base64.b64encode(byte_data), 'latin-1')
    else:
        raise UnicodeEncodeError("Unknown escape option")

Response: Decode given bytes using specified escaping method.


Instruction:
How would you explain what the following Python 3 function does
def apply_escape_bytes(byte_data, escape, skip_printable=False):

    if isnumber(byte_data):
        if skip_printable:
            return num2bytes(byte_data)
        else:
            byte_data = num2bytes(byte_data)
    else:
        assert (isinstance(byte_data, type(b'')))
        if skip_printable and all(0x20 <= bval(ch) <= 0x7E for ch in byte_data):
            escape = STRING_ESCAPE_RAW

    if escape == STRING_ESCAPE_RAW:
        return byte_data
    elif escape == STRING_ESCAPE_PRINT:
        return escape_ascii_bytes(byte_data)
    elif escape == STRING_ESCAPE_UTF8:
        return codecs.encode(escape_utf8(byte_data), 'utf-8')
    elif escape == STRING_ESCAPE_BASE64:
        return base64.b64encode(byte_data)
    else:
        raise UnicodeEncodeError("Unknown escape option")

Response: Apply the specified escape method on the given byte - like object with bytes.


Instruction:
Create a Python 3 function to
decode module id to string based on @antirez moduleTypeNameByID function from redis / src / module. c

Response:
def _decode_module_id(self, module_id):
        """
        decode module id to string
        based on @antirez moduleTypeNameByID function from redis/src/module.c
        :param module_id: 64bit integer
        :return: string
        """
        name = [''] * 9
        module_id >>= 10
        for i in reversed(range(9)):
            name[i] = self.charset[module_id & 63]
            module_id >>= 6
        return ''.join(name)


Instruction:
How would you code a function in Python 3 to
return the step value in format suitable for display.

Response:
def format_step(step, zero_prefix=False):
    """Return the step value in format suitable for display."""
    if isinstance(step, int):
        return "{:06}".format(step) if zero_prefix else "{}".format(step)
    elif isinstance(step, tuple):
        return "{:04}:{:06}".format(*step) if zero_prefix else "{}:{}".format(*step)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def log(self, step, **kwargs):
        assert isinstance(step, (int, tuple)), "Step must be an int or a tuple of two ints"
        self.step = step
        # Any new metrics we haven't seen before?
        self.metrics |= set(kwargs.keys())
        # Insert (or update) record of the step
        if step not in self.history:
            self.history[step] = {}
        self.history[step].update({k:utils.to_data(v) for k, v in kwargs.items()})
        # Update step timestamp
        self.history[step]["__timestamp__"] = time.time()

Response: Record the metrics at a specific step.


Instruction:
How would you code a function in Python 3 to
return the total time between when the first and last steps where logged. This usually correspondspnods to the total training time where the training time is logged.

Response:
def get_total_time(self):
        """Returns the total period between when the first and last steps
        where logged. This usually correspnods to the total training time
        if there were no gaps in the training.
        """
        first_step = self.steps[0]
        last_step = self.steps[-1]
        seconds = self.history[last_step]["__timestamp__"] \
                  - self.history[first_step]["__timestamp__"]
        return datetime.timedelta(seconds=seconds)


Instruction:
Here you have a function in Python 3, explain what it does
def outgoing(self, node):
        nodes = node if isinstance(node, list) else [node]
        node_ids = [self.id(n) for n in nodes]
        # Find edges outgoing from this group but not incoming to it
        outgoing = [self[e[1]] for e in self.edges
                    if e[0] in node_ids and e[1] not in node_ids]
        return outgoing

Response: Returns nodes connecting out of the given node or list of nodes.


Instruction:
Can you tell what is the following Python 3 function doing
def siblings(self, node):
        incoming = self.incoming(node)
        # TODO: Not handling the case of multiple incoming nodes yet
        if len(incoming) == 1:
            incoming = incoming[0]
            siblings = self.outgoing(incoming)
            return siblings
        else:
            return [node]

Response:
Returns all nodes that share the same parent with
        the given node including the itself.


Instruction:
Make a summary of the following Python 3 code
def remove(self, nodes):
        nodes = nodes if isinstance(nodes, list) else [nodes]
        for node in nodes:
            k = self.id(node)
            self.edges = list(filter(lambda e: e[0] != k and e[1] != k, self.edges))
            del self.nodes[k]

Response: Remove a node and its edges.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def search(self, pattern):
        for node in self.nodes.values():
            match, following = pattern.match(self, node)
            if match:
                return match, following
        return [], None

Response:
Searches the graph for a sub - graph that matches the given pattern
       . Returns a list of the first match it finds.


Instruction:
Create a Python 3 function to
make up an ID for a sequence of nodes.

Response:
def sequence_id(self, sequence):
        """Make up an ID for a sequence (list) of nodes.
        Note: `getrandbits()` is very uninformative as a "readable" ID. Here, we build a name
        such that when the mouse hovers over the drawn node in Jupyter, one can figure out
        which original nodes make up the sequence. This is actually quite useful.
        """
        if self.meaningful_ids:
            # TODO: This might fail if the ID becomes too long
            return "><".join([node.id for node in sequence])
        else:
            return getrandbits(64)


Instruction:
Explain what the following Python 3 code does
def build_dot(self):
        from graphviz import Digraph

        # Build GraphViz Digraph
        dot = Digraph()
        dot.attr("graph",
                 bgcolor=self.theme["background_color"],
                 color=self.theme["outline_color"],
                 fontsize=self.theme["font_size"],
                 fontcolor=self.theme["font_color"],
                 fontname=self.theme["font_name"],
                 margin=self.theme["margin"],
                 rankdir="LR",
                 pad=self.theme["padding"])
        dot.attr("node", shape="box",
                 style="filled", margin="0,0",
                 fillcolor=self.theme["fill_color"],
                 color=self.theme["outline_color"],
                 fontsize=self.theme["font_size"],
                 fontcolor=self.theme["font_color"],
                 fontname=self.theme["font_name"])
        dot.attr("edge", style="solid",
                 color=self.theme["outline_color"],
                 fontsize=self.theme["font_size"],
                 fontcolor=self.theme["font_color"],
                 fontname=self.theme["font_name"])

        for k, n in self.nodes.items():
            label = "
{}
".format(n.title)
            if n.caption:
                label += "
{}
".format(n.caption)
            if n.repeat > 1:
                label += "
x{}
".format(n.repeat)
            label = "<
" + label + "
>"
            dot.node(str(k), label)
        for a, b, label in self.edges:
            if isinstance(label, (list, tuple)):
                label = "x".join([str(l or "?") for l in label])

            dot.edge(str(a), str(b), label)
        return dot

Response: Generate a GraphViz Dot object.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def to_data(value):
    # TODO: Use get_framework() for better detection.
    if value.__class__.__module__.startswith("torch"):
        import torch
        if isinstance(value, torch.nn.parameter.Parameter):
            value = value.data
        if isinstance(value, torch.Tensor):
            if value.requires_grad:
                value = value.detach()
            value = value.cpu().numpy().copy()
        # If 0-dim array, convert to scalar
        if not value.shape:
            value = value.item()
    # Convert Numpy scalar types to Python types
    if value.__class__.__module__ == "numpy" and value.__class__.__name__ != "ndarray":
        value = value.item()
    return value

Response: Standardize data types. Converts PyTorch tensors to Numpy arrays and Numpy scalars to Python scalars.


Instruction:
Create a Python 3 function for
liking print but recognizes tensors and arrays and show more details about them.

Response:
def write(*args):
    """Like print(), but recognizes tensors and arrays and show
    more details about them.

    Example:
        hl.write("My Tensor", my_tensor)

        Prints:
            My Tensor  float32 (10, 3, 224, 224)  min: 0.0  max: 1.0
    """
    s = ""
    for a in args:
        # Convert tensors to Numpy arrays
        a = to_data(a)

        if isinstance(a, np.ndarray):
            # Numpy Array
            s += ("\t" if s else "") + "Tensor  {} {}  min: {:.3f}  max: {:.3f}".format(
                a.dtype, a.shape, a.min(), a.max())
            print(s)
            s = ""
        elif isinstance(a, list):
            s += ("\t" if s else "") + "list    len: {}  {}".format(len(a), a[:10])
        else:
            s += (" " if s else "") + str(a)
    if s:
        print(s)


Instruction:
Can you create a Python 3 function that
normalizes an image to [ 0 1 ) range.

Response:
def norm(image):
    """Normalize an image to [0, 1] range."""
    min_value = image.min()
    max_value = image.max()
    if min_value == max_value:
        return image - min_value
    return (image - min_value) / (max_value - min_value)


Instruction:
Can you generate the documentation for the following Python 3 function
def show_images(images, titles=None, cols=5, **kwargs):
    # The images param can be a list or an array

    titles = titles or [""] * len(images)
    rows = math.ceil(len(images) / cols)
    height_ratio = 1.2 * (rows/cols) * (0.5 if type(images[0]) is not np.ndarray else 1)
    plt.figure(figsize=(11, 11 * height_ratio))
    i = 1
    for image, title in zip(images, titles):
        plt.subplot(rows, cols, i)
        plt.axis("off")
        # Is image a list? If so, merge them into one image.
        if type(image) is not np.ndarray:
            image = [norm(g) for g in image]
            image = np.concatenate(image, axis=1)
        else:
            image = norm(image)
        plt.title(title, fontsize=9)
        plt.imshow(image, cmap="Greys_r", **kwargs)
        i += 1
    plt.tight_layout(h_pad=0, w_pad=0)

Response: Show the images in a single order tree structure.


Instruction:
Here you have a function in Python 3, explain what it does
def draw_summary(self, history, title=""):
        # Generate summary string
        time_str = str(history.get_total_time()).split(".")[0]  # remove microseconds
        summary = "Step: {}      Time: {}".format(history.step, time_str)
        if title:
            summary = title + "\n\n" + summary
        self.figure.suptitle(summary)

Response:
Inserts a text summary at the top that lists the number of steps and total
        training time.


Instruction:
Can you tell what is the following Python 3 function doing
def draw_plot(self, metrics, labels=None, ylabel=""):
        metrics = metrics if isinstance(metrics, list) else [metrics]
        # Loop through metrics
        title = ""
        for i, m in enumerate(metrics):
            label = labels[i] if labels else m.name
            # TODO: use a standard formating function for values
            title += ("   " if title else "") + "{}: {}".format(label, m.data[-1])
            self.ax.plot(m.formatted_steps, m.data, label=label)
        self.ax.set_title(title)
        self.ax.set_ylabel(ylabel)
        self.ax.legend()
        self.ax.set_xlabel("Steps")
        self.ax.xaxis.set_major_locator(plt.AutoLocator())

Response: Draws the history of the entries in the given metrics.


Instruction:
Create a Python 3 function to
display a series of images at different time steps.

Response:
def draw_image(self, metric, limit=5):
        """Display a series of images at different time steps."""
        rows = 1
        cols = limit
        self.ax.axis("off")
        # Take the Axes gridspec and divide it into a grid
        gs = matplotlib.gridspec.GridSpecFromSubplotSpec(
            rows, cols, subplot_spec=self.gs)
        # Loop through images in last few steps
        for i, image in enumerate(metric.data[-cols:]):
            ax = self.figure.add_subplot(gs[0, i])
            ax.axis('off')
            ax.set_title(metric.formatted_steps[-cols:][i])
            ax.imshow(norm(image))


Instruction:
Can you tell what is the following Python 3 function doing
def draw_hist(self, metric, title=""):
        # TODO: assert isinstance(list(values.values())[0], np.ndarray)

        rows = 1
        cols = 1
        limit = 10  # max steps to show

        # We need a 3D projection Subplot, so ignore the one provided to
        # as an create a new one.
        ax = self.figure.add_subplot(self.gs, projection="3d")
        ax.view_init(30, -80)

        # Compute histograms
        verts = []
        area_colors = []
        edge_colors = []
        for i, s in enumerate(metric.steps[-limit:]):
            hist, edges = np.histogram(metric.data[-i-1:])
            # X is bin centers
            x = np.diff(edges)/2 + edges[:-1]
            # Y is hist values
            y = hist
            x = np.concatenate([x[0:1], x, x[-1:]])
            y = np.concatenate([[0], y, [0]])

            # Ranges
            if i == 0:
                x_min = x.min()
                x_max = x.max()
                y_min = y.min()
                y_max = y.max()
            x_min = np.minimum(x_min, x.min())
            x_max = np.maximum(x_max, x.max())
            y_min = np.minimum(y_min, y.min())
            y_max = np.maximum(y_max, y.max())

            alpha = 0.8 * (i+1) / min(limit, len(metric.steps))
            verts.append(list(zip(x, y)))
            area_colors.append(np.array(self.theme["hist_color"] + [alpha]))
            edge_colors.append(np.array(self.theme["hist_outline_color"] + [alpha]))

        poly = PolyCollection(verts, facecolors=area_colors, edgecolors=edge_colors)
        ax.add_collection3d(poly, zs=list(range(min(limit, len(metric.steps)))), zdir='y')

        ax.set_xlim(x_min, x_max)
        ax.set_ylim(0, limit)
        ax.set_yticklabels(metric.formatted_steps[-limit:])
        ax.set_zlim(y_min, y_max)
        ax.set_title(metric.name)

Response: Draw a series of histograms over the selected keys over different training steps.


Instruction:
Make a summary of the following Python 3 code
def _load(self, dataset='train'):
        data, labels = None, None
        if dataset is 'train':
            files = [os.path.join(self.cifar10_dir, 'data_batch_%d' % i) for i in range(1, 6)]
        else:
            files = [os.path.join(self.cifar10_dir, 'test_batch')]

        for file in files:
            if not os.path.exists(file):
                raise FileNotFoundError('Failed to find file: ' + file)

        # Load the data from the batch files
        for file in files:
            with open(file, 'rb') as f:
                cifar10 = pickle.load(f, encoding='latin1')

            if labels is None:
                labels = np.array(cifar10['labels'])
            else:
                labels = np.concatenate((labels, cifar10['labels']), axis=0)

            if data is None:
                data = cifar10['data']
            else:
                data = np.concatenate((data, cifar10['data']), axis=0)

        # Adapt the format of the data to our convnet
        data = np.array(data, dtype=float) / 255.0
        data = data.reshape([-1, self.num_channels, self.img_size, self.img_size])
        data = data.transpose([0, 2, 3, 1])

        # One-hot encode labels (see https://stackoverflow.com/a/42874726)
        labels = np.eye(self.num_classes)[np.array(labels).reshape(-1)]

        if dataset is 'train':
            self._train_data, self._train_labels = data, labels
        else:
            self._test_data, self._test_labels = data, labels

Response: Loads the data in memory.


Instruction:
Write a Python 3 function that can
build a simple convnet with the last op containing the predictions.

Response:
def model(self, inputs, mode='train'):
        """Build a simple convnet (BN before ReLU).
        Args:
            inputs: a tensor of size [batch_size, height, width, channels]
            mode: string in ['train', 'test']
        Returns:
            the last op containing the predictions
        Note:
            Best score
            Step:  7015 - Epoch: 18/20 - best batch acc: 0.8984 - loss: 1.5656
            Worst score
            Step:  7523 - Epoch: 20/20 - best batch acc: 0.7734 - loss: 1.6874
        """
        # Extract features
        training = (mode == 'train')
        with tf.variable_scope('conv1') as scope:
            conv = tf.layers.conv2d(inputs=inputs, filters=16, kernel_size=[3, 3], padding='SAME')
            bn = tf.layers.batch_normalization(inputs=conv, training=training)
            bn = tf.nn.relu(bn)
            conv = tf.layers.conv2d(inputs=bn, filters=16, kernel_size=[3, 3], padding='SAME')
            bn = tf.layers.batch_normalization(inputs=conv, training=training)
            bn = tf.nn.relu(bn)
            pool = tf.layers.max_pooling2d(bn, pool_size=[2, 2], strides=2, padding='SAME', name=scope.name)

        with tf.variable_scope('conv2') as scope:
            conv = tf.layers.conv2d(inputs=pool, filters=32, kernel_size=[3, 3], padding='SAME')
            bn = tf.layers.batch_normalization(inputs=conv, training=training)
            bn = tf.nn.relu(bn)
            conv = tf.layers.conv2d(inputs=bn, filters=32, kernel_size=[3, 3], padding='SAME')
            bn = tf.layers.batch_normalization(inputs=conv, training=training)
            bn = tf.nn.relu(bn)
            pool = tf.layers.max_pooling2d(bn, pool_size=[2, 2], strides=2, padding='SAME', name=scope.name)

        with tf.variable_scope('conv3') as scope:
            conv = tf.layers.conv2d(inputs=pool, filters=32, kernel_size=[3, 3], padding='SAME')
            bn = tf.layers.batch_normalization(inputs=conv, training=training)
            bn = tf.nn.relu(bn)
            conv = tf.layers.conv2d(inputs=bn, filters=32, kernel_size=[3, 3], padding='SAME')
            bn = tf.layers.batch_normalization(inputs=conv, training=training)
            bn = tf.nn.relu(bn)
            pool = tf.layers.max_pooling2d(bn, pool_size=[2, 2], strides=2, padding='SAME', name=scope.name)

        # Classify
        with tf.variable_scope('fc') as scope:
            flat = tf.layers.flatten(pool)
            fc = tf.layers.dense(inputs=flat, units=32, activation=tf.nn.relu)
            softmax = tf.layers.dense(inputs=fc, units=self.num_classes, activation=tf.nn.softmax)

        return softmax


Instruction:
Can you generate a brief explanation for the following Python 3 code
def maybe_download_and_extract(self):
        if not os.path.exists(self.cifar10_dir):

            if not os.path.exists(self.data_dir):
                os.makedirs(self.data_dir)

            def _progress(count, block_size, total_size):
                status_msg = '\r>> Downloading {} {:>3}%   '
                sys.stdout.write(status_msg.format(self.cifar10_tarball, float(count * block_size) / total_size * 100.0))
                sys.stdout.flush()

            file_path, _ = urlretrieve(CIFAR10_URL, self.cifar10_tarball, _progress)

            stat_info = os.stat(file_path)
            print('\nSuccessfully downloaded', file_path, stat_info.st_size, 'bytes.\n')

            tarfile.open(file_path, 'r:gz').extractall(self.data_dir)

Response: Download and extract the tarball from Alex Krizhevsky s website.


Instruction:
Given the following Python 3 function, write the documentation
def dump_pytorch_graph(graph):
    f = "{:25} {:40}   {} -> {}"
    print(f.format("kind", "scopeName", "inputs", "outputs"))
    for node in graph.nodes():
        print(f.format(node.kind(), node.scopeName(),
                       [i.unique() for i in node.inputs()],
                       [i.unique() for i in node.outputs()]
                       ))

Response: List all the nodes in a PyTorch graph.


Instruction:
Here you have a function in Python 3, explain what it does
def pytorch_id(node):
    # After ONNX simplification, the scopeName is not unique anymore
    # so append node outputs to guarantee uniqueness
    return node.scopeName() + "/outputs/" + "/".join([o.uniqueName() for o in node.outputs()])

Response: Returns a unique ID for a node.


Instruction:
Write a Python 3 function for
returning the output shape of the given Pytorch node.

Response:
def get_shape(torch_node):
    """Return the output shape of the given Pytorch node."""
    # Extract node output shape from the node string representation
    # This is a hack because there doesn't seem to be an official way to do it.
    # See my quesiton in the PyTorch forum:
    # https://discuss.pytorch.org/t/node-output-shape-from-trace-graph/24351/2
    # TODO: find a better way to extract output shape
    # TODO: Assuming the node has one output. Update if we encounter a multi-output node.
    m = re.match(r".*Float\(([\d\s\,]+)\).*", str(next(torch_node.outputs())))
    if m:
        shape = m.group(1)
        shape = shape.split(",")
        shape = tuple(map(int, shape))
    else:
        shape = None
    return shape


Instruction:
Here you have a function in Python 3, explain what it does
def dump_tf_graph(tfgraph, tfgraphdef):
    print("Nodes ({})".format(len(tfgraphdef.node)))
    f = "{:15} {:59} {:20} {}"
    print(f.format("kind", "scopeName", "shape", "inputs"))
    for node in tfgraphdef.node:
        scopename = node.name
        kind = node.op
        inputs = node.input
        shape = tf.graph_util.tensor_shape_from_node_def_name(tfgraph, scopename)
        print(f.format(kind, scopename, str(shape), inputs))

Response: Dump a TF graph to stdout.


Instruction:
Can you implement a function in Python 3 that
converts a TF graph to a directed graph.

Response:
def import_graph(hl_graph, tf_graph, output=None, verbose=False):
    """Convert TF graph to directed graph
    tfgraph: A TF Graph object.
    output: Name of the output node (string).
    verbose: Set to True for debug print output
    """
    # Get clean(er) list of nodes
    graph_def = tf_graph.as_graph_def(add_shapes=True)
    graph_def = tf.graph_util.remove_training_nodes(graph_def)

    # Dump list of TF nodes (DEBUG only)
    if verbose:
        dump_tf_graph(tf_graph, graph_def)

    # Loop through nodes and build the matching directed graph
    for tf_node in graph_def.node:
        # Read node details
        try:
            op,  uid, name, shape, params = import_node(tf_node, tf_graph, verbose)
        except:
            if verbose:
                logging.exception("Failed to read node {}".format(tf_node))
            continue

        # Add node
        hl_node = Node(uid=uid, name=name, op=op, output_shape=shape, params=params)
        hl_graph.add_node(hl_node)

        # Add edges
        for target_node in graph_def.node:
            target_inputs = target_node.input
            if uid in target_node.input:
                hl_graph.add_edge_by_id(uid, target_node.name, shape)
    return hl_graph


Instruction:
Create a Python 3 function for
sending FetchRequests for all assigned partitions that do not already have an in - flight fetch or pending fetch data. Returns a list of Futures that resolves to a FetchResponse

Response:
def send_fetches(self):
        """Send FetchRequests for all assigned partitions that do not already have
        an in-flight fetch or pending fetch data.

        Returns:
            List of Futures: each future resolves to a FetchResponse
        """
        futures = []
        for node_id, request in six.iteritems(self._create_fetch_requests()):
            if self._client.ready(node_id):
                log.debug("Sending FetchRequest to node %s", node_id)
                future = self._client.send(node_id, request)
                future.add_callback(self._handle_fetch_response, request, time.time())
                future.add_errback(log.error, 'Fetch to node %s failed: %s', node_id)
                futures.append(future)
        self._fetch_futures.extend(futures)
        self._clean_done_fetch_futures()
        return futures


Instruction:
Can you implement a function in Python 3 that
resets offsets for any partitions which are awaiting an explicit reset.

Response:
def reset_offsets_if_needed(self, partitions):
        """Lookup and set offsets for any partitions which are awaiting an
        explicit reset.

        Arguments:
            partitions (set of TopicPartitions): the partitions to reset
        """
        for tp in partitions:
            # TODO: If there are several offsets to reset, we could submit offset requests in parallel
            if self._subscriptions.is_assigned(tp) and self._subscriptions.is_offset_reset_needed(tp):
                self._reset_offset(tp)


Instruction:
How would you implement a function in Python 3 that
updates the fetch positions for the provided partitions.

Response:
def update_fetch_positions(self, partitions):
        """Update the fetch positions for the provided partitions.

        Arguments:
            partitions (list of TopicPartitions): partitions to update

        Raises:
            NoOffsetForPartitionError: if no offset is stored for a given
                partition and no reset policy is available
        """
        # reset the fetch position to the committed position
        for tp in partitions:
            if not self._subscriptions.is_assigned(tp):
                log.warning("partition %s is not assigned - skipping offset"
                            " update", tp)
                continue
            elif self._subscriptions.is_fetchable(tp):
                log.warning("partition %s is still fetchable -- skipping offset"
                            " update", tp)
                continue

            if self._subscriptions.is_offset_reset_needed(tp):
                self._reset_offset(tp)
            elif self._subscriptions.assignment[tp].committed is None:
                # there's no committed position, so we need to reset with the
                # default strategy
                self._subscriptions.need_offset_reset(tp)
                self._reset_offset(tp)
            else:
                committed = self._subscriptions.assignment[tp].committed
                log.debug("Resetting offset for partition %s to the committed"
                          " offset %s", tp, committed)
                self._subscriptions.seek(tp, committed)


Instruction:
How would you code a function in Python 3 to
reset the offset for the given topic partition.

Response:
def _reset_offset(self, partition):
        """Reset offsets for the given partition using the offset reset strategy.

        Arguments:
            partition (TopicPartition): the partition that needs reset offset

        Raises:
            NoOffsetForPartitionError: if no offset reset strategy is defined
        """
        timestamp = self._subscriptions.assignment[partition].reset_strategy
        if timestamp is OffsetResetStrategy.EARLIEST:
            strategy = 'earliest'
        elif timestamp is OffsetResetStrategy.LATEST:
            strategy = 'latest'
        else:
            raise NoOffsetForPartitionError(partition)

        log.debug("Resetting offset for partition %s to %s offset.",
                  partition, strategy)
        offsets = self._retrieve_offsets({partition: timestamp})
        if partition not in offsets:
            raise NoOffsetForPartitionError(partition)
        offset = offsets[partition][0]

        # we might lose the assignment while fetching the offset,
        # so check it is still active
        if self._subscriptions.is_assigned(partition):
            self._subscriptions.seek(partition, offset)


Instruction:
Create a Python 3 function for
fetching offsets for each topic partition passed in timestamps.

Response:
def _retrieve_offsets(self, timestamps, timeout_ms=float("inf")):
        """Fetch offset for each partition passed in ``timestamps`` map.

        Blocks until offsets are obtained, a non-retriable exception is raised
        or ``timeout_ms`` passed.

        Arguments:
            timestamps: {TopicPartition: int} dict with timestamps to fetch
                offsets by. -1 for the latest available, -2 for the earliest
                available. Otherwise timestamp is treated as epoch miliseconds.

        Returns:
            {TopicPartition: (int, int)}: Mapping of partition to
                retrieved offset and timestamp. If offset does not exist for
                the provided timestamp, that partition will be missing from
                this mapping.
        """
        if not timestamps:
            return {}

        start_time = time.time()
        remaining_ms = timeout_ms
        while remaining_ms > 0:
            future = self._send_offset_requests(timestamps)
            self._client.poll(future=future, timeout_ms=remaining_ms)

            if future.succeeded():
                return future.value
            if not future.retriable():
                raise future.exception  # pylint: disable-msg=raising-bad-type

            elapsed_ms = (time.time() - start_time) * 1000
            remaining_ms = timeout_ms - elapsed_ms
            if remaining_ms < 0:
                break

            if future.exception.invalid_metadata:
                refresh_future = self._client.cluster.request_update()
                self._client.poll(future=refresh_future, timeout_ms=remaining_ms)
            else:
                time.sleep(self.config['retry_backoff_ms'] / 1000.0)

            elapsed_ms = (time.time() - start_time) * 1000
            remaining_ms = timeout_ms - elapsed_ms

        raise Errors.KafkaTimeoutError(
            "Failed to get offsets by timestamps in %s ms" % (timeout_ms,))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def fetched_records(self, max_records=None):
        if max_records is None:
            max_records = self.config['max_poll_records']
        assert max_records > 0

        drained = collections.defaultdict(list)
        records_remaining = max_records

        while records_remaining > 0:
            if not self._next_partition_records:
                if not self._completed_fetches:
                    break
                completion = self._completed_fetches.popleft()
                self._next_partition_records = self._parse_fetched_data(completion)
            else:
                records_remaining -= self._append(drained,
                                                  self._next_partition_records,
                                                  records_remaining)
        return dict(drained), bool(self._completed_fetches)

Response: Returns previously fetched records and updates consumed offsets.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _message_generator(self):
        while self._next_partition_records or self._completed_fetches:

            if not self._next_partition_records:
                completion = self._completed_fetches.popleft()
                self._next_partition_records = self._parse_fetched_data(completion)
                continue

            # Send additional FetchRequests when the internal queue is low
            # this should enable moderate pipelining
            if len(self._completed_fetches) <= self.config['iterator_refetch_records']:
                self.send_fetches()

            tp = self._next_partition_records.topic_partition

            # We can ignore any prior signal to drop pending message sets
            # because we are starting from a fresh one where fetch_offset == position
            # i.e., the user seek()'d to this position
            self._subscriptions.assignment[tp].drop_pending_message_set = False

            for msg in self._next_partition_records.take():

                # Because we are in a generator, it is possible for
                # subscription state to change between yield calls
                # so we need to re-check on each loop
                # this should catch assignment changes, pauses
                # and resets via seek_to_beginning / seek_to_end
                if not self._subscriptions.is_fetchable(tp):
                    log.debug("Not returning fetched records for partition %s"
                              " since it is no longer fetchable", tp)
                    self._next_partition_records = None
                    break

                # If there is a seek during message iteration,
                # we should stop unpacking this message set and
                # wait for a new fetch response that aligns with the
                # new seek position
                elif self._subscriptions.assignment[tp].drop_pending_message_set:
                    log.debug("Skipping remainder of message set for partition %s", tp)
                    self._subscriptions.assignment[tp].drop_pending_message_set = False
                    self._next_partition_records = None
                    break

                # Compressed messagesets may include earlier messages
                elif msg.offset < self._subscriptions.assignment[tp].position:
                    log.debug("Skipping message offset: %s (expecting %s)",
                              msg.offset,
                              self._subscriptions.assignment[tp].position)
                    continue

                self._subscriptions.assignment[tp].position = msg.offset + 1
                yield msg

            self._next_partition_records = None

Response: Iterate over fetched_records and yield messages from the broker.


Instruction:
Can you generate the documentation for the following Python 3 function
def _send_offset_requests(self, timestamps):
        timestamps_by_node = collections.defaultdict(dict)
        for partition, timestamp in six.iteritems(timestamps):
            node_id = self._client.cluster.leader_for_partition(partition)
            if node_id is None:
                self._client.add_topic(partition.topic)
                log.debug("Partition %s is unknown for fetching offset,"
                          " wait for metadata refresh", partition)
                return Future().failure(Errors.StaleMetadata(partition))
            elif node_id == -1:
                log.debug("Leader for partition %s unavailable for fetching "
                          "offset, wait for metadata refresh", partition)
                return Future().failure(
                    Errors.LeaderNotAvailableError(partition))
            else:
                timestamps_by_node[node_id][partition] = timestamp

        # Aggregate results until we have all
        list_offsets_future = Future()
        responses = []
        node_count = len(timestamps_by_node)

        def on_success(value):
            responses.append(value)
            if len(responses) == node_count:
                offsets = {}
                for r in responses:
                    offsets.update(r)
                list_offsets_future.success(offsets)

        def on_fail(err):
            if not list_offsets_future.is_done:
                list_offsets_future.failure(err)

        for node_id, timestamps in six.iteritems(timestamps_by_node):
            _f = self._send_offset_request(node_id, timestamps)
            _f.add_callback(on_success)
            _f.add_errback(on_fail)
        return list_offsets_future

Response: Send offsets for each partition in timestamps dict to each node.


Instruction:
Can you write a function in Python 3 where it
creates fetch requests for all assigned partitions grouped by node.

Response:
def _create_fetch_requests(self):
        """Create fetch requests for all assigned partitions, grouped by node.

        FetchRequests skipped if no leader, or node has requests in flight

        Returns:
            dict: {node_id: FetchRequest, ...} (version depends on api_version)
        """
        # create the fetch info as a dict of lists of partition info tuples
        # which can be passed to FetchRequest() via .items()
        fetchable = collections.defaultdict(lambda: collections.defaultdict(list))

        for partition in self._fetchable_partitions():
            node_id = self._client.cluster.leader_for_partition(partition)

            # advance position for any deleted compacted messages if required
            if self._subscriptions.assignment[partition].last_offset_from_message_batch:
                next_offset_from_batch_header = self._subscriptions.assignment[partition].last_offset_from_message_batch + 1
                if next_offset_from_batch_header > self._subscriptions.assignment[partition].position:
                    log.debug(
                        "Advance position for partition %s from %s to %s (last message batch location plus one)"
                        " to correct for deleted compacted messages",
                        partition, self._subscriptions.assignment[partition].position, next_offset_from_batch_header)
                    self._subscriptions.assignment[partition].position = next_offset_from_batch_header

            position = self._subscriptions.assignment[partition].position

            # fetch if there is a leader and no in-flight requests
            if node_id is None or node_id == -1:
                log.debug("No leader found for partition %s."
                          " Requesting metadata update", partition)
                self._client.cluster.request_update()

            elif self._client.in_flight_request_count(node_id) == 0:
                partition_info = (
                    partition.partition,
                    position,
                    self.config['max_partition_fetch_bytes']
                )
                fetchable[node_id][partition.topic].append(partition_info)
                log.debug("Adding fetch request for partition %s at offset %d",
                          partition, position)
            else:
                log.log(0, "Skipping fetch for partition %s because there is an inflight request to node %s",
                        partition, node_id)

        if self.config['api_version'] >= (0, 11, 0):
            version = 4
        elif self.config['api_version'] >= (0, 10, 1):
            version = 3
        elif self.config['api_version'] >= (0, 10):
            version = 2
        elif self.config['api_version'] == (0, 9):
            version = 1
        else:
            version = 0
        requests = {}
        for node_id, partition_data in six.iteritems(fetchable):
            if version < 3:
                requests[node_id] = FetchRequest[version](
                    -1,  # replica_id
                    self.config['fetch_max_wait_ms'],
                    self.config['fetch_min_bytes'],
                    partition_data.items())
            else:
                # As of version == 3 partitions will be returned in order as
                # they are requested, so to avoid starvation with
                # `fetch_max_bytes` option we need this shuffle
                # NOTE: we do have partition_data in random order due to usage
                #       of unordered structures like dicts, but that does not
                #       guarantee equal distribution, and starting in Python3.6
                #       dicts retain insert order.
                partition_data = list(partition_data.items())
                random.shuffle(partition_data)
                if version == 3:
                    requests[node_id] = FetchRequest[version](
                        -1,  # replica_id
                        self.config['fetch_max_wait_ms'],
                        self.config['fetch_min_bytes'],
                        self.config['fetch_max_bytes'],
                        partition_data)
                else:
                    requests[node_id] = FetchRequest[version](
                        -1,  # replica_id
                        self.config['fetch_max_wait_ms'],
                        self.config['fetch_min_bytes'],
                        self.config['fetch_max_bytes'],
                        self._isolation_level,
                        partition_data)
        return requests


Instruction:
How would you explain what the following Python 3 function does
def _handle_fetch_response(self, request, send_time, response):
        fetch_offsets = {}
        for topic, partitions in request.topics:
            for partition_data in partitions:
                partition, offset = partition_data[:2]
                fetch_offsets[TopicPartition(topic, partition)] = offset

        partitions = set([TopicPartition(topic, partition_data[0])
                          for topic, partitions in response.topics
                          for partition_data in partitions])
        metric_aggregator = FetchResponseMetricAggregator(self._sensors, partitions)

        # randomized ordering should improve balance for short-lived consumers
        random.shuffle(response.topics)
        for topic, partitions in response.topics:
            random.shuffle(partitions)
            for partition_data in partitions:
                tp = TopicPartition(topic, partition_data[0])
                completed_fetch = CompletedFetch(
                    tp, fetch_offsets[tp],
                    response.API_VERSION,
                    partition_data[1:],
                    metric_aggregator
                )
                self._completed_fetches.append(completed_fetch)

        if response.API_VERSION >= 1:
            self._sensors.fetch_throttle_time_sensor.record(response.throttle_time_ms)
        self._sensors.fetch_latency.record((time.time() - send_time) * 1000)

Response: The callback for fetching a new topic from the server.


Instruction:
Given the following Python 3 function, write the documentation
def record(self, partition, num_bytes, num_records):
        self.unrecorded_partitions.remove(partition)
        self.total_bytes += num_bytes
        self.total_records += num_records

        # once all expected partitions from the fetch have reported in, record the metrics
        if not self.unrecorded_partitions:
            self.sensors.bytes_fetched.record(self.total_bytes)
            self.sensors.records_fetched.record(self.total_records)

Response: Record the current metrics for the given partition.


Instruction:
Can you generate the documentation for the following Python 3 function
def _auto_commit(self):

        # Check if we are supposed to do an auto-commit
        if not self.auto_commit or self.auto_commit_every_n is None:
            return

        if self.count_since_commit >= self.auto_commit_every_n:
            self.commit()

Response: Check if we have to commit based on number of messages and commit


Instruction:
Can you generate a brief explanation for the following Python 3 code
def pending(self, partitions=None):
        if partitions is None:
            partitions = self.offsets.keys()

        total = 0
        reqs = []

        for partition in partitions:
            reqs.append(OffsetRequestPayload(self.topic, partition, -1, 1))

        resps = self.client.send_offset_request(reqs)
        for resp in resps:
            partition = resp.partition
            pending = resp.offsets[0]
            offset = self.offsets[partition]
            total += pending - offset

        return total

Response: Gets the number of pending messages for the specified partitions


Instruction:
Can you generate a brief explanation for the following Python 3 code
def murmur2(data):
    # Python2 bytes is really a str, causing the bitwise operations below to fail
    # so convert to bytearray.
    if six.PY2:
        data = bytearray(bytes(data))

    length = len(data)
    seed = 0x9747b28c
    # 'm' and 'r' are mixing constants generated offline.
    # They're not really 'magic', they just happen to work well.
    m = 0x5bd1e995
    r = 24

    # Initialize the hash to a random value
    h = seed ^ length
    length4 = length // 4

    for i in range(length4):
        i4 = i * 4
        k = ((data[i4 + 0] & 0xff) +
            ((data[i4 + 1] & 0xff) << 8) +
            ((data[i4 + 2] & 0xff) << 16) +
            ((data[i4 + 3] & 0xff) << 24))
        k &= 0xffffffff
        k *= m
        k &= 0xffffffff
        k ^= (k % 0x100000000) >> r # k ^= k >>> r
        k &= 0xffffffff
        k *= m
        k &= 0xffffffff

        h *= m
        h &= 0xffffffff
        h ^= k
        h &= 0xffffffff

    # Handle the last few bytes of the input array
    extra_bytes = length % 4
    if extra_bytes >= 3:
        h ^= (data[(length & ~3) + 2] & 0xff) << 16
        h &= 0xffffffff
    if extra_bytes >= 2:
        h ^= (data[(length & ~3) + 1] & 0xff) << 8
        h &= 0xffffffff
    if extra_bytes >= 1:
        h ^= (data[length & ~3] & 0xff)
        h &= 0xffffffff
        h *= m
        h &= 0xffffffff

    h ^= (h % 0x100000000) >> 13 # h >>> 13;
    h &= 0xffffffff
    h *= m
    h &= 0xffffffff
    h ^= (h % 0x100000000) >> 15 # h >>> 15;
    h &= 0xffffffff

    return h

Response: Pure - python Murmur2 implementation.


Instruction:
Write a Python 3 script for
fetching the specified number of messages from the queue and return them as a list of Message objects.

Response:
def get_messages(self, count=1, block=True, timeout=10):
        """
        Fetch the specified number of messages

        Keyword Arguments:
            count: Indicates the maximum number of messages to be fetched
            block: If True, the API will block till all messages are fetched.
                If block is a positive integer the API will block until that
                many messages are fetched.
            timeout: When blocking is requested the function will block for
                the specified time (in seconds) until count messages is
                fetched. If None, it will block forever.
        """
        messages = []

        # Give a size hint to the consumers. Each consumer process will fetch
        # a maximum of "count" messages. This will fetch more messages than
        # necessary, but these will not be committed to kafka. Also, the extra
        # messages can be provided in subsequent runs
        self.size.value = count
        self.events.pause.clear()

        if timeout is not None:
            max_time = time.time() + timeout

        new_offsets = {}
        while count > 0 and (timeout is None or timeout > 0):
            # Trigger consumption only if the queue is empty
            # By doing this, we will ensure that consumers do not
            # go into overdrive and keep consuming thousands of
            # messages when the user might need only a few
            if self.queue.empty():
                self.events.start.set()

            block_next_call = block is True or block > len(messages)
            try:
                partition, message = self.queue.get(block_next_call,
                                                    timeout)
            except queue.Empty:
                break

            _msg = (partition, message) if self.partition_info else message
            messages.append(_msg)
            new_offsets[partition] = message.offset + 1
            count -= 1
            if timeout is not None:
                timeout = max_time - time.time()

        self.size.value = 0
        self.events.start.clear()
        self.events.pause.set()

        # Update and commit offsets if necessary
        self.offsets.update(new_offsets)
        self.count_since_commit += len(messages)
        self._auto_commit()

        return messages


Instruction:
Can you generate the documentation for the following Python 3 function
def mark(self, partition, offset):
        max_offset = max(offset + 1, self.high_water_mark.get(partition, 0))

        self.logger.debug("Setting high-water mark to: %s",
                          {partition: max_offset})

        self.high_water_mark[partition] = max_offset

Response: Set the high - water mark for the given partition.


Instruction:
Given the following Python 3 function, write the documentation
def commit(self):
        if self.high_water_mark:
            self.logger.info("Committing offsets: %s", self.high_water_mark)
            self.commit_partition_offsets(self.high_water_mark)
            self.update_consumer_offsets(self.high_water_mark)
        else:
            self.update_consumer_offsets(self.initial_offsets)

Response: Commit the offsets of this context.


Instruction:
Make a summary of the following Python 3 code
def rollback(self):
        self.logger.info("Rolling back context: %s", self.initial_offsets)
        self.update_consumer_offsets(self.initial_offsets)

Response: Roll back the consumer context.


Instruction:
Create a Python 3 function to
commit explicit partition and offset pairs.

Response:
def commit_partition_offsets(self, partition_offsets):
        """
        Commit explicit partition/offset pairs.
        """
        self.logger.debug("Committing partition offsets: %s", partition_offsets)

        commit_requests = [
            OffsetCommitRequestPayload(self.consumer.topic, partition, offset, None)
            for partition, offset in partition_offsets.items()
        ]
        commit_responses = self.consumer.client.send_offset_commit_request(
            self.consumer.group,
            commit_requests,
        )
        for commit_response in commit_responses:
            check_error(commit_response)


Instruction:
Can you generate the documentation for the following Python 3 function
def update_consumer_offsets(self, partition_offsets):
        self.logger.debug("Updating consumer offsets to: %s", partition_offsets)

        for partition, offset in partition_offsets.items():
            self.consumer.offsets[partition] = offset

        # consumer keeps other offset states beyond its `offsets` dictionary,
        # a relative seek with zero delta forces the consumer to reset to the
        # current value of the `offsets` dictionary
        self.consumer.seek(0, 1)

Response: Update consumer offsets to explicit positions.


Instruction:
Can you tell what is the following Python 3 function doing
def coordinator(self):
        if self.coordinator_id is None:
            return None
        elif self._client.is_disconnected(self.coordinator_id):
            self.coordinator_dead('Node Disconnected')
            return None
        else:
            return self.coordinator_id

Response: Get the current coordinator id


Instruction:
Implement a Python 3 function for
blocking until the coordinator is ready for this group.

Response:
def ensure_coordinator_ready(self):
        """Block until the coordinator for this group is known
        (and we have an active connection -- java client uses unsent queue).
        """
        with self._client._lock, self._lock:
            while self.coordinator_unknown():

                # Prior to 0.8.2 there was no group coordinator
                # so we will just pick a node at random and treat
                # it as the "coordinator"
                if self.config['api_version'] < (0, 8, 2):
                    self.coordinator_id = self._client.least_loaded_node()
                    if self.coordinator_id is not None:
                        self._client.maybe_connect(self.coordinator_id)
                    continue

                future = self.lookup_coordinator()
                self._client.poll(future=future)

                if future.failed():
                    if future.retriable():
                        if getattr(future.exception, 'invalid_metadata', False):
                            log.debug('Requesting metadata for group coordinator request: %s', future.exception)
                            metadata_update = self._client.cluster.request_update()
                            self._client.poll(future=metadata_update)
                        else:
                            time.sleep(self.config['retry_backoff_ms'] / 1000)
                    else:
                        raise future.exception


Instruction:
Can you create a Python 3 function that
polls the heartbeat thread for the given entry.

Response:
def poll_heartbeat(self):
        """
        Check the status of the heartbeat thread (if it is active) and indicate
        the liveness of the client. This must be called periodically after
        joining with :meth:`.ensure_active_group` to ensure that the member stays
        in the group. If an interval of time longer than the provided rebalance
        timeout (max_poll_interval_ms) expires without calling this method, then
        the client will proactively leave the group.

        Raises: RuntimeError for unexpected errors raised from the heartbeat thread
        """
        with self._lock:
            if self._heartbeat_thread is not None:
                if self._heartbeat_thread.failed:
                    # set the heartbeat thread to None and raise an exception.
                    # If the user catches it, the next call to ensure_active_group()
                    # will spawn a new heartbeat thread.
                    cause = self._heartbeat_thread.failed
                    self._heartbeat_thread = None
                    raise cause  # pylint: disable-msg=raising-bad-type

                # Awake the heartbeat thread if needed
                if self.heartbeat.should_heartbeat():
                    self._lock.notify()
                self.heartbeat.poll()


Instruction:
Can you generate the documentation for the following Python 3 function
def ensure_active_group(self):
        with self._client._lock, self._lock:
            if self._heartbeat_thread is None:
                self._start_heartbeat_thread()

            while self.need_rejoin() or self._rejoin_incomplete():
                self.ensure_coordinator_ready()

                # call on_join_prepare if needed. We set a flag
                # to make sure that we do not call it a second
                # time if the client is woken up before a pending
                # rebalance completes. This must be called on each
                # iteration of the loop because an event requiring
                # a rebalance (such as a metadata refresh which
                # changes the matched subscription set) can occur
                # while another rebalance is still in progress.
                if not self.rejoining:
                    self._on_join_prepare(self._generation.generation_id,
                                          self._generation.member_id)
                    self.rejoining = True

                # ensure that there are no pending requests to the coordinator.
                # This is important in particular to avoid resending a pending
                # JoinGroup request.
                while not self.coordinator_unknown():
                    if not self._client.in_flight_request_count(self.coordinator_id):
                        break
                    self._client.poll()
                else:
                    continue

                # we store the join future in case we are woken up by the user
                # after beginning the rebalance in the call to poll below.
                # This ensures that we do not mistakenly attempt to rejoin
                # before the pending rebalance has completed.
                if self.join_future is None:
                    # Fence off the heartbeat thread explicitly so that it cannot
                    # interfere with the join group. Note that this must come after
                    # the call to _on_join_prepare since we must be able to continue
                    # sending heartbeats if that callback takes some time.
                    self._heartbeat_thread.disable()

                    self.state = MemberState.REBALANCING
                    future = self._send_join_group_request()

                    self.join_future = future  # this should happen before adding callbacks

                    # handle join completion in the callback so that the
                    # callback will be invoked even if the consumer is woken up
                    # before finishing the rebalance
                    future.add_callback(self._handle_join_success)

                    # we handle failures below after the request finishes.
                    # If the join completes after having been woken up, the
                    # exception is ignored and we will rejoin
                    future.add_errback(self._handle_join_failure)

                else:
                    future = self.join_future

                self._client.poll(future=future)

                if future.succeeded():
                    self._on_join_complete(self._generation.generation_id,
                                           self._generation.member_id,
                                           self._generation.protocol,
                                           future.value)
                    self.join_future = None
                    self.rejoining = False

                else:
                    self.join_future = None
                    exception = future.exception
                    if isinstance(exception, (Errors.UnknownMemberIdError,
                                              Errors.RebalanceInProgressError,
                                              Errors.IllegalGenerationError)):
                        continue
                    elif not future.retriable():
                        raise exception  # pylint: disable-msg=raising-bad-type
                    time.sleep(self.config['retry_backoff_ms'] / 1000)

Response: Ensure that the group is active.


Instruction:
Write a Python 3 script to
send a JoinGroupRequest request to the coordinator.

Response:
def _send_join_group_request(self):
        """Join the group and return the assignment for the next generation.

        This function handles both JoinGroup and SyncGroup, delegating to
        :meth:`._perform_assignment` if elected leader by the coordinator.

        Returns:
            Future: resolves to the encoded-bytes assignment returned from the
                group leader
        """
        if self.coordinator_unknown():
            e = Errors.GroupCoordinatorNotAvailableError(self.coordinator_id)
            return Future().failure(e)

        elif not self._client.ready(self.coordinator_id, metadata_priority=False):
            e = Errors.NodeNotReadyError(self.coordinator_id)
            return Future().failure(e)

        # send a join group request to the coordinator
        log.info("(Re-)joining group %s", self.group_id)
        member_metadata = [
            (protocol, metadata if isinstance(metadata, bytes) else metadata.encode())
            for protocol, metadata in self.group_protocols()
        ]
        if self.config['api_version'] < (0, 9):
            raise Errors.KafkaError('JoinGroupRequest api requires 0.9+ brokers')
        elif (0, 9) <= self.config['api_version'] < (0, 10, 1):
            request = JoinGroupRequest[0](
                self.group_id,
                self.config['session_timeout_ms'],
                self._generation.member_id,
                self.protocol_type(),
                member_metadata)
        elif (0, 10, 1) <= self.config['api_version'] < (0, 11, 0):
            request = JoinGroupRequest[1](
                self.group_id,
                self.config['session_timeout_ms'],
                self.config['max_poll_interval_ms'],
                self._generation.member_id,
                self.protocol_type(),
                member_metadata)
        else:
            request = JoinGroupRequest[2](
                self.group_id,
                self.config['session_timeout_ms'],
                self.config['max_poll_interval_ms'],
                self._generation.member_id,
                self.protocol_type(),
                member_metadata)

        # create the request for the coordinator
        log.debug("Sending JoinGroup (%s) to coordinator %s", request, self.coordinator_id)
        future = Future()
        _f = self._client.send(self.coordinator_id, request)
        _f.add_callback(self._handle_join_group_response, future, time.time())
        _f.add_errback(self._failed_request, self.coordinator_id,
                       request, future)
        return future


Instruction:
Can you tell what is the following Python 3 function doing
def _on_join_leader(self, response):
        try:
            group_assignment = self._perform_assignment(response.leader_id,
                                                        response.group_protocol,
                                                        response.members)
        except Exception as e:
            return Future().failure(e)

        version = 0 if self.config['api_version'] < (0, 11, 0) else 1
        request = SyncGroupRequest[version](
            self.group_id,
            self._generation.generation_id,
            self._generation.member_id,
            [(member_id,
              assignment if isinstance(assignment, bytes) else assignment.encode())
             for member_id, assignment in six.iteritems(group_assignment)])

        log.debug("Sending leader SyncGroup for group %s to coordinator %s: %s",
                  self.group_id, self.coordinator_id, request)
        return self._send_sync_group_request(request)

Response:
Perform leader synchronization and send back the assignment
        for the group via SyncGroupRequest


Instruction:
Write a Python 3 function that can
send a request to the broker to discover the current coordinator for the group.

Response:
def _send_group_coordinator_request(self):
        """Discover the current coordinator for the group.

        Returns:
            Future: resolves to the node id of the coordinator
        """
        node_id = self._client.least_loaded_node()
        if node_id is None:
            return Future().failure(Errors.NoBrokersAvailable())

        elif not self._client.ready(node_id, metadata_priority=False):
            e = Errors.NodeNotReadyError(node_id)
            return Future().failure(e)

        log.debug("Sending group coordinator request for group %s to broker %s",
                  self.group_id, node_id)
        request = GroupCoordinatorRequest[0](self.group_id)
        future = Future()
        _f = self._client.send(node_id, request)
        _f.add_callback(self._handle_group_coordinator_response, future)
        _f.add_errback(self._failed_request, node_id, request, future)
        return future


Instruction:
Can you tell what is the following Python 3 function doing
def coordinator_dead(self, error):
        if self.coordinator_id is not None:
            log.warning("Marking the coordinator dead (node %s) for group %s: %s.",
                        self.coordinator_id, self.group_id, error)
            self.coordinator_id = None

Response: Mark the current coordinator as dead.


Instruction:
How would you explain what the following Python 3 function does
def generation(self):
        with self._lock:
            if self.state is not MemberState.STABLE:
                return None
            return self._generation

Response: Get the current generation state if the group is stable.


Instruction:
Write a Python 3 script for
resetting the generation and memberId because we have fallen out of the group.

Response:
def reset_generation(self):
        """Reset the generation and memberId because we have fallen out of the group."""
        with self._lock:
            self._generation = Generation.NO_GENERATION
            self.rejoin_needed = True
            self.state = MemberState.UNJOINED


Instruction:
Can you generate the documentation for the following Python 3 function
def maybe_leave_group(self):
        with self._client._lock, self._lock:
            if (not self.coordinator_unknown()
                and self.state is not MemberState.UNJOINED
                and self._generation is not Generation.NO_GENERATION):

                # this is a minimal effort attempt to leave the group. we do not
                # attempt any resending if the request fails or times out.
                log.info('Leaving consumer group (%s).', self.group_id)
                version = 0 if self.config['api_version'] < (0, 11, 0) else 1
                request = LeaveGroupRequest[version](self.group_id, self._generation.member_id)
                future = self._client.send(self.coordinator_id, request)
                future.add_callback(self._handle_leave_group_response)
                future.add_errback(log.error, "LeaveGroup request failed: %s")
                self._client.poll(future=future)

            self.reset_generation()

Response: Leave the current group and reset local generation and memberId.


Instruction:
How would you code a function in Python 3 to
send a heartbeat request to the group

Response:
def _send_heartbeat_request(self):
        """Send a heartbeat request"""
        if self.coordinator_unknown():
            e = Errors.GroupCoordinatorNotAvailableError(self.coordinator_id)
            return Future().failure(e)

        elif not self._client.ready(self.coordinator_id, metadata_priority=False):
            e = Errors.NodeNotReadyError(self.coordinator_id)
            return Future().failure(e)

        version = 0 if self.config['api_version'] < (0, 11, 0) else 1
        request = HeartbeatRequest[version](self.group_id,
                                            self._generation.generation_id,
                                            self._generation.member_id)
        log.debug("Heartbeat: %s[%s] %s", request.group, request.generation_id, request.member_id)  # pylint: disable-msg=no-member
        future = Future()
        _f = self._client.send(self.coordinator_id, request)
        _f.add_callback(self._handle_heartbeat_response, future, time.time())
        _f.add_errback(self._failed_request, self.coordinator_id,
                       request, future)
        return future


Instruction:
Can you tell what is the following Python 3 function doing
def metric_name(self, name, group, description='', tags=None):
        combined_tags = dict(self.config.tags)
        combined_tags.update(tags or {})
        return MetricName(name, group, description, combined_tags)

Response: Creates a MetricName object with the given name group description and tags.


Instruction:
Can you tell what is the following Python 3 function doing
def sensor(self, name, config=None,
               inactive_sensor_expiration_time_seconds=sys.maxsize,
               parents=None):
        """
        Get or create a sensor with the given unique name and zero or
        more parent sensors. All parent sensors will receive every value
        recorded with this sensor.

        Arguments:
            name (str): The name of the sensor
            config (MetricConfig, optional): A default configuration to use
                for this sensor for metrics that don't have their own config
            inactive_sensor_expiration_time_seconds (int, optional):
                If no value if recorded on the Sensor for this duration of
                time, it is eligible for removal
            parents (list of Sensor): The parent sensors

        Returns:
            Sensor: The sensor that is created
        """
        sensor = self.get_sensor(name)
        if sensor:
            return sensor

        with self._lock:
            sensor = self.get_sensor(name)
            if not sensor:
                sensor = Sensor(self, name, parents, config or self.config,
                                inactive_sensor_expiration_time_seconds)
                self._sensors[name] = sensor
                if parents:
                    for parent in parents:
                        children = self._children_sensors.get(parent)
                        if not children:
                            children = []
                            self._children_sensors[parent] = children
                        children.append(sensor)
                logger.debug('Added sensor with name %s', name)
            return sensor

Response: Get or create a Sensor with the given unique name and configuration.


Instruction:
How would you implement a function in Python 3 that
removes a sensor from the cache.

Response:
def remove_sensor(self, name):
        """
        Remove a sensor (if it exists), associated metrics and its children.

        Arguments:
            name (str): The name of the sensor to be removed
        """
        sensor = self._sensors.get(name)
        if sensor:
            child_sensors = None
            with sensor._lock:
                with self._lock:
                    val = self._sensors.pop(name, None)
                    if val and val == sensor:
                        for metric in sensor.metrics:
                            self.remove_metric(metric.metric_name)
                        logger.debug('Removed sensor with name %s', name)
                        child_sensors = self._children_sensors.pop(sensor, None)
            if child_sensors:
                for child_sensor in child_sensors:
                    self.remove_sensor(child_sensor.name)


Instruction:
Implement a function in Python 3 to
add a metric to monitor an object that implements measurable. This metric will be added to any existing sensors.

Response:
def add_metric(self, metric_name, measurable, config=None):
        """
        Add a metric to monitor an object that implements measurable.
        This metric won't be associated with any sensor.
        This is a way to expose existing values as metrics.

        Arguments:
            metricName (MetricName): The name of the metric
            measurable (AbstractMeasurable): The measurable that will be
                measured by this metric
            config (MetricConfig, optional): The configuration to use when
                measuring this measurable
        """
        # NOTE there was a lock here, but i don't think it's needed
        metric = KafkaMetric(metric_name, measurable, config or self.config)
        self.register_metric(metric)


Instruction:
Write a Python 3 script to
remove a metric from the internal cache if it exists and return it.

Response:
def remove_metric(self, metric_name):
        """
        Remove a metric if it exists and return it. Return None otherwise.
        If a metric is removed, `metric_removal` will be invoked
        for each reporter.

        Arguments:
            metric_name (MetricName): The name of the metric

        Returns:
            KafkaMetric: the removed `KafkaMetric` or None if no such
                metric exists
        """
        with self._lock:
            metric = self._metrics.pop(metric_name, None)
            if metric:
                for reporter in self._reporters:
                    reporter.metric_removal(metric)
            return metric


Instruction:
How would you implement a function in Python 3 that
adds a MetricReporter to the list of available metrics.

Response:
def add_reporter(self, reporter):
        """Add a MetricReporter"""
        with self._lock:
            reporter.init(list(self.metrics.values()))
            self._reporters.append(reporter)


Instruction:
Write a Python 3 function that can
close this metrics repository.

Response:
def close(self):
        """Close this metrics repository."""
        for reporter in self._reporters:
            reporter.close()

        self._metrics.clear()


Instruction:
How would you code a function in Python 3 to
construct a Snappy Message containing multiple Messages

Response:
def create_snappy_message(payloads, key=None):
    """
    Construct a Snappy Message containing multiple Messages

    The given payloads will be encoded, compressed, and sent as a single atomic
    message to Kafka.

    Arguments:
        payloads: list(bytes), a list of payload to send be sent to Kafka
        key: bytes, a key used for partition routing (optional)

    """
    message_set = KafkaProtocol._encode_message_set(
        [create_message(payload, pl_key) for payload, pl_key in payloads])

    snapped = snappy_encode(message_set)
    codec = ATTRIBUTE_CODEC_MASK & CODEC_SNAPPY

    return kafka.structs.Message(0, 0x00 | codec, key, snapped)


Instruction:
How would you code a function in Python 3 to
create a message set using the given codec.

Response:
def create_message_set(messages, codec=CODEC_NONE, key=None, compresslevel=None):
    """Create a message set using the given codec.

    If codec is CODEC_NONE, return a list of raw Kafka messages. Otherwise,
    return a list containing a single codec-encoded message.
    """
    if codec == CODEC_NONE:
        return [create_message(m, k) for m, k in messages]
    elif codec == CODEC_GZIP:
        return [create_gzip_message(messages, key, compresslevel)]
    elif codec == CODEC_SNAPPY:
        return [create_snappy_message(messages, key)]
    else:
        raise UnsupportedCodecError("Codec 0x%02x unsupported" % (codec,))


Instruction:
Here you have a function in Python 3, explain what it does
def _encode_message_header(cls, client_id, correlation_id, request_key,
                               version=0):
        """
        Encode the common request envelope
        """
        return struct.pack('>hhih%ds' % len(client_id),
                           request_key,          # ApiKey
                           version,              # ApiVersion
                           correlation_id,       # CorrelationId
                           len(client_id),       # ClientId size
                           client_id)

Response: Encode the message header.


Instruction:
Can you tell what is the following Python 3 function doing
def _encode_message_set(cls, messages):
        message_set = []
        for message in messages:
            encoded_message = KafkaProtocol._encode_message(message)
            message_set.append(struct.pack('>qi%ds' % len(encoded_message), 0,
                                           len(encoded_message),
                                           encoded_message))
        return b''.join(message_set)

Response: Encode a list of messages into a message set.


Instruction:
How would you explain what the following Python 3 function does
def encode_produce_request(cls, payloads=(), acks=1, timeout=1000):
        if acks not in (1, 0, -1):
            raise ValueError('ProduceRequest acks (%s) must be 1, 0, -1' % acks)

        topics = []
        for topic, topic_payloads in group_by_topic_and_partition(payloads).items():
            topic_msgs = []
            for partition, payload in topic_payloads.items():
                partition_msgs = []
                for msg in payload.messages:
                    m = kafka.protocol.message.Message(
                          msg.value, key=msg.key,
                          magic=msg.magic, attributes=msg.attributes
                    )
                    partition_msgs.append((0, m.encode()))
                topic_msgs.append((partition, MessageSet.encode(partition_msgs, prepend_size=False)))
            topics.append((topic, topic_msgs))

        return kafka.protocol.produce.ProduceRequest[0](
            required_acks=acks,
            timeout=timeout,
            topics=topics
        )

Response: Encodes ProduceRequest struct into a ProduceRequest structure.


Instruction:
Implement a Python 3 function for
decoding ProduceResponse to ProduceResponsePayloads

Response:
def decode_produce_response(cls, response):
        """
        Decode ProduceResponse to ProduceResponsePayload

        Arguments:
            response: ProduceResponse

        Return: list of ProduceResponsePayload
        """
        return [
            kafka.structs.ProduceResponsePayload(topic, partition, error, offset)
            for topic, partitions in response.topics
            for partition, error, offset in partitions
        ]


Instruction:
Can you generate the documentation for the following Python 3 function
def encode_fetch_request(cls, payloads=(), max_wait_time=100, min_bytes=4096):
        return kafka.protocol.fetch.FetchRequest[0](
            replica_id=-1,
            max_wait_time=max_wait_time,
            min_bytes=min_bytes,
            topics=[(
                topic,
                [(
                    partition,
                    payload.offset,
                    payload.max_bytes)
                for partition, payload in topic_payloads.items()])
            for topic, topic_payloads in group_by_topic_and_partition(payloads).items()])

Response: Encodes a FetchRequest struct into a FetchRequest struct.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def decode_fetch_response(cls, response):
        return [
            kafka.structs.FetchResponsePayload(
                topic, partition, error, highwater_offset, [
                    offset_and_msg
                    for offset_and_msg in cls.decode_message_set(messages)])
            for topic, partitions in response.topics
                for partition, error, highwater_offset, messages in partitions
        ]

Response: Decodes FetchResponse struct to FetchResponsePayloads


Instruction:
Make a summary of the following Python 3 code
def decode_offset_response(cls, response):
        return [
            kafka.structs.OffsetResponsePayload(topic, partition, error, tuple(offsets))
            for topic, partitions in response.topics
            for partition, error, offsets in partitions
        ]

Response: Decodes OffsetResponse into OffsetResponsePayloads


Instruction:
Here you have a function in Python 3, explain what it does
def decode_list_offset_response(cls, response):
        return [
            kafka.structs.ListOffsetResponsePayload(topic, partition, error, timestamp, offset)
            for topic, partitions in response.topics
            for partition, error, timestamp, offset in partitions
        ]

Response: Decode OffsetResponse_v2 into ListOffsetResponsePayloads


Instruction:
How would you implement a function in Python 3 that
encodes a MetadataRequest object

Response:
def encode_metadata_request(cls, topics=(), payloads=None):
        """
        Encode a MetadataRequest

        Arguments:
            topics: list of strings
        """
        if payloads is not None:
            topics = payloads

        return kafka.protocol.metadata.MetadataRequest[0](topics)


Instruction:
Can you tell what is the following Python 3 function doing
def encode_consumer_metadata_request(cls, client_id, correlation_id, payloads):
        message = []
        message.append(cls._encode_message_header(client_id, correlation_id,
                                                  KafkaProtocol.CONSUMER_METADATA_KEY))
        message.append(struct.pack('>h%ds' % len(payloads), len(payloads), payloads))

        msg = b''.join(message)
        return write_int_string(msg)

Response: Encode a ConsumerMetadataRequest object.


Instruction:
Make a summary of the following Python 3 code
def decode_consumer_metadata_response(cls, data):
        ((correlation_id, error, nodeId), cur) = relative_unpack('>ihi', data, 0)
        (host, cur) = read_short_string(data, cur)
        ((port,), cur) = relative_unpack('>i', data, cur)

        return kafka.structs.ConsumerMetadataResponse(error, nodeId, host, port)

Response: Decodes bytes to a kafka. structs. ConsumerMetadataResponse object


Instruction:
Here you have a function in Python 3, explain what it does
def encode_offset_commit_request(cls, group, payloads):
        return kafka.protocol.commit.OffsetCommitRequest[0](
            consumer_group=group,
            topics=[(
                topic,
                [(
                    partition,
                    payload.offset,
                    payload.metadata)
                for partition, payload in six.iteritems(topic_payloads)])
            for topic, topic_payloads in six.iteritems(group_by_topic_and_partition(payloads))])

Response: Encode an OffsetCommitRequest struct that is used to commit the offsets for the specified topic and partition.


Instruction:
Can you generate the documentation for the following Python 3 function
def decode_offset_commit_response(cls, response):
        return [
            kafka.structs.OffsetCommitResponsePayload(topic, partition, error)
            for topic, partitions in response.topics
            for partition, error in partitions
        ]

Response: Decodes an OffsetCommitResponse to an OffsetCommitResponsePayload


Instruction:
Can you generate a brief explanation for the following Python 3 code
def encode_offset_fetch_request(cls, group, payloads, from_kafka=False):
        version = 1 if from_kafka else 0
        return kafka.protocol.commit.OffsetFetchRequest[version](
            consumer_group=group,
            topics=[(
                topic,
                list(topic_payloads.keys()))
            for topic, topic_payloads in six.iteritems(group_by_topic_and_partition(payloads))])

Response: Encode an OffsetFetchRequest struct.


Instruction:
Explain what the following Python 3 code does
def decode_offset_fetch_response(cls, response):
        return [
            kafka.structs.OffsetFetchResponsePayload(
                topic, partition, offset, metadata, error
            )
            for topic, partitions in response.topics
            for partition, offset, metadata, error in partitions
        ]

Response: Decode OffsetFetchResponse to OffsetFetchResponsePayloads


Instruction:
Create a Python 3 function for
returning a file descriptor from a file object.

Response:
def _fileobj_lookup(self, fileobj):
        """Return a file descriptor from a file object.

        This wraps _fileobj_to_fd() to do an exhaustive search in case
        the object is invalid but we still have it in our map.  This
        is used by unregister() so we can unregister an object that
        was previously registered even if it is closed.  It is also
        used by _SelectorMapping.
        """
        try:
            return _fileobj_to_fd(fileobj)
        except ValueError:
            # Do an exhaustive search.
            for key in self._fd_to_key.values():
                if key.fileobj is fileobj:
                    return key.fd
            # Raise ValueError after all.
            raise


Instruction:
Explain what the following Python 3 code does
def _check_forest(self, sensors):
        if self in sensors:
            raise ValueError('Circular dependency in sensors: %s is its own'
                             'parent.' % (self.name,))
        sensors.add(self)
        for parent in self._parents:
            parent._check_forest(sensors)

Response: Validate that this sensor doesn t end up referencing itself.