Instruction:
Given the following Python 3 function, write the documentation
def load_isd_hourly_temp_data(
        self,
        start,
        end,
        read_from_cache=True,
        write_to_cache=True,
        error_on_missing_years=True,
    ):
        """ Load resampled hourly ISD temperature data from start date to end date (inclusive).

        This is the primary convenience method for loading resampled hourly ISD temperature data.

        Parameters
        ----------
        start : datetime.datetime
            The earliest date from which to load data.
        end : datetime.datetime
            The latest date until which to load data.
        read_from_cache : bool
            Whether or not to load data from cache.
        write_to_cache : bool
            Whether or not to write newly loaded data to cache.
        """
        return load_isd_hourly_temp_data(
            self.usaf_id,
            start,
            end,
            read_from_cache=read_from_cache,
            write_to_cache=write_to_cache,
            error_on_missing_years=error_on_missing_years,
        )

Response: Load ISD hourly temperature data from start to end date.


Instruction:
Given the following Python 3 function, write the documentation
def load_isd_daily_temp_data(
        self, start, end, read_from_cache=True, write_to_cache=True
    ):
        """ Load resampled daily ISD temperature data from start date to end date (inclusive).

        This is the primary convenience method for loading resampled daily ISD temperature data.

        Parameters
        ----------
        start : datetime.datetime
            The earliest date from which to load data.
        end : datetime.datetime
            The latest date until which to load data.
        read_from_cache : bool
            Whether or not to load data from cache.
        write_to_cache : bool
            Whether or not to write newly loaded data to cache.
        """
        return load_isd_daily_temp_data(
            self.usaf_id,
            start,
            end,
            read_from_cache=read_from_cache,
            write_to_cache=write_to_cache,
        )

Response: Load resampled daily ISD temperature data from start date to end date.


Instruction:
Can you write a function in Python 3 where it
loads resampled GSOD temperature data from start date to end date.

Response:
def load_gsod_daily_temp_data(
        self, start, end, read_from_cache=True, write_to_cache=True
    ):
        """ Load resampled daily GSOD temperature data from start date to end date (inclusive).

        This is the primary convenience method for loading resampled daily GSOD temperature data.

        Parameters
        ----------
        start : datetime.datetime
            The earliest date from which to load data.
        end : datetime.datetime
            The latest date until which to load data.
        read_from_cache : bool
            Whether or not to load data from cache.
        write_to_cache : bool
            Whether or not to write newly loaded data to cache.
        """
        return load_gsod_daily_temp_data(
            self.usaf_id,
            start,
            end,
            read_from_cache=read_from_cache,
            write_to_cache=write_to_cache,
        )


Instruction:
Here you have a function in Python 3, explain what it does
def load_tmy3_hourly_temp_data(
        self, start, end, read_from_cache=True, write_to_cache=True
    ):
        """ Load hourly TMY3 temperature data from start date to end date (inclusive).

        This is the primary convenience method for loading hourly TMY3 temperature data.

        Parameters
        ----------
        start : datetime.datetime
            The earliest date from which to load data.
        end : datetime.datetime
            The latest date until which to load data.
        read_from_cache : bool
            Whether or not to load data from cache.
        write_to_cache : bool
            Whether or not to write newly loaded data to cache.
        """
        return load_tmy3_hourly_temp_data(
            self.usaf_id,
            start,
            end,
            read_from_cache=read_from_cache,
            write_to_cache=write_to_cache,
        )

Response: Load TMY3 hourly temperature data from start to end date.


Instruction:
Can you generate the documentation for the following Python 3 function
def load_cz2010_hourly_temp_data(
        self, start, end, read_from_cache=True, write_to_cache=True
    ):
        """ Load hourly CZ2010 temperature data from start date to end date (inclusive).

        This is the primary convenience method for loading hourly CZ2010 temperature data.

        Parameters
        ----------
        start : datetime.datetime
            The earliest date from which to load data.
        end : datetime.datetime
            The latest date until which to load data.
        read_from_cache : bool
            Whether or not to load data from cache.
        write_to_cache : bool
            Whether or not to write newly loaded data to cache.
        """
        return load_cz2010_hourly_temp_data(
            self.usaf_id,
            start,
            end,
            read_from_cache=read_from_cache,
            write_to_cache=write_to_cache,
        )

Response: Load hourly CZ2010 temperature data from start date to end date.


Instruction:
How would you explain what the following Python 3 function does
def plot_station_mapping(
    target_latitude,
    target_longitude,
    isd_station,
    distance_meters,
    target_label="target",
):  # pragma: no cover
    """ Plots this mapping on a map."""
    try:
        import matplotlib.pyplot as plt
    except ImportError:
        raise ImportError("Plotting requires matplotlib.")

    try:
        import cartopy.crs as ccrs
        import cartopy.feature as cfeature
        import cartopy.io.img_tiles as cimgt
    except ImportError:
        raise ImportError("Plotting requires cartopy.")

    lat, lng = isd_station.coords
    t_lat, t_lng = float(target_latitude), float(target_longitude)

    # fiture
    fig = plt.figure(figsize=(16, 8))

    # axes
    tiles = cimgt.StamenTerrain()
    ax = plt.subplot(1, 1, 1, projection=tiles.crs)

    # offsets for labels
    x_max = max([lng, t_lng])
    x_min = min([lng, t_lng])
    x_diff = x_max - x_min

    y_max = max([lat, t_lat])
    y_min = min([lat, t_lat])
    y_diff = y_max - y_min

    xoffset = x_diff * 0.05
    yoffset = y_diff * 0.05

    # minimum
    left = x_min - x_diff * 0.5
    right = x_max + x_diff * 0.5
    bottom = y_min - y_diff * 0.3
    top = y_max + y_diff * 0.3

    width_ratio = 2.
    height_ratio = 1.

    if (right - left) / (top - bottom) > width_ratio / height_ratio:
        # too short
        goal = (right - left) * height_ratio / width_ratio
        diff = goal - (top - bottom)
        bottom = bottom - diff / 2.
        top = top + diff / 2.
    else:
        # too skinny
        goal = (top - bottom) * width_ratio / height_ratio
        diff = goal - (right - left)
        left = left - diff / 2.
        right = right + diff / 2.

    ax.set_extent([left, right, bottom, top])

    # determine zoom level
    # tile size at level 1 = 64 km
    # level 2 = 32 km, level 3 = 16 km, etc, i.e. 128/(2^n) km
    N_TILES = 600  # (how many tiles approximately fit in distance)
    km = distance_meters / 1000.0
    zoom_level = int(np.log2(128 * N_TILES / km))

    ax.add_image(tiles, zoom_level)

    # line between
    plt.plot(
        [lng, t_lng],
        [lat, t_lat],
        linestyle="-",
        dashes=[2, 2],
        transform=ccrs.Geodetic(),
    )

    # station
    ax.plot(lng, lat, "ko", markersize=7, transform=ccrs.Geodetic())

    # target
    ax.plot(t_lng, t_lat, "ro", markersize=7, transform=ccrs.Geodetic())

    # station label
    station_label = "{} ({})".format(isd_station.usaf_id, isd_station.name)
    ax.text(lng + xoffset, lat + yoffset, station_label, transform=ccrs.Geodetic())

    # target label
    ax.text(t_lng + xoffset, t_lat + yoffset, target_label, transform=ccrs.Geodetic())

    # distance labels
    mid_lng = (lng + t_lng) / 2
    mid_lat = (lat + t_lat) / 2
    dist_text = "{:.01f} km".format(km)
    ax.text(mid_lng + xoffset, mid_lat + yoffset, dist_text, transform=ccrs.Geodetic())

    plt.show()

Response: Plots this mapping on a map.


Instruction:
How would you implement a function in Python 3 that
plots a list of mapping results on a map.

Response:
def plot_station_mappings(mapping_results):  # pragma: no cover
    """ Plot a list of mapping results on a map.

    Requires matplotlib and cartopy.

    Parameters
    ----------
    mapping_results : list of MappingResult objects
        Mapping results to plot
    """
    try:
        import matplotlib.pyplot as plt
    except ImportError:
        raise ImportError("Plotting requires matplotlib.")

    try:
        import cartopy.crs as ccrs
        import cartopy.feature as cfeature
    except ImportError:
        raise ImportError("Plotting requires cartopy.")

    lats = []
    lngs = []
    t_lats = []
    t_lngs = []
    n_discards = 0
    for mapping_result in mapping_results:
        if not mapping_result.is_empty():
            lat, lng = mapping_result.isd_station.coords
            t_lat, t_lng = map(float, mapping_result.target_coords)
            lats.append(lat)
            lngs.append(lng)
            t_lats.append(t_lat)
            t_lngs.append(t_lng)
        else:
            n_discards += 1

    print("Discarded {} empty mappings".format(n_discards))

    # figure
    fig = plt.figure(figsize=(60, 60))

    # axes
    ax = plt.subplot(1, 1, 1, projection=ccrs.Mercator())

    # offsets for labels
    all_lngs = lngs + t_lngs
    all_lats = lats + t_lats
    x_max = max(all_lngs)  # lists
    x_min = min(all_lngs)
    x_diff = x_max - x_min

    y_max = max(all_lats)
    y_min = min(all_lats)
    y_diff = y_max - y_min

    # minimum
    x_pad = 0.1 * x_diff
    y_pad = 0.1 * y_diff
    left = x_min - x_pad
    right = x_max + x_pad
    bottom = y_min - y_pad
    top = y_max + y_pad

    width_ratio = 2.
    height_ratio = 1.

    if (right - left) / (top - bottom) > height_ratio / width_ratio:
        # too short
        goal = (right - left) * height_ratio / width_ratio
        diff = goal - (top - bottom)
        bottom = bottom - diff / 2.
        top = top + diff / 2.
    else:
        # too skinny
        goal = (top - bottom) * width_ratio / height_ratio
        diff = goal - (right - left)
        left = left - diff / 2.
        right = right + diff / 2.

    left = max(left, -179.9)
    right = min(right, 179.9)
    bottom = max([bottom, -89.9])
    top = min([top, 89.9])

    ax.set_extent([left, right, bottom, top])

    # OCEAN
    ax.add_feature(
        cfeature.NaturalEarthFeature(
            "physical",
            "ocean",
            "50m",
            edgecolor="face",
            facecolor=cfeature.COLORS["water"],
        )
    )

    # LAND
    ax.add_feature(
        cfeature.NaturalEarthFeature(
            "physical",
            "land",
            "50m",
            edgecolor="face",
            facecolor=cfeature.COLORS["land"],
        )
    )

    # BORDERS
    ax.add_feature(
        cfeature.NaturalEarthFeature(
            "cultural",
            "admin_0_boundary_lines_land",
            "50m",
            edgecolor="black",
            facecolor="none",
        )
    )

    # LAKES
    ax.add_feature(
        cfeature.NaturalEarthFeature(
            "physical",
            "lakes",
            "50m",
            edgecolor="face",
            facecolor=cfeature.COLORS["water"],
        )
    )

    # COASTLINE
    ax.add_feature(
        cfeature.NaturalEarthFeature(
            "physical", "coastline", "50m", edgecolor="black", facecolor="none"
        )
    )

    # lines between
    # for lat, t_lat, lng, t_lng in zip(lats, t_lats, lngs, t_lngs):
    ax.plot(
        [lngs, t_lngs],
        [lats, t_lats],
        color="k",
        linestyle="-",
        transform=ccrs.Geodetic(),
        linewidth=0.3,
    )

    # stations
    ax.plot(lngs, lats, "bo", markersize=1, transform=ccrs.Geodetic())

    plt.title("Location to weather station mapping")

    plt.show()


Instruction:
Can you generate the documentation for the following Python 3 function
def execute(self, input_data):
        ''' Execute the ViewPDF worker '''

        # Just a small check to make sure we haven't been called on the wrong file type
        if (input_data['meta']['type_tag'] != 'pdf'):
            return {'error': self.__class__.__name__+': called on '+input_data['meta']['type_tag']}

        view = {}
        view['strings'] = input_data['strings']['string_list'][:5]
        view.update(input_data['meta'])
        return view

Response: Execute the ViewPDF worker


Instruction:
How would you implement a function in Python 3 that
builds a graph of nodes and edges from a conn. log file.

Response:
def conn_log_graph(self, stream):
        ''' Build up a graph (nodes and edges from a Bro conn.log) '''
        conn_log = list(stream)
        print 'Entering conn_log_graph...(%d rows)' % len(conn_log)
        for row in stream:

            # Add the connection id with service as one of the labels
            self.add_node(row['uid'], row['uid'][:6], ['conn_id', row['service']])

            # Add the originating host
            self.add_node(row['id.orig_h'], row['id.orig_h'], ['ip', 'origin'])

            # Add the response host
            self.add_node(row['id.resp_h'], row['id.resp_h'], ['ip', 'response'])

            # Add the ip->connection relationships
            self.add_rel(row['uid'], row['id.orig_h'], 'origin')
            self.add_rel(row['uid'], row['id.resp_h'], 'response')


Instruction:
Make a summary of the following Python 3 code
def dns_log_graph(self, stream):
        ''' Build up a graph (nodes and edges from a Bro dns.log) '''
        dns_log = list(stream)
        print 'Entering dns_log_graph...(%d rows)' % len(dns_log)
        for row in dns_log:

            # Skip '-' hosts
            if (row['id.orig_h'] == '-'):
                continue

            # Add the originating host
            self.add_node(row['id.orig_h'], row['id.orig_h'], ['host', 'origin'])

            # Add the query host
            self.add_node(row['query'], row['query'], ['host', 'dns_query'])

            # The relationship between origin host and query
            self.add_rel(row['id.orig_h'], row['query'], 'dns_query')

            # Add the DNS answers as hosts and add the relationships
            for answer in row['answers'].split(','):
                self.add_node(answer, answer, ['host'])
                self.add_rel(row['query'], answer, row['qtype_name'])

Response: Build a graph of nodes and edges from a Bro dns. log file.


Instruction:
Can you generate the documentation for the following Python 3 function
def weird_log_graph(self, stream):
        ''' Build up a graph (nodes and edges from a Bro weird.log) '''
        weird_log = list(stream)
        print 'Entering weird_log_graph...(%d rows)' % len(weird_log)

        # Here we're just going to capture that something weird
        # happened between two hosts
        weird_pairs = set()
        for row in weird_log:
            weird_pairs.add((row['id.orig_h'], row['id.resp_h']))

        # Okay now make the weird node for each pair
        for pair in weird_pairs:

            # Skip '-' hosts
            if (pair[0] == '-'):
                continue

            # Add the originating host
            self.add_node(pair[0], pair[0], ['host', 'origin'])

            # Add the response host
            self.add_node(pair[1], pair[1], ['host'])

            # Add a weird node
            weird_name = 'weird'+pair[0]+'_'+pair[1]
            self.add_node(weird_name, 'weird', ['weird'])

            # The relationships between the nodes
            self.add_rel(pair[0], weird_name, 'weird')
            self.add_rel(weird_name, pair[1], 'weird')

Response: Build a graph of nodes and edges from a Bro weird. log file.


Instruction:
Create a Python 3 function for
building a graph of nodes and edges from a Bro files. log file stream.

Response:
def files_log_graph(self, stream):
        ''' Build up a graph (nodes and edges from a Bro files.log) '''
        file_log = list(stream)
        print 'Entering file_log_graph...(%d rows)' % len(file_log)
        for row in file_log:

            # If the mime-type is interesting add the uri and the host->uri->host relationships
            if row['mime_type'] not in self.exclude_mime_types:

                # Check for weird conditions
                if (row['total_bytes'] == '-'):
                    continue
                if ('-' in row['md5']):
                    continue

                # Check for missing bytes
                if row['missing_bytes']:
                    labels = ['file','missing']
                else:
                    labels = ['file']

                # Make the file node name kewl
                name = '%6s  %s  %.0f-KB' % (row['md5'][:6], row['mime_type'], row['total_bytes']/1024.0)
                if row['missing_bytes']:
                    name += '*'
                name = name.replace('application/','')

                # Add the file node
                self.add_node(row['md5'], name, labels)

                # Add the tx_host
                self.add_node(row['tx_hosts'], row['tx_hosts'], ['host'])

                # Add the file->tx_host relationship
                self.add_rel(row['tx_hosts'], row['md5'], 'file')


Instruction:
Make a summary of the following Python 3 code
def execute(self, input_data):
        ''' Execute the URL worker '''
        string_output = input_data['strings']['string_list']
        flatten = ' '.join(string_output)
        urls = self.url_match.findall(flatten)
        return {'url_list': urls}

Response: Execute the URL worker


Instruction:
Can you generate the documentation for the following Python 3 function
def all_files_in_directory(path):
    file_list = []
    for dirname, dirnames, filenames in os.walk(path):
        for filename in filenames:
            file_list.append(os.path.join(dirname, filename))
    return file_list

Response: Recursively ist all files under a directory


Instruction:
How would you explain what the following Python 3 function does
def load_all_plugins(self):

        # Go through the existing python files in the plugin directory
        self.plugin_path = os.path.realpath(self.plugin_dir)
        sys.path.append(self.plugin_dir)
        print '<<< Plugin Manager >>>'
        for f in [os.path.join(self.plugin_dir, child) for child in os.listdir(self.plugin_dir)]:

            # Skip certain files
            if '.DS_Store' in f or '__init__.py' in f:
                continue

            # Add the plugin
            self.add_plugin(f)

Response: Load all the plugins in the plugin directory


Instruction:
Write a Python 3 function for
removing a deleted plugin.

Response:
def remove_plugin(self, f):
        """Remvoing a deleted plugin.

        Args:
            f: the filepath for the plugin.
        """
        if f.endswith('.py'):
            plugin_name = os.path.splitext(os.path.basename(f))[0]
            print '- %s %sREMOVED' % (plugin_name, color.Red)
            print '\t%sNote: still in memory, restart Workbench to remove...%s' % \
                  (color.Yellow, color.Normal)


Instruction:
Create a Python 3 function for
adding and verifying a plugin.

Response:
def add_plugin(self, f):
        """Adding and verifying plugin.

        Args:
            f: the filepath for the plugin.
        """
        if f.endswith('.py'):

            # Just the basename without extension
            plugin_name = os.path.splitext(os.path.basename(f))[0]

            # It's possible the plugin has been modified and needs to be reloaded
            if plugin_name in sys.modules:
                try:
                    handler = reload(sys.modules[plugin_name])
                    print'\t- %s %sRELOAD%s' % (plugin_name, color.Yellow, color.Normal)
                except ImportError, error:
                    print 'Failed to import plugin: %s (%s)' % (plugin_name, error)
                    return
            else:
                # Not already loaded so try to import it
                try:
                    handler = __import__(plugin_name, globals(), locals(), [], -1)
                except ImportError, error:
                    print 'Failed to import plugin: %s (%s)' % (plugin_name, error)
                    return

            # Run the handler through plugin validation
            plugin = self.validate(handler)
            print '\t- %s %sOK%s' % (plugin_name, color.Green, color.Normal)
            if plugin:

                # Okay must be successfully loaded so capture the plugin meta-data,
                # modification time and register the plugin through the callback
                plugin['name'] = plugin_name
                plugin['dependencies'] = plugin['class'].dependencies
                plugin['docstring'] = plugin['class'].__doc__
                plugin['mod_time'] = datetime.utcfromtimestamp(os.path.getmtime(f))

                # Plugin may accept sample_sets as input
                try:
                    plugin['sample_set_input'] = getattr(plugin['class'], 'sample_set_input')
                except AttributeError:
                    plugin['sample_set_input'] = False

                # Now pass the plugin back to workbench
                self.plugin_callback(plugin)


Instruction:
Make a summary of the following Python 3 code
def validate(self, handler):

        # Check for the test method first
        test_method = self.plugin_test_validation(handler)
        if not test_method:
            return None

        # Here we iterate through the classes found in the module and pick
        # the first one that satisfies the validation
        for name, plugin_class in inspect.getmembers(handler, inspect.isclass):
            if self.plugin_class_validation(plugin_class):
                return {'class':plugin_class, 'test':test_method}

        # If we're here the plugin didn't pass validation
        print 'Failure for plugin: %s' % (handler.__name__)
        print 'Validation Error: Worker class is required to have a dependencies list and an execute method'
        return None

Response:
Validate the plugin, each plugin must have the following:
            1) The worker class must have an execute method: execute(self, input_data).
            2) The worker class must have a dependencies list (even if it's empty).
            3) The file must have a top level test() method.

        Args:
            handler: the loaded plugin.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def store_sample(self, sample_bytes, filename, type_tag):

        # Temp sanity check for old clients
        if len(filename) > 1000:
            print 'switched bytes/filename... %s %s' % (sample_bytes[:100], filename[:100])
            exit(1)

        sample_info = {}

        # Compute the MD5 hash
        sample_info['md5'] = hashlib.md5(sample_bytes).hexdigest()

        # Check if sample already exists
        if self.has_sample(sample_info['md5']):
            return sample_info['md5']

        # Run the periodic operations
        self.periodic_ops()

        # Check if we need to expire anything
        self.expire_data()

        # Okay start populating the sample for adding to the data store
        # Filename, length, import time and type_tag
        sample_info['filename'] = filename
        sample_info['length'] = len(sample_bytes)
        sample_info['import_time'] = datetime.datetime.utcnow()
        sample_info['type_tag'] = type_tag

        # Random customer for now
        import random
        sample_info['customer'] = random.choice(['Mega Corp', 'Huge Inc', 'BearTron', 'Dorseys Mom'])

        # Push the file into the MongoDB GridFS
        sample_info['__grid_fs'] = self.gridfs_handle.put(sample_bytes)
        self.database[self.sample_collection].insert(sample_info)

        # Print info
        print 'Sample Storage: %.2f out of %.2f MB' % (self.sample_storage_size(), self.samples_cap)

        # Return the sample md5
        return sample_info['md5']

Response: Store a sample into the datastore.


Instruction:
Create a Python 3 function for
getting the storage size of the samples storage collection.

Response:
def sample_storage_size(self):
        """Get the storage size of the samples storage collection."""

        try:
            coll_stats = self.database.command('collStats', 'fs.chunks')
            sample_storage_size = coll_stats['size']/1024.0/1024.0
            return sample_storage_size
        except pymongo.errors.OperationFailure:
            return 0


Instruction:
Can you generate the documentation for the following Python 3 function
def expire_data(self):

        # Do we need to start deleting stuff?
        while self.sample_storage_size() > self.samples_cap:

            # This should return the 'oldest' record in samples
            record = self.database[self.sample_collection].find().sort('import_time',pymongo.ASCENDING).limit(1)[0]
            self.remove_sample(record['md5'])

Response: Expire data within the samples collection.


Instruction:
Make a summary of the following Python 3 code
def remove_sample(self, md5):

        # Grab the sample
        record = self.database[self.sample_collection].find_one({'md5': md5})
        if not record:
            return

        # Delete it
        print 'Deleting sample: %s (%.2f MB)...' % (record['md5'], record['length']/1024.0/1024.0)
        self.database[self.sample_collection].remove({'md5': record['md5']})
        self.gridfs_handle.delete(record['__grid_fs'])

        # Print info
        print 'Sample Storage: %.2f out of %.2f MB' % (self.sample_storage_size(), self.samples_cap)

Response: Delete a specific sample from the database


Instruction:
Implement a function in Python 3 to
clean data in preparation for serialization.

Response:
def clean_for_serialization(self, data):
        """Clean data in preparation for serialization.

        Deletes items having key either a BSON, datetime, dict or a list instance, or
        starting with __.

        Args:
            data: Sample data to be serialized.

        Returns:
            Cleaned data dictionary.
        """

        if isinstance(data, dict):
            for k in data.keys():
                if (k.startswith('__')):
                    del data[k]
                elif isinstance(data[k], bson.objectid.ObjectId):
                    del data[k]
                elif isinstance(data[k], datetime.datetime):
                    data[k] = data[k].isoformat()+'Z'
                elif isinstance(data[k], dict):
                    data[k] = self.clean_for_serialization(data[k])
                elif isinstance(data[k], list):
                    data[k] = [self.clean_for_serialization(item) for item in data[k]]
        return data


Instruction:
Write a Python 3 script for
cleaning data in preparation for storage.

Response:
def clean_for_storage(self, data):
        """Clean data in preparation for storage.

        Deletes items with key having a '.' or is '_id'. Also deletes those items
        whose value is a dictionary or a list.

        Args:
            data: Sample data dictionary to be cleaned.

        Returns:
            Cleaned data dictionary.
        """
        data = self.data_to_unicode(data)
        if isinstance(data, dict):
            for k in dict(data).keys():
                if k == '_id':
                    del data[k]
                    continue
                if '.' in k:
                    new_k = k.replace('.', '_')
                    data[new_k] = data[k]
                    del data[k]
                    k = new_k
                if isinstance(data[k], dict):
                    data[k] = self.clean_for_storage(data[k])
                elif isinstance(data[k], list):
                    data[k] = [self.clean_for_storage(item) for item in data[k]]
        return data


Instruction:
Write a Python 3 script to
support partial md5s return the full md5 with this method

Response:
def get_full_md5(self, partial_md5, collection):
        """Support partial/short md5s, return the full md5 with this method"""
        print 'Notice: Performing slow md5 search...'
        starts_with = '%s.*' % partial_md5
        sample_info = self.database[collection].find_one({'md5': {'$regex' : starts_with}},{'md5':1})
        return sample_info['md5'] if sample_info else None


Instruction:
Make a summary of the following Python 3 code
def get_sample(self, md5):

        # Support 'short' md5s but don't waste performance if the full md5 is provided
        if len(md5) < 32:
            md5 = self.get_full_md5(md5, self.sample_collection)

        # Grab the sample
        sample_info = self.database[self.sample_collection].find_one({'md5': md5})
        if not sample_info:
            return None

        # Get the raw bytes from GridFS (note: this could fail)
        try:
            grid_fs_id = sample_info['__grid_fs']
            sample_info = self.clean_for_serialization(sample_info)
            sample_info.update({'raw_bytes':self.gridfs_handle.get(grid_fs_id).read()})
            return sample_info
        except gridfs.errors.CorruptGridFile:
            # If we don't have the gridfs files, delete the entry from samples
            self.database[self.sample_collection].update({'md5': md5}, {'md5': None})
            return None

Response: Get the sample from the data store.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_sample_window(self, type_tag, size=10):

        # Convert size to MB
        size = size * 1024 * 1024

        # Grab all the samples of type=type_tag, sort by import_time (newest to oldest)
        cursor = self.database[self.sample_collection].find({'type_tag': type_tag},
            {'md5': 1,'length': 1}).sort('import_time',pymongo.DESCENDING)
        total_size = 0
        md5_list = []
        for item in cursor:
            if total_size > size:
                return md5_list
            md5_list.append(item['md5'])
            total_size += item['length']

        # If you get this far you don't have 'size' amount of data
        # so just return what you've got
        return md5_list

Response: Get a window of samples of type type_tag not to exceed size.


Instruction:
Given the following Python 3 function, write the documentation
def has_sample(self, md5):

        # The easiest thing is to simply get the sample and if that
        # succeeds than return True, else return False
        sample = self.get_sample(md5)
        return True if sample else False

Response: Checks if the data store has this sample with the given md5.


Instruction:
Explain what the following Python 3 code does
def _list_samples(self, predicate=None):
        cursor = self.database[self.sample_collection].find(predicate, {'_id':0, 'md5':1})
        return [item['md5'] for item in cursor]

Response: List all samples that meet the predicate or all.


Instruction:
Write a Python 3 script for
listing all samples that match the tags or all if tags are not specified.

Response:
def tag_match(self, tags=None):
        """List all samples that match the tags or all if tags are not specified.

        Args:
            tags: Match samples against these tags (or all if not specified)

        Returns:
            List of the md5s for the matching samples
        """
        if 'tags' not in self.database.collection_names():
            print 'Warning: Searching on non-existance tags collection'
            return None
        if not tags:
            cursor = self.database['tags'].find({}, {'_id':0, 'md5':1})
        else:
            cursor = self.database['tags'].find({'tags': {'$in': tags}}, {'_id':0, 'md5':1})

        # We have the tags, now make sure we only return those md5 which
        # also exist in the samples collection
        tag_md5s = set([item['md5'] for item in cursor])
        sample_md5s = set(item['md5'] for item in self.database['samples'].find({}, {'_id':0, 'md5':1}))
        return list(tag_md5s.intersection(sample_md5s))


Instruction:
Can you tell what is the following Python 3 function doing
def tags_all(self):
        if 'tags' not in self.database.collection_names():
            print 'Warning: Searching on non-existance tags collection'
            return None

        cursor = self.database['tags'].find({}, {'_id':0, 'md5':1, 'tags':1})
        return [item for item in cursor]

Response: Returns a list of the tags and md5s for all samples


Instruction:
Explain what the following Python 3 code does
def store_work_results(self, results, collection, md5):

        # Make sure the md5 and time stamp is on the data before storing
        results['md5'] = md5
        results['__time_stamp'] = datetime.datetime.utcnow()

        # If the data doesn't have a 'mod_time' field add one now
        if 'mod_time' not in results:
            results['mod_time'] = results['__time_stamp']

        # Fixme: Occasionally a capped collection will not let you update with a
        #        larger object, if you have MongoDB 2.6 or above this shouldn't
        #        really happen, so for now just kinda punting and giving a message.
        try:
            self.database[collection].update({'md5':md5}, self.clean_for_storage(results), True)
        except pymongo.errors.OperationFailure:
            #self.database[collection].insert({'md5':md5}, self.clean_for_storage(results), True)
            print 'Could not update exising object in capped collection, punting...'
            print 'collection: %s md5:%s' % (collection, md5)

Response: Store the results of the worker in the database.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def all_sample_md5s(self, type_tag=None):

        if type_tag:
            cursor = self.database[self.sample_collection].find({'type_tag': type_tag}, {'md5': 1, '_id': 0})
        else:
            cursor = self.database[self.sample_collection].find({}, {'md5': 1, '_id': 0})
        return [match.values()[0] for match in cursor]

Response: Return a list of all md5s matching the type_tag.


Instruction:
Can you generate the documentation for the following Python 3 function
def clear_worker_output(self):

        print 'Dropping all of the worker output collections... Whee!'
        # Get all the collections in the workbench database
        all_c = self.database.collection_names()

        # Remove collections that we don't want to cap
        try:
            all_c.remove('system.indexes')
            all_c.remove('fs.chunks')
            all_c.remove('fs.files')
            all_c.remove('sample_set')
            all_c.remove('tags')
            all_c.remove(self.sample_collection)
        except ValueError:
            print 'Catching a benign exception thats expected...'

        for collection in all_c:
            self.database.drop_collection(collection)

Response: Removes all of the worker output collections


Instruction:
Explain what the following Python 3 code does
def periodic_ops(self):

        # Only run every 30 seconds
        if (time.time() - self.last_ops_run) < 30:
            return

        try:

            # Reset last ops run
            self.last_ops_run = time.time()
            print 'Running Periodic Ops'

            # Get all the collections in the workbench database
            all_c = self.database.collection_names()

            # Remove collections that we don't want to cap
            try:
                all_c.remove('system.indexes')
                all_c.remove('fs.chunks')
                all_c.remove('fs.files')
                all_c.remove('info')
                all_c.remove('tags')
                all_c.remove(self.sample_collection)
            except ValueError:
                print 'Catching a benign exception thats expected...'

            # Convert collections to capped if desired
            if self.worker_cap:
                size = self.worker_cap * pow(1024, 2)  # MegaBytes per collection
                for collection in all_c:
                    self.database.command('convertToCapped', collection, size=size)

            # Loop through all collections ensuring they have an index on MD5s
            for collection in all_c:
                self.database[collection].ensure_index('md5')

            # Add required indexes for samples collection
            self.database[self.sample_collection].create_index('import_time')

            # Create an index on tags
            self.database['tags'].create_index('tags')

        # Mongo may throw an autoreconnect exception so catch it and just return
        # the autoreconnect means that some operations didn't get executed but
        # because this method gets called every 30 seconds no biggy...
        except pymongo.errors.AutoReconnect as e:
            print 'Warning: MongoDB raised an AutoReconnect...' % e
            return
        except Exception as e:
            print 'Critical: MongoDB raised an exception' % e
            return

Response: Run periodic operations on the data store.


Instruction:
Can you generate the documentation for the following Python 3 function
def to_unicode(self, s):

        # Fixme: This is total horseshit
        if isinstance(s, unicode):
            return s
        if isinstance(s, str):
            return unicode(s, errors='ignore')

        # Just return the original object
        return s

Response: Convert an elementary datatype to unicode.


Instruction:
Make a summary of the following Python 3 code
def execute(self, input_data):

        # Spin up SWF class
        swf = SWF()

        # Get the raw_bytes
        raw_bytes = input_data['sample']['raw_bytes']

        # Parse it
        swf.parse(StringIO(raw_bytes))

        # Header info
        head = swf.header
        output = {'version':head.version,'file_length':head.file_length,'frame_count':head.frame_count,
                  'frame_rate':head.frame_rate,'frame_size':head.frame_size.__str__(),'compressed':head.compressed}

        # Loop through all the tags
        output['tags'] = [tag.__str__() for tag in swf.tags]

        # Add the meta data to the output
        output.update(input_data['meta'])
        return output

        '''
        # Map all tag names to indexes
        tag_map = {tag.name:index for tag,index in enumerate(swf.tags)}

        # FileAttribute Info
        file_attr_tag = swf.tags[tag_map]

        '''
        '''

        # Build up return data structure
        output = {name:value for name,value in locals().iteritems()
                if name not in ['self', 'input_data','raw_bytes']}
        output.update(input_data['meta'])
        return output
        '''

Response: Execute the SWF command.


Instruction:
Here you have a function in Python 3, explain what it does
def grab_server_args():

    workbench_conf = ConfigParser.ConfigParser()
    config_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config.ini')
    workbench_conf.read(config_path)
    server = workbench_conf.get('workbench', 'server_uri')
    port = workbench_conf.get('workbench', 'server_port')

    # Collect args from the command line
    parser = argparse.ArgumentParser()
    parser.add_argument('-s', '--server', type=str, default=server, help='location of workbench server')
    parser.add_argument('-p', '--port', type=int, default=port, help='port used by workbench server')
    args, commands = parser.parse_known_args()
    server = str(args.server)
    port = str(args.port)

    return {'server':server, 'port':port, 'commands': commands}

Response: Grab server info from command line


Instruction:
Here you have a function in Python 3, explain what it does
def _make_attachment(self, attachment, str_encoding=None):
        is_inline_image = False
        if isinstance(attachment, MIMEBase):
            name = attachment.get_filename()
            content = attachment.get_payload(decode=True)
            mimetype = attachment.get_content_type()

            if attachment.get_content_maintype() == 'image' and attachment['Content-ID'] is not None:
                is_inline_image = True
                name = attachment['Content-ID']
        else:
            (name, content, mimetype) = attachment

        # Guess missing mimetype from filename, borrowed from
        # django.core.mail.EmailMessage._create_attachment()
        if mimetype is None and name is not None:
            mimetype, _ = mimetypes.guess_type(name)
        if mimetype is None:
            mimetype = DEFAULT_ATTACHMENT_MIME_TYPE

        try:
            # noinspection PyUnresolvedReferences
            if isinstance(content, unicode):
                # Python 2.x unicode string
                content = content.encode(str_encoding)
        except NameError:
            # Python 3 doesn't differentiate between strings and unicode
            # Convert python3 unicode str to bytes attachment:
            if isinstance(content, str):
                content = content.encode(str_encoding)

        content_b64 = b64encode(content)

        mj_attachment = {
            'Content-type': mimetype,
            'Filename': name or '',
            'content': content_b64.decode('ascii'),
        }
        return mj_attachment, is_inline_image

Response: Returns a dict of email. attachments item formatted for sending with Mailjet


Instruction:
Given the following Python 3 function, write the documentation
def index_data(self, data, index_name, doc_type):

        # Index the data (which needs to be a dict/object) if it's not
        # we're going to toss an exception
        if not isinstance(data, dict):
            raise RuntimeError('Index failed, data needs to be a dict!')

        try:
            self.els_search.index(index=index_name, doc_type=doc_type, body=data)
        except Exception, error:
            print 'Index failed: %s' % str(error)
            raise RuntimeError('Index failed: %s' % str(error))

Response: Take an arbitrary dictionary of data and index it with ELS.


Instruction:
Write a Python 3 function that can
search the given index with the given ELS query.

Response:
def search(self, index_name, query):
        """Search the given index_name with the given ELS query.

        Args:
            index_name: Name of the Index
            query: The string to be searched.

        Returns:
            List of results.

        Raises:
            RuntimeError: When the search query fails.
        """

        try:
            results = self.els_search.search(index=index_name, body=query)
            return results
        except Exception, error:
            error_str = 'Query failed: %s\n' % str(error)
            error_str += '\nIs there a dynamic script in the query?, see www.elasticsearch.org'
            print error_str
            raise RuntimeError(error_str)


Instruction:
Create a Python 3 function for
indexing data in Stub Indexer.

Response:
def index_data(self, data, index_name, doc_type):
        """Index data in Stub Indexer."""

        print 'ELS Stub Indexer getting called...'
        print '%s %s %s %s' % (self, data, index_name, doc_type)


Instruction:
How would you explain what the following Python 3 function does
def execute(self, input_data):
        ''' Execute method '''
        my_ssdeep = input_data['meta_deep']['ssdeep']
        my_md5 = input_data['meta_deep']['md5']

        # For every PE sample in the database compute my ssdeep fuzzy match
        sample_set = self.workbench.generate_sample_set('exe')
        results = self.workbench.set_work_request('meta_deep', sample_set, ['md5','ssdeep'])
        sim_list = []
        for result in results:
            if result['md5'] != my_md5:
                sim_list.append({'md5':result['md5'], 'sim':ssd.compare(my_ssdeep, result['ssdeep'])})

        # Sort and return the sim_list (with some logic for threshold)
        sim_list.sort(key=itemgetter('sim'), reverse=True)
        output_list = [sim for sim in sim_list if sim['sim'] > 0]
        return {'md5': my_md5, 'sim_list':output_list}

Response: Execute method of the cluster_cache_get_cluster_entry method


Instruction:
Write a Python 3 function that can
execute the help_formatter command

Response:
def execute(self, input_data):
        ''' Do CLI formatting and coloring based on the type_tag '''
        input_data = input_data['help_base']
        type_tag = input_data['type_tag']

        # Standard help text
        if type_tag == 'help':
            output = '%s%s%s' % (color.LightBlue, input_data['help'], color.Normal)

        # Worker
        elif type_tag == 'worker':
            output = '%s%s' % (color.Yellow, input_data['name'])
            output += '\n    %sInput: %s%s%s' % (color.LightBlue, color.Green, input_data['dependencies'], color.Normal)
            output += '\n    %s%s' % (color.Green, input_data['docstring'])

        # Command
        elif type_tag == 'command':
            output = '%s%s%s %s' % (color.Yellow, input_data['command'], color.LightBlue, input_data['sig'])
            output += '\n    %s%s%s' % (color.Green, input_data['docstring'], color.Normal)

        # WTF: Alert on unknown type_tag and return a string of the input_data
        else:
            print 'Alert: help_formatter worker received malformed object: %s' % str(input_data)
            output = '\n%s%s%s' % (color.Red, str(input_data), color.Normal)

        # Return the formatted and colored help
        return {'help': output}


Instruction:
Here you have a function in Python 3, explain what it does
def run():

    # Load the configuration file relative to this script location
    config_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config.ini')
    workbench_conf = ConfigParser.ConfigParser()
    config_ini = workbench_conf.read(config_path)
    if not config_ini:
        print 'Could not locate config.ini file, tried %s : exiting...' % config_path
        exit(1)

    # Pull configuration settings
    datastore_uri = workbench_conf.get('workbench', 'datastore_uri')
    database = workbench_conf.get('workbench', 'database')
    worker_cap = workbench_conf.getint('workbench', 'worker_cap')
    samples_cap = workbench_conf.getint('workbench', 'samples_cap')

    # Spin up Workbench ZeroRPC
    try:
        store_args = {'uri': datastore_uri, 'database': database, 'worker_cap':worker_cap, 'samples_cap':samples_cap}
        workbench = zerorpc.Server(WorkBench(store_args=store_args), name='workbench', heartbeat=60)
        workbench.bind('tcp://0.0.0.0:4242')
        print '\nWorkbench is ready and feeling super duper!'
        gevent_signal(signal.SIGTERM, workbench.stop)
        gevent_signal(signal.SIGINT, workbench.stop)
        gevent_signal(signal.SIGKILL, workbench.stop)
        workbench.run()
        print '\nWorkbench Server Shutting Down... and dreaming of sheep...'

    except zmq.error.ZMQError:
        print '\nInfo: Could not start Workbench server (no worries, probably already running...)\n'

Response: Run the workbench server


Instruction:
Can you generate a brief explanation for the following Python 3 code
def store_sample(self, input_bytes, filename, type_tag):

        # If the sample comes in with an unknown type_tag try to determine it
        if type_tag == 'unknown':
            print 'Info: Unknown File -- Trying to Determine Type...'
            type_tag = self.guess_type_tag(input_bytes, filename)

        # Do we have a compressed sample? If so decompress it
        if type_tag == 'lz4':
            input_bytes = lz4.loads(input_bytes)

        # Store the sample
        md5 = self.data_store.store_sample(input_bytes, filename, type_tag)

        # Add the type_tags to tags
        if type_tag != 'lz4':
            self.add_tags(md5, type_tag)

        return md5

Response: Store a sample into the DataStore.


Instruction:
Here you have a function in Python 3, explain what it does
def get_sample(self, md5):
        # First we try a sample, if we can't find one we try getting a sample_set.
        sample = self.data_store.get_sample(md5)
        if not sample:
            return {'sample_set': {'md5_list': self.get_sample_set(md5)}}
        return {'sample': sample}

Response: Get a sample from the DataStore.


Instruction:
Can you write a function in Python 3 where it
returns True if the md5 represent a sample_set False otherwise

Response:
def is_sample_set(self, md5):
        """ Does the md5 represent a sample_set?
            Args:
                md5: the md5 of the sample_set
            Returns:
                True/False
        """
        try:
            self.get_sample_set(md5)
            return True
        except WorkBench.DataNotFound:
            return False


Instruction:
Write a Python 3 function for
getting a sample from the DataStore.

Response:
def get_sample_window(self, type_tag, size):
        """ Get a sample from the DataStore.
            Args:
                type_tag: the type of samples ('pcap','exe','pdf')
                size: the size of the window in MegaBytes (10 = 10MB)
            Returns:
                A sample_set handle which represents the newest samples within the size window
        """
        md5_list = self.data_store.get_sample_window(type_tag, size)
        return self.store_sample_set(md5_list)


Instruction:
Write a Python 3 function that can
combine the samples together and store the result in the file.

Response:
def combine_samples(self, md5_list, filename, type_tag):
        """Combine samples together. This may have various use cases the most significant
           involving a bunch of sample 'chunks' got uploaded and now we combine them together

            Args:
                md5_list: The list of md5s to combine, order matters!
                filename: name of the file (used purely as meta data not for lookup)
                type_tag: ('exe','pcap','pdf','json','swf', or ...)
            Returns:
                the computed md5 of the combined samples
        """
        total_bytes = ""
        for md5 in md5_list:
            total_bytes += self.get_sample(md5)['sample']['raw_bytes']
            self.remove_sample(md5)

        # Store it
        return self.store_sample(total_bytes, filename, type_tag)


Instruction:
Given the following Python 3 function, write the documentation
def stream_sample(self, md5, kwargs=None):

        # Get the max_rows if specified
        max_rows = kwargs.get('max_rows', None) if kwargs else None

        # Grab the sample and it's raw bytes
        sample = self.get_sample(md5)['sample']
        raw_bytes = sample['raw_bytes']

        # Figure out the type of file to be streamed
        type_tag = sample['type_tag']
        if type_tag == 'bro':
            bro_log = bro_log_reader.BroLogReader(convert_datetimes=False)
            mem_file = StringIO(raw_bytes)
            generator = bro_log.read_log(mem_file)
            return generator
        elif type_tag == 'els_query':
            els_log = json.loads(raw_bytes)
            # Try to determine a couple of different types of ELS query results
            if 'fields' in els_log['hits']['hits'][0]:
                generator = (row['fields'] for row in els_log['hits']['hits'][:max_rows])
            else:
                generator = (row['_source'] for row in els_log['hits']['hits'][:max_rows])
            return generator
        elif type_tag == 'log':
            generator = ({'row':row} for row in raw_bytes.split('\n')[:max_rows])
            return generator
        elif type_tag == 'json':
            generator = (row for row in json.loads(raw_bytes)[:max_rows])
            return generator
        else:
            raise RuntimeError('Cannot stream file %s with type_tag:%s' % (md5, type_tag))

Response: Stream the sample by giving back a generator that yields rows of the file.


Instruction:
Can you write a function in Python 3 where it
returns a dataframe from the DataStore. This is just a convenience method that uses get_sample internally.

Response:
def get_dataframe(self, md5, compress='lz4'):
        """Return a dataframe from the DataStore. This is just a convenience method
           that uses get_sample internally.
            Args:
                md5: the md5 of the dataframe
                compress: compression to use: (defaults to 'lz4' but can be set to None)
            Returns:
                A msgpack'd Pandas DataFrame
            Raises:
                Workbench.DataNotFound if the dataframe is not found.
        """
        # First we try a sample, if we can't find one we try getting a sample_set.
        sample = self.data_store.get_sample(md5)
        if not sample:
            raise WorkBench.DataNotFound("Could not find %s in the data store", md5)
        if not compress:
            return sample['raw_bytes']
        else:
            compress_df = lz4.dumps(sample['raw_bytes'])
            print 'Info: DataFrame compression %.0f%%' % (len(compress_df)*100.0/float(len(sample['raw_bytes'])))
            return compress_df


Instruction:
Make a summary of the following Python 3 code
def guess_type_tag(self, input_bytes, filename):
        mime_to_type = {'application/jar': 'jar',
                        'application/java-archive': 'jar',
                        'application/octet-stream': 'data',
                        'application/pdf': 'pdf',
                        'application/vnd.ms-cab-compressed': 'cab',
                        'application/vnd.ms-fontobject': 'ms_font',
                        'application/vnd.tcpdump.pcap': 'pcap',
                        'application/x-dosexec': 'exe',
                        'application/x-empty': 'empty',
                        'application/x-shockwave-flash': 'swf',
                        'application/xml': 'xml',
                        'application/zip': 'zip',
                        'image/gif': 'gif',
                        'text/html': 'html',
                        'image/jpeg': 'jpg',
                        'image/png': 'png',
                        'image/x-icon': 'icon',
                        'text/plain': 'txt'
                        }

        # See what filemagic can determine
        with magic.Magic(flags=magic.MAGIC_MIME_TYPE) as mag:
            mime_type = mag.id_buffer(input_bytes[:1024])
            if mime_type in mime_to_type:
                type_tag = mime_to_type[mime_type]

                # If we get 'data' back look at the filename
                if type_tag == 'data':
                    print 'Info: File -- Trying to Determine Type from filename...'
                    ext = os.path.splitext(filename)[1][1:]
                    if ext in ['mem','vmem']:
                        type_tag = 'mem'
                    else:
                        print 'Alert: Failed to Determine Type for %s' % filename
                        exit(1) # Temp
                return type_tag
            else:
                print 'Alert: Sample Type could not be Determined'
                return 'unknown'

Response: Try to guess the type of the sample from the file.


Instruction:
How would you explain what the following Python 3 function does
def add_tags(self, md5, tags):
        if not tags: return
        tag_set = set(self.get_tags(md5)) if self.get_tags(md5) else set()
        if isinstance(tags, str):
            tags = [tags]
        for tag in tags:
            tag_set.add(tag)
        self.data_store.store_work_results({'tags': list(tag_set)}, 'tags', md5)

Response: Add tags to this sample


Instruction:
Here you have a function in Python 3, explain what it does
def set_tags(self, md5, tags):
        if isinstance(tags, str):
            tags = [tags]
        tag_set = set(tags)
        self.data_store.store_work_results({'tags': list(tag_set)}, 'tags', md5)

Response: Set the tags for this sample


Instruction:
Write a Python 3 script for
getting tags for this sample

Response:
def get_tags(self, md5):
        """Get tags for this sample"""
        tag_data = self.data_store.get_work_results('tags', md5)
        return tag_data['tags'] if tag_data else None


Instruction:
Can you tell what is the following Python 3 function doing
def index_sample(self, md5, index_name):
        generator = self.stream_sample(md5)
        for row in generator:
            self.indexer.index_data(row, index_name)

Response: Index a stored sample with the Indexer.


Instruction:
Given the following Python 3 function, write the documentation
def index_worker_output(self, worker_name, md5, index_name, subfield):

        # Grab the data
        if subfield:
            data = self.work_request(worker_name, md5)[worker_name][subfield]
        else:
            data = self.work_request(worker_name, md5)[worker_name]

        # Okay now index the data
        self.indexer.index_data(data, index_name=index_name, doc_type='unknown')

Response: Index the data in the index_name with the Indexer.


Instruction:
Can you write a function in Python 3 where it
adds a node to the database.

Response:
def add_node(self, node_id, name, labels):
        """ Add a node to the graph with name and labels.
            Args:
                node_id: the unique node_id e.g. 'www.evil4u.com'
                name: the display name of the node e.g. 'evil4u'
                labels: a list of labels e.g. ['domain','evil']
            Returns:
                Nothing
        """
        self.neo_db.add_node(node_id, name, labels)


Instruction:
Create a Python 3 function to
add a relationship between two nodes.

Response:
def add_rel(self, source_id, target_id, rel):
        """ Add a relationship: source, target must already exist (see add_node)
            'rel' is the name of the relationship 'contains' or whatever.
            Args:
                source_id: the unique node_id of the source
                target_id: the unique node_id of the target
                rel: name of the relationship
            Returns:
                Nothing
        """
        self.neo_db.add_rel(source_id, target_id, rel)


Instruction:
Given the following Python 3 function, write the documentation
def clear_db(self):
        self.data_store.clear_db()

        # Have the plugin manager reload all the plugins
        self.plugin_manager.load_all_plugins()

        # Store information about commands and workbench
        self._store_information()

Response: Clears the Main Database of all samples and worker output.


Instruction:
How would you explain what the following Python 3 function does
def clear_worker_output(self):
        self.data_store.clear_worker_output()

        # Have the plugin manager reload all the plugins
        self.plugin_manager.load_all_plugins()

        # Store information about commands and workbench
        self._store_information()

Response:
Drops all of the worker output collections
            Args : None


Instruction:
Make a summary of the following Python 3 code
def work_request(self, worker_name, md5, subkeys=None):

        # Pull the worker output
        work_results = self._recursive_work_resolver(worker_name, md5)

        # Subkeys (Fixme this is super klutzy)
        if subkeys:
            if isinstance(subkeys, str):
                subkeys = [subkeys]
            try:
                sub_results = {}
                for subkey in subkeys:
                    tmp = work_results[worker_name]

                    # Traverse any subkeys
                    for key in subkey.split('.')[:-1]:
                        tmp = tmp[key]

                    # Last subkey
                    key = subkey.split('.')[-1]
                    if key == '*':
                        for key in tmp.keys():
                            sub_results[key] = tmp[key]
                    else:
                        sub_results[key] = tmp[key]

                # Set the output
                work_results = sub_results

            except (KeyError, TypeError):
                raise RuntimeError('Could not get one or more subkeys for: %s' % (work_results))

        # Clean it and ship it
        return self.data_store.clean_for_serialization(work_results)

Response: This function is used to make a work request for an existing stored sample.


Instruction:
Can you create a Python 3 function that
makes a work request for an existing stored sample (or sample_set). Args: worker_name: 'strings', 'pe_features', whatever sample_set: the md5 of a sample_set in the Workbench data store subkeys: just get a subkey of the output: 'foo' or 'foo.bar' (None for all) Returns: The output is a generator of the results of the worker output for the sample_set

Response:
def set_work_request(self, worker_name, sample_set, subkeys=None):
        """ Make a work request for an existing stored sample (or sample_set).
            Args:
                worker_name: 'strings', 'pe_features', whatever
                sample_set: the md5 of a sample_set in the Workbench data store
                subkeys: just get a subkey of the output: 'foo' or 'foo.bar' (None for all)
            Returns:
                The output is a generator of the results of the worker output for the sample_set
        """

        # Does worker support sample_set_input?
        if self.plugin_meta[worker_name]['sample_set_input']:
            yield self.work_request(worker_name, sample_set, subkeys)

        # Loop through all the md5s and return a generator with yield
        else:
            md5_list = self.get_sample_set(sample_set)
            for md5 in md5_list:
                if subkeys:
                    yield self.work_request(worker_name, md5, subkeys)
                else:
                    yield self.work_request(worker_name, md5)[worker_name]


Instruction:
Can you tell what is the following Python 3 function doing
def store_sample_set(self, md5_list):

        # Sanity check
        if not md5_list:
            print 'Warning: Trying to store an empty sample_set'
            return None

        # Remove any duplicates
        md5_list = list(set(md5_list))

        for md5 in md5_list:
            if not self.has_sample(md5):
                raise RuntimeError('%s: Not found! All items in sample_set must be in the datastore' % (md5))
        set_md5 = hashlib.md5(str(md5_list)).hexdigest()
        self._store_work_results({'md5_list':md5_list}, 'sample_set', set_md5)
        return set_md5

Response: Store a sample set in the data store.


Instruction:
Explain what the following Python 3 code does
def generate_sample_set(self, tags=None):
        if isinstance(tags, str):
            tags = [tags]
        md5_list = self.data_store.tag_match(tags)
        return self.store_sample_set(md5_list)

Response: Generate a sample_set that maches the tags or all if tags are not specified.


Instruction:
Make a summary of the following Python 3 code
def help(self, topic=None):
        if not topic:
            topic = 'workbench'

        # It's possible to ask for help on something that doesn't exist
        # so we'll catch the exception and push back an object that
        # indicates we didn't find what they were asking for
        try:
            return self.work_request('help_formatter', topic)['help_formatter']['help']
        except WorkBench.DataNotFound as e:

            # Okay this is a bit tricky we want to give the user a nice error
            # message that has both the md5 of what they were looking for and
            # a nice informative message that explains what might have happened
            sample_md5 = e.args[0]
            return '%s%s\n\t%s%s%s' % (color.Yellow, sample_md5, color.Green, e.message(), color.Normal)

Response: Returns the formatted help for the given topic


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _help_basic(self):
        help =  '%sWorkbench: Getting started...' % (color.Yellow)
        help += '\n%sStore a sample into Workbench:'  % (color.Green)
        help += '\n\t%s$ workbench.store_sample(raw_bytes, filename, type_tag)' % (color.LightBlue)
        help += '\n\n%sNotice store_sample returns an md5 of the sample...'% (color.Yellow)
        help += '\n%sRun workers on the sample (view, meta, whatever...):'  % (color.Green)
        help += '\n\t%s$ workbench.work_request(\'view\', md5)%s' % (color.LightBlue, color.Normal)
        return help

Response: Help for Workbench Basics


Instruction:
Implement a Python 3 function for
helping on all available commands

Response:
def _help_commands(self):
        """ Help on all the available commands """
        help =  'Workbench Commands:'
        for command in self.list_all_commands():
            full_help = self.work_request('help_formatter', command)['help_formatter']['help']
            compact_help = full_help.split('\n')[:2]
            help += '\n\n%s' % '\n'.join(compact_help)
        return help


Instruction:
Explain what the following Python 3 code does
def _help_workers(self):
        help =  'Workbench Workers:'
        for worker in self.list_all_workers():
            full_help = self.work_request('help_formatter', worker)['help_formatter']['help']
            compact_help = full_help.split('\n')[:4]
            help += '\n\n%s' % '\n'.join(compact_help)
        return help

Response: Help on all the available workers


Instruction:
Explain what the following Python 3 code does
def list_all_commands(self):
        commands = [name for name, _ in inspect.getmembers(self, predicate=inspect.isroutine) if not name.startswith('_')]
        return commands

Response: Returns a list of all the Workbench commands


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_info(self, component):

        # Grab it, clean it and ship it
        work_results = self._get_work_results('info', component)
        return self.data_store.clean_for_serialization(work_results)

Response: Get the information about this component


Instruction:
How would you explain what the following Python 3 function does
def store_info(self, info_dict, component, type_tag):

        # Enforce dictionary input
        if not isinstance(info_dict, dict):
            print 'Critical: info_dict must be a python dictionary, got %s' % type(info_dict)
            return

        # Ensure values are not functions/methods/classes
        info_storage = {key:value for key, value in info_dict.iteritems() if not hasattr(value, '__call__')}

        # Place the type_tag on it and store it
        info_storage['type_tag'] = type_tag
        self._store_work_results(info_storage, 'info', component)

Response: Store information about a specific component.


Instruction:
Can you tell what is the following Python 3 function doing
def _store_information(self):

        print '<<< Generating Information Storage >>>'

        # Stores information on Workbench commands and signatures
        for name, meth in inspect.getmembers(self, predicate=inspect.isroutine):
            if not name.startswith('_'):
                info = {'command': name, 'sig': str(funcsigs.signature(meth)), 'docstring': meth.__doc__}
                self.store_info(info, name, type_tag='command')

        # Stores help text into the workbench information system
        self.store_info({'help': '<<< Workbench Server Version %s >>>' % self.version}, 'version', type_tag='help')
        self.store_info({'help': self._help_workbench()}, 'workbench', type_tag='help')
        self.store_info({'help': self._help_basic()}, 'basic', type_tag='help')
        self.store_info({'help': self._help_commands()}, 'commands', type_tag='help')
        self.store_info({'help': self._help_workers()}, 'workers', type_tag='help')

Response: Stores information about Workbench commands and workers and help text into the system s information.


Instruction:
Can you generate the documentation for the following Python 3 function
def _new_plugin(self, plugin):

        # First store the plugin info into our data store
        self.store_info(plugin, plugin['name'], type_tag='worker')

        # Place it into our active plugin list
        self.plugin_meta[plugin['name']] = plugin

Response: Internal method for handling new plugins.


Instruction:
Can you tell what is the following Python 3 function doing
def _store_work_results(self, results, collection, md5):
        self.data_store.store_work_results(results, collection, md5)

Response: Internal method to store the work results in the data store.


Instruction:
How would you explain what the following Python 3 function does
def _recursive_work_resolver(self, worker_name, md5):

        # Looking for the sample?
        if worker_name == 'sample':
            return self.get_sample(md5)

        # Looking for info?
        if worker_name == 'info':
            return self._get_work_results('info', md5)

        # Looking for tags?
        if worker_name == 'tags':
            return self._get_work_results('tags', md5)

        # Do I actually have this plugin? (might have failed, etc)
        if (worker_name not in self.plugin_meta):
            print 'Alert: Request for non-existing or failed plugin: %s' % (worker_name)
            return {}

        # If the results exist and the time_stamp is newer than the entire work_chain, I'm done
        collection = self.plugin_meta[worker_name]['name']
        try:
            work_results = self._get_work_results(collection, md5)
            work_chain_mod_time = self._work_chain_mod_time(worker_name)
            if work_chain_mod_time < work_results[collection]['__time_stamp']:
                return work_results
            elif self.VERBOSE:
                print 'VERBOSE: %s work_chain is newer than data' % (worker_name)
        except WorkBench.DataNotFound:
            if self.VERBOSE:
                print 'Verbose: %s data not found generating' % (worker_name)

        # Okay either need to generate (or re-generate) the work results
        dependencies = self.plugin_meta[worker_name]['dependencies']
        dependant_results = {}
        for dependency in dependencies:
            dependant_results.update(self._recursive_work_resolver(dependency, md5))
        if self.VERBOSE:
            print 'Verbose: new work for plugin: %s' % (worker_name)
        work_results = self.plugin_meta[worker_name]['class']().execute(dependant_results)

        # Enforce dictionary output
        if not isinstance(work_results, dict):
            print 'Critical: Plugin %s MUST produce a python dictionary!' % worker_name
            return None

        # Store the results and return
        self._store_work_results(work_results, collection, md5)
        return self._get_work_results(collection, md5)

Response: Internal method that recursively generates the work results for the given worker_name and md5.


Instruction:
Given the following Python 3 function, write the documentation
def load_sample(self, file_path, tags=None):

        # Recommend a tag
        if not tags:
            print '\n%sRequired: Add a list of tags when you load samples (put \'unknown\' if you must). \
                   \n\t%sExamples: [\'bad\'], [\'good\'], [\'bad\',\'aptz13\']%s' % (color.Yellow, color.Green, color.Normal)
            return

        # Do they want everything under a directory?
        if os.path.isdir(file_path):
            file_list = self._all_files_in_directory(file_path)
        else:
            file_list = [file_path]

        # Upload the files into workbench
        md5_list = []
        for path in file_list:
            with open(path, 'rb') as my_file:
                raw_bytes = my_file.read()
                md5 = hashlib.md5(raw_bytes).hexdigest()
                if not self.workbench.has_sample(md5):
                    print '%sStreaming Sample...%s' % (color.LightPurple, color.Normal)
                    basename = os.path.basename(path)
                    md5 = self.streamer.stream_to_workbench(raw_bytes, basename, 'unknown', tags)

                print '\n%s  %s%s %sLocked and Loaded...%s\n' % \
                      (self.beer, color.LightPurple, md5[:6], color.Yellow, color.Normal)

                # Add tags to the sample
                self.workbench.add_tags(md5, tags)
                md5_list.append(md5)

        # Pivot on the sample_set
        set_md5 = self.workbench.store_sample_set(md5_list)
        self.pivot(set_md5, '_'.join(tags))

        # Dump out tag information
        self.tags()

Response: Load a sample into a workbench object


Instruction:
Here you have a function in Python 3, explain what it does
def pivot(self, md5, tag=''):

        # Is the md5 a tag?
        ss = self.workbench.generate_sample_set(md5)
        if ss:
            tag = md5 if not tag else tag
            md5 = ss

        # Is the md5 a sample_set?
        if self.workbench.is_sample_set(md5):

            # Is the sample_set one sample?
            ss = self.workbench.get_sample_set(md5)
            if len(ss) == 1:
                md5 = ss[0]
            deco = '(%s:%d)' % (tag, len(ss))
            self.ipshell.push({'prompt_deco': deco})
        else:
            deco = '(%s:1)' % tag
            self.ipshell.push({'prompt_deco': deco})

        # Set the new md5
        self.session.md5 = md5
        self.session.short_md5 = md5[:6]
        self.ipshell.push({'md5': self.session.md5})
        self.ipshell.push({'short_md5': self.session.short_md5})

Response: Pivot on an md5


Instruction:
Given the following Python 3 function, write the documentation
def tags(self):
        '''Display tag information for all samples in database'''
        tags = self.workbench.get_all_tags()
        if not tags:
            return
        tag_df = pd.DataFrame(tags)
        tag_df = self.vectorize(tag_df, 'tags')
        print '\n%sSamples in Database%s' % (color.LightPurple, color.Normal)
        self.top_corr(tag_df)

Response: Display tag information for all samples in database


Instruction:
Here you have a function in Python 3, explain what it does
def pull_df(self, md5):
        try:
            _packed_df = self.workbench.get_dataframe(md5)
            _df = pd.read_msgpack(lz4.loads(_packed_df))
            return _df
        except zerorpc.exceptions.RemoteError as e:
            return repr_to_str_decorator.r_to_s(self._data_not_found)(e)

Response: Wrapper for the Workbench get_dataframe method


Instruction:
Can you tell what is the following Python 3 function doing
def flatten(self, df, column_name):
        _exp_list = [[md5, x] for md5, value_list in zip(df['md5'], df[column_name]) for x in value_list]
        return pd.DataFrame(_exp_list, columns=['md5',column_name])

Response: Flatten a column in the dataframe that contains lists


Instruction:
Can you generate a brief explanation for the following Python 3 code
def top_corr(self, df):
        tag_freq = df.sum()
        tag_freq.sort(ascending=False)

        corr = df.corr().fillna(1)
        corr_dict = corr.to_dict()
        for tag, count in tag_freq.iteritems():
            print '  %s%s: %s%s%s  (' % (color.Green, tag, color.LightBlue, count, color.Normal),
            tag_corrs = sorted(corr_dict[tag].iteritems(), key=operator.itemgetter(1), reverse=True)
            for corr_tag, value in tag_corrs[:5]:
                if corr_tag != tag and (value > .2):
                    print '%s%s:%s%.1f' % (color.Green, corr_tag, color.LightBlue, value),
            print '%s)' % color.Normal

Response: Give aggregation counts and correlations


Instruction:
Can you generate the documentation for the following Python 3 function
def search(self, tags=None):
        if isinstance(tags, str):
            tags = [tags]
        return self.workbench.generate_sample_set(tags)

Response: Wrapper for the Workbench search method


Instruction:
Can you generate the documentation for the following Python 3 function
def versions(self):
        print '%s<<< Workbench CLI Version %s >>>%s' % (color.LightBlue, self.version, color.Normal)
        print self.workbench.help('version')

Response: Announce Versions of CLI and Server


Instruction:
Given the following Python 3 function, write the documentation
def run(self):
        ''' Running the workbench CLI '''

        # Announce versions
        self.versions()

        # Sample/Tag info and Help
        self.tags()
        print '\n%s' % self.workbench.help('cli')

        # Now that we have the Workbench connection spun up, we register some stuff
        # with the embedded IPython interpreter and than spin it up
        # cfg = IPython.config.loader.Config()
        cfg = Config()
        cfg.InteractiveShellEmbed.autocall = 2
        cfg.InteractiveShellEmbed.colors = 'Linux'
        cfg.InteractiveShellEmbed.color_info = True
        cfg.InteractiveShellEmbed.autoindent = True
        cfg.InteractiveShellEmbed.deep_reload = True
        cfg.PromptManager.in_template = (
            r'{color.LightPurple}{short_md5}{color.Yellow}{prompt_deco}{color.LightBlue} Workbench{color.Green}[\#]> ')
        # cfg.PromptManager.out_template = ''

        # Create the IPython shell
        self.ipshell = IPython.terminal.embed.InteractiveShellEmbed(
            config=cfg, banner1='', exit_msg='\nWorkbench has SuperCowPowers...')

        # Register our transformer, the shell will use this for 'shortcut' commands
        auto_quoter = auto_quote_xform.AutoQuoteTransformer(self.ipshell, self.ipshell.prefilter_manager)
        auto_quoter.register_command_set(self.command_set)

        # Setting up some Pandas options
        pd.set_option('display.width', 140)
        pd.set_option('max_colwidth', 15)

        # Start up the shell with our set of workbench commands
        self.ipshell(local_ns=self.command_dict)

Response: Runs the workbench CLI


Instruction:
How would you explain what the following Python 3 function does
def _connect(self, server_info):

        # First we do a temp connect with a short heartbeat
        _tmp_connect = zerorpc.Client(timeout=300, heartbeat=2)
        _tmp_connect.connect('tcp://'+server_info['server']+':'+server_info['port'])
        try:
            _tmp_connect._zerorpc_name()
            _tmp_connect.close()
            del _tmp_connect
        except zerorpc.exceptions.LostRemote:
            print '%sError: Could not connect to Workbench Server at %s:%s%s' % \
                  (color.Red, server_info['server'], server_info['port'], color.Normal)
            sys.exit(1)

        # Okay do the real connection
        if self.workbench:
            self.workbench.close()
        self.workbench = zerorpc.Client(timeout=300, heartbeat=60)
        self.workbench.connect('tcp://'+server_info['server']+':'+server_info['port'])
        print '\n%s<<< Connected: %s:%s >>>%s' % (color.Green, server_info['server'], server_info['port'], color.Normal)

Response: Connect to the workbench server


Instruction:
Given the following Python 3 function, write the documentation
def _progress_print(self, sent, total):
        percent = min(int(sent*100.0/total), 100)
        sys.stdout.write('\r{0}[{1}{2}] {3}{4}%{5}'.
                         format(color.Green, '#'*(percent/2), ' '*(50-percent/2), color.Yellow, percent, color.Normal))
        sys.stdout.flush()

Response: Print the current progress of the current upload.


Instruction:
Write a Python 3 function that can
wrap for a work_request to workbench

Response:
def _work_request(self, worker, md5=None):
        """Wrapper for a work_request to workbench"""

        # I'm sure there's a better way to do this
        if not md5 and not self.session.md5:
            return 'Must call worker with an md5 argument...'
        elif not md5:
            md5 = self.session.md5

        # Is the md5 a sample_set?
        if self.workbench.is_sample_set(md5):
            return self.workbench.set_work_request(worker, md5)

        # Make the work_request with worker and md5 args
        try:
            return self.workbench.work_request(worker, md5)
        except zerorpc.exceptions.RemoteError as e:
            return repr_to_str_decorator.r_to_s(self._data_not_found)(e)


Instruction:
Here you have a function in Python 3, explain what it does
def _generate_command_dict(self):

        # First add all the workers
        commands = {}
        for worker in self.workbench.list_all_workers():
            commands[worker] = lambda md5=None, worker=worker: self._work_request(worker, md5)

        # Next add all the commands
        for command in self.workbench.list_all_commands():
            # Fixme: is there a better way to get the lambda function from ZeroRPC
            commands[command] = self.workbench.__getattr__(command)

        # Now the general commands which are often overloads
        # for some of the workbench commands
        general = {
            'workbench': self.workbench,
            'help': self._help,
            'load_sample': self.load_sample,
            'pull_df': self.pull_df,
            'flatten': self.flatten,
            'vectorize': self.vectorize,
            'top_corr': self.top_corr,
            'tags': self.tags,
            'pivot': self.pivot,
            'search': self.search,
            'reconnect': lambda info=self.server_info: self._connect(info),
            'version': self.versions,
            'versions': self.versions,
            'short_md5': self.session.short_md5,
            'prompt_deco': self.session.prompt_deco
        }
        commands.update(general)

        # Return the list of workbench commands
        return commands

Response:
Generate a dict of functions that will be used to make the workbench with a bunch of shortcuts
            and helper/alias functions that will make the shell MUCH easier.


Instruction:
Make a summary of the following Python 3 code
def _register_info(self):
        # Stores information on Workbench commands and signatures
        for name, meth in inspect.getmembers(self, predicate=inspect.isroutine):
            if not name.startswith('_') and name != 'run':
                info = {'command': name, 'sig': str(funcsigs.signature(meth)), 'docstring': meth.__doc__}
                self.workbench.store_info(info, name, 'command')

        # Register help information
        self.workbench.store_info({'help': self.help.help_cli()}, 'cli', 'help')
        self.workbench.store_info({'help': self.help.help_cli_basic()}, 'cli_basic', 'help')
        self.workbench.store_info({'help': self.help.help_cli_search()}, 'search', 'help')
        self.workbench.store_info({'help': self.help.help_dataframe()}, 'dataframe', 'help')
        self.workbench.store_info({'help': self.help.help_dataframe_memory()}, 'dataframe_memory', 'help')
        self.workbench.store_info({'help': self.help.help_dataframe_pe()}, 'dataframe_pe', 'help')

Response: Register local methods in the Workbench Information system


Instruction:
Can you generate the documentation for the following Python 3 function
def transform(self, line, _continue_prompt):

        # Capture the original line
        orig_line = line

        # Get tokens from all the currently active namespace
        ns_token_set = set([token for nspace in self.shell.all_ns_refs for token in nspace])

        # Build up token set and info out of the incoming line
        token_list = re.split(' |;|,|(|)|\'|"', line)
        token_list = [item for item in token_list if item != None and item != '']
        num_tokens = len(token_list)
        first_token = token_list[0]
        token_set = set(token_list)

        # Very conservative logic (but possibly flawed)
        # 1) Lines with any of these symbols ; , ' " ( ) aren't touched
        # 2) Need to have more than one token
        # 3) First token in line must be in the workbench command set
        # 4) If first token is 'help' than all other tokens are quoted
        # 5) Otherwise only tokens that are not in any of the namespace are quoted

        # Fixme: Horse shit temp hack for load_sample
        # 0) If load_sample do special processing
        if first_token == 'load_sample':
            # If the second arg isn't in namespace quote it
            if token_list[1] not in ns_token_set:
                line = line.replace(token_list[1], '"'+token_list[1]+'",')
            return line

        # Fixme: Horse shit temp hack for pivot
        # 0) If pivot do special processing
        if first_token == 'pivot':
            # Quote all other tokens
            for token in token_list:
                if token not in ns_token_set:
                    line = line.replace(token, '"' + token + '",')
            return line

        # 1) Lines with any of these symbols ; , ' " ( ) aren't touched
        skip_symbols = [';', ',', '\'', '"', '(', ')']
        if any([sym in line for sym in skip_symbols]):
            return line

        # 2) Need to have more than one token
        # 3) First token in line must be in the workbench command set
        if num_tokens > 1 and first_token in self.command_set:

            # 4) If first token is 'help' than all other tokens are quoted
            if first_token == 'help':
                token_set.remove('help')
                for token in token_set:
                    line = line.replace(token, '"'+token+'"')

            # 5) Otherwise only tokens that are not in any of the namespace are quoted
            else: # Not help
                for token in token_set:
                    if token not in ns_token_set:
                        line = line.replace(token, '"'+token+'"')

        # Return the processed line
        return line

Response: Shortcut Workbench commands by using auto - quotes


Instruction:
Can you generate the documentation for the following Python 3 function
def decodes(self, s: str) -> BioCCollection:
        tree = etree.parse(io.BytesIO(bytes(s, encoding='UTF-8')))
        collection = self.__parse_collection(tree.getroot())
        collection.encoding = tree.docinfo.encoding
        collection.standalone = tree.docinfo.standalone
        collection.version = tree.docinfo.xml_version
        return collection

Response: Deserialize a string representation of a BioC collection into a BioC object.


Instruction:
Can you generate the documentation for the following Python 3 function
def decode(self, fp: TextIO) -> BioCCollection:
        # utf8_parser = etree.XMLParser(encoding='utf-8')
        tree = etree.parse(fp)
        collection = self.__parse_collection(tree.getroot())
        collection.encoding = tree.docinfo.encoding
        collection.standalone = tree.docinfo.standalone
        collection.version = tree.docinfo.xml_version
        return collection

Response: Deserialize a file - like object containing a BioC collection.


Instruction:
Implement a function in Python 3 to
execute the plugin and return the output

Response:
def execute(self, input_data):
        ''' Execute method '''

        # Grab the raw bytes of the sample
        raw_bytes = input_data['sample']['raw_bytes']

        # Spin up the rekall session and render components
        session = MemSession(raw_bytes)
        renderer = WorkbenchRenderer(session=session)

        # Run the plugin
        session.RunPlugin(self.plugin_name, renderer=renderer)

        return renderer.get_output()


Instruction:
Explain what the following Python 3 code does
def process_row(cls, data, column_map):
        row = {}
        for key,value in data.iteritems():
            if not value:
                value = '-'
            elif isinstance(value, list):
                value = value[1]
            elif isinstance(value, dict):
                if 'type_name' in value:
                    if 'UnixTimeStamp' in value['type_name']:
                        value = datetime.datetime.utcfromtimestamp(value['epoch'])
                        if value == datetime.datetime(1970, 1, 1, 0, 0):
                            value = '-'

            # Assume the value is somehow well formed when we get here
            row[column_map[key]] = value
        return row

Response: Process the row data from Rekall


Instruction:
How would you explain what the following Python 3 function does
def start(self, plugin_name=None, kwargs=None):
        self.output = [] # Start basically resets the output data
        super(WorkbenchRenderer, self).start(plugin_name=plugin_name)
        return self

Response: Start method that initial data structures and store some meta data.


Instruction:
Explain what the following Python 3 code does
def format(self, formatstring, *args):

        # Make a new section
        if self.incoming_section:
            self.SendMessage(['s', {'name': args}])
            self.incoming_section = False

Response: Presentation Information from the Plugin


Instruction:
How would you code a function in Python 3 to
open a file for writing or reading.

Response:
def open(self, directory=None, filename=None, mode="rb"):
        """Opens a file for writing or reading."""
        path = os.path.join(directory, filename)
        return open(path, mode)


Instruction:
Implement a Python 3 function for
yielding compressed chunks from a data array

Response:
def _file_chunks(self, data, chunk_size):
        """ Yield compressed chunks from a data array"""
        for i in xrange(0, len(data), chunk_size):
            yield self.compressor(data[i:i+chunk_size])


Instruction:
How would you explain what the following Python 3 function does
def stream_to_workbench(self, raw_bytes, filename, type_tag, tags):
        md5_list = []
        sent_bytes = 0
        total_bytes = len(raw_bytes)
        for chunk in self._file_chunks(raw_bytes, self.chunk_size):
            md5_list.append(self.workbench.store_sample(chunk, filename, self.compress_ident))
            sent_bytes += self.chunk_size
            self.progress(sent_bytes, total_bytes)

        # Now we just ask Workbench to combine these
        full_md5 = self.workbench.combine_samples(md5_list, filename, type_tag)

        # Add the tags
        self.workbench.add_tags(full_md5, tags)

        # Return the md5 of the finalized sample
        return full_md5

Response: Stream a large file into chunks and send them to Workbench


Instruction:
How would you implement a function in Python 3 that
serializes a BioC object to a JSON formatted str.

Response:
def dumps(obj, **kwargs) -> str:
    """
    Serialize a BioC ``obj`` to a JSON formatted ``str``.
    """
    return json.dumps(obj, cls=BioCJSONEncoder, **kwargs)


Instruction:
How would you code a function in Python 3 to
serialize obj as a JSON formatted stream to fp.

Response:
def dump(obj, fp, **kwargs):
    """
    Serialize obj as a JSON formatted stream to fp (a .write()-supporting file-like object)
    """
    return json.dump(obj, fp, cls=BioCJSONEncoder, **kwargs)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def write(self, obj: BioCDocument or BioCPassage or BioCSentence):
        if self.level == DOCUMENT and not isinstance(obj, BioCDocument):
            raise ValueError
        if self.level == PASSAGE and not isinstance(obj, BioCPassage):
            raise ValueError
        if self.level == SENTENCE and not isinstance(obj, BioCSentence):
            raise ValueError
        self.writer.write(BioCJSONEncoder().default(obj))

Response: Encode and write a single object.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def execute(self, input_data):
        ''' Execute method '''

        # Spin up the rekall adapter
        adapter = RekallAdapter()
        adapter.set_plugin_name(self.plugin_name)
        rekall_output = adapter.execute(input_data)

        # Process the output data
        for line in rekall_output:

            if line['type'] == 'm':  # Meta
                self.output['meta'] = line['data']
            elif line['type'] == 's': # New Session (Table)
                self.current_table_name = line['data']['name'][1]
            elif line['type'] == 't': # New Table Headers (column names)
                self.column_map = {item['cname']: item['name'] if 'name' in item else item['cname'] for item in line['data']}
            elif line['type'] == 'r': # Row

                # Add the row to our current table
                row = RekallAdapter.process_row(line['data'], self.column_map)
                self.output['tables'][self.current_table_name].append(row)
            else:
                print 'Note: Ignoring rekall message of type %s: %s' % (line['type'], line['data'])

        # All done
        return self.output

Response: Execute the rekall adapter and return the output


Instruction:
Can you generate the documentation for the following Python 3 function
def execute(self, input_data):
        ''' Execute the ViewMemoryDeep worker '''

        # Aggregate the output from all the memory workers, clearly this could be kewler
        output = input_data['view_memory']
        output['tables'] = {}
        for data in [input_data[key] for key in ViewMemoryDeep.dependencies]:
            for name,table in data['tables'].iteritems():
                output['tables'].update({name: table})
        return output

Response: Execute the ViewMemoryDeep worker


Instruction:
Here you have a function in Python 3, explain what it does
def execute(self, input_data):
        ''' Execute the ViewMemory worker '''

        # Aggregate the output from all the memory workers into concise summary info
        output = {'meta': input_data['mem_meta']['tables']['info']}
        output['connscan'] = list(set([item['Remote Address'] for item in input_data['mem_connscan']['tables']['connscan']]))
        pslist_md5s = {self.file_to_pid(item['filename']): item['md5'] for item in input_data['mem_procdump']['tables']['dumped_files']}
        output['pslist'] = ['PPID: %d PID: %d Name: %s - %s' % (item['PPID'], item['PID'], item['Name'], pslist_md5s[item['PID']])
                            for item in input_data['mem_pslist']['tables']['pslist']]
        return output

Response: Execute the ViewMemory worker


Instruction:
Can you write a function in Python 3 where it
stores a config value in a dictionary

Response:
def store(self, name, value, atype, new_name=None, multiplier=None, allowed_values=None):
        ''' store a config value in a dictionary, these values are used to populate a trasnfer spec
            validation -- check type, check allowed values and rename if required '''
        if value is not None:
            _bad_type = (not isinstance(value, atype))
            if not _bad_type:
                # special case
                _bad_type = (isinstance(value, bool) and atype == int)

            if _bad_type:
                # could be a special value
                if allowed_values and value in allowed_values:
                    allowed_values = None
                else:
                    raise ValueError("%s should be value of type (%s)" % (name, atype.__name__))

            if allowed_values:
                if isinstance(value, str):
                    if value not in allowed_values:
                        raise ValueError("%s can be %s" % (name, allowed_values))
                elif isinstance(value, int):
                    if isinstance(allowed_values[0], int):
                        if value < allowed_values[0]:
                            raise ValueError("%s must be >= %d" % (name, allowed_values[0]))

            _val = value if not multiplier else (multiplier * value)
            _name = name if not new_name else new_name
            self._dict[_name] = _val


Instruction:
Given the following Python 3 function, write the documentation
def multi_session(self):
        ''' convert the multi_session param a number '''
        _val = 0
        if "multi_session" in self._dict:
            _val = self._dict["multi_session"]
            if str(_val).lower() == 'all':
                _val = -1

        return int(_val)

Response: convert the multi_session param a number


Instruction:
How would you code a function in Python 3 to
get the Aspera connection details on Aspera enabled buckets

Response:
def _raw_aspera_metadata(self, bucket):
        ''' get the Aspera connection details on Aspera enabled buckets '''
        response = self._client.get_bucket_aspera(Bucket=bucket)

        # Parse metadata from response
        aspera_access_key = response['AccessKey']['Id']
        aspera_secret_key = response['AccessKey']['Secret']
        ats_endpoint = response['ATSEndpoint']

        return aspera_access_key, aspera_secret_key, ats_endpoint


Instruction:
How would you implement a function in Python 3 that
makes hhtp call to Aspera to fetch back trasnfer spec

Response:
def _fetch_transfer_spec(self, node_action, token, bucket_name, paths):
        ''' make hhtp call to Aspera to fetch back trasnfer spec '''
        aspera_access_key, aspera_secret_key, ats_endpoint = self._get_aspera_metadata(bucket_name)

        _headers = {'accept': "application/json",
                    'Content-Type': "application/json"}

        credentials = {'type': 'token',
                       'token': {'delegated_refresh_token': token}}

        _url = ats_endpoint
        _headers['X-Aspera-Storage-Credentials'] = json.dumps(credentials)
        _data = {'transfer_requests': [
                {'transfer_request': {'paths': paths, 'tags': {'aspera': {
                 'node': {'storage_credentials': credentials}}}}}]}

        _session = requests.Session()
        _response = _session.post(url=_url + "/files/" + node_action,
                                  auth=(aspera_access_key, aspera_secret_key),
                                  headers=_headers, json=_data, verify=self._config.verify_ssl)
        return _response