Instruction:
Can you tell what is the following Python 3 function doing
def order_categories(unordered_cats):
  '''
  If categories are strings, then simple ordering is fine.
  If categories are values then I'll need to order based on their values.
  The final ordering is given as the original categories (including titles) in a
  ordered list.
  '''

  no_titles = remove_titles(unordered_cats)

  all_are_numbers = check_all_numbers(no_titles)

  if all_are_numbers:
    ordered_cats = order_cats_based_on_values(unordered_cats, no_titles)
  else:
    ordered_cats = sorted(unordered_cats)

  return ordered_cats

Response:
Order the categories in a
.


Instruction:
Can you write a function in Python 3 where it
loads file as a string.

Response:
def load_file_as_string(self, file_string, filename=''):
    '''
    Load file as a string.
    '''
    load_data.load_file_as_string(self, file_string, filename=filename)


Instruction:
Write a Python 3 function for
loading Clustergrammer s dat format saved as JSON.

Response:
def load_data_file_to_net(self, filename):
    '''
    Load Clustergrammer's dat format (saved as JSON).
    '''
    inst_dat = self.load_json_to_dict(filename)
    load_data.load_data_to_net(self, inst_dat)


Instruction:
How would you implement a function in Python 3 that
loads Pandas DataFrame. Load Pandas DataFrame.

Response:
def load_df(self, df):
    '''
    Load Pandas DataFrame.
    '''
    # self.__init__()
    self.reset()

    df_dict = {}
    df_dict['mat'] = deepcopy(df)
    # always define category colors if applicable when loading a df
    data_formats.df_to_dat(self, df_dict, define_cat_colors=True)


Instruction:
Write a Python 3 script to
load Pandas DataFrame into a dat file.

Response:
def df_to_dat(self, df, define_cat_colors=False):
    '''
    Load Pandas DataFrame (will be deprecated).
    '''
    data_formats.df_to_dat(self, df, define_cat_colors)


Instruction:
Given the following Python 3 function, write the documentation
def widget(self, which_viz='viz'):
    '''
    Generate a widget visualization using the widget. The export_viz_to_widget
    method passes the visualization JSON to the instantiated widget, which is
    returned and visualized on the front-end.
    '''
    if hasattr(self, 'widget_class') == True:

      # run clustering if necessary
      if len(self.viz['row_nodes']) == 0:
        self.cluster()

      self.widget_instance = self.widget_class(network = self.export_viz_to_widget(which_viz))

      return self.widget_instance
    else:
      print('Can not make widget because Network has no attribute widget_class')
      print('Please instantiate Network with clustergrammer_widget using: Network(clustergrammer_widget)')

Response: Generate a widget visualization using the widget class


Instruction:
How would you explain what the following Python 3 function does
def widget_df(self):
    '''
    Export a DataFrame from the front-end visualization. For instance, a user
    can filter to show only a single cluster using the dendrogram and then
    get a dataframe of this cluster using the widget_df method.
    '''

    if hasattr(self, 'widget_instance') == True:

      if self.widget_instance.mat_string != '':

        tmp_net = deepcopy(Network())

        df_string = self.widget_instance.mat_string

        tmp_net.load_file_as_string(df_string)

        df = tmp_net.export_df()

        return df

      else:
        return self.export_df()

    else:
      if hasattr(self, 'widget_class') == True:
        print('Please make the widget before exporting the widget DataFrame.')
        print('Do this using the widget method: net.widget()')

      else:
        print('Can not make widget because Network has no attribute widget_class')
        print('Please instantiate Network with clustergrammer_widget using: Network(clustergrammer_widget)')

Response: Export a DataFrame from the front - end visualization.


Instruction:
Can you tell what is the following Python 3 function doing
def write_json_to_file(self, net_type, filename, indent='no-indent'):
    '''
    Save dat or viz as a JSON to file.
    '''
    export_data.write_json_to_file(self, net_type, filename, indent)

Response: Save dat or viz as a JSON to file.


Instruction:
How would you implement a function in Python 3 that
filters a network based on the sum across rows or columns.

Response:
def filter_sum(self, inst_rc, threshold, take_abs=True):
    '''
    Filter a network's rows or columns based on the sum across rows or columns.
    '''
    inst_df = self.dat_to_df()
    if inst_rc == 'row':
      inst_df = run_filter.df_filter_row_sum(inst_df, threshold, take_abs)
    elif inst_rc == 'col':
      inst_df = run_filter.df_filter_col_sum(inst_df, threshold, take_abs)
    self.df_to_dat(inst_df)


Instruction:
How would you implement a function in Python 3 that
filters the matrix rows or columns based on sum / variance and only keep the top N.

Response:
def filter_N_top(self, inst_rc, N_top, rank_type='sum'):
    '''
    Filter the matrix rows or columns based on sum/variance, and only keep the top
    N.
    '''
    inst_df = self.dat_to_df()

    inst_df = run_filter.filter_N_top(inst_rc, inst_df, N_top, rank_type)

    self.df_to_dat(inst_df)


Instruction:
Here you have a function in Python 3, explain what it does
def filter_threshold(self, inst_rc, threshold, num_occur=1):
    '''
    Filter the matrix rows or columns based on num_occur values being above a
    threshold (in absolute value).
    '''
    inst_df = self.dat_to_df()

    inst_df = run_filter.filter_threshold(inst_df, inst_rc, threshold,
      num_occur)

    self.df_to_dat(inst_df)

Response: Filter the matrix rows or columns based on num_occur values being above threshold.


Instruction:
Create a Python 3 function for
filtering the matrix based on their category.

Response:
def filter_cat(self, axis, cat_index, cat_name):
    '''
    Filter the matrix based on their category. cat_index is the index of the category, the first category has index=1.
    '''
    run_filter.filter_cat(self, axis, cat_index, cat_name)


Instruction:
Can you tell what is the following Python 3 function doing
def clip(self, lower=None, upper=None):
    '''
    Trim values at input thresholds using pandas function
    '''
    df = self.export_df()
    df = df.clip(lower=lower, upper=upper)
    self.load_df(df)

Response: Trim values at input thresholds using pandas function


Instruction:
How would you implement a function in Python 3 that
normalizes the matrix rows or columns using Z - score or Quantile Normalization.

Response:
def normalize(self, df=None, norm_type='zscore', axis='row', keep_orig=False):
    '''
    Normalize the matrix rows or columns using Z-score (zscore) or Quantile Normalization (qn). Users can optionally pass in a DataFrame to be normalized (and this will be incorporated into the Network object).
    '''
    normalize_fun.run_norm(self, df, norm_type, axis, keep_orig)


Instruction:
Explain what the following Python 3 code does
def random_sample(self, num_samples, df=None, replace=False, weights=None, random_state=100, axis='row'):
    '''
    Return random sample of matrix.
    '''

    if df is None:
      df = self.dat_to_df()

    if axis == 'row':
      axis = 0
    if axis == 'col':
      axis = 1

    df = self.export_df()
    df = df.sample(n=num_samples, replace=replace, weights=weights, random_state=random_state,  axis=axis)

    self.load_df(df)

Response: Return random samples from the matrix.


Instruction:
Write a Python 3 function that can
add categories to rows or columns of the specified axis.

Response:
def add_cats(self, axis, cat_data):
    '''
    Add categories to rows or columns using cat_data array of objects. Each object in cat_data is a dictionary with one key (category title) and value (rows/column names) that have this category. Categories will be added onto the existing categories and will be added in the order of the objects in the array.

    Example ``cat_data``::

        [
          {
            "title": "First Category",
            "cats": {
              "true": [
                "ROS1",
                "AAK1"
              ]
            }
          },
          {
            "title": "Second Category",
            "cats": {
              "something": [
                "PDK4"
              ]
            }
          }
        ]

    '''
    for inst_data in cat_data:
      categories.add_cats(self, axis, inst_data)


Instruction:
Can you tell what is the following Python 3 function doing
def enrichrgram(self, lib, axis='row'):
    '''
    Add Enrichr gene enrichment results to your visualization (where your rows
    are genes). Run enrichrgram before clustering to incldue enrichment results
    as row categories. Enrichrgram can also be run on the front-end using the
    Enrichr logo at the top left.

    Set lib to the Enrichr library that you want to use for enrichment analysis.
    Libraries included:

      * ChEA_2016
      * KEA_2015
      * ENCODE_TF_ChIP-seq_2015
      * ENCODE_Histone_Modifications_2015
      * Disease_Perturbations_from_GEO_up
      * Disease_Perturbations_from_GEO_down
      * GO_Molecular_Function_2015
      * GO_Biological_Process_2015
      * GO_Cellular_Component_2015
      * Reactome_2016
      * KEGG_2016
      * MGI_Mammalian_Phenotype_Level_4
      * LINCS_L1000_Chem_Pert_up
      * LINCS_L1000_Chem_Pert_down

    '''

    df = self.export_df()
    df, bar_info = enr_fun.add_enrichr_cats(df, axis, lib)
    self.load_df(df)

    self.dat['enrichrgram_lib'] = lib
    self.dat['row_cat_bars'] = bar_info

Response: Add enrichrgram results to the visualization.


Instruction:
Write a Python 3 script for
loading gene expression data from 10x in sparse matrix format and returns a Pandas dataframe containing the gene name and unique id.

Response:
def load_gene_exp_to_df(inst_path):
    '''
    Loads gene expression data from 10x in sparse matrix format and returns a
    Pandas dataframe
    '''

    import pandas as pd
    from scipy import io
    from scipy import sparse
    from ast import literal_eval as make_tuple

    # matrix
    Matrix = io.mmread( inst_path + 'matrix.mtx')
    mat = Matrix.todense()

    # genes
    filename = inst_path + 'genes.tsv'
    f = open(filename, 'r')
    lines = f.readlines()
    f.close()

    # # add unique id to all genes
    # genes = []
    # unique_id = 0
    # for inst_line in lines:
    #     inst_line = inst_line.strip().split()

    #     if len(inst_line) > 1:
    #       inst_gene = inst_line[1]
    #     else:
    #       inst_gene = inst_line[0]

    #     genes.append(inst_gene + '_' + str(unique_id))
    #     unique_id = unique_id + 1

    # add unique id only to duplicate genes
    ini_genes = []
    for inst_line in lines:
        inst_line = inst_line.strip().split()
        if len(inst_line) > 1:
          inst_gene = inst_line[1]
        else:
          inst_gene = inst_line[0]
        ini_genes.append(inst_gene)

    gene_name_count = pd.Series(ini_genes).value_counts()
    duplicate_genes = gene_name_count[gene_name_count > 1].index.tolist()

    dup_index = {}
    genes = []
    for inst_row in ini_genes:

      # add index to non-unique genes
      if inst_row in duplicate_genes:

        # calc_non-unque index
        if inst_row not in dup_index:
          dup_index[inst_row] = 1
        else:
          dup_index[inst_row] = dup_index[inst_row] + 1

        new_row = inst_row + '_' + str(dup_index[inst_row])

      else:
        new_row = inst_row

      genes.append(new_row)

    # barcodes
    filename = inst_path + 'barcodes.tsv'
    f = open(filename, 'r')
    lines = f.readlines()
    f.close()

    cell_barcodes = []
    for inst_bc in lines:
        inst_bc = inst_bc.strip().split('\t')

        # remove dash from barcodes if necessary
        if '-' in inst_bc[0]:
          inst_bc[0] = inst_bc[0].split('-')[0]

        cell_barcodes.append(inst_bc[0])

    # parse tuples if necessary
    try:
        cell_barcodes = [make_tuple(x) for x in cell_barcodes]
    except:
        pass

    try:
        genes = [make_tuple(x) for x in genes]
    except:
        pass

    # make dataframe
    df = pd.DataFrame(mat, index=genes, columns=cell_barcodes)

    return df


Instruction:
How would you code a function in Python 3 to
calculate the similarity of samples from the same and different categories.

Response:
def sim_same_and_diff_category_samples(self, df, cat_index=1, dist_type='cosine',
                                         equal_var=False, plot_roc=True,
                                         precalc_dist=False, calc_roc=True):
      '''
      Calculate the similarity of samples from the same and different categories. The
      cat_index gives the index of the category, where 1 in the first category
      '''

      cols = df.columns.tolist()

      if type(precalc_dist) == bool:
          # compute distnace between rows (transpose to get cols as rows)
          dist_arr = 1 - pdist(df.transpose(), metric=dist_type)
      else:
          dist_arr = precalc_dist

      # generate sample names with categories
      sample_combos = list(combinations(range(df.shape[1]),2))

      sample_names = [str(ind) + '_same' if cols[x[0]][cat_index] == cols[x[1]][cat_index] else str(ind) + '_different' for ind, x in enumerate(sample_combos)]

      ser_dist = pd.Series(data=dist_arr, index=sample_names)

      # find same-cat sample comparisons
      same_cat = [x for x in sample_names if x.split('_')[1] == 'same']

      # find diff-cat sample comparisons
      diff_cat = [x for x in sample_names if x.split('_')[1] == 'different']

      # make series of same and diff category sample comparisons
      ser_same = ser_dist[same_cat]
      ser_same.name = 'Same Category'
      ser_diff = ser_dist[diff_cat]
      ser_diff.name = 'Different Category'

      sim_dict = {}
      roc_data = {}
      sim_data = {}

      sim_dict['same'] = ser_same
      sim_dict['diff'] = ser_diff

      pval_dict = {}
      ttest_stat, pval_dict['ttest'] = ttest_ind(ser_diff, ser_same, equal_var=equal_var)

      ttest_stat, pval_dict['mannwhitney'] = mannwhitneyu(ser_diff, ser_same)

      if calc_roc:
          # calc AUC
          true_index = list(np.ones(sim_dict['same'].shape[0]))
          false_index = list(np.zeros(sim_dict['diff'].shape[0]))
          y_true = true_index + false_index

          true_val = list(sim_dict['same'].get_values())
          false_val = list(sim_dict['diff'].get_values())
          y_score = true_val + false_val

          fpr, tpr, thresholds = roc_curve(y_true, y_score)

          inst_auc = auc(fpr, tpr)

          if plot_roc:
              plt.figure()
              plt.plot(fpr, tpr)
              plt.plot([0, 1], [0, 1], color='navy', linestyle='--')
              plt.figure(figsize=(10,10))

              print('AUC', inst_auc)

          roc_data['true'] = y_true
          roc_data['score'] = y_score
          roc_data['fpr'] = fpr
          roc_data['tpr'] = tpr
          roc_data['thresholds'] = thresholds
          roc_data['auc'] = inst_auc

      sim_data['sim_dict'] = sim_dict
      sim_data['pval_dict'] = pval_dict
      sim_data['roc_data'] = roc_data

      return sim_data


Instruction:
Given the following Python 3 function, write the documentation
def generate_signatures(self, df_ini, category_level, pval_cutoff=0.05,
                          num_top_dims=False, verbose=True, equal_var=False):

      ''' Generate signatures for column categories '''

      df_t = df_ini.transpose()

      # remove columns with constant values
      df_t = df_t.loc[:, (df_t != df_t.iloc[0]).any()]

      df = self.row_tuple_to_multiindex(df_t)

      cell_types = sorted(list(set(df.index.get_level_values(category_level).tolist())))

      keep_genes = []
      keep_genes_dict = {}
      gene_pval_dict = {}
      all_fold_info = {}

      for inst_ct in cell_types:

          inst_ct_mat = df.xs(key=inst_ct, level=category_level)
          inst_other_mat = df.drop(inst_ct, level=category_level)

          # save mean values and fold change
          fold_info = {}
          fold_info['cluster_mean'] = inst_ct_mat.mean()
          fold_info['other_mean'] = inst_other_mat.mean()
          fold_info['log2_fold'] = fold_info['cluster_mean']/fold_info['other_mean']
          fold_info['log2_fold'] = fold_info['log2_fold'].apply(np.log2)
          all_fold_info[inst_ct] = fold_info

          inst_stats, inst_pvals = ttest_ind(inst_ct_mat, inst_other_mat, axis=0, equal_var=equal_var)

          ser_pval = pd.Series(data=inst_pvals, index=df.columns.tolist()).sort_values()

          if num_top_dims == False:
              ser_pval_keep = ser_pval[ser_pval < pval_cutoff]
          else:
              ser_pval_keep = ser_pval[:num_top_dims]

          gene_pval_dict[inst_ct] = ser_pval_keep

          inst_keep = ser_pval_keep.index.tolist()
          keep_genes.extend(inst_keep)
          keep_genes_dict[inst_ct] = inst_keep

      keep_genes = sorted(list(set(keep_genes)))

      df_gbm = df.groupby(level=category_level).mean().transpose()
      cols = df_gbm.columns.tolist()
      new_cols = []
      for inst_col in cols:
          new_col = (inst_col, category_level + ': ' + inst_col)
          new_cols.append(new_col)
      df_gbm.columns = new_cols

      df_sig = df_gbm.ix[keep_genes]

      if len(keep_genes) == 0 and verbose:
          print('found no informative dimensions')

      df_gene_pval = pd.concat(gene_pval_dict, axis=1, sort=False)

      return df_sig, keep_genes_dict, df_gene_pval, all_fold_info

Response: Generate signatures for column categories.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def predict_cats_from_sigs(self, df_data_ini, df_sig_ini, dist_type='cosine', predict_level='Predict Category',
                             truth_level=1, unknown_thresh=-1):
      ''' Predict category using signature '''

      keep_rows = df_sig_ini.index.tolist()
      data_rows = df_data_ini.index.tolist()

      common_rows = list(set(data_rows).intersection(keep_rows))

      df_data = deepcopy(df_data_ini.ix[common_rows])
      df_sig = deepcopy(df_sig_ini.ix[common_rows])

      # calculate sim_mat of df_data and df_sig
      cell_types = df_sig.columns.tolist()
      barcodes = df_data.columns.tolist()
      sim_mat = 1 - pairwise_distances(df_sig.transpose(), df_data.transpose(), metric=dist_type)
      df_sim = pd.DataFrame(data=sim_mat, index=cell_types, columns=barcodes).transpose()

      # get the top column value (most similar signature)
      df_sim_top = df_sim.idxmax(axis=1)

      # get the maximum similarity of a cell to a cell type definition
      max_sim = df_sim.max(axis=1)

      unknown_cells = max_sim[max_sim < unknown_thresh].index.tolist()

      # assign unknown cells (need category of same name)
      df_sim_top[unknown_cells] = 'Unknown'

      # add predicted category name to top list
      top_list = df_sim_top.get_values()
      top_list = [ predict_level + ': ' + x[0] if type(x) is tuple else predict_level + ': ' + x  for x in top_list]

      # add cell type category to input data
      df_cat = deepcopy(df_data)
      cols = df_cat.columns.tolist()
      new_cols = []

      # check whether the columns have the true category available
      has_truth = False
      if type(cols[0]) is tuple:
          has_truth = True

      if has_truth:
          new_cols = [tuple(list(a) + [b]) for a,b in zip(cols, top_list)]
      else:
          new_cols = [tuple([a] + [b]) for a,b in zip(cols, top_list)]

      # transfer new categories
      df_cat.columns = new_cols

      # keep track of true and predicted labels
      y_info = {}
      y_info['true'] = []
      y_info['pred'] = []

      if has_truth:
          y_info['true'] = [x[truth_level].split(': ')[1] for x in cols]
          y_info['pred'] = [x.split(': ')[1] for x in top_list]

      return df_cat, df_sim.transpose(), y_info

Response: Predict a category using a signature.


Instruction:
Make a summary of the following Python 3 code
def confusion_matrix_and_correct_series(self, y_info):
      ''' Generate confusion matrix from y_info '''

      a = deepcopy(y_info['true'])
      true_count = dict((i, a.count(i)) for i in set(a))

      a = deepcopy(y_info['pred'])
      pred_count = dict((i, a.count(i)) for i in set(a))

      sorted_cats = sorted(list(set(y_info['true'] + y_info['pred'])))
      conf_mat = confusion_matrix(y_info['true'], y_info['pred'], sorted_cats)
      df_conf = pd.DataFrame(conf_mat, index=sorted_cats, columns=sorted_cats)

      total_correct = np.trace(df_conf)
      total_pred = df_conf.sum().sum()
      fraction_correct = total_correct/float(total_pred)

      # calculate ser_correct
      correct_list = []
      cat_counts = df_conf.sum(axis=1)
      all_cols = df_conf.columns.tolist()
      for inst_cat in all_cols:
          inst_correct = df_conf[inst_cat].loc[inst_cat] / cat_counts[inst_cat]
          correct_list.append(inst_correct)

      ser_correct = pd.Series(data=correct_list, index=all_cols)

      populations = {}
      populations['true'] = true_count
      populations['pred'] = pred_count

      return df_conf, populations, ser_correct, fraction_correct

Response: Generate a confusion matrix from y_info and correct the series.


Instruction:
Write a Python 3 script to
load data into nodes and mat convert mat to numpy array

Response:
def load_data_to_net(net, inst_net):
  ''' load data into nodes and mat, also convert mat to numpy array'''
  net.dat['nodes'] = inst_net['nodes']
  net.dat['mat'] = inst_net['mat']
  data_formats.mat_to_numpy_arr(net)


Instruction:
Here you have a function in Python 3, explain what it does
def export_net_json(net, net_type, indent='no-indent'):
  ''' export json string of dat '''
  import json
  from copy import deepcopy

  if net_type == 'dat':
    exp_dict = deepcopy(net.dat)

    if type(exp_dict['mat']) is not list:
      exp_dict['mat'] = exp_dict['mat'].tolist()
      if 'mat_orig' in exp_dict:
        exp_dict['mat_orig'] = exp_dict['mat_orig'].tolist()

  elif net_type == 'viz':
    exp_dict = net.viz

  elif net_type == 'sim_row':
    exp_dict = net.sim['row']

  elif net_type == 'sim_col':
    exp_dict = net.sim['col']

  # make json
  if indent == 'indent':
    exp_json = json.dumps(exp_dict, indent=2)
  else:
    exp_json = json.dumps(exp_dict)

  return exp_json

Response: export json string of dat


Instruction:
Can you generate a brief explanation for the following Python 3 code
def write_matrix_to_tsv(net, filename=None, df=None):
  '''
  This will export the matrix in net.dat or a dataframe (optional df in
  arguments) as a tsv file. Row/column categories will be saved as tuples in
  tsv, which can be read back into the network object.
  '''
  import pandas as pd

  if df is None:
    df = net.dat_to_df()

  return df['mat'].to_csv(filename, sep='\t')

Response: This will write the matrix in net. dat or a dataframe to a tsv file.


Instruction:
Given the following Python 3 function, write the documentation
def main(net, df=None):
  '''
  Run in load_data module (which runs when file is loaded or dataframe is loaded),
  check for duplicate row/col names, and add index to names if necesary
  '''
  if df is None:
    df = net.export_df()

  # rows
  #############
  rows = df.index.tolist()
  if type(rows[0]) is str:

    if len(rows) != len(list(set(rows))):
      new_rows = add_index_list(rows)
      df.index = new_rows

  elif type(rows[0]) is tuple:

    row_names = []
    for inst_row in rows:
      row_names.append(inst_row[0])

    if len(row_names) != len(list(set(row_names))):
      row_names = add_index_list(row_names)

      # add back to tuple
      new_rows = []
      for inst_index in range(len(rows)):
        inst_row = rows[inst_index]
        new_row = list(inst_row)
        new_row[0] = row_names[inst_index]
        new_row = tuple(new_row)
        new_rows.append(new_row)

      df.index = new_rows

  # cols
  #############
  cols = df.columns.tolist()
  if type(cols[0]) is str:

    # list column names
    if len(cols) != len(list(set(cols))):
      new_cols = add_index_list(cols)
      df.columns = new_cols

  elif type(cols[0]) is tuple:

    col_names = []
    for inst_col in cols:
      col_names.append(inst_col[0])

    if len(col_names) != len(list(set(col_names))):
      col_names = add_index_list(col_names)

      # add back to tuple
      new_cols = []
      for inst_index in range(len(cols)):
        inst_col = cols[inst_index]
        new_col = list(inst_col)
        new_col[0] = col_names[inst_index]
        new_col = tuple(new_col)
        new_cols.append(new_col)

      df.columns = new_cols

  # return dataframe with unique names
  return df

Response:
Main function for the necesary
 module.


Instruction:
How would you explain what the following Python 3 function does
def viz_json(net, dendro=True, links=False):
  ''' make the dictionary for the clustergram.js visualization '''
  from . import calc_clust
  import numpy as np

  all_dist = calc_clust.group_cutoffs()

  for inst_rc in net.dat['nodes']:

    inst_keys = net.dat['node_info'][inst_rc]
    all_cats = [x for x in inst_keys if 'cat-' in x]

    for i in range(len(net.dat['nodes'][inst_rc])):
      inst_dict = {}
      inst_dict['name'] = net.dat['nodes'][inst_rc][i]
      inst_dict['ini'] = net.dat['node_info'][inst_rc]['ini'][i]
      inst_dict['clust'] = net.dat['node_info'][inst_rc]['clust'].index(i)
      inst_dict['rank'] = net.dat['node_info'][inst_rc]['rank'][i]

      if 'rankvar' in inst_keys:
        inst_dict['rankvar'] = net.dat['node_info'][inst_rc]['rankvar'][i]

      # fix for similarity matrix
      if len(all_cats) > 0:

        for inst_name_cat in all_cats:

          actual_cat_name = net.dat['node_info'][inst_rc][inst_name_cat][i]
          inst_dict[inst_name_cat] = actual_cat_name

          check_pval = 'pval_'+inst_name_cat.replace('-','_')

          if check_pval in net.dat['node_info'][inst_rc]:
            tmp_pval_name = inst_name_cat.replace('-','_') + '_pval'
            inst_dict[tmp_pval_name] = net.dat['node_info'][inst_rc][check_pval][actual_cat_name]

          tmp_index_name = inst_name_cat.replace('-', '_') + '_index'

          inst_dict[tmp_index_name] = net.dat['node_info'][inst_rc] \
              [tmp_index_name][i]

      if len(net.dat['node_info'][inst_rc]['value']) > 0:
        inst_dict['value'] = net.dat['node_info'][inst_rc]['value'][i]

      if len(net.dat['node_info'][inst_rc]['info']) > 0:
        inst_dict['info'] = net.dat['node_info'][inst_rc]['info'][i]

      if dendro is True:
        inst_dict['group'] = []
        for tmp_dist in all_dist:
          tmp_dist = str(tmp_dist).replace('.', '')
          tmp_append = float(
              net.dat['node_info'][inst_rc]['group'][tmp_dist][i])
          inst_dict['group'].append(tmp_append)

      net.viz[inst_rc + '_nodes'].append(inst_dict)

  mat_types = ['mat', 'mat_orig', 'mat_info', 'mat_hl', 'mat_up', 'mat_dn']

  # save data as links or mat
  ###########################
  if links is True:
    for i in range(len(net.dat['nodes']['row'])):
      for j in range(len(net.dat['nodes']['col'])):

        inst_dict = {}
        inst_dict['source'] = i
        inst_dict['target'] = j
        inst_dict['value'] = float(net.dat['mat'][i, j])

        if 'mat_up' in net.dat:
          inst_dict['value_up'] = net.dat['mat_up'][i, j]
          inst_dict['value_dn'] = net.dat['mat_dn'][i, j]

        if 'mat_orig' in net.dat:
          inst_dict['value_orig'] = net.dat['mat_orig'][i, j]

          if np.isnan(inst_dict['value_orig']):
            inst_dict['value_orig'] = 'NaN'

        if 'mat_info' in net.dat:
          inst_dict['info'] = net.dat['mat_info'][str((i, j))]

        if 'mat_hl' in net.dat:
          inst_dict['highlight'] = net.dat['mat_hl'][i, j]

        net.viz['links'].append(inst_dict)

  else:
    for inst_mat in mat_types:
      if inst_mat in net.dat:
        net.viz[inst_mat] = net.dat[inst_mat].tolist()

Response: make the dictionary for the clustergram. js visualization


Instruction:
Given the following Python 3 function, write the documentation
def install_npm(path=None, build_dir=None, source_dir=None, build_cmd='build', force=False, npm=None):

    class NPM(BaseCommand):
        description = 'install package.json dependencies using npm'

        def run(self):
            if skip_npm:
                log.info('Skipping npm-installation')
                return
            node_package = path or HERE
            node_modules = pjoin(node_package, 'node_modules')
            is_yarn = os.path.exists(pjoin(node_package, 'yarn.lock'))

            npm_cmd = npm

            if npm is None:
                if is_yarn:
                    npm_cmd = ['yarn']
                else:
                    npm_cmd = ['npm']

            if not which(npm_cmd[0]):
                log.error("`{0}` unavailable.  If you're running this command "
                          "using sudo, make sure `{0}` is availble to sudo"
                          .format(npm_cmd[0]))
                return

            if force or is_stale(node_modules, pjoin(node_package, 'package.json')):
                log.info('Installing build dependencies with npm.  This may '
                         'take a while...')
                run(npm_cmd + ['install'], cwd=node_package)
            if build_dir and source_dir and not force:
                should_build = is_stale(build_dir, source_dir)
            else:
                should_build = True
            if should_build:
                run(npm_cmd + ['run', build_cmd], cwd=node_package)

    return NPM

Response: Returns a Command that can be used to install npm dependencies.


Instruction:
How would you implement a function in Python 3 that
joins paths for glob processing

Response:
def _glob_pjoin(*parts):
    """Join paths for glob processing"""
    if parts[0] in ('.', ''):
        parts = parts[1:]
    return pjoin(*parts).replace(os.sep, '/')


Instruction:
Can you generate the documentation for the following Python 3 function
def _get_data_files(data_specs, existing, top=HERE):
    # Extract the existing data files into a staging object.
    file_data = defaultdict(list)
    for (path, files) in existing or []:
        file_data[path] = files

    # Extract the files and assign them to the proper data
    # files path.
    for (path, dname, pattern) in data_specs or []:
        if os.path.isabs(dname):
            dname = os.path.relpath(dname, top)
        dname = dname.replace(os.sep, '/')
        offset = 0 if dname in ('.', '') else len(dname) + 1
        files = _get_files(_glob_pjoin(dname, pattern), top=top)
        for fname in files:
            # Normalize the path.
            root = os.path.dirname(fname)
            full_path = _glob_pjoin(path, root[offset:])
            print(dname, root, full_path, offset)
            if full_path.endswith('/'):
                full_path = full_path[:-1]
            file_data[full_path].append(fname)

    # Construct the data files spec.
    data_files = []
    for (path, files) in file_data.items():
        data_files.append((path, files))
    return data_files

Response: Expand data file specs into valid data files metadata.


Instruction:
Write a Python 3 function that can
expand file patterns to a list of paths.

Response:
def _get_files(file_patterns, top=HERE):
    """Expand file patterns to a list of paths.

    Parameters
    -----------
    file_patterns: list or str
        A list of glob patterns for the data file locations.
        The globs can be recursive if they include a `**`.
        They should be relative paths from the top directory or
        absolute paths.
    top: str
        the directory to consider for data files

    Note:
    Files in `node_modules` are ignored.
    """
    if not isinstance(file_patterns, (list, tuple)):
        file_patterns = [file_patterns]

    for i, p in enumerate(file_patterns):
        if os.path.isabs(p):
            file_patterns[i] = os.path.relpath(p, top)

    matchers = [_compile_pattern(p) for p in file_patterns]

    files = set()

    for root, dirnames, filenames in os.walk(top):
        # Don't recurse into node_modules
        if 'node_modules' in dirnames:
            dirnames.remove('node_modules')
        for m in matchers:
            for filename in filenames:
                fn = os.path.relpath(_glob_pjoin(root, filename), top)
                fn = fn.replace(os.sep, '/')
                if m(fn):
                    files.add(fn.replace(os.sep, '/'))

    return list(files)


Instruction:
Write a Python 3 function that can
expand file patterns to a list of package_data paths.

Response:
def _get_package_data(root, file_patterns=None):
    """Expand file patterns to a list of `package_data` paths.

    Parameters
    -----------
    root: str
        The relative path to the package root from `HERE`.
    file_patterns: list or str, optional
        A list of glob patterns for the data file locations.
        The globs can be recursive if they include a `**`.
        They should be relative paths from the root or
        absolute paths.  If not given, all files will be used.

    Note:
    Files in `node_modules` are ignored.
    """
    if file_patterns is None:
        file_patterns = ['*']
    return _get_files(file_patterns, _glob_pjoin(HERE, root))


Instruction:
Can you generate the documentation for the following Python 3 function
def df_filter_row_sum(df, threshold, take_abs=True):
  ''' filter rows in matrix at some threshold
  and remove columns that have a sum below this threshold '''

  from copy import deepcopy
  from .__init__ import Network
  net = Network()

  if take_abs is True:
    df_copy = deepcopy(df['mat'].abs())
  else:
    df_copy = deepcopy(df['mat'])

  ini_rows = df_copy.index.values.tolist()
  df_copy = df_copy.transpose()
  tmp_sum = df_copy.sum(axis=0)
  tmp_sum = tmp_sum.abs()
  tmp_sum.sort_values(inplace=True, ascending=False)

  tmp_sum = tmp_sum[tmp_sum > threshold]
  keep_rows = sorted(tmp_sum.index.values.tolist())

  if len(keep_rows) < len(ini_rows):
    df['mat'] = grab_df_subset(df['mat'], keep_rows=keep_rows)

    if 'mat_up' in df:
      df['mat_up'] = grab_df_subset(df['mat_up'], keep_rows=keep_rows)
      df['mat_dn'] = grab_df_subset(df['mat_dn'], keep_rows=keep_rows)

    if 'mat_orig' in df:
      df['mat_orig'] = grab_df_subset(df['mat_orig'], keep_rows=keep_rows)

  return df

Response:
filter rows in matrix at some threshold
  and remove columns that have a sum below this threshold


Instruction:
Create a Python 3 function for
filtering columns in matrix at some threshold and remove rows that have all zero values

Response:
def df_filter_col_sum(df, threshold, take_abs=True):
  ''' filter columns in matrix at some threshold
  and remove rows that have all zero values '''

  from copy import deepcopy
  from .__init__ import Network
  net = Network()

  if take_abs is True:
    df_copy = deepcopy(df['mat'].abs())
  else:
    df_copy = deepcopy(df['mat'])

  df_copy = df_copy.transpose()
  df_copy = df_copy[df_copy.sum(axis=1) > threshold]
  df_copy = df_copy.transpose()
  df_copy = df_copy[df_copy.sum(axis=1) > 0]

  if take_abs is True:
    inst_rows = df_copy.index.tolist()
    inst_cols = df_copy.columns.tolist()
    df['mat'] = grab_df_subset(df['mat'], inst_rows, inst_cols)

    if 'mat_up' in df:
      df['mat_up'] = grab_df_subset(df['mat_up'], inst_rows, inst_cols)
      df['mat_dn'] = grab_df_subset(df['mat_dn'], inst_rows, inst_cols)

    if 'mat_orig' in df:
      df['mat_orig'] = grab_df_subset(df['mat_orig'], inst_rows, inst_cols)

  else:
    df['mat'] = df_copy

  return df


Instruction:
Here you have a function in Python 3, explain what it does
def filter_threshold(df, inst_rc, threshold, num_occur=1):
  '''
  Filter a network's rows or cols based on num_occur values being above a
  threshold (in absolute_value)
  '''
  from copy import deepcopy

  inst_df = deepcopy(df['mat'])

  if inst_rc == 'col':
    inst_df = inst_df.transpose()

  inst_df = inst_df.abs()

  ini_rows = inst_df.index.values.tolist()

  inst_df[inst_df < threshold] = 0
  inst_df[inst_df >= threshold] = 1

  tmp_sum = inst_df.sum(axis=1)

  tmp_sum = tmp_sum[tmp_sum >= num_occur]

  keep_names = tmp_sum.index.values.tolist()

  if inst_rc == 'row':
    if len(keep_names) < len(ini_rows):
      df['mat'] = grab_df_subset(df['mat'], keep_rows=keep_names)

      if 'mat_up' in df:
        df['mat_up'] = grab_df_subset(df['mat_up'], keep_rows=keep_names)
        df['mat_dn'] = grab_df_subset(df['mat_dn'], keep_rows=keep_names)

      if 'mat_orig' in df:
        df['mat_orig'] = grab_df_subset(df['mat_orig'], keep_rows=keep_names)

  elif inst_rc == 'col':
    inst_df = inst_df.transpose()

    inst_rows = inst_df.index.values.tolist()
    inst_cols = keep_names

    df['mat'] = grab_df_subset(df['mat'], inst_rows, inst_cols)

    if 'mat_up' in df:
      df['mat_up'] = grab_df_subset(df['mat_up'], inst_rows, inst_cols)
      df['mat_dn'] = grab_df_subset(df['mat_dn'], inst_rows, inst_cols)

    if 'mat_orig' in df:
      df['mat_orig'] = grab_df_subset(df['mat_orig'], inst_rows, inst_cols)

  return df

Response: Filter a network based on num_occur values being above threshold


Instruction:
Can you tell what is the following Python 3 function doing
def make_clust(net, dist_type='cosine', run_clustering=True, dendro=True,
                          requested_views=['pct_row_sum', 'N_row_sum'],
                          linkage_type='average', sim_mat=False, filter_sim=0.1,
                          calc_cat_pval=False, sim_mat_views=['N_row_sum'],
                          run_enrichr=None, enrichrgram=None):
  '''
  This will calculate multiple views of a clustergram by filtering the
  data and clustering after each filtering. This filtering will keep the top
  N rows based on some quantity (sum, num-non-zero, etc).
  '''
  from copy import deepcopy
  import scipy
  from . import calc_clust, run_filter, make_views, make_sim_mat, cat_pval
  from . import enrichr_functions as enr_fun

  df = net.dat_to_df()

  threshold = 0.0001
  df = run_filter.df_filter_row_sum(df, threshold)
  df = run_filter.df_filter_col_sum(df, threshold)

  # default setting
  define_cat_colors = False

  if run_enrichr is not None:
    df = enr_fun.add_enrichr_cats(df, 'row', run_enrichr)

    define_cat_colors = True

  # calculate initial view with no row filtering
  net.df_to_dat(df, define_cat_colors=True)

  inst_dm = calc_clust.cluster_row_and_col(net, dist_type=dist_type,
                                linkage_type=linkage_type,
                                run_clustering=run_clustering,
                                dendro=dendro, ignore_cat=False,
                                calc_cat_pval=calc_cat_pval)

  all_views = []
  send_df = deepcopy(df)

  if 'N_row_sum' in requested_views:
    all_views = make_views.N_rows(net, send_df, all_views,
                                  dist_type=dist_type, rank_type='sum')

  if 'N_row_var' in requested_views:
    all_views = make_views.N_rows(net, send_df, all_views,
                                  dist_type=dist_type, rank_type='var')

  if 'pct_row_sum' in requested_views:
    all_views = make_views.pct_rows(net, send_df, all_views,
                                    dist_type=dist_type, rank_type='sum')

  if 'pct_row_var' in requested_views:
    all_views = make_views.pct_rows(net, send_df, all_views,
                                    dist_type=dist_type, rank_type='var')

  which_sim = []

  if sim_mat == True:
    which_sim = ['row', 'col']
  elif sim_mat == 'row':
    which_sim = ['row']
  elif sim_mat == 'col':
    which_sim = ['col']

  if sim_mat is not False:
    sim_net = make_sim_mat.main(net, inst_dm, which_sim, filter_sim, sim_mat_views)

    net.sim = {}

    for inst_rc in which_sim:
      net.sim[inst_rc] = sim_net[inst_rc].viz

      if inst_rc == 'row':
        other_rc = 'col'
      elif inst_rc == 'col':
        other_rc = 'row'

      # keep track of cat_colors
      net.sim[inst_rc]['cat_colors'][inst_rc] = net.viz['cat_colors'][inst_rc]
      net.sim[inst_rc]['cat_colors'][other_rc] = net.viz['cat_colors'][inst_rc]

  else:
    net.sim = {}

  net.viz['views'] = all_views

  if enrichrgram != None:
    # toggle enrichrgram functionality from back-end
    net.viz['enrichrgram'] = enrichrgram

  if 'enrichrgram_lib' in net.dat:
    net.viz['enrichrgram'] = True
    net.viz['enrichrgram_lib'] = net.dat['enrichrgram_lib']

  if 'row_cat_bars' in net.dat:
    net.viz['row_cat_bars'] = net.dat['row_cat_bars']

Response: This function calculates multiple views of a clustergram from a network.


Instruction:
Can you generate the documentation for the following Python 3 function
def infer_flags(bytecode, is_async=False):
    flags = CompilerFlags(0)
    if not isinstance(bytecode, (_bytecode.Bytecode,
                                 _bytecode.ConcreteBytecode,
                                 _bytecode.ControlFlowGraph)):
        msg = ('Expected a Bytecode, ConcreteBytecode or ControlFlowGraph '
               'instance not %s')
        raise ValueError(msg % bytecode)

    instructions = (bytecode.get_instructions()
                    if isinstance(bytecode, _bytecode.ControlFlowGraph) else
                    bytecode)
    instr_names = {i.name for i in instructions
                   if not isinstance(i, (_bytecode.SetLineno,
                                         _bytecode.Label))}

    if not (instr_names & {'STORE_NAME', 'LOAD_NAME', 'DELETE_NAME'}):
        flags |= CompilerFlags.OPTIMIZED

    flags |= bytecode.flags & (CompilerFlags.NEWLOCALS |
                               CompilerFlags.VARARGS |
                               CompilerFlags.VARKEYWORDS |
                               CompilerFlags.NESTED)

    if instr_names & {'YIELD_VALUE', 'YIELD_FROM'}:
        if not is_async and not bytecode.flags & CompilerFlags.ASYNC_GENERATOR:
            flags |= CompilerFlags.GENERATOR
        else:
            flags |= CompilerFlags.ASYNC_GENERATOR

    if not (instr_names & {'LOAD_CLOSURE', 'LOAD_DEREF', 'STORE_DEREF',
                           'DELETE_DEREF', 'LOAD_CLASSDEREF'}):
        flags |= CompilerFlags.NOFREE

    if (not (bytecode.flags & CompilerFlags.ITERABLE_COROUTINE or
             flags & CompilerFlags.ASYNC_GENERATOR) and
            (instr_names & {'GET_AWAITABLE', 'GET_AITER', 'GET_ANEXT',
                            'BEFORE_ASYNC_WITH', 'SETUP_ASYNC_WITH'} or
             bytecode.flags & CompilerFlags.COROUTINE)):
        flags |= CompilerFlags.COROUTINE

    flags |= bytecode.flags & CompilerFlags.ITERABLE_COROUTINE

    flags |= bytecode.flags & CompilerFlags.FUTURE_GENERATOR_STOP

    if ([bool(flags & getattr(CompilerFlags, k))
         for k in ('COROUTINE', 'ITERABLE_COROUTINE', 'GENERATOR',
                   'ASYNC_GENERATOR')].count(True) > 1):
        raise ValueError("Code should not have more than one of the "
                         "following flag set : generator, coroutine, "
                         "iterable coroutine and async generator, got:"
                         "%s" % flags)

    return flags

Response: Infer the proper flags for a given bytecode.


Instruction:
Create a Python 3 function to
modify the instruction in - place.

Response:
def set(self, name, arg=UNSET):
        """Modify the instruction in-place.

        Replace name and arg attributes. Don't modify lineno.
        """
        self._set(name, arg, self._lineno)


Instruction:
How would you explain what the following Python 3 function does
def setup(tf, order=None):
    global _tf_version_string, _tf_version
    _tf_version_string = tf.__version__
    _tf_version = _parse_tf_version(_tf_version_string)

    if order is not None:
        optimize(order)

Response:
Sets up global variables to adapt to peculiarities of
    different tensorflow versions.


Instruction:
Make a summary of the following Python 3 code
def optimize(order):
    if not isinstance(order, (list, tuple)):
        order = [order]

    for op in Operation.__subclasses__():
        for impl in order:
            if impl in op.impls:
                op.use_impl(impl)
                break

Response: Optimize the order of the available resource types.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def print_tensor(td_tensor, indent="|   ", max_depth=-1, depth=0):
    offset = depth * indent
    line = "td tensor: %s" % td_tensor.name
    if td_tensor.value is not None:
        line += " (%s)" % (",".join(str(i) for i in td_tensor.value.shape),)

    print(offset + line)

    if td_tensor.op and (max_depth < 0 or max_depth > depth):
        print_op(td_tensor.op, indent=indent, max_depth=max_depth, depth=depth+1)

Response: Prints a tensor to stdout.


Instruction:
Here you have a function in Python 3, explain what it does
def print_op(td_op, indent="|   ", max_depth=-1, depth=0):
    offset = depth * indent
    line = "td op: %s (%s)" % (td_op.name, ",".join(td_op.types))

    print(offset + line)

    if max_depth < 0 or max_depth > depth:
        for td_tensor in td_op.inputs:
            print_tensor(td_tensor, indent=indent, max_depth=max_depth, depth=depth+1)

Response: Prints a single Operation object.


Instruction:
Given the following Python 3 function, write the documentation
def print_tf_tensor(tf_tensor, indent="|   ", max_depth=-1, depth=0):
    offset = depth * indent
    shape = tuple(int(i) for i in tf_tensor.get_shape())
    line = "tf tensor: %s (%s)" % (tf_tensor.name, ",".join(str(i) for i in shape))

    print(offset + line)

    if tf_tensor.op and (max_depth < 0 or max_depth > depth):
        print_tf_op(tf_tensor.op, indent=indent, max_depth=max_depth, depth=depth+1)

Response: Prints a tensor to stdout.


Instruction:
Create a Python 3 function for
printing a tensorflow operation to stdout.

Response:
def print_tf_op(tf_op, indent="|   ", max_depth=-1, depth=0):
    """ print_tf_op(tf_tensor, indent="    ", max_depth=-1)
    Prints the dependency graph of a tensorflow operation *tf_op*, where each new level is indented
    by *indent*. When *max_depth* is positive, the graph is truncated at that depth, where each
    tensor and each op count as a level.
    """
    offset = depth * indent
    line = "tf op: %s (%s)" % (tf_op.name, tf_op.type)

    print(offset + line)

    if max_depth < 0 or max_depth > depth:
        for tf_tensor in tf_op.inputs:
            print_tf_tensor(tf_tensor, indent=indent, max_depth=max_depth, depth=depth+1)


Instruction:
Can you tell what is the following Python 3 function doing
def LinSpace(start, stop, num):
    return np.linspace(start, stop, num=num, dtype=np.float32),

Response: Linspace implementation of the LinearSpace operator.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def Range(start, limit, delta):
    return np.arange(start, limit, delta, dtype=np.int32),

Response: Returns an array of the specified range.


Instruction:
Given the following Python 3 function, write the documentation
def RandomUniformInt(shape, minval, maxval, seed):
    if seed:
        np.random.seed(seed)
    return np.random.randint(minval, maxval, size=shape),

Response: Random uniform int op.


Instruction:
How would you implement a function in Python 3 that
ranks of the array.

Response:
def Rank(a):
    """
    Rank op.
    """
    return np.array([len(a.shape)], dtype=np.int32),


Instruction:
Can you create a Python 3 function that
squeezes op. i. e. removes singular axes.

Response:
def Squeeze(a, squeeze_dims):
    """
    Squeeze op, i.e. removes singular axes.
    """
    if not squeeze_dims:
        squeeze_dims = list(range(len(a.shape)))
    slices = [(0 if (dim == 1 and i in squeeze_dims) else slice(None)) \
              for i, dim in enumerate(a.shape)]
    return np.copy(a)[slices],


Instruction:
Make a summary of the following Python 3 code
def ExpandDims(a, dim):
    shape = list(a.shape)
    if dim >= 0:
        shape.insert(dim, 1)
    else:
        shape.insert(len(shape) + dim + 1, 1)
    return np.copy(a).reshape(*shape),

Response: Expand dim op, i.e. add singular axis at dim.


Instruction:
Write a Python 3 script for
slicing a array by given indices.

Response:
def Slice(a, begin, size):
    """
    Slicing op.
    """
    return np.copy(a)[[slice(*tpl) for tpl in zip(begin, begin+size)]],


Instruction:
Implement a function in Python 3 to
split op with n splits.

Response:
def Split(axis, a, n):
    """
    Split op with n splits.
    """
    return tuple(np.split(np.copy(a), n, axis=axis))


Instruction:
How would you explain what the following Python 3 function does
def SplitV(a, splits, axis):
    return tuple(np.split(np.copy(a), np.cumsum(splits), axis=axis))

Response: Split op with multiple split sizes.


Instruction:
Implement a function in Python 3 to
batch diag op that returns only the diagonal elements.

Response:
def MatrixDiagPart(a):
    """
    Batched diag op that returns only the diagonal elements.
    """
    r = np.zeros(a.shape[:-2] + (min(a.shape[-2:]),))
    for coord in np.ndindex(a.shape[:-2]):
        pos = coord + (Ellipsis,)
        r[pos] = np.diagonal(a[pos])
    return r,


Instruction:
Create a Python 3 function for
selfing Adjoint Eigen decomp op.

Response:
def SelfAdjointEig(a):
    """
    Eigen decomp op.
    """
    shape = list(a.shape)
    shape[-2] += 1
    return np.append(*np.linalg.eig(a)).reshape(*shape),


Instruction:
Can you generate the documentation for the following Python 3 function
def Svd(a, uv, full):
    u, s, v = np.linalg.svd(a, full_matrices=full, compute_uv=uv)
    return s, u, v

Response: Single value decomp op.


Instruction:
How would you code a function in Python 3 to
compute prod along specified axis.

Response:
def Prod(a, axis, keep_dims):
    """
    Prod reduction op.
    """
    return np.prod(a, axis=axis if not isinstance(axis, np.ndarray) else tuple(axis),
                   keepdims=keep_dims),


Instruction:
Can you tell what is the following Python 3 function doing
def SparseSegmentSqrtN(a, idxs, ids):
    func = lambda _idxs: np.divide(reduce(np.add, a[idxs][_idxs]), np.math.sqrt(len(_idxs)))
    return seg_map(func, a, ids),

Response: Sparse segmented sum / sqrt N


Instruction:
Here you have a function in Python 3, explain what it does
def AvgPool3D(a, k, strides, padding):
    patches = _pool_patches(a, k, strides, padding.decode("ascii"))
    return np.average(patches, axis=tuple(range(-len(k), 0))),

Response: Average 3D pooling op.


Instruction:
Here you have a function in Python 3, explain what it does
def get(self, *names, **kwargs):
        tensors = tuple(self._get(name, **kwargs) for name in names)
        return tensors[0] if len(names) == 1 else tensors

Response: Get a set of tensors given by names using a deep lookup within the current model.


Instruction:
How would you implement a function in Python 3 that
adds a new root tensor for a given key.

Response:
def add(self, tensor, tf_sess=None, key=None, **kwargs):
        """
        Adds a new root *tensor* for a *key* which, if *None*, defaults to a consecutive number.
        When *tensor* is not an instance of :py:class:`Tensor` but an instance of
        ``tensorflow.Tensor``, it is converted first. In that case, *tf_sess* should be a valid
        tensorflow session and *kwargs* are forwarded to the :py:class:`Tensor` constructor.
        """
        if not isinstance(tensor, Tensor):
            tensor = Tensor(tensor, tf_sess, **kwargs)

        if key is None:
            if len(self.roots) == 0:
                key = 0
            else:
                key = max(self.roots.keys()) + 1

        self.roots[key] = tensor


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load(self, path):
        path = os.path.expandvars(os.path.expanduser(path))
        with open(path, "rb") as f:
            roots = pickle.load(f)

        for key, tensor in roots.items():
            self.add(tensor, key=key)

Response: Loads all tensors from a file defined by path and adds them to the set.


Instruction:
Explain what the following Python 3 code does
def save(self, path):
        path = os.path.expandvars(os.path.expanduser(path))
        with open(path, "wb") as f:
            pickle.dump(self.roots, f)

Response: Saves all the tensors of the root set to a file defined by path.


Instruction:
Explain what the following Python 3 code does
def get(self, *names, **kwargs):
        # create empty tensor ensembles with our method
        tensor_ensembles = [TensorEnsemble([], self.method) for name in names]

        # loop over models, collect and add tensors
        for model in self.models:
            tensors = model.get(*names, **kwargs)
            if not isinstance(tensors, tuple):
                tensors = (tensors,)
            for i, t in enumerate(tensors if isinstance(tensors, tuple) else (tensors,)):
                tensor_ensembles[i].tensors.append(t)

        return tensor_ensembles[0] if len(names) == 1 else tuple(tensor_ensembles)

Response:
Get one or more tensor ensembles given by names using a deep
        lookup within all read models.


Instruction:
Here you have a function in Python 3, explain what it does
def load(self, paths):
        for path in paths:
            self.models.append(Model(path))

Response: Loads models from a list of paths.


Instruction:
Write a Python 3 script to
evaluate all contained tensors using a feed_dict and returns the ensemble value.

Response:
def eval(self, feed_dict=None):
        """
        Evaluates all contained tensors using a *feed_dict* and returns the ensemble value. The keys
        of *feed_dict* must be tensor ensembles. Its values can be batches, i.e., numpy arrays, or
        lists or tuples of batches. In the latter case, these lists or tuples must have the same
        length as the list of stored tensors as they will be mapped.
        """
        # first, check that the length of all feed_dict keys match our own length
        for tensor_ensemble in feed_dict:
            if len(tensor_ensemble.tensors) != len(self.tensors):
                raise EnsembleMismatchException("incompatible lengths of tensors: %d, %d" \
                                                % (len(self.tensors), len(tensor_ensemble.tensors)))

        # create a joined uuid
        _uuid = uuid4()

        # prepare feed_dicts
        feed_dicts = [{} for _ in range(len(self.tensors))]
        for tensor_ensemble, value in feed_dict.items():
            for i, tensor in enumerate(tensor_ensemble.tensors):
                if tensor is not None:
                    feed_dicts[i][tensor] = value[i] if isinstance(value, (list, tuple)) else value

        # eval all tensors
        values = [t.eval(feed_dict=d, _uuid=_uuid) for t, d in zip(self.tensors, feed_dicts)]

        # return the computed ensemble value
        return self.func(values)


Instruction:
Can you tell what is the following Python 3 function doing
def pressed_keys(self):
        # make a list of all the keys that are detected
        pressed = []

        # set all pins pins to be inputs w/pullups
        for pin in self.row_pins+self.col_pins:
            pin.direction = Direction.INPUT
            pin.pull = Pull.UP

        for row in range(len(self.row_pins)):
            # set one row low at a time
            self.row_pins[row].direction = Direction.OUTPUT
            self.row_pins[row].value = False
            # check the column pins, which ones are pulled down
            for col in range(len(self.col_pins)):
                if not self.col_pins[col].value:
                    pressed.append(self.keys[row][col])
            # reset the pin to be an input
            self.row_pins[row].direction = Direction.INPUT
            self.row_pins[row].pull = Pull.UP
        return pressed

Response:
An array containing all the keys that are pressed from the initalized
            list - of - lists passed in during creation


Instruction:
Make a summary of the following Python 3 code
def get_stdev(self, asset_type):
        load_times = []
        # Handle edge cases like TTFB
        if asset_type == 'ttfb':
            for page in self.pages:
                if page.time_to_first_byte is not None:
                    load_times.append(page.time_to_first_byte)
        elif asset_type not in self.asset_types and asset_type != 'page':
            raise ValueError('asset_type must be one of:\nttfb\n{0}'.format(
                    '\n'.join(self.asset_types)))
        else:
            load_times = self.get_load_times(asset_type)

        if not load_times or not sum(load_times):
            return 0
        return round(stdev(load_times),
                     self.decimal_precision)

Response: Calculates the standard deviation for a set of asset types.


Instruction:
How would you code a function in Python 3 to
return a list of all the pages of all the parser objects.

Response:
def pages(self):
        """
        The aggregate pages of all the parser objects.
        """
        pages = []
        for har_dict in self.har_data:
            har_parser = HarParser(har_data=har_dict)
            if self.page_id:
                for page in har_parser.pages:
                    if page.page_id == self.page_id:
                        pages.append(page)
            else:
                pages = pages + har_parser.pages
        return pages


Instruction:
Can you generate a brief explanation for the following Python 3 code
def time_to_first_byte(self):
        ttfb = []
        for page in self.pages:
            if page.time_to_first_byte is not None:
                ttfb.append(page.time_to_first_byte)
        return round(mean(ttfb), self.decimal_precision)

Response: Returns the aggregate time to first byte for all pages.


Instruction:
Can you create a Python 3 function that
returns the average load time for all runs.

Response:
def page_load_time(self):
        """
        The average total load time for all runs (not weighted).
        """
        load_times = self.get_load_times('page')
        return round(mean(load_times), self.decimal_precision)


Instruction:
Write a Python 3 script to
return aggregate javascript load time.

Response:
def js_load_time(self):
        """
        Returns aggregate javascript load time.
        """
        load_times = self.get_load_times('js')
        return round(mean(load_times), self.decimal_precision)


Instruction:
Write a Python 3 script to
return aggregate css load time for all pages.

Response:
def css_load_time(self):
        """
        Returns aggregate css load time for all pages.
        """
        load_times = self.get_load_times('css')
        return round(mean(load_times), self.decimal_precision)


Instruction:
Can you tell what is the following Python 3 function doing
def image_load_time(self):
        load_times = self.get_load_times('image')
        return round(mean(load_times), self.decimal_precision)

Response: Returns aggregate image load time for all pages.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def html_load_time(self):
        load_times = self.get_load_times('html')
        return round(mean(load_times), self.decimal_precision)

Response: Returns aggregate html load time for all pages.


Instruction:
Here you have a function in Python 3, explain what it does
def audio_load_time(self):
        load_times = self.get_load_times('audio')
        return round(mean(load_times), self.decimal_precision)

Response: Returns aggregate audio load time for all pages.


Instruction:
Here you have a function in Python 3, explain what it does
def video_load_time(self):
        load_times = self.get_load_times('video')
        return round(mean(load_times), self.decimal_precision)

Response: Returns aggregate video load time for all pages.


Instruction:
Write a Python 3 function that can
function to match headers.

Response:
def match_headers(self, entry, header_type, header, value, regex=True):
        """
        Function to match headers.

        Since the output of headers might use different case, like:

            'content-type' vs 'Content-Type'

        This function is case-insensitive

        :param entry: entry object
        :param header_type: ``str`` of header type. Valid values:

            * 'request'
            * 'response'

        :param header: ``str`` of the header to search for
        :param value: ``str`` of value to search for
        :param regex: ``bool`` indicating whether to use regex or exact match

        :returns: a ``bool`` indicating whether a match was found
        """
        if header_type not in entry:
            raise ValueError('Invalid header_type, should be either:\n\n'
                             '* \'request\'\n*\'response\'')

        # TODO - headers are empty in some HAR data.... need fallbacks here
        for h in entry[header_type]['headers']:
            if h['name'].lower() == header.lower() and h['value'] is not None:
                if regex and re.search(value, h['value'], flags=re.IGNORECASE):
                    return True
                elif value == h['value']:
                    return True
        return False


Instruction:
Implement a Python 3 function for
returning True if the content type of a request matches the mimeType metadata.

Response:
def match_content_type(entry, content_type, regex=True):
        """
        Matches the content type of a request using the mimeType metadata.

        :param entry: ``dict`` of a single entry from a HarPage
        :param content_type: ``str`` of regex to use for finding content type
        :param regex: ``bool`` indicating whether to use regex or exact match.
        """
        mimeType = entry['response']['content']['mimeType']

        if regex and re.search(content_type, mimeType, flags=re.IGNORECASE):
            return True

        elif content_type == mimeType:
            return True

        return False


Instruction:
Implement a function in Python 3 to
return True if the entry with the given request type matches the given request_type.

Response:
def match_request_type(self, entry, request_type, regex=True):
        """
        Helper function that returns entries with a request type
        matching the given `request_type` argument.

        :param entry: entry object to analyze
        :param request_type: ``str`` of request type to match
        :param regex: ``bool`` indicating whether to use a regex or string match
        """
        if regex:
            return re.search(request_type, entry['request']['method'],
                             flags=re.IGNORECASE) is not None
        else:
            return entry['request']['method'] == request_type


Instruction:
Can you implement a function in Python 3 that
creates a dictionary of the asset timeline for the requested assets.

Response:
def create_asset_timeline(self, asset_list):
        """
        Returns a ``dict`` of the timeline for the requested assets. The key is
        a datetime object (down to the millisecond) of ANY time where at least
        one of the requested assets was loaded. The value is a ``list`` of ALL
        assets that were loading at that time.

        :param asset_list: ``list`` of the assets to create a timeline for.
        """
        results = dict()
        for asset in asset_list:
            time_key = dateutil.parser.parse(asset['startedDateTime'])
            load_time = int(asset['time'])
            # Add the start time and asset to the results dict
            if time_key in results:
                results[time_key].append(asset)
            else:
                results[time_key] = [asset]
            # For each millisecond the asset was loading, insert the asset
            # into the appropriate key of the results dict. Starting the range()
            # index at 1 because we already inserted the first millisecond.
            for _ in range(1, load_time):
                time_key = time_key + datetime.timedelta(milliseconds=1)
                if time_key in results:
                    results[time_key].append(asset)
                else:
                    results[time_key] = [asset]

        return results


Instruction:
Can you generate the documentation for the following Python 3 function
def pages(self):
        # Start with a page object for unknown entries if the HAR data has
        # any entries with no page ID
        pages = []
        if any('pageref' not in entry for entry in self.har_data['entries']):
            pages.append(HarPage('unknown', har_parser=self))
        for har_page in self.har_data['pages']:
            page = HarPage(har_page['id'], har_parser=self)
            pages.append(page)

        return pages

Response:
This is a list of HarPage objects each of which represents a page
        from the HAR file.


Instruction:
Given the following Python 3 function, write the documentation
def filter_entries(self, request_type=None, content_type=None,
                       status_code=None, http_version=None, regex=True):
        """
        Returns a ``list`` of entry objects based on the filter criteria.

        :param request_type: ``str`` of request type (i.e. - GET or POST)
        :param content_type: ``str`` of regex to use for finding content type
        :param status_code: ``int`` of the desired status code
        :param http_version: ``str`` of HTTP version of request
        :param regex: ``bool`` indicating whether to use regex or exact match.
        """
        results = []

        for entry in self.entries:
            """
            So yea... this is a bit ugly. We are looking for:

                * The request type using self._match_request_type()
                * The content type using self._match_headers()
                * The HTTP response status code using self._match_status_code()
                * The HTTP version using self._match_headers()

            Oh lords of python.... please forgive my soul
            """
            valid_entry = True
            p = self.parser
            if request_type is not None and not p.match_request_type(
                    entry, request_type, regex=regex):
                valid_entry = False
            if content_type is not None:
                if not self.parser.match_content_type(entry, content_type,
                                                      regex=regex):
                    valid_entry = False
            if status_code is not None and not p.match_status_code(
                    entry, status_code, regex=regex):
                valid_entry = False
            if http_version is not None and not p.match_http_version(
                    entry, http_version, regex=regex):
                valid_entry = False

            if valid_entry:
                results.append(entry)

        return results

Response: Returns a list of entries based on the filter criteria.


Instruction:
Can you write a function in Python 3 where it
returns the total size of a collection of entries.

Response:
def get_total_size(self, entries):
        """
        Returns the total size of a collection of entries.

        :param entries: ``list`` of entries to calculate the total size of.
        """
        size = 0
        for entry in entries:
            if entry['response']['bodySize'] > 0:
                size += entry['response']['bodySize']
        return size


Instruction:
Can you generate the documentation for the following Python 3 function
def get_total_size_trans(self, entries):
        size = 0
        for entry in entries:
            if entry['response']['_transferSize'] > 0:
                size += entry['response']['_transferSize']
        return size

Response: Returns the total size of a collection of entries - transferred.


Instruction:
Explain what the following Python 3 code does
def time_to_first_byte(self):
        # The unknown page is just a placeholder for entries with no page ID.
        # As such, it would not have a TTFB
        if self.page_id == 'unknown':
            return None
        ttfb = 0
        for entry in self.entries:
            if entry['response']['status'] == 200:
                for k, v in iteritems(entry['timings']):
                    if k != 'receive':
                        if v > 0:
                            ttfb += v
                break
            else:
                ttfb += entry['time']

        return ttfb

Response: Return the time to first byte of the page request in ms.


Instruction:
Make a summary of the following Python 3 code
def group_by(self, *fields):
        fields = self._expand_group_by_fields(self.model, fields)
        return self._clone(klass=GroupByQuerySet, setup=True, _fields=fields)

Response: Returns a new queryset with the specified fields grouped by the current queryset.


Instruction:
Implement a Python 3 function for
returning the cached data built from instance raw _values as a dictionary.

Response:
def _data(self):
        """
        Cached data built from instance raw _values as a dictionary.
        """
        d = {}

        # Iterate all keys and values
        for k, v in self._row_values.items():
            # Split related model fields
            attrs = k.rsplit('__', 1)

            # Set value depending case
            if len(attrs) == 2:
                # Related model field, store nested
                fk, fn = attrs
                if fk not in d:
                    d[fk] = {}
                d[fk][fn] = v

            else:
                # Own model field, store directly
                d[k] = v

        # Return (+cache) data
        return d


Instruction:
Write a Python 3 script for
populating instance with given.

Response:
def _set_values(self):
        """
        Populate instance with given.
        """
        # Iterate all keys and values in data
        for k, v in self._data.items():
            # If it's a dict, process it (it's probably instance data)
            if isinstance(v, dict):
                try:
                    # Get related model from field (follow path)
                    rel_model = self._model
                    for attr in k.split('__'):
                        rel_model = getattr(rel_model, attr).field.related_model

                except AttributeError:
                    # Not a model, maybe it is a dict field (?)
                    pass

                else:
                    # Model, first shorten field name
                    k = k.replace('__', '_')

                    # Now init instance if required (not if we got ID None)
                    if 'id' in v and v['id'] is None:
                        # This means we grouped by ID, if it's none then FK is None
                        v = None

                    else:
                        # Either we have ID or we didn't group by ID, use instance
                        v = rel_model(**v)

            # Set value
            setattr(self, k, v)


Instruction:
Can you generate the documentation for the following Python 3 function
def group_by(self, *fields):
        fields = self._expand_group_by_fields(self.model, fields)
        clone = self._values(*fields)
        clone._iterable_class = GroupByIterable
        return clone

Response: Returns a new queryset with the given fields grouped by the current queryset.


Instruction:
Implement a Python 3 function for
expanding FK fields into all related object s fields to avoid future lookups.

Response:
def _expand_group_by_fields(cls, model, fields):
        """
        Expand FK fields into all related object's fields to avoid future
        lookups.

        :param fields: fields to "group by"
        :return: expanded fields
        """
        # Containers for resulting fields and related model fields
        res = []
        related = {}

        # Add own fields and populate related fields
        for field_name in fields:
            if '__' in field_name:
                # Related model field: append to related model's fields
                fk_field_name, related_field = field_name.split('__', 1)
                if fk_field_name not in related:
                    related[fk_field_name] = [related_field]
                else:
                    related[fk_field_name].append(related_field)

            else:
                # Simple field, get the field instance
                model_field = model._meta.get_field(field_name)

                if isinstance(model_field, (ForeignKey, ManyToManyField)):
                    # It's a related field, get model
                    related_model = model_field.related_model

                    # Append all its fields with the correct prefix
                    res.extend('{}__{}'.format(field_name, f.column)
                               for f in related_model._meta.fields)

                else:
                    # It's a common field, just append it
                    res.append(field_name)

        # Resolve all related fields
        for fk_field_name, field_names in related.items():
            # Get field
            fk = model._meta.get_field(fk_field_name)

            # Get all fields for that related model
            related_fields = cls._expand_group_by_fields(fk.related_model,
                                                         field_names)

            # Append them with the correct prefix
            res.extend('{}__{}'.format(fk_field_name, f) for f in related_fields)

        # Return all fields
        return res


Instruction:
Can you generate the documentation for the following Python 3 function
def write(self, addr, data):
        '''Write to dummy memory

        Parameters
        ----------
        addr : int
            The register address.
        data : list, tuple
            Data (byte array) to be written.

        Returns
        -------
        nothing
        '''
        logger.debug(
            "Dummy SiTransferLayer.write addr: %s data: %s" % (hex(addr), data))
        for curr_addr, d in enumerate(data, start=addr):
            self.mem[curr_addr] = array.array('B', [d])[0]

Response: Write to dummy memory.


Instruction:
How would you code a function in Python 3 to
read the data from memory.

Response:
def read(self, addr, size):
        '''
        Parameters
        ----------
        addr : int
            The register address.
        size : int
            Length of data to be read (number of bytes).

        Returns
        -------
        array : array
            Data (byte array) read from memory. Returns 0 for each byte if it hasn't been written to.
        '''
        logger.debug("Dummy SiTransferLayer.read addr: %s size: %s" % (hex(addr), size))
        return array.array('B', [self.mem[curr_addr] if curr_addr in self.mem else 0 for curr_addr in range(addr, addr + size)])


Instruction:
Write a Python 3 function that can
write a value to a memory address and offset.

Response:
def set_value(self, value, addr, size, offset, **kwargs):
        '''Writing a value of any arbitrary size (max. unsigned int 64) and offset to a register

        Parameters
        ----------
        value : int, str
            The register value (int, long, bit string) to be written.
        addr : int
            The register address.
        size : int
            Bit size/length of the value to be written to the register.
        offset : int
            Offset of the value to be written to the register (in number of bits).

        Returns
        -------
        nothing
        '''
        div_offset, mod_offset = divmod(offset, 8)
        div_size, mod_size = divmod(size + mod_offset, 8)
        if mod_size:
            div_size += 1
        if mod_offset == 0 and mod_size == 0:
            reg = BitLogic.from_value(0, size=div_size * 8)
        else:
            ret = self._intf.read(self._base_addr + addr + div_offset, size=div_size)
            reg = BitLogic()
            reg.frombytes(tobytes(ret))
        reg[size + mod_offset - 1:mod_offset] = value
        self._intf.write(self._base_addr + addr + div_offset, data=array.array('B', reg.tobytes()))


Instruction:
Write a Python 3 script for
reading a value from a register of any arbitrary size and offset from a register.

Response:
def get_value(self, addr, size, offset, **kwargs):
        '''Reading a value of any arbitrary size (max. unsigned int 64) and offset from a register

        Parameters
        ----------
        addr : int
            The register address.
        size : int
            Bit size/length of the value.
        offset : int
            Offset of the value to be written to the register (in number of bits).

        Returns
        -------
        reg : int
            Register value.
        '''
        div_offset, mod_offset = divmod(offset, 8)
        div_size, mod_size = divmod(size + mod_offset, 8)
        if mod_size:
            div_size += 1
        ret = self._intf.read(self._base_addr + addr + div_offset, size=div_size)
        reg = BitLogic()
        reg.frombytes(tobytes(ret))
        return reg[size + mod_offset - 1:mod_offset].tovalue()


Instruction:
Make a summary of the following Python 3 code
def set_bytes(self, data, addr, **kwargs):
        '''Writing bytes of any arbitrary size

        Parameters
        ----------
        data : iterable
            The data (byte array) to be written.
        addr : int
            The register address.

        Returns
        -------
        nothing
        '''
        self._intf.write(self._conf['base_addr'] + addr, data)

Response: Writes bytes of any arbitrary size to the specified address.


Instruction:
Explain what the following Python 3 code does
def get_bytes(self, addr, size, **kwargs):
        '''Reading bytes of any arbitrary size

        Parameters
        ----------.
        addr : int
            The register address.
        size : int
            Byte length of the value.

        Returns
        -------
        data : iterable
            Byte array.
        '''
        return self._intf.read(self._conf['base_addr'] + addr, size)

Response: Reads bytes of any arbitrary size from the specified address.


Instruction:
Create a Python 3 function to
write the current entry to the file

Response:
def write(self, size=None):
        """
        to call start() automatically, set yaml file as follows:
        registers:
          - name        : CCPD_PCB
            type        : StdRegister
            hw_driver   : CCPD_PCB_SPI
            size        : 32
            auto_start  : True  <------ add this
            fields: ......
        """
        if size is None:
            self._drv.set_data(self.tobytes())
        else:
            self._drv.set_data(self.tobytes()[:size])

        if "auto_start" in self._conf:
            if self._conf["auto_start"]:
                self._drv.start()


Instruction:
Make a summary of the following Python 3 code
def from_value(cls, value, size=None, fmt='Q', **kwargs):
        '''
        Factory method

        For format characters see: https://docs.python.org/2/library/struct.html
        '''
        bl = cls(**kwargs)  # size is 0 by default
        bl.fromvalue(value=value, size=size, fmt=fmt)
        return bl

Response:
Factory method
        Factory method
       FromValue


Instruction:
Can you tell what is the following Python 3 function doing
def fromvalue(self, value, size=None, fmt='Q'):
        '''
        Append from a int/long number.
        '''
        if size and value.bit_length() > size:
            raise TypeError('Value is too big for given size')
        self.frombytes(struct.pack(fmt, value))
        if size:
            if not isinstance(size, integer_types) or not size > 0:
                raise TypeError('Size must be greater than zero')
            if size > self.length():
                bitarray.extend(self, (size - self.length()) * [0])
            else:
                bitarray.__delitem__(self, slice(size, self.length()))

Response: Append from a int or long number.


Instruction:
Can you write a function in Python 3 where it
converts bitstring to a int or long number.

Response:
def tovalue(self, fmt='Q'):
        '''
        Convert bitstring to a int/long number.
        '''
        format_size = struct.calcsize(fmt)
        if self.length() > format_size * 8:
            raise TypeError('Cannot convert to number')
        ba = self.copy()
        ba.extend((format_size * 8 - self.length()) * [0])
        return struct.unpack_from(fmt, ba.tobytes())[0]


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _swap_slice_indices(self, slc, make_slice=False):
        '''Swap slice indices

        Change slice indices from Verilog slicing (e.g. IEEE 1800-2012) to Python slicing.
        '''
        try:
            start = slc.start
            stop = slc.stop
            slc_step = slc.step
        except AttributeError:
            if make_slice:
                if slc < 0:
                    slc += self.length()
                return slice(slc, slc + 1)
            else:
                return slc
        else:
            if not start and start != 0:
                slc_stop = self.length()
            elif start < 0:
                slc_stop = self.length() + start + 1
            else:
                slc_stop = start + 1
            if not stop and stop != 0:
                slc_start = 0
            elif stop < 0:
                slc_start = self.length() + stop
            else:
                slc_start = stop
            return slice(slc_start, slc_stop, slc_step)

Response: Swap slice indices from Verilog slicing ( e. g. IEEE 1800 -2012 to Python slicing.


Instruction:
Explain what the following Python 3 code does
def _run_seq(self, size):

        # Write the sequence to the sequence generator (hw driver)
        self['SEQ'].write(size) #write pattern to memory

        self['SEQ'].set_size(size)  # set size
        self['SEQ'].set_repeat(1) # set repeat
        for _ in range(1):
            self['SEQ'].start() # start

            while not self['SEQ'].get_done():
                #time.sleep(0.1)
                print("Wait for done...")

Response: Runs the sequence generator.


Instruction:
Can you create a Python 3 function that
clears all the stuff that are not set.

Response:
def _clear_strobes(self):
        """
        Resets the "enable" and "load" output streams to all 0.

        """
        #reset some stuff
        self['SEQ']['GLOBAL_SHIFT_EN'].setall(False)
        self['SEQ']['GLOBAL_CTR_LD'].setall(False)
        self['SEQ']['GLOBAL_DAC_LD'].setall(False)
        self['SEQ']['PIXEL_SHIFT_EN'].setall(False)
        self['SEQ']['INJECTION'].setall(False)


Instruction:
How would you code a function in Python 3 to
set data for outgoing stream

Response:
def set_data(self, data, addr=0):
        '''
        Sets data for outgoing stream
        '''
        if self._mem_bytes < len(data):
            raise ValueError('Size of data (%d bytes) is too big for memory (%d bytes)' % (len(data), self._mem_bytes))
        self._intf.write(self._conf['base_addr'] + self._spi_mem_offset + addr, data)


Instruction:
Can you tell what is the following Python 3 function doing
def get_data(self, size=None, addr=None):
        '''
        Gets data for incoming stream
        '''
        # readback memory offset
        if addr is None:
            addr = self._mem_bytes

        if size and self._mem_bytes < size:
            raise ValueError('Size is too big')

        if size is None:
            return self._intf.read(self._conf['base_addr'] + self._spi_mem_offset + addr, self._mem_bytes)
        else:
            return self._intf.read(self._conf['base_addr'] + self._spi_mem_offset + addr, size)

Response: Gets data for incoming stream


Instruction:
Can you write a function in Python 3 where it
initializes serial device. Parameters of serial.Serial: http://pyserial.sourceforge.net/pyserial_api.html Plus termination string parameter eol

Response:
def init(self):
        '''
        Initialize serial device.
        Parameters of serial.Serial: http://pyserial.sourceforge.net/pyserial_api.html
        Plus termination string parameter eol
        '''
        super(Serial, self).init()
        self.read_termination = self._init.get('read_termination', None)
        self.write_termination = self._init.get('write_termination', self.read_termination)
        try:
            self.read_termination = bytes(self.read_termination, 'utf-8')
            self.write_termination = bytes(self.write_termination, 'utf-8')
        except TypeError as e:
            logger.debug(e)
        self.timeout = self._init.get('timeout', None)  # timeout of 0 returns immediately

        self._port = serial.Serial(**{key: value for key, value in self._init.items() if key not in ("read_termination", "write_termination")})


Instruction:
How would you explain what the following Python 3 function does
def read_eeprom_calibration(self):  # use default values for temperature, EEPROM values are usually not calibrated and random
        '''Reading EEPROM calibration for sources and regulators
        '''
        header = self.get_format()
        if header == self.HEADER_GPAC:
            data = self._read_eeprom(self.CAL_DATA_ADDR, size=calcsize(self.CAL_DATA_GPAC_FORMAT))
            for idx, channel in enumerate(self._ch_cal.iterkeys()):
                ch_data = data[idx * calcsize(self.CAL_DATA_CH_GPAC_FORMAT):(idx + 1) * calcsize(self.CAL_DATA_CH_GPAC_FORMAT)]
                values = unpack_from(self.CAL_DATA_CH_GPAC_FORMAT, ch_data)
                self._ch_cal[channel]['name'] = "".join([c for c in values[0] if (c in string.printable)])  # values[0].strip()
                self._ch_cal[channel]['default'] = values[1]
                self._ch_cal[channel]['min'] = values[2]
                self._ch_cal[channel]['max'] = values[3]
                self._ch_cal[channel]['ADCI']['gain'] = -values[4]  # fix gain sign
                self._ch_cal[channel]['ADCI']['offset'] = values[5]
                self._ch_cal[channel]['ADCV']['gain'] = values[6]
                self._ch_cal[channel]['ADCV']['offset'] = values[7]
                self._ch_cal[channel]['DAC']['gain'] = values[8]
                self._ch_cal[channel]['DAC']['offset'] = values[9]
                self._ch_cal[channel]['limit'] = values[10]
        else:
            raise ValueError('EEPROM data format not supported (header: %s)' % header)

Response: Reads the EEPROM calibration for sources and regulators.


Instruction:
How would you code a function in Python 3 to
read the voltage from the ADC.

Response:
def get_voltage(self, channel, unit='V'):
        '''Reading voltage
        '''
        adc_ch = self._ch_map[channel]['ADCV']['adc_ch']
        address = self._ch_map[channel]['ADCV']['address']
        raw = self._get_adc_value(address=address)[adc_ch]

        dac_offset = self._ch_cal[channel]['ADCV']['offset']
        dac_gain = self._ch_cal[channel]['ADCV']['gain']

        voltage = ((raw - dac_offset) / dac_gain)

        if unit == 'raw':
            return raw
        elif unit == 'V':
            return voltage / 1000
        elif unit == 'mV':
            return voltage
        else:
            raise TypeError("Invalid unit type.")


Instruction:
Can you write a function in Python 3 where it
reads current from the ADC.

Response:
def get_current(self, channel, unit='A'):
        '''Reading current
        '''
        values = self._get_adc_value(address=self._ch_map[channel]['ADCI']['address'])
        raw = values[self._ch_map[channel]['ADCI']['adc_ch']]
        dac_offset = self._ch_cal[channel]['ADCI']['offset']
        dac_gain = self._ch_cal[channel]['ADCI']['gain']

        if 'PWR' in channel:
            current = ((raw - dac_offset) / dac_gain)

            if unit == 'raw':
                return raw
            elif unit == 'A':
                return current / 1000
            elif unit == 'mA':
                return current
            elif unit == 'uA':
                return current * 1000
            else:
                raise TypeError("Invalid unit type.")
        else:
            voltage = values[self._ch_map[channel]['ADCV']['adc_ch']]
            current = (((raw - voltage) - dac_offset) / dac_gain)

            if unit == 'raw':
                return raw
            elif unit == 'A':
                return current / 1000000
            elif unit == 'mA':
                return current / 1000
            elif unit == 'uA':
                return current
            else:
                raise TypeError("Invalid unit type.")


Instruction:
Create a Python 3 function to
enable or disable output of power channel.

Response:
def set_enable(self, channel, value):
        '''Enable/Disable output of power channel
        '''
        try:
            bit = self._ch_map[channel]['GPIOEN']['bit']
        except KeyError:
            raise ValueError('set_enable() not supported for channel %s' % channel)
        self._set_power_gpio_value(bit=bit, value=value)


Instruction:
Write a Python 3 function for
returning True if the channel is over current.

Response:
def get_over_current(self, channel):
        '''Reading over current status of power channel
        '''
        try:
            bit = self._ch_map[channel]['GPIOOC']['bit']
        except KeyError:
            raise ValueError('get_over_current() not supported for channel %s' % channel)
        return not self._get_power_gpio_value(bit)


Instruction:
Create a Python 3 function for
setting the current limit for the specified channel.

Response:
def set_current_limit(self, channel, value, unit='A'):
        '''Setting current limit

        Note: same limit for all channels.
        '''
        # TODO: add units / calibration
        if unit == 'raw':
            value = value
        elif unit == 'A':
            value = int(value * 1000 * self.CURRENT_LIMIT_GAIN)
        elif unit == 'mA':
            value = int(value * self.CURRENT_LIMIT_GAIN)
        elif unit == 'uA':
            value = int(value / 1000 * self.CURRENT_LIMIT_GAIN)
        else:
            raise TypeError("Invalid unit type.")

        I2cAnalogChannel._set_dac_value(self, address=self.CURRENT_LIMIT_DAC_SLAVE_ADD, dac_ch=self.CURRENT_LIMIT_DAC_CH, value=value)


Instruction:
Write a Python 3 script for
setting the current of current source to the given value.

Response:
def set_current(self, channel, value, unit='A'):
        '''Setting current of current source
        '''
        dac_offset = self._ch_cal[channel]['DAC']['offset']
        dac_gain = self._ch_cal[channel]['DAC']['gain']
        if unit == 'raw':
            value = value
        elif unit == 'A':
            value = int((-value * 1000000 - dac_offset) / dac_gain)  # fix sign of output
        elif unit == 'mA':
            value = int((-value * 1000 - dac_offset) / dac_gain)  # fix sign of output
        elif unit == 'uA':
            value = int((-value - dac_offset) / dac_gain)  # fix sign of output
        else:
            raise TypeError("Invalid unit type.")

        self._set_dac_value(channel=channel, value=value)


Instruction:
Here you have a function in Python 3, explain what it does
def _calculateParameters(self, fout):
        q_d_f = 0

        '''
        fout = fref * (p_total / q_total) * (1 / div)

        p_total = 2 * ((p_counter + 4) + p_0)     [16..1023]
        q_total = q_counter + 2                   [2..129]
        div = [2,(3),4..127]

        constraints:

        f_ref * p_total / q_total = [100..400] MHz
        f_ref / q_total > 0.25 MHz
        '''
        for self.q_counter in range(128):
            self.q_total = self.q_counter + 2
            if (self.fref / self.q_total) < 0.25:  # PLL constraint
                break
            for self.div in range(2, 128):
                q_d_f = self.q_total * self.div * fout
                if isinstance(q_d_f, six.integer_types) and q_d_f > (15 * self.fref):  # = f0 * p
                    if int(q_d_f) % int(self.fref) == 0:  # p, q, and d found
                        self.p_total = q_d_f / self.fref
                        while self.p_total <= 16:  # counter constraint
                            self.p_total = self.p_total * 2
                            self.div = self.div * 2
                            if self.div > 127:
                                break
                            if self.p_total > 1023:
                                break
                        if ((self.fref * self.p_total / self.q_total) < 100 or (self.fref * self.p_total / self.q_total) > 400):  # PLL constraint
                            break
                        if int(self.p_total) % 2 == 0:
                            self.p_0 = 0
                        else:
                            self.p_0 = 1
                        self.p_counter = ((int(self.p_total) - self.p_0) / 2) - 4  # set p counter value

                        if self.div == 2:
                            self.clk1SRC = 0x02
                            self.div1N = 4
                        else:
                            if self.div == 3:
                                self.clk1SRC = 0x03
                                self.div1N = 6
                            else:
                                self.clk1SRC = 0x01
                                self.div1N = self.div

                        if self.p_total <= 44:
                            self.chg_pump = 0
                        else:
                            if self.p_total <= 479:
                                self.chg_pump = 1
                            else:
                                if self.p_total <= 639:
                                    self.chg_pump = 2
                                else:
                                    if self.p_total <= 799:
                                        self.chg_pump = 3
                                    else:
                                        if self.p_total <= 1023:
                                            self.chg_pump = 4
                        ftest = self.fref * self.p_total / self.q_total * 1 / self.div
                        fvco = self.fref * self.p_total / self.q_total
                        logger.info('PLL frequency set to ' + str(ftest) + ' MHz' + ' (VCO @ ' + str(fvco) + ' MHz)')
                        return True
        logger.error('MIO_PLL: Could not find PLL parameters')
        return False

Response: This method calculates the parameters of the base class.


Instruction:
Write a Python 3 script to
initialize the device. Parameters of visa.ResourceManager().open_resource()

Response:
def init(self):
        '''
        Initialize the device.
        Parameters of visa.ResourceManager().open_resource()
        '''
        super(Visa, self).init()
        backend = self._init.get('backend', '')  # Empty string means std. backend (NI VISA)
        rm = visa.ResourceManager(backend)
        try:
            logger.info('BASIL VISA TL with %s backend found the following devices: %s', backend, ", ".join(rm.list_resources()))
        except NotImplementedError:  # some backends do not always implement the list_resources function
            logger.info('BASIL VISA TL with %s backend', backend)
        self._resource = rm.open_resource(**{key: value for key, value in self._init.items() if key not in ("backend",)})


Instruction:
Write a Python 3 script to
read data in BRAM. returns numpy array.

Response:
def get_data(self):
        ''' Reading data in BRAM.

        Returns
        -------
        array : numpy.ndarray
            Array of unsigned integers (32 bit).
        '''
        fifo_int_size_1 = self.FIFO_INT_SIZE
        fifo_int_size_2 = self.FIFO_INT_SIZE
        if fifo_int_size_1 > fifo_int_size_2:
            fifo_int_size = fifo_int_size_2  # use smaller chunk
            logger.warning("Reading wrong FIFO size. Expected: %d <= %d" % (fifo_int_size_1, fifo_int_size_2))
        else:
            fifo_int_size = fifo_int_size_1  # use smaller chunk
        return np.frombuffer(self._intf.read(self._conf['base_data_addr'], size=4 * fifo_int_size), dtype=np.dtype('<u4'))


Instruction:
Make a summary of the following Python 3 code
def read_external(self, address):
        self.bus.RD_B <= 1
        self.bus.ADD <= self._x
        self.bus.BUS_DATA <= self._high_impedence
        for _ in range(5):
            yield RisingEdge(self.clock)

        yield RisingEdge(self.clock)
        self.bus.ADD <= address + 0x4000

        yield RisingEdge(self.clock)
        self.bus.RD_B <= 0
        yield RisingEdge(self.clock)
        self.bus.RD_B <= 0
        yield ReadOnly()
        result = self.bus.BUS_DATA.value.integer
        yield RisingEdge(self.clock)
        self.bus.RD_B <= 1

        yield RisingEdge(self.clock)
        self.bus.RD_B <= 1
        self.bus.ADD <= self._x

        yield RisingEdge(self.clock)

        for _ in range(5):
            yield RisingEdge(self.clock)

        raise ReturnValue(result)

Response: A testbench interface that reads an external bus address.


Instruction:
Given the following Python 3 function, write the documentation
def write_external(self, address, value):
        self.bus.WR_B <= 1
        self.bus.ADD <= self._x

        for _ in range(5):
            yield RisingEdge(self.clock)

        yield RisingEdge(self.clock)
        self.bus.ADD <= address + 0x4000
        self.bus.BUS_DATA <= int(value)
        yield Timer(1)  # This is hack for iverilog
        self.bus.ADD <= address + 0x4000
        self.bus.BUS_DATA <= int(value)
        yield RisingEdge(self.clock)
        self.bus.WR_B <= 0
        yield Timer(1)  # This is hack for iverilog
        self.bus.BUS_DATA <= int(value)
        self.bus.WR_B <= 0
        yield RisingEdge(self.clock)
        self.bus.WR_B <= 0
        yield Timer(1)  # This is hack for iverilog
        self.bus.BUS_DATA <= int(value)
        self.bus.WR_B <= 0
        yield RisingEdge(self.clock)
        self.bus.WR_B <= 1
        self.bus.BUS_DATA <= self._high_impedence
        yield Timer(1)  # This is hack for iverilog
        self.bus.WR_B <= 1
        self.bus.BUS_DATA <= self._high_impedence
        yield RisingEdge(self.clock)
        self.bus.WR_B <= 1
        self.bus.ADD <= self._x

        for _ in range(5):
            yield RisingEdge(self.clock)

Response: Write external data to the bus.


Instruction:
Explain what the following Python 3 code does
def _read_eeprom(self, address, size):
        '''Read EEPROM
        '''
        self._intf.write(self._base_addr + self.CAL_EEPROM_ADD, array('B', pack('>H', address & 0x3FFF)))  # 14-bit address, 16384 bytes

        n_pages, n_bytes = divmod(size, self.CAL_EEPROM_PAGE_SIZE)
        data = array('B')
        for _ in range(n_pages):
            data.extend(self._intf.read(self._base_addr + self.CAL_EEPROM_ADD | 1, size=self.CAL_EEPROM_PAGE_SIZE))

        if n_bytes > 0:
            data.extend(self._intf.read(self._base_addr + self.CAL_EEPROM_ADD | 1, size=n_bytes))

        return data

Response: Read the EEPROM at the specified address and size.