Instruction:
Given the following Python 3 function, write the documentation
def delete_nodes(self, nodes: Iterable[str]):
        for n in nodes:
            if self.has_node(n):
                self.remove_node(n)

Response:
Iterate over a set of nodes and remove the ones that are present in
            the graph.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def delete_node(self, node: str):
        if self.has_node(node):
            self.remove_node(node)

Response: Removes a node from the graph if it is in the graph.


Instruction:
Can you create a Python 3 function that
removes an edge from the graph if it is in the graph.

Response:
def delete_edge(self, source: str, target: str):
        """ Removes an edge if it is in the graph. """
        if self.has_edge(source, target):
            self.remove_edge(source, target)


Instruction:
Implement a function in Python 3 to
iterate over a set of edges and remove the ones that are present in the graph.

Response:
def delete_edges(self, edges: Iterable[Tuple[str, str]]):
        """ Iterate over a set of edges and remove the ones that are present in
        the graph. """
        for edge in edges:
            if self.has_edge(*edge):
                self.remove_edge(*edge)


Instruction:
Explain what the following Python 3 code does
def prune(self, cutoff: int = 2):

        # Remove redundant paths.
        for node_pair in tqdm(list(permutations(self.nodes(), 2))):
            paths = [
                list(pairwise(path))
                for path in nx.all_simple_paths(self, *node_pair, cutoff)
            ]
            if len(paths) > 1:
                for path in paths:
                    if len(path) == 1:
                        self.delete_edge(*path[0])
                        if any(self.degree(n) == 0 for n in path[0]):
                            self.add_edge(*path[0])
                        break

Response: Removes redundant paths from the CAG.


Instruction:
Make a summary of the following Python 3 code
def merge_nodes(self, n1: str, n2: str, same_polarity: bool = True):

        for p in self.predecessors(n1):
            for st in self[p][n1]["InfluenceStatements"]:
                if not same_polarity:
                    st.obj_delta["polarity"] = -st.obj_delta["polarity"]
                st.obj.db_refs["UN"][0] = (n2, st.obj.db_refs["UN"][0][1])

            if not self.has_edge(p, n2):
                self.add_edge(p, n2)
                self[p][n2]["InfluenceStatements"] = self[p][n1][
                    "InfluenceStatements"
                ]

            else:
                self[p][n2]["InfluenceStatements"] += self[p][n1][
                    "InfluenceStatements"
                ]

        for s in self.successors(n1):
            for st in self.edges[n1, s]["InfluenceStatements"]:
                if not same_polarity:
                    st.subj_delta["polarity"] = -st.subj_delta["polarity"]
                st.subj.db_refs["UN"][0] = (n2, st.subj.db_refs["UN"][0][1])

            if not self.has_edge(n2, s):
                self.add_edge(n2, s)
                self[n2][s]["InfluenceStatements"] = self[n1][s][
                    "InfluenceStatements"
                ]
            else:
                self[n2][s]["InfluenceStatements"] += self[n1][s][
                    "InfluenceStatements"
                ]

        self.remove_node(n1)

Response: Merge two nodes into one.


Instruction:
Can you implement a function in Python 3 that
returns a new subgraph of the analysis graph for a single concept.

Response:
def get_subgraph_for_concept(
        self, concept: str, depth: int = 1, reverse: bool = False
    ):
        """ Returns a new subgraph of the analysis graph for a single concept.

        Args:
            concept: The concept that the subgraph will be centered around.
            depth: The depth to which the depth-first search must be performed.

            reverse: Sets the direction of causal influence flow to examine.
                Setting this to False (default) will search for upstream causal
                influences, and setting it to True will search for
                downstream causal influences.

        Returns:
            AnalysisGraph
        """

        nodeset = {concept}

        if reverse:
            func = self.predecessors
        else:
            func = self.successors
        for i in range(depth):
            nodeset.update(
                chain.from_iterable([list(func(n)) for n in nodeset])
            )

        return AnalysisGraph(self.subgraph(nodeset).copy())


Instruction:
Can you implement a function in Python 3 that
returns an AnalysisGraph comprised of simple paths between the source and target.

Response:
def get_subgraph_for_concept_pair(
        self, source: str, target: str, cutoff: Optional[int] = None
    ):
        """ Get subgraph comprised of simple paths between the source and the
        target.

        Args:
            source
            target
            cutoff
        """
        paths = nx.all_simple_paths(self, source, target, cutoff=cutoff)
        return AnalysisGraph(self.subgraph(set(chain.from_iterable(paths))))


Instruction:
How would you explain what the following Python 3 function does
def get_subgraph_for_concept_pairs(
        self, concepts: List[str], cutoff: Optional[int] = None
    ):
        """ Get subgraph comprised of simple paths between the source and the
        target.

        Args:
            concepts
            cutoff
        """
        path_generator = (
            nx.all_simple_paths(self, source, target, cutoff=cutoff)
            for source, target in permutations(concepts, 2)
        )
        paths = chain.from_iterable(path_generator)
        return AnalysisGraph(self.subgraph(set(chain.from_iterable(paths))))

Response: Returns an AnalysisGraph comprised of simple paths between the source and target and the given concepts.


Instruction:
Implement a Python 3 function for
inserting the model into the SQLite3 database associated with the ICM REST API.

Response:
def to_sql(self, app=None, last_known_value_date: Optional[date] = None):
        """ Inserts the model into the SQLite3 database associated with Delphi,
        for use with the ICM REST API. """

        from delphi.apps.rest_api import create_app, db

        self.assign_uuids_to_nodes_and_edges()
        icm_metadata = ICMMetadata(
            id=self.id,
            created=self.dateCreated,
            estimatedNumberOfPrimitives=len(self.nodes) + len(self.edges),
            createdByUser_id=1,
            lastAccessedByUser_id=1,
            lastUpdatedByUser_id=1,
        )
        if last_known_value_date is None:
            today = date.today().isoformat()
        else:
            today = last_known_value_date.isoformat()
        default_latent_var_value = 1.0
        causal_primitives = []
        nodeset = {n.split("/")[-1] for n in self.nodes}
        simplified_labels = len(nodeset) == len(self)
        for n in self.nodes(data=True):
            n[1]["rv"] = LatentVar(n[0])
            n[1]["update_function"] = self.default_update_function
            rv = n[1]["rv"]
            rv.dataset = [default_latent_var_value for _ in range(self.res)]

            causal_variable = CausalVariable(
                id=n[1]["id"],
                model_id=self.id,
                units="",
                namespaces={},
                auxiliaryProperties=[],
                label=n[0].split("/")[-1].replace("_", " ").capitalize()
                if simplified_labels
                else n[0],
                description=n[0],
                lastUpdated=today,
                confidence=1.0,
                lastKnownValue={
                    "active": "ACTIVE",
                    "trend": None,
                    "time": today,
                    "value": {
                        "baseType": "FloatValue",
                        "value": n[1]["rv"].dataset[0],
                    },
                },
                range={
                    "baseType": "FloatRange",
                    "range": {"min": 0, "max": 5, "step": 0.1},
                },
            )
            causal_primitives.append(causal_variable)

        max_mean_betas = max(
            [abs(np.median(e[2]["βs"])) for e in self.edges(data=True)]
        )
        for e in self.edges(data=True):
            causal_relationship = CausalRelationship(
                id=e[2]["id"],
                namespaces={},
                source={
                    "id": self.nodes[e[0]]["id"],
                    "baseType": "CausalVariable",
                },
                target={
                    "id": self.nodes[e[1]]["id"],
                    "baseType": "CausalVariable",
                },
                model_id=self.id,
                auxiliaryProperties=[],
                lastUpdated=today,
                types=["causal"],
                description="",
                confidence=np.mean(
                    [s.belief for s in e[2]["InfluenceStatements"]]
                ),
                label="",
                strength=abs(np.median(e[2]["βs"]) / max_mean_betas),
                reinforcement=(
                    True
                    if np.mean(
                        [
                            stmt.subj_delta["polarity"]
                            * stmt.obj_delta["polarity"]
                            for stmt in e[2]["InfluenceStatements"]
                        ]
                    )
                    > 0
                    else False
                ),
            )
            causal_primitives.append(causal_relationship)
        evidences = []
        for edge in self.edges(data=True):
            for stmt in edge[2]["InfluenceStatements"]:
                for ev in stmt.evidence:
                    evidence = Evidence(
                        id=str(uuid4()),
                        causalrelationship_id=edge[2]["id"],
                        # TODO - make source and target appear in CauseEx HMI
                        description=(ev.text),
                    )
                    evidences.append(evidence)

        if app is None:
            app = create_app()

        with app.app_context():
            db.drop_all()
            db.create_all()
            db.session.add(icm_metadata)
            db.session.add(DelphiModel(id=self.id, model=self))
            for causal_primitive in causal_primitives:
                db.session.add(causal_primitive)
            for evidence in evidences:
                db.session.add(evidence)
            db.session.commit()


Instruction:
Can you generate the documentation for the following Python 3 function
def to_agraph(
        self,
        indicators: bool = False,
        indicator_values: bool = False,
        nodes_to_highlight=None,
        *args,
        **kwargs,
    ):
        """ Exports the CAG as a pygraphviz AGraph for visualization.

        Args:
            indicators: Whether to display indicators in the AGraph
            indicator_values: Whether to display indicator values in the AGraph
            nodes_to_highlight: Nodes to highlight in the AGraph.
        Returns:
            A PyGraphviz AGraph representation of the AnalysisGraph.
        """

        from delphi.utils.misc import choose_font

        FONT = choose_font()
        A = nx.nx_agraph.to_agraph(self)
        A.graph_attr.update(
            {
                "dpi": 227,
                "fontsize": 20,
                "rankdir": kwargs.get("rankdir", "TB"),
                "fontname": FONT,
                "overlap": "scale",
                "splines": True,
            }
        )

        A.node_attr.update(
            {
                "shape": "rectangle",
                "color": "black",
                # "color": "#650021",
                "style": "rounded",
                "fontname": FONT,
            }
        )

        nodes_with_indicators = [
            n
            for n in self.nodes(data=True)
            if n[1].get("indicators") is not None
        ]

        n_max = max(
            [
                sum([len(s.evidence) for s in e[2]["InfluenceStatements"]])
                for e in self.edges(data=True)
            ]
        )

        nodeset = {n.split("/")[-1] for n in self.nodes}

        simplified_labels = len(nodeset) == len(self)
        color_str = "#650021"
        for n in self.nodes(data=True):
            if kwargs.get("values"):
                node_label = (
                    n[0].capitalize().replace("_", " ")
                    + " ("
                    + str(np.mean(n[1]["rv"].dataset))
                    + ")"
                )
            else:
                node_label = (
                    n[0].split("/")[-1].replace("_", " ").capitalize()
                    if simplified_labels
                    else n[0]
                )
            A.add_node(n[0], label=node_label)

        if list(self.edges(data=True))[0][2].get("βs") is not None:
            max_median_betas = max(
                [abs(np.median(e[2]["βs"])) for e in self.edges(data=True)]
            )

        for e in self.edges(data=True):
            # Calculate reinforcement (ad-hoc!)

            sts = e[2]["InfluenceStatements"]
            total_evidence_pieces = sum([len(s.evidence) for s in sts])
            reinforcement = (
                sum(
                    [
                        stmt.overall_polarity() * len(stmt.evidence)
                        for stmt in sts
                    ]
                )
                / total_evidence_pieces
            )
            opacity = total_evidence_pieces / n_max
            h = (opacity * 255).hex()

            if list(self.edges(data=True))[0][2].get("βs") is not None:
                penwidth = 3 * abs(np.median(e[2]["βs"]) / max_median_betas)
                cmap = cm.Greens if reinforcement > 0 else cm.Reds
                c_str = (
                    matplotlib.colors.rgb2hex(cmap(abs(reinforcement)))
                    + h[4:6]
                )
            else:
                penwidth = 1
                c_str = "black"

            A.add_edge(e[0], e[1], color=c_str, penwidth=penwidth)

        # Drawing indicator variables

        if indicators:
            for n in nodes_with_indicators:
                for indicator_name, ind in n[1]["indicators"].items():
                    node_label = _insert_line_breaks(
                        ind.name.replace("_", " "), 30
                    )
                    if indicator_values:
                        if ind.unit is not None:
                            units = f" {ind.unit}"
                        else:
                            units = ""

                        if ind.mean is not None:
                            ind_value = "{:.2f}".format(ind.mean)
                            node_label = (
                                f"{node_label}\n{ind_value} {ind.unit}"
                                f"\nSource: {ind.source}"
                                f"\nAggregation axes: {ind.aggaxes}"
                                f"\nAggregation method: {ind.aggregation_method}"
                            )

                    A.add_node(
                        indicator_name,
                        style="rounded, filled",
                        fillcolor="lightblue",
                        label=node_label,
                    )
                    A.add_edge(n[0], indicator_name, color="royalblue4")

        if nodes_to_highlight is not None:
            nodes = kwargs.pop("nodes_to_highlight")
            if isinstance(nodes, list):
                for n in nodes:
                    if n in A.nodes():
                        A.add_node(n, fontcolor="royalblue")
            elif isinstance(nodes, str):
                if n in A.nodes():
                    A.add_node(nodes, fontcolor="royalblue")

        if kwargs.get("graph_label") is not None:
            A.graph_attr["label"] = kwargs["graph_label"]

        return A

Response: Exports the CAG as a pygraphviz AGraph.


Instruction:
Make a summary of the following Python 3 code
def dump(node, annotate_fields=True, include_attributes=False, indent="  "):

    def _format(node, level=0):
        if isinstance(node, ast.AST):
            fields = [(a, _format(b, level)) for a, b in ast.iter_fields(node)]
            if include_attributes and node._attributes:
                fields.extend(
                    [
                        (a, _format(getattr(node, a), level))
                        for a in node._attributes
                    ]
                )
            return "".join(
                [
                    node.__class__.__name__,
                    "(",
                    ", ".join(
                        ("%s=%s" % field for field in fields)
                        if annotate_fields
                        else (b for a, b in fields)
                    ),
                    ")",
                ]
            )
        elif isinstance(node, list):
            lines = ["["]
            lines.extend(
                (
                    indent * (level + 2) + _format(x, level + 2) + ","
                    for x in node
                )
            )
            if len(lines) > 1:
                lines.append(indent * (level + 1) + "]")
            else:
                lines[-1] += "]"
            return "\n".join(lines)
        return repr(node)

    if not isinstance(node, ast.AST):
        raise TypeError("expected AST, got %r" % node.__class__.__name__)
    return _format(node)

Response: Return a formatted string of the tree in node.


Instruction:
Create a Python 3 function for
creating a Python dict representing the PGM and its additional metadata for PGM output.

Response:
def create_pgm_dict(
    lambdaFile: str,
    asts: List,
    file_name: str,
    mode_mapper_dict: dict,
    save_file=False,
) -> Dict:
    """ Create a Python dict representing the PGM, with additional metadata for
    JSON output. """

    lambdaStrings = ["import math\n\n"]
    state = PGMState(lambdaStrings)
    generator = GrFNGenerator()
    generator.mode_mapper = mode_mapper_dict
    pgm = generator.genPgm(asts, state, {}, "")[0]
    if pgm.get("start"):
        pgm["start"] = pgm["start"][0]
    else:
        pgm["start"] = generator.function_defs[-1]

    pgm["source"] = [[get_path(file_name, "source")]]

    # dateCreated stores the date and time on which the lambda and PGM file was created.
    # It is stored in YYYMMDD format
    pgm["dateCreated"] = f"{datetime.today().strftime('%Y%m%d')}"

    with open(lambdaFile, "w") as f:
        f.write("".join(lambdaStrings))

    # View the PGM file that will be used to build a scope tree
    if save_file:
        json.dump(pgm, open(file_name[:file_name.rfind(".")] + ".json", "w"))

    return pgm


Instruction:
Write a Python 3 script to
filter preassembled statements according to certain rules.

Response:
def filter_and_process_statements(
    sts,
    grounding_score_cutoff: float = 0.8,
    belief_score_cutoff: float = 0.85,
    concepts_of_interest: List[str] = [],
):

    """ Filter preassembled statements according to certain rules. """
    filtered_sts = []
    counters = {}

    def update_counter(counter_name):
        if counter_name in counters:
            counters[counter_name] += 1
        else:
            counters[counter_name] = 1

    for s in tqdm(sts):

        update_counter("Original number of statements")

        # Apply belief score threshold cutoff

        if not s.belief > belief_score_cutoff:
            continue
        update_counter(f"Statements with belief score > {belief_score_cutoff}")

        # Select statements with UN groundings

        if s.subj.db_refs.get("UN") is None or s.obj.db_refs.get("UN") is None:
            continue
        update_counter("Statements with UN groundings")

        # Apply grounding score cutoffs

        if not all(
            x[1] > grounding_score_cutoff
            for x in (y.db_refs["UN"][0] for y in (s.subj, s.obj))
        ):
            continue
        update_counter(
            f"Statements with subj and obj grounding scores > {grounding_score_cutoff}"
        )

        # Assign default polarities

        if s.subj_delta["polarity"] is None:
            s.subj_delta["polarity"] = 1
        if s.obj_delta["polarity"] is None:
            s.obj_delta["polarity"] = 1

        filtered_sts.append(s)

    for k, v in counters.items():
        print(f"{k}: {v}")

    return filtered_sts


Instruction:
Can you generate a brief explanation for the following Python 3 code
def create_CAG_with_indicators(input, output, filename="CAG_with_indicators.pdf"):
    with open(input, "rb") as f:
        G = pickle.load(f)
    G.map_concepts_to_indicators(min_temporal_res="month")
    G.set_indicator("UN/events/weather/precipitation", "Historical Average Total Daily Rainfall (Maize)", "DSSAT")
    G.set_indicator("UN/events/human/agriculture/food_production",
            "Historical Production (Maize)", "DSSAT")
    G.set_indicator("UN/entities/human/food/food_security", "IPC Phase Classification", "FEWSNET")
    G.set_indicator("UN/entities/food_availability", "Production, Meat indigenous, total", "FAO")
    G.set_indicator("UN/entities/human/financial/economic/market", "Inflation Rate", "ieconomics.com")
    G.set_indicator("UN/events/human/death", "Battle-related deaths", "WDI")
    with open(output, "wb") as f:
        pickle.dump(G, f)

Response: Create a CAG with mapped indicators.


Instruction:
Write a Python 3 function that can
execute the GrFN over a particular set of inputs and returns the result.

Response:
def run(
        self,
        inputs: Dict[str, Union[float, Iterable]],
        torch_size: Optional[int] = None,
    ) -> Union[float, Iterable]:
        """Executes the GrFN over a particular set of inputs and returns the
        result.

        Args:
            inputs: Input set where keys are the names of input nodes in the
              GrFN and each key points to a set of input values (or just one).

        Returns:
            A set of outputs from executing the GrFN, one for every set of
            inputs.
        """
        # Set input values
        for i in self.inputs:
            self.nodes[i]["value"] = inputs[i]

        for func_set in self.function_sets:
            for func_name in func_set:
                lambda_fn = self.nodes[func_name]["lambda_fn"]
                output_node = list(self.successors(func_name))[0]

                signature = self.nodes[func_name]["func_inputs"]
                input_values = [self.nodes[n]["value"] for n in signature]
                res = lambda_fn(*input_values)

                if torch_size is not None and len(signature) == 0:
                    self.nodes[output_node]["value"] = torch.tensor(
                        [res] * torch_size, dtype=torch.double
                    )
                else:
                    self.nodes[output_node]["value"] = res

        # Return the output
        return self.nodes[self.output_node]["value"]


Instruction:
Can you generate the documentation for the following Python 3 function
def to_CAG(self):

        G = nx.DiGraph()
        for (name, attrs) in self.nodes(data=True):
            if attrs["type"] == "variable":
                for pred_fn in self.predecessors(name):
                    if not any(
                        fn_type in pred_fn
                        for fn_type in ("condition", "decision")
                    ):
                        for pred_var in self.predecessors(pred_fn):
                            G.add_node(
                                self.nodes[pred_var]["basename"],
                                **self.nodes[pred_var],
                            )
                            G.add_node(attrs["basename"], **attrs)
                            G.add_edge(
                                self.nodes[pred_var]["basename"],
                                attrs["basename"],
                            )
                if attrs["is_loop_index"]:
                    G.add_edge(attrs["basename"], attrs["basename"])

        return G

Response: Export the current state of the variable to a Causal Analysis Graph.


Instruction:
Implement a function in Python 3 to
build a GrFN object from a JSON file.

Response:
def from_json_and_lambdas(cls, file: str, lambdas):
        """Builds a GrFN from a JSON object.

        Args:
            cls: The class variable for object creation.
            file: Filename of a GrFN JSON file.

        Returns:
            type: A GroundedFunctionNetwork object.

        """
        with open(file, "r") as f:
            data = json.load(f)

        return cls.from_dict(data, lambdas)


Instruction:
Explain what the following Python 3 code does
def from_dict(cls, data: Dict, lambdas):
        functions = {d["name"]: d for d in data["functions"]}
        G = nx.DiGraph()
        scope_tree = nx.DiGraph()

        def add_variable_node(
            basename: str, parent: str, index: int, is_loop_index: bool = False
        ):
            G.add_node(
                f"{parent}::{basename}_{index}",
                type="variable",
                color="maroon",
                parent=parent,
                label=f"{basename}_{index}",
                basename=basename,
                is_loop_index=is_loop_index,
                padding=15,
                value=None,
            )

        def find_correct_scope(cur_scope, var_name):
            if cur_scope.parent is None:
                return cur_scope.name, -1

            new_scope = cur_scope.parent

            if var_name in new_scope.variables:
                return new_scope.name, new_scope.variables[var_name]
            else:
                return find_correct_scope(new_scope, var_name)

        def process_container(
            scope, loop_index_variable: Optional[str] = None
        ):
            for stmt in scope.body:
                if "name" in stmt:
                    stmt_type = functions[stmt["name"]]["type"]
                    if stmt_type in ("assign", "condition", "decision"):
                        if stmt_type == "assign":
                            if "body" in functions[stmt["name"]]:
                                stmt_type = "literal"

                        output = stmt["output"]
                        scope.variables[output["variable"]] = output["index"]
                        add_variable_node(
                            output["variable"], scope.name, output["index"]
                        )
                        G.add_edge(
                            stmt["name"],
                            f"{scope.name}::{output['variable']}_{output['index']}",
                        )

                        ordered_inputs = list()
                        for input in stmt.get("input", []):
                            parent = scope.name
                            index = input["index"]
                            base_name = input["variable"]
                            if (
                                index == -1
                                and base_name != loop_index_variable
                            ):
                                parent, index = find_correct_scope(
                                    scope, base_name
                                )

                            add_variable_node(
                                base_name,
                                parent,
                                index,
                                base_name == loop_index_variable,
                            )
                            node_name = f"{parent}::{base_name}_{index}"
                            ordered_inputs.append(node_name)
                            G.add_edge(node_name, stmt["name"])

                        G.add_node(
                            stmt["name"],
                            type="function",
                            lambda_fn=getattr(lambdas, stmt["name"]),
                            func_inputs=ordered_inputs,
                            shape="rectangle",
                            parent=scope.name,
                            label=stmt_type[0].upper(),
                            padding=10,
                        )

                    elif stmt_type == "loop_plate":
                        index_variable = functions[stmt["name"]][
                            "index_variable"
                        ]
                        scope_tree.add_node(stmt["name"], color="blue")
                        scope_tree.add_edge(scope.name, stmt["name"])
                        new_scope = ScopeNode(
                            functions[stmt["name"]], parent=scope
                        )
                        process_container(
                            new_scope, loop_index_variable=index_variable
                        )
                    else:
                        pass
                elif "function" in stmt and stmt["function"] != "print":
                    scope_tree.add_node(stmt["function"], color="green")
                    scope_tree.add_edge(scope.name, stmt["function"])
                    new_scope = ScopeNode(
                        functions[stmt["function"]], parent=scope
                    )
                    process_container(new_scope)

        root = data["start"]
        scope_tree.add_node(root, color="green")
        cur_scope = ScopeNode(functions[root])
        process_container(cur_scope)
        return cls(G, scope_tree)

Response: Builds a GrFN object from a set of extracted function data objects and a set of lambdas.


Instruction:
How would you explain what the following Python 3 function does
def from_python_file(
        cls, python_file, lambdas_path, json_filename: str, stem: str
    ):
        """Builds GrFN object from Python file."""
        with open(python_file, "r") as f:
            pySrc = f.read()
        return cls.from_python_src(pySrc, lambdas_path, json_filename, stem)

Response: Builds GrFN object from Python file.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def from_python_src(
        cls,
        pySrc,
        lambdas_path,
        json_filename: str,
        stem: str,
        save_file: bool = False,
    ):
        """Builds GrFN object from Python source code."""
        asts = [ast.parse(pySrc)]
        pgm_dict = genPGM.create_pgm_dict(
            lambdas_path,
            asts,
            json_filename,
            {"FileName": f"{stem}.py"},  # HACK
        )
        lambdas = importlib.__import__(stem + "_lambdas")
        return cls.from_dict(pgm_dict, lambdas)

Response: Builds GrFN object from Python source code.


Instruction:
Write a Python 3 script to
build a GrFN object from a Fortran file.

Response:
def from_fortran_file(cls, fortran_file: str, tmpdir: str = "."):
        """Builds GrFN object from a Fortran program."""
        stem = Path(fortran_file).stem
        if tmpdir == "." and "/" in fortran_file:
            tmpdir = Path(fortran_file).parent
        preprocessed_fortran_file = f"{tmpdir}/{stem}_preprocessed.f"
        lambdas_path = f"{tmpdir}/{stem}_lambdas.py"
        json_filename = stem + ".json"

        with open(fortran_file, "r") as f:
            inputLines = f.readlines()

        with open(preprocessed_fortran_file, "w") as f:
            f.write(preprocessor.process(inputLines))

        xml_string = sp.run(
            [
                "java",
                "fortran.ofp.FrontEnd",
                "--class",
                "fortran.ofp.XMLPrinter",
                "--verbosity",
                "0",
                preprocessed_fortran_file,
            ],
            stdout=sp.PIPE,
        ).stdout
        trees = [ET.fromstring(xml_string)]
        comments = get_comments.get_comments(preprocessed_fortran_file)
        os.remove(preprocessed_fortran_file)
        xml_to_json_translator = translate.XMLToJSONTranslator()
        outputDict = xml_to_json_translator.analyze(trees, comments)
        pySrc = pyTranslate.create_python_source_list(outputDict)[0][0]

        G = cls.from_python_src(pySrc, lambdas_path, json_filename, stem)
        return G


Instruction:
Can you write a function in Python 3 where it
creates a GroundedFunctionNetwork instance from a string with raw Fortran source code.

Response:
def from_fortran_src(cls, fortran_src: str, dir: str = "."):
        """ Create a GroundedFunctionNetwork instance from a string with raw
        Fortran code.

        Args:
            fortran_src: A string with Fortran source code.
            dir: (Optional) - the directory in which the temporary Fortran file
                will be created (make sure you have write permission!) Defaults to
                the current directory.
        Returns:
            A GroundedFunctionNetwork instance
        """
        import tempfile
        fp = tempfile.NamedTemporaryFile('w+t', delete=False, dir=dir)
        fp.writelines(fortran_src)
        fp.close()
        G = cls.from_fortran_file(fp.name, dir)
        os.remove(fp.name)
        return G


Instruction:
Here you have a function in Python 3, explain what it does
def clear(self):
        for n in self.nodes():
            if self.nodes[n]["type"] == "variable":
                self.nodes[n]["value"] = None
            elif self.nodes[n]["type"] == "function":
                self.nodes[n]["func_visited"] = False

Response: Clear variable nodes for next computation.


Instruction:
Here you have a function in Python 3, explain what it does
def to_FIB(self, other):

        if not isinstance(other, GroundedFunctionNetwork):
            raise TypeError(
                f"Expected GroundedFunctionNetwork, but got {type(other)}"
            )

        def shortname(var):
            return var[var.find("::") + 2 : var.rfind("_")]

        def shortname_vars(graph, shortname):
            return [v for v in graph.nodes() if shortname in v]

        this_var_nodes = [
            shortname(n)
            for (n, d) in self.nodes(data=True)
            if d["type"] == "variable"
        ]
        other_var_nodes = [
            shortname(n)
            for (n, d) in other.nodes(data=True)
            if d["type"] == "variable"
        ]

        shared_vars = set(this_var_nodes).intersection(set(other_var_nodes))
        full_shared_vars = {
            full_var
            for shared_var in shared_vars
            for full_var in shortname_vars(self, shared_var)
        }

        return ForwardInfluenceBlanket(self, full_shared_vars)

Response: Creates a ForwardInfluenceBlanket object representing the intersection of this model with the input model.


Instruction:
Write a Python 3 function that can
export to a PyGraphviz AGraph object.

Response:
def to_agraph(self):
        """ Export to a PyGraphviz AGraph object. """
        A = nx.nx_agraph.to_agraph(self)
        A.graph_attr.update(
            {"dpi": 227, "fontsize": 20, "fontname": "Menlo", "rankdir": "TB"}
        )
        A.node_attr.update({"fontname": "Menlo"})

        def build_tree(cluster_name, root_graph):
            subgraph_nodes = [
                n[0]
                for n in self.nodes(data=True)
                if n[1]["parent"] == cluster_name
            ]
            root_graph.add_nodes_from(subgraph_nodes)
            subgraph = root_graph.add_subgraph(
                subgraph_nodes,
                name=f"cluster_{cluster_name}",
                label=cluster_name,
                style="bold, rounded",
            )
            for n in self.scope_tree.successors(cluster_name):
                build_tree(n, subgraph)

        root = [n for n, d in self.scope_tree.in_degree() if d == 0][0]
        build_tree(root, A)
        return A


Instruction:
Given the following Python 3 function, write the documentation
def to_CAG_agraph(self):
        CAG = self.to_CAG()
        A = nx.nx_agraph.to_agraph(CAG)
        A.graph_attr.update({"dpi": 227, "fontsize": 20, "fontname": "Menlo"})
        A.node_attr.update(
            {
                "shape": "rectangle",
                "color": "#650021",
                "style": "rounded",
                "fontname": "Gill Sans",
            }
        )
        A.edge_attr.update({"color": "#650021", "arrowsize": 0.5})
        return A

Response: Returns a variable - only view of the GrFN object in the form of an AGraph.


Instruction:
How would you code a function in Python 3 to
build a PyGraphviz AGraph object corresponding to a call graph of functions.

Response:
def to_call_agraph(self):
        """ Build a PyGraphviz AGraph object corresponding to a call graph of
        functions. """

        A = nx.nx_agraph.to_agraph(self.call_graph)
        A.graph_attr.update({"dpi": 227, "fontsize": 20, "fontname": "Menlo"})
        A.node_attr.update(
            {"shape": "rectangle", "color": "#650021", "style": "rounded"}
        )
        A.edge_attr.update({"color": "#650021", "arrowsize": 0.5})
        return A


Instruction:
Can you implement a function in Python 3 that
executes the FIB over a particular set of inputs and returns the result.

Response:
def run(
        self,
        inputs: Dict[str, Union[float, Iterable]],
        covers: Dict[str, Union[float, Iterable]],
        torch_size: Optional[int] = None,
    ) -> Union[float, Iterable]:
        """Executes the FIB over a particular set of inputs and returns the
        result.
        Args:
            inputs: Input set where keys are the names of input nodes in the
              GrFN and each key points to a set of input values (or just one).
        Returns:
            A set of outputs from executing the GrFN, one for every set of
            inputs.
        """
        # Abort run if covers does not match our expected cover set
        if len(covers) != len(self.cover_nodes):
            raise ValueError("Incorrect number of cover values.")

        # Set the cover node values
        for node_name, val in covers.items():
            self.nodes[node_name]["value"] = val

        return super().run(inputs, torch_size)


Instruction:
Explain what the following Python 3 code does
def S2_surface(self, sizes, bounds, presets, covers, use_torch=False,
            num_samples = 10):
        """Calculates the sensitivity surface of a GrFN for the two variables with
        the highest S2 index.

        Args:
            num_samples: Number of samples for sensitivity analysis.
            sizes: Tuple of (number of x inputs, number of y inputs).
            bounds: Set of bounds for GrFN inputs.
            presets: Set of standard values for GrFN inputs.

        Returns:
            Tuple:
                Tuple: The names of the two variables that were selected
                Tuple: The X, Y vectors of eval values
                Z: The numpy matrix of output evaluations

        """
        args = self.inputs
        Si = self.sobol_analysis(
            num_samples,
            {
                "num_vars": len(args),
                "names": args,
                "bounds": [bounds[arg] for arg in args],
            },
            covers
        )
        S2 = Si["S2"]
        (s2_max, v1, v2) = get_max_s2_sensitivity(S2)

        x_var = args[v1]
        y_var = args[v2]
        search_space = [(x_var, bounds[x_var]), (y_var, bounds[y_var])]
        preset_vals = {
            arg: presets[arg]
            for i, arg in enumerate(args)
            if i != v1 and i != v2
        }

        X = np.linspace(*search_space[0][1], sizes[0])
        Y = np.linspace(*search_space[1][1], sizes[1])

        if use_torch:
            Xm, Ym = torch.meshgrid(torch.tensor(X), torch.tensor(Y))
            inputs = {n: torch.full_like(Xm, v) for n, v in presets.items()}
            inputs.update({search_space[0][0]: Xm, search_space[1][0]: Ym})
            Z = self.run(inputs, covers).numpy()
        else:
            Xm, Ym = np.meshgrid(X, Y)
            Z = np.zeros((len(X), len(Y)))
            for x, y in itertools.product(range(len(X)), range(len(Y))):
                inputs = {n: v for n, v in presets.items()}
                inputs.update({search_space[0][0]: x, search_space[1][0]: y})
                Z[x][y] = self.run(inputs, covers)

        return X, Y, Z, x_var, y_var

Response: Calculates the sensitivity surface of a GrFN for the two variables with the highest S2 index.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def process_direct_map(self, root, state) -> List[Dict]:

        val = {"tag": root.tag, "args": []}
        for node in root:
            val["args"] += self.parseTree(node, state)
        return [val]

Response:
Handles tags that are mapped directly from xml to IR with no
        additional processing other than recursive translation of any child
            nodes.


Instruction:
Create a Python 3 function to
parse the XML tree recursively to generate a JSON AST which can be ingested by other scripts.

Response:
def parseTree(self, root, state: ParseState) -> List[Dict]:
        """
        Parses the XML ast tree recursively to generate a JSON AST
        which can be ingested by other scripts to generate Python
        scripts.

        Args:
            root: The current root of the tree.
            state: The current state of the tree defined by an object of the
                ParseState class.

        Returns:
                ast: A JSON ast that defines the structure of the Fortran file.
        """

        if root.tag in self.AST_TAG_HANDLERS:
            return self.AST_TAG_HANDLERS[root.tag](root, state)

        elif root.tag in self.libRtns:
            return self.process_libRtn(root, state)

        else:
            prog = []
            for node in root:
                prog += self.parseTree(node, state)
            return prog


Instruction:
Explain what the following Python 3 code does
def loadFunction(self, root):
        for element in root.iter():
            if element.tag == "function":
                self.functionList.append(element.attrib["name"])

Response: Loads a list with all the functions in the Fortran File.


Instruction:
Can you tell what is the following Python 3 function doing
def analyze(
        self, trees: List[ET.ElementTree], comments: OrderedDict
    ) -> Dict:
        outputDict = {}
        ast = []

        # Parse through the ast once to identify and grab all the functions
        # present in the Fortran file.
        for tree in trees:
            self.loadFunction(tree)

        # Parse through the ast tree a second time to convert the XML ast
        # format to a format that can be used to generate Python statements.
        for tree in trees:
            ast += self.parseTree(tree, ParseState())

        """
        Find the entry point for the Fortran file.
        The entry point for a conventional Fortran file is always the PROGRAM
        section. This 'if' statement checks for the presence of a PROGRAM
        segment.

        If not found, the entry point can be any of the functions or
        subroutines in the file. So, all the functions and subroutines of the
        program are listed and included as the possible entry point.
        """
        if self.entryPoint:
            entry = {"program": self.entryPoint[0]}
        else:
            entry = {}
            if self.functionList:
                entry["function"] = self.functionList
            if self.subroutineList:
                entry["subroutine"] = self.subroutineList

        # Load the functions list and Fortran ast to a single data structure
        # which can be pickled and hence is portable across various scripts and
        # usages.
        outputDict["ast"] = ast
        outputDict["functionList"] = self.functionList
        outputDict["comments"] = comments
        return outputDict

Response: This function will load the functions and subroutines and return a dictionary of the information that can be used to generate the Fortran file.


Instruction:
How would you explain what the following Python 3 function does
def main():
    parser = ArgumentParser(
        description="Dynamic Bayes Net Executable Model",
        formatter_class=ArgumentDefaultsHelpFormatter,
    )

    def add_flag(short_arg: str, long_arg: str, help: str):
        parser.add_argument(
            "-" + short_arg, "--" + long_arg, help=help, action="store_true"
        )

    subparsers = parser.add_subparsers()

    parser_execute = subparsers.add_parser("execute", help="Model execution")
    parser_execute.set_defaults(func=execute)

    # ==========================================================================
    # Model execution options
    # ==========================================================================

    parser_execute.add_argument(
        "--input_dressed_cag",
        help="Path to the input dressed cag",
        type=str,
        default="delphi_model.pkl",
    )

    parser_execute.add_argument(
        "--steps",
        help="Number of time steps to take",
        type=partial(positive_int, "steps"),
        default=5,
    )

    parser_execute.add_argument(
        "--samples",
        help="Number of sequences to sample",
        type=partial(positive_int, "samples"),
        default=100,
    )

    parser_execute.add_argument(
        "--output_sequences",
        help="Output file containing sampled sequences",
        type=str,
        default="dbn_sampled_sequences.csv",
    )

    parser_execute.add_argument(
        "--input_variables",
        help="Path to the variables of the input dressed cag",
        type=str,
        default="bmi_config.txt",
    )

    args = parser.parse_args()

    if len(sys.argv) == 1:
        parser.print_help()

    else:
        args.func(args)

Response: The main function of the CLI.


Instruction:
Write a Python 3 function that can
construct FAO variable ontology for use with Eidos.

Response:
def construct_FAO_ontology():
    """ Construct FAO variable ontology for use with Eidos. """
    df = pd.read_csv("south_sudan_data_fao.csv")
    gb = df.groupby("Element")

    d = [
        {
            "events": [
                {
                    k: [
                        {e: [process_variable_name(k, e)]}
                        for e in list(set(gb.get_group(k)["Item"].tolist()))
                    ]
                }
                for k in gb.groups.keys()
            ]
        }
    ]

    yaml = YAML()
    yaml.default_flow_style = False

    with open("fao_variable_ontology.yml", "w") as f:
        yaml.dump(d, f)


Instruction:
How would you code a function in Python 3 to
drill down into an edge in the analysis graph and inspect its provenance.

Response:
def inspect_edge(G: AnalysisGraph, source: str, target: str):
    """ 'Drill down' into an edge in the analysis graph and inspect its
    provenance. This function prints the provenance.

    Args:
        G
        source
        target
    """

    return create_statement_inspection_table(
        G[source][target]["InfluenceStatements"]
    )


Instruction:
How would you explain what the following Python 3 function does
def _get_edge_sentences(
    G: AnalysisGraph, source: str, target: str
) -> List[str]:
    """ Return the sentences that led to the construction of a specified edge.

    Args:
        G
        source: The source of the edge.
        target: The target of the edge.
    """

    return chain.from_iterable(
        [
            [repr(e.text) for e in s.evidence]
            for s in G.edges[source, target]["InfluenceStatements"]
        ]
    )

Response: Returns the list of sentences that led to the construction of a specified edge.


Instruction:
Write a Python 3 script for
returning the NodeType given a name of a JSON function object.

Response:
def get_node_type(type_str):
    """Returns the NodeType given a name of a JSON function object."""
    if type_str == "container":
        return NodeType.CONTAINER
    elif type_str == "loop_plate":
        return NodeType.LOOP
    elif type_str == "assign":
        return NodeType.ASSIGN
    elif type_str == "condition":
        return NodeType.CONDITION
    elif type_str == "decision":
        return NodeType.DECISION
    else:
        raise ValueError("Unrecognized type string: ", type_str)


Instruction:
Can you tell what is the following Python 3 function doing
def list_data_type(type_list):
    data_type = []
    for item in type_list:
        match = re.match(r"(\d+)(.+)", item)
        if not match:
            reps = 1
            if item[0] in "FfEegG":
                data_type.append("REAL")
            elif item[0] in "Ii":
                data_type.append("INTEGER")
        else:
            reps = match.group(1)
            fmt = match.group(2)
            if "(" in fmt and "," in fmt:
                fmt = fmt[1:-1].split(",")
            elif "(" in fmt:
                fmt = [fmt[1:-1]]
            else:
                fmt = [fmt]
            for i in range(int(reps)):
                for ft in fmt:
                    if ft[0] in "FfEegG":
                        data_type.append("REAL")
                    elif ft[0] in "Ii":
                        data_type.append("INTEGER")
    return data_type

Response: This function takes a list of format specifiers and returns a list of data types represented by the format specifiers.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def init_read_line(self):
        format_list = self._format_list
        self._re_cvt = self.match_input_fmt(format_list)
        regexp0_str = "".join([subs[0] for subs in self._re_cvt])
        self._regexp_str = regexp0_str
        self._re = re.compile(regexp0_str)
        self._match_exps = [
            subs[1] for subs in self._re_cvt if subs[1] is not None
        ]
        self._divisors = [subs[2] for subs in self._re_cvt if subs[2] is not None]
        self._in_cvt_fns = [
            subs[3] for subs in self._re_cvt if subs[3] is not None
        ]
        self._read_line_init = True

Response: Initialize the internal state of the read line that is relevant to input matching


Instruction:
Write a Python 3 function that can
initialize the internal state of the write line

Response:
def init_write_line(self):
        """init_write_line() initializes fields relevant to output generation"""
        format_list = self._format_list
        output_info = self.gen_output_fmt(format_list)
        self._output_fmt = "".join([sub[0] for sub in output_info])
        self._out_gen_fmt = [sub[1] for sub in output_info if sub[1] is not None]
        self._out_widths = [sub[2] for sub in output_info if sub[2] is not None]
        self._write_line_init = True


Instruction:
Make a summary of the following Python 3 code
def read_line(self, line):

        if not self._read_line_init:
            self.init_read_line()

        match = self._re.match(line)
        assert match is not None, f"Format mismatch (line = {line})"

        matched_values = []
        for i in range(self._re.groups):
            cvt_re = self._match_exps[i]
            cvt_div = self._divisors[i]
            cvt_fn = self._in_cvt_fns[i]
            match_str = match.group(i + 1)

            match0 = re.match(cvt_re, match_str)
            if match0 is not None:
                if cvt_fn == "float":
                    if "." in match_str:
                        val = float(match_str)
                    else:
                        val = int(match_str) / cvt_div
                elif cvt_fn == "int":
                    val = int(match_str)
                else:
                    sys.stderr.write(
                        f"Unrecognized conversion function: {cvt_fn}\n"
                    )
            else:
                sys.stderr.write(
                    f"Format conversion failed: {match_str}\n"
                )

            matched_values.append(val)

        return tuple(matched_values)

Response: Match a line of input according to the format specified and return a tuple of the matched values


Instruction:
Create a Python 3 function for
writing a line of output according to the format specified to generate

Response:
def write_line(self, values):
        """
        Process a list of values according to the format specified to generate
        a line of output.
        """

        if not self._write_line_init:
            self.init_write_line()

        if len(self._out_widths) > len(values):
            raise For2PyError(f"ERROR: too few values for format {self._format_list}\n")

        out_strs = []
        for i in range(len(self._out_widths)):
            out_fmt = self._out_gen_fmt[i]
            out_width = self._out_widths[i]

            out_val = out_fmt.format(values[i])
            if len(out_val) > out_width:  # value too big for field
                out_val = "*" * out_width

            out_strs.append(out_val)

        out_str_exp = (
            '"' + self._output_fmt + '".format' + str(tuple(out_strs))
        )
        out_str = eval(out_str_exp)
        return out_str + "\n"


Instruction:
Given the following Python 3 function, write the documentation
def match_input_fmt(self, fmt_list):

        rexp_list = []
        for fmt in fmt_list:
            rexp_list.extend(self.match_input_fmt_1(fmt))

        return rexp_list

Response:
Given a list of Fortran format specifiers e. g. I5 2X F4. 1 returns a list of tuples for matching an input
        string against those format specifiers.


Instruction:
Can you create a Python 3 function that
gives a single format specifier, e.g., '2X', 'I5', etc., this function constructs a list of tuples for matching against that specifier. Each element of this list is a tuple (xtract_re, cvt_re, divisor, cvt_fn) where: xtract_re is a regular expression that extracts an input field of the requisite width; cvt_re is a regular expression that matches the character sequence extracted by xtract_re against the specified format; divisor is the value to divide by in order to get the appropriate number of decimal places if a decimal point is not given in the input value (meaningful only for floats); and cvt_fn is a string denoting the function to be used to convert the matched string to a value.

Response:
def match_input_fmt_1(self, fmt):
        """
        Given a single format specifier, e.g., '2X', 'I5', etc., this function
        constructs a list of tuples for matching against that specifier. Each
        element of this list is a tuple

               (xtract_re, cvt_re, divisor, cvt_fn)

        where:

           xtract_re is a regular expression that extracts an input field of
                   the requisite width;
           cvt_re is a regular expression that matches the character sequence
                   extracted by xtract_re against the specified format;
           divisor is the value to divide by in order to get the appropriate
                   number of decimal places if a decimal point is not given
                   in the input value (meaningful only for floats); and
           cvt_fn is a string denoting the function to be used to convert the
                   matched string to a value.
        """

        # first, remove any surrounding space
        fmt = fmt.strip()

        # get any leading digits indicating repetition
        match = re.match("(\d+)(.+)", fmt)
        if match is None:
            reps = 1
        else:
            reps = int(match.group(1))
            fmt = match.group(2)

        if fmt[0] == "(":  # process parenthesized format list recursively
            fmt = fmt[1:-1]
            fmt_list = fmt.split(",")
            rexp = self.match_input_fmt(fmt_list)
        else:
            if fmt[0] in "iI":  # integer
                sz = fmt[1:]
                xtract_rexp = "(.{" + sz + "})"  # r.e. for extraction
                leading_sp = " *"
                optional_sign = "-?"
                rexp0 = "\d+"
                rexp1 = leading_sp + optional_sign + rexp0  # r.e. for matching
                divisor = 1
                rexp = [(xtract_rexp, rexp1, divisor, "int")]

            elif fmt[0] in "xX":  # skip
                xtract_rexp = "."  # r.e. for extraction
                rexp = [(xtract_rexp, None, None, None)]

            elif fmt[0] in "fF":  # floating point
                idx0 = fmt.find(".")
                sz = fmt[1:idx0]
                divisor = 10 ** (int(fmt[idx0 + 1 :]))
                xtract_rexp = "(.{" + sz + "})"  # r.e. for extraction
                leading_sp = " *"
                optional_sign = "-?"
                rexp0 = "\d+(\.\d+)?"
                rexp1 = leading_sp + optional_sign + rexp0  # r.e. for matching
                rexp = [(xtract_rexp, rexp1, divisor, "float")]
            else:
                raise For2PyError(
                    f"ERROR: Unrecognized format specifier {fmt}\n"
                )

        # replicate the regular expression by the repetition factor in the format
        rexp *= reps

        return rexp


Instruction:
How would you implement a function in Python 3 that
gives a list of Fortran format specifiers e. g. I5 2X F4. 1 returns a list of tuples for constructing an output string based on those format specifiers.

Response:
def gen_output_fmt(self, fmt_list):
        """given a list of Fortran format specifiers, e.g., ['I5', '2X', 'F4.1'],
        this function constructs a list of tuples for constructing an output
        string based on those format specifiers."""

        rexp_list = []
        for fmt in fmt_list:
            rexp_list.extend(self.gen_output_fmt_1(fmt))

        return rexp_list


Instruction:
Make a summary of the following Python 3 code
def gen_output_fmt_1(self, fmt):

        # first, remove any surrounding space
        fmt = fmt.strip()

        # get any leading digits indicating repetition
        match = re.match("(\d+)(.+)", fmt)
        if match is None:
            reps = 1
        else:
            reps = int(match.group(1))
            fmt = match.group(2)

        if fmt[0] == "(":  # process parenthesized format list recursively
            fmt = fmt[1:-1]
            fmt_list = fmt.split(",")
            rexp = self.gen_output_fmt(fmt_list)
        else:
            if fmt[0] in "iI":  # integer
                sz = fmt[1:]
                gen_fmt = "{}"
                cvt_fmt = "{:" + str(sz) + "d}"
                rexp = [(gen_fmt, cvt_fmt, int(sz))]

            elif fmt[0] in "xX":
                gen_fmt = " "
                rexp = [(gen_fmt, None, None)]

            elif fmt[0] in "eEfFgG":  # various floating point formats
                idx0 = fmt.find(".")
                sz = fmt[1:idx0]
                suffix = fmt[idx0 + 1 :]
                # The 'E' and G formats can optionally specify the width of
                # the exponent, e.g.: 'E15.3E2'.  For now we ignore any such
                # the exponent width -- but if it's there, we need to extract
                # the sequence of digits before it.
                m = re.match("(\d+).*", suffix)
                assert m is not None, f"Improper format? '{fmt}'"
                prec = m.group(1)
                gen_fmt = "{}"
                cvt_fmt = "{:" + sz + "." + prec + fmt[0] + "}"
                rexp = [(gen_fmt, cvt_fmt, int(sz))]

            elif fmt[0] in "pP":  # scaling factor
                # For now we ignore scaling: there are lots of other things we
                # need to spend time on.  To fix later if necessary.
                rest_of_fmt = fmt[1:]
                rexp = self.gen_output_fmt_1(rest_of_fmt)

            elif fmt[0] in "'\"":  # character string
                sz = len(fmt) - 2  # -2 for the quote at either end
                gen_fmt = fmt[1:-1]
                rexp = [(gen_fmt, None, None)]

            elif fmt[0] == "/":  # newlines
                gen_fmt = "\\n" * len(fmt)
                rexp = [(gen_fmt, None, None)]

            else:
                raise For2PyError(
                    f"ERROR: Unrecognized format specifier {fmt}\n"
                )

        # replicate the regular expression by the repetition factor in the format
        rexp *= reps

        return rexp

Response: given a single format specifier get_output_fmt_1() constructs and returns a list of tuples for matching against that specifier.


Instruction:
How would you code a function in Python 3 to
construct a conditional probability density function for a particular base object.

Response:
def constructConditionalPDF(
    gb, rs: np.ndarray, e: Tuple[str, str, Dict]
) -> gaussian_kde:
    """ Construct a conditional probability density function for a particular
    AnalysisGraph edge. """

    adjective_response_dict = {}
    all_θs = []

    # Setting σ_X and σ_Y that are in Eq. 1.21 of the model document.
    # This assumes that the real-valued variables representing the abstract
    # concepts are on the order of 1.0.
    # TODO Make this more general.

    σ_X = σ_Y = 0.1

    for stmt in e[2]["InfluenceStatements"]:
        for ev in stmt.evidence:
            # To account for discrepancy between Hume and Eidos extractions
            if ev.annotations.get("subj_adjectives") is not None:
                for subj_adjective in ev.annotations["subj_adjectives"]:
                    if (
                        subj_adjective in gb.groups
                        and subj_adjective not in adjective_response_dict
                    ):
                        adjective_response_dict[subj_adjective] = get_respdevs(
                            gb.get_group(subj_adjective)
                        )
                    rs_subj = stmt.subj_delta[
                        "polarity"
                    ] * adjective_response_dict.get(subj_adjective, rs)

                    for obj_adjective in ev.annotations["obj_adjectives"]:
                        if (
                            obj_adjective in gb.groups
                            and obj_adjective not in adjective_response_dict
                        ):
                            adjective_response_dict[
                                obj_adjective
                            ] = get_respdevs(gb.get_group(obj_adjective))

                        rs_obj = stmt.obj_delta[
                            "polarity"
                        ] * adjective_response_dict.get(obj_adjective, rs)

                        xs1, ys1 = np.meshgrid(rs_subj, rs_obj, indexing="xy")
                        θs = np.arctan2(σ_Y * ys1.flatten(), xs1.flatten())
                        all_θs.append(θs)

            # Prior
            xs1, ys1 = np.meshgrid(
                stmt.subj_delta["polarity"] * rs,
                stmt.obj_delta["polarity"] * rs,
                indexing="xy",
            )
            # TODO - make the setting of σ_X and σ_Y more automated
            θs = np.arctan2(σ_Y * ys1.flatten(), σ_X * xs1.flatten())

    # all_θs.append(θs)
    # return gaussian_kde(np.concatenate(all_θs))

    if len(all_θs) == 0:
        all_θs.append(θs)
        return gaussian_kde(all_θs)
    else:
        return gaussian_kde(np.concatenate(all_θs))


Instruction:
Create a Python 3 function to
process the variable name to make it more human - readable.

Response:
def get_variable_and_source(x: str):
    """ Process the variable name to make it more human-readable. """
    xs = x.replace("\/", "|").split("/")
    xs = [x.replace("|", "/") for x in xs]
    if xs[0] == "FAO":
        return " ".join(xs[2:]), xs[0]
    else:
        return xs[-1], xs[0]


Instruction:
Here you have a function in Python 3, explain what it does
def construct_concept_to_indicator_mapping(n: int = 1) -> Dict[str, List[str]]:

    df = pd.read_sql_table("concept_to_indicator_mapping", con=engine)
    gb = df.groupby("Concept")

    _dict = {
        k: [get_variable_and_source(x) for x in take(n, v["Indicator"].values)]
        for k, v in gb
    }
    return _dict

Response: Create a dictionary mapping high - level concepts to low - level indicators.


Instruction:
Here you have a function in Python 3, explain what it does
def setup(self, builder: Builder):
        self.config = builder.configuration.mortality
        self.population_view = builder.population.get_view(['alive'], query="alive == 'alive'")
        self.randomness = builder.randomness.get_stream('mortality')

        self.mortality_rate = builder.value.register_rate_producer('mortality_rate', source=self.base_mortality_rate)

        builder.event.register_listener('time_step', self.determine_deaths)

Response: Performs this component s simulation setup.


Instruction:
Write a Python 3 script for
computing the base mortality rate for every individual in the index.

Response:
def base_mortality_rate(self, index: pd.Index) -> pd.Series:
        """Computes the base mortality rate for every individual.

        Parameters
        ----------
        index :
            A representation of the simulants to compute the base mortality
            rate for.

        Returns
        -------
            The base mortality rate for all simulants in the index.
        """
        return pd.Series(self.config.mortality_rate, index=index)


Instruction:
Can you create a Python 3 function that
determines who dies each time step.

Response:
def determine_deaths(self, event: Event):
        """Determines who dies each time step.

        Parameters
        ----------
        event :
            An event object emitted by the simulation containing an index
            representing the simulants affected by the event and timing
            information.
        """
        effective_rate = self.mortality_rate(event.index)
        effective_probability = 1 - np.exp(-effective_rate)
        draw = self.randomness.get_draw(event.index)
        affected_simulants = draw < effective_probability
        self.population_view.update(pd.Series('dead', index=event.index[affected_simulants]))


Instruction:
Explain what the following Python 3 code does
def _prep_components(component_list: Sequence[str]) -> List[Tuple[str, Tuple[str]]]:
    components = []
    for c in component_list:
        path, args_plus = c.split('(')
        cleaned_args = _clean_args(args_plus[:-1].split(','), path)
        components.append((path, cleaned_args))
    return components

Response: Transform component description strings into tuples of component paths and required arguments.


Instruction:
Here you have a function in Python 3, explain what it does
def get_components(self, component_config: Union[ConfigTree, List]) -> List:
        if isinstance(component_config, ConfigTree):
            component_list = self.parse_component_config(component_config.to_dict())
        else:  # Components were specified in a list rather than a tree.
            component_list = component_config
        component_list = _prep_components(component_list)
        return _import_and_instantiate_components(component_list)

Response: Extracts component specifications from configuration information and returns initialized components.


Instruction:
Can you generate the documentation for the following Python 3 function
def parse_component_config(self, component_config: Dict[str, Union[Dict, List]]) -> List[str]:
        return _parse_component_config(component_config)

Response: Parses a hierarchical component specification into a list of standardized component definitions.


Instruction:
Create a Python 3 function to
load the SAR format logfile in ASCII format.

Response:
def load_file(self):
        """
        Loads SAR format logfile in ASCII format (sarXX).
            :return: ``True`` if loading and parsing of file went fine, \
            ``False`` if it failed (at any point)
        """

        # We first split file into pieces
        searchunks = self._split_file()

        if searchunks:

            # And then we parse pieces into meaningful data
            usage = self._parse_file(searchunks)

            if 'CPU' in usage:
                return False

            self._sarinfo = usage
            del usage
            return True

        else:
            return False


Instruction:
Implement a Python 3 function for
returning parsed sar info of the current CPU.

Response:
def get_sar_info(self):
        """
        Returns parsed sar info
            :return: ``Dictionary``-style list of SAR data
        """

        try:
            test = self._sarinfo["CPU"]
            del test

        except KeyError:
            file_parsed = self.load_file()
            if file_parsed:
                return self._sarinfo
            else:
                return False

        except:
            ### DEBUG
            traceback.print_exc()
            return False

        return self._sarinfo


Instruction:
Can you write a function in Python 3 where it
splits the file into sections separated by the type of info we need for it.

Response:
def _split_file(self, data=''):
        """
        Splits SAR output or SAR output file (in ASCII format) in order to
        extract info we need for it, in the format we want.
            :param data: Input data instead of file
            :type data: str.
            :return: ``List``-style of SAR file sections separated by
                the type of info they contain (SAR file sections) without
                parsing what is exactly what at this point
        """

        # Filename passed checks through __init__
        if ((self.__filename and os.access(self.__filename, os.R_OK))
                or data != ''):

            fhandle = None

            if data == '':
                try:
                    fhandle = os.open(self.__filename, os.O_RDONLY)
                except OSError:
                    print(("Couldn't open file %s" % self.__filename))
                    fhandle = None

            if fhandle or data != '':

                datalength = 0

                # Dealing with mmap difference on Windows and Linux
                if platform.system() == 'Windows':
                    dataprot = mmap.ACCESS_READ
                else:
                    dataprot = mmap.PROT_READ

                if data != '':
                    fhandle = -1
                    datalength = len(data)

                    if platform.system() == 'Windows':
                        dataprot = mmap.ACCESS_READ | mmap.ACCESS_WRITE
                    else:
                        dataprot = mmap.PROT_READ | mmap.PROT_WRITE

                try:
                    if platform.system() == 'Windows':
                        sarmap = mmap.mmap(
                            fhandle, length=datalength, access=dataprot
                        )
                    else:
                        sarmap = mmap.mmap(
                            fhandle, length=datalength, prot=dataprot
                        )
                    if data != '':

                        sarmap.write(data)
                        sarmap.flush()
                        sarmap.seek(0, os.SEEK_SET)

                except (TypeError, IndexError):
                    if data == '':
                        os.close(fhandle)
                    traceback.print_exc()
                    # sys.exit(-1)
                    return False

                # Here we'll store chunks of SAR file, unparsed
                searchunks = []
                oldchunkpos = 0
                dlpos = sarmap.find('\n\n', 0)
                size = 0

                if data == '':
                    # We can do mmap.size() only on read-only mmaps
                    size = sarmap.size()
                else:
                    # Otherwise, if data was passed to us,
                    # we measure its length
                    len(data)

                # oldchunkpos = dlpos

                while dlpos > -1:  # mmap.find() returns -1 on failure.

                    tempchunk = sarmap.read(dlpos - oldchunkpos)
                    searchunks.append(tempchunk.strip())

                    # We remember position, add 2 for 2 DD's
                    # (newspaces in production). We have to remember
                    # relative value
                    oldchunkpos += (dlpos - oldchunkpos) + 2

                    # We position to new place, to be behind \n\n
                    # we've looked for.
                    try:
                        sarmap.seek(2, os.SEEK_CUR)
                    except ValueError:
                        print(('Out of bounds (%s)!\n' % (sarmap.tell())))
                    # Now we repeat find.
                    dlpos = sarmap.find("\n\n")

                # If it wasn't the end of file, we want last piece of it
                if oldchunkpos < size:
                    tempchunk = sarmap[oldchunkpos:]
                    searchunks.append(tempchunk.strip())

                sarmap.close()

            if fhandle != -1:
                os.close(fhandle)

            if searchunks:
                return searchunks
            else:
                return False

        return False


Instruction:
Can you generate the documentation for the following Python 3 function
def _parse_file(self, sar_parts):
        usage = {}
        output = {}

        # If sar_parts is a list
        if type(sar_parts) is list:
            restart_pattern = re.compile(PATTERN_RESTART)

            """ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! """
            """              ********** ATTENTION *******            """
            """ THERE CAN BE MORE THAN ONE SAME SECTION IN ONE FILE  """
            """ IF SYSTEM WAS REBOOTED DURING THE DAY                """
            """ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! """

            for PATTERNSNAME in ALL_PATTERNS:
                patterns = ALL_PATTERNS[PATTERNSNAME]
                rgxpattern = re.compile(patterns['PATTERN'])

                for part in sar_parts:

                    if rgxpattern.search(part):

                        if PATTERNSNAME in usage:
                            usage[PATTERNSNAME] += '\n' + part
                        else:
                            usage[PATTERNSNAME] = part
                            try:
                                first_line = part.split('\n')[0]
                            except IndexError:
                                first_line = part

                            self.__fields[PATTERNSNAME] = self.__find_column(patterns['FIELDS'], first_line)

                    # Try to match restart time
                    if restart_pattern.search(part):
                        pieces = part.split()
                        self.__restart_times.append(pieces[0])
                        del pieces

            del sar_parts

            # Now we have parts pulled out and combined, do further
            # processing.
            for PATTERNSNAME in ALL_PATTERNS:
                patterns = ALL_PATTERNS[PATTERNSNAME]
                output[PATTERNSNAME] = self.__split_info(usage[PATTERNSNAME], PATTERNSNAME, patterns)
            del usage

            return output

        return output

Response: Parses the SAR file and returns a dictionary of information.


Instruction:
Here you have a function in Python 3, explain what it does
def __find_column(self, column_names, part_first_line):
        part_parts = part_first_line.split()

        ### DEBUG
        # print("Parts: %s" % (part_parts))

        return_dict = {}

        counter = 0
        for piece in part_parts:
            for colname in column_names:
                pattern_re = re.compile(colname)
                if pattern_re.search(piece):
                    return_dict[colname] = counter
                    break
            counter += 1

        # Verify the content of the return dictionary, fill the blanks
        # with -1s :-)
        for colver in column_names:
            try:
                tempval = return_dict[colver]
                del tempval
            except KeyError:
                return_dict[colver] = None

        return return_dict

Response: Find the column for the column_name in sar type definition and returns its index.


Instruction:
Can you tell what is the following Python 3 function doing
def __split_info(self, info_part, patternsname, patterns):

        pattern = patterns['PATTERN']
        if pattern == '':
            return False

        return_dict = {}

        pattern_re = re.compile(pattern)

        for part_line in info_part.split('\n'):

            if part_line.strip() != '' and not pattern_re.search(part_line):

                # Take care of AM/PM timestamps in SAR file
                is_24hr = True
                is_AM = False

                if part_line[9:11] == 'AM':
                    is_24hr = False
                    is_AM = True
                elif part_line[9:11] == 'PM':
                    is_24hr = False
                    is_AM = False

                if is_24hr is False:
                    part_line =  ('%s_%s XX %s' % (part_line[:8], part_line[9:11], part_line[12:]))

                # Line is not empty, nor it's header.
                # let's hit the road Jack!
                elems = part_line.split()
                full_time = elems[0].strip()

                if full_time != "Average:":

                    # Convert time to 24hr format if needed
                    if is_24hr is False:
                        full_time = full_time[:-3]

                        # 12 is a bitch in AM/PM notation
                        if full_time[:2] == '12':
                            if is_AM is True:
                                full_time = ('%s:%s' % ('00', full_time[3:]))
                            is_AM = not is_AM

                        if is_AM is False and full_time[0:2] != '00':
                            hours = int(full_time[:2]) + 12
                            hours = ('%02d' % (hours,))
                            full_time = ('%s:%s' % (hours, full_time[3:]))

                    try:
                        blah = return_dict[full_time]
                        del blah
                    except KeyError:
                        return_dict[full_time] = {}

                    fields = self.__fields[patternsname]
                    pairs = patterns["PAIRS"]

                    for sectionname in pairs.iterkeys():

                        value = elems[fields[pairs[sectionname]]]

                        if sectionname == 'membuffer' or \
                                sectionname == 'memcache' or \
                                sectionname == 'memfree' or \
                                sectionname == 'memused' or \
                                sectionname == 'swapfree' or \
                                sectionname == 'swapused':
                            value = int(value)
                        else:
                            value = float(value)

                        if patternsname == 'CPU':
                            cpuid = elems[(1 if is_24hr is True else 2)]
                            try:
                                blah = return_dict[full_time][cpuid]
                                del blah
                            except KeyError:
                                return_dict[full_time][cpuid] = {}
                            return_dict[full_time][cpuid][sectionname] = \
                                value
                        else:
                            return_dict[full_time][sectionname] = value

        return return_dict

Response: Splits info part into logical stuff


Instruction:
Implement a function in Python 3 to
parse the ISO - style date of the SAR file and returns the ISO - style date of the SAR file.

Response:
def __get_filedate(self):
        """
        Parses (extracts) date of SAR data, from the SAR output file itself.
            :return: ISO-style (YYYY-MM-DD) date from SAR file
        """

        if os.access(self.__filename, os.R_OK):

            # Read first line of the file
            try:
                sar_file = open(self.__filename, "r")

            except OSError:
                ### DEBUG
                traceback.print_exc()
                return False

            except:
                ### DEBUG
                traceback.print_exc()
                return False

            firstline = sar_file.readline()
            info = firstline.split()
            sar_file.close()

            try:
                self.__file_date = info[3]

            except KeyError:
                self.__file_date = ''
                return False

            except:
                ### DEBUG
                traceback.print_exc()
                return False

            return True

        return False


Instruction:
Create a Python 3 function for
moving a population between different states using information from a TransitionSet.

Response:
def _next_state(index, event_time, transition_set, population_view):
    """Moves a population between different states using information from a `TransitionSet`.

    Parameters
    ----------
    index : iterable of ints
        An iterable of integer labels for the simulants.
    event_time : pandas.Timestamp
        When this transition is occurring.
    transition_set : TransitionSet
        A set of potential transitions available to the simulants.
    population_view : vivarium.framework.population.PopulationView
        A view of the internal state of the simulation.
    """

    if len(transition_set) == 0 or index.empty:
        return

    outputs, decisions = transition_set.choose_new_state(index)
    groups = _groupby_new_state(index, outputs, decisions)

    if groups:
        for output, affected_index in sorted(groups, key=lambda x: str(x[0])):
            if output == 'null_transition':
                pass
            elif isinstance(output, Transient):
                if not isinstance(output, State):
                    raise ValueError('Invalid transition output: {}'.format(output))
                output.transition_effect(affected_index, event_time, population_view)
                output.next_state(affected_index, event_time, population_view)
            elif isinstance(output, State):
                output.transition_effect(affected_index, event_time, population_view)
            else:
                raise ValueError('Invalid transition output: {}'.format(output))


Instruction:
Implement a Python 3 function for
grouping the simulants in the index by their new output state.

Response:
def _groupby_new_state(index, outputs, decisions):
    """Groups the simulants in the index by their new output state.

    Parameters
    ----------
    index : iterable of ints
        An iterable of integer labels for the simulants.
    outputs : iterable
        A list of possible output states.
    decisions : `pandas.Series`
        A series containing the name of the next state for each simulant in the index.

    Returns
    -------
    iterable of 2-tuples
        The first item in each tuple is the name of an output state and the second item
        is a `pandas.Index` representing the simulants to transition into that state.
    """
    output_map = {o: i for i, o in enumerate(outputs)}
    groups = pd.Series(index).groupby([output_map[d] for d in decisions])
    results = [(outputs[i], pd.Index(sub_group.values)) for i, sub_group in groups]
    selected_outputs = [o for o, _ in results]
    for output in outputs:
        if output not in selected_outputs:
            results.append((output, pd.Index([])))
    return results


Instruction:
How would you code a function in Python 3 to
move a population between different states using information this state s transition_set.

Response:
def next_state(self, index, event_time, population_view):
        """Moves a population between different states using information this state's `transition_set`.

        Parameters
        ----------
        index : iterable of ints
            An iterable of integer labels for the simulants.
        event_time : pandas.Timestamp
            When this transition is occurring.
        population_view : vivarium.framework.population.PopulationView
            A view of the internal state of the simulation.
        """
        return _next_state(index, event_time, self.transition_set, population_view)


Instruction:
Make a summary of the following Python 3 code
def transition_effect(self, index, event_time, population_view):
        population_view.update(pd.Series(self.state_id, index=index))
        self._transition_side_effect(index, event_time)

Response: Updates the simulation state and triggers any side - effects associated with entering this state.


Instruction:
Can you generate the documentation for the following Python 3 function
def add_transition(self, output,
                       probability_func=lambda index: np.ones(len(index), dtype=float),
                       triggered=Trigger.NOT_TRIGGERED):
        """Builds a transition from this state to the given state.

        output : State
            The end state after the transition.
        """
        t = Transition(self, output, probability_func=probability_func, triggered=triggered)
        self.transition_set.append(t)
        return t

Response: Builds a transition from this state to the given state.


Instruction:
Can you implement a function in Python 3 that
performs this component s simulation setup and return sub - components.

Response:
def setup(self, builder):
        """Performs this component's simulation setup and return sub-components.

        Parameters
        ----------
        builder : `engine.Builder`
            Interface to several simulation tools including access to common random
            number generation, in particular.

        Returns
        -------
        iterable
            This component's sub-components.
        """
        builder.components.add_components(self.transitions)
        self.random = builder.randomness.get_stream(self.key)


Instruction:
Can you implement a function in Python 3 that
selects a new state for each simulant in the index.

Response:
def choose_new_state(self, index):
        """Chooses a new state for each simulant in the index.

        Parameters
        ----------
        index : iterable of ints
            An iterable of integer labels for the simulants.

        Returns
        -------
        outputs : list
            The possible end states of this set of transitions.
        decisions: `pandas.Series`
            A series containing the name of the next state for each simulant in the index.
        """
        outputs, probabilities = zip(*[(transition.output_state, np.array(transition.probability(index)))
                                       for transition in self.transitions])
        probabilities = np.transpose(probabilities)
        outputs, probabilities = self._normalize_probabilities(outputs, probabilities)
        return outputs, self.random.choice(index, outputs, probabilities)


Instruction:
Write a Python 3 function for
normalizing probabilities to sum to 1 and add a null transition if desired.

Response:
def _normalize_probabilities(self, outputs, probabilities):
        """Normalize probabilities to sum to 1 and add a null transition if desired.

        Parameters
        ----------
        outputs : iterable
            List of possible end states corresponding to this containers transitions.
        probabilities : iterable of iterables
            A set of probability weights whose columns correspond to the end states in `outputs`
            and whose rows correspond to each simulant undergoing the transition.

        Returns
        -------
        outputs: list
            The original output list expanded to include a null transition (a transition back
            to the starting state) if requested.
        probabilities : ndarray
            The original probabilities rescaled to sum to 1 and potentially expanded to
            include a null transition weight.
        """
        outputs = list(outputs)
        total = np.sum(probabilities, axis=1)
        if self.allow_null_transition or not np.any(total):
            if np.any(total > 1+1e-08):  # Accommodate rounding errors
                raise ValueError(
                    "Null transition requested with un-normalized probability weights: {}".format(probabilities))
            total[total > 1] = 1  # Correct allowed rounding errors.
            probabilities = np.concatenate([probabilities, (1-total)[:, np.newaxis]], axis=1)
            outputs.append('null_transition')
        return outputs, probabilities/(np.sum(probabilities, axis=1)[:, np.newaxis])


Instruction:
Here you have a function in Python 3, explain what it does
def setup(self, builder):
        builder.components.add_components(self.states)
        self.population_view = builder.population.get_view([self.state_column])

Response: Performs this component s simulation setup and return sub - components.


Instruction:
Implement a function in Python 3 to
find the population in each state and moves them to the next state.

Response:
def transition(self, index, event_time):
        """Finds the population in each state and moves them to the next state.

        Parameters
        ----------
        index : iterable of ints
            An iterable of integer labels for the simulants.
        event_time : pandas.Timestamp
            The time at which this transition occurs.
        """
        for state, affected in self._get_state_pops(index):
            if not affected.empty:
                state.next_state(affected.index, event_time, self.population_view.subview([self.state_column]))


Instruction:
Can you generate the documentation for the following Python 3 function
def to_dot(self):
        from graphviz import Digraph
        dot = Digraph(format='png')
        for state in self.states:
            if isinstance(state, TransientState):
                dot.node(state.state_id, style='dashed')
            else:
                dot.node(state.state_id)
            for transition in state.transition_set:
                dot.edge(state.state_id, transition.output.state_id, transition.label())
        return dot

Response: Returns a graphviz. Digraph representation of this state machine.


Instruction:
Here you have a function in Python 3, explain what it does
def setup_components(self, builder, configuration):
        self._managers = _setup_components(builder, self._managers, configuration)
        self._components = _setup_components(builder, self._components, configuration)

Response: Apply component level configuration defaults to the global config and run setup methods on the components and managers.


Instruction:
Explain what the following Python 3 code does
def get_dataset(catalog, identifier=None, title=None):

    msg = "Se requiere un 'identifier' o 'title' para buscar el dataset."
    assert identifier or title, msg
    catalog = read_catalog_obj(catalog)

    # búsqueda optimizada por identificador
    if identifier:
        try:
            return _get_dataset_by_identifier(catalog, identifier)
        except BaseException:
            try:
                catalog._build_index()
                return _get_dataset_by_identifier(catalog, identifier)
            except BaseException:
                filtered_datasets = get_datasets(
                    catalog, {"dataset": {"identifier": identifier}})
    elif title:  # TODO: is this required?
        filtered_datasets = get_datasets(
            catalog, {"dataset": {"title": title}})

    if len(filtered_datasets) > 1:
        if identifier:
            raise ce.DatasetIdRepetitionError(
                identifier, filtered_datasets)
        elif title:
            # TODO: Improve exceptions module!
            raise ce.DatasetTitleRepetitionError(title, filtered_datasets)
    elif len(filtered_datasets) == 0:
        return None
    else:
        return filtered_datasets[0]

Response: Devuelve un Dataset del catálogo.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_catalog_metadata(catalog, exclude_meta_fields=None):
    exclude_meta_fields = exclude_meta_fields or []
    catalog_dict_copy = catalog.copy()
    del catalog_dict_copy["dataset"]

    for excluded_meta_field in exclude_meta_fields:
        catalog_dict_copy.pop(excluded_meta_field, None)

    return catalog_dict_copy

Response: Devuelve sólo la metadata de nivel catálogo.


Instruction:
Here you have a function in Python 3, explain what it does
def field_to_markdown(field):
    if "title" in field:
        field_title = "**{}**".format(field["title"])
    else:
        raise Exception("Es necesario un `title` para describir un campo.")

    field_type = " ({})".format(field["type"]) if "type" in field else ""
    field_desc = ": {}".format(
        field["description"]) if "description" in field else ""

    text_template = "{title}{type}{description}"
    text = text_template.format(title=field_title, type=field_type,
                                description=field_desc)

    return text

Response: Genera texto en markdown a partir de los metadatos de un campo.


Instruction:
Can you tell what is the following Python 3 function doing
def _dataset_report_helper(cls, dataset, catalog_homepage=None):
        publisher_name = helpers.traverse_dict(dataset, ["publisher", "name"])

        languages = cls._stringify_list(dataset.get("language"))
        super_themes = cls._stringify_list(dataset.get("superTheme"))
        themes = cls._stringify_list(dataset.get("theme"))

        def _stringify_distribution(distribution):
            title = distribution.get("title")
            url = distribution.get("downloadURL")

            return "\"{}\": {}".format(title, url)

        distributions = [d for d in dataset["distribution"]
                         if isinstance(d, dict)]

        # crea lista de distribuciones
        distributions_list = None
        if isinstance(distributions, list):
            distributions_strings = [
                _stringify_distribution(d) for d in distributions
            ]
            distributions_list = "\n\n".join(distributions_strings)

        # crea lista de formatos
        distributions_formats = json.dumps(
            helpers.count_distribution_formats_dataset(dataset))

        fields = OrderedDict()
        fields["dataset_identifier"] = dataset.get("identifier")
        fields["dataset_title"] = dataset.get("title")
        fields["dataset_accrualPeriodicity"] = dataset.get(
            "accrualPeriodicity")
        fields["dataset_description"] = dataset.get("description")
        fields["dataset_publisher_name"] = publisher_name
        fields["dataset_superTheme"] = super_themes
        fields["dataset_theme"] = themes
        fields["dataset_landingPage"] = dataset.get("landingPage")
        fields["dataset_landingPage_generated"] = cls._generate_landingPage(
            catalog_homepage, dataset.get("identifier")
        )
        fields["dataset_issued"] = dataset.get("issued")
        fields["dataset_modified"] = dataset.get("modified")
        fields["distributions_formats"] = distributions_formats
        fields["distributions_list"] = distributions_list
        fields["dataset_license"] = dataset.get("license")
        fields["dataset_language"] = languages
        fields["dataset_spatial"] = dataset.get("spatial")
        fields["dataset_temporal"] = dataset.get("temporal")

        return fields

Response: A helper method that returns a dict that contains the metadata for a dataset.


Instruction:
Given the following Python 3 function, write the documentation
def _catalog_report_helper(catalog, catalog_validation, url, catalog_id,
                               catalog_org):
        """Toma un dict con la metadata de un catálogo, y devuelve un dict con
        los valores que catalog_report() usa para reportar sobre él.

        Args:
            catalog (dict): Diccionario con la metadata de un catálogo.
            validation (dict): Resultado, únicamente a nivel catálogo, de la
                validación completa de `catalog`.

        Returns:
            dict: Diccionario con los campos a nivel catálogo que requiere
                catalog_report().
        """
        fields = OrderedDict()
        fields["catalog_metadata_url"] = url
        fields["catalog_federation_id"] = catalog_id
        fields["catalog_federation_org"] = catalog_org
        fields["catalog_title"] = catalog.get("title")
        fields["catalog_description"] = catalog.get("description")
        fields["valid_catalog_metadata"] = (
            1 if catalog_validation["status"] == "OK" else 0)

        return fields

Response: Helper function for catalog_report.


Instruction:
Can you tell what is the following Python 3 function doing
def _dataset_report(
            self, dataset, dataset_validation, dataset_index,
            catalog_fields, harvest='none', report=None, catalog_homepage=None
    ):
        """ Genera una línea del `catalog_report`, correspondiente a un dataset
        de los que conforman el catálogo analizado."""

        # hace un breve análisis de qa al dataset
        good_qa, notes = self._dataset_qa(dataset)

        dataset_report = OrderedDict(catalog_fields)
        dataset_report["valid_dataset_metadata"] = (
            1 if dataset_validation["status"] == "OK" else 0)
        dataset_report["dataset_index"] = dataset_index

        if isinstance(harvest, list):
            dataset_report["harvest"] = 1 if dataset["title"] in harvest else 0
        elif harvest == 'all':
            dataset_report["harvest"] = 1
        elif harvest == 'none':
            dataset_report["harvest"] = 0
        elif harvest == 'valid':
            dataset_report["harvest"] = (
                int(dataset_report["valid_dataset_metadata"]))
        elif harvest == 'good':
            valid_metadata = int(dataset_report["valid_dataset_metadata"]) == 1
            dataset_report["harvest"] = 1 if valid_metadata and good_qa else 0

        elif harvest == 'report':
            if not report:
                raise ValueError("""
Usted eligio 'report' como criterio de harvest, pero no proveyo un valor para
el argumento 'report'. Por favor, intentelo nuevamente.""")

            datasets_to_harvest = self._extract_datasets_to_harvest(report)
            dataset_report["harvest"] = (
                1 if (dataset_report["catalog_metadata_url"],
                      dataset.get("title")) in datasets_to_harvest
                else 0)
        else:
            raise ValueError("""
{} no es un criterio de harvest reconocido. Pruebe con 'all', 'none', 'valid' o
'report'.""".format(harvest))

        dataset_report.update(
            self._dataset_report_helper(
                dataset, catalog_homepage=catalog_homepage)
        )

        dataset_report["notas"] = "\n\n".join(notes)

        return dataset_report.copy()

Response: Genera una línea del dataset_report.