Instruction:
Can you tell what is the following Python 3 function doing
def org_find_apps(org_id,
                  name=None,
                  name_mode='exact',
                  category=None,
                  all_versions=None,
                  published=None,
                  created_by=None,
                  developer=None,
                  authorized_user=None,
                  created_after=None,
                  created_before=None,
                  modified_after=None,
                  modified_before=None,
                  describe=False,
                  limit=None,
                  return_handler=False,
                  first_page_size=100,
                  **kwargs):
    """
    :param name: Name of the app (also see *name_mode*)
    :type name: string
    :param name_mode: Method by which to interpret the *name* field
        "exact": exact match,
        "glob": use "*" and "?" as wildcards,
        "regexp": interpret as a regular expression
    :type name_mode: string
    :param category: If specified, only returns apps that are in the specified category
    :type category: string
    :param all_versions: Whether to return all versions of each app or just the version tagged "default"
    :type all_versions: boolean
    :param published: If specified, only returns results that have the specified publish status
        True for published apps,
        False for unpublished apps
    :type published: boolean
    :param created_by: If specified, only returns app versions that were created by the specified user
        (of the form "user-USERNAME")
    :type created_by: string
    :param developer: If specified, only returns apps for which the specified user (of the form "user-USERNAME")
        is a developer
    :type developer: string
    :param authorized_user: If specified, only returns apps for which the specified user (either a user ID, org ID,
        or the string "PUBLIC") appears in the app's list of authorized users
    :type authorized_user: string
    :param created_after: Timestamp after which each result was last created (see note accompanying
        :meth:`find_data_objects()` for interpretation)
    :type created_after: int or string
    :param created_before: Timestamp before which each result was last created (see note accompanying
        :meth:`find_data_objects()` for interpretation)
    :type created_before: int or string
    :param modified_after: Timestamp after which each result was last modified (see note accompanying
        :meth:`find_data_objects()` for interpretation)
    :type modified_after: int or string
    :param modified_before: Timestamp before which each result was last modified (see note accompanying
        :meth:`find_data_objects()` for interpretation)
    :type modified_before: int or string
    :param describe: Controls whether to also return the output of
        calling describe() on each app. Supply False to omit describe
        output, True to obtain the default describe output, or a dict to
        be supplied as the describe call input (which may be used to
        customize the set of fields that is returned)
    :type describe: bool or dict
    :param limit: The maximum number of results to be returned (if not specified, the number of results is unlimited)
    :type limit: int
    :param first_page_size: The number of results that the initial API call will return. Subsequent calls will raise
        this by multiplying by 2 up to a maximum of 1000.
    :type first_page_size: int
    :param return_handler: If True, yields results as dxpy object handlers (otherwise, yields each result as a dict
        with keys "id" and "project")
    :type return_handler: boolean
    :rtype: generator

    Returns a generator that yields all apps that match the query. It
    transparently handles paging through the result set if necessary.
    For all parameters that are omitted, the search is not restricted by
    the corresponding field.

    """

    query = {}
    if name is not None:
        if name_mode == 'exact':
            query['name'] = name
        elif name_mode == 'glob':
            query['name'] = {'glob': name}
        elif name_mode == 'regexp':
            query['name'] = {'regexp': name}
        else:
            raise DXError('find_apps: Unexpected value found for argument name_mode')
    if category is not None:
        query["category"] = category
    if all_versions is not None:
        query["allVersions"] = all_versions
    if published is not None:
        query["published"] = published
    if created_by is not None:
        query["createdBy"] = created_by
    if developer is not None:
        query["developer"] = developer
    if authorized_user is not None:
        query["authorizedUser"] = authorized_user
    if modified_after is not None or modified_before is not None:
        query["modified"] = {}
        if modified_after is not None:
            query["modified"]["after"] = dxpy.utils.normalize_time_input(modified_after)
        if modified_before is not None:
            query["modified"]["before"] = dxpy.utils.normalize_time_input(modified_before)
    if created_after is not None or created_before is not None:
        query["created"] = {}
        if created_after is not None:
            query["created"]["after"] = dxpy.utils.normalize_time_input(created_after)
        if created_before is not None:
            query["created"]["before"] = dxpy.utils.normalize_time_input(created_before)
    if describe is not None and describe is not False:
        query["describe"] = describe
    if limit is not None:
        query["limit"] = limit

    return _org_find(dxpy.api.org_find_apps, org_id, query)

Response: This function returns a list of apps that are available for the org.


Instruction:
Can you implement a function in Python 3 that
returns a generator that yields all orgs matching the query.

Response:
def find_orgs(query, first_page_size=10):
    """
    :param query: The input to the /system/findOrgs API method.
    :type query: dict

    :param first_page_size: The number of results that the initial
        /system/findOrgs API call will return; default 10, max 1000. Subsequent
        calls will raise the number of returned results exponentially up to a
        max of 1000.
    :type first_page_size: int

    :rtype: generator

    Returns a generator that yields all orgs matching the specified query. Will
    transparently handle pagination as necessary.
    """
    return _find(dxpy.api.system_find_orgs, query, limit=None,
                 return_handler=False, first_page_size=first_page_size)


Instruction:
Can you generate the documentation for the following Python 3 function
def _create_dirs(idir, dirs):
    '''
    Create a set of directories, so we could store the input files.
    For example, seq1 could be stored under:
        /in/seq1/NC_001122.fasta

    TODO: this call could fail, we need to report a reasonable error code

    Note that we create a directory for every file array, even if
    it has zero inputs.
    '''
    # create the <idir> itself
    file_load_utils.ensure_dir(idir)
    # create each subdir
    for d in dirs:
        file_load_utils.ensure_dir(os.path.join(idir, d))

Response: Create a set of directories that can be used to store the input files.


Instruction:
Implement a Python 3 function for
generating a dict of values for the downloaded files.

Response:
def _gen_helper_dict(filtered_inputs):
    '''
    Create a dict of values for the downloaded files. This is similar to the variables created
    when running a bash app.
    '''

    file_key_descs, _ignore = file_load_utils.analyze_bash_vars(
        file_load_utils.get_input_json_file(), None)

    flattened_dict = {}

    def add_if_no_collision(key, value, dict_):
        if key not in dict_:
            dict_[key] = value

    for input_ in filtered_inputs:
        if input_ not in file_key_descs:
            continue
        input_var_dict = file_key_descs[input_]
        add_if_no_collision(input_ + '_path', input_var_dict["path"], flattened_dict)
        add_if_no_collision(input_ + '_name', input_var_dict["basename"], flattened_dict)
        add_if_no_collision(input_ + '_prefix', input_var_dict["prefix"], flattened_dict)

    return flattened_dict


Instruction:
Write a Python 3 script to
get the number of threads that can be used in parallel.

Response:
def _get_num_parallel_threads(max_threads, num_cores, mem_available_mb):
    '''
    Ensure at least ~1.2 GB memory per thread, see PTFM-18767
    '''
    return min(max_threads, num_cores, max(int(mem_available_mb/1200), 1))


Instruction:
Given the following Python 3 function, write the documentation
def refill_paragraphs(string, ignored_prefix='    '):
    paragraphs = string.split('\n\n')
    refilled_paragraphs = [fill(paragraph) if not paragraph.startswith(ignored_prefix) else paragraph for paragraph in paragraphs]
    return '\n\n'.join(refilled_paragraphs).strip('\n')

Response:
Refills the given text where the text is composed of paragraphs
    separated by blank lines.


Instruction:
Here you have a function in Python 3, explain what it does
def format_find_results(args, results):
    if args.json:
        print(json.dumps(list(results), indent=4))
    elif args.brief:
        for result in results:
            print(result['id'])
    else:
        if args.func.__name__ in ("find_projects", "org_find_projects"):
            _format_find_projects_results(results)
        elif args.func.__name__ in ("org_find_members"):
            _format_find_org_members_results(results)
        elif args.func.__name__ in ("org_find_apps"):  # should have "find_apps" here one day
            _format_find_apps_results(results, verbose=args.verbose)

Response: Formats the output of dx find... commands for args. json and args. brief arguments.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def main(**kwargs):
    draw_lines()
    draw_histogram()
    draw_bar_chart()
    destination = "-r /report"
    if use_html:
        generate_html()
        command = "dx-build-report-html {h} {d}".format(h=html_filename, d=destination)
    else:
        command = "dx-build-report-html {l} {b} {h} {d}".format(l=lines_filename, b=bars_filename, h=histogram_filename, d=destination)
    sub_output = json.loads(subprocess.check_output(command, shell=True))
    output = {}
    output["report"] = dxpy.dxlink(sub_output["recordId"])
    return output

Response: Generates a couple of simple graphs and optionally generates an HTML file to upload them


Instruction:
Write a Python 3 function that can
draw a histogram of random numbers.

Response:
def draw_histogram():
    """
    Uses sample code from http://matplotlib.org/1.2.1/examples/api/histogram_demo.html
    """

    mu, sigma = 100, 15
    x = mu + sigma * numpy.random.randn(10000)
    fig = pyplot.figure()
    ax = fig.add_subplot(111)
    n, bins, patches = ax.hist(x, 50, normed=1, facecolor='green', alpha=0.75)

    ax.set_xlabel('Random number generated')
    ax.set_ylabel('Probability')
    ax.set_xlim(40, 160)
    ax.set_ylim(0, 0.03)
    ax.grid(True)

    pyplot.savefig(histogram_filename)


Instruction:
How would you code a function in Python 3 to
draw a bar chart for the next N elements.

Response:
def draw_bar_chart():
    """
    Uses sample code from http://matplotlib.org/1.2.1/examples/api/barchart_demo.html
    """
    N = 5
    menMeans = (20, 35, 30, 35, 27)
    menStd = (2, 3, 4, 1, 2)

    ind = numpy.arange(N)
    width = 0.35

    fig = pyplot.figure()
    ax = fig.add_subplot(111)
    rects1 = ax.bar(ind, menMeans, width, color='r', yerr=menStd)

    womenMeans = (25, 32, 34, 20, 25)
    womenStd = (3, 5, 2, 3, 3)
    rects2 = ax.bar(ind+width, womenMeans, width, color='y', yerr=womenStd)

    ax.set_ylabel('Scores')
    ax.set_title('Scores by group and color')
    ax.set_xticks(ind+width)
    ax.set_xticklabels(('G1', 'G2', 'G3', 'G4', 'G5'))

    ax.legend((rects1[0], rects2[0]), ('Red', 'Yellow'))

    def autolabel(rects):
        for rect in rects:
            height = rect.get_height()
            ax.text(rect.get_x()+rect.get_width()/2., 1.05*height, '%d' % int(height),
                    ha='center', va='bottom')

    autolabel(rects1)
    autolabel(rects2)
    pyplot.savefig(bars_filename)


Instruction:
Given the following Python 3 function, write the documentation
def draw_lines():
    r = numpy.random.randn(200)
    fig = pyplot.figure()
    ax = fig.add_subplot(111)
    ax.plot(r)

    ax.grid(True)

    pyplot.savefig(lines_filename)

Response: Draw a line between a set of random values


Instruction:
Can you tell what is the following Python 3 function doing
def generate_html():
    html_file = open(html_filename, "w")
    html_file.write("")
    html_file.write("
Here are some graphs for you!
")
    for image in [lines_filename, bars_filename, histogram_filename]:
        html_file.write("

{0}

".format(image))
    html_file.write("")
    html_file.close()

Response: Generate an HTML file incorporating the images produced by this script


Instruction:
Here you have a function in Python 3, explain what it does
def _safe_unicode(o):
    def clean(s):
        return u''.join([c if c in ASCII_PRINTABLE else '?' for c in s])
    if USING_PYTHON2:
        try:
            return unicode(o)
        except:
            try:
                s = str(o)
                try:
                    return s.decode("utf-8")
                except:
                    return clean(s[:2048]) + u" [Raw error message: " + unicode(s.encode("hex"), 'utf-8') + u"]"
            except:
                return u"(Unable to decode Python exception message)"
    else:
        return str(o)

Response: Returns an equivalent unicode object for the given object.


Instruction:
Write a Python 3 script to
format the specified exception into a string that can be used in the log file.

Response:
def _format_exception_message(e):
    """
    Formats the specified exception.
    """
    # Prevent duplication of "AppError" in places that print "AppError"
    # and then this formatted string
    if isinstance(e, dxpy.AppError):
        return _safe_unicode(e)
    if USING_PYTHON2:
        return unicode(e.__class__.__name__, 'utf-8') + ": " + _safe_unicode(e)
    else:
        return e.__class__.__name__ + ": " + _safe_unicode(e)


Instruction:
Write a Python 3 script for
triggering the execution environment entry point processor. Use this function in the program entry point code: .. code-block:: python import dxpy @dxpy.entry_point('main') def hello(i): pass dxpy.run() This method may be used to invoke the program either in a production environment (inside the execution environment) or for local debugging (in the debug harness), as follows: If the environment variable *DX_JOB_ID* is set, the processor retrieves the job with that ID from the API server. The job's *function* field indicates the function name to be invoked. That function name is looked up in the table of all methods decorated with *@dxpy.entry_point('name')* in the module from which :func:`run()` was called, and the matching method is invoked (with the job's input supplied as parameters). This is the mode of operation used in the DNAnexus execution environment. .. warning:: The parameters *function_name* and *function_input* are disregarded in this mode of operation. If the environment variable *DX_JOB_ID* is not set, the function name may be given in *function_name*; if not set, it is set by the environment variable *DX_TEST_FUNCTION*. The function input may be given in *function_input*; if not set, it is set by the local file *job_input.json* which is expected to be present. The absence of *DX_JOB_ID* signals to :func:`run()` that execution is happening in the debug harness. In this mode of operation, all calls to :func:`dxpy.bindings.dxjob.new_dxjob()` (and higher level handler methods which use it) are intercepted, and :func:`run()` is invoked instead with appropriate inputs.

Response:
def run(function_name=None, function_input=None):
    """Triggers the execution environment entry point processor.

    Use this function in the program entry point code:

    .. code-block:: python

       import dxpy

       @dxpy.entry_point('main')
       def hello(i):
           pass

       dxpy.run()

    This method may be used to invoke the program either in a production
    environment (inside the execution environment) or for local
    debugging (in the debug harness), as follows:

    If the environment variable *DX_JOB_ID* is set, the processor
    retrieves the job with that ID from the API server. The job's
    *function* field indicates the function name to be invoked. That
    function name is looked up in the table of all methods decorated
    with *@dxpy.entry_point('name')* in the module from which
    :func:`run()` was called, and the matching method is invoked (with
    the job's input supplied as parameters). This is the mode of
    operation used in the DNAnexus execution environment.

    .. warning::

       The parameters *function_name* and *function_input* are
       disregarded in this mode of operation.

    If the environment variable *DX_JOB_ID* is not set, the function
    name may be given in *function_name*; if not set, it is set by the
    environment variable *DX_TEST_FUNCTION*. The function input may be
    given in *function_input*; if not set, it is set by the local file
    *job_input.json* which is expected to be present.

    The absence of *DX_JOB_ID* signals to :func:`run()` that execution
    is happening in the debug harness. In this mode of operation, all
    calls to :func:`dxpy.bindings.dxjob.new_dxjob()` (and higher level
    handler methods which use it) are intercepted, and :func:`run()` is
    invoked instead with appropriate inputs.

    """

    global RUN_COUNT
    RUN_COUNT += 1

    dx_working_dir = os.getcwd()

    if dxpy.JOB_ID is not None:
        logging.basicConfig()

        try:
            logging.getLogger().addHandler(dxpy.DXLogHandler())
        except dxpy.exceptions.DXError:
            print("TODO: FIXME: the EE client should die if logging is not available")

        job = dxpy.describe(dxpy.JOB_ID)
    else:
        if function_name is None:
            function_name = os.environ.get('DX_TEST_FUNCTION', 'main')
        if function_input is None:
            with open("job_input.json", "r") as fh:
                function_input = json.load(fh)

        job = {'function': function_name, 'input': function_input}

    with open("job_error_reserved_space", "w") as fh:
        fh.write("This file contains reserved space for writing job errors in case the filesystem becomes full.\n" + " "*1024*64)

    print("Invoking", job.get('function'), "with", job.get('input'))

    try:
        result = ENTRY_POINT_TABLE[job['function']](**job['input'])
    except dxpy.AppError as e:
        save_error(e, dx_working_dir, error_type="AppError")
        raise
    except Exception as e:
        save_error(e, dx_working_dir)
        raise

    if result is not None:
        # TODO: protect against client removing its original working directory
        os.chdir(dx_working_dir)
        if USING_PYTHON2:
            # On python-2 we need to use binary mode
            with open("job_output.json", "wb") as fh:
                json.dump(result, fh, indent=2, cls=DXJSONEncoder)
                fh.write(b"\n")
        else:
            with open("job_output.json", "w") as fh:
                json.dump(result, fh, indent=2, cls=DXJSONEncoder)
                fh.write("\n")

    return result


Instruction:
Can you generate a brief explanation for the following Python 3 code
def entry_point(entry_point_name):
    def wrap(f):
        ENTRY_POINT_TABLE[entry_point_name] = f
        @wraps(f)
        def wrapped_f(*args, **kwargs):
            return f(*args, **kwargs)
        return wrapped_f
    return wrap

Response: Use this to decorate a DNAnexus execution environment entry point.


Instruction:
Write a Python 3 script for
getting the workflow ID and project ID from a given path.

Response:
def get_workflow_id_and_project(path):
    '''
    :param path: a path or ID to a workflow object
    :type path: string
    :returns: tuple of (workflow ID, project ID)

    Returns the workflow and project IDs from the given path if
    available; otherwise, exits with an appropriate error message.
    '''
    project, _folderpath, entity_result = try_call(resolve_existing_path, path, expected='entity')
    try:
        if entity_result is None or not entity_result['id'].startswith('workflow-'):
            raise DXCLIError('Could not resolve "' + path + '" to a workflow object')
    except:
        err_exit()
    return entity_result['id'], project


Instruction:
Can you generate a brief explanation for the following Python 3 code
def user_info(authserver_host=None, authserver_port=None):
    authserver = get_auth_server_name(authserver_host, authserver_port)
    return DXHTTPRequest(authserver + "/system/getUserInfo", {}, prepend_srv=False)

Response:
Returns the result of the user_info call against the specified auth
    server.


Instruction:
Write a Python 3 script for
returning the input directory for the current node.

Response:
def get_input_dir(job_homedir=None):
    '''
    :param job_homedir: explicit value for home directory, used for testing purposes
    :rtype: string
    :returns: path to input directory

    Returns the input directory, where all inputs are downloaded
    '''
    if job_homedir is not None:
        home_dir = job_homedir
    else:
        home_dir = environ.get('HOME')
    idir = os.path.join(home_dir, 'in')
    return idir


Instruction:
Can you implement a function in Python 3 that
returns the output directory where all outputs are created and uploaded from the base directory.

Response:
def get_output_dir(job_homedir=None):
    '''
    :param job_homedir: explicit value for home directory, used for testing purposes
    :rtype: string
    :returns: path to output directory

    Returns the output directory, where all outputs are created, and
    uploaded from
    '''
    if job_homedir is not None:
        home_dir = job_homedir
    else:
        home_dir = environ.get('HOME')
    odir = os.path.join(home_dir, 'out')
    return odir


Instruction:
Make a summary of the following Python 3 code
def rm_output_json_file():
    path = get_output_json_file()
    try:
        os.remove(path)
    except OSError as e:
        if e.errno == errno.ENOENT:
            pass
        else:
            raise

Response: Remove the output json file.


Instruction:
Implement a function in Python 3 to
ensure that the directory path exists and is a file.

Response:
def ensure_dir(path):
    """
    :param path: path to directory to be created

    Create a directory if it does not already exist.
    """
    if not os.path.exists(path):
        # path does not exist, create the directory
        os.mkdir(path)
    else:
        # The path exists, check that it is not a file
        if os.path.isfile(path):
            raise Exception("Path %s already exists, and it is a file, not a directory" % path)


Instruction:
Can you write a function in Python 3 where it
makes a valid Unix filename on Unix systems.

Response:
def make_unix_filename(fname):
    """
    :param fname: the basename of a file (e.g., xxx in /zzz/yyy/xxx).
    :returns: a valid unix filename
    :rtype: string
    :raises: DXError if the filename is invalid on a Unix system

    The problem being solved here is that *fname* is a python string, it
    may contain characters that are invalid for a file name. We replace all the slashes with %2F.
    Another issue, is that the user may choose an invalid name. Since we focus
    on Unix systems, the only possibilies are "." and "..".
    """
    # sanity check for filenames
    bad_filenames = [".", ".."]
    if fname in bad_filenames:
        raise DXError("Invalid filename {}".format(fname))
    return fname.replace('/', '%2F')


Instruction:
Write a Python 3 script for
extracting list of files from the input directory and create a set of directories to create and a set of files with sources and destinations.

Response:
def get_job_input_filenames(job_input_file):
    """Extract list of files, returns a set of directories to create, and
    a set of files, with sources and destinations. The paths created are
    relative to the input directory.

    Note: we go through file names inside arrays, and create a
    separate subdirectory for each. This avoids clobbering files when
    duplicate filenames appear in an array.
    """
    def get_input_hash():
        with open(job_input_file) as fh:
            job_input = json.load(fh)
            return job_input
    job_input = get_input_hash()

    files = collections.defaultdict(list)  # dictionary, with empty lists as default elements
    dirs = []  # directories to create under <idir>

    # Local function for adding a file to the list of files to be created
    # for example:
    #    iname == "seq1"
    #    subdir == "015"
    #    value == { "$dnanexus_link": {
    #       "project": "project-BKJfY1j0b06Z4y8PX8bQ094f",
    #       "id": "file-BKQGkgQ0b06xG5560GGQ001B"
    #    }
    # will create a record describing that the file should
    # be downloaded into seq1/015/<filename>
    def add_file(iname, subdir, value):
        if not dxpy.is_dxlink(value):
            return
        handler = dxpy.get_handler(value)
        if not isinstance(handler, dxpy.DXFile):
            return
        filename = make_unix_filename(handler.name)
        trg_dir = iname
        if subdir is not None:
            trg_dir = os.path.join(trg_dir, subdir)
        files[iname].append({'trg_fname': os.path.join(trg_dir, filename),
                             'handler': handler,
                             'src_file_id': handler.id})
        dirs.append(trg_dir)

    # An array of inputs, for a single key. A directory
    # will be created per array entry. For example, if the input key is
    # FOO, and the inputs are {A, B, C}.vcf then, the directory structure
    # will be:
    #   <idir>/FOO/00/A.vcf
    #   <idir>/FOO/01/B.vcf
    #   <idir>/FOO/02/C.vcf
    def add_file_array(input_name, links):
        num_files = len(links)
        if num_files == 0:
            return
        num_digits = len(str(num_files - 1))
        dirs.append(input_name)
        for i, link in enumerate(links):
            subdir = str(i).zfill(num_digits)
            add_file(input_name, subdir, link)

    for input_name, value in list(job_input.items()):
        if isinstance(value, list):
            # This is a file array
            add_file_array(input_name, value)
        else:
            add_file(input_name, None, value)

    ## create a dictionary of the all non-file elements
    rest_hash = {key: val for key, val in list(job_input.items()) if key not in files}
    return dirs, files, rest_hash


Instruction:
Here you have a function in Python 3, explain what it does
def get_input_spec_patterns():
    ''' Extract the inputSpec patterns, if they exist -- modifed from dx-upload-all-outputs

    Returns a dict of all patterns, with keys equal to the respective
    input parameter names.
    '''
    input_spec = None
    if 'DX_JOB_ID' in environ:
        # works in the cloud, not locally
        job_desc = dxpy.describe(dxpy.JOB_ID)
        if job_desc["function"] == "main":
            # The input spec does not apply for subjobs
            desc = dxpy.describe(job_desc.get("app", job_desc.get("applet")))
            if "inputSpec" in desc:
                input_spec = desc["inputSpec"]
    elif 'DX_TEST_DXAPP_JSON' in environ:
        # works only locally
        path_to_dxapp_json = environ['DX_TEST_DXAPP_JSON']
        with open(path_to_dxapp_json) as fd:
            dxapp_json = json.load(fd)
            input_spec = dxapp_json.get('inputSpec')

    # convert to a dictionary. Each entry in the input spec
    # has {name, class} attributes.
    if input_spec is None:
        return {}

    # For each field name, return its patterns.
    # Make sure a pattern is legal, ignore illegal patterns.
    def is_legal_pattern(pattern):
        return "*" in pattern
    patterns_dict = {}
    for spec in input_spec:
        name = spec['name']
        if 'patterns' in spec:
            patterns_dict[name] = []
            for p in spec['patterns']:
                if is_legal_pattern(p):
                    patterns_dict[name].append(p)
    return patterns_dict

Response: Extract the inputSpec patterns if they exist and return a dictionary of all patterns.


Instruction:
Explain what the following Python 3 code does
def gen_bash_vars(job_input_file, job_homedir=None, check_name_collision=True):
    file_key_descs, rest_hash = analyze_bash_vars(job_input_file, job_homedir)

    def string_of_elem(elem):
        result = None
        if isinstance(elem, basestring):
            result = elem
        elif isinstance(elem, dxpy.DXFile):
            result = json.dumps(dxpy.dxlink(elem))
        else:
            result = json.dumps(elem)
        return pipes.quote(result)

    def string_of_value(val):
        if isinstance(val, list):
            string = " ".join([string_of_elem(vitem) for vitem in val])
            return "( {} )".format(string)
        else:
            return string_of_elem(val)

    var_defs_hash = {}

    def gen_text_line_and_name_collision(key, val):
        ''' In the absence of a name collision, create a line describing a bash variable.
        '''
        if check_name_collision:
            if key not in environ and key not in var_defs_hash:
                var_defs_hash[key] = val
            else:
                sys.stderr.write(dxpy.utils.printing.fill(
                    "Creating environment variable ({}) would cause a name collision".format(key)
                ) + "\n")
        else:
            var_defs_hash[key] = val

    # Processing non-file variables before the file variables. This priorities them,
    # so that in case of name collisions, the file-variables will be dropped.
    for key, desc in list(rest_hash.items()):
        gen_text_line_and_name_collision(key, string_of_value(desc))
    for file_key, desc in list(file_key_descs.items()):
        gen_text_line_and_name_collision(file_key, string_of_value(desc['handler']))
        gen_text_line_and_name_collision(file_key + "_name", string_of_value(desc['basename']))
        gen_text_line_and_name_collision(file_key + "_prefix", string_of_value(desc['prefix']))
        gen_text_line_and_name_collision(file_key + "_path", string_of_value(desc['path']))

    return var_defs_hash

Response: Generates a list of lines for each bash variable in the job input file.


Instruction:
Write a Python 3 script to
wait for a future that completes.

Response:
def wait_for_a_future(futures, print_traceback=False):
    """
    Return the next future that completes.  If a KeyboardInterrupt is
    received, then the entire process is exited immediately.  See
    wait_for_all_futures for more notes.
    """
    while True:
        try:
            future = next(concurrent.futures.as_completed(futures, timeout=THREAD_TIMEOUT_MAX))
            break
        except concurrent.futures.TimeoutError:
            pass
        except KeyboardInterrupt:
            if print_traceback:
                traceback.print_stack()
            else:
                print('')
            os._exit(os.EX_IOERR)

    return future


Instruction:
Can you generate a brief explanation for the following Python 3 code
def wait_for_all_futures(futures, print_traceback=False):
    try:
        while True:
            waited_futures = concurrent.futures.wait(futures, timeout=60)
            if len(waited_futures.not_done) == 0:
                break
    except KeyboardInterrupt:
        if print_traceback:
            traceback.print_stack()
        else:
            print('')
        os._exit(os.EX_IOERR)

Response: Wait indefinitely for all futures in the input iterable to complete.


Instruction:
Can you generate the documentation for the following Python 3 function
def response_iterator(request_iterator, thread_pool, max_active_tasks=None, do_first_task_sequentially=True):
    tasks_in_progress = collections.deque()
    if max_active_tasks is None:
        max_active_tasks = cpu_count()

    # The following two functions facilitate GC by not adding extra variables to the enclosing scope.
    def submit_task(task_iterator, executor, futures_queue):
        retval  = next(task_iterator, None)
        if retval is None:
            return False
        task_callable, task_args, task_kwargs = retval
        task_future = executor.submit(task_callable, *task_args, **task_kwargs)
        futures_queue.append(task_future)
        return True

    def next_result(tasks_in_progress):
        future = tasks_in_progress.popleft()
        try:
            result = future.result(timeout=THREAD_TIMEOUT_MAX)
        except KeyboardInterrupt:
            print('')
            os._exit(os.EX_IOERR)
        return result

    if do_first_task_sequentially:
        task_callable, task_args, task_kwargs = next(request_iterator)
        yield task_callable(*task_args, **task_kwargs)

    for _i in range(max_active_tasks):
        retval = submit_task(request_iterator, thread_pool, tasks_in_progress)
        if not retval:
            break

    while len(tasks_in_progress) > 0:
        result = next_result(tasks_in_progress)
        submit_task(request_iterator, thread_pool, tasks_in_progress)
        yield result
        del result

Response: Returns an iterator that yields the results of the request and returns the result of the request.


Instruction:
Explain what the following Python 3 code does
def normalize_time_input(t, future=False, default_unit='ms'):
    error_msg = 'Error: Expected an int timestamp, a date format (e.g. YYYY-MM-DD), or an int with a single-letter suffix (s=seconds, m=minutes, h=hours, d=days, w=weeks, M=months, y=years; e.g. "-10d" indicates 10 days ago); but got {t}'
    if isinstance(t, basestring) and t.isdigit():
        t = int(t)

    if isinstance(t, basestring):
        try:
            t = normalize_timedelta(t)
        except ValueError:
            try:
                t = int(time.mktime(dateutil.parser.parse(t).timetuple())*1000)
                assert t > 0
            except (ValueError, OverflowError, AssertionError):
                raise ValueError(error_msg.format(t=t))
    elif isinstance(t, numbers.Integral):
        units_multipliers = {'ms': 1, 's': 1000}
        if default_unit not in units_multipliers:
            raise ValueError("Expected default_unit to be one of 's' or 'ms'")
        t = t * units_multipliers[default_unit]
    else:
        raise ValueError(error_msg.format(t=t))
    now = int(time.time()*1000)
    if t < 0 or (future and t < now):
        t += now
    return t

Response: Normalizes a time input to a number of milliseconds since epoch.


Instruction:
How would you explain what the following Python 3 function does
def normalize_timedelta(timedelta):
    try:
        return int(timedelta) * 1000
    except ValueError as e:
        t, suffix = timedelta[:-1], timedelta[-1:]
        suffix_multipliers = {'s': 1000, 'm': 1000*60, 'h': 1000*60*60, 'd': 1000*60*60*24, 'w': 1000*60*60*24*7,
                              'M': 1000*60*60*24*30, 'y': 1000*60*60*24*365}
        if suffix not in suffix_multipliers:
            raise ValueError()
        return int(t) * suffix_multipliers[suffix]

Response: Normalizes a datetime timedelta string to an integer in milliseconds.


Instruction:
Here you have a function in Python 3, explain what it does
def _dict_raise_on_duplicates(ordered_pairs):
    d = {}
    for k, v in ordered_pairs:
        if k in d:
           raise ValueError("duplicate key: %r" % (k,))
        else:
           d[k] = v
    return d

Response: Return a dictionary with the keys in ordered_pairs.


Instruction:
How would you explain what the following Python 3 function does
def update_nonce(input_params):
        '''
        Static method to return a copy of the input dictionary with an
        additional unique nonce
        :param input: an input dictionary that may be empty
        :type input: dict
        :returns an extended copy of the input with an additional nonce field

        The input dictionary is updated with a nonce only if does not already
        have a non empty nonce
        '''
        input_cp = input_params.copy()
        if len(input_cp.get('nonce', '')) == 0:
            input_cp['nonce'] = str(Nonce())
        return input_cp

Response: Static method to update the nonce field of the input dictionary with an additional unique nonce


Instruction:
Explain what the following Python 3 code does
def inline_documentation_files(json_spec, src_dir):
    # Inline description from a readme file
    if 'description' not in json_spec:
        readme_filename = None
        for filename in 'README.md', 'Readme.md', 'readme.md':
            if os.path.exists(os.path.join(src_dir, filename)):
                readme_filename = filename
                break
        if readme_filename is not None:
            with open(os.path.join(src_dir, readme_filename)) as fh:
                json_spec['description'] = fh.read()

    # Inline developerNotes from Readme.developer.md
    if 'developerNotes' not in json_spec:
        for filename in 'README.developer.md', 'Readme.developer.md', 'readme.developer.md':
            if os.path.exists(os.path.join(src_dir, filename)):
                with open(os.path.join(src_dir, filename)) as fh:
                    json_spec['developerNotes'] = fh.read()
                break

Response: Inline the contents of the README. md file into the JSON_SPEC dictionary.


Instruction:
Explain what the following Python 3 code does
def verify_developer_rights(prefixed_name):
    assert(prefixed_name.startswith('app-') or prefixed_name.startswith('globalworkflow-'))

    if prefixed_name.partition('-')[0] == 'app':
        exception_type = dxpy.app_builder.AppBuilderException
        describe_method = dxpy.api.app_describe
        exception_msg = \
            'An app with the given name already exists and you are not a developer of that app'
    else:
        exception_type = dxpy.workflow_builder.WorkflowBuilderException
        describe_method = dxpy.api.global_workflow_describe
        exception_msg = \
            'A global workflow with the given name already exists and you are not a developer of that workflow'

    name_already_exists = True
    is_developer = False
    version = None
    executable_id = None
    FoundExecutable = collections.namedtuple('FoundExecutable', ['name', 'version', 'id'])
    try:
        describe_output = describe_method(prefixed_name,
                                          input_params={"fields": {"isDeveloperFor": True,
                                                                   "version": True,
                                                                   "id": True}})
        is_developer = describe_output['isDeveloperFor']
        version = describe_output['version']
        executable_id = describe_output['id']
    except dxpy.exceptions.DXAPIError as e:
        if e.name == 'ResourceNotFound':
            name_already_exists = False
        elif e.name == 'PermissionDenied':
            raise exception_type(exception_msg)
        else:
            raise e

    if not name_already_exists:
        # This app/workflow doesn't exist yet so its creation will succeed
        # (or at least, not fail on the basis of the ACL).
        return FoundExecutable(name=None, version=None, id=None)

    name_without_prefix = prefixed_name.partition('-')[2]
    if not is_developer:
        raise exception_type('You are not a developer for {n}'.format(n=name_without_prefix))

    return FoundExecutable(name=name_without_prefix, version=version, id=executable_id)

Response: Verifies that the current user has the given developer rights to the app or global workflow with the given name.


Instruction:
Given the following Python 3 function, write the documentation
def assert_consistent_regions(from_spec, from_command_line, builder_exception):
    if from_spec is None or from_command_line is None:
        return
    if set(from_spec) != set(from_command_line):
        raise builder_exception("--region and the 'regionalOptions' key in the JSON file do not agree")

Response: Ensures that the regions passed in from_spec and from_command_line are the same.


Instruction:
Make a summary of the following Python 3 code
def assert_consistent_reg_options(exec_type, json_spec, executable_builder_exeception):
    reg_options_spec = json_spec.get('regionalOptions')
    json_fn = 'dxapp.json' if exec_type == 'app' else 'dxworkflow.json'

    if not isinstance(reg_options_spec, dict):
        raise executable_builder_exeception("The field 'regionalOptions' in  must be a mapping")
    if not reg_options_spec:
        raise executable_builder_exeception(
            "The field 'regionalOptions' in " + json_fn + " must be a non-empty mapping")
    regional_options_list = list(reg_options_spec.items())
    for region, opts_for_region in regional_options_list:
        if not isinstance(opts_for_region, dict):
            raise executable_builder_exeception("The field 'regionalOptions['" + region +
                            "']' in " + json_fn + " must be a mapping")
        if set(opts_for_region.keys()) != set(regional_options_list[0][1].keys()):
            if set(opts_for_region.keys()) - set(regional_options_list[0][1].keys()):
                with_key, without_key = region, regional_options_list[0][0]
                key_name = next(iter(set(opts_for_region.keys()) - set(regional_options_list[0][1].keys())))
            else:
                with_key, without_key = regional_options_list[0][0], region
                key_name = next(iter(set(regional_options_list[0][1].keys()) - set(opts_for_region.keys())))
            raise executable_builder_exeception(
                "All regions in regionalOptions must specify the same options; " +
                "%s was given for %s but not for %s" % (key_name, with_key, without_key)
            )

        if exec_type == 'app':
            for key in opts_for_region:
                if key in json_spec.get('runSpec', {}):
                    raise executable_builder_exeception(
                    key + " cannot be given in both runSpec and in regional options for " + region)

Response: Validates the regionalOptions field and verifies all the regions used in the current workflow are in the same options.


Instruction:
How would you explain what the following Python 3 function does
def get_enabled_regions(exec_type, json_spec, from_command_line, executable_builder_exeception):

    from_spec = json_spec.get('regionalOptions')

    if from_spec is not None:
        assert_consistent_reg_options(exec_type, json_spec, executable_builder_exeception)

    assert_consistent_regions(from_spec, from_command_line, executable_builder_exeception)

    enabled_regions = None
    if from_spec is not None:
        enabled_regions = from_spec.keys()
    elif from_command_line is not None:
        enabled_regions = from_command_line

    return enabled_regions

Response: Returns a list of regions that are enabled for the global executable.


Instruction:
Can you generate the documentation for the following Python 3 function
def describe(self, fields=None, **kwargs):
        describe_input = {}
        if fields is not None:
            describe_input['fields'] = fields
        self._desc = dxpy.api.analysis_describe(self._dxid, describe_input, **kwargs)
        return self._desc

Response: Returns the description of the current object.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def add_tags(self, tags, **kwargs):

        dxpy.api.analysis_add_tags(self._dxid, {"tags": tags}, **kwargs)

Response: Adds each of the specified tags to the analysis.


Instruction:
Write a Python 3 script for
removing the specified tags from the analysis.

Response:
def remove_tags(self, tags, **kwargs):
        """
        :param tags: Tags to remove from the analysis
        :type tags: list of strings

        Removes each of the specified tags from the analysis. Takes
        no action for tags that the analysis does not currently have.

        """

        dxpy.api.analysis_remove_tags(self._dxid, {"tags": tags}, **kwargs)


Instruction:
Write a Python 3 function for
setting the properties of the current analysis for the given key - value pairs.

Response:
def set_properties(self, properties, **kwargs):
        """
        :param properties: Property names and values given as key-value pairs of strings
        :type properties: dict

        Given key-value pairs in *properties* for property names and
        values, the properties are set on the analysis for the given
        property names. Any property with a value of :const:`None`
        indicates the property will be deleted.

        .. note:: Any existing properties not mentioned in *properties*
           are not modified by this method.

        """

        dxpy.api.analysis_set_properties(self._dxid, {"properties": properties}, **kwargs)


Instruction:
Can you write a function in Python 3 where it
returns a dictionary containing a valid reference to an output of this analysis.

Response:
def get_output_ref(self, field, index=None, metadata=None):
        '''
        :param field: Output field name of this analysis
        :type field: string
        :param index: If the referenced field is an array, optionally specify an index (starting from 0) to indicate a particular member of the array
        :type index: int
        :param metadata: If the referenced field is of a data object class, a string indicating the metadata that should be read, e.g. "name", "properties.propkey", "details.refgenome"
        :type metadata: string

        Returns a dict containing a valid reference to an output of this analysis.
        '''

        link = {"$dnanexus_link": {"analysis": self._dxid, "field": field}}
        if index is not None:
            link["$dnanexus_link"]["index"] = index
        if metadata is not None:
            link["$dnanexus_link"]["metadata"] = metadata
        return link


Instruction:
Write a Python 3 script for
checking the suggestions guidelines for any data objects that can be found in the specified app. json file and raises an exception if any of the violations are not found.

Response:
def _check_suggestions(app_json, publish=False):
    """
    Examines the specified dxapp.json file and warns about any
    violations of suggestions guidelines.

    :raises: AppBuilderException for data objects that could not be found
    """
    for input_field in app_json.get('inputSpec', []):
        for suggestion in input_field.get('suggestions', []):
            if 'project' in suggestion:
                try:
                    project = dxpy.api.project_describe(suggestion['project'], {"permissions": True})
                    if 'PUBLIC' not in project['permissions'] and publish:
                        logger.warn('Project {name} NOT PUBLIC!'.format(name=project['name']))
                except dxpy.exceptions.DXAPIError as e:
                    if e.code == 404:
                        logger.warn('Suggested project {name} does not exist, or not accessible by user'.format(
                                     name=suggestion['project']))
                if 'path' in suggestion:
                    try:
                        check_folder_exists(suggestion['project'], suggestion['path'], '')
                    except ResolutionError as e:
                        logger.warn('Folder {path} could not be found in project {project}'.format(
                                     path=suggestion['path'], project=suggestion['project']))
            if '$dnanexus_link' in suggestion:
                if suggestion['$dnanexus_link'].startswith(('file-', 'record-')):
                    try:
                        dnanexus_link = dxpy.describe(suggestion['$dnanexus_link'])
                    except dxpy.exceptions.DXAPIError as e:
                        if e.code == 404:
                            raise dxpy.app_builder.AppBuilderException(
                                'Suggested object {name} could not be found'.format(
                                    name=suggestion['$dnanexus_link']))
                    except Exception as e:
                        raise dxpy.app_builder.AppBuilderException(str(e))
            if 'value' in suggestion and isinstance(suggestion["value"], dict):
                if '$dnanexus_link' in suggestion['value']:
                    # Check if we have JSON or string
                    if isinstance(suggestion['value']['$dnanexus_link'], dict):
                        if 'project' in suggestion['value']['$dnanexus_link']:
                            try:
                                dxpy.api.project_describe(suggestion['value']['$dnanexus_link']['project'])
                            except dxpy.exceptions.DXAPIError as e:
                                if e.code == 404:
                                    logger.warn('Suggested project {name} does not exist, or not accessible by user'.format(
                                                 name=suggestion['value']['$dnanexus_link']['project']))
                    elif isinstance(suggestion['value']['$dnanexus_link'], basestring):
                        if suggestion['value']['$dnanexus_link'].startswith(('file-', 'record-')):
                            try:
                                dnanexus_link = dxpy.describe(suggestion['value']['$dnanexus_link'])
                            except dxpy.exceptions.DXAPIError as e:
                                if e.code == 404:
                                    raise dxpy.app_builder.AppBuilderException(
                                        'Suggested object {name} could not be found'.format(
                                            name=suggestion['value']['$dnanexus_link']))
                            except Exception as e:
                                raise dxpy.app_builder.AppBuilderException(str(e))


Instruction:
Make a summary of the following Python 3 code
def _lint(dxapp_json_filename, mode):

    def _find_readme(dirname):
        for basename in ['README.md', 'Readme.md', 'readme.md']:
            if os.path.exists(os.path.join(dirname, basename)):
                return os.path.join(dirname, basename)
        return None

    app_spec = json.load(open(dxapp_json_filename))

    dirname = os.path.basename(os.path.dirname(os.path.abspath(dxapp_json_filename)))

    if mode == "app":
        if 'title' not in app_spec:
            logger.warn('app is missing a title, please add one in the "title" field of dxapp.json')

        if 'summary' in app_spec:
            if app_spec['summary'].endswith('.'):
                logger.warn('summary "%s" should be a short phrase not ending in a period' % (app_spec['summary'],))
        else:
            logger.warn('app is missing a summary, please add one in the "summary" field of dxapp.json')

        readme_filename = _find_readme(os.path.dirname(dxapp_json_filename))
        if 'description' in app_spec:
            if readme_filename:
                raise dxpy.app_builder.AppBuilderException('Description was provided both in Readme.md '
                          'and in the "description" field of {file}. Please consolidate content in Readme.md '
                          'and remove the "description" field.'.format(file=dxapp_json_filename))
            if not app_spec['description'].strip().endswith('.'):
                logger.warn('"description" field should be written in complete sentences and end with a period')
        else:
            if readme_filename is None:
                logger.warn("app is missing a description, please supply one in README.md")
        if 'categories' in app_spec:
            for category in app_spec['categories']:
                if category not in APP_CATEGORIES:
                    logger.warn('app has unrecognized category "%s"' % (category,))
                if category == 'Import':
                    if 'title' in app_spec and not app_spec['title'].endswith('Importer'):
                        logger.warn('title "%s" should end in "Importer"' % (app_spec['title'],))
                if category == 'Export':
                    if 'title' in app_spec and not app_spec['title'].endswith('Exporter'):
                        logger.warn('title "%s" should end in "Exporter"' % (app_spec['title'],))

    if 'name' in app_spec:
        if app_spec['name'] != app_spec['name'].lower():
            logger.warn('name "%s" should be all lowercase' % (app_spec['name'],))
        if dirname != app_spec['name']:
            logger.warn('app name "%s" does not match containing directory "%s"' % (app_spec['name'], dirname))
    else:
        logger.warn('app is missing a name, please add one in the "name" field of dxapp.json')

    if 'version' in app_spec:
        if not dxpy.executable_builder.GLOBAL_EXEC_VERSION_RE.match(app_spec['version']):
            logger.warn('"version" %s should be semver compliant (e.g. of the form X.Y.Z)' % (app_spec['version'],))

    # Note that identical checks are performed on the server side (and
    # will cause the app build to fail), but the checks here are printed
    # sooner and multiple naming problems can be detected in a single
    # pass.
    if 'inputSpec' in app_spec:
        for i, input_field in enumerate(app_spec['inputSpec']):
            if not re.match("^[a-zA-Z_][0-9a-zA-Z_]*$", input_field['name']):
                logger.error('input %d has illegal name "%s" (must match ^[a-zA-Z_][0-9a-zA-Z_]*$)' % (i, input_field['name']))
    else:
        logger.warn("dxapp.json contains no input specification (inputSpec). Your applet will not be usable as an " +
                    "app, runnable from the GUI, or composable using workflows.")
    if 'outputSpec' in app_spec:
        for i, output_field in enumerate(app_spec['outputSpec']):
            if not re.match("^[a-zA-Z_][0-9a-zA-Z_]*$", output_field['name']):
                logger.error('output %d has illegal name "%s" (must match ^[a-zA-Z_][0-9a-zA-Z_]*$)' % (i, output_field['name']))
    else:
        logger.warn("dxapp.json contains no output specification (outputSpec). Your applet will not be usable as an " +
                    "app, runnable from the GUI, or composable using workflows.")

Response: Check the dxapp. json file and raise an exception if it is not.


Instruction:
Create a Python 3 function to
check that the code is in CODE parses as LANG.

Response:
def _check_syntax(code, lang, temp_dir, enforce=True):
    """
    Checks that the code whose text is in CODE parses as LANG.

    Raises DXSyntaxError if there is a problem and "enforce" is True.
    """
    # This function needs the language to be explicitly set, so we can
    # generate an appropriate temp filename.
    if lang == 'python2.7':
        temp_basename = 'inlined_code_from_dxapp_json.py'
    elif lang == 'bash':
        temp_basename = 'inlined_code_from_dxapp_json.sh'
    else:
        raise ValueError('lang must be one of "python2.7" or "bash"')
    # Dump the contents out to a temporary file, then call _check_file_syntax.
    with open(os.path.join(temp_dir, temp_basename), 'w') as ofile:
        ofile.write(code)
    _check_file_syntax(os.path.join(temp_dir, temp_basename), temp_dir, override_lang=lang, enforce=enforce)


Instruction:
Create a Python 3 function for
checking that the code in FILENAME parses and checks that the language of the file is correct.

Response:
def _check_file_syntax(filename, temp_dir, override_lang=None, enforce=True):
    """
    Checks that the code in FILENAME parses, attempting to autodetect
    the language if necessary.

    Raises IOError if the file cannot be read.

    Raises DXSyntaxError if there is a problem and "enforce" is True.
    """
    def check_python(filename):
        # Generate a semi-recognizable name to write the pyc to. Of
        # course it's possible that different files being scanned could
        # have the same basename, so this path won't be unique, but the
        # checks don't run concurrently so this shouldn't cause any
        # problems.
        pyc_path = os.path.join(temp_dir, os.path.basename(filename) + ".pyc")
        try:
            if USING_PYTHON2:
                filename = filename.encode(sys.getfilesystemencoding())
            py_compile.compile(filename, cfile=pyc_path, doraise=True)
        finally:
            try:
                os.unlink(pyc_path)
            except OSError:
                pass
    def check_bash(filename):
        if platform.system() == 'Windows':
            logging.warn(
                    'Skipping bash syntax check due to unavailability of bash on Windows.')
        else:
            subprocess.check_output(["/bin/bash", "-n", filename], stderr=subprocess.STDOUT)

    if override_lang == 'python2.7':
        checker_fn = check_python
    elif override_lang == 'bash':
        checker_fn = check_bash
    elif filename.endswith('.py'):
        checker_fn = check_python
    elif filename.endswith('.sh'):
        checker_fn = check_bash
    else:
        # Ignore other kinds of files.
        return

    # Do a test read of the file to catch errors like the file not
    # existing or not being readable.
    open(filename)

    try:
        checker_fn(filename)
    except subprocess.CalledProcessError as e:
        print(filename + " has a syntax error! Interpreter output:", file=sys.stderr)
        for line in e.output.strip("\n").split("\n"):
            print("  " + line.rstrip("\n"), file=sys.stderr)
        if enforce:
            raise DXSyntaxError(filename + " has a syntax error")
    except py_compile.PyCompileError as e:
        print(filename + " has a syntax error! Interpreter output:", file=sys.stderr)
        print("  " + e.msg.strip(), file=sys.stderr)
        if enforce:
            raise DXSyntaxError(e.msg.strip())


Instruction:
Can you implement a function in Python 3 that
checks that the dxapp. json file exists and parses as the type it is going to be parsed.

Response:
def _verify_app_source_dir_impl(src_dir, temp_dir, mode, enforce=True):
    """Performs syntax and lint checks on the app source.

    Precondition: the dxapp.json file exists and can be parsed.
    """
    _lint(os.path.join(src_dir, "dxapp.json"), mode)

    # Check that the entry point file parses as the type it is going to
    # be interpreted as. The extension is irrelevant.
    manifest = json.load(open(os.path.join(src_dir, "dxapp.json")))
    if "runSpec" in manifest:
        if "interpreter" not in manifest['runSpec']:
            raise dxpy.app_builder.AppBuilderException('runSpec.interpreter field was not present')

        if "release" not in manifest['runSpec'] or "distribution" not in manifest['runSpec']:
            warn_message = 'runSpec.distribution or runSpec.release was not present. These fields '
            warn_message += 'will be required in a future version of the API. Recommended value '
            warn_message += 'for distribution is \"Ubuntu\" and release - \"14.04\".'
            logger.warn(warn_message)

        if manifest['runSpec']['interpreter'] in ["python2.7", "bash"]:
            if "file" in manifest['runSpec']:
                entry_point_file = os.path.abspath(os.path.join(src_dir, manifest['runSpec']['file']))
                try:
                    _check_file_syntax(entry_point_file, temp_dir, override_lang=manifest['runSpec']['interpreter'], enforce=enforce)
                except IOError as e:
                    raise dxpy.app_builder.AppBuilderException(
                        'Could not open runSpec.file=%r. The problem was: %s' % (entry_point_file, e))
                except DXSyntaxError:
                    raise dxpy.app_builder.AppBuilderException('Entry point file %s has syntax errors, see above for details. Rerun with --no-check-syntax to proceed anyway.' % (entry_point_file,))
            elif "code" in manifest['runSpec']:
                try:
                    _check_syntax(manifest['runSpec']['code'], manifest['runSpec']['interpreter'], temp_dir, enforce=enforce)
                except DXSyntaxError:
                    raise dxpy.app_builder.AppBuilderException('Code in runSpec.code has syntax errors, see above for details. Rerun with --no-check-syntax to proceed anyway.')

            cluster_bootstrap_scripts = _get_all_cluster_bootstrap_script_names(manifest)
            for filename in cluster_bootstrap_scripts:
                abs_filename = os.path.abspath(os.path.join(src_dir, filename))
                try:
                    _check_file_syntax(abs_filename,
                                       temp_dir,
                                       override_lang=manifest['runSpec']['interpreter'],
                                       enforce=enforce)
                except IOError as e:
                    raise dxpy.app_builder.AppBuilderException(
                        'Could not open cluster bootstrap script %r. The problem was: %s' % (abs_filename, e))
                except DXSyntaxError:
                    raise dxpy.app_builder.AppBuilderException('Code in cluster bootstrapScript %r has syntax errors, see above for details. Rerun with --no-check-syntax to proceed anyway.' % filename)

        if 'execDepends' in manifest['runSpec']:
            if not isinstance(manifest['runSpec']['execDepends'], list):
                raise dxpy.app_builder.AppBuilderException('Expected runSpec.execDepends to be an array. Rerun with --no-check-syntax to proceed anyway.')
            if not all(isinstance(dep, dict) for dep in manifest['runSpec']['execDepends']):
                raise dxpy.app_builder.AppBuilderException('Expected runSpec.execDepends to be an array of hashes. Rerun with --no-check-syntax to proceed anyway.')
            if any(dep.get('package_manager', 'apt') != 'apt' for dep in manifest['runSpec']['execDepends']):
                if not isinstance(manifest.get('access'), dict) or 'network' not in manifest['access']:
                    msg = '\n'.join(['runSpec.execDepends specifies non-APT dependencies, but no network access spec is given.',
                    'Add {"access": {"network": ["*"]}} to allow dependencies to install.',
                    'See https://wiki.dnanexus.com/Developer-Tutorials/Request-Additional-App-Resources#Network-Access.',
                    'Rerun with --no-check-syntax to proceed anyway.'])
                    raise dxpy.app_builder.AppBuilderException(msg)

    if 'authorizedUsers' in manifest:
        if not isinstance(manifest['authorizedUsers'], list) or isinstance(manifest['authorizedUsers'], basestring):
            raise dxpy.app_builder.AppBuilderException('Expected authorizedUsers to be a list of strings')
        for thing in manifest['authorizedUsers']:
            if thing != 'PUBLIC' and (not isinstance(thing, basestring) or not re.match("^(org-|user-)", thing)):
                raise dxpy.app_builder.AppBuilderException('authorizedUsers field contains an entry which is not either the string "PUBLIC" or a user or org ID')

    if "pricingPolicy" in manifest:
        error_message = "\"pricingPolicy\" at the top level is not accepted. It must be specified "
        error_message += "under the \"regionalOptions\" field in all enabled regions of the app"
        raise dxpy.app_builder.AppBuilderException(error_message)

    # Check all other files that are going to be in the resources tree.
    # For these we detect the language based on the filename extension.
    # Obviously this check can have false positives, since the app can
    # execute (or not execute!) all these files in whatever way it
    # wishes, e.g. it could use Python != 2.7 or some non-bash shell.
    # Consequently errors here are non-fatal.
    files_with_problems = []
    for dirpath, dirnames, filenames in os.walk(os.path.abspath(os.path.join(src_dir, "resources"))):
        for filename in filenames:
            # On Mac OS, the resource fork for "FILE.EXT" gets tarred up
            # as a file named "._FILE.EXT". To a naive check this
            # appears to be a file of the same extension. Therefore, we
            # exclude these from syntax checking since they are likely
            # to not parse as whatever language they appear to be.
            if not filename.startswith("._"):
                try:
                    _check_file_syntax(os.path.join(dirpath, filename), temp_dir, enforce=True)
                except IOError as e:
                    raise dxpy.app_builder.AppBuilderException(
                        'Could not open file in resources directory %r. The problem was: %s' %
                        (os.path.join(dirpath, filename), e)
                    )
                except DXSyntaxError:
                    # Suppresses errors from _check_file_syntax so we
                    # only print a nice error message
                    files_with_problems.append(os.path.join(dirpath, filename))

    if files_with_problems:
        # Make a message of the form:
        #    "/path/to/my/app.py"
        # OR "/path/to/my/app.py and 3 other files"
        files_str = files_with_problems[0] if len(files_with_problems) == 1 else (files_with_problems[0] + " and " + str(len(files_with_problems) - 1) + " other file" + ("s" if len(files_with_problems) > 2 else ""))
        logging.warn('%s contained syntax errors, see above for details' % (files_str,))


Instruction:
Explain what the following Python 3 code does
def _verify_app_source_dir(src_dir, mode, enforce=True):
    temp_dir = tempfile.mkdtemp(prefix='dx-build_tmp')
    try:
        _verify_app_source_dir_impl(src_dir, temp_dir, mode, enforce=enforce)
    finally:
        shutil.rmtree(temp_dir)

Response: Performs syntax and lint checks on the app source directory.


Instruction:
Can you implement a function in Python 3 that
parses the contents of dxapp. json and returns a dict of the parsed contents. Raises AppBuilderException if this cannot be done.

Response:
def _parse_app_spec(src_dir):
    """Returns the parsed contents of dxapp.json.

    Raises either AppBuilderException or a parser error (exit codes 3 or
    2 respectively) if this cannot be done.
    """
    if not os.path.isdir(src_dir):
        parser.error("%s is not a directory" % src_dir)
    if not os.path.exists(os.path.join(src_dir, "dxapp.json")):
        raise dxpy.app_builder.AppBuilderException("Directory %s does not contain dxapp.json: not a valid DNAnexus app source directory" % src_dir)
    with open(os.path.join(src_dir, "dxapp.json")) as app_desc:
        try:
            return json_load_raise_on_duplicates(app_desc)
        except Exception as e:
            raise dxpy.app_builder.AppBuilderException("Could not parse dxapp.json file as JSON: " + str(e.args))


Instruction:
Here you have a function in Python 3, explain what it does
def _build_app(args, extra_args):

    if not args.remote:
        # LOCAL BUILD

        try:
            output = build_and_upload_locally(
                args.src_dir,
                args.mode,
                overwrite=args.overwrite,
                archive=args.archive,
                publish=args.publish,
                destination_override=args.destination,
                version_override=args.version_override,
                bill_to_override=args.bill_to,
                use_temp_build_project=args.use_temp_build_project,
                do_parallel_build=args.parallel_build,
                do_version_autonumbering=args.version_autonumbering,
                do_try_update=args.update,
                dx_toolkit_autodep=args.dx_toolkit_autodep,
                do_check_syntax=args.check_syntax,
                ensure_upload=args.ensure_upload,
                force_symlinks=args.force_symlinks,
                dry_run=args.dry_run,
                confirm=args.confirm,
                return_object_dump=args.json,
                region=args.region,
                **extra_args
                )

            if output is not None and args.run is None:
                print(json.dumps(output))
        except dxpy.app_builder.AppBuilderException as e:
            # AppBuilderException represents errors during app or applet building
            # that could reasonably have been anticipated by the user.
            print("Error: %s" % (e.args,), file=sys.stderr)
            sys.exit(3)
        except dxpy.exceptions.DXAPIError as e:
            print("Error: %s" % (e,), file=sys.stderr)
            sys.exit(3)

        if args.dry_run:
            return None

        return output['id']

    else:
        # REMOTE BUILD

        try:
            app_json = _parse_app_spec(args.src_dir)
            _check_suggestions(app_json, publish=args.publish)
            _verify_app_source_dir(args.src_dir, args.mode)
            if args.mode == "app" and not args.dry_run:
                dxpy.executable_builder.verify_developer_rights('app-' + app_json['name'])
        except dxpy.app_builder.AppBuilderException as e:
            print("Error: %s" % (e.args,), file=sys.stderr)
            sys.exit(3)

        # The following flags might be useful in conjunction with
        # --remote. To enable these, we need to learn how to pass these
        # options through to the interior call of dx_build_app(let).
        if args.dry_run:
            parser.error('--remote cannot be combined with --dry-run')
        if args.overwrite:
            parser.error('--remote cannot be combined with --overwrite/-f')
        if args.archive:
            parser.error('--remote cannot be combined with --archive/-a')

        # The following flags are probably not useful in conjunction
        # with --remote.
        if args.json:
            parser.error('--remote cannot be combined with --json')
        if not args.use_temp_build_project:
            parser.error('--remote cannot be combined with --no-temp-build-project')

        if isinstance(args.region, list) and len(args.region) > 1:
            parser.error('--region can only be specified once for remote builds')
        region = args.region[0] if args.region is not None else None

        more_kwargs = {}
        if args.version_override:
            more_kwargs['version_override'] = args.version_override
        if args.bill_to:
            more_kwargs['bill_to_override'] = args.bill_to
        if not args.version_autonumbering:
            more_kwargs['do_version_autonumbering'] = False
        if not args.update:
            more_kwargs['do_try_update'] = False
        if not args.parallel_build:
            more_kwargs['do_parallel_build'] = False
        if not args.check_syntax:
            more_kwargs['do_check_syntax'] = False

        return _build_app_remote(args.mode, args.src_dir, destination_override=args.destination,
                                 publish=args.publish, dx_toolkit_autodep=args.dx_toolkit_autodep,
                                 region=region, watch=args.watch, **more_kwargs)

Response:
Builds an app or applet and returns the resulting executable ID
   .


Instruction:
Given the following Python 3 function, write the documentation
def set_id(self, dxid=None, name=None, alias=None):
        '''
        :param dxid: App ID
        :type dxid: string
        :param name: App name
        :type name: string
        :param alias: App version or tag
        :type alias: string
        :raises: :exc:`~dxpy.exceptions.DXError` if *dxid* and some other input are both given or if neither *dxid* nor *name* are given

        Discards the currently stored ID and associates the handler
        with the requested parameters.  Note that if *dxid* is given,
        the other fields should not be given, and if *name* is given,
        *alias* has default value "default".

        '''
        self._dxid = None
        self._name = None
        self._alias = None
        if dxid is not None:
            if name is not None or alias is not None:
                raise DXError("Did not expect name or alias to be given if dxid is given")
            verify_string_dxid(dxid, self._class)
            self._dxid = dxid
        elif name is not None:
            self._name = name
            if not isinstance(name, basestring):
                raise DXError("App name needs to be a string: %r" % (name,))
            if alias is not None:
                if not isinstance(alias, basestring):
                    raise DXError("App alias needs to be a string: %r" % (alias,))
                self._alias = alias
            else:
                self._alias = 'default'

Response: Sets the ID of the currently stored handler.


Instruction:
Can you write a function in Python 3 where it
returns the object ID of the currently associated app.

Response:
def get_id(self):
        '''
        :returns: Object ID of associated app
        :rtype: string

        Returns the object ID of the app that the handler is currently
        associated with.
        '''
        if self._dxid is not None:
            return self._dxid
        else:
            return 'app-' + self._name + '/' + self._alias


Instruction:
Make a summary of the following Python 3 code
def new(self, **kwargs):
        '''
        :param initializeFrom: ID of an existing app object from which to initialize the app
        :type initializeFrom: string
        :param applet: ID of the applet that the app will be created from
        :type applet: string
        :param name: Name of the app (inherits from *initializeFrom* if possible)
        :type name: string
        :param title: Title or brand name of the app (optional)
        :type title: string
        :param summary: A short description of the app (optional)
        :type summary: string
        :param description: An extended description of the app (optional)
        :type description: string
        :param details: Arbitrary JSON to be associated with the app (optional)
        :type details: dict or list
        :param version: Version number
        :type version: string
        :param bill_to: ID of the user or organization who will own the app and be billed for its space usage (optional if an app with this name already exists)
        :type bill_to: string
        :param access: Access specification (optional)
        :type access: dict
        :param resources: Specifies what is to be put into the app's resources container. Must be a string containing a project ID, or a list containing object IDs. (optional)
        :type resources: string or list

        .. note:: It is highly recommended that the higher-level module
           :mod:`dxpy.app_builder` or (preferably) its frontend `dx build --create-app
           <https://wiki.dnanexus.com/Command-Line-Client/Index-of-dx-Commands#build>`_
           be used instead for app creation.

        Creates an app with the given parameters by using the specified
        applet or app as a base and overriding its attributes. See the
        API documentation for the `/app/new
        <https://wiki.dnanexus.com/API-Specification-v1.0.0/Apps#API-method%3A-%2Fapp%2Fnew>`_
        method for more info.

        Exactly one of *initializeFrom* and *applet* must be provided.

        The app is only available to its developers until
        :meth:`publish()` is called, and is not run until :meth:`run()`
        is called.

        '''
        #TODO: add support for regionalOptions (and deprecate top-level applet and resources)
        dx_hash = {}
        if 'applet' not in kwargs and 'initializeFrom' not in kwargs:
            raise DXError("%s: One of the keyword arguments %s and %s is required" % (self.__class__.__name__, 'applet', 'initializeFrom'))

        for field in ['version']:
            if field not in kwargs:
                raise DXError("%s: Keyword argument %s is required" % (self.__class__.__name__, field))
            dx_hash[field] = kwargs[field]
            del kwargs[field]

        for field in 'initializeFrom', 'applet', 'name', 'title', 'summary', 'description', 'billing', 'access', 'resources':
            if field in kwargs:
                dx_hash[field] = kwargs[field]
                del kwargs[field]

        if "bill_to" in kwargs:
            dx_hash['billTo'] = kwargs['bill_to']
            del kwargs["bill_to"]

        resp = dxpy.api.app_new(dx_hash, **kwargs)
        self.set_id(dxid=resp["id"])

Response: Creates a new app with the given parameters.


Instruction:
Can you implement a function in Python 3 that
returns a dict with the description of the app.

Response:
def describe(self, fields=None, **kwargs):
        '''
        :param fields: Hash where the keys are field names that should be returned, and values should be set to True (default is that all fields are returned)
        :type fields: dict
        :returns: Description of the remote app object
        :rtype: dict

        Returns a dict with a description of the app. The result
        includes the key-value pairs as specified in the API
        documentation for the `/app-xxxx/describe
        <https://wiki.dnanexus.com/API-Specification-v1.0.0/Apps#API-method%253A-%252Fapp-xxxx%255B%252Fyyyy%255D%252Fdescribe>`_
        method.

        '''
        describe_input = {}
        if fields:
            describe_input['fields'] = fields
        if self._dxid is not None:
            self._desc = dxpy.api.app_describe(self._dxid, input_params=describe_input, **kwargs)
        else:
            self._desc = dxpy.api.app_describe('app-' + self._name, alias=self._alias,
                                               input_params=describe_input, **kwargs)

        return self._desc


Instruction:
Write a Python 3 function that can
update the contents of the applet s resources and details of the app s resources container.

Response:
def update(self, **kwargs):
        '''
        :param applet: ID of the applet to replace the app's contents with
        :type applet: string
        :param details: Metadata to store with the app (optional)
        :type details: dict or list
        :param access: Access specification (optional)
        :type access: dict
        :param resources: Specifies what is to be put into the app's resources container. Must be a string containing a project ID, or a list containing object IDs. (optional)
        :type resources: string or list

        Updates the parameters of an existing app. See the API
        documentation for the `/app/update
        <https://wiki.dnanexus.com/API-Specification-v1.0.0/Apps#API-method%253A-%252Fapp-xxxx%255B%252Fyyyy%255D%252Fupdate>`_
        method for more info.

        The current user must be a developer of the app.

        '''
        updates = {}
        for field in 'applet', 'billing', 'access', 'resources', 'details':
            if field in kwargs:
                updates[field] = kwargs[field]
                del kwargs[field]

        if self._dxid is not None:
            resp = dxpy.api.app_update(self._dxid, input_params=updates, **kwargs)
        else:
            resp = dxpy.api.app_update('app-' + self._name, alias=self._alias,
                                       input_params=updates, **kwargs)


Instruction:
Given the following Python 3 function, write the documentation
def add_tags(self, tags, **kwargs):
        if self._dxid is not None:
            return dxpy.api.app_add_tags(self._dxid, input_params={"tags": tags}, **kwargs)
        else:
            return dxpy.api.app_add_tags('app-' + self._name, alias=self._alias,
                                         input_params={"tags": tags}, **kwargs)

Response: Adds the specified tags to the app.


Instruction:
Can you create a Python 3 function that
removes the specified tags from the app.

Response:
def remove_tags(self, tags, **kwargs):
        """
        :param tags: Tags to remove from the app
        :type tags: array

        Removes the specified application name tags (aliases) from this
        app, so that it is no longer addressable by those aliases.

        The current user must be a developer of the app.

        """
        if self._dxid is not None:
            return dxpy.api.app_remove_tags(self._dxid, input_params={"tags": tags}, **kwargs)
        else:
            return dxpy.api.app_remove_tags('app-' + self._name, alias=self._alias,
                                            input_params={"tags": tags}, **kwargs)


Instruction:
Write a Python 3 script to
install the app in the current user s account.

Response:
def install(self, **kwargs):
        """
        Installs the app in the current user's account.
        """
        if self._dxid is not None:
            return dxpy.api.app_install(self._dxid, **kwargs)
        else:
            return dxpy.api.app_install('app-' + self._name, alias=self._alias, **kwargs)


Instruction:
Can you tell what is the following Python 3 function doing
def get(self, **kwargs):
        if self._dxid is not None:
            return dxpy.api.app_get(self._dxid, **kwargs)
        else:
            return dxpy.api.app_get('app-' + self._name, alias=self._alias, **kwargs)

Response: Returns the full specification of the remote app object.


Instruction:
How would you explain what the following Python 3 function does
def delete(self, **kwargs):
        if self._dxid is not None:
            return dxpy.api.app_delete(self._dxid, **kwargs)
        else:
            return dxpy.api.app_delete('app-' + self._name, alias=self._alias, **kwargs)

Response: Removes this app object from the platform.


Instruction:
Write a Python 3 function for
creating a new job that executes the function main of this app with the given input *app_input*.

Response:
def run(self, app_input, *args, **kwargs):
        """
        Creates a new job that executes the function "main" of this app with
        the given input *app_input*.

        See :meth:`dxpy.bindings.dxapplet.DXExecutable.run` for the available
        args.
        """
        # Rename app_input to preserve API compatibility when calling
        # DXApp.run(app_input=...).
        return super(DXApp, self).run(app_input, *args, **kwargs)


Instruction:
Can you implement a function in Python 3 that
parses a dxworkflow. json file into a list of dictionaries.

Response:
def _parse_executable_spec(src_dir, json_file_name, parser):
    """
    Returns the parsed contents of a json specification.
    Raises WorkflowBuilderException (exit code 3) if this cannot be done.
    """
    if not os.path.isdir(src_dir):
        parser.error("{} is not a directory".format(src_dir))

    if not os.path.exists(os.path.join(src_dir, json_file_name)):
        raise WorkflowBuilderException(
            "Directory {} does not contain dxworkflow.json: not a valid DNAnexus workflow source directory"
            .format(src_dir))

    with open(os.path.join(src_dir, json_file_name)) as desc:
        try:
            return json_load_raise_on_duplicates(desc)
        except Exception as e:
            raise WorkflowBuilderException("Could not parse {} file as JSON: {}".format(json_file_name, e.args))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _get_destination_project(json_spec, args, build_project_id=None):
    if build_project_id:
        return build_project_id
    if 'project' in json_spec:
        return json_spec['project']
    if dxpy.WORKSPACE_ID:
        return dxpy.WORKSPACE_ID
    error_msg = "Can't create a workflow without specifying a destination project; "
    error_msg += "please use the -d/--destination flag to explicitly specify a project"
    raise WorkflowBuilderException(error_msg)

Response: Returns the project in which the workflow should be created.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _get_destination_folder(json_spec, folder_name=None):
    dest_folder = folder_name or json_spec.get('folder') or '/'
    if not dest_folder.endswith('/'):
        dest_folder = dest_folder + '/'
    return dest_folder

Response: Returns the folder name of the project where the workflow should be created.


Instruction:
Create a Python 3 function to
set the categories on the given global workflow.

Response:
def _set_categories_on_workflow(global_workflow_id, categories_to_set):
    """
    Note: Categories are set on the workflow series level,
    i.e. the same set applies to all versions.
    """
    assert(isinstance(categories_to_set, list))

    existing_categories = dxpy.api.global_workflow_list_categories(global_workflow_id)['categories']
    categories_to_add = set(categories_to_set).difference(set(existing_categories))
    categories_to_remove = set(existing_categories).difference(set(categories_to_set))
    if categories_to_add:
        dxpy.api.global_workflow_add_categories(global_workflow_id,
                                                input_params={'categories': list(categories_to_add)})
    if categories_to_remove:
        dxpy.api.global_workflow_remove_categories(global_workflow_id,
                                                   input_params={'categories': list(categories_to_remove)})


Instruction:
Can you create a Python 3 function that
returns True if a global workflow with the given name and version already exists in the platform.

Response:
def _version_exists(json_spec, name=None, version=None):
    """
    Returns True if a global workflow with the given name and version
    already exists in the platform and the user has developer rights
    to the workflow. "name" and "version" can be passed if we already
    made a "describe" API call on the global workflow and so know the
    requested name and version already exists.
    """
    requested_name = json_spec['name']
    requested_version = json_spec['version']

    if requested_name == name and requested_version == version:
        return True
    else:
        try:
            desc_output = dxpy.api.global_workflow_describe('globalworkflow-' + json_spec['name'],
                                                            alias=json_spec['version'],
                                                            input_params={"fields": {"name": True,
                                                                                "version": True}})
            return desc_output['name'] == json_spec['name'] and desc_output['version'] == json_spec['version']
        except dxpy.exceptions.DXAPIError:
            return False
        except:
            raise


Instruction:
Explain what the following Python 3 code does
def validate_ignore_reuse(stages, ignore_reuse_stages):
    if not isinstance(ignore_reuse_stages, list):
         raise WorkflowBuilderException('"IgnoreReuse must be a list of strings - stage IDs or "*"')

    ignore_reuse_set = set(ignore_reuse_stages)
    if '*' in ignore_reuse_set and ignore_reuse_set == 1:
        return

    stage_ids = set([stage.get('id') for stage in stages])
    for ignored in ignore_reuse_set:
        if ignored not in stage_ids:
            raise WorkflowBuilderException(
                'Stage with ID {} not found. Add a matching "id" for the stage you wish to set ignoreReuse for'.format(ignored))

Response: Checks if the given list of stages contains a valid entry in the order they are not used.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _get_validated_stages(stages):
    if not isinstance(stages, list):
        raise WorkflowBuilderException("Stages must be specified as a list of dictionaries")
    validated_stages = []
    for index, stage in enumerate(stages):
        validated_stages.append(_get_validated_stage(stage, index))
    return validated_stages

Response: Validates the given stages and returns a list of dictionaries.


Instruction:
How would you explain what the following Python 3 function does
def _validate_json_for_regular_workflow(json_spec, args):
    validated = {}
    override_project_id, override_folder, override_workflow_name = \
        dxpy.executable_builder.get_parsed_destination(args.destination)
    validated['project'] = _get_destination_project(json_spec, args, override_project_id)
    validated['folder'] = _get_destination_folder(json_spec, override_folder)

    workflow_name = _get_workflow_name(json_spec, override_workflow_name)
    if not workflow_name:
        print('Warning: workflow name is not specified')
    else:
        validated['name'] = workflow_name
    return validated

Response: Validates the JSON spec for a regular project - based workflow.


Instruction:
How would you code a function in Python 3 to
validate the JSON spec for a global workflow.

Response:
def _validate_json_for_global_workflow(json_spec, args):
    """
    Validates fields used for building a global workflow.
    Since building a global workflow is done after all the underlying workflows
    are built, which may be time-consuming, we validate as much as possible here.
    """
    # TODO: verify the billTo can build the workflow
    # TODO: if the global workflow build fails add an option to interactively change billto
    # TODO: (or other simple fields) instead of failing altogether
    # TODO: get a confirmation before building a workflow that may be costly
    if 'name' not in json_spec:
        raise WorkflowBuilderException(
            "dxworkflow.json contains no 'name' field, but it is required to build a global workflow")
    if not dxpy.executable_builder.GLOBAL_EXEC_NAME_RE.match(json_spec['name']):
        raise WorkflowBuilderException(
            "The name of your workflow must match /^[a-zA-Z0-9._-]+$/")
    if json_spec['name'] != json_spec['name'].lower():
        logger.warn('workflow name "{}" should be all lowercase'.format(json_spec['name']))

    if 'version' not in json_spec:
        raise WorkflowBuilderException(
            "dxworkflow.json contains no 'version' field, but it is required to build a global workflow")
    if not dxpy.executable_builder.GLOBAL_EXEC_VERSION_RE.match(json_spec['version']):
        logger.warn('"version" {} should be semver compliant (e.g. of the form X.Y.Z)'.format(json_spec['version']))

    if 'details' in json_spec:
        if not isinstance(json_spec['details'], dict):
            raise WorkflowBuilderException(
                'The field "details" must be a dictionary')

    if 'regionalOptions' in json_spec:
        if not (isinstance(json_spec['regionalOptions'], dict)
                and json_spec['regionalOptions']
                and all([isinstance(i, dict) for i in json_spec['regionalOptions'].values()])):
            raise WorkflowBuilderException(
                'The field "regionalOptions" must be a non-empty dictionary whose values are dictionaries')

    if args.bill_to:
        json_spec["billTo"] = args.bill_to


Instruction:
Implement a Python 3 function for
validating dxworkflow. json and returns the json that can be sent with the dxworkflow API or globalworkflow API or dxworkflow. new API or dxworkflow. new API or dxworkflow. new API or dxworkflow. new API.

Response:
def _get_validated_json(json_spec, args):
    """
    Validates dxworkflow.json and returns the json that can be sent with the
    /workflow/new API or /globalworkflow/new request.
    """
    if not json_spec:
        return
    if not args:
        return

    validated_spec = copy.deepcopy(json_spec)

    # print ignored keys if present in json_spec
    unsupported_keys = _get_unsupported_keys(validated_spec.keys(), SUPPORTED_KEYS)
    if len(unsupported_keys) > 0:
        logger.warn(
            "Warning: the following root level fields are not supported and will be ignored: {}"
                .format(", ".join(unsupported_keys)))

    if 'stages' in validated_spec:
        validated_spec['stages'] = _get_validated_stages(validated_spec['stages'])

    if 'name' in validated_spec:
        if args.src_dir != validated_spec['name']:
            logger.warn(
                'workflow name "%s" does not match containing directory "%s"' % (validated_spec['name'], args.src_dir))

    if 'ignoreReuse' in validated_spec:
        validate_ignore_reuse(validated_spec['stages'], validated_spec['ignoreReuse'])

    validated_documentation_fields = _get_validated_json_for_build_or_update(validated_spec, args)
    validated_spec.update(validated_documentation_fields)

    # Project-based workflow specific validation
    if args.mode == 'workflow':
        validated = _validate_json_for_regular_workflow(json_spec, args)
        validated_spec.update(validated)

    # Global workflow specific validation
    if args.mode == 'globalworkflow':
        _validate_json_for_global_workflow(validated_spec, args)

    return validated_spec


Instruction:
Write a Python 3 script for
validating that the json_spec is valid and returns the json_spec that can be used to build or update the current version of the current project.

Response:
def _get_validated_json_for_build_or_update(json_spec, args):
    """
    Validates those fields that can be used when either building
    a new version (of a local, project-based workflow) or updating
    an existing version (of a global workflow).
    """
    validated = copy.deepcopy(json_spec)

    dxpy.executable_builder.inline_documentation_files(validated, args.src_dir)

    if 'title' not in json_spec:
        logger.warn("dxworkflow.json is missing a title, please add one in the 'title' field")

    if 'summary' not in json_spec:
        logger.warn("dxworkflow.json is missing a summary, please add one in the 'summary' field")
    else:
        if json_spec['summary'].endswith('.'):
            logger.warn("summary {} should be a short phrase not ending in a period".format(json_spec['summary'],))

    return validated


Instruction:
Create a Python 3 function to
check if the global workflow regions and the regions of the executables of the workflow are all matching.

Response:
def _assert_executable_regions_match(workflow_enabled_regions, workflow_spec):
    """
    Check if the global workflow regions and the regions of stages (apps) match.
    If the workflow contains any applets, the workflow can be currently enabled
    in only one region - the region in which the applets are stored.
    """
    executables = [i.get("executable") for i in workflow_spec.get("stages")]

    for exect in executables:

        if exect.startswith("applet-") and len(workflow_enabled_regions) > 1:
            raise WorkflowBuilderException("Building a global workflow with applets in more than one region is not yet supported.")

        elif exect.startswith("app-"):
            app_regional_options = dxpy.api.app_describe(exect, input_params={"fields": {"regionalOptions": True}})
            app_regions = set(app_regional_options['regionalOptions'].keys())
            if not workflow_enabled_regions.issubset(app_regions):
                additional_workflow_regions = workflow_enabled_regions - app_regions
                mesg = "The app {} is enabled in regions {} while the global workflow in {}.".format(
                    exect, ", ".join(app_regions), ", ".join(workflow_enabled_regions))
                mesg += " The workflow will not be able to run in {}.".format(", ".join(additional_workflow_regions))
                mesg += " If you are a developer of the app, you can enable the app in {} to run the workflow in that region(s).".format(
                    ", ".join(additional_workflow_regions))
                logger.warn(mesg)

        elif exect.startswith("workflow-"):
             # We recurse to check the regions of the executables of the inner workflow
            inner_workflow_spec = dxpy.api.workflow_describe(exect)
            _assert_executable_regions_match(workflow_enabled_regions, inner_workflow_spec)

        elif exect.startswith("globalworkflow-"):
            raise WorkflowBuilderException("Building a global workflow with nested global workflows is not yet supported")


Instruction:
How would you implement a function in Python 3 that
builds a regular workflow.

Response:
def _build_regular_workflow(json_spec):
    """
    Precondition: json_spec must be validated
    """
    workflow_id = dxpy.api.workflow_new(json_spec)["id"]
    dxpy.api.workflow_close(workflow_id)
    return workflow_id


Instruction:
Create a Python 3 function to
return a set of regions in which the global workflow should be enabled.

Response:
def _get_validated_enabled_regions(json_spec, from_command_line):
    """
    Returns a set of regions (region names) in which the global workflow
    should be enabled. Also validates and synchronizes the regions
    passed via CLI argument and in the regionalOptions field.
    """
    enabled_regions = dxpy.executable_builder.get_enabled_regions('globalworkflow',
                                                                  json_spec,
                                                                  from_command_line,
                                                                  WorkflowBuilderException)
    if not enabled_regions:
        enabled_regions = []
        if not dxpy.WORKSPACE_ID:
            msg = "A context project must be selected to enable a workflow in the project's region."
            msg += " You can use 'dx select' to select a project. Otherwise you can use --region option"
            msg += " to select a region in which the workflow should be enabled"
            raise(WorkflowBuilderException(msg))
        region = dxpy.api.project_describe(dxpy.WORKSPACE_ID,
                                           input_params={"fields": {"region": True}})["region"]

        enabled_regions.append(region)

    if not enabled_regions:
        raise AssertionError("This workflow should be enabled in at least one region")

    return set(enabled_regions)