Instruction:
Implement a function in Python 3 to
while `Seq` is sequential, `phi.dsl.Expression.List` allows you to split the computation and get back a list with the result of each path. While the list literal should be the most incarnation of this expresion, it can actually be any iterable (implements `__iter__`) that is not a tuple and yields a valid expresion. The expression k = List(f, g) is equivalent to k = lambda x: [ f(x), g(x) ] In general, the following rules apply after compilation: **General Branching** List(f0, f1, ..., fn) is equivalent to lambda x: [ f0(x), f1(x), ..., fn(x) ] **Composing & Branching** It is interesting to see how braching interacts with composing. The expression Seq(f, List(g, h)) is *almost* equivalent to List( Seq(f, g), Seq(f, h) ) As you see its as if `f` where distributed over the List. We say *almost* because their implementation is different def _lambda(x): x = f(x) return [ g(x), h(x) ] vs lambda x: [ g(f(x)), h(f(x)) ] As you see `f` is only executed once in the first one. Both should yield the same result if `f` is a pure function. ### Examples form phi import P, List avg_word_length = P.Pipe( "1 22 333", lambda s: s.split(' '), # ['1', '22', '333'] lambda l: map(len, l), # [1, 2, 3] List( sum # 1 + 2 + 3 == 6 , len # len([1, 2, 3]) == 3 ), lambda l: l[0] / l[1] # sum / len == 6 / 3 == 2 ) assert avg_word_length == 2 The previous could also be done more briefly like this form phi import P, Obj, List avg_word_length = P.Pipe( "1 22 333", Obj .split(' ') # ['1', '22', '333'] .map(len) # [1, 2, 3] .List( sum #sum([1, 2, 3]) == 6 , len #len([1, 2, 3]) == 3 ), P[0] / P[1] #6 / 3 == 2 ) assert avg_word_length == 2 In the example above the last expression P[0] / P[1] works for a couple of reasons 1. The previous expression returns a list 2. In general the expression `P[x]` compiles to a function with the form `lambda obj: obj[x]` 3. The class `Expression` (the class from which the object `P` inherits) overrides most operators to create functions easily. For example, the expression (P * 2) / (P + 1) compile to a function of the form lambda x: (x * 2) / (x + 1) Check out the documentatio for Phi [lambdas](https://cgarciae.github.io/phi/lambdas.m.html).

Response:
def List(self, *branches, **kwargs):
        """
While `Seq` is sequential, `phi.dsl.Expression.List` allows you to split the computation and get back a list with the result of each path. While the list literal should be the most incarnation of this expresion, it can actually be any iterable (implements `__iter__`) that is not a tuple and yields a valid expresion.

The expression

    k = List(f, g)

is equivalent to

    k = lambda x: [ f(x), g(x) ]

In general, the following rules apply after compilation:

**General Branching**

    List(f0, f1, ..., fn)

is equivalent to

    lambda x: [ f0(x), f1(x), ..., fn(x) ]

**Composing & Branching**

It is interesting to see how braching interacts with composing. The expression

    Seq(f, List(g, h))

is *almost* equivalent to

    List( Seq(f, g), Seq(f, h) )

As you see its as if `f` where distributed over the List. We say *almost* because their implementation is different

    def _lambda(x):
        x = f(x)
        return [ g(x), h(x) ]

vs

    lambda x: [ g(f(x)), h(f(x)) ]

As you see `f` is only executed once in the first one. Both should yield the same result if `f` is a pure function.

### Examples

    form phi import P, List

    avg_word_length = P.Pipe(
        "1 22 333",
        lambda s: s.split(' '), # ['1', '22', '333']
        lambda l: map(len, l), # [1, 2, 3]
        List(
            sum # 1 + 2 + 3 == 6
        ,
            len # len([1, 2, 3]) == 3
        ),
        lambda l: l[0] / l[1] # sum / len == 6 / 3 == 2
    )

    assert avg_word_length == 2

The previous could also be done more briefly like this

    form phi import P, Obj, List

    avg_word_length = P.Pipe(
        "1 22 333", Obj
        .split(' ')  # ['1', '22', '333']
        .map(len)    # [1, 2, 3]
        .List(
            sum  #sum([1, 2, 3]) == 6
        ,
            len  #len([1, 2, 3]) == 3
        ),
        P[0] / P[1]  #6 / 3 == 2
    )

    assert avg_word_length == 2

In the example above the last expression

    P[0] / P[1]

works for a couple of reasons

1. The previous expression returns a list
2. In general the expression `P[x]` compiles to a function with the form `lambda obj: obj[x]`
3. The class `Expression` (the class from which the object `P` inherits) overrides most operators to create functions easily. For example, the expression

    (P * 2) / (P + 1)

compile to a function of the form

    lambda x: (x * 2) / (x + 1)

Check out the documentatio for Phi [lambdas](https://cgarciae.github.io/phi/lambdas.m.html).

        """
        gs = [ _parse(code)._f for code in branches ]

        def h(x, state):
            ys = []
            for g in gs:
                y, state = g(x, state)
                ys.append(y)

            return (ys, state)

        return self.__then__(h, **kwargs)


Instruction:
Can you write a function in Python 3 where it
returns a new sequence that is the sequence of the given elements.

Response:
def Seq(self, *sequence, **kwargs):
        """
`Seq` is used to express function composition. The expression

    Seq(f, g)

be equivalent to

    lambda x: g(f(x))

As you see, its a little different from the mathematical definition. Excecution order flow from left to right, this makes reading and reasoning about code way more easy. This bahaviour is based upon the `|>` (pipe) operator found in languages like F#, Elixir and Elm. You can pack as many expressions as you like and they will be applied in order to the data that is passed through them when compiled an excecuted.

In general, the following rules apply for Seq:

**General Sequence**

    Seq(f0, f1, ..., fn-1, fn)

is equivalent to

    lambda x: fn(fn-1(...(f1(f0(x)))))

**Single Function**

    Seq(f)

is equivalent to

    f

**Identity**

The empty Seq

    Seq()

is equivalent to

    lambda x: x

### Examples

    from phi import P, Seq

    f = Seq(
        P * 2,
        P + 1,
        P ** 2
    )

    assert f(1) == 9 # ((1 * 2) + 1) ** 2

The previous example using `P.Pipe`

    from phi import P

    assert 9 == P.Pipe(
        1,
        P * 2,  #1 * 2 == 2
        P + 1,  #2 + 1 == 3
        P ** 2  #3 ** 2 == 9
    )
        """
        fs = [ _parse(elem)._f for elem in sequence ]

        def g(x, state):
            return functools.reduce(lambda args, f: f(*args), fs, (x, state))

        return self.__then__(g, **kwargs)


Instruction:
Explain what the following Python 3 code does
def With(self, context_manager, *body, **kwargs):
        context_f = _parse(context_manager)._f
        body_f = E.Seq(*body)._f

        def g(x, state):
            context, state = context_f(x, state)
            with context as scope:
                with _WithContextManager(scope):
                    return body_f(x, state)

        return self.__then__(g, **kwargs)

Response: A simple context manager with a given DSL.


Instruction:
Explain what the following Python 3 code does
def ReadList(self, *branches, **kwargs):
        branches = map(lambda x: E.Read(x) if isinstance(x, str) else x, branches)

        return self.List(*branches, **kwargs)

Response: A version of phi. dsl. Expression. List that handles the case where a string argument x is translated to Read.


Instruction:
Here you have a function in Python 3, explain what it does
def Write(self, *state_args, **state_dict):
        if len(state_dict) + len(state_args) < 1:
            raise Exception("Please include at-least 1 state variable, got {0} and {1}".format(state_args, state_dict))

        if len(state_dict) > 1:
            raise Exception("Please include at-most 1 keyword argument expression, got {0}".format(state_dict))

        if len(state_dict) > 0:
            state_key = next(iter(state_dict.keys()))
            write_expr = state_dict[state_key]

            state_args += (state_key,)

            expr = self >> write_expr

        else:
            expr = self

        def g(x, state):
            update = { key: x for key in state_args }
            state = utils.merge(state, update)

            #side effect for convenience
            _StateContextManager.REFS.update(state)

            return x, state

        return expr.__then__(g)

Response: See phi. dsl. Expression. Write


Instruction:
Write a Python 3 script to
return a new Seq with the given value.

Response:
def Val(self, val, **kwargs):
        """
The expression

    Val(a)

is equivalent to the constant function

    lambda x: a

All expression in this module interprete values that are not functions as constant functions using `Val`, for example

    Seq(1, P + 1)

is equivalent to

    Seq(Val(1), P + 1)

The previous expression as a whole is a constant function since it will return `2` no matter what input you give it.
        """
        f = utils.lift(lambda z: val)

        return self.__then__(f, **kwargs)


Instruction:
How would you explain what the following Python 3 function does
def Else(self, *Else, **kwargs):
        root = self._root
        ast = self._ast

        next_else = E.Seq(*Else)._f
        ast = _add_else(ast, next_else)

        g = _compile_if(ast)

        return root.__then__(g, **kwargs)

Response: See phi. dsl. Expression. If


Instruction:
Can you generate a brief explanation for the following Python 3 code
def length(string, until=None):
    if until is None:
        return sum(1 for _ in GraphemeIterator(string))

    iterator = graphemes(string)
    count = 0
    while True:
        try:
            if count >= until:
                break
            next(iterator)
        except StopIteration:
            break
        else:
            count += 1

    return count

Response: Returns the length of the string.


Instruction:
Can you generate the documentation for the following Python 3 function
def slice(string, start=None, end=None):

    if start is None:
        start = 0
    if end is not None and start >= end:
        return ""

    if start < 0:
        raise NotImplementedError("Negative indexing is currently not supported.")

    sum_ = 0
    start_index = None
    for grapheme_index, grapheme_length in enumerate(grapheme_lengths(string)):
        if grapheme_index == start:
            start_index = sum_
        elif grapheme_index == end:
            return string[start_index:sum_]
        sum_ += grapheme_length

    if start_index is not None:
        return string[start_index:]

    return ""

Response: Returns a substring of the given string counting graphemes instead of codepoints.


Instruction:
Implement a function in Python 3 to
return True if the grapheme sequence of graphemes in substring is also present in string.

Response:
def contains(string, substring):
    """
    Returns true if the sequence of graphemes in substring is also present in string.

    This differs from the normal python `in` operator, since the python operator will return
    true if the sequence of codepoints are withing the other string without considering
    grapheme boundaries.

    Performance notes: Very fast if `substring not in string`, since that also means that
    the same graphemes can not be in the two strings. Otherwise this function has linear time
    complexity in relation to the string length. It will traverse the sequence of graphemes until
    a match is found, so it will generally perform better for grapheme sequences that match early.

    >>> "🇸🇪" in "🇪🇸🇪🇪"
    True
    >>> grapheme.contains("🇪🇸🇪🇪", "🇸🇪")
    False
    """
    if substring not in string:
        return False

    substr_graphemes = list(graphemes(substring))

    if len(substr_graphemes) == 0:
        return True
    elif len(substr_graphemes) == 1:
        return substr_graphemes[0] in graphemes(string)
    else:
        str_iter = graphemes(string)
        str_sub_part = []
        for _ in range(len(substr_graphemes)):
            try:
                str_sub_part.append(next(str_iter))
            except StopIteration:
                return False

        for g in str_iter:
            if str_sub_part == substr_graphemes:
                return True

            str_sub_part.append(g)
            str_sub_part.pop(0)
        return str_sub_part == substr_graphemes


Instruction:
How would you code a function in Python 3 to
like str. startswith but checks that the given string starts with the given prefixes sequence of graphemes.

Response:
def startswith(string, prefix):
    """
    Like str.startswith, but also checks that the string starts with the given prefixes sequence of graphemes.

    str.startswith may return true for a prefix that is not visually represented as a prefix if a grapheme cluster
    is continued after the prefix ends.

    >>> grapheme.startswith("✊🏾", "✊")
    False
    >>> "✊🏾".startswith("✊")
    True
    """
    return string.startswith(prefix) and safe_split_index(string, len(prefix)) == len(prefix)


Instruction:
Can you tell what is the following Python 3 function doing
def endswith(string, suffix):
    expected_index = len(string) - len(suffix)
    return string.endswith(suffix) and safe_split_index(string, expected_index) == expected_index

Response: Like str. endswith but checks that the given string ends with the given suffix.


Instruction:
Can you create a Python 3 function that
returns the highest index up to `max_len` at which the given string can be sliced, without breaking a grapheme. This is useful for when you want to split or take a substring from a string, and don't really care about the exact grapheme length, but don't want to risk breaking existing graphemes. This function does normally not traverse the full grapheme sequence up to the given length, so it can be used for arbitrarily long strings and high `max_len`s. However, some grapheme boundaries depend on the previous state, so the worst case performance is O(n). In practice, it's only very long non-broken sequences of country flags (represented as Regional Indicators) that will perform badly. The return value will always be between `0` and `len(string)`. >>> string = "tamil நி (ni)" >>> i = grapheme.safe_split_index(string, 7) >>> i 6 >>> string[:i] 'tamil ' >>> string[i:] 'நி (ni)'

Response:
def safe_split_index(string, max_len):
    """
    Returns the highest index up to `max_len` at which the given string can be sliced, without breaking a grapheme.

    This is useful for when you want to split or take a substring from a string, and don't really care about
    the exact grapheme length, but don't want to risk breaking existing graphemes.

    This function does normally not traverse the full grapheme sequence up to the given length, so it can be used
    for arbitrarily long strings and high `max_len`s. However, some grapheme boundaries depend on the previous state,
    so the worst case performance is O(n). In practice, it's only very long non-broken sequences of country flags
    (represented as Regional Indicators) that will perform badly.

    The return value will always be between `0` and `len(string)`.

    >>> string = "tamil நி (ni)"
    >>> i = grapheme.safe_split_index(string, 7)
    >>> i
    6
    >>> string[:i]
    'tamil '
    >>> string[i:]
    'நி (ni)'
    """
    last_index = get_last_certain_break_index(string, max_len)
    for l in grapheme_lengths(string[last_index:]):
        if last_index + l > max_len:
            break
        last_index += l
    return last_index


Instruction:
Can you generate a brief explanation for the following Python 3 code
def readB1logfile(filename):
    dic = dict()
    # try to open. If this fails, an exception is raised
    with open(filename, 'rt', encoding='utf-8') as f:
        for l in f:
            l = l.strip()
            if l[0] in '#!%\'':
                continue  # treat this line as a comment
            try:
                # find the first tuple in _logfile_data where the first element of the
                # tuple is the starting of the line.
                ld = [ld_ for ld_ in _logfile_data if l.split(
                    ':', 1)[0].strip() == ld_[0]][0]
            except IndexError:
                # line is not recognized. We can still try to load it: find the first
                # semicolon. If found, the part of the line before it is stripped
                # from whitespaces and will be the key. The part after it is stripped
                # from whitespaces and parsed with misc.parse_number(). If no
                if ':' in l:
                    key, val = [x.strip() for x in l.split(':', 1)]
                    val = misc.parse_number(val)
                    dic[key] = val
                    try:
                        # fix the character encoding in files written by a
                        # previous version of this software.
                        dic[key] = dic[key].encode('latin2').decode('utf-8')
                    except (UnicodeDecodeError, UnicodeEncodeError, AttributeError):
                        pass
                else:
                    dic[l.strip()] = True
                continue
            try:
                reader = ld[3]
            except IndexError:
                reader = str
            rhs = l.split(':', 1)[1].strip()
            try:
                vals = reader(rhs)
            except ValueError:
                if rhs.lower() == 'none':
                    vals = None
                else:
                    raise
            if isinstance(ld[1], tuple):
                # more than one field names. The reader function should return a
                # tuple here, a value for each field.
                if len(vals) != len(ld[1]):
                    raise ValueError(
                        'Cannot read %d values from line %s in file!' % (len(ld[1]), l))
                dic.update(dict(list(zip(ld[1], vals))))
            else:
                dic[ld[1]] = vals
    dic['__Origin__'] = 'B1 log'
    dic['__particle__'] = 'photon'
    return dic

Response: Read the B1 logfile and return a dictionary.


Instruction:
Given the following Python 3 function, write the documentation
def writeB1logfile(filename, data):
    allkeys = list(data.keys())
    f = open(filename, 'wt', encoding='utf-8')
    for ld in _logfile_data:  # process each line
        linebegin = ld[0]
        fieldnames = ld[1]
        # set the default formatter if it is not given
        if len(ld) < 3:
            formatter = str
        elif ld[2] is None:
            formatter = str
        else:
            formatter = ld[2]
        # this will contain the formatted values.
        formatted = ''
        if isinstance(fieldnames, str):
            # scalar field name, just one field. Formatter should be a
            # callable.
            if fieldnames not in allkeys:
                # this field has already been processed
                continue
            try:
                formatted = formatter(data[fieldnames])
            except KeyError:
                # field not found in param structure
                continue
        elif isinstance(fieldnames, tuple):
            # more than one field names in a tuple. In this case, formatter can
            # be a tuple of callables...
            if all([(fn not in allkeys) for fn in fieldnames]):
                # if all the fields have been processed:
                continue
            if isinstance(formatter, tuple) and len(formatter) == len(fieldnames):
                formatted = ' '.join([ft(data[fn])
                                      for ft, fn in zip(formatter, fieldnames)])
            # ...or a single callable...
            elif not isinstance(formatter, tuple):
                formatted = formatter([data[fn] for fn in fieldnames])
            # ...otherwise raise an exception.
            else:
                raise SyntaxError('Programming error: formatter should be a scalar or a tuple\
of the same length as the field names in logfile_data.')
        else:  # fieldnames is neither a string, nor a tuple.
            raise SyntaxError(
                'Invalid syntax (programming error) in logfile_data in writeparamfile().')
        # try to get the values
        linetowrite = linebegin + ':\t' + formatted + '\n'
        f.write(linetowrite)
        if isinstance(fieldnames, tuple):
            for fn in fieldnames:  # remove the params treated.
                if fn in allkeys:
                    allkeys.remove(fn)
        else:
            if fieldnames in allkeys:
                allkeys.remove(fieldnames)
    # write untreated params
    for k in allkeys:
        linetowrite = k + ':\t' + str(data[k]) + '\n'
        f.write(linetowrite)

    f.close()

Response: Write a header structure into a B1 logfile.


Instruction:
Can you implement a function in Python 3 that
reads beamline B1 header data from file.

Response:
def readB1header(filename):
    """Read beamline B1 (HASYLAB, Hamburg) header data

    Input
    -----
    filename: string
        the file name. If ends with ``.gz``, it is fed through a ``gunzip``
        filter

    Output
    ------
    A header dictionary.

    Examples
    --------
    read header data from 'ORG000123.DAT'::

        header=readB1header('ORG00123.DAT')
    """
    # Planck's constant times speed of light: incorrect
    # constant in the old program on hasjusi1, which was
    # taken over by the measurement program, to keep
    # compatibility with that.
    hed = {}
    if libconfig.LENGTH_UNIT == 'A':
        jusifaHC = 12396.4
    elif libconfig.LENGTH_UNIT == 'nm':
        jusifaHC = 1239.64
    else:
        raise NotImplementedError(
            'Invalid length unit: ' + str(libconfig.LENGTH_UNIT))

    if filename.upper().endswith('.GZ'):
        fid = gzip.GzipFile(filename, 'r')
    else:
        fid = open(filename, 'rt')
    lines = fid.readlines()
    fid.close()
    hed['FSN'] = int(lines[0].strip())
    hed['Hour'] = int(lines[17].strip())
    hed['Minutes'] = int(lines[18].strip())
    hed['Month'] = int(lines[19].strip())
    hed['Day'] = int(lines[20].strip())
    hed['Year'] = int(lines[21].strip()) + 2000
    hed['FSNref1'] = int(lines[23].strip())
    hed['FSNdc'] = int(lines[24].strip())
    hed['FSNsensitivity'] = int(lines[25].strip())
    hed['FSNempty'] = int(lines[26].strip())
    hed['FSNref2'] = int(lines[27].strip())
    hed['Monitor'] = float(lines[31].strip())
    hed['Anode'] = float(lines[32].strip())
    hed['MeasTime'] = float(lines[33].strip())
    hed['Temperature'] = float(lines[34].strip())
    hed['BeamPosX'] = float(lines[36].strip())
    hed['BeamPosY'] = float(lines[37].strip())
    hed['Transm'] = float(lines[41].strip())
    hed['Wavelength'] = float(lines[43].strip())
    hed['Energy'] = jusifaHC / hed['Wavelength']
    hed['Dist'] = float(lines[46].strip())
    hed['XPixel'] = 1 / float(lines[49].strip())
    hed['YPixel'] = 1 / float(lines[50].strip())
    hed['Title'] = lines[53].strip().replace(' ', '_').replace('-', '_')
    hed['MonitorDORIS'] = float(lines[56].strip())  # aka. DORIS counter
    hed['Owner'] = lines[57].strip()
    hed['RotXSample'] = float(lines[59].strip())
    hed['RotYSample'] = float(lines[60].strip())
    hed['PosSample'] = float(lines[61].strip())
    hed['DetPosX'] = float(lines[62].strip())
    hed['DetPosY'] = float(lines[63].strip())
    hed['MonitorPIEZO'] = float(lines[64].strip())  # aka. PIEZO counter
    hed['BeamsizeX'] = float(lines[66].strip())
    hed['BeamsizeY'] = float(lines[67].strip())
    hed['PosRef'] = float(lines[70].strip())
    hed['Monochromator1Rot'] = float(lines[77].strip())
    hed['Monochromator2Rot'] = float(lines[78].strip())
    hed['Heidenhain1'] = float(lines[79].strip())
    hed['Heidenhain2'] = float(lines[80].strip())
    hed['Current1'] = float(lines[81].strip())
    hed['Current2'] = float(lines[82].strip())
    hed['Detector'] = 'Unknown'
    hed['PixelSize'] = (hed['XPixel'] + hed['YPixel']) / 2.0

    hed['AnodeError'] = math.sqrt(hed['Anode'])
    hed['TransmError'] = 0
    hed['MonitorError'] = math.sqrt(hed['Monitor'])
    hed['MonitorPIEZOError'] = math.sqrt(hed['MonitorPIEZO'])
    hed['MonitorDORISError'] = math.sqrt(hed['MonitorDORIS'])
    hed['Date'] = datetime.datetime(
        hed['Year'], hed['Month'], hed['Day'], hed['Hour'], hed['Minutes'])
    hed['__Origin__'] = 'B1 original'
    hed['__particle__'] = 'photon'
    return hed


Instruction:
Can you generate the documentation for the following Python 3 function
def readehf(filename):
    f = open(filename, 'r')
    edf = {}
    if not f.readline().strip().startswith('{'):
        raise ValueError('Invalid file format.')
    for l in f:
        l = l.strip()
        if not l:
            continue
        if l.endswith('}'):
            break  # last line of header
        try:
            left, right = l.split('=', 1)
        except ValueError:
            raise ValueError('Invalid line: ' + l)
        left = left.strip()
        right = right.strip()
        if not right.endswith(';'):
            raise ValueError(
                'Invalid line (does not end with a semicolon): ' + l)
        right = right[:-1].strip()
        m = re.match('^(?P<left>.*)~(?P<continuation>\d+)$', left)
        if m is not None:
            edf[m.group('left')] = edf[m.group('left')] + right
        else:
            edf[left] = _readedf_extractline(left, right)
    f.close()
    edf['FileName'] = filename
    edf['__Origin__'] = 'EDF ID02'
    edf['__particle__'] = 'photon'
    return edf

Response: Read the EDF header from a file and return a dictionary of the EDF header structure.


Instruction:
Write a Python 3 script for
reading the header data from a Bessy Data Format v1 file.

Response:
def readbhfv1(filename, load_data=False, bdfext='.bdf', bhfext='.bhf'):
    """Read header data from bdf/bhf file (Bessy Data Format v1)

    Input:
        filename: the name of the file
        load_data: if the matrices are to be loaded

    Output:
        bdf: the BDF header structure

    Adapted the bdf_read.m macro from Sylvio Haas.
    """
    # strip the bhf or bdf extension if there.
    if filename.endswith(bdfext):
        basename = filename[:-len(bdfext)]
    elif filename.endswith(bhfext):
        basename = filename[:-len(bhfext)]
    else:  # assume a single file of header and data.
        basename, bhfext = os.path.splitext(filename)
        bdfext = bhfext
    headername = basename + bhfext
    dataname = basename + bdfext
    bdf = {}
    bdf['his'] = []  # empty list for history
    bdf['C'] = {}  # empty list for bdf file descriptions
    namelists = {}
    valuelists = {}
    with open(headername, 'rb') as fid:  # if fails, an exception is raised
        for line in fid:
            if not line.strip():
                continue  # empty line
            mat = line.split(None, 1)
            prefix = mat[0]
            if prefix == '#C':
                left, right = mat[1].split('=', 1)
                left = left.strip()
                right = right.strip()
                if left in ['xdim', 'ydim']:
                    bdf[left] = int(right)
                elif left in ['type', 'bdf']:
                    bdf[left] = right
                if left in ['Sendtime']:
                    bdf['C'][left] = float(right)
                elif left in ['xdim', 'ydim']:
                    bdf['C'][left] = int(right)
                else:
                    bdf['C'][left] = misc.parse_number(right)
            elif prefix.startswith("#H"):
                bdf['his'].append(mat[1])
#            elif prefix.startswith("#DATA"):
#                if not load_data:
#                    break
#                darray = np.fromfile(fid, dtype = bdf['type'], count = int(bdf['xdim'] * bdf['ydim']))
#                bdf['data'] = np.rot90((darray.reshape(bdf['xdim'], bdf['ydim'])).astype('double').T, 1).copy() # this weird transformation is needed to get the matrix in the same form as bdf_read.m gets it.
#            elif prefix.startswith('#ERROR'):
#                if not load_data:
#                    break
#                darray = np.fromfile(fid, dtype = bdf['type'], count = int(bdf['xdim'] * bdf['ydim']))
#                bdf['error'] = np.rot90((darray.reshape(bdf['xdim'], bdf['ydim'])).astype('double').T, 1).copy()
            else:
                for prf in ['M', 'G', 'S', 'T']:
                    if prefix.startswith('#C%sL' % prf):
                        if prf not in namelists:
                            namelists[prf] = []
                        namelists[prf].extend(mat[1].split())
                    elif prefix.startswith('#C%sV' % prf):
                        if prf not in valuelists:
                            valuelists[prf] = []
                        valuelists[prf].extend([float(x)
                                                for x in mat[1].split()])
                    else:
                        continue
    for dictname, prfname in zip(['M', 'CG', 'CS', 'CT'], ['M', 'G', 'S', 'T']):
        bdf[dictname] = dict(
            list(zip(namelists[prfname], valuelists[prfname])))
    bdf['__Origin__'] = 'BDFv1'
    bdf['__particle__'] = 'photon'

    if load_data:
        f = open(dataname, 'r')
        try:
            s = f.read()
        except IOError as ioe:
            # an ugly bug (M$ KB899149) in W!nd0w$ causes an error if loading too
            # large a file from a network drive and opening it read-only.
            if ioe.errno == 22:
                f.close()
                try:
                    # one work-around is to open it read-write.
                    f = open(dataname, 'r+b')
                    s = f.read()
                except IOError:
                    # if this does not work, inform the user to either obtain
                    # write permission for that file or copy it to a local
                    # drive
                    f.close()
                    raise IOError(22, """
You were probably trying to open a read-only file from a network drive on
Windows, weren\'t you? There is a bug in Windows causing this error
(see http://support.microsoft.com/default.aspx?scid=kb;en-us;899149).
To work around this, please either obtain write permission for that file
(I won't write anything to it, I promise!!!) or copy it to a local drive.
Sorry for the inconvenience.""", ioe.filename)
        datasets = re.findall(
            '#\s*(?P<name>\w+)\[(?P<xsize>\d+):(?P<ysize>\d+)\]', s)
        names = [d[0] for d in datasets]
        xsize = [int(d[1]) for d in datasets]
        ysize = [int(d[2]) for d in datasets]
        dt = np.dtype(bdf['type'])
        for i in range(len(datasets)):
            start = s.find('#%s' % names[i])
            if i < len(datasets) - 1:
                end = s.find('#%s' % (names[i + 1]))
            else:
                end = len(s)
            s1 = s[start:end]
            datasize = xsize[i] * ysize[i] * dt.itemsize
            if datasize > len(s1):
                # assume we are dealing with a BOOL matrix
                bdf[names[i]] = np.fromstring(
                    s1[-xsize[i] * ysize[i]:], dtype=np.uint8)
            else:
                bdf[names[i]] = np.fromstring(
                    s1[-xsize[i] * ysize[i] * dt.itemsize:], dtype=dt)
            # conversion: Matlab saves the array in Fortran-style ordering (columns first).
            # Python however loads in C-style: rows first. We need to take care:
            #   1) reshape from linear to (ysize,xsize) and not (xsize,ysize)
            #   2) transpose (swaps columns and rows)
            # After these operations, we only have to rotate this counter-clockwise by 90
            # degrees because bdf2_write rotates by +270 degrees before saving.
            bdf[names[i]] = np.rot90(
                bdf[names[i]].reshape((ysize[i], xsize[i]), order='F'), 1)

    return bdf


Instruction:
Can you generate a brief explanation for the following Python 3 code
def readmarheader(filename):
    with open(filename, 'rb') as f:
        intheader = np.fromstring(f.read(10 * 4), np.int32)
        floatheader = np.fromstring(f.read(15 * 4), '<f4')
        strheader = f.read(24)
        f.read(4)
        otherstrings = [f.read(16) for i in range(29)]
    return {'Xsize': intheader[0], 'Ysize': intheader[1], 'MeasTime': intheader[8],
            'BeamPosX': floatheader[7], 'BeamPosY': floatheader[8],
            'Wavelength': floatheader[9], 'Dist': floatheader[10],
            '__Origin__': 'MarResearch .image', 'recordlength': intheader[2],
            'highintensitypixels': intheader[4],
            'highintensityrecords': intheader[5],
            'Date': dateutil.parser.parse(strheader),
            'Detector': 'MARCCD', '__particle__': 'photon'}

Response: Read a header from a MarResearch. image file.


Instruction:
Can you generate the documentation for the following Python 3 function
def readBerSANS(filename):
    hed = {'Comment': ''}
    translate = {'Lambda': 'Wavelength',
                 'Title': 'Owner',
                 'SampleName': 'Title',
                 'BeamcenterX': 'BeamPosY',
                 'BeamcenterY': 'BeamPosX',
                 'Time': 'MeasTime',
                 'TotalTime': 'MeasTime',
                 'Moni1': 'Monitor',
                 'Moni2': 'Monitor',
                 'Moni': 'Monitor',
                 'Transmission': 'Transm',
                 }
    with open(filename, 'rt') as f:
        comment_next = False
        for l in f:
            l = l.strip()
            if comment_next:
                hed['Comment'] = hed['Comment'] + '\n' + l
                comment_next = False
            elif l.startswith('%Counts'):
                break
            elif l.startswith('%Comment'):
                comment_next = True
            elif l.startswith('%'):
                continue
            elif l.split('=', 1)[0] in translate:
                hed[translate[l.split('=', 1)[0]]] = misc.parse_number(
                    l.split('=', 1)[1])
            else:
                try:
                    hed[l.split('=', 1)[0]] = misc.parse_number(
                        l.split('=', 1)[1])
                except IndexError:
                    print(l.split('=', 1))
    if 'FileName' in hed:
        m = re.match('D(\d+)\.(\d+)', hed['FileName'])
        if m is not None:
            hed['FSN'] = int(m.groups()[0])
            hed['suffix'] = int(m.groups()[1])
    if 'FileDate' in hed:
        hed['Date'] = dateutil.parser.parse(hed['FileDate'])
    if 'FileTime' in hed:
        hed['Date'] = datetime.datetime.combine(
            hed['Date'].date(), dateutil.parser.parse(hed['FileTime']).time())
    hed['__Origin__'] = 'BerSANS'
    if 'SD' in hed:
        hed['Dist'] = hed['SD'] * 1000
    if hed['Comment'].startswith('\n'):
        hed['Comment'] = hed['Comment'][1:]
    hed['__particle__'] = 'neutron'
    hed['Wavelength'] *= 10  # convert from nanometres to Angstroems
    return hed

Response: Read a header from a BerSANS file.


Instruction:
Make a summary of the following Python 3 code
def Cosine(x, a, omega, phi, y0):
    return a * np.cos(x * omega + phi) + y0

Response: Cosine function for frequency - based variables.


Instruction:
Can you implement a function in Python 3 that
seconds order polynomial of a set of terms.

Response:
def Square(x, a, b, c):
    """Second order polynomial

    Inputs:
    -------
        ``x``: independent variable
        ``a``: coefficient of the second-order term
        ``b``: coefficient of the first-order term
        ``c``: additive constant

    Formula:
    --------
        ``a*x^2 + b*x + c``
    """
    return a * x ** 2 + b * x + c


Instruction:
How would you code a function in Python 3 to
third order polynomial of a set of terms.

Response:
def Cube(x, a, b, c, d):
    """Third order polynomial

    Inputs:
    -------
        ``x``: independent variable
        ``a``: coefficient of the third-order term
        ``b``: coefficient of the second-order term
        ``c``: coefficient of the first-order term
        ``d``: additive constant

    Formula:
    --------
        ``a*x^3 + b*x^2 + c*x + d``
    """
    return a * x ** 3 + b * x ** 2 + c * x + d


Instruction:
Can you generate a brief explanation for the following Python 3 code
def Exponential(x, a, tau, y0):
    return np.exp(x / tau) * a + y0

Response: Exponential function that returns the log - likelihood of a time - series.


Instruction:
How would you explain what the following Python 3 function does
def Lorentzian(x, a, x0, sigma, y0):
    return a / (1 + ((x - x0) / sigma) ** 2) + y0

Response:
Lorentzian peak

    Inputs:
    -------
        ``x``: independent variable
        ``a``: scaling factor (extremal value)
        ``x0``: center
        ``sigma``: half width at half maximum
        ``y0``: additive constant

    Formula:
    --------
        ``a/(1+((x-x0)/sigma)^2)+y0``


Instruction:
Explain what the following Python 3 code does
def LogNormal(x, a, mu, sigma):
    return a / np.sqrt(2 * np.pi * sigma ** 2 * x ** 2) *\
        np.exp(-(np.log(x) - mu) ** 2 / (2 * sigma ** 2))

Response: PDF of a log - normal distribution.


Instruction:
Can you generate the documentation for the following Python 3 function
def radintpix(data, dataerr, bcx, bcy, mask=None, pix=None, returnavgpix=False,
              phi0=0, dphi=0, returnmask=False, symmetric_sector=False,
              doslice=False, errorpropagation=2, autoqrange_linear=True):
    """Radial integration (averaging) on the detector plane

    Inputs:
        data: scattering pattern matrix (np.ndarray, dtype: np.double)
        dataerr: error matrix (np.ndarray, dtype: np.double; or None)
        bcx, bcy: beam position, counting from 1
        mask: mask matrix (np.ndarray, dtype: np.uint8)
        pix: pixel distance values (abscissa) from origin. If None,
            auto-determine.
        returnavgpix: if the averaged pixel values should be returned
        phi0: starting angle (radian) for sector integration. If doslice is True,
            this is the angle of the slice.
        dphi: angular width (radian) of the sector or width (pixels) of the
            slice. If negative or zero, full radial average is requested.
        returnmask: if the effective mask matrix is to be returned
        symmetric_sector: the sector defined by phi0+pi is also to be used for
            integration.
        doslice: if slicing is to be done instead of sector averaging.
        autoqrange_linear: if the automatically determined q-range is to be
            linspace-d. Otherwise log10 spacing will be applied.

    Outputs: pix, Intensity, [Error], Area, [mask]
        Error is only returned if dataerr is not None
        mask is only returned if returnmask is True

    Relies heavily (completely) on radint().
    """
    if isinstance(data, np.ndarray):
        data = data.astype(np.double)
    if isinstance(dataerr, np.ndarray):
        dataerr = dataerr.astype(np.double)
    if isinstance(mask, np.ndarray):
        mask = mask.astype(np.uint8)
    return radint(data, dataerr, -1, -1, -1,
                  1.0 * bcx, 1.0 * bcy, mask, pix, returnavgpix,
                  phi0, dphi, returnmask, symmetric_sector, doslice, False, errorpropagation, autoqrange_linear)

Response: Radial integration of a single image on the detector plane.


Instruction:
Write a Python 3 function that can
find all subdirs of a directory.

Response:
def find_subdirs(startdir='.', recursion_depth=None):
    """Find all subdirectory of a directory.

    Inputs:
        startdir: directory to start with. Defaults to the current folder.
        recursion_depth: number of levels to traverse. None is infinite.

    Output: a list of absolute names of subfolders.

    Examples:
        >>> find_subdirs('dir',0)  # returns just ['dir']

        >>> find_subdirs('dir',1)  # returns all direct (first-level) subdirs
                                   # of 'dir'.
    """
    startdir = os.path.expanduser(startdir)
    direct_subdirs = [os.path.join(startdir, x) for x in os.listdir(
        startdir) if os.path.isdir(os.path.join(startdir, x))]
    if recursion_depth is None:
        next_recursion_depth = None
    else:
        next_recursion_depth = recursion_depth - 1
    if (recursion_depth is not None) and (recursion_depth <= 1):
        return [startdir] + direct_subdirs
    else:
        subdirs = []
        for d in direct_subdirs:
            subdirs.extend(find_subdirs(d, next_recursion_depth))
        return [startdir] + subdirs


Instruction:
How would you explain what the following Python 3 function does
def findpeak(x, y, dy=None, position=None, hwhm=None, baseline=None, amplitude=None, curve='Lorentz'):
    warnings.warn('Function findpeak() is deprecated, please use findpeak_single() instead.', DeprecationWarning)
    pos, hwhm, baseline, ampl = findpeak_single(x, y, dy, position, hwhm, baseline, amplitude, curve)
    return pos.val, pos.err, hwhm.val, hwhm.err, baseline.val, baseline.err, ampl.val, ampl.err

Response: This function finds a positive peak in the dataset.


Instruction:
Can you generate the documentation for the following Python 3 function
def findpeak_single(x, y, dy=None, position=None, hwhm=None, baseline=None, amplitude=None, curve='Lorentz',
                    return_stat=False, signs=(-1, 1), return_x=None):
    """Find a (positive or negative) peak in the dataset.

    Inputs:
        x, y, dy: abscissa, ordinate and the error of the ordinate (can be None)
        position, hwhm, baseline, amplitude: first guesses for the named parameters
        curve: 'Gauss' or 'Lorentz' (default)
        return_stat: return fitting statistics from easylsq.nlsq_fit()
        signs: a tuple, can be (1,), (-1,), (1,-1). Will try these signs for the peak amplitude
        return_x: abscissa on which the fitted function form has to be evaluated

    Outputs:
        peak position, hwhm, baseline, amplitude[, stat][, peakfunction]

        where:
            peak position, hwhm, baseline, amplitude are ErrorValue instances.
            stat is the statistics dictionary, returned only if return_stat is True
            peakfunction is the fitted peak evaluated at return_x if it is not None.

    Notes:
        A Gauss or a Lorentz curve is fitted, depending on the value of 'curve'. The abscissa
        should be sorted, ascending.
    """
    y_orig=y
    if dy is None: dy = np.ones_like(x)
    if curve.upper().startswith('GAUSS'):
        def fitfunc(x_, amplitude_, position_, hwhm_, baseline_):
            return amplitude_ * np.exp(-0.5 * (x_ - position_) ** 2 / hwhm_ ** 2) + baseline_
    elif curve.upper().startswith('LORENTZ'):
        def fitfunc(x_, amplitude_, position_, hwhm_, baseline_):
            return amplitude_ * hwhm_ ** 2 / (hwhm_ ** 2 + (position_ - x_) ** 2) + baseline_
    else:
        raise ValueError('Invalid curve type: {}'.format(curve))
    results=[]
    # we try fitting a positive and a negative peak and return the better fit (where R2 is larger)
    for sign in signs:
        init_params={'position':position,'hwhm':hwhm,'baseline':baseline,'amplitude':amplitude}
        y = y_orig * sign
        if init_params['position'] is None: init_params['position'] = x[y == y.max()][0]
        if init_params['hwhm'] is None: init_params['hwhm'] = 0.5 * (x.max() - x.min())
        if init_params['baseline'] is None: init_params['baseline'] = y.min()
        if init_params['amplitude'] is None: init_params['amplitude'] = y.max() - init_params['baseline']
        results.append(nlsq_fit(x, y, dy, fitfunc, (init_params['amplitude'],
                                                   init_params['position'],
                                                   init_params['hwhm'],
                                                   init_params['baseline']))+(sign,))
    max_R2=max([r[2]['R2'] for r in results])
    p,dp,stat,sign=[r for r in results if r[2]['R2']==max_R2][0]
    retval = [ErrorValue(p[1], dp[1]), ErrorValue(abs(p[2]), dp[2]), sign * ErrorValue(p[3], dp[3]),
              sign * ErrorValue(p[0], dp[0])]
    if return_stat:
        stat['func_value'] = stat['func_value'] * sign
        retval.append(stat)
    if return_x is not None:
        retval.append(sign * fitfunc(return_x, p[0], p[1], p[2], p[3]))
    return tuple(retval)

Response: Find a single peak in the dataset.


Instruction:
Given the following Python 3 function, write the documentation
def findpeak_multi(x, y, dy, N, Ntolerance, Nfit=None, curve='Lorentz', return_xfit=False, return_stat=False):
    if Nfit is None:
        Nfit = N
    # find points where the curve grows for N points before them and
    # decreases for N points after them. To accomplish this, we create
    # an indicator array of the sign of the first derivative.
    sgndiff = np.sign(np.diff(y))
    xdiff = x[:-1]  # associate difference values to the lower 'x' value.
    pix = np.arange(len(x) - 1)  # pixel coordinates create an indicator
    # array as the sum of sgndiff shifted left and right.  whenever an
    # element of this is 2*N, it fulfills the criteria above.
    indicator = np.zeros(len(sgndiff) - 2 * N)
    for i in range(2 * N):
        indicator += np.sign(N - i) * sgndiff[i:-2 * N + i]
    # add the last one, since the indexing is different (would be
    # [2*N:0], which is not what we want)
    indicator += -sgndiff[2 * N:]
    # find the positions (indices) of the peak. The strict criteria is
    # relaxed somewhat by using the Ntolerance value. Note the use of
    # 2*Ntolerance, since each outlier point creates two outliers in
    # sgndiff (-1 insted of +1 and vice versa).
    peakpospix = pix[N:-N][indicator >= 2 * N - 2 * Ntolerance]
    ypeak = y[peakpospix]
    # Now refine the found positions by least-squares fitting. But
    # first we have to sort out other non-peaks, i.e. found points
    # which have other found points with higher values in their [-N,N]
    # neighbourhood.
    pos = []; ampl = []; hwhm = []; baseline = []; xfit = []; stat = []
    dy1 = None
    for i in range(len(ypeak)):
        if not [j for j in list(range(i + 1, len(ypeak))) + list(range(0, i)) if abs(peakpospix[j] - peakpospix[i]) <= N and ypeak[i] < ypeak[j]]:
            # only leave maxima.
            idx = peakpospix[i]
            if dy is not None:
                dy1 = dy[(idx - Nfit):(idx + Nfit + 1)]
            xfit_ = x[(idx - Nfit):(idx + Nfit + 1)]
            pos_, hwhm_, baseline_, ampl_, stat_ = findpeak_single(xfit_, y[(idx - Nfit):(idx + Nfit + 1)], dy1, position=x[idx], return_stat=True)

            stat.append(stat_)
            xfit.append(xfit_)
            pos.append(pos_)
            ampl.append(ampl_)
            hwhm.append(hwhm_)
            baseline.append(baseline_)
    results = [pos, hwhm, baseline, ampl]
    if return_xfit:
        results.append(xfit)
    if return_stat:
        results.append(stat)
    return tuple(results)

Response: Find multiple peaks in the dataset given by vectors x and y.


Instruction:
Make a summary of the following Python 3 code
def findpeak_asymmetric(x, y, dy=None, curve='Lorentz', return_x=None, init_parameters=None):
    idx = np.logical_and(np.isfinite(x), np.isfinite(y))
    if dy is not None:
        idx = np.logical_and(idx, np.isfinite(dy))
    x=x[idx]
    y=y[idx]
    if dy is not None:
        dy=dy[idx]
    if curve.lower().startswith('loren'):
        lorentzian = True
    elif curve.lower().startswith('gauss'):
        lorentzian = False
    else:
        raise ValueError('Unknown peak type {}'.format(curve))

    def peakfunc(pars, x, lorentzian=True):
        x0, sigma1, sigma2, C, A = pars
        result = np.empty_like(x)
        if lorentzian:
            result[x < x0] = A * sigma1 ** 2 / (sigma1 ** 2 + (x0 - x[x < x0]) ** 2) + C
            result[x >= x0] = A * sigma2 ** 2 / (sigma2 ** 2 + (x0 - x[x >= x0]) ** 2) + C
        else:
            result[x < x0] = A * np.exp(-(x[x < x0] - x0) ** 2 / (2 * sigma1 ** 2))
            result[x >= x0] = A * np.exp(-(x[x >= x0] - x0) ** 2 / (2 * sigma1 ** 2))
        return result

    def fitfunc(pars, x, y, dy, lorentzian=True):
        yfit = peakfunc(pars, x, lorentzian)
        if dy is None:
            return yfit - y
        else:
            return (yfit - y) / dy
    if init_parameters is not None:
        pos, hwhmleft, hwhmright, baseline, amplitude = [float(x) for x in init_parameters]
    else:
        baseline = y.min()
        amplitude = y.max() - baseline
        hwhmleft = hwhmright = (x.max() - x.min()) * 0.5
        pos = x[np.argmax(y)]
    #print([pos,hwhm,hwhm,baseline,amplitude])
    result = scipy.optimize.least_squares(fitfunc, [pos, hwhmleft, hwhmright, baseline, amplitude],
                                          args=(x, y, dy, lorentzian),
                                          bounds=([x.min(), 0, 0, -np.inf, 0],
                                                  [x.max(), np.inf, np.inf, np.inf, np.inf]))
#    print(result.x[0], result.x[1], result.x[2], result.x[3], result.x[4], result.message, result.success)
    if not result.success:
        raise RuntimeError('Error while peak fitting: {}'.format(result.message))
    if dy is None:
        ret = (result.x[0], result.x[1], result.x[2], result.x[3], result.x[4])
    else:
        # noinspection PyTupleAssignmentBalance
        _, s, VT = svd(result.jac, full_matrices=False)
        threshold = np.finfo(float).eps * max(result.jac.shape) * s[0]
        s = s[s > threshold]
        VT = VT[:s.size]
        pcov = np.dot(VT.T / s ** 2, VT)
        ret = tuple([ErrorValue(result.x[i], pcov[i, i] ** 0.5) for i in range(5)])
    if return_x is not None:
        ret = ret + (peakfunc([float(x) for x in ret], return_x, lorentzian),)
    return ret

Response: This function finds an asymmetric peak in the dataset.


Instruction:
Can you implement a function in Python 3 that
reads the next spec scan in the file.

Response:
def readspecscan(f, number=None):
    """Read the next spec scan in the file, which starts at the current position."""
    scan = None
    scannumber = None
    while True:
        l = f.readline()
        if l.startswith('#S'):
            scannumber = int(l[2:].split()[0])
            if not ((number is None) or (number == scannumber)):
                # break the loop, will skip to the next empty line after this
                # loop
                break
            if scan is None:
                scan = {}
            scan['number'] = scannumber
            scan['command'] = l[2:].split(None, 1)[1].strip()
            scan['data'] = []
        elif l.startswith('#C'):
            scan['comment'] = l[2:].strip()
        elif l.startswith('#D'):
            scan['datestring'] = l[2:].strip()
        elif l.startswith('#T'):
            scan['countingtime'] = float(l[2:].split()[0])
            scan['scantimeunits'] = l[2:].split()[1].strip()
        elif l.startswith('#M'):
            scan['countingcounts'] = float(l[2:].split()[0])
        elif l.startswith('#G'):
            if 'G' not in scan:
                scan['G'] = []
            scan['G'].extend([float(x) for x in l.split()[1:]])
        elif l.startswith('#P'):
            if 'positions' not in scan:
                scan['positions'] = []
            scan['positions'].extend([float(x) for x in l.split()[1:]])
        elif l.startswith('#Q'):
            pass
        elif l.startswith('#N'):
            n = [float(x) for x in l[2:].strip().split()]
            if len(n) == 1:
                scan['N'] = n[0]
            else:
                scan['N'] = n
        elif l.startswith('#L'):
            scan['Columns'] = [x.strip() for x in l[3:].split('  ')]
        elif not l:
            # end of file
            if scan is None:
                raise SpecFileEOF
            else:
                break
        elif not l.strip():
            break  # empty line, end of scan in file.
        elif l.startswith('#'):
            # ignore other lines starting with a hashmark.
            continue
        else:
            scan['data'].append(tuple(float(x) for x in l.split()))
    while l.strip():
        l = f.readline()
    if scan is not None:
        scan['data'] = np.array(
            scan['data'], dtype=list(zip(scan['Columns'], itertools.repeat(np.float))))
        return scan
    else:
        return scannumber


Instruction:
Can you write a function in Python 3 where it
opens a SPEC file and read its content into a dict.

Response:
def readspec(filename, read_scan=None):
    """Open a SPEC file and read its content

    Inputs:

        filename: string
            the file to open

        read_scan: None, 'all' or integer
            the index of scan to be read from the file. If None, no scan should be read. If
            'all', all scans should be read. If a number, just the scan with that number
            should be read.

    Output:
        the data in the spec file in a dict.
    """
    with open(filename, 'rt') as f:
        sf = {'motors': [], 'maxscannumber': 0}
        sf['originalfilename'] = filename
        lastscannumber = None
        while True:
            l = f.readline()
            if l.startswith('#F'):
                sf['filename'] = l[2:].strip()
            elif l.startswith('#E'):
                sf['epoch'] = int(l[2:].strip())
                sf['datetime'] = datetime.datetime.fromtimestamp(sf['epoch'])
            elif l.startswith('#D'):
                sf['datestring'] = l[2:].strip()
            elif l.startswith('#C'):
                sf['comment'] = l[2:].strip()
            elif l.startswith('#O'):
                try:
                    l = l.split(None, 1)[1]
                except IndexError:
                    continue
                if 'motors' not in list(sf.keys()):
                    sf['motors'] = []
                sf['motors'].extend([x.strip() for x in l.split('  ')])
            elif not l.strip():
                # empty line, signifies the end of the header part. The next
                # line will be a scan.
                break
        sf['scans'] = {}
        if read_scan is not None:
            if read_scan == 'all':
                nr = None
            else:
                nr = read_scan
            try:
                while True:
                    s = readspecscan(f, nr)
                    if isinstance(s, dict):
                        sf['scans'][s['number']] = s
                        if nr is not None:
                            break
                        sf['maxscannumber'] = max(
                            sf['maxscannumber'], s['number'])
                    elif s is not None:
                        sf['maxscannumber'] = max(sf['maxscannumber'], s)
            except SpecFileEOF:
                pass
        else:
            while True:
                l = f.readline()
                if not l:
                    break
                if l.startswith('#S'):
                    n = int(l[2:].split()[0])
                    sf['maxscannumber'] = max(sf['maxscannumber'], n)
        for n in sf['scans']:
            s = sf['scans'][n]
            s['motors'] = sf['motors']
            if 'comment' not in s:
                s['comment'] = sf['comment']
            if 'positions' not in s:
                s['positions'] = [None] * len(sf['motors'])
    return sf


Instruction:
Write a Python 3 function for
reading a file from beamline B1 HASYLAB and return a dictionary. The keys are self - planatory. The values are self - planatory.

Response:
def readabt(filename, dirs='.'):
    """Read abt_*.fio type files from beamline B1, HASYLAB.

    Input:
        filename: the name of the file.
        dirs: directories to search for files in

    Output:
        A dictionary. The fields are self-explanatory.
    """
    # resolve filename
    filename = misc.findfileindirs(filename, dirs)
    f = open(filename, 'rt')
    abt = {'offsetcorrected': False, 'params': {}, 'columns': [], 'data': [], 'title': '<no_title>',
           'offsets': {}, 'filename': filename}
    readingmode = ''
    for l in f:
        l = l.strip()
        if l.startswith('!') or len(l) == 0:
            continue
        elif l.startswith('%c'):
            readingmode = 'comments'
        elif l.startswith('%p'):
            readingmode = 'params'
        elif l.startswith('%d'):
            readingmode = 'data'
        elif readingmode == 'comments':
            m = re.match(
                r'(?P<scantype>\w+)-Scan started at (?P<startdate>\d+-\w+-\d+) (?P<starttime>\d+:\d+:\d+), ended (?P<endtime>\d+:\d+:\d+)', l)
            if m:
                abt.update(m.groupdict())
                continue
            else:
                m = re.match(r'Name: (?P<name>\w+)', l)
                if m:
                    abt.update(m.groupdict())
                    m1 = re.search(r'from (?P<from>\d+(?:.\d+)?)', l)
                    if m1:
                        abt.update(m1.groupdict())
                    m1 = re.search(r'to (?P<to>\d+(?:.\d+)?)', l)
                    if m1:
                        abt.update(m1.groupdict())
                    m1 = re.search(r'by (?P<by>\d+(?:.\d+)?)', l)
                    if m1:
                        abt.update(m1.groupdict())
                    m1 = re.search(r'sampling (?P<sampling>\d+(?:.\d+)?)', l)
                    if m1:
                        abt.update(m1.groupdict())
                    continue
            if l.find('Counter readings are offset corrected') >= 0:
                abt['offsetcorrected'] = True
                readingmode = 'offsets'
                continue
            # if we reach here in 'comments' mode, this is the title line
            abt['title'] = l
            continue
        elif readingmode == 'offsets':
            m = re.findall(r'(\w+)\s(\d+(?:.\d+)?)', l)
            if m:
                abt['offsets'].update(dict(m))
                for k in abt['offsets']:
                    abt['offsets'][k] = float(abt['offsets'][k])
        elif readingmode == 'params':
            abt['params'][l.split('=')[0].strip()] = float(
                l.split('=')[1].strip())
        elif readingmode == 'data':
            if l.startswith('Col'):
                abt['columns'].append(l.split()[2])
            else:
                abt['data'].append([float(x) for x in l.split()])
    f.close()
    # some post-processing
    # remove common prefix from column names
    maxcolnamelen = max(len(c) for c in abt['columns'])
    l = 1
    for l in range(1, maxcolnamelen):
        if len(set([c[:l] for c in abt['columns']])) > 1:
            break
    abt['columns'] = [c[l - 1:] for c in abt['columns']]
    # represent data as a structured array
    dt = np.dtype(list(zip(abt['columns'], itertools.repeat(np.double))))
    abt['data'] = np.array(abt['data'], dtype=np.double).view(dt)
    # dates and times in datetime formats
    monthnames = ['Jan', 'Feb', 'Mar', 'Apr', 'May',
                  'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
    for m, i in zip(monthnames, itertools.count(1)):
        abt['startdate'] = abt['startdate'].replace(m, str(i))
    abt['startdate'] = datetime.date(
        *reversed([int(x) for x in abt['startdate'].split('-')]))
    abt['starttime'] = datetime.time(
        *[int(x) for x in abt['starttime'].split(':')])
    abt['endtime'] = datetime.time(
        *[int(x) for x in abt['endtime'].split(':')])
    abt['start'] = datetime.datetime.combine(
        abt['startdate'], abt['starttime'])
    if abt['endtime'] <= abt['starttime']:
        abt['end'] = datetime.datetime.combine(
            abt['startdate'] + datetime.timedelta(1), abt['endtime'])
    else:
        abt['end'] = datetime.datetime.combine(
            abt['startdate'], abt['endtime'])
    del abt['starttime']
    del abt['startdate']
    del abt['endtime']
    # convert some fields to float
    for k in ['from', 'to', 'by', 'sampling']:
        if k in abt:
            abt[k] = float(abt[k])
    # change space and dash in title to underscore
    abt['title'] = abt['title'].replace('-', '_').replace(' ', '_')
    return abt


Instruction:
Explain what the following Python 3 code does
def energy(self) -> ErrorValue:
        return (ErrorValue(*(scipy.constants.physical_constants['speed of light in vacuum'][0::2])) *
                ErrorValue(*(scipy.constants.physical_constants['Planck constant in eV s'][0::2])) /
                scipy.constants.nano /
                self.wavelength)

Response: X - ray energy


Instruction:
Given the following Python 3 function, write the documentation
def maskname(self) -> Optional[str]:
        try:
            maskid = self._data['maskname']
            if not maskid.endswith('.mat'):
                maskid = maskid + '.mat'
            return maskid
        except KeyError:
            return None

Response: Name of the mask matrix file.


Instruction:
Can you create a Python 3 function that
finds the beam center with the gravity method.

Response:
def findbeam_gravity(data, mask):
    """Find beam center with the "gravity" method

    Inputs:
        data: scattering image
        mask: mask matrix

    Output:
        a vector of length 2 with the x (row) and y (column) coordinates
         of the origin, starting from 1
    """
    # for each row and column find the center of gravity
    data1 = data.copy()  # take a copy, because elements will be tampered with
    data1[mask == 0] = 0  # set masked elements to zero
    # vector of x (row) coordinates
    x = np.arange(data1.shape[0])
    # vector of y (column) coordinates
    y = np.arange(data1.shape[1])
    # two column vectors, both containing ones. The length of onex and
    # oney corresponds to length of x and y, respectively.
    onex = np.ones_like(x)
    oney = np.ones_like(y)
    # Multiply the matrix with x. Each element of the resulting column
    # vector will contain the center of gravity of the corresponding row
    # in the matrix, multiplied by the "weight". Thus: nix_i=sum_j( A_ij
    # * x_j). If we divide this by spamx_i=sum_j(A_ij), then we get the
    # center of gravity. The length of this column vector is len(y).
    nix = np.dot(x, data1)
    spamx = np.dot(onex, data1)
    # indices where both nix and spamx is nonzero.
    goodx = ((nix != 0) & (spamx != 0))
    # trim y, nix and spamx by goodx, eliminate invalid points.
    nix = nix[goodx]
    spamx = spamx[goodx]

    # now do the same for the column direction.
    niy = np.dot(data1, y)
    spamy = np.dot(data1, oney)
    goody = ((niy != 0) & (spamy != 0))
    niy = niy[goody]
    spamy = spamy[goody]
    # column coordinate of the center in each row will be contained in
    # ycent, the row coordinate of the center in each column will be
    # in xcent.
    ycent = nix / spamx
    xcent = niy / spamy
    # return the mean values as the centers.
    return [xcent.mean(), ycent.mean()]


Instruction:
How would you code a function in Python 3 to
find beam center with the slices method.

Response:
def findbeam_slices(data, orig_initial, mask=None, maxiter=0, epsfcn=0.001,
                    dmin=0, dmax=np.inf, sector_width=np.pi / 9.0, extent=10, callback=None):
    """Find beam center with the "slices" method

    Inputs:
        data: scattering matrix
        orig_initial: estimated value for x (row) and y (column)
            coordinates of the beam center, starting from 1.
        mask: mask matrix. If None, nothing will be masked. Otherwise it
            should be of the same size as data. Nonzero means non-masked.
        maxiter: maximum number of iterations for scipy.optimize.leastsq
        epsfcn: input for scipy.optimize.leastsq
        dmin: disregard pixels nearer to the origin than this
        dmax: disregard pixels farther from the origin than this
        sector_width: width of sectors in radians
        extent: approximate distance of the current and the real origin in pixels.
            Too high a value makes the fitting procedure unstable. Too low a value
            does not permit to move away the current origin.
        callback: callback function (expects no arguments)

    Output:
        a vector of length 2 with the x (row) and y (column) coordinates
         of the origin.
    """
    if mask is None:
        mask = np.ones(data.shape)
    data = data.astype(np.double)

    def targetfunc(orig, data, mask, orig_orig, callback):
        # integrate four sectors
        I = [None] * 4
        p, Ints, A = radint_nsector(data, None, -1, -1, -1, orig[0] + orig_orig[0], orig[1] + orig_orig[1], mask=mask,
                                    phi0=np.pi / 4 - 0.5 * sector_width, dphi=sector_width,
                                    Nsector=4)
        minpix = max(max(p.min(0).tolist()), dmin)
        maxpix = min(min(p.max(0).tolist()), dmax)
        if (maxpix < minpix):
            raise ValueError('The four slices do not overlap! Please give a\
 better approximation for the origin or use another centering method.')
        for i in range(4):
            I[i] = Ints[:, i][(p[:, i] >= minpix) & (p[:, i] <= maxpix)]
        ret = ((I[0] - I[2]) ** 2 + (I[1] - I[3]) ** 2) / (maxpix - minpix)
        if callback is not None:
            callback()
        return ret
    orig = scipy.optimize.leastsq(targetfunc, np.array([extent, extent]),
                                  args=(data, 1 - mask.astype(np.uint8),
                                        np.array(orig_initial) - extent, callback),
                                  maxfev=maxiter, epsfcn=0.01)
    return orig[0] + np.array(orig_initial) - extent


Instruction:
Here you have a function in Python 3, explain what it does
def findbeam_azimuthal(data, orig_initial, mask=None, maxiter=100, Ntheta=50,
                       dmin=0, dmax=np.inf, extent=10, callback=None):
    """Find beam center using azimuthal integration

    Inputs:
        data: scattering matrix
        orig_initial: estimated value for x (row) and y (column)
            coordinates of the beam center, starting from 1.
        mask: mask matrix. If None, nothing will be masked. Otherwise it
            should be of the same size as data. Nonzero means non-masked.
        maxiter: maximum number of iterations for scipy.optimize.fmin
        Ntheta: the number of theta points for the azimuthal integration
        dmin: pixels nearer to the origin than this will be excluded from
            the azimuthal integration
        dmax: pixels farther from the origin than this will be excluded from
            the azimuthal integration
        extent: approximate distance of the current and the real origin in pixels.
            Too high a value makes the fitting procedure unstable. Too low a value
            does not permit to move away the current origin.
        callback: callback function (expects no arguments)
    Output:
        a vector of length 2 with the x and y coordinates of the origin,
            starting from 1
    """
    if mask is None:
        mask = np.ones(data.shape)
    data = data.astype(np.double)

    def targetfunc(orig, data, mask, orig_orig, callback):
        def sinfun(p, x, y):
            return (y - np.sin(x + p[1]) * p[0] - p[2]) / np.sqrt(len(x))
        t, I, a = azimintpix(data, None, orig[
                             0] + orig_orig[0], orig[1] + orig_orig[1], mask.astype('uint8'), Ntheta, dmin, dmax)
        if len(a) > (a > 0).sum():
            raise ValueError('findbeam_azimuthal: non-complete azimuthal average, please consider changing dmin, dmax and/or orig_initial!')
        p = ((I.max() - I.min()) / 2.0, t[I == I.max()][0], I.mean())
        p = scipy.optimize.leastsq(sinfun, p, (t, I))[0]
        # print "findbeam_azimuthal: orig=",orig,"amplitude=",abs(p[0])
        if callback is not None:
            callback()
        return abs(p[0])
    orig1 = scipy.optimize.fmin(targetfunc, np.array([extent, extent]),
                                args=(data, 1 - mask, np.array(orig_initial) - extent,
                                      callback), maxiter=maxiter, disp=0)
    return orig1 + np.array(orig_initial) - extent

Response: This function finds the beam center using azimuthal integration.


Instruction:
Make a summary of the following Python 3 code
def findbeam_azimuthal_fold(data, orig_initial, mask=None, maxiter=100,
                            Ntheta=50, dmin=0, dmax=np.inf, extent=10, callback=None):
    """Find beam center using azimuthal integration and folding

    Inputs:
        data: scattering matrix
        orig_initial: estimated value for x (row) and y (column)
            coordinates of the beam center, starting from 1.
        mask: mask matrix. If None, nothing will be masked. Otherwise it
            should be of the same size as data. Nonzero means non-masked.
        maxiter: maximum number of iterations for scipy.optimize.fmin
        Ntheta: the number of theta points for the azimuthal integration.
            Should be even!
        dmin: pixels nearer to the origin than this will be excluded from
            the azimuthal integration
        dmax: pixels farther from the origin than this will be excluded from
            the azimuthal integration
        extent: approximate distance of the current and the real origin in pixels.
            Too high a value makes the fitting procedure unstable. Too low a value
            does not permit to move away the current origin.
        callback: callback function (expects no arguments)
    Output:
        a vector of length 2 with the x and y coordinates of the origin,
            starting from 1
    """
    if Ntheta % 2:
        raise ValueError('Ntheta should be even!')
    if mask is None:
        mask = np.ones_like(data).astype(np.uint8)
    data = data.astype(np.double)
    # the function to minimize is the sum of squared difference of two halves of
    # the azimuthal integral.

    def targetfunc(orig, data, mask, orig_orig, callback):
        I = azimintpix(data, None, orig[
                       0] + orig_orig[0], orig[1] + orig_orig[1], mask, Ntheta, dmin, dmax)[1]
        if callback is not None:
            callback()
        return np.sum((I[:Ntheta / 2] - I[Ntheta / 2:]) ** 2) / Ntheta
    orig1 = scipy.optimize.fmin(targetfunc, np.array([extent, extent]),
                                args=(data, 1 - mask, np.array(orig_initial) - extent, callback), maxiter=maxiter, disp=0)
    return orig1 + np.array(orig_initial) - extent

Response: This function finds the beam center using azimuthal integration and folding.


Instruction:
Write a Python 3 script for
finding beam with 2D weighting of semitransparent beamstop area.

Response:
def findbeam_semitransparent(data, pri, threshold=0.05):
    """Find beam with 2D weighting of semitransparent beamstop area

    Inputs:
        data: scattering matrix
        pri: list of four: [xmin,xmax,ymin,ymax] for the borders of the beam
            area under the semitransparent beamstop. X corresponds to the column
            index (ie. A[Y,X] is the element of A from the Xth column and the
            Yth row). You can get these by zooming on the figure and retrieving
            the result of axis() (like in Matlab)
        threshold: do not count pixels if their intensity falls below
            max_intensity*threshold. max_intensity is the highest count rate
            in the current row or column, respectively. Set None to disable
            this feature.

    Outputs: bcx,bcy
        the x and y coordinates of the primary beam
    """
    rowmin = np.floor(min(pri[2:]))
    rowmax = np.ceil(max(pri[2:]))
    colmin = np.floor(min(pri[:2]))
    colmax = np.ceil(max(pri[:2]))

    if threshold is not None:
        # beam area on the scattering image
        B = data[rowmin:rowmax, colmin:colmax]
        # print B.shape
        # row and column indices
        Ri = np.arange(rowmin, rowmax)
        Ci = np.arange(colmin, colmax)
        # print len(Ri)
        # print len(Ci)
        Ravg = B.mean(1)  # average over column index, will be a concave curve
        Cavg = B.mean(0)  # average over row index, will be a concave curve
        # find the maxima im both directions and their positions
        maxR = Ravg.max()
        maxRpos = Ravg.argmax()
        maxC = Cavg.max()
        maxCpos = Cavg.argmax()
        # cut off pixels which are smaller than threshold*peak_height
        Rmin = Ri[
            ((Ravg - Ravg[0]) >= ((maxR - Ravg[0]) * threshold)) & (Ri < maxRpos)][0]
        Rmax = Ri[
            ((Ravg - Ravg[-1]) >= ((maxR - Ravg[-1]) * threshold)) & (Ri > maxRpos)][-1]
        Cmin = Ci[
            ((Cavg - Cavg[0]) >= ((maxC - Cavg[0]) * threshold)) & (Ci < maxCpos)][0]
        Cmax = Ci[
            ((Cavg - Cavg[-1]) >= ((maxC - Cavg[-1]) * threshold)) & (Ci > maxCpos)][-1]
    else:
        Rmin = rowmin
        Rmax = rowmax
        Cmin = colmin
        Cmax = colmax
    d = data[Rmin:Rmax + 1, Cmin:Cmax + 1]
    x = np.arange(Rmin, Rmax + 1)
    y = np.arange(Cmin, Cmax + 1)
    bcx = (d.sum(1) * x).sum() / d.sum()
    bcy = (d.sum(0) * y).sum() / d.sum()
    return bcx, bcy


Instruction:
Write a Python 3 script to
find the beam in the radial average.

Response:
def findbeam_radialpeak(data, orig_initial, mask, rmin, rmax, maxiter=100,
                        drive_by='amplitude', extent=10, callback=None):
    """Find the beam by minimizing the width of a peak in the radial average.

    Inputs:
        data: scattering matrix
        orig_initial: first guess for the origin
        mask: mask matrix. Nonzero is non-masked.
        rmin,rmax: distance from the origin (in pixels) of the peak range.
        drive_by: 'hwhm' to minimize the hwhm of the peak or 'amplitude' to
            maximize the peak amplitude
        extent: approximate distance of the current and the real origin in pixels.
            Too high a value makes the fitting procedure unstable. Too low a value
            does not permit to move away the current origin.
        callback: callback function (expects no arguments)
    Outputs:
        the beam coordinates

    Notes:
        A Gaussian will be fitted.
    """
    orig_initial = np.array(orig_initial)
    mask = 1 - mask.astype(np.uint8)
    data = data.astype(np.double)
    pix = np.arange(rmin * 1.0, rmax * 1.0, 1)
    if drive_by.lower() == 'hwhm':
        def targetfunc(orig, data, mask, orig_orig, callback):
            I = radintpix(
                data, None, orig[0] + orig_orig[0], orig[1] + orig_orig[1], mask, pix)[1]
            hwhm = float(misc.findpeak_single(pix, I)[1])
            # print orig[0] + orig_orig[0], orig[1] + orig_orig[1], p
            if callback is not None:
                callback()
            return abs(hwhm)
    elif drive_by.lower() == 'amplitude':
        def targetfunc(orig, data, mask, orig_orig, callback):
            I = radintpix(
                data, None, orig[0] + orig_orig[0], orig[1] + orig_orig[1], mask, pix)[1]
            fp = misc.findpeak_single(pix, I)
            height = -float(fp[2] + fp[3])
            # print orig[0] + orig_orig[0], orig[1] + orig_orig[1], p
            if callback is not None:
                callback()
            return height
    else:
        raise ValueError('Invalid argument for drive_by %s' % drive_by)
    orig1 = scipy.optimize.fmin(targetfunc, np.array([extent, extent]),
                                args=(
                                    data, mask, orig_initial - extent, callback),
                                maxiter=maxiter, disp=0)
    return np.array(orig_initial) - extent + orig1


Instruction:
How would you code a function in Python 3 to
find the beam of a Guinier scattering.

Response:
def findbeam_Guinier(data, orig_initial, mask, rmin, rmax, maxiter=100,
                     extent=10, callback=None):
    """Find the beam by minimizing the width of a Gaussian centered at the
    origin (i.e. maximizing the radius of gyration in a Guinier scattering).

    Inputs:
        data: scattering matrix
        orig_initial: first guess for the origin
        mask: mask matrix. Nonzero is non-masked.
        rmin,rmax: distance from the origin (in pixels) of the Guinier range.
        extent: approximate distance of the current and the real origin in pixels.
            Too high a value makes the fitting procedure unstable. Too low a value
            does not permit to move away the current origin.
        callback: callback function (expects no arguments)
    Outputs:
        the beam coordinates

    Notes:
        A Gaussian with its will be fitted.
    """
    orig_initial = np.array(orig_initial)
    mask = 1 - mask.astype(np.uint8)
    data = data.astype(np.double)
    pix = np.arange(rmin * 1.0, rmax * 1.0, 1)
    pix2 = pix ** 2

    def targetfunc(orig, data, mask, orig_orig, callback):
        I = radintpix(
            data, None, orig[0] + orig_orig[0], orig[1] + orig_orig[1], mask, pix)[1]
        p = np.polyfit(pix2, np.log(I), 1)[0]
        if callback is not None:
            callback()
        return p
    orig1 = scipy.optimize.fmin(targetfunc, np.array([extent, extent]),
                                args=(
                                    data, mask, orig_initial - extent, callback),
                                maxiter=maxiter, disp=0)
    return np.array(orig_initial) - extent + orig1


Instruction:
Write a Python 3 function that can
find the beam of a Guinier scattering.

Response:
def findbeam_powerlaw(data, orig_initial, mask, rmin, rmax, maxiter=100,
                      drive_by='R2', extent=10, callback=None):
    """Find the beam by minimizing the width of a Gaussian centered at the
    origin (i.e. maximizing the radius of gyration in a Guinier scattering).

    Inputs:
        data: scattering matrix
        orig_initial: first guess for the origin
        mask: mask matrix. Nonzero is non-masked.
        rmin,rmax: distance from the origin (in pixels) of the fitting range
        drive_by: 'R2' or 'Chi2'
        extent: approximate distance of the current and the real origin in pixels.
            Too high a value makes the fitting procedure unstable. Too low a value
            does not permit to move away the current origin.
        callback: callback function (expects no arguments)
    Outputs:
        the beam coordinates

    Notes:
        A power-law will be fitted
    """
    orig_initial = np.array(orig_initial)
    mask = 1 - mask.astype(np.uint8)
    data = data.astype(np.double)
    pix = np.arange(rmin * 1.0, rmax * 1.0, 1)

    def targetfunc(orig, data, mask, orig_orig, callback):
        I, E = radintpix(
            data, None, orig[0] + orig_orig[0], orig[1] + orig_orig[1], mask, pix)[1:3]
        p, dp, stat = misc.easylsq.nlsq_fit(
            pix, I, E, lambda q, A, alpha: A * q ** alpha, [1.0, -3.0])
        if callback is not None:
            callback()
        #        print(orig, orig_orig, orig + orig_orig, stat[drive_by])
        if drive_by == 'R2':
            return 1 - stat['R2']
        elif drive_by.startswith('Chi2'):
            return stat[drive_by]
    orig1 = scipy.optimize.fmin(targetfunc, np.array([extent, extent]),
                                args=(
                                    data, mask, orig_initial - extent, callback),
                                maxiter=maxiter, disp=False)
    return np.array(orig_initial) - extent + orig1


Instruction:
Can you generate the documentation for the following Python 3 function
def maskname(self) -> Optional[str]:
        mask = self._data['geometry']['mask']
        if os.path.abspath(mask):
            mask = os.path.split(mask)[-1]
        return mask

Response: Name of the mask matrix file.


Instruction:
Can you tell what is the following Python 3 function doing
def fit(self, fitfunction, parinit, unfittableparameters=(), *args, **kwargs):
        kwargs['otherparameters'] = unfittableparameters
        fitter = Fitter(fitfunction, parinit, self.q, self.Intensity, self.qError, self.Error, *args, **kwargs)
        fixedvalues = [[None, p][isinstance(p, FixedParameter)] for p in parinit]
        fitter.fixparameters(fixedvalues)
        fitter.fit()
        pars = fitter.parameters()
        uncs = fitter.uncertainties()
        stats = fitter.stats()
        results = [ErrorValue(p, u) for p, u in zip(pars, uncs)] + [stats, type(self)(self.q, stats['func_value'])]
        return results

Response: Perform a nonlinear least - squares fit using sastool. misc. fitter. Fitter. fit


Instruction:
Can you tell what is the following Python 3 function doing
def momentum(self, exponent=1, errorrequested=True):
        y = self.Intensity * self.q ** exponent
        m = np.trapz(y, self.q)
        if errorrequested:
            err = self.Error * self.q ** exponent
            dm = errtrapz(self.q, err)
            return ErrorValue(m, dm)
        else:
            return m

Response: Calculate the momentum of the object.


Instruction:
Explain what the following Python 3 code does
def scalefactor(self, other, qmin=None, qmax=None, Npoints=None):
        if qmin is None:
            qmin = max(self.q.min(), other.q.min())
        if qmax is None:
            xmax = min(self.q.max(), other.q.max())
        data1 = self.trim(qmin, qmax)
        data2 = other.trim(qmin, qmax)
        if Npoints is None:
            Npoints = min(len(data1), len(data2))
        commonx = np.linspace(
                max(data1.q.min(), data2.q.min()), min(data2.q.max(), data1.q.max()), Npoints)
        data1 = data1.interpolate(commonx)
        data2 = data2.interpolate(commonx)
        return nonlinear_odr(data1.Intensity, data2.Intensity, data1.Error, data2.Error, lambda x, a: a * x, [1])[0]

Response: Calculates a scaling factor by which this curve is to be multiplied to best fit the other one.


Instruction:
Explain what the following Python 3 code does
def _substitute_fixed_parameters_covar(self, covar):
        covar_resolved = np.empty((len(self._fixed_parameters), len(self._fixed_parameters)))
        indices_of_fixed_parameters = [i for i in range(len(self.parameters())) if
                                       self._fixed_parameters[i] is not None]
        indices_of_free_parameters = [i for i in range(len(self.parameters())) if self._fixed_parameters[i] is None]
        for i in range(covar_resolved.shape[0]):
            if i in indices_of_fixed_parameters:
                # the i-eth argument was fixed. This means that the row and column corresponding to this argument
                # must be None
                covar_resolved[i, :] = 0
                continue
            for j in range(covar_resolved.shape[1]):
                if j in indices_of_fixed_parameters:
                    covar_resolved[:, j] = 0
                    continue
                covar_resolved[i, j] = covar[indices_of_free_parameters.index(i), indices_of_free_parameters.index(j)]
        return covar_resolved

Response: Insert fixed parameters in a covariance matrix


Instruction:
Given the following Python 3 function, write the documentation
def loadmask(self, filename: str) -> np.ndarray:
        mask = scipy.io.loadmat(self.find_file(filename, what='mask'))
        maskkey = [k for k in mask.keys() if not (k.startswith('_') or k.endswith('_'))][0]
        return mask[maskkey].astype(np.bool)

Response: Load a mask file.


Instruction:
How would you implement a function in Python 3 that
loads a radial scattering curve

Response:
def loadcurve(self, fsn: int) -> classes2.Curve:
        """Load a radial scattering curve"""
        return classes2.Curve.new_from_file(self.find_file(self._exposureclass + '_%05d.txt' % fsn))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def readcbf(name, load_header=False, load_data=True, for_nexus=False):
    with open(name, 'rb') as f:
        cbfbin = f.read()
    datastart = cbfbin.find(b'\x0c\x1a\x04\xd5') + 4
    hed = [x.strip() for x in cbfbin[:datastart].split(b'\n')]
    header = {}
    readingmode = None
    for i in range(len(hed)):
        if not hed[i]:
            # skip empty header lines
            continue
        elif hed[i] == b';':
            continue
        elif hed[i].startswith(b'_array_data.header_convention'):
            header['CBF_header_convention'] = str(hed[i][
                len(b'_array_data.header_convention'):].strip().replace(b'"', b''), encoding='utf-8')
        elif hed[i].startswith(b'_array_data.header_contents'):
            readingmode = 'PilatusHeader'
        elif hed[i].startswith(b'_array_data.data'):
            readingmode = 'CIFHeader'
        elif readingmode == 'PilatusHeader':
            if not hed[i].startswith(b'#'):
                continue
            line = hed[i].strip()[1:].strip()
            try:
                # try to interpret the line as the date.
                header['CBF_Date'] = dateutil.parser.parse(line)
                header['Date'] = header['CBF_Date']
                continue
            except (ValueError, TypeError):
                # eat exception: if we cannot parse this line as a date, try
                # another format.
                pass
            treated = False
            for sep in (b':', b'='):
                if treated:
                    continue
                if line.count(sep) == 1:
                    name, value = tuple(x.strip() for x in line.split(sep, 1))
                    try:
                        m = re.match(
                            b'^(?P<number>-?(\d+(.\d+)?(e-?\d+)?))\s+(?P<unit>m|s|counts|eV)$', value).groupdict()
                        value = float(m['number'])
                        m['unit'] = str(m['unit'], encoding='utf-8')
                    except AttributeError:
                        # the regex did not match the string, thus re.match()
                        # returned None.
                        pass
                    header[str(name, 'utf-8')] = value
                    treated = True
            if treated:
                continue
            if line.startswith(b'Pixel_size'):
                header['XPixel'], header['YPixel'] = tuple(
                    [float(a.strip().split(b' ')[0]) * 1000 for a in line[len(b'Pixel_size'):].split(b'x')])
            else:
                try:
                    m = re.match(
                        b'^(?P<label>[a-zA-Z0-9,_\.\-!\?\ ]*?)\s+(?P<number>-?(\d+(.\d+)?(e-?\d+)?))\s+(?P<unit>m|s|counts|eV)$', line).groupdict()
                except AttributeError:
                    pass
                else:
                    m['label'] = str(m['label'], 'utf-8')
                    m['unit'] = str(m['unit'], encoding='utf-8')
                    if m['unit'] == b'counts':
                        header[m['label']] = int(m['number'])
                    else:
                        header[m['label']] = float(m['number'])
                    if 'sensor' in m['label'] and 'thickness' in m['label']:
                        header[m['label']] *= 1e6
        elif readingmode == 'CIFHeader':
            line = hed[i]
            for sep in (b':', b'='):
                if line.count(sep) == 1:
                    label, content = tuple(x.strip()
                                           for x in line.split(sep, 1))
                    if b'"' in content:
                        content = content.replace(b'"', b'')
                    try:
                        content = int(content)
                    except ValueError:
                        content = str(content, encoding='utf-8')
                    header['CBF_' + str(label, encoding='utf-8')] = content

        else:
            pass
    ret = []
    if load_data:
        if header['CBF_X-Binary-Element-Type'] != 'signed 32-bit integer':
            raise NotImplementedError(
                'element type is not "signed 32-bit integer" in CBF, but %s.' % header['CBF_X-Binary-Element-Type'])
        if header['CBF_conversions'] != 'x-CBF_BYTE_OFFSET':
            raise NotImplementedError(
                'compression is not "x-CBF_BYTE_OFFSET" in CBF!')
        dim1 = header['CBF_X-Binary-Size-Fastest-Dimension']
        dim2 = header['CBF_X-Binary-Size-Second-Dimension']
        nbytes = header['CBF_X-Binary-Size']
        cbfdata = cbfdecompress(
            bytearray(cbfbin[datastart:datastart + nbytes]), dim1, dim2, for_nexus)
        ret.append(cbfdata)
    if load_header:
        ret.append(header)
    return tuple(ret)

Response:
Reads a cbf file from a Dectris PILATUS
    detector.


Instruction:
Write a Python 3 function for
reading a Bessy Data Format v1 file.

Response:
def readbdfv1(filename, bdfext='.bdf', bhfext='.bhf'):
    """Read bdf file (Bessy Data Format v1)

    Input
    -----
    filename: string
        the name of the file

    Output
    ------
    the BDF structure in a dict

    Notes
    -----
    This is an adaptation of the bdf_read.m macro of Sylvio Haas.
    """
    return header.readbhfv1(filename, True, bdfext, bhfext)


Instruction:
Can you write a function in Python 3 where it
reads corrected intensity and error matrices from a file.

Response:
def readint2dnorm(filename):
    """Read corrected intensity and error matrices (Matlab mat or numpy npz
    format for Beamline B1 (HASYLAB/DORISIII))

    Input
    -----
    filename: string
        the name of the file

    Outputs
    -------
    two ``np.ndarray``-s, the Intensity and the Error matrices

    File formats supported:
    -----------------------

    ``.mat``
        Matlab MAT file, with (at least) two fields: Intensity and Error

    ``.npz``
        Numpy zip file, with (at least) two fields: Intensity and Error

    other
        the file is opened with ``np.loadtxt``. The error matrix is tried
        to be loaded from the file ``<name>_error<ext>`` where the intensity was
        loaded from file ``<name><ext>``. I.e. if ``somedir/matrix.dat`` is given,
        the existence of ``somedir/matrix_error.dat`` is checked. If not found,
        None is returned for the error matrix.

    Notes
    -----
    The non-existence of the Intensity matrix results in an exception. If the
    Error matrix does not exist, None is returned for it.
    """
    # the core of read2dintfile
    if filename.upper().endswith('.MAT'):  # Matlab
        m = scipy.io.loadmat(filename)
    elif filename.upper().endswith('.NPZ'):  # Numpy
        m = np.load(filename)
    else:  # loadtxt
        m = {'Intensity': np.loadtxt(filename)}
        name, ext = os.path.splitext(filename)
        errorfilename = name + '_error' + ext
        if os.path.exists(errorfilename):
            m['Error'] = np.loadtxt(errorfilename)
    Intensity = m['Intensity']
    try:
        Error = m['Error']
        return Intensity, Error
    except:
        return Intensity, None


Instruction:
Can you implement a function in Python 3 that
saves the intensity and error matrices to a file.

Response:
def writeint2dnorm(filename, Intensity, Error=None):
    """Save the intensity and error matrices to a file

    Inputs
    ------
    filename: string
        the name of the file
    Intensity: np.ndarray
        the intensity matrix
    Error: np.ndarray, optional
        the error matrix (can be ``None``, if no error matrix is to be saved)

    Output
    ------
    None
    """
    whattosave = {'Intensity': Intensity}
    if Error is not None:
        whattosave['Error'] = Error
    if filename.upper().endswith('.NPZ'):
        np.savez(filename, **whattosave)
    elif filename.upper().endswith('.MAT'):
        scipy.io.savemat(filename, whattosave)
    else:  # text file
        np.savetxt(filename, Intensity)
        if Error is not None:
            name, ext = os.path.splitext(filename)
            np.savetxt(name + '_error' + ext, Error)


Instruction:
Create a Python 3 function to
try to load a maskfile from a matlab R matrix file.

Response:
def readmask(filename, fieldname=None):
    """Try to load a maskfile from a matlab(R) matrix file

    Inputs
    ------
    filename: string
        the input file name
    fieldname: string, optional
        field in the mat file. None to autodetect.

    Outputs
    -------
    the mask in a numpy array of type np.uint8
    """
    f = scipy.io.loadmat(filename)
    if fieldname is not None:
        return f[fieldname].astype(np.uint8)
    else:
        validkeys = [
            k for k in list(f.keys()) if not (k.startswith('_') and k.endswith('_'))]
        if len(validkeys) < 1:
            raise ValueError('mask file contains no masks!')
        if len(validkeys) > 1:
            raise ValueError('mask file contains multiple masks!')
        return f[validkeys[0]].astype(np.uint8)


Instruction:
Can you generate the documentation for the following Python 3 function
def readedf(filename):
    edf = header.readehf(filename)
    f = open(filename, 'rb')
    f.read(edf['EDF_HeaderSize'])  # skip header.
    if edf['DataType'] == 'FloatValue':
        dtype = np.float32
    else:
        raise NotImplementedError(
            'Not supported data type: %s' % edf['DataType'])
    edf['data'] = np.fromstring(f.read(edf['EDF_BinarySize']), dtype).reshape(
        edf['Dim_1'], edf['Dim_2'])
    return edf

Response: Read an ESRF file and return a dict with the data as a numpy array.


Instruction:
Given the following Python 3 function, write the documentation
def readbdfv2(filename, bdfext='.bdf', bhfext='.bhf'):
    datas = header.readbhfv2(filename, True, bdfext, bhfext)
    return datas

Response: Read a version 2 Bessy Data File


Instruction:
Can you write a function in Python 3 where it
reads a two - dimensional scattering pattern from a MarResearch. image file.

Response:
def readmar(filename):
    """Read a two-dimensional scattering pattern from a MarResearch .image file.
    """
    hed = header.readmarheader(filename)
    with open(filename, 'rb') as f:
        h = f.read(hed['recordlength'])
        data = np.fromstring(
            f.read(2 * hed['Xsize'] * hed['Ysize']), '<u2').astype(np.float64)
        if hed['highintensitypixels'] > 0:
            raise NotImplementedError(
                'Intensities over 65535 are not yet supported!')
        data = data.reshape(hed['Xsize'], hed['Ysize'])
    return data, hed


Instruction:
Create a Python 3 function to
write a version 2 BDF file with the given data.

Response:
def writebdfv2(filename, bdf, bdfext='.bdf', bhfext='.bhf'):
    """Write a version 2 Bessy Data File

    Inputs
    ------
    filename: string
        the name of the output file. One can give the complete header or
        datafile name or just the base name without the extensions.
    bdf: dict
        the BDF structure (in the same format as loaded by ``readbdfv2()``
    bdfext: string, optional
        the extension of the data file
    bhfext: string, optional
        the extension of the header file

    Output
    ------
    None

    Notes
    -----
    BDFv2 header and scattering data are stored separately in the header and
    the data files. Given the file name both are saved.
    """
    if filename.endswith(bdfext):
        basename = filename[:-len(bdfext)]
    elif filename.endswith(bhfext):
        basename = filename[:-len(bhfext)]
    else:
        basename = filename
    header.writebhfv2(basename + '.bhf', bdf)
    f = open(basename + '.bdf', 'wb')
    keys = ['RAWDATA', 'RAWERROR', 'CORRDATA', 'CORRERROR', 'NANDATA']
    keys.extend(
        [x for x in list(bdf.keys()) if isinstance(bdf[x], np.ndarray) and x not in keys])
    for k in keys:
        if k not in list(bdf.keys()):
            continue
        f.write('#%s[%d:%d]\n' % (k, bdf['xdim'], bdf['ydim']))
        f.write(np.rot90(bdf[k], 3).astype('float32').tostring(order='F'))
    f.close()


Instruction:
Explain what the following Python 3 code does
def rebinmask(mask, binx, biny, enlarge=False):
    if not enlarge and ((mask.shape[0] % binx) or (mask.shape[1] % biny)):
        raise ValueError(
            'The number of pixels of the mask matrix should be divisible by the binning in each direction!')
    if enlarge:
        return mask.repeat(binx, axis=0).repeat(biny, axis=1)
    else:
        return mask[::binx, ::biny]

Response: Re - bin a mask matrix.


Instruction:
How would you explain what the following Python 3 function does
def fill_padding(padded_string):
    # type: (bytes) -> bytes
    """
    Fill up missing padding in a string.

    This function makes sure that the string has length which is multiplication of 4,
    and if not, fills the missing places with dots.

    :param str padded_string: string to be decoded that might miss padding dots.
    :return: properly padded string
    :rtype: str
    """
    length = len(padded_string)
    reminder = len(padded_string) % 4
    if reminder:
        return padded_string.ljust(length + 4 - reminder, b'.')
    return padded_string

Response: Fills up missing padding in a string.


Instruction:
Write a Python 3 function for
decoding a querystringsafe base64 string or a regular base64 string.

Response:
def decode(encoded):
    # type: (bytes) -> bytes
    """
    Decode the result of querystringsafe_base64_encode or a regular base64.

    .. note ::
        As a regular base64 string does not contain dots, replacing dots with
        equal signs does basically noting to it. Also,
        base64.urlsafe_b64decode allows to decode both safe and unsafe base64.
        Therefore this function may also be used to decode the regular base64.

    :param (str, unicode) encoded: querystringsafe_base64 string or unicode
    :rtype: str, bytes
    :return: decoded string
    """
    padded_string = fill_padding(encoded)
    return urlsafe_b64decode(padded_string.replace(b'.', b'='))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def normalize_listargument(arg):
    if isinstance(arg, np.ndarray):
        return arg.flatten()
    if isinstance(arg, str):
        return [arg]
    if isinstance(arg, list) or isinstance(arg, tuple) or isinstance(arg, dict) or isinstance(arg, set):
        return list(arg)
    return [arg]

Response:
Check if arg is iterable and if not make a list of it. Numpy arrays are flattened and
        converted to lists.


Instruction:
How would you implement a function in Python 3 that
tries to auto - detect the numeric type of the value.

Response:
def parse_number(val, use_dateutilparser=False):
    """Try to auto-detect the numeric type of the value. First a conversion to
    int is tried. If this fails float is tried, and if that fails too, unicode()
    is executed. If this also fails, a ValueError is raised.
    """
    if use_dateutilparser:
        funcs = [int, float, parse_list_from_string,
                 dateutil.parser.parse, str]
    else:
        funcs = [int, float, parse_list_from_string, str]
    if (val.strip().startswith("'") and val.strip().endswith("'")) or (val.strip().startswith('"') and val.strip().endswith('"')):
        return val[1:-1]
    for f in funcs:
        try:
            return f(val)
        # eat exception
        except (ValueError, UnicodeEncodeError, UnicodeDecodeError) as ve:
            pass
    raise ValueError('Cannot parse number:', val)