PythonSTB
/

pypi312

@@ -1,402 +0,0 @@
-================================================================================
-  PYARROW - USER GUIDE (Android Python STB) - Generated by RIMI
-================================================================================
-  Covers: what pyarrow is, install/verify, arrays, tables, types, IPC,
-          CSV, JSON, Feather, compute, filesystem, and Android-specific notes.
-  Written for: Python 3.12 (RIMI build) on Android
-  Version:     pyarrow 25.0.1 (Arrow C++ 25.0.1)
-  Scripts dir: /storage/emulated/0/PythonSTB/Scripts/
-  Installed:   /data/user/0/com.pythonstb.rimi/files/python/lib/python3.12/site-packages/
-================================================================================
---------------------------------------------------------------------------------
-1) WHAT IS PYARROW?
---------------------------------------------------------------------------------
-PyArrow is the Python bindings for Apache Arrow — a cross-language development
-platform for in-memory columnar data. It provides:
-  - Apache Arrow arrays and tables (columnar memory format)
-  - Zero-copy data interchange between Python, pandas, and other languages
-  - IPC (Inter-Process Communication) for fast serialization
-  - CSV, JSON, and Feather file readers/writers
-  - Compute kernels (sum, mean, cast, filter, sort, etc.)
-  - A local filesystem abstraction
-Arrow is the foundation behind many modern data tools:
-  pandas 2.x uses Arrow as its default backend
-  Polars, DuckDB, and Spark all speak Arrow natively
-    import pyarrow as pa
-    print(pa.__version__)          # 25.0.1
-    print(pa.cpp_version)          # 25.0.1
---------------------------------------------------------------------------------
-2) INSTALL / VERIFY
---------------------------------------------------------------------------------
-Install:
-    pip install pyarrow-25.0.1-cp312-cp312-android_24_x86_64.whl
-        or:  pyarrow-25.0.1-cp312-cp312-android_24_arm64_v8a.whl
-Quick smoke test:
-    import pyarrow as pa
-    a = pa.array([1, 2, 3])
-    print(a)                       # [<pyarrow.Int64Scalar: 1>, ...]
-    print("version:", pa.__version__)
-    print("cpp:", pa.cpp_version)
-Expected output:
-    [1, 2, 3]
-    version: 25.0.1
-    cpp: 25.0.1
---------------------------------------------------------------------------------
-3) ARRAYS - CREATION BASICS
---------------------------------------------------------------------------------
-    import pyarrow as pa
-    # from Python lists
-    a = pa.array([1, 2, 3])                      # int64
-    b = pa.array([1.0, 2.5, 3.7])               # float64
-    c = pa.array(["hello", "world"])             # string
-    d = pa.array([True, False, True])            # bool
-    # explicit type
-    e = pa.array([1, 2, 3], type=pa.int32())
-    f = pa.array([1, 2, 3], type=pa.float64())
-    # with nulls
-    g = pa.array([1, None, 3], type=pa.int64())
-    g.null_count                # 1
-    # from numpy
-    import numpy as np
-    h = pa.array(np.arange(10))   # numpy -> Arrow (zero-copy for numeric types)
-    # important attributes
-    a.type          # DataType: int64
-    a.dtype         # same as .type
-    len(a)          # 3
-    a.to_pylist()   # [1, 2, 3]
-    a.as_py()       # [1, 2, 3] (same for scalar)
---------------------------------------------------------------------------------
-4) TYPES
---------------------------------------------------------------------------------
-    pa.int8()       pa.int16()      pa.int32()      pa.int64()
-    pa.uint8()      pa.uint16()     pa.uint32()     pa.uint64()
-    pa.float16()    pa.float32()    pa.float64()
-    pa.bool_()
-    pa.string()     pa.large_string()
-    pa.binary()     pa.large_binary()
-    pa.date32()     pa.date64()
-    pa.timestamp("ns")             # nanosecond timestamp
-    pa.duration("ms")              # millisecond duration
-    pa.null()                      # all-null type
-    pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.string())])
-    # check type
-    a = pa.array([1, 2])
-    a.type == pa.int64()           # True
-    isinstance(a.type, pa.DataType) # True
-    # convert between types
-    a = pa.array([1, 2, 3], type=pa.int64())
-    b = a.cast(pa.float64())       # explicit cast
-    c = a.cast(pa.int32())         # downcast
---------------------------------------------------------------------------------
-5) TABLES
---------------------------------------------------------------------------------
-    import pyarrow as pa
-    # from dict
-    t = pa.table({
-        "id":    [1, 2, 3],
-        "name":  ["alice", "bob", "charlie"],
-        "score": [95.5, 87.0, 92.3],
-    })
-    t.num_rows              # 3
-    t.num_columns           # 3
-    t.column_names          # ["id", "name", "score"]
-    t.schema                # id: int64, name: string, score: float64
-    # access columns
-    t.column("id")          # <pyarrow.lib.ChunkedArray ...>
-    t.column("id").to_pylist()  # [1, 2, 3]
-    # convert to pandas
-    df = t.to_pandas()
-    print(df)
-    #    id     name  score
-    # 0   1    alice   95.5
-    # 1   2      bob   87.0
-    # 2   3  charlie   92.3
-    # from pandas
-    t2 = pa.Table.from_pandas(df)
-    # slice
-    t.slice(0, 2)           # first 2 rows
-    # combine tables
-    t3 = pa.concat_tables([t, t])
-    # sort
-    t.sort_by("score", descending=True)
---------------------------------------------------------------------------------
-6) IPC (Inter-Process Communication)
---------------------------------------------------------------------------------
-IPC is Arrow's fast binary serialization format — much faster than CSV or JSON.
-    import pyarrow as pa
-    import pyarrow.ipc as ipc
-    t = pa.table({"x": [1, 2, 3], "y": ["a", "b", "c"]})
-    # --- File format (random access) ---
-    path = "/storage/emulated/0/Download/data.arrow"
-    writer = ipc.new_file(path, t.schema)
-    writer.write_table(t)
-    writer.close()
-    reader = ipc.open_file(path)
-    t2 = reader.read_all()
-    assert t.equals(t2)
-    # --- Stream format (sequential access, smaller header) ---
-    path2 = "/storage/emulated/0/Download/data_stream.arrow"
-    writer = ipc.new_stream(path2, t.schema)
-    writer.write_table(t)
-    writer.close()
-    reader = ipc.open_stream(path2)
-    t3 = reader.read_all()
-    assert t.equals(t3)
-    # --- In-memory IPC (for zero-copy between processes) ---
-    sink = ipc.BufferOutputStream()
-    writer = ipc.new_stream(sink, t.schema)
-    writer.write_table(t)
-    writer.close()
-    buf = sink.getvalue().to_pybytes()   # bytes
-    # send buf over socket / pipe, then:
-    reader = ipc.open_stream(pa.BufferReader(buf))
-    t4 = reader.read_all()
---------------------------------------------------------------------------------
-7) CSV
---------------------------------------------------------------------------------
-    import pyarrow as pa
-    import pyarrow.csv as pcsv
-    t = pa.table({"a": [1, 2], "b": [3.0, 4.0]})
-    # write
-    pcsv.write_csv(t, "/storage/emulated/0/Download/data.csv")
-    # read (basic)
-    t2 = pcsv.read_csv("/storage/emulated/0/Download/data.csv")
-    # read with options
-    read_opts = pcsv.ReadOptions(column_names=["x", "y"])
-    convert_opts = pcsv.ConvertOptions(
-        column_types={"x": pa.int64(), "y": pa.float64()},
-        null_values=["NA", "NULL"],
-    )
-    t3 = pcsv.read_csv(
-        "/storage/emulated/0/Download/data.csv",
-        read_options=read_opts,
-        convert_options=convert_opts,
-    )
-    # read as pandas directly
-    df = pcsv.read_csv("/storage/emulated/0/Download/data.csv").to_pandas()
---------------------------------------------------------------------------------
-8) JSON
---------------------------------------------------------------------------------
-    import pyarrow as pa
-    import pyarrow.json as pjson
-    # JSON Lines (one JSON object per line) — preferred for tabular data
-    # file content:
-    # {"a": 1, "b": "hello"}
-    # {"a": 2, "b": "world"}
-    t = pjson.read_json("/storage/emulated/0/Download/data.json")
-    # t: 2 rows, columns ["a", "b"]
-    # types inferred automatically (a: int64, b: string)
---------------------------------------------------------------------------------
-9) FEATHER (fast columnar format)
---------------------------------------------------------------------------------
-Feather is Arrow's columnar format optimized for pandas read/write.
-Faster than CSV, smaller than IPC for single-table files.
-    import pyarrow as pa
-    import pyarrow.feather as pf
-    t = pa.table({"x": [1, 2, 3], "y": [4.0, 5.0, 6.0]})
-    # write
-    pf.write_feather(t, "/storage/emulated/0/Download/data.feather")
-    # read
-    t2 = pf.read_feather("/storage/emulated/0/Download/data.feather")
-    assert t.equals(t2)
-    # read as pandas
-    df = pf.read_feather("/storage/emulated/0/Download/data.feather")
-    # older Feather v1 format also supported (pandas read_feather兼容)
---------------------------------------------------------------------------------
-10) COMPUTE FUNCTIONS
---------------------------------------------------------------------------------
-    import pyarrow as pa
-    import pyarrow.compute as pc
-    a = pa.array([1, 2, 3, 4, 5])
-    # aggregations
-    pc.sum(a)            # 15
-    pc.mean(a)           # 3.0
-    pc.min(a)            # 1
-    pc.max(a)            # 5
-    pc.count(a)          # 5
-    # element-wise
-    pc.add(a, pa.array([10, 10, 10, 10, 10]))
-    pc.multiply(a, pa.scalar(2))
-    pc.sqrt(a.cast(pa.float64()))
-    pc.abs(pa.array([-1, 2, -3]))
-    # comparison / filtering
-    mask = pc.greater(a, 3)               # [False, False, False, True, True]
-    pc.filter(a, mask)                    # [4, 5]
-    pc.sum(mask.as_py()) if hasattr(mask, 'as_py') else pc.sum(mask)  # 2
-    # casting
-    b = pc.cast(a, to=pa.float64())
-    c = pc.cast(pa.array(["1", "2", "3"]), to=pa.int64())
-    # sort
-    pc.sort(pa.array([3, 1, 4, 1, 5]))   # [1, 1, 3, 4, 5]
-    # unique / value_counts
-    pc.unique(pa.array([1, 2, 1, 3, 2]))  # [1, 2, 3]
-    pc.value_counts(pa.array([1, 1, 2]))  # [{values: [1,2], counts: [2,1]}]
-    # string ops (via compute)
-    names = pa.array(["alice", "BOB", "charlie"])
-    pc.upper(names)                        # ["ALICE", "BOB", "CHARLIE"]
-    pc.utf8_length(names)                  # [5, 3, 7]
-    pc.utf8_lower(names)                   # ["alice", "bob", "charlie"]
---------------------------------------------------------------------------------
-11) FILESYSTEM
---------------------------------------------------------------------------------
-    import pyarrow.fs as pfs
-    # local filesystem
-    local = pfs.LocalFileSystem()
-    # list directory
-    local.get_file_info(pfs.FileSelector("/storage/emulated/0/Download"))
-    # file info
-    meta = local.get_file_info("/storage/emulated/0/Download/data.csv")
-    meta.type    # FileType.File
-    meta.size    # bytes
-    meta.mtime   # modification time
-    # read / write
-    with local.open_output_stream("/storage/emulated/0/Download/test.txt") as f:
-        f.write(b"hello arrow")
-    with local.open_input_stream("/storage/emulated/0/Download/test.txt") as f:
-        content = f.read()
-    # content == b"hello arrow"
-    # create / delete
-    local.create_dir("/storage/emulated/0/Download/testdir")
-    local.delete_dir("/storage/emulated/0/Download/testdir")
---------------------------------------------------------------------------------
-12) ANDROID-SPECIFIC NOTES
---------------------------------------------------------------------------------
-  - libarrow_python.so preload:
-    The __init__.py is patched to call ctypes.CDLL("libarrow_python.so") before
-    importing any Cython modules. This is required because Android's linker
-    cannot find shared libraries the same way as Linux desktop. The preload
-    ensures symbols from libarrow_python.so are available when the .so extension
-    modules load.
-  - Static Arrow C++:
-    Arrow C++ (libarrow.a, libarrow_compute.a) is linked statically into
-    pyarrow's .so files. Only libarrow_python.so is a separate shared library
-    that needs preloading.
-  - Dependencies:
-    Only numpy is required. Install numpy first before using pyarrow.
-  - Wheels are tagged android_24_arm64_v8a and android_24_x86_64.
-    Install the wheel matching your device ABI:
-      arm64 phone -> android_24_arm64_v8a
-      x86_64 emulator -> android_24_x86_64
-  - Temp files for IPC/CSV/Feather:
-    Write to /storage/emulated/0/Download/ or use tempfile.gettempdir().
---------------------------------------------------------------------------------
-13) TESTED FEATURES
---------------------------------------------------------------------------------
-  v  import pyarrow 25.0.1
-  v  import numpy (dependency)
-  v  import submodules: compute, csv, json, feather, fs, ipc
-  v  array creation: int64, float64, string
-  v  null handling
-  v  type system: int64, float64, string, bool
-  v  table creation, schema inspection
-  v  table column access + to_pandas
-  v  IPC file round-trip
-  v  IPC stream round-trip
-  v  CSV write / read
-  v  CSV custom options (column_names, column_types)
-  v  JSON read
-  v  Feather round-trip
-  v  compute: sum, cast, filter, add, multiply
-  v  LocalFileSystem get_file_info
-================================================================================
-  Generated by RIMI
-================================================================================