Instructions to use giancap61/AXEL-AI_Virtual_Race_Engineer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M # Run inference directly in the terminal: llama cli -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M # Run inference directly in the terminal: llama cli -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Use Docker
docker model run hf.co/giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with Ollama:
ollama run hf.co/giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
- Unsloth Desktop
- Pi
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with Docker Model Runner:
docker model run hf.co/giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
- Lemonade
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Run and chat with the model
lemonade run user.AXEL-AI_Virtual_Race_Engineer-Q5_K_M
List all available models
lemonade list
- Hermes Agent
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use giancap61/AXEL-AI_Virtual_Race_Engineer with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "giancap61/AXEL-AI_Virtual_Race_Engineer:Q5_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Add files using upload-large-folder tool
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/INSTALLER +1 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/METADATA +84 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/RECORD +57 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/WHEEL +4 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/entry_points.txt +3 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/__init__.py +38 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/_identifier.py +6 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/ext.py +870 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/filters.py +1873 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/loaders.py +693 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/meta.py +112 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/nativetypes.py +130 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/nodes.py +1206 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/optimizer.py +48 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/parser.py +1049 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/py.typed +0 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/runtime.py +1062 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/sandbox.py +436 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/tests.py +256 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/utils.py +766 -0
- TTS/Kokoro/.venv/Lib/site-packages/jinja2/visitor.py +92 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/INSTALLER +1 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/METADATA +172 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/RECORD +217 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/WHEEL +5 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/top_level.txt +1 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/__init__.py +163 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_cloudpickle_wrapper.py +18 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_dask.py +381 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_memmapping_reducer.py +715 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_multiprocessing_helpers.py +51 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_parallel_backends.py +753 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_store_backends.py +500 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/_utils.py +120 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/backports.py +195 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/compressor.py +572 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/disk.py +131 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/executor.py +131 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/func_inspect.py +379 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/hashing.py +270 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/logger.py +159 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/memory.py +1242 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle.py +756 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_compat.py +250 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_utils.py +291 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/parallel.py +2075 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/pool.py +362 -0
- TTS/Kokoro/.venv/Lib/site-packages/joblib/testing.py +96 -0
- TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/INSTALLER +1 -0
- TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/METADATA +170 -0
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/INSTALLER
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
pip
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/METADATA
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Metadata-Version: 2.4
|
| 2 |
+
Name: Jinja2
|
| 3 |
+
Version: 3.1.6
|
| 4 |
+
Summary: A very fast and expressive template engine.
|
| 5 |
+
Maintainer-email: Pallets <contact@palletsprojects.com>
|
| 6 |
+
Requires-Python: >=3.7
|
| 7 |
+
Description-Content-Type: text/markdown
|
| 8 |
+
Classifier: Development Status :: 5 - Production/Stable
|
| 9 |
+
Classifier: Environment :: Web Environment
|
| 10 |
+
Classifier: Intended Audience :: Developers
|
| 11 |
+
Classifier: License :: OSI Approved :: BSD License
|
| 12 |
+
Classifier: Operating System :: OS Independent
|
| 13 |
+
Classifier: Programming Language :: Python
|
| 14 |
+
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
|
| 15 |
+
Classifier: Topic :: Text Processing :: Markup :: HTML
|
| 16 |
+
Classifier: Typing :: Typed
|
| 17 |
+
License-File: LICENSE.txt
|
| 18 |
+
Requires-Dist: MarkupSafe>=2.0
|
| 19 |
+
Requires-Dist: Babel>=2.7 ; extra == "i18n"
|
| 20 |
+
Project-URL: Changes, https://jinja.palletsprojects.com/changes/
|
| 21 |
+
Project-URL: Chat, https://discord.gg/pallets
|
| 22 |
+
Project-URL: Documentation, https://jinja.palletsprojects.com/
|
| 23 |
+
Project-URL: Donate, https://palletsprojects.com/donate
|
| 24 |
+
Project-URL: Source, https://github.com/pallets/jinja/
|
| 25 |
+
Provides-Extra: i18n
|
| 26 |
+
|
| 27 |
+
# Jinja
|
| 28 |
+
|
| 29 |
+
Jinja is a fast, expressive, extensible templating engine. Special
|
| 30 |
+
placeholders in the template allow writing code similar to Python
|
| 31 |
+
syntax. Then the template is passed data to render the final document.
|
| 32 |
+
|
| 33 |
+
It includes:
|
| 34 |
+
|
| 35 |
+
- Template inheritance and inclusion.
|
| 36 |
+
- Define and import macros within templates.
|
| 37 |
+
- HTML templates can use autoescaping to prevent XSS from untrusted
|
| 38 |
+
user input.
|
| 39 |
+
- A sandboxed environment can safely render untrusted templates.
|
| 40 |
+
- AsyncIO support for generating templates and calling async
|
| 41 |
+
functions.
|
| 42 |
+
- I18N support with Babel.
|
| 43 |
+
- Templates are compiled to optimized Python code just-in-time and
|
| 44 |
+
cached, or can be compiled ahead-of-time.
|
| 45 |
+
- Exceptions point to the correct line in templates to make debugging
|
| 46 |
+
easier.
|
| 47 |
+
- Extensible filters, tests, functions, and even syntax.
|
| 48 |
+
|
| 49 |
+
Jinja's philosophy is that while application logic belongs in Python if
|
| 50 |
+
possible, it shouldn't make the template designer's job difficult by
|
| 51 |
+
restricting functionality too much.
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
## In A Nutshell
|
| 55 |
+
|
| 56 |
+
```jinja
|
| 57 |
+
{% extends "base.html" %}
|
| 58 |
+
{% block title %}Members{% endblock %}
|
| 59 |
+
{% block content %}
|
| 60 |
+
<ul>
|
| 61 |
+
{% for user in users %}
|
| 62 |
+
<li><a href="{{ user.url }}">{{ user.username }}</a></li>
|
| 63 |
+
{% endfor %}
|
| 64 |
+
</ul>
|
| 65 |
+
{% endblock %}
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
## Donate
|
| 69 |
+
|
| 70 |
+
The Pallets organization develops and supports Jinja and other popular
|
| 71 |
+
packages. In order to grow the community of contributors and users, and
|
| 72 |
+
allow the maintainers to devote more time to the projects, [please
|
| 73 |
+
donate today][].
|
| 74 |
+
|
| 75 |
+
[please donate today]: https://palletsprojects.com/donate
|
| 76 |
+
|
| 77 |
+
## Contributing
|
| 78 |
+
|
| 79 |
+
See our [detailed contributing documentation][contrib] for many ways to
|
| 80 |
+
contribute, including reporting issues, requesting features, asking or answering
|
| 81 |
+
questions, and making PRs.
|
| 82 |
+
|
| 83 |
+
[contrib]: https://palletsprojects.com/contributing/
|
| 84 |
+
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/RECORD
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
jinja2-3.1.6.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
|
| 2 |
+
jinja2-3.1.6.dist-info/METADATA,sha256=aMVUj7Z8QTKhOJjZsx7FDGvqKr3ZFdkh8hQ1XDpkmcg,2871
|
| 3 |
+
jinja2-3.1.6.dist-info/RECORD,,
|
| 4 |
+
jinja2-3.1.6.dist-info/WHEEL,sha256=_2ozNFCLWc93bK4WKHCO-eDUENDlo-dgc9cU3qokYO4,82
|
| 5 |
+
jinja2-3.1.6.dist-info/entry_points.txt,sha256=OL85gYU1eD8cuPlikifFngXpeBjaxl6rIJ8KkC_3r-I,58
|
| 6 |
+
jinja2-3.1.6.dist-info/licenses/LICENSE.txt,sha256=O0nc7kEF6ze6wQ-vG-JgQI_oXSUrjp3y4JefweCUQ3s,1475
|
| 7 |
+
jinja2/__init__.py,sha256=xxepO9i7DHsqkQrgBEduLtfoz2QCuT6_gbL4XSN1hbU,1928
|
| 8 |
+
jinja2/__pycache__/__init__.cpython-312.pyc,,
|
| 9 |
+
jinja2/__pycache__/_identifier.cpython-312.pyc,,
|
| 10 |
+
jinja2/__pycache__/async_utils.cpython-312.pyc,,
|
| 11 |
+
jinja2/__pycache__/bccache.cpython-312.pyc,,
|
| 12 |
+
jinja2/__pycache__/compiler.cpython-312.pyc,,
|
| 13 |
+
jinja2/__pycache__/constants.cpython-312.pyc,,
|
| 14 |
+
jinja2/__pycache__/debug.cpython-312.pyc,,
|
| 15 |
+
jinja2/__pycache__/defaults.cpython-312.pyc,,
|
| 16 |
+
jinja2/__pycache__/environment.cpython-312.pyc,,
|
| 17 |
+
jinja2/__pycache__/exceptions.cpython-312.pyc,,
|
| 18 |
+
jinja2/__pycache__/ext.cpython-312.pyc,,
|
| 19 |
+
jinja2/__pycache__/filters.cpython-312.pyc,,
|
| 20 |
+
jinja2/__pycache__/idtracking.cpython-312.pyc,,
|
| 21 |
+
jinja2/__pycache__/lexer.cpython-312.pyc,,
|
| 22 |
+
jinja2/__pycache__/loaders.cpython-312.pyc,,
|
| 23 |
+
jinja2/__pycache__/meta.cpython-312.pyc,,
|
| 24 |
+
jinja2/__pycache__/nativetypes.cpython-312.pyc,,
|
| 25 |
+
jinja2/__pycache__/nodes.cpython-312.pyc,,
|
| 26 |
+
jinja2/__pycache__/optimizer.cpython-312.pyc,,
|
| 27 |
+
jinja2/__pycache__/parser.cpython-312.pyc,,
|
| 28 |
+
jinja2/__pycache__/runtime.cpython-312.pyc,,
|
| 29 |
+
jinja2/__pycache__/sandbox.cpython-312.pyc,,
|
| 30 |
+
jinja2/__pycache__/tests.cpython-312.pyc,,
|
| 31 |
+
jinja2/__pycache__/utils.cpython-312.pyc,,
|
| 32 |
+
jinja2/__pycache__/visitor.cpython-312.pyc,,
|
| 33 |
+
jinja2/_identifier.py,sha256=_zYctNKzRqlk_murTNlzrju1FFJL7Va_Ijqqd7ii2lU,1958
|
| 34 |
+
jinja2/async_utils.py,sha256=vK-PdsuorOMnWSnEkT3iUJRIkTnYgO2T6MnGxDgHI5o,2834
|
| 35 |
+
jinja2/bccache.py,sha256=gh0qs9rulnXo0PhX5jTJy2UHzI8wFnQ63o_vw7nhzRg,14061
|
| 36 |
+
jinja2/compiler.py,sha256=9RpCQl5X88BHllJiPsHPh295Hh0uApvwFJNQuutULeM,74131
|
| 37 |
+
jinja2/constants.py,sha256=GMoFydBF_kdpaRKPoM5cl5MviquVRLVyZtfp5-16jg0,1433
|
| 38 |
+
jinja2/debug.py,sha256=CnHqCDHd-BVGvti_8ZsTolnXNhA3ECsY-6n_2pwU8Hw,6297
|
| 39 |
+
jinja2/defaults.py,sha256=boBcSw78h-lp20YbaXSJsqkAI2uN_mD_TtCydpeq5wU,1267
|
| 40 |
+
jinja2/environment.py,sha256=9nhrP7Ch-NbGX00wvyr4yy-uhNHq2OCc60ggGrni_fk,61513
|
| 41 |
+
jinja2/exceptions.py,sha256=ioHeHrWwCWNaXX1inHmHVblvc4haO7AXsjCp3GfWvx0,5071
|
| 42 |
+
jinja2/ext.py,sha256=5PF5eHfh8mXAIxXHHRB2xXbXohi8pE3nHSOxa66uS7E,31875
|
| 43 |
+
jinja2/filters.py,sha256=PQ_Egd9n9jSgtnGQYyF4K5j2nYwhUIulhPnyimkdr-k,55212
|
| 44 |
+
jinja2/idtracking.py,sha256=-ll5lIp73pML3ErUYiIJj7tdmWxcH_IlDv3yA_hiZYo,10555
|
| 45 |
+
jinja2/lexer.py,sha256=LYiYio6br-Tep9nPcupWXsPEtjluw3p1mU-lNBVRUfk,29786
|
| 46 |
+
jinja2/loaders.py,sha256=wIrnxjvcbqh5VwW28NSkfotiDq8qNCxIOSFbGUiSLB4,24055
|
| 47 |
+
jinja2/meta.py,sha256=OTDPkaFvU2Hgvx-6akz7154F8BIWaRmvJcBFvwopHww,4397
|
| 48 |
+
jinja2/nativetypes.py,sha256=7GIGALVJgdyL80oZJdQUaUfwSt5q2lSSZbXt0dNf_M4,4210
|
| 49 |
+
jinja2/nodes.py,sha256=m1Duzcr6qhZI8JQ6VyJgUNinjAf5bQzijSmDnMsvUx8,34579
|
| 50 |
+
jinja2/optimizer.py,sha256=rJnCRlQ7pZsEEmMhsQDgC_pKyDHxP5TPS6zVPGsgcu8,1651
|
| 51 |
+
jinja2/parser.py,sha256=lLOFy3sEmHc5IaEHRiH1sQVnId2moUQzhyeJZTtdY30,40383
|
| 52 |
+
jinja2/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
| 53 |
+
jinja2/runtime.py,sha256=gDk-GvdriJXqgsGbHgrcKTP0Yp6zPXzhzrIpCFH3jAU,34249
|
| 54 |
+
jinja2/sandbox.py,sha256=Mw2aitlY2I8la7FYhcX2YG9BtUYcLnD0Gh3d29cDWrY,15009
|
| 55 |
+
jinja2/tests.py,sha256=VLsBhVFnWg-PxSBz1MhRnNWgP1ovXk3neO1FLQMeC9Q,5926
|
| 56 |
+
jinja2/utils.py,sha256=rRp3o9e7ZKS4fyrWRbELyLcpuGVTFcnooaOa1qx_FIk,24129
|
| 57 |
+
jinja2/visitor.py,sha256=EcnL1PIwf_4RVCOMxsRNuR8AXHbS1qfAdMOE2ngKJz4,3557
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/WHEEL
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Wheel-Version: 1.0
|
| 2 |
+
Generator: flit 3.11.0
|
| 3 |
+
Root-Is-Purelib: true
|
| 4 |
+
Tag: py3-none-any
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/entry_points.txt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[babel.extractors]
|
| 2 |
+
jinja2=jinja2.ext:babel_extract[i18n]
|
| 3 |
+
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/__init__.py
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Jinja is a template engine written in pure Python. It provides a
|
| 2 |
+
non-XML syntax that supports inline expressions and an optional
|
| 3 |
+
sandboxed environment.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
from .bccache import BytecodeCache as BytecodeCache
|
| 7 |
+
from .bccache import FileSystemBytecodeCache as FileSystemBytecodeCache
|
| 8 |
+
from .bccache import MemcachedBytecodeCache as MemcachedBytecodeCache
|
| 9 |
+
from .environment import Environment as Environment
|
| 10 |
+
from .environment import Template as Template
|
| 11 |
+
from .exceptions import TemplateAssertionError as TemplateAssertionError
|
| 12 |
+
from .exceptions import TemplateError as TemplateError
|
| 13 |
+
from .exceptions import TemplateNotFound as TemplateNotFound
|
| 14 |
+
from .exceptions import TemplateRuntimeError as TemplateRuntimeError
|
| 15 |
+
from .exceptions import TemplatesNotFound as TemplatesNotFound
|
| 16 |
+
from .exceptions import TemplateSyntaxError as TemplateSyntaxError
|
| 17 |
+
from .exceptions import UndefinedError as UndefinedError
|
| 18 |
+
from .loaders import BaseLoader as BaseLoader
|
| 19 |
+
from .loaders import ChoiceLoader as ChoiceLoader
|
| 20 |
+
from .loaders import DictLoader as DictLoader
|
| 21 |
+
from .loaders import FileSystemLoader as FileSystemLoader
|
| 22 |
+
from .loaders import FunctionLoader as FunctionLoader
|
| 23 |
+
from .loaders import ModuleLoader as ModuleLoader
|
| 24 |
+
from .loaders import PackageLoader as PackageLoader
|
| 25 |
+
from .loaders import PrefixLoader as PrefixLoader
|
| 26 |
+
from .runtime import ChainableUndefined as ChainableUndefined
|
| 27 |
+
from .runtime import DebugUndefined as DebugUndefined
|
| 28 |
+
from .runtime import make_logging_undefined as make_logging_undefined
|
| 29 |
+
from .runtime import StrictUndefined as StrictUndefined
|
| 30 |
+
from .runtime import Undefined as Undefined
|
| 31 |
+
from .utils import clear_caches as clear_caches
|
| 32 |
+
from .utils import is_undefined as is_undefined
|
| 33 |
+
from .utils import pass_context as pass_context
|
| 34 |
+
from .utils import pass_environment as pass_environment
|
| 35 |
+
from .utils import pass_eval_context as pass_eval_context
|
| 36 |
+
from .utils import select_autoescape as select_autoescape
|
| 37 |
+
|
| 38 |
+
__version__ = "3.1.6"
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/_identifier.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import re
|
| 2 |
+
|
| 3 |
+
# generated by scripts/generate_identifier_pattern.py
|
| 4 |
+
pattern = re.compile(
|
| 5 |
+
r"[\w·̀-ͯ·҃-֑҇-ׇֽֿׁׂׅׄؐ-ًؚ-ٰٟۖ-ۜ۟-۪ۤۧۨ-ܑۭܰ-݊ަ-ް߫-߽߳ࠖ-࠙ࠛ-ࠣࠥ-ࠧࠩ-࡙࠭-࡛࣓-ࣣ࣡-ःऺ-़ा-ॏ॑-ॗॢॣঁ-ঃ়া-ৄেৈো-্ৗৢৣ৾ਁ-ਃ਼ਾ-ੂੇੈੋ-੍ੑੰੱੵઁ-ઃ઼ા-ૅે-ૉો-્ૢૣૺ-૿ଁ-ଃ଼ା-ୄେୈୋ-୍ୖୗୢୣஂா-ூெ-ைொ-்ௗఀ-ఄా-ౄె-ైొ-్ౕౖౢౣಁ-ಃ಼ಾ-ೄೆ-ೈೊ-್ೕೖೢೣഀ-ഃ഻഼ാ-ൄെ-ൈൊ-്ൗൢൣංඃ්ා-ුූෘ-ෟෲෳัิ-ฺ็-๎ັິ-ູົຼ່-ໍ༹༘༙༵༷༾༿ཱ-྄྆྇ྍ-ྗྙ-ྼ࿆ါ-ှၖ-ၙၞ-ၠၢ-ၤၧ-ၭၱ-ၴႂ-ႍႏႚ-ႝ፝-፟ᜒ-᜔ᜲ-᜴ᝒᝓᝲᝳ឴-៓៝᠋-᠍ᢅᢆᢩᤠ-ᤫᤰ-᤻ᨗ-ᨛᩕ-ᩞ᩠-᩿᩼᪰-᪽ᬀ-ᬄ᬴-᭄᭫-᭳ᮀ-ᮂᮡ-ᮭ᯦-᯳ᰤ-᰷᳐-᳔᳒-᳨᳭ᳲ-᳴᳷-᳹᷀-᷹᷻-᷿‿⁀⁔⃐-⃥⃜⃡-⃰℘℮⳯-⵿⳱ⷠ-〪ⷿ-゙゚〯꙯ꙴ-꙽ꚞꚟ꛰꛱ꠂ꠆ꠋꠣ-ꠧꢀꢁꢴ-ꣅ꣠-꣱ꣿꤦ-꤭ꥇ-꥓ꦀ-ꦃ꦳-꧀ꧥꨩ-ꨶꩃꩌꩍꩻ-ꩽꪰꪲ-ꪴꪷꪸꪾ꪿꫁ꫫ-ꫯꫵ꫶ꯣ-ꯪ꯬꯭ﬞ︀-️︠-︯︳︴﹍-﹏_𐇽𐋠𐍶-𐍺𐨁-𐨃𐨅𐨆𐨌-𐨏𐨸-𐨿𐨺𐫦𐫥𐴤-𐽆𐴧-𐽐𑀀-𑀂𑀸-𑁆𑁿-𑂂𑂰-𑂺𑄀-𑄂𑄧-𑄴𑅅𑅆𑅳𑆀-𑆂𑆳-𑇀𑇉-𑇌𑈬-𑈷𑈾𑋟-𑋪𑌀-𑌃𑌻𑌼𑌾-𑍄𑍇𑍈𑍋-𑍍𑍗𑍢𑍣𑍦-𑍬𑍰-𑍴𑐵-𑑆𑑞𑒰-𑓃𑖯-𑖵𑖸-𑗀𑗜𑗝𑘰-𑙀𑚫-𑚷𑜝-𑜫𑠬-𑠺𑨁-𑨊𑨳-𑨹𑨻-𑨾𑩇𑩑-𑩛𑪊-𑪙𑰯-𑰶𑰸-𑰿𑲒-𑲧𑲩-𑲶𑴱-𑴶𑴺𑴼𑴽𑴿-𑵅𑵇𑶊-𑶎𑶐𑶑𑶓-𑶗𑻳-𑻶𖫰-𖫴𖬰-𖬶𖽑-𖽾𖾏-𖾒𛲝𛲞𝅥-𝅩𝅭-𝅲𝅻-𝆂𝆅-𝆋𝆪-𝆭𝉂-𝉄𝨀-𝨶𝨻-𝩬𝩵𝪄𝪛-𝪟𝪡-𝪯𞀀-𞀆𞀈-𞀘𞀛-𞀡𞀣𞀤𞀦-𞣐𞀪-𞣖𞥄-𞥊󠄀-󠇯]+" # noqa: B950
|
| 6 |
+
)
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/ext.py
ADDED
|
@@ -0,0 +1,870 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Extension API for adding custom tags and behavior."""
|
| 2 |
+
|
| 3 |
+
import pprint
|
| 4 |
+
import re
|
| 5 |
+
import typing as t
|
| 6 |
+
|
| 7 |
+
from markupsafe import Markup
|
| 8 |
+
|
| 9 |
+
from . import defaults
|
| 10 |
+
from . import nodes
|
| 11 |
+
from .environment import Environment
|
| 12 |
+
from .exceptions import TemplateAssertionError
|
| 13 |
+
from .exceptions import TemplateSyntaxError
|
| 14 |
+
from .runtime import concat # type: ignore
|
| 15 |
+
from .runtime import Context
|
| 16 |
+
from .runtime import Undefined
|
| 17 |
+
from .utils import import_string
|
| 18 |
+
from .utils import pass_context
|
| 19 |
+
|
| 20 |
+
if t.TYPE_CHECKING:
|
| 21 |
+
import typing_extensions as te
|
| 22 |
+
|
| 23 |
+
from .lexer import Token
|
| 24 |
+
from .lexer import TokenStream
|
| 25 |
+
from .parser import Parser
|
| 26 |
+
|
| 27 |
+
class _TranslationsBasic(te.Protocol):
|
| 28 |
+
def gettext(self, message: str) -> str: ...
|
| 29 |
+
|
| 30 |
+
def ngettext(self, singular: str, plural: str, n: int) -> str:
|
| 31 |
+
pass
|
| 32 |
+
|
| 33 |
+
class _TranslationsContext(_TranslationsBasic):
|
| 34 |
+
def pgettext(self, context: str, message: str) -> str: ...
|
| 35 |
+
|
| 36 |
+
def npgettext(
|
| 37 |
+
self, context: str, singular: str, plural: str, n: int
|
| 38 |
+
) -> str: ...
|
| 39 |
+
|
| 40 |
+
_SupportedTranslations = t.Union[_TranslationsBasic, _TranslationsContext]
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
# I18N functions available in Jinja templates. If the I18N library
|
| 44 |
+
# provides ugettext, it will be assigned to gettext.
|
| 45 |
+
GETTEXT_FUNCTIONS: t.Tuple[str, ...] = (
|
| 46 |
+
"_",
|
| 47 |
+
"gettext",
|
| 48 |
+
"ngettext",
|
| 49 |
+
"pgettext",
|
| 50 |
+
"npgettext",
|
| 51 |
+
)
|
| 52 |
+
_ws_re = re.compile(r"\s*\n\s*")
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
class Extension:
|
| 56 |
+
"""Extensions can be used to add extra functionality to the Jinja template
|
| 57 |
+
system at the parser level. Custom extensions are bound to an environment
|
| 58 |
+
but may not store environment specific data on `self`. The reason for
|
| 59 |
+
this is that an extension can be bound to another environment (for
|
| 60 |
+
overlays) by creating a copy and reassigning the `environment` attribute.
|
| 61 |
+
|
| 62 |
+
As extensions are created by the environment they cannot accept any
|
| 63 |
+
arguments for configuration. One may want to work around that by using
|
| 64 |
+
a factory function, but that is not possible as extensions are identified
|
| 65 |
+
by their import name. The correct way to configure the extension is
|
| 66 |
+
storing the configuration values on the environment. Because this way the
|
| 67 |
+
environment ends up acting as central configuration storage the
|
| 68 |
+
attributes may clash which is why extensions have to ensure that the names
|
| 69 |
+
they choose for configuration are not too generic. ``prefix`` for example
|
| 70 |
+
is a terrible name, ``fragment_cache_prefix`` on the other hand is a good
|
| 71 |
+
name as includes the name of the extension (fragment cache).
|
| 72 |
+
"""
|
| 73 |
+
|
| 74 |
+
identifier: t.ClassVar[str]
|
| 75 |
+
|
| 76 |
+
def __init_subclass__(cls) -> None:
|
| 77 |
+
cls.identifier = f"{cls.__module__}.{cls.__name__}"
|
| 78 |
+
|
| 79 |
+
#: if this extension parses this is the list of tags it's listening to.
|
| 80 |
+
tags: t.Set[str] = set()
|
| 81 |
+
|
| 82 |
+
#: the priority of that extension. This is especially useful for
|
| 83 |
+
#: extensions that preprocess values. A lower value means higher
|
| 84 |
+
#: priority.
|
| 85 |
+
#:
|
| 86 |
+
#: .. versionadded:: 2.4
|
| 87 |
+
priority = 100
|
| 88 |
+
|
| 89 |
+
def __init__(self, environment: Environment) -> None:
|
| 90 |
+
self.environment = environment
|
| 91 |
+
|
| 92 |
+
def bind(self, environment: Environment) -> "te.Self":
|
| 93 |
+
"""Create a copy of this extension bound to another environment."""
|
| 94 |
+
rv = object.__new__(self.__class__)
|
| 95 |
+
rv.__dict__.update(self.__dict__)
|
| 96 |
+
rv.environment = environment
|
| 97 |
+
return rv
|
| 98 |
+
|
| 99 |
+
def preprocess(
|
| 100 |
+
self, source: str, name: t.Optional[str], filename: t.Optional[str] = None
|
| 101 |
+
) -> str:
|
| 102 |
+
"""This method is called before the actual lexing and can be used to
|
| 103 |
+
preprocess the source. The `filename` is optional. The return value
|
| 104 |
+
must be the preprocessed source.
|
| 105 |
+
"""
|
| 106 |
+
return source
|
| 107 |
+
|
| 108 |
+
def filter_stream(
|
| 109 |
+
self, stream: "TokenStream"
|
| 110 |
+
) -> t.Union["TokenStream", t.Iterable["Token"]]:
|
| 111 |
+
"""It's passed a :class:`~jinja2.lexer.TokenStream` that can be used
|
| 112 |
+
to filter tokens returned. This method has to return an iterable of
|
| 113 |
+
:class:`~jinja2.lexer.Token`\\s, but it doesn't have to return a
|
| 114 |
+
:class:`~jinja2.lexer.TokenStream`.
|
| 115 |
+
"""
|
| 116 |
+
return stream
|
| 117 |
+
|
| 118 |
+
def parse(self, parser: "Parser") -> t.Union[nodes.Node, t.List[nodes.Node]]:
|
| 119 |
+
"""If any of the :attr:`tags` matched this method is called with the
|
| 120 |
+
parser as first argument. The token the parser stream is pointing at
|
| 121 |
+
is the name token that matched. This method has to return one or a
|
| 122 |
+
list of multiple nodes.
|
| 123 |
+
"""
|
| 124 |
+
raise NotImplementedError()
|
| 125 |
+
|
| 126 |
+
def attr(
|
| 127 |
+
self, name: str, lineno: t.Optional[int] = None
|
| 128 |
+
) -> nodes.ExtensionAttribute:
|
| 129 |
+
"""Return an attribute node for the current extension. This is useful
|
| 130 |
+
to pass constants on extensions to generated template code.
|
| 131 |
+
|
| 132 |
+
::
|
| 133 |
+
|
| 134 |
+
self.attr('_my_attribute', lineno=lineno)
|
| 135 |
+
"""
|
| 136 |
+
return nodes.ExtensionAttribute(self.identifier, name, lineno=lineno)
|
| 137 |
+
|
| 138 |
+
def call_method(
|
| 139 |
+
self,
|
| 140 |
+
name: str,
|
| 141 |
+
args: t.Optional[t.List[nodes.Expr]] = None,
|
| 142 |
+
kwargs: t.Optional[t.List[nodes.Keyword]] = None,
|
| 143 |
+
dyn_args: t.Optional[nodes.Expr] = None,
|
| 144 |
+
dyn_kwargs: t.Optional[nodes.Expr] = None,
|
| 145 |
+
lineno: t.Optional[int] = None,
|
| 146 |
+
) -> nodes.Call:
|
| 147 |
+
"""Call a method of the extension. This is a shortcut for
|
| 148 |
+
:meth:`attr` + :class:`jinja2.nodes.Call`.
|
| 149 |
+
"""
|
| 150 |
+
if args is None:
|
| 151 |
+
args = []
|
| 152 |
+
if kwargs is None:
|
| 153 |
+
kwargs = []
|
| 154 |
+
return nodes.Call(
|
| 155 |
+
self.attr(name, lineno=lineno),
|
| 156 |
+
args,
|
| 157 |
+
kwargs,
|
| 158 |
+
dyn_args,
|
| 159 |
+
dyn_kwargs,
|
| 160 |
+
lineno=lineno,
|
| 161 |
+
)
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
@pass_context
|
| 165 |
+
def _gettext_alias(
|
| 166 |
+
__context: Context, *args: t.Any, **kwargs: t.Any
|
| 167 |
+
) -> t.Union[t.Any, Undefined]:
|
| 168 |
+
return __context.call(__context.resolve("gettext"), *args, **kwargs)
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
def _make_new_gettext(func: t.Callable[[str], str]) -> t.Callable[..., str]:
|
| 172 |
+
@pass_context
|
| 173 |
+
def gettext(__context: Context, __string: str, **variables: t.Any) -> str:
|
| 174 |
+
rv = __context.call(func, __string)
|
| 175 |
+
if __context.eval_ctx.autoescape:
|
| 176 |
+
rv = Markup(rv)
|
| 177 |
+
# Always treat as a format string, even if there are no
|
| 178 |
+
# variables. This makes translation strings more consistent
|
| 179 |
+
# and predictable. This requires escaping
|
| 180 |
+
return rv % variables # type: ignore
|
| 181 |
+
|
| 182 |
+
return gettext
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
def _make_new_ngettext(func: t.Callable[[str, str, int], str]) -> t.Callable[..., str]:
|
| 186 |
+
@pass_context
|
| 187 |
+
def ngettext(
|
| 188 |
+
__context: Context,
|
| 189 |
+
__singular: str,
|
| 190 |
+
__plural: str,
|
| 191 |
+
__num: int,
|
| 192 |
+
**variables: t.Any,
|
| 193 |
+
) -> str:
|
| 194 |
+
variables.setdefault("num", __num)
|
| 195 |
+
rv = __context.call(func, __singular, __plural, __num)
|
| 196 |
+
if __context.eval_ctx.autoescape:
|
| 197 |
+
rv = Markup(rv)
|
| 198 |
+
# Always treat as a format string, see gettext comment above.
|
| 199 |
+
return rv % variables # type: ignore
|
| 200 |
+
|
| 201 |
+
return ngettext
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
def _make_new_pgettext(func: t.Callable[[str, str], str]) -> t.Callable[..., str]:
|
| 205 |
+
@pass_context
|
| 206 |
+
def pgettext(
|
| 207 |
+
__context: Context, __string_ctx: str, __string: str, **variables: t.Any
|
| 208 |
+
) -> str:
|
| 209 |
+
variables.setdefault("context", __string_ctx)
|
| 210 |
+
rv = __context.call(func, __string_ctx, __string)
|
| 211 |
+
|
| 212 |
+
if __context.eval_ctx.autoescape:
|
| 213 |
+
rv = Markup(rv)
|
| 214 |
+
|
| 215 |
+
# Always treat as a format string, see gettext comment above.
|
| 216 |
+
return rv % variables # type: ignore
|
| 217 |
+
|
| 218 |
+
return pgettext
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def _make_new_npgettext(
|
| 222 |
+
func: t.Callable[[str, str, str, int], str],
|
| 223 |
+
) -> t.Callable[..., str]:
|
| 224 |
+
@pass_context
|
| 225 |
+
def npgettext(
|
| 226 |
+
__context: Context,
|
| 227 |
+
__string_ctx: str,
|
| 228 |
+
__singular: str,
|
| 229 |
+
__plural: str,
|
| 230 |
+
__num: int,
|
| 231 |
+
**variables: t.Any,
|
| 232 |
+
) -> str:
|
| 233 |
+
variables.setdefault("context", __string_ctx)
|
| 234 |
+
variables.setdefault("num", __num)
|
| 235 |
+
rv = __context.call(func, __string_ctx, __singular, __plural, __num)
|
| 236 |
+
|
| 237 |
+
if __context.eval_ctx.autoescape:
|
| 238 |
+
rv = Markup(rv)
|
| 239 |
+
|
| 240 |
+
# Always treat as a format string, see gettext comment above.
|
| 241 |
+
return rv % variables # type: ignore
|
| 242 |
+
|
| 243 |
+
return npgettext
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
class InternationalizationExtension(Extension):
|
| 247 |
+
"""This extension adds gettext support to Jinja."""
|
| 248 |
+
|
| 249 |
+
tags = {"trans"}
|
| 250 |
+
|
| 251 |
+
# TODO: the i18n extension is currently reevaluating values in a few
|
| 252 |
+
# situations. Take this example:
|
| 253 |
+
# {% trans count=something() %}{{ count }} foo{% pluralize
|
| 254 |
+
# %}{{ count }} fooss{% endtrans %}
|
| 255 |
+
# something is called twice here. One time for the gettext value and
|
| 256 |
+
# the other time for the n-parameter of the ngettext function.
|
| 257 |
+
|
| 258 |
+
def __init__(self, environment: Environment) -> None:
|
| 259 |
+
super().__init__(environment)
|
| 260 |
+
environment.globals["_"] = _gettext_alias
|
| 261 |
+
environment.extend(
|
| 262 |
+
install_gettext_translations=self._install,
|
| 263 |
+
install_null_translations=self._install_null,
|
| 264 |
+
install_gettext_callables=self._install_callables,
|
| 265 |
+
uninstall_gettext_translations=self._uninstall,
|
| 266 |
+
extract_translations=self._extract,
|
| 267 |
+
newstyle_gettext=False,
|
| 268 |
+
)
|
| 269 |
+
|
| 270 |
+
def _install(
|
| 271 |
+
self, translations: "_SupportedTranslations", newstyle: t.Optional[bool] = None
|
| 272 |
+
) -> None:
|
| 273 |
+
# ugettext and ungettext are preferred in case the I18N library
|
| 274 |
+
# is providing compatibility with older Python versions.
|
| 275 |
+
gettext = getattr(translations, "ugettext", None)
|
| 276 |
+
if gettext is None:
|
| 277 |
+
gettext = translations.gettext
|
| 278 |
+
ngettext = getattr(translations, "ungettext", None)
|
| 279 |
+
if ngettext is None:
|
| 280 |
+
ngettext = translations.ngettext
|
| 281 |
+
|
| 282 |
+
pgettext = getattr(translations, "pgettext", None)
|
| 283 |
+
npgettext = getattr(translations, "npgettext", None)
|
| 284 |
+
self._install_callables(
|
| 285 |
+
gettext, ngettext, newstyle=newstyle, pgettext=pgettext, npgettext=npgettext
|
| 286 |
+
)
|
| 287 |
+
|
| 288 |
+
def _install_null(self, newstyle: t.Optional[bool] = None) -> None:
|
| 289 |
+
import gettext
|
| 290 |
+
|
| 291 |
+
translations = gettext.NullTranslations()
|
| 292 |
+
|
| 293 |
+
if hasattr(translations, "pgettext"):
|
| 294 |
+
# Python < 3.8
|
| 295 |
+
pgettext = translations.pgettext
|
| 296 |
+
else:
|
| 297 |
+
|
| 298 |
+
def pgettext(c: str, s: str) -> str: # type: ignore[misc]
|
| 299 |
+
return s
|
| 300 |
+
|
| 301 |
+
if hasattr(translations, "npgettext"):
|
| 302 |
+
npgettext = translations.npgettext
|
| 303 |
+
else:
|
| 304 |
+
|
| 305 |
+
def npgettext(c: str, s: str, p: str, n: int) -> str: # type: ignore[misc]
|
| 306 |
+
return s if n == 1 else p
|
| 307 |
+
|
| 308 |
+
self._install_callables(
|
| 309 |
+
gettext=translations.gettext,
|
| 310 |
+
ngettext=translations.ngettext,
|
| 311 |
+
newstyle=newstyle,
|
| 312 |
+
pgettext=pgettext,
|
| 313 |
+
npgettext=npgettext,
|
| 314 |
+
)
|
| 315 |
+
|
| 316 |
+
def _install_callables(
|
| 317 |
+
self,
|
| 318 |
+
gettext: t.Callable[[str], str],
|
| 319 |
+
ngettext: t.Callable[[str, str, int], str],
|
| 320 |
+
newstyle: t.Optional[bool] = None,
|
| 321 |
+
pgettext: t.Optional[t.Callable[[str, str], str]] = None,
|
| 322 |
+
npgettext: t.Optional[t.Callable[[str, str, str, int], str]] = None,
|
| 323 |
+
) -> None:
|
| 324 |
+
if newstyle is not None:
|
| 325 |
+
self.environment.newstyle_gettext = newstyle # type: ignore
|
| 326 |
+
if self.environment.newstyle_gettext: # type: ignore
|
| 327 |
+
gettext = _make_new_gettext(gettext)
|
| 328 |
+
ngettext = _make_new_ngettext(ngettext)
|
| 329 |
+
|
| 330 |
+
if pgettext is not None:
|
| 331 |
+
pgettext = _make_new_pgettext(pgettext)
|
| 332 |
+
|
| 333 |
+
if npgettext is not None:
|
| 334 |
+
npgettext = _make_new_npgettext(npgettext)
|
| 335 |
+
|
| 336 |
+
self.environment.globals.update(
|
| 337 |
+
gettext=gettext, ngettext=ngettext, pgettext=pgettext, npgettext=npgettext
|
| 338 |
+
)
|
| 339 |
+
|
| 340 |
+
def _uninstall(self, translations: "_SupportedTranslations") -> None:
|
| 341 |
+
for key in ("gettext", "ngettext", "pgettext", "npgettext"):
|
| 342 |
+
self.environment.globals.pop(key, None)
|
| 343 |
+
|
| 344 |
+
def _extract(
|
| 345 |
+
self,
|
| 346 |
+
source: t.Union[str, nodes.Template],
|
| 347 |
+
gettext_functions: t.Sequence[str] = GETTEXT_FUNCTIONS,
|
| 348 |
+
) -> t.Iterator[
|
| 349 |
+
t.Tuple[int, str, t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]]]
|
| 350 |
+
]:
|
| 351 |
+
if isinstance(source, str):
|
| 352 |
+
source = self.environment.parse(source)
|
| 353 |
+
return extract_from_ast(source, gettext_functions)
|
| 354 |
+
|
| 355 |
+
def parse(self, parser: "Parser") -> t.Union[nodes.Node, t.List[nodes.Node]]:
|
| 356 |
+
"""Parse a translatable tag."""
|
| 357 |
+
lineno = next(parser.stream).lineno
|
| 358 |
+
|
| 359 |
+
context = None
|
| 360 |
+
context_token = parser.stream.next_if("string")
|
| 361 |
+
|
| 362 |
+
if context_token is not None:
|
| 363 |
+
context = context_token.value
|
| 364 |
+
|
| 365 |
+
# find all the variables referenced. Additionally a variable can be
|
| 366 |
+
# defined in the body of the trans block too, but this is checked at
|
| 367 |
+
# a later state.
|
| 368 |
+
plural_expr: t.Optional[nodes.Expr] = None
|
| 369 |
+
plural_expr_assignment: t.Optional[nodes.Assign] = None
|
| 370 |
+
num_called_num = False
|
| 371 |
+
variables: t.Dict[str, nodes.Expr] = {}
|
| 372 |
+
trimmed = None
|
| 373 |
+
while parser.stream.current.type != "block_end":
|
| 374 |
+
if variables:
|
| 375 |
+
parser.stream.expect("comma")
|
| 376 |
+
|
| 377 |
+
# skip colon for python compatibility
|
| 378 |
+
if parser.stream.skip_if("colon"):
|
| 379 |
+
break
|
| 380 |
+
|
| 381 |
+
token = parser.stream.expect("name")
|
| 382 |
+
if token.value in variables:
|
| 383 |
+
parser.fail(
|
| 384 |
+
f"translatable variable {token.value!r} defined twice.",
|
| 385 |
+
token.lineno,
|
| 386 |
+
exc=TemplateAssertionError,
|
| 387 |
+
)
|
| 388 |
+
|
| 389 |
+
# expressions
|
| 390 |
+
if parser.stream.current.type == "assign":
|
| 391 |
+
next(parser.stream)
|
| 392 |
+
variables[token.value] = var = parser.parse_expression()
|
| 393 |
+
elif trimmed is None and token.value in ("trimmed", "notrimmed"):
|
| 394 |
+
trimmed = token.value == "trimmed"
|
| 395 |
+
continue
|
| 396 |
+
else:
|
| 397 |
+
variables[token.value] = var = nodes.Name(token.value, "load")
|
| 398 |
+
|
| 399 |
+
if plural_expr is None:
|
| 400 |
+
if isinstance(var, nodes.Call):
|
| 401 |
+
plural_expr = nodes.Name("_trans", "load")
|
| 402 |
+
variables[token.value] = plural_expr
|
| 403 |
+
plural_expr_assignment = nodes.Assign(
|
| 404 |
+
nodes.Name("_trans", "store"), var
|
| 405 |
+
)
|
| 406 |
+
else:
|
| 407 |
+
plural_expr = var
|
| 408 |
+
num_called_num = token.value == "num"
|
| 409 |
+
|
| 410 |
+
parser.stream.expect("block_end")
|
| 411 |
+
|
| 412 |
+
plural = None
|
| 413 |
+
have_plural = False
|
| 414 |
+
referenced = set()
|
| 415 |
+
|
| 416 |
+
# now parse until endtrans or pluralize
|
| 417 |
+
singular_names, singular = self._parse_block(parser, True)
|
| 418 |
+
if singular_names:
|
| 419 |
+
referenced.update(singular_names)
|
| 420 |
+
if plural_expr is None:
|
| 421 |
+
plural_expr = nodes.Name(singular_names[0], "load")
|
| 422 |
+
num_called_num = singular_names[0] == "num"
|
| 423 |
+
|
| 424 |
+
# if we have a pluralize block, we parse that too
|
| 425 |
+
if parser.stream.current.test("name:pluralize"):
|
| 426 |
+
have_plural = True
|
| 427 |
+
next(parser.stream)
|
| 428 |
+
if parser.stream.current.type != "block_end":
|
| 429 |
+
token = parser.stream.expect("name")
|
| 430 |
+
if token.value not in variables:
|
| 431 |
+
parser.fail(
|
| 432 |
+
f"unknown variable {token.value!r} for pluralization",
|
| 433 |
+
token.lineno,
|
| 434 |
+
exc=TemplateAssertionError,
|
| 435 |
+
)
|
| 436 |
+
plural_expr = variables[token.value]
|
| 437 |
+
num_called_num = token.value == "num"
|
| 438 |
+
parser.stream.expect("block_end")
|
| 439 |
+
plural_names, plural = self._parse_block(parser, False)
|
| 440 |
+
next(parser.stream)
|
| 441 |
+
referenced.update(plural_names)
|
| 442 |
+
else:
|
| 443 |
+
next(parser.stream)
|
| 444 |
+
|
| 445 |
+
# register free names as simple name expressions
|
| 446 |
+
for name in referenced:
|
| 447 |
+
if name not in variables:
|
| 448 |
+
variables[name] = nodes.Name(name, "load")
|
| 449 |
+
|
| 450 |
+
if not have_plural:
|
| 451 |
+
plural_expr = None
|
| 452 |
+
elif plural_expr is None:
|
| 453 |
+
parser.fail("pluralize without variables", lineno)
|
| 454 |
+
|
| 455 |
+
if trimmed is None:
|
| 456 |
+
trimmed = self.environment.policies["ext.i18n.trimmed"]
|
| 457 |
+
if trimmed:
|
| 458 |
+
singular = self._trim_whitespace(singular)
|
| 459 |
+
if plural:
|
| 460 |
+
plural = self._trim_whitespace(plural)
|
| 461 |
+
|
| 462 |
+
node = self._make_node(
|
| 463 |
+
singular,
|
| 464 |
+
plural,
|
| 465 |
+
context,
|
| 466 |
+
variables,
|
| 467 |
+
plural_expr,
|
| 468 |
+
bool(referenced),
|
| 469 |
+
num_called_num and have_plural,
|
| 470 |
+
)
|
| 471 |
+
node.set_lineno(lineno)
|
| 472 |
+
if plural_expr_assignment is not None:
|
| 473 |
+
return [plural_expr_assignment, node]
|
| 474 |
+
else:
|
| 475 |
+
return node
|
| 476 |
+
|
| 477 |
+
def _trim_whitespace(self, string: str, _ws_re: t.Pattern[str] = _ws_re) -> str:
|
| 478 |
+
return _ws_re.sub(" ", string.strip())
|
| 479 |
+
|
| 480 |
+
def _parse_block(
|
| 481 |
+
self, parser: "Parser", allow_pluralize: bool
|
| 482 |
+
) -> t.Tuple[t.List[str], str]:
|
| 483 |
+
"""Parse until the next block tag with a given name."""
|
| 484 |
+
referenced = []
|
| 485 |
+
buf = []
|
| 486 |
+
|
| 487 |
+
while True:
|
| 488 |
+
if parser.stream.current.type == "data":
|
| 489 |
+
buf.append(parser.stream.current.value.replace("%", "%%"))
|
| 490 |
+
next(parser.stream)
|
| 491 |
+
elif parser.stream.current.type == "variable_begin":
|
| 492 |
+
next(parser.stream)
|
| 493 |
+
name = parser.stream.expect("name").value
|
| 494 |
+
referenced.append(name)
|
| 495 |
+
buf.append(f"%({name})s")
|
| 496 |
+
parser.stream.expect("variable_end")
|
| 497 |
+
elif parser.stream.current.type == "block_begin":
|
| 498 |
+
next(parser.stream)
|
| 499 |
+
block_name = (
|
| 500 |
+
parser.stream.current.value
|
| 501 |
+
if parser.stream.current.type == "name"
|
| 502 |
+
else None
|
| 503 |
+
)
|
| 504 |
+
if block_name == "endtrans":
|
| 505 |
+
break
|
| 506 |
+
elif block_name == "pluralize":
|
| 507 |
+
if allow_pluralize:
|
| 508 |
+
break
|
| 509 |
+
parser.fail(
|
| 510 |
+
"a translatable section can have only one pluralize section"
|
| 511 |
+
)
|
| 512 |
+
elif block_name == "trans":
|
| 513 |
+
parser.fail(
|
| 514 |
+
"trans blocks can't be nested; did you mean `endtrans`?"
|
| 515 |
+
)
|
| 516 |
+
parser.fail(
|
| 517 |
+
f"control structures in translatable sections are not allowed; "
|
| 518 |
+
f"saw `{block_name}`"
|
| 519 |
+
)
|
| 520 |
+
elif parser.stream.eos:
|
| 521 |
+
parser.fail("unclosed translation block")
|
| 522 |
+
else:
|
| 523 |
+
raise RuntimeError("internal parser error")
|
| 524 |
+
|
| 525 |
+
return referenced, concat(buf)
|
| 526 |
+
|
| 527 |
+
def _make_node(
|
| 528 |
+
self,
|
| 529 |
+
singular: str,
|
| 530 |
+
plural: t.Optional[str],
|
| 531 |
+
context: t.Optional[str],
|
| 532 |
+
variables: t.Dict[str, nodes.Expr],
|
| 533 |
+
plural_expr: t.Optional[nodes.Expr],
|
| 534 |
+
vars_referenced: bool,
|
| 535 |
+
num_called_num: bool,
|
| 536 |
+
) -> nodes.Output:
|
| 537 |
+
"""Generates a useful node from the data provided."""
|
| 538 |
+
newstyle = self.environment.newstyle_gettext # type: ignore
|
| 539 |
+
node: nodes.Expr
|
| 540 |
+
|
| 541 |
+
# no variables referenced? no need to escape for old style
|
| 542 |
+
# gettext invocations only if there are vars.
|
| 543 |
+
if not vars_referenced and not newstyle:
|
| 544 |
+
singular = singular.replace("%%", "%")
|
| 545 |
+
if plural:
|
| 546 |
+
plural = plural.replace("%%", "%")
|
| 547 |
+
|
| 548 |
+
func_name = "gettext"
|
| 549 |
+
func_args: t.List[nodes.Expr] = [nodes.Const(singular)]
|
| 550 |
+
|
| 551 |
+
if context is not None:
|
| 552 |
+
func_args.insert(0, nodes.Const(context))
|
| 553 |
+
func_name = f"p{func_name}"
|
| 554 |
+
|
| 555 |
+
if plural_expr is not None:
|
| 556 |
+
func_name = f"n{func_name}"
|
| 557 |
+
func_args.extend((nodes.Const(plural), plural_expr))
|
| 558 |
+
|
| 559 |
+
node = nodes.Call(nodes.Name(func_name, "load"), func_args, [], None, None)
|
| 560 |
+
|
| 561 |
+
# in case newstyle gettext is used, the method is powerful
|
| 562 |
+
# enough to handle the variable expansion and autoescape
|
| 563 |
+
# handling itself
|
| 564 |
+
if newstyle:
|
| 565 |
+
for key, value in variables.items():
|
| 566 |
+
# the function adds that later anyways in case num was
|
| 567 |
+
# called num, so just skip it.
|
| 568 |
+
if num_called_num and key == "num":
|
| 569 |
+
continue
|
| 570 |
+
node.kwargs.append(nodes.Keyword(key, value))
|
| 571 |
+
|
| 572 |
+
# otherwise do that here
|
| 573 |
+
else:
|
| 574 |
+
# mark the return value as safe if we are in an
|
| 575 |
+
# environment with autoescaping turned on
|
| 576 |
+
node = nodes.MarkSafeIfAutoescape(node)
|
| 577 |
+
if variables:
|
| 578 |
+
node = nodes.Mod(
|
| 579 |
+
node,
|
| 580 |
+
nodes.Dict(
|
| 581 |
+
[
|
| 582 |
+
nodes.Pair(nodes.Const(key), value)
|
| 583 |
+
for key, value in variables.items()
|
| 584 |
+
]
|
| 585 |
+
),
|
| 586 |
+
)
|
| 587 |
+
return nodes.Output([node])
|
| 588 |
+
|
| 589 |
+
|
| 590 |
+
class ExprStmtExtension(Extension):
|
| 591 |
+
"""Adds a `do` tag to Jinja that works like the print statement just
|
| 592 |
+
that it doesn't print the return value.
|
| 593 |
+
"""
|
| 594 |
+
|
| 595 |
+
tags = {"do"}
|
| 596 |
+
|
| 597 |
+
def parse(self, parser: "Parser") -> nodes.ExprStmt:
|
| 598 |
+
node = nodes.ExprStmt(lineno=next(parser.stream).lineno)
|
| 599 |
+
node.node = parser.parse_tuple()
|
| 600 |
+
return node
|
| 601 |
+
|
| 602 |
+
|
| 603 |
+
class LoopControlExtension(Extension):
|
| 604 |
+
"""Adds break and continue to the template engine."""
|
| 605 |
+
|
| 606 |
+
tags = {"break", "continue"}
|
| 607 |
+
|
| 608 |
+
def parse(self, parser: "Parser") -> t.Union[nodes.Break, nodes.Continue]:
|
| 609 |
+
token = next(parser.stream)
|
| 610 |
+
if token.value == "break":
|
| 611 |
+
return nodes.Break(lineno=token.lineno)
|
| 612 |
+
return nodes.Continue(lineno=token.lineno)
|
| 613 |
+
|
| 614 |
+
|
| 615 |
+
class DebugExtension(Extension):
|
| 616 |
+
"""A ``{% debug %}`` tag that dumps the available variables,
|
| 617 |
+
filters, and tests.
|
| 618 |
+
|
| 619 |
+
.. code-block:: html+jinja
|
| 620 |
+
|
| 621 |
+
<pre>{% debug %}</pre>
|
| 622 |
+
|
| 623 |
+
.. code-block:: text
|
| 624 |
+
|
| 625 |
+
{'context': {'cycler': <class 'jinja2.utils.Cycler'>,
|
| 626 |
+
...,
|
| 627 |
+
'namespace': <class 'jinja2.utils.Namespace'>},
|
| 628 |
+
'filters': ['abs', 'attr', 'batch', 'capitalize', 'center', 'count', 'd',
|
| 629 |
+
..., 'urlencode', 'urlize', 'wordcount', 'wordwrap', 'xmlattr'],
|
| 630 |
+
'tests': ['!=', '<', '<=', '==', '>', '>=', 'callable', 'defined',
|
| 631 |
+
..., 'odd', 'sameas', 'sequence', 'string', 'undefined', 'upper']}
|
| 632 |
+
|
| 633 |
+
.. versionadded:: 2.11.0
|
| 634 |
+
"""
|
| 635 |
+
|
| 636 |
+
tags = {"debug"}
|
| 637 |
+
|
| 638 |
+
def parse(self, parser: "Parser") -> nodes.Output:
|
| 639 |
+
lineno = parser.stream.expect("name:debug").lineno
|
| 640 |
+
context = nodes.ContextReference()
|
| 641 |
+
result = self.call_method("_render", [context], lineno=lineno)
|
| 642 |
+
return nodes.Output([result], lineno=lineno)
|
| 643 |
+
|
| 644 |
+
def _render(self, context: Context) -> str:
|
| 645 |
+
result = {
|
| 646 |
+
"context": context.get_all(),
|
| 647 |
+
"filters": sorted(self.environment.filters.keys()),
|
| 648 |
+
"tests": sorted(self.environment.tests.keys()),
|
| 649 |
+
}
|
| 650 |
+
|
| 651 |
+
# Set the depth since the intent is to show the top few names.
|
| 652 |
+
return pprint.pformat(result, depth=3, compact=True)
|
| 653 |
+
|
| 654 |
+
|
| 655 |
+
def extract_from_ast(
|
| 656 |
+
ast: nodes.Template,
|
| 657 |
+
gettext_functions: t.Sequence[str] = GETTEXT_FUNCTIONS,
|
| 658 |
+
babel_style: bool = True,
|
| 659 |
+
) -> t.Iterator[
|
| 660 |
+
t.Tuple[int, str, t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]]]
|
| 661 |
+
]:
|
| 662 |
+
"""Extract localizable strings from the given template node. Per
|
| 663 |
+
default this function returns matches in babel style that means non string
|
| 664 |
+
parameters as well as keyword arguments are returned as `None`. This
|
| 665 |
+
allows Babel to figure out what you really meant if you are using
|
| 666 |
+
gettext functions that allow keyword arguments for placeholder expansion.
|
| 667 |
+
If you don't want that behavior set the `babel_style` parameter to `False`
|
| 668 |
+
which causes only strings to be returned and parameters are always stored
|
| 669 |
+
in tuples. As a consequence invalid gettext calls (calls without a single
|
| 670 |
+
string parameter or string parameters after non-string parameters) are
|
| 671 |
+
skipped.
|
| 672 |
+
|
| 673 |
+
This example explains the behavior:
|
| 674 |
+
|
| 675 |
+
>>> from jinja2 import Environment
|
| 676 |
+
>>> env = Environment()
|
| 677 |
+
>>> node = env.parse('{{ (_("foo"), _(), ngettext("foo", "bar", 42)) }}')
|
| 678 |
+
>>> list(extract_from_ast(node))
|
| 679 |
+
[(1, '_', 'foo'), (1, '_', ()), (1, 'ngettext', ('foo', 'bar', None))]
|
| 680 |
+
>>> list(extract_from_ast(node, babel_style=False))
|
| 681 |
+
[(1, '_', ('foo',)), (1, 'ngettext', ('foo', 'bar'))]
|
| 682 |
+
|
| 683 |
+
For every string found this function yields a ``(lineno, function,
|
| 684 |
+
message)`` tuple, where:
|
| 685 |
+
|
| 686 |
+
* ``lineno`` is the number of the line on which the string was found,
|
| 687 |
+
* ``function`` is the name of the ``gettext`` function used (if the
|
| 688 |
+
string was extracted from embedded Python code), and
|
| 689 |
+
* ``message`` is the string, or a tuple of strings for functions
|
| 690 |
+
with multiple string arguments.
|
| 691 |
+
|
| 692 |
+
This extraction function operates on the AST and is because of that unable
|
| 693 |
+
to extract any comments. For comment support you have to use the babel
|
| 694 |
+
extraction interface or extract comments yourself.
|
| 695 |
+
"""
|
| 696 |
+
out: t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]]
|
| 697 |
+
|
| 698 |
+
for node in ast.find_all(nodes.Call):
|
| 699 |
+
if (
|
| 700 |
+
not isinstance(node.node, nodes.Name)
|
| 701 |
+
or node.node.name not in gettext_functions
|
| 702 |
+
):
|
| 703 |
+
continue
|
| 704 |
+
|
| 705 |
+
strings: t.List[t.Optional[str]] = []
|
| 706 |
+
|
| 707 |
+
for arg in node.args:
|
| 708 |
+
if isinstance(arg, nodes.Const) and isinstance(arg.value, str):
|
| 709 |
+
strings.append(arg.value)
|
| 710 |
+
else:
|
| 711 |
+
strings.append(None)
|
| 712 |
+
|
| 713 |
+
for _ in node.kwargs:
|
| 714 |
+
strings.append(None)
|
| 715 |
+
if node.dyn_args is not None:
|
| 716 |
+
strings.append(None)
|
| 717 |
+
if node.dyn_kwargs is not None:
|
| 718 |
+
strings.append(None)
|
| 719 |
+
|
| 720 |
+
if not babel_style:
|
| 721 |
+
out = tuple(x for x in strings if x is not None)
|
| 722 |
+
|
| 723 |
+
if not out:
|
| 724 |
+
continue
|
| 725 |
+
else:
|
| 726 |
+
if len(strings) == 1:
|
| 727 |
+
out = strings[0]
|
| 728 |
+
else:
|
| 729 |
+
out = tuple(strings)
|
| 730 |
+
|
| 731 |
+
yield node.lineno, node.node.name, out
|
| 732 |
+
|
| 733 |
+
|
| 734 |
+
class _CommentFinder:
|
| 735 |
+
"""Helper class to find comments in a token stream. Can only
|
| 736 |
+
find comments for gettext calls forwards. Once the comment
|
| 737 |
+
from line 4 is found, a comment for line 1 will not return a
|
| 738 |
+
usable value.
|
| 739 |
+
"""
|
| 740 |
+
|
| 741 |
+
def __init__(
|
| 742 |
+
self, tokens: t.Sequence[t.Tuple[int, str, str]], comment_tags: t.Sequence[str]
|
| 743 |
+
) -> None:
|
| 744 |
+
self.tokens = tokens
|
| 745 |
+
self.comment_tags = comment_tags
|
| 746 |
+
self.offset = 0
|
| 747 |
+
self.last_lineno = 0
|
| 748 |
+
|
| 749 |
+
def find_backwards(self, offset: int) -> t.List[str]:
|
| 750 |
+
try:
|
| 751 |
+
for _, token_type, token_value in reversed(
|
| 752 |
+
self.tokens[self.offset : offset]
|
| 753 |
+
):
|
| 754 |
+
if token_type in ("comment", "linecomment"):
|
| 755 |
+
try:
|
| 756 |
+
prefix, comment = token_value.split(None, 1)
|
| 757 |
+
except ValueError:
|
| 758 |
+
continue
|
| 759 |
+
if prefix in self.comment_tags:
|
| 760 |
+
return [comment.rstrip()]
|
| 761 |
+
return []
|
| 762 |
+
finally:
|
| 763 |
+
self.offset = offset
|
| 764 |
+
|
| 765 |
+
def find_comments(self, lineno: int) -> t.List[str]:
|
| 766 |
+
if not self.comment_tags or self.last_lineno > lineno:
|
| 767 |
+
return []
|
| 768 |
+
for idx, (token_lineno, _, _) in enumerate(self.tokens[self.offset :]):
|
| 769 |
+
if token_lineno > lineno:
|
| 770 |
+
return self.find_backwards(self.offset + idx)
|
| 771 |
+
return self.find_backwards(len(self.tokens))
|
| 772 |
+
|
| 773 |
+
|
| 774 |
+
def babel_extract(
|
| 775 |
+
fileobj: t.BinaryIO,
|
| 776 |
+
keywords: t.Sequence[str],
|
| 777 |
+
comment_tags: t.Sequence[str],
|
| 778 |
+
options: t.Dict[str, t.Any],
|
| 779 |
+
) -> t.Iterator[
|
| 780 |
+
t.Tuple[
|
| 781 |
+
int, str, t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]], t.List[str]
|
| 782 |
+
]
|
| 783 |
+
]:
|
| 784 |
+
"""Babel extraction method for Jinja templates.
|
| 785 |
+
|
| 786 |
+
.. versionchanged:: 2.3
|
| 787 |
+
Basic support for translation comments was added. If `comment_tags`
|
| 788 |
+
is now set to a list of keywords for extraction, the extractor will
|
| 789 |
+
try to find the best preceding comment that begins with one of the
|
| 790 |
+
keywords. For best results, make sure to not have more than one
|
| 791 |
+
gettext call in one line of code and the matching comment in the
|
| 792 |
+
same line or the line before.
|
| 793 |
+
|
| 794 |
+
.. versionchanged:: 2.5.1
|
| 795 |
+
The `newstyle_gettext` flag can be set to `True` to enable newstyle
|
| 796 |
+
gettext calls.
|
| 797 |
+
|
| 798 |
+
.. versionchanged:: 2.7
|
| 799 |
+
A `silent` option can now be provided. If set to `False` template
|
| 800 |
+
syntax errors are propagated instead of being ignored.
|
| 801 |
+
|
| 802 |
+
:param fileobj: the file-like object the messages should be extracted from
|
| 803 |
+
:param keywords: a list of keywords (i.e. function names) that should be
|
| 804 |
+
recognized as translation functions
|
| 805 |
+
:param comment_tags: a list of translator tags to search for and include
|
| 806 |
+
in the results.
|
| 807 |
+
:param options: a dictionary of additional options (optional)
|
| 808 |
+
:return: an iterator over ``(lineno, funcname, message, comments)`` tuples.
|
| 809 |
+
(comments will be empty currently)
|
| 810 |
+
"""
|
| 811 |
+
extensions: t.Dict[t.Type[Extension], None] = {}
|
| 812 |
+
|
| 813 |
+
for extension_name in options.get("extensions", "").split(","):
|
| 814 |
+
extension_name = extension_name.strip()
|
| 815 |
+
|
| 816 |
+
if not extension_name:
|
| 817 |
+
continue
|
| 818 |
+
|
| 819 |
+
extensions[import_string(extension_name)] = None
|
| 820 |
+
|
| 821 |
+
if InternationalizationExtension not in extensions:
|
| 822 |
+
extensions[InternationalizationExtension] = None
|
| 823 |
+
|
| 824 |
+
def getbool(options: t.Mapping[str, str], key: str, default: bool = False) -> bool:
|
| 825 |
+
return options.get(key, str(default)).lower() in {"1", "on", "yes", "true"}
|
| 826 |
+
|
| 827 |
+
silent = getbool(options, "silent", True)
|
| 828 |
+
environment = Environment(
|
| 829 |
+
options.get("block_start_string", defaults.BLOCK_START_STRING),
|
| 830 |
+
options.get("block_end_string", defaults.BLOCK_END_STRING),
|
| 831 |
+
options.get("variable_start_string", defaults.VARIABLE_START_STRING),
|
| 832 |
+
options.get("variable_end_string", defaults.VARIABLE_END_STRING),
|
| 833 |
+
options.get("comment_start_string", defaults.COMMENT_START_STRING),
|
| 834 |
+
options.get("comment_end_string", defaults.COMMENT_END_STRING),
|
| 835 |
+
options.get("line_statement_prefix") or defaults.LINE_STATEMENT_PREFIX,
|
| 836 |
+
options.get("line_comment_prefix") or defaults.LINE_COMMENT_PREFIX,
|
| 837 |
+
getbool(options, "trim_blocks", defaults.TRIM_BLOCKS),
|
| 838 |
+
getbool(options, "lstrip_blocks", defaults.LSTRIP_BLOCKS),
|
| 839 |
+
defaults.NEWLINE_SEQUENCE,
|
| 840 |
+
getbool(options, "keep_trailing_newline", defaults.KEEP_TRAILING_NEWLINE),
|
| 841 |
+
tuple(extensions),
|
| 842 |
+
cache_size=0,
|
| 843 |
+
auto_reload=False,
|
| 844 |
+
)
|
| 845 |
+
|
| 846 |
+
if getbool(options, "trimmed"):
|
| 847 |
+
environment.policies["ext.i18n.trimmed"] = True
|
| 848 |
+
if getbool(options, "newstyle_gettext"):
|
| 849 |
+
environment.newstyle_gettext = True # type: ignore
|
| 850 |
+
|
| 851 |
+
source = fileobj.read().decode(options.get("encoding", "utf-8"))
|
| 852 |
+
try:
|
| 853 |
+
node = environment.parse(source)
|
| 854 |
+
tokens = list(environment.lex(environment.preprocess(source)))
|
| 855 |
+
except TemplateSyntaxError:
|
| 856 |
+
if not silent:
|
| 857 |
+
raise
|
| 858 |
+
# skip templates with syntax errors
|
| 859 |
+
return
|
| 860 |
+
|
| 861 |
+
finder = _CommentFinder(tokens, comment_tags)
|
| 862 |
+
for lineno, func, message in extract_from_ast(node, keywords):
|
| 863 |
+
yield lineno, func, message, finder.find_comments(lineno)
|
| 864 |
+
|
| 865 |
+
|
| 866 |
+
#: nicer import names
|
| 867 |
+
i18n = InternationalizationExtension
|
| 868 |
+
do = ExprStmtExtension
|
| 869 |
+
loopcontrols = LoopControlExtension
|
| 870 |
+
debug = DebugExtension
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/filters.py
ADDED
|
@@ -0,0 +1,1873 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Built-in template filters used with the ``|`` operator."""
|
| 2 |
+
|
| 3 |
+
import math
|
| 4 |
+
import random
|
| 5 |
+
import re
|
| 6 |
+
import typing
|
| 7 |
+
import typing as t
|
| 8 |
+
from collections import abc
|
| 9 |
+
from inspect import getattr_static
|
| 10 |
+
from itertools import chain
|
| 11 |
+
from itertools import groupby
|
| 12 |
+
|
| 13 |
+
from markupsafe import escape
|
| 14 |
+
from markupsafe import Markup
|
| 15 |
+
from markupsafe import soft_str
|
| 16 |
+
|
| 17 |
+
from .async_utils import async_variant
|
| 18 |
+
from .async_utils import auto_aiter
|
| 19 |
+
from .async_utils import auto_await
|
| 20 |
+
from .async_utils import auto_to_list
|
| 21 |
+
from .exceptions import FilterArgumentError
|
| 22 |
+
from .runtime import Undefined
|
| 23 |
+
from .utils import htmlsafe_json_dumps
|
| 24 |
+
from .utils import pass_context
|
| 25 |
+
from .utils import pass_environment
|
| 26 |
+
from .utils import pass_eval_context
|
| 27 |
+
from .utils import pformat
|
| 28 |
+
from .utils import url_quote
|
| 29 |
+
from .utils import urlize
|
| 30 |
+
|
| 31 |
+
if t.TYPE_CHECKING:
|
| 32 |
+
import typing_extensions as te
|
| 33 |
+
|
| 34 |
+
from .environment import Environment
|
| 35 |
+
from .nodes import EvalContext
|
| 36 |
+
from .runtime import Context
|
| 37 |
+
from .sandbox import SandboxedEnvironment # noqa: F401
|
| 38 |
+
|
| 39 |
+
class HasHTML(te.Protocol):
|
| 40 |
+
def __html__(self) -> str:
|
| 41 |
+
pass
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
F = t.TypeVar("F", bound=t.Callable[..., t.Any])
|
| 45 |
+
K = t.TypeVar("K")
|
| 46 |
+
V = t.TypeVar("V")
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def ignore_case(value: V) -> V:
|
| 50 |
+
"""For use as a postprocessor for :func:`make_attrgetter`. Converts strings
|
| 51 |
+
to lowercase and returns other types as-is."""
|
| 52 |
+
if isinstance(value, str):
|
| 53 |
+
return t.cast(V, value.lower())
|
| 54 |
+
|
| 55 |
+
return value
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def make_attrgetter(
|
| 59 |
+
environment: "Environment",
|
| 60 |
+
attribute: t.Optional[t.Union[str, int]],
|
| 61 |
+
postprocess: t.Optional[t.Callable[[t.Any], t.Any]] = None,
|
| 62 |
+
default: t.Optional[t.Any] = None,
|
| 63 |
+
) -> t.Callable[[t.Any], t.Any]:
|
| 64 |
+
"""Returns a callable that looks up the given attribute from a
|
| 65 |
+
passed object with the rules of the environment. Dots are allowed
|
| 66 |
+
to access attributes of attributes. Integer parts in paths are
|
| 67 |
+
looked up as integers.
|
| 68 |
+
"""
|
| 69 |
+
parts = _prepare_attribute_parts(attribute)
|
| 70 |
+
|
| 71 |
+
def attrgetter(item: t.Any) -> t.Any:
|
| 72 |
+
for part in parts:
|
| 73 |
+
item = environment.getitem(item, part)
|
| 74 |
+
|
| 75 |
+
if default is not None and isinstance(item, Undefined):
|
| 76 |
+
item = default
|
| 77 |
+
|
| 78 |
+
if postprocess is not None:
|
| 79 |
+
item = postprocess(item)
|
| 80 |
+
|
| 81 |
+
return item
|
| 82 |
+
|
| 83 |
+
return attrgetter
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def make_multi_attrgetter(
|
| 87 |
+
environment: "Environment",
|
| 88 |
+
attribute: t.Optional[t.Union[str, int]],
|
| 89 |
+
postprocess: t.Optional[t.Callable[[t.Any], t.Any]] = None,
|
| 90 |
+
) -> t.Callable[[t.Any], t.List[t.Any]]:
|
| 91 |
+
"""Returns a callable that looks up the given comma separated
|
| 92 |
+
attributes from a passed object with the rules of the environment.
|
| 93 |
+
Dots are allowed to access attributes of each attribute. Integer
|
| 94 |
+
parts in paths are looked up as integers.
|
| 95 |
+
|
| 96 |
+
The value returned by the returned callable is a list of extracted
|
| 97 |
+
attribute values.
|
| 98 |
+
|
| 99 |
+
Examples of attribute: "attr1,attr2", "attr1.inner1.0,attr2.inner2.0", etc.
|
| 100 |
+
"""
|
| 101 |
+
if isinstance(attribute, str):
|
| 102 |
+
split: t.Sequence[t.Union[str, int, None]] = attribute.split(",")
|
| 103 |
+
else:
|
| 104 |
+
split = [attribute]
|
| 105 |
+
|
| 106 |
+
parts = [_prepare_attribute_parts(item) for item in split]
|
| 107 |
+
|
| 108 |
+
def attrgetter(item: t.Any) -> t.List[t.Any]:
|
| 109 |
+
items = [None] * len(parts)
|
| 110 |
+
|
| 111 |
+
for i, attribute_part in enumerate(parts):
|
| 112 |
+
item_i = item
|
| 113 |
+
|
| 114 |
+
for part in attribute_part:
|
| 115 |
+
item_i = environment.getitem(item_i, part)
|
| 116 |
+
|
| 117 |
+
if postprocess is not None:
|
| 118 |
+
item_i = postprocess(item_i)
|
| 119 |
+
|
| 120 |
+
items[i] = item_i
|
| 121 |
+
|
| 122 |
+
return items
|
| 123 |
+
|
| 124 |
+
return attrgetter
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
def _prepare_attribute_parts(
|
| 128 |
+
attr: t.Optional[t.Union[str, int]],
|
| 129 |
+
) -> t.List[t.Union[str, int]]:
|
| 130 |
+
if attr is None:
|
| 131 |
+
return []
|
| 132 |
+
|
| 133 |
+
if isinstance(attr, str):
|
| 134 |
+
return [int(x) if x.isdigit() else x for x in attr.split(".")]
|
| 135 |
+
|
| 136 |
+
return [attr]
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def do_forceescape(value: "t.Union[str, HasHTML]") -> Markup:
|
| 140 |
+
"""Enforce HTML escaping. This will probably double escape variables."""
|
| 141 |
+
if hasattr(value, "__html__"):
|
| 142 |
+
value = t.cast("HasHTML", value).__html__()
|
| 143 |
+
|
| 144 |
+
return escape(str(value))
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def do_urlencode(
|
| 148 |
+
value: t.Union[str, t.Mapping[str, t.Any], t.Iterable[t.Tuple[str, t.Any]]],
|
| 149 |
+
) -> str:
|
| 150 |
+
"""Quote data for use in a URL path or query using UTF-8.
|
| 151 |
+
|
| 152 |
+
Basic wrapper around :func:`urllib.parse.quote` when given a
|
| 153 |
+
string, or :func:`urllib.parse.urlencode` for a dict or iterable.
|
| 154 |
+
|
| 155 |
+
:param value: Data to quote. A string will be quoted directly. A
|
| 156 |
+
dict or iterable of ``(key, value)`` pairs will be joined as a
|
| 157 |
+
query string.
|
| 158 |
+
|
| 159 |
+
When given a string, "/" is not quoted. HTTP servers treat "/" and
|
| 160 |
+
"%2F" equivalently in paths. If you need quoted slashes, use the
|
| 161 |
+
``|replace("/", "%2F")`` filter.
|
| 162 |
+
|
| 163 |
+
.. versionadded:: 2.7
|
| 164 |
+
"""
|
| 165 |
+
if isinstance(value, str) or not isinstance(value, abc.Iterable):
|
| 166 |
+
return url_quote(value)
|
| 167 |
+
|
| 168 |
+
if isinstance(value, dict):
|
| 169 |
+
items: t.Iterable[t.Tuple[str, t.Any]] = value.items()
|
| 170 |
+
else:
|
| 171 |
+
items = value # type: ignore
|
| 172 |
+
|
| 173 |
+
return "&".join(
|
| 174 |
+
f"{url_quote(k, for_qs=True)}={url_quote(v, for_qs=True)}" for k, v in items
|
| 175 |
+
)
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
@pass_eval_context
|
| 179 |
+
def do_replace(
|
| 180 |
+
eval_ctx: "EvalContext", s: str, old: str, new: str, count: t.Optional[int] = None
|
| 181 |
+
) -> str:
|
| 182 |
+
"""Return a copy of the value with all occurrences of a substring
|
| 183 |
+
replaced with a new one. The first argument is the substring
|
| 184 |
+
that should be replaced, the second is the replacement string.
|
| 185 |
+
If the optional third argument ``count`` is given, only the first
|
| 186 |
+
``count`` occurrences are replaced:
|
| 187 |
+
|
| 188 |
+
.. sourcecode:: jinja
|
| 189 |
+
|
| 190 |
+
{{ "Hello World"|replace("Hello", "Goodbye") }}
|
| 191 |
+
-> Goodbye World
|
| 192 |
+
|
| 193 |
+
{{ "aaaaargh"|replace("a", "d'oh, ", 2) }}
|
| 194 |
+
-> d'oh, d'oh, aaargh
|
| 195 |
+
"""
|
| 196 |
+
if count is None:
|
| 197 |
+
count = -1
|
| 198 |
+
|
| 199 |
+
if not eval_ctx.autoescape:
|
| 200 |
+
return str(s).replace(str(old), str(new), count)
|
| 201 |
+
|
| 202 |
+
if (
|
| 203 |
+
hasattr(old, "__html__")
|
| 204 |
+
or hasattr(new, "__html__")
|
| 205 |
+
and not hasattr(s, "__html__")
|
| 206 |
+
):
|
| 207 |
+
s = escape(s)
|
| 208 |
+
else:
|
| 209 |
+
s = soft_str(s)
|
| 210 |
+
|
| 211 |
+
return s.replace(soft_str(old), soft_str(new), count)
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
def do_upper(s: str) -> str:
|
| 215 |
+
"""Convert a value to uppercase."""
|
| 216 |
+
return soft_str(s).upper()
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
def do_lower(s: str) -> str:
|
| 220 |
+
"""Convert a value to lowercase."""
|
| 221 |
+
return soft_str(s).lower()
|
| 222 |
+
|
| 223 |
+
|
| 224 |
+
def do_items(value: t.Union[t.Mapping[K, V], Undefined]) -> t.Iterator[t.Tuple[K, V]]:
|
| 225 |
+
"""Return an iterator over the ``(key, value)`` items of a mapping.
|
| 226 |
+
|
| 227 |
+
``x|items`` is the same as ``x.items()``, except if ``x`` is
|
| 228 |
+
undefined an empty iterator is returned.
|
| 229 |
+
|
| 230 |
+
This filter is useful if you expect the template to be rendered with
|
| 231 |
+
an implementation of Jinja in another programming language that does
|
| 232 |
+
not have a ``.items()`` method on its mapping type.
|
| 233 |
+
|
| 234 |
+
.. code-block:: html+jinja
|
| 235 |
+
|
| 236 |
+
<dl>
|
| 237 |
+
{% for key, value in my_dict|items %}
|
| 238 |
+
<dt>{{ key }}
|
| 239 |
+
<dd>{{ value }}
|
| 240 |
+
{% endfor %}
|
| 241 |
+
</dl>
|
| 242 |
+
|
| 243 |
+
.. versionadded:: 3.1
|
| 244 |
+
"""
|
| 245 |
+
if isinstance(value, Undefined):
|
| 246 |
+
return
|
| 247 |
+
|
| 248 |
+
if not isinstance(value, abc.Mapping):
|
| 249 |
+
raise TypeError("Can only get item pairs from a mapping.")
|
| 250 |
+
|
| 251 |
+
yield from value.items()
|
| 252 |
+
|
| 253 |
+
|
| 254 |
+
# Check for characters that would move the parser state from key to value.
|
| 255 |
+
# https://html.spec.whatwg.org/#attribute-name-state
|
| 256 |
+
_attr_key_re = re.compile(r"[\s/>=]", flags=re.ASCII)
|
| 257 |
+
|
| 258 |
+
|
| 259 |
+
@pass_eval_context
|
| 260 |
+
def do_xmlattr(
|
| 261 |
+
eval_ctx: "EvalContext", d: t.Mapping[str, t.Any], autospace: bool = True
|
| 262 |
+
) -> str:
|
| 263 |
+
"""Create an SGML/XML attribute string based on the items in a dict.
|
| 264 |
+
|
| 265 |
+
**Values** that are neither ``none`` nor ``undefined`` are automatically
|
| 266 |
+
escaped, safely allowing untrusted user input.
|
| 267 |
+
|
| 268 |
+
User input should not be used as **keys** to this filter. If any key
|
| 269 |
+
contains a space, ``/`` solidus, ``>`` greater-than sign, or ``=`` equals
|
| 270 |
+
sign, this fails with a ``ValueError``. Regardless of this, user input
|
| 271 |
+
should never be used as keys to this filter, or must be separately validated
|
| 272 |
+
first.
|
| 273 |
+
|
| 274 |
+
.. sourcecode:: html+jinja
|
| 275 |
+
|
| 276 |
+
<ul{{ {'class': 'my_list', 'missing': none,
|
| 277 |
+
'id': 'list-%d'|format(variable)}|xmlattr }}>
|
| 278 |
+
...
|
| 279 |
+
</ul>
|
| 280 |
+
|
| 281 |
+
Results in something like this:
|
| 282 |
+
|
| 283 |
+
.. sourcecode:: html
|
| 284 |
+
|
| 285 |
+
<ul class="my_list" id="list-42">
|
| 286 |
+
...
|
| 287 |
+
</ul>
|
| 288 |
+
|
| 289 |
+
As you can see it automatically prepends a space in front of the item
|
| 290 |
+
if the filter returned something unless the second parameter is false.
|
| 291 |
+
|
| 292 |
+
.. versionchanged:: 3.1.4
|
| 293 |
+
Keys with ``/`` solidus, ``>`` greater-than sign, or ``=`` equals sign
|
| 294 |
+
are not allowed.
|
| 295 |
+
|
| 296 |
+
.. versionchanged:: 3.1.3
|
| 297 |
+
Keys with spaces are not allowed.
|
| 298 |
+
"""
|
| 299 |
+
items = []
|
| 300 |
+
|
| 301 |
+
for key, value in d.items():
|
| 302 |
+
if value is None or isinstance(value, Undefined):
|
| 303 |
+
continue
|
| 304 |
+
|
| 305 |
+
if _attr_key_re.search(key) is not None:
|
| 306 |
+
raise ValueError(f"Invalid character in attribute name: {key!r}")
|
| 307 |
+
|
| 308 |
+
items.append(f'{escape(key)}="{escape(value)}"')
|
| 309 |
+
|
| 310 |
+
rv = " ".join(items)
|
| 311 |
+
|
| 312 |
+
if autospace and rv:
|
| 313 |
+
rv = " " + rv
|
| 314 |
+
|
| 315 |
+
if eval_ctx.autoescape:
|
| 316 |
+
rv = Markup(rv)
|
| 317 |
+
|
| 318 |
+
return rv
|
| 319 |
+
|
| 320 |
+
|
| 321 |
+
def do_capitalize(s: str) -> str:
|
| 322 |
+
"""Capitalize a value. The first character will be uppercase, all others
|
| 323 |
+
lowercase.
|
| 324 |
+
"""
|
| 325 |
+
return soft_str(s).capitalize()
|
| 326 |
+
|
| 327 |
+
|
| 328 |
+
_word_beginning_split_re = re.compile(r"([-\s({\[<]+)")
|
| 329 |
+
|
| 330 |
+
|
| 331 |
+
def do_title(s: str) -> str:
|
| 332 |
+
"""Return a titlecased version of the value. I.e. words will start with
|
| 333 |
+
uppercase letters, all remaining characters are lowercase.
|
| 334 |
+
"""
|
| 335 |
+
return "".join(
|
| 336 |
+
[
|
| 337 |
+
item[0].upper() + item[1:].lower()
|
| 338 |
+
for item in _word_beginning_split_re.split(soft_str(s))
|
| 339 |
+
if item
|
| 340 |
+
]
|
| 341 |
+
)
|
| 342 |
+
|
| 343 |
+
|
| 344 |
+
def do_dictsort(
|
| 345 |
+
value: t.Mapping[K, V],
|
| 346 |
+
case_sensitive: bool = False,
|
| 347 |
+
by: 'te.Literal["key", "value"]' = "key",
|
| 348 |
+
reverse: bool = False,
|
| 349 |
+
) -> t.List[t.Tuple[K, V]]:
|
| 350 |
+
"""Sort a dict and yield (key, value) pairs. Python dicts may not
|
| 351 |
+
be in the order you want to display them in, so sort them first.
|
| 352 |
+
|
| 353 |
+
.. sourcecode:: jinja
|
| 354 |
+
|
| 355 |
+
{% for key, value in mydict|dictsort %}
|
| 356 |
+
sort the dict by key, case insensitive
|
| 357 |
+
|
| 358 |
+
{% for key, value in mydict|dictsort(reverse=true) %}
|
| 359 |
+
sort the dict by key, case insensitive, reverse order
|
| 360 |
+
|
| 361 |
+
{% for key, value in mydict|dictsort(true) %}
|
| 362 |
+
sort the dict by key, case sensitive
|
| 363 |
+
|
| 364 |
+
{% for key, value in mydict|dictsort(false, 'value') %}
|
| 365 |
+
sort the dict by value, case insensitive
|
| 366 |
+
"""
|
| 367 |
+
if by == "key":
|
| 368 |
+
pos = 0
|
| 369 |
+
elif by == "value":
|
| 370 |
+
pos = 1
|
| 371 |
+
else:
|
| 372 |
+
raise FilterArgumentError('You can only sort by either "key" or "value"')
|
| 373 |
+
|
| 374 |
+
def sort_func(item: t.Tuple[t.Any, t.Any]) -> t.Any:
|
| 375 |
+
value = item[pos]
|
| 376 |
+
|
| 377 |
+
if not case_sensitive:
|
| 378 |
+
value = ignore_case(value)
|
| 379 |
+
|
| 380 |
+
return value
|
| 381 |
+
|
| 382 |
+
return sorted(value.items(), key=sort_func, reverse=reverse)
|
| 383 |
+
|
| 384 |
+
|
| 385 |
+
@pass_environment
|
| 386 |
+
def do_sort(
|
| 387 |
+
environment: "Environment",
|
| 388 |
+
value: "t.Iterable[V]",
|
| 389 |
+
reverse: bool = False,
|
| 390 |
+
case_sensitive: bool = False,
|
| 391 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 392 |
+
) -> "t.List[V]":
|
| 393 |
+
"""Sort an iterable using Python's :func:`sorted`.
|
| 394 |
+
|
| 395 |
+
.. sourcecode:: jinja
|
| 396 |
+
|
| 397 |
+
{% for city in cities|sort %}
|
| 398 |
+
...
|
| 399 |
+
{% endfor %}
|
| 400 |
+
|
| 401 |
+
:param reverse: Sort descending instead of ascending.
|
| 402 |
+
:param case_sensitive: When sorting strings, sort upper and lower
|
| 403 |
+
case separately.
|
| 404 |
+
:param attribute: When sorting objects or dicts, an attribute or
|
| 405 |
+
key to sort by. Can use dot notation like ``"address.city"``.
|
| 406 |
+
Can be a list of attributes like ``"age,name"``.
|
| 407 |
+
|
| 408 |
+
The sort is stable, it does not change the relative order of
|
| 409 |
+
elements that compare equal. This makes it is possible to chain
|
| 410 |
+
sorts on different attributes and ordering.
|
| 411 |
+
|
| 412 |
+
.. sourcecode:: jinja
|
| 413 |
+
|
| 414 |
+
{% for user in users|sort(attribute="name")
|
| 415 |
+
|sort(reverse=true, attribute="age") %}
|
| 416 |
+
...
|
| 417 |
+
{% endfor %}
|
| 418 |
+
|
| 419 |
+
As a shortcut to chaining when the direction is the same for all
|
| 420 |
+
attributes, pass a comma separate list of attributes.
|
| 421 |
+
|
| 422 |
+
.. sourcecode:: jinja
|
| 423 |
+
|
| 424 |
+
{% for user in users|sort(attribute="age,name") %}
|
| 425 |
+
...
|
| 426 |
+
{% endfor %}
|
| 427 |
+
|
| 428 |
+
.. versionchanged:: 2.11.0
|
| 429 |
+
The ``attribute`` parameter can be a comma separated list of
|
| 430 |
+
attributes, e.g. ``"age,name"``.
|
| 431 |
+
|
| 432 |
+
.. versionchanged:: 2.6
|
| 433 |
+
The ``attribute`` parameter was added.
|
| 434 |
+
"""
|
| 435 |
+
key_func = make_multi_attrgetter(
|
| 436 |
+
environment, attribute, postprocess=ignore_case if not case_sensitive else None
|
| 437 |
+
)
|
| 438 |
+
return sorted(value, key=key_func, reverse=reverse)
|
| 439 |
+
|
| 440 |
+
|
| 441 |
+
@pass_environment
|
| 442 |
+
def sync_do_unique(
|
| 443 |
+
environment: "Environment",
|
| 444 |
+
value: "t.Iterable[V]",
|
| 445 |
+
case_sensitive: bool = False,
|
| 446 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 447 |
+
) -> "t.Iterator[V]":
|
| 448 |
+
"""Returns a list of unique items from the given iterable.
|
| 449 |
+
|
| 450 |
+
.. sourcecode:: jinja
|
| 451 |
+
|
| 452 |
+
{{ ['foo', 'bar', 'foobar', 'FooBar']|unique|list }}
|
| 453 |
+
-> ['foo', 'bar', 'foobar']
|
| 454 |
+
|
| 455 |
+
The unique items are yielded in the same order as their first occurrence in
|
| 456 |
+
the iterable passed to the filter.
|
| 457 |
+
|
| 458 |
+
:param case_sensitive: Treat upper and lower case strings as distinct.
|
| 459 |
+
:param attribute: Filter objects with unique values for this attribute.
|
| 460 |
+
"""
|
| 461 |
+
getter = make_attrgetter(
|
| 462 |
+
environment, attribute, postprocess=ignore_case if not case_sensitive else None
|
| 463 |
+
)
|
| 464 |
+
seen = set()
|
| 465 |
+
|
| 466 |
+
for item in value:
|
| 467 |
+
key = getter(item)
|
| 468 |
+
|
| 469 |
+
if key not in seen:
|
| 470 |
+
seen.add(key)
|
| 471 |
+
yield item
|
| 472 |
+
|
| 473 |
+
|
| 474 |
+
@async_variant(sync_do_unique) # type: ignore
|
| 475 |
+
async def do_unique(
|
| 476 |
+
environment: "Environment",
|
| 477 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 478 |
+
case_sensitive: bool = False,
|
| 479 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 480 |
+
) -> "t.Iterator[V]":
|
| 481 |
+
return sync_do_unique(
|
| 482 |
+
environment, await auto_to_list(value), case_sensitive, attribute
|
| 483 |
+
)
|
| 484 |
+
|
| 485 |
+
|
| 486 |
+
def _min_or_max(
|
| 487 |
+
environment: "Environment",
|
| 488 |
+
value: "t.Iterable[V]",
|
| 489 |
+
func: "t.Callable[..., V]",
|
| 490 |
+
case_sensitive: bool,
|
| 491 |
+
attribute: t.Optional[t.Union[str, int]],
|
| 492 |
+
) -> "t.Union[V, Undefined]":
|
| 493 |
+
it = iter(value)
|
| 494 |
+
|
| 495 |
+
try:
|
| 496 |
+
first = next(it)
|
| 497 |
+
except StopIteration:
|
| 498 |
+
return environment.undefined("No aggregated item, sequence was empty.")
|
| 499 |
+
|
| 500 |
+
key_func = make_attrgetter(
|
| 501 |
+
environment, attribute, postprocess=ignore_case if not case_sensitive else None
|
| 502 |
+
)
|
| 503 |
+
return func(chain([first], it), key=key_func)
|
| 504 |
+
|
| 505 |
+
|
| 506 |
+
@pass_environment
|
| 507 |
+
def do_min(
|
| 508 |
+
environment: "Environment",
|
| 509 |
+
value: "t.Iterable[V]",
|
| 510 |
+
case_sensitive: bool = False,
|
| 511 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 512 |
+
) -> "t.Union[V, Undefined]":
|
| 513 |
+
"""Return the smallest item from the sequence.
|
| 514 |
+
|
| 515 |
+
.. sourcecode:: jinja
|
| 516 |
+
|
| 517 |
+
{{ [1, 2, 3]|min }}
|
| 518 |
+
-> 1
|
| 519 |
+
|
| 520 |
+
:param case_sensitive: Treat upper and lower case strings as distinct.
|
| 521 |
+
:param attribute: Get the object with the min value of this attribute.
|
| 522 |
+
"""
|
| 523 |
+
return _min_or_max(environment, value, min, case_sensitive, attribute)
|
| 524 |
+
|
| 525 |
+
|
| 526 |
+
@pass_environment
|
| 527 |
+
def do_max(
|
| 528 |
+
environment: "Environment",
|
| 529 |
+
value: "t.Iterable[V]",
|
| 530 |
+
case_sensitive: bool = False,
|
| 531 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 532 |
+
) -> "t.Union[V, Undefined]":
|
| 533 |
+
"""Return the largest item from the sequence.
|
| 534 |
+
|
| 535 |
+
.. sourcecode:: jinja
|
| 536 |
+
|
| 537 |
+
{{ [1, 2, 3]|max }}
|
| 538 |
+
-> 3
|
| 539 |
+
|
| 540 |
+
:param case_sensitive: Treat upper and lower case strings as distinct.
|
| 541 |
+
:param attribute: Get the object with the max value of this attribute.
|
| 542 |
+
"""
|
| 543 |
+
return _min_or_max(environment, value, max, case_sensitive, attribute)
|
| 544 |
+
|
| 545 |
+
|
| 546 |
+
def do_default(
|
| 547 |
+
value: V,
|
| 548 |
+
default_value: V = "", # type: ignore
|
| 549 |
+
boolean: bool = False,
|
| 550 |
+
) -> V:
|
| 551 |
+
"""If the value is undefined it will return the passed default value,
|
| 552 |
+
otherwise the value of the variable:
|
| 553 |
+
|
| 554 |
+
.. sourcecode:: jinja
|
| 555 |
+
|
| 556 |
+
{{ my_variable|default('my_variable is not defined') }}
|
| 557 |
+
|
| 558 |
+
This will output the value of ``my_variable`` if the variable was
|
| 559 |
+
defined, otherwise ``'my_variable is not defined'``. If you want
|
| 560 |
+
to use default with variables that evaluate to false you have to
|
| 561 |
+
set the second parameter to `true`:
|
| 562 |
+
|
| 563 |
+
.. sourcecode:: jinja
|
| 564 |
+
|
| 565 |
+
{{ ''|default('the string was empty', true) }}
|
| 566 |
+
|
| 567 |
+
.. versionchanged:: 2.11
|
| 568 |
+
It's now possible to configure the :class:`~jinja2.Environment` with
|
| 569 |
+
:class:`~jinja2.ChainableUndefined` to make the `default` filter work
|
| 570 |
+
on nested elements and attributes that may contain undefined values
|
| 571 |
+
in the chain without getting an :exc:`~jinja2.UndefinedError`.
|
| 572 |
+
"""
|
| 573 |
+
if isinstance(value, Undefined) or (boolean and not value):
|
| 574 |
+
return default_value
|
| 575 |
+
|
| 576 |
+
return value
|
| 577 |
+
|
| 578 |
+
|
| 579 |
+
@pass_eval_context
|
| 580 |
+
def sync_do_join(
|
| 581 |
+
eval_ctx: "EvalContext",
|
| 582 |
+
value: t.Iterable[t.Any],
|
| 583 |
+
d: str = "",
|
| 584 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 585 |
+
) -> str:
|
| 586 |
+
"""Return a string which is the concatenation of the strings in the
|
| 587 |
+
sequence. The separator between elements is an empty string per
|
| 588 |
+
default, you can define it with the optional parameter:
|
| 589 |
+
|
| 590 |
+
.. sourcecode:: jinja
|
| 591 |
+
|
| 592 |
+
{{ [1, 2, 3]|join('|') }}
|
| 593 |
+
-> 1|2|3
|
| 594 |
+
|
| 595 |
+
{{ [1, 2, 3]|join }}
|
| 596 |
+
-> 123
|
| 597 |
+
|
| 598 |
+
It is also possible to join certain attributes of an object:
|
| 599 |
+
|
| 600 |
+
.. sourcecode:: jinja
|
| 601 |
+
|
| 602 |
+
{{ users|join(', ', attribute='username') }}
|
| 603 |
+
|
| 604 |
+
.. versionadded:: 2.6
|
| 605 |
+
The `attribute` parameter was added.
|
| 606 |
+
"""
|
| 607 |
+
if attribute is not None:
|
| 608 |
+
value = map(make_attrgetter(eval_ctx.environment, attribute), value)
|
| 609 |
+
|
| 610 |
+
# no automatic escaping? joining is a lot easier then
|
| 611 |
+
if not eval_ctx.autoescape:
|
| 612 |
+
return str(d).join(map(str, value))
|
| 613 |
+
|
| 614 |
+
# if the delimiter doesn't have an html representation we check
|
| 615 |
+
# if any of the items has. If yes we do a coercion to Markup
|
| 616 |
+
if not hasattr(d, "__html__"):
|
| 617 |
+
value = list(value)
|
| 618 |
+
do_escape = False
|
| 619 |
+
|
| 620 |
+
for idx, item in enumerate(value):
|
| 621 |
+
if hasattr(item, "__html__"):
|
| 622 |
+
do_escape = True
|
| 623 |
+
else:
|
| 624 |
+
value[idx] = str(item)
|
| 625 |
+
|
| 626 |
+
if do_escape:
|
| 627 |
+
d = escape(d)
|
| 628 |
+
else:
|
| 629 |
+
d = str(d)
|
| 630 |
+
|
| 631 |
+
return d.join(value)
|
| 632 |
+
|
| 633 |
+
# no html involved, to normal joining
|
| 634 |
+
return soft_str(d).join(map(soft_str, value))
|
| 635 |
+
|
| 636 |
+
|
| 637 |
+
@async_variant(sync_do_join) # type: ignore
|
| 638 |
+
async def do_join(
|
| 639 |
+
eval_ctx: "EvalContext",
|
| 640 |
+
value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
|
| 641 |
+
d: str = "",
|
| 642 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 643 |
+
) -> str:
|
| 644 |
+
return sync_do_join(eval_ctx, await auto_to_list(value), d, attribute)
|
| 645 |
+
|
| 646 |
+
|
| 647 |
+
def do_center(value: str, width: int = 80) -> str:
|
| 648 |
+
"""Centers the value in a field of a given width."""
|
| 649 |
+
return soft_str(value).center(width)
|
| 650 |
+
|
| 651 |
+
|
| 652 |
+
@pass_environment
|
| 653 |
+
def sync_do_first(
|
| 654 |
+
environment: "Environment", seq: "t.Iterable[V]"
|
| 655 |
+
) -> "t.Union[V, Undefined]":
|
| 656 |
+
"""Return the first item of a sequence."""
|
| 657 |
+
try:
|
| 658 |
+
return next(iter(seq))
|
| 659 |
+
except StopIteration:
|
| 660 |
+
return environment.undefined("No first item, sequence was empty.")
|
| 661 |
+
|
| 662 |
+
|
| 663 |
+
@async_variant(sync_do_first) # type: ignore
|
| 664 |
+
async def do_first(
|
| 665 |
+
environment: "Environment", seq: "t.Union[t.AsyncIterable[V], t.Iterable[V]]"
|
| 666 |
+
) -> "t.Union[V, Undefined]":
|
| 667 |
+
try:
|
| 668 |
+
return await auto_aiter(seq).__anext__()
|
| 669 |
+
except StopAsyncIteration:
|
| 670 |
+
return environment.undefined("No first item, sequence was empty.")
|
| 671 |
+
|
| 672 |
+
|
| 673 |
+
@pass_environment
|
| 674 |
+
def do_last(
|
| 675 |
+
environment: "Environment", seq: "t.Reversible[V]"
|
| 676 |
+
) -> "t.Union[V, Undefined]":
|
| 677 |
+
"""Return the last item of a sequence.
|
| 678 |
+
|
| 679 |
+
Note: Does not work with generators. You may want to explicitly
|
| 680 |
+
convert it to a list:
|
| 681 |
+
|
| 682 |
+
.. sourcecode:: jinja
|
| 683 |
+
|
| 684 |
+
{{ data | selectattr('name', '==', 'Jinja') | list | last }}
|
| 685 |
+
"""
|
| 686 |
+
try:
|
| 687 |
+
return next(iter(reversed(seq)))
|
| 688 |
+
except StopIteration:
|
| 689 |
+
return environment.undefined("No last item, sequence was empty.")
|
| 690 |
+
|
| 691 |
+
|
| 692 |
+
# No async do_last, it may not be safe in async mode.
|
| 693 |
+
|
| 694 |
+
|
| 695 |
+
@pass_context
|
| 696 |
+
def do_random(context: "Context", seq: "t.Sequence[V]") -> "t.Union[V, Undefined]":
|
| 697 |
+
"""Return a random item from the sequence."""
|
| 698 |
+
try:
|
| 699 |
+
return random.choice(seq)
|
| 700 |
+
except IndexError:
|
| 701 |
+
return context.environment.undefined("No random item, sequence was empty.")
|
| 702 |
+
|
| 703 |
+
|
| 704 |
+
def do_filesizeformat(value: t.Union[str, float, int], binary: bool = False) -> str:
|
| 705 |
+
"""Format the value like a 'human-readable' file size (i.e. 13 kB,
|
| 706 |
+
4.1 MB, 102 Bytes, etc). Per default decimal prefixes are used (Mega,
|
| 707 |
+
Giga, etc.), if the second parameter is set to `True` the binary
|
| 708 |
+
prefixes are used (Mebi, Gibi).
|
| 709 |
+
"""
|
| 710 |
+
bytes = float(value)
|
| 711 |
+
base = 1024 if binary else 1000
|
| 712 |
+
prefixes = [
|
| 713 |
+
("KiB" if binary else "kB"),
|
| 714 |
+
("MiB" if binary else "MB"),
|
| 715 |
+
("GiB" if binary else "GB"),
|
| 716 |
+
("TiB" if binary else "TB"),
|
| 717 |
+
("PiB" if binary else "PB"),
|
| 718 |
+
("EiB" if binary else "EB"),
|
| 719 |
+
("ZiB" if binary else "ZB"),
|
| 720 |
+
("YiB" if binary else "YB"),
|
| 721 |
+
]
|
| 722 |
+
|
| 723 |
+
if bytes == 1:
|
| 724 |
+
return "1 Byte"
|
| 725 |
+
elif bytes < base:
|
| 726 |
+
return f"{int(bytes)} Bytes"
|
| 727 |
+
else:
|
| 728 |
+
for i, prefix in enumerate(prefixes):
|
| 729 |
+
unit = base ** (i + 2)
|
| 730 |
+
|
| 731 |
+
if bytes < unit:
|
| 732 |
+
return f"{base * bytes / unit:.1f} {prefix}"
|
| 733 |
+
|
| 734 |
+
return f"{base * bytes / unit:.1f} {prefix}"
|
| 735 |
+
|
| 736 |
+
|
| 737 |
+
def do_pprint(value: t.Any) -> str:
|
| 738 |
+
"""Pretty print a variable. Useful for debugging."""
|
| 739 |
+
return pformat(value)
|
| 740 |
+
|
| 741 |
+
|
| 742 |
+
_uri_scheme_re = re.compile(r"^([\w.+-]{2,}:(/){0,2})$")
|
| 743 |
+
|
| 744 |
+
|
| 745 |
+
@pass_eval_context
|
| 746 |
+
def do_urlize(
|
| 747 |
+
eval_ctx: "EvalContext",
|
| 748 |
+
value: str,
|
| 749 |
+
trim_url_limit: t.Optional[int] = None,
|
| 750 |
+
nofollow: bool = False,
|
| 751 |
+
target: t.Optional[str] = None,
|
| 752 |
+
rel: t.Optional[str] = None,
|
| 753 |
+
extra_schemes: t.Optional[t.Iterable[str]] = None,
|
| 754 |
+
) -> str:
|
| 755 |
+
"""Convert URLs in text into clickable links.
|
| 756 |
+
|
| 757 |
+
This may not recognize links in some situations. Usually, a more
|
| 758 |
+
comprehensive formatter, such as a Markdown library, is a better
|
| 759 |
+
choice.
|
| 760 |
+
|
| 761 |
+
Works on ``http://``, ``https://``, ``www.``, ``mailto:``, and email
|
| 762 |
+
addresses. Links with trailing punctuation (periods, commas, closing
|
| 763 |
+
parentheses) and leading punctuation (opening parentheses) are
|
| 764 |
+
recognized excluding the punctuation. Email addresses that include
|
| 765 |
+
header fields are not recognized (for example,
|
| 766 |
+
``mailto:address@example.com?cc=copy@example.com``).
|
| 767 |
+
|
| 768 |
+
:param value: Original text containing URLs to link.
|
| 769 |
+
:param trim_url_limit: Shorten displayed URL values to this length.
|
| 770 |
+
:param nofollow: Add the ``rel=nofollow`` attribute to links.
|
| 771 |
+
:param target: Add the ``target`` attribute to links.
|
| 772 |
+
:param rel: Add the ``rel`` attribute to links.
|
| 773 |
+
:param extra_schemes: Recognize URLs that start with these schemes
|
| 774 |
+
in addition to the default behavior. Defaults to
|
| 775 |
+
``env.policies["urlize.extra_schemes"]``, which defaults to no
|
| 776 |
+
extra schemes.
|
| 777 |
+
|
| 778 |
+
.. versionchanged:: 3.0
|
| 779 |
+
The ``extra_schemes`` parameter was added.
|
| 780 |
+
|
| 781 |
+
.. versionchanged:: 3.0
|
| 782 |
+
Generate ``https://`` links for URLs without a scheme.
|
| 783 |
+
|
| 784 |
+
.. versionchanged:: 3.0
|
| 785 |
+
The parsing rules were updated. Recognize email addresses with
|
| 786 |
+
or without the ``mailto:`` scheme. Validate IP addresses. Ignore
|
| 787 |
+
parentheses and brackets in more cases.
|
| 788 |
+
|
| 789 |
+
.. versionchanged:: 2.8
|
| 790 |
+
The ``target`` parameter was added.
|
| 791 |
+
"""
|
| 792 |
+
policies = eval_ctx.environment.policies
|
| 793 |
+
rel_parts = set((rel or "").split())
|
| 794 |
+
|
| 795 |
+
if nofollow:
|
| 796 |
+
rel_parts.add("nofollow")
|
| 797 |
+
|
| 798 |
+
rel_parts.update((policies["urlize.rel"] or "").split())
|
| 799 |
+
rel = " ".join(sorted(rel_parts)) or None
|
| 800 |
+
|
| 801 |
+
if target is None:
|
| 802 |
+
target = policies["urlize.target"]
|
| 803 |
+
|
| 804 |
+
if extra_schemes is None:
|
| 805 |
+
extra_schemes = policies["urlize.extra_schemes"] or ()
|
| 806 |
+
|
| 807 |
+
for scheme in extra_schemes:
|
| 808 |
+
if _uri_scheme_re.fullmatch(scheme) is None:
|
| 809 |
+
raise FilterArgumentError(f"{scheme!r} is not a valid URI scheme prefix.")
|
| 810 |
+
|
| 811 |
+
rv = urlize(
|
| 812 |
+
value,
|
| 813 |
+
trim_url_limit=trim_url_limit,
|
| 814 |
+
rel=rel,
|
| 815 |
+
target=target,
|
| 816 |
+
extra_schemes=extra_schemes,
|
| 817 |
+
)
|
| 818 |
+
|
| 819 |
+
if eval_ctx.autoescape:
|
| 820 |
+
rv = Markup(rv)
|
| 821 |
+
|
| 822 |
+
return rv
|
| 823 |
+
|
| 824 |
+
|
| 825 |
+
def do_indent(
|
| 826 |
+
s: str, width: t.Union[int, str] = 4, first: bool = False, blank: bool = False
|
| 827 |
+
) -> str:
|
| 828 |
+
"""Return a copy of the string with each line indented by 4 spaces. The
|
| 829 |
+
first line and blank lines are not indented by default.
|
| 830 |
+
|
| 831 |
+
:param width: Number of spaces, or a string, to indent by.
|
| 832 |
+
:param first: Don't skip indenting the first line.
|
| 833 |
+
:param blank: Don't skip indenting empty lines.
|
| 834 |
+
|
| 835 |
+
.. versionchanged:: 3.0
|
| 836 |
+
``width`` can be a string.
|
| 837 |
+
|
| 838 |
+
.. versionchanged:: 2.10
|
| 839 |
+
Blank lines are not indented by default.
|
| 840 |
+
|
| 841 |
+
Rename the ``indentfirst`` argument to ``first``.
|
| 842 |
+
"""
|
| 843 |
+
if isinstance(width, str):
|
| 844 |
+
indention = width
|
| 845 |
+
else:
|
| 846 |
+
indention = " " * width
|
| 847 |
+
|
| 848 |
+
newline = "\n"
|
| 849 |
+
|
| 850 |
+
if isinstance(s, Markup):
|
| 851 |
+
indention = Markup(indention)
|
| 852 |
+
newline = Markup(newline)
|
| 853 |
+
|
| 854 |
+
s += newline # this quirk is necessary for splitlines method
|
| 855 |
+
|
| 856 |
+
if blank:
|
| 857 |
+
rv = (newline + indention).join(s.splitlines())
|
| 858 |
+
else:
|
| 859 |
+
lines = s.splitlines()
|
| 860 |
+
rv = lines.pop(0)
|
| 861 |
+
|
| 862 |
+
if lines:
|
| 863 |
+
rv += newline + newline.join(
|
| 864 |
+
indention + line if line else line for line in lines
|
| 865 |
+
)
|
| 866 |
+
|
| 867 |
+
if first:
|
| 868 |
+
rv = indention + rv
|
| 869 |
+
|
| 870 |
+
return rv
|
| 871 |
+
|
| 872 |
+
|
| 873 |
+
@pass_environment
|
| 874 |
+
def do_truncate(
|
| 875 |
+
env: "Environment",
|
| 876 |
+
s: str,
|
| 877 |
+
length: int = 255,
|
| 878 |
+
killwords: bool = False,
|
| 879 |
+
end: str = "...",
|
| 880 |
+
leeway: t.Optional[int] = None,
|
| 881 |
+
) -> str:
|
| 882 |
+
"""Return a truncated copy of the string. The length is specified
|
| 883 |
+
with the first parameter which defaults to ``255``. If the second
|
| 884 |
+
parameter is ``true`` the filter will cut the text at length. Otherwise
|
| 885 |
+
it will discard the last word. If the text was in fact
|
| 886 |
+
truncated it will append an ellipsis sign (``"..."``). If you want a
|
| 887 |
+
different ellipsis sign than ``"..."`` you can specify it using the
|
| 888 |
+
third parameter. Strings that only exceed the length by the tolerance
|
| 889 |
+
margin given in the fourth parameter will not be truncated.
|
| 890 |
+
|
| 891 |
+
.. sourcecode:: jinja
|
| 892 |
+
|
| 893 |
+
{{ "foo bar baz qux"|truncate(9) }}
|
| 894 |
+
-> "foo..."
|
| 895 |
+
{{ "foo bar baz qux"|truncate(9, True) }}
|
| 896 |
+
-> "foo ba..."
|
| 897 |
+
{{ "foo bar baz qux"|truncate(11) }}
|
| 898 |
+
-> "foo bar baz qux"
|
| 899 |
+
{{ "foo bar baz qux"|truncate(11, False, '...', 0) }}
|
| 900 |
+
-> "foo bar..."
|
| 901 |
+
|
| 902 |
+
The default leeway on newer Jinja versions is 5 and was 0 before but
|
| 903 |
+
can be reconfigured globally.
|
| 904 |
+
"""
|
| 905 |
+
if leeway is None:
|
| 906 |
+
leeway = env.policies["truncate.leeway"]
|
| 907 |
+
|
| 908 |
+
assert length >= len(end), f"expected length >= {len(end)}, got {length}"
|
| 909 |
+
assert leeway >= 0, f"expected leeway >= 0, got {leeway}"
|
| 910 |
+
|
| 911 |
+
if len(s) <= length + leeway:
|
| 912 |
+
return s
|
| 913 |
+
|
| 914 |
+
if killwords:
|
| 915 |
+
return s[: length - len(end)] + end
|
| 916 |
+
|
| 917 |
+
result = s[: length - len(end)].rsplit(" ", 1)[0]
|
| 918 |
+
return result + end
|
| 919 |
+
|
| 920 |
+
|
| 921 |
+
@pass_environment
|
| 922 |
+
def do_wordwrap(
|
| 923 |
+
environment: "Environment",
|
| 924 |
+
s: str,
|
| 925 |
+
width: int = 79,
|
| 926 |
+
break_long_words: bool = True,
|
| 927 |
+
wrapstring: t.Optional[str] = None,
|
| 928 |
+
break_on_hyphens: bool = True,
|
| 929 |
+
) -> str:
|
| 930 |
+
"""Wrap a string to the given width. Existing newlines are treated
|
| 931 |
+
as paragraphs to be wrapped separately.
|
| 932 |
+
|
| 933 |
+
:param s: Original text to wrap.
|
| 934 |
+
:param width: Maximum length of wrapped lines.
|
| 935 |
+
:param break_long_words: If a word is longer than ``width``, break
|
| 936 |
+
it across lines.
|
| 937 |
+
:param break_on_hyphens: If a word contains hyphens, it may be split
|
| 938 |
+
across lines.
|
| 939 |
+
:param wrapstring: String to join each wrapped line. Defaults to
|
| 940 |
+
:attr:`Environment.newline_sequence`.
|
| 941 |
+
|
| 942 |
+
.. versionchanged:: 2.11
|
| 943 |
+
Existing newlines are treated as paragraphs wrapped separately.
|
| 944 |
+
|
| 945 |
+
.. versionchanged:: 2.11
|
| 946 |
+
Added the ``break_on_hyphens`` parameter.
|
| 947 |
+
|
| 948 |
+
.. versionchanged:: 2.7
|
| 949 |
+
Added the ``wrapstring`` parameter.
|
| 950 |
+
"""
|
| 951 |
+
import textwrap
|
| 952 |
+
|
| 953 |
+
if wrapstring is None:
|
| 954 |
+
wrapstring = environment.newline_sequence
|
| 955 |
+
|
| 956 |
+
# textwrap.wrap doesn't consider existing newlines when wrapping.
|
| 957 |
+
# If the string has a newline before width, wrap will still insert
|
| 958 |
+
# a newline at width, resulting in a short line. Instead, split and
|
| 959 |
+
# wrap each paragraph individually.
|
| 960 |
+
return wrapstring.join(
|
| 961 |
+
[
|
| 962 |
+
wrapstring.join(
|
| 963 |
+
textwrap.wrap(
|
| 964 |
+
line,
|
| 965 |
+
width=width,
|
| 966 |
+
expand_tabs=False,
|
| 967 |
+
replace_whitespace=False,
|
| 968 |
+
break_long_words=break_long_words,
|
| 969 |
+
break_on_hyphens=break_on_hyphens,
|
| 970 |
+
)
|
| 971 |
+
)
|
| 972 |
+
for line in s.splitlines()
|
| 973 |
+
]
|
| 974 |
+
)
|
| 975 |
+
|
| 976 |
+
|
| 977 |
+
_word_re = re.compile(r"\w+")
|
| 978 |
+
|
| 979 |
+
|
| 980 |
+
def do_wordcount(s: str) -> int:
|
| 981 |
+
"""Count the words in that string."""
|
| 982 |
+
return len(_word_re.findall(soft_str(s)))
|
| 983 |
+
|
| 984 |
+
|
| 985 |
+
def do_int(value: t.Any, default: int = 0, base: int = 10) -> int:
|
| 986 |
+
"""Convert the value into an integer. If the
|
| 987 |
+
conversion doesn't work it will return ``0``. You can
|
| 988 |
+
override this default using the first parameter. You
|
| 989 |
+
can also override the default base (10) in the second
|
| 990 |
+
parameter, which handles input with prefixes such as
|
| 991 |
+
0b, 0o and 0x for bases 2, 8 and 16 respectively.
|
| 992 |
+
The base is ignored for decimal numbers and non-string values.
|
| 993 |
+
"""
|
| 994 |
+
try:
|
| 995 |
+
if isinstance(value, str):
|
| 996 |
+
return int(value, base)
|
| 997 |
+
|
| 998 |
+
return int(value)
|
| 999 |
+
except (TypeError, ValueError):
|
| 1000 |
+
# this quirk is necessary so that "42.23"|int gives 42.
|
| 1001 |
+
try:
|
| 1002 |
+
return int(float(value))
|
| 1003 |
+
except (TypeError, ValueError, OverflowError):
|
| 1004 |
+
return default
|
| 1005 |
+
|
| 1006 |
+
|
| 1007 |
+
def do_float(value: t.Any, default: float = 0.0) -> float:
|
| 1008 |
+
"""Convert the value into a floating point number. If the
|
| 1009 |
+
conversion doesn't work it will return ``0.0``. You can
|
| 1010 |
+
override this default using the first parameter.
|
| 1011 |
+
"""
|
| 1012 |
+
try:
|
| 1013 |
+
return float(value)
|
| 1014 |
+
except (TypeError, ValueError):
|
| 1015 |
+
return default
|
| 1016 |
+
|
| 1017 |
+
|
| 1018 |
+
def do_format(value: str, *args: t.Any, **kwargs: t.Any) -> str:
|
| 1019 |
+
"""Apply the given values to a `printf-style`_ format string, like
|
| 1020 |
+
``string % values``.
|
| 1021 |
+
|
| 1022 |
+
.. sourcecode:: jinja
|
| 1023 |
+
|
| 1024 |
+
{{ "%s, %s!"|format(greeting, name) }}
|
| 1025 |
+
Hello, World!
|
| 1026 |
+
|
| 1027 |
+
In most cases it should be more convenient and efficient to use the
|
| 1028 |
+
``%`` operator or :meth:`str.format`.
|
| 1029 |
+
|
| 1030 |
+
.. code-block:: text
|
| 1031 |
+
|
| 1032 |
+
{{ "%s, %s!" % (greeting, name) }}
|
| 1033 |
+
{{ "{}, {}!".format(greeting, name) }}
|
| 1034 |
+
|
| 1035 |
+
.. _printf-style: https://docs.python.org/library/stdtypes.html
|
| 1036 |
+
#printf-style-string-formatting
|
| 1037 |
+
"""
|
| 1038 |
+
if args and kwargs:
|
| 1039 |
+
raise FilterArgumentError(
|
| 1040 |
+
"can't handle positional and keyword arguments at the same time"
|
| 1041 |
+
)
|
| 1042 |
+
|
| 1043 |
+
return soft_str(value) % (kwargs or args)
|
| 1044 |
+
|
| 1045 |
+
|
| 1046 |
+
def do_trim(value: str, chars: t.Optional[str] = None) -> str:
|
| 1047 |
+
"""Strip leading and trailing characters, by default whitespace."""
|
| 1048 |
+
return soft_str(value).strip(chars)
|
| 1049 |
+
|
| 1050 |
+
|
| 1051 |
+
def do_striptags(value: "t.Union[str, HasHTML]") -> str:
|
| 1052 |
+
"""Strip SGML/XML tags and replace adjacent whitespace by one space."""
|
| 1053 |
+
if hasattr(value, "__html__"):
|
| 1054 |
+
value = t.cast("HasHTML", value).__html__()
|
| 1055 |
+
|
| 1056 |
+
return Markup(str(value)).striptags()
|
| 1057 |
+
|
| 1058 |
+
|
| 1059 |
+
def sync_do_slice(
|
| 1060 |
+
value: "t.Collection[V]", slices: int, fill_with: "t.Optional[V]" = None
|
| 1061 |
+
) -> "t.Iterator[t.List[V]]":
|
| 1062 |
+
"""Slice an iterator and return a list of lists containing
|
| 1063 |
+
those items. Useful if you want to create a div containing
|
| 1064 |
+
three ul tags that represent columns:
|
| 1065 |
+
|
| 1066 |
+
.. sourcecode:: html+jinja
|
| 1067 |
+
|
| 1068 |
+
<div class="columnwrapper">
|
| 1069 |
+
{%- for column in items|slice(3) %}
|
| 1070 |
+
<ul class="column-{{ loop.index }}">
|
| 1071 |
+
{%- for item in column %}
|
| 1072 |
+
<li>{{ item }}</li>
|
| 1073 |
+
{%- endfor %}
|
| 1074 |
+
</ul>
|
| 1075 |
+
{%- endfor %}
|
| 1076 |
+
</div>
|
| 1077 |
+
|
| 1078 |
+
If you pass it a second argument it's used to fill missing
|
| 1079 |
+
values on the last iteration.
|
| 1080 |
+
"""
|
| 1081 |
+
seq = list(value)
|
| 1082 |
+
length = len(seq)
|
| 1083 |
+
items_per_slice = length // slices
|
| 1084 |
+
slices_with_extra = length % slices
|
| 1085 |
+
offset = 0
|
| 1086 |
+
|
| 1087 |
+
for slice_number in range(slices):
|
| 1088 |
+
start = offset + slice_number * items_per_slice
|
| 1089 |
+
|
| 1090 |
+
if slice_number < slices_with_extra:
|
| 1091 |
+
offset += 1
|
| 1092 |
+
|
| 1093 |
+
end = offset + (slice_number + 1) * items_per_slice
|
| 1094 |
+
tmp = seq[start:end]
|
| 1095 |
+
|
| 1096 |
+
if fill_with is not None and slice_number >= slices_with_extra:
|
| 1097 |
+
tmp.append(fill_with)
|
| 1098 |
+
|
| 1099 |
+
yield tmp
|
| 1100 |
+
|
| 1101 |
+
|
| 1102 |
+
@async_variant(sync_do_slice) # type: ignore
|
| 1103 |
+
async def do_slice(
|
| 1104 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1105 |
+
slices: int,
|
| 1106 |
+
fill_with: t.Optional[t.Any] = None,
|
| 1107 |
+
) -> "t.Iterator[t.List[V]]":
|
| 1108 |
+
return sync_do_slice(await auto_to_list(value), slices, fill_with)
|
| 1109 |
+
|
| 1110 |
+
|
| 1111 |
+
def do_batch(
|
| 1112 |
+
value: "t.Iterable[V]", linecount: int, fill_with: "t.Optional[V]" = None
|
| 1113 |
+
) -> "t.Iterator[t.List[V]]":
|
| 1114 |
+
"""
|
| 1115 |
+
A filter that batches items. It works pretty much like `slice`
|
| 1116 |
+
just the other way round. It returns a list of lists with the
|
| 1117 |
+
given number of items. If you provide a second parameter this
|
| 1118 |
+
is used to fill up missing items. See this example:
|
| 1119 |
+
|
| 1120 |
+
.. sourcecode:: html+jinja
|
| 1121 |
+
|
| 1122 |
+
<table>
|
| 1123 |
+
{%- for row in items|batch(3, ' ') %}
|
| 1124 |
+
<tr>
|
| 1125 |
+
{%- for column in row %}
|
| 1126 |
+
<td>{{ column }}</td>
|
| 1127 |
+
{%- endfor %}
|
| 1128 |
+
</tr>
|
| 1129 |
+
{%- endfor %}
|
| 1130 |
+
</table>
|
| 1131 |
+
"""
|
| 1132 |
+
tmp: t.List[V] = []
|
| 1133 |
+
|
| 1134 |
+
for item in value:
|
| 1135 |
+
if len(tmp) == linecount:
|
| 1136 |
+
yield tmp
|
| 1137 |
+
tmp = []
|
| 1138 |
+
|
| 1139 |
+
tmp.append(item)
|
| 1140 |
+
|
| 1141 |
+
if tmp:
|
| 1142 |
+
if fill_with is not None and len(tmp) < linecount:
|
| 1143 |
+
tmp += [fill_with] * (linecount - len(tmp))
|
| 1144 |
+
|
| 1145 |
+
yield tmp
|
| 1146 |
+
|
| 1147 |
+
|
| 1148 |
+
def do_round(
|
| 1149 |
+
value: float,
|
| 1150 |
+
precision: int = 0,
|
| 1151 |
+
method: 'te.Literal["common", "ceil", "floor"]' = "common",
|
| 1152 |
+
) -> float:
|
| 1153 |
+
"""Round the number to a given precision. The first
|
| 1154 |
+
parameter specifies the precision (default is ``0``), the
|
| 1155 |
+
second the rounding method:
|
| 1156 |
+
|
| 1157 |
+
- ``'common'`` rounds either up or down
|
| 1158 |
+
- ``'ceil'`` always rounds up
|
| 1159 |
+
- ``'floor'`` always rounds down
|
| 1160 |
+
|
| 1161 |
+
If you don't specify a method ``'common'`` is used.
|
| 1162 |
+
|
| 1163 |
+
.. sourcecode:: jinja
|
| 1164 |
+
|
| 1165 |
+
{{ 42.55|round }}
|
| 1166 |
+
-> 43.0
|
| 1167 |
+
{{ 42.55|round(1, 'floor') }}
|
| 1168 |
+
-> 42.5
|
| 1169 |
+
|
| 1170 |
+
Note that even if rounded to 0 precision, a float is returned. If
|
| 1171 |
+
you need a real integer, pipe it through `int`:
|
| 1172 |
+
|
| 1173 |
+
.. sourcecode:: jinja
|
| 1174 |
+
|
| 1175 |
+
{{ 42.55|round|int }}
|
| 1176 |
+
-> 43
|
| 1177 |
+
"""
|
| 1178 |
+
if method not in {"common", "ceil", "floor"}:
|
| 1179 |
+
raise FilterArgumentError("method must be common, ceil or floor")
|
| 1180 |
+
|
| 1181 |
+
if method == "common":
|
| 1182 |
+
return round(value, precision)
|
| 1183 |
+
|
| 1184 |
+
func = getattr(math, method)
|
| 1185 |
+
return t.cast(float, func(value * (10**precision)) / (10**precision))
|
| 1186 |
+
|
| 1187 |
+
|
| 1188 |
+
class _GroupTuple(t.NamedTuple):
|
| 1189 |
+
grouper: t.Any
|
| 1190 |
+
list: t.List[t.Any]
|
| 1191 |
+
|
| 1192 |
+
# Use the regular tuple repr to hide this subclass if users print
|
| 1193 |
+
# out the value during debugging.
|
| 1194 |
+
def __repr__(self) -> str:
|
| 1195 |
+
return tuple.__repr__(self)
|
| 1196 |
+
|
| 1197 |
+
def __str__(self) -> str:
|
| 1198 |
+
return tuple.__str__(self)
|
| 1199 |
+
|
| 1200 |
+
|
| 1201 |
+
@pass_environment
|
| 1202 |
+
def sync_do_groupby(
|
| 1203 |
+
environment: "Environment",
|
| 1204 |
+
value: "t.Iterable[V]",
|
| 1205 |
+
attribute: t.Union[str, int],
|
| 1206 |
+
default: t.Optional[t.Any] = None,
|
| 1207 |
+
case_sensitive: bool = False,
|
| 1208 |
+
) -> "t.List[_GroupTuple]":
|
| 1209 |
+
"""Group a sequence of objects by an attribute using Python's
|
| 1210 |
+
:func:`itertools.groupby`. The attribute can use dot notation for
|
| 1211 |
+
nested access, like ``"address.city"``. Unlike Python's ``groupby``,
|
| 1212 |
+
the values are sorted first so only one group is returned for each
|
| 1213 |
+
unique value.
|
| 1214 |
+
|
| 1215 |
+
For example, a list of ``User`` objects with a ``city`` attribute
|
| 1216 |
+
can be rendered in groups. In this example, ``grouper`` refers to
|
| 1217 |
+
the ``city`` value of the group.
|
| 1218 |
+
|
| 1219 |
+
.. sourcecode:: html+jinja
|
| 1220 |
+
|
| 1221 |
+
<ul>{% for city, items in users|groupby("city") %}
|
| 1222 |
+
<li>{{ city }}
|
| 1223 |
+
<ul>{% for user in items %}
|
| 1224 |
+
<li>{{ user.name }}
|
| 1225 |
+
{% endfor %}</ul>
|
| 1226 |
+
</li>
|
| 1227 |
+
{% endfor %}</ul>
|
| 1228 |
+
|
| 1229 |
+
``groupby`` yields namedtuples of ``(grouper, list)``, which
|
| 1230 |
+
can be used instead of the tuple unpacking above. ``grouper`` is the
|
| 1231 |
+
value of the attribute, and ``list`` is the items with that value.
|
| 1232 |
+
|
| 1233 |
+
.. sourcecode:: html+jinja
|
| 1234 |
+
|
| 1235 |
+
<ul>{% for group in users|groupby("city") %}
|
| 1236 |
+
<li>{{ group.grouper }}: {{ group.list|join(", ") }}
|
| 1237 |
+
{% endfor %}</ul>
|
| 1238 |
+
|
| 1239 |
+
You can specify a ``default`` value to use if an object in the list
|
| 1240 |
+
does not have the given attribute.
|
| 1241 |
+
|
| 1242 |
+
.. sourcecode:: jinja
|
| 1243 |
+
|
| 1244 |
+
<ul>{% for city, items in users|groupby("city", default="NY") %}
|
| 1245 |
+
<li>{{ city }}: {{ items|map(attribute="name")|join(", ") }}</li>
|
| 1246 |
+
{% endfor %}</ul>
|
| 1247 |
+
|
| 1248 |
+
Like the :func:`~jinja-filters.sort` filter, sorting and grouping is
|
| 1249 |
+
case-insensitive by default. The ``key`` for each group will have
|
| 1250 |
+
the case of the first item in that group of values. For example, if
|
| 1251 |
+
a list of users has cities ``["CA", "NY", "ca"]``, the "CA" group
|
| 1252 |
+
will have two values. This can be disabled by passing
|
| 1253 |
+
``case_sensitive=True``.
|
| 1254 |
+
|
| 1255 |
+
.. versionchanged:: 3.1
|
| 1256 |
+
Added the ``case_sensitive`` parameter. Sorting and grouping is
|
| 1257 |
+
case-insensitive by default, matching other filters that do
|
| 1258 |
+
comparisons.
|
| 1259 |
+
|
| 1260 |
+
.. versionchanged:: 3.0
|
| 1261 |
+
Added the ``default`` parameter.
|
| 1262 |
+
|
| 1263 |
+
.. versionchanged:: 2.6
|
| 1264 |
+
The attribute supports dot notation for nested access.
|
| 1265 |
+
"""
|
| 1266 |
+
expr = make_attrgetter(
|
| 1267 |
+
environment,
|
| 1268 |
+
attribute,
|
| 1269 |
+
postprocess=ignore_case if not case_sensitive else None,
|
| 1270 |
+
default=default,
|
| 1271 |
+
)
|
| 1272 |
+
out = [
|
| 1273 |
+
_GroupTuple(key, list(values))
|
| 1274 |
+
for key, values in groupby(sorted(value, key=expr), expr)
|
| 1275 |
+
]
|
| 1276 |
+
|
| 1277 |
+
if not case_sensitive:
|
| 1278 |
+
# Return the real key from the first value instead of the lowercase key.
|
| 1279 |
+
output_expr = make_attrgetter(environment, attribute, default=default)
|
| 1280 |
+
out = [_GroupTuple(output_expr(values[0]), values) for _, values in out]
|
| 1281 |
+
|
| 1282 |
+
return out
|
| 1283 |
+
|
| 1284 |
+
|
| 1285 |
+
@async_variant(sync_do_groupby) # type: ignore
|
| 1286 |
+
async def do_groupby(
|
| 1287 |
+
environment: "Environment",
|
| 1288 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1289 |
+
attribute: t.Union[str, int],
|
| 1290 |
+
default: t.Optional[t.Any] = None,
|
| 1291 |
+
case_sensitive: bool = False,
|
| 1292 |
+
) -> "t.List[_GroupTuple]":
|
| 1293 |
+
expr = make_attrgetter(
|
| 1294 |
+
environment,
|
| 1295 |
+
attribute,
|
| 1296 |
+
postprocess=ignore_case if not case_sensitive else None,
|
| 1297 |
+
default=default,
|
| 1298 |
+
)
|
| 1299 |
+
out = [
|
| 1300 |
+
_GroupTuple(key, await auto_to_list(values))
|
| 1301 |
+
for key, values in groupby(sorted(await auto_to_list(value), key=expr), expr)
|
| 1302 |
+
]
|
| 1303 |
+
|
| 1304 |
+
if not case_sensitive:
|
| 1305 |
+
# Return the real key from the first value instead of the lowercase key.
|
| 1306 |
+
output_expr = make_attrgetter(environment, attribute, default=default)
|
| 1307 |
+
out = [_GroupTuple(output_expr(values[0]), values) for _, values in out]
|
| 1308 |
+
|
| 1309 |
+
return out
|
| 1310 |
+
|
| 1311 |
+
|
| 1312 |
+
@pass_environment
|
| 1313 |
+
def sync_do_sum(
|
| 1314 |
+
environment: "Environment",
|
| 1315 |
+
iterable: "t.Iterable[V]",
|
| 1316 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 1317 |
+
start: V = 0, # type: ignore
|
| 1318 |
+
) -> V:
|
| 1319 |
+
"""Returns the sum of a sequence of numbers plus the value of parameter
|
| 1320 |
+
'start' (which defaults to 0). When the sequence is empty it returns
|
| 1321 |
+
start.
|
| 1322 |
+
|
| 1323 |
+
It is also possible to sum up only certain attributes:
|
| 1324 |
+
|
| 1325 |
+
.. sourcecode:: jinja
|
| 1326 |
+
|
| 1327 |
+
Total: {{ items|sum(attribute='price') }}
|
| 1328 |
+
|
| 1329 |
+
.. versionchanged:: 2.6
|
| 1330 |
+
The ``attribute`` parameter was added to allow summing up over
|
| 1331 |
+
attributes. Also the ``start`` parameter was moved on to the right.
|
| 1332 |
+
"""
|
| 1333 |
+
if attribute is not None:
|
| 1334 |
+
iterable = map(make_attrgetter(environment, attribute), iterable)
|
| 1335 |
+
|
| 1336 |
+
return sum(iterable, start) # type: ignore[no-any-return, call-overload]
|
| 1337 |
+
|
| 1338 |
+
|
| 1339 |
+
@async_variant(sync_do_sum) # type: ignore
|
| 1340 |
+
async def do_sum(
|
| 1341 |
+
environment: "Environment",
|
| 1342 |
+
iterable: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1343 |
+
attribute: t.Optional[t.Union[str, int]] = None,
|
| 1344 |
+
start: V = 0, # type: ignore
|
| 1345 |
+
) -> V:
|
| 1346 |
+
rv = start
|
| 1347 |
+
|
| 1348 |
+
if attribute is not None:
|
| 1349 |
+
func = make_attrgetter(environment, attribute)
|
| 1350 |
+
else:
|
| 1351 |
+
|
| 1352 |
+
def func(x: V) -> V:
|
| 1353 |
+
return x
|
| 1354 |
+
|
| 1355 |
+
async for item in auto_aiter(iterable):
|
| 1356 |
+
rv += func(item)
|
| 1357 |
+
|
| 1358 |
+
return rv
|
| 1359 |
+
|
| 1360 |
+
|
| 1361 |
+
def sync_do_list(value: "t.Iterable[V]") -> "t.List[V]":
|
| 1362 |
+
"""Convert the value into a list. If it was a string the returned list
|
| 1363 |
+
will be a list of characters.
|
| 1364 |
+
"""
|
| 1365 |
+
return list(value)
|
| 1366 |
+
|
| 1367 |
+
|
| 1368 |
+
@async_variant(sync_do_list) # type: ignore
|
| 1369 |
+
async def do_list(value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]") -> "t.List[V]":
|
| 1370 |
+
return await auto_to_list(value)
|
| 1371 |
+
|
| 1372 |
+
|
| 1373 |
+
def do_mark_safe(value: str) -> Markup:
|
| 1374 |
+
"""Mark the value as safe which means that in an environment with automatic
|
| 1375 |
+
escaping enabled this variable will not be escaped.
|
| 1376 |
+
"""
|
| 1377 |
+
return Markup(value)
|
| 1378 |
+
|
| 1379 |
+
|
| 1380 |
+
def do_mark_unsafe(value: str) -> str:
|
| 1381 |
+
"""Mark a value as unsafe. This is the reverse operation for :func:`safe`."""
|
| 1382 |
+
return str(value)
|
| 1383 |
+
|
| 1384 |
+
|
| 1385 |
+
@typing.overload
|
| 1386 |
+
def do_reverse(value: str) -> str: ...
|
| 1387 |
+
|
| 1388 |
+
|
| 1389 |
+
@typing.overload
|
| 1390 |
+
def do_reverse(value: "t.Iterable[V]") -> "t.Iterable[V]": ...
|
| 1391 |
+
|
| 1392 |
+
|
| 1393 |
+
def do_reverse(value: t.Union[str, t.Iterable[V]]) -> t.Union[str, t.Iterable[V]]:
|
| 1394 |
+
"""Reverse the object or return an iterator that iterates over it the other
|
| 1395 |
+
way round.
|
| 1396 |
+
"""
|
| 1397 |
+
if isinstance(value, str):
|
| 1398 |
+
return value[::-1]
|
| 1399 |
+
|
| 1400 |
+
try:
|
| 1401 |
+
return reversed(value) # type: ignore
|
| 1402 |
+
except TypeError:
|
| 1403 |
+
try:
|
| 1404 |
+
rv = list(value)
|
| 1405 |
+
rv.reverse()
|
| 1406 |
+
return rv
|
| 1407 |
+
except TypeError as e:
|
| 1408 |
+
raise FilterArgumentError("argument must be iterable") from e
|
| 1409 |
+
|
| 1410 |
+
|
| 1411 |
+
@pass_environment
|
| 1412 |
+
def do_attr(
|
| 1413 |
+
environment: "Environment", obj: t.Any, name: str
|
| 1414 |
+
) -> t.Union[Undefined, t.Any]:
|
| 1415 |
+
"""Get an attribute of an object. ``foo|attr("bar")`` works like
|
| 1416 |
+
``foo.bar``, but returns undefined instead of falling back to ``foo["bar"]``
|
| 1417 |
+
if the attribute doesn't exist.
|
| 1418 |
+
|
| 1419 |
+
See :ref:`Notes on subscriptions <notes-on-subscriptions>` for more details.
|
| 1420 |
+
"""
|
| 1421 |
+
# Environment.getattr will fall back to obj[name] if obj.name doesn't exist.
|
| 1422 |
+
# But we want to call env.getattr to get behavior such as sandboxing.
|
| 1423 |
+
# Determine if the attr exists first, so we know the fallback won't trigger.
|
| 1424 |
+
try:
|
| 1425 |
+
# This avoids executing properties/descriptors, but misses __getattr__
|
| 1426 |
+
# and __getattribute__ dynamic attrs.
|
| 1427 |
+
getattr_static(obj, name)
|
| 1428 |
+
except AttributeError:
|
| 1429 |
+
# This finds dynamic attrs, and we know it's not a descriptor at this point.
|
| 1430 |
+
if not hasattr(obj, name):
|
| 1431 |
+
return environment.undefined(obj=obj, name=name)
|
| 1432 |
+
|
| 1433 |
+
return environment.getattr(obj, name)
|
| 1434 |
+
|
| 1435 |
+
|
| 1436 |
+
@typing.overload
|
| 1437 |
+
def sync_do_map(
|
| 1438 |
+
context: "Context",
|
| 1439 |
+
value: t.Iterable[t.Any],
|
| 1440 |
+
name: str,
|
| 1441 |
+
*args: t.Any,
|
| 1442 |
+
**kwargs: t.Any,
|
| 1443 |
+
) -> t.Iterable[t.Any]: ...
|
| 1444 |
+
|
| 1445 |
+
|
| 1446 |
+
@typing.overload
|
| 1447 |
+
def sync_do_map(
|
| 1448 |
+
context: "Context",
|
| 1449 |
+
value: t.Iterable[t.Any],
|
| 1450 |
+
*,
|
| 1451 |
+
attribute: str = ...,
|
| 1452 |
+
default: t.Optional[t.Any] = None,
|
| 1453 |
+
) -> t.Iterable[t.Any]: ...
|
| 1454 |
+
|
| 1455 |
+
|
| 1456 |
+
@pass_context
|
| 1457 |
+
def sync_do_map(
|
| 1458 |
+
context: "Context", value: t.Iterable[t.Any], *args: t.Any, **kwargs: t.Any
|
| 1459 |
+
) -> t.Iterable[t.Any]:
|
| 1460 |
+
"""Applies a filter on a sequence of objects or looks up an attribute.
|
| 1461 |
+
This is useful when dealing with lists of objects but you are really
|
| 1462 |
+
only interested in a certain value of it.
|
| 1463 |
+
|
| 1464 |
+
The basic usage is mapping on an attribute. Imagine you have a list
|
| 1465 |
+
of users but you are only interested in a list of usernames:
|
| 1466 |
+
|
| 1467 |
+
.. sourcecode:: jinja
|
| 1468 |
+
|
| 1469 |
+
Users on this page: {{ users|map(attribute='username')|join(', ') }}
|
| 1470 |
+
|
| 1471 |
+
You can specify a ``default`` value to use if an object in the list
|
| 1472 |
+
does not have the given attribute.
|
| 1473 |
+
|
| 1474 |
+
.. sourcecode:: jinja
|
| 1475 |
+
|
| 1476 |
+
{{ users|map(attribute="username", default="Anonymous")|join(", ") }}
|
| 1477 |
+
|
| 1478 |
+
Alternatively you can let it invoke a filter by passing the name of the
|
| 1479 |
+
filter and the arguments afterwards. A good example would be applying a
|
| 1480 |
+
text conversion filter on a sequence:
|
| 1481 |
+
|
| 1482 |
+
.. sourcecode:: jinja
|
| 1483 |
+
|
| 1484 |
+
Users on this page: {{ titles|map('lower')|join(', ') }}
|
| 1485 |
+
|
| 1486 |
+
Similar to a generator comprehension such as:
|
| 1487 |
+
|
| 1488 |
+
.. code-block:: python
|
| 1489 |
+
|
| 1490 |
+
(u.username for u in users)
|
| 1491 |
+
(getattr(u, "username", "Anonymous") for u in users)
|
| 1492 |
+
(do_lower(x) for x in titles)
|
| 1493 |
+
|
| 1494 |
+
.. versionchanged:: 2.11.0
|
| 1495 |
+
Added the ``default`` parameter.
|
| 1496 |
+
|
| 1497 |
+
.. versionadded:: 2.7
|
| 1498 |
+
"""
|
| 1499 |
+
if value:
|
| 1500 |
+
func = prepare_map(context, args, kwargs)
|
| 1501 |
+
|
| 1502 |
+
for item in value:
|
| 1503 |
+
yield func(item)
|
| 1504 |
+
|
| 1505 |
+
|
| 1506 |
+
@typing.overload
|
| 1507 |
+
def do_map(
|
| 1508 |
+
context: "Context",
|
| 1509 |
+
value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
|
| 1510 |
+
name: str,
|
| 1511 |
+
*args: t.Any,
|
| 1512 |
+
**kwargs: t.Any,
|
| 1513 |
+
) -> t.Iterable[t.Any]: ...
|
| 1514 |
+
|
| 1515 |
+
|
| 1516 |
+
@typing.overload
|
| 1517 |
+
def do_map(
|
| 1518 |
+
context: "Context",
|
| 1519 |
+
value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
|
| 1520 |
+
*,
|
| 1521 |
+
attribute: str = ...,
|
| 1522 |
+
default: t.Optional[t.Any] = None,
|
| 1523 |
+
) -> t.Iterable[t.Any]: ...
|
| 1524 |
+
|
| 1525 |
+
|
| 1526 |
+
@async_variant(sync_do_map) # type: ignore
|
| 1527 |
+
async def do_map(
|
| 1528 |
+
context: "Context",
|
| 1529 |
+
value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
|
| 1530 |
+
*args: t.Any,
|
| 1531 |
+
**kwargs: t.Any,
|
| 1532 |
+
) -> t.AsyncIterable[t.Any]:
|
| 1533 |
+
if value:
|
| 1534 |
+
func = prepare_map(context, args, kwargs)
|
| 1535 |
+
|
| 1536 |
+
async for item in auto_aiter(value):
|
| 1537 |
+
yield await auto_await(func(item))
|
| 1538 |
+
|
| 1539 |
+
|
| 1540 |
+
@pass_context
|
| 1541 |
+
def sync_do_select(
|
| 1542 |
+
context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
|
| 1543 |
+
) -> "t.Iterator[V]":
|
| 1544 |
+
"""Filters a sequence of objects by applying a test to each object,
|
| 1545 |
+
and only selecting the objects with the test succeeding.
|
| 1546 |
+
|
| 1547 |
+
If no test is specified, each object will be evaluated as a boolean.
|
| 1548 |
+
|
| 1549 |
+
Example usage:
|
| 1550 |
+
|
| 1551 |
+
.. sourcecode:: jinja
|
| 1552 |
+
|
| 1553 |
+
{{ numbers|select("odd") }}
|
| 1554 |
+
{{ numbers|select("odd") }}
|
| 1555 |
+
{{ numbers|select("divisibleby", 3) }}
|
| 1556 |
+
{{ numbers|select("lessthan", 42) }}
|
| 1557 |
+
{{ strings|select("equalto", "mystring") }}
|
| 1558 |
+
|
| 1559 |
+
Similar to a generator comprehension such as:
|
| 1560 |
+
|
| 1561 |
+
.. code-block:: python
|
| 1562 |
+
|
| 1563 |
+
(n for n in numbers if test_odd(n))
|
| 1564 |
+
(n for n in numbers if test_divisibleby(n, 3))
|
| 1565 |
+
|
| 1566 |
+
.. versionadded:: 2.7
|
| 1567 |
+
"""
|
| 1568 |
+
return select_or_reject(context, value, args, kwargs, lambda x: x, False)
|
| 1569 |
+
|
| 1570 |
+
|
| 1571 |
+
@async_variant(sync_do_select) # type: ignore
|
| 1572 |
+
async def do_select(
|
| 1573 |
+
context: "Context",
|
| 1574 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1575 |
+
*args: t.Any,
|
| 1576 |
+
**kwargs: t.Any,
|
| 1577 |
+
) -> "t.AsyncIterator[V]":
|
| 1578 |
+
return async_select_or_reject(context, value, args, kwargs, lambda x: x, False)
|
| 1579 |
+
|
| 1580 |
+
|
| 1581 |
+
@pass_context
|
| 1582 |
+
def sync_do_reject(
|
| 1583 |
+
context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
|
| 1584 |
+
) -> "t.Iterator[V]":
|
| 1585 |
+
"""Filters a sequence of objects by applying a test to each object,
|
| 1586 |
+
and rejecting the objects with the test succeeding.
|
| 1587 |
+
|
| 1588 |
+
If no test is specified, each object will be evaluated as a boolean.
|
| 1589 |
+
|
| 1590 |
+
Example usage:
|
| 1591 |
+
|
| 1592 |
+
.. sourcecode:: jinja
|
| 1593 |
+
|
| 1594 |
+
{{ numbers|reject("odd") }}
|
| 1595 |
+
|
| 1596 |
+
Similar to a generator comprehension such as:
|
| 1597 |
+
|
| 1598 |
+
.. code-block:: python
|
| 1599 |
+
|
| 1600 |
+
(n for n in numbers if not test_odd(n))
|
| 1601 |
+
|
| 1602 |
+
.. versionadded:: 2.7
|
| 1603 |
+
"""
|
| 1604 |
+
return select_or_reject(context, value, args, kwargs, lambda x: not x, False)
|
| 1605 |
+
|
| 1606 |
+
|
| 1607 |
+
@async_variant(sync_do_reject) # type: ignore
|
| 1608 |
+
async def do_reject(
|
| 1609 |
+
context: "Context",
|
| 1610 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1611 |
+
*args: t.Any,
|
| 1612 |
+
**kwargs: t.Any,
|
| 1613 |
+
) -> "t.AsyncIterator[V]":
|
| 1614 |
+
return async_select_or_reject(context, value, args, kwargs, lambda x: not x, False)
|
| 1615 |
+
|
| 1616 |
+
|
| 1617 |
+
@pass_context
|
| 1618 |
+
def sync_do_selectattr(
|
| 1619 |
+
context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
|
| 1620 |
+
) -> "t.Iterator[V]":
|
| 1621 |
+
"""Filters a sequence of objects by applying a test to the specified
|
| 1622 |
+
attribute of each object, and only selecting the objects with the
|
| 1623 |
+
test succeeding.
|
| 1624 |
+
|
| 1625 |
+
If no test is specified, the attribute's value will be evaluated as
|
| 1626 |
+
a boolean.
|
| 1627 |
+
|
| 1628 |
+
Example usage:
|
| 1629 |
+
|
| 1630 |
+
.. sourcecode:: jinja
|
| 1631 |
+
|
| 1632 |
+
{{ users|selectattr("is_active") }}
|
| 1633 |
+
{{ users|selectattr("email", "none") }}
|
| 1634 |
+
|
| 1635 |
+
Similar to a generator comprehension such as:
|
| 1636 |
+
|
| 1637 |
+
.. code-block:: python
|
| 1638 |
+
|
| 1639 |
+
(user for user in users if user.is_active)
|
| 1640 |
+
(user for user in users if test_none(user.email))
|
| 1641 |
+
|
| 1642 |
+
.. versionadded:: 2.7
|
| 1643 |
+
"""
|
| 1644 |
+
return select_or_reject(context, value, args, kwargs, lambda x: x, True)
|
| 1645 |
+
|
| 1646 |
+
|
| 1647 |
+
@async_variant(sync_do_selectattr) # type: ignore
|
| 1648 |
+
async def do_selectattr(
|
| 1649 |
+
context: "Context",
|
| 1650 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1651 |
+
*args: t.Any,
|
| 1652 |
+
**kwargs: t.Any,
|
| 1653 |
+
) -> "t.AsyncIterator[V]":
|
| 1654 |
+
return async_select_or_reject(context, value, args, kwargs, lambda x: x, True)
|
| 1655 |
+
|
| 1656 |
+
|
| 1657 |
+
@pass_context
|
| 1658 |
+
def sync_do_rejectattr(
|
| 1659 |
+
context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
|
| 1660 |
+
) -> "t.Iterator[V]":
|
| 1661 |
+
"""Filters a sequence of objects by applying a test to the specified
|
| 1662 |
+
attribute of each object, and rejecting the objects with the test
|
| 1663 |
+
succeeding.
|
| 1664 |
+
|
| 1665 |
+
If no test is specified, the attribute's value will be evaluated as
|
| 1666 |
+
a boolean.
|
| 1667 |
+
|
| 1668 |
+
.. sourcecode:: jinja
|
| 1669 |
+
|
| 1670 |
+
{{ users|rejectattr("is_active") }}
|
| 1671 |
+
{{ users|rejectattr("email", "none") }}
|
| 1672 |
+
|
| 1673 |
+
Similar to a generator comprehension such as:
|
| 1674 |
+
|
| 1675 |
+
.. code-block:: python
|
| 1676 |
+
|
| 1677 |
+
(user for user in users if not user.is_active)
|
| 1678 |
+
(user for user in users if not test_none(user.email))
|
| 1679 |
+
|
| 1680 |
+
.. versionadded:: 2.7
|
| 1681 |
+
"""
|
| 1682 |
+
return select_or_reject(context, value, args, kwargs, lambda x: not x, True)
|
| 1683 |
+
|
| 1684 |
+
|
| 1685 |
+
@async_variant(sync_do_rejectattr) # type: ignore
|
| 1686 |
+
async def do_rejectattr(
|
| 1687 |
+
context: "Context",
|
| 1688 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1689 |
+
*args: t.Any,
|
| 1690 |
+
**kwargs: t.Any,
|
| 1691 |
+
) -> "t.AsyncIterator[V]":
|
| 1692 |
+
return async_select_or_reject(context, value, args, kwargs, lambda x: not x, True)
|
| 1693 |
+
|
| 1694 |
+
|
| 1695 |
+
@pass_eval_context
|
| 1696 |
+
def do_tojson(
|
| 1697 |
+
eval_ctx: "EvalContext", value: t.Any, indent: t.Optional[int] = None
|
| 1698 |
+
) -> Markup:
|
| 1699 |
+
"""Serialize an object to a string of JSON, and mark it safe to
|
| 1700 |
+
render in HTML. This filter is only for use in HTML documents.
|
| 1701 |
+
|
| 1702 |
+
The returned string is safe to render in HTML documents and
|
| 1703 |
+
``<script>`` tags. The exception is in HTML attributes that are
|
| 1704 |
+
double quoted; either use single quotes or the ``|forceescape``
|
| 1705 |
+
filter.
|
| 1706 |
+
|
| 1707 |
+
:param value: The object to serialize to JSON.
|
| 1708 |
+
:param indent: The ``indent`` parameter passed to ``dumps``, for
|
| 1709 |
+
pretty-printing the value.
|
| 1710 |
+
|
| 1711 |
+
.. versionadded:: 2.9
|
| 1712 |
+
"""
|
| 1713 |
+
policies = eval_ctx.environment.policies
|
| 1714 |
+
dumps = policies["json.dumps_function"]
|
| 1715 |
+
kwargs = policies["json.dumps_kwargs"]
|
| 1716 |
+
|
| 1717 |
+
if indent is not None:
|
| 1718 |
+
kwargs = kwargs.copy()
|
| 1719 |
+
kwargs["indent"] = indent
|
| 1720 |
+
|
| 1721 |
+
return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)
|
| 1722 |
+
|
| 1723 |
+
|
| 1724 |
+
def prepare_map(
|
| 1725 |
+
context: "Context", args: t.Tuple[t.Any, ...], kwargs: t.Dict[str, t.Any]
|
| 1726 |
+
) -> t.Callable[[t.Any], t.Any]:
|
| 1727 |
+
if not args and "attribute" in kwargs:
|
| 1728 |
+
attribute = kwargs.pop("attribute")
|
| 1729 |
+
default = kwargs.pop("default", None)
|
| 1730 |
+
|
| 1731 |
+
if kwargs:
|
| 1732 |
+
raise FilterArgumentError(
|
| 1733 |
+
f"Unexpected keyword argument {next(iter(kwargs))!r}"
|
| 1734 |
+
)
|
| 1735 |
+
|
| 1736 |
+
func = make_attrgetter(context.environment, attribute, default=default)
|
| 1737 |
+
else:
|
| 1738 |
+
try:
|
| 1739 |
+
name = args[0]
|
| 1740 |
+
args = args[1:]
|
| 1741 |
+
except LookupError:
|
| 1742 |
+
raise FilterArgumentError("map requires a filter argument") from None
|
| 1743 |
+
|
| 1744 |
+
def func(item: t.Any) -> t.Any:
|
| 1745 |
+
return context.environment.call_filter(
|
| 1746 |
+
name, item, args, kwargs, context=context
|
| 1747 |
+
)
|
| 1748 |
+
|
| 1749 |
+
return func
|
| 1750 |
+
|
| 1751 |
+
|
| 1752 |
+
def prepare_select_or_reject(
|
| 1753 |
+
context: "Context",
|
| 1754 |
+
args: t.Tuple[t.Any, ...],
|
| 1755 |
+
kwargs: t.Dict[str, t.Any],
|
| 1756 |
+
modfunc: t.Callable[[t.Any], t.Any],
|
| 1757 |
+
lookup_attr: bool,
|
| 1758 |
+
) -> t.Callable[[t.Any], t.Any]:
|
| 1759 |
+
if lookup_attr:
|
| 1760 |
+
try:
|
| 1761 |
+
attr = args[0]
|
| 1762 |
+
except LookupError:
|
| 1763 |
+
raise FilterArgumentError("Missing parameter for attribute name") from None
|
| 1764 |
+
|
| 1765 |
+
transfunc = make_attrgetter(context.environment, attr)
|
| 1766 |
+
off = 1
|
| 1767 |
+
else:
|
| 1768 |
+
off = 0
|
| 1769 |
+
|
| 1770 |
+
def transfunc(x: V) -> V:
|
| 1771 |
+
return x
|
| 1772 |
+
|
| 1773 |
+
try:
|
| 1774 |
+
name = args[off]
|
| 1775 |
+
args = args[1 + off :]
|
| 1776 |
+
|
| 1777 |
+
def func(item: t.Any) -> t.Any:
|
| 1778 |
+
return context.environment.call_test(name, item, args, kwargs, context)
|
| 1779 |
+
|
| 1780 |
+
except LookupError:
|
| 1781 |
+
func = bool # type: ignore
|
| 1782 |
+
|
| 1783 |
+
return lambda item: modfunc(func(transfunc(item)))
|
| 1784 |
+
|
| 1785 |
+
|
| 1786 |
+
def select_or_reject(
|
| 1787 |
+
context: "Context",
|
| 1788 |
+
value: "t.Iterable[V]",
|
| 1789 |
+
args: t.Tuple[t.Any, ...],
|
| 1790 |
+
kwargs: t.Dict[str, t.Any],
|
| 1791 |
+
modfunc: t.Callable[[t.Any], t.Any],
|
| 1792 |
+
lookup_attr: bool,
|
| 1793 |
+
) -> "t.Iterator[V]":
|
| 1794 |
+
if value:
|
| 1795 |
+
func = prepare_select_or_reject(context, args, kwargs, modfunc, lookup_attr)
|
| 1796 |
+
|
| 1797 |
+
for item in value:
|
| 1798 |
+
if func(item):
|
| 1799 |
+
yield item
|
| 1800 |
+
|
| 1801 |
+
|
| 1802 |
+
async def async_select_or_reject(
|
| 1803 |
+
context: "Context",
|
| 1804 |
+
value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
|
| 1805 |
+
args: t.Tuple[t.Any, ...],
|
| 1806 |
+
kwargs: t.Dict[str, t.Any],
|
| 1807 |
+
modfunc: t.Callable[[t.Any], t.Any],
|
| 1808 |
+
lookup_attr: bool,
|
| 1809 |
+
) -> "t.AsyncIterator[V]":
|
| 1810 |
+
if value:
|
| 1811 |
+
func = prepare_select_or_reject(context, args, kwargs, modfunc, lookup_attr)
|
| 1812 |
+
|
| 1813 |
+
async for item in auto_aiter(value):
|
| 1814 |
+
if func(item):
|
| 1815 |
+
yield item
|
| 1816 |
+
|
| 1817 |
+
|
| 1818 |
+
FILTERS = {
|
| 1819 |
+
"abs": abs,
|
| 1820 |
+
"attr": do_attr,
|
| 1821 |
+
"batch": do_batch,
|
| 1822 |
+
"capitalize": do_capitalize,
|
| 1823 |
+
"center": do_center,
|
| 1824 |
+
"count": len,
|
| 1825 |
+
"d": do_default,
|
| 1826 |
+
"default": do_default,
|
| 1827 |
+
"dictsort": do_dictsort,
|
| 1828 |
+
"e": escape,
|
| 1829 |
+
"escape": escape,
|
| 1830 |
+
"filesizeformat": do_filesizeformat,
|
| 1831 |
+
"first": do_first,
|
| 1832 |
+
"float": do_float,
|
| 1833 |
+
"forceescape": do_forceescape,
|
| 1834 |
+
"format": do_format,
|
| 1835 |
+
"groupby": do_groupby,
|
| 1836 |
+
"indent": do_indent,
|
| 1837 |
+
"int": do_int,
|
| 1838 |
+
"join": do_join,
|
| 1839 |
+
"last": do_last,
|
| 1840 |
+
"length": len,
|
| 1841 |
+
"list": do_list,
|
| 1842 |
+
"lower": do_lower,
|
| 1843 |
+
"items": do_items,
|
| 1844 |
+
"map": do_map,
|
| 1845 |
+
"min": do_min,
|
| 1846 |
+
"max": do_max,
|
| 1847 |
+
"pprint": do_pprint,
|
| 1848 |
+
"random": do_random,
|
| 1849 |
+
"reject": do_reject,
|
| 1850 |
+
"rejectattr": do_rejectattr,
|
| 1851 |
+
"replace": do_replace,
|
| 1852 |
+
"reverse": do_reverse,
|
| 1853 |
+
"round": do_round,
|
| 1854 |
+
"safe": do_mark_safe,
|
| 1855 |
+
"select": do_select,
|
| 1856 |
+
"selectattr": do_selectattr,
|
| 1857 |
+
"slice": do_slice,
|
| 1858 |
+
"sort": do_sort,
|
| 1859 |
+
"string": soft_str,
|
| 1860 |
+
"striptags": do_striptags,
|
| 1861 |
+
"sum": do_sum,
|
| 1862 |
+
"title": do_title,
|
| 1863 |
+
"trim": do_trim,
|
| 1864 |
+
"truncate": do_truncate,
|
| 1865 |
+
"unique": do_unique,
|
| 1866 |
+
"upper": do_upper,
|
| 1867 |
+
"urlencode": do_urlencode,
|
| 1868 |
+
"urlize": do_urlize,
|
| 1869 |
+
"wordcount": do_wordcount,
|
| 1870 |
+
"wordwrap": do_wordwrap,
|
| 1871 |
+
"xmlattr": do_xmlattr,
|
| 1872 |
+
"tojson": do_tojson,
|
| 1873 |
+
}
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/loaders.py
ADDED
|
@@ -0,0 +1,693 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""API and implementations for loading templates from different data
|
| 2 |
+
sources.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import importlib.util
|
| 6 |
+
import os
|
| 7 |
+
import posixpath
|
| 8 |
+
import sys
|
| 9 |
+
import typing as t
|
| 10 |
+
import weakref
|
| 11 |
+
import zipimport
|
| 12 |
+
from collections import abc
|
| 13 |
+
from hashlib import sha1
|
| 14 |
+
from importlib import import_module
|
| 15 |
+
from types import ModuleType
|
| 16 |
+
|
| 17 |
+
from .exceptions import TemplateNotFound
|
| 18 |
+
from .utils import internalcode
|
| 19 |
+
|
| 20 |
+
if t.TYPE_CHECKING:
|
| 21 |
+
from .environment import Environment
|
| 22 |
+
from .environment import Template
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def split_template_path(template: str) -> t.List[str]:
|
| 26 |
+
"""Split a path into segments and perform a sanity check. If it detects
|
| 27 |
+
'..' in the path it will raise a `TemplateNotFound` error.
|
| 28 |
+
"""
|
| 29 |
+
pieces = []
|
| 30 |
+
for piece in template.split("/"):
|
| 31 |
+
if (
|
| 32 |
+
os.path.sep in piece
|
| 33 |
+
or (os.path.altsep and os.path.altsep in piece)
|
| 34 |
+
or piece == os.path.pardir
|
| 35 |
+
):
|
| 36 |
+
raise TemplateNotFound(template)
|
| 37 |
+
elif piece and piece != ".":
|
| 38 |
+
pieces.append(piece)
|
| 39 |
+
return pieces
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
class BaseLoader:
|
| 43 |
+
"""Baseclass for all loaders. Subclass this and override `get_source` to
|
| 44 |
+
implement a custom loading mechanism. The environment provides a
|
| 45 |
+
`get_template` method that calls the loader's `load` method to get the
|
| 46 |
+
:class:`Template` object.
|
| 47 |
+
|
| 48 |
+
A very basic example for a loader that looks up templates on the file
|
| 49 |
+
system could look like this::
|
| 50 |
+
|
| 51 |
+
from jinja2 import BaseLoader, TemplateNotFound
|
| 52 |
+
from os.path import join, exists, getmtime
|
| 53 |
+
|
| 54 |
+
class MyLoader(BaseLoader):
|
| 55 |
+
|
| 56 |
+
def __init__(self, path):
|
| 57 |
+
self.path = path
|
| 58 |
+
|
| 59 |
+
def get_source(self, environment, template):
|
| 60 |
+
path = join(self.path, template)
|
| 61 |
+
if not exists(path):
|
| 62 |
+
raise TemplateNotFound(template)
|
| 63 |
+
mtime = getmtime(path)
|
| 64 |
+
with open(path) as f:
|
| 65 |
+
source = f.read()
|
| 66 |
+
return source, path, lambda: mtime == getmtime(path)
|
| 67 |
+
"""
|
| 68 |
+
|
| 69 |
+
#: if set to `False` it indicates that the loader cannot provide access
|
| 70 |
+
#: to the source of templates.
|
| 71 |
+
#:
|
| 72 |
+
#: .. versionadded:: 2.4
|
| 73 |
+
has_source_access = True
|
| 74 |
+
|
| 75 |
+
def get_source(
|
| 76 |
+
self, environment: "Environment", template: str
|
| 77 |
+
) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
|
| 78 |
+
"""Get the template source, filename and reload helper for a template.
|
| 79 |
+
It's passed the environment and template name and has to return a
|
| 80 |
+
tuple in the form ``(source, filename, uptodate)`` or raise a
|
| 81 |
+
`TemplateNotFound` error if it can't locate the template.
|
| 82 |
+
|
| 83 |
+
The source part of the returned tuple must be the source of the
|
| 84 |
+
template as a string. The filename should be the name of the
|
| 85 |
+
file on the filesystem if it was loaded from there, otherwise
|
| 86 |
+
``None``. The filename is used by Python for the tracebacks
|
| 87 |
+
if no loader extension is used.
|
| 88 |
+
|
| 89 |
+
The last item in the tuple is the `uptodate` function. If auto
|
| 90 |
+
reloading is enabled it's always called to check if the template
|
| 91 |
+
changed. No arguments are passed so the function must store the
|
| 92 |
+
old state somewhere (for example in a closure). If it returns `False`
|
| 93 |
+
the template will be reloaded.
|
| 94 |
+
"""
|
| 95 |
+
if not self.has_source_access:
|
| 96 |
+
raise RuntimeError(
|
| 97 |
+
f"{type(self).__name__} cannot provide access to the source"
|
| 98 |
+
)
|
| 99 |
+
raise TemplateNotFound(template)
|
| 100 |
+
|
| 101 |
+
def list_templates(self) -> t.List[str]:
|
| 102 |
+
"""Iterates over all templates. If the loader does not support that
|
| 103 |
+
it should raise a :exc:`TypeError` which is the default behavior.
|
| 104 |
+
"""
|
| 105 |
+
raise TypeError("this loader cannot iterate over all templates")
|
| 106 |
+
|
| 107 |
+
@internalcode
|
| 108 |
+
def load(
|
| 109 |
+
self,
|
| 110 |
+
environment: "Environment",
|
| 111 |
+
name: str,
|
| 112 |
+
globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
|
| 113 |
+
) -> "Template":
|
| 114 |
+
"""Loads a template. This method looks up the template in the cache
|
| 115 |
+
or loads one by calling :meth:`get_source`. Subclasses should not
|
| 116 |
+
override this method as loaders working on collections of other
|
| 117 |
+
loaders (such as :class:`PrefixLoader` or :class:`ChoiceLoader`)
|
| 118 |
+
will not call this method but `get_source` directly.
|
| 119 |
+
"""
|
| 120 |
+
code = None
|
| 121 |
+
if globals is None:
|
| 122 |
+
globals = {}
|
| 123 |
+
|
| 124 |
+
# first we try to get the source for this template together
|
| 125 |
+
# with the filename and the uptodate function.
|
| 126 |
+
source, filename, uptodate = self.get_source(environment, name)
|
| 127 |
+
|
| 128 |
+
# try to load the code from the bytecode cache if there is a
|
| 129 |
+
# bytecode cache configured.
|
| 130 |
+
bcc = environment.bytecode_cache
|
| 131 |
+
if bcc is not None:
|
| 132 |
+
bucket = bcc.get_bucket(environment, name, filename, source)
|
| 133 |
+
code = bucket.code
|
| 134 |
+
|
| 135 |
+
# if we don't have code so far (not cached, no longer up to
|
| 136 |
+
# date) etc. we compile the template
|
| 137 |
+
if code is None:
|
| 138 |
+
code = environment.compile(source, name, filename)
|
| 139 |
+
|
| 140 |
+
# if the bytecode cache is available and the bucket doesn't
|
| 141 |
+
# have a code so far, we give the bucket the new code and put
|
| 142 |
+
# it back to the bytecode cache.
|
| 143 |
+
if bcc is not None and bucket.code is None:
|
| 144 |
+
bucket.code = code
|
| 145 |
+
bcc.set_bucket(bucket)
|
| 146 |
+
|
| 147 |
+
return environment.template_class.from_code(
|
| 148 |
+
environment, code, globals, uptodate
|
| 149 |
+
)
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
class FileSystemLoader(BaseLoader):
|
| 153 |
+
"""Load templates from a directory in the file system.
|
| 154 |
+
|
| 155 |
+
The path can be relative or absolute. Relative paths are relative to
|
| 156 |
+
the current working directory.
|
| 157 |
+
|
| 158 |
+
.. code-block:: python
|
| 159 |
+
|
| 160 |
+
loader = FileSystemLoader("templates")
|
| 161 |
+
|
| 162 |
+
A list of paths can be given. The directories will be searched in
|
| 163 |
+
order, stopping at the first matching template.
|
| 164 |
+
|
| 165 |
+
.. code-block:: python
|
| 166 |
+
|
| 167 |
+
loader = FileSystemLoader(["/override/templates", "/default/templates"])
|
| 168 |
+
|
| 169 |
+
:param searchpath: A path, or list of paths, to the directory that
|
| 170 |
+
contains the templates.
|
| 171 |
+
:param encoding: Use this encoding to read the text from template
|
| 172 |
+
files.
|
| 173 |
+
:param followlinks: Follow symbolic links in the path.
|
| 174 |
+
|
| 175 |
+
.. versionchanged:: 2.8
|
| 176 |
+
Added the ``followlinks`` parameter.
|
| 177 |
+
"""
|
| 178 |
+
|
| 179 |
+
def __init__(
|
| 180 |
+
self,
|
| 181 |
+
searchpath: t.Union[
|
| 182 |
+
str, "os.PathLike[str]", t.Sequence[t.Union[str, "os.PathLike[str]"]]
|
| 183 |
+
],
|
| 184 |
+
encoding: str = "utf-8",
|
| 185 |
+
followlinks: bool = False,
|
| 186 |
+
) -> None:
|
| 187 |
+
if not isinstance(searchpath, abc.Iterable) or isinstance(searchpath, str):
|
| 188 |
+
searchpath = [searchpath]
|
| 189 |
+
|
| 190 |
+
self.searchpath = [os.fspath(p) for p in searchpath]
|
| 191 |
+
self.encoding = encoding
|
| 192 |
+
self.followlinks = followlinks
|
| 193 |
+
|
| 194 |
+
def get_source(
|
| 195 |
+
self, environment: "Environment", template: str
|
| 196 |
+
) -> t.Tuple[str, str, t.Callable[[], bool]]:
|
| 197 |
+
pieces = split_template_path(template)
|
| 198 |
+
|
| 199 |
+
for searchpath in self.searchpath:
|
| 200 |
+
# Use posixpath even on Windows to avoid "drive:" or UNC
|
| 201 |
+
# segments breaking out of the search directory.
|
| 202 |
+
filename = posixpath.join(searchpath, *pieces)
|
| 203 |
+
|
| 204 |
+
if os.path.isfile(filename):
|
| 205 |
+
break
|
| 206 |
+
else:
|
| 207 |
+
plural = "path" if len(self.searchpath) == 1 else "paths"
|
| 208 |
+
paths_str = ", ".join(repr(p) for p in self.searchpath)
|
| 209 |
+
raise TemplateNotFound(
|
| 210 |
+
template,
|
| 211 |
+
f"{template!r} not found in search {plural}: {paths_str}",
|
| 212 |
+
)
|
| 213 |
+
|
| 214 |
+
with open(filename, encoding=self.encoding) as f:
|
| 215 |
+
contents = f.read()
|
| 216 |
+
|
| 217 |
+
mtime = os.path.getmtime(filename)
|
| 218 |
+
|
| 219 |
+
def uptodate() -> bool:
|
| 220 |
+
try:
|
| 221 |
+
return os.path.getmtime(filename) == mtime
|
| 222 |
+
except OSError:
|
| 223 |
+
return False
|
| 224 |
+
|
| 225 |
+
# Use normpath to convert Windows altsep to sep.
|
| 226 |
+
return contents, os.path.normpath(filename), uptodate
|
| 227 |
+
|
| 228 |
+
def list_templates(self) -> t.List[str]:
|
| 229 |
+
found = set()
|
| 230 |
+
for searchpath in self.searchpath:
|
| 231 |
+
walk_dir = os.walk(searchpath, followlinks=self.followlinks)
|
| 232 |
+
for dirpath, _, filenames in walk_dir:
|
| 233 |
+
for filename in filenames:
|
| 234 |
+
template = (
|
| 235 |
+
os.path.join(dirpath, filename)[len(searchpath) :]
|
| 236 |
+
.strip(os.path.sep)
|
| 237 |
+
.replace(os.path.sep, "/")
|
| 238 |
+
)
|
| 239 |
+
if template[:2] == "./":
|
| 240 |
+
template = template[2:]
|
| 241 |
+
if template not in found:
|
| 242 |
+
found.add(template)
|
| 243 |
+
return sorted(found)
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
if sys.version_info >= (3, 13):
|
| 247 |
+
|
| 248 |
+
def _get_zipimporter_files(z: t.Any) -> t.Dict[str, object]:
|
| 249 |
+
try:
|
| 250 |
+
get_files = z._get_files
|
| 251 |
+
except AttributeError as e:
|
| 252 |
+
raise TypeError(
|
| 253 |
+
"This zip import does not have the required"
|
| 254 |
+
" metadata to list templates."
|
| 255 |
+
) from e
|
| 256 |
+
return get_files()
|
| 257 |
+
else:
|
| 258 |
+
|
| 259 |
+
def _get_zipimporter_files(z: t.Any) -> t.Dict[str, object]:
|
| 260 |
+
try:
|
| 261 |
+
files = z._files
|
| 262 |
+
except AttributeError as e:
|
| 263 |
+
raise TypeError(
|
| 264 |
+
"This zip import does not have the required"
|
| 265 |
+
" metadata to list templates."
|
| 266 |
+
) from e
|
| 267 |
+
return files # type: ignore[no-any-return]
|
| 268 |
+
|
| 269 |
+
|
| 270 |
+
class PackageLoader(BaseLoader):
|
| 271 |
+
"""Load templates from a directory in a Python package.
|
| 272 |
+
|
| 273 |
+
:param package_name: Import name of the package that contains the
|
| 274 |
+
template directory.
|
| 275 |
+
:param package_path: Directory within the imported package that
|
| 276 |
+
contains the templates.
|
| 277 |
+
:param encoding: Encoding of template files.
|
| 278 |
+
|
| 279 |
+
The following example looks up templates in the ``pages`` directory
|
| 280 |
+
within the ``project.ui`` package.
|
| 281 |
+
|
| 282 |
+
.. code-block:: python
|
| 283 |
+
|
| 284 |
+
loader = PackageLoader("project.ui", "pages")
|
| 285 |
+
|
| 286 |
+
Only packages installed as directories (standard pip behavior) or
|
| 287 |
+
zip/egg files (less common) are supported. The Python API for
|
| 288 |
+
introspecting data in packages is too limited to support other
|
| 289 |
+
installation methods the way this loader requires.
|
| 290 |
+
|
| 291 |
+
There is limited support for :pep:`420` namespace packages. The
|
| 292 |
+
template directory is assumed to only be in one namespace
|
| 293 |
+
contributor. Zip files contributing to a namespace are not
|
| 294 |
+
supported.
|
| 295 |
+
|
| 296 |
+
.. versionchanged:: 3.0
|
| 297 |
+
No longer uses ``setuptools`` as a dependency.
|
| 298 |
+
|
| 299 |
+
.. versionchanged:: 3.0
|
| 300 |
+
Limited PEP 420 namespace package support.
|
| 301 |
+
"""
|
| 302 |
+
|
| 303 |
+
def __init__(
|
| 304 |
+
self,
|
| 305 |
+
package_name: str,
|
| 306 |
+
package_path: "str" = "templates",
|
| 307 |
+
encoding: str = "utf-8",
|
| 308 |
+
) -> None:
|
| 309 |
+
package_path = os.path.normpath(package_path).rstrip(os.path.sep)
|
| 310 |
+
|
| 311 |
+
# normpath preserves ".", which isn't valid in zip paths.
|
| 312 |
+
if package_path == os.path.curdir:
|
| 313 |
+
package_path = ""
|
| 314 |
+
elif package_path[:2] == os.path.curdir + os.path.sep:
|
| 315 |
+
package_path = package_path[2:]
|
| 316 |
+
|
| 317 |
+
self.package_path = package_path
|
| 318 |
+
self.package_name = package_name
|
| 319 |
+
self.encoding = encoding
|
| 320 |
+
|
| 321 |
+
# Make sure the package exists. This also makes namespace
|
| 322 |
+
# packages work, otherwise get_loader returns None.
|
| 323 |
+
import_module(package_name)
|
| 324 |
+
spec = importlib.util.find_spec(package_name)
|
| 325 |
+
assert spec is not None, "An import spec was not found for the package."
|
| 326 |
+
loader = spec.loader
|
| 327 |
+
assert loader is not None, "A loader was not found for the package."
|
| 328 |
+
self._loader = loader
|
| 329 |
+
self._archive = None
|
| 330 |
+
|
| 331 |
+
if isinstance(loader, zipimport.zipimporter):
|
| 332 |
+
self._archive = loader.archive
|
| 333 |
+
pkgdir = next(iter(spec.submodule_search_locations)) # type: ignore
|
| 334 |
+
template_root = os.path.join(pkgdir, package_path).rstrip(os.path.sep)
|
| 335 |
+
else:
|
| 336 |
+
roots: t.List[str] = []
|
| 337 |
+
|
| 338 |
+
# One element for regular packages, multiple for namespace
|
| 339 |
+
# packages, or None for single module file.
|
| 340 |
+
if spec.submodule_search_locations:
|
| 341 |
+
roots.extend(spec.submodule_search_locations)
|
| 342 |
+
# A single module file, use the parent directory instead.
|
| 343 |
+
elif spec.origin is not None:
|
| 344 |
+
roots.append(os.path.dirname(spec.origin))
|
| 345 |
+
|
| 346 |
+
if not roots:
|
| 347 |
+
raise ValueError(
|
| 348 |
+
f"The {package_name!r} package was not installed in a"
|
| 349 |
+
" way that PackageLoader understands."
|
| 350 |
+
)
|
| 351 |
+
|
| 352 |
+
for root in roots:
|
| 353 |
+
root = os.path.join(root, package_path)
|
| 354 |
+
|
| 355 |
+
if os.path.isdir(root):
|
| 356 |
+
template_root = root
|
| 357 |
+
break
|
| 358 |
+
else:
|
| 359 |
+
raise ValueError(
|
| 360 |
+
f"PackageLoader could not find a {package_path!r} directory"
|
| 361 |
+
f" in the {package_name!r} package."
|
| 362 |
+
)
|
| 363 |
+
|
| 364 |
+
self._template_root = template_root
|
| 365 |
+
|
| 366 |
+
def get_source(
|
| 367 |
+
self, environment: "Environment", template: str
|
| 368 |
+
) -> t.Tuple[str, str, t.Optional[t.Callable[[], bool]]]:
|
| 369 |
+
# Use posixpath even on Windows to avoid "drive:" or UNC
|
| 370 |
+
# segments breaking out of the search directory. Use normpath to
|
| 371 |
+
# convert Windows altsep to sep.
|
| 372 |
+
p = os.path.normpath(
|
| 373 |
+
posixpath.join(self._template_root, *split_template_path(template))
|
| 374 |
+
)
|
| 375 |
+
up_to_date: t.Optional[t.Callable[[], bool]]
|
| 376 |
+
|
| 377 |
+
if self._archive is None:
|
| 378 |
+
# Package is a directory.
|
| 379 |
+
if not os.path.isfile(p):
|
| 380 |
+
raise TemplateNotFound(template)
|
| 381 |
+
|
| 382 |
+
with open(p, "rb") as f:
|
| 383 |
+
source = f.read()
|
| 384 |
+
|
| 385 |
+
mtime = os.path.getmtime(p)
|
| 386 |
+
|
| 387 |
+
def up_to_date() -> bool:
|
| 388 |
+
return os.path.isfile(p) and os.path.getmtime(p) == mtime
|
| 389 |
+
|
| 390 |
+
else:
|
| 391 |
+
# Package is a zip file.
|
| 392 |
+
try:
|
| 393 |
+
source = self._loader.get_data(p) # type: ignore
|
| 394 |
+
except OSError as e:
|
| 395 |
+
raise TemplateNotFound(template) from e
|
| 396 |
+
|
| 397 |
+
# Could use the zip's mtime for all template mtimes, but
|
| 398 |
+
# would need to safely reload the module if it's out of
|
| 399 |
+
# date, so just report it as always current.
|
| 400 |
+
up_to_date = None
|
| 401 |
+
|
| 402 |
+
return source.decode(self.encoding), p, up_to_date
|
| 403 |
+
|
| 404 |
+
def list_templates(self) -> t.List[str]:
|
| 405 |
+
results: t.List[str] = []
|
| 406 |
+
|
| 407 |
+
if self._archive is None:
|
| 408 |
+
# Package is a directory.
|
| 409 |
+
offset = len(self._template_root)
|
| 410 |
+
|
| 411 |
+
for dirpath, _, filenames in os.walk(self._template_root):
|
| 412 |
+
dirpath = dirpath[offset:].lstrip(os.path.sep)
|
| 413 |
+
results.extend(
|
| 414 |
+
os.path.join(dirpath, name).replace(os.path.sep, "/")
|
| 415 |
+
for name in filenames
|
| 416 |
+
)
|
| 417 |
+
else:
|
| 418 |
+
files = _get_zipimporter_files(self._loader)
|
| 419 |
+
|
| 420 |
+
# Package is a zip file.
|
| 421 |
+
prefix = (
|
| 422 |
+
self._template_root[len(self._archive) :].lstrip(os.path.sep)
|
| 423 |
+
+ os.path.sep
|
| 424 |
+
)
|
| 425 |
+
offset = len(prefix)
|
| 426 |
+
|
| 427 |
+
for name in files:
|
| 428 |
+
# Find names under the templates directory that aren't directories.
|
| 429 |
+
if name.startswith(prefix) and name[-1] != os.path.sep:
|
| 430 |
+
results.append(name[offset:].replace(os.path.sep, "/"))
|
| 431 |
+
|
| 432 |
+
results.sort()
|
| 433 |
+
return results
|
| 434 |
+
|
| 435 |
+
|
| 436 |
+
class DictLoader(BaseLoader):
|
| 437 |
+
"""Loads a template from a Python dict mapping template names to
|
| 438 |
+
template source. This loader is useful for unittesting:
|
| 439 |
+
|
| 440 |
+
>>> loader = DictLoader({'index.html': 'source here'})
|
| 441 |
+
|
| 442 |
+
Because auto reloading is rarely useful this is disabled by default.
|
| 443 |
+
"""
|
| 444 |
+
|
| 445 |
+
def __init__(self, mapping: t.Mapping[str, str]) -> None:
|
| 446 |
+
self.mapping = mapping
|
| 447 |
+
|
| 448 |
+
def get_source(
|
| 449 |
+
self, environment: "Environment", template: str
|
| 450 |
+
) -> t.Tuple[str, None, t.Callable[[], bool]]:
|
| 451 |
+
if template in self.mapping:
|
| 452 |
+
source = self.mapping[template]
|
| 453 |
+
return source, None, lambda: source == self.mapping.get(template)
|
| 454 |
+
raise TemplateNotFound(template)
|
| 455 |
+
|
| 456 |
+
def list_templates(self) -> t.List[str]:
|
| 457 |
+
return sorted(self.mapping)
|
| 458 |
+
|
| 459 |
+
|
| 460 |
+
class FunctionLoader(BaseLoader):
|
| 461 |
+
"""A loader that is passed a function which does the loading. The
|
| 462 |
+
function receives the name of the template and has to return either
|
| 463 |
+
a string with the template source, a tuple in the form ``(source,
|
| 464 |
+
filename, uptodatefunc)`` or `None` if the template does not exist.
|
| 465 |
+
|
| 466 |
+
>>> def load_template(name):
|
| 467 |
+
... if name == 'index.html':
|
| 468 |
+
... return '...'
|
| 469 |
+
...
|
| 470 |
+
>>> loader = FunctionLoader(load_template)
|
| 471 |
+
|
| 472 |
+
The `uptodatefunc` is a function that is called if autoreload is enabled
|
| 473 |
+
and has to return `True` if the template is still up to date. For more
|
| 474 |
+
details have a look at :meth:`BaseLoader.get_source` which has the same
|
| 475 |
+
return value.
|
| 476 |
+
"""
|
| 477 |
+
|
| 478 |
+
def __init__(
|
| 479 |
+
self,
|
| 480 |
+
load_func: t.Callable[
|
| 481 |
+
[str],
|
| 482 |
+
t.Optional[
|
| 483 |
+
t.Union[
|
| 484 |
+
str, t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]
|
| 485 |
+
]
|
| 486 |
+
],
|
| 487 |
+
],
|
| 488 |
+
) -> None:
|
| 489 |
+
self.load_func = load_func
|
| 490 |
+
|
| 491 |
+
def get_source(
|
| 492 |
+
self, environment: "Environment", template: str
|
| 493 |
+
) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
|
| 494 |
+
rv = self.load_func(template)
|
| 495 |
+
|
| 496 |
+
if rv is None:
|
| 497 |
+
raise TemplateNotFound(template)
|
| 498 |
+
|
| 499 |
+
if isinstance(rv, str):
|
| 500 |
+
return rv, None, None
|
| 501 |
+
|
| 502 |
+
return rv
|
| 503 |
+
|
| 504 |
+
|
| 505 |
+
class PrefixLoader(BaseLoader):
|
| 506 |
+
"""A loader that is passed a dict of loaders where each loader is bound
|
| 507 |
+
to a prefix. The prefix is delimited from the template by a slash per
|
| 508 |
+
default, which can be changed by setting the `delimiter` argument to
|
| 509 |
+
something else::
|
| 510 |
+
|
| 511 |
+
loader = PrefixLoader({
|
| 512 |
+
'app1': PackageLoader('mypackage.app1'),
|
| 513 |
+
'app2': PackageLoader('mypackage.app2')
|
| 514 |
+
})
|
| 515 |
+
|
| 516 |
+
By loading ``'app1/index.html'`` the file from the app1 package is loaded,
|
| 517 |
+
by loading ``'app2/index.html'`` the file from the second.
|
| 518 |
+
"""
|
| 519 |
+
|
| 520 |
+
def __init__(
|
| 521 |
+
self, mapping: t.Mapping[str, BaseLoader], delimiter: str = "/"
|
| 522 |
+
) -> None:
|
| 523 |
+
self.mapping = mapping
|
| 524 |
+
self.delimiter = delimiter
|
| 525 |
+
|
| 526 |
+
def get_loader(self, template: str) -> t.Tuple[BaseLoader, str]:
|
| 527 |
+
try:
|
| 528 |
+
prefix, name = template.split(self.delimiter, 1)
|
| 529 |
+
loader = self.mapping[prefix]
|
| 530 |
+
except (ValueError, KeyError) as e:
|
| 531 |
+
raise TemplateNotFound(template) from e
|
| 532 |
+
return loader, name
|
| 533 |
+
|
| 534 |
+
def get_source(
|
| 535 |
+
self, environment: "Environment", template: str
|
| 536 |
+
) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
|
| 537 |
+
loader, name = self.get_loader(template)
|
| 538 |
+
try:
|
| 539 |
+
return loader.get_source(environment, name)
|
| 540 |
+
except TemplateNotFound as e:
|
| 541 |
+
# re-raise the exception with the correct filename here.
|
| 542 |
+
# (the one that includes the prefix)
|
| 543 |
+
raise TemplateNotFound(template) from e
|
| 544 |
+
|
| 545 |
+
@internalcode
|
| 546 |
+
def load(
|
| 547 |
+
self,
|
| 548 |
+
environment: "Environment",
|
| 549 |
+
name: str,
|
| 550 |
+
globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
|
| 551 |
+
) -> "Template":
|
| 552 |
+
loader, local_name = self.get_loader(name)
|
| 553 |
+
try:
|
| 554 |
+
return loader.load(environment, local_name, globals)
|
| 555 |
+
except TemplateNotFound as e:
|
| 556 |
+
# re-raise the exception with the correct filename here.
|
| 557 |
+
# (the one that includes the prefix)
|
| 558 |
+
raise TemplateNotFound(name) from e
|
| 559 |
+
|
| 560 |
+
def list_templates(self) -> t.List[str]:
|
| 561 |
+
result = []
|
| 562 |
+
for prefix, loader in self.mapping.items():
|
| 563 |
+
for template in loader.list_templates():
|
| 564 |
+
result.append(prefix + self.delimiter + template)
|
| 565 |
+
return result
|
| 566 |
+
|
| 567 |
+
|
| 568 |
+
class ChoiceLoader(BaseLoader):
|
| 569 |
+
"""This loader works like the `PrefixLoader` just that no prefix is
|
| 570 |
+
specified. If a template could not be found by one loader the next one
|
| 571 |
+
is tried.
|
| 572 |
+
|
| 573 |
+
>>> loader = ChoiceLoader([
|
| 574 |
+
... FileSystemLoader('/path/to/user/templates'),
|
| 575 |
+
... FileSystemLoader('/path/to/system/templates')
|
| 576 |
+
... ])
|
| 577 |
+
|
| 578 |
+
This is useful if you want to allow users to override builtin templates
|
| 579 |
+
from a different location.
|
| 580 |
+
"""
|
| 581 |
+
|
| 582 |
+
def __init__(self, loaders: t.Sequence[BaseLoader]) -> None:
|
| 583 |
+
self.loaders = loaders
|
| 584 |
+
|
| 585 |
+
def get_source(
|
| 586 |
+
self, environment: "Environment", template: str
|
| 587 |
+
) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
|
| 588 |
+
for loader in self.loaders:
|
| 589 |
+
try:
|
| 590 |
+
return loader.get_source(environment, template)
|
| 591 |
+
except TemplateNotFound:
|
| 592 |
+
pass
|
| 593 |
+
raise TemplateNotFound(template)
|
| 594 |
+
|
| 595 |
+
@internalcode
|
| 596 |
+
def load(
|
| 597 |
+
self,
|
| 598 |
+
environment: "Environment",
|
| 599 |
+
name: str,
|
| 600 |
+
globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
|
| 601 |
+
) -> "Template":
|
| 602 |
+
for loader in self.loaders:
|
| 603 |
+
try:
|
| 604 |
+
return loader.load(environment, name, globals)
|
| 605 |
+
except TemplateNotFound:
|
| 606 |
+
pass
|
| 607 |
+
raise TemplateNotFound(name)
|
| 608 |
+
|
| 609 |
+
def list_templates(self) -> t.List[str]:
|
| 610 |
+
found = set()
|
| 611 |
+
for loader in self.loaders:
|
| 612 |
+
found.update(loader.list_templates())
|
| 613 |
+
return sorted(found)
|
| 614 |
+
|
| 615 |
+
|
| 616 |
+
class _TemplateModule(ModuleType):
|
| 617 |
+
"""Like a normal module but with support for weak references"""
|
| 618 |
+
|
| 619 |
+
|
| 620 |
+
class ModuleLoader(BaseLoader):
|
| 621 |
+
"""This loader loads templates from precompiled templates.
|
| 622 |
+
|
| 623 |
+
Example usage:
|
| 624 |
+
|
| 625 |
+
>>> loader = ModuleLoader('/path/to/compiled/templates')
|
| 626 |
+
|
| 627 |
+
Templates can be precompiled with :meth:`Environment.compile_templates`.
|
| 628 |
+
"""
|
| 629 |
+
|
| 630 |
+
has_source_access = False
|
| 631 |
+
|
| 632 |
+
def __init__(
|
| 633 |
+
self,
|
| 634 |
+
path: t.Union[
|
| 635 |
+
str, "os.PathLike[str]", t.Sequence[t.Union[str, "os.PathLike[str]"]]
|
| 636 |
+
],
|
| 637 |
+
) -> None:
|
| 638 |
+
package_name = f"_jinja2_module_templates_{id(self):x}"
|
| 639 |
+
|
| 640 |
+
# create a fake module that looks for the templates in the
|
| 641 |
+
# path given.
|
| 642 |
+
mod = _TemplateModule(package_name)
|
| 643 |
+
|
| 644 |
+
if not isinstance(path, abc.Iterable) or isinstance(path, str):
|
| 645 |
+
path = [path]
|
| 646 |
+
|
| 647 |
+
mod.__path__ = [os.fspath(p) for p in path]
|
| 648 |
+
|
| 649 |
+
sys.modules[package_name] = weakref.proxy(
|
| 650 |
+
mod, lambda x: sys.modules.pop(package_name, None)
|
| 651 |
+
)
|
| 652 |
+
|
| 653 |
+
# the only strong reference, the sys.modules entry is weak
|
| 654 |
+
# so that the garbage collector can remove it once the
|
| 655 |
+
# loader that created it goes out of business.
|
| 656 |
+
self.module = mod
|
| 657 |
+
self.package_name = package_name
|
| 658 |
+
|
| 659 |
+
@staticmethod
|
| 660 |
+
def get_template_key(name: str) -> str:
|
| 661 |
+
return "tmpl_" + sha1(name.encode("utf-8")).hexdigest()
|
| 662 |
+
|
| 663 |
+
@staticmethod
|
| 664 |
+
def get_module_filename(name: str) -> str:
|
| 665 |
+
return ModuleLoader.get_template_key(name) + ".py"
|
| 666 |
+
|
| 667 |
+
@internalcode
|
| 668 |
+
def load(
|
| 669 |
+
self,
|
| 670 |
+
environment: "Environment",
|
| 671 |
+
name: str,
|
| 672 |
+
globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
|
| 673 |
+
) -> "Template":
|
| 674 |
+
key = self.get_template_key(name)
|
| 675 |
+
module = f"{self.package_name}.{key}"
|
| 676 |
+
mod = getattr(self.module, module, None)
|
| 677 |
+
|
| 678 |
+
if mod is None:
|
| 679 |
+
try:
|
| 680 |
+
mod = __import__(module, None, None, ["root"])
|
| 681 |
+
except ImportError as e:
|
| 682 |
+
raise TemplateNotFound(name) from e
|
| 683 |
+
|
| 684 |
+
# remove the entry from sys.modules, we only want the attribute
|
| 685 |
+
# on the module object we have stored on the loader.
|
| 686 |
+
sys.modules.pop(module, None)
|
| 687 |
+
|
| 688 |
+
if globals is None:
|
| 689 |
+
globals = {}
|
| 690 |
+
|
| 691 |
+
return environment.template_class.from_module_dict(
|
| 692 |
+
environment, mod.__dict__, globals
|
| 693 |
+
)
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/meta.py
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Functions that expose information about templates that might be
|
| 2 |
+
interesting for introspection.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import typing as t
|
| 6 |
+
|
| 7 |
+
from . import nodes
|
| 8 |
+
from .compiler import CodeGenerator
|
| 9 |
+
from .compiler import Frame
|
| 10 |
+
|
| 11 |
+
if t.TYPE_CHECKING:
|
| 12 |
+
from .environment import Environment
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class TrackingCodeGenerator(CodeGenerator):
|
| 16 |
+
"""We abuse the code generator for introspection."""
|
| 17 |
+
|
| 18 |
+
def __init__(self, environment: "Environment") -> None:
|
| 19 |
+
super().__init__(environment, "<introspection>", "<introspection>")
|
| 20 |
+
self.undeclared_identifiers: t.Set[str] = set()
|
| 21 |
+
|
| 22 |
+
def write(self, x: str) -> None:
|
| 23 |
+
"""Don't write."""
|
| 24 |
+
|
| 25 |
+
def enter_frame(self, frame: Frame) -> None:
|
| 26 |
+
"""Remember all undeclared identifiers."""
|
| 27 |
+
super().enter_frame(frame)
|
| 28 |
+
|
| 29 |
+
for _, (action, param) in frame.symbols.loads.items():
|
| 30 |
+
if action == "resolve" and param not in self.environment.globals:
|
| 31 |
+
self.undeclared_identifiers.add(param)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def find_undeclared_variables(ast: nodes.Template) -> t.Set[str]:
|
| 35 |
+
"""Returns a set of all variables in the AST that will be looked up from
|
| 36 |
+
the context at runtime. Because at compile time it's not known which
|
| 37 |
+
variables will be used depending on the path the execution takes at
|
| 38 |
+
runtime, all variables are returned.
|
| 39 |
+
|
| 40 |
+
>>> from jinja2 import Environment, meta
|
| 41 |
+
>>> env = Environment()
|
| 42 |
+
>>> ast = env.parse('{% set foo = 42 %}{{ bar + foo }}')
|
| 43 |
+
>>> meta.find_undeclared_variables(ast) == {'bar'}
|
| 44 |
+
True
|
| 45 |
+
|
| 46 |
+
.. admonition:: Implementation
|
| 47 |
+
|
| 48 |
+
Internally the code generator is used for finding undeclared variables.
|
| 49 |
+
This is good to know because the code generator might raise a
|
| 50 |
+
:exc:`TemplateAssertionError` during compilation and as a matter of
|
| 51 |
+
fact this function can currently raise that exception as well.
|
| 52 |
+
"""
|
| 53 |
+
codegen = TrackingCodeGenerator(ast.environment) # type: ignore
|
| 54 |
+
codegen.visit(ast)
|
| 55 |
+
return codegen.undeclared_identifiers
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
_ref_types = (nodes.Extends, nodes.FromImport, nodes.Import, nodes.Include)
|
| 59 |
+
_RefType = t.Union[nodes.Extends, nodes.FromImport, nodes.Import, nodes.Include]
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def find_referenced_templates(ast: nodes.Template) -> t.Iterator[t.Optional[str]]:
|
| 63 |
+
"""Finds all the referenced templates from the AST. This will return an
|
| 64 |
+
iterator over all the hardcoded template extensions, inclusions and
|
| 65 |
+
imports. If dynamic inheritance or inclusion is used, `None` will be
|
| 66 |
+
yielded.
|
| 67 |
+
|
| 68 |
+
>>> from jinja2 import Environment, meta
|
| 69 |
+
>>> env = Environment()
|
| 70 |
+
>>> ast = env.parse('{% extends "layout.html" %}{% include helper %}')
|
| 71 |
+
>>> list(meta.find_referenced_templates(ast))
|
| 72 |
+
['layout.html', None]
|
| 73 |
+
|
| 74 |
+
This function is useful for dependency tracking. For example if you want
|
| 75 |
+
to rebuild parts of the website after a layout template has changed.
|
| 76 |
+
"""
|
| 77 |
+
template_name: t.Any
|
| 78 |
+
|
| 79 |
+
for node in ast.find_all(_ref_types):
|
| 80 |
+
template: nodes.Expr = node.template # type: ignore
|
| 81 |
+
|
| 82 |
+
if not isinstance(template, nodes.Const):
|
| 83 |
+
# a tuple with some non consts in there
|
| 84 |
+
if isinstance(template, (nodes.Tuple, nodes.List)):
|
| 85 |
+
for template_name in template.items:
|
| 86 |
+
# something const, only yield the strings and ignore
|
| 87 |
+
# non-string consts that really just make no sense
|
| 88 |
+
if isinstance(template_name, nodes.Const):
|
| 89 |
+
if isinstance(template_name.value, str):
|
| 90 |
+
yield template_name.value
|
| 91 |
+
# something dynamic in there
|
| 92 |
+
else:
|
| 93 |
+
yield None
|
| 94 |
+
# something dynamic we don't know about here
|
| 95 |
+
else:
|
| 96 |
+
yield None
|
| 97 |
+
continue
|
| 98 |
+
# constant is a basestring, direct template name
|
| 99 |
+
if isinstance(template.value, str):
|
| 100 |
+
yield template.value
|
| 101 |
+
# a tuple or list (latter *should* not happen) made of consts,
|
| 102 |
+
# yield the consts that are strings. We could warn here for
|
| 103 |
+
# non string values
|
| 104 |
+
elif isinstance(node, nodes.Include) and isinstance(
|
| 105 |
+
template.value, (tuple, list)
|
| 106 |
+
):
|
| 107 |
+
for template_name in template.value:
|
| 108 |
+
if isinstance(template_name, str):
|
| 109 |
+
yield template_name
|
| 110 |
+
# something else we don't care about, we could warn here
|
| 111 |
+
else:
|
| 112 |
+
yield None
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/nativetypes.py
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import typing as t
|
| 2 |
+
from ast import literal_eval
|
| 3 |
+
from ast import parse
|
| 4 |
+
from itertools import chain
|
| 5 |
+
from itertools import islice
|
| 6 |
+
from types import GeneratorType
|
| 7 |
+
|
| 8 |
+
from . import nodes
|
| 9 |
+
from .compiler import CodeGenerator
|
| 10 |
+
from .compiler import Frame
|
| 11 |
+
from .compiler import has_safe_repr
|
| 12 |
+
from .environment import Environment
|
| 13 |
+
from .environment import Template
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def native_concat(values: t.Iterable[t.Any]) -> t.Optional[t.Any]:
|
| 17 |
+
"""Return a native Python type from the list of compiled nodes. If
|
| 18 |
+
the result is a single node, its value is returned. Otherwise, the
|
| 19 |
+
nodes are concatenated as strings. If the result can be parsed with
|
| 20 |
+
:func:`ast.literal_eval`, the parsed value is returned. Otherwise,
|
| 21 |
+
the string is returned.
|
| 22 |
+
|
| 23 |
+
:param values: Iterable of outputs to concatenate.
|
| 24 |
+
"""
|
| 25 |
+
head = list(islice(values, 2))
|
| 26 |
+
|
| 27 |
+
if not head:
|
| 28 |
+
return None
|
| 29 |
+
|
| 30 |
+
if len(head) == 1:
|
| 31 |
+
raw = head[0]
|
| 32 |
+
if not isinstance(raw, str):
|
| 33 |
+
return raw
|
| 34 |
+
else:
|
| 35 |
+
if isinstance(values, GeneratorType):
|
| 36 |
+
values = chain(head, values)
|
| 37 |
+
raw = "".join([str(v) for v in values])
|
| 38 |
+
|
| 39 |
+
try:
|
| 40 |
+
return literal_eval(
|
| 41 |
+
# In Python 3.10+ ast.literal_eval removes leading spaces/tabs
|
| 42 |
+
# from the given string. For backwards compatibility we need to
|
| 43 |
+
# parse the string ourselves without removing leading spaces/tabs.
|
| 44 |
+
parse(raw, mode="eval")
|
| 45 |
+
)
|
| 46 |
+
except (ValueError, SyntaxError, MemoryError):
|
| 47 |
+
return raw
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
class NativeCodeGenerator(CodeGenerator):
|
| 51 |
+
"""A code generator which renders Python types by not adding
|
| 52 |
+
``str()`` around output nodes.
|
| 53 |
+
"""
|
| 54 |
+
|
| 55 |
+
@staticmethod
|
| 56 |
+
def _default_finalize(value: t.Any) -> t.Any:
|
| 57 |
+
return value
|
| 58 |
+
|
| 59 |
+
def _output_const_repr(self, group: t.Iterable[t.Any]) -> str:
|
| 60 |
+
return repr("".join([str(v) for v in group]))
|
| 61 |
+
|
| 62 |
+
def _output_child_to_const(
|
| 63 |
+
self, node: nodes.Expr, frame: Frame, finalize: CodeGenerator._FinalizeInfo
|
| 64 |
+
) -> t.Any:
|
| 65 |
+
const = node.as_const(frame.eval_ctx)
|
| 66 |
+
|
| 67 |
+
if not has_safe_repr(const):
|
| 68 |
+
raise nodes.Impossible()
|
| 69 |
+
|
| 70 |
+
if isinstance(node, nodes.TemplateData):
|
| 71 |
+
return const
|
| 72 |
+
|
| 73 |
+
return finalize.const(const) # type: ignore
|
| 74 |
+
|
| 75 |
+
def _output_child_pre(
|
| 76 |
+
self, node: nodes.Expr, frame: Frame, finalize: CodeGenerator._FinalizeInfo
|
| 77 |
+
) -> None:
|
| 78 |
+
if finalize.src is not None:
|
| 79 |
+
self.write(finalize.src)
|
| 80 |
+
|
| 81 |
+
def _output_child_post(
|
| 82 |
+
self, node: nodes.Expr, frame: Frame, finalize: CodeGenerator._FinalizeInfo
|
| 83 |
+
) -> None:
|
| 84 |
+
if finalize.src is not None:
|
| 85 |
+
self.write(")")
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
class NativeEnvironment(Environment):
|
| 89 |
+
"""An environment that renders templates to native Python types."""
|
| 90 |
+
|
| 91 |
+
code_generator_class = NativeCodeGenerator
|
| 92 |
+
concat = staticmethod(native_concat) # type: ignore
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
class NativeTemplate(Template):
|
| 96 |
+
environment_class = NativeEnvironment
|
| 97 |
+
|
| 98 |
+
def render(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
|
| 99 |
+
"""Render the template to produce a native Python type. If the
|
| 100 |
+
result is a single node, its value is returned. Otherwise, the
|
| 101 |
+
nodes are concatenated as strings. If the result can be parsed
|
| 102 |
+
with :func:`ast.literal_eval`, the parsed value is returned.
|
| 103 |
+
Otherwise, the string is returned.
|
| 104 |
+
"""
|
| 105 |
+
ctx = self.new_context(dict(*args, **kwargs))
|
| 106 |
+
|
| 107 |
+
try:
|
| 108 |
+
return self.environment_class.concat( # type: ignore
|
| 109 |
+
self.root_render_func(ctx)
|
| 110 |
+
)
|
| 111 |
+
except Exception:
|
| 112 |
+
return self.environment.handle_exception()
|
| 113 |
+
|
| 114 |
+
async def render_async(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
|
| 115 |
+
if not self.environment.is_async:
|
| 116 |
+
raise RuntimeError(
|
| 117 |
+
"The environment was not created with async mode enabled."
|
| 118 |
+
)
|
| 119 |
+
|
| 120 |
+
ctx = self.new_context(dict(*args, **kwargs))
|
| 121 |
+
|
| 122 |
+
try:
|
| 123 |
+
return self.environment_class.concat( # type: ignore
|
| 124 |
+
[n async for n in self.root_render_func(ctx)] # type: ignore
|
| 125 |
+
)
|
| 126 |
+
except Exception:
|
| 127 |
+
return self.environment.handle_exception()
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
NativeEnvironment.template_class = NativeTemplate
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/nodes.py
ADDED
|
@@ -0,0 +1,1206 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""AST nodes generated by the parser for the compiler. Also provides
|
| 2 |
+
some node tree helper functions used by the parser and compiler in order
|
| 3 |
+
to normalize nodes.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
import inspect
|
| 7 |
+
import operator
|
| 8 |
+
import typing as t
|
| 9 |
+
from collections import deque
|
| 10 |
+
|
| 11 |
+
from markupsafe import Markup
|
| 12 |
+
|
| 13 |
+
from .utils import _PassArg
|
| 14 |
+
|
| 15 |
+
if t.TYPE_CHECKING:
|
| 16 |
+
import typing_extensions as te
|
| 17 |
+
|
| 18 |
+
from .environment import Environment
|
| 19 |
+
|
| 20 |
+
_NodeBound = t.TypeVar("_NodeBound", bound="Node")
|
| 21 |
+
|
| 22 |
+
_binop_to_func: t.Dict[str, t.Callable[[t.Any, t.Any], t.Any]] = {
|
| 23 |
+
"*": operator.mul,
|
| 24 |
+
"/": operator.truediv,
|
| 25 |
+
"//": operator.floordiv,
|
| 26 |
+
"**": operator.pow,
|
| 27 |
+
"%": operator.mod,
|
| 28 |
+
"+": operator.add,
|
| 29 |
+
"-": operator.sub,
|
| 30 |
+
}
|
| 31 |
+
|
| 32 |
+
_uaop_to_func: t.Dict[str, t.Callable[[t.Any], t.Any]] = {
|
| 33 |
+
"not": operator.not_,
|
| 34 |
+
"+": operator.pos,
|
| 35 |
+
"-": operator.neg,
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
_cmpop_to_func: t.Dict[str, t.Callable[[t.Any, t.Any], t.Any]] = {
|
| 39 |
+
"eq": operator.eq,
|
| 40 |
+
"ne": operator.ne,
|
| 41 |
+
"gt": operator.gt,
|
| 42 |
+
"gteq": operator.ge,
|
| 43 |
+
"lt": operator.lt,
|
| 44 |
+
"lteq": operator.le,
|
| 45 |
+
"in": lambda a, b: a in b,
|
| 46 |
+
"notin": lambda a, b: a not in b,
|
| 47 |
+
}
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
class Impossible(Exception):
|
| 51 |
+
"""Raised if the node could not perform a requested action."""
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
class NodeType(type):
|
| 55 |
+
"""A metaclass for nodes that handles the field and attribute
|
| 56 |
+
inheritance. fields and attributes from the parent class are
|
| 57 |
+
automatically forwarded to the child."""
|
| 58 |
+
|
| 59 |
+
def __new__(mcs, name, bases, d): # type: ignore
|
| 60 |
+
for attr in "fields", "attributes":
|
| 61 |
+
storage: t.List[t.Tuple[str, ...]] = []
|
| 62 |
+
storage.extend(getattr(bases[0] if bases else object, attr, ()))
|
| 63 |
+
storage.extend(d.get(attr, ()))
|
| 64 |
+
assert len(bases) <= 1, "multiple inheritance not allowed"
|
| 65 |
+
assert len(storage) == len(set(storage)), "layout conflict"
|
| 66 |
+
d[attr] = tuple(storage)
|
| 67 |
+
d.setdefault("abstract", False)
|
| 68 |
+
return type.__new__(mcs, name, bases, d)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
class EvalContext:
|
| 72 |
+
"""Holds evaluation time information. Custom attributes can be attached
|
| 73 |
+
to it in extensions.
|
| 74 |
+
"""
|
| 75 |
+
|
| 76 |
+
def __init__(
|
| 77 |
+
self, environment: "Environment", template_name: t.Optional[str] = None
|
| 78 |
+
) -> None:
|
| 79 |
+
self.environment = environment
|
| 80 |
+
if callable(environment.autoescape):
|
| 81 |
+
self.autoescape = environment.autoescape(template_name)
|
| 82 |
+
else:
|
| 83 |
+
self.autoescape = environment.autoescape
|
| 84 |
+
self.volatile = False
|
| 85 |
+
|
| 86 |
+
def save(self) -> t.Mapping[str, t.Any]:
|
| 87 |
+
return self.__dict__.copy()
|
| 88 |
+
|
| 89 |
+
def revert(self, old: t.Mapping[str, t.Any]) -> None:
|
| 90 |
+
self.__dict__.clear()
|
| 91 |
+
self.__dict__.update(old)
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def get_eval_context(node: "Node", ctx: t.Optional[EvalContext]) -> EvalContext:
|
| 95 |
+
if ctx is None:
|
| 96 |
+
if node.environment is None:
|
| 97 |
+
raise RuntimeError(
|
| 98 |
+
"if no eval context is passed, the node must have an"
|
| 99 |
+
" attached environment."
|
| 100 |
+
)
|
| 101 |
+
return EvalContext(node.environment)
|
| 102 |
+
return ctx
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
class Node(metaclass=NodeType):
|
| 106 |
+
"""Baseclass for all Jinja nodes. There are a number of nodes available
|
| 107 |
+
of different types. There are four major types:
|
| 108 |
+
|
| 109 |
+
- :class:`Stmt`: statements
|
| 110 |
+
- :class:`Expr`: expressions
|
| 111 |
+
- :class:`Helper`: helper nodes
|
| 112 |
+
- :class:`Template`: the outermost wrapper node
|
| 113 |
+
|
| 114 |
+
All nodes have fields and attributes. Fields may be other nodes, lists,
|
| 115 |
+
or arbitrary values. Fields are passed to the constructor as regular
|
| 116 |
+
positional arguments, attributes as keyword arguments. Each node has
|
| 117 |
+
two attributes: `lineno` (the line number of the node) and `environment`.
|
| 118 |
+
The `environment` attribute is set at the end of the parsing process for
|
| 119 |
+
all nodes automatically.
|
| 120 |
+
"""
|
| 121 |
+
|
| 122 |
+
fields: t.Tuple[str, ...] = ()
|
| 123 |
+
attributes: t.Tuple[str, ...] = ("lineno", "environment")
|
| 124 |
+
abstract = True
|
| 125 |
+
|
| 126 |
+
lineno: int
|
| 127 |
+
environment: t.Optional["Environment"]
|
| 128 |
+
|
| 129 |
+
def __init__(self, *fields: t.Any, **attributes: t.Any) -> None:
|
| 130 |
+
if self.abstract:
|
| 131 |
+
raise TypeError("abstract nodes are not instantiable")
|
| 132 |
+
if fields:
|
| 133 |
+
if len(fields) != len(self.fields):
|
| 134 |
+
if not self.fields:
|
| 135 |
+
raise TypeError(f"{type(self).__name__!r} takes 0 arguments")
|
| 136 |
+
raise TypeError(
|
| 137 |
+
f"{type(self).__name__!r} takes 0 or {len(self.fields)}"
|
| 138 |
+
f" argument{'s' if len(self.fields) != 1 else ''}"
|
| 139 |
+
)
|
| 140 |
+
for name, arg in zip(self.fields, fields):
|
| 141 |
+
setattr(self, name, arg)
|
| 142 |
+
for attr in self.attributes:
|
| 143 |
+
setattr(self, attr, attributes.pop(attr, None))
|
| 144 |
+
if attributes:
|
| 145 |
+
raise TypeError(f"unknown attribute {next(iter(attributes))!r}")
|
| 146 |
+
|
| 147 |
+
def iter_fields(
|
| 148 |
+
self,
|
| 149 |
+
exclude: t.Optional[t.Container[str]] = None,
|
| 150 |
+
only: t.Optional[t.Container[str]] = None,
|
| 151 |
+
) -> t.Iterator[t.Tuple[str, t.Any]]:
|
| 152 |
+
"""This method iterates over all fields that are defined and yields
|
| 153 |
+
``(key, value)`` tuples. Per default all fields are returned, but
|
| 154 |
+
it's possible to limit that to some fields by providing the `only`
|
| 155 |
+
parameter or to exclude some using the `exclude` parameter. Both
|
| 156 |
+
should be sets or tuples of field names.
|
| 157 |
+
"""
|
| 158 |
+
for name in self.fields:
|
| 159 |
+
if (
|
| 160 |
+
(exclude is None and only is None)
|
| 161 |
+
or (exclude is not None and name not in exclude)
|
| 162 |
+
or (only is not None and name in only)
|
| 163 |
+
):
|
| 164 |
+
try:
|
| 165 |
+
yield name, getattr(self, name)
|
| 166 |
+
except AttributeError:
|
| 167 |
+
pass
|
| 168 |
+
|
| 169 |
+
def iter_child_nodes(
|
| 170 |
+
self,
|
| 171 |
+
exclude: t.Optional[t.Container[str]] = None,
|
| 172 |
+
only: t.Optional[t.Container[str]] = None,
|
| 173 |
+
) -> t.Iterator["Node"]:
|
| 174 |
+
"""Iterates over all direct child nodes of the node. This iterates
|
| 175 |
+
over all fields and yields the values of they are nodes. If the value
|
| 176 |
+
of a field is a list all the nodes in that list are returned.
|
| 177 |
+
"""
|
| 178 |
+
for _, item in self.iter_fields(exclude, only):
|
| 179 |
+
if isinstance(item, list):
|
| 180 |
+
for n in item:
|
| 181 |
+
if isinstance(n, Node):
|
| 182 |
+
yield n
|
| 183 |
+
elif isinstance(item, Node):
|
| 184 |
+
yield item
|
| 185 |
+
|
| 186 |
+
def find(self, node_type: t.Type[_NodeBound]) -> t.Optional[_NodeBound]:
|
| 187 |
+
"""Find the first node of a given type. If no such node exists the
|
| 188 |
+
return value is `None`.
|
| 189 |
+
"""
|
| 190 |
+
for result in self.find_all(node_type):
|
| 191 |
+
return result
|
| 192 |
+
|
| 193 |
+
return None
|
| 194 |
+
|
| 195 |
+
def find_all(
|
| 196 |
+
self, node_type: t.Union[t.Type[_NodeBound], t.Tuple[t.Type[_NodeBound], ...]]
|
| 197 |
+
) -> t.Iterator[_NodeBound]:
|
| 198 |
+
"""Find all the nodes of a given type. If the type is a tuple,
|
| 199 |
+
the check is performed for any of the tuple items.
|
| 200 |
+
"""
|
| 201 |
+
for child in self.iter_child_nodes():
|
| 202 |
+
if isinstance(child, node_type):
|
| 203 |
+
yield child # type: ignore
|
| 204 |
+
yield from child.find_all(node_type)
|
| 205 |
+
|
| 206 |
+
def set_ctx(self, ctx: str) -> "Node":
|
| 207 |
+
"""Reset the context of a node and all child nodes. Per default the
|
| 208 |
+
parser will all generate nodes that have a 'load' context as it's the
|
| 209 |
+
most common one. This method is used in the parser to set assignment
|
| 210 |
+
targets and other nodes to a store context.
|
| 211 |
+
"""
|
| 212 |
+
todo = deque([self])
|
| 213 |
+
while todo:
|
| 214 |
+
node = todo.popleft()
|
| 215 |
+
if "ctx" in node.fields:
|
| 216 |
+
node.ctx = ctx # type: ignore
|
| 217 |
+
todo.extend(node.iter_child_nodes())
|
| 218 |
+
return self
|
| 219 |
+
|
| 220 |
+
def set_lineno(self, lineno: int, override: bool = False) -> "Node":
|
| 221 |
+
"""Set the line numbers of the node and children."""
|
| 222 |
+
todo = deque([self])
|
| 223 |
+
while todo:
|
| 224 |
+
node = todo.popleft()
|
| 225 |
+
if "lineno" in node.attributes:
|
| 226 |
+
if node.lineno is None or override:
|
| 227 |
+
node.lineno = lineno
|
| 228 |
+
todo.extend(node.iter_child_nodes())
|
| 229 |
+
return self
|
| 230 |
+
|
| 231 |
+
def set_environment(self, environment: "Environment") -> "Node":
|
| 232 |
+
"""Set the environment for all nodes."""
|
| 233 |
+
todo = deque([self])
|
| 234 |
+
while todo:
|
| 235 |
+
node = todo.popleft()
|
| 236 |
+
node.environment = environment
|
| 237 |
+
todo.extend(node.iter_child_nodes())
|
| 238 |
+
return self
|
| 239 |
+
|
| 240 |
+
def __eq__(self, other: t.Any) -> bool:
|
| 241 |
+
if type(self) is not type(other):
|
| 242 |
+
return NotImplemented
|
| 243 |
+
|
| 244 |
+
return tuple(self.iter_fields()) == tuple(other.iter_fields())
|
| 245 |
+
|
| 246 |
+
__hash__ = object.__hash__
|
| 247 |
+
|
| 248 |
+
def __repr__(self) -> str:
|
| 249 |
+
args_str = ", ".join(f"{a}={getattr(self, a, None)!r}" for a in self.fields)
|
| 250 |
+
return f"{type(self).__name__}({args_str})"
|
| 251 |
+
|
| 252 |
+
def dump(self) -> str:
|
| 253 |
+
def _dump(node: t.Union[Node, t.Any]) -> None:
|
| 254 |
+
if not isinstance(node, Node):
|
| 255 |
+
buf.append(repr(node))
|
| 256 |
+
return
|
| 257 |
+
|
| 258 |
+
buf.append(f"nodes.{type(node).__name__}(")
|
| 259 |
+
if not node.fields:
|
| 260 |
+
buf.append(")")
|
| 261 |
+
return
|
| 262 |
+
for idx, field in enumerate(node.fields):
|
| 263 |
+
if idx:
|
| 264 |
+
buf.append(", ")
|
| 265 |
+
value = getattr(node, field)
|
| 266 |
+
if isinstance(value, list):
|
| 267 |
+
buf.append("[")
|
| 268 |
+
for idx, item in enumerate(value):
|
| 269 |
+
if idx:
|
| 270 |
+
buf.append(", ")
|
| 271 |
+
_dump(item)
|
| 272 |
+
buf.append("]")
|
| 273 |
+
else:
|
| 274 |
+
_dump(value)
|
| 275 |
+
buf.append(")")
|
| 276 |
+
|
| 277 |
+
buf: t.List[str] = []
|
| 278 |
+
_dump(self)
|
| 279 |
+
return "".join(buf)
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
class Stmt(Node):
|
| 283 |
+
"""Base node for all statements."""
|
| 284 |
+
|
| 285 |
+
abstract = True
|
| 286 |
+
|
| 287 |
+
|
| 288 |
+
class Helper(Node):
|
| 289 |
+
"""Nodes that exist in a specific context only."""
|
| 290 |
+
|
| 291 |
+
abstract = True
|
| 292 |
+
|
| 293 |
+
|
| 294 |
+
class Template(Node):
|
| 295 |
+
"""Node that represents a template. This must be the outermost node that
|
| 296 |
+
is passed to the compiler.
|
| 297 |
+
"""
|
| 298 |
+
|
| 299 |
+
fields = ("body",)
|
| 300 |
+
body: t.List[Node]
|
| 301 |
+
|
| 302 |
+
|
| 303 |
+
class Output(Stmt):
|
| 304 |
+
"""A node that holds multiple expressions which are then printed out.
|
| 305 |
+
This is used both for the `print` statement and the regular template data.
|
| 306 |
+
"""
|
| 307 |
+
|
| 308 |
+
fields = ("nodes",)
|
| 309 |
+
nodes: t.List["Expr"]
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
class Extends(Stmt):
|
| 313 |
+
"""Represents an extends statement."""
|
| 314 |
+
|
| 315 |
+
fields = ("template",)
|
| 316 |
+
template: "Expr"
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
class For(Stmt):
|
| 320 |
+
"""The for loop. `target` is the target for the iteration (usually a
|
| 321 |
+
:class:`Name` or :class:`Tuple`), `iter` the iterable. `body` is a list
|
| 322 |
+
of nodes that are used as loop-body, and `else_` a list of nodes for the
|
| 323 |
+
`else` block. If no else node exists it has to be an empty list.
|
| 324 |
+
|
| 325 |
+
For filtered nodes an expression can be stored as `test`, otherwise `None`.
|
| 326 |
+
"""
|
| 327 |
+
|
| 328 |
+
fields = ("target", "iter", "body", "else_", "test", "recursive")
|
| 329 |
+
target: Node
|
| 330 |
+
iter: Node
|
| 331 |
+
body: t.List[Node]
|
| 332 |
+
else_: t.List[Node]
|
| 333 |
+
test: t.Optional[Node]
|
| 334 |
+
recursive: bool
|
| 335 |
+
|
| 336 |
+
|
| 337 |
+
class If(Stmt):
|
| 338 |
+
"""If `test` is true, `body` is rendered, else `else_`."""
|
| 339 |
+
|
| 340 |
+
fields = ("test", "body", "elif_", "else_")
|
| 341 |
+
test: Node
|
| 342 |
+
body: t.List[Node]
|
| 343 |
+
elif_: t.List["If"]
|
| 344 |
+
else_: t.List[Node]
|
| 345 |
+
|
| 346 |
+
|
| 347 |
+
class Macro(Stmt):
|
| 348 |
+
"""A macro definition. `name` is the name of the macro, `args` a list of
|
| 349 |
+
arguments and `defaults` a list of defaults if there are any. `body` is
|
| 350 |
+
a list of nodes for the macro body.
|
| 351 |
+
"""
|
| 352 |
+
|
| 353 |
+
fields = ("name", "args", "defaults", "body")
|
| 354 |
+
name: str
|
| 355 |
+
args: t.List["Name"]
|
| 356 |
+
defaults: t.List["Expr"]
|
| 357 |
+
body: t.List[Node]
|
| 358 |
+
|
| 359 |
+
|
| 360 |
+
class CallBlock(Stmt):
|
| 361 |
+
"""Like a macro without a name but a call instead. `call` is called with
|
| 362 |
+
the unnamed macro as `caller` argument this node holds.
|
| 363 |
+
"""
|
| 364 |
+
|
| 365 |
+
fields = ("call", "args", "defaults", "body")
|
| 366 |
+
call: "Call"
|
| 367 |
+
args: t.List["Name"]
|
| 368 |
+
defaults: t.List["Expr"]
|
| 369 |
+
body: t.List[Node]
|
| 370 |
+
|
| 371 |
+
|
| 372 |
+
class FilterBlock(Stmt):
|
| 373 |
+
"""Node for filter sections."""
|
| 374 |
+
|
| 375 |
+
fields = ("body", "filter")
|
| 376 |
+
body: t.List[Node]
|
| 377 |
+
filter: "Filter"
|
| 378 |
+
|
| 379 |
+
|
| 380 |
+
class With(Stmt):
|
| 381 |
+
"""Specific node for with statements. In older versions of Jinja the
|
| 382 |
+
with statement was implemented on the base of the `Scope` node instead.
|
| 383 |
+
|
| 384 |
+
.. versionadded:: 2.9.3
|
| 385 |
+
"""
|
| 386 |
+
|
| 387 |
+
fields = ("targets", "values", "body")
|
| 388 |
+
targets: t.List["Expr"]
|
| 389 |
+
values: t.List["Expr"]
|
| 390 |
+
body: t.List[Node]
|
| 391 |
+
|
| 392 |
+
|
| 393 |
+
class Block(Stmt):
|
| 394 |
+
"""A node that represents a block.
|
| 395 |
+
|
| 396 |
+
.. versionchanged:: 3.0.0
|
| 397 |
+
the `required` field was added.
|
| 398 |
+
"""
|
| 399 |
+
|
| 400 |
+
fields = ("name", "body", "scoped", "required")
|
| 401 |
+
name: str
|
| 402 |
+
body: t.List[Node]
|
| 403 |
+
scoped: bool
|
| 404 |
+
required: bool
|
| 405 |
+
|
| 406 |
+
|
| 407 |
+
class Include(Stmt):
|
| 408 |
+
"""A node that represents the include tag."""
|
| 409 |
+
|
| 410 |
+
fields = ("template", "with_context", "ignore_missing")
|
| 411 |
+
template: "Expr"
|
| 412 |
+
with_context: bool
|
| 413 |
+
ignore_missing: bool
|
| 414 |
+
|
| 415 |
+
|
| 416 |
+
class Import(Stmt):
|
| 417 |
+
"""A node that represents the import tag."""
|
| 418 |
+
|
| 419 |
+
fields = ("template", "target", "with_context")
|
| 420 |
+
template: "Expr"
|
| 421 |
+
target: str
|
| 422 |
+
with_context: bool
|
| 423 |
+
|
| 424 |
+
|
| 425 |
+
class FromImport(Stmt):
|
| 426 |
+
"""A node that represents the from import tag. It's important to not
|
| 427 |
+
pass unsafe names to the name attribute. The compiler translates the
|
| 428 |
+
attribute lookups directly into getattr calls and does *not* use the
|
| 429 |
+
subscript callback of the interface. As exported variables may not
|
| 430 |
+
start with double underscores (which the parser asserts) this is not a
|
| 431 |
+
problem for regular Jinja code, but if this node is used in an extension
|
| 432 |
+
extra care must be taken.
|
| 433 |
+
|
| 434 |
+
The list of names may contain tuples if aliases are wanted.
|
| 435 |
+
"""
|
| 436 |
+
|
| 437 |
+
fields = ("template", "names", "with_context")
|
| 438 |
+
template: "Expr"
|
| 439 |
+
names: t.List[t.Union[str, t.Tuple[str, str]]]
|
| 440 |
+
with_context: bool
|
| 441 |
+
|
| 442 |
+
|
| 443 |
+
class ExprStmt(Stmt):
|
| 444 |
+
"""A statement that evaluates an expression and discards the result."""
|
| 445 |
+
|
| 446 |
+
fields = ("node",)
|
| 447 |
+
node: Node
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
class Assign(Stmt):
|
| 451 |
+
"""Assigns an expression to a target."""
|
| 452 |
+
|
| 453 |
+
fields = ("target", "node")
|
| 454 |
+
target: "Expr"
|
| 455 |
+
node: Node
|
| 456 |
+
|
| 457 |
+
|
| 458 |
+
class AssignBlock(Stmt):
|
| 459 |
+
"""Assigns a block to a target."""
|
| 460 |
+
|
| 461 |
+
fields = ("target", "filter", "body")
|
| 462 |
+
target: "Expr"
|
| 463 |
+
filter: t.Optional["Filter"]
|
| 464 |
+
body: t.List[Node]
|
| 465 |
+
|
| 466 |
+
|
| 467 |
+
class Expr(Node):
|
| 468 |
+
"""Baseclass for all expressions."""
|
| 469 |
+
|
| 470 |
+
abstract = True
|
| 471 |
+
|
| 472 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 473 |
+
"""Return the value of the expression as constant or raise
|
| 474 |
+
:exc:`Impossible` if this was not possible.
|
| 475 |
+
|
| 476 |
+
An :class:`EvalContext` can be provided, if none is given
|
| 477 |
+
a default context is created which requires the nodes to have
|
| 478 |
+
an attached environment.
|
| 479 |
+
|
| 480 |
+
.. versionchanged:: 2.4
|
| 481 |
+
the `eval_ctx` parameter was added.
|
| 482 |
+
"""
|
| 483 |
+
raise Impossible()
|
| 484 |
+
|
| 485 |
+
def can_assign(self) -> bool:
|
| 486 |
+
"""Check if it's possible to assign something to this node."""
|
| 487 |
+
return False
|
| 488 |
+
|
| 489 |
+
|
| 490 |
+
class BinExpr(Expr):
|
| 491 |
+
"""Baseclass for all binary expressions."""
|
| 492 |
+
|
| 493 |
+
fields = ("left", "right")
|
| 494 |
+
left: Expr
|
| 495 |
+
right: Expr
|
| 496 |
+
operator: str
|
| 497 |
+
abstract = True
|
| 498 |
+
|
| 499 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 500 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 501 |
+
|
| 502 |
+
# intercepted operators cannot be folded at compile time
|
| 503 |
+
if (
|
| 504 |
+
eval_ctx.environment.sandboxed
|
| 505 |
+
and self.operator in eval_ctx.environment.intercepted_binops # type: ignore
|
| 506 |
+
):
|
| 507 |
+
raise Impossible()
|
| 508 |
+
f = _binop_to_func[self.operator]
|
| 509 |
+
try:
|
| 510 |
+
return f(self.left.as_const(eval_ctx), self.right.as_const(eval_ctx))
|
| 511 |
+
except Exception as e:
|
| 512 |
+
raise Impossible() from e
|
| 513 |
+
|
| 514 |
+
|
| 515 |
+
class UnaryExpr(Expr):
|
| 516 |
+
"""Baseclass for all unary expressions."""
|
| 517 |
+
|
| 518 |
+
fields = ("node",)
|
| 519 |
+
node: Expr
|
| 520 |
+
operator: str
|
| 521 |
+
abstract = True
|
| 522 |
+
|
| 523 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 524 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 525 |
+
|
| 526 |
+
# intercepted operators cannot be folded at compile time
|
| 527 |
+
if (
|
| 528 |
+
eval_ctx.environment.sandboxed
|
| 529 |
+
and self.operator in eval_ctx.environment.intercepted_unops # type: ignore
|
| 530 |
+
):
|
| 531 |
+
raise Impossible()
|
| 532 |
+
f = _uaop_to_func[self.operator]
|
| 533 |
+
try:
|
| 534 |
+
return f(self.node.as_const(eval_ctx))
|
| 535 |
+
except Exception as e:
|
| 536 |
+
raise Impossible() from e
|
| 537 |
+
|
| 538 |
+
|
| 539 |
+
class Name(Expr):
|
| 540 |
+
"""Looks up a name or stores a value in a name.
|
| 541 |
+
The `ctx` of the node can be one of the following values:
|
| 542 |
+
|
| 543 |
+
- `store`: store a value in the name
|
| 544 |
+
- `load`: load that name
|
| 545 |
+
- `param`: like `store` but if the name was defined as function parameter.
|
| 546 |
+
"""
|
| 547 |
+
|
| 548 |
+
fields = ("name", "ctx")
|
| 549 |
+
name: str
|
| 550 |
+
ctx: str
|
| 551 |
+
|
| 552 |
+
def can_assign(self) -> bool:
|
| 553 |
+
return self.name not in {"true", "false", "none", "True", "False", "None"}
|
| 554 |
+
|
| 555 |
+
|
| 556 |
+
class NSRef(Expr):
|
| 557 |
+
"""Reference to a namespace value assignment"""
|
| 558 |
+
|
| 559 |
+
fields = ("name", "attr")
|
| 560 |
+
name: str
|
| 561 |
+
attr: str
|
| 562 |
+
|
| 563 |
+
def can_assign(self) -> bool:
|
| 564 |
+
# We don't need any special checks here; NSRef assignments have a
|
| 565 |
+
# runtime check to ensure the target is a namespace object which will
|
| 566 |
+
# have been checked already as it is created using a normal assignment
|
| 567 |
+
# which goes through a `Name` node.
|
| 568 |
+
return True
|
| 569 |
+
|
| 570 |
+
|
| 571 |
+
class Literal(Expr):
|
| 572 |
+
"""Baseclass for literals."""
|
| 573 |
+
|
| 574 |
+
abstract = True
|
| 575 |
+
|
| 576 |
+
|
| 577 |
+
class Const(Literal):
|
| 578 |
+
"""All constant values. The parser will return this node for simple
|
| 579 |
+
constants such as ``42`` or ``"foo"`` but it can be used to store more
|
| 580 |
+
complex values such as lists too. Only constants with a safe
|
| 581 |
+
representation (objects where ``eval(repr(x)) == x`` is true).
|
| 582 |
+
"""
|
| 583 |
+
|
| 584 |
+
fields = ("value",)
|
| 585 |
+
value: t.Any
|
| 586 |
+
|
| 587 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 588 |
+
return self.value
|
| 589 |
+
|
| 590 |
+
@classmethod
|
| 591 |
+
def from_untrusted(
|
| 592 |
+
cls,
|
| 593 |
+
value: t.Any,
|
| 594 |
+
lineno: t.Optional[int] = None,
|
| 595 |
+
environment: "t.Optional[Environment]" = None,
|
| 596 |
+
) -> "Const":
|
| 597 |
+
"""Return a const object if the value is representable as
|
| 598 |
+
constant value in the generated code, otherwise it will raise
|
| 599 |
+
an `Impossible` exception.
|
| 600 |
+
"""
|
| 601 |
+
from .compiler import has_safe_repr
|
| 602 |
+
|
| 603 |
+
if not has_safe_repr(value):
|
| 604 |
+
raise Impossible()
|
| 605 |
+
return cls(value, lineno=lineno, environment=environment)
|
| 606 |
+
|
| 607 |
+
|
| 608 |
+
class TemplateData(Literal):
|
| 609 |
+
"""A constant template string."""
|
| 610 |
+
|
| 611 |
+
fields = ("data",)
|
| 612 |
+
data: str
|
| 613 |
+
|
| 614 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> str:
|
| 615 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 616 |
+
if eval_ctx.volatile:
|
| 617 |
+
raise Impossible()
|
| 618 |
+
if eval_ctx.autoescape:
|
| 619 |
+
return Markup(self.data)
|
| 620 |
+
return self.data
|
| 621 |
+
|
| 622 |
+
|
| 623 |
+
class Tuple(Literal):
|
| 624 |
+
"""For loop unpacking and some other things like multiple arguments
|
| 625 |
+
for subscripts. Like for :class:`Name` `ctx` specifies if the tuple
|
| 626 |
+
is used for loading the names or storing.
|
| 627 |
+
"""
|
| 628 |
+
|
| 629 |
+
fields = ("items", "ctx")
|
| 630 |
+
items: t.List[Expr]
|
| 631 |
+
ctx: str
|
| 632 |
+
|
| 633 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Tuple[t.Any, ...]:
|
| 634 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 635 |
+
return tuple(x.as_const(eval_ctx) for x in self.items)
|
| 636 |
+
|
| 637 |
+
def can_assign(self) -> bool:
|
| 638 |
+
for item in self.items:
|
| 639 |
+
if not item.can_assign():
|
| 640 |
+
return False
|
| 641 |
+
return True
|
| 642 |
+
|
| 643 |
+
|
| 644 |
+
class List(Literal):
|
| 645 |
+
"""Any list literal such as ``[1, 2, 3]``"""
|
| 646 |
+
|
| 647 |
+
fields = ("items",)
|
| 648 |
+
items: t.List[Expr]
|
| 649 |
+
|
| 650 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.List[t.Any]:
|
| 651 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 652 |
+
return [x.as_const(eval_ctx) for x in self.items]
|
| 653 |
+
|
| 654 |
+
|
| 655 |
+
class Dict(Literal):
|
| 656 |
+
"""Any dict literal such as ``{1: 2, 3: 4}``. The items must be a list of
|
| 657 |
+
:class:`Pair` nodes.
|
| 658 |
+
"""
|
| 659 |
+
|
| 660 |
+
fields = ("items",)
|
| 661 |
+
items: t.List["Pair"]
|
| 662 |
+
|
| 663 |
+
def as_const(
|
| 664 |
+
self, eval_ctx: t.Optional[EvalContext] = None
|
| 665 |
+
) -> t.Dict[t.Any, t.Any]:
|
| 666 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 667 |
+
return dict(x.as_const(eval_ctx) for x in self.items)
|
| 668 |
+
|
| 669 |
+
|
| 670 |
+
class Pair(Helper):
|
| 671 |
+
"""A key, value pair for dicts."""
|
| 672 |
+
|
| 673 |
+
fields = ("key", "value")
|
| 674 |
+
key: Expr
|
| 675 |
+
value: Expr
|
| 676 |
+
|
| 677 |
+
def as_const(
|
| 678 |
+
self, eval_ctx: t.Optional[EvalContext] = None
|
| 679 |
+
) -> t.Tuple[t.Any, t.Any]:
|
| 680 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 681 |
+
return self.key.as_const(eval_ctx), self.value.as_const(eval_ctx)
|
| 682 |
+
|
| 683 |
+
|
| 684 |
+
class Keyword(Helper):
|
| 685 |
+
"""A key, value pair for keyword arguments where key is a string."""
|
| 686 |
+
|
| 687 |
+
fields = ("key", "value")
|
| 688 |
+
key: str
|
| 689 |
+
value: Expr
|
| 690 |
+
|
| 691 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Tuple[str, t.Any]:
|
| 692 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 693 |
+
return self.key, self.value.as_const(eval_ctx)
|
| 694 |
+
|
| 695 |
+
|
| 696 |
+
class CondExpr(Expr):
|
| 697 |
+
"""A conditional expression (inline if expression). (``{{
|
| 698 |
+
foo if bar else baz }}``)
|
| 699 |
+
"""
|
| 700 |
+
|
| 701 |
+
fields = ("test", "expr1", "expr2")
|
| 702 |
+
test: Expr
|
| 703 |
+
expr1: Expr
|
| 704 |
+
expr2: t.Optional[Expr]
|
| 705 |
+
|
| 706 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 707 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 708 |
+
if self.test.as_const(eval_ctx):
|
| 709 |
+
return self.expr1.as_const(eval_ctx)
|
| 710 |
+
|
| 711 |
+
# if we evaluate to an undefined object, we better do that at runtime
|
| 712 |
+
if self.expr2 is None:
|
| 713 |
+
raise Impossible()
|
| 714 |
+
|
| 715 |
+
return self.expr2.as_const(eval_ctx)
|
| 716 |
+
|
| 717 |
+
|
| 718 |
+
def args_as_const(
|
| 719 |
+
node: t.Union["_FilterTestCommon", "Call"], eval_ctx: t.Optional[EvalContext]
|
| 720 |
+
) -> t.Tuple[t.List[t.Any], t.Dict[t.Any, t.Any]]:
|
| 721 |
+
args = [x.as_const(eval_ctx) for x in node.args]
|
| 722 |
+
kwargs = dict(x.as_const(eval_ctx) for x in node.kwargs)
|
| 723 |
+
|
| 724 |
+
if node.dyn_args is not None:
|
| 725 |
+
try:
|
| 726 |
+
args.extend(node.dyn_args.as_const(eval_ctx))
|
| 727 |
+
except Exception as e:
|
| 728 |
+
raise Impossible() from e
|
| 729 |
+
|
| 730 |
+
if node.dyn_kwargs is not None:
|
| 731 |
+
try:
|
| 732 |
+
kwargs.update(node.dyn_kwargs.as_const(eval_ctx))
|
| 733 |
+
except Exception as e:
|
| 734 |
+
raise Impossible() from e
|
| 735 |
+
|
| 736 |
+
return args, kwargs
|
| 737 |
+
|
| 738 |
+
|
| 739 |
+
class _FilterTestCommon(Expr):
|
| 740 |
+
fields = ("node", "name", "args", "kwargs", "dyn_args", "dyn_kwargs")
|
| 741 |
+
node: Expr
|
| 742 |
+
name: str
|
| 743 |
+
args: t.List[Expr]
|
| 744 |
+
kwargs: t.List[Pair]
|
| 745 |
+
dyn_args: t.Optional[Expr]
|
| 746 |
+
dyn_kwargs: t.Optional[Expr]
|
| 747 |
+
abstract = True
|
| 748 |
+
_is_filter = True
|
| 749 |
+
|
| 750 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 751 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 752 |
+
|
| 753 |
+
if eval_ctx.volatile:
|
| 754 |
+
raise Impossible()
|
| 755 |
+
|
| 756 |
+
if self._is_filter:
|
| 757 |
+
env_map = eval_ctx.environment.filters
|
| 758 |
+
else:
|
| 759 |
+
env_map = eval_ctx.environment.tests
|
| 760 |
+
|
| 761 |
+
func = env_map.get(self.name)
|
| 762 |
+
pass_arg = _PassArg.from_obj(func) # type: ignore
|
| 763 |
+
|
| 764 |
+
if func is None or pass_arg is _PassArg.context:
|
| 765 |
+
raise Impossible()
|
| 766 |
+
|
| 767 |
+
if eval_ctx.environment.is_async and (
|
| 768 |
+
getattr(func, "jinja_async_variant", False) is True
|
| 769 |
+
or inspect.iscoroutinefunction(func)
|
| 770 |
+
):
|
| 771 |
+
raise Impossible()
|
| 772 |
+
|
| 773 |
+
args, kwargs = args_as_const(self, eval_ctx)
|
| 774 |
+
args.insert(0, self.node.as_const(eval_ctx))
|
| 775 |
+
|
| 776 |
+
if pass_arg is _PassArg.eval_context:
|
| 777 |
+
args.insert(0, eval_ctx)
|
| 778 |
+
elif pass_arg is _PassArg.environment:
|
| 779 |
+
args.insert(0, eval_ctx.environment)
|
| 780 |
+
|
| 781 |
+
try:
|
| 782 |
+
return func(*args, **kwargs)
|
| 783 |
+
except Exception as e:
|
| 784 |
+
raise Impossible() from e
|
| 785 |
+
|
| 786 |
+
|
| 787 |
+
class Filter(_FilterTestCommon):
|
| 788 |
+
"""Apply a filter to an expression. ``name`` is the name of the
|
| 789 |
+
filter, the other fields are the same as :class:`Call`.
|
| 790 |
+
|
| 791 |
+
If ``node`` is ``None``, the filter is being used in a filter block
|
| 792 |
+
and is applied to the content of the block.
|
| 793 |
+
"""
|
| 794 |
+
|
| 795 |
+
node: t.Optional[Expr] # type: ignore
|
| 796 |
+
|
| 797 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 798 |
+
if self.node is None:
|
| 799 |
+
raise Impossible()
|
| 800 |
+
|
| 801 |
+
return super().as_const(eval_ctx=eval_ctx)
|
| 802 |
+
|
| 803 |
+
|
| 804 |
+
class Test(_FilterTestCommon):
|
| 805 |
+
"""Apply a test to an expression. ``name`` is the name of the test,
|
| 806 |
+
the other field are the same as :class:`Call`.
|
| 807 |
+
|
| 808 |
+
.. versionchanged:: 3.0
|
| 809 |
+
``as_const`` shares the same logic for filters and tests. Tests
|
| 810 |
+
check for volatile, async, and ``@pass_context`` etc.
|
| 811 |
+
decorators.
|
| 812 |
+
"""
|
| 813 |
+
|
| 814 |
+
_is_filter = False
|
| 815 |
+
|
| 816 |
+
|
| 817 |
+
class Call(Expr):
|
| 818 |
+
"""Calls an expression. `args` is a list of arguments, `kwargs` a list
|
| 819 |
+
of keyword arguments (list of :class:`Keyword` nodes), and `dyn_args`
|
| 820 |
+
and `dyn_kwargs` has to be either `None` or a node that is used as
|
| 821 |
+
node for dynamic positional (``*args``) or keyword (``**kwargs``)
|
| 822 |
+
arguments.
|
| 823 |
+
"""
|
| 824 |
+
|
| 825 |
+
fields = ("node", "args", "kwargs", "dyn_args", "dyn_kwargs")
|
| 826 |
+
node: Expr
|
| 827 |
+
args: t.List[Expr]
|
| 828 |
+
kwargs: t.List[Keyword]
|
| 829 |
+
dyn_args: t.Optional[Expr]
|
| 830 |
+
dyn_kwargs: t.Optional[Expr]
|
| 831 |
+
|
| 832 |
+
|
| 833 |
+
class Getitem(Expr):
|
| 834 |
+
"""Get an attribute or item from an expression and prefer the item."""
|
| 835 |
+
|
| 836 |
+
fields = ("node", "arg", "ctx")
|
| 837 |
+
node: Expr
|
| 838 |
+
arg: Expr
|
| 839 |
+
ctx: str
|
| 840 |
+
|
| 841 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 842 |
+
if self.ctx != "load":
|
| 843 |
+
raise Impossible()
|
| 844 |
+
|
| 845 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 846 |
+
|
| 847 |
+
try:
|
| 848 |
+
return eval_ctx.environment.getitem(
|
| 849 |
+
self.node.as_const(eval_ctx), self.arg.as_const(eval_ctx)
|
| 850 |
+
)
|
| 851 |
+
except Exception as e:
|
| 852 |
+
raise Impossible() from e
|
| 853 |
+
|
| 854 |
+
|
| 855 |
+
class Getattr(Expr):
|
| 856 |
+
"""Get an attribute or item from an expression that is a ascii-only
|
| 857 |
+
bytestring and prefer the attribute.
|
| 858 |
+
"""
|
| 859 |
+
|
| 860 |
+
fields = ("node", "attr", "ctx")
|
| 861 |
+
node: Expr
|
| 862 |
+
attr: str
|
| 863 |
+
ctx: str
|
| 864 |
+
|
| 865 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 866 |
+
if self.ctx != "load":
|
| 867 |
+
raise Impossible()
|
| 868 |
+
|
| 869 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 870 |
+
|
| 871 |
+
try:
|
| 872 |
+
return eval_ctx.environment.getattr(self.node.as_const(eval_ctx), self.attr)
|
| 873 |
+
except Exception as e:
|
| 874 |
+
raise Impossible() from e
|
| 875 |
+
|
| 876 |
+
|
| 877 |
+
class Slice(Expr):
|
| 878 |
+
"""Represents a slice object. This must only be used as argument for
|
| 879 |
+
:class:`Subscript`.
|
| 880 |
+
"""
|
| 881 |
+
|
| 882 |
+
fields = ("start", "stop", "step")
|
| 883 |
+
start: t.Optional[Expr]
|
| 884 |
+
stop: t.Optional[Expr]
|
| 885 |
+
step: t.Optional[Expr]
|
| 886 |
+
|
| 887 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> slice:
|
| 888 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 889 |
+
|
| 890 |
+
def const(obj: t.Optional[Expr]) -> t.Optional[t.Any]:
|
| 891 |
+
if obj is None:
|
| 892 |
+
return None
|
| 893 |
+
return obj.as_const(eval_ctx)
|
| 894 |
+
|
| 895 |
+
return slice(const(self.start), const(self.stop), const(self.step))
|
| 896 |
+
|
| 897 |
+
|
| 898 |
+
class Concat(Expr):
|
| 899 |
+
"""Concatenates the list of expressions provided after converting
|
| 900 |
+
them to strings.
|
| 901 |
+
"""
|
| 902 |
+
|
| 903 |
+
fields = ("nodes",)
|
| 904 |
+
nodes: t.List[Expr]
|
| 905 |
+
|
| 906 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> str:
|
| 907 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 908 |
+
return "".join(str(x.as_const(eval_ctx)) for x in self.nodes)
|
| 909 |
+
|
| 910 |
+
|
| 911 |
+
class Compare(Expr):
|
| 912 |
+
"""Compares an expression with some other expressions. `ops` must be a
|
| 913 |
+
list of :class:`Operand`\\s.
|
| 914 |
+
"""
|
| 915 |
+
|
| 916 |
+
fields = ("expr", "ops")
|
| 917 |
+
expr: Expr
|
| 918 |
+
ops: t.List["Operand"]
|
| 919 |
+
|
| 920 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 921 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 922 |
+
result = value = self.expr.as_const(eval_ctx)
|
| 923 |
+
|
| 924 |
+
try:
|
| 925 |
+
for op in self.ops:
|
| 926 |
+
new_value = op.expr.as_const(eval_ctx)
|
| 927 |
+
result = _cmpop_to_func[op.op](value, new_value)
|
| 928 |
+
|
| 929 |
+
if not result:
|
| 930 |
+
return False
|
| 931 |
+
|
| 932 |
+
value = new_value
|
| 933 |
+
except Exception as e:
|
| 934 |
+
raise Impossible() from e
|
| 935 |
+
|
| 936 |
+
return result
|
| 937 |
+
|
| 938 |
+
|
| 939 |
+
class Operand(Helper):
|
| 940 |
+
"""Holds an operator and an expression."""
|
| 941 |
+
|
| 942 |
+
fields = ("op", "expr")
|
| 943 |
+
op: str
|
| 944 |
+
expr: Expr
|
| 945 |
+
|
| 946 |
+
|
| 947 |
+
class Mul(BinExpr):
|
| 948 |
+
"""Multiplies the left with the right node."""
|
| 949 |
+
|
| 950 |
+
operator = "*"
|
| 951 |
+
|
| 952 |
+
|
| 953 |
+
class Div(BinExpr):
|
| 954 |
+
"""Divides the left by the right node."""
|
| 955 |
+
|
| 956 |
+
operator = "/"
|
| 957 |
+
|
| 958 |
+
|
| 959 |
+
class FloorDiv(BinExpr):
|
| 960 |
+
"""Divides the left by the right node and converts the
|
| 961 |
+
result into an integer by truncating.
|
| 962 |
+
"""
|
| 963 |
+
|
| 964 |
+
operator = "//"
|
| 965 |
+
|
| 966 |
+
|
| 967 |
+
class Add(BinExpr):
|
| 968 |
+
"""Add the left to the right node."""
|
| 969 |
+
|
| 970 |
+
operator = "+"
|
| 971 |
+
|
| 972 |
+
|
| 973 |
+
class Sub(BinExpr):
|
| 974 |
+
"""Subtract the right from the left node."""
|
| 975 |
+
|
| 976 |
+
operator = "-"
|
| 977 |
+
|
| 978 |
+
|
| 979 |
+
class Mod(BinExpr):
|
| 980 |
+
"""Left modulo right."""
|
| 981 |
+
|
| 982 |
+
operator = "%"
|
| 983 |
+
|
| 984 |
+
|
| 985 |
+
class Pow(BinExpr):
|
| 986 |
+
"""Left to the power of right."""
|
| 987 |
+
|
| 988 |
+
operator = "**"
|
| 989 |
+
|
| 990 |
+
|
| 991 |
+
class And(BinExpr):
|
| 992 |
+
"""Short circuited AND."""
|
| 993 |
+
|
| 994 |
+
operator = "and"
|
| 995 |
+
|
| 996 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 997 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 998 |
+
return self.left.as_const(eval_ctx) and self.right.as_const(eval_ctx)
|
| 999 |
+
|
| 1000 |
+
|
| 1001 |
+
class Or(BinExpr):
|
| 1002 |
+
"""Short circuited OR."""
|
| 1003 |
+
|
| 1004 |
+
operator = "or"
|
| 1005 |
+
|
| 1006 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
|
| 1007 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 1008 |
+
return self.left.as_const(eval_ctx) or self.right.as_const(eval_ctx)
|
| 1009 |
+
|
| 1010 |
+
|
| 1011 |
+
class Not(UnaryExpr):
|
| 1012 |
+
"""Negate the expression."""
|
| 1013 |
+
|
| 1014 |
+
operator = "not"
|
| 1015 |
+
|
| 1016 |
+
|
| 1017 |
+
class Neg(UnaryExpr):
|
| 1018 |
+
"""Make the expression negative."""
|
| 1019 |
+
|
| 1020 |
+
operator = "-"
|
| 1021 |
+
|
| 1022 |
+
|
| 1023 |
+
class Pos(UnaryExpr):
|
| 1024 |
+
"""Make the expression positive (noop for most expressions)"""
|
| 1025 |
+
|
| 1026 |
+
operator = "+"
|
| 1027 |
+
|
| 1028 |
+
|
| 1029 |
+
# Helpers for extensions
|
| 1030 |
+
|
| 1031 |
+
|
| 1032 |
+
class EnvironmentAttribute(Expr):
|
| 1033 |
+
"""Loads an attribute from the environment object. This is useful for
|
| 1034 |
+
extensions that want to call a callback stored on the environment.
|
| 1035 |
+
"""
|
| 1036 |
+
|
| 1037 |
+
fields = ("name",)
|
| 1038 |
+
name: str
|
| 1039 |
+
|
| 1040 |
+
|
| 1041 |
+
class ExtensionAttribute(Expr):
|
| 1042 |
+
"""Returns the attribute of an extension bound to the environment.
|
| 1043 |
+
The identifier is the identifier of the :class:`Extension`.
|
| 1044 |
+
|
| 1045 |
+
This node is usually constructed by calling the
|
| 1046 |
+
:meth:`~jinja2.ext.Extension.attr` method on an extension.
|
| 1047 |
+
"""
|
| 1048 |
+
|
| 1049 |
+
fields = ("identifier", "name")
|
| 1050 |
+
identifier: str
|
| 1051 |
+
name: str
|
| 1052 |
+
|
| 1053 |
+
|
| 1054 |
+
class ImportedName(Expr):
|
| 1055 |
+
"""If created with an import name the import name is returned on node
|
| 1056 |
+
access. For example ``ImportedName('cgi.escape')`` returns the `escape`
|
| 1057 |
+
function from the cgi module on evaluation. Imports are optimized by the
|
| 1058 |
+
compiler so there is no need to assign them to local variables.
|
| 1059 |
+
"""
|
| 1060 |
+
|
| 1061 |
+
fields = ("importname",)
|
| 1062 |
+
importname: str
|
| 1063 |
+
|
| 1064 |
+
|
| 1065 |
+
class InternalName(Expr):
|
| 1066 |
+
"""An internal name in the compiler. You cannot create these nodes
|
| 1067 |
+
yourself but the parser provides a
|
| 1068 |
+
:meth:`~jinja2.parser.Parser.free_identifier` method that creates
|
| 1069 |
+
a new identifier for you. This identifier is not available from the
|
| 1070 |
+
template and is not treated specially by the compiler.
|
| 1071 |
+
"""
|
| 1072 |
+
|
| 1073 |
+
fields = ("name",)
|
| 1074 |
+
name: str
|
| 1075 |
+
|
| 1076 |
+
def __init__(self) -> None:
|
| 1077 |
+
raise TypeError(
|
| 1078 |
+
"Can't create internal names. Use the "
|
| 1079 |
+
"`free_identifier` method on a parser."
|
| 1080 |
+
)
|
| 1081 |
+
|
| 1082 |
+
|
| 1083 |
+
class MarkSafe(Expr):
|
| 1084 |
+
"""Mark the wrapped expression as safe (wrap it as `Markup`)."""
|
| 1085 |
+
|
| 1086 |
+
fields = ("expr",)
|
| 1087 |
+
expr: Expr
|
| 1088 |
+
|
| 1089 |
+
def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> Markup:
|
| 1090 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 1091 |
+
return Markup(self.expr.as_const(eval_ctx))
|
| 1092 |
+
|
| 1093 |
+
|
| 1094 |
+
class MarkSafeIfAutoescape(Expr):
|
| 1095 |
+
"""Mark the wrapped expression as safe (wrap it as `Markup`) but
|
| 1096 |
+
only if autoescaping is active.
|
| 1097 |
+
|
| 1098 |
+
.. versionadded:: 2.5
|
| 1099 |
+
"""
|
| 1100 |
+
|
| 1101 |
+
fields = ("expr",)
|
| 1102 |
+
expr: Expr
|
| 1103 |
+
|
| 1104 |
+
def as_const(
|
| 1105 |
+
self, eval_ctx: t.Optional[EvalContext] = None
|
| 1106 |
+
) -> t.Union[Markup, t.Any]:
|
| 1107 |
+
eval_ctx = get_eval_context(self, eval_ctx)
|
| 1108 |
+
if eval_ctx.volatile:
|
| 1109 |
+
raise Impossible()
|
| 1110 |
+
expr = self.expr.as_const(eval_ctx)
|
| 1111 |
+
if eval_ctx.autoescape:
|
| 1112 |
+
return Markup(expr)
|
| 1113 |
+
return expr
|
| 1114 |
+
|
| 1115 |
+
|
| 1116 |
+
class ContextReference(Expr):
|
| 1117 |
+
"""Returns the current template context. It can be used like a
|
| 1118 |
+
:class:`Name` node, with a ``'load'`` ctx and will return the
|
| 1119 |
+
current :class:`~jinja2.runtime.Context` object.
|
| 1120 |
+
|
| 1121 |
+
Here an example that assigns the current template name to a
|
| 1122 |
+
variable named `foo`::
|
| 1123 |
+
|
| 1124 |
+
Assign(Name('foo', ctx='store'),
|
| 1125 |
+
Getattr(ContextReference(), 'name'))
|
| 1126 |
+
|
| 1127 |
+
This is basically equivalent to using the
|
| 1128 |
+
:func:`~jinja2.pass_context` decorator when using the high-level
|
| 1129 |
+
API, which causes a reference to the context to be passed as the
|
| 1130 |
+
first argument to a function.
|
| 1131 |
+
"""
|
| 1132 |
+
|
| 1133 |
+
|
| 1134 |
+
class DerivedContextReference(Expr):
|
| 1135 |
+
"""Return the current template context including locals. Behaves
|
| 1136 |
+
exactly like :class:`ContextReference`, but includes local
|
| 1137 |
+
variables, such as from a ``for`` loop.
|
| 1138 |
+
|
| 1139 |
+
.. versionadded:: 2.11
|
| 1140 |
+
"""
|
| 1141 |
+
|
| 1142 |
+
|
| 1143 |
+
class Continue(Stmt):
|
| 1144 |
+
"""Continue a loop."""
|
| 1145 |
+
|
| 1146 |
+
|
| 1147 |
+
class Break(Stmt):
|
| 1148 |
+
"""Break a loop."""
|
| 1149 |
+
|
| 1150 |
+
|
| 1151 |
+
class Scope(Stmt):
|
| 1152 |
+
"""An artificial scope."""
|
| 1153 |
+
|
| 1154 |
+
fields = ("body",)
|
| 1155 |
+
body: t.List[Node]
|
| 1156 |
+
|
| 1157 |
+
|
| 1158 |
+
class OverlayScope(Stmt):
|
| 1159 |
+
"""An overlay scope for extensions. This is a largely unoptimized scope
|
| 1160 |
+
that however can be used to introduce completely arbitrary variables into
|
| 1161 |
+
a sub scope from a dictionary or dictionary like object. The `context`
|
| 1162 |
+
field has to evaluate to a dictionary object.
|
| 1163 |
+
|
| 1164 |
+
Example usage::
|
| 1165 |
+
|
| 1166 |
+
OverlayScope(context=self.call_method('get_context'),
|
| 1167 |
+
body=[...])
|
| 1168 |
+
|
| 1169 |
+
.. versionadded:: 2.10
|
| 1170 |
+
"""
|
| 1171 |
+
|
| 1172 |
+
fields = ("context", "body")
|
| 1173 |
+
context: Expr
|
| 1174 |
+
body: t.List[Node]
|
| 1175 |
+
|
| 1176 |
+
|
| 1177 |
+
class EvalContextModifier(Stmt):
|
| 1178 |
+
"""Modifies the eval context. For each option that should be modified,
|
| 1179 |
+
a :class:`Keyword` has to be added to the :attr:`options` list.
|
| 1180 |
+
|
| 1181 |
+
Example to change the `autoescape` setting::
|
| 1182 |
+
|
| 1183 |
+
EvalContextModifier(options=[Keyword('autoescape', Const(True))])
|
| 1184 |
+
"""
|
| 1185 |
+
|
| 1186 |
+
fields = ("options",)
|
| 1187 |
+
options: t.List[Keyword]
|
| 1188 |
+
|
| 1189 |
+
|
| 1190 |
+
class ScopedEvalContextModifier(EvalContextModifier):
|
| 1191 |
+
"""Modifies the eval context and reverts it later. Works exactly like
|
| 1192 |
+
:class:`EvalContextModifier` but will only modify the
|
| 1193 |
+
:class:`~jinja2.nodes.EvalContext` for nodes in the :attr:`body`.
|
| 1194 |
+
"""
|
| 1195 |
+
|
| 1196 |
+
fields = ("body",)
|
| 1197 |
+
body: t.List[Node]
|
| 1198 |
+
|
| 1199 |
+
|
| 1200 |
+
# make sure nobody creates custom nodes
|
| 1201 |
+
def _failing_new(*args: t.Any, **kwargs: t.Any) -> "te.NoReturn":
|
| 1202 |
+
raise TypeError("can't create custom node types")
|
| 1203 |
+
|
| 1204 |
+
|
| 1205 |
+
NodeType.__new__ = staticmethod(_failing_new) # type: ignore
|
| 1206 |
+
del _failing_new
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/optimizer.py
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The optimizer tries to constant fold expressions and modify the AST
|
| 2 |
+
in place so that it should be faster to evaluate.
|
| 3 |
+
|
| 4 |
+
Because the AST does not contain all the scoping information and the
|
| 5 |
+
compiler has to find that out, we cannot do all the optimizations we
|
| 6 |
+
want. For example, loop unrolling doesn't work because unrolled loops
|
| 7 |
+
would have a different scope. The solution would be a second syntax tree
|
| 8 |
+
that stored the scoping rules.
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
import typing as t
|
| 12 |
+
|
| 13 |
+
from . import nodes
|
| 14 |
+
from .visitor import NodeTransformer
|
| 15 |
+
|
| 16 |
+
if t.TYPE_CHECKING:
|
| 17 |
+
from .environment import Environment
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def optimize(node: nodes.Node, environment: "Environment") -> nodes.Node:
|
| 21 |
+
"""The context hint can be used to perform an static optimization
|
| 22 |
+
based on the context given."""
|
| 23 |
+
optimizer = Optimizer(environment)
|
| 24 |
+
return t.cast(nodes.Node, optimizer.visit(node))
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
class Optimizer(NodeTransformer):
|
| 28 |
+
def __init__(self, environment: "t.Optional[Environment]") -> None:
|
| 29 |
+
self.environment = environment
|
| 30 |
+
|
| 31 |
+
def generic_visit(
|
| 32 |
+
self, node: nodes.Node, *args: t.Any, **kwargs: t.Any
|
| 33 |
+
) -> nodes.Node:
|
| 34 |
+
node = super().generic_visit(node, *args, **kwargs)
|
| 35 |
+
|
| 36 |
+
# Do constant folding. Some other nodes besides Expr have
|
| 37 |
+
# as_const, but folding them causes errors later on.
|
| 38 |
+
if isinstance(node, nodes.Expr):
|
| 39 |
+
try:
|
| 40 |
+
return nodes.Const.from_untrusted(
|
| 41 |
+
node.as_const(args[0] if args else None),
|
| 42 |
+
lineno=node.lineno,
|
| 43 |
+
environment=self.environment,
|
| 44 |
+
)
|
| 45 |
+
except nodes.Impossible:
|
| 46 |
+
pass
|
| 47 |
+
|
| 48 |
+
return node
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/parser.py
ADDED
|
@@ -0,0 +1,1049 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Parse tokens from the lexer into nodes for the compiler."""
|
| 2 |
+
|
| 3 |
+
import typing
|
| 4 |
+
import typing as t
|
| 5 |
+
|
| 6 |
+
from . import nodes
|
| 7 |
+
from .exceptions import TemplateAssertionError
|
| 8 |
+
from .exceptions import TemplateSyntaxError
|
| 9 |
+
from .lexer import describe_token
|
| 10 |
+
from .lexer import describe_token_expr
|
| 11 |
+
|
| 12 |
+
if t.TYPE_CHECKING:
|
| 13 |
+
import typing_extensions as te
|
| 14 |
+
|
| 15 |
+
from .environment import Environment
|
| 16 |
+
|
| 17 |
+
_ImportInclude = t.TypeVar("_ImportInclude", nodes.Import, nodes.Include)
|
| 18 |
+
_MacroCall = t.TypeVar("_MacroCall", nodes.Macro, nodes.CallBlock)
|
| 19 |
+
|
| 20 |
+
_statement_keywords = frozenset(
|
| 21 |
+
[
|
| 22 |
+
"for",
|
| 23 |
+
"if",
|
| 24 |
+
"block",
|
| 25 |
+
"extends",
|
| 26 |
+
"print",
|
| 27 |
+
"macro",
|
| 28 |
+
"include",
|
| 29 |
+
"from",
|
| 30 |
+
"import",
|
| 31 |
+
"set",
|
| 32 |
+
"with",
|
| 33 |
+
"autoescape",
|
| 34 |
+
]
|
| 35 |
+
)
|
| 36 |
+
_compare_operators = frozenset(["eq", "ne", "lt", "lteq", "gt", "gteq"])
|
| 37 |
+
|
| 38 |
+
_math_nodes: t.Dict[str, t.Type[nodes.Expr]] = {
|
| 39 |
+
"add": nodes.Add,
|
| 40 |
+
"sub": nodes.Sub,
|
| 41 |
+
"mul": nodes.Mul,
|
| 42 |
+
"div": nodes.Div,
|
| 43 |
+
"floordiv": nodes.FloorDiv,
|
| 44 |
+
"mod": nodes.Mod,
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
class Parser:
|
| 49 |
+
"""This is the central parsing class Jinja uses. It's passed to
|
| 50 |
+
extensions and can be used to parse expressions or statements.
|
| 51 |
+
"""
|
| 52 |
+
|
| 53 |
+
def __init__(
|
| 54 |
+
self,
|
| 55 |
+
environment: "Environment",
|
| 56 |
+
source: str,
|
| 57 |
+
name: t.Optional[str] = None,
|
| 58 |
+
filename: t.Optional[str] = None,
|
| 59 |
+
state: t.Optional[str] = None,
|
| 60 |
+
) -> None:
|
| 61 |
+
self.environment = environment
|
| 62 |
+
self.stream = environment._tokenize(source, name, filename, state)
|
| 63 |
+
self.name = name
|
| 64 |
+
self.filename = filename
|
| 65 |
+
self.closed = False
|
| 66 |
+
self.extensions: t.Dict[
|
| 67 |
+
str, t.Callable[[Parser], t.Union[nodes.Node, t.List[nodes.Node]]]
|
| 68 |
+
] = {}
|
| 69 |
+
for extension in environment.iter_extensions():
|
| 70 |
+
for tag in extension.tags:
|
| 71 |
+
self.extensions[tag] = extension.parse
|
| 72 |
+
self._last_identifier = 0
|
| 73 |
+
self._tag_stack: t.List[str] = []
|
| 74 |
+
self._end_token_stack: t.List[t.Tuple[str, ...]] = []
|
| 75 |
+
|
| 76 |
+
def fail(
|
| 77 |
+
self,
|
| 78 |
+
msg: str,
|
| 79 |
+
lineno: t.Optional[int] = None,
|
| 80 |
+
exc: t.Type[TemplateSyntaxError] = TemplateSyntaxError,
|
| 81 |
+
) -> "te.NoReturn":
|
| 82 |
+
"""Convenience method that raises `exc` with the message, passed
|
| 83 |
+
line number or last line number as well as the current name and
|
| 84 |
+
filename.
|
| 85 |
+
"""
|
| 86 |
+
if lineno is None:
|
| 87 |
+
lineno = self.stream.current.lineno
|
| 88 |
+
raise exc(msg, lineno, self.name, self.filename)
|
| 89 |
+
|
| 90 |
+
def _fail_ut_eof(
|
| 91 |
+
self,
|
| 92 |
+
name: t.Optional[str],
|
| 93 |
+
end_token_stack: t.List[t.Tuple[str, ...]],
|
| 94 |
+
lineno: t.Optional[int],
|
| 95 |
+
) -> "te.NoReturn":
|
| 96 |
+
expected: t.Set[str] = set()
|
| 97 |
+
for exprs in end_token_stack:
|
| 98 |
+
expected.update(map(describe_token_expr, exprs))
|
| 99 |
+
if end_token_stack:
|
| 100 |
+
currently_looking: t.Optional[str] = " or ".join(
|
| 101 |
+
map(repr, map(describe_token_expr, end_token_stack[-1]))
|
| 102 |
+
)
|
| 103 |
+
else:
|
| 104 |
+
currently_looking = None
|
| 105 |
+
|
| 106 |
+
if name is None:
|
| 107 |
+
message = ["Unexpected end of template."]
|
| 108 |
+
else:
|
| 109 |
+
message = [f"Encountered unknown tag {name!r}."]
|
| 110 |
+
|
| 111 |
+
if currently_looking:
|
| 112 |
+
if name is not None and name in expected:
|
| 113 |
+
message.append(
|
| 114 |
+
"You probably made a nesting mistake. Jinja is expecting this tag,"
|
| 115 |
+
f" but currently looking for {currently_looking}."
|
| 116 |
+
)
|
| 117 |
+
else:
|
| 118 |
+
message.append(
|
| 119 |
+
f"Jinja was looking for the following tags: {currently_looking}."
|
| 120 |
+
)
|
| 121 |
+
|
| 122 |
+
if self._tag_stack:
|
| 123 |
+
message.append(
|
| 124 |
+
"The innermost block that needs to be closed is"
|
| 125 |
+
f" {self._tag_stack[-1]!r}."
|
| 126 |
+
)
|
| 127 |
+
|
| 128 |
+
self.fail(" ".join(message), lineno)
|
| 129 |
+
|
| 130 |
+
def fail_unknown_tag(
|
| 131 |
+
self, name: str, lineno: t.Optional[int] = None
|
| 132 |
+
) -> "te.NoReturn":
|
| 133 |
+
"""Called if the parser encounters an unknown tag. Tries to fail
|
| 134 |
+
with a human readable error message that could help to identify
|
| 135 |
+
the problem.
|
| 136 |
+
"""
|
| 137 |
+
self._fail_ut_eof(name, self._end_token_stack, lineno)
|
| 138 |
+
|
| 139 |
+
def fail_eof(
|
| 140 |
+
self,
|
| 141 |
+
end_tokens: t.Optional[t.Tuple[str, ...]] = None,
|
| 142 |
+
lineno: t.Optional[int] = None,
|
| 143 |
+
) -> "te.NoReturn":
|
| 144 |
+
"""Like fail_unknown_tag but for end of template situations."""
|
| 145 |
+
stack = list(self._end_token_stack)
|
| 146 |
+
if end_tokens is not None:
|
| 147 |
+
stack.append(end_tokens)
|
| 148 |
+
self._fail_ut_eof(None, stack, lineno)
|
| 149 |
+
|
| 150 |
+
def is_tuple_end(
|
| 151 |
+
self, extra_end_rules: t.Optional[t.Tuple[str, ...]] = None
|
| 152 |
+
) -> bool:
|
| 153 |
+
"""Are we at the end of a tuple?"""
|
| 154 |
+
if self.stream.current.type in ("variable_end", "block_end", "rparen"):
|
| 155 |
+
return True
|
| 156 |
+
elif extra_end_rules is not None:
|
| 157 |
+
return self.stream.current.test_any(extra_end_rules) # type: ignore
|
| 158 |
+
return False
|
| 159 |
+
|
| 160 |
+
def free_identifier(self, lineno: t.Optional[int] = None) -> nodes.InternalName:
|
| 161 |
+
"""Return a new free identifier as :class:`~jinja2.nodes.InternalName`."""
|
| 162 |
+
self._last_identifier += 1
|
| 163 |
+
rv = object.__new__(nodes.InternalName)
|
| 164 |
+
nodes.Node.__init__(rv, f"fi{self._last_identifier}", lineno=lineno)
|
| 165 |
+
return rv
|
| 166 |
+
|
| 167 |
+
def parse_statement(self) -> t.Union[nodes.Node, t.List[nodes.Node]]:
|
| 168 |
+
"""Parse a single statement."""
|
| 169 |
+
token = self.stream.current
|
| 170 |
+
if token.type != "name":
|
| 171 |
+
self.fail("tag name expected", token.lineno)
|
| 172 |
+
self._tag_stack.append(token.value)
|
| 173 |
+
pop_tag = True
|
| 174 |
+
try:
|
| 175 |
+
if token.value in _statement_keywords:
|
| 176 |
+
f = getattr(self, f"parse_{self.stream.current.value}")
|
| 177 |
+
return f() # type: ignore
|
| 178 |
+
if token.value == "call":
|
| 179 |
+
return self.parse_call_block()
|
| 180 |
+
if token.value == "filter":
|
| 181 |
+
return self.parse_filter_block()
|
| 182 |
+
ext = self.extensions.get(token.value)
|
| 183 |
+
if ext is not None:
|
| 184 |
+
return ext(self)
|
| 185 |
+
|
| 186 |
+
# did not work out, remove the token we pushed by accident
|
| 187 |
+
# from the stack so that the unknown tag fail function can
|
| 188 |
+
# produce a proper error message.
|
| 189 |
+
self._tag_stack.pop()
|
| 190 |
+
pop_tag = False
|
| 191 |
+
self.fail_unknown_tag(token.value, token.lineno)
|
| 192 |
+
finally:
|
| 193 |
+
if pop_tag:
|
| 194 |
+
self._tag_stack.pop()
|
| 195 |
+
|
| 196 |
+
def parse_statements(
|
| 197 |
+
self, end_tokens: t.Tuple[str, ...], drop_needle: bool = False
|
| 198 |
+
) -> t.List[nodes.Node]:
|
| 199 |
+
"""Parse multiple statements into a list until one of the end tokens
|
| 200 |
+
is reached. This is used to parse the body of statements as it also
|
| 201 |
+
parses template data if appropriate. The parser checks first if the
|
| 202 |
+
current token is a colon and skips it if there is one. Then it checks
|
| 203 |
+
for the block end and parses until if one of the `end_tokens` is
|
| 204 |
+
reached. Per default the active token in the stream at the end of
|
| 205 |
+
the call is the matched end token. If this is not wanted `drop_needle`
|
| 206 |
+
can be set to `True` and the end token is removed.
|
| 207 |
+
"""
|
| 208 |
+
# the first token may be a colon for python compatibility
|
| 209 |
+
self.stream.skip_if("colon")
|
| 210 |
+
|
| 211 |
+
# in the future it would be possible to add whole code sections
|
| 212 |
+
# by adding some sort of end of statement token and parsing those here.
|
| 213 |
+
self.stream.expect("block_end")
|
| 214 |
+
result = self.subparse(end_tokens)
|
| 215 |
+
|
| 216 |
+
# we reached the end of the template too early, the subparser
|
| 217 |
+
# does not check for this, so we do that now
|
| 218 |
+
if self.stream.current.type == "eof":
|
| 219 |
+
self.fail_eof(end_tokens)
|
| 220 |
+
|
| 221 |
+
if drop_needle:
|
| 222 |
+
next(self.stream)
|
| 223 |
+
return result
|
| 224 |
+
|
| 225 |
+
def parse_set(self) -> t.Union[nodes.Assign, nodes.AssignBlock]:
|
| 226 |
+
"""Parse an assign statement."""
|
| 227 |
+
lineno = next(self.stream).lineno
|
| 228 |
+
target = self.parse_assign_target(with_namespace=True)
|
| 229 |
+
if self.stream.skip_if("assign"):
|
| 230 |
+
expr = self.parse_tuple()
|
| 231 |
+
return nodes.Assign(target, expr, lineno=lineno)
|
| 232 |
+
filter_node = self.parse_filter(None)
|
| 233 |
+
body = self.parse_statements(("name:endset",), drop_needle=True)
|
| 234 |
+
return nodes.AssignBlock(target, filter_node, body, lineno=lineno)
|
| 235 |
+
|
| 236 |
+
def parse_for(self) -> nodes.For:
|
| 237 |
+
"""Parse a for loop."""
|
| 238 |
+
lineno = self.stream.expect("name:for").lineno
|
| 239 |
+
target = self.parse_assign_target(extra_end_rules=("name:in",))
|
| 240 |
+
self.stream.expect("name:in")
|
| 241 |
+
iter = self.parse_tuple(
|
| 242 |
+
with_condexpr=False, extra_end_rules=("name:recursive",)
|
| 243 |
+
)
|
| 244 |
+
test = None
|
| 245 |
+
if self.stream.skip_if("name:if"):
|
| 246 |
+
test = self.parse_expression()
|
| 247 |
+
recursive = self.stream.skip_if("name:recursive")
|
| 248 |
+
body = self.parse_statements(("name:endfor", "name:else"))
|
| 249 |
+
if next(self.stream).value == "endfor":
|
| 250 |
+
else_ = []
|
| 251 |
+
else:
|
| 252 |
+
else_ = self.parse_statements(("name:endfor",), drop_needle=True)
|
| 253 |
+
return nodes.For(target, iter, body, else_, test, recursive, lineno=lineno)
|
| 254 |
+
|
| 255 |
+
def parse_if(self) -> nodes.If:
|
| 256 |
+
"""Parse an if construct."""
|
| 257 |
+
node = result = nodes.If(lineno=self.stream.expect("name:if").lineno)
|
| 258 |
+
while True:
|
| 259 |
+
node.test = self.parse_tuple(with_condexpr=False)
|
| 260 |
+
node.body = self.parse_statements(("name:elif", "name:else", "name:endif"))
|
| 261 |
+
node.elif_ = []
|
| 262 |
+
node.else_ = []
|
| 263 |
+
token = next(self.stream)
|
| 264 |
+
if token.test("name:elif"):
|
| 265 |
+
node = nodes.If(lineno=self.stream.current.lineno)
|
| 266 |
+
result.elif_.append(node)
|
| 267 |
+
continue
|
| 268 |
+
elif token.test("name:else"):
|
| 269 |
+
result.else_ = self.parse_statements(("name:endif",), drop_needle=True)
|
| 270 |
+
break
|
| 271 |
+
return result
|
| 272 |
+
|
| 273 |
+
def parse_with(self) -> nodes.With:
|
| 274 |
+
node = nodes.With(lineno=next(self.stream).lineno)
|
| 275 |
+
targets: t.List[nodes.Expr] = []
|
| 276 |
+
values: t.List[nodes.Expr] = []
|
| 277 |
+
while self.stream.current.type != "block_end":
|
| 278 |
+
if targets:
|
| 279 |
+
self.stream.expect("comma")
|
| 280 |
+
target = self.parse_assign_target()
|
| 281 |
+
target.set_ctx("param")
|
| 282 |
+
targets.append(target)
|
| 283 |
+
self.stream.expect("assign")
|
| 284 |
+
values.append(self.parse_expression())
|
| 285 |
+
node.targets = targets
|
| 286 |
+
node.values = values
|
| 287 |
+
node.body = self.parse_statements(("name:endwith",), drop_needle=True)
|
| 288 |
+
return node
|
| 289 |
+
|
| 290 |
+
def parse_autoescape(self) -> nodes.Scope:
|
| 291 |
+
node = nodes.ScopedEvalContextModifier(lineno=next(self.stream).lineno)
|
| 292 |
+
node.options = [nodes.Keyword("autoescape", self.parse_expression())]
|
| 293 |
+
node.body = self.parse_statements(("name:endautoescape",), drop_needle=True)
|
| 294 |
+
return nodes.Scope([node])
|
| 295 |
+
|
| 296 |
+
def parse_block(self) -> nodes.Block:
|
| 297 |
+
node = nodes.Block(lineno=next(self.stream).lineno)
|
| 298 |
+
node.name = self.stream.expect("name").value
|
| 299 |
+
node.scoped = self.stream.skip_if("name:scoped")
|
| 300 |
+
node.required = self.stream.skip_if("name:required")
|
| 301 |
+
|
| 302 |
+
# common problem people encounter when switching from django
|
| 303 |
+
# to jinja. we do not support hyphens in block names, so let's
|
| 304 |
+
# raise a nicer error message in that case.
|
| 305 |
+
if self.stream.current.type == "sub":
|
| 306 |
+
self.fail(
|
| 307 |
+
"Block names in Jinja have to be valid Python identifiers and may not"
|
| 308 |
+
" contain hyphens, use an underscore instead."
|
| 309 |
+
)
|
| 310 |
+
|
| 311 |
+
node.body = self.parse_statements(("name:endblock",), drop_needle=True)
|
| 312 |
+
|
| 313 |
+
# enforce that required blocks only contain whitespace or comments
|
| 314 |
+
# by asserting that the body, if not empty, is just TemplateData nodes
|
| 315 |
+
# with whitespace data
|
| 316 |
+
if node.required:
|
| 317 |
+
for body_node in node.body:
|
| 318 |
+
if not isinstance(body_node, nodes.Output) or any(
|
| 319 |
+
not isinstance(output_node, nodes.TemplateData)
|
| 320 |
+
or not output_node.data.isspace()
|
| 321 |
+
for output_node in body_node.nodes
|
| 322 |
+
):
|
| 323 |
+
self.fail("Required blocks can only contain comments or whitespace")
|
| 324 |
+
|
| 325 |
+
self.stream.skip_if("name:" + node.name)
|
| 326 |
+
return node
|
| 327 |
+
|
| 328 |
+
def parse_extends(self) -> nodes.Extends:
|
| 329 |
+
node = nodes.Extends(lineno=next(self.stream).lineno)
|
| 330 |
+
node.template = self.parse_expression()
|
| 331 |
+
return node
|
| 332 |
+
|
| 333 |
+
def parse_import_context(
|
| 334 |
+
self, node: _ImportInclude, default: bool
|
| 335 |
+
) -> _ImportInclude:
|
| 336 |
+
if self.stream.current.test_any(
|
| 337 |
+
"name:with", "name:without"
|
| 338 |
+
) and self.stream.look().test("name:context"):
|
| 339 |
+
node.with_context = next(self.stream).value == "with"
|
| 340 |
+
self.stream.skip()
|
| 341 |
+
else:
|
| 342 |
+
node.with_context = default
|
| 343 |
+
return node
|
| 344 |
+
|
| 345 |
+
def parse_include(self) -> nodes.Include:
|
| 346 |
+
node = nodes.Include(lineno=next(self.stream).lineno)
|
| 347 |
+
node.template = self.parse_expression()
|
| 348 |
+
if self.stream.current.test("name:ignore") and self.stream.look().test(
|
| 349 |
+
"name:missing"
|
| 350 |
+
):
|
| 351 |
+
node.ignore_missing = True
|
| 352 |
+
self.stream.skip(2)
|
| 353 |
+
else:
|
| 354 |
+
node.ignore_missing = False
|
| 355 |
+
return self.parse_import_context(node, True)
|
| 356 |
+
|
| 357 |
+
def parse_import(self) -> nodes.Import:
|
| 358 |
+
node = nodes.Import(lineno=next(self.stream).lineno)
|
| 359 |
+
node.template = self.parse_expression()
|
| 360 |
+
self.stream.expect("name:as")
|
| 361 |
+
node.target = self.parse_assign_target(name_only=True).name
|
| 362 |
+
return self.parse_import_context(node, False)
|
| 363 |
+
|
| 364 |
+
def parse_from(self) -> nodes.FromImport:
|
| 365 |
+
node = nodes.FromImport(lineno=next(self.stream).lineno)
|
| 366 |
+
node.template = self.parse_expression()
|
| 367 |
+
self.stream.expect("name:import")
|
| 368 |
+
node.names = []
|
| 369 |
+
|
| 370 |
+
def parse_context() -> bool:
|
| 371 |
+
if self.stream.current.value in {
|
| 372 |
+
"with",
|
| 373 |
+
"without",
|
| 374 |
+
} and self.stream.look().test("name:context"):
|
| 375 |
+
node.with_context = next(self.stream).value == "with"
|
| 376 |
+
self.stream.skip()
|
| 377 |
+
return True
|
| 378 |
+
return False
|
| 379 |
+
|
| 380 |
+
while True:
|
| 381 |
+
if node.names:
|
| 382 |
+
self.stream.expect("comma")
|
| 383 |
+
if self.stream.current.type == "name":
|
| 384 |
+
if parse_context():
|
| 385 |
+
break
|
| 386 |
+
target = self.parse_assign_target(name_only=True)
|
| 387 |
+
if target.name.startswith("_"):
|
| 388 |
+
self.fail(
|
| 389 |
+
"names starting with an underline can not be imported",
|
| 390 |
+
target.lineno,
|
| 391 |
+
exc=TemplateAssertionError,
|
| 392 |
+
)
|
| 393 |
+
if self.stream.skip_if("name:as"):
|
| 394 |
+
alias = self.parse_assign_target(name_only=True)
|
| 395 |
+
node.names.append((target.name, alias.name))
|
| 396 |
+
else:
|
| 397 |
+
node.names.append(target.name)
|
| 398 |
+
if parse_context() or self.stream.current.type != "comma":
|
| 399 |
+
break
|
| 400 |
+
else:
|
| 401 |
+
self.stream.expect("name")
|
| 402 |
+
if not hasattr(node, "with_context"):
|
| 403 |
+
node.with_context = False
|
| 404 |
+
return node
|
| 405 |
+
|
| 406 |
+
def parse_signature(self, node: _MacroCall) -> None:
|
| 407 |
+
args = node.args = []
|
| 408 |
+
defaults = node.defaults = []
|
| 409 |
+
self.stream.expect("lparen")
|
| 410 |
+
while self.stream.current.type != "rparen":
|
| 411 |
+
if args:
|
| 412 |
+
self.stream.expect("comma")
|
| 413 |
+
arg = self.parse_assign_target(name_only=True)
|
| 414 |
+
arg.set_ctx("param")
|
| 415 |
+
if self.stream.skip_if("assign"):
|
| 416 |
+
defaults.append(self.parse_expression())
|
| 417 |
+
elif defaults:
|
| 418 |
+
self.fail("non-default argument follows default argument")
|
| 419 |
+
args.append(arg)
|
| 420 |
+
self.stream.expect("rparen")
|
| 421 |
+
|
| 422 |
+
def parse_call_block(self) -> nodes.CallBlock:
|
| 423 |
+
node = nodes.CallBlock(lineno=next(self.stream).lineno)
|
| 424 |
+
if self.stream.current.type == "lparen":
|
| 425 |
+
self.parse_signature(node)
|
| 426 |
+
else:
|
| 427 |
+
node.args = []
|
| 428 |
+
node.defaults = []
|
| 429 |
+
|
| 430 |
+
call_node = self.parse_expression()
|
| 431 |
+
if not isinstance(call_node, nodes.Call):
|
| 432 |
+
self.fail("expected call", node.lineno)
|
| 433 |
+
node.call = call_node
|
| 434 |
+
node.body = self.parse_statements(("name:endcall",), drop_needle=True)
|
| 435 |
+
return node
|
| 436 |
+
|
| 437 |
+
def parse_filter_block(self) -> nodes.FilterBlock:
|
| 438 |
+
node = nodes.FilterBlock(lineno=next(self.stream).lineno)
|
| 439 |
+
node.filter = self.parse_filter(None, start_inline=True) # type: ignore
|
| 440 |
+
node.body = self.parse_statements(("name:endfilter",), drop_needle=True)
|
| 441 |
+
return node
|
| 442 |
+
|
| 443 |
+
def parse_macro(self) -> nodes.Macro:
|
| 444 |
+
node = nodes.Macro(lineno=next(self.stream).lineno)
|
| 445 |
+
node.name = self.parse_assign_target(name_only=True).name
|
| 446 |
+
self.parse_signature(node)
|
| 447 |
+
node.body = self.parse_statements(("name:endmacro",), drop_needle=True)
|
| 448 |
+
return node
|
| 449 |
+
|
| 450 |
+
def parse_print(self) -> nodes.Output:
|
| 451 |
+
node = nodes.Output(lineno=next(self.stream).lineno)
|
| 452 |
+
node.nodes = []
|
| 453 |
+
while self.stream.current.type != "block_end":
|
| 454 |
+
if node.nodes:
|
| 455 |
+
self.stream.expect("comma")
|
| 456 |
+
node.nodes.append(self.parse_expression())
|
| 457 |
+
return node
|
| 458 |
+
|
| 459 |
+
@typing.overload
|
| 460 |
+
def parse_assign_target(
|
| 461 |
+
self, with_tuple: bool = ..., name_only: "te.Literal[True]" = ...
|
| 462 |
+
) -> nodes.Name: ...
|
| 463 |
+
|
| 464 |
+
@typing.overload
|
| 465 |
+
def parse_assign_target(
|
| 466 |
+
self,
|
| 467 |
+
with_tuple: bool = True,
|
| 468 |
+
name_only: bool = False,
|
| 469 |
+
extra_end_rules: t.Optional[t.Tuple[str, ...]] = None,
|
| 470 |
+
with_namespace: bool = False,
|
| 471 |
+
) -> t.Union[nodes.NSRef, nodes.Name, nodes.Tuple]: ...
|
| 472 |
+
|
| 473 |
+
def parse_assign_target(
|
| 474 |
+
self,
|
| 475 |
+
with_tuple: bool = True,
|
| 476 |
+
name_only: bool = False,
|
| 477 |
+
extra_end_rules: t.Optional[t.Tuple[str, ...]] = None,
|
| 478 |
+
with_namespace: bool = False,
|
| 479 |
+
) -> t.Union[nodes.NSRef, nodes.Name, nodes.Tuple]:
|
| 480 |
+
"""Parse an assignment target. As Jinja allows assignments to
|
| 481 |
+
tuples, this function can parse all allowed assignment targets. Per
|
| 482 |
+
default assignments to tuples are parsed, that can be disable however
|
| 483 |
+
by setting `with_tuple` to `False`. If only assignments to names are
|
| 484 |
+
wanted `name_only` can be set to `True`. The `extra_end_rules`
|
| 485 |
+
parameter is forwarded to the tuple parsing function. If
|
| 486 |
+
`with_namespace` is enabled, a namespace assignment may be parsed.
|
| 487 |
+
"""
|
| 488 |
+
target: nodes.Expr
|
| 489 |
+
|
| 490 |
+
if name_only:
|
| 491 |
+
token = self.stream.expect("name")
|
| 492 |
+
target = nodes.Name(token.value, "store", lineno=token.lineno)
|
| 493 |
+
else:
|
| 494 |
+
if with_tuple:
|
| 495 |
+
target = self.parse_tuple(
|
| 496 |
+
simplified=True,
|
| 497 |
+
extra_end_rules=extra_end_rules,
|
| 498 |
+
with_namespace=with_namespace,
|
| 499 |
+
)
|
| 500 |
+
else:
|
| 501 |
+
target = self.parse_primary(with_namespace=with_namespace)
|
| 502 |
+
|
| 503 |
+
target.set_ctx("store")
|
| 504 |
+
|
| 505 |
+
if not target.can_assign():
|
| 506 |
+
self.fail(
|
| 507 |
+
f"can't assign to {type(target).__name__.lower()!r}", target.lineno
|
| 508 |
+
)
|
| 509 |
+
|
| 510 |
+
return target # type: ignore
|
| 511 |
+
|
| 512 |
+
def parse_expression(self, with_condexpr: bool = True) -> nodes.Expr:
|
| 513 |
+
"""Parse an expression. Per default all expressions are parsed, if
|
| 514 |
+
the optional `with_condexpr` parameter is set to `False` conditional
|
| 515 |
+
expressions are not parsed.
|
| 516 |
+
"""
|
| 517 |
+
if with_condexpr:
|
| 518 |
+
return self.parse_condexpr()
|
| 519 |
+
return self.parse_or()
|
| 520 |
+
|
| 521 |
+
def parse_condexpr(self) -> nodes.Expr:
|
| 522 |
+
lineno = self.stream.current.lineno
|
| 523 |
+
expr1 = self.parse_or()
|
| 524 |
+
expr3: t.Optional[nodes.Expr]
|
| 525 |
+
|
| 526 |
+
while self.stream.skip_if("name:if"):
|
| 527 |
+
expr2 = self.parse_or()
|
| 528 |
+
if self.stream.skip_if("name:else"):
|
| 529 |
+
expr3 = self.parse_condexpr()
|
| 530 |
+
else:
|
| 531 |
+
expr3 = None
|
| 532 |
+
expr1 = nodes.CondExpr(expr2, expr1, expr3, lineno=lineno)
|
| 533 |
+
lineno = self.stream.current.lineno
|
| 534 |
+
return expr1
|
| 535 |
+
|
| 536 |
+
def parse_or(self) -> nodes.Expr:
|
| 537 |
+
lineno = self.stream.current.lineno
|
| 538 |
+
left = self.parse_and()
|
| 539 |
+
while self.stream.skip_if("name:or"):
|
| 540 |
+
right = self.parse_and()
|
| 541 |
+
left = nodes.Or(left, right, lineno=lineno)
|
| 542 |
+
lineno = self.stream.current.lineno
|
| 543 |
+
return left
|
| 544 |
+
|
| 545 |
+
def parse_and(self) -> nodes.Expr:
|
| 546 |
+
lineno = self.stream.current.lineno
|
| 547 |
+
left = self.parse_not()
|
| 548 |
+
while self.stream.skip_if("name:and"):
|
| 549 |
+
right = self.parse_not()
|
| 550 |
+
left = nodes.And(left, right, lineno=lineno)
|
| 551 |
+
lineno = self.stream.current.lineno
|
| 552 |
+
return left
|
| 553 |
+
|
| 554 |
+
def parse_not(self) -> nodes.Expr:
|
| 555 |
+
if self.stream.current.test("name:not"):
|
| 556 |
+
lineno = next(self.stream).lineno
|
| 557 |
+
return nodes.Not(self.parse_not(), lineno=lineno)
|
| 558 |
+
return self.parse_compare()
|
| 559 |
+
|
| 560 |
+
def parse_compare(self) -> nodes.Expr:
|
| 561 |
+
lineno = self.stream.current.lineno
|
| 562 |
+
expr = self.parse_math1()
|
| 563 |
+
ops = []
|
| 564 |
+
while True:
|
| 565 |
+
token_type = self.stream.current.type
|
| 566 |
+
if token_type in _compare_operators:
|
| 567 |
+
next(self.stream)
|
| 568 |
+
ops.append(nodes.Operand(token_type, self.parse_math1()))
|
| 569 |
+
elif self.stream.skip_if("name:in"):
|
| 570 |
+
ops.append(nodes.Operand("in", self.parse_math1()))
|
| 571 |
+
elif self.stream.current.test("name:not") and self.stream.look().test(
|
| 572 |
+
"name:in"
|
| 573 |
+
):
|
| 574 |
+
self.stream.skip(2)
|
| 575 |
+
ops.append(nodes.Operand("notin", self.parse_math1()))
|
| 576 |
+
else:
|
| 577 |
+
break
|
| 578 |
+
lineno = self.stream.current.lineno
|
| 579 |
+
if not ops:
|
| 580 |
+
return expr
|
| 581 |
+
return nodes.Compare(expr, ops, lineno=lineno)
|
| 582 |
+
|
| 583 |
+
def parse_math1(self) -> nodes.Expr:
|
| 584 |
+
lineno = self.stream.current.lineno
|
| 585 |
+
left = self.parse_concat()
|
| 586 |
+
while self.stream.current.type in ("add", "sub"):
|
| 587 |
+
cls = _math_nodes[self.stream.current.type]
|
| 588 |
+
next(self.stream)
|
| 589 |
+
right = self.parse_concat()
|
| 590 |
+
left = cls(left, right, lineno=lineno)
|
| 591 |
+
lineno = self.stream.current.lineno
|
| 592 |
+
return left
|
| 593 |
+
|
| 594 |
+
def parse_concat(self) -> nodes.Expr:
|
| 595 |
+
lineno = self.stream.current.lineno
|
| 596 |
+
args = [self.parse_math2()]
|
| 597 |
+
while self.stream.current.type == "tilde":
|
| 598 |
+
next(self.stream)
|
| 599 |
+
args.append(self.parse_math2())
|
| 600 |
+
if len(args) == 1:
|
| 601 |
+
return args[0]
|
| 602 |
+
return nodes.Concat(args, lineno=lineno)
|
| 603 |
+
|
| 604 |
+
def parse_math2(self) -> nodes.Expr:
|
| 605 |
+
lineno = self.stream.current.lineno
|
| 606 |
+
left = self.parse_pow()
|
| 607 |
+
while self.stream.current.type in ("mul", "div", "floordiv", "mod"):
|
| 608 |
+
cls = _math_nodes[self.stream.current.type]
|
| 609 |
+
next(self.stream)
|
| 610 |
+
right = self.parse_pow()
|
| 611 |
+
left = cls(left, right, lineno=lineno)
|
| 612 |
+
lineno = self.stream.current.lineno
|
| 613 |
+
return left
|
| 614 |
+
|
| 615 |
+
def parse_pow(self) -> nodes.Expr:
|
| 616 |
+
lineno = self.stream.current.lineno
|
| 617 |
+
left = self.parse_unary()
|
| 618 |
+
while self.stream.current.type == "pow":
|
| 619 |
+
next(self.stream)
|
| 620 |
+
right = self.parse_unary()
|
| 621 |
+
left = nodes.Pow(left, right, lineno=lineno)
|
| 622 |
+
lineno = self.stream.current.lineno
|
| 623 |
+
return left
|
| 624 |
+
|
| 625 |
+
def parse_unary(self, with_filter: bool = True) -> nodes.Expr:
|
| 626 |
+
token_type = self.stream.current.type
|
| 627 |
+
lineno = self.stream.current.lineno
|
| 628 |
+
node: nodes.Expr
|
| 629 |
+
|
| 630 |
+
if token_type == "sub":
|
| 631 |
+
next(self.stream)
|
| 632 |
+
node = nodes.Neg(self.parse_unary(False), lineno=lineno)
|
| 633 |
+
elif token_type == "add":
|
| 634 |
+
next(self.stream)
|
| 635 |
+
node = nodes.Pos(self.parse_unary(False), lineno=lineno)
|
| 636 |
+
else:
|
| 637 |
+
node = self.parse_primary()
|
| 638 |
+
node = self.parse_postfix(node)
|
| 639 |
+
if with_filter:
|
| 640 |
+
node = self.parse_filter_expr(node)
|
| 641 |
+
return node
|
| 642 |
+
|
| 643 |
+
def parse_primary(self, with_namespace: bool = False) -> nodes.Expr:
|
| 644 |
+
"""Parse a name or literal value. If ``with_namespace`` is enabled, also
|
| 645 |
+
parse namespace attr refs, for use in assignments."""
|
| 646 |
+
token = self.stream.current
|
| 647 |
+
node: nodes.Expr
|
| 648 |
+
if token.type == "name":
|
| 649 |
+
next(self.stream)
|
| 650 |
+
if token.value in ("true", "false", "True", "False"):
|
| 651 |
+
node = nodes.Const(token.value in ("true", "True"), lineno=token.lineno)
|
| 652 |
+
elif token.value in ("none", "None"):
|
| 653 |
+
node = nodes.Const(None, lineno=token.lineno)
|
| 654 |
+
elif with_namespace and self.stream.current.type == "dot":
|
| 655 |
+
# If namespace attributes are allowed at this point, and the next
|
| 656 |
+
# token is a dot, produce a namespace reference.
|
| 657 |
+
next(self.stream)
|
| 658 |
+
attr = self.stream.expect("name")
|
| 659 |
+
node = nodes.NSRef(token.value, attr.value, lineno=token.lineno)
|
| 660 |
+
else:
|
| 661 |
+
node = nodes.Name(token.value, "load", lineno=token.lineno)
|
| 662 |
+
elif token.type == "string":
|
| 663 |
+
next(self.stream)
|
| 664 |
+
buf = [token.value]
|
| 665 |
+
lineno = token.lineno
|
| 666 |
+
while self.stream.current.type == "string":
|
| 667 |
+
buf.append(self.stream.current.value)
|
| 668 |
+
next(self.stream)
|
| 669 |
+
node = nodes.Const("".join(buf), lineno=lineno)
|
| 670 |
+
elif token.type in ("integer", "float"):
|
| 671 |
+
next(self.stream)
|
| 672 |
+
node = nodes.Const(token.value, lineno=token.lineno)
|
| 673 |
+
elif token.type == "lparen":
|
| 674 |
+
next(self.stream)
|
| 675 |
+
node = self.parse_tuple(explicit_parentheses=True)
|
| 676 |
+
self.stream.expect("rparen")
|
| 677 |
+
elif token.type == "lbracket":
|
| 678 |
+
node = self.parse_list()
|
| 679 |
+
elif token.type == "lbrace":
|
| 680 |
+
node = self.parse_dict()
|
| 681 |
+
else:
|
| 682 |
+
self.fail(f"unexpected {describe_token(token)!r}", token.lineno)
|
| 683 |
+
return node
|
| 684 |
+
|
| 685 |
+
def parse_tuple(
|
| 686 |
+
self,
|
| 687 |
+
simplified: bool = False,
|
| 688 |
+
with_condexpr: bool = True,
|
| 689 |
+
extra_end_rules: t.Optional[t.Tuple[str, ...]] = None,
|
| 690 |
+
explicit_parentheses: bool = False,
|
| 691 |
+
with_namespace: bool = False,
|
| 692 |
+
) -> t.Union[nodes.Tuple, nodes.Expr]:
|
| 693 |
+
"""Works like `parse_expression` but if multiple expressions are
|
| 694 |
+
delimited by a comma a :class:`~jinja2.nodes.Tuple` node is created.
|
| 695 |
+
This method could also return a regular expression instead of a tuple
|
| 696 |
+
if no commas where found.
|
| 697 |
+
|
| 698 |
+
The default parsing mode is a full tuple. If `simplified` is `True`
|
| 699 |
+
only names and literals are parsed; ``with_namespace`` allows namespace
|
| 700 |
+
attr refs as well. The `no_condexpr` parameter is forwarded to
|
| 701 |
+
:meth:`parse_expression`.
|
| 702 |
+
|
| 703 |
+
Because tuples do not require delimiters and may end in a bogus comma
|
| 704 |
+
an extra hint is needed that marks the end of a tuple. For example
|
| 705 |
+
for loops support tuples between `for` and `in`. In that case the
|
| 706 |
+
`extra_end_rules` is set to ``['name:in']``.
|
| 707 |
+
|
| 708 |
+
`explicit_parentheses` is true if the parsing was triggered by an
|
| 709 |
+
expression in parentheses. This is used to figure out if an empty
|
| 710 |
+
tuple is a valid expression or not.
|
| 711 |
+
"""
|
| 712 |
+
lineno = self.stream.current.lineno
|
| 713 |
+
if simplified:
|
| 714 |
+
|
| 715 |
+
def parse() -> nodes.Expr:
|
| 716 |
+
return self.parse_primary(with_namespace=with_namespace)
|
| 717 |
+
|
| 718 |
+
else:
|
| 719 |
+
|
| 720 |
+
def parse() -> nodes.Expr:
|
| 721 |
+
return self.parse_expression(with_condexpr=with_condexpr)
|
| 722 |
+
|
| 723 |
+
args: t.List[nodes.Expr] = []
|
| 724 |
+
is_tuple = False
|
| 725 |
+
|
| 726 |
+
while True:
|
| 727 |
+
if args:
|
| 728 |
+
self.stream.expect("comma")
|
| 729 |
+
if self.is_tuple_end(extra_end_rules):
|
| 730 |
+
break
|
| 731 |
+
args.append(parse())
|
| 732 |
+
if self.stream.current.type == "comma":
|
| 733 |
+
is_tuple = True
|
| 734 |
+
else:
|
| 735 |
+
break
|
| 736 |
+
lineno = self.stream.current.lineno
|
| 737 |
+
|
| 738 |
+
if not is_tuple:
|
| 739 |
+
if args:
|
| 740 |
+
return args[0]
|
| 741 |
+
|
| 742 |
+
# if we don't have explicit parentheses, an empty tuple is
|
| 743 |
+
# not a valid expression. This would mean nothing (literally
|
| 744 |
+
# nothing) in the spot of an expression would be an empty
|
| 745 |
+
# tuple.
|
| 746 |
+
if not explicit_parentheses:
|
| 747 |
+
self.fail(
|
| 748 |
+
"Expected an expression,"
|
| 749 |
+
f" got {describe_token(self.stream.current)!r}"
|
| 750 |
+
)
|
| 751 |
+
|
| 752 |
+
return nodes.Tuple(args, "load", lineno=lineno)
|
| 753 |
+
|
| 754 |
+
def parse_list(self) -> nodes.List:
|
| 755 |
+
token = self.stream.expect("lbracket")
|
| 756 |
+
items: t.List[nodes.Expr] = []
|
| 757 |
+
while self.stream.current.type != "rbracket":
|
| 758 |
+
if items:
|
| 759 |
+
self.stream.expect("comma")
|
| 760 |
+
if self.stream.current.type == "rbracket":
|
| 761 |
+
break
|
| 762 |
+
items.append(self.parse_expression())
|
| 763 |
+
self.stream.expect("rbracket")
|
| 764 |
+
return nodes.List(items, lineno=token.lineno)
|
| 765 |
+
|
| 766 |
+
def parse_dict(self) -> nodes.Dict:
|
| 767 |
+
token = self.stream.expect("lbrace")
|
| 768 |
+
items: t.List[nodes.Pair] = []
|
| 769 |
+
while self.stream.current.type != "rbrace":
|
| 770 |
+
if items:
|
| 771 |
+
self.stream.expect("comma")
|
| 772 |
+
if self.stream.current.type == "rbrace":
|
| 773 |
+
break
|
| 774 |
+
key = self.parse_expression()
|
| 775 |
+
self.stream.expect("colon")
|
| 776 |
+
value = self.parse_expression()
|
| 777 |
+
items.append(nodes.Pair(key, value, lineno=key.lineno))
|
| 778 |
+
self.stream.expect("rbrace")
|
| 779 |
+
return nodes.Dict(items, lineno=token.lineno)
|
| 780 |
+
|
| 781 |
+
def parse_postfix(self, node: nodes.Expr) -> nodes.Expr:
|
| 782 |
+
while True:
|
| 783 |
+
token_type = self.stream.current.type
|
| 784 |
+
if token_type == "dot" or token_type == "lbracket":
|
| 785 |
+
node = self.parse_subscript(node)
|
| 786 |
+
# calls are valid both after postfix expressions (getattr
|
| 787 |
+
# and getitem) as well as filters and tests
|
| 788 |
+
elif token_type == "lparen":
|
| 789 |
+
node = self.parse_call(node)
|
| 790 |
+
else:
|
| 791 |
+
break
|
| 792 |
+
return node
|
| 793 |
+
|
| 794 |
+
def parse_filter_expr(self, node: nodes.Expr) -> nodes.Expr:
|
| 795 |
+
while True:
|
| 796 |
+
token_type = self.stream.current.type
|
| 797 |
+
if token_type == "pipe":
|
| 798 |
+
node = self.parse_filter(node) # type: ignore
|
| 799 |
+
elif token_type == "name" and self.stream.current.value == "is":
|
| 800 |
+
node = self.parse_test(node)
|
| 801 |
+
# calls are valid both after postfix expressions (getattr
|
| 802 |
+
# and getitem) as well as filters and tests
|
| 803 |
+
elif token_type == "lparen":
|
| 804 |
+
node = self.parse_call(node)
|
| 805 |
+
else:
|
| 806 |
+
break
|
| 807 |
+
return node
|
| 808 |
+
|
| 809 |
+
def parse_subscript(
|
| 810 |
+
self, node: nodes.Expr
|
| 811 |
+
) -> t.Union[nodes.Getattr, nodes.Getitem]:
|
| 812 |
+
token = next(self.stream)
|
| 813 |
+
arg: nodes.Expr
|
| 814 |
+
|
| 815 |
+
if token.type == "dot":
|
| 816 |
+
attr_token = self.stream.current
|
| 817 |
+
next(self.stream)
|
| 818 |
+
if attr_token.type == "name":
|
| 819 |
+
return nodes.Getattr(
|
| 820 |
+
node, attr_token.value, "load", lineno=token.lineno
|
| 821 |
+
)
|
| 822 |
+
elif attr_token.type != "integer":
|
| 823 |
+
self.fail("expected name or number", attr_token.lineno)
|
| 824 |
+
arg = nodes.Const(attr_token.value, lineno=attr_token.lineno)
|
| 825 |
+
return nodes.Getitem(node, arg, "load", lineno=token.lineno)
|
| 826 |
+
if token.type == "lbracket":
|
| 827 |
+
args: t.List[nodes.Expr] = []
|
| 828 |
+
while self.stream.current.type != "rbracket":
|
| 829 |
+
if args:
|
| 830 |
+
self.stream.expect("comma")
|
| 831 |
+
args.append(self.parse_subscribed())
|
| 832 |
+
self.stream.expect("rbracket")
|
| 833 |
+
if len(args) == 1:
|
| 834 |
+
arg = args[0]
|
| 835 |
+
else:
|
| 836 |
+
arg = nodes.Tuple(args, "load", lineno=token.lineno)
|
| 837 |
+
return nodes.Getitem(node, arg, "load", lineno=token.lineno)
|
| 838 |
+
self.fail("expected subscript expression", token.lineno)
|
| 839 |
+
|
| 840 |
+
def parse_subscribed(self) -> nodes.Expr:
|
| 841 |
+
lineno = self.stream.current.lineno
|
| 842 |
+
args: t.List[t.Optional[nodes.Expr]]
|
| 843 |
+
|
| 844 |
+
if self.stream.current.type == "colon":
|
| 845 |
+
next(self.stream)
|
| 846 |
+
args = [None]
|
| 847 |
+
else:
|
| 848 |
+
node = self.parse_expression()
|
| 849 |
+
if self.stream.current.type != "colon":
|
| 850 |
+
return node
|
| 851 |
+
next(self.stream)
|
| 852 |
+
args = [node]
|
| 853 |
+
|
| 854 |
+
if self.stream.current.type == "colon":
|
| 855 |
+
args.append(None)
|
| 856 |
+
elif self.stream.current.type not in ("rbracket", "comma"):
|
| 857 |
+
args.append(self.parse_expression())
|
| 858 |
+
else:
|
| 859 |
+
args.append(None)
|
| 860 |
+
|
| 861 |
+
if self.stream.current.type == "colon":
|
| 862 |
+
next(self.stream)
|
| 863 |
+
if self.stream.current.type not in ("rbracket", "comma"):
|
| 864 |
+
args.append(self.parse_expression())
|
| 865 |
+
else:
|
| 866 |
+
args.append(None)
|
| 867 |
+
else:
|
| 868 |
+
args.append(None)
|
| 869 |
+
|
| 870 |
+
return nodes.Slice(lineno=lineno, *args) # noqa: B026
|
| 871 |
+
|
| 872 |
+
def parse_call_args(
|
| 873 |
+
self,
|
| 874 |
+
) -> t.Tuple[
|
| 875 |
+
t.List[nodes.Expr],
|
| 876 |
+
t.List[nodes.Keyword],
|
| 877 |
+
t.Optional[nodes.Expr],
|
| 878 |
+
t.Optional[nodes.Expr],
|
| 879 |
+
]:
|
| 880 |
+
token = self.stream.expect("lparen")
|
| 881 |
+
args = []
|
| 882 |
+
kwargs = []
|
| 883 |
+
dyn_args = None
|
| 884 |
+
dyn_kwargs = None
|
| 885 |
+
require_comma = False
|
| 886 |
+
|
| 887 |
+
def ensure(expr: bool) -> None:
|
| 888 |
+
if not expr:
|
| 889 |
+
self.fail("invalid syntax for function call expression", token.lineno)
|
| 890 |
+
|
| 891 |
+
while self.stream.current.type != "rparen":
|
| 892 |
+
if require_comma:
|
| 893 |
+
self.stream.expect("comma")
|
| 894 |
+
|
| 895 |
+
# support for trailing comma
|
| 896 |
+
if self.stream.current.type == "rparen":
|
| 897 |
+
break
|
| 898 |
+
|
| 899 |
+
if self.stream.current.type == "mul":
|
| 900 |
+
ensure(dyn_args is None and dyn_kwargs is None)
|
| 901 |
+
next(self.stream)
|
| 902 |
+
dyn_args = self.parse_expression()
|
| 903 |
+
elif self.stream.current.type == "pow":
|
| 904 |
+
ensure(dyn_kwargs is None)
|
| 905 |
+
next(self.stream)
|
| 906 |
+
dyn_kwargs = self.parse_expression()
|
| 907 |
+
else:
|
| 908 |
+
if (
|
| 909 |
+
self.stream.current.type == "name"
|
| 910 |
+
and self.stream.look().type == "assign"
|
| 911 |
+
):
|
| 912 |
+
# Parsing a kwarg
|
| 913 |
+
ensure(dyn_kwargs is None)
|
| 914 |
+
key = self.stream.current.value
|
| 915 |
+
self.stream.skip(2)
|
| 916 |
+
value = self.parse_expression()
|
| 917 |
+
kwargs.append(nodes.Keyword(key, value, lineno=value.lineno))
|
| 918 |
+
else:
|
| 919 |
+
# Parsing an arg
|
| 920 |
+
ensure(dyn_args is None and dyn_kwargs is None and not kwargs)
|
| 921 |
+
args.append(self.parse_expression())
|
| 922 |
+
|
| 923 |
+
require_comma = True
|
| 924 |
+
|
| 925 |
+
self.stream.expect("rparen")
|
| 926 |
+
return args, kwargs, dyn_args, dyn_kwargs
|
| 927 |
+
|
| 928 |
+
def parse_call(self, node: nodes.Expr) -> nodes.Call:
|
| 929 |
+
# The lparen will be expected in parse_call_args, but the lineno
|
| 930 |
+
# needs to be recorded before the stream is advanced.
|
| 931 |
+
token = self.stream.current
|
| 932 |
+
args, kwargs, dyn_args, dyn_kwargs = self.parse_call_args()
|
| 933 |
+
return nodes.Call(node, args, kwargs, dyn_args, dyn_kwargs, lineno=token.lineno)
|
| 934 |
+
|
| 935 |
+
def parse_filter(
|
| 936 |
+
self, node: t.Optional[nodes.Expr], start_inline: bool = False
|
| 937 |
+
) -> t.Optional[nodes.Expr]:
|
| 938 |
+
while self.stream.current.type == "pipe" or start_inline:
|
| 939 |
+
if not start_inline:
|
| 940 |
+
next(self.stream)
|
| 941 |
+
token = self.stream.expect("name")
|
| 942 |
+
name = token.value
|
| 943 |
+
while self.stream.current.type == "dot":
|
| 944 |
+
next(self.stream)
|
| 945 |
+
name += "." + self.stream.expect("name").value
|
| 946 |
+
if self.stream.current.type == "lparen":
|
| 947 |
+
args, kwargs, dyn_args, dyn_kwargs = self.parse_call_args()
|
| 948 |
+
else:
|
| 949 |
+
args = []
|
| 950 |
+
kwargs = []
|
| 951 |
+
dyn_args = dyn_kwargs = None
|
| 952 |
+
node = nodes.Filter(
|
| 953 |
+
node, name, args, kwargs, dyn_args, dyn_kwargs, lineno=token.lineno
|
| 954 |
+
)
|
| 955 |
+
start_inline = False
|
| 956 |
+
return node
|
| 957 |
+
|
| 958 |
+
def parse_test(self, node: nodes.Expr) -> nodes.Expr:
|
| 959 |
+
token = next(self.stream)
|
| 960 |
+
if self.stream.current.test("name:not"):
|
| 961 |
+
next(self.stream)
|
| 962 |
+
negated = True
|
| 963 |
+
else:
|
| 964 |
+
negated = False
|
| 965 |
+
name = self.stream.expect("name").value
|
| 966 |
+
while self.stream.current.type == "dot":
|
| 967 |
+
next(self.stream)
|
| 968 |
+
name += "." + self.stream.expect("name").value
|
| 969 |
+
dyn_args = dyn_kwargs = None
|
| 970 |
+
kwargs: t.List[nodes.Keyword] = []
|
| 971 |
+
if self.stream.current.type == "lparen":
|
| 972 |
+
args, kwargs, dyn_args, dyn_kwargs = self.parse_call_args()
|
| 973 |
+
elif self.stream.current.type in {
|
| 974 |
+
"name",
|
| 975 |
+
"string",
|
| 976 |
+
"integer",
|
| 977 |
+
"float",
|
| 978 |
+
"lparen",
|
| 979 |
+
"lbracket",
|
| 980 |
+
"lbrace",
|
| 981 |
+
} and not self.stream.current.test_any("name:else", "name:or", "name:and"):
|
| 982 |
+
if self.stream.current.test("name:is"):
|
| 983 |
+
self.fail("You cannot chain multiple tests with is")
|
| 984 |
+
arg_node = self.parse_primary()
|
| 985 |
+
arg_node = self.parse_postfix(arg_node)
|
| 986 |
+
args = [arg_node]
|
| 987 |
+
else:
|
| 988 |
+
args = []
|
| 989 |
+
node = nodes.Test(
|
| 990 |
+
node, name, args, kwargs, dyn_args, dyn_kwargs, lineno=token.lineno
|
| 991 |
+
)
|
| 992 |
+
if negated:
|
| 993 |
+
node = nodes.Not(node, lineno=token.lineno)
|
| 994 |
+
return node
|
| 995 |
+
|
| 996 |
+
def subparse(
|
| 997 |
+
self, end_tokens: t.Optional[t.Tuple[str, ...]] = None
|
| 998 |
+
) -> t.List[nodes.Node]:
|
| 999 |
+
body: t.List[nodes.Node] = []
|
| 1000 |
+
data_buffer: t.List[nodes.Node] = []
|
| 1001 |
+
add_data = data_buffer.append
|
| 1002 |
+
|
| 1003 |
+
if end_tokens is not None:
|
| 1004 |
+
self._end_token_stack.append(end_tokens)
|
| 1005 |
+
|
| 1006 |
+
def flush_data() -> None:
|
| 1007 |
+
if data_buffer:
|
| 1008 |
+
lineno = data_buffer[0].lineno
|
| 1009 |
+
body.append(nodes.Output(data_buffer[:], lineno=lineno))
|
| 1010 |
+
del data_buffer[:]
|
| 1011 |
+
|
| 1012 |
+
try:
|
| 1013 |
+
while self.stream:
|
| 1014 |
+
token = self.stream.current
|
| 1015 |
+
if token.type == "data":
|
| 1016 |
+
if token.value:
|
| 1017 |
+
add_data(nodes.TemplateData(token.value, lineno=token.lineno))
|
| 1018 |
+
next(self.stream)
|
| 1019 |
+
elif token.type == "variable_begin":
|
| 1020 |
+
next(self.stream)
|
| 1021 |
+
add_data(self.parse_tuple(with_condexpr=True))
|
| 1022 |
+
self.stream.expect("variable_end")
|
| 1023 |
+
elif token.type == "block_begin":
|
| 1024 |
+
flush_data()
|
| 1025 |
+
next(self.stream)
|
| 1026 |
+
if end_tokens is not None and self.stream.current.test_any(
|
| 1027 |
+
*end_tokens
|
| 1028 |
+
):
|
| 1029 |
+
return body
|
| 1030 |
+
rv = self.parse_statement()
|
| 1031 |
+
if isinstance(rv, list):
|
| 1032 |
+
body.extend(rv)
|
| 1033 |
+
else:
|
| 1034 |
+
body.append(rv)
|
| 1035 |
+
self.stream.expect("block_end")
|
| 1036 |
+
else:
|
| 1037 |
+
raise AssertionError("internal parsing error")
|
| 1038 |
+
|
| 1039 |
+
flush_data()
|
| 1040 |
+
finally:
|
| 1041 |
+
if end_tokens is not None:
|
| 1042 |
+
self._end_token_stack.pop()
|
| 1043 |
+
return body
|
| 1044 |
+
|
| 1045 |
+
def parse(self) -> nodes.Template:
|
| 1046 |
+
"""Parse the whole template into a `Template` node."""
|
| 1047 |
+
result = nodes.Template(self.subparse(), lineno=1)
|
| 1048 |
+
result.set_environment(self.environment)
|
| 1049 |
+
return result
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/py.typed
ADDED
|
File without changes
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/runtime.py
ADDED
|
@@ -0,0 +1,1062 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The runtime functions and state used by compiled templates."""
|
| 2 |
+
|
| 3 |
+
import functools
|
| 4 |
+
import sys
|
| 5 |
+
import typing as t
|
| 6 |
+
from collections import abc
|
| 7 |
+
from itertools import chain
|
| 8 |
+
|
| 9 |
+
from markupsafe import escape # noqa: F401
|
| 10 |
+
from markupsafe import Markup
|
| 11 |
+
from markupsafe import soft_str
|
| 12 |
+
|
| 13 |
+
from .async_utils import auto_aiter
|
| 14 |
+
from .async_utils import auto_await # noqa: F401
|
| 15 |
+
from .exceptions import TemplateNotFound # noqa: F401
|
| 16 |
+
from .exceptions import TemplateRuntimeError # noqa: F401
|
| 17 |
+
from .exceptions import UndefinedError
|
| 18 |
+
from .nodes import EvalContext
|
| 19 |
+
from .utils import _PassArg
|
| 20 |
+
from .utils import concat
|
| 21 |
+
from .utils import internalcode
|
| 22 |
+
from .utils import missing
|
| 23 |
+
from .utils import Namespace # noqa: F401
|
| 24 |
+
from .utils import object_type_repr
|
| 25 |
+
from .utils import pass_eval_context
|
| 26 |
+
|
| 27 |
+
V = t.TypeVar("V")
|
| 28 |
+
F = t.TypeVar("F", bound=t.Callable[..., t.Any])
|
| 29 |
+
|
| 30 |
+
if t.TYPE_CHECKING:
|
| 31 |
+
import logging
|
| 32 |
+
|
| 33 |
+
import typing_extensions as te
|
| 34 |
+
|
| 35 |
+
from .environment import Environment
|
| 36 |
+
|
| 37 |
+
class LoopRenderFunc(te.Protocol):
|
| 38 |
+
def __call__(
|
| 39 |
+
self,
|
| 40 |
+
reciter: t.Iterable[V],
|
| 41 |
+
loop_render_func: "LoopRenderFunc",
|
| 42 |
+
depth: int = 0,
|
| 43 |
+
) -> str: ...
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
# these variables are exported to the template runtime
|
| 47 |
+
exported = [
|
| 48 |
+
"LoopContext",
|
| 49 |
+
"TemplateReference",
|
| 50 |
+
"Macro",
|
| 51 |
+
"Markup",
|
| 52 |
+
"TemplateRuntimeError",
|
| 53 |
+
"missing",
|
| 54 |
+
"escape",
|
| 55 |
+
"markup_join",
|
| 56 |
+
"str_join",
|
| 57 |
+
"identity",
|
| 58 |
+
"TemplateNotFound",
|
| 59 |
+
"Namespace",
|
| 60 |
+
"Undefined",
|
| 61 |
+
"internalcode",
|
| 62 |
+
]
|
| 63 |
+
async_exported = [
|
| 64 |
+
"AsyncLoopContext",
|
| 65 |
+
"auto_aiter",
|
| 66 |
+
"auto_await",
|
| 67 |
+
]
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def identity(x: V) -> V:
|
| 71 |
+
"""Returns its argument. Useful for certain things in the
|
| 72 |
+
environment.
|
| 73 |
+
"""
|
| 74 |
+
return x
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def markup_join(seq: t.Iterable[t.Any]) -> str:
|
| 78 |
+
"""Concatenation that escapes if necessary and converts to string."""
|
| 79 |
+
buf = []
|
| 80 |
+
iterator = map(soft_str, seq)
|
| 81 |
+
for arg in iterator:
|
| 82 |
+
buf.append(arg)
|
| 83 |
+
if hasattr(arg, "__html__"):
|
| 84 |
+
return Markup("").join(chain(buf, iterator))
|
| 85 |
+
return concat(buf)
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def str_join(seq: t.Iterable[t.Any]) -> str:
|
| 89 |
+
"""Simple args to string conversion and concatenation."""
|
| 90 |
+
return concat(map(str, seq))
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def new_context(
|
| 94 |
+
environment: "Environment",
|
| 95 |
+
template_name: t.Optional[str],
|
| 96 |
+
blocks: t.Dict[str, t.Callable[["Context"], t.Iterator[str]]],
|
| 97 |
+
vars: t.Optional[t.Dict[str, t.Any]] = None,
|
| 98 |
+
shared: bool = False,
|
| 99 |
+
globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
|
| 100 |
+
locals: t.Optional[t.Mapping[str, t.Any]] = None,
|
| 101 |
+
) -> "Context":
|
| 102 |
+
"""Internal helper for context creation."""
|
| 103 |
+
if vars is None:
|
| 104 |
+
vars = {}
|
| 105 |
+
if shared:
|
| 106 |
+
parent = vars
|
| 107 |
+
else:
|
| 108 |
+
parent = dict(globals or (), **vars)
|
| 109 |
+
if locals:
|
| 110 |
+
# if the parent is shared a copy should be created because
|
| 111 |
+
# we don't want to modify the dict passed
|
| 112 |
+
if shared:
|
| 113 |
+
parent = dict(parent)
|
| 114 |
+
for key, value in locals.items():
|
| 115 |
+
if value is not missing:
|
| 116 |
+
parent[key] = value
|
| 117 |
+
return environment.context_class(
|
| 118 |
+
environment, parent, template_name, blocks, globals=globals
|
| 119 |
+
)
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
class TemplateReference:
|
| 123 |
+
"""The `self` in templates."""
|
| 124 |
+
|
| 125 |
+
def __init__(self, context: "Context") -> None:
|
| 126 |
+
self.__context = context
|
| 127 |
+
|
| 128 |
+
def __getitem__(self, name: str) -> t.Any:
|
| 129 |
+
blocks = self.__context.blocks[name]
|
| 130 |
+
return BlockReference(name, self.__context, blocks, 0)
|
| 131 |
+
|
| 132 |
+
def __repr__(self) -> str:
|
| 133 |
+
return f"<{type(self).__name__} {self.__context.name!r}>"
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def _dict_method_all(dict_method: F) -> F:
|
| 137 |
+
@functools.wraps(dict_method)
|
| 138 |
+
def f_all(self: "Context") -> t.Any:
|
| 139 |
+
return dict_method(self.get_all())
|
| 140 |
+
|
| 141 |
+
return t.cast(F, f_all)
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
@abc.Mapping.register
|
| 145 |
+
class Context:
|
| 146 |
+
"""The template context holds the variables of a template. It stores the
|
| 147 |
+
values passed to the template and also the names the template exports.
|
| 148 |
+
Creating instances is neither supported nor useful as it's created
|
| 149 |
+
automatically at various stages of the template evaluation and should not
|
| 150 |
+
be created by hand.
|
| 151 |
+
|
| 152 |
+
The context is immutable. Modifications on :attr:`parent` **must not**
|
| 153 |
+
happen and modifications on :attr:`vars` are allowed from generated
|
| 154 |
+
template code only. Template filters and global functions marked as
|
| 155 |
+
:func:`pass_context` get the active context passed as first argument
|
| 156 |
+
and are allowed to access the context read-only.
|
| 157 |
+
|
| 158 |
+
The template context supports read only dict operations (`get`,
|
| 159 |
+
`keys`, `values`, `items`, `iterkeys`, `itervalues`, `iteritems`,
|
| 160 |
+
`__getitem__`, `__contains__`). Additionally there is a :meth:`resolve`
|
| 161 |
+
method that doesn't fail with a `KeyError` but returns an
|
| 162 |
+
:class:`Undefined` object for missing variables.
|
| 163 |
+
"""
|
| 164 |
+
|
| 165 |
+
def __init__(
|
| 166 |
+
self,
|
| 167 |
+
environment: "Environment",
|
| 168 |
+
parent: t.Dict[str, t.Any],
|
| 169 |
+
name: t.Optional[str],
|
| 170 |
+
blocks: t.Dict[str, t.Callable[["Context"], t.Iterator[str]]],
|
| 171 |
+
globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
|
| 172 |
+
):
|
| 173 |
+
self.parent = parent
|
| 174 |
+
self.vars: t.Dict[str, t.Any] = {}
|
| 175 |
+
self.environment: Environment = environment
|
| 176 |
+
self.eval_ctx = EvalContext(self.environment, name)
|
| 177 |
+
self.exported_vars: t.Set[str] = set()
|
| 178 |
+
self.name = name
|
| 179 |
+
self.globals_keys = set() if globals is None else set(globals)
|
| 180 |
+
|
| 181 |
+
# create the initial mapping of blocks. Whenever template inheritance
|
| 182 |
+
# takes place the runtime will update this mapping with the new blocks
|
| 183 |
+
# from the template.
|
| 184 |
+
self.blocks = {k: [v] for k, v in blocks.items()}
|
| 185 |
+
|
| 186 |
+
def super(
|
| 187 |
+
self, name: str, current: t.Callable[["Context"], t.Iterator[str]]
|
| 188 |
+
) -> t.Union["BlockReference", "Undefined"]:
|
| 189 |
+
"""Render a parent block."""
|
| 190 |
+
try:
|
| 191 |
+
blocks = self.blocks[name]
|
| 192 |
+
index = blocks.index(current) + 1
|
| 193 |
+
blocks[index]
|
| 194 |
+
except LookupError:
|
| 195 |
+
return self.environment.undefined(
|
| 196 |
+
f"there is no parent block called {name!r}.", name="super"
|
| 197 |
+
)
|
| 198 |
+
return BlockReference(name, self, blocks, index)
|
| 199 |
+
|
| 200 |
+
def get(self, key: str, default: t.Any = None) -> t.Any:
|
| 201 |
+
"""Look up a variable by name, or return a default if the key is
|
| 202 |
+
not found.
|
| 203 |
+
|
| 204 |
+
:param key: The variable name to look up.
|
| 205 |
+
:param default: The value to return if the key is not found.
|
| 206 |
+
"""
|
| 207 |
+
try:
|
| 208 |
+
return self[key]
|
| 209 |
+
except KeyError:
|
| 210 |
+
return default
|
| 211 |
+
|
| 212 |
+
def resolve(self, key: str) -> t.Union[t.Any, "Undefined"]:
|
| 213 |
+
"""Look up a variable by name, or return an :class:`Undefined`
|
| 214 |
+
object if the key is not found.
|
| 215 |
+
|
| 216 |
+
If you need to add custom behavior, override
|
| 217 |
+
:meth:`resolve_or_missing`, not this method. The various lookup
|
| 218 |
+
functions use that method, not this one.
|
| 219 |
+
|
| 220 |
+
:param key: The variable name to look up.
|
| 221 |
+
"""
|
| 222 |
+
rv = self.resolve_or_missing(key)
|
| 223 |
+
|
| 224 |
+
if rv is missing:
|
| 225 |
+
return self.environment.undefined(name=key)
|
| 226 |
+
|
| 227 |
+
return rv
|
| 228 |
+
|
| 229 |
+
def resolve_or_missing(self, key: str) -> t.Any:
|
| 230 |
+
"""Look up a variable by name, or return a ``missing`` sentinel
|
| 231 |
+
if the key is not found.
|
| 232 |
+
|
| 233 |
+
Override this method to add custom lookup behavior.
|
| 234 |
+
:meth:`resolve`, :meth:`get`, and :meth:`__getitem__` use this
|
| 235 |
+
method. Don't call this method directly.
|
| 236 |
+
|
| 237 |
+
:param key: The variable name to look up.
|
| 238 |
+
"""
|
| 239 |
+
if key in self.vars:
|
| 240 |
+
return self.vars[key]
|
| 241 |
+
|
| 242 |
+
if key in self.parent:
|
| 243 |
+
return self.parent[key]
|
| 244 |
+
|
| 245 |
+
return missing
|
| 246 |
+
|
| 247 |
+
def get_exported(self) -> t.Dict[str, t.Any]:
|
| 248 |
+
"""Get a new dict with the exported variables."""
|
| 249 |
+
return {k: self.vars[k] for k in self.exported_vars}
|
| 250 |
+
|
| 251 |
+
def get_all(self) -> t.Dict[str, t.Any]:
|
| 252 |
+
"""Return the complete context as dict including the exported
|
| 253 |
+
variables. For optimizations reasons this might not return an
|
| 254 |
+
actual copy so be careful with using it.
|
| 255 |
+
"""
|
| 256 |
+
if not self.vars:
|
| 257 |
+
return self.parent
|
| 258 |
+
if not self.parent:
|
| 259 |
+
return self.vars
|
| 260 |
+
return dict(self.parent, **self.vars)
|
| 261 |
+
|
| 262 |
+
@internalcode
|
| 263 |
+
def call(
|
| 264 |
+
__self,
|
| 265 |
+
__obj: t.Callable[..., t.Any],
|
| 266 |
+
*args: t.Any,
|
| 267 |
+
**kwargs: t.Any, # noqa: B902
|
| 268 |
+
) -> t.Union[t.Any, "Undefined"]:
|
| 269 |
+
"""Call the callable with the arguments and keyword arguments
|
| 270 |
+
provided but inject the active context or environment as first
|
| 271 |
+
argument if the callable has :func:`pass_context` or
|
| 272 |
+
:func:`pass_environment`.
|
| 273 |
+
"""
|
| 274 |
+
if __debug__:
|
| 275 |
+
__traceback_hide__ = True # noqa
|
| 276 |
+
|
| 277 |
+
# Allow callable classes to take a context
|
| 278 |
+
if (
|
| 279 |
+
hasattr(__obj, "__call__") # noqa: B004
|
| 280 |
+
and _PassArg.from_obj(__obj.__call__) is not None
|
| 281 |
+
):
|
| 282 |
+
__obj = __obj.__call__
|
| 283 |
+
|
| 284 |
+
pass_arg = _PassArg.from_obj(__obj)
|
| 285 |
+
|
| 286 |
+
if pass_arg is _PassArg.context:
|
| 287 |
+
# the active context should have access to variables set in
|
| 288 |
+
# loops and blocks without mutating the context itself
|
| 289 |
+
if kwargs.get("_loop_vars"):
|
| 290 |
+
__self = __self.derived(kwargs["_loop_vars"])
|
| 291 |
+
if kwargs.get("_block_vars"):
|
| 292 |
+
__self = __self.derived(kwargs["_block_vars"])
|
| 293 |
+
args = (__self,) + args
|
| 294 |
+
elif pass_arg is _PassArg.eval_context:
|
| 295 |
+
args = (__self.eval_ctx,) + args
|
| 296 |
+
elif pass_arg is _PassArg.environment:
|
| 297 |
+
args = (__self.environment,) + args
|
| 298 |
+
|
| 299 |
+
kwargs.pop("_block_vars", None)
|
| 300 |
+
kwargs.pop("_loop_vars", None)
|
| 301 |
+
|
| 302 |
+
try:
|
| 303 |
+
return __obj(*args, **kwargs)
|
| 304 |
+
except StopIteration:
|
| 305 |
+
return __self.environment.undefined(
|
| 306 |
+
"value was undefined because a callable raised a"
|
| 307 |
+
" StopIteration exception"
|
| 308 |
+
)
|
| 309 |
+
|
| 310 |
+
def derived(self, locals: t.Optional[t.Dict[str, t.Any]] = None) -> "Context":
|
| 311 |
+
"""Internal helper function to create a derived context. This is
|
| 312 |
+
used in situations where the system needs a new context in the same
|
| 313 |
+
template that is independent.
|
| 314 |
+
"""
|
| 315 |
+
context = new_context(
|
| 316 |
+
self.environment, self.name, {}, self.get_all(), True, None, locals
|
| 317 |
+
)
|
| 318 |
+
context.eval_ctx = self.eval_ctx
|
| 319 |
+
context.blocks.update((k, list(v)) for k, v in self.blocks.items())
|
| 320 |
+
return context
|
| 321 |
+
|
| 322 |
+
keys = _dict_method_all(dict.keys)
|
| 323 |
+
values = _dict_method_all(dict.values)
|
| 324 |
+
items = _dict_method_all(dict.items)
|
| 325 |
+
|
| 326 |
+
def __contains__(self, name: str) -> bool:
|
| 327 |
+
return name in self.vars or name in self.parent
|
| 328 |
+
|
| 329 |
+
def __getitem__(self, key: str) -> t.Any:
|
| 330 |
+
"""Look up a variable by name with ``[]`` syntax, or raise a
|
| 331 |
+
``KeyError`` if the key is not found.
|
| 332 |
+
"""
|
| 333 |
+
item = self.resolve_or_missing(key)
|
| 334 |
+
|
| 335 |
+
if item is missing:
|
| 336 |
+
raise KeyError(key)
|
| 337 |
+
|
| 338 |
+
return item
|
| 339 |
+
|
| 340 |
+
def __repr__(self) -> str:
|
| 341 |
+
return f"<{type(self).__name__} {self.get_all()!r} of {self.name!r}>"
|
| 342 |
+
|
| 343 |
+
|
| 344 |
+
class BlockReference:
|
| 345 |
+
"""One block on a template reference."""
|
| 346 |
+
|
| 347 |
+
def __init__(
|
| 348 |
+
self,
|
| 349 |
+
name: str,
|
| 350 |
+
context: "Context",
|
| 351 |
+
stack: t.List[t.Callable[["Context"], t.Iterator[str]]],
|
| 352 |
+
depth: int,
|
| 353 |
+
) -> None:
|
| 354 |
+
self.name = name
|
| 355 |
+
self._context = context
|
| 356 |
+
self._stack = stack
|
| 357 |
+
self._depth = depth
|
| 358 |
+
|
| 359 |
+
@property
|
| 360 |
+
def super(self) -> t.Union["BlockReference", "Undefined"]:
|
| 361 |
+
"""Super the block."""
|
| 362 |
+
if self._depth + 1 >= len(self._stack):
|
| 363 |
+
return self._context.environment.undefined(
|
| 364 |
+
f"there is no parent block called {self.name!r}.", name="super"
|
| 365 |
+
)
|
| 366 |
+
return BlockReference(self.name, self._context, self._stack, self._depth + 1)
|
| 367 |
+
|
| 368 |
+
@internalcode
|
| 369 |
+
async def _async_call(self) -> str:
|
| 370 |
+
rv = self._context.environment.concat( # type: ignore
|
| 371 |
+
[x async for x in self._stack[self._depth](self._context)] # type: ignore
|
| 372 |
+
)
|
| 373 |
+
|
| 374 |
+
if self._context.eval_ctx.autoescape:
|
| 375 |
+
return Markup(rv)
|
| 376 |
+
|
| 377 |
+
return rv
|
| 378 |
+
|
| 379 |
+
@internalcode
|
| 380 |
+
def __call__(self) -> str:
|
| 381 |
+
if self._context.environment.is_async:
|
| 382 |
+
return self._async_call() # type: ignore
|
| 383 |
+
|
| 384 |
+
rv = self._context.environment.concat( # type: ignore
|
| 385 |
+
self._stack[self._depth](self._context)
|
| 386 |
+
)
|
| 387 |
+
|
| 388 |
+
if self._context.eval_ctx.autoescape:
|
| 389 |
+
return Markup(rv)
|
| 390 |
+
|
| 391 |
+
return rv
|
| 392 |
+
|
| 393 |
+
|
| 394 |
+
class LoopContext:
|
| 395 |
+
"""A wrapper iterable for dynamic ``for`` loops, with information
|
| 396 |
+
about the loop and iteration.
|
| 397 |
+
"""
|
| 398 |
+
|
| 399 |
+
#: Current iteration of the loop, starting at 0.
|
| 400 |
+
index0 = -1
|
| 401 |
+
|
| 402 |
+
_length: t.Optional[int] = None
|
| 403 |
+
_after: t.Any = missing
|
| 404 |
+
_current: t.Any = missing
|
| 405 |
+
_before: t.Any = missing
|
| 406 |
+
_last_changed_value: t.Any = missing
|
| 407 |
+
|
| 408 |
+
def __init__(
|
| 409 |
+
self,
|
| 410 |
+
iterable: t.Iterable[V],
|
| 411 |
+
undefined: t.Type["Undefined"],
|
| 412 |
+
recurse: t.Optional["LoopRenderFunc"] = None,
|
| 413 |
+
depth0: int = 0,
|
| 414 |
+
) -> None:
|
| 415 |
+
"""
|
| 416 |
+
:param iterable: Iterable to wrap.
|
| 417 |
+
:param undefined: :class:`Undefined` class to use for next and
|
| 418 |
+
previous items.
|
| 419 |
+
:param recurse: The function to render the loop body when the
|
| 420 |
+
loop is marked recursive.
|
| 421 |
+
:param depth0: Incremented when looping recursively.
|
| 422 |
+
"""
|
| 423 |
+
self._iterable = iterable
|
| 424 |
+
self._iterator = self._to_iterator(iterable)
|
| 425 |
+
self._undefined = undefined
|
| 426 |
+
self._recurse = recurse
|
| 427 |
+
#: How many levels deep a recursive loop currently is, starting at 0.
|
| 428 |
+
self.depth0 = depth0
|
| 429 |
+
|
| 430 |
+
@staticmethod
|
| 431 |
+
def _to_iterator(iterable: t.Iterable[V]) -> t.Iterator[V]:
|
| 432 |
+
return iter(iterable)
|
| 433 |
+
|
| 434 |
+
@property
|
| 435 |
+
def length(self) -> int:
|
| 436 |
+
"""Length of the iterable.
|
| 437 |
+
|
| 438 |
+
If the iterable is a generator or otherwise does not have a
|
| 439 |
+
size, it is eagerly evaluated to get a size.
|
| 440 |
+
"""
|
| 441 |
+
if self._length is not None:
|
| 442 |
+
return self._length
|
| 443 |
+
|
| 444 |
+
try:
|
| 445 |
+
self._length = len(self._iterable) # type: ignore
|
| 446 |
+
except TypeError:
|
| 447 |
+
iterable = list(self._iterator)
|
| 448 |
+
self._iterator = self._to_iterator(iterable)
|
| 449 |
+
self._length = len(iterable) + self.index + (self._after is not missing)
|
| 450 |
+
|
| 451 |
+
return self._length
|
| 452 |
+
|
| 453 |
+
def __len__(self) -> int:
|
| 454 |
+
return self.length
|
| 455 |
+
|
| 456 |
+
@property
|
| 457 |
+
def depth(self) -> int:
|
| 458 |
+
"""How many levels deep a recursive loop currently is, starting at 1."""
|
| 459 |
+
return self.depth0 + 1
|
| 460 |
+
|
| 461 |
+
@property
|
| 462 |
+
def index(self) -> int:
|
| 463 |
+
"""Current iteration of the loop, starting at 1."""
|
| 464 |
+
return self.index0 + 1
|
| 465 |
+
|
| 466 |
+
@property
|
| 467 |
+
def revindex0(self) -> int:
|
| 468 |
+
"""Number of iterations from the end of the loop, ending at 0.
|
| 469 |
+
|
| 470 |
+
Requires calculating :attr:`length`.
|
| 471 |
+
"""
|
| 472 |
+
return self.length - self.index
|
| 473 |
+
|
| 474 |
+
@property
|
| 475 |
+
def revindex(self) -> int:
|
| 476 |
+
"""Number of iterations from the end of the loop, ending at 1.
|
| 477 |
+
|
| 478 |
+
Requires calculating :attr:`length`.
|
| 479 |
+
"""
|
| 480 |
+
return self.length - self.index0
|
| 481 |
+
|
| 482 |
+
@property
|
| 483 |
+
def first(self) -> bool:
|
| 484 |
+
"""Whether this is the first iteration of the loop."""
|
| 485 |
+
return self.index0 == 0
|
| 486 |
+
|
| 487 |
+
def _peek_next(self) -> t.Any:
|
| 488 |
+
"""Return the next element in the iterable, or :data:`missing`
|
| 489 |
+
if the iterable is exhausted. Only peeks one item ahead, caching
|
| 490 |
+
the result in :attr:`_last` for use in subsequent checks. The
|
| 491 |
+
cache is reset when :meth:`__next__` is called.
|
| 492 |
+
"""
|
| 493 |
+
if self._after is not missing:
|
| 494 |
+
return self._after
|
| 495 |
+
|
| 496 |
+
self._after = next(self._iterator, missing)
|
| 497 |
+
return self._after
|
| 498 |
+
|
| 499 |
+
@property
|
| 500 |
+
def last(self) -> bool:
|
| 501 |
+
"""Whether this is the last iteration of the loop.
|
| 502 |
+
|
| 503 |
+
Causes the iterable to advance early. See
|
| 504 |
+
:func:`itertools.groupby` for issues this can cause.
|
| 505 |
+
The :func:`groupby` filter avoids that issue.
|
| 506 |
+
"""
|
| 507 |
+
return self._peek_next() is missing
|
| 508 |
+
|
| 509 |
+
@property
|
| 510 |
+
def previtem(self) -> t.Union[t.Any, "Undefined"]:
|
| 511 |
+
"""The item in the previous iteration. Undefined during the
|
| 512 |
+
first iteration.
|
| 513 |
+
"""
|
| 514 |
+
if self.first:
|
| 515 |
+
return self._undefined("there is no previous item")
|
| 516 |
+
|
| 517 |
+
return self._before
|
| 518 |
+
|
| 519 |
+
@property
|
| 520 |
+
def nextitem(self) -> t.Union[t.Any, "Undefined"]:
|
| 521 |
+
"""The item in the next iteration. Undefined during the last
|
| 522 |
+
iteration.
|
| 523 |
+
|
| 524 |
+
Causes the iterable to advance early. See
|
| 525 |
+
:func:`itertools.groupby` for issues this can cause.
|
| 526 |
+
The :func:`jinja-filters.groupby` filter avoids that issue.
|
| 527 |
+
"""
|
| 528 |
+
rv = self._peek_next()
|
| 529 |
+
|
| 530 |
+
if rv is missing:
|
| 531 |
+
return self._undefined("there is no next item")
|
| 532 |
+
|
| 533 |
+
return rv
|
| 534 |
+
|
| 535 |
+
def cycle(self, *args: V) -> V:
|
| 536 |
+
"""Return a value from the given args, cycling through based on
|
| 537 |
+
the current :attr:`index0`.
|
| 538 |
+
|
| 539 |
+
:param args: One or more values to cycle through.
|
| 540 |
+
"""
|
| 541 |
+
if not args:
|
| 542 |
+
raise TypeError("no items for cycling given")
|
| 543 |
+
|
| 544 |
+
return args[self.index0 % len(args)]
|
| 545 |
+
|
| 546 |
+
def changed(self, *value: t.Any) -> bool:
|
| 547 |
+
"""Return ``True`` if previously called with a different value
|
| 548 |
+
(including when called for the first time).
|
| 549 |
+
|
| 550 |
+
:param value: One or more values to compare to the last call.
|
| 551 |
+
"""
|
| 552 |
+
if self._last_changed_value != value:
|
| 553 |
+
self._last_changed_value = value
|
| 554 |
+
return True
|
| 555 |
+
|
| 556 |
+
return False
|
| 557 |
+
|
| 558 |
+
def __iter__(self) -> "LoopContext":
|
| 559 |
+
return self
|
| 560 |
+
|
| 561 |
+
def __next__(self) -> t.Tuple[t.Any, "LoopContext"]:
|
| 562 |
+
if self._after is not missing:
|
| 563 |
+
rv = self._after
|
| 564 |
+
self._after = missing
|
| 565 |
+
else:
|
| 566 |
+
rv = next(self._iterator)
|
| 567 |
+
|
| 568 |
+
self.index0 += 1
|
| 569 |
+
self._before = self._current
|
| 570 |
+
self._current = rv
|
| 571 |
+
return rv, self
|
| 572 |
+
|
| 573 |
+
@internalcode
|
| 574 |
+
def __call__(self, iterable: t.Iterable[V]) -> str:
|
| 575 |
+
"""When iterating over nested data, render the body of the loop
|
| 576 |
+
recursively with the given inner iterable data.
|
| 577 |
+
|
| 578 |
+
The loop must have the ``recursive`` marker for this to work.
|
| 579 |
+
"""
|
| 580 |
+
if self._recurse is None:
|
| 581 |
+
raise TypeError(
|
| 582 |
+
"The loop must have the 'recursive' marker to be called recursively."
|
| 583 |
+
)
|
| 584 |
+
|
| 585 |
+
return self._recurse(iterable, self._recurse, depth=self.depth)
|
| 586 |
+
|
| 587 |
+
def __repr__(self) -> str:
|
| 588 |
+
return f"<{type(self).__name__} {self.index}/{self.length}>"
|
| 589 |
+
|
| 590 |
+
|
| 591 |
+
class AsyncLoopContext(LoopContext):
|
| 592 |
+
_iterator: t.AsyncIterator[t.Any] # type: ignore
|
| 593 |
+
|
| 594 |
+
@staticmethod
|
| 595 |
+
def _to_iterator( # type: ignore
|
| 596 |
+
iterable: t.Union[t.Iterable[V], t.AsyncIterable[V]],
|
| 597 |
+
) -> t.AsyncIterator[V]:
|
| 598 |
+
return auto_aiter(iterable)
|
| 599 |
+
|
| 600 |
+
@property
|
| 601 |
+
async def length(self) -> int: # type: ignore
|
| 602 |
+
if self._length is not None:
|
| 603 |
+
return self._length
|
| 604 |
+
|
| 605 |
+
try:
|
| 606 |
+
self._length = len(self._iterable) # type: ignore
|
| 607 |
+
except TypeError:
|
| 608 |
+
iterable = [x async for x in self._iterator]
|
| 609 |
+
self._iterator = self._to_iterator(iterable)
|
| 610 |
+
self._length = len(iterable) + self.index + (self._after is not missing)
|
| 611 |
+
|
| 612 |
+
return self._length
|
| 613 |
+
|
| 614 |
+
@property
|
| 615 |
+
async def revindex0(self) -> int: # type: ignore
|
| 616 |
+
return await self.length - self.index
|
| 617 |
+
|
| 618 |
+
@property
|
| 619 |
+
async def revindex(self) -> int: # type: ignore
|
| 620 |
+
return await self.length - self.index0
|
| 621 |
+
|
| 622 |
+
async def _peek_next(self) -> t.Any:
|
| 623 |
+
if self._after is not missing:
|
| 624 |
+
return self._after
|
| 625 |
+
|
| 626 |
+
try:
|
| 627 |
+
self._after = await self._iterator.__anext__()
|
| 628 |
+
except StopAsyncIteration:
|
| 629 |
+
self._after = missing
|
| 630 |
+
|
| 631 |
+
return self._after
|
| 632 |
+
|
| 633 |
+
@property
|
| 634 |
+
async def last(self) -> bool: # type: ignore
|
| 635 |
+
return await self._peek_next() is missing
|
| 636 |
+
|
| 637 |
+
@property
|
| 638 |
+
async def nextitem(self) -> t.Union[t.Any, "Undefined"]:
|
| 639 |
+
rv = await self._peek_next()
|
| 640 |
+
|
| 641 |
+
if rv is missing:
|
| 642 |
+
return self._undefined("there is no next item")
|
| 643 |
+
|
| 644 |
+
return rv
|
| 645 |
+
|
| 646 |
+
def __aiter__(self) -> "AsyncLoopContext":
|
| 647 |
+
return self
|
| 648 |
+
|
| 649 |
+
async def __anext__(self) -> t.Tuple[t.Any, "AsyncLoopContext"]:
|
| 650 |
+
if self._after is not missing:
|
| 651 |
+
rv = self._after
|
| 652 |
+
self._after = missing
|
| 653 |
+
else:
|
| 654 |
+
rv = await self._iterator.__anext__()
|
| 655 |
+
|
| 656 |
+
self.index0 += 1
|
| 657 |
+
self._before = self._current
|
| 658 |
+
self._current = rv
|
| 659 |
+
return rv, self
|
| 660 |
+
|
| 661 |
+
|
| 662 |
+
class Macro:
|
| 663 |
+
"""Wraps a macro function."""
|
| 664 |
+
|
| 665 |
+
def __init__(
|
| 666 |
+
self,
|
| 667 |
+
environment: "Environment",
|
| 668 |
+
func: t.Callable[..., str],
|
| 669 |
+
name: str,
|
| 670 |
+
arguments: t.List[str],
|
| 671 |
+
catch_kwargs: bool,
|
| 672 |
+
catch_varargs: bool,
|
| 673 |
+
caller: bool,
|
| 674 |
+
default_autoescape: t.Optional[bool] = None,
|
| 675 |
+
):
|
| 676 |
+
self._environment = environment
|
| 677 |
+
self._func = func
|
| 678 |
+
self._argument_count = len(arguments)
|
| 679 |
+
self.name = name
|
| 680 |
+
self.arguments = arguments
|
| 681 |
+
self.catch_kwargs = catch_kwargs
|
| 682 |
+
self.catch_varargs = catch_varargs
|
| 683 |
+
self.caller = caller
|
| 684 |
+
self.explicit_caller = "caller" in arguments
|
| 685 |
+
|
| 686 |
+
if default_autoescape is None:
|
| 687 |
+
if callable(environment.autoescape):
|
| 688 |
+
default_autoescape = environment.autoescape(None)
|
| 689 |
+
else:
|
| 690 |
+
default_autoescape = environment.autoescape
|
| 691 |
+
|
| 692 |
+
self._default_autoescape = default_autoescape
|
| 693 |
+
|
| 694 |
+
@internalcode
|
| 695 |
+
@pass_eval_context
|
| 696 |
+
def __call__(self, *args: t.Any, **kwargs: t.Any) -> str:
|
| 697 |
+
# This requires a bit of explanation, In the past we used to
|
| 698 |
+
# decide largely based on compile-time information if a macro is
|
| 699 |
+
# safe or unsafe. While there was a volatile mode it was largely
|
| 700 |
+
# unused for deciding on escaping. This turns out to be
|
| 701 |
+
# problematic for macros because whether a macro is safe depends not
|
| 702 |
+
# on the escape mode when it was defined, but rather when it was used.
|
| 703 |
+
#
|
| 704 |
+
# Because however we export macros from the module system and
|
| 705 |
+
# there are historic callers that do not pass an eval context (and
|
| 706 |
+
# will continue to not pass one), we need to perform an instance
|
| 707 |
+
# check here.
|
| 708 |
+
#
|
| 709 |
+
# This is considered safe because an eval context is not a valid
|
| 710 |
+
# argument to callables otherwise anyway. Worst case here is
|
| 711 |
+
# that if no eval context is passed we fall back to the compile
|
| 712 |
+
# time autoescape flag.
|
| 713 |
+
if args and isinstance(args[0], EvalContext):
|
| 714 |
+
autoescape = args[0].autoescape
|
| 715 |
+
args = args[1:]
|
| 716 |
+
else:
|
| 717 |
+
autoescape = self._default_autoescape
|
| 718 |
+
|
| 719 |
+
# try to consume the positional arguments
|
| 720 |
+
arguments = list(args[: self._argument_count])
|
| 721 |
+
off = len(arguments)
|
| 722 |
+
|
| 723 |
+
# For information why this is necessary refer to the handling
|
| 724 |
+
# of caller in the `macro_body` handler in the compiler.
|
| 725 |
+
found_caller = False
|
| 726 |
+
|
| 727 |
+
# if the number of arguments consumed is not the number of
|
| 728 |
+
# arguments expected we start filling in keyword arguments
|
| 729 |
+
# and defaults.
|
| 730 |
+
if off != self._argument_count:
|
| 731 |
+
for name in self.arguments[len(arguments) :]:
|
| 732 |
+
try:
|
| 733 |
+
value = kwargs.pop(name)
|
| 734 |
+
except KeyError:
|
| 735 |
+
value = missing
|
| 736 |
+
if name == "caller":
|
| 737 |
+
found_caller = True
|
| 738 |
+
arguments.append(value)
|
| 739 |
+
else:
|
| 740 |
+
found_caller = self.explicit_caller
|
| 741 |
+
|
| 742 |
+
# it's important that the order of these arguments does not change
|
| 743 |
+
# if not also changed in the compiler's `function_scoping` method.
|
| 744 |
+
# the order is caller, keyword arguments, positional arguments!
|
| 745 |
+
if self.caller and not found_caller:
|
| 746 |
+
caller = kwargs.pop("caller", None)
|
| 747 |
+
if caller is None:
|
| 748 |
+
caller = self._environment.undefined("No caller defined", name="caller")
|
| 749 |
+
arguments.append(caller)
|
| 750 |
+
|
| 751 |
+
if self.catch_kwargs:
|
| 752 |
+
arguments.append(kwargs)
|
| 753 |
+
elif kwargs:
|
| 754 |
+
if "caller" in kwargs:
|
| 755 |
+
raise TypeError(
|
| 756 |
+
f"macro {self.name!r} was invoked with two values for the special"
|
| 757 |
+
" caller argument. This is most likely a bug."
|
| 758 |
+
)
|
| 759 |
+
raise TypeError(
|
| 760 |
+
f"macro {self.name!r} takes no keyword argument {next(iter(kwargs))!r}"
|
| 761 |
+
)
|
| 762 |
+
if self.catch_varargs:
|
| 763 |
+
arguments.append(args[self._argument_count :])
|
| 764 |
+
elif len(args) > self._argument_count:
|
| 765 |
+
raise TypeError(
|
| 766 |
+
f"macro {self.name!r} takes not more than"
|
| 767 |
+
f" {len(self.arguments)} argument(s)"
|
| 768 |
+
)
|
| 769 |
+
|
| 770 |
+
return self._invoke(arguments, autoescape)
|
| 771 |
+
|
| 772 |
+
async def _async_invoke(self, arguments: t.List[t.Any], autoescape: bool) -> str:
|
| 773 |
+
rv = await self._func(*arguments) # type: ignore
|
| 774 |
+
|
| 775 |
+
if autoescape:
|
| 776 |
+
return Markup(rv)
|
| 777 |
+
|
| 778 |
+
return rv # type: ignore
|
| 779 |
+
|
| 780 |
+
def _invoke(self, arguments: t.List[t.Any], autoescape: bool) -> str:
|
| 781 |
+
if self._environment.is_async:
|
| 782 |
+
return self._async_invoke(arguments, autoescape) # type: ignore
|
| 783 |
+
|
| 784 |
+
rv = self._func(*arguments)
|
| 785 |
+
|
| 786 |
+
if autoescape:
|
| 787 |
+
rv = Markup(rv)
|
| 788 |
+
|
| 789 |
+
return rv
|
| 790 |
+
|
| 791 |
+
def __repr__(self) -> str:
|
| 792 |
+
name = "anonymous" if self.name is None else repr(self.name)
|
| 793 |
+
return f"<{type(self).__name__} {name}>"
|
| 794 |
+
|
| 795 |
+
|
| 796 |
+
class Undefined:
|
| 797 |
+
"""The default undefined type. This can be printed, iterated, and treated as
|
| 798 |
+
a boolean. Any other operation will raise an :exc:`UndefinedError`.
|
| 799 |
+
|
| 800 |
+
>>> foo = Undefined(name='foo')
|
| 801 |
+
>>> str(foo)
|
| 802 |
+
''
|
| 803 |
+
>>> not foo
|
| 804 |
+
True
|
| 805 |
+
>>> foo + 42
|
| 806 |
+
Traceback (most recent call last):
|
| 807 |
+
...
|
| 808 |
+
jinja2.exceptions.UndefinedError: 'foo' is undefined
|
| 809 |
+
"""
|
| 810 |
+
|
| 811 |
+
__slots__ = (
|
| 812 |
+
"_undefined_hint",
|
| 813 |
+
"_undefined_obj",
|
| 814 |
+
"_undefined_name",
|
| 815 |
+
"_undefined_exception",
|
| 816 |
+
)
|
| 817 |
+
|
| 818 |
+
def __init__(
|
| 819 |
+
self,
|
| 820 |
+
hint: t.Optional[str] = None,
|
| 821 |
+
obj: t.Any = missing,
|
| 822 |
+
name: t.Optional[str] = None,
|
| 823 |
+
exc: t.Type[TemplateRuntimeError] = UndefinedError,
|
| 824 |
+
) -> None:
|
| 825 |
+
self._undefined_hint = hint
|
| 826 |
+
self._undefined_obj = obj
|
| 827 |
+
self._undefined_name = name
|
| 828 |
+
self._undefined_exception = exc
|
| 829 |
+
|
| 830 |
+
@property
|
| 831 |
+
def _undefined_message(self) -> str:
|
| 832 |
+
"""Build a message about the undefined value based on how it was
|
| 833 |
+
accessed.
|
| 834 |
+
"""
|
| 835 |
+
if self._undefined_hint:
|
| 836 |
+
return self._undefined_hint
|
| 837 |
+
|
| 838 |
+
if self._undefined_obj is missing:
|
| 839 |
+
return f"{self._undefined_name!r} is undefined"
|
| 840 |
+
|
| 841 |
+
if not isinstance(self._undefined_name, str):
|
| 842 |
+
return (
|
| 843 |
+
f"{object_type_repr(self._undefined_obj)} has no"
|
| 844 |
+
f" element {self._undefined_name!r}"
|
| 845 |
+
)
|
| 846 |
+
|
| 847 |
+
return (
|
| 848 |
+
f"{object_type_repr(self._undefined_obj)!r} has no"
|
| 849 |
+
f" attribute {self._undefined_name!r}"
|
| 850 |
+
)
|
| 851 |
+
|
| 852 |
+
@internalcode
|
| 853 |
+
def _fail_with_undefined_error(
|
| 854 |
+
self, *args: t.Any, **kwargs: t.Any
|
| 855 |
+
) -> "te.NoReturn":
|
| 856 |
+
"""Raise an :exc:`UndefinedError` when operations are performed
|
| 857 |
+
on the undefined value.
|
| 858 |
+
"""
|
| 859 |
+
raise self._undefined_exception(self._undefined_message)
|
| 860 |
+
|
| 861 |
+
@internalcode
|
| 862 |
+
def __getattr__(self, name: str) -> t.Any:
|
| 863 |
+
# Raise AttributeError on requests for names that appear to be unimplemented
|
| 864 |
+
# dunder methods to keep Python's internal protocol probing behaviors working
|
| 865 |
+
# properly in cases where another exception type could cause unexpected or
|
| 866 |
+
# difficult-to-diagnose failures.
|
| 867 |
+
if name[:2] == "__" and name[-2:] == "__":
|
| 868 |
+
raise AttributeError(name)
|
| 869 |
+
|
| 870 |
+
return self._fail_with_undefined_error()
|
| 871 |
+
|
| 872 |
+
__add__ = __radd__ = __sub__ = __rsub__ = _fail_with_undefined_error
|
| 873 |
+
__mul__ = __rmul__ = __div__ = __rdiv__ = _fail_with_undefined_error
|
| 874 |
+
__truediv__ = __rtruediv__ = _fail_with_undefined_error
|
| 875 |
+
__floordiv__ = __rfloordiv__ = _fail_with_undefined_error
|
| 876 |
+
__mod__ = __rmod__ = _fail_with_undefined_error
|
| 877 |
+
__pos__ = __neg__ = _fail_with_undefined_error
|
| 878 |
+
__call__ = __getitem__ = _fail_with_undefined_error
|
| 879 |
+
__lt__ = __le__ = __gt__ = __ge__ = _fail_with_undefined_error
|
| 880 |
+
__int__ = __float__ = __complex__ = _fail_with_undefined_error
|
| 881 |
+
__pow__ = __rpow__ = _fail_with_undefined_error
|
| 882 |
+
|
| 883 |
+
def __eq__(self, other: t.Any) -> bool:
|
| 884 |
+
return type(self) is type(other)
|
| 885 |
+
|
| 886 |
+
def __ne__(self, other: t.Any) -> bool:
|
| 887 |
+
return not self.__eq__(other)
|
| 888 |
+
|
| 889 |
+
def __hash__(self) -> int:
|
| 890 |
+
return id(type(self))
|
| 891 |
+
|
| 892 |
+
def __str__(self) -> str:
|
| 893 |
+
return ""
|
| 894 |
+
|
| 895 |
+
def __len__(self) -> int:
|
| 896 |
+
return 0
|
| 897 |
+
|
| 898 |
+
def __iter__(self) -> t.Iterator[t.Any]:
|
| 899 |
+
yield from ()
|
| 900 |
+
|
| 901 |
+
async def __aiter__(self) -> t.AsyncIterator[t.Any]:
|
| 902 |
+
for _ in ():
|
| 903 |
+
yield
|
| 904 |
+
|
| 905 |
+
def __bool__(self) -> bool:
|
| 906 |
+
return False
|
| 907 |
+
|
| 908 |
+
def __repr__(self) -> str:
|
| 909 |
+
return "Undefined"
|
| 910 |
+
|
| 911 |
+
|
| 912 |
+
def make_logging_undefined(
|
| 913 |
+
logger: t.Optional["logging.Logger"] = None, base: t.Type[Undefined] = Undefined
|
| 914 |
+
) -> t.Type[Undefined]:
|
| 915 |
+
"""Given a logger object this returns a new undefined class that will
|
| 916 |
+
log certain failures. It will log iterations and printing. If no
|
| 917 |
+
logger is given a default logger is created.
|
| 918 |
+
|
| 919 |
+
Example::
|
| 920 |
+
|
| 921 |
+
logger = logging.getLogger(__name__)
|
| 922 |
+
LoggingUndefined = make_logging_undefined(
|
| 923 |
+
logger=logger,
|
| 924 |
+
base=Undefined
|
| 925 |
+
)
|
| 926 |
+
|
| 927 |
+
.. versionadded:: 2.8
|
| 928 |
+
|
| 929 |
+
:param logger: the logger to use. If not provided, a default logger
|
| 930 |
+
is created.
|
| 931 |
+
:param base: the base class to add logging functionality to. This
|
| 932 |
+
defaults to :class:`Undefined`.
|
| 933 |
+
"""
|
| 934 |
+
if logger is None:
|
| 935 |
+
import logging
|
| 936 |
+
|
| 937 |
+
logger = logging.getLogger(__name__)
|
| 938 |
+
logger.addHandler(logging.StreamHandler(sys.stderr))
|
| 939 |
+
|
| 940 |
+
def _log_message(undef: Undefined) -> None:
|
| 941 |
+
logger.warning("Template variable warning: %s", undef._undefined_message)
|
| 942 |
+
|
| 943 |
+
class LoggingUndefined(base): # type: ignore
|
| 944 |
+
__slots__ = ()
|
| 945 |
+
|
| 946 |
+
def _fail_with_undefined_error( # type: ignore
|
| 947 |
+
self, *args: t.Any, **kwargs: t.Any
|
| 948 |
+
) -> "te.NoReturn":
|
| 949 |
+
try:
|
| 950 |
+
super()._fail_with_undefined_error(*args, **kwargs)
|
| 951 |
+
except self._undefined_exception as e:
|
| 952 |
+
logger.error("Template variable error: %s", e) # type: ignore
|
| 953 |
+
raise e
|
| 954 |
+
|
| 955 |
+
def __str__(self) -> str:
|
| 956 |
+
_log_message(self)
|
| 957 |
+
return super().__str__() # type: ignore
|
| 958 |
+
|
| 959 |
+
def __iter__(self) -> t.Iterator[t.Any]:
|
| 960 |
+
_log_message(self)
|
| 961 |
+
return super().__iter__() # type: ignore
|
| 962 |
+
|
| 963 |
+
def __bool__(self) -> bool:
|
| 964 |
+
_log_message(self)
|
| 965 |
+
return super().__bool__() # type: ignore
|
| 966 |
+
|
| 967 |
+
return LoggingUndefined
|
| 968 |
+
|
| 969 |
+
|
| 970 |
+
class ChainableUndefined(Undefined):
|
| 971 |
+
"""An undefined that is chainable, where both ``__getattr__`` and
|
| 972 |
+
``__getitem__`` return itself rather than raising an
|
| 973 |
+
:exc:`UndefinedError`.
|
| 974 |
+
|
| 975 |
+
>>> foo = ChainableUndefined(name='foo')
|
| 976 |
+
>>> str(foo.bar['baz'])
|
| 977 |
+
''
|
| 978 |
+
>>> foo.bar['baz'] + 42
|
| 979 |
+
Traceback (most recent call last):
|
| 980 |
+
...
|
| 981 |
+
jinja2.exceptions.UndefinedError: 'foo' is undefined
|
| 982 |
+
|
| 983 |
+
.. versionadded:: 2.11.0
|
| 984 |
+
"""
|
| 985 |
+
|
| 986 |
+
__slots__ = ()
|
| 987 |
+
|
| 988 |
+
def __html__(self) -> str:
|
| 989 |
+
return str(self)
|
| 990 |
+
|
| 991 |
+
def __getattr__(self, name: str) -> "ChainableUndefined":
|
| 992 |
+
# Raise AttributeError on requests for names that appear to be unimplemented
|
| 993 |
+
# dunder methods to avoid confusing Python with truthy non-method objects that
|
| 994 |
+
# do not implement the protocol being probed for. e.g., copy.copy(Undefined())
|
| 995 |
+
# fails spectacularly if getattr(Undefined(), '__setstate__') returns an
|
| 996 |
+
# Undefined object instead of raising AttributeError to signal that it does not
|
| 997 |
+
# support that style of object initialization.
|
| 998 |
+
if name[:2] == "__" and name[-2:] == "__":
|
| 999 |
+
raise AttributeError(name)
|
| 1000 |
+
|
| 1001 |
+
return self
|
| 1002 |
+
|
| 1003 |
+
def __getitem__(self, _name: str) -> "ChainableUndefined": # type: ignore[override]
|
| 1004 |
+
return self
|
| 1005 |
+
|
| 1006 |
+
|
| 1007 |
+
class DebugUndefined(Undefined):
|
| 1008 |
+
"""An undefined that returns the debug info when printed.
|
| 1009 |
+
|
| 1010 |
+
>>> foo = DebugUndefined(name='foo')
|
| 1011 |
+
>>> str(foo)
|
| 1012 |
+
'{{ foo }}'
|
| 1013 |
+
>>> not foo
|
| 1014 |
+
True
|
| 1015 |
+
>>> foo + 42
|
| 1016 |
+
Traceback (most recent call last):
|
| 1017 |
+
...
|
| 1018 |
+
jinja2.exceptions.UndefinedError: 'foo' is undefined
|
| 1019 |
+
"""
|
| 1020 |
+
|
| 1021 |
+
__slots__ = ()
|
| 1022 |
+
|
| 1023 |
+
def __str__(self) -> str:
|
| 1024 |
+
if self._undefined_hint:
|
| 1025 |
+
message = f"undefined value printed: {self._undefined_hint}"
|
| 1026 |
+
|
| 1027 |
+
elif self._undefined_obj is missing:
|
| 1028 |
+
message = self._undefined_name # type: ignore
|
| 1029 |
+
|
| 1030 |
+
else:
|
| 1031 |
+
message = (
|
| 1032 |
+
f"no such element: {object_type_repr(self._undefined_obj)}"
|
| 1033 |
+
f"[{self._undefined_name!r}]"
|
| 1034 |
+
)
|
| 1035 |
+
|
| 1036 |
+
return f"{{{{ {message} }}}}"
|
| 1037 |
+
|
| 1038 |
+
|
| 1039 |
+
class StrictUndefined(Undefined):
|
| 1040 |
+
"""An undefined that barks on print and iteration as well as boolean
|
| 1041 |
+
tests and all kinds of comparisons. In other words: you can do nothing
|
| 1042 |
+
with it except checking if it's defined using the `defined` test.
|
| 1043 |
+
|
| 1044 |
+
>>> foo = StrictUndefined(name='foo')
|
| 1045 |
+
>>> str(foo)
|
| 1046 |
+
Traceback (most recent call last):
|
| 1047 |
+
...
|
| 1048 |
+
jinja2.exceptions.UndefinedError: 'foo' is undefined
|
| 1049 |
+
>>> not foo
|
| 1050 |
+
Traceback (most recent call last):
|
| 1051 |
+
...
|
| 1052 |
+
jinja2.exceptions.UndefinedError: 'foo' is undefined
|
| 1053 |
+
>>> foo + 42
|
| 1054 |
+
Traceback (most recent call last):
|
| 1055 |
+
...
|
| 1056 |
+
jinja2.exceptions.UndefinedError: 'foo' is undefined
|
| 1057 |
+
"""
|
| 1058 |
+
|
| 1059 |
+
__slots__ = ()
|
| 1060 |
+
__iter__ = __str__ = __len__ = Undefined._fail_with_undefined_error
|
| 1061 |
+
__eq__ = __ne__ = __bool__ = __hash__ = Undefined._fail_with_undefined_error
|
| 1062 |
+
__contains__ = Undefined._fail_with_undefined_error
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/sandbox.py
ADDED
|
@@ -0,0 +1,436 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""A sandbox layer that ensures unsafe operations cannot be performed.
|
| 2 |
+
Useful when the template itself comes from an untrusted source.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import operator
|
| 6 |
+
import types
|
| 7 |
+
import typing as t
|
| 8 |
+
from _string import formatter_field_name_split # type: ignore
|
| 9 |
+
from collections import abc
|
| 10 |
+
from collections import deque
|
| 11 |
+
from functools import update_wrapper
|
| 12 |
+
from string import Formatter
|
| 13 |
+
|
| 14 |
+
from markupsafe import EscapeFormatter
|
| 15 |
+
from markupsafe import Markup
|
| 16 |
+
|
| 17 |
+
from .environment import Environment
|
| 18 |
+
from .exceptions import SecurityError
|
| 19 |
+
from .runtime import Context
|
| 20 |
+
from .runtime import Undefined
|
| 21 |
+
|
| 22 |
+
F = t.TypeVar("F", bound=t.Callable[..., t.Any])
|
| 23 |
+
|
| 24 |
+
#: maximum number of items a range may produce
|
| 25 |
+
MAX_RANGE = 100000
|
| 26 |
+
|
| 27 |
+
#: Unsafe function attributes.
|
| 28 |
+
UNSAFE_FUNCTION_ATTRIBUTES: t.Set[str] = set()
|
| 29 |
+
|
| 30 |
+
#: Unsafe method attributes. Function attributes are unsafe for methods too.
|
| 31 |
+
UNSAFE_METHOD_ATTRIBUTES: t.Set[str] = set()
|
| 32 |
+
|
| 33 |
+
#: unsafe generator attributes.
|
| 34 |
+
UNSAFE_GENERATOR_ATTRIBUTES = {"gi_frame", "gi_code"}
|
| 35 |
+
|
| 36 |
+
#: unsafe attributes on coroutines
|
| 37 |
+
UNSAFE_COROUTINE_ATTRIBUTES = {"cr_frame", "cr_code"}
|
| 38 |
+
|
| 39 |
+
#: unsafe attributes on async generators
|
| 40 |
+
UNSAFE_ASYNC_GENERATOR_ATTRIBUTES = {"ag_code", "ag_frame"}
|
| 41 |
+
|
| 42 |
+
_mutable_spec: t.Tuple[t.Tuple[t.Type[t.Any], t.FrozenSet[str]], ...] = (
|
| 43 |
+
(
|
| 44 |
+
abc.MutableSet,
|
| 45 |
+
frozenset(
|
| 46 |
+
[
|
| 47 |
+
"add",
|
| 48 |
+
"clear",
|
| 49 |
+
"difference_update",
|
| 50 |
+
"discard",
|
| 51 |
+
"pop",
|
| 52 |
+
"remove",
|
| 53 |
+
"symmetric_difference_update",
|
| 54 |
+
"update",
|
| 55 |
+
]
|
| 56 |
+
),
|
| 57 |
+
),
|
| 58 |
+
(
|
| 59 |
+
abc.MutableMapping,
|
| 60 |
+
frozenset(["clear", "pop", "popitem", "setdefault", "update"]),
|
| 61 |
+
),
|
| 62 |
+
(
|
| 63 |
+
abc.MutableSequence,
|
| 64 |
+
frozenset(
|
| 65 |
+
["append", "clear", "pop", "reverse", "insert", "sort", "extend", "remove"]
|
| 66 |
+
),
|
| 67 |
+
),
|
| 68 |
+
(
|
| 69 |
+
deque,
|
| 70 |
+
frozenset(
|
| 71 |
+
[
|
| 72 |
+
"append",
|
| 73 |
+
"appendleft",
|
| 74 |
+
"clear",
|
| 75 |
+
"extend",
|
| 76 |
+
"extendleft",
|
| 77 |
+
"pop",
|
| 78 |
+
"popleft",
|
| 79 |
+
"remove",
|
| 80 |
+
"rotate",
|
| 81 |
+
]
|
| 82 |
+
),
|
| 83 |
+
),
|
| 84 |
+
)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def safe_range(*args: int) -> range:
|
| 88 |
+
"""A range that can't generate ranges with a length of more than
|
| 89 |
+
MAX_RANGE items.
|
| 90 |
+
"""
|
| 91 |
+
rng = range(*args)
|
| 92 |
+
|
| 93 |
+
if len(rng) > MAX_RANGE:
|
| 94 |
+
raise OverflowError(
|
| 95 |
+
"Range too big. The sandbox blocks ranges larger than"
|
| 96 |
+
f" MAX_RANGE ({MAX_RANGE})."
|
| 97 |
+
)
|
| 98 |
+
|
| 99 |
+
return rng
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def unsafe(f: F) -> F:
|
| 103 |
+
"""Marks a function or method as unsafe.
|
| 104 |
+
|
| 105 |
+
.. code-block: python
|
| 106 |
+
|
| 107 |
+
@unsafe
|
| 108 |
+
def delete(self):
|
| 109 |
+
pass
|
| 110 |
+
"""
|
| 111 |
+
f.unsafe_callable = True # type: ignore
|
| 112 |
+
return f
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def is_internal_attribute(obj: t.Any, attr: str) -> bool:
|
| 116 |
+
"""Test if the attribute given is an internal python attribute. For
|
| 117 |
+
example this function returns `True` for the `func_code` attribute of
|
| 118 |
+
python objects. This is useful if the environment method
|
| 119 |
+
:meth:`~SandboxedEnvironment.is_safe_attribute` is overridden.
|
| 120 |
+
|
| 121 |
+
>>> from jinja2.sandbox import is_internal_attribute
|
| 122 |
+
>>> is_internal_attribute(str, "mro")
|
| 123 |
+
True
|
| 124 |
+
>>> is_internal_attribute(str, "upper")
|
| 125 |
+
False
|
| 126 |
+
"""
|
| 127 |
+
if isinstance(obj, types.FunctionType):
|
| 128 |
+
if attr in UNSAFE_FUNCTION_ATTRIBUTES:
|
| 129 |
+
return True
|
| 130 |
+
elif isinstance(obj, types.MethodType):
|
| 131 |
+
if attr in UNSAFE_FUNCTION_ATTRIBUTES or attr in UNSAFE_METHOD_ATTRIBUTES:
|
| 132 |
+
return True
|
| 133 |
+
elif isinstance(obj, type):
|
| 134 |
+
if attr == "mro":
|
| 135 |
+
return True
|
| 136 |
+
elif isinstance(obj, (types.CodeType, types.TracebackType, types.FrameType)):
|
| 137 |
+
return True
|
| 138 |
+
elif isinstance(obj, types.GeneratorType):
|
| 139 |
+
if attr in UNSAFE_GENERATOR_ATTRIBUTES:
|
| 140 |
+
return True
|
| 141 |
+
elif hasattr(types, "CoroutineType") and isinstance(obj, types.CoroutineType):
|
| 142 |
+
if attr in UNSAFE_COROUTINE_ATTRIBUTES:
|
| 143 |
+
return True
|
| 144 |
+
elif hasattr(types, "AsyncGeneratorType") and isinstance(
|
| 145 |
+
obj, types.AsyncGeneratorType
|
| 146 |
+
):
|
| 147 |
+
if attr in UNSAFE_ASYNC_GENERATOR_ATTRIBUTES:
|
| 148 |
+
return True
|
| 149 |
+
return attr.startswith("__")
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def modifies_known_mutable(obj: t.Any, attr: str) -> bool:
|
| 153 |
+
"""This function checks if an attribute on a builtin mutable object
|
| 154 |
+
(list, dict, set or deque) or the corresponding ABCs would modify it
|
| 155 |
+
if called.
|
| 156 |
+
|
| 157 |
+
>>> modifies_known_mutable({}, "clear")
|
| 158 |
+
True
|
| 159 |
+
>>> modifies_known_mutable({}, "keys")
|
| 160 |
+
False
|
| 161 |
+
>>> modifies_known_mutable([], "append")
|
| 162 |
+
True
|
| 163 |
+
>>> modifies_known_mutable([], "index")
|
| 164 |
+
False
|
| 165 |
+
|
| 166 |
+
If called with an unsupported object, ``False`` is returned.
|
| 167 |
+
|
| 168 |
+
>>> modifies_known_mutable("foo", "upper")
|
| 169 |
+
False
|
| 170 |
+
"""
|
| 171 |
+
for typespec, unsafe in _mutable_spec:
|
| 172 |
+
if isinstance(obj, typespec):
|
| 173 |
+
return attr in unsafe
|
| 174 |
+
return False
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
class SandboxedEnvironment(Environment):
|
| 178 |
+
"""The sandboxed environment. It works like the regular environment but
|
| 179 |
+
tells the compiler to generate sandboxed code. Additionally subclasses of
|
| 180 |
+
this environment may override the methods that tell the runtime what
|
| 181 |
+
attributes or functions are safe to access.
|
| 182 |
+
|
| 183 |
+
If the template tries to access insecure code a :exc:`SecurityError` is
|
| 184 |
+
raised. However also other exceptions may occur during the rendering so
|
| 185 |
+
the caller has to ensure that all exceptions are caught.
|
| 186 |
+
"""
|
| 187 |
+
|
| 188 |
+
sandboxed = True
|
| 189 |
+
|
| 190 |
+
#: default callback table for the binary operators. A copy of this is
|
| 191 |
+
#: available on each instance of a sandboxed environment as
|
| 192 |
+
#: :attr:`binop_table`
|
| 193 |
+
default_binop_table: t.Dict[str, t.Callable[[t.Any, t.Any], t.Any]] = {
|
| 194 |
+
"+": operator.add,
|
| 195 |
+
"-": operator.sub,
|
| 196 |
+
"*": operator.mul,
|
| 197 |
+
"/": operator.truediv,
|
| 198 |
+
"//": operator.floordiv,
|
| 199 |
+
"**": operator.pow,
|
| 200 |
+
"%": operator.mod,
|
| 201 |
+
}
|
| 202 |
+
|
| 203 |
+
#: default callback table for the unary operators. A copy of this is
|
| 204 |
+
#: available on each instance of a sandboxed environment as
|
| 205 |
+
#: :attr:`unop_table`
|
| 206 |
+
default_unop_table: t.Dict[str, t.Callable[[t.Any], t.Any]] = {
|
| 207 |
+
"+": operator.pos,
|
| 208 |
+
"-": operator.neg,
|
| 209 |
+
}
|
| 210 |
+
|
| 211 |
+
#: a set of binary operators that should be intercepted. Each operator
|
| 212 |
+
#: that is added to this set (empty by default) is delegated to the
|
| 213 |
+
#: :meth:`call_binop` method that will perform the operator. The default
|
| 214 |
+
#: operator callback is specified by :attr:`binop_table`.
|
| 215 |
+
#:
|
| 216 |
+
#: The following binary operators are interceptable:
|
| 217 |
+
#: ``//``, ``%``, ``+``, ``*``, ``-``, ``/``, and ``**``
|
| 218 |
+
#:
|
| 219 |
+
#: The default operation form the operator table corresponds to the
|
| 220 |
+
#: builtin function. Intercepted calls are always slower than the native
|
| 221 |
+
#: operator call, so make sure only to intercept the ones you are
|
| 222 |
+
#: interested in.
|
| 223 |
+
#:
|
| 224 |
+
#: .. versionadded:: 2.6
|
| 225 |
+
intercepted_binops: t.FrozenSet[str] = frozenset()
|
| 226 |
+
|
| 227 |
+
#: a set of unary operators that should be intercepted. Each operator
|
| 228 |
+
#: that is added to this set (empty by default) is delegated to the
|
| 229 |
+
#: :meth:`call_unop` method that will perform the operator. The default
|
| 230 |
+
#: operator callback is specified by :attr:`unop_table`.
|
| 231 |
+
#:
|
| 232 |
+
#: The following unary operators are interceptable: ``+``, ``-``
|
| 233 |
+
#:
|
| 234 |
+
#: The default operation form the operator table corresponds to the
|
| 235 |
+
#: builtin function. Intercepted calls are always slower than the native
|
| 236 |
+
#: operator call, so make sure only to intercept the ones you are
|
| 237 |
+
#: interested in.
|
| 238 |
+
#:
|
| 239 |
+
#: .. versionadded:: 2.6
|
| 240 |
+
intercepted_unops: t.FrozenSet[str] = frozenset()
|
| 241 |
+
|
| 242 |
+
def __init__(self, *args: t.Any, **kwargs: t.Any) -> None:
|
| 243 |
+
super().__init__(*args, **kwargs)
|
| 244 |
+
self.globals["range"] = safe_range
|
| 245 |
+
self.binop_table = self.default_binop_table.copy()
|
| 246 |
+
self.unop_table = self.default_unop_table.copy()
|
| 247 |
+
|
| 248 |
+
def is_safe_attribute(self, obj: t.Any, attr: str, value: t.Any) -> bool:
|
| 249 |
+
"""The sandboxed environment will call this method to check if the
|
| 250 |
+
attribute of an object is safe to access. Per default all attributes
|
| 251 |
+
starting with an underscore are considered private as well as the
|
| 252 |
+
special attributes of internal python objects as returned by the
|
| 253 |
+
:func:`is_internal_attribute` function.
|
| 254 |
+
"""
|
| 255 |
+
return not (attr.startswith("_") or is_internal_attribute(obj, attr))
|
| 256 |
+
|
| 257 |
+
def is_safe_callable(self, obj: t.Any) -> bool:
|
| 258 |
+
"""Check if an object is safely callable. By default callables
|
| 259 |
+
are considered safe unless decorated with :func:`unsafe`.
|
| 260 |
+
|
| 261 |
+
This also recognizes the Django convention of setting
|
| 262 |
+
``func.alters_data = True``.
|
| 263 |
+
"""
|
| 264 |
+
return not (
|
| 265 |
+
getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
|
| 266 |
+
)
|
| 267 |
+
|
| 268 |
+
def call_binop(
|
| 269 |
+
self, context: Context, operator: str, left: t.Any, right: t.Any
|
| 270 |
+
) -> t.Any:
|
| 271 |
+
"""For intercepted binary operator calls (:meth:`intercepted_binops`)
|
| 272 |
+
this function is executed instead of the builtin operator. This can
|
| 273 |
+
be used to fine tune the behavior of certain operators.
|
| 274 |
+
|
| 275 |
+
.. versionadded:: 2.6
|
| 276 |
+
"""
|
| 277 |
+
return self.binop_table[operator](left, right)
|
| 278 |
+
|
| 279 |
+
def call_unop(self, context: Context, operator: str, arg: t.Any) -> t.Any:
|
| 280 |
+
"""For intercepted unary operator calls (:meth:`intercepted_unops`)
|
| 281 |
+
this function is executed instead of the builtin operator. This can
|
| 282 |
+
be used to fine tune the behavior of certain operators.
|
| 283 |
+
|
| 284 |
+
.. versionadded:: 2.6
|
| 285 |
+
"""
|
| 286 |
+
return self.unop_table[operator](arg)
|
| 287 |
+
|
| 288 |
+
def getitem(
|
| 289 |
+
self, obj: t.Any, argument: t.Union[str, t.Any]
|
| 290 |
+
) -> t.Union[t.Any, Undefined]:
|
| 291 |
+
"""Subscribe an object from sandboxed code."""
|
| 292 |
+
try:
|
| 293 |
+
return obj[argument]
|
| 294 |
+
except (TypeError, LookupError):
|
| 295 |
+
if isinstance(argument, str):
|
| 296 |
+
try:
|
| 297 |
+
attr = str(argument)
|
| 298 |
+
except Exception:
|
| 299 |
+
pass
|
| 300 |
+
else:
|
| 301 |
+
try:
|
| 302 |
+
value = getattr(obj, attr)
|
| 303 |
+
except AttributeError:
|
| 304 |
+
pass
|
| 305 |
+
else:
|
| 306 |
+
fmt = self.wrap_str_format(value)
|
| 307 |
+
if fmt is not None:
|
| 308 |
+
return fmt
|
| 309 |
+
if self.is_safe_attribute(obj, argument, value):
|
| 310 |
+
return value
|
| 311 |
+
return self.unsafe_undefined(obj, argument)
|
| 312 |
+
return self.undefined(obj=obj, name=argument)
|
| 313 |
+
|
| 314 |
+
def getattr(self, obj: t.Any, attribute: str) -> t.Union[t.Any, Undefined]:
|
| 315 |
+
"""Subscribe an object from sandboxed code and prefer the
|
| 316 |
+
attribute. The attribute passed *must* be a bytestring.
|
| 317 |
+
"""
|
| 318 |
+
try:
|
| 319 |
+
value = getattr(obj, attribute)
|
| 320 |
+
except AttributeError:
|
| 321 |
+
try:
|
| 322 |
+
return obj[attribute]
|
| 323 |
+
except (TypeError, LookupError):
|
| 324 |
+
pass
|
| 325 |
+
else:
|
| 326 |
+
fmt = self.wrap_str_format(value)
|
| 327 |
+
if fmt is not None:
|
| 328 |
+
return fmt
|
| 329 |
+
if self.is_safe_attribute(obj, attribute, value):
|
| 330 |
+
return value
|
| 331 |
+
return self.unsafe_undefined(obj, attribute)
|
| 332 |
+
return self.undefined(obj=obj, name=attribute)
|
| 333 |
+
|
| 334 |
+
def unsafe_undefined(self, obj: t.Any, attribute: str) -> Undefined:
|
| 335 |
+
"""Return an undefined object for unsafe attributes."""
|
| 336 |
+
return self.undefined(
|
| 337 |
+
f"access to attribute {attribute!r} of"
|
| 338 |
+
f" {type(obj).__name__!r} object is unsafe.",
|
| 339 |
+
name=attribute,
|
| 340 |
+
obj=obj,
|
| 341 |
+
exc=SecurityError,
|
| 342 |
+
)
|
| 343 |
+
|
| 344 |
+
def wrap_str_format(self, value: t.Any) -> t.Optional[t.Callable[..., str]]:
|
| 345 |
+
"""If the given value is a ``str.format`` or ``str.format_map`` method,
|
| 346 |
+
return a new function than handles sandboxing. This is done at access
|
| 347 |
+
rather than in :meth:`call`, so that calls made without ``call`` are
|
| 348 |
+
also sandboxed.
|
| 349 |
+
"""
|
| 350 |
+
if not isinstance(
|
| 351 |
+
value, (types.MethodType, types.BuiltinMethodType)
|
| 352 |
+
) or value.__name__ not in ("format", "format_map"):
|
| 353 |
+
return None
|
| 354 |
+
|
| 355 |
+
f_self: t.Any = value.__self__
|
| 356 |
+
|
| 357 |
+
if not isinstance(f_self, str):
|
| 358 |
+
return None
|
| 359 |
+
|
| 360 |
+
str_type: t.Type[str] = type(f_self)
|
| 361 |
+
is_format_map = value.__name__ == "format_map"
|
| 362 |
+
formatter: SandboxedFormatter
|
| 363 |
+
|
| 364 |
+
if isinstance(f_self, Markup):
|
| 365 |
+
formatter = SandboxedEscapeFormatter(self, escape=f_self.escape)
|
| 366 |
+
else:
|
| 367 |
+
formatter = SandboxedFormatter(self)
|
| 368 |
+
|
| 369 |
+
vformat = formatter.vformat
|
| 370 |
+
|
| 371 |
+
def wrapper(*args: t.Any, **kwargs: t.Any) -> str:
|
| 372 |
+
if is_format_map:
|
| 373 |
+
if kwargs:
|
| 374 |
+
raise TypeError("format_map() takes no keyword arguments")
|
| 375 |
+
|
| 376 |
+
if len(args) != 1:
|
| 377 |
+
raise TypeError(
|
| 378 |
+
f"format_map() takes exactly one argument ({len(args)} given)"
|
| 379 |
+
)
|
| 380 |
+
|
| 381 |
+
kwargs = args[0]
|
| 382 |
+
args = ()
|
| 383 |
+
|
| 384 |
+
return str_type(vformat(f_self, args, kwargs))
|
| 385 |
+
|
| 386 |
+
return update_wrapper(wrapper, value)
|
| 387 |
+
|
| 388 |
+
def call(
|
| 389 |
+
__self, # noqa: B902
|
| 390 |
+
__context: Context,
|
| 391 |
+
__obj: t.Any,
|
| 392 |
+
*args: t.Any,
|
| 393 |
+
**kwargs: t.Any,
|
| 394 |
+
) -> t.Any:
|
| 395 |
+
"""Call an object from sandboxed code."""
|
| 396 |
+
|
| 397 |
+
# the double prefixes are to avoid double keyword argument
|
| 398 |
+
# errors when proxying the call.
|
| 399 |
+
if not __self.is_safe_callable(__obj):
|
| 400 |
+
raise SecurityError(f"{__obj!r} is not safely callable")
|
| 401 |
+
return __context.call(__obj, *args, **kwargs)
|
| 402 |
+
|
| 403 |
+
|
| 404 |
+
class ImmutableSandboxedEnvironment(SandboxedEnvironment):
|
| 405 |
+
"""Works exactly like the regular `SandboxedEnvironment` but does not
|
| 406 |
+
permit modifications on the builtin mutable objects `list`, `set`, and
|
| 407 |
+
`dict` by using the :func:`modifies_known_mutable` function.
|
| 408 |
+
"""
|
| 409 |
+
|
| 410 |
+
def is_safe_attribute(self, obj: t.Any, attr: str, value: t.Any) -> bool:
|
| 411 |
+
if not super().is_safe_attribute(obj, attr, value):
|
| 412 |
+
return False
|
| 413 |
+
|
| 414 |
+
return not modifies_known_mutable(obj, attr)
|
| 415 |
+
|
| 416 |
+
|
| 417 |
+
class SandboxedFormatter(Formatter):
|
| 418 |
+
def __init__(self, env: Environment, **kwargs: t.Any) -> None:
|
| 419 |
+
self._env = env
|
| 420 |
+
super().__init__(**kwargs)
|
| 421 |
+
|
| 422 |
+
def get_field(
|
| 423 |
+
self, field_name: str, args: t.Sequence[t.Any], kwargs: t.Mapping[str, t.Any]
|
| 424 |
+
) -> t.Tuple[t.Any, str]:
|
| 425 |
+
first, rest = formatter_field_name_split(field_name)
|
| 426 |
+
obj = self.get_value(first, args, kwargs)
|
| 427 |
+
for is_attr, i in rest:
|
| 428 |
+
if is_attr:
|
| 429 |
+
obj = self._env.getattr(obj, i)
|
| 430 |
+
else:
|
| 431 |
+
obj = self._env.getitem(obj, i)
|
| 432 |
+
return obj, first
|
| 433 |
+
|
| 434 |
+
|
| 435 |
+
class SandboxedEscapeFormatter(SandboxedFormatter, EscapeFormatter):
|
| 436 |
+
pass
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/tests.py
ADDED
|
@@ -0,0 +1,256 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Built-in template tests used with the ``is`` operator."""
|
| 2 |
+
|
| 3 |
+
import operator
|
| 4 |
+
import typing as t
|
| 5 |
+
from collections import abc
|
| 6 |
+
from numbers import Number
|
| 7 |
+
|
| 8 |
+
from .runtime import Undefined
|
| 9 |
+
from .utils import pass_environment
|
| 10 |
+
|
| 11 |
+
if t.TYPE_CHECKING:
|
| 12 |
+
from .environment import Environment
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def test_odd(value: int) -> bool:
|
| 16 |
+
"""Return true if the variable is odd."""
|
| 17 |
+
return value % 2 == 1
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def test_even(value: int) -> bool:
|
| 21 |
+
"""Return true if the variable is even."""
|
| 22 |
+
return value % 2 == 0
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def test_divisibleby(value: int, num: int) -> bool:
|
| 26 |
+
"""Check if a variable is divisible by a number."""
|
| 27 |
+
return value % num == 0
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def test_defined(value: t.Any) -> bool:
|
| 31 |
+
"""Return true if the variable is defined:
|
| 32 |
+
|
| 33 |
+
.. sourcecode:: jinja
|
| 34 |
+
|
| 35 |
+
{% if variable is defined %}
|
| 36 |
+
value of variable: {{ variable }}
|
| 37 |
+
{% else %}
|
| 38 |
+
variable is not defined
|
| 39 |
+
{% endif %}
|
| 40 |
+
|
| 41 |
+
See the :func:`default` filter for a simple way to set undefined
|
| 42 |
+
variables.
|
| 43 |
+
"""
|
| 44 |
+
return not isinstance(value, Undefined)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def test_undefined(value: t.Any) -> bool:
|
| 48 |
+
"""Like :func:`defined` but the other way round."""
|
| 49 |
+
return isinstance(value, Undefined)
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
@pass_environment
|
| 53 |
+
def test_filter(env: "Environment", value: str) -> bool:
|
| 54 |
+
"""Check if a filter exists by name. Useful if a filter may be
|
| 55 |
+
optionally available.
|
| 56 |
+
|
| 57 |
+
.. code-block:: jinja
|
| 58 |
+
|
| 59 |
+
{% if 'markdown' is filter %}
|
| 60 |
+
{{ value | markdown }}
|
| 61 |
+
{% else %}
|
| 62 |
+
{{ value }}
|
| 63 |
+
{% endif %}
|
| 64 |
+
|
| 65 |
+
.. versionadded:: 3.0
|
| 66 |
+
"""
|
| 67 |
+
return value in env.filters
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
@pass_environment
|
| 71 |
+
def test_test(env: "Environment", value: str) -> bool:
|
| 72 |
+
"""Check if a test exists by name. Useful if a test may be
|
| 73 |
+
optionally available.
|
| 74 |
+
|
| 75 |
+
.. code-block:: jinja
|
| 76 |
+
|
| 77 |
+
{% if 'loud' is test %}
|
| 78 |
+
{% if value is loud %}
|
| 79 |
+
{{ value|upper }}
|
| 80 |
+
{% else %}
|
| 81 |
+
{{ value|lower }}
|
| 82 |
+
{% endif %}
|
| 83 |
+
{% else %}
|
| 84 |
+
{{ value }}
|
| 85 |
+
{% endif %}
|
| 86 |
+
|
| 87 |
+
.. versionadded:: 3.0
|
| 88 |
+
"""
|
| 89 |
+
return value in env.tests
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def test_none(value: t.Any) -> bool:
|
| 93 |
+
"""Return true if the variable is none."""
|
| 94 |
+
return value is None
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def test_boolean(value: t.Any) -> bool:
|
| 98 |
+
"""Return true if the object is a boolean value.
|
| 99 |
+
|
| 100 |
+
.. versionadded:: 2.11
|
| 101 |
+
"""
|
| 102 |
+
return value is True or value is False
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def test_false(value: t.Any) -> bool:
|
| 106 |
+
"""Return true if the object is False.
|
| 107 |
+
|
| 108 |
+
.. versionadded:: 2.11
|
| 109 |
+
"""
|
| 110 |
+
return value is False
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def test_true(value: t.Any) -> bool:
|
| 114 |
+
"""Return true if the object is True.
|
| 115 |
+
|
| 116 |
+
.. versionadded:: 2.11
|
| 117 |
+
"""
|
| 118 |
+
return value is True
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
# NOTE: The existing 'number' test matches booleans and floats
|
| 122 |
+
def test_integer(value: t.Any) -> bool:
|
| 123 |
+
"""Return true if the object is an integer.
|
| 124 |
+
|
| 125 |
+
.. versionadded:: 2.11
|
| 126 |
+
"""
|
| 127 |
+
return isinstance(value, int) and value is not True and value is not False
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
# NOTE: The existing 'number' test matches booleans and integers
|
| 131 |
+
def test_float(value: t.Any) -> bool:
|
| 132 |
+
"""Return true if the object is a float.
|
| 133 |
+
|
| 134 |
+
.. versionadded:: 2.11
|
| 135 |
+
"""
|
| 136 |
+
return isinstance(value, float)
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def test_lower(value: str) -> bool:
|
| 140 |
+
"""Return true if the variable is lowercased."""
|
| 141 |
+
return str(value).islower()
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
def test_upper(value: str) -> bool:
|
| 145 |
+
"""Return true if the variable is uppercased."""
|
| 146 |
+
return str(value).isupper()
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def test_string(value: t.Any) -> bool:
|
| 150 |
+
"""Return true if the object is a string."""
|
| 151 |
+
return isinstance(value, str)
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def test_mapping(value: t.Any) -> bool:
|
| 155 |
+
"""Return true if the object is a mapping (dict etc.).
|
| 156 |
+
|
| 157 |
+
.. versionadded:: 2.6
|
| 158 |
+
"""
|
| 159 |
+
return isinstance(value, abc.Mapping)
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def test_number(value: t.Any) -> bool:
|
| 163 |
+
"""Return true if the variable is a number."""
|
| 164 |
+
return isinstance(value, Number)
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def test_sequence(value: t.Any) -> bool:
|
| 168 |
+
"""Return true if the variable is a sequence. Sequences are variables
|
| 169 |
+
that are iterable.
|
| 170 |
+
"""
|
| 171 |
+
try:
|
| 172 |
+
len(value)
|
| 173 |
+
value.__getitem__ # noqa B018
|
| 174 |
+
except Exception:
|
| 175 |
+
return False
|
| 176 |
+
|
| 177 |
+
return True
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
def test_sameas(value: t.Any, other: t.Any) -> bool:
|
| 181 |
+
"""Check if an object points to the same memory address than another
|
| 182 |
+
object:
|
| 183 |
+
|
| 184 |
+
.. sourcecode:: jinja
|
| 185 |
+
|
| 186 |
+
{% if foo.attribute is sameas false %}
|
| 187 |
+
the foo attribute really is the `False` singleton
|
| 188 |
+
{% endif %}
|
| 189 |
+
"""
|
| 190 |
+
return value is other
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
def test_iterable(value: t.Any) -> bool:
|
| 194 |
+
"""Check if it's possible to iterate over an object."""
|
| 195 |
+
try:
|
| 196 |
+
iter(value)
|
| 197 |
+
except TypeError:
|
| 198 |
+
return False
|
| 199 |
+
|
| 200 |
+
return True
|
| 201 |
+
|
| 202 |
+
|
| 203 |
+
def test_escaped(value: t.Any) -> bool:
|
| 204 |
+
"""Check if the value is escaped."""
|
| 205 |
+
return hasattr(value, "__html__")
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
def test_in(value: t.Any, seq: t.Container[t.Any]) -> bool:
|
| 209 |
+
"""Check if value is in seq.
|
| 210 |
+
|
| 211 |
+
.. versionadded:: 2.10
|
| 212 |
+
"""
|
| 213 |
+
return value in seq
|
| 214 |
+
|
| 215 |
+
|
| 216 |
+
TESTS = {
|
| 217 |
+
"odd": test_odd,
|
| 218 |
+
"even": test_even,
|
| 219 |
+
"divisibleby": test_divisibleby,
|
| 220 |
+
"defined": test_defined,
|
| 221 |
+
"undefined": test_undefined,
|
| 222 |
+
"filter": test_filter,
|
| 223 |
+
"test": test_test,
|
| 224 |
+
"none": test_none,
|
| 225 |
+
"boolean": test_boolean,
|
| 226 |
+
"false": test_false,
|
| 227 |
+
"true": test_true,
|
| 228 |
+
"integer": test_integer,
|
| 229 |
+
"float": test_float,
|
| 230 |
+
"lower": test_lower,
|
| 231 |
+
"upper": test_upper,
|
| 232 |
+
"string": test_string,
|
| 233 |
+
"mapping": test_mapping,
|
| 234 |
+
"number": test_number,
|
| 235 |
+
"sequence": test_sequence,
|
| 236 |
+
"iterable": test_iterable,
|
| 237 |
+
"callable": callable,
|
| 238 |
+
"sameas": test_sameas,
|
| 239 |
+
"escaped": test_escaped,
|
| 240 |
+
"in": test_in,
|
| 241 |
+
"==": operator.eq,
|
| 242 |
+
"eq": operator.eq,
|
| 243 |
+
"equalto": operator.eq,
|
| 244 |
+
"!=": operator.ne,
|
| 245 |
+
"ne": operator.ne,
|
| 246 |
+
">": operator.gt,
|
| 247 |
+
"gt": operator.gt,
|
| 248 |
+
"greaterthan": operator.gt,
|
| 249 |
+
"ge": operator.ge,
|
| 250 |
+
">=": operator.ge,
|
| 251 |
+
"<": operator.lt,
|
| 252 |
+
"lt": operator.lt,
|
| 253 |
+
"lessthan": operator.lt,
|
| 254 |
+
"<=": operator.le,
|
| 255 |
+
"le": operator.le,
|
| 256 |
+
}
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/utils.py
ADDED
|
@@ -0,0 +1,766 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import enum
|
| 2 |
+
import json
|
| 3 |
+
import os
|
| 4 |
+
import re
|
| 5 |
+
import typing as t
|
| 6 |
+
from collections import abc
|
| 7 |
+
from collections import deque
|
| 8 |
+
from random import choice
|
| 9 |
+
from random import randrange
|
| 10 |
+
from threading import Lock
|
| 11 |
+
from types import CodeType
|
| 12 |
+
from urllib.parse import quote_from_bytes
|
| 13 |
+
|
| 14 |
+
import markupsafe
|
| 15 |
+
|
| 16 |
+
if t.TYPE_CHECKING:
|
| 17 |
+
import typing_extensions as te
|
| 18 |
+
|
| 19 |
+
F = t.TypeVar("F", bound=t.Callable[..., t.Any])
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
class _MissingType:
|
| 23 |
+
def __repr__(self) -> str:
|
| 24 |
+
return "missing"
|
| 25 |
+
|
| 26 |
+
def __reduce__(self) -> str:
|
| 27 |
+
return "missing"
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
missing: t.Any = _MissingType()
|
| 31 |
+
"""Special singleton representing missing values for the runtime."""
|
| 32 |
+
|
| 33 |
+
internal_code: t.MutableSet[CodeType] = set()
|
| 34 |
+
|
| 35 |
+
concat = "".join
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def pass_context(f: F) -> F:
|
| 39 |
+
"""Pass the :class:`~jinja2.runtime.Context` as the first argument
|
| 40 |
+
to the decorated function when called while rendering a template.
|
| 41 |
+
|
| 42 |
+
Can be used on functions, filters, and tests.
|
| 43 |
+
|
| 44 |
+
If only ``Context.eval_context`` is needed, use
|
| 45 |
+
:func:`pass_eval_context`. If only ``Context.environment`` is
|
| 46 |
+
needed, use :func:`pass_environment`.
|
| 47 |
+
|
| 48 |
+
.. versionadded:: 3.0.0
|
| 49 |
+
Replaces ``contextfunction`` and ``contextfilter``.
|
| 50 |
+
"""
|
| 51 |
+
f.jinja_pass_arg = _PassArg.context # type: ignore
|
| 52 |
+
return f
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def pass_eval_context(f: F) -> F:
|
| 56 |
+
"""Pass the :class:`~jinja2.nodes.EvalContext` as the first argument
|
| 57 |
+
to the decorated function when called while rendering a template.
|
| 58 |
+
See :ref:`eval-context`.
|
| 59 |
+
|
| 60 |
+
Can be used on functions, filters, and tests.
|
| 61 |
+
|
| 62 |
+
If only ``EvalContext.environment`` is needed, use
|
| 63 |
+
:func:`pass_environment`.
|
| 64 |
+
|
| 65 |
+
.. versionadded:: 3.0.0
|
| 66 |
+
Replaces ``evalcontextfunction`` and ``evalcontextfilter``.
|
| 67 |
+
"""
|
| 68 |
+
f.jinja_pass_arg = _PassArg.eval_context # type: ignore
|
| 69 |
+
return f
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def pass_environment(f: F) -> F:
|
| 73 |
+
"""Pass the :class:`~jinja2.Environment` as the first argument to
|
| 74 |
+
the decorated function when called while rendering a template.
|
| 75 |
+
|
| 76 |
+
Can be used on functions, filters, and tests.
|
| 77 |
+
|
| 78 |
+
.. versionadded:: 3.0.0
|
| 79 |
+
Replaces ``environmentfunction`` and ``environmentfilter``.
|
| 80 |
+
"""
|
| 81 |
+
f.jinja_pass_arg = _PassArg.environment # type: ignore
|
| 82 |
+
return f
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
class _PassArg(enum.Enum):
|
| 86 |
+
context = enum.auto()
|
| 87 |
+
eval_context = enum.auto()
|
| 88 |
+
environment = enum.auto()
|
| 89 |
+
|
| 90 |
+
@classmethod
|
| 91 |
+
def from_obj(cls, obj: F) -> t.Optional["_PassArg"]:
|
| 92 |
+
if hasattr(obj, "jinja_pass_arg"):
|
| 93 |
+
return obj.jinja_pass_arg # type: ignore
|
| 94 |
+
|
| 95 |
+
return None
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def internalcode(f: F) -> F:
|
| 99 |
+
"""Marks the function as internally used"""
|
| 100 |
+
internal_code.add(f.__code__)
|
| 101 |
+
return f
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def is_undefined(obj: t.Any) -> bool:
|
| 105 |
+
"""Check if the object passed is undefined. This does nothing more than
|
| 106 |
+
performing an instance check against :class:`Undefined` but looks nicer.
|
| 107 |
+
This can be used for custom filters or tests that want to react to
|
| 108 |
+
undefined variables. For example a custom default filter can look like
|
| 109 |
+
this::
|
| 110 |
+
|
| 111 |
+
def default(var, default=''):
|
| 112 |
+
if is_undefined(var):
|
| 113 |
+
return default
|
| 114 |
+
return var
|
| 115 |
+
"""
|
| 116 |
+
from .runtime import Undefined
|
| 117 |
+
|
| 118 |
+
return isinstance(obj, Undefined)
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def consume(iterable: t.Iterable[t.Any]) -> None:
|
| 122 |
+
"""Consumes an iterable without doing anything with it."""
|
| 123 |
+
for _ in iterable:
|
| 124 |
+
pass
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
def clear_caches() -> None:
|
| 128 |
+
"""Jinja keeps internal caches for environments and lexers. These are
|
| 129 |
+
used so that Jinja doesn't have to recreate environments and lexers all
|
| 130 |
+
the time. Normally you don't have to care about that but if you are
|
| 131 |
+
measuring memory consumption you may want to clean the caches.
|
| 132 |
+
"""
|
| 133 |
+
from .environment import get_spontaneous_environment
|
| 134 |
+
from .lexer import _lexer_cache
|
| 135 |
+
|
| 136 |
+
get_spontaneous_environment.cache_clear()
|
| 137 |
+
_lexer_cache.clear()
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
def import_string(import_name: str, silent: bool = False) -> t.Any:
|
| 141 |
+
"""Imports an object based on a string. This is useful if you want to
|
| 142 |
+
use import paths as endpoints or something similar. An import path can
|
| 143 |
+
be specified either in dotted notation (``xml.sax.saxutils.escape``)
|
| 144 |
+
or with a colon as object delimiter (``xml.sax.saxutils:escape``).
|
| 145 |
+
|
| 146 |
+
If the `silent` is True the return value will be `None` if the import
|
| 147 |
+
fails.
|
| 148 |
+
|
| 149 |
+
:return: imported object
|
| 150 |
+
"""
|
| 151 |
+
try:
|
| 152 |
+
if ":" in import_name:
|
| 153 |
+
module, obj = import_name.split(":", 1)
|
| 154 |
+
elif "." in import_name:
|
| 155 |
+
module, _, obj = import_name.rpartition(".")
|
| 156 |
+
else:
|
| 157 |
+
return __import__(import_name)
|
| 158 |
+
return getattr(__import__(module, None, None, [obj]), obj)
|
| 159 |
+
except (ImportError, AttributeError):
|
| 160 |
+
if not silent:
|
| 161 |
+
raise
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def open_if_exists(filename: str, mode: str = "rb") -> t.Optional[t.IO[t.Any]]:
|
| 165 |
+
"""Returns a file descriptor for the filename if that file exists,
|
| 166 |
+
otherwise ``None``.
|
| 167 |
+
"""
|
| 168 |
+
if not os.path.isfile(filename):
|
| 169 |
+
return None
|
| 170 |
+
|
| 171 |
+
return open(filename, mode)
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
def object_type_repr(obj: t.Any) -> str:
|
| 175 |
+
"""Returns the name of the object's type. For some recognized
|
| 176 |
+
singletons the name of the object is returned instead. (For
|
| 177 |
+
example for `None` and `Ellipsis`).
|
| 178 |
+
"""
|
| 179 |
+
if obj is None:
|
| 180 |
+
return "None"
|
| 181 |
+
elif obj is Ellipsis:
|
| 182 |
+
return "Ellipsis"
|
| 183 |
+
|
| 184 |
+
cls = type(obj)
|
| 185 |
+
|
| 186 |
+
if cls.__module__ == "builtins":
|
| 187 |
+
return f"{cls.__name__} object"
|
| 188 |
+
|
| 189 |
+
return f"{cls.__module__}.{cls.__name__} object"
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def pformat(obj: t.Any) -> str:
|
| 193 |
+
"""Format an object using :func:`pprint.pformat`."""
|
| 194 |
+
from pprint import pformat
|
| 195 |
+
|
| 196 |
+
return pformat(obj)
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
_http_re = re.compile(
|
| 200 |
+
r"""
|
| 201 |
+
^
|
| 202 |
+
(
|
| 203 |
+
(https?://|www\.) # scheme or www
|
| 204 |
+
(([\w%-]+\.)+)? # subdomain
|
| 205 |
+
(
|
| 206 |
+
[a-z]{2,63} # basic tld
|
| 207 |
+
|
|
| 208 |
+
xn--[\w%]{2,59} # idna tld
|
| 209 |
+
)
|
| 210 |
+
|
|
| 211 |
+
([\w%-]{2,63}\.)+ # basic domain
|
| 212 |
+
(com|net|int|edu|gov|org|info|mil) # basic tld
|
| 213 |
+
|
|
| 214 |
+
(https?://) # scheme
|
| 215 |
+
(
|
| 216 |
+
(([\d]{1,3})(\.[\d]{1,3}){3}) # IPv4
|
| 217 |
+
|
|
| 218 |
+
(\[([\da-f]{0,4}:){2}([\da-f]{0,4}:?){1,6}]) # IPv6
|
| 219 |
+
)
|
| 220 |
+
)
|
| 221 |
+
(?::[\d]{1,5})? # port
|
| 222 |
+
(?:[/?#]\S*)? # path, query, and fragment
|
| 223 |
+
$
|
| 224 |
+
""",
|
| 225 |
+
re.IGNORECASE | re.VERBOSE,
|
| 226 |
+
)
|
| 227 |
+
_email_re = re.compile(r"^\S+@\w[\w.-]*\.\w+$")
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
def urlize(
|
| 231 |
+
text: str,
|
| 232 |
+
trim_url_limit: t.Optional[int] = None,
|
| 233 |
+
rel: t.Optional[str] = None,
|
| 234 |
+
target: t.Optional[str] = None,
|
| 235 |
+
extra_schemes: t.Optional[t.Iterable[str]] = None,
|
| 236 |
+
) -> str:
|
| 237 |
+
"""Convert URLs in text into clickable links.
|
| 238 |
+
|
| 239 |
+
This may not recognize links in some situations. Usually, a more
|
| 240 |
+
comprehensive formatter, such as a Markdown library, is a better
|
| 241 |
+
choice.
|
| 242 |
+
|
| 243 |
+
Works on ``http://``, ``https://``, ``www.``, ``mailto:``, and email
|
| 244 |
+
addresses. Links with trailing punctuation (periods, commas, closing
|
| 245 |
+
parentheses) and leading punctuation (opening parentheses) are
|
| 246 |
+
recognized excluding the punctuation. Email addresses that include
|
| 247 |
+
header fields are not recognized (for example,
|
| 248 |
+
``mailto:address@example.com?cc=copy@example.com``).
|
| 249 |
+
|
| 250 |
+
:param text: Original text containing URLs to link.
|
| 251 |
+
:param trim_url_limit: Shorten displayed URL values to this length.
|
| 252 |
+
:param target: Add the ``target`` attribute to links.
|
| 253 |
+
:param rel: Add the ``rel`` attribute to links.
|
| 254 |
+
:param extra_schemes: Recognize URLs that start with these schemes
|
| 255 |
+
in addition to the default behavior.
|
| 256 |
+
|
| 257 |
+
.. versionchanged:: 3.0
|
| 258 |
+
The ``extra_schemes`` parameter was added.
|
| 259 |
+
|
| 260 |
+
.. versionchanged:: 3.0
|
| 261 |
+
Generate ``https://`` links for URLs without a scheme.
|
| 262 |
+
|
| 263 |
+
.. versionchanged:: 3.0
|
| 264 |
+
The parsing rules were updated. Recognize email addresses with
|
| 265 |
+
or without the ``mailto:`` scheme. Validate IP addresses. Ignore
|
| 266 |
+
parentheses and brackets in more cases.
|
| 267 |
+
"""
|
| 268 |
+
if trim_url_limit is not None:
|
| 269 |
+
|
| 270 |
+
def trim_url(x: str) -> str:
|
| 271 |
+
if len(x) > trim_url_limit:
|
| 272 |
+
return f"{x[:trim_url_limit]}..."
|
| 273 |
+
|
| 274 |
+
return x
|
| 275 |
+
|
| 276 |
+
else:
|
| 277 |
+
|
| 278 |
+
def trim_url(x: str) -> str:
|
| 279 |
+
return x
|
| 280 |
+
|
| 281 |
+
words = re.split(r"(\s+)", str(markupsafe.escape(text)))
|
| 282 |
+
rel_attr = f' rel="{markupsafe.escape(rel)}"' if rel else ""
|
| 283 |
+
target_attr = f' target="{markupsafe.escape(target)}"' if target else ""
|
| 284 |
+
|
| 285 |
+
for i, word in enumerate(words):
|
| 286 |
+
head, middle, tail = "", word, ""
|
| 287 |
+
match = re.match(r"^([(<]|<)+", middle)
|
| 288 |
+
|
| 289 |
+
if match:
|
| 290 |
+
head = match.group()
|
| 291 |
+
middle = middle[match.end() :]
|
| 292 |
+
|
| 293 |
+
# Unlike lead, which is anchored to the start of the string,
|
| 294 |
+
# need to check that the string ends with any of the characters
|
| 295 |
+
# before trying to match all of them, to avoid backtracking.
|
| 296 |
+
if middle.endswith((")", ">", ".", ",", "\n", ">")):
|
| 297 |
+
match = re.search(r"([)>.,\n]|>)+$", middle)
|
| 298 |
+
|
| 299 |
+
if match:
|
| 300 |
+
tail = match.group()
|
| 301 |
+
middle = middle[: match.start()]
|
| 302 |
+
|
| 303 |
+
# Prefer balancing parentheses in URLs instead of ignoring a
|
| 304 |
+
# trailing character.
|
| 305 |
+
for start_char, end_char in ("(", ")"), ("<", ">"), ("<", ">"):
|
| 306 |
+
start_count = middle.count(start_char)
|
| 307 |
+
|
| 308 |
+
if start_count <= middle.count(end_char):
|
| 309 |
+
# Balanced, or lighter on the left
|
| 310 |
+
continue
|
| 311 |
+
|
| 312 |
+
# Move as many as possible from the tail to balance
|
| 313 |
+
for _ in range(min(start_count, tail.count(end_char))):
|
| 314 |
+
end_index = tail.index(end_char) + len(end_char)
|
| 315 |
+
# Move anything in the tail before the end char too
|
| 316 |
+
middle += tail[:end_index]
|
| 317 |
+
tail = tail[end_index:]
|
| 318 |
+
|
| 319 |
+
if _http_re.match(middle):
|
| 320 |
+
if middle.startswith("https://") or middle.startswith("http://"):
|
| 321 |
+
middle = (
|
| 322 |
+
f'<a href="{middle}"{rel_attr}{target_attr}>{trim_url(middle)}</a>'
|
| 323 |
+
)
|
| 324 |
+
else:
|
| 325 |
+
middle = (
|
| 326 |
+
f'<a href="https://{middle}"{rel_attr}{target_attr}>'
|
| 327 |
+
f"{trim_url(middle)}</a>"
|
| 328 |
+
)
|
| 329 |
+
|
| 330 |
+
elif middle.startswith("mailto:") and _email_re.match(middle[7:]):
|
| 331 |
+
middle = f'<a href="{middle}">{middle[7:]}</a>'
|
| 332 |
+
|
| 333 |
+
elif (
|
| 334 |
+
"@" in middle
|
| 335 |
+
and not middle.startswith("www.")
|
| 336 |
+
# ignore values like `@a@b`
|
| 337 |
+
and not middle.startswith("@")
|
| 338 |
+
and ":" not in middle
|
| 339 |
+
and _email_re.match(middle)
|
| 340 |
+
):
|
| 341 |
+
middle = f'<a href="mailto:{middle}">{middle}</a>'
|
| 342 |
+
|
| 343 |
+
elif extra_schemes is not None:
|
| 344 |
+
for scheme in extra_schemes:
|
| 345 |
+
if middle != scheme and middle.startswith(scheme):
|
| 346 |
+
middle = f'<a href="{middle}"{rel_attr}{target_attr}>{middle}</a>'
|
| 347 |
+
|
| 348 |
+
words[i] = f"{head}{middle}{tail}"
|
| 349 |
+
|
| 350 |
+
return "".join(words)
|
| 351 |
+
|
| 352 |
+
|
| 353 |
+
def generate_lorem_ipsum(
|
| 354 |
+
n: int = 5, html: bool = True, min: int = 20, max: int = 100
|
| 355 |
+
) -> str:
|
| 356 |
+
"""Generate some lorem ipsum for the template."""
|
| 357 |
+
from .constants import LOREM_IPSUM_WORDS
|
| 358 |
+
|
| 359 |
+
words = LOREM_IPSUM_WORDS.split()
|
| 360 |
+
result = []
|
| 361 |
+
|
| 362 |
+
for _ in range(n):
|
| 363 |
+
next_capitalized = True
|
| 364 |
+
last_comma = last_fullstop = 0
|
| 365 |
+
word = None
|
| 366 |
+
last = None
|
| 367 |
+
p = []
|
| 368 |
+
|
| 369 |
+
# each paragraph contains out of 20 to 100 words.
|
| 370 |
+
for idx, _ in enumerate(range(randrange(min, max))):
|
| 371 |
+
while True:
|
| 372 |
+
word = choice(words)
|
| 373 |
+
if word != last:
|
| 374 |
+
last = word
|
| 375 |
+
break
|
| 376 |
+
if next_capitalized:
|
| 377 |
+
word = word.capitalize()
|
| 378 |
+
next_capitalized = False
|
| 379 |
+
# add commas
|
| 380 |
+
if idx - randrange(3, 8) > last_comma:
|
| 381 |
+
last_comma = idx
|
| 382 |
+
last_fullstop += 2
|
| 383 |
+
word += ","
|
| 384 |
+
# add end of sentences
|
| 385 |
+
if idx - randrange(10, 20) > last_fullstop:
|
| 386 |
+
last_comma = last_fullstop = idx
|
| 387 |
+
word += "."
|
| 388 |
+
next_capitalized = True
|
| 389 |
+
p.append(word)
|
| 390 |
+
|
| 391 |
+
# ensure that the paragraph ends with a dot.
|
| 392 |
+
p_str = " ".join(p)
|
| 393 |
+
|
| 394 |
+
if p_str.endswith(","):
|
| 395 |
+
p_str = p_str[:-1] + "."
|
| 396 |
+
elif not p_str.endswith("."):
|
| 397 |
+
p_str += "."
|
| 398 |
+
|
| 399 |
+
result.append(p_str)
|
| 400 |
+
|
| 401 |
+
if not html:
|
| 402 |
+
return "\n\n".join(result)
|
| 403 |
+
return markupsafe.Markup(
|
| 404 |
+
"\n".join(f"<p>{markupsafe.escape(x)}</p>" for x in result)
|
| 405 |
+
)
|
| 406 |
+
|
| 407 |
+
|
| 408 |
+
def url_quote(obj: t.Any, charset: str = "utf-8", for_qs: bool = False) -> str:
|
| 409 |
+
"""Quote a string for use in a URL using the given charset.
|
| 410 |
+
|
| 411 |
+
:param obj: String or bytes to quote. Other types are converted to
|
| 412 |
+
string then encoded to bytes using the given charset.
|
| 413 |
+
:param charset: Encode text to bytes using this charset.
|
| 414 |
+
:param for_qs: Quote "/" and use "+" for spaces.
|
| 415 |
+
"""
|
| 416 |
+
if not isinstance(obj, bytes):
|
| 417 |
+
if not isinstance(obj, str):
|
| 418 |
+
obj = str(obj)
|
| 419 |
+
|
| 420 |
+
obj = obj.encode(charset)
|
| 421 |
+
|
| 422 |
+
safe = b"" if for_qs else b"/"
|
| 423 |
+
rv = quote_from_bytes(obj, safe)
|
| 424 |
+
|
| 425 |
+
if for_qs:
|
| 426 |
+
rv = rv.replace("%20", "+")
|
| 427 |
+
|
| 428 |
+
return rv
|
| 429 |
+
|
| 430 |
+
|
| 431 |
+
@abc.MutableMapping.register
|
| 432 |
+
class LRUCache:
|
| 433 |
+
"""A simple LRU Cache implementation."""
|
| 434 |
+
|
| 435 |
+
# this is fast for small capacities (something below 1000) but doesn't
|
| 436 |
+
# scale. But as long as it's only used as storage for templates this
|
| 437 |
+
# won't do any harm.
|
| 438 |
+
|
| 439 |
+
def __init__(self, capacity: int) -> None:
|
| 440 |
+
self.capacity = capacity
|
| 441 |
+
self._mapping: t.Dict[t.Any, t.Any] = {}
|
| 442 |
+
self._queue: te.Deque[t.Any] = deque()
|
| 443 |
+
self._postinit()
|
| 444 |
+
|
| 445 |
+
def _postinit(self) -> None:
|
| 446 |
+
# alias all queue methods for faster lookup
|
| 447 |
+
self._popleft = self._queue.popleft
|
| 448 |
+
self._pop = self._queue.pop
|
| 449 |
+
self._remove = self._queue.remove
|
| 450 |
+
self._wlock = Lock()
|
| 451 |
+
self._append = self._queue.append
|
| 452 |
+
|
| 453 |
+
def __getstate__(self) -> t.Mapping[str, t.Any]:
|
| 454 |
+
return {
|
| 455 |
+
"capacity": self.capacity,
|
| 456 |
+
"_mapping": self._mapping,
|
| 457 |
+
"_queue": self._queue,
|
| 458 |
+
}
|
| 459 |
+
|
| 460 |
+
def __setstate__(self, d: t.Mapping[str, t.Any]) -> None:
|
| 461 |
+
self.__dict__.update(d)
|
| 462 |
+
self._postinit()
|
| 463 |
+
|
| 464 |
+
def __getnewargs__(self) -> t.Tuple[t.Any, ...]:
|
| 465 |
+
return (self.capacity,)
|
| 466 |
+
|
| 467 |
+
def copy(self) -> "te.Self":
|
| 468 |
+
"""Return a shallow copy of the instance."""
|
| 469 |
+
rv = self.__class__(self.capacity)
|
| 470 |
+
rv._mapping.update(self._mapping)
|
| 471 |
+
rv._queue.extend(self._queue)
|
| 472 |
+
return rv
|
| 473 |
+
|
| 474 |
+
def get(self, key: t.Any, default: t.Any = None) -> t.Any:
|
| 475 |
+
"""Return an item from the cache dict or `default`"""
|
| 476 |
+
try:
|
| 477 |
+
return self[key]
|
| 478 |
+
except KeyError:
|
| 479 |
+
return default
|
| 480 |
+
|
| 481 |
+
def setdefault(self, key: t.Any, default: t.Any = None) -> t.Any:
|
| 482 |
+
"""Set `default` if the key is not in the cache otherwise
|
| 483 |
+
leave unchanged. Return the value of this key.
|
| 484 |
+
"""
|
| 485 |
+
try:
|
| 486 |
+
return self[key]
|
| 487 |
+
except KeyError:
|
| 488 |
+
self[key] = default
|
| 489 |
+
return default
|
| 490 |
+
|
| 491 |
+
def clear(self) -> None:
|
| 492 |
+
"""Clear the cache."""
|
| 493 |
+
with self._wlock:
|
| 494 |
+
self._mapping.clear()
|
| 495 |
+
self._queue.clear()
|
| 496 |
+
|
| 497 |
+
def __contains__(self, key: t.Any) -> bool:
|
| 498 |
+
"""Check if a key exists in this cache."""
|
| 499 |
+
return key in self._mapping
|
| 500 |
+
|
| 501 |
+
def __len__(self) -> int:
|
| 502 |
+
"""Return the current size of the cache."""
|
| 503 |
+
return len(self._mapping)
|
| 504 |
+
|
| 505 |
+
def __repr__(self) -> str:
|
| 506 |
+
return f"<{type(self).__name__} {self._mapping!r}>"
|
| 507 |
+
|
| 508 |
+
def __getitem__(self, key: t.Any) -> t.Any:
|
| 509 |
+
"""Get an item from the cache. Moves the item up so that it has the
|
| 510 |
+
highest priority then.
|
| 511 |
+
|
| 512 |
+
Raise a `KeyError` if it does not exist.
|
| 513 |
+
"""
|
| 514 |
+
with self._wlock:
|
| 515 |
+
rv = self._mapping[key]
|
| 516 |
+
|
| 517 |
+
if self._queue[-1] != key:
|
| 518 |
+
try:
|
| 519 |
+
self._remove(key)
|
| 520 |
+
except ValueError:
|
| 521 |
+
# if something removed the key from the container
|
| 522 |
+
# when we read, ignore the ValueError that we would
|
| 523 |
+
# get otherwise.
|
| 524 |
+
pass
|
| 525 |
+
|
| 526 |
+
self._append(key)
|
| 527 |
+
|
| 528 |
+
return rv
|
| 529 |
+
|
| 530 |
+
def __setitem__(self, key: t.Any, value: t.Any) -> None:
|
| 531 |
+
"""Sets the value for an item. Moves the item up so that it
|
| 532 |
+
has the highest priority then.
|
| 533 |
+
"""
|
| 534 |
+
with self._wlock:
|
| 535 |
+
if key in self._mapping:
|
| 536 |
+
self._remove(key)
|
| 537 |
+
elif len(self._mapping) == self.capacity:
|
| 538 |
+
del self._mapping[self._popleft()]
|
| 539 |
+
|
| 540 |
+
self._append(key)
|
| 541 |
+
self._mapping[key] = value
|
| 542 |
+
|
| 543 |
+
def __delitem__(self, key: t.Any) -> None:
|
| 544 |
+
"""Remove an item from the cache dict.
|
| 545 |
+
Raise a `KeyError` if it does not exist.
|
| 546 |
+
"""
|
| 547 |
+
with self._wlock:
|
| 548 |
+
del self._mapping[key]
|
| 549 |
+
|
| 550 |
+
try:
|
| 551 |
+
self._remove(key)
|
| 552 |
+
except ValueError:
|
| 553 |
+
pass
|
| 554 |
+
|
| 555 |
+
def items(self) -> t.Iterable[t.Tuple[t.Any, t.Any]]:
|
| 556 |
+
"""Return a list of items."""
|
| 557 |
+
result = [(key, self._mapping[key]) for key in list(self._queue)]
|
| 558 |
+
result.reverse()
|
| 559 |
+
return result
|
| 560 |
+
|
| 561 |
+
def values(self) -> t.Iterable[t.Any]:
|
| 562 |
+
"""Return a list of all values."""
|
| 563 |
+
return [x[1] for x in self.items()]
|
| 564 |
+
|
| 565 |
+
def keys(self) -> t.Iterable[t.Any]:
|
| 566 |
+
"""Return a list of all keys ordered by most recent usage."""
|
| 567 |
+
return list(self)
|
| 568 |
+
|
| 569 |
+
def __iter__(self) -> t.Iterator[t.Any]:
|
| 570 |
+
return reversed(tuple(self._queue))
|
| 571 |
+
|
| 572 |
+
def __reversed__(self) -> t.Iterator[t.Any]:
|
| 573 |
+
"""Iterate over the keys in the cache dict, oldest items
|
| 574 |
+
coming first.
|
| 575 |
+
"""
|
| 576 |
+
return iter(tuple(self._queue))
|
| 577 |
+
|
| 578 |
+
__copy__ = copy
|
| 579 |
+
|
| 580 |
+
|
| 581 |
+
def select_autoescape(
|
| 582 |
+
enabled_extensions: t.Collection[str] = ("html", "htm", "xml"),
|
| 583 |
+
disabled_extensions: t.Collection[str] = (),
|
| 584 |
+
default_for_string: bool = True,
|
| 585 |
+
default: bool = False,
|
| 586 |
+
) -> t.Callable[[t.Optional[str]], bool]:
|
| 587 |
+
"""Intelligently sets the initial value of autoescaping based on the
|
| 588 |
+
filename of the template. This is the recommended way to configure
|
| 589 |
+
autoescaping if you do not want to write a custom function yourself.
|
| 590 |
+
|
| 591 |
+
If you want to enable it for all templates created from strings or
|
| 592 |
+
for all templates with `.html` and `.xml` extensions::
|
| 593 |
+
|
| 594 |
+
from jinja2 import Environment, select_autoescape
|
| 595 |
+
env = Environment(autoescape=select_autoescape(
|
| 596 |
+
enabled_extensions=('html', 'xml'),
|
| 597 |
+
default_for_string=True,
|
| 598 |
+
))
|
| 599 |
+
|
| 600 |
+
Example configuration to turn it on at all times except if the template
|
| 601 |
+
ends with `.txt`::
|
| 602 |
+
|
| 603 |
+
from jinja2 import Environment, select_autoescape
|
| 604 |
+
env = Environment(autoescape=select_autoescape(
|
| 605 |
+
disabled_extensions=('txt',),
|
| 606 |
+
default_for_string=True,
|
| 607 |
+
default=True,
|
| 608 |
+
))
|
| 609 |
+
|
| 610 |
+
The `enabled_extensions` is an iterable of all the extensions that
|
| 611 |
+
autoescaping should be enabled for. Likewise `disabled_extensions` is
|
| 612 |
+
a list of all templates it should be disabled for. If a template is
|
| 613 |
+
loaded from a string then the default from `default_for_string` is used.
|
| 614 |
+
If nothing matches then the initial value of autoescaping is set to the
|
| 615 |
+
value of `default`.
|
| 616 |
+
|
| 617 |
+
For security reasons this function operates case insensitive.
|
| 618 |
+
|
| 619 |
+
.. versionadded:: 2.9
|
| 620 |
+
"""
|
| 621 |
+
enabled_patterns = tuple(f".{x.lstrip('.').lower()}" for x in enabled_extensions)
|
| 622 |
+
disabled_patterns = tuple(f".{x.lstrip('.').lower()}" for x in disabled_extensions)
|
| 623 |
+
|
| 624 |
+
def autoescape(template_name: t.Optional[str]) -> bool:
|
| 625 |
+
if template_name is None:
|
| 626 |
+
return default_for_string
|
| 627 |
+
template_name = template_name.lower()
|
| 628 |
+
if template_name.endswith(enabled_patterns):
|
| 629 |
+
return True
|
| 630 |
+
if template_name.endswith(disabled_patterns):
|
| 631 |
+
return False
|
| 632 |
+
return default
|
| 633 |
+
|
| 634 |
+
return autoescape
|
| 635 |
+
|
| 636 |
+
|
| 637 |
+
def htmlsafe_json_dumps(
|
| 638 |
+
obj: t.Any, dumps: t.Optional[t.Callable[..., str]] = None, **kwargs: t.Any
|
| 639 |
+
) -> markupsafe.Markup:
|
| 640 |
+
"""Serialize an object to a string of JSON with :func:`json.dumps`,
|
| 641 |
+
then replace HTML-unsafe characters with Unicode escapes and mark
|
| 642 |
+
the result safe with :class:`~markupsafe.Markup`.
|
| 643 |
+
|
| 644 |
+
This is available in templates as the ``|tojson`` filter.
|
| 645 |
+
|
| 646 |
+
The following characters are escaped: ``<``, ``>``, ``&``, ``'``.
|
| 647 |
+
|
| 648 |
+
The returned string is safe to render in HTML documents and
|
| 649 |
+
``<script>`` tags. The exception is in HTML attributes that are
|
| 650 |
+
double quoted; either use single quotes or the ``|forceescape``
|
| 651 |
+
filter.
|
| 652 |
+
|
| 653 |
+
:param obj: The object to serialize to JSON.
|
| 654 |
+
:param dumps: The ``dumps`` function to use. Defaults to
|
| 655 |
+
``env.policies["json.dumps_function"]``, which defaults to
|
| 656 |
+
:func:`json.dumps`.
|
| 657 |
+
:param kwargs: Extra arguments to pass to ``dumps``. Merged onto
|
| 658 |
+
``env.policies["json.dumps_kwargs"]``.
|
| 659 |
+
|
| 660 |
+
.. versionchanged:: 3.0
|
| 661 |
+
The ``dumper`` parameter is renamed to ``dumps``.
|
| 662 |
+
|
| 663 |
+
.. versionadded:: 2.9
|
| 664 |
+
"""
|
| 665 |
+
if dumps is None:
|
| 666 |
+
dumps = json.dumps
|
| 667 |
+
|
| 668 |
+
return markupsafe.Markup(
|
| 669 |
+
dumps(obj, **kwargs)
|
| 670 |
+
.replace("<", "\\u003c")
|
| 671 |
+
.replace(">", "\\u003e")
|
| 672 |
+
.replace("&", "\\u0026")
|
| 673 |
+
.replace("'", "\\u0027")
|
| 674 |
+
)
|
| 675 |
+
|
| 676 |
+
|
| 677 |
+
class Cycler:
|
| 678 |
+
"""Cycle through values by yield them one at a time, then restarting
|
| 679 |
+
once the end is reached. Available as ``cycler`` in templates.
|
| 680 |
+
|
| 681 |
+
Similar to ``loop.cycle``, but can be used outside loops or across
|
| 682 |
+
multiple loops. For example, render a list of folders and files in a
|
| 683 |
+
list, alternating giving them "odd" and "even" classes.
|
| 684 |
+
|
| 685 |
+
.. code-block:: html+jinja
|
| 686 |
+
|
| 687 |
+
{% set row_class = cycler("odd", "even") %}
|
| 688 |
+
<ul class="browser">
|
| 689 |
+
{% for folder in folders %}
|
| 690 |
+
<li class="folder {{ row_class.next() }}">{{ folder }}
|
| 691 |
+
{% endfor %}
|
| 692 |
+
{% for file in files %}
|
| 693 |
+
<li class="file {{ row_class.next() }}">{{ file }}
|
| 694 |
+
{% endfor %}
|
| 695 |
+
</ul>
|
| 696 |
+
|
| 697 |
+
:param items: Each positional argument will be yielded in the order
|
| 698 |
+
given for each cycle.
|
| 699 |
+
|
| 700 |
+
.. versionadded:: 2.1
|
| 701 |
+
"""
|
| 702 |
+
|
| 703 |
+
def __init__(self, *items: t.Any) -> None:
|
| 704 |
+
if not items:
|
| 705 |
+
raise RuntimeError("at least one item has to be provided")
|
| 706 |
+
self.items = items
|
| 707 |
+
self.pos = 0
|
| 708 |
+
|
| 709 |
+
def reset(self) -> None:
|
| 710 |
+
"""Resets the current item to the first item."""
|
| 711 |
+
self.pos = 0
|
| 712 |
+
|
| 713 |
+
@property
|
| 714 |
+
def current(self) -> t.Any:
|
| 715 |
+
"""Return the current item. Equivalent to the item that will be
|
| 716 |
+
returned next time :meth:`next` is called.
|
| 717 |
+
"""
|
| 718 |
+
return self.items[self.pos]
|
| 719 |
+
|
| 720 |
+
def next(self) -> t.Any:
|
| 721 |
+
"""Return the current item, then advance :attr:`current` to the
|
| 722 |
+
next item.
|
| 723 |
+
"""
|
| 724 |
+
rv = self.current
|
| 725 |
+
self.pos = (self.pos + 1) % len(self.items)
|
| 726 |
+
return rv
|
| 727 |
+
|
| 728 |
+
__next__ = next
|
| 729 |
+
|
| 730 |
+
|
| 731 |
+
class Joiner:
|
| 732 |
+
"""A joining helper for templates."""
|
| 733 |
+
|
| 734 |
+
def __init__(self, sep: str = ", ") -> None:
|
| 735 |
+
self.sep = sep
|
| 736 |
+
self.used = False
|
| 737 |
+
|
| 738 |
+
def __call__(self) -> str:
|
| 739 |
+
if not self.used:
|
| 740 |
+
self.used = True
|
| 741 |
+
return ""
|
| 742 |
+
return self.sep
|
| 743 |
+
|
| 744 |
+
|
| 745 |
+
class Namespace:
|
| 746 |
+
"""A namespace object that can hold arbitrary attributes. It may be
|
| 747 |
+
initialized from a dictionary or with keyword arguments."""
|
| 748 |
+
|
| 749 |
+
def __init__(*args: t.Any, **kwargs: t.Any) -> None: # noqa: B902
|
| 750 |
+
self, args = args[0], args[1:]
|
| 751 |
+
self.__attrs = dict(*args, **kwargs)
|
| 752 |
+
|
| 753 |
+
def __getattribute__(self, name: str) -> t.Any:
|
| 754 |
+
# __class__ is needed for the awaitable check in async mode
|
| 755 |
+
if name in {"_Namespace__attrs", "__class__"}:
|
| 756 |
+
return object.__getattribute__(self, name)
|
| 757 |
+
try:
|
| 758 |
+
return self.__attrs[name]
|
| 759 |
+
except KeyError:
|
| 760 |
+
raise AttributeError(name) from None
|
| 761 |
+
|
| 762 |
+
def __setitem__(self, name: str, value: t.Any) -> None:
|
| 763 |
+
self.__attrs[name] = value
|
| 764 |
+
|
| 765 |
+
def __repr__(self) -> str:
|
| 766 |
+
return f"<Namespace {self.__attrs!r}>"
|
TTS/Kokoro/.venv/Lib/site-packages/jinja2/visitor.py
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""API for traversing the AST nodes. Implemented by the compiler and
|
| 2 |
+
meta introspection.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import typing as t
|
| 6 |
+
|
| 7 |
+
from .nodes import Node
|
| 8 |
+
|
| 9 |
+
if t.TYPE_CHECKING:
|
| 10 |
+
import typing_extensions as te
|
| 11 |
+
|
| 12 |
+
class VisitCallable(te.Protocol):
|
| 13 |
+
def __call__(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.Any: ...
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
class NodeVisitor:
|
| 17 |
+
"""Walks the abstract syntax tree and call visitor functions for every
|
| 18 |
+
node found. The visitor functions may return values which will be
|
| 19 |
+
forwarded by the `visit` method.
|
| 20 |
+
|
| 21 |
+
Per default the visitor functions for the nodes are ``'visit_'`` +
|
| 22 |
+
class name of the node. So a `TryFinally` node visit function would
|
| 23 |
+
be `visit_TryFinally`. This behavior can be changed by overriding
|
| 24 |
+
the `get_visitor` function. If no visitor function exists for a node
|
| 25 |
+
(return value `None`) the `generic_visit` visitor is used instead.
|
| 26 |
+
"""
|
| 27 |
+
|
| 28 |
+
def get_visitor(self, node: Node) -> "t.Optional[VisitCallable]":
|
| 29 |
+
"""Return the visitor function for this node or `None` if no visitor
|
| 30 |
+
exists for this node. In that case the generic visit function is
|
| 31 |
+
used instead.
|
| 32 |
+
"""
|
| 33 |
+
return getattr(self, f"visit_{type(node).__name__}", None)
|
| 34 |
+
|
| 35 |
+
def visit(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.Any:
|
| 36 |
+
"""Visit a node."""
|
| 37 |
+
f = self.get_visitor(node)
|
| 38 |
+
|
| 39 |
+
if f is not None:
|
| 40 |
+
return f(node, *args, **kwargs)
|
| 41 |
+
|
| 42 |
+
return self.generic_visit(node, *args, **kwargs)
|
| 43 |
+
|
| 44 |
+
def generic_visit(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.Any:
|
| 45 |
+
"""Called if no explicit visitor function exists for a node."""
|
| 46 |
+
for child_node in node.iter_child_nodes():
|
| 47 |
+
self.visit(child_node, *args, **kwargs)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
class NodeTransformer(NodeVisitor):
|
| 51 |
+
"""Walks the abstract syntax tree and allows modifications of nodes.
|
| 52 |
+
|
| 53 |
+
The `NodeTransformer` will walk the AST and use the return value of the
|
| 54 |
+
visitor functions to replace or remove the old node. If the return
|
| 55 |
+
value of the visitor function is `None` the node will be removed
|
| 56 |
+
from the previous location otherwise it's replaced with the return
|
| 57 |
+
value. The return value may be the original node in which case no
|
| 58 |
+
replacement takes place.
|
| 59 |
+
"""
|
| 60 |
+
|
| 61 |
+
def generic_visit(self, node: Node, *args: t.Any, **kwargs: t.Any) -> Node:
|
| 62 |
+
for field, old_value in node.iter_fields():
|
| 63 |
+
if isinstance(old_value, list):
|
| 64 |
+
new_values = []
|
| 65 |
+
for value in old_value:
|
| 66 |
+
if isinstance(value, Node):
|
| 67 |
+
value = self.visit(value, *args, **kwargs)
|
| 68 |
+
if value is None:
|
| 69 |
+
continue
|
| 70 |
+
elif not isinstance(value, Node):
|
| 71 |
+
new_values.extend(value)
|
| 72 |
+
continue
|
| 73 |
+
new_values.append(value)
|
| 74 |
+
old_value[:] = new_values
|
| 75 |
+
elif isinstance(old_value, Node):
|
| 76 |
+
new_node = self.visit(old_value, *args, **kwargs)
|
| 77 |
+
if new_node is None:
|
| 78 |
+
delattr(node, field)
|
| 79 |
+
else:
|
| 80 |
+
setattr(node, field, new_node)
|
| 81 |
+
return node
|
| 82 |
+
|
| 83 |
+
def visit_list(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.List[Node]:
|
| 84 |
+
"""As transformers may return lists in some places this method
|
| 85 |
+
can be used to enforce a list as return value.
|
| 86 |
+
"""
|
| 87 |
+
rv = self.visit(node, *args, **kwargs)
|
| 88 |
+
|
| 89 |
+
if not isinstance(rv, list):
|
| 90 |
+
return [rv]
|
| 91 |
+
|
| 92 |
+
return rv
|
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/INSTALLER
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
pip
|
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/METADATA
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Metadata-Version: 2.4
|
| 2 |
+
Name: joblib
|
| 3 |
+
Version: 1.5.3
|
| 4 |
+
Summary: Lightweight pipelining with Python functions
|
| 5 |
+
Author-email: Gael Varoquaux <gael.varoquaux@normalesup.org>
|
| 6 |
+
License-Expression: BSD-3-Clause
|
| 7 |
+
Project-URL: Homepage, https://joblib.readthedocs.io
|
| 8 |
+
Project-URL: Source, https://github.com/joblib/joblib
|
| 9 |
+
Platform: any
|
| 10 |
+
Classifier: Development Status :: 5 - Production/Stable
|
| 11 |
+
Classifier: Environment :: Console
|
| 12 |
+
Classifier: Intended Audience :: Developers
|
| 13 |
+
Classifier: Intended Audience :: Science/Research
|
| 14 |
+
Classifier: Intended Audience :: Education
|
| 15 |
+
Classifier: Operating System :: OS Independent
|
| 16 |
+
Classifier: Programming Language :: Python :: 3
|
| 17 |
+
Classifier: Programming Language :: Python :: 3.9
|
| 18 |
+
Classifier: Programming Language :: Python :: 3.10
|
| 19 |
+
Classifier: Programming Language :: Python :: 3.11
|
| 20 |
+
Classifier: Programming Language :: Python :: 3.12
|
| 21 |
+
Classifier: Programming Language :: Python :: 3.13
|
| 22 |
+
Classifier: Topic :: Scientific/Engineering
|
| 23 |
+
Classifier: Topic :: Utilities
|
| 24 |
+
Classifier: Topic :: Software Development :: Libraries
|
| 25 |
+
Requires-Python: >=3.9
|
| 26 |
+
Description-Content-Type: text/x-rst
|
| 27 |
+
License-File: LICENSE.txt
|
| 28 |
+
Dynamic: license-file
|
| 29 |
+
|
| 30 |
+
|PyPi| |CIStatus| |ReadTheDocs| |Codecov|
|
| 31 |
+
|
| 32 |
+
.. |PyPi| image:: https://badge.fury.io/py/joblib.svg
|
| 33 |
+
:target: https://badge.fury.io/py/joblib
|
| 34 |
+
:alt: Joblib version
|
| 35 |
+
|
| 36 |
+
.. |CIStatus| image:: https://github.com/joblib/joblib/actions/workflows/test.yml/badge.svg
|
| 37 |
+
:target: https://github.com/joblib/joblib/actions/workflows/test.yml?query=branch%3Amain
|
| 38 |
+
:alt: CI status
|
| 39 |
+
|
| 40 |
+
.. |ReadTheDocs| image:: https://readthedocs.org/projects/joblib/badge/?version=latest
|
| 41 |
+
:target: https://joblib.readthedocs.io/en/latest/?badge=latest
|
| 42 |
+
:alt: Documentation Status
|
| 43 |
+
|
| 44 |
+
.. |Codecov| image:: https://codecov.io/gh/joblib/joblib/branch/main/graph/badge.svg
|
| 45 |
+
:target: https://codecov.io/gh/joblib/joblib
|
| 46 |
+
:alt: Codecov coverage
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
The homepage of joblib with user documentation is located on:
|
| 50 |
+
|
| 51 |
+
https://joblib.readthedocs.io
|
| 52 |
+
|
| 53 |
+
Getting the latest code
|
| 54 |
+
=======================
|
| 55 |
+
|
| 56 |
+
To get the latest code using git, simply type::
|
| 57 |
+
|
| 58 |
+
git clone https://github.com/joblib/joblib.git
|
| 59 |
+
|
| 60 |
+
If you don't have git installed, you can download a zip
|
| 61 |
+
of the latest code: https://github.com/joblib/joblib/archive/refs/heads/main.zip
|
| 62 |
+
|
| 63 |
+
Installing
|
| 64 |
+
==========
|
| 65 |
+
|
| 66 |
+
You can use `pip` to install joblib from any directory::
|
| 67 |
+
|
| 68 |
+
pip install joblib
|
| 69 |
+
|
| 70 |
+
or install it in editable mode from the source directory::
|
| 71 |
+
|
| 72 |
+
pip install -e .
|
| 73 |
+
|
| 74 |
+
Dependencies
|
| 75 |
+
============
|
| 76 |
+
|
| 77 |
+
- Joblib has no mandatory dependencies besides Python (supported versions are
|
| 78 |
+
3.9+).
|
| 79 |
+
- Joblib has an optional dependency on Numpy (at least version 1.6.1) for array
|
| 80 |
+
manipulation.
|
| 81 |
+
- Joblib includes its own vendored copy of
|
| 82 |
+
`loky <https://github.com/tomMoral/loky>`_ for process management.
|
| 83 |
+
- Joblib can efficiently dump and load numpy arrays but does not require numpy
|
| 84 |
+
to be installed.
|
| 85 |
+
- Joblib has an optional dependency on
|
| 86 |
+
`python-lz4 <https://pypi.python.org/pypi/lz4>`_ as a faster alternative to
|
| 87 |
+
zlib and gzip for compressed serialization.
|
| 88 |
+
- Joblib has an optional dependency on psutil to mitigate memory leaks in
|
| 89 |
+
parallel worker processes.
|
| 90 |
+
- Some examples require external dependencies such as pandas. See the
|
| 91 |
+
instructions in the `Building the docs`_ section for details.
|
| 92 |
+
|
| 93 |
+
Workflow to contribute
|
| 94 |
+
======================
|
| 95 |
+
|
| 96 |
+
To contribute to joblib, first create an account on `github
|
| 97 |
+
<https://github.com/>`_. Once this is done, fork the `joblib repository
|
| 98 |
+
<https://github.com/joblib/joblib>`_ to have your own repository,
|
| 99 |
+
clone it using ``git clone``. Make your changes in a branch of your clone, push
|
| 100 |
+
them to your github account, test them locally, and when you are happy with
|
| 101 |
+
them, send a pull request to the main repository.
|
| 102 |
+
|
| 103 |
+
You can use `pre-commit <https://pre-commit.com/#install>`_ to run code style checks
|
| 104 |
+
before each commit::
|
| 105 |
+
|
| 106 |
+
pip install pre-commit
|
| 107 |
+
pre-commit install
|
| 108 |
+
|
| 109 |
+
pre-commit checks can be disabled for a single commit with::
|
| 110 |
+
|
| 111 |
+
git commit -n
|
| 112 |
+
|
| 113 |
+
Running the test suite
|
| 114 |
+
======================
|
| 115 |
+
|
| 116 |
+
To run the test suite, you need the pytest (version >= 3) and coverage modules.
|
| 117 |
+
Run the test suite using::
|
| 118 |
+
|
| 119 |
+
pytest joblib
|
| 120 |
+
|
| 121 |
+
from the root of the project.
|
| 122 |
+
|
| 123 |
+
Building the docs
|
| 124 |
+
=================
|
| 125 |
+
|
| 126 |
+
To build the docs you need to have sphinx (>=1.4) and some dependencies
|
| 127 |
+
installed::
|
| 128 |
+
|
| 129 |
+
pip install -U -r .readthedocs-requirements.txt
|
| 130 |
+
|
| 131 |
+
The docs can then be built with the following command::
|
| 132 |
+
|
| 133 |
+
make doc
|
| 134 |
+
|
| 135 |
+
The html docs are located in the ``doc/_build/html`` directory.
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
Making a source tarball
|
| 139 |
+
=======================
|
| 140 |
+
|
| 141 |
+
To create a source tarball, eg for packaging or distributing, run the
|
| 142 |
+
following command::
|
| 143 |
+
|
| 144 |
+
pip install build
|
| 145 |
+
python -m build --sdist
|
| 146 |
+
|
| 147 |
+
The tarball will be created in the `dist` directory. This command will create
|
| 148 |
+
the resulting tarball that can be installed with no extra dependencies than the
|
| 149 |
+
Python standard library.
|
| 150 |
+
|
| 151 |
+
Making a release and uploading it to PyPI
|
| 152 |
+
=========================================
|
| 153 |
+
|
| 154 |
+
This command is only run by project manager, to make a release, and
|
| 155 |
+
upload in to PyPI::
|
| 156 |
+
|
| 157 |
+
pip install build
|
| 158 |
+
python -m build --sdist --wheel
|
| 159 |
+
twine upload dist/*
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
Note that the documentation should automatically get updated at each git
|
| 163 |
+
push. If that is not the case, try building th doc locally and resolve
|
| 164 |
+
any doc build error (in particular when running the examples).
|
| 165 |
+
|
| 166 |
+
Updating the changelog
|
| 167 |
+
======================
|
| 168 |
+
|
| 169 |
+
Changes are listed in the CHANGES.rst file. They must be manually updated
|
| 170 |
+
but, the following git command may be used to generate the lines::
|
| 171 |
+
|
| 172 |
+
git log --abbrev-commit --date=short --no-merges --sparse
|
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/RECORD
ADDED
|
@@ -0,0 +1,217 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
joblib-1.5.3.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
|
| 2 |
+
joblib-1.5.3.dist-info/METADATA,sha256=zunGCJauTnqWWoQPveUNER0vgBj2EgHR7S6fw91Ig-Y,5542
|
| 3 |
+
joblib-1.5.3.dist-info/RECORD,,
|
| 4 |
+
joblib-1.5.3.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
|
| 5 |
+
joblib-1.5.3.dist-info/licenses/LICENSE.txt,sha256=QmEpEcGHLF5LQ_auDo7llGfNNQMyJBz3LOkGQCZPrmo,1527
|
| 6 |
+
joblib-1.5.3.dist-info/top_level.txt,sha256=P0LsoZ45gBL7ckL4lqQt7tdbrHD4xlVYhffmhHeeT_U,7
|
| 7 |
+
joblib/__init__.py,sha256=W-0Q8UNWm7rSJmZA9eq-ihckKM6JRKvOm1l8SN44piQ,5337
|
| 8 |
+
joblib/__pycache__/__init__.cpython-312.pyc,,
|
| 9 |
+
joblib/__pycache__/_cloudpickle_wrapper.cpython-312.pyc,,
|
| 10 |
+
joblib/__pycache__/_dask.cpython-312.pyc,,
|
| 11 |
+
joblib/__pycache__/_memmapping_reducer.cpython-312.pyc,,
|
| 12 |
+
joblib/__pycache__/_multiprocessing_helpers.cpython-312.pyc,,
|
| 13 |
+
joblib/__pycache__/_parallel_backends.cpython-312.pyc,,
|
| 14 |
+
joblib/__pycache__/_store_backends.cpython-312.pyc,,
|
| 15 |
+
joblib/__pycache__/_utils.cpython-312.pyc,,
|
| 16 |
+
joblib/__pycache__/backports.cpython-312.pyc,,
|
| 17 |
+
joblib/__pycache__/compressor.cpython-312.pyc,,
|
| 18 |
+
joblib/__pycache__/disk.cpython-312.pyc,,
|
| 19 |
+
joblib/__pycache__/executor.cpython-312.pyc,,
|
| 20 |
+
joblib/__pycache__/func_inspect.cpython-312.pyc,,
|
| 21 |
+
joblib/__pycache__/hashing.cpython-312.pyc,,
|
| 22 |
+
joblib/__pycache__/logger.cpython-312.pyc,,
|
| 23 |
+
joblib/__pycache__/memory.cpython-312.pyc,,
|
| 24 |
+
joblib/__pycache__/numpy_pickle.cpython-312.pyc,,
|
| 25 |
+
joblib/__pycache__/numpy_pickle_compat.cpython-312.pyc,,
|
| 26 |
+
joblib/__pycache__/numpy_pickle_utils.cpython-312.pyc,,
|
| 27 |
+
joblib/__pycache__/parallel.cpython-312.pyc,,
|
| 28 |
+
joblib/__pycache__/pool.cpython-312.pyc,,
|
| 29 |
+
joblib/__pycache__/testing.cpython-312.pyc,,
|
| 30 |
+
joblib/_cloudpickle_wrapper.py,sha256=HSFxIio3jiGnwVCstAa6obbxs4-5aRAIMDDUAA-cDPc,416
|
| 31 |
+
joblib/_dask.py,sha256=xUYA_2VVc0LvPavSiFy8M7TZc6KF0lIxcQhng5kPaXU,13217
|
| 32 |
+
joblib/_memmapping_reducer.py,sha256=AZ6dqA6fXlm4-ehBCf9m1nq43jUPKman4_2whrOButc,28553
|
| 33 |
+
joblib/_multiprocessing_helpers.py,sha256=f8-Vf_8ildmdg991eLz8xk4DJJFTS_bcrhj6CgQ4lxU,1878
|
| 34 |
+
joblib/_parallel_backends.py,sha256=fgy_FgZiKeNvTWr4wKbSX4kUNx2YD6m7p5O1J96xhb4,28766
|
| 35 |
+
joblib/_store_backends.py,sha256=xJWN4xvM1LU8kZXk3LEJwIjeEoj6zOmESsPHWmlUkPw,17918
|
| 36 |
+
joblib/_utils.py,sha256=bE4abMGL08HTLtVeisrww8QjvCllMLHVT2qbmYtAGaA,3248
|
| 37 |
+
joblib/backports.py,sha256=mITpG-yuEADimg89_LCdUY9QH9a5xQHsRNJnd7BmAMo,5450
|
| 38 |
+
joblib/compressor.py,sha256=GDDVJmeOBqftc6tMkDupryojHhk_jIV8WrNoMiTxTdQ,19281
|
| 39 |
+
joblib/disk.py,sha256=1J5hhMsCP5LDW65luTtArUxsMAJRrPB6wxSWf6GeBns,4332
|
| 40 |
+
joblib/executor.py,sha256=fbVmE_KKywjJcIKmHO9k8M3VkaMqZXEP4YXBRz_p6xU,5229
|
| 41 |
+
joblib/externals/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
| 42 |
+
joblib/externals/__pycache__/__init__.cpython-312.pyc,,
|
| 43 |
+
joblib/externals/cloudpickle/__init__.py,sha256=ErKyvqqAErOkbCowrizhGpN9DN-TaD22QtqI9CQ9sDw,308
|
| 44 |
+
joblib/externals/cloudpickle/__pycache__/__init__.cpython-312.pyc,,
|
| 45 |
+
joblib/externals/cloudpickle/__pycache__/cloudpickle.cpython-312.pyc,,
|
| 46 |
+
joblib/externals/cloudpickle/__pycache__/cloudpickle_fast.cpython-312.pyc,,
|
| 47 |
+
joblib/externals/cloudpickle/cloudpickle.py,sha256=KcLA_uAxp2KuMJb9vaTVve7FLFUQu7mYkD3dCscxeRc,58874
|
| 48 |
+
joblib/externals/cloudpickle/cloudpickle_fast.py,sha256=AI5ZKf2AbLNxD8lXyLDpKZyzeZ2ofFtdK1ZWFq_ec1c,323
|
| 49 |
+
joblib/externals/loky/__init__.py,sha256=8LzBTFpYfRFrjD1loIQpRF9QQ_8wwEkssJI6hYcGbfE,1105
|
| 50 |
+
joblib/externals/loky/__pycache__/__init__.cpython-312.pyc,,
|
| 51 |
+
joblib/externals/loky/__pycache__/_base.cpython-312.pyc,,
|
| 52 |
+
joblib/externals/loky/__pycache__/cloudpickle_wrapper.cpython-312.pyc,,
|
| 53 |
+
joblib/externals/loky/__pycache__/initializers.cpython-312.pyc,,
|
| 54 |
+
joblib/externals/loky/__pycache__/process_executor.cpython-312.pyc,,
|
| 55 |
+
joblib/externals/loky/__pycache__/reusable_executor.cpython-312.pyc,,
|
| 56 |
+
joblib/externals/loky/_base.py,sha256=LsQnEoKWKGhdeqGhMc68Aqwz4MrTnEs20KAYbFiUHzo,1057
|
| 57 |
+
joblib/externals/loky/backend/__init__.py,sha256=Ix9KThV1CYk7-M5OQnJ_A_JrrrWJ-Jowa-HMMeGbp18,312
|
| 58 |
+
joblib/externals/loky/backend/__pycache__/__init__.cpython-312.pyc,,
|
| 59 |
+
joblib/externals/loky/backend/__pycache__/_posix_reduction.cpython-312.pyc,,
|
| 60 |
+
joblib/externals/loky/backend/__pycache__/_win_reduction.cpython-312.pyc,,
|
| 61 |
+
joblib/externals/loky/backend/__pycache__/context.cpython-312.pyc,,
|
| 62 |
+
joblib/externals/loky/backend/__pycache__/fork_exec.cpython-312.pyc,,
|
| 63 |
+
joblib/externals/loky/backend/__pycache__/popen_loky_posix.cpython-312.pyc,,
|
| 64 |
+
joblib/externals/loky/backend/__pycache__/popen_loky_win32.cpython-312.pyc,,
|
| 65 |
+
joblib/externals/loky/backend/__pycache__/process.cpython-312.pyc,,
|
| 66 |
+
joblib/externals/loky/backend/__pycache__/queues.cpython-312.pyc,,
|
| 67 |
+
joblib/externals/loky/backend/__pycache__/reduction.cpython-312.pyc,,
|
| 68 |
+
joblib/externals/loky/backend/__pycache__/resource_tracker.cpython-312.pyc,,
|
| 69 |
+
joblib/externals/loky/backend/__pycache__/spawn.cpython-312.pyc,,
|
| 70 |
+
joblib/externals/loky/backend/__pycache__/synchronize.cpython-312.pyc,,
|
| 71 |
+
joblib/externals/loky/backend/__pycache__/utils.cpython-312.pyc,,
|
| 72 |
+
joblib/externals/loky/backend/_posix_reduction.py,sha256=xgCSrIaLI0k_MI0XNOBSp5e1ox1WN9idgrWbkWpMUr4,1776
|
| 73 |
+
joblib/externals/loky/backend/_win_reduction.py,sha256=WmNB0NXtyJ_o_WzfPUEGh5dPhXIeI6FkEnFNXUxO2ws,683
|
| 74 |
+
joblib/externals/loky/backend/context.py,sha256=RPdZvzkEk7iA0rtdAILSHNzl6wsHpm6XD6IL30owAPE,14284
|
| 75 |
+
joblib/externals/loky/backend/fork_exec.py,sha256=4DZ1iLBB-21rlg3Z4Kh9DTVZj35JPaWFE5rzWZaSDxk,2319
|
| 76 |
+
joblib/externals/loky/backend/popen_loky_posix.py,sha256=3G-2_-ovZtjWcHI-xSyW5zQjAZ-_Z9IGjzY1RrZH4nc,5541
|
| 77 |
+
joblib/externals/loky/backend/popen_loky_win32.py,sha256=bYkhRA0w8qUcYFwoezeGwcnlCocEdheWXc6SZ-_rVxo,5325
|
| 78 |
+
joblib/externals/loky/backend/process.py,sha256=4-Y94EoIrg4btsjTNxUBHAHhR96Nrugn_7_PGL6aU50,2018
|
| 79 |
+
joblib/externals/loky/backend/queues.py,sha256=eETFvbPHwKfdoYyOgNQCyKq_Zlm-lzH3fwwpUIh-_4U,7322
|
| 80 |
+
joblib/externals/loky/backend/reduction.py,sha256=861drQAefXTJjfFWAEWmYAS315d8lAyqWx0RgyxFw_0,6926
|
| 81 |
+
joblib/externals/loky/backend/resource_tracker.py,sha256=Jzbmb8otLR7etqhefKuZxAs1VvT1jV8d5Zev8vUcV6s,15403
|
| 82 |
+
joblib/externals/loky/backend/spawn.py,sha256=t4PzEJ3tjwoF9t8qnQUF9R7Q-LmBpDBIcHURWNznz8M,8626
|
| 83 |
+
joblib/externals/loky/backend/synchronize.py,sha256=nlDwBoLZB93m_l55qfZM_Ql-4L84PSYimoQqt5TzpDk,11768
|
| 84 |
+
joblib/externals/loky/backend/utils.py,sha256=RVsxqyET4TJdbjc9uUHJmfhlQ2v4Uq-fiT_5b5rfC0s,5757
|
| 85 |
+
joblib/externals/loky/cloudpickle_wrapper.py,sha256=jUnfhXI3qMXTlCeTUzpABQlv0VOLMJL1V7fpRlq2LgU,3609
|
| 86 |
+
joblib/externals/loky/initializers.py,sha256=dtKtRsJUmVwiJu0yZ-Ih0m8PvW_MxmouG7mShEcsStc,2567
|
| 87 |
+
joblib/externals/loky/process_executor.py,sha256=QPSKet0OCAWr6g_2fHwPt4yjQaAJsjfeJYFPiKhS9RE,52348
|
| 88 |
+
joblib/externals/loky/reusable_executor.py,sha256=d9ksrTnJS8549Oq50iG08u5pEhuMbhQ3oSYUSq0twNQ,10863
|
| 89 |
+
joblib/func_inspect.py,sha256=bhm_GpBe3H_Dmw5ripzP5BalA6wbq7ZFI3SEuAQbfek,14017
|
| 90 |
+
joblib/hashing.py,sha256=38MM0zRl0Ebk78Ij6cMdrQ8ibYZP0pCJxu3L4Yrw1sc,10694
|
| 91 |
+
joblib/logger.py,sha256=HK06qwNWJYInYIIXFYINAKCxjYxi0hoX45ckNKkogHQ,5342
|
| 92 |
+
joblib/memory.py,sha256=va7zWG9s_X6eE-Cm1junrH-QwKTnguin5cEJIhUXo98,45404
|
| 93 |
+
joblib/numpy_pickle.py,sha256=N_wQMf6_vgI71nRYLne0dc2kO6dfh0lkTaOZn8Tq5Hc,28791
|
| 94 |
+
joblib/numpy_pickle_compat.py,sha256=JOlSfMT1uDIztOyQ3dzYgp5fGVnzPVWBCqXjdIZsjLQ,8451
|
| 95 |
+
joblib/numpy_pickle_utils.py,sha256=j3GlI25QFvo-DTPn7uRptu-NtW16ztHM0DuglyQyEDI,9497
|
| 96 |
+
joblib/parallel.py,sha256=SkJYk-cTHC8oMvZU79SDXV61IZ10YIHbBYhrHB47yM8,86989
|
| 97 |
+
joblib/pool.py,sha256=JTc00PEAyPayo8mHdktmburp5OBsnNxwSQI4zzvtKYs,14134
|
| 98 |
+
joblib/test/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
| 99 |
+
joblib/test/__pycache__/__init__.cpython-312.pyc,,
|
| 100 |
+
joblib/test/__pycache__/common.cpython-312.pyc,,
|
| 101 |
+
joblib/test/__pycache__/test_backports.cpython-312.pyc,,
|
| 102 |
+
joblib/test/__pycache__/test_cloudpickle_wrapper.cpython-312.pyc,,
|
| 103 |
+
joblib/test/__pycache__/test_config.cpython-312.pyc,,
|
| 104 |
+
joblib/test/__pycache__/test_dask.cpython-312.pyc,,
|
| 105 |
+
joblib/test/__pycache__/test_disk.cpython-312.pyc,,
|
| 106 |
+
joblib/test/__pycache__/test_func_inspect.cpython-312.pyc,,
|
| 107 |
+
joblib/test/__pycache__/test_func_inspect_special_encoding.cpython-312.pyc,,
|
| 108 |
+
joblib/test/__pycache__/test_hashing.cpython-312.pyc,,
|
| 109 |
+
joblib/test/__pycache__/test_init.cpython-312.pyc,,
|
| 110 |
+
joblib/test/__pycache__/test_logger.cpython-312.pyc,,
|
| 111 |
+
joblib/test/__pycache__/test_memmapping.cpython-312.pyc,,
|
| 112 |
+
joblib/test/__pycache__/test_memory.cpython-312.pyc,,
|
| 113 |
+
joblib/test/__pycache__/test_memory_async.cpython-312.pyc,,
|
| 114 |
+
joblib/test/__pycache__/test_missing_multiprocessing.cpython-312.pyc,,
|
| 115 |
+
joblib/test/__pycache__/test_module.cpython-312.pyc,,
|
| 116 |
+
joblib/test/__pycache__/test_numpy_pickle.cpython-312.pyc,,
|
| 117 |
+
joblib/test/__pycache__/test_numpy_pickle_compat.cpython-312.pyc,,
|
| 118 |
+
joblib/test/__pycache__/test_numpy_pickle_utils.cpython-312.pyc,,
|
| 119 |
+
joblib/test/__pycache__/test_parallel.cpython-312.pyc,,
|
| 120 |
+
joblib/test/__pycache__/test_store_backends.cpython-312.pyc,,
|
| 121 |
+
joblib/test/__pycache__/test_testing.cpython-312.pyc,,
|
| 122 |
+
joblib/test/__pycache__/test_utils.cpython-312.pyc,,
|
| 123 |
+
joblib/test/__pycache__/testutils.cpython-312.pyc,,
|
| 124 |
+
joblib/test/common.py,sha256=vpjpcJgMbmr8H3skc3qsr_KC-u-ZnhVFRk2vAxmJqvA,2102
|
| 125 |
+
joblib/test/data/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
| 126 |
+
joblib/test/data/__pycache__/__init__.cpython-312.pyc,,
|
| 127 |
+
joblib/test/data/__pycache__/create_numpy_pickle.cpython-312.pyc,,
|
| 128 |
+
joblib/test/data/create_numpy_pickle.py,sha256=vZE7JNye9o0gYaxrn1555av6Igee0KeXacAWKNRhsu8,3334
|
| 129 |
+
joblib/test/data/joblib_0.10.0_compressed_pickle_py27_np16.gz,sha256=QYRH6Q2DSGVorjCSqWCxjTWCMOJKyew4Nl2qmfQVvQ8,769
|
| 130 |
+
joblib/test/data/joblib_0.10.0_compressed_pickle_py27_np17.gz,sha256=ofTozM_KlPJa50TR8FCwc09mMmO6OO0GQhgUBLNIsXs,757
|
| 131 |
+
joblib/test/data/joblib_0.10.0_compressed_pickle_py33_np18.gz,sha256=2eIVeA-XjOaT5IEQ6tI2UuHG3hwhiRciMmkBmPcIh4g,792
|
| 132 |
+
joblib/test/data/joblib_0.10.0_compressed_pickle_py34_np19.gz,sha256=Gr2z_1tVWDH1H3_wCVHmakknf8KqeHKT8Yz4d1vmUCM,794
|
| 133 |
+
joblib/test/data/joblib_0.10.0_compressed_pickle_py35_np19.gz,sha256=pWw_xuDbOkECqu1KGf1OFU7s2VbzC2v5F5iXhE7TwB4,790
|
| 134 |
+
joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl,sha256=icRQjj374B-AHk5znxre0T9oWUHokoHIBQ8MqKo8l-U,986
|
| 135 |
+
joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.bz2,sha256=oYQVIyMiUxyRgWSuBBSOvCWKzToA-kUpcoQWdV4UoV4,997
|
| 136 |
+
joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.gzip,sha256=Jpv3iGcDgKTv-O4nZsUreIbUK7qnt2cugZ-VMgNeEDQ,798
|
| 137 |
+
joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.lzma,sha256=c0wu0x8pPv4BcStj7pE61rZpf68FLG_pNzQZ4e82zH8,660
|
| 138 |
+
joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.xz,sha256=77FG1FDG0GHQav-1bxc4Tn9ky6ubUW_MbE0_iGmz5wc,712
|
| 139 |
+
joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl,sha256=4GTC7s_cWNVShERn2nvVbspZYJgyK_0man4TEqvdVzU,1068
|
| 140 |
+
joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.bz2,sha256=6G1vbs_iYmz2kYJ6w4qB1k7D67UnxUMus0S4SWeBtFo,1000
|
| 141 |
+
joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.gzip,sha256=tlRUWeJS1BXmcwtLNSNK9L0hDHekFl07CqWxTShinmY,831
|
| 142 |
+
joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.lzma,sha256=CorPwnfv3rR5hjNtJI01-sEBMOnkSxNlRVaWTszMopA,694
|
| 143 |
+
joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.xz,sha256=Dppj3MffOKsKETeptEtDaxPOv6MA6xnbpK5LzlDQ-oE,752
|
| 144 |
+
joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl,sha256=HL5Fb1uR9aPLjjhoOPJ2wwM1Qyo1FCZoYYd2HVw0Fos,1068
|
| 145 |
+
joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.bz2,sha256=Pyr2fqZnwfUxXdyrBr-kRwBYY8HA_Yi7fgSguKy5pUs,1021
|
| 146 |
+
joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.gzip,sha256=os8NJjQI9FhnlZM-Ay9dX_Uo35gZnoJCgQSIVvcBPfE,831
|
| 147 |
+
joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.lzma,sha256=Q_0y43qU7_GqAabJ8y3PWVhOisurnCAq3GzuCu04V58,697
|
| 148 |
+
joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.xz,sha256=BNfmiQfpeLVpdfkwlJK4hJ5Cpgl0vreVyekyc5d_PNM,752
|
| 149 |
+
joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl,sha256=l7nvLolhBDIdPFznOz3lBHiMOPBPCMi1bXop1tFSCpY,1068
|
| 150 |
+
joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.bz2,sha256=pqGpuIS-ZU4uP8mkglHs8MaSDiVcPy7l3XHYJSppRgY,1005
|
| 151 |
+
joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.gzip,sha256=YRFXE6LEb6qK72yPqnXdqQVY8Ts8xKUS9PWQKhLxWvk,833
|
| 152 |
+
joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.lzma,sha256=Bf7gCUeTuTjCkbcIdyZYz69irblX4SAVQEzxCnMQhNU,701
|
| 153 |
+
joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.xz,sha256=As8w2LGWwwNmKy3QNdKljK63Yq46gjRf_RJ0lh5_WqA,752
|
| 154 |
+
joblib/test/data/joblib_0.11.0_compressed_pickle_py36_np111.gz,sha256=1WrnXDqDoNEPYOZX1Q5Wr2463b8vVV6fw4Wm5S4bMt4,800
|
| 155 |
+
joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl,sha256=XmsOFxeC1f1aYdGETclG6yfF9rLoB11DayOAhDMULrw,1068
|
| 156 |
+
joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.bz2,sha256=vI2yWb50LKL_NgZyd_XkoD5teIg93uI42mWnx9ee-AQ,991
|
| 157 |
+
joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.gzip,sha256=1WrnXDqDoNEPYOZX1Q5Wr2463b8vVV6fw4Wm5S4bMt4,800
|
| 158 |
+
joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.lzma,sha256=IWA0JlZG2ur53HgTUDl1m7q79dcVq6b0VOq33gKoJU0,715
|
| 159 |
+
joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.xz,sha256=3Xh_NbMZdBjYx7ynfJ3Fyke28izSRSSzzNB0z5D4k9Y,752
|
| 160 |
+
joblib/test/data/joblib_0.8.4_compressed_pickle_py27_np17.gz,sha256=Sp-ZT7i6pj5on2gbptszu7RarzJpOmHJ67UKOmCPQMg,659
|
| 161 |
+
joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np16.gz,sha256=NLtDrvo2XIH0KvUUAvhOqMeoXEjGW0IuTk_osu5XiDw,658
|
| 162 |
+
joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np17.gz,sha256=NLtDrvo2XIH0KvUUAvhOqMeoXEjGW0IuTk_osu5XiDw,658
|
| 163 |
+
joblib/test/data/joblib_0.9.2_compressed_pickle_py34_np19.gz,sha256=nzO9iiGkG3KbBdrF3usOho8higkrDj_lmICUzxZyF_Y,673
|
| 164 |
+
joblib/test/data/joblib_0.9.2_compressed_pickle_py35_np19.gz,sha256=nzO9iiGkG3KbBdrF3usOho8higkrDj_lmICUzxZyF_Y,673
|
| 165 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl,sha256=naijdk2xIeKdIa3mfJw0JlmOdtiN6uRM1yOJg6-M73M,670
|
| 166 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
|
| 167 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
|
| 168 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_03.npy,sha256=oMRa4qKJhBy-uiRDt-uqOzHAqencxzKUrKVynaAJJAU,236
|
| 169 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
|
| 170 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl,sha256=LynX8dLOygfxDfFywOgm7wgWOhSxLG7z-oDsU6X83Dw,670
|
| 171 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
|
| 172 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
|
| 173 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_03.npy,sha256=oMRa4qKJhBy-uiRDt-uqOzHAqencxzKUrKVynaAJJAU,236
|
| 174 |
+
joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
|
| 175 |
+
joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl,sha256=w9TLxpDTzp5TI6cU6lRvMsAasXEChcQgGE9s30sm_CU,691
|
| 176 |
+
joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
|
| 177 |
+
joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
|
| 178 |
+
joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_03.npy,sha256=jt6aZKUrJdfbMJUJVsl47As5MrfRSs1avGMhbmS6vec,307
|
| 179 |
+
joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
|
| 180 |
+
joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl,sha256=ilOBAOaulLFvKrD32S1NfnpiK-LfzA9rC3O2I7xROuI,691
|
| 181 |
+
joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
|
| 182 |
+
joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
|
| 183 |
+
joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_03.npy,sha256=jt6aZKUrJdfbMJUJVsl47As5MrfRSs1avGMhbmS6vec,307
|
| 184 |
+
joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
|
| 185 |
+
joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl,sha256=WfDVIqKcMzzh1gSAshIfzBoIpdLdZQuG79yYf5kfpOo,691
|
| 186 |
+
joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
|
| 187 |
+
joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
|
| 188 |
+
joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_03.npy,sha256=jt6aZKUrJdfbMJUJVsl47As5MrfRSs1avGMhbmS6vec,307
|
| 189 |
+
joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
|
| 190 |
+
joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz,sha256=8jYfWJsx0oY2J-3LlmEigK5cClnJSW2J2rfeSTZw-Ts,802
|
| 191 |
+
joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_01.npy.z,sha256=YT9VvT3sEl2uWlOyvH2CkyE9Sok4od9O3kWtgeuUUqE,43
|
| 192 |
+
joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_02.npy.z,sha256=txA5RDI0PRuiU_UNKY8pGp-zQgQQ9vaVvMi60hOPaVs,43
|
| 193 |
+
joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_03.npy.z,sha256=d3AwICvU2MpSNjh2aPIsdJeGZLlDjANAF1Soa6uM0Po,37
|
| 194 |
+
joblib/test/test_backports.py,sha256=ONt0JUPV1etZCO9DTLur1h84XmgHZYK_k73qmp4kRgg,1175
|
| 195 |
+
joblib/test/test_cloudpickle_wrapper.py,sha256=9jx3hqNVO9GXdVHCxr9mN-GiLR0XK-O5d6YPaaG8Y14,729
|
| 196 |
+
joblib/test/test_config.py,sha256=1Z102AO7Gb8Z8mHYahnZy2fxBA-9_vY0ZtWyNNk1cf4,5255
|
| 197 |
+
joblib/test/test_dask.py,sha256=X2MBEYvz5WQwzGZRN04JNgk_75iIHF96yA1F1t1sK_Y,22932
|
| 198 |
+
joblib/test/test_disk.py,sha256=0EaWGENlosrqwrSZvquPQw3jhqay1KD1NRlQ6YLHOOM,2223
|
| 199 |
+
joblib/test/test_func_inspect.py,sha256=RsORR-j48SfXrNBQbb5i-SdmfU7zk2Mr0IKvcu8m1tw,9314
|
| 200 |
+
joblib/test/test_func_inspect_special_encoding.py,sha256=5xILDjSO-xtjQAMLvMeVD-L7IG4ZURb2gvBiShaDE78,145
|
| 201 |
+
joblib/test/test_hashing.py,sha256=wZeTJMX8C8ua3fJsKAI7MKtperUfZf1fLt0ZaOjvSKw,15820
|
| 202 |
+
joblib/test/test_init.py,sha256=Y6y6Hcqa_cqwQ8S8ozUQ180y_RfkRajfZ_fDp2UXgbw,423
|
| 203 |
+
joblib/test/test_logger.py,sha256=FA9ohTNcqIFViQK60_rwZ5PEGL2zoYN5qBOrDwFqVzI,941
|
| 204 |
+
joblib/test/test_memmapping.py,sha256=z0aanbEs3yCDKShyW3IYlLkTARwdvqVTb4beTPRFmjk,43731
|
| 205 |
+
joblib/test/test_memory.py,sha256=yHoTaoYsWEaDo0CQLNHrBwY5cCpYtIFc3Hi5PArB0iM,51011
|
| 206 |
+
joblib/test/test_memory_async.py,sha256=tUoCI9dngR2AuJjAAKXElJIiz2Qm4AJGdXKn9c8lWaM,5245
|
| 207 |
+
joblib/test/test_missing_multiprocessing.py,sha256=FVoS91krFZogIoDFScyZuJPpaeiq6O-aLAxug0qCQyY,1171
|
| 208 |
+
joblib/test/test_module.py,sha256=IABzz5JmdeY_Adk_vZ0776JN94Ra7tWxDA7DPDNdJKI,1942
|
| 209 |
+
joblib/test/test_numpy_pickle.py,sha256=QExCnBSG-EXdVKnoDkJjNFk6kbX0FDeGeR50wtLHiso,42130
|
| 210 |
+
joblib/test/test_numpy_pickle_compat.py,sha256=paMz1G3Fr9SHdjFmKcG1ec6B5h_S-XE6WRtfHmX9r50,609
|
| 211 |
+
joblib/test/test_numpy_pickle_utils.py,sha256=iB2Ve1TYYUEN3DQiNB5qUxk_QxeIXl7Jpgv4TwkFWTY,382
|
| 212 |
+
joblib/test/test_parallel.py,sha256=_13kli8GYyclwh2QsxysXrRJa44o3gb3FEpSY61ag94,78095
|
| 213 |
+
joblib/test/test_store_backends.py,sha256=DyK1f7PTSPErzhk27gaRoMe2UQrstIz6fnvZh4hKIf0,3057
|
| 214 |
+
joblib/test/test_testing.py,sha256=jL-Ph5pzUJSXOgY2rqbjMRp2y3i3CCWmEi-Lbw4Wzr8,2520
|
| 215 |
+
joblib/test/test_utils.py,sha256=qSD3IJnIGyLiJlUSbEQ0ILVqwG18LW6u50BqTJ3Cn3o,1015
|
| 216 |
+
joblib/test/testutils.py,sha256=A1bm-A5Ydis2iZJVI2-r3aFKUufWR42NZ8Yttrp8mzg,252
|
| 217 |
+
joblib/testing.py,sha256=lK8HOBvrpXcTYUCSvpE-M2ede_dTVJzcmyw-9BrBsOc,3029
|
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/WHEEL
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Wheel-Version: 1.0
|
| 2 |
+
Generator: setuptools (80.9.0)
|
| 3 |
+
Root-Is-Purelib: true
|
| 4 |
+
Tag: py3-none-any
|
| 5 |
+
|
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/top_level.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
joblib
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/__init__.py
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Joblib is a set of tools to provide **lightweight pipelining in
|
| 2 |
+
Python**. In particular:
|
| 3 |
+
|
| 4 |
+
1. transparent disk-caching of functions and lazy re-evaluation
|
| 5 |
+
(memoize pattern)
|
| 6 |
+
|
| 7 |
+
2. easy simple parallel computing
|
| 8 |
+
|
| 9 |
+
Joblib is optimized to be **fast** and **robust** on large
|
| 10 |
+
data in particular and has specific optimizations for `numpy` arrays. It is
|
| 11 |
+
**BSD-licensed**.
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
==================== ===============================================
|
| 15 |
+
**Documentation:** https://joblib.readthedocs.io
|
| 16 |
+
|
| 17 |
+
**Download:** https://pypi.python.org/pypi/joblib#downloads
|
| 18 |
+
|
| 19 |
+
**Source code:** https://github.com/joblib/joblib
|
| 20 |
+
|
| 21 |
+
**Report issues:** https://github.com/joblib/joblib/issues
|
| 22 |
+
==================== ===============================================
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
Vision
|
| 26 |
+
--------
|
| 27 |
+
|
| 28 |
+
The vision is to provide tools to easily achieve better performance and
|
| 29 |
+
reproducibility when working with long running jobs.
|
| 30 |
+
|
| 31 |
+
* **Avoid computing the same thing twice**: code is often rerun again and
|
| 32 |
+
again, for instance when prototyping computational-heavy jobs (as in
|
| 33 |
+
scientific development), but hand-crafted solutions to alleviate this
|
| 34 |
+
issue are error-prone and often lead to unreproducible results.
|
| 35 |
+
|
| 36 |
+
* **Persist to disk transparently**: efficiently persisting
|
| 37 |
+
arbitrary objects containing large data is hard. Using
|
| 38 |
+
joblib's caching mechanism avoids hand-written persistence and
|
| 39 |
+
implicitly links the file on disk to the execution context of
|
| 40 |
+
the original Python object. As a result, joblib's persistence is
|
| 41 |
+
good for resuming an application status or computational job, eg
|
| 42 |
+
after a crash.
|
| 43 |
+
|
| 44 |
+
Joblib addresses these problems while **leaving your code and your flow
|
| 45 |
+
control as unmodified as possible** (no framework, no new paradigms).
|
| 46 |
+
|
| 47 |
+
Main features
|
| 48 |
+
------------------
|
| 49 |
+
|
| 50 |
+
1) **Transparent and fast disk-caching of output value:** a memoize or
|
| 51 |
+
make-like functionality for Python functions that works well for
|
| 52 |
+
arbitrary Python objects, including very large numpy arrays. Separate
|
| 53 |
+
persistence and flow-execution logic from domain logic or algorithmic
|
| 54 |
+
code by writing the operations as a set of steps with well-defined
|
| 55 |
+
inputs and outputs: Python functions. Joblib can save their
|
| 56 |
+
computation to disk and rerun it only if necessary::
|
| 57 |
+
|
| 58 |
+
>>> from joblib import Memory
|
| 59 |
+
>>> location = 'your_cache_dir_goes_here'
|
| 60 |
+
>>> mem = Memory(location, verbose=1)
|
| 61 |
+
>>> import numpy as np
|
| 62 |
+
>>> a = np.vander(np.arange(3)).astype(float)
|
| 63 |
+
>>> square = mem.cache(np.square)
|
| 64 |
+
>>> b = square(a) # doctest: +ELLIPSIS
|
| 65 |
+
______________________________________________________________________...
|
| 66 |
+
[Memory] Calling ...square...
|
| 67 |
+
square(array([[0., 0., 1.],
|
| 68 |
+
[1., 1., 1.],
|
| 69 |
+
[4., 2., 1.]]))
|
| 70 |
+
_________________________________________________...square - ...s, 0.0min
|
| 71 |
+
|
| 72 |
+
>>> c = square(a)
|
| 73 |
+
>>> # The above call did not trigger an evaluation
|
| 74 |
+
|
| 75 |
+
2) **Embarrassingly parallel helper:** to make it easy to write readable
|
| 76 |
+
parallel code and debug it quickly::
|
| 77 |
+
|
| 78 |
+
>>> from joblib import Parallel, delayed
|
| 79 |
+
>>> from math import sqrt
|
| 80 |
+
>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
|
| 81 |
+
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
3) **Fast compressed Persistence**: a replacement for pickle to work
|
| 85 |
+
efficiently on Python objects containing large data (
|
| 86 |
+
*joblib.dump* & *joblib.load* ).
|
| 87 |
+
|
| 88 |
+
..
|
| 89 |
+
>>> import shutil ; shutil.rmtree(location)
|
| 90 |
+
|
| 91 |
+
"""
|
| 92 |
+
|
| 93 |
+
# PEP0440 compatible formatted version, see:
|
| 94 |
+
# https://www.python.org/dev/peps/pep-0440/
|
| 95 |
+
#
|
| 96 |
+
# Generic release markers:
|
| 97 |
+
# X.Y
|
| 98 |
+
# X.Y.Z # For bugfix releases
|
| 99 |
+
#
|
| 100 |
+
# Admissible pre-release markers:
|
| 101 |
+
# X.YaN # Alpha release
|
| 102 |
+
# X.YbN # Beta release
|
| 103 |
+
# X.YrcN # Release Candidate
|
| 104 |
+
# X.Y # Final release
|
| 105 |
+
#
|
| 106 |
+
# Dev branch marker is: 'X.Y.dev' or 'X.Y.devN' where N is an integer.
|
| 107 |
+
# 'X.Y.dev0' is the canonical version of 'X.Y.dev'
|
| 108 |
+
#
|
| 109 |
+
__version__ = "1.5.3"
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
import os
|
| 113 |
+
|
| 114 |
+
from ._cloudpickle_wrapper import wrap_non_picklable_objects
|
| 115 |
+
from ._parallel_backends import ParallelBackendBase
|
| 116 |
+
from ._store_backends import StoreBackendBase
|
| 117 |
+
from .compressor import register_compressor
|
| 118 |
+
from .hashing import hash
|
| 119 |
+
from .logger import Logger, PrintTime
|
| 120 |
+
from .memory import MemorizedResult, Memory, expires_after, register_store_backend
|
| 121 |
+
from .numpy_pickle import dump, load
|
| 122 |
+
from .parallel import (
|
| 123 |
+
Parallel,
|
| 124 |
+
cpu_count,
|
| 125 |
+
delayed,
|
| 126 |
+
effective_n_jobs,
|
| 127 |
+
parallel_backend,
|
| 128 |
+
parallel_config,
|
| 129 |
+
register_parallel_backend,
|
| 130 |
+
)
|
| 131 |
+
|
| 132 |
+
__all__ = [
|
| 133 |
+
# On-disk result caching
|
| 134 |
+
"Memory",
|
| 135 |
+
"MemorizedResult",
|
| 136 |
+
"expires_after",
|
| 137 |
+
# Parallel code execution
|
| 138 |
+
"Parallel",
|
| 139 |
+
"delayed",
|
| 140 |
+
"cpu_count",
|
| 141 |
+
"effective_n_jobs",
|
| 142 |
+
"wrap_non_picklable_objects",
|
| 143 |
+
# Context to change the backend globally
|
| 144 |
+
"parallel_config",
|
| 145 |
+
"parallel_backend",
|
| 146 |
+
# Helpers to define and register store/parallel backends
|
| 147 |
+
"ParallelBackendBase",
|
| 148 |
+
"StoreBackendBase",
|
| 149 |
+
"register_compressor",
|
| 150 |
+
"register_parallel_backend",
|
| 151 |
+
"register_store_backend",
|
| 152 |
+
# Helpers kept for backward compatibility
|
| 153 |
+
"PrintTime",
|
| 154 |
+
"Logger",
|
| 155 |
+
"hash",
|
| 156 |
+
"dump",
|
| 157 |
+
"load",
|
| 158 |
+
]
|
| 159 |
+
|
| 160 |
+
|
| 161 |
+
# Workaround issue discovered in intel-openmp 2019.5:
|
| 162 |
+
# https://github.com/ContinuumIO/anaconda-issues/issues/11294
|
| 163 |
+
os.environ.setdefault("KMP_INIT_AT_FORK", "FALSE")
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_cloudpickle_wrapper.py
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Small shim of loky's cloudpickle_wrapper to avoid failure when
|
| 3 |
+
multiprocessing is not available.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
from ._multiprocessing_helpers import mp
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def _my_wrap_non_picklable_objects(obj, keep_wrapper=True):
|
| 10 |
+
return obj
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
if mp is not None:
|
| 14 |
+
from .externals.loky import wrap_non_picklable_objects
|
| 15 |
+
else:
|
| 16 |
+
wrap_non_picklable_objects = _my_wrap_non_picklable_objects
|
| 17 |
+
|
| 18 |
+
__all__ = ["wrap_non_picklable_objects"]
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_dask.py
ADDED
|
@@ -0,0 +1,381 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import absolute_import, division, print_function
|
| 2 |
+
|
| 3 |
+
import asyncio
|
| 4 |
+
import concurrent.futures
|
| 5 |
+
import contextlib
|
| 6 |
+
import time
|
| 7 |
+
import weakref
|
| 8 |
+
from uuid import uuid4
|
| 9 |
+
|
| 10 |
+
from ._utils import (
|
| 11 |
+
_retrieve_traceback_capturing_wrapped_call,
|
| 12 |
+
_TracebackCapturingWrapper,
|
| 13 |
+
)
|
| 14 |
+
from .parallel import AutoBatchingMixin, ParallelBackendBase, parallel_config
|
| 15 |
+
|
| 16 |
+
try:
|
| 17 |
+
import dask
|
| 18 |
+
import distributed
|
| 19 |
+
except ImportError:
|
| 20 |
+
dask = None
|
| 21 |
+
distributed = None
|
| 22 |
+
|
| 23 |
+
if dask is not None and distributed is not None:
|
| 24 |
+
from dask.distributed import (
|
| 25 |
+
Client,
|
| 26 |
+
as_completed,
|
| 27 |
+
get_client,
|
| 28 |
+
rejoin,
|
| 29 |
+
secede,
|
| 30 |
+
)
|
| 31 |
+
from dask.sizeof import sizeof
|
| 32 |
+
from dask.utils import funcname
|
| 33 |
+
from distributed.utils import thread_state
|
| 34 |
+
|
| 35 |
+
try:
|
| 36 |
+
# asyncio.TimeoutError, Python3-only error thrown by recent versions of
|
| 37 |
+
# distributed
|
| 38 |
+
from distributed.utils import TimeoutError as _TimeoutError
|
| 39 |
+
except ImportError:
|
| 40 |
+
from tornado.gen import TimeoutError as _TimeoutError
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def is_weakrefable(obj):
|
| 44 |
+
try:
|
| 45 |
+
weakref.ref(obj)
|
| 46 |
+
return True
|
| 47 |
+
except TypeError:
|
| 48 |
+
return False
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
class _WeakKeyDictionary:
|
| 52 |
+
"""A variant of weakref.WeakKeyDictionary for unhashable objects.
|
| 53 |
+
|
| 54 |
+
This datastructure is used to store futures for broadcasted data objects
|
| 55 |
+
such as large numpy arrays or pandas dataframes that are not hashable and
|
| 56 |
+
therefore cannot be used as keys of traditional python dicts.
|
| 57 |
+
|
| 58 |
+
Furthermore using a dict with id(array) as key is not safe because the
|
| 59 |
+
Python is likely to reuse id of recently collected arrays.
|
| 60 |
+
"""
|
| 61 |
+
|
| 62 |
+
def __init__(self):
|
| 63 |
+
self._data = {}
|
| 64 |
+
|
| 65 |
+
def __getitem__(self, obj):
|
| 66 |
+
ref, val = self._data[id(obj)]
|
| 67 |
+
if ref() is not obj:
|
| 68 |
+
# In case of a race condition with on_destroy.
|
| 69 |
+
raise KeyError(obj)
|
| 70 |
+
return val
|
| 71 |
+
|
| 72 |
+
def __setitem__(self, obj, value):
|
| 73 |
+
key = id(obj)
|
| 74 |
+
try:
|
| 75 |
+
ref, _ = self._data[key]
|
| 76 |
+
if ref() is not obj:
|
| 77 |
+
# In case of race condition with on_destroy.
|
| 78 |
+
raise KeyError(obj)
|
| 79 |
+
except KeyError:
|
| 80 |
+
# Insert the new entry in the mapping along with a weakref
|
| 81 |
+
# callback to automatically delete the entry from the mapping
|
| 82 |
+
# as soon as the object used as key is garbage collected.
|
| 83 |
+
def on_destroy(_):
|
| 84 |
+
del self._data[key]
|
| 85 |
+
|
| 86 |
+
ref = weakref.ref(obj, on_destroy)
|
| 87 |
+
self._data[key] = ref, value
|
| 88 |
+
|
| 89 |
+
def __len__(self):
|
| 90 |
+
return len(self._data)
|
| 91 |
+
|
| 92 |
+
def clear(self):
|
| 93 |
+
self._data.clear()
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def _funcname(x):
|
| 97 |
+
try:
|
| 98 |
+
if isinstance(x, list):
|
| 99 |
+
x = x[0][0]
|
| 100 |
+
except Exception:
|
| 101 |
+
pass
|
| 102 |
+
return funcname(x)
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def _make_tasks_summary(tasks):
|
| 106 |
+
"""Summarize of list of (func, args, kwargs) function calls"""
|
| 107 |
+
unique_funcs = {func for func, args, kwargs in tasks}
|
| 108 |
+
|
| 109 |
+
if len(unique_funcs) == 1:
|
| 110 |
+
mixed = False
|
| 111 |
+
else:
|
| 112 |
+
mixed = True
|
| 113 |
+
return len(tasks), mixed, _funcname(tasks)
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
class Batch:
|
| 117 |
+
"""dask-compatible wrapper that executes a batch of tasks"""
|
| 118 |
+
|
| 119 |
+
def __init__(self, tasks):
|
| 120 |
+
# collect some metadata from the tasks to ease Batch calls
|
| 121 |
+
# introspection when debugging
|
| 122 |
+
self._num_tasks, self._mixed, self._funcname = _make_tasks_summary(tasks)
|
| 123 |
+
|
| 124 |
+
def __call__(self, tasks=None):
|
| 125 |
+
results = []
|
| 126 |
+
with parallel_config(backend="dask"):
|
| 127 |
+
for func, args, kwargs in tasks:
|
| 128 |
+
results.append(func(*args, **kwargs))
|
| 129 |
+
return results
|
| 130 |
+
|
| 131 |
+
def __repr__(self):
|
| 132 |
+
descr = f"batch_of_{self._funcname}_{self._num_tasks}_calls"
|
| 133 |
+
if self._mixed:
|
| 134 |
+
descr = "mixed_" + descr
|
| 135 |
+
return descr
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def _joblib_probe_task():
|
| 139 |
+
# Noop used by the joblib connector to probe when workers are ready.
|
| 140 |
+
pass
|
| 141 |
+
|
| 142 |
+
|
| 143 |
+
class DaskDistributedBackend(AutoBatchingMixin, ParallelBackendBase):
|
| 144 |
+
MIN_IDEAL_BATCH_DURATION = 0.2
|
| 145 |
+
MAX_IDEAL_BATCH_DURATION = 1.0
|
| 146 |
+
supports_retrieve_callback = True
|
| 147 |
+
default_n_jobs = -1
|
| 148 |
+
|
| 149 |
+
def __init__(
|
| 150 |
+
self,
|
| 151 |
+
scheduler_host=None,
|
| 152 |
+
scatter=None,
|
| 153 |
+
client=None,
|
| 154 |
+
loop=None,
|
| 155 |
+
wait_for_workers_timeout=10,
|
| 156 |
+
**submit_kwargs,
|
| 157 |
+
):
|
| 158 |
+
super().__init__()
|
| 159 |
+
|
| 160 |
+
if distributed is None:
|
| 161 |
+
msg = (
|
| 162 |
+
"You are trying to use 'dask' as a joblib parallel backend "
|
| 163 |
+
"but dask is not installed. Please install dask "
|
| 164 |
+
"to fix this error."
|
| 165 |
+
)
|
| 166 |
+
raise ValueError(msg)
|
| 167 |
+
|
| 168 |
+
if client is None:
|
| 169 |
+
if scheduler_host:
|
| 170 |
+
client = Client(scheduler_host, loop=loop, set_as_default=False)
|
| 171 |
+
else:
|
| 172 |
+
try:
|
| 173 |
+
client = get_client()
|
| 174 |
+
except ValueError as e:
|
| 175 |
+
msg = (
|
| 176 |
+
"To use Joblib with Dask first create a Dask Client"
|
| 177 |
+
"\n\n"
|
| 178 |
+
" from dask.distributed import Client\n"
|
| 179 |
+
" client = Client()\n"
|
| 180 |
+
"or\n"
|
| 181 |
+
" client = Client('scheduler-address:8786')"
|
| 182 |
+
)
|
| 183 |
+
raise ValueError(msg) from e
|
| 184 |
+
|
| 185 |
+
self.client = client
|
| 186 |
+
|
| 187 |
+
if scatter is not None and not isinstance(scatter, (list, tuple)):
|
| 188 |
+
raise TypeError(
|
| 189 |
+
"scatter must be a list/tuple, got `%s`" % type(scatter).__name__
|
| 190 |
+
)
|
| 191 |
+
|
| 192 |
+
if scatter is not None and len(scatter) > 0:
|
| 193 |
+
# Keep a reference to the scattered data to keep the ids the same
|
| 194 |
+
self._scatter = list(scatter)
|
| 195 |
+
scattered = self.client.scatter(scatter, broadcast=True)
|
| 196 |
+
self.data_futures = {id(x): f for x, f in zip(scatter, scattered)}
|
| 197 |
+
else:
|
| 198 |
+
self._scatter = []
|
| 199 |
+
self.data_futures = {}
|
| 200 |
+
self.wait_for_workers_timeout = wait_for_workers_timeout
|
| 201 |
+
self.submit_kwargs = submit_kwargs
|
| 202 |
+
self.waiting_futures = as_completed(
|
| 203 |
+
[], loop=client.loop, with_results=True, raise_errors=False
|
| 204 |
+
)
|
| 205 |
+
self._results = {}
|
| 206 |
+
self._callbacks = {}
|
| 207 |
+
|
| 208 |
+
async def _collect(self):
|
| 209 |
+
while self._continue:
|
| 210 |
+
async for future, result in self.waiting_futures:
|
| 211 |
+
cf_future = self._results.pop(future)
|
| 212 |
+
callback = self._callbacks.pop(future)
|
| 213 |
+
if future.status == "error":
|
| 214 |
+
typ, exc, tb = result
|
| 215 |
+
cf_future.set_exception(exc)
|
| 216 |
+
else:
|
| 217 |
+
cf_future.set_result(result)
|
| 218 |
+
callback(result)
|
| 219 |
+
await asyncio.sleep(0.01)
|
| 220 |
+
|
| 221 |
+
def __reduce__(self):
|
| 222 |
+
return (DaskDistributedBackend, ())
|
| 223 |
+
|
| 224 |
+
def get_nested_backend(self):
|
| 225 |
+
return DaskDistributedBackend(client=self.client), -1
|
| 226 |
+
|
| 227 |
+
def configure(self, n_jobs=1, parallel=None, **backend_args):
|
| 228 |
+
self.parallel = parallel
|
| 229 |
+
return self.effective_n_jobs(n_jobs)
|
| 230 |
+
|
| 231 |
+
def start_call(self):
|
| 232 |
+
self._continue = True
|
| 233 |
+
self.client.loop.add_callback(self._collect)
|
| 234 |
+
self.call_data_futures = _WeakKeyDictionary()
|
| 235 |
+
|
| 236 |
+
def stop_call(self):
|
| 237 |
+
# The explicit call to clear is required to break a cycling reference
|
| 238 |
+
# to the futures.
|
| 239 |
+
self._continue = False
|
| 240 |
+
# wait for the future collection routine (self._backend._collect) to
|
| 241 |
+
# finish in order to limit asyncio warnings due to aborting _collect
|
| 242 |
+
# during a following backend termination call
|
| 243 |
+
time.sleep(0.01)
|
| 244 |
+
self.call_data_futures.clear()
|
| 245 |
+
|
| 246 |
+
def effective_n_jobs(self, n_jobs):
|
| 247 |
+
effective_n_jobs = sum(self.client.ncores().values())
|
| 248 |
+
if effective_n_jobs != 0 or not self.wait_for_workers_timeout:
|
| 249 |
+
return effective_n_jobs
|
| 250 |
+
|
| 251 |
+
# If there is no worker, schedule a probe task to wait for the workers
|
| 252 |
+
# to come up and be available. If the dask cluster is in adaptive mode
|
| 253 |
+
# task might cause the cluster to provision some workers.
|
| 254 |
+
try:
|
| 255 |
+
self.client.submit(_joblib_probe_task).result(
|
| 256 |
+
timeout=self.wait_for_workers_timeout
|
| 257 |
+
)
|
| 258 |
+
except _TimeoutError as e:
|
| 259 |
+
error_msg = (
|
| 260 |
+
"DaskDistributedBackend has no worker after {} seconds. "
|
| 261 |
+
"Make sure that workers are started and can properly connect "
|
| 262 |
+
"to the scheduler and increase the joblib/dask connection "
|
| 263 |
+
"timeout with:\n\n"
|
| 264 |
+
"parallel_config(backend='dask', wait_for_workers_timeout={})"
|
| 265 |
+
).format(
|
| 266 |
+
self.wait_for_workers_timeout,
|
| 267 |
+
max(10, 2 * self.wait_for_workers_timeout),
|
| 268 |
+
)
|
| 269 |
+
raise TimeoutError(error_msg) from e
|
| 270 |
+
return sum(self.client.ncores().values())
|
| 271 |
+
|
| 272 |
+
async def _to_func_args(self, func):
|
| 273 |
+
itemgetters = dict()
|
| 274 |
+
|
| 275 |
+
# Futures that are dynamically generated during a single call to
|
| 276 |
+
# Parallel.__call__.
|
| 277 |
+
call_data_futures = getattr(self, "call_data_futures", None)
|
| 278 |
+
|
| 279 |
+
async def maybe_to_futures(args):
|
| 280 |
+
out = []
|
| 281 |
+
for arg in args:
|
| 282 |
+
arg_id = id(arg)
|
| 283 |
+
if arg_id in itemgetters:
|
| 284 |
+
out.append(itemgetters[arg_id])
|
| 285 |
+
continue
|
| 286 |
+
|
| 287 |
+
f = self.data_futures.get(arg_id, None)
|
| 288 |
+
if f is None and call_data_futures is not None:
|
| 289 |
+
try:
|
| 290 |
+
f = await call_data_futures[arg]
|
| 291 |
+
except KeyError:
|
| 292 |
+
pass
|
| 293 |
+
if f is None:
|
| 294 |
+
if is_weakrefable(arg) and sizeof(arg) > 1e3:
|
| 295 |
+
# Automatically scatter large objects to some of
|
| 296 |
+
# the workers to avoid duplicated data transfers.
|
| 297 |
+
# Rely on automated inter-worker data stealing if
|
| 298 |
+
# more workers need to reuse this data
|
| 299 |
+
# concurrently.
|
| 300 |
+
# set hash=False - nested scatter calls (i.e
|
| 301 |
+
# calling client.scatter inside a dask worker)
|
| 302 |
+
# using hash=True often raise CancelledError,
|
| 303 |
+
# see dask/distributed#3703
|
| 304 |
+
_coro = self.client.scatter(
|
| 305 |
+
arg, asynchronous=True, hash=False
|
| 306 |
+
)
|
| 307 |
+
# Centralize the scattering of identical arguments
|
| 308 |
+
# between concurrent apply_async callbacks by
|
| 309 |
+
# exposing the running coroutine in
|
| 310 |
+
# call_data_futures before it completes.
|
| 311 |
+
t = asyncio.Task(_coro)
|
| 312 |
+
call_data_futures[arg] = t
|
| 313 |
+
|
| 314 |
+
f = await t
|
| 315 |
+
|
| 316 |
+
if f is not None:
|
| 317 |
+
out.append(f)
|
| 318 |
+
else:
|
| 319 |
+
out.append(arg)
|
| 320 |
+
return out
|
| 321 |
+
|
| 322 |
+
tasks = []
|
| 323 |
+
for f, args, kwargs in func.items:
|
| 324 |
+
args = list(await maybe_to_futures(args))
|
| 325 |
+
kwargs = dict(zip(kwargs.keys(), await maybe_to_futures(kwargs.values())))
|
| 326 |
+
tasks.append((f, args, kwargs))
|
| 327 |
+
|
| 328 |
+
return (Batch(tasks), tasks)
|
| 329 |
+
|
| 330 |
+
def apply_async(self, func, callback=None):
|
| 331 |
+
cf_future = concurrent.futures.Future()
|
| 332 |
+
cf_future.get = cf_future.result # achieve AsyncResult API
|
| 333 |
+
|
| 334 |
+
async def f(func, callback):
|
| 335 |
+
batch, tasks = await self._to_func_args(func)
|
| 336 |
+
key = f"{repr(batch)}-{uuid4().hex}"
|
| 337 |
+
|
| 338 |
+
dask_future = self.client.submit(
|
| 339 |
+
_TracebackCapturingWrapper(batch),
|
| 340 |
+
tasks=tasks,
|
| 341 |
+
key=key,
|
| 342 |
+
**self.submit_kwargs,
|
| 343 |
+
)
|
| 344 |
+
self.waiting_futures.add(dask_future)
|
| 345 |
+
self._callbacks[dask_future] = callback
|
| 346 |
+
self._results[dask_future] = cf_future
|
| 347 |
+
|
| 348 |
+
self.client.loop.add_callback(f, func, callback)
|
| 349 |
+
|
| 350 |
+
return cf_future
|
| 351 |
+
|
| 352 |
+
def retrieve_result_callback(self, out):
|
| 353 |
+
return _retrieve_traceback_capturing_wrapped_call(out)
|
| 354 |
+
|
| 355 |
+
def abort_everything(self, ensure_ready=True):
|
| 356 |
+
"""Tell the client to cancel any task submitted via this instance
|
| 357 |
+
|
| 358 |
+
joblib.Parallel will never access those results
|
| 359 |
+
"""
|
| 360 |
+
with self.waiting_futures.lock:
|
| 361 |
+
self.waiting_futures.futures.clear()
|
| 362 |
+
while not self.waiting_futures.queue.empty():
|
| 363 |
+
self.waiting_futures.queue.get()
|
| 364 |
+
|
| 365 |
+
@contextlib.contextmanager
|
| 366 |
+
def retrieval_context(self):
|
| 367 |
+
"""Override ParallelBackendBase.retrieval_context to avoid deadlocks.
|
| 368 |
+
|
| 369 |
+
This removes thread from the worker's thread pool (using 'secede').
|
| 370 |
+
Seceding avoids deadlock in nested parallelism settings.
|
| 371 |
+
"""
|
| 372 |
+
# See 'joblib.Parallel.__call__' and 'joblib.Parallel.retrieve' for how
|
| 373 |
+
# this is used.
|
| 374 |
+
if hasattr(thread_state, "execution_state"):
|
| 375 |
+
# we are in a worker. Secede to avoid deadlock.
|
| 376 |
+
secede()
|
| 377 |
+
|
| 378 |
+
yield
|
| 379 |
+
|
| 380 |
+
if hasattr(thread_state, "execution_state"):
|
| 381 |
+
rejoin()
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_memmapping_reducer.py
ADDED
|
@@ -0,0 +1,715 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Reducer using memory mapping for numpy arrays
|
| 3 |
+
"""
|
| 4 |
+
# Author: Thomas Moreau <thomas.moreau.2010@gmail.com>
|
| 5 |
+
# Copyright: 2017, Thomas Moreau
|
| 6 |
+
# License: BSD 3 clause
|
| 7 |
+
|
| 8 |
+
import atexit
|
| 9 |
+
import errno
|
| 10 |
+
import os
|
| 11 |
+
import stat
|
| 12 |
+
import tempfile
|
| 13 |
+
import threading
|
| 14 |
+
import time
|
| 15 |
+
import warnings
|
| 16 |
+
import weakref
|
| 17 |
+
from mmap import mmap
|
| 18 |
+
from multiprocessing import util
|
| 19 |
+
from pickle import HIGHEST_PROTOCOL, PicklingError, dumps, loads, whichmodule
|
| 20 |
+
from uuid import uuid4
|
| 21 |
+
|
| 22 |
+
try:
|
| 23 |
+
WindowsError
|
| 24 |
+
except NameError:
|
| 25 |
+
WindowsError = type(None)
|
| 26 |
+
|
| 27 |
+
try:
|
| 28 |
+
import numpy as np
|
| 29 |
+
from numpy.lib.stride_tricks import as_strided
|
| 30 |
+
except ImportError:
|
| 31 |
+
np = None
|
| 32 |
+
|
| 33 |
+
from .backports import make_memmap
|
| 34 |
+
from .disk import delete_folder
|
| 35 |
+
from .externals.loky.backend import resource_tracker
|
| 36 |
+
from .numpy_pickle import dump, load, load_temporary_memmap
|
| 37 |
+
|
| 38 |
+
# Some system have a ramdisk mounted by default, we can use it instead of /tmp
|
| 39 |
+
# as the default folder to dump big arrays to share with subprocesses.
|
| 40 |
+
SYSTEM_SHARED_MEM_FS = "/dev/shm"
|
| 41 |
+
|
| 42 |
+
# Minimal number of bytes available on SYSTEM_SHARED_MEM_FS to consider using
|
| 43 |
+
# it as the default folder to dump big arrays to share with subprocesses.
|
| 44 |
+
SYSTEM_SHARED_MEM_FS_MIN_SIZE = int(2e9)
|
| 45 |
+
|
| 46 |
+
# Folder and file permissions to chmod temporary files generated by the
|
| 47 |
+
# memmapping pool. Only the owner of the Python process can access the
|
| 48 |
+
# temporary files and folder.
|
| 49 |
+
FOLDER_PERMISSIONS = stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
|
| 50 |
+
FILE_PERMISSIONS = stat.S_IRUSR | stat.S_IWUSR
|
| 51 |
+
|
| 52 |
+
# Set used in joblib workers, referencing the filenames of temporary memmaps
|
| 53 |
+
# created by joblib to speed up data communication. In child processes, we add
|
| 54 |
+
# a finalizer to these memmaps that sends a maybe_unlink call to the
|
| 55 |
+
# resource_tracker, in order to free main memory as fast as possible.
|
| 56 |
+
JOBLIB_MMAPS = set()
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def _log_and_unlink(filename):
|
| 60 |
+
from .externals.loky.backend.resource_tracker import _resource_tracker
|
| 61 |
+
|
| 62 |
+
util.debug(
|
| 63 |
+
"[FINALIZER CALL] object mapping to {} about to be deleted,"
|
| 64 |
+
" decrementing the refcount of the file (pid: {})".format(
|
| 65 |
+
os.path.basename(filename), os.getpid()
|
| 66 |
+
)
|
| 67 |
+
)
|
| 68 |
+
_resource_tracker.maybe_unlink(filename, "file")
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def add_maybe_unlink_finalizer(memmap):
|
| 72 |
+
util.debug(
|
| 73 |
+
"[FINALIZER ADD] adding finalizer to {} (id {}, filename {}, pid {})".format(
|
| 74 |
+
type(memmap), id(memmap), os.path.basename(memmap.filename), os.getpid()
|
| 75 |
+
)
|
| 76 |
+
)
|
| 77 |
+
weakref.finalize(memmap, _log_and_unlink, memmap.filename)
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def unlink_file(filename):
|
| 81 |
+
"""Wrapper around os.unlink with a retry mechanism.
|
| 82 |
+
|
| 83 |
+
The retry mechanism has been implemented primarily to overcome a race
|
| 84 |
+
condition happening during the finalizer of a np.memmap: when a process
|
| 85 |
+
holding the last reference to a mmap-backed np.memmap/np.array is about to
|
| 86 |
+
delete this array (and close the reference), it sends a maybe_unlink
|
| 87 |
+
request to the resource_tracker. This request can be processed faster than
|
| 88 |
+
it takes for the last reference of the memmap to be closed, yielding (on
|
| 89 |
+
Windows) a PermissionError in the resource_tracker loop.
|
| 90 |
+
"""
|
| 91 |
+
NUM_RETRIES = 10
|
| 92 |
+
for retry_no in range(1, NUM_RETRIES + 1):
|
| 93 |
+
try:
|
| 94 |
+
os.unlink(filename)
|
| 95 |
+
break
|
| 96 |
+
except PermissionError:
|
| 97 |
+
util.debug(
|
| 98 |
+
"[ResourceTracker] tried to unlink {}, got PermissionError".format(
|
| 99 |
+
filename
|
| 100 |
+
)
|
| 101 |
+
)
|
| 102 |
+
if retry_no == NUM_RETRIES:
|
| 103 |
+
raise
|
| 104 |
+
else:
|
| 105 |
+
time.sleep(0.2)
|
| 106 |
+
except FileNotFoundError:
|
| 107 |
+
# In case of a race condition when deleting the temporary folder,
|
| 108 |
+
# avoid noisy FileNotFoundError exception in the resource tracker.
|
| 109 |
+
pass
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
resource_tracker._CLEANUP_FUNCS["file"] = unlink_file
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
class _WeakArrayKeyMap:
|
| 116 |
+
"""A variant of weakref.WeakKeyDictionary for unhashable numpy arrays.
|
| 117 |
+
|
| 118 |
+
This datastructure will be used with numpy arrays as obj keys, therefore we
|
| 119 |
+
do not use the __get__ / __set__ methods to avoid any conflict with the
|
| 120 |
+
numpy fancy indexing syntax.
|
| 121 |
+
"""
|
| 122 |
+
|
| 123 |
+
def __init__(self):
|
| 124 |
+
self._data = {}
|
| 125 |
+
|
| 126 |
+
def get(self, obj):
|
| 127 |
+
ref, val = self._data[id(obj)]
|
| 128 |
+
if ref() is not obj:
|
| 129 |
+
# In case of race condition with on_destroy: could never be
|
| 130 |
+
# triggered by the joblib tests with CPython.
|
| 131 |
+
raise KeyError(obj)
|
| 132 |
+
return val
|
| 133 |
+
|
| 134 |
+
def set(self, obj, value):
|
| 135 |
+
key = id(obj)
|
| 136 |
+
try:
|
| 137 |
+
ref, _ = self._data[key]
|
| 138 |
+
if ref() is not obj:
|
| 139 |
+
# In case of race condition with on_destroy: could never be
|
| 140 |
+
# triggered by the joblib tests with CPython.
|
| 141 |
+
raise KeyError(obj)
|
| 142 |
+
except KeyError:
|
| 143 |
+
# Insert the new entry in the mapping along with a weakref
|
| 144 |
+
# callback to automatically delete the entry from the mapping
|
| 145 |
+
# as soon as the object used as key is garbage collected.
|
| 146 |
+
def on_destroy(_):
|
| 147 |
+
del self._data[key]
|
| 148 |
+
|
| 149 |
+
ref = weakref.ref(obj, on_destroy)
|
| 150 |
+
self._data[key] = ref, value
|
| 151 |
+
|
| 152 |
+
def __getstate__(self):
|
| 153 |
+
raise PicklingError("_WeakArrayKeyMap is not pickleable")
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
###############################################################################
|
| 157 |
+
# Support for efficient transient pickling of numpy data structures
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def _get_backing_memmap(a):
|
| 161 |
+
"""Recursively look up the original np.memmap instance base if any."""
|
| 162 |
+
b = getattr(a, "base", None)
|
| 163 |
+
if b is None:
|
| 164 |
+
# TODO: check scipy sparse datastructure if scipy is installed
|
| 165 |
+
# a nor its descendants do not have a memmap base
|
| 166 |
+
return None
|
| 167 |
+
|
| 168 |
+
elif isinstance(b, mmap):
|
| 169 |
+
# a is already a real memmap instance.
|
| 170 |
+
return a
|
| 171 |
+
|
| 172 |
+
else:
|
| 173 |
+
# Recursive exploration of the base ancestry
|
| 174 |
+
return _get_backing_memmap(b)
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def _get_temp_dir(pool_folder_name, temp_folder=None):
|
| 178 |
+
"""Get the full path to a subfolder inside the temporary folder.
|
| 179 |
+
|
| 180 |
+
Parameters
|
| 181 |
+
----------
|
| 182 |
+
pool_folder_name : str
|
| 183 |
+
Sub-folder name used for the serialization of a pool instance.
|
| 184 |
+
|
| 185 |
+
temp_folder: str, optional
|
| 186 |
+
Folder to be used by the pool for memmapping large arrays
|
| 187 |
+
for sharing memory with worker processes. If None, this will try in
|
| 188 |
+
order:
|
| 189 |
+
|
| 190 |
+
- a folder pointed by the JOBLIB_TEMP_FOLDER environment
|
| 191 |
+
variable,
|
| 192 |
+
- /dev/shm if the folder exists and is writable: this is a
|
| 193 |
+
RAMdisk filesystem available by default on modern Linux
|
| 194 |
+
distributions,
|
| 195 |
+
- the default system temporary folder that can be
|
| 196 |
+
overridden with TMP, TMPDIR or TEMP environment
|
| 197 |
+
variables, typically /tmp under Unix operating systems.
|
| 198 |
+
|
| 199 |
+
Returns
|
| 200 |
+
-------
|
| 201 |
+
pool_folder : str
|
| 202 |
+
full path to the temporary folder
|
| 203 |
+
use_shared_mem : bool
|
| 204 |
+
whether the temporary folder is written to the system shared memory
|
| 205 |
+
folder or some other temporary folder.
|
| 206 |
+
"""
|
| 207 |
+
use_shared_mem = False
|
| 208 |
+
if temp_folder is None:
|
| 209 |
+
temp_folder = os.environ.get("JOBLIB_TEMP_FOLDER", None)
|
| 210 |
+
if temp_folder is None:
|
| 211 |
+
if os.path.exists(SYSTEM_SHARED_MEM_FS) and hasattr(os, "statvfs"):
|
| 212 |
+
try:
|
| 213 |
+
shm_stats = os.statvfs(SYSTEM_SHARED_MEM_FS)
|
| 214 |
+
available_nbytes = shm_stats.f_bsize * shm_stats.f_bavail
|
| 215 |
+
if available_nbytes > SYSTEM_SHARED_MEM_FS_MIN_SIZE:
|
| 216 |
+
# Try to see if we have write access to the shared mem
|
| 217 |
+
# folder only if it is reasonably large (that is 2GB or
|
| 218 |
+
# more).
|
| 219 |
+
temp_folder = SYSTEM_SHARED_MEM_FS
|
| 220 |
+
pool_folder = os.path.join(temp_folder, pool_folder_name)
|
| 221 |
+
if not os.path.exists(pool_folder):
|
| 222 |
+
os.makedirs(pool_folder)
|
| 223 |
+
use_shared_mem = True
|
| 224 |
+
except (IOError, OSError):
|
| 225 |
+
# Missing rights in the /dev/shm partition, fallback to regular
|
| 226 |
+
# temp folder.
|
| 227 |
+
temp_folder = None
|
| 228 |
+
if temp_folder is None:
|
| 229 |
+
# Fallback to the default tmp folder, typically /tmp
|
| 230 |
+
temp_folder = tempfile.gettempdir()
|
| 231 |
+
temp_folder = os.path.abspath(os.path.expanduser(temp_folder))
|
| 232 |
+
pool_folder = os.path.join(temp_folder, pool_folder_name)
|
| 233 |
+
return pool_folder, use_shared_mem
|
| 234 |
+
|
| 235 |
+
|
| 236 |
+
def has_shareable_memory(a):
|
| 237 |
+
"""Return True if a is backed by some mmap buffer directly or not."""
|
| 238 |
+
return _get_backing_memmap(a) is not None
|
| 239 |
+
|
| 240 |
+
|
| 241 |
+
def _strided_from_memmap(
|
| 242 |
+
filename,
|
| 243 |
+
dtype,
|
| 244 |
+
mode,
|
| 245 |
+
offset,
|
| 246 |
+
order,
|
| 247 |
+
shape,
|
| 248 |
+
strides,
|
| 249 |
+
total_buffer_len,
|
| 250 |
+
unlink_on_gc_collect,
|
| 251 |
+
):
|
| 252 |
+
"""Reconstruct an array view on a memory mapped file."""
|
| 253 |
+
if mode == "w+":
|
| 254 |
+
# Do not zero the original data when unpickling
|
| 255 |
+
mode = "r+"
|
| 256 |
+
|
| 257 |
+
if strides is None:
|
| 258 |
+
# Simple, contiguous memmap
|
| 259 |
+
return make_memmap(
|
| 260 |
+
filename,
|
| 261 |
+
dtype=dtype,
|
| 262 |
+
shape=shape,
|
| 263 |
+
mode=mode,
|
| 264 |
+
offset=offset,
|
| 265 |
+
order=order,
|
| 266 |
+
unlink_on_gc_collect=unlink_on_gc_collect,
|
| 267 |
+
)
|
| 268 |
+
else:
|
| 269 |
+
# For non-contiguous data, memmap the total enclosing buffer and then
|
| 270 |
+
# extract the non-contiguous view with the stride-tricks API
|
| 271 |
+
base = make_memmap(
|
| 272 |
+
filename,
|
| 273 |
+
dtype=dtype,
|
| 274 |
+
shape=total_buffer_len,
|
| 275 |
+
offset=offset,
|
| 276 |
+
mode=mode,
|
| 277 |
+
order=order,
|
| 278 |
+
unlink_on_gc_collect=unlink_on_gc_collect,
|
| 279 |
+
)
|
| 280 |
+
return as_strided(base, shape=shape, strides=strides)
|
| 281 |
+
|
| 282 |
+
|
| 283 |
+
def _reduce_memmap_backed(a, m):
|
| 284 |
+
"""Pickling reduction for memmap backed arrays.
|
| 285 |
+
|
| 286 |
+
a is expected to be an instance of np.ndarray (or np.memmap)
|
| 287 |
+
m is expected to be an instance of np.memmap on the top of the ``base``
|
| 288 |
+
attribute ancestry of a. ``m.base`` should be the real python mmap object.
|
| 289 |
+
"""
|
| 290 |
+
# offset that comes from the striding differences between a and m
|
| 291 |
+
util.debug(
|
| 292 |
+
"[MEMMAP REDUCE] reducing a memmap-backed array (shape, {}, pid: {})".format(
|
| 293 |
+
a.shape, os.getpid()
|
| 294 |
+
)
|
| 295 |
+
)
|
| 296 |
+
try:
|
| 297 |
+
from numpy.lib.array_utils import byte_bounds
|
| 298 |
+
except (ModuleNotFoundError, ImportError):
|
| 299 |
+
# Backward-compat for numpy < 2.0
|
| 300 |
+
from numpy import byte_bounds
|
| 301 |
+
a_start, a_end = byte_bounds(a)
|
| 302 |
+
m_start = byte_bounds(m)[0]
|
| 303 |
+
offset = a_start - m_start
|
| 304 |
+
|
| 305 |
+
# offset from the backing memmap
|
| 306 |
+
offset += m.offset
|
| 307 |
+
|
| 308 |
+
# 1D arrays are both F and C contiguous, so only set the flag in
|
| 309 |
+
# higher dimensions. See https://github.com/joblib/joblib/pull/1704.
|
| 310 |
+
if m.ndim > 1 and m.flags["F_CONTIGUOUS"]:
|
| 311 |
+
order = "F"
|
| 312 |
+
else:
|
| 313 |
+
# The backing memmap buffer is necessarily contiguous hence C if not
|
| 314 |
+
# Fortran
|
| 315 |
+
order = "C"
|
| 316 |
+
|
| 317 |
+
if a.flags["F_CONTIGUOUS"] or a.flags["C_CONTIGUOUS"]:
|
| 318 |
+
# If the array is a contiguous view, no need to pass the strides
|
| 319 |
+
strides = None
|
| 320 |
+
total_buffer_len = None
|
| 321 |
+
else:
|
| 322 |
+
# Compute the total number of items to map from which the strided
|
| 323 |
+
# view will be extracted.
|
| 324 |
+
strides = a.strides
|
| 325 |
+
total_buffer_len = (a_end - a_start) // a.itemsize
|
| 326 |
+
|
| 327 |
+
return (
|
| 328 |
+
_strided_from_memmap,
|
| 329 |
+
(
|
| 330 |
+
m.filename,
|
| 331 |
+
a.dtype,
|
| 332 |
+
m.mode,
|
| 333 |
+
offset,
|
| 334 |
+
order,
|
| 335 |
+
a.shape,
|
| 336 |
+
strides,
|
| 337 |
+
total_buffer_len,
|
| 338 |
+
False,
|
| 339 |
+
),
|
| 340 |
+
)
|
| 341 |
+
|
| 342 |
+
|
| 343 |
+
def reduce_array_memmap_backward(a):
|
| 344 |
+
"""reduce a np.array or a np.memmap from a child process"""
|
| 345 |
+
m = _get_backing_memmap(a)
|
| 346 |
+
if isinstance(m, np.memmap) and m.filename not in JOBLIB_MMAPS:
|
| 347 |
+
# if a is backed by a memmaped file, reconstruct a using the
|
| 348 |
+
# memmaped file.
|
| 349 |
+
return _reduce_memmap_backed(a, m)
|
| 350 |
+
else:
|
| 351 |
+
# a is either a regular (not memmap-backed) numpy array, or an array
|
| 352 |
+
# backed by a shared temporary file created by joblib. In the latter
|
| 353 |
+
# case, in order to limit the lifespan of these temporary files, we
|
| 354 |
+
# serialize the memmap as a regular numpy array, and decref the
|
| 355 |
+
# file backing the memmap (done implicitly in a previously registered
|
| 356 |
+
# finalizer, see ``unlink_on_gc_collect`` for more details)
|
| 357 |
+
return (loads, (dumps(np.asarray(a), protocol=HIGHEST_PROTOCOL),))
|
| 358 |
+
|
| 359 |
+
|
| 360 |
+
class ArrayMemmapForwardReducer(object):
|
| 361 |
+
"""Reducer callable to dump large arrays to memmap files.
|
| 362 |
+
|
| 363 |
+
Parameters
|
| 364 |
+
----------
|
| 365 |
+
max_nbytes: int
|
| 366 |
+
Threshold to trigger memmapping of large arrays to files created
|
| 367 |
+
a folder.
|
| 368 |
+
temp_folder_resolver: callable
|
| 369 |
+
An callable in charge of resolving a temporary folder name where files
|
| 370 |
+
for backing memmapped arrays are created.
|
| 371 |
+
mmap_mode: 'r', 'r+' or 'c'
|
| 372 |
+
Mode for the created memmap datastructure. See the documentation of
|
| 373 |
+
numpy.memmap for more details. Note: 'w+' is coerced to 'r+'
|
| 374 |
+
automatically to avoid zeroing the data on unpickling.
|
| 375 |
+
verbose: int, optional, 0 by default
|
| 376 |
+
If verbose > 0, memmap creations are logged.
|
| 377 |
+
If verbose > 1, both memmap creations, reuse and array pickling are
|
| 378 |
+
logged.
|
| 379 |
+
prewarm: bool, optional, False by default.
|
| 380 |
+
Force a read on newly memmapped array to make sure that OS pre-cache it
|
| 381 |
+
memory. This can be useful to avoid concurrent disk access when the
|
| 382 |
+
same data array is passed to different worker processes.
|
| 383 |
+
"""
|
| 384 |
+
|
| 385 |
+
def __init__(
|
| 386 |
+
self,
|
| 387 |
+
max_nbytes,
|
| 388 |
+
temp_folder_resolver,
|
| 389 |
+
mmap_mode,
|
| 390 |
+
unlink_on_gc_collect,
|
| 391 |
+
verbose=0,
|
| 392 |
+
prewarm=True,
|
| 393 |
+
):
|
| 394 |
+
self._max_nbytes = max_nbytes
|
| 395 |
+
self._temp_folder_resolver = temp_folder_resolver
|
| 396 |
+
self._mmap_mode = mmap_mode
|
| 397 |
+
self.verbose = int(verbose)
|
| 398 |
+
if prewarm == "auto":
|
| 399 |
+
self._prewarm = not self._temp_folder.startswith(SYSTEM_SHARED_MEM_FS)
|
| 400 |
+
else:
|
| 401 |
+
self._prewarm = prewarm
|
| 402 |
+
self._prewarm = prewarm
|
| 403 |
+
self._memmaped_arrays = _WeakArrayKeyMap()
|
| 404 |
+
self._temporary_memmaped_filenames = set()
|
| 405 |
+
self._unlink_on_gc_collect = unlink_on_gc_collect
|
| 406 |
+
|
| 407 |
+
@property
|
| 408 |
+
def _temp_folder(self):
|
| 409 |
+
return self._temp_folder_resolver()
|
| 410 |
+
|
| 411 |
+
def __reduce__(self):
|
| 412 |
+
# The ArrayMemmapForwardReducer is passed to the children processes: it
|
| 413 |
+
# needs to be pickled but the _WeakArrayKeyMap need to be skipped as
|
| 414 |
+
# it's only guaranteed to be consistent with the parent process memory
|
| 415 |
+
# garbage collection.
|
| 416 |
+
# Although this reducer is pickled, it is not needed in its destination
|
| 417 |
+
# process (child processes), as we only use this reducer to send
|
| 418 |
+
# memmaps from the parent process to the children processes. For this
|
| 419 |
+
# reason, we can afford skipping the resolver, (which would otherwise
|
| 420 |
+
# be unpicklable), and pass it as None instead.
|
| 421 |
+
args = (self._max_nbytes, None, self._mmap_mode, self._unlink_on_gc_collect)
|
| 422 |
+
kwargs = {
|
| 423 |
+
"verbose": self.verbose,
|
| 424 |
+
"prewarm": self._prewarm,
|
| 425 |
+
}
|
| 426 |
+
return ArrayMemmapForwardReducer, args, kwargs
|
| 427 |
+
|
| 428 |
+
def __call__(self, a):
|
| 429 |
+
m = _get_backing_memmap(a)
|
| 430 |
+
if m is not None and isinstance(m, np.memmap):
|
| 431 |
+
# a is already backed by a memmap file, let's reuse it directly
|
| 432 |
+
return _reduce_memmap_backed(a, m)
|
| 433 |
+
|
| 434 |
+
if (
|
| 435 |
+
not a.dtype.hasobject
|
| 436 |
+
and self._max_nbytes is not None
|
| 437 |
+
and a.nbytes > self._max_nbytes
|
| 438 |
+
):
|
| 439 |
+
# check that the folder exists (lazily create the pool temp folder
|
| 440 |
+
# if required)
|
| 441 |
+
try:
|
| 442 |
+
os.makedirs(self._temp_folder)
|
| 443 |
+
os.chmod(self._temp_folder, FOLDER_PERMISSIONS)
|
| 444 |
+
except OSError as e:
|
| 445 |
+
if e.errno != errno.EEXIST:
|
| 446 |
+
raise e
|
| 447 |
+
|
| 448 |
+
try:
|
| 449 |
+
basename = self._memmaped_arrays.get(a)
|
| 450 |
+
except KeyError:
|
| 451 |
+
# Generate a new unique random filename. The process and thread
|
| 452 |
+
# ids are only useful for debugging purpose and to make it
|
| 453 |
+
# easier to cleanup orphaned files in case of hard process
|
| 454 |
+
# kill (e.g. by "kill -9" or segfault).
|
| 455 |
+
basename = "{}-{}-{}.pkl".format(
|
| 456 |
+
os.getpid(), id(threading.current_thread()), uuid4().hex
|
| 457 |
+
)
|
| 458 |
+
self._memmaped_arrays.set(a, basename)
|
| 459 |
+
filename = os.path.join(self._temp_folder, basename)
|
| 460 |
+
|
| 461 |
+
# In case the same array with the same content is passed several
|
| 462 |
+
# times to the pool subprocess children, serialize it only once
|
| 463 |
+
|
| 464 |
+
is_new_memmap = filename not in self._temporary_memmaped_filenames
|
| 465 |
+
|
| 466 |
+
# add the memmap to the list of temporary memmaps created by joblib
|
| 467 |
+
self._temporary_memmaped_filenames.add(filename)
|
| 468 |
+
|
| 469 |
+
if self._unlink_on_gc_collect:
|
| 470 |
+
# Bump reference count of the memmap by 1 to account for
|
| 471 |
+
# shared usage of the memmap by a child process. The
|
| 472 |
+
# corresponding decref call will be executed upon calling
|
| 473 |
+
# resource_tracker.maybe_unlink, registered as a finalizer in
|
| 474 |
+
# the child.
|
| 475 |
+
# the incref/decref calls here are only possible when the child
|
| 476 |
+
# and the parent share the same resource_tracker. It is not the
|
| 477 |
+
# case for the multiprocessing backend, but it does not matter
|
| 478 |
+
# because unlinking a memmap from a child process is only
|
| 479 |
+
# useful to control the memory usage of long-lasting child
|
| 480 |
+
# processes, while the multiprocessing-based pools terminate
|
| 481 |
+
# their workers at the end of a map() call.
|
| 482 |
+
resource_tracker.register(filename, "file")
|
| 483 |
+
|
| 484 |
+
if is_new_memmap:
|
| 485 |
+
# Incref each temporary memmap created by joblib one extra
|
| 486 |
+
# time. This means that these memmaps will only be deleted
|
| 487 |
+
# once an extra maybe_unlink() is called, which is done once
|
| 488 |
+
# all the jobs have completed (or been canceled) in the
|
| 489 |
+
# Parallel._terminate_backend() method.
|
| 490 |
+
resource_tracker.register(filename, "file")
|
| 491 |
+
|
| 492 |
+
if not os.path.exists(filename):
|
| 493 |
+
util.debug(
|
| 494 |
+
"[ARRAY DUMP] Pickling new array (shape={}, dtype={}) "
|
| 495 |
+
"creating a new memmap at {}".format(a.shape, a.dtype, filename)
|
| 496 |
+
)
|
| 497 |
+
for dumped_filename in dump(a, filename):
|
| 498 |
+
os.chmod(dumped_filename, FILE_PERMISSIONS)
|
| 499 |
+
|
| 500 |
+
if self._prewarm:
|
| 501 |
+
# Warm up the data by accessing it. This operation ensures
|
| 502 |
+
# that the disk access required to create the memmapping
|
| 503 |
+
# file are performed in the reducing process and avoids
|
| 504 |
+
# concurrent memmap creation in multiple children
|
| 505 |
+
# processes.
|
| 506 |
+
load(filename, mmap_mode=self._mmap_mode).max()
|
| 507 |
+
|
| 508 |
+
else:
|
| 509 |
+
util.debug(
|
| 510 |
+
"[ARRAY DUMP] Pickling known array (shape={}, dtype={}) "
|
| 511 |
+
"reusing memmap file: {}".format(
|
| 512 |
+
a.shape, a.dtype, os.path.basename(filename)
|
| 513 |
+
)
|
| 514 |
+
)
|
| 515 |
+
|
| 516 |
+
# The worker process will use joblib.load to memmap the data
|
| 517 |
+
return (
|
| 518 |
+
load_temporary_memmap,
|
| 519 |
+
(filename, self._mmap_mode, self._unlink_on_gc_collect),
|
| 520 |
+
)
|
| 521 |
+
else:
|
| 522 |
+
# do not convert a into memmap, let pickler do its usual copy with
|
| 523 |
+
# the default system pickler
|
| 524 |
+
util.debug(
|
| 525 |
+
"[ARRAY DUMP] Pickling array (NO MEMMAPPING) (shape={}, "
|
| 526 |
+
" dtype={}).".format(a.shape, a.dtype)
|
| 527 |
+
)
|
| 528 |
+
return (loads, (dumps(a, protocol=HIGHEST_PROTOCOL),))
|
| 529 |
+
|
| 530 |
+
|
| 531 |
+
def get_memmapping_reducers(
|
| 532 |
+
forward_reducers=None,
|
| 533 |
+
backward_reducers=None,
|
| 534 |
+
temp_folder_resolver=None,
|
| 535 |
+
max_nbytes=1e6,
|
| 536 |
+
mmap_mode="r",
|
| 537 |
+
verbose=0,
|
| 538 |
+
prewarm=False,
|
| 539 |
+
unlink_on_gc_collect=True,
|
| 540 |
+
**kwargs,
|
| 541 |
+
):
|
| 542 |
+
"""Construct a pair of memmapping reducer linked to a tmpdir.
|
| 543 |
+
|
| 544 |
+
This function manage the creation and the clean up of the temporary folders
|
| 545 |
+
underlying the memory maps and should be use to get the reducers necessary
|
| 546 |
+
to construct joblib pool or executor.
|
| 547 |
+
"""
|
| 548 |
+
if forward_reducers is None:
|
| 549 |
+
forward_reducers = dict()
|
| 550 |
+
if backward_reducers is None:
|
| 551 |
+
backward_reducers = dict()
|
| 552 |
+
|
| 553 |
+
if np is not None:
|
| 554 |
+
# Register smart numpy.ndarray reducers that detects memmap backed
|
| 555 |
+
# arrays and that is also able to dump to memmap large in-memory
|
| 556 |
+
# arrays over the max_nbytes threshold
|
| 557 |
+
forward_reduce_ndarray = ArrayMemmapForwardReducer(
|
| 558 |
+
max_nbytes,
|
| 559 |
+
temp_folder_resolver,
|
| 560 |
+
mmap_mode,
|
| 561 |
+
unlink_on_gc_collect,
|
| 562 |
+
verbose,
|
| 563 |
+
prewarm=prewarm,
|
| 564 |
+
)
|
| 565 |
+
forward_reducers[np.ndarray] = forward_reduce_ndarray
|
| 566 |
+
forward_reducers[np.memmap] = forward_reduce_ndarray
|
| 567 |
+
|
| 568 |
+
# Communication from child process to the parent process always
|
| 569 |
+
# pickles in-memory numpy.ndarray without dumping them as memmap
|
| 570 |
+
# to avoid confusing the caller and make it tricky to collect the
|
| 571 |
+
# temporary folder
|
| 572 |
+
backward_reducers[np.ndarray] = reduce_array_memmap_backward
|
| 573 |
+
backward_reducers[np.memmap] = reduce_array_memmap_backward
|
| 574 |
+
|
| 575 |
+
return forward_reducers, backward_reducers
|
| 576 |
+
|
| 577 |
+
|
| 578 |
+
class TemporaryResourcesManager(object):
|
| 579 |
+
"""Stateful object able to manage temporary folder and pickles
|
| 580 |
+
|
| 581 |
+
It exposes:
|
| 582 |
+
- a per-context folder name resolving API that memmap-based reducers will
|
| 583 |
+
rely on to know where to pickle the temporary memmaps
|
| 584 |
+
- a temporary file/folder management API that internally uses the
|
| 585 |
+
resource_tracker.
|
| 586 |
+
"""
|
| 587 |
+
|
| 588 |
+
def __init__(self, temp_folder_root=None, context_id=None):
|
| 589 |
+
self._current_temp_folder = None
|
| 590 |
+
self._temp_folder_root = temp_folder_root
|
| 591 |
+
self._use_shared_mem = None
|
| 592 |
+
self._cached_temp_folders = dict()
|
| 593 |
+
self._id = uuid4().hex
|
| 594 |
+
self._finalizers = {}
|
| 595 |
+
if context_id is None:
|
| 596 |
+
# It would be safer to not assign a default context id (less silent
|
| 597 |
+
# bugs), but doing this while maintaining backward compatibility
|
| 598 |
+
# with the previous, context-unaware version get_memmaping_executor
|
| 599 |
+
# exposes too many low-level details.
|
| 600 |
+
context_id = uuid4().hex
|
| 601 |
+
self.set_current_context(context_id)
|
| 602 |
+
|
| 603 |
+
def set_current_context(self, context_id):
|
| 604 |
+
self._current_context_id = context_id
|
| 605 |
+
self.register_new_context(context_id)
|
| 606 |
+
|
| 607 |
+
def register_new_context(self, context_id):
|
| 608 |
+
# Prepare a sub-folder name specific to a context (usually a unique id
|
| 609 |
+
# generated by each instance of the Parallel class). Do not create in
|
| 610 |
+
# advance to spare FS write access if no array is to be dumped).
|
| 611 |
+
if context_id in self._cached_temp_folders:
|
| 612 |
+
return
|
| 613 |
+
else:
|
| 614 |
+
# During its lifecycle, one Parallel object can have several
|
| 615 |
+
# executors associated to it (for instance, if a loky worker raises
|
| 616 |
+
# an exception, joblib shutdowns the executor and instantly
|
| 617 |
+
# recreates a new one before raising the error - see
|
| 618 |
+
# ``ensure_ready``. Because we don't want two executors tied to
|
| 619 |
+
# the same Parallel object (and thus the same context id) to
|
| 620 |
+
# register/use/delete the same folder, we also add an id specific
|
| 621 |
+
# to the current Manager (and thus specific to its associated
|
| 622 |
+
# executor) to the folder name.
|
| 623 |
+
new_folder_name = "joblib_memmapping_folder_{}_{}_{}".format(
|
| 624 |
+
os.getpid(), self._id, context_id
|
| 625 |
+
)
|
| 626 |
+
new_folder_path, _ = _get_temp_dir(new_folder_name, self._temp_folder_root)
|
| 627 |
+
self.register_folder_finalizer(new_folder_path, context_id)
|
| 628 |
+
self._cached_temp_folders[context_id] = new_folder_path
|
| 629 |
+
|
| 630 |
+
def resolve_temp_folder_name(self):
|
| 631 |
+
"""Return a folder name specific to the currently activated context"""
|
| 632 |
+
return self._cached_temp_folders[self._current_context_id]
|
| 633 |
+
|
| 634 |
+
# resource management API
|
| 635 |
+
|
| 636 |
+
def register_folder_finalizer(self, pool_subfolder, context_id):
|
| 637 |
+
# Register the garbage collector at program exit in case caller forgets
|
| 638 |
+
# to call terminate explicitly: note we do not pass any reference to
|
| 639 |
+
# ensure that this callback won't prevent garbage collection of
|
| 640 |
+
# parallel instance and related file handler resources such as POSIX
|
| 641 |
+
# semaphores and pipes
|
| 642 |
+
pool_module_name = whichmodule(delete_folder, "delete_folder")
|
| 643 |
+
resource_tracker.register(pool_subfolder, "folder")
|
| 644 |
+
|
| 645 |
+
def _cleanup():
|
| 646 |
+
# In some cases the Python runtime seems to set delete_folder to
|
| 647 |
+
# None just before exiting when accessing the delete_folder
|
| 648 |
+
# function from the closure namespace. So instead we reimport
|
| 649 |
+
# the delete_folder function explicitly.
|
| 650 |
+
# https://github.com/joblib/joblib/issues/328
|
| 651 |
+
# We cannot just use from 'joblib.pool import delete_folder'
|
| 652 |
+
# because joblib should only use relative imports to allow
|
| 653 |
+
# easy vendoring.
|
| 654 |
+
delete_folder = __import__(
|
| 655 |
+
pool_module_name, fromlist=["delete_folder"]
|
| 656 |
+
).delete_folder
|
| 657 |
+
try:
|
| 658 |
+
delete_folder(pool_subfolder, allow_non_empty=True)
|
| 659 |
+
resource_tracker.unregister(pool_subfolder, "folder")
|
| 660 |
+
except OSError:
|
| 661 |
+
warnings.warn(
|
| 662 |
+
"Failed to delete temporary folder: {}".format(pool_subfolder)
|
| 663 |
+
)
|
| 664 |
+
|
| 665 |
+
self._finalizers[context_id] = atexit.register(_cleanup)
|
| 666 |
+
|
| 667 |
+
def _clean_temporary_resources(
|
| 668 |
+
self, context_id=None, force=False, allow_non_empty=False
|
| 669 |
+
):
|
| 670 |
+
"""Clean temporary resources created by a process-based pool"""
|
| 671 |
+
if context_id is None:
|
| 672 |
+
# Iterates over a copy of the cache keys to avoid Error due to
|
| 673 |
+
# iterating over a changing size dictionary.
|
| 674 |
+
for context_id in list(self._cached_temp_folders):
|
| 675 |
+
self._clean_temporary_resources(
|
| 676 |
+
context_id, force=force, allow_non_empty=allow_non_empty
|
| 677 |
+
)
|
| 678 |
+
else:
|
| 679 |
+
temp_folder = self._cached_temp_folders.get(context_id)
|
| 680 |
+
if temp_folder and os.path.exists(temp_folder):
|
| 681 |
+
for filename in os.listdir(temp_folder):
|
| 682 |
+
if force:
|
| 683 |
+
# Some workers have failed and the ref counted might
|
| 684 |
+
# be off. The workers should have shut down by this
|
| 685 |
+
# time so forcefully clean up the files.
|
| 686 |
+
resource_tracker.unregister(
|
| 687 |
+
os.path.join(temp_folder, filename), "file"
|
| 688 |
+
)
|
| 689 |
+
else:
|
| 690 |
+
resource_tracker.maybe_unlink(
|
| 691 |
+
os.path.join(temp_folder, filename), "file"
|
| 692 |
+
)
|
| 693 |
+
|
| 694 |
+
# When forcing clean-up, try to delete the folder even if some
|
| 695 |
+
# files are still in it. Otherwise, try to delete the folder
|
| 696 |
+
allow_non_empty |= force
|
| 697 |
+
|
| 698 |
+
# Clean up the folder if possible, either if it is empty or
|
| 699 |
+
# if none of the files in it are in used and allow_non_empty.
|
| 700 |
+
try:
|
| 701 |
+
delete_folder(temp_folder, allow_non_empty=allow_non_empty)
|
| 702 |
+
# Forget the folder once it has been deleted
|
| 703 |
+
self._cached_temp_folders.pop(context_id, None)
|
| 704 |
+
resource_tracker.unregister(temp_folder, "folder")
|
| 705 |
+
|
| 706 |
+
# Also cancel the finalizers that gets triggered at gc.
|
| 707 |
+
finalizer = self._finalizers.pop(context_id, None)
|
| 708 |
+
if finalizer is not None:
|
| 709 |
+
atexit.unregister(finalizer)
|
| 710 |
+
|
| 711 |
+
except OSError:
|
| 712 |
+
# Temporary folder cannot be deleted right now.
|
| 713 |
+
# This folder will be cleaned up by an atexit
|
| 714 |
+
# finalizer registered by the memmapping_reducer.
|
| 715 |
+
pass
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_multiprocessing_helpers.py
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Helper module to factorize the conditional multiprocessing import logic
|
| 2 |
+
|
| 3 |
+
We use a distinct module to simplify import statements and avoid introducing
|
| 4 |
+
circular dependencies (for instance for the assert_spawning name).
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import os
|
| 8 |
+
import warnings
|
| 9 |
+
|
| 10 |
+
# Obtain possible configuration from the environment, assuming 1 (on)
|
| 11 |
+
# by default, upon 0 set to None. Should instructively fail if some non
|
| 12 |
+
# 0/1 value is set.
|
| 13 |
+
mp = int(os.environ.get("JOBLIB_MULTIPROCESSING", 1)) or None
|
| 14 |
+
if mp:
|
| 15 |
+
try:
|
| 16 |
+
import _multiprocessing # noqa
|
| 17 |
+
import multiprocessing as mp
|
| 18 |
+
except ImportError:
|
| 19 |
+
mp = None
|
| 20 |
+
|
| 21 |
+
# 2nd stage: validate that locking is available on the system and
|
| 22 |
+
# issue a warning if not
|
| 23 |
+
if mp is not None:
|
| 24 |
+
try:
|
| 25 |
+
# try to create a named semaphore using SemLock to make sure they are
|
| 26 |
+
# available on this platform. We use the low level object
|
| 27 |
+
# _multiprocessing.SemLock to avoid spawning a resource tracker on
|
| 28 |
+
# Unix system or changing the default backend.
|
| 29 |
+
import tempfile
|
| 30 |
+
from _multiprocessing import SemLock
|
| 31 |
+
|
| 32 |
+
_rand = tempfile._RandomNameSequence()
|
| 33 |
+
for i in range(100):
|
| 34 |
+
try:
|
| 35 |
+
name = "/joblib-{}-{}".format(os.getpid(), next(_rand))
|
| 36 |
+
_sem = SemLock(0, 0, 1, name=name, unlink=True)
|
| 37 |
+
del _sem # cleanup
|
| 38 |
+
break
|
| 39 |
+
except FileExistsError as e: # pragma: no cover
|
| 40 |
+
if i >= 99:
|
| 41 |
+
raise FileExistsError("cannot find name for semaphore") from e
|
| 42 |
+
except (FileExistsError, AttributeError, ImportError, OSError) as e:
|
| 43 |
+
mp = None
|
| 44 |
+
warnings.warn("%s. joblib will operate in serial mode" % (e,))
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
# 3rd stage: backward compat for the assert_spawning helper
|
| 48 |
+
if mp is not None:
|
| 49 |
+
from multiprocessing.context import assert_spawning
|
| 50 |
+
else:
|
| 51 |
+
assert_spawning = None
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_parallel_backends.py
ADDED
|
@@ -0,0 +1,753 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Backends for embarrassingly parallel code.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import contextlib
|
| 6 |
+
import gc
|
| 7 |
+
import os
|
| 8 |
+
import threading
|
| 9 |
+
import warnings
|
| 10 |
+
from abc import ABCMeta, abstractmethod
|
| 11 |
+
|
| 12 |
+
from ._multiprocessing_helpers import mp
|
| 13 |
+
from ._utils import (
|
| 14 |
+
_retrieve_traceback_capturing_wrapped_call,
|
| 15 |
+
_TracebackCapturingWrapper,
|
| 16 |
+
)
|
| 17 |
+
|
| 18 |
+
if mp is not None:
|
| 19 |
+
from multiprocessing.pool import ThreadPool
|
| 20 |
+
|
| 21 |
+
from .executor import get_memmapping_executor
|
| 22 |
+
|
| 23 |
+
# Import loky only if multiprocessing is present
|
| 24 |
+
from .externals.loky import cpu_count, process_executor
|
| 25 |
+
from .externals.loky.process_executor import ShutdownExecutorError
|
| 26 |
+
from .pool import MemmappingPool
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
class ParallelBackendBase(metaclass=ABCMeta):
|
| 30 |
+
"""Helper abc which defines all methods a ParallelBackend must implement"""
|
| 31 |
+
|
| 32 |
+
default_n_jobs = 1
|
| 33 |
+
|
| 34 |
+
supports_inner_max_num_threads = False
|
| 35 |
+
|
| 36 |
+
# This flag was introduced for backward compatibility reasons.
|
| 37 |
+
# New backends should always set it to True and implement the
|
| 38 |
+
# `retrieve_result_callback` method.
|
| 39 |
+
supports_retrieve_callback = False
|
| 40 |
+
|
| 41 |
+
@property
|
| 42 |
+
def supports_return_generator(self):
|
| 43 |
+
return self.supports_retrieve_callback
|
| 44 |
+
|
| 45 |
+
@property
|
| 46 |
+
def supports_timeout(self):
|
| 47 |
+
return self.supports_retrieve_callback
|
| 48 |
+
|
| 49 |
+
nesting_level = None
|
| 50 |
+
|
| 51 |
+
def __init__(
|
| 52 |
+
self, nesting_level=None, inner_max_num_threads=None, **backend_kwargs
|
| 53 |
+
):
|
| 54 |
+
super().__init__()
|
| 55 |
+
self.nesting_level = nesting_level
|
| 56 |
+
self.inner_max_num_threads = inner_max_num_threads
|
| 57 |
+
self.backend_kwargs = backend_kwargs
|
| 58 |
+
|
| 59 |
+
MAX_NUM_THREADS_VARS = [
|
| 60 |
+
"OMP_NUM_THREADS",
|
| 61 |
+
"OPENBLAS_NUM_THREADS",
|
| 62 |
+
"MKL_NUM_THREADS",
|
| 63 |
+
"BLIS_NUM_THREADS",
|
| 64 |
+
"VECLIB_MAXIMUM_THREADS",
|
| 65 |
+
"NUMBA_NUM_THREADS",
|
| 66 |
+
"NUMEXPR_NUM_THREADS",
|
| 67 |
+
]
|
| 68 |
+
|
| 69 |
+
TBB_ENABLE_IPC_VAR = "ENABLE_IPC"
|
| 70 |
+
|
| 71 |
+
@abstractmethod
|
| 72 |
+
def effective_n_jobs(self, n_jobs):
|
| 73 |
+
"""Determine the number of jobs that can actually run in parallel
|
| 74 |
+
|
| 75 |
+
n_jobs is the number of workers requested by the callers. Passing
|
| 76 |
+
n_jobs=-1 means requesting all available workers for instance matching
|
| 77 |
+
the number of CPU cores on the worker host(s).
|
| 78 |
+
|
| 79 |
+
This method should return a guesstimate of the number of workers that
|
| 80 |
+
can actually perform work concurrently. The primary use case is to make
|
| 81 |
+
it possible for the caller to know in how many chunks to slice the
|
| 82 |
+
work.
|
| 83 |
+
|
| 84 |
+
In general working on larger data chunks is more efficient (less
|
| 85 |
+
scheduling overhead and better use of CPU cache prefetching heuristics)
|
| 86 |
+
as long as all the workers have enough work to do.
|
| 87 |
+
"""
|
| 88 |
+
|
| 89 |
+
def apply_async(self, func, callback=None):
|
| 90 |
+
"""Deprecated: implement `submit` instead."""
|
| 91 |
+
raise NotImplementedError("Implement `submit` instead.")
|
| 92 |
+
|
| 93 |
+
def submit(self, func, callback=None):
|
| 94 |
+
"""Schedule a function to be run and return a future-like object.
|
| 95 |
+
|
| 96 |
+
This method should return a future-like object that allow tracking
|
| 97 |
+
the progress of the task.
|
| 98 |
+
|
| 99 |
+
If ``supports_retrieve_callback`` is False, the return value of this
|
| 100 |
+
method is passed to ``retrieve_result`` instead of calling
|
| 101 |
+
``retrieve_result_callback``.
|
| 102 |
+
|
| 103 |
+
Parameters
|
| 104 |
+
----------
|
| 105 |
+
func: callable
|
| 106 |
+
The function to be run in parallel.
|
| 107 |
+
|
| 108 |
+
callback: callable
|
| 109 |
+
A callable that will be called when the task is completed. This callable
|
| 110 |
+
is a wrapper around ``retrieve_result_callback``. This should be added
|
| 111 |
+
to the future-like object returned by this method, so that the callback
|
| 112 |
+
is called when the task is completed.
|
| 113 |
+
|
| 114 |
+
For future-like backends, this can be achieved with something like
|
| 115 |
+
``future.add_done_callback(callback)``.
|
| 116 |
+
|
| 117 |
+
Returns
|
| 118 |
+
-------
|
| 119 |
+
future: future-like
|
| 120 |
+
A future-like object to track the execution of the submitted function.
|
| 121 |
+
"""
|
| 122 |
+
warnings.warn(
|
| 123 |
+
"`apply_async` is deprecated, implement and use `submit` instead.",
|
| 124 |
+
DeprecationWarning,
|
| 125 |
+
)
|
| 126 |
+
return self.apply_async(func, callback)
|
| 127 |
+
|
| 128 |
+
def retrieve_result_callback(self, out):
|
| 129 |
+
"""Called within the callback function passed to `submit`.
|
| 130 |
+
|
| 131 |
+
This method can customise how the result of the function is retrieved
|
| 132 |
+
from the future-like object.
|
| 133 |
+
|
| 134 |
+
Parameters
|
| 135 |
+
----------
|
| 136 |
+
future: future-like
|
| 137 |
+
The future-like object returned by the `submit` method.
|
| 138 |
+
|
| 139 |
+
Returns
|
| 140 |
+
-------
|
| 141 |
+
result: object
|
| 142 |
+
The result of the function executed in parallel.
|
| 143 |
+
"""
|
| 144 |
+
|
| 145 |
+
def retrieve_result(self, out, timeout=None):
|
| 146 |
+
"""Hook to retrieve the result when support_retrieve_callback=False.
|
| 147 |
+
|
| 148 |
+
The argument `out` is the result of the `submit` call. This method
|
| 149 |
+
should return the result of the computation or raise an exception if
|
| 150 |
+
the computation failed.
|
| 151 |
+
"""
|
| 152 |
+
if self.supports_timeout:
|
| 153 |
+
return out.get(timeout=timeout)
|
| 154 |
+
else:
|
| 155 |
+
return out.get()
|
| 156 |
+
|
| 157 |
+
def configure(
|
| 158 |
+
self, n_jobs=1, parallel=None, prefer=None, require=None, **backend_kwargs
|
| 159 |
+
):
|
| 160 |
+
"""Reconfigure the backend and return the number of workers.
|
| 161 |
+
|
| 162 |
+
This makes it possible to reuse an existing backend instance for
|
| 163 |
+
successive independent calls to Parallel with different parameters.
|
| 164 |
+
"""
|
| 165 |
+
self.parallel = parallel
|
| 166 |
+
return self.effective_n_jobs(n_jobs)
|
| 167 |
+
|
| 168 |
+
def start_call(self):
|
| 169 |
+
"""Call-back method called at the beginning of a Parallel call"""
|
| 170 |
+
|
| 171 |
+
def stop_call(self):
|
| 172 |
+
"""Call-back method called at the end of a Parallel call"""
|
| 173 |
+
|
| 174 |
+
def terminate(self):
|
| 175 |
+
"""Shutdown the workers and free the shared memory."""
|
| 176 |
+
|
| 177 |
+
def compute_batch_size(self):
|
| 178 |
+
"""Determine the optimal batch size"""
|
| 179 |
+
return 1
|
| 180 |
+
|
| 181 |
+
def batch_completed(self, batch_size, duration):
|
| 182 |
+
"""Callback indicate how long it took to run a batch"""
|
| 183 |
+
|
| 184 |
+
def abort_everything(self, ensure_ready=True):
|
| 185 |
+
"""Abort any running tasks
|
| 186 |
+
|
| 187 |
+
This is called when an exception has been raised when executing a task
|
| 188 |
+
and all the remaining tasks will be ignored and can therefore be
|
| 189 |
+
aborted to spare computation resources.
|
| 190 |
+
|
| 191 |
+
If ensure_ready is True, the backend should be left in an operating
|
| 192 |
+
state as future tasks might be re-submitted via that same backend
|
| 193 |
+
instance.
|
| 194 |
+
|
| 195 |
+
If ensure_ready is False, the implementer of this method can decide
|
| 196 |
+
to leave the backend in a closed / terminated state as no new task
|
| 197 |
+
are expected to be submitted to this backend.
|
| 198 |
+
|
| 199 |
+
Setting ensure_ready to False is an optimization that can be leveraged
|
| 200 |
+
when aborting tasks via killing processes from a local process pool
|
| 201 |
+
managed by the backend it-self: if we expect no new tasks, there is no
|
| 202 |
+
point in re-creating new workers.
|
| 203 |
+
"""
|
| 204 |
+
# Does nothing by default: to be overridden in subclasses when
|
| 205 |
+
# canceling tasks is possible.
|
| 206 |
+
pass
|
| 207 |
+
|
| 208 |
+
def get_nested_backend(self):
|
| 209 |
+
"""Backend instance to be used by nested Parallel calls.
|
| 210 |
+
|
| 211 |
+
By default a thread-based backend is used for the first level of
|
| 212 |
+
nesting. Beyond, switch to sequential backend to avoid spawning too
|
| 213 |
+
many threads on the host.
|
| 214 |
+
"""
|
| 215 |
+
nesting_level = getattr(self, "nesting_level", 0) + 1
|
| 216 |
+
if nesting_level > 1:
|
| 217 |
+
return SequentialBackend(nesting_level=nesting_level), None
|
| 218 |
+
else:
|
| 219 |
+
return ThreadingBackend(nesting_level=nesting_level), None
|
| 220 |
+
|
| 221 |
+
def _prepare_worker_env(self, n_jobs):
|
| 222 |
+
"""Return environment variables limiting threadpools in external libs.
|
| 223 |
+
|
| 224 |
+
This function return a dict containing environment variables to pass
|
| 225 |
+
when creating a pool of process. These environment variables limit the
|
| 226 |
+
number of threads to `n_threads` for OpenMP, MKL, Accelerated and
|
| 227 |
+
OpenBLAS libraries in the child processes.
|
| 228 |
+
"""
|
| 229 |
+
explicit_n_threads = self.inner_max_num_threads
|
| 230 |
+
default_n_threads = max(cpu_count() // n_jobs, 1)
|
| 231 |
+
|
| 232 |
+
# Set the inner environment variables to self.inner_max_num_threads if
|
| 233 |
+
# it is given. Else, default to cpu_count // n_jobs unless the variable
|
| 234 |
+
# is already present in the parent process environment.
|
| 235 |
+
env = {}
|
| 236 |
+
for var in self.MAX_NUM_THREADS_VARS:
|
| 237 |
+
if explicit_n_threads is None:
|
| 238 |
+
var_value = os.environ.get(var, default_n_threads)
|
| 239 |
+
else:
|
| 240 |
+
var_value = explicit_n_threads
|
| 241 |
+
|
| 242 |
+
env[var] = str(var_value)
|
| 243 |
+
|
| 244 |
+
if self.TBB_ENABLE_IPC_VAR not in os.environ:
|
| 245 |
+
# To avoid over-subscription when using TBB, let the TBB schedulers
|
| 246 |
+
# use Inter Process Communication to coordinate:
|
| 247 |
+
env[self.TBB_ENABLE_IPC_VAR] = "1"
|
| 248 |
+
return env
|
| 249 |
+
|
| 250 |
+
@contextlib.contextmanager
|
| 251 |
+
def retrieval_context(self):
|
| 252 |
+
"""Context manager to manage an execution context.
|
| 253 |
+
|
| 254 |
+
Calls to Parallel.retrieve will be made inside this context.
|
| 255 |
+
|
| 256 |
+
By default, this does nothing. It may be useful for subclasses to
|
| 257 |
+
handle nested parallelism. In particular, it may be required to avoid
|
| 258 |
+
deadlocks if a backend manages a fixed number of workers, when those
|
| 259 |
+
workers may be asked to do nested Parallel calls. Without
|
| 260 |
+
'retrieval_context' this could lead to deadlock, as all the workers
|
| 261 |
+
managed by the backend may be "busy" waiting for the nested parallel
|
| 262 |
+
calls to finish, but the backend has no free workers to execute those
|
| 263 |
+
tasks.
|
| 264 |
+
"""
|
| 265 |
+
yield
|
| 266 |
+
|
| 267 |
+
@staticmethod
|
| 268 |
+
def in_main_thread():
|
| 269 |
+
return isinstance(threading.current_thread(), threading._MainThread)
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
class SequentialBackend(ParallelBackendBase):
|
| 273 |
+
"""A ParallelBackend which will execute all batches sequentially.
|
| 274 |
+
|
| 275 |
+
Does not use/create any threading objects, and hence has minimal
|
| 276 |
+
overhead. Used when n_jobs == 1.
|
| 277 |
+
"""
|
| 278 |
+
|
| 279 |
+
uses_threads = True
|
| 280 |
+
supports_timeout = False
|
| 281 |
+
supports_retrieve_callback = False
|
| 282 |
+
supports_sharedmem = True
|
| 283 |
+
|
| 284 |
+
def effective_n_jobs(self, n_jobs):
|
| 285 |
+
"""Determine the number of jobs which are going to run in parallel"""
|
| 286 |
+
if n_jobs == 0:
|
| 287 |
+
raise ValueError("n_jobs == 0 in Parallel has no meaning")
|
| 288 |
+
return 1
|
| 289 |
+
|
| 290 |
+
def submit(self, func, callback=None):
|
| 291 |
+
"""Schedule a func to be run"""
|
| 292 |
+
raise RuntimeError("Should never be called for SequentialBackend.")
|
| 293 |
+
|
| 294 |
+
def retrieve_result_callback(self, out):
|
| 295 |
+
raise RuntimeError("Should never be called for SequentialBackend.")
|
| 296 |
+
|
| 297 |
+
def get_nested_backend(self):
|
| 298 |
+
# import is not top level to avoid cyclic import errors.
|
| 299 |
+
from .parallel import get_active_backend
|
| 300 |
+
|
| 301 |
+
# SequentialBackend should neither change the nesting level, the
|
| 302 |
+
# default backend or the number of jobs. Just return the current one.
|
| 303 |
+
return get_active_backend()
|
| 304 |
+
|
| 305 |
+
|
| 306 |
+
class PoolManagerMixin(object):
|
| 307 |
+
"""A helper class for managing pool of workers."""
|
| 308 |
+
|
| 309 |
+
_pool = None
|
| 310 |
+
|
| 311 |
+
def effective_n_jobs(self, n_jobs):
|
| 312 |
+
"""Determine the number of jobs which are going to run in parallel"""
|
| 313 |
+
if n_jobs == 0:
|
| 314 |
+
raise ValueError("n_jobs == 0 in Parallel has no meaning")
|
| 315 |
+
elif mp is None or n_jobs is None:
|
| 316 |
+
# multiprocessing is not available or disabled, fallback
|
| 317 |
+
# to sequential mode
|
| 318 |
+
return 1
|
| 319 |
+
elif n_jobs < 0:
|
| 320 |
+
n_jobs = max(cpu_count() + 1 + n_jobs, 1)
|
| 321 |
+
return n_jobs
|
| 322 |
+
|
| 323 |
+
def terminate(self):
|
| 324 |
+
"""Shutdown the process or thread pool"""
|
| 325 |
+
if self._pool is not None:
|
| 326 |
+
self._pool.close()
|
| 327 |
+
self._pool.terminate() # terminate does a join()
|
| 328 |
+
self._pool = None
|
| 329 |
+
|
| 330 |
+
def _get_pool(self):
|
| 331 |
+
"""Used by `submit` to make it possible to implement lazy init"""
|
| 332 |
+
return self._pool
|
| 333 |
+
|
| 334 |
+
def submit(self, func, callback=None):
|
| 335 |
+
"""Schedule a func to be run"""
|
| 336 |
+
# Here, we need a wrapper to avoid crashes on KeyboardInterruptErrors.
|
| 337 |
+
# We also call the callback on error, to make sure the pool does not
|
| 338 |
+
# wait on crashed jobs.
|
| 339 |
+
return self._get_pool().apply_async(
|
| 340 |
+
_TracebackCapturingWrapper(func),
|
| 341 |
+
(),
|
| 342 |
+
callback=callback,
|
| 343 |
+
error_callback=callback,
|
| 344 |
+
)
|
| 345 |
+
|
| 346 |
+
def retrieve_result_callback(self, result):
|
| 347 |
+
"""Mimic concurrent.futures results, raising an error if needed."""
|
| 348 |
+
# In the multiprocessing Pool API, the callback are called with the
|
| 349 |
+
# result value as an argument so `result`(`out`) is the output of
|
| 350 |
+
# job.get(). It's either the result or the exception raised while
|
| 351 |
+
# collecting the result.
|
| 352 |
+
return _retrieve_traceback_capturing_wrapped_call(result)
|
| 353 |
+
|
| 354 |
+
def abort_everything(self, ensure_ready=True):
|
| 355 |
+
"""Shutdown the pool and restart a new one with the same parameters"""
|
| 356 |
+
self.terminate()
|
| 357 |
+
if ensure_ready:
|
| 358 |
+
self.configure(
|
| 359 |
+
n_jobs=self.parallel.n_jobs,
|
| 360 |
+
parallel=self.parallel,
|
| 361 |
+
**self.parallel._backend_kwargs,
|
| 362 |
+
)
|
| 363 |
+
|
| 364 |
+
|
| 365 |
+
class AutoBatchingMixin(object):
|
| 366 |
+
"""A helper class for automagically batching jobs."""
|
| 367 |
+
|
| 368 |
+
# In seconds, should be big enough to hide multiprocessing dispatching
|
| 369 |
+
# overhead.
|
| 370 |
+
# This settings was found by running benchmarks/bench_auto_batching.py
|
| 371 |
+
# with various parameters on various platforms.
|
| 372 |
+
MIN_IDEAL_BATCH_DURATION = 0.2
|
| 373 |
+
|
| 374 |
+
# Should not be too high to avoid stragglers: long jobs running alone
|
| 375 |
+
# on a single worker while other workers have no work to process any more.
|
| 376 |
+
MAX_IDEAL_BATCH_DURATION = 2
|
| 377 |
+
|
| 378 |
+
# Batching counters default values
|
| 379 |
+
_DEFAULT_EFFECTIVE_BATCH_SIZE = 1
|
| 380 |
+
_DEFAULT_SMOOTHED_BATCH_DURATION = 0.0
|
| 381 |
+
|
| 382 |
+
def __init__(self, **kwargs):
|
| 383 |
+
super().__init__(**kwargs)
|
| 384 |
+
self._effective_batch_size = self._DEFAULT_EFFECTIVE_BATCH_SIZE
|
| 385 |
+
self._smoothed_batch_duration = self._DEFAULT_SMOOTHED_BATCH_DURATION
|
| 386 |
+
|
| 387 |
+
def compute_batch_size(self):
|
| 388 |
+
"""Determine the optimal batch size"""
|
| 389 |
+
old_batch_size = self._effective_batch_size
|
| 390 |
+
batch_duration = self._smoothed_batch_duration
|
| 391 |
+
if batch_duration > 0 and batch_duration < self.MIN_IDEAL_BATCH_DURATION:
|
| 392 |
+
# The current batch size is too small: the duration of the
|
| 393 |
+
# processing of a batch of task is not large enough to hide
|
| 394 |
+
# the scheduling overhead.
|
| 395 |
+
ideal_batch_size = int(
|
| 396 |
+
old_batch_size * self.MIN_IDEAL_BATCH_DURATION / batch_duration
|
| 397 |
+
)
|
| 398 |
+
# Multiply by two to limit oscilations between min and max.
|
| 399 |
+
ideal_batch_size *= 2
|
| 400 |
+
|
| 401 |
+
# dont increase the batch size too fast to limit huge batch sizes
|
| 402 |
+
# potentially leading to starving worker
|
| 403 |
+
batch_size = min(2 * old_batch_size, ideal_batch_size)
|
| 404 |
+
|
| 405 |
+
batch_size = max(batch_size, 1)
|
| 406 |
+
|
| 407 |
+
self._effective_batch_size = batch_size
|
| 408 |
+
if self.parallel.verbose >= 10:
|
| 409 |
+
self.parallel._print(
|
| 410 |
+
f"Batch computation too fast ({batch_duration}s.) "
|
| 411 |
+
f"Setting batch_size={batch_size}."
|
| 412 |
+
)
|
| 413 |
+
elif batch_duration > self.MAX_IDEAL_BATCH_DURATION and old_batch_size >= 2:
|
| 414 |
+
# The current batch size is too big. If we schedule overly long
|
| 415 |
+
# running batches some CPUs might wait with nothing left to do
|
| 416 |
+
# while a couple of CPUs a left processing a few long running
|
| 417 |
+
# batches. Better reduce the batch size a bit to limit the
|
| 418 |
+
# likelihood of scheduling such stragglers.
|
| 419 |
+
|
| 420 |
+
# decrease the batch size quickly to limit potential starving
|
| 421 |
+
ideal_batch_size = int(
|
| 422 |
+
old_batch_size * self.MIN_IDEAL_BATCH_DURATION / batch_duration
|
| 423 |
+
)
|
| 424 |
+
# Multiply by two to limit oscilations between min and max.
|
| 425 |
+
batch_size = max(2 * ideal_batch_size, 1)
|
| 426 |
+
self._effective_batch_size = batch_size
|
| 427 |
+
if self.parallel.verbose >= 10:
|
| 428 |
+
self.parallel._print(
|
| 429 |
+
f"Batch computation too slow ({batch_duration}s.) "
|
| 430 |
+
f"Setting batch_size={batch_size}."
|
| 431 |
+
)
|
| 432 |
+
else:
|
| 433 |
+
# No batch size adjustment
|
| 434 |
+
batch_size = old_batch_size
|
| 435 |
+
|
| 436 |
+
if batch_size != old_batch_size:
|
| 437 |
+
# Reset estimation of the smoothed mean batch duration: this
|
| 438 |
+
# estimate is updated in the multiprocessing apply_async
|
| 439 |
+
# CallBack as long as the batch_size is constant. Therefore
|
| 440 |
+
# we need to reset the estimate whenever we re-tune the batch
|
| 441 |
+
# size.
|
| 442 |
+
self._smoothed_batch_duration = self._DEFAULT_SMOOTHED_BATCH_DURATION
|
| 443 |
+
|
| 444 |
+
return batch_size
|
| 445 |
+
|
| 446 |
+
def batch_completed(self, batch_size, duration):
|
| 447 |
+
"""Callback indicate how long it took to run a batch"""
|
| 448 |
+
if batch_size == self._effective_batch_size:
|
| 449 |
+
# Update the smoothed streaming estimate of the duration of a batch
|
| 450 |
+
# from dispatch to completion
|
| 451 |
+
old_duration = self._smoothed_batch_duration
|
| 452 |
+
if old_duration == self._DEFAULT_SMOOTHED_BATCH_DURATION:
|
| 453 |
+
# First record of duration for this batch size after the last
|
| 454 |
+
# reset.
|
| 455 |
+
new_duration = duration
|
| 456 |
+
else:
|
| 457 |
+
# Update the exponentially weighted average of the duration of
|
| 458 |
+
# batch for the current effective size.
|
| 459 |
+
new_duration = 0.8 * old_duration + 0.2 * duration
|
| 460 |
+
self._smoothed_batch_duration = new_duration
|
| 461 |
+
|
| 462 |
+
def reset_batch_stats(self):
|
| 463 |
+
"""Reset batch statistics to default values.
|
| 464 |
+
|
| 465 |
+
This avoids interferences with future jobs.
|
| 466 |
+
"""
|
| 467 |
+
self._effective_batch_size = self._DEFAULT_EFFECTIVE_BATCH_SIZE
|
| 468 |
+
self._smoothed_batch_duration = self._DEFAULT_SMOOTHED_BATCH_DURATION
|
| 469 |
+
|
| 470 |
+
|
| 471 |
+
class ThreadingBackend(PoolManagerMixin, ParallelBackendBase):
|
| 472 |
+
"""A ParallelBackend which will use a thread pool to execute batches in.
|
| 473 |
+
|
| 474 |
+
This is a low-overhead backend but it suffers from the Python Global
|
| 475 |
+
Interpreter Lock if the called function relies a lot on Python objects.
|
| 476 |
+
Mostly useful when the execution bottleneck is a compiled extension that
|
| 477 |
+
explicitly releases the GIL (for instance a Cython loop wrapped in a "with
|
| 478 |
+
nogil" block or an expensive call to a library such as NumPy).
|
| 479 |
+
|
| 480 |
+
The actual thread pool is lazily initialized: the actual thread pool
|
| 481 |
+
construction is delayed to the first call to apply_async.
|
| 482 |
+
|
| 483 |
+
ThreadingBackend is used as the default backend for nested calls.
|
| 484 |
+
"""
|
| 485 |
+
|
| 486 |
+
supports_retrieve_callback = True
|
| 487 |
+
uses_threads = True
|
| 488 |
+
supports_sharedmem = True
|
| 489 |
+
|
| 490 |
+
def configure(self, n_jobs=1, parallel=None, **backend_kwargs):
|
| 491 |
+
"""Build a process or thread pool and return the number of workers"""
|
| 492 |
+
n_jobs = self.effective_n_jobs(n_jobs)
|
| 493 |
+
if n_jobs == 1:
|
| 494 |
+
# Avoid unnecessary overhead and use sequential backend instead.
|
| 495 |
+
raise FallbackToBackend(SequentialBackend(nesting_level=self.nesting_level))
|
| 496 |
+
self.parallel = parallel
|
| 497 |
+
self._n_jobs = n_jobs
|
| 498 |
+
return n_jobs
|
| 499 |
+
|
| 500 |
+
def _get_pool(self):
|
| 501 |
+
"""Lazily initialize the thread pool
|
| 502 |
+
|
| 503 |
+
The actual pool of worker threads is only initialized at the first
|
| 504 |
+
call to apply_async.
|
| 505 |
+
"""
|
| 506 |
+
if self._pool is None:
|
| 507 |
+
self._pool = ThreadPool(self._n_jobs)
|
| 508 |
+
return self._pool
|
| 509 |
+
|
| 510 |
+
|
| 511 |
+
class MultiprocessingBackend(PoolManagerMixin, AutoBatchingMixin, ParallelBackendBase):
|
| 512 |
+
"""A ParallelBackend which will use a multiprocessing.Pool.
|
| 513 |
+
|
| 514 |
+
Will introduce some communication and memory overhead when exchanging
|
| 515 |
+
input and output data with the with the worker Python processes.
|
| 516 |
+
However, does not suffer from the Python Global Interpreter Lock.
|
| 517 |
+
"""
|
| 518 |
+
|
| 519 |
+
supports_retrieve_callback = True
|
| 520 |
+
supports_return_generator = False
|
| 521 |
+
|
| 522 |
+
def effective_n_jobs(self, n_jobs):
|
| 523 |
+
"""Determine the number of jobs which are going to run in parallel.
|
| 524 |
+
|
| 525 |
+
This also checks if we are attempting to create a nested parallel
|
| 526 |
+
loop.
|
| 527 |
+
"""
|
| 528 |
+
if mp is None:
|
| 529 |
+
return 1
|
| 530 |
+
|
| 531 |
+
if mp.current_process().daemon:
|
| 532 |
+
# Daemonic processes cannot have children
|
| 533 |
+
if n_jobs != 1:
|
| 534 |
+
if inside_dask_worker():
|
| 535 |
+
msg = (
|
| 536 |
+
"Inside a Dask worker with daemon=True, "
|
| 537 |
+
"setting n_jobs=1.\nPossible work-arounds:\n"
|
| 538 |
+
"- dask.config.set("
|
| 539 |
+
"{'distributed.worker.daemon': False})"
|
| 540 |
+
"- set the environment variable "
|
| 541 |
+
"DASK_DISTRIBUTED__WORKER__DAEMON=False\n"
|
| 542 |
+
"before creating your Dask cluster."
|
| 543 |
+
)
|
| 544 |
+
else:
|
| 545 |
+
msg = (
|
| 546 |
+
"Multiprocessing-backed parallel loops "
|
| 547 |
+
"cannot be nested, setting n_jobs=1"
|
| 548 |
+
)
|
| 549 |
+
warnings.warn(msg, stacklevel=3)
|
| 550 |
+
return 1
|
| 551 |
+
|
| 552 |
+
if process_executor._CURRENT_DEPTH > 0:
|
| 553 |
+
# Mixing loky and multiprocessing in nested loop is not supported
|
| 554 |
+
if n_jobs != 1:
|
| 555 |
+
warnings.warn(
|
| 556 |
+
"Multiprocessing-backed parallel loops cannot be nested,"
|
| 557 |
+
" below loky, setting n_jobs=1",
|
| 558 |
+
stacklevel=3,
|
| 559 |
+
)
|
| 560 |
+
return 1
|
| 561 |
+
|
| 562 |
+
elif not (self.in_main_thread() or self.nesting_level == 0):
|
| 563 |
+
# Prevent posix fork inside in non-main posix threads
|
| 564 |
+
if n_jobs != 1:
|
| 565 |
+
warnings.warn(
|
| 566 |
+
"Multiprocessing-backed parallel loops cannot be nested"
|
| 567 |
+
" below threads, setting n_jobs=1",
|
| 568 |
+
stacklevel=3,
|
| 569 |
+
)
|
| 570 |
+
return 1
|
| 571 |
+
|
| 572 |
+
return super(MultiprocessingBackend, self).effective_n_jobs(n_jobs)
|
| 573 |
+
|
| 574 |
+
def configure(
|
| 575 |
+
self,
|
| 576 |
+
n_jobs=1,
|
| 577 |
+
parallel=None,
|
| 578 |
+
prefer=None,
|
| 579 |
+
require=None,
|
| 580 |
+
**memmapping_pool_kwargs,
|
| 581 |
+
):
|
| 582 |
+
"""Build a process or thread pool and return the number of workers"""
|
| 583 |
+
n_jobs = self.effective_n_jobs(n_jobs)
|
| 584 |
+
if n_jobs == 1:
|
| 585 |
+
raise FallbackToBackend(SequentialBackend(nesting_level=self.nesting_level))
|
| 586 |
+
|
| 587 |
+
memmapping_pool_kwargs = {
|
| 588 |
+
**self.backend_kwargs,
|
| 589 |
+
**memmapping_pool_kwargs,
|
| 590 |
+
}
|
| 591 |
+
|
| 592 |
+
# Make sure to free as much memory as possible before forking
|
| 593 |
+
gc.collect()
|
| 594 |
+
self._pool = MemmappingPool(n_jobs, **memmapping_pool_kwargs)
|
| 595 |
+
self.parallel = parallel
|
| 596 |
+
return n_jobs
|
| 597 |
+
|
| 598 |
+
def terminate(self):
|
| 599 |
+
"""Shutdown the process or thread pool"""
|
| 600 |
+
super(MultiprocessingBackend, self).terminate()
|
| 601 |
+
self.reset_batch_stats()
|
| 602 |
+
|
| 603 |
+
|
| 604 |
+
class LokyBackend(AutoBatchingMixin, ParallelBackendBase):
|
| 605 |
+
"""Managing pool of workers with loky instead of multiprocessing."""
|
| 606 |
+
|
| 607 |
+
supports_retrieve_callback = True
|
| 608 |
+
supports_inner_max_num_threads = True
|
| 609 |
+
|
| 610 |
+
def configure(
|
| 611 |
+
self,
|
| 612 |
+
n_jobs=1,
|
| 613 |
+
parallel=None,
|
| 614 |
+
prefer=None,
|
| 615 |
+
require=None,
|
| 616 |
+
idle_worker_timeout=None,
|
| 617 |
+
**memmapping_executor_kwargs,
|
| 618 |
+
):
|
| 619 |
+
"""Build a process executor and return the number of workers"""
|
| 620 |
+
n_jobs = self.effective_n_jobs(n_jobs)
|
| 621 |
+
if n_jobs == 1:
|
| 622 |
+
raise FallbackToBackend(SequentialBackend(nesting_level=self.nesting_level))
|
| 623 |
+
|
| 624 |
+
memmapping_executor_kwargs = {
|
| 625 |
+
**self.backend_kwargs,
|
| 626 |
+
**memmapping_executor_kwargs,
|
| 627 |
+
}
|
| 628 |
+
|
| 629 |
+
# Prohibit the use of 'timeout' in the LokyBackend, as 'idle_worker_timeout'
|
| 630 |
+
# better describes the backend's behavior.
|
| 631 |
+
if "timeout" in memmapping_executor_kwargs:
|
| 632 |
+
raise ValueError(
|
| 633 |
+
"The 'timeout' parameter is not supported by the LokyBackend. "
|
| 634 |
+
"Please use the `idle_worker_timeout` parameter instead."
|
| 635 |
+
)
|
| 636 |
+
if idle_worker_timeout is None:
|
| 637 |
+
idle_worker_timeout = self.backend_kwargs.get("idle_worker_timeout", 300)
|
| 638 |
+
|
| 639 |
+
self._workers = get_memmapping_executor(
|
| 640 |
+
n_jobs,
|
| 641 |
+
timeout=idle_worker_timeout,
|
| 642 |
+
env=self._prepare_worker_env(n_jobs=n_jobs),
|
| 643 |
+
context_id=parallel._id,
|
| 644 |
+
**memmapping_executor_kwargs,
|
| 645 |
+
)
|
| 646 |
+
self.parallel = parallel
|
| 647 |
+
return n_jobs
|
| 648 |
+
|
| 649 |
+
def effective_n_jobs(self, n_jobs):
|
| 650 |
+
"""Determine the number of jobs which are going to run in parallel"""
|
| 651 |
+
if n_jobs == 0:
|
| 652 |
+
raise ValueError("n_jobs == 0 in Parallel has no meaning")
|
| 653 |
+
elif mp is None or n_jobs is None:
|
| 654 |
+
# multiprocessing is not available or disabled, fallback
|
| 655 |
+
# to sequential mode
|
| 656 |
+
return 1
|
| 657 |
+
elif mp.current_process().daemon:
|
| 658 |
+
# Daemonic processes cannot have children
|
| 659 |
+
if n_jobs != 1:
|
| 660 |
+
if inside_dask_worker():
|
| 661 |
+
msg = (
|
| 662 |
+
"Inside a Dask worker with daemon=True, "
|
| 663 |
+
"setting n_jobs=1.\nPossible work-arounds:\n"
|
| 664 |
+
"- dask.config.set("
|
| 665 |
+
"{'distributed.worker.daemon': False})\n"
|
| 666 |
+
"- set the environment variable "
|
| 667 |
+
"DASK_DISTRIBUTED__WORKER__DAEMON=False\n"
|
| 668 |
+
"before creating your Dask cluster."
|
| 669 |
+
)
|
| 670 |
+
else:
|
| 671 |
+
msg = (
|
| 672 |
+
"Loky-backed parallel loops cannot be called in a"
|
| 673 |
+
" multiprocessing, setting n_jobs=1"
|
| 674 |
+
)
|
| 675 |
+
warnings.warn(msg, stacklevel=3)
|
| 676 |
+
|
| 677 |
+
return 1
|
| 678 |
+
elif not (self.in_main_thread() or self.nesting_level == 0):
|
| 679 |
+
# Prevent posix fork inside in non-main posix threads
|
| 680 |
+
if n_jobs != 1:
|
| 681 |
+
warnings.warn(
|
| 682 |
+
"Loky-backed parallel loops cannot be nested below "
|
| 683 |
+
"threads, setting n_jobs=1",
|
| 684 |
+
stacklevel=3,
|
| 685 |
+
)
|
| 686 |
+
return 1
|
| 687 |
+
elif n_jobs < 0:
|
| 688 |
+
n_jobs = max(cpu_count() + 1 + n_jobs, 1)
|
| 689 |
+
return n_jobs
|
| 690 |
+
|
| 691 |
+
def submit(self, func, callback=None):
|
| 692 |
+
"""Schedule a func to be run"""
|
| 693 |
+
future = self._workers.submit(func)
|
| 694 |
+
if callback is not None:
|
| 695 |
+
future.add_done_callback(callback)
|
| 696 |
+
return future
|
| 697 |
+
|
| 698 |
+
def retrieve_result_callback(self, future):
|
| 699 |
+
"""Retrieve the result, here out is the future given by submit"""
|
| 700 |
+
try:
|
| 701 |
+
return future.result()
|
| 702 |
+
except ShutdownExecutorError:
|
| 703 |
+
raise RuntimeError(
|
| 704 |
+
"The executor underlying Parallel has been shutdown. "
|
| 705 |
+
"This is likely due to the garbage collection of a previous "
|
| 706 |
+
"generator from a call to Parallel with return_as='generator'."
|
| 707 |
+
" Make sure the generator is not garbage collected when "
|
| 708 |
+
"submitting a new job or that it is first properly exhausted."
|
| 709 |
+
)
|
| 710 |
+
|
| 711 |
+
def terminate(self):
|
| 712 |
+
if self._workers is not None:
|
| 713 |
+
# Don't terminate the workers as we want to reuse them in later
|
| 714 |
+
# calls, but cleanup the temporary resources that the Parallel call
|
| 715 |
+
# created. This 'hack' requires a private, low-level operation.
|
| 716 |
+
self._workers._temp_folder_manager._clean_temporary_resources(
|
| 717 |
+
context_id=self.parallel._id, force=False
|
| 718 |
+
)
|
| 719 |
+
self._workers = None
|
| 720 |
+
|
| 721 |
+
self.reset_batch_stats()
|
| 722 |
+
|
| 723 |
+
def abort_everything(self, ensure_ready=True):
|
| 724 |
+
"""Shutdown the workers and restart a new one with the same parameters"""
|
| 725 |
+
self._workers.terminate(kill_workers=True)
|
| 726 |
+
self._workers = None
|
| 727 |
+
|
| 728 |
+
if ensure_ready:
|
| 729 |
+
self.configure(n_jobs=self.parallel.n_jobs, parallel=self.parallel)
|
| 730 |
+
|
| 731 |
+
|
| 732 |
+
class FallbackToBackend(Exception):
|
| 733 |
+
"""Raised when configuration should fallback to another backend"""
|
| 734 |
+
|
| 735 |
+
def __init__(self, backend):
|
| 736 |
+
self.backend = backend
|
| 737 |
+
|
| 738 |
+
|
| 739 |
+
def inside_dask_worker():
|
| 740 |
+
"""Check whether the current function is executed inside a Dask worker."""
|
| 741 |
+
# This function can not be in joblib._dask because there would be a
|
| 742 |
+
# circular import:
|
| 743 |
+
# _dask imports _parallel_backend that imports _dask ...
|
| 744 |
+
try:
|
| 745 |
+
from distributed import get_worker
|
| 746 |
+
except ImportError:
|
| 747 |
+
return False
|
| 748 |
+
|
| 749 |
+
try:
|
| 750 |
+
get_worker()
|
| 751 |
+
return True
|
| 752 |
+
except ValueError:
|
| 753 |
+
return False
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_store_backends.py
ADDED
|
@@ -0,0 +1,500 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Storage providers backends for Memory caching."""
|
| 2 |
+
|
| 3 |
+
import collections
|
| 4 |
+
import datetime
|
| 5 |
+
import json
|
| 6 |
+
import operator
|
| 7 |
+
import os
|
| 8 |
+
import os.path
|
| 9 |
+
import re
|
| 10 |
+
import shutil
|
| 11 |
+
import threading
|
| 12 |
+
import time
|
| 13 |
+
import uuid
|
| 14 |
+
import warnings
|
| 15 |
+
from abc import ABCMeta, abstractmethod
|
| 16 |
+
from pickle import PicklingError
|
| 17 |
+
|
| 18 |
+
from . import numpy_pickle
|
| 19 |
+
from .backports import concurrency_safe_rename
|
| 20 |
+
from .disk import memstr_to_bytes, mkdirp, rm_subdirs
|
| 21 |
+
from .logger import format_time
|
| 22 |
+
|
| 23 |
+
CacheItemInfo = collections.namedtuple("CacheItemInfo", "path size last_access")
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
class CacheWarning(Warning):
|
| 27 |
+
"""Warning to capture dump failures except for PicklingError."""
|
| 28 |
+
|
| 29 |
+
pass
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def concurrency_safe_write(object_to_write, filename, write_func):
|
| 33 |
+
"""Writes an object into a unique file in a concurrency-safe way."""
|
| 34 |
+
# Temporary name is composed of UUID, process_id and thread_id to avoid
|
| 35 |
+
# collisions due to concurrent write.
|
| 36 |
+
# UUID is unique across nodes and time and help avoid collisions, even if
|
| 37 |
+
# the cache folder is shared by several Python processes with the same pid and
|
| 38 |
+
# thread id on different nodes of a cluster for instance.
|
| 39 |
+
thread_id = id(threading.current_thread())
|
| 40 |
+
temporary_filename = f"{filename}.{uuid.uuid4().hex}-{os.getpid()}-{thread_id}"
|
| 41 |
+
|
| 42 |
+
write_func(object_to_write, temporary_filename)
|
| 43 |
+
|
| 44 |
+
return temporary_filename
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
class StoreBackendBase(metaclass=ABCMeta):
|
| 48 |
+
"""Helper Abstract Base Class which defines all methods that
|
| 49 |
+
a StorageBackend must implement."""
|
| 50 |
+
|
| 51 |
+
location = None
|
| 52 |
+
|
| 53 |
+
@abstractmethod
|
| 54 |
+
def _open_item(self, f, mode):
|
| 55 |
+
"""Opens an item on the store and return a file-like object.
|
| 56 |
+
|
| 57 |
+
This method is private and only used by the StoreBackendMixin object.
|
| 58 |
+
|
| 59 |
+
Parameters
|
| 60 |
+
----------
|
| 61 |
+
f: a file-like object
|
| 62 |
+
The file-like object where an item is stored and retrieved
|
| 63 |
+
mode: string, optional
|
| 64 |
+
the mode in which the file-like object is opened allowed valued are
|
| 65 |
+
'rb', 'wb'
|
| 66 |
+
|
| 67 |
+
Returns
|
| 68 |
+
-------
|
| 69 |
+
a file-like object
|
| 70 |
+
"""
|
| 71 |
+
|
| 72 |
+
@abstractmethod
|
| 73 |
+
def _item_exists(self, location):
|
| 74 |
+
"""Checks if an item location exists in the store.
|
| 75 |
+
|
| 76 |
+
This method is private and only used by the StoreBackendMixin object.
|
| 77 |
+
|
| 78 |
+
Parameters
|
| 79 |
+
----------
|
| 80 |
+
location: string
|
| 81 |
+
The location of an item. On a filesystem, this corresponds to the
|
| 82 |
+
absolute path, including the filename, of a file.
|
| 83 |
+
|
| 84 |
+
Returns
|
| 85 |
+
-------
|
| 86 |
+
True if the item exists, False otherwise
|
| 87 |
+
"""
|
| 88 |
+
|
| 89 |
+
@abstractmethod
|
| 90 |
+
def _move_item(self, src, dst):
|
| 91 |
+
"""Moves an item from src to dst in the store.
|
| 92 |
+
|
| 93 |
+
This method is private and only used by the StoreBackendMixin object.
|
| 94 |
+
|
| 95 |
+
Parameters
|
| 96 |
+
----------
|
| 97 |
+
src: string
|
| 98 |
+
The source location of an item
|
| 99 |
+
dst: string
|
| 100 |
+
The destination location of an item
|
| 101 |
+
"""
|
| 102 |
+
|
| 103 |
+
@abstractmethod
|
| 104 |
+
def create_location(self, location):
|
| 105 |
+
"""Creates a location on the store.
|
| 106 |
+
|
| 107 |
+
Parameters
|
| 108 |
+
----------
|
| 109 |
+
location: string
|
| 110 |
+
The location in the store. On a filesystem, this corresponds to a
|
| 111 |
+
directory.
|
| 112 |
+
"""
|
| 113 |
+
|
| 114 |
+
@abstractmethod
|
| 115 |
+
def clear_location(self, location):
|
| 116 |
+
"""Clears a location on the store.
|
| 117 |
+
|
| 118 |
+
Parameters
|
| 119 |
+
----------
|
| 120 |
+
location: string
|
| 121 |
+
The location in the store. On a filesystem, this corresponds to a
|
| 122 |
+
directory or a filename absolute path
|
| 123 |
+
"""
|
| 124 |
+
|
| 125 |
+
@abstractmethod
|
| 126 |
+
def get_items(self):
|
| 127 |
+
"""Returns the whole list of items available in the store.
|
| 128 |
+
|
| 129 |
+
Returns
|
| 130 |
+
-------
|
| 131 |
+
The list of items identified by their ids (e.g filename in a
|
| 132 |
+
filesystem).
|
| 133 |
+
"""
|
| 134 |
+
|
| 135 |
+
@abstractmethod
|
| 136 |
+
def configure(self, location, verbose=0, backend_options=dict()):
|
| 137 |
+
"""Configures the store.
|
| 138 |
+
|
| 139 |
+
Parameters
|
| 140 |
+
----------
|
| 141 |
+
location: string
|
| 142 |
+
The base location used by the store. On a filesystem, this
|
| 143 |
+
corresponds to a directory.
|
| 144 |
+
verbose: int
|
| 145 |
+
The level of verbosity of the store
|
| 146 |
+
backend_options: dict
|
| 147 |
+
Contains a dictionary of named parameters used to configure the
|
| 148 |
+
store backend.
|
| 149 |
+
"""
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
class StoreBackendMixin(object):
|
| 153 |
+
"""Class providing all logic for managing the store in a generic way.
|
| 154 |
+
|
| 155 |
+
The StoreBackend subclass has to implement 3 methods: create_location,
|
| 156 |
+
clear_location and configure. The StoreBackend also has to provide
|
| 157 |
+
a private _open_item, _item_exists and _move_item methods. The _open_item
|
| 158 |
+
method has to have the same signature as the builtin open and return a
|
| 159 |
+
file-like object.
|
| 160 |
+
"""
|
| 161 |
+
|
| 162 |
+
def load_item(self, call_id, verbose=1, timestamp=None, metadata=None):
|
| 163 |
+
"""Load an item from the store given its id as a list of str."""
|
| 164 |
+
full_path = os.path.join(self.location, *call_id)
|
| 165 |
+
|
| 166 |
+
if verbose > 1:
|
| 167 |
+
ts_string = (
|
| 168 |
+
"{: <16}".format(format_time(time.time() - timestamp))
|
| 169 |
+
if timestamp is not None
|
| 170 |
+
else ""
|
| 171 |
+
)
|
| 172 |
+
signature = os.path.basename(call_id[0])
|
| 173 |
+
if metadata is not None and "input_args" in metadata:
|
| 174 |
+
kwargs = ", ".join(
|
| 175 |
+
"{}={}".format(*item) for item in metadata["input_args"].items()
|
| 176 |
+
)
|
| 177 |
+
signature += "({})".format(kwargs)
|
| 178 |
+
msg = "[Memory]{}: Loading {}".format(ts_string, signature)
|
| 179 |
+
if verbose < 10:
|
| 180 |
+
print("{0}...".format(msg))
|
| 181 |
+
else:
|
| 182 |
+
print("{0} from {1}".format(msg, full_path))
|
| 183 |
+
|
| 184 |
+
mmap_mode = None if not hasattr(self, "mmap_mode") else self.mmap_mode
|
| 185 |
+
|
| 186 |
+
filename = os.path.join(full_path, "output.pkl")
|
| 187 |
+
if not self._item_exists(filename):
|
| 188 |
+
raise KeyError(
|
| 189 |
+
"Non-existing item (may have been "
|
| 190 |
+
"cleared).\nFile %s does not exist" % filename
|
| 191 |
+
)
|
| 192 |
+
|
| 193 |
+
# file-like object cannot be used when mmap_mode is set
|
| 194 |
+
if mmap_mode is None:
|
| 195 |
+
with self._open_item(filename, "rb") as f:
|
| 196 |
+
item = numpy_pickle.load(f)
|
| 197 |
+
else:
|
| 198 |
+
item = numpy_pickle.load(filename, mmap_mode=mmap_mode)
|
| 199 |
+
return item
|
| 200 |
+
|
| 201 |
+
def dump_item(self, call_id, item, verbose=1):
|
| 202 |
+
"""Dump an item in the store at the id given as a list of str."""
|
| 203 |
+
try:
|
| 204 |
+
item_path = os.path.join(self.location, *call_id)
|
| 205 |
+
if not self._item_exists(item_path):
|
| 206 |
+
self.create_location(item_path)
|
| 207 |
+
filename = os.path.join(item_path, "output.pkl")
|
| 208 |
+
if verbose > 10:
|
| 209 |
+
print("Persisting in %s" % item_path)
|
| 210 |
+
|
| 211 |
+
def write_func(to_write, dest_filename):
|
| 212 |
+
with self._open_item(dest_filename, "wb") as f:
|
| 213 |
+
try:
|
| 214 |
+
numpy_pickle.dump(to_write, f, compress=self.compress)
|
| 215 |
+
except PicklingError as e:
|
| 216 |
+
# TODO(1.5) turn into error
|
| 217 |
+
warnings.warn(
|
| 218 |
+
"Unable to cache to disk: failed to pickle "
|
| 219 |
+
"output. In version 1.5 this will raise an "
|
| 220 |
+
f"exception. Exception: {e}.",
|
| 221 |
+
FutureWarning,
|
| 222 |
+
)
|
| 223 |
+
|
| 224 |
+
self._concurrency_safe_write(item, filename, write_func)
|
| 225 |
+
except Exception as e: # noqa: E722
|
| 226 |
+
warnings.warn(
|
| 227 |
+
"Unable to cache to disk. Possibly a race condition in the "
|
| 228 |
+
f"creation of the directory. Exception: {e}.",
|
| 229 |
+
CacheWarning,
|
| 230 |
+
)
|
| 231 |
+
|
| 232 |
+
def clear_item(self, call_id):
|
| 233 |
+
"""Clear the item at the id, given as a list of str."""
|
| 234 |
+
item_path = os.path.join(self.location, *call_id)
|
| 235 |
+
if self._item_exists(item_path):
|
| 236 |
+
self.clear_location(item_path)
|
| 237 |
+
|
| 238 |
+
def contains_item(self, call_id):
|
| 239 |
+
"""Check if there is an item at the id, given as a list of str."""
|
| 240 |
+
item_path = os.path.join(self.location, *call_id)
|
| 241 |
+
filename = os.path.join(item_path, "output.pkl")
|
| 242 |
+
|
| 243 |
+
return self._item_exists(filename)
|
| 244 |
+
|
| 245 |
+
def get_item_info(self, call_id):
|
| 246 |
+
"""Return information about item."""
|
| 247 |
+
return {"location": os.path.join(self.location, *call_id)}
|
| 248 |
+
|
| 249 |
+
def get_metadata(self, call_id):
|
| 250 |
+
"""Return actual metadata of an item."""
|
| 251 |
+
try:
|
| 252 |
+
item_path = os.path.join(self.location, *call_id)
|
| 253 |
+
filename = os.path.join(item_path, "metadata.json")
|
| 254 |
+
with self._open_item(filename, "rb") as f:
|
| 255 |
+
return json.loads(f.read().decode("utf-8"))
|
| 256 |
+
except: # noqa: E722
|
| 257 |
+
return {}
|
| 258 |
+
|
| 259 |
+
def store_metadata(self, call_id, metadata):
|
| 260 |
+
"""Store metadata of a computation."""
|
| 261 |
+
try:
|
| 262 |
+
item_path = os.path.join(self.location, *call_id)
|
| 263 |
+
self.create_location(item_path)
|
| 264 |
+
filename = os.path.join(item_path, "metadata.json")
|
| 265 |
+
|
| 266 |
+
def write_func(to_write, dest_filename):
|
| 267 |
+
with self._open_item(dest_filename, "wb") as f:
|
| 268 |
+
f.write(json.dumps(to_write).encode("utf-8"))
|
| 269 |
+
|
| 270 |
+
self._concurrency_safe_write(metadata, filename, write_func)
|
| 271 |
+
except: # noqa: E722
|
| 272 |
+
pass
|
| 273 |
+
|
| 274 |
+
def contains_path(self, call_id):
|
| 275 |
+
"""Check cached function is available in store."""
|
| 276 |
+
func_path = os.path.join(self.location, *call_id)
|
| 277 |
+
return self.object_exists(func_path)
|
| 278 |
+
|
| 279 |
+
def clear_path(self, call_id):
|
| 280 |
+
"""Clear all items with a common path in the store."""
|
| 281 |
+
func_path = os.path.join(self.location, *call_id)
|
| 282 |
+
if self._item_exists(func_path):
|
| 283 |
+
self.clear_location(func_path)
|
| 284 |
+
|
| 285 |
+
def store_cached_func_code(self, call_id, func_code=None):
|
| 286 |
+
"""Store the code of the cached function."""
|
| 287 |
+
func_path = os.path.join(self.location, *call_id)
|
| 288 |
+
if not self._item_exists(func_path):
|
| 289 |
+
self.create_location(func_path)
|
| 290 |
+
|
| 291 |
+
if func_code is not None:
|
| 292 |
+
filename = os.path.join(func_path, "func_code.py")
|
| 293 |
+
with self._open_item(filename, "wb") as f:
|
| 294 |
+
f.write(func_code.encode("utf-8"))
|
| 295 |
+
|
| 296 |
+
def get_cached_func_code(self, call_id):
|
| 297 |
+
"""Store the code of the cached function."""
|
| 298 |
+
filename = os.path.join(self.location, *call_id, "func_code.py")
|
| 299 |
+
try:
|
| 300 |
+
with self._open_item(filename, "rb") as f:
|
| 301 |
+
return f.read().decode("utf-8")
|
| 302 |
+
except: # noqa: E722
|
| 303 |
+
raise
|
| 304 |
+
|
| 305 |
+
def get_cached_func_info(self, call_id):
|
| 306 |
+
"""Return information related to the cached function if it exists."""
|
| 307 |
+
return {"location": os.path.join(self.location, *call_id)}
|
| 308 |
+
|
| 309 |
+
def clear(self):
|
| 310 |
+
"""Clear the whole store content."""
|
| 311 |
+
self.clear_location(self.location)
|
| 312 |
+
|
| 313 |
+
def enforce_store_limits(self, bytes_limit, items_limit=None, age_limit=None):
|
| 314 |
+
"""
|
| 315 |
+
Remove the store's oldest files to enforce item, byte, and age limits.
|
| 316 |
+
"""
|
| 317 |
+
items_to_delete = self._get_items_to_delete(bytes_limit, items_limit, age_limit)
|
| 318 |
+
|
| 319 |
+
for item in items_to_delete:
|
| 320 |
+
if self.verbose > 10:
|
| 321 |
+
print("Deleting item {0}".format(item))
|
| 322 |
+
try:
|
| 323 |
+
self.clear_location(item.path)
|
| 324 |
+
except OSError:
|
| 325 |
+
# Even with ignore_errors=True shutil.rmtree can raise OSError
|
| 326 |
+
# with:
|
| 327 |
+
# [Errno 116] Stale file handle if another process has deleted
|
| 328 |
+
# the folder already.
|
| 329 |
+
pass
|
| 330 |
+
|
| 331 |
+
def _get_items_to_delete(self, bytes_limit, items_limit=None, age_limit=None):
|
| 332 |
+
"""
|
| 333 |
+
Get items to delete to keep the store under size, file, & age limits.
|
| 334 |
+
"""
|
| 335 |
+
if isinstance(bytes_limit, str):
|
| 336 |
+
bytes_limit = memstr_to_bytes(bytes_limit)
|
| 337 |
+
|
| 338 |
+
items = self.get_items()
|
| 339 |
+
if not items:
|
| 340 |
+
return []
|
| 341 |
+
|
| 342 |
+
size = sum(item.size for item in items)
|
| 343 |
+
|
| 344 |
+
if bytes_limit is not None:
|
| 345 |
+
to_delete_size = size - bytes_limit
|
| 346 |
+
else:
|
| 347 |
+
to_delete_size = 0
|
| 348 |
+
|
| 349 |
+
if items_limit is not None:
|
| 350 |
+
to_delete_items = len(items) - items_limit
|
| 351 |
+
else:
|
| 352 |
+
to_delete_items = 0
|
| 353 |
+
|
| 354 |
+
if age_limit is not None:
|
| 355 |
+
older_item = min(item.last_access for item in items)
|
| 356 |
+
if age_limit.total_seconds() < 0:
|
| 357 |
+
raise ValueError("age_limit has to be a positive timedelta")
|
| 358 |
+
deadline = datetime.datetime.now() - age_limit
|
| 359 |
+
else:
|
| 360 |
+
deadline = None
|
| 361 |
+
|
| 362 |
+
if (
|
| 363 |
+
to_delete_size <= 0
|
| 364 |
+
and to_delete_items <= 0
|
| 365 |
+
and (deadline is None or older_item > deadline)
|
| 366 |
+
):
|
| 367 |
+
return []
|
| 368 |
+
|
| 369 |
+
# We want to delete first the cache items that were accessed a
|
| 370 |
+
# long time ago
|
| 371 |
+
items.sort(key=operator.attrgetter("last_access"))
|
| 372 |
+
|
| 373 |
+
items_to_delete = []
|
| 374 |
+
size_so_far = 0
|
| 375 |
+
items_so_far = 0
|
| 376 |
+
|
| 377 |
+
for item in items:
|
| 378 |
+
if (
|
| 379 |
+
(size_so_far >= to_delete_size)
|
| 380 |
+
and items_so_far >= to_delete_items
|
| 381 |
+
and (deadline is None or deadline < item.last_access)
|
| 382 |
+
):
|
| 383 |
+
break
|
| 384 |
+
|
| 385 |
+
items_to_delete.append(item)
|
| 386 |
+
size_so_far += item.size
|
| 387 |
+
items_so_far += 1
|
| 388 |
+
|
| 389 |
+
return items_to_delete
|
| 390 |
+
|
| 391 |
+
def _concurrency_safe_write(self, to_write, filename, write_func):
|
| 392 |
+
"""Writes an object into a file in a concurrency-safe way."""
|
| 393 |
+
temporary_filename = concurrency_safe_write(to_write, filename, write_func)
|
| 394 |
+
self._move_item(temporary_filename, filename)
|
| 395 |
+
|
| 396 |
+
def __repr__(self):
|
| 397 |
+
"""Printable representation of the store location."""
|
| 398 |
+
return '{class_name}(location="{location}")'.format(
|
| 399 |
+
class_name=self.__class__.__name__, location=self.location
|
| 400 |
+
)
|
| 401 |
+
|
| 402 |
+
|
| 403 |
+
class FileSystemStoreBackend(StoreBackendBase, StoreBackendMixin):
|
| 404 |
+
"""A StoreBackend used with local or network file systems."""
|
| 405 |
+
|
| 406 |
+
_open_item = staticmethod(open)
|
| 407 |
+
_item_exists = staticmethod(os.path.exists)
|
| 408 |
+
_move_item = staticmethod(concurrency_safe_rename)
|
| 409 |
+
|
| 410 |
+
def clear_location(self, location):
|
| 411 |
+
"""Delete location on store."""
|
| 412 |
+
if location == self.location:
|
| 413 |
+
rm_subdirs(location)
|
| 414 |
+
else:
|
| 415 |
+
shutil.rmtree(location, ignore_errors=True)
|
| 416 |
+
|
| 417 |
+
def create_location(self, location):
|
| 418 |
+
"""Create object location on store"""
|
| 419 |
+
mkdirp(location)
|
| 420 |
+
|
| 421 |
+
def get_items(self):
|
| 422 |
+
"""Returns the whole list of items available in the store."""
|
| 423 |
+
items = []
|
| 424 |
+
|
| 425 |
+
for dirpath, _, filenames in os.walk(self.location):
|
| 426 |
+
is_cache_hash_dir = re.match("[a-f0-9]{32}", os.path.basename(dirpath))
|
| 427 |
+
|
| 428 |
+
if is_cache_hash_dir:
|
| 429 |
+
output_filename = os.path.join(dirpath, "output.pkl")
|
| 430 |
+
try:
|
| 431 |
+
last_access = os.path.getatime(output_filename)
|
| 432 |
+
except OSError:
|
| 433 |
+
try:
|
| 434 |
+
last_access = os.path.getatime(dirpath)
|
| 435 |
+
except OSError:
|
| 436 |
+
# The directory has already been deleted
|
| 437 |
+
continue
|
| 438 |
+
|
| 439 |
+
last_access = datetime.datetime.fromtimestamp(last_access)
|
| 440 |
+
try:
|
| 441 |
+
full_filenames = [os.path.join(dirpath, fn) for fn in filenames]
|
| 442 |
+
dirsize = sum(os.path.getsize(fn) for fn in full_filenames)
|
| 443 |
+
except OSError:
|
| 444 |
+
# Either output_filename or one of the files in
|
| 445 |
+
# dirpath does not exist any more. We assume this
|
| 446 |
+
# directory is being cleaned by another process already
|
| 447 |
+
continue
|
| 448 |
+
|
| 449 |
+
items.append(CacheItemInfo(dirpath, dirsize, last_access))
|
| 450 |
+
|
| 451 |
+
return items
|
| 452 |
+
|
| 453 |
+
def configure(self, location, verbose=1, backend_options=None):
|
| 454 |
+
"""Configure the store backend.
|
| 455 |
+
|
| 456 |
+
For this backend, valid store options are 'compress' and 'mmap_mode'
|
| 457 |
+
"""
|
| 458 |
+
if backend_options is None:
|
| 459 |
+
backend_options = {}
|
| 460 |
+
|
| 461 |
+
# setup location directory
|
| 462 |
+
self.location = location
|
| 463 |
+
if not os.path.exists(self.location):
|
| 464 |
+
mkdirp(self.location)
|
| 465 |
+
|
| 466 |
+
# Automatically add `.gitignore` file to the cache folder.
|
| 467 |
+
# XXX: the condition is necessary because in `Memory.__init__`, the user
|
| 468 |
+
# passed `location` param is modified to be either `{location}` or
|
| 469 |
+
# `{location}/joblib` depending on input type (`pathlib.Path` vs `str`).
|
| 470 |
+
# The proper resolution of this inconsistency is tracked in:
|
| 471 |
+
# https://github.com/joblib/joblib/issues/1684
|
| 472 |
+
cache_directory = (
|
| 473 |
+
os.path.dirname(location)
|
| 474 |
+
if os.path.dirname(location) and os.path.basename(location) == "joblib"
|
| 475 |
+
else location
|
| 476 |
+
)
|
| 477 |
+
gitignore = os.path.join(cache_directory, ".gitignore")
|
| 478 |
+
if not os.path.exists(gitignore):
|
| 479 |
+
try:
|
| 480 |
+
with open(gitignore, "w") as file:
|
| 481 |
+
file.write("# Created by joblib automatically.\n")
|
| 482 |
+
file.write("*\n")
|
| 483 |
+
except OSError as e:
|
| 484 |
+
warnings.warn(f"Unable to write {gitignore}. Exception: {e}.")
|
| 485 |
+
|
| 486 |
+
# item can be stored compressed for faster I/O
|
| 487 |
+
self.compress = backend_options.get("compress", False)
|
| 488 |
+
|
| 489 |
+
# FileSystemStoreBackend can be used with mmap_mode options under
|
| 490 |
+
# certain conditions.
|
| 491 |
+
mmap_mode = backend_options.get("mmap_mode")
|
| 492 |
+
if self.compress and mmap_mode is not None:
|
| 493 |
+
warnings.warn(
|
| 494 |
+
"Compressed items cannot be memmapped in a "
|
| 495 |
+
"filesystem store. Option will be ignored.",
|
| 496 |
+
stacklevel=2,
|
| 497 |
+
)
|
| 498 |
+
|
| 499 |
+
self.mmap_mode = mmap_mode
|
| 500 |
+
self.verbose = verbose
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/_utils.py
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Adapted from https://stackoverflow.com/a/9558001/2536294
|
| 2 |
+
|
| 3 |
+
import ast
|
| 4 |
+
import functools
|
| 5 |
+
import operator as op
|
| 6 |
+
from dataclasses import dataclass
|
| 7 |
+
|
| 8 |
+
from ._multiprocessing_helpers import mp
|
| 9 |
+
|
| 10 |
+
if mp is not None:
|
| 11 |
+
from .externals.loky.process_executor import _ExceptionWithTraceback
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
# supported operators
|
| 15 |
+
operators = {
|
| 16 |
+
ast.Add: op.add,
|
| 17 |
+
ast.Sub: op.sub,
|
| 18 |
+
ast.Mult: op.mul,
|
| 19 |
+
ast.Div: op.truediv,
|
| 20 |
+
ast.FloorDiv: op.floordiv,
|
| 21 |
+
ast.Mod: op.mod,
|
| 22 |
+
ast.Pow: op.pow,
|
| 23 |
+
ast.USub: op.neg,
|
| 24 |
+
}
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def eval_expr(expr):
|
| 28 |
+
"""Somewhat safely evaluate an arithmetic expression.
|
| 29 |
+
|
| 30 |
+
>>> eval_expr('2*6')
|
| 31 |
+
12
|
| 32 |
+
>>> eval_expr('2**6')
|
| 33 |
+
64
|
| 34 |
+
>>> eval_expr('1 + 2*3**(4) / (6 + -7)')
|
| 35 |
+
-161.0
|
| 36 |
+
|
| 37 |
+
Raises ValueError if the expression is invalid, too long
|
| 38 |
+
or its computation involves too large values.
|
| 39 |
+
"""
|
| 40 |
+
# Restrict the length of the expression to avoid potential Python crashes
|
| 41 |
+
# as per the documentation of ast.parse.
|
| 42 |
+
max_length = 30
|
| 43 |
+
if len(expr) > max_length:
|
| 44 |
+
raise ValueError(
|
| 45 |
+
f"Expression {expr[:max_length]!r}... is too long. "
|
| 46 |
+
f"Max length is {max_length}, got {len(expr)}."
|
| 47 |
+
)
|
| 48 |
+
try:
|
| 49 |
+
return eval_(ast.parse(expr, mode="eval").body)
|
| 50 |
+
except (TypeError, SyntaxError, OverflowError, KeyError) as e:
|
| 51 |
+
raise ValueError(
|
| 52 |
+
f"{expr!r} is not a valid or supported arithmetic expression."
|
| 53 |
+
) from e
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def limit(max_=None):
|
| 57 |
+
"""Return decorator that limits allowed returned values."""
|
| 58 |
+
|
| 59 |
+
def decorator(func):
|
| 60 |
+
@functools.wraps(func)
|
| 61 |
+
def wrapper(*args, **kwargs):
|
| 62 |
+
ret = func(*args, **kwargs)
|
| 63 |
+
try:
|
| 64 |
+
mag = abs(ret)
|
| 65 |
+
except TypeError:
|
| 66 |
+
pass # not applicable
|
| 67 |
+
else:
|
| 68 |
+
if mag > max_:
|
| 69 |
+
raise ValueError(
|
| 70 |
+
f"Numeric literal {ret} is too large, max is {max_}."
|
| 71 |
+
)
|
| 72 |
+
return ret
|
| 73 |
+
|
| 74 |
+
return wrapper
|
| 75 |
+
|
| 76 |
+
return decorator
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
@limit(max_=10**6)
|
| 80 |
+
def eval_(node):
|
| 81 |
+
if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
|
| 82 |
+
return node.value
|
| 83 |
+
elif isinstance(node, ast.BinOp): # <left> <operator> <right>
|
| 84 |
+
return operators[type(node.op)](eval_(node.left), eval_(node.right))
|
| 85 |
+
elif isinstance(node, ast.UnaryOp): # <operator> <operand> e.g., -1
|
| 86 |
+
return operators[type(node.op)](eval_(node.operand))
|
| 87 |
+
else:
|
| 88 |
+
raise TypeError(node)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
@dataclass(frozen=True)
|
| 92 |
+
class _Sentinel:
|
| 93 |
+
"""A sentinel to mark a parameter as not explicitly set"""
|
| 94 |
+
|
| 95 |
+
default_value: object
|
| 96 |
+
|
| 97 |
+
def __repr__(self):
|
| 98 |
+
return f"default({self.default_value!r})"
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
class _TracebackCapturingWrapper:
|
| 102 |
+
"""Protect function call and return error with traceback."""
|
| 103 |
+
|
| 104 |
+
def __init__(self, func):
|
| 105 |
+
self.func = func
|
| 106 |
+
|
| 107 |
+
def __call__(self, **kwargs):
|
| 108 |
+
try:
|
| 109 |
+
return self.func(**kwargs)
|
| 110 |
+
except BaseException as e:
|
| 111 |
+
return _ExceptionWithTraceback(e)
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def _retrieve_traceback_capturing_wrapped_call(out):
|
| 115 |
+
if isinstance(out, _ExceptionWithTraceback):
|
| 116 |
+
rebuild, args = out.__reduce__()
|
| 117 |
+
out = rebuild(*args)
|
| 118 |
+
if isinstance(out, BaseException):
|
| 119 |
+
raise out
|
| 120 |
+
return out
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/backports.py
ADDED
|
@@ -0,0 +1,195 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Backports of fixes for joblib dependencies
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import os
|
| 6 |
+
import re
|
| 7 |
+
import time
|
| 8 |
+
from multiprocessing import util
|
| 9 |
+
from os.path import basename
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
class Version:
|
| 13 |
+
"""Backport from deprecated distutils
|
| 14 |
+
|
| 15 |
+
We maintain this backport to avoid introducing a new dependency on
|
| 16 |
+
`packaging`.
|
| 17 |
+
|
| 18 |
+
We might rexplore this choice in the future if all major Python projects
|
| 19 |
+
introduce a dependency on packaging anyway.
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
def __init__(self, vstring=None):
|
| 23 |
+
if vstring:
|
| 24 |
+
self.parse(vstring)
|
| 25 |
+
|
| 26 |
+
def __repr__(self):
|
| 27 |
+
return "%s ('%s')" % (self.__class__.__name__, str(self))
|
| 28 |
+
|
| 29 |
+
def __eq__(self, other):
|
| 30 |
+
c = self._cmp(other)
|
| 31 |
+
if c is NotImplemented:
|
| 32 |
+
return c
|
| 33 |
+
return c == 0
|
| 34 |
+
|
| 35 |
+
def __lt__(self, other):
|
| 36 |
+
c = self._cmp(other)
|
| 37 |
+
if c is NotImplemented:
|
| 38 |
+
return c
|
| 39 |
+
return c < 0
|
| 40 |
+
|
| 41 |
+
def __le__(self, other):
|
| 42 |
+
c = self._cmp(other)
|
| 43 |
+
if c is NotImplemented:
|
| 44 |
+
return c
|
| 45 |
+
return c <= 0
|
| 46 |
+
|
| 47 |
+
def __gt__(self, other):
|
| 48 |
+
c = self._cmp(other)
|
| 49 |
+
if c is NotImplemented:
|
| 50 |
+
return c
|
| 51 |
+
return c > 0
|
| 52 |
+
|
| 53 |
+
def __ge__(self, other):
|
| 54 |
+
c = self._cmp(other)
|
| 55 |
+
if c is NotImplemented:
|
| 56 |
+
return c
|
| 57 |
+
return c >= 0
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
class LooseVersion(Version):
|
| 61 |
+
"""Backport from deprecated distutils
|
| 62 |
+
|
| 63 |
+
We maintain this backport to avoid introducing a new dependency on
|
| 64 |
+
`packaging`.
|
| 65 |
+
|
| 66 |
+
We might rexplore this choice in the future if all major Python projects
|
| 67 |
+
introduce a dependency on packaging anyway.
|
| 68 |
+
"""
|
| 69 |
+
|
| 70 |
+
component_re = re.compile(r"(\d+ | [a-z]+ | \.)", re.VERBOSE)
|
| 71 |
+
|
| 72 |
+
def __init__(self, vstring=None):
|
| 73 |
+
if vstring:
|
| 74 |
+
self.parse(vstring)
|
| 75 |
+
|
| 76 |
+
def parse(self, vstring):
|
| 77 |
+
# I've given up on thinking I can reconstruct the version string
|
| 78 |
+
# from the parsed tuple -- so I just store the string here for
|
| 79 |
+
# use by __str__
|
| 80 |
+
self.vstring = vstring
|
| 81 |
+
components = [x for x in self.component_re.split(vstring) if x and x != "."]
|
| 82 |
+
for i, obj in enumerate(components):
|
| 83 |
+
try:
|
| 84 |
+
components[i] = int(obj)
|
| 85 |
+
except ValueError:
|
| 86 |
+
pass
|
| 87 |
+
|
| 88 |
+
self.version = components
|
| 89 |
+
|
| 90 |
+
def __str__(self):
|
| 91 |
+
return self.vstring
|
| 92 |
+
|
| 93 |
+
def __repr__(self):
|
| 94 |
+
return "LooseVersion ('%s')" % str(self)
|
| 95 |
+
|
| 96 |
+
def _cmp(self, other):
|
| 97 |
+
if isinstance(other, str):
|
| 98 |
+
other = LooseVersion(other)
|
| 99 |
+
elif not isinstance(other, LooseVersion):
|
| 100 |
+
return NotImplemented
|
| 101 |
+
|
| 102 |
+
if self.version == other.version:
|
| 103 |
+
return 0
|
| 104 |
+
if self.version < other.version:
|
| 105 |
+
return -1
|
| 106 |
+
if self.version > other.version:
|
| 107 |
+
return 1
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
try:
|
| 111 |
+
import numpy as np
|
| 112 |
+
|
| 113 |
+
def make_memmap(
|
| 114 |
+
filename,
|
| 115 |
+
dtype="uint8",
|
| 116 |
+
mode="r+",
|
| 117 |
+
offset=0,
|
| 118 |
+
shape=None,
|
| 119 |
+
order="C",
|
| 120 |
+
unlink_on_gc_collect=False,
|
| 121 |
+
):
|
| 122 |
+
"""Custom memmap constructor compatible with numpy.memmap.
|
| 123 |
+
|
| 124 |
+
This function:
|
| 125 |
+
- is a backport the numpy memmap offset fix (See
|
| 126 |
+
https://github.com/numpy/numpy/pull/8443 for more details.
|
| 127 |
+
The numpy fix is available starting numpy 1.13)
|
| 128 |
+
- adds ``unlink_on_gc_collect``, which specifies explicitly whether
|
| 129 |
+
the process re-constructing the memmap owns a reference to the
|
| 130 |
+
underlying file. If set to True, it adds a finalizer to the
|
| 131 |
+
newly-created memmap that sends a maybe_unlink request for the
|
| 132 |
+
memmaped file to resource_tracker.
|
| 133 |
+
"""
|
| 134 |
+
util.debug(
|
| 135 |
+
"[MEMMAP READ] creating a memmap (shape {}, filename {}, pid {})".format(
|
| 136 |
+
shape, basename(filename), os.getpid()
|
| 137 |
+
)
|
| 138 |
+
)
|
| 139 |
+
|
| 140 |
+
mm = np.memmap(
|
| 141 |
+
filename, dtype=dtype, mode=mode, offset=offset, shape=shape, order=order
|
| 142 |
+
)
|
| 143 |
+
if LooseVersion(np.__version__) < "1.13":
|
| 144 |
+
mm.offset = offset
|
| 145 |
+
if unlink_on_gc_collect:
|
| 146 |
+
from ._memmapping_reducer import add_maybe_unlink_finalizer
|
| 147 |
+
|
| 148 |
+
add_maybe_unlink_finalizer(mm)
|
| 149 |
+
return mm
|
| 150 |
+
except ImportError:
|
| 151 |
+
|
| 152 |
+
def make_memmap(
|
| 153 |
+
filename,
|
| 154 |
+
dtype="uint8",
|
| 155 |
+
mode="r+",
|
| 156 |
+
offset=0,
|
| 157 |
+
shape=None,
|
| 158 |
+
order="C",
|
| 159 |
+
unlink_on_gc_collect=False,
|
| 160 |
+
):
|
| 161 |
+
raise NotImplementedError(
|
| 162 |
+
"'joblib.backports.make_memmap' should not be used "
|
| 163 |
+
"if numpy is not installed."
|
| 164 |
+
)
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
if os.name == "nt":
|
| 168 |
+
# https://github.com/joblib/joblib/issues/540
|
| 169 |
+
access_denied_errors = (5, 13)
|
| 170 |
+
from os import replace
|
| 171 |
+
|
| 172 |
+
def concurrency_safe_rename(src, dst):
|
| 173 |
+
"""Renames ``src`` into ``dst`` overwriting ``dst`` if it exists.
|
| 174 |
+
|
| 175 |
+
On Windows os.replace can yield permission errors if executed by two
|
| 176 |
+
different processes.
|
| 177 |
+
"""
|
| 178 |
+
max_sleep_time = 1
|
| 179 |
+
total_sleep_time = 0
|
| 180 |
+
sleep_time = 0.001
|
| 181 |
+
while total_sleep_time < max_sleep_time:
|
| 182 |
+
try:
|
| 183 |
+
replace(src, dst)
|
| 184 |
+
break
|
| 185 |
+
except Exception as exc:
|
| 186 |
+
if getattr(exc, "winerror", None) in access_denied_errors:
|
| 187 |
+
time.sleep(sleep_time)
|
| 188 |
+
total_sleep_time += sleep_time
|
| 189 |
+
sleep_time *= 2
|
| 190 |
+
else:
|
| 191 |
+
raise
|
| 192 |
+
else:
|
| 193 |
+
raise
|
| 194 |
+
else:
|
| 195 |
+
from os import replace as concurrency_safe_rename # noqa
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/compressor.py
ADDED
|
@@ -0,0 +1,572 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Classes and functions for managing compressors."""
|
| 2 |
+
|
| 3 |
+
import io
|
| 4 |
+
import zlib
|
| 5 |
+
|
| 6 |
+
from joblib.backports import LooseVersion
|
| 7 |
+
|
| 8 |
+
try:
|
| 9 |
+
from threading import RLock
|
| 10 |
+
except ImportError:
|
| 11 |
+
from dummy_threading import RLock
|
| 12 |
+
|
| 13 |
+
try:
|
| 14 |
+
import bz2
|
| 15 |
+
except ImportError:
|
| 16 |
+
bz2 = None
|
| 17 |
+
|
| 18 |
+
try:
|
| 19 |
+
import lz4
|
| 20 |
+
from lz4.frame import LZ4FrameFile
|
| 21 |
+
except ImportError:
|
| 22 |
+
lz4 = None
|
| 23 |
+
|
| 24 |
+
try:
|
| 25 |
+
import lzma
|
| 26 |
+
except ImportError:
|
| 27 |
+
lzma = None
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
LZ4_NOT_INSTALLED_ERROR = (
|
| 31 |
+
"LZ4 is not installed. Install it with pip: https://python-lz4.readthedocs.io/"
|
| 32 |
+
)
|
| 33 |
+
|
| 34 |
+
# Registered compressors
|
| 35 |
+
_COMPRESSORS = {}
|
| 36 |
+
|
| 37 |
+
# Magic numbers of supported compression file formats.
|
| 38 |
+
_ZFILE_PREFIX = b"ZF" # used with pickle files created before 0.9.3.
|
| 39 |
+
_ZLIB_PREFIX = b"\x78"
|
| 40 |
+
_GZIP_PREFIX = b"\x1f\x8b"
|
| 41 |
+
_BZ2_PREFIX = b"BZ"
|
| 42 |
+
_XZ_PREFIX = b"\xfd\x37\x7a\x58\x5a"
|
| 43 |
+
_LZMA_PREFIX = b"\x5d\x00"
|
| 44 |
+
_LZ4_PREFIX = b"\x04\x22\x4d\x18"
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def register_compressor(compressor_name, compressor, force=False):
|
| 48 |
+
"""Register a new compressor.
|
| 49 |
+
|
| 50 |
+
Parameters
|
| 51 |
+
----------
|
| 52 |
+
compressor_name: str.
|
| 53 |
+
The name of the compressor.
|
| 54 |
+
compressor: CompressorWrapper
|
| 55 |
+
An instance of a 'CompressorWrapper'.
|
| 56 |
+
"""
|
| 57 |
+
global _COMPRESSORS
|
| 58 |
+
if not isinstance(compressor_name, str):
|
| 59 |
+
raise ValueError(
|
| 60 |
+
"Compressor name should be a string, '{}' given.".format(compressor_name)
|
| 61 |
+
)
|
| 62 |
+
|
| 63 |
+
if not isinstance(compressor, CompressorWrapper):
|
| 64 |
+
raise ValueError(
|
| 65 |
+
"Compressor should implement the CompressorWrapper "
|
| 66 |
+
"interface, '{}' given.".format(compressor)
|
| 67 |
+
)
|
| 68 |
+
|
| 69 |
+
if compressor.fileobj_factory is not None and (
|
| 70 |
+
not hasattr(compressor.fileobj_factory, "read")
|
| 71 |
+
or not hasattr(compressor.fileobj_factory, "write")
|
| 72 |
+
or not hasattr(compressor.fileobj_factory, "seek")
|
| 73 |
+
or not hasattr(compressor.fileobj_factory, "tell")
|
| 74 |
+
):
|
| 75 |
+
raise ValueError(
|
| 76 |
+
"Compressor 'fileobj_factory' attribute should "
|
| 77 |
+
"implement the file object interface, '{}' given.".format(
|
| 78 |
+
compressor.fileobj_factory
|
| 79 |
+
)
|
| 80 |
+
)
|
| 81 |
+
|
| 82 |
+
if compressor_name in _COMPRESSORS and not force:
|
| 83 |
+
raise ValueError("Compressor '{}' already registered.".format(compressor_name))
|
| 84 |
+
|
| 85 |
+
_COMPRESSORS[compressor_name] = compressor
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
class CompressorWrapper:
|
| 89 |
+
"""A wrapper around a compressor file object.
|
| 90 |
+
|
| 91 |
+
Attributes
|
| 92 |
+
----------
|
| 93 |
+
obj: a file-like object
|
| 94 |
+
The object must implement the buffer interface and will be used
|
| 95 |
+
internally to compress/decompress the data.
|
| 96 |
+
prefix: bytestring
|
| 97 |
+
A bytestring corresponding to the magic number that identifies the
|
| 98 |
+
file format associated to the compressor.
|
| 99 |
+
extension: str
|
| 100 |
+
The file extension used to automatically select this compressor during
|
| 101 |
+
a dump to a file.
|
| 102 |
+
"""
|
| 103 |
+
|
| 104 |
+
def __init__(self, obj, prefix=b"", extension=""):
|
| 105 |
+
self.fileobj_factory = obj
|
| 106 |
+
self.prefix = prefix
|
| 107 |
+
self.extension = extension
|
| 108 |
+
|
| 109 |
+
def compressor_file(self, fileobj, compresslevel=None):
|
| 110 |
+
"""Returns an instance of a compressor file object."""
|
| 111 |
+
if compresslevel is None:
|
| 112 |
+
return self.fileobj_factory(fileobj, "wb")
|
| 113 |
+
else:
|
| 114 |
+
return self.fileobj_factory(fileobj, "wb", compresslevel=compresslevel)
|
| 115 |
+
|
| 116 |
+
def decompressor_file(self, fileobj):
|
| 117 |
+
"""Returns an instance of a decompressor file object."""
|
| 118 |
+
return self.fileobj_factory(fileobj, "rb")
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
class BZ2CompressorWrapper(CompressorWrapper):
|
| 122 |
+
prefix = _BZ2_PREFIX
|
| 123 |
+
extension = ".bz2"
|
| 124 |
+
|
| 125 |
+
def __init__(self):
|
| 126 |
+
if bz2 is not None:
|
| 127 |
+
self.fileobj_factory = bz2.BZ2File
|
| 128 |
+
else:
|
| 129 |
+
self.fileobj_factory = None
|
| 130 |
+
|
| 131 |
+
def _check_versions(self):
|
| 132 |
+
if bz2 is None:
|
| 133 |
+
raise ValueError(
|
| 134 |
+
"bz2 module is not compiled on your python standard library."
|
| 135 |
+
)
|
| 136 |
+
|
| 137 |
+
def compressor_file(self, fileobj, compresslevel=None):
|
| 138 |
+
"""Returns an instance of a compressor file object."""
|
| 139 |
+
self._check_versions()
|
| 140 |
+
if compresslevel is None:
|
| 141 |
+
return self.fileobj_factory(fileobj, "wb")
|
| 142 |
+
else:
|
| 143 |
+
return self.fileobj_factory(fileobj, "wb", compresslevel=compresslevel)
|
| 144 |
+
|
| 145 |
+
def decompressor_file(self, fileobj):
|
| 146 |
+
"""Returns an instance of a decompressor file object."""
|
| 147 |
+
self._check_versions()
|
| 148 |
+
fileobj = self.fileobj_factory(fileobj, "rb")
|
| 149 |
+
return fileobj
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
class LZMACompressorWrapper(CompressorWrapper):
|
| 153 |
+
prefix = _LZMA_PREFIX
|
| 154 |
+
extension = ".lzma"
|
| 155 |
+
_lzma_format_name = "FORMAT_ALONE"
|
| 156 |
+
|
| 157 |
+
def __init__(self):
|
| 158 |
+
if lzma is not None:
|
| 159 |
+
self.fileobj_factory = lzma.LZMAFile
|
| 160 |
+
self._lzma_format = getattr(lzma, self._lzma_format_name)
|
| 161 |
+
else:
|
| 162 |
+
self.fileobj_factory = None
|
| 163 |
+
|
| 164 |
+
def _check_versions(self):
|
| 165 |
+
if lzma is None:
|
| 166 |
+
raise ValueError(
|
| 167 |
+
"lzma module is not compiled on your python standard library."
|
| 168 |
+
)
|
| 169 |
+
|
| 170 |
+
def compressor_file(self, fileobj, compresslevel=None):
|
| 171 |
+
"""Returns an instance of a compressor file object."""
|
| 172 |
+
if compresslevel is None:
|
| 173 |
+
return self.fileobj_factory(fileobj, "wb", format=self._lzma_format)
|
| 174 |
+
else:
|
| 175 |
+
return self.fileobj_factory(
|
| 176 |
+
fileobj, "wb", format=self._lzma_format, preset=compresslevel
|
| 177 |
+
)
|
| 178 |
+
|
| 179 |
+
def decompressor_file(self, fileobj):
|
| 180 |
+
"""Returns an instance of a decompressor file object."""
|
| 181 |
+
return lzma.LZMAFile(fileobj, "rb")
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
class XZCompressorWrapper(LZMACompressorWrapper):
|
| 185 |
+
prefix = _XZ_PREFIX
|
| 186 |
+
extension = ".xz"
|
| 187 |
+
_lzma_format_name = "FORMAT_XZ"
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
class LZ4CompressorWrapper(CompressorWrapper):
|
| 191 |
+
prefix = _LZ4_PREFIX
|
| 192 |
+
extension = ".lz4"
|
| 193 |
+
|
| 194 |
+
def __init__(self):
|
| 195 |
+
if lz4 is not None:
|
| 196 |
+
self.fileobj_factory = LZ4FrameFile
|
| 197 |
+
else:
|
| 198 |
+
self.fileobj_factory = None
|
| 199 |
+
|
| 200 |
+
def _check_versions(self):
|
| 201 |
+
if lz4 is None:
|
| 202 |
+
raise ValueError(LZ4_NOT_INSTALLED_ERROR)
|
| 203 |
+
lz4_version = lz4.__version__
|
| 204 |
+
if lz4_version.startswith("v"):
|
| 205 |
+
lz4_version = lz4_version[1:]
|
| 206 |
+
if LooseVersion(lz4_version) < LooseVersion("0.19"):
|
| 207 |
+
raise ValueError(LZ4_NOT_INSTALLED_ERROR)
|
| 208 |
+
|
| 209 |
+
def compressor_file(self, fileobj, compresslevel=None):
|
| 210 |
+
"""Returns an instance of a compressor file object."""
|
| 211 |
+
self._check_versions()
|
| 212 |
+
if compresslevel is None:
|
| 213 |
+
return self.fileobj_factory(fileobj, "wb")
|
| 214 |
+
else:
|
| 215 |
+
return self.fileobj_factory(fileobj, "wb", compression_level=compresslevel)
|
| 216 |
+
|
| 217 |
+
def decompressor_file(self, fileobj):
|
| 218 |
+
"""Returns an instance of a decompressor file object."""
|
| 219 |
+
self._check_versions()
|
| 220 |
+
return self.fileobj_factory(fileobj, "rb")
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
###############################################################################
|
| 224 |
+
# base file compression/decompression object definition
|
| 225 |
+
_MODE_CLOSED = 0
|
| 226 |
+
_MODE_READ = 1
|
| 227 |
+
_MODE_READ_EOF = 2
|
| 228 |
+
_MODE_WRITE = 3
|
| 229 |
+
_BUFFER_SIZE = 8192
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
class BinaryZlibFile(io.BufferedIOBase):
|
| 233 |
+
"""A file object providing transparent zlib (de)compression.
|
| 234 |
+
|
| 235 |
+
TODO python2_drop: is it still needed since we dropped Python 2 support A
|
| 236 |
+
BinaryZlibFile can act as a wrapper for an existing file object, or refer
|
| 237 |
+
directly to a named file on disk.
|
| 238 |
+
|
| 239 |
+
Note that BinaryZlibFile provides only a *binary* file interface: data read
|
| 240 |
+
is returned as bytes, and data to be written should be given as bytes.
|
| 241 |
+
|
| 242 |
+
This object is an adaptation of the BZ2File object and is compatible with
|
| 243 |
+
versions of python >= 2.7.
|
| 244 |
+
|
| 245 |
+
If filename is a str or bytes object, it gives the name
|
| 246 |
+
of the file to be opened. Otherwise, it should be a file object,
|
| 247 |
+
which will be used to read or write the compressed data.
|
| 248 |
+
|
| 249 |
+
mode can be 'rb' for reading (default) or 'wb' for (over)writing
|
| 250 |
+
|
| 251 |
+
If mode is 'wb', compresslevel can be a number between 1
|
| 252 |
+
and 9 specifying the level of compression: 1 produces the least
|
| 253 |
+
compression, and 9 produces the most compression. 3 is the default.
|
| 254 |
+
"""
|
| 255 |
+
|
| 256 |
+
wbits = zlib.MAX_WBITS
|
| 257 |
+
|
| 258 |
+
def __init__(self, filename, mode="rb", compresslevel=3):
|
| 259 |
+
# This lock must be recursive, so that BufferedIOBase's
|
| 260 |
+
# readline(), readlines() and writelines() don't deadlock.
|
| 261 |
+
self._lock = RLock()
|
| 262 |
+
self._fp = None
|
| 263 |
+
self._closefp = False
|
| 264 |
+
self._mode = _MODE_CLOSED
|
| 265 |
+
self._pos = 0
|
| 266 |
+
self._size = -1
|
| 267 |
+
self.compresslevel = compresslevel
|
| 268 |
+
|
| 269 |
+
if not isinstance(compresslevel, int) or not (1 <= compresslevel <= 9):
|
| 270 |
+
raise ValueError(
|
| 271 |
+
"'compresslevel' must be an integer "
|
| 272 |
+
"between 1 and 9. You provided 'compresslevel={}'".format(compresslevel)
|
| 273 |
+
)
|
| 274 |
+
|
| 275 |
+
if mode == "rb":
|
| 276 |
+
self._mode = _MODE_READ
|
| 277 |
+
self._decompressor = zlib.decompressobj(self.wbits)
|
| 278 |
+
self._buffer = b""
|
| 279 |
+
self._buffer_offset = 0
|
| 280 |
+
elif mode == "wb":
|
| 281 |
+
self._mode = _MODE_WRITE
|
| 282 |
+
self._compressor = zlib.compressobj(
|
| 283 |
+
self.compresslevel, zlib.DEFLATED, self.wbits, zlib.DEF_MEM_LEVEL, 0
|
| 284 |
+
)
|
| 285 |
+
else:
|
| 286 |
+
raise ValueError("Invalid mode: %r" % (mode,))
|
| 287 |
+
|
| 288 |
+
if isinstance(filename, str):
|
| 289 |
+
self._fp = io.open(filename, mode)
|
| 290 |
+
self._closefp = True
|
| 291 |
+
elif hasattr(filename, "read") or hasattr(filename, "write"):
|
| 292 |
+
self._fp = filename
|
| 293 |
+
else:
|
| 294 |
+
raise TypeError("filename must be a str or bytes object, or a file")
|
| 295 |
+
|
| 296 |
+
def close(self):
|
| 297 |
+
"""Flush and close the file.
|
| 298 |
+
|
| 299 |
+
May be called more than once without error. Once the file is
|
| 300 |
+
closed, any other operation on it will raise a ValueError.
|
| 301 |
+
"""
|
| 302 |
+
with self._lock:
|
| 303 |
+
if self._mode == _MODE_CLOSED:
|
| 304 |
+
return
|
| 305 |
+
try:
|
| 306 |
+
if self._mode in (_MODE_READ, _MODE_READ_EOF):
|
| 307 |
+
self._decompressor = None
|
| 308 |
+
elif self._mode == _MODE_WRITE:
|
| 309 |
+
self._fp.write(self._compressor.flush())
|
| 310 |
+
self._compressor = None
|
| 311 |
+
finally:
|
| 312 |
+
try:
|
| 313 |
+
if self._closefp:
|
| 314 |
+
self._fp.close()
|
| 315 |
+
finally:
|
| 316 |
+
self._fp = None
|
| 317 |
+
self._closefp = False
|
| 318 |
+
self._mode = _MODE_CLOSED
|
| 319 |
+
self._buffer = b""
|
| 320 |
+
self._buffer_offset = 0
|
| 321 |
+
|
| 322 |
+
@property
|
| 323 |
+
def closed(self):
|
| 324 |
+
"""True if this file is closed."""
|
| 325 |
+
return self._mode == _MODE_CLOSED
|
| 326 |
+
|
| 327 |
+
def fileno(self):
|
| 328 |
+
"""Return the file descriptor for the underlying file."""
|
| 329 |
+
self._check_not_closed()
|
| 330 |
+
return self._fp.fileno()
|
| 331 |
+
|
| 332 |
+
def seekable(self):
|
| 333 |
+
"""Return whether the file supports seeking."""
|
| 334 |
+
return self.readable() and self._fp.seekable()
|
| 335 |
+
|
| 336 |
+
def readable(self):
|
| 337 |
+
"""Return whether the file was opened for reading."""
|
| 338 |
+
self._check_not_closed()
|
| 339 |
+
return self._mode in (_MODE_READ, _MODE_READ_EOF)
|
| 340 |
+
|
| 341 |
+
def writable(self):
|
| 342 |
+
"""Return whether the file was opened for writing."""
|
| 343 |
+
self._check_not_closed()
|
| 344 |
+
return self._mode == _MODE_WRITE
|
| 345 |
+
|
| 346 |
+
# Mode-checking helper functions.
|
| 347 |
+
|
| 348 |
+
def _check_not_closed(self):
|
| 349 |
+
if self.closed:
|
| 350 |
+
fname = getattr(self._fp, "name", None)
|
| 351 |
+
msg = "I/O operation on closed file"
|
| 352 |
+
if fname is not None:
|
| 353 |
+
msg += " {}".format(fname)
|
| 354 |
+
msg += "."
|
| 355 |
+
raise ValueError(msg)
|
| 356 |
+
|
| 357 |
+
def _check_can_read(self):
|
| 358 |
+
if self._mode not in (_MODE_READ, _MODE_READ_EOF):
|
| 359 |
+
self._check_not_closed()
|
| 360 |
+
raise io.UnsupportedOperation("File not open for reading")
|
| 361 |
+
|
| 362 |
+
def _check_can_write(self):
|
| 363 |
+
if self._mode != _MODE_WRITE:
|
| 364 |
+
self._check_not_closed()
|
| 365 |
+
raise io.UnsupportedOperation("File not open for writing")
|
| 366 |
+
|
| 367 |
+
def _check_can_seek(self):
|
| 368 |
+
if self._mode not in (_MODE_READ, _MODE_READ_EOF):
|
| 369 |
+
self._check_not_closed()
|
| 370 |
+
raise io.UnsupportedOperation(
|
| 371 |
+
"Seeking is only supported on files open for reading"
|
| 372 |
+
)
|
| 373 |
+
if not self._fp.seekable():
|
| 374 |
+
raise io.UnsupportedOperation(
|
| 375 |
+
"The underlying file object does not support seeking"
|
| 376 |
+
)
|
| 377 |
+
|
| 378 |
+
# Fill the readahead buffer if it is empty. Returns False on EOF.
|
| 379 |
+
def _fill_buffer(self):
|
| 380 |
+
if self._mode == _MODE_READ_EOF:
|
| 381 |
+
return False
|
| 382 |
+
# Depending on the input data, our call to the decompressor may not
|
| 383 |
+
# return any data. In this case, try again after reading another block.
|
| 384 |
+
while self._buffer_offset == len(self._buffer):
|
| 385 |
+
try:
|
| 386 |
+
rawblock = self._decompressor.unused_data or self._fp.read(_BUFFER_SIZE)
|
| 387 |
+
if not rawblock:
|
| 388 |
+
raise EOFError
|
| 389 |
+
except EOFError:
|
| 390 |
+
# End-of-stream marker and end of file. We're good.
|
| 391 |
+
self._mode = _MODE_READ_EOF
|
| 392 |
+
self._size = self._pos
|
| 393 |
+
return False
|
| 394 |
+
else:
|
| 395 |
+
self._buffer = self._decompressor.decompress(rawblock)
|
| 396 |
+
self._buffer_offset = 0
|
| 397 |
+
return True
|
| 398 |
+
|
| 399 |
+
# Read data until EOF.
|
| 400 |
+
# If return_data is false, consume the data without returning it.
|
| 401 |
+
def _read_all(self, return_data=True):
|
| 402 |
+
# The loop assumes that _buffer_offset is 0. Ensure that this is true.
|
| 403 |
+
self._buffer = self._buffer[self._buffer_offset :]
|
| 404 |
+
self._buffer_offset = 0
|
| 405 |
+
|
| 406 |
+
blocks = []
|
| 407 |
+
while self._fill_buffer():
|
| 408 |
+
if return_data:
|
| 409 |
+
blocks.append(self._buffer)
|
| 410 |
+
self._pos += len(self._buffer)
|
| 411 |
+
self._buffer = b""
|
| 412 |
+
if return_data:
|
| 413 |
+
return b"".join(blocks)
|
| 414 |
+
|
| 415 |
+
# Read a block of up to n bytes.
|
| 416 |
+
# If return_data is false, consume the data without returning it.
|
| 417 |
+
def _read_block(self, n_bytes, return_data=True):
|
| 418 |
+
# If we have enough data buffered, return immediately.
|
| 419 |
+
end = self._buffer_offset + n_bytes
|
| 420 |
+
if end <= len(self._buffer):
|
| 421 |
+
data = self._buffer[self._buffer_offset : end]
|
| 422 |
+
self._buffer_offset = end
|
| 423 |
+
self._pos += len(data)
|
| 424 |
+
return data if return_data else None
|
| 425 |
+
|
| 426 |
+
# The loop assumes that _buffer_offset is 0. Ensure that this is true.
|
| 427 |
+
self._buffer = self._buffer[self._buffer_offset :]
|
| 428 |
+
self._buffer_offset = 0
|
| 429 |
+
|
| 430 |
+
blocks = []
|
| 431 |
+
while n_bytes > 0 and self._fill_buffer():
|
| 432 |
+
if n_bytes < len(self._buffer):
|
| 433 |
+
data = self._buffer[:n_bytes]
|
| 434 |
+
self._buffer_offset = n_bytes
|
| 435 |
+
else:
|
| 436 |
+
data = self._buffer
|
| 437 |
+
self._buffer = b""
|
| 438 |
+
if return_data:
|
| 439 |
+
blocks.append(data)
|
| 440 |
+
self._pos += len(data)
|
| 441 |
+
n_bytes -= len(data)
|
| 442 |
+
if return_data:
|
| 443 |
+
return b"".join(blocks)
|
| 444 |
+
|
| 445 |
+
def read(self, size=-1):
|
| 446 |
+
"""Read up to size uncompressed bytes from the file.
|
| 447 |
+
|
| 448 |
+
If size is negative or omitted, read until EOF is reached.
|
| 449 |
+
Returns b'' if the file is already at EOF.
|
| 450 |
+
"""
|
| 451 |
+
with self._lock:
|
| 452 |
+
self._check_can_read()
|
| 453 |
+
if size == 0:
|
| 454 |
+
return b""
|
| 455 |
+
elif size < 0:
|
| 456 |
+
return self._read_all()
|
| 457 |
+
else:
|
| 458 |
+
return self._read_block(size)
|
| 459 |
+
|
| 460 |
+
def readinto(self, b):
|
| 461 |
+
"""Read up to len(b) bytes into b.
|
| 462 |
+
|
| 463 |
+
Returns the number of bytes read (0 for EOF).
|
| 464 |
+
"""
|
| 465 |
+
with self._lock:
|
| 466 |
+
return io.BufferedIOBase.readinto(self, b)
|
| 467 |
+
|
| 468 |
+
def write(self, data):
|
| 469 |
+
"""Write a byte string to the file.
|
| 470 |
+
|
| 471 |
+
Returns the number of uncompressed bytes written, which is
|
| 472 |
+
always len(data). Note that due to buffering, the file on disk
|
| 473 |
+
may not reflect the data written until close() is called.
|
| 474 |
+
"""
|
| 475 |
+
with self._lock:
|
| 476 |
+
self._check_can_write()
|
| 477 |
+
# Convert data type if called by io.BufferedWriter.
|
| 478 |
+
if isinstance(data, memoryview):
|
| 479 |
+
data = data.tobytes()
|
| 480 |
+
|
| 481 |
+
compressed = self._compressor.compress(data)
|
| 482 |
+
self._fp.write(compressed)
|
| 483 |
+
self._pos += len(data)
|
| 484 |
+
return len(data)
|
| 485 |
+
|
| 486 |
+
# Rewind the file to the beginning of the data stream.
|
| 487 |
+
def _rewind(self):
|
| 488 |
+
self._fp.seek(0, 0)
|
| 489 |
+
self._mode = _MODE_READ
|
| 490 |
+
self._pos = 0
|
| 491 |
+
self._decompressor = zlib.decompressobj(self.wbits)
|
| 492 |
+
self._buffer = b""
|
| 493 |
+
self._buffer_offset = 0
|
| 494 |
+
|
| 495 |
+
def seek(self, offset, whence=0):
|
| 496 |
+
"""Change the file position.
|
| 497 |
+
|
| 498 |
+
The new position is specified by offset, relative to the
|
| 499 |
+
position indicated by whence. Values for whence are:
|
| 500 |
+
|
| 501 |
+
0: start of stream (default); offset must not be negative
|
| 502 |
+
1: current stream position
|
| 503 |
+
2: end of stream; offset must not be positive
|
| 504 |
+
|
| 505 |
+
Returns the new file position.
|
| 506 |
+
|
| 507 |
+
Note that seeking is emulated, so depending on the parameters,
|
| 508 |
+
this operation may be extremely slow.
|
| 509 |
+
"""
|
| 510 |
+
with self._lock:
|
| 511 |
+
self._check_can_seek()
|
| 512 |
+
|
| 513 |
+
# Recalculate offset as an absolute file position.
|
| 514 |
+
if whence == 0:
|
| 515 |
+
pass
|
| 516 |
+
elif whence == 1:
|
| 517 |
+
offset = self._pos + offset
|
| 518 |
+
elif whence == 2:
|
| 519 |
+
# Seeking relative to EOF - we need to know the file's size.
|
| 520 |
+
if self._size < 0:
|
| 521 |
+
self._read_all(return_data=False)
|
| 522 |
+
offset = self._size + offset
|
| 523 |
+
else:
|
| 524 |
+
raise ValueError("Invalid value for whence: %s" % (whence,))
|
| 525 |
+
|
| 526 |
+
# Make it so that offset is the number of bytes to skip forward.
|
| 527 |
+
if offset < self._pos:
|
| 528 |
+
self._rewind()
|
| 529 |
+
else:
|
| 530 |
+
offset -= self._pos
|
| 531 |
+
|
| 532 |
+
# Read and discard data until we reach the desired position.
|
| 533 |
+
self._read_block(offset, return_data=False)
|
| 534 |
+
|
| 535 |
+
return self._pos
|
| 536 |
+
|
| 537 |
+
def tell(self):
|
| 538 |
+
"""Return the current file position."""
|
| 539 |
+
with self._lock:
|
| 540 |
+
self._check_not_closed()
|
| 541 |
+
return self._pos
|
| 542 |
+
|
| 543 |
+
|
| 544 |
+
class ZlibCompressorWrapper(CompressorWrapper):
|
| 545 |
+
def __init__(self):
|
| 546 |
+
CompressorWrapper.__init__(
|
| 547 |
+
self, obj=BinaryZlibFile, prefix=_ZLIB_PREFIX, extension=".z"
|
| 548 |
+
)
|
| 549 |
+
|
| 550 |
+
|
| 551 |
+
class BinaryGzipFile(BinaryZlibFile):
|
| 552 |
+
"""A file object providing transparent gzip (de)compression.
|
| 553 |
+
|
| 554 |
+
If filename is a str or bytes object, it gives the name
|
| 555 |
+
of the file to be opened. Otherwise, it should be a file object,
|
| 556 |
+
which will be used to read or write the compressed data.
|
| 557 |
+
|
| 558 |
+
mode can be 'rb' for reading (default) or 'wb' for (over)writing
|
| 559 |
+
|
| 560 |
+
If mode is 'wb', compresslevel can be a number between 1
|
| 561 |
+
and 9 specifying the level of compression: 1 produces the least
|
| 562 |
+
compression, and 9 produces the most compression. 3 is the default.
|
| 563 |
+
"""
|
| 564 |
+
|
| 565 |
+
wbits = 31 # zlib compressor/decompressor wbits value for gzip format.
|
| 566 |
+
|
| 567 |
+
|
| 568 |
+
class GzipCompressorWrapper(CompressorWrapper):
|
| 569 |
+
def __init__(self):
|
| 570 |
+
CompressorWrapper.__init__(
|
| 571 |
+
self, obj=BinaryGzipFile, prefix=_GZIP_PREFIX, extension=".gz"
|
| 572 |
+
)
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/disk.py
ADDED
|
@@ -0,0 +1,131 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Disk management utilities.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
# Authors: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 6 |
+
# Lars Buitinck
|
| 7 |
+
# Copyright (c) 2010 Gael Varoquaux
|
| 8 |
+
# License: BSD Style, 3 clauses.
|
| 9 |
+
|
| 10 |
+
import errno
|
| 11 |
+
import os
|
| 12 |
+
import shutil
|
| 13 |
+
import sys
|
| 14 |
+
import time
|
| 15 |
+
from multiprocessing import util
|
| 16 |
+
|
| 17 |
+
try:
|
| 18 |
+
WindowsError
|
| 19 |
+
except NameError:
|
| 20 |
+
WindowsError = OSError
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def disk_used(path):
|
| 24 |
+
"""Return the disk usage in a directory."""
|
| 25 |
+
size = 0
|
| 26 |
+
for file in os.listdir(path) + ["."]:
|
| 27 |
+
stat = os.stat(os.path.join(path, file))
|
| 28 |
+
if hasattr(stat, "st_blocks"):
|
| 29 |
+
size += stat.st_blocks * 512
|
| 30 |
+
else:
|
| 31 |
+
# on some platform st_blocks is not available (e.g., Windows)
|
| 32 |
+
# approximate by rounding to next multiple of 512
|
| 33 |
+
size += (stat.st_size // 512 + 1) * 512
|
| 34 |
+
# We need to convert to int to avoid having longs on some systems (we
|
| 35 |
+
# don't want longs to avoid problems we SQLite)
|
| 36 |
+
return int(size / 1024.0)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def memstr_to_bytes(text):
|
| 40 |
+
"""Convert a memory text to its value in bytes."""
|
| 41 |
+
kilo = 1024
|
| 42 |
+
units = dict(K=kilo, M=kilo**2, G=kilo**3)
|
| 43 |
+
try:
|
| 44 |
+
size = int(units[text[-1]] * float(text[:-1]))
|
| 45 |
+
except (KeyError, ValueError) as e:
|
| 46 |
+
raise ValueError(
|
| 47 |
+
"Invalid literal for size give: %s (type %s) should be "
|
| 48 |
+
"alike '10G', '500M', '50K'." % (text, type(text))
|
| 49 |
+
) from e
|
| 50 |
+
return size
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def mkdirp(d):
|
| 54 |
+
"""Ensure directory d exists (like mkdir -p on Unix)
|
| 55 |
+
No guarantee that the directory is writable.
|
| 56 |
+
"""
|
| 57 |
+
try:
|
| 58 |
+
os.makedirs(d)
|
| 59 |
+
except OSError as e:
|
| 60 |
+
if e.errno != errno.EEXIST:
|
| 61 |
+
raise
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
# if a rmtree operation fails in rm_subdirs, wait for this much time (in secs),
|
| 65 |
+
# then retry up to RM_SUBDIRS_N_RETRY times. If it still fails, raise the
|
| 66 |
+
# exception. this mechanism ensures that the sub-process gc have the time to
|
| 67 |
+
# collect and close the memmaps before we fail.
|
| 68 |
+
RM_SUBDIRS_RETRY_TIME = 0.1
|
| 69 |
+
RM_SUBDIRS_N_RETRY = 10
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def rm_subdirs(path, onerror=None):
|
| 73 |
+
"""Remove all subdirectories in this path.
|
| 74 |
+
|
| 75 |
+
The directory indicated by `path` is left in place, and its subdirectories
|
| 76 |
+
are erased.
|
| 77 |
+
|
| 78 |
+
If onerror is set, it is called to handle the error with arguments (func,
|
| 79 |
+
path, exc_info) where func is os.listdir, os.remove, or os.rmdir;
|
| 80 |
+
path is the argument to that function that caused it to fail; and
|
| 81 |
+
exc_info is a tuple returned by sys.exc_info(). If onerror is None,
|
| 82 |
+
an exception is raised.
|
| 83 |
+
"""
|
| 84 |
+
|
| 85 |
+
# NOTE this code is adapted from the one in shutil.rmtree, and is
|
| 86 |
+
# just as fast
|
| 87 |
+
|
| 88 |
+
names = []
|
| 89 |
+
try:
|
| 90 |
+
names = os.listdir(path)
|
| 91 |
+
except os.error:
|
| 92 |
+
if onerror is not None:
|
| 93 |
+
onerror(os.listdir, path, sys.exc_info())
|
| 94 |
+
else:
|
| 95 |
+
raise
|
| 96 |
+
|
| 97 |
+
for name in names:
|
| 98 |
+
fullname = os.path.join(path, name)
|
| 99 |
+
delete_folder(fullname, onerror=onerror)
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def delete_folder(folder_path, onerror=None, allow_non_empty=True):
|
| 103 |
+
"""Utility function to cleanup a temporary folder if it still exists."""
|
| 104 |
+
if os.path.isdir(folder_path):
|
| 105 |
+
if onerror is not None:
|
| 106 |
+
shutil.rmtree(folder_path, False, onerror)
|
| 107 |
+
else:
|
| 108 |
+
# allow the rmtree to fail once, wait and re-try.
|
| 109 |
+
# if the error is raised again, fail
|
| 110 |
+
err_count = 0
|
| 111 |
+
while True:
|
| 112 |
+
files = os.listdir(folder_path)
|
| 113 |
+
try:
|
| 114 |
+
if len(files) == 0 or allow_non_empty:
|
| 115 |
+
shutil.rmtree(folder_path, ignore_errors=False, onerror=None)
|
| 116 |
+
util.debug("Successfully deleted {}".format(folder_path))
|
| 117 |
+
break
|
| 118 |
+
else:
|
| 119 |
+
raise OSError(
|
| 120 |
+
"Expected empty folder {} but got {} files.".format(
|
| 121 |
+
folder_path, len(files)
|
| 122 |
+
)
|
| 123 |
+
)
|
| 124 |
+
except (OSError, WindowsError):
|
| 125 |
+
err_count += 1
|
| 126 |
+
if err_count > RM_SUBDIRS_N_RETRY:
|
| 127 |
+
# the folder cannot be deleted right now. It maybe
|
| 128 |
+
# because some temporary files have not been deleted
|
| 129 |
+
# yet.
|
| 130 |
+
raise
|
| 131 |
+
time.sleep(RM_SUBDIRS_RETRY_TIME)
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/executor.py
ADDED
|
@@ -0,0 +1,131 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Utility function to construct a loky.ReusableExecutor with custom pickler.
|
| 2 |
+
|
| 3 |
+
This module provides efficient ways of working with data stored in
|
| 4 |
+
shared memory with numpy.memmap arrays without inducing any memory
|
| 5 |
+
copy between the parent and child processes.
|
| 6 |
+
"""
|
| 7 |
+
# Author: Thomas Moreau <thomas.moreau.2010@gmail.com>
|
| 8 |
+
# Copyright: 2017, Thomas Moreau
|
| 9 |
+
# License: BSD 3 clause
|
| 10 |
+
|
| 11 |
+
from ._memmapping_reducer import TemporaryResourcesManager, get_memmapping_reducers
|
| 12 |
+
from .externals.loky.reusable_executor import _ReusablePoolExecutor
|
| 13 |
+
|
| 14 |
+
_executor_args = None
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def get_memmapping_executor(n_jobs, **kwargs):
|
| 18 |
+
return MemmappingExecutor.get_memmapping_executor(n_jobs, **kwargs)
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class MemmappingExecutor(_ReusablePoolExecutor):
|
| 22 |
+
@classmethod
|
| 23 |
+
def get_memmapping_executor(
|
| 24 |
+
cls,
|
| 25 |
+
n_jobs,
|
| 26 |
+
timeout=300,
|
| 27 |
+
initializer=None,
|
| 28 |
+
initargs=(),
|
| 29 |
+
env=None,
|
| 30 |
+
temp_folder=None,
|
| 31 |
+
context_id=None,
|
| 32 |
+
**backend_args,
|
| 33 |
+
):
|
| 34 |
+
"""Factory for ReusableExecutor with automatic memmapping for large
|
| 35 |
+
numpy arrays.
|
| 36 |
+
"""
|
| 37 |
+
global _executor_args
|
| 38 |
+
# Check if we can reuse the executor here instead of deferring the test
|
| 39 |
+
# to loky as the reducers are objects that changes at each call.
|
| 40 |
+
executor_args = backend_args.copy()
|
| 41 |
+
executor_args.update(env if env else {})
|
| 42 |
+
executor_args.update(
|
| 43 |
+
dict(timeout=timeout, initializer=initializer, initargs=initargs)
|
| 44 |
+
)
|
| 45 |
+
reuse = _executor_args is None or _executor_args == executor_args
|
| 46 |
+
_executor_args = executor_args
|
| 47 |
+
|
| 48 |
+
manager = TemporaryResourcesManager(temp_folder)
|
| 49 |
+
|
| 50 |
+
# reducers access the temporary folder in which to store temporary
|
| 51 |
+
# pickles through a call to manager.resolve_temp_folder_name. resolving
|
| 52 |
+
# the folder name dynamically is useful to use different folders across
|
| 53 |
+
# calls of a same reusable executor
|
| 54 |
+
job_reducers, result_reducers = get_memmapping_reducers(
|
| 55 |
+
unlink_on_gc_collect=True,
|
| 56 |
+
temp_folder_resolver=manager.resolve_temp_folder_name,
|
| 57 |
+
**backend_args,
|
| 58 |
+
)
|
| 59 |
+
_executor, executor_is_reused = super().get_reusable_executor(
|
| 60 |
+
n_jobs,
|
| 61 |
+
job_reducers=job_reducers,
|
| 62 |
+
result_reducers=result_reducers,
|
| 63 |
+
reuse=reuse,
|
| 64 |
+
timeout=timeout,
|
| 65 |
+
initializer=initializer,
|
| 66 |
+
initargs=initargs,
|
| 67 |
+
env=env,
|
| 68 |
+
)
|
| 69 |
+
|
| 70 |
+
if not executor_is_reused:
|
| 71 |
+
# Only set a _temp_folder_manager for new executors. Reused
|
| 72 |
+
# executors already have a _temporary_folder_manager that must not
|
| 73 |
+
# be re-assigned like that because it is referenced in various
|
| 74 |
+
# places in the reducing machinery of the executor.
|
| 75 |
+
_executor._temp_folder_manager = manager
|
| 76 |
+
|
| 77 |
+
if context_id is not None:
|
| 78 |
+
# Only register the specified context once we know which manager
|
| 79 |
+
# the current executor is using, in order to not register an atexit
|
| 80 |
+
# finalizer twice for the same folder.
|
| 81 |
+
_executor._temp_folder_manager.register_new_context(context_id)
|
| 82 |
+
|
| 83 |
+
return _executor
|
| 84 |
+
|
| 85 |
+
def terminate(self, kill_workers=False):
|
| 86 |
+
self.shutdown(kill_workers=kill_workers)
|
| 87 |
+
|
| 88 |
+
# When workers are killed in a brutal manner, they cannot execute the
|
| 89 |
+
# finalizer of their shared memmaps. The refcount of those memmaps may
|
| 90 |
+
# be off by an unknown number, so instead of decref'ing them, we force
|
| 91 |
+
# delete the whole temporary folder, and unregister them. There is no
|
| 92 |
+
# risk of PermissionError at folder deletion because at this
|
| 93 |
+
# point, all child processes are dead, so all references to temporary
|
| 94 |
+
# memmaps are closed. Otherwise, just try to delete as much as possible
|
| 95 |
+
# with allow_non_empty=True but if we can't, it will be clean up later
|
| 96 |
+
# on by the resource_tracker.
|
| 97 |
+
with self._submit_resize_lock:
|
| 98 |
+
self._temp_folder_manager._clean_temporary_resources(
|
| 99 |
+
force=kill_workers, allow_non_empty=True
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
@property
|
| 103 |
+
def _temp_folder(self):
|
| 104 |
+
# Legacy property in tests. could be removed if we refactored the
|
| 105 |
+
# memmapping tests. SHOULD ONLY BE USED IN TESTS!
|
| 106 |
+
# We cache this property because it is called late in the tests - at
|
| 107 |
+
# this point, all context have been unregistered, and
|
| 108 |
+
# resolve_temp_folder_name raises an error.
|
| 109 |
+
if getattr(self, "_cached_temp_folder", None) is not None:
|
| 110 |
+
return self._cached_temp_folder
|
| 111 |
+
else:
|
| 112 |
+
self._cached_temp_folder = (
|
| 113 |
+
self._temp_folder_manager.resolve_temp_folder_name()
|
| 114 |
+
) # noqa
|
| 115 |
+
return self._cached_temp_folder
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
class _TestingMemmappingExecutor(MemmappingExecutor):
|
| 119 |
+
"""Wrapper around ReusableExecutor to ease memmapping testing with Pool
|
| 120 |
+
and Executor. This is only for testing purposes.
|
| 121 |
+
|
| 122 |
+
"""
|
| 123 |
+
|
| 124 |
+
def apply_async(self, func, args):
|
| 125 |
+
"""Schedule a func to be run"""
|
| 126 |
+
future = self.submit(func, *args)
|
| 127 |
+
future.get = future.result
|
| 128 |
+
return future
|
| 129 |
+
|
| 130 |
+
def map(self, f, *args):
|
| 131 |
+
return list(super().map(f, *args))
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/func_inspect.py
ADDED
|
@@ -0,0 +1,379 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
My own variation on function-specific inspect-like features.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
# Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 6 |
+
# Copyright (c) 2009 Gael Varoquaux
|
| 7 |
+
# License: BSD Style, 3 clauses.
|
| 8 |
+
|
| 9 |
+
import collections
|
| 10 |
+
import inspect
|
| 11 |
+
import os
|
| 12 |
+
import re
|
| 13 |
+
import warnings
|
| 14 |
+
from itertools import islice
|
| 15 |
+
from tokenize import open as open_py_source
|
| 16 |
+
|
| 17 |
+
from .logger import pformat
|
| 18 |
+
|
| 19 |
+
full_argspec_fields = (
|
| 20 |
+
"args varargs varkw defaults kwonlyargs kwonlydefaults annotations"
|
| 21 |
+
)
|
| 22 |
+
full_argspec_type = collections.namedtuple("FullArgSpec", full_argspec_fields)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def get_func_code(func):
|
| 26 |
+
"""Attempts to retrieve a reliable function code hash.
|
| 27 |
+
|
| 28 |
+
The reason we don't use inspect.getsource is that it caches the
|
| 29 |
+
source, whereas we want this to be modified on the fly when the
|
| 30 |
+
function is modified.
|
| 31 |
+
|
| 32 |
+
Returns
|
| 33 |
+
-------
|
| 34 |
+
func_code: string
|
| 35 |
+
The function code
|
| 36 |
+
source_file: string
|
| 37 |
+
The path to the file in which the function is defined.
|
| 38 |
+
first_line: int
|
| 39 |
+
The first line of the code in the source file.
|
| 40 |
+
|
| 41 |
+
Notes
|
| 42 |
+
------
|
| 43 |
+
This function does a bit more magic than inspect, and is thus
|
| 44 |
+
more robust.
|
| 45 |
+
"""
|
| 46 |
+
source_file = None
|
| 47 |
+
try:
|
| 48 |
+
code = func.__code__
|
| 49 |
+
source_file = code.co_filename
|
| 50 |
+
if not os.path.exists(source_file):
|
| 51 |
+
# Use inspect for lambda functions and functions defined in an
|
| 52 |
+
# interactive shell, or in doctests
|
| 53 |
+
source_code = "".join(inspect.getsourcelines(func)[0])
|
| 54 |
+
line_no = 1
|
| 55 |
+
if source_file.startswith("<doctest "):
|
| 56 |
+
source_file, line_no = re.match(
|
| 57 |
+
r"\<doctest (.*\.rst)\[(.*)\]\>", source_file
|
| 58 |
+
).groups()
|
| 59 |
+
line_no = int(line_no)
|
| 60 |
+
source_file = "<doctest %s>" % source_file
|
| 61 |
+
return source_code, source_file, line_no
|
| 62 |
+
# Try to retrieve the source code.
|
| 63 |
+
with open_py_source(source_file) as source_file_obj:
|
| 64 |
+
first_line = code.co_firstlineno
|
| 65 |
+
# All the lines after the function definition:
|
| 66 |
+
source_lines = list(islice(source_file_obj, first_line - 1, None))
|
| 67 |
+
return "".join(inspect.getblock(source_lines)), source_file, first_line
|
| 68 |
+
except: # noqa: E722
|
| 69 |
+
# If the source code fails, we use the hash. This is fragile and
|
| 70 |
+
# might change from one session to another.
|
| 71 |
+
if hasattr(func, "__code__"):
|
| 72 |
+
# Python 3.X
|
| 73 |
+
return str(func.__code__.__hash__()), source_file, -1
|
| 74 |
+
else:
|
| 75 |
+
# Weird objects like numpy ufunc don't have __code__
|
| 76 |
+
# This is fragile, as quite often the id of the object is
|
| 77 |
+
# in the repr, so it might not persist across sessions,
|
| 78 |
+
# however it will work for ufuncs.
|
| 79 |
+
return repr(func), source_file, -1
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def _clean_win_chars(string):
|
| 83 |
+
"""Windows cannot encode some characters in filename."""
|
| 84 |
+
import urllib
|
| 85 |
+
|
| 86 |
+
if hasattr(urllib, "quote"):
|
| 87 |
+
quote = urllib.quote
|
| 88 |
+
else:
|
| 89 |
+
# In Python 3, quote is elsewhere
|
| 90 |
+
import urllib.parse
|
| 91 |
+
|
| 92 |
+
quote = urllib.parse.quote
|
| 93 |
+
for char in ("<", ">", "!", ":", "\\"):
|
| 94 |
+
string = string.replace(char, quote(char))
|
| 95 |
+
return string
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def get_func_name(func, resolv_alias=True, win_characters=True):
|
| 99 |
+
"""Return the function import path (as a list of module names), and
|
| 100 |
+
a name for the function.
|
| 101 |
+
|
| 102 |
+
Parameters
|
| 103 |
+
----------
|
| 104 |
+
func: callable
|
| 105 |
+
The func to inspect
|
| 106 |
+
resolv_alias: boolean, optional
|
| 107 |
+
If true, possible local aliases are indicated.
|
| 108 |
+
win_characters: boolean, optional
|
| 109 |
+
If true, substitute special characters using urllib.quote
|
| 110 |
+
This is useful in Windows, as it cannot encode some filenames
|
| 111 |
+
"""
|
| 112 |
+
if hasattr(func, "__module__"):
|
| 113 |
+
module = func.__module__
|
| 114 |
+
else:
|
| 115 |
+
try:
|
| 116 |
+
module = inspect.getmodule(func)
|
| 117 |
+
except TypeError:
|
| 118 |
+
if hasattr(func, "__class__"):
|
| 119 |
+
module = func.__class__.__module__
|
| 120 |
+
else:
|
| 121 |
+
module = "unknown"
|
| 122 |
+
if module is None:
|
| 123 |
+
# Happens in doctests, eg
|
| 124 |
+
module = ""
|
| 125 |
+
if module == "__main__":
|
| 126 |
+
try:
|
| 127 |
+
filename = os.path.abspath(inspect.getsourcefile(func))
|
| 128 |
+
except: # noqa: E722
|
| 129 |
+
filename = None
|
| 130 |
+
if filename is not None:
|
| 131 |
+
# mangling of full path to filename
|
| 132 |
+
parts = filename.split(os.sep)
|
| 133 |
+
if parts[-1].startswith("<ipython-input"):
|
| 134 |
+
# We're in a IPython (or notebook) session. parts[-1] comes
|
| 135 |
+
# from func.__code__.co_filename and is of the form
|
| 136 |
+
# <ipython-input-N-XYZ>, where:
|
| 137 |
+
# - N is the cell number where the function was defined
|
| 138 |
+
# - XYZ is a hash representing the function's code (and name).
|
| 139 |
+
# It will be consistent across sessions and kernel restarts,
|
| 140 |
+
# and will change if the function's code/name changes
|
| 141 |
+
# We remove N so that cache is properly hit if the cell where
|
| 142 |
+
# the func is defined is re-exectuted.
|
| 143 |
+
# The XYZ hash should avoid collisions between functions with
|
| 144 |
+
# the same name, both within the same notebook but also across
|
| 145 |
+
# notebooks
|
| 146 |
+
split = parts[-1].split("-")
|
| 147 |
+
parts[-1] = "-".join(split[:2] + split[3:])
|
| 148 |
+
elif len(parts) > 2 and parts[-2].startswith("ipykernel_"):
|
| 149 |
+
# In a notebook session (ipykernel). Filename seems to be 'xyz'
|
| 150 |
+
# of above. parts[-2] has the structure ipykernel_XXXXXX where
|
| 151 |
+
# XXXXXX is a six-digit number identifying the current run (?).
|
| 152 |
+
# If we split it off, the function again has the same
|
| 153 |
+
# identifier across runs.
|
| 154 |
+
parts[-2] = "ipykernel"
|
| 155 |
+
filename = "-".join(parts)
|
| 156 |
+
if filename.endswith(".py"):
|
| 157 |
+
filename = filename[:-3]
|
| 158 |
+
module = module + "-" + filename
|
| 159 |
+
module = module.split(".")
|
| 160 |
+
if hasattr(func, "func_name"):
|
| 161 |
+
name = func.func_name
|
| 162 |
+
elif hasattr(func, "__name__"):
|
| 163 |
+
name = func.__name__
|
| 164 |
+
else:
|
| 165 |
+
name = "unknown"
|
| 166 |
+
# Hack to detect functions not defined at the module-level
|
| 167 |
+
if resolv_alias:
|
| 168 |
+
# TODO: Maybe add a warning here?
|
| 169 |
+
if hasattr(func, "func_globals") and name in func.func_globals:
|
| 170 |
+
if func.func_globals[name] is not func:
|
| 171 |
+
name = "%s-alias" % name
|
| 172 |
+
if hasattr(func, "__qualname__") and func.__qualname__ != name:
|
| 173 |
+
# Extend the module name in case of nested functions to avoid
|
| 174 |
+
# (module, name) collisions
|
| 175 |
+
module.extend(func.__qualname__.split(".")[:-1])
|
| 176 |
+
if inspect.ismethod(func):
|
| 177 |
+
# We need to add the name of the class
|
| 178 |
+
if hasattr(func, "im_class"):
|
| 179 |
+
klass = func.im_class
|
| 180 |
+
module.append(klass.__name__)
|
| 181 |
+
if os.name == "nt" and win_characters:
|
| 182 |
+
# Windows can't encode certain characters in filenames
|
| 183 |
+
name = _clean_win_chars(name)
|
| 184 |
+
module = [_clean_win_chars(s) for s in module]
|
| 185 |
+
return module, name
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def _signature_str(function_name, arg_sig):
|
| 189 |
+
"""Helper function to output a function signature"""
|
| 190 |
+
return "{}{}".format(function_name, arg_sig)
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
def _function_called_str(function_name, args, kwargs):
|
| 194 |
+
"""Helper function to output a function call"""
|
| 195 |
+
template_str = "{0}({1}, {2})"
|
| 196 |
+
|
| 197 |
+
args_str = repr(args)[1:-1]
|
| 198 |
+
kwargs_str = ", ".join("%s=%s" % (k, v) for k, v in kwargs.items())
|
| 199 |
+
return template_str.format(function_name, args_str, kwargs_str)
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def filter_args(func, ignore_lst, args=(), kwargs=dict()):
|
| 203 |
+
"""Filters the given args and kwargs using a list of arguments to
|
| 204 |
+
ignore, and a function specification.
|
| 205 |
+
|
| 206 |
+
Parameters
|
| 207 |
+
----------
|
| 208 |
+
func: callable
|
| 209 |
+
Function giving the argument specification
|
| 210 |
+
ignore_lst: list of strings
|
| 211 |
+
List of arguments to ignore (either a name of an argument
|
| 212 |
+
in the function spec, or '*', or '**')
|
| 213 |
+
*args: list
|
| 214 |
+
Positional arguments passed to the function.
|
| 215 |
+
**kwargs: dict
|
| 216 |
+
Keyword arguments passed to the function
|
| 217 |
+
|
| 218 |
+
Returns
|
| 219 |
+
-------
|
| 220 |
+
filtered_args: list
|
| 221 |
+
List of filtered positional and keyword arguments.
|
| 222 |
+
"""
|
| 223 |
+
args = list(args)
|
| 224 |
+
if isinstance(ignore_lst, str):
|
| 225 |
+
# Catch a common mistake
|
| 226 |
+
raise ValueError(
|
| 227 |
+
"ignore_lst must be a list of parameters to ignore "
|
| 228 |
+
"%s (type %s) was given" % (ignore_lst, type(ignore_lst))
|
| 229 |
+
)
|
| 230 |
+
# Special case for functools.partial objects
|
| 231 |
+
if not inspect.ismethod(func) and not inspect.isfunction(func):
|
| 232 |
+
if ignore_lst:
|
| 233 |
+
warnings.warn(
|
| 234 |
+
"Cannot inspect object %s, ignore list will not work." % func,
|
| 235 |
+
stacklevel=2,
|
| 236 |
+
)
|
| 237 |
+
return {"*": args, "**": kwargs}
|
| 238 |
+
arg_sig = inspect.signature(func)
|
| 239 |
+
arg_names = []
|
| 240 |
+
arg_defaults = []
|
| 241 |
+
arg_kwonlyargs = []
|
| 242 |
+
arg_varargs = None
|
| 243 |
+
arg_varkw = None
|
| 244 |
+
for param in arg_sig.parameters.values():
|
| 245 |
+
if param.kind is param.POSITIONAL_OR_KEYWORD:
|
| 246 |
+
arg_names.append(param.name)
|
| 247 |
+
elif param.kind is param.KEYWORD_ONLY:
|
| 248 |
+
arg_names.append(param.name)
|
| 249 |
+
arg_kwonlyargs.append(param.name)
|
| 250 |
+
elif param.kind is param.VAR_POSITIONAL:
|
| 251 |
+
arg_varargs = param.name
|
| 252 |
+
elif param.kind is param.VAR_KEYWORD:
|
| 253 |
+
arg_varkw = param.name
|
| 254 |
+
if param.default is not param.empty:
|
| 255 |
+
arg_defaults.append(param.default)
|
| 256 |
+
if inspect.ismethod(func):
|
| 257 |
+
# First argument is 'self', it has been removed by Python
|
| 258 |
+
# we need to add it back:
|
| 259 |
+
args = [
|
| 260 |
+
func.__self__,
|
| 261 |
+
] + args
|
| 262 |
+
# func is an instance method, inspect.signature(func) does not
|
| 263 |
+
# include self, we need to fetch it from the class method, i.e
|
| 264 |
+
# func.__func__
|
| 265 |
+
class_method_sig = inspect.signature(func.__func__)
|
| 266 |
+
self_name = next(iter(class_method_sig.parameters))
|
| 267 |
+
arg_names = [self_name] + arg_names
|
| 268 |
+
# XXX: Maybe I need an inspect.isbuiltin to detect C-level methods, such
|
| 269 |
+
# as on ndarrays.
|
| 270 |
+
|
| 271 |
+
_, name = get_func_name(func, resolv_alias=False)
|
| 272 |
+
arg_dict = dict()
|
| 273 |
+
arg_position = -1
|
| 274 |
+
for arg_position, arg_name in enumerate(arg_names):
|
| 275 |
+
if arg_position < len(args):
|
| 276 |
+
# Positional argument or keyword argument given as positional
|
| 277 |
+
if arg_name not in arg_kwonlyargs:
|
| 278 |
+
arg_dict[arg_name] = args[arg_position]
|
| 279 |
+
else:
|
| 280 |
+
raise ValueError(
|
| 281 |
+
"Keyword-only parameter '%s' was passed as "
|
| 282 |
+
"positional parameter for %s:\n"
|
| 283 |
+
" %s was called."
|
| 284 |
+
% (
|
| 285 |
+
arg_name,
|
| 286 |
+
_signature_str(name, arg_sig),
|
| 287 |
+
_function_called_str(name, args, kwargs),
|
| 288 |
+
)
|
| 289 |
+
)
|
| 290 |
+
|
| 291 |
+
else:
|
| 292 |
+
position = arg_position - len(arg_names)
|
| 293 |
+
if arg_name in kwargs:
|
| 294 |
+
arg_dict[arg_name] = kwargs[arg_name]
|
| 295 |
+
else:
|
| 296 |
+
try:
|
| 297 |
+
arg_dict[arg_name] = arg_defaults[position]
|
| 298 |
+
except (IndexError, KeyError) as e:
|
| 299 |
+
# Missing argument
|
| 300 |
+
raise ValueError(
|
| 301 |
+
"Wrong number of arguments for %s:\n"
|
| 302 |
+
" %s was called."
|
| 303 |
+
% (
|
| 304 |
+
_signature_str(name, arg_sig),
|
| 305 |
+
_function_called_str(name, args, kwargs),
|
| 306 |
+
)
|
| 307 |
+
) from e
|
| 308 |
+
|
| 309 |
+
varkwargs = dict()
|
| 310 |
+
for arg_name, arg_value in sorted(kwargs.items()):
|
| 311 |
+
if arg_name in arg_dict:
|
| 312 |
+
arg_dict[arg_name] = arg_value
|
| 313 |
+
elif arg_varkw is not None:
|
| 314 |
+
varkwargs[arg_name] = arg_value
|
| 315 |
+
else:
|
| 316 |
+
raise TypeError(
|
| 317 |
+
"Ignore list for %s() contains an unexpected "
|
| 318 |
+
"keyword argument '%s'" % (name, arg_name)
|
| 319 |
+
)
|
| 320 |
+
|
| 321 |
+
if arg_varkw is not None:
|
| 322 |
+
arg_dict["**"] = varkwargs
|
| 323 |
+
if arg_varargs is not None:
|
| 324 |
+
varargs = args[arg_position + 1 :]
|
| 325 |
+
arg_dict["*"] = varargs
|
| 326 |
+
|
| 327 |
+
# Now remove the arguments to be ignored
|
| 328 |
+
for item in ignore_lst:
|
| 329 |
+
if item in arg_dict:
|
| 330 |
+
arg_dict.pop(item)
|
| 331 |
+
else:
|
| 332 |
+
raise ValueError(
|
| 333 |
+
"Ignore list: argument '%s' is not defined for "
|
| 334 |
+
"function %s" % (item, _signature_str(name, arg_sig))
|
| 335 |
+
)
|
| 336 |
+
# XXX: Return a sorted list of pairs?
|
| 337 |
+
return arg_dict
|
| 338 |
+
|
| 339 |
+
|
| 340 |
+
def _format_arg(arg):
|
| 341 |
+
formatted_arg = pformat(arg, indent=2)
|
| 342 |
+
if len(formatted_arg) > 1500:
|
| 343 |
+
formatted_arg = "%s..." % formatted_arg[:700]
|
| 344 |
+
return formatted_arg
|
| 345 |
+
|
| 346 |
+
|
| 347 |
+
def format_signature(func, *args, **kwargs):
|
| 348 |
+
# XXX: Should this use inspect.formatargvalues/formatargspec?
|
| 349 |
+
module, name = get_func_name(func)
|
| 350 |
+
module = [m for m in module if m]
|
| 351 |
+
if module:
|
| 352 |
+
module.append(name)
|
| 353 |
+
module_path = ".".join(module)
|
| 354 |
+
else:
|
| 355 |
+
module_path = name
|
| 356 |
+
arg_str = list()
|
| 357 |
+
previous_length = 0
|
| 358 |
+
for arg in args:
|
| 359 |
+
formatted_arg = _format_arg(arg)
|
| 360 |
+
if previous_length > 80:
|
| 361 |
+
formatted_arg = "\n%s" % formatted_arg
|
| 362 |
+
previous_length = len(formatted_arg)
|
| 363 |
+
arg_str.append(formatted_arg)
|
| 364 |
+
arg_str.extend(["%s=%s" % (v, _format_arg(i)) for v, i in kwargs.items()])
|
| 365 |
+
arg_str = ", ".join(arg_str)
|
| 366 |
+
|
| 367 |
+
signature = "%s(%s)" % (name, arg_str)
|
| 368 |
+
return module_path, signature
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
def format_call(func, args, kwargs, object_name="Memory"):
|
| 372 |
+
"""Returns a nicely formatted statement displaying the function
|
| 373 |
+
call with the given arguments.
|
| 374 |
+
"""
|
| 375 |
+
path, signature = format_signature(func, *args, **kwargs)
|
| 376 |
+
msg = "%s\n[%s] Calling %s...\n%s" % (80 * "_", object_name, path, signature)
|
| 377 |
+
return msg
|
| 378 |
+
# XXX: Not using logging framework
|
| 379 |
+
# self.debug(msg)
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/hashing.py
ADDED
|
@@ -0,0 +1,270 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Fast cryptographic hash of Python objects, with a special case for fast
|
| 3 |
+
hashing of numpy arrays.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
# Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 7 |
+
# Copyright (c) 2009 Gael Varoquaux
|
| 8 |
+
# License: BSD Style, 3 clauses.
|
| 9 |
+
|
| 10 |
+
import decimal
|
| 11 |
+
import hashlib
|
| 12 |
+
import io
|
| 13 |
+
import pickle
|
| 14 |
+
import struct
|
| 15 |
+
import sys
|
| 16 |
+
import types
|
| 17 |
+
|
| 18 |
+
Pickler = pickle._Pickler
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class _ConsistentSet(object):
|
| 22 |
+
"""Class used to ensure the hash of Sets is preserved
|
| 23 |
+
whatever the order of its items.
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
def __init__(self, set_sequence):
|
| 27 |
+
# Forces order of elements in set to ensure consistent hash.
|
| 28 |
+
try:
|
| 29 |
+
# Trying first to order the set assuming the type of elements is
|
| 30 |
+
# consistent and orderable.
|
| 31 |
+
# This fails on python 3 when elements are unorderable
|
| 32 |
+
# but we keep it in a try as it's faster.
|
| 33 |
+
self._sequence = sorted(set_sequence)
|
| 34 |
+
except (TypeError, decimal.InvalidOperation):
|
| 35 |
+
# If elements are unorderable, sorting them using their hash.
|
| 36 |
+
# This is slower but works in any case.
|
| 37 |
+
self._sequence = sorted((hash(e) for e in set_sequence))
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
class _MyHash(object):
|
| 41 |
+
"""Class used to hash objects that won't normally pickle"""
|
| 42 |
+
|
| 43 |
+
def __init__(self, *args):
|
| 44 |
+
self.args = args
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
class Hasher(Pickler):
|
| 48 |
+
"""A subclass of pickler, to do cryptographic hashing, rather than
|
| 49 |
+
pickling. This is used to produce a unique hash of the given
|
| 50 |
+
Python object that is not necessarily cryptographically secure.
|
| 51 |
+
"""
|
| 52 |
+
|
| 53 |
+
def __init__(self, hash_name="md5"):
|
| 54 |
+
self.stream = io.BytesIO()
|
| 55 |
+
# By default we want a pickle protocol that only changes with
|
| 56 |
+
# the major python version and not the minor one
|
| 57 |
+
protocol = 3
|
| 58 |
+
Pickler.__init__(self, self.stream, protocol=protocol)
|
| 59 |
+
# Initialise the hash obj
|
| 60 |
+
self._hash = hashlib.new(hash_name, usedforsecurity=False)
|
| 61 |
+
|
| 62 |
+
def hash(self, obj, return_digest=True):
|
| 63 |
+
try:
|
| 64 |
+
self.dump(obj)
|
| 65 |
+
except pickle.PicklingError as e:
|
| 66 |
+
e.args += ("PicklingError while hashing %r: %r" % (obj, e),)
|
| 67 |
+
raise
|
| 68 |
+
dumps = self.stream.getvalue()
|
| 69 |
+
self._hash.update(dumps)
|
| 70 |
+
if return_digest:
|
| 71 |
+
return self._hash.hexdigest()
|
| 72 |
+
|
| 73 |
+
def save(self, obj):
|
| 74 |
+
if isinstance(obj, (types.MethodType, type({}.pop))):
|
| 75 |
+
# the Pickler cannot pickle instance methods; here we decompose
|
| 76 |
+
# them into components that make them uniquely identifiable
|
| 77 |
+
if hasattr(obj, "__func__"):
|
| 78 |
+
func_name = obj.__func__.__name__
|
| 79 |
+
else:
|
| 80 |
+
func_name = obj.__name__
|
| 81 |
+
inst = obj.__self__
|
| 82 |
+
if type(inst) is type(pickle):
|
| 83 |
+
obj = _MyHash(func_name, inst.__name__)
|
| 84 |
+
elif inst is None:
|
| 85 |
+
# type(None) or type(module) do not pickle
|
| 86 |
+
obj = _MyHash(func_name, inst)
|
| 87 |
+
else:
|
| 88 |
+
cls = obj.__self__.__class__
|
| 89 |
+
obj = _MyHash(func_name, inst, cls)
|
| 90 |
+
Pickler.save(self, obj)
|
| 91 |
+
|
| 92 |
+
def memoize(self, obj):
|
| 93 |
+
# We want hashing to be sensitive to value instead of reference.
|
| 94 |
+
# For example we want ['aa', 'aa'] and ['aa', 'aaZ'[:2]]
|
| 95 |
+
# to hash to the same value and that's why we disable memoization
|
| 96 |
+
# for strings
|
| 97 |
+
if isinstance(obj, (bytes, str)):
|
| 98 |
+
return
|
| 99 |
+
Pickler.memoize(self, obj)
|
| 100 |
+
|
| 101 |
+
# The dispatch table of the pickler is not accessible in Python
|
| 102 |
+
# 3, as these lines are only bugware for IPython, we skip them.
|
| 103 |
+
def save_global(self, obj, name=None, pack=struct.pack):
|
| 104 |
+
# We have to override this method in order to deal with objects
|
| 105 |
+
# defined interactively in IPython that are not injected in
|
| 106 |
+
# __main__
|
| 107 |
+
kwargs = dict(name=name, pack=pack)
|
| 108 |
+
del kwargs["pack"]
|
| 109 |
+
try:
|
| 110 |
+
Pickler.save_global(self, obj, **kwargs)
|
| 111 |
+
except pickle.PicklingError:
|
| 112 |
+
Pickler.save_global(self, obj, **kwargs)
|
| 113 |
+
module = getattr(obj, "__module__", None)
|
| 114 |
+
if module == "__main__":
|
| 115 |
+
my_name = name
|
| 116 |
+
if my_name is None:
|
| 117 |
+
my_name = obj.__name__
|
| 118 |
+
mod = sys.modules[module]
|
| 119 |
+
if not hasattr(mod, my_name):
|
| 120 |
+
# IPython doesn't inject the variables define
|
| 121 |
+
# interactively in __main__
|
| 122 |
+
setattr(mod, my_name, obj)
|
| 123 |
+
|
| 124 |
+
dispatch = Pickler.dispatch.copy()
|
| 125 |
+
# builtin
|
| 126 |
+
dispatch[type(len)] = save_global
|
| 127 |
+
# type
|
| 128 |
+
dispatch[type(object)] = save_global
|
| 129 |
+
# classobj
|
| 130 |
+
dispatch[type(Pickler)] = save_global
|
| 131 |
+
# function
|
| 132 |
+
dispatch[type(pickle.dump)] = save_global
|
| 133 |
+
|
| 134 |
+
# We use *args in _batch_setitems signature because _batch_setitems has an
|
| 135 |
+
# additional 'obj' argument in Python 3.14
|
| 136 |
+
def _batch_setitems(self, items, *args):
|
| 137 |
+
# forces order of keys in dict to ensure consistent hash.
|
| 138 |
+
try:
|
| 139 |
+
# Trying first to compare dict assuming the type of keys is
|
| 140 |
+
# consistent and orderable.
|
| 141 |
+
# This fails on python 3 when keys are unorderable
|
| 142 |
+
# but we keep it in a try as it's faster.
|
| 143 |
+
Pickler._batch_setitems(self, iter(sorted(items)), *args)
|
| 144 |
+
except TypeError:
|
| 145 |
+
# If keys are unorderable, sorting them using their hash. This is
|
| 146 |
+
# slower but works in any case.
|
| 147 |
+
Pickler._batch_setitems(
|
| 148 |
+
self, iter(sorted((hash(k), v) for k, v in items)), *args
|
| 149 |
+
)
|
| 150 |
+
|
| 151 |
+
def save_set(self, set_items):
|
| 152 |
+
# forces order of items in Set to ensure consistent hash
|
| 153 |
+
Pickler.save(self, _ConsistentSet(set_items))
|
| 154 |
+
|
| 155 |
+
dispatch[type(set())] = save_set
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
class NumpyHasher(Hasher):
|
| 159 |
+
"""Special case the hasher for when numpy is loaded."""
|
| 160 |
+
|
| 161 |
+
def __init__(self, hash_name="md5", coerce_mmap=False):
|
| 162 |
+
"""
|
| 163 |
+
Parameters
|
| 164 |
+
----------
|
| 165 |
+
hash_name: string
|
| 166 |
+
The hash algorithm to be used
|
| 167 |
+
coerce_mmap: boolean
|
| 168 |
+
Make no difference between np.memmap and np.ndarray
|
| 169 |
+
objects.
|
| 170 |
+
"""
|
| 171 |
+
self.coerce_mmap = coerce_mmap
|
| 172 |
+
Hasher.__init__(self, hash_name=hash_name)
|
| 173 |
+
# delayed import of numpy, to avoid tight coupling
|
| 174 |
+
import numpy as np
|
| 175 |
+
|
| 176 |
+
self.np = np
|
| 177 |
+
if hasattr(np, "getbuffer"):
|
| 178 |
+
self._getbuffer = np.getbuffer
|
| 179 |
+
else:
|
| 180 |
+
self._getbuffer = memoryview
|
| 181 |
+
|
| 182 |
+
def save(self, obj):
|
| 183 |
+
"""Subclass the save method, to hash ndarray subclass, rather
|
| 184 |
+
than pickling them. Off course, this is a total abuse of
|
| 185 |
+
the Pickler class.
|
| 186 |
+
"""
|
| 187 |
+
if isinstance(obj, self.np.ndarray) and not obj.dtype.hasobject:
|
| 188 |
+
# Compute a hash of the object
|
| 189 |
+
# The update function of the hash requires a c_contiguous buffer.
|
| 190 |
+
if obj.shape == ():
|
| 191 |
+
# 0d arrays need to be flattened because viewing them as bytes
|
| 192 |
+
# raises a ValueError exception.
|
| 193 |
+
obj_c_contiguous = obj.flatten()
|
| 194 |
+
elif obj.flags.c_contiguous:
|
| 195 |
+
obj_c_contiguous = obj
|
| 196 |
+
elif obj.flags.f_contiguous:
|
| 197 |
+
obj_c_contiguous = obj.T
|
| 198 |
+
else:
|
| 199 |
+
# Cater for non-single-segment arrays: this creates a
|
| 200 |
+
# copy, and thus alleviates this issue.
|
| 201 |
+
# XXX: There might be a more efficient way of doing this
|
| 202 |
+
obj_c_contiguous = obj.flatten()
|
| 203 |
+
|
| 204 |
+
# memoryview is not supported for some dtypes, e.g. datetime64, see
|
| 205 |
+
# https://github.com/numpy/numpy/issues/4983. The
|
| 206 |
+
# workaround is to view the array as bytes before
|
| 207 |
+
# taking the memoryview.
|
| 208 |
+
self._hash.update(self._getbuffer(obj_c_contiguous.view(self.np.uint8)))
|
| 209 |
+
|
| 210 |
+
# We store the class, to be able to distinguish between
|
| 211 |
+
# Objects with the same binary content, but different
|
| 212 |
+
# classes.
|
| 213 |
+
if self.coerce_mmap and isinstance(obj, self.np.memmap):
|
| 214 |
+
# We don't make the difference between memmap and
|
| 215 |
+
# normal ndarrays, to be able to reload previously
|
| 216 |
+
# computed results with memmap.
|
| 217 |
+
klass = self.np.ndarray
|
| 218 |
+
else:
|
| 219 |
+
klass = obj.__class__
|
| 220 |
+
# We also return the dtype and the shape, to distinguish
|
| 221 |
+
# different views on the same data with different dtypes.
|
| 222 |
+
|
| 223 |
+
# The object will be pickled by the pickler hashed at the end.
|
| 224 |
+
obj = (klass, ("HASHED", obj.dtype, obj.shape, obj.strides))
|
| 225 |
+
elif isinstance(obj, self.np.dtype):
|
| 226 |
+
# numpy.dtype consistent hashing is tricky to get right. This comes
|
| 227 |
+
# from the fact that atomic np.dtype objects are interned:
|
| 228 |
+
# ``np.dtype('f4') is np.dtype('f4')``. The situation is
|
| 229 |
+
# complicated by the fact that this interning does not resist a
|
| 230 |
+
# simple pickle.load/dump roundtrip:
|
| 231 |
+
# ``pickle.loads(pickle.dumps(np.dtype('f4'))) is not
|
| 232 |
+
# np.dtype('f4') Because pickle relies on memoization during
|
| 233 |
+
# pickling, it is easy to
|
| 234 |
+
# produce different hashes for seemingly identical objects, such as
|
| 235 |
+
# ``[np.dtype('f4'), np.dtype('f4')]``
|
| 236 |
+
# and ``[np.dtype('f4'), pickle.loads(pickle.dumps('f4'))]``.
|
| 237 |
+
# To prevent memoization from interfering with hashing, we isolate
|
| 238 |
+
# the serialization (and thus the pickle memoization) of each dtype
|
| 239 |
+
# using each time a different ``pickle.dumps`` call unrelated to
|
| 240 |
+
# the current Hasher instance.
|
| 241 |
+
self._hash.update("_HASHED_DTYPE".encode("utf-8"))
|
| 242 |
+
self._hash.update(pickle.dumps(obj))
|
| 243 |
+
return
|
| 244 |
+
Hasher.save(self, obj)
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
def hash(obj, hash_name="md5", coerce_mmap=False):
|
| 248 |
+
"""Quick calculation of a hash to identify uniquely Python objects
|
| 249 |
+
containing numpy arrays.
|
| 250 |
+
|
| 251 |
+
Parameters
|
| 252 |
+
----------
|
| 253 |
+
hash_name: 'md5' or 'sha1'
|
| 254 |
+
Hashing algorithm used. sha1 is supposedly safer, but md5 is
|
| 255 |
+
faster.
|
| 256 |
+
coerce_mmap: boolean
|
| 257 |
+
Make no difference between np.memmap and np.ndarray
|
| 258 |
+
"""
|
| 259 |
+
valid_hash_names = ("md5", "sha1")
|
| 260 |
+
if hash_name not in valid_hash_names:
|
| 261 |
+
raise ValueError(
|
| 262 |
+
"Valid options for 'hash_name' are {}. Got hash_name={!r} instead.".format(
|
| 263 |
+
valid_hash_names, hash_name
|
| 264 |
+
)
|
| 265 |
+
)
|
| 266 |
+
if "numpy" in sys.modules:
|
| 267 |
+
hasher = NumpyHasher(hash_name=hash_name, coerce_mmap=coerce_mmap)
|
| 268 |
+
else:
|
| 269 |
+
hasher = Hasher(hash_name=hash_name)
|
| 270 |
+
return hasher.hash(obj)
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/logger.py
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Helpers for logging.
|
| 3 |
+
|
| 4 |
+
This module needs much love to become useful.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
# Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 8 |
+
# Copyright (c) 2008 Gael Varoquaux
|
| 9 |
+
# License: BSD Style, 3 clauses.
|
| 10 |
+
|
| 11 |
+
from __future__ import print_function
|
| 12 |
+
|
| 13 |
+
import logging
|
| 14 |
+
import os
|
| 15 |
+
import pprint
|
| 16 |
+
import shutil
|
| 17 |
+
import sys
|
| 18 |
+
import time
|
| 19 |
+
|
| 20 |
+
from .disk import mkdirp
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def _squeeze_time(t):
|
| 24 |
+
"""Remove .1s to the time under Windows: this is the time it take to
|
| 25 |
+
stat files. This is needed to make results similar to timings under
|
| 26 |
+
Unix, for tests
|
| 27 |
+
"""
|
| 28 |
+
if sys.platform.startswith("win"):
|
| 29 |
+
return max(0, t - 0.1)
|
| 30 |
+
else:
|
| 31 |
+
return t
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def format_time(t):
|
| 35 |
+
t = _squeeze_time(t)
|
| 36 |
+
return "%.1fs, %.1fmin" % (t, t / 60.0)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def short_format_time(t):
|
| 40 |
+
t = _squeeze_time(t)
|
| 41 |
+
if t > 60:
|
| 42 |
+
return "%4.1fmin" % (t / 60.0)
|
| 43 |
+
else:
|
| 44 |
+
return " %5.1fs" % (t)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def pformat(obj, indent=0, depth=3):
|
| 48 |
+
if "numpy" in sys.modules:
|
| 49 |
+
import numpy as np
|
| 50 |
+
|
| 51 |
+
print_options = np.get_printoptions()
|
| 52 |
+
np.set_printoptions(precision=6, threshold=64, edgeitems=1)
|
| 53 |
+
else:
|
| 54 |
+
print_options = None
|
| 55 |
+
out = pprint.pformat(obj, depth=depth, indent=indent)
|
| 56 |
+
if print_options:
|
| 57 |
+
np.set_printoptions(**print_options)
|
| 58 |
+
return out
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
###############################################################################
|
| 62 |
+
# class `Logger`
|
| 63 |
+
###############################################################################
|
| 64 |
+
class Logger(object):
|
| 65 |
+
"""Base class for logging messages."""
|
| 66 |
+
|
| 67 |
+
def __init__(self, depth=3, name=None):
|
| 68 |
+
"""
|
| 69 |
+
Parameters
|
| 70 |
+
----------
|
| 71 |
+
depth: int, optional
|
| 72 |
+
The depth of objects printed.
|
| 73 |
+
name: str, optional
|
| 74 |
+
The namespace to log to. If None, defaults to joblib.
|
| 75 |
+
"""
|
| 76 |
+
self.depth = depth
|
| 77 |
+
self._name = name if name else "joblib"
|
| 78 |
+
|
| 79 |
+
def warn(self, msg):
|
| 80 |
+
logging.getLogger(self._name).warning("[%s]: %s" % (self, msg))
|
| 81 |
+
|
| 82 |
+
def info(self, msg):
|
| 83 |
+
logging.info("[%s]: %s" % (self, msg))
|
| 84 |
+
|
| 85 |
+
def debug(self, msg):
|
| 86 |
+
# XXX: This conflicts with the debug flag used in children class
|
| 87 |
+
logging.getLogger(self._name).debug("[%s]: %s" % (self, msg))
|
| 88 |
+
|
| 89 |
+
def format(self, obj, indent=0):
|
| 90 |
+
"""Return the formatted representation of the object."""
|
| 91 |
+
return pformat(obj, indent=indent, depth=self.depth)
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
###############################################################################
|
| 95 |
+
# class `PrintTime`
|
| 96 |
+
###############################################################################
|
| 97 |
+
class PrintTime(object):
|
| 98 |
+
"""Print and log messages while keeping track of time."""
|
| 99 |
+
|
| 100 |
+
def __init__(self, logfile=None, logdir=None):
|
| 101 |
+
if logfile is not None and logdir is not None:
|
| 102 |
+
raise ValueError("Cannot specify both logfile and logdir")
|
| 103 |
+
# XXX: Need argument docstring
|
| 104 |
+
self.last_time = time.time()
|
| 105 |
+
self.start_time = self.last_time
|
| 106 |
+
if logdir is not None:
|
| 107 |
+
logfile = os.path.join(logdir, "joblib.log")
|
| 108 |
+
self.logfile = logfile
|
| 109 |
+
if logfile is not None:
|
| 110 |
+
mkdirp(os.path.dirname(logfile))
|
| 111 |
+
if os.path.exists(logfile):
|
| 112 |
+
# Rotate the logs
|
| 113 |
+
for i in range(1, 9):
|
| 114 |
+
try:
|
| 115 |
+
shutil.move(logfile + ".%i" % i, logfile + ".%i" % (i + 1))
|
| 116 |
+
except: # noqa: E722
|
| 117 |
+
"No reason failing here"
|
| 118 |
+
# Use a copy rather than a move, so that a process
|
| 119 |
+
# monitoring this file does not get lost.
|
| 120 |
+
try:
|
| 121 |
+
shutil.copy(logfile, logfile + ".1")
|
| 122 |
+
except: # noqa: E722
|
| 123 |
+
"No reason failing here"
|
| 124 |
+
try:
|
| 125 |
+
with open(logfile, "w") as logfile:
|
| 126 |
+
logfile.write("\nLogging joblib python script\n")
|
| 127 |
+
logfile.write("\n---%s---\n" % time.ctime(self.last_time))
|
| 128 |
+
except: # noqa: E722
|
| 129 |
+
""" Multiprocessing writing to files can create race
|
| 130 |
+
conditions. Rather fail silently than crash the
|
| 131 |
+
computation.
|
| 132 |
+
"""
|
| 133 |
+
# XXX: We actually need a debug flag to disable this
|
| 134 |
+
# silent failure.
|
| 135 |
+
|
| 136 |
+
def __call__(self, msg="", total=False):
|
| 137 |
+
"""Print the time elapsed between the last call and the current
|
| 138 |
+
call, with an optional message.
|
| 139 |
+
"""
|
| 140 |
+
if not total:
|
| 141 |
+
time_lapse = time.time() - self.last_time
|
| 142 |
+
full_msg = "%s: %s" % (msg, format_time(time_lapse))
|
| 143 |
+
else:
|
| 144 |
+
# FIXME: Too much logic duplicated
|
| 145 |
+
time_lapse = time.time() - self.start_time
|
| 146 |
+
full_msg = "%s: %.2fs, %.1f min" % (msg, time_lapse, time_lapse / 60)
|
| 147 |
+
print(full_msg, file=sys.stderr)
|
| 148 |
+
if self.logfile is not None:
|
| 149 |
+
try:
|
| 150 |
+
with open(self.logfile, "a") as f:
|
| 151 |
+
print(full_msg, file=f)
|
| 152 |
+
except: # noqa: E722
|
| 153 |
+
""" Multiprocessing writing to files can create race
|
| 154 |
+
conditions. Rather fail silently than crash the
|
| 155 |
+
calculation.
|
| 156 |
+
"""
|
| 157 |
+
# XXX: We actually need a debug flag to disable this
|
| 158 |
+
# silent failure.
|
| 159 |
+
self.last_time = time.time()
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/memory.py
ADDED
|
@@ -0,0 +1,1242 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
A context object for caching a function's return value each time it
|
| 3 |
+
is called with the same input arguments.
|
| 4 |
+
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
# Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 8 |
+
# Copyright (c) 2009 Gael Varoquaux
|
| 9 |
+
# License: BSD Style, 3 clauses.
|
| 10 |
+
|
| 11 |
+
import asyncio
|
| 12 |
+
import datetime
|
| 13 |
+
import functools
|
| 14 |
+
import inspect
|
| 15 |
+
import logging
|
| 16 |
+
import os
|
| 17 |
+
import pathlib
|
| 18 |
+
import pydoc
|
| 19 |
+
import re
|
| 20 |
+
import textwrap
|
| 21 |
+
import time
|
| 22 |
+
import tokenize
|
| 23 |
+
import traceback
|
| 24 |
+
import warnings
|
| 25 |
+
import weakref
|
| 26 |
+
|
| 27 |
+
from . import hashing
|
| 28 |
+
from ._store_backends import (
|
| 29 |
+
CacheWarning, # noqa
|
| 30 |
+
FileSystemStoreBackend,
|
| 31 |
+
StoreBackendBase,
|
| 32 |
+
)
|
| 33 |
+
from .func_inspect import (
|
| 34 |
+
filter_args,
|
| 35 |
+
format_call,
|
| 36 |
+
format_signature,
|
| 37 |
+
get_func_code,
|
| 38 |
+
get_func_name,
|
| 39 |
+
)
|
| 40 |
+
from .logger import Logger, format_time, pformat
|
| 41 |
+
|
| 42 |
+
FIRST_LINE_TEXT = "# first line:"
|
| 43 |
+
|
| 44 |
+
# TODO: The following object should have a data store object as a sub
|
| 45 |
+
# object, and the interface to persist and query should be separated in
|
| 46 |
+
# the data store.
|
| 47 |
+
#
|
| 48 |
+
# This would enable creating 'Memory' objects with a different logic for
|
| 49 |
+
# pickling that would simply span a MemorizedFunc with the same
|
| 50 |
+
# store (or do we want to copy it to avoid cross-talks?), for instance to
|
| 51 |
+
# implement HDF5 pickling.
|
| 52 |
+
|
| 53 |
+
# TODO: Same remark for the logger, and probably use the Python logging
|
| 54 |
+
# mechanism.
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def extract_first_line(func_code):
|
| 58 |
+
"""Extract the first line information from the function code
|
| 59 |
+
text if available.
|
| 60 |
+
"""
|
| 61 |
+
if func_code.startswith(FIRST_LINE_TEXT):
|
| 62 |
+
func_code = func_code.split("\n")
|
| 63 |
+
first_line = int(func_code[0][len(FIRST_LINE_TEXT) :])
|
| 64 |
+
func_code = "\n".join(func_code[1:])
|
| 65 |
+
else:
|
| 66 |
+
first_line = -1
|
| 67 |
+
return func_code, first_line
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
class JobLibCollisionWarning(UserWarning):
|
| 71 |
+
"""Warn that there might be a collision between names of functions."""
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
_STORE_BACKENDS = {"local": FileSystemStoreBackend}
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def register_store_backend(backend_name, backend):
|
| 78 |
+
"""Extend available store backends.
|
| 79 |
+
|
| 80 |
+
The Memory, MemorizeResult and MemorizeFunc objects are designed to be
|
| 81 |
+
agnostic to the type of store used behind. By default, the local file
|
| 82 |
+
system is used but this function gives the possibility to extend joblib's
|
| 83 |
+
memory pattern with other types of storage such as cloud storage (S3, GCS,
|
| 84 |
+
OpenStack, HadoopFS, etc) or blob DBs.
|
| 85 |
+
|
| 86 |
+
Parameters
|
| 87 |
+
----------
|
| 88 |
+
backend_name: str
|
| 89 |
+
The name identifying the store backend being registered. For example,
|
| 90 |
+
'local' is used with FileSystemStoreBackend.
|
| 91 |
+
backend: StoreBackendBase subclass
|
| 92 |
+
The name of a class that implements the StoreBackendBase interface.
|
| 93 |
+
|
| 94 |
+
"""
|
| 95 |
+
if not isinstance(backend_name, str):
|
| 96 |
+
raise ValueError(
|
| 97 |
+
"Store backend name should be a string, '{0}' given.".format(backend_name)
|
| 98 |
+
)
|
| 99 |
+
if backend is None or not issubclass(backend, StoreBackendBase):
|
| 100 |
+
raise ValueError(
|
| 101 |
+
"Store backend should inherit StoreBackendBase, '{0}' given.".format(
|
| 102 |
+
backend
|
| 103 |
+
)
|
| 104 |
+
)
|
| 105 |
+
|
| 106 |
+
_STORE_BACKENDS[backend_name] = backend
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def _store_backend_factory(backend, location, verbose=0, backend_options=None):
|
| 110 |
+
"""Return the correct store object for the given location."""
|
| 111 |
+
if backend_options is None:
|
| 112 |
+
backend_options = {}
|
| 113 |
+
|
| 114 |
+
if isinstance(location, pathlib.Path):
|
| 115 |
+
location = str(location)
|
| 116 |
+
|
| 117 |
+
if isinstance(location, StoreBackendBase):
|
| 118 |
+
return location
|
| 119 |
+
elif isinstance(location, str):
|
| 120 |
+
obj = None
|
| 121 |
+
location = os.path.expanduser(location)
|
| 122 |
+
# The location is not a local file system, we look in the
|
| 123 |
+
# registered backends if there's one matching the given backend
|
| 124 |
+
# name.
|
| 125 |
+
for backend_key, backend_obj in _STORE_BACKENDS.items():
|
| 126 |
+
if backend == backend_key:
|
| 127 |
+
obj = backend_obj()
|
| 128 |
+
|
| 129 |
+
# By default, we assume the FileSystemStoreBackend can be used if no
|
| 130 |
+
# matching backend could be found.
|
| 131 |
+
if obj is None:
|
| 132 |
+
raise TypeError(
|
| 133 |
+
"Unknown location {0} or backend {1}".format(location, backend)
|
| 134 |
+
)
|
| 135 |
+
|
| 136 |
+
# The store backend is configured with the extra named parameters,
|
| 137 |
+
# some of them are specific to the underlying store backend.
|
| 138 |
+
obj.configure(location, verbose=verbose, backend_options=backend_options)
|
| 139 |
+
return obj
|
| 140 |
+
elif location is not None:
|
| 141 |
+
warnings.warn(
|
| 142 |
+
"Instantiating a backend using a {} as a location is not "
|
| 143 |
+
"supported by joblib. Returning None instead.".format(
|
| 144 |
+
location.__class__.__name__
|
| 145 |
+
),
|
| 146 |
+
UserWarning,
|
| 147 |
+
)
|
| 148 |
+
|
| 149 |
+
return None
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def _build_func_identifier(func):
|
| 153 |
+
"""Build a roughly unique identifier for the cached function."""
|
| 154 |
+
modules, funcname = get_func_name(func)
|
| 155 |
+
# We reuse historical fs-like way of building a function identifier
|
| 156 |
+
return os.path.join(*modules, funcname)
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
# An in-memory store to avoid looking at the disk-based function
|
| 160 |
+
# source code to check if a function definition has changed
|
| 161 |
+
_FUNCTION_HASHES = weakref.WeakKeyDictionary()
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
###############################################################################
|
| 165 |
+
# class `MemorizedResult`
|
| 166 |
+
###############################################################################
|
| 167 |
+
class MemorizedResult(Logger):
|
| 168 |
+
"""Object representing a cached value.
|
| 169 |
+
|
| 170 |
+
Attributes
|
| 171 |
+
----------
|
| 172 |
+
location: str
|
| 173 |
+
The location of joblib cache. Depends on the store backend used.
|
| 174 |
+
|
| 175 |
+
func: function or str
|
| 176 |
+
function whose output is cached. The string case is intended only for
|
| 177 |
+
instantiation based on the output of repr() on another instance.
|
| 178 |
+
(namely eval(repr(memorized_instance)) works).
|
| 179 |
+
|
| 180 |
+
argument_hash: str
|
| 181 |
+
hash of the function arguments.
|
| 182 |
+
|
| 183 |
+
backend: str
|
| 184 |
+
Type of store backend for reading/writing cache files.
|
| 185 |
+
Default is 'local'.
|
| 186 |
+
|
| 187 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
|
| 188 |
+
The memmapping mode used when loading from cache numpy arrays. See
|
| 189 |
+
numpy.load for the meaning of the different values.
|
| 190 |
+
|
| 191 |
+
verbose: int
|
| 192 |
+
verbosity level (0 means no message).
|
| 193 |
+
|
| 194 |
+
timestamp, metadata: string
|
| 195 |
+
for internal use only.
|
| 196 |
+
"""
|
| 197 |
+
|
| 198 |
+
def __init__(
|
| 199 |
+
self,
|
| 200 |
+
location,
|
| 201 |
+
call_id,
|
| 202 |
+
backend="local",
|
| 203 |
+
mmap_mode=None,
|
| 204 |
+
verbose=0,
|
| 205 |
+
timestamp=None,
|
| 206 |
+
metadata=None,
|
| 207 |
+
):
|
| 208 |
+
Logger.__init__(self)
|
| 209 |
+
self._call_id = call_id
|
| 210 |
+
self.store_backend = _store_backend_factory(backend, location, verbose=verbose)
|
| 211 |
+
self.mmap_mode = mmap_mode
|
| 212 |
+
|
| 213 |
+
if metadata is not None:
|
| 214 |
+
self.metadata = metadata
|
| 215 |
+
else:
|
| 216 |
+
self.metadata = self.store_backend.get_metadata(self._call_id)
|
| 217 |
+
|
| 218 |
+
self.duration = self.metadata.get("duration", None)
|
| 219 |
+
self.verbose = verbose
|
| 220 |
+
self.timestamp = timestamp
|
| 221 |
+
|
| 222 |
+
@property
|
| 223 |
+
def func(self):
|
| 224 |
+
return self.func_id
|
| 225 |
+
|
| 226 |
+
@property
|
| 227 |
+
def func_id(self):
|
| 228 |
+
return self._call_id[0]
|
| 229 |
+
|
| 230 |
+
@property
|
| 231 |
+
def args_id(self):
|
| 232 |
+
return self._call_id[1]
|
| 233 |
+
|
| 234 |
+
def get(self):
|
| 235 |
+
"""Read value from cache and return it."""
|
| 236 |
+
try:
|
| 237 |
+
return self.store_backend.load_item(
|
| 238 |
+
self._call_id,
|
| 239 |
+
timestamp=self.timestamp,
|
| 240 |
+
metadata=self.metadata,
|
| 241 |
+
verbose=self.verbose,
|
| 242 |
+
)
|
| 243 |
+
except ValueError as exc:
|
| 244 |
+
new_exc = KeyError(
|
| 245 |
+
"Error while trying to load a MemorizedResult's value. "
|
| 246 |
+
"It seems that this folder is corrupted : {}".format(
|
| 247 |
+
os.path.join(self.store_backend.location, *self._call_id)
|
| 248 |
+
)
|
| 249 |
+
)
|
| 250 |
+
raise new_exc from exc
|
| 251 |
+
|
| 252 |
+
def clear(self):
|
| 253 |
+
"""Clear value from cache"""
|
| 254 |
+
self.store_backend.clear_item(self._call_id)
|
| 255 |
+
|
| 256 |
+
def __repr__(self):
|
| 257 |
+
return '{}(location="{}", func="{}", args_id="{}")'.format(
|
| 258 |
+
self.__class__.__name__, self.store_backend.location, *self._call_id
|
| 259 |
+
)
|
| 260 |
+
|
| 261 |
+
def __getstate__(self):
|
| 262 |
+
state = self.__dict__.copy()
|
| 263 |
+
state["timestamp"] = None
|
| 264 |
+
return state
|
| 265 |
+
|
| 266 |
+
|
| 267 |
+
class NotMemorizedResult(object):
|
| 268 |
+
"""Class representing an arbitrary value.
|
| 269 |
+
|
| 270 |
+
This class is a replacement for MemorizedResult when there is no cache.
|
| 271 |
+
"""
|
| 272 |
+
|
| 273 |
+
__slots__ = ("value", "valid")
|
| 274 |
+
|
| 275 |
+
def __init__(self, value):
|
| 276 |
+
self.value = value
|
| 277 |
+
self.valid = True
|
| 278 |
+
|
| 279 |
+
def get(self):
|
| 280 |
+
if self.valid:
|
| 281 |
+
return self.value
|
| 282 |
+
else:
|
| 283 |
+
raise KeyError("No value stored.")
|
| 284 |
+
|
| 285 |
+
def clear(self):
|
| 286 |
+
self.valid = False
|
| 287 |
+
self.value = None
|
| 288 |
+
|
| 289 |
+
def __repr__(self):
|
| 290 |
+
if self.valid:
|
| 291 |
+
return "{class_name}({value})".format(
|
| 292 |
+
class_name=self.__class__.__name__, value=pformat(self.value)
|
| 293 |
+
)
|
| 294 |
+
else:
|
| 295 |
+
return self.__class__.__name__ + " with no value"
|
| 296 |
+
|
| 297 |
+
# __getstate__ and __setstate__ are required because of __slots__
|
| 298 |
+
def __getstate__(self):
|
| 299 |
+
return {"valid": self.valid, "value": self.value}
|
| 300 |
+
|
| 301 |
+
def __setstate__(self, state):
|
| 302 |
+
self.valid = state["valid"]
|
| 303 |
+
self.value = state["value"]
|
| 304 |
+
|
| 305 |
+
|
| 306 |
+
###############################################################################
|
| 307 |
+
# class `NotMemorizedFunc`
|
| 308 |
+
###############################################################################
|
| 309 |
+
class NotMemorizedFunc(object):
|
| 310 |
+
"""No-op object decorating a function.
|
| 311 |
+
|
| 312 |
+
This class replaces MemorizedFunc when there is no cache. It provides an
|
| 313 |
+
identical API but does not write anything on disk.
|
| 314 |
+
|
| 315 |
+
Attributes
|
| 316 |
+
----------
|
| 317 |
+
func: callable
|
| 318 |
+
Original undecorated function.
|
| 319 |
+
"""
|
| 320 |
+
|
| 321 |
+
# Should be a light as possible (for speed)
|
| 322 |
+
def __init__(self, func):
|
| 323 |
+
self.func = func
|
| 324 |
+
|
| 325 |
+
def __call__(self, *args, **kwargs):
|
| 326 |
+
return self.func(*args, **kwargs)
|
| 327 |
+
|
| 328 |
+
def call_and_shelve(self, *args, **kwargs):
|
| 329 |
+
return NotMemorizedResult(self.func(*args, **kwargs))
|
| 330 |
+
|
| 331 |
+
def __repr__(self):
|
| 332 |
+
return "{0}(func={1})".format(self.__class__.__name__, self.func)
|
| 333 |
+
|
| 334 |
+
def clear(self, warn=True):
|
| 335 |
+
# Argument "warn" is for compatibility with MemorizedFunc.clear
|
| 336 |
+
pass
|
| 337 |
+
|
| 338 |
+
def call(self, *args, **kwargs):
|
| 339 |
+
return self.func(*args, **kwargs), {}
|
| 340 |
+
|
| 341 |
+
def check_call_in_cache(self, *args, **kwargs):
|
| 342 |
+
return False
|
| 343 |
+
|
| 344 |
+
|
| 345 |
+
###############################################################################
|
| 346 |
+
# class `AsyncNotMemorizedFunc`
|
| 347 |
+
###############################################################################
|
| 348 |
+
class AsyncNotMemorizedFunc(NotMemorizedFunc):
|
| 349 |
+
async def call_and_shelve(self, *args, **kwargs):
|
| 350 |
+
return NotMemorizedResult(await self.func(*args, **kwargs))
|
| 351 |
+
|
| 352 |
+
|
| 353 |
+
###############################################################################
|
| 354 |
+
# class `MemorizedFunc`
|
| 355 |
+
###############################################################################
|
| 356 |
+
class MemorizedFunc(Logger):
|
| 357 |
+
"""Callable object decorating a function for caching its return value
|
| 358 |
+
each time it is called.
|
| 359 |
+
|
| 360 |
+
Methods are provided to inspect the cache or clean it.
|
| 361 |
+
|
| 362 |
+
Attributes
|
| 363 |
+
----------
|
| 364 |
+
func: callable
|
| 365 |
+
The original, undecorated, function.
|
| 366 |
+
|
| 367 |
+
location: string
|
| 368 |
+
The location of joblib cache. Depends on the store backend used.
|
| 369 |
+
|
| 370 |
+
backend: str
|
| 371 |
+
Type of store backend for reading/writing cache files.
|
| 372 |
+
Default is 'local', in which case the location is the path to a
|
| 373 |
+
disk storage.
|
| 374 |
+
|
| 375 |
+
ignore: list or None
|
| 376 |
+
List of variable names to ignore when choosing whether to
|
| 377 |
+
recompute.
|
| 378 |
+
|
| 379 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
|
| 380 |
+
The memmapping mode used when loading from cache
|
| 381 |
+
numpy arrays. See numpy.load for the meaning of the different
|
| 382 |
+
values.
|
| 383 |
+
|
| 384 |
+
compress: boolean, or integer
|
| 385 |
+
Whether to zip the stored data on disk. If an integer is
|
| 386 |
+
given, it should be between 1 and 9, and sets the amount
|
| 387 |
+
of compression. Note that compressed arrays cannot be
|
| 388 |
+
read by memmapping.
|
| 389 |
+
|
| 390 |
+
verbose: int, optional
|
| 391 |
+
The verbosity flag, controls messages that are issued as
|
| 392 |
+
the function is evaluated.
|
| 393 |
+
|
| 394 |
+
cache_validation_callback: callable, optional
|
| 395 |
+
Callable to check if a result in cache is valid or is to be recomputed.
|
| 396 |
+
When the function is called with arguments for which a cache exists,
|
| 397 |
+
the callback is called with the cache entry's metadata as its sole
|
| 398 |
+
argument. If it returns True, the cached result is returned, else the
|
| 399 |
+
cache for these arguments is cleared and the result is recomputed.
|
| 400 |
+
"""
|
| 401 |
+
|
| 402 |
+
# ------------------------------------------------------------------------
|
| 403 |
+
# Public interface
|
| 404 |
+
# ------------------------------------------------------------------------
|
| 405 |
+
|
| 406 |
+
def __init__(
|
| 407 |
+
self,
|
| 408 |
+
func,
|
| 409 |
+
location,
|
| 410 |
+
backend="local",
|
| 411 |
+
ignore=None,
|
| 412 |
+
mmap_mode=None,
|
| 413 |
+
compress=False,
|
| 414 |
+
verbose=1,
|
| 415 |
+
timestamp=None,
|
| 416 |
+
cache_validation_callback=None,
|
| 417 |
+
):
|
| 418 |
+
Logger.__init__(self)
|
| 419 |
+
self.mmap_mode = mmap_mode
|
| 420 |
+
self.compress = compress
|
| 421 |
+
self.func = func
|
| 422 |
+
self.cache_validation_callback = cache_validation_callback
|
| 423 |
+
self.func_id = _build_func_identifier(func)
|
| 424 |
+
self.ignore = ignore if ignore is not None else []
|
| 425 |
+
self._verbose = verbose
|
| 426 |
+
|
| 427 |
+
# retrieve store object from backend type and location.
|
| 428 |
+
self.store_backend = _store_backend_factory(
|
| 429 |
+
backend,
|
| 430 |
+
location,
|
| 431 |
+
verbose=verbose,
|
| 432 |
+
backend_options=dict(compress=compress, mmap_mode=mmap_mode),
|
| 433 |
+
)
|
| 434 |
+
if self.store_backend is not None:
|
| 435 |
+
# Create func directory on demand.
|
| 436 |
+
self.store_backend.store_cached_func_code([self.func_id])
|
| 437 |
+
|
| 438 |
+
self.timestamp = timestamp if timestamp is not None else time.time()
|
| 439 |
+
try:
|
| 440 |
+
functools.update_wrapper(self, func)
|
| 441 |
+
except Exception:
|
| 442 |
+
pass # Objects like ufunc don't like that
|
| 443 |
+
if inspect.isfunction(func):
|
| 444 |
+
doc = pydoc.TextDoc().document(func)
|
| 445 |
+
# Remove blank line
|
| 446 |
+
doc = doc.replace("\n", "\n\n", 1)
|
| 447 |
+
# Strip backspace-overprints for compatibility with autodoc
|
| 448 |
+
doc = re.sub("\x08.", "", doc)
|
| 449 |
+
else:
|
| 450 |
+
# Pydoc does a poor job on other objects
|
| 451 |
+
doc = func.__doc__
|
| 452 |
+
self.__doc__ = "Memoized version of %s" % doc
|
| 453 |
+
|
| 454 |
+
self._func_code_info = None
|
| 455 |
+
self._func_code_id = None
|
| 456 |
+
|
| 457 |
+
def _is_in_cache_and_valid(self, call_id):
|
| 458 |
+
"""Check if the function call is cached and valid for given arguments.
|
| 459 |
+
|
| 460 |
+
- Compare the function code with the one from the cached function,
|
| 461 |
+
asserting if it has changed.
|
| 462 |
+
- Check if the function call is present in the cache.
|
| 463 |
+
- Call `cache_validation_callback` for user define cache validation.
|
| 464 |
+
|
| 465 |
+
Returns True if the function call is in cache and can be used, and
|
| 466 |
+
returns False otherwise.
|
| 467 |
+
"""
|
| 468 |
+
# Check if the code of the function has changed
|
| 469 |
+
if not self._check_previous_func_code(stacklevel=4):
|
| 470 |
+
return False
|
| 471 |
+
|
| 472 |
+
# Check if this specific call is in the cache
|
| 473 |
+
if not self.store_backend.contains_item(call_id):
|
| 474 |
+
return False
|
| 475 |
+
|
| 476 |
+
# Call the user defined cache validation callback
|
| 477 |
+
metadata = self.store_backend.get_metadata(call_id)
|
| 478 |
+
if (
|
| 479 |
+
self.cache_validation_callback is not None
|
| 480 |
+
and not self.cache_validation_callback(metadata)
|
| 481 |
+
):
|
| 482 |
+
self.store_backend.clear_item(call_id)
|
| 483 |
+
return False
|
| 484 |
+
|
| 485 |
+
return True
|
| 486 |
+
|
| 487 |
+
def _cached_call(self, args, kwargs, shelving):
|
| 488 |
+
"""Call wrapped function and cache result, or read cache if available.
|
| 489 |
+
|
| 490 |
+
This function returns the wrapped function output or a reference to
|
| 491 |
+
the cached result.
|
| 492 |
+
|
| 493 |
+
Arguments:
|
| 494 |
+
----------
|
| 495 |
+
|
| 496 |
+
args, kwargs: list and dict
|
| 497 |
+
input arguments for wrapped function
|
| 498 |
+
|
| 499 |
+
shelving: bool
|
| 500 |
+
True when called via the call_and_shelve function.
|
| 501 |
+
|
| 502 |
+
|
| 503 |
+
Returns
|
| 504 |
+
-------
|
| 505 |
+
output: Output of the wrapped function if shelving is false, or a
|
| 506 |
+
MemorizedResult reference to the value if shelving is true.
|
| 507 |
+
metadata: dict containing the metadata associated with the call.
|
| 508 |
+
"""
|
| 509 |
+
args_id = self._get_args_id(*args, **kwargs)
|
| 510 |
+
call_id = (self.func_id, args_id)
|
| 511 |
+
_, func_name = get_func_name(self.func)
|
| 512 |
+
func_info = self.store_backend.get_cached_func_info([self.func_id])
|
| 513 |
+
location = func_info["location"]
|
| 514 |
+
|
| 515 |
+
if self._verbose >= 20:
|
| 516 |
+
logging.basicConfig(level=logging.INFO)
|
| 517 |
+
_, signature = format_signature(self.func, *args, **kwargs)
|
| 518 |
+
self.info(
|
| 519 |
+
textwrap.dedent(
|
| 520 |
+
f"""
|
| 521 |
+
Querying {func_name} with signature
|
| 522 |
+
{signature}.
|
| 523 |
+
|
| 524 |
+
(argument hash {args_id})
|
| 525 |
+
|
| 526 |
+
The store location is {location}.
|
| 527 |
+
"""
|
| 528 |
+
)
|
| 529 |
+
)
|
| 530 |
+
|
| 531 |
+
# Compare the function code with the previous to see if the
|
| 532 |
+
# function code has changed and check if the results are present in
|
| 533 |
+
# the cache.
|
| 534 |
+
if self._is_in_cache_and_valid(call_id):
|
| 535 |
+
if shelving:
|
| 536 |
+
return self._get_memorized_result(call_id), {}
|
| 537 |
+
|
| 538 |
+
try:
|
| 539 |
+
start_time = time.time()
|
| 540 |
+
output = self._load_item(call_id)
|
| 541 |
+
if self._verbose > 4:
|
| 542 |
+
self._print_duration(
|
| 543 |
+
time.time() - start_time, context="cache loaded "
|
| 544 |
+
)
|
| 545 |
+
return output, {}
|
| 546 |
+
except Exception:
|
| 547 |
+
# XXX: Should use an exception logger
|
| 548 |
+
_, signature = format_signature(self.func, *args, **kwargs)
|
| 549 |
+
self.warn(
|
| 550 |
+
"Exception while loading results for {}\n {}".format(
|
| 551 |
+
signature, traceback.format_exc()
|
| 552 |
+
)
|
| 553 |
+
)
|
| 554 |
+
|
| 555 |
+
if self._verbose > 10:
|
| 556 |
+
self.warn(
|
| 557 |
+
f"Computing func {func_name}, argument hash {args_id} "
|
| 558 |
+
f"in location {location}"
|
| 559 |
+
)
|
| 560 |
+
|
| 561 |
+
# Returns the output but not the metadata
|
| 562 |
+
return self._call(call_id, args, kwargs, shelving)
|
| 563 |
+
|
| 564 |
+
@property
|
| 565 |
+
def func_code_info(self):
|
| 566 |
+
# 3-tuple property containing: the function source code, source file,
|
| 567 |
+
# and first line of the code inside the source file
|
| 568 |
+
if hasattr(self.func, "__code__"):
|
| 569 |
+
if self._func_code_id is None:
|
| 570 |
+
self._func_code_id = id(self.func.__code__)
|
| 571 |
+
elif id(self.func.__code__) != self._func_code_id:
|
| 572 |
+
# Be robust to dynamic reassignments of self.func.__code__
|
| 573 |
+
self._func_code_info = None
|
| 574 |
+
|
| 575 |
+
if self._func_code_info is None:
|
| 576 |
+
# Cache the source code of self.func . Provided that get_func_code
|
| 577 |
+
# (which should be called once on self) gets called in the process
|
| 578 |
+
# in which self.func was defined, this caching mechanism prevents
|
| 579 |
+
# undesired cache clearing when the cached function is called in
|
| 580 |
+
# an environment where the introspection utilities get_func_code
|
| 581 |
+
# relies on do not work (typically, in joblib child processes).
|
| 582 |
+
# See #1035 for more info
|
| 583 |
+
# TODO (pierreglaser): do the same with get_func_name?
|
| 584 |
+
self._func_code_info = get_func_code(self.func)
|
| 585 |
+
return self._func_code_info
|
| 586 |
+
|
| 587 |
+
def call_and_shelve(self, *args, **kwargs):
|
| 588 |
+
"""Call wrapped function, cache result and return a reference.
|
| 589 |
+
|
| 590 |
+
This method returns a reference to the cached result instead of the
|
| 591 |
+
result itself. The reference object is small and picklable, allowing
|
| 592 |
+
to send or store it easily. Call .get() on reference object to get
|
| 593 |
+
result.
|
| 594 |
+
|
| 595 |
+
Returns
|
| 596 |
+
-------
|
| 597 |
+
cached_result: MemorizedResult or NotMemorizedResult
|
| 598 |
+
reference to the value returned by the wrapped function. The
|
| 599 |
+
class "NotMemorizedResult" is used when there is no cache
|
| 600 |
+
activated (e.g. location=None in Memory).
|
| 601 |
+
"""
|
| 602 |
+
# Return the wrapped output, without the metadata
|
| 603 |
+
return self._cached_call(args, kwargs, shelving=True)[0]
|
| 604 |
+
|
| 605 |
+
def __call__(self, *args, **kwargs):
|
| 606 |
+
# Return the output, without the metadata
|
| 607 |
+
return self._cached_call(args, kwargs, shelving=False)[0]
|
| 608 |
+
|
| 609 |
+
def __getstate__(self):
|
| 610 |
+
# Make sure self.func's source is introspected prior to being pickled -
|
| 611 |
+
# code introspection utilities typically do not work inside child
|
| 612 |
+
# processes
|
| 613 |
+
_ = self.func_code_info
|
| 614 |
+
|
| 615 |
+
# We don't store the timestamp when pickling, to avoid the hash
|
| 616 |
+
# depending from it.
|
| 617 |
+
state = self.__dict__.copy()
|
| 618 |
+
state["timestamp"] = None
|
| 619 |
+
|
| 620 |
+
# Invalidate the code id as id(obj) will be different in the child
|
| 621 |
+
state["_func_code_id"] = None
|
| 622 |
+
|
| 623 |
+
return state
|
| 624 |
+
|
| 625 |
+
def check_call_in_cache(self, *args, **kwargs):
|
| 626 |
+
"""Check if the function call is cached and valid for given arguments.
|
| 627 |
+
|
| 628 |
+
Does not call the function or do any work besides function inspection
|
| 629 |
+
and argument hashing.
|
| 630 |
+
|
| 631 |
+
- Compare the function code with the one from the cached function,
|
| 632 |
+
asserting if it has changed.
|
| 633 |
+
- Check if the function call is present in the cache.
|
| 634 |
+
- Call `cache_validation_callback` for user define cache validation.
|
| 635 |
+
|
| 636 |
+
Returns
|
| 637 |
+
-------
|
| 638 |
+
is_call_in_cache: bool
|
| 639 |
+
Whether or not the function call is in cache and can be used.
|
| 640 |
+
"""
|
| 641 |
+
call_id = (self.func_id, self._get_args_id(*args, **kwargs))
|
| 642 |
+
return self._is_in_cache_and_valid(call_id)
|
| 643 |
+
|
| 644 |
+
# ------------------------------------------------------------------------
|
| 645 |
+
# Private interface
|
| 646 |
+
# ------------------------------------------------------------------------
|
| 647 |
+
|
| 648 |
+
def _get_args_id(self, *args, **kwargs):
|
| 649 |
+
"""Return the input parameter hash of a result."""
|
| 650 |
+
return hashing.hash(
|
| 651 |
+
filter_args(self.func, self.ignore, args, kwargs),
|
| 652 |
+
coerce_mmap=self.mmap_mode is not None,
|
| 653 |
+
)
|
| 654 |
+
|
| 655 |
+
def _hash_func(self):
|
| 656 |
+
"""Hash a function to key the online cache"""
|
| 657 |
+
func_code_h = hash(getattr(self.func, "__code__", None))
|
| 658 |
+
return id(self.func), hash(self.func), func_code_h
|
| 659 |
+
|
| 660 |
+
def _write_func_code(self, func_code, first_line):
|
| 661 |
+
"""Write the function code and the filename to a file."""
|
| 662 |
+
# We store the first line because the filename and the function
|
| 663 |
+
# name is not always enough to identify a function: people
|
| 664 |
+
# sometimes have several functions named the same way in a
|
| 665 |
+
# file. This is bad practice, but joblib should be robust to bad
|
| 666 |
+
# practice.
|
| 667 |
+
func_code = "%s %i\n%s" % (FIRST_LINE_TEXT, first_line, func_code)
|
| 668 |
+
self.store_backend.store_cached_func_code([self.func_id], func_code)
|
| 669 |
+
|
| 670 |
+
# Also store in the in-memory store of function hashes
|
| 671 |
+
is_named_callable = (
|
| 672 |
+
hasattr(self.func, "__name__") and self.func.__name__ != "<lambda>"
|
| 673 |
+
)
|
| 674 |
+
if is_named_callable:
|
| 675 |
+
# Don't do this for lambda functions or strange callable
|
| 676 |
+
# objects, as it ends up being too fragile
|
| 677 |
+
func_hash = self._hash_func()
|
| 678 |
+
try:
|
| 679 |
+
_FUNCTION_HASHES[self.func] = func_hash
|
| 680 |
+
except TypeError:
|
| 681 |
+
# Some callable are not hashable
|
| 682 |
+
pass
|
| 683 |
+
|
| 684 |
+
def _check_previous_func_code(self, stacklevel=2):
|
| 685 |
+
"""
|
| 686 |
+
stacklevel is the depth a which this function is called, to
|
| 687 |
+
issue useful warnings to the user.
|
| 688 |
+
"""
|
| 689 |
+
# First check if our function is in the in-memory store.
|
| 690 |
+
# Using the in-memory store not only makes things faster, but it
|
| 691 |
+
# also renders us robust to variations of the files when the
|
| 692 |
+
# in-memory version of the code does not vary
|
| 693 |
+
try:
|
| 694 |
+
if self.func in _FUNCTION_HASHES:
|
| 695 |
+
# We use as an identifier the id of the function and its
|
| 696 |
+
# hash. This is more likely to falsely change than have hash
|
| 697 |
+
# collisions, thus we are on the safe side.
|
| 698 |
+
func_hash = self._hash_func()
|
| 699 |
+
if func_hash == _FUNCTION_HASHES[self.func]:
|
| 700 |
+
return True
|
| 701 |
+
except TypeError:
|
| 702 |
+
# Some callables are not hashable
|
| 703 |
+
pass
|
| 704 |
+
|
| 705 |
+
# Here, we go through some effort to be robust to dynamically
|
| 706 |
+
# changing code and collision. We cannot inspect.getsource
|
| 707 |
+
# because it is not reliable when using IPython's magic "%run".
|
| 708 |
+
func_code, source_file, first_line = self.func_code_info
|
| 709 |
+
try:
|
| 710 |
+
old_func_code, old_first_line = extract_first_line(
|
| 711 |
+
self.store_backend.get_cached_func_code([self.func_id])
|
| 712 |
+
)
|
| 713 |
+
except (IOError, OSError): # some backend can also raise OSError
|
| 714 |
+
self._write_func_code(func_code, first_line)
|
| 715 |
+
return False
|
| 716 |
+
if old_func_code == func_code:
|
| 717 |
+
return True
|
| 718 |
+
|
| 719 |
+
# We have differing code, is this because we are referring to
|
| 720 |
+
# different functions, or because the function we are referring to has
|
| 721 |
+
# changed?
|
| 722 |
+
|
| 723 |
+
_, func_name = get_func_name(
|
| 724 |
+
self.func, resolv_alias=False, win_characters=False
|
| 725 |
+
)
|
| 726 |
+
if old_first_line == first_line == -1 or func_name == "<lambda>":
|
| 727 |
+
if not first_line == -1:
|
| 728 |
+
func_description = "{0} ({1}:{2})".format(
|
| 729 |
+
func_name, source_file, first_line
|
| 730 |
+
)
|
| 731 |
+
else:
|
| 732 |
+
func_description = func_name
|
| 733 |
+
warnings.warn(
|
| 734 |
+
JobLibCollisionWarning(
|
| 735 |
+
"Cannot detect name collisions for function '{0}'".format(
|
| 736 |
+
func_description
|
| 737 |
+
)
|
| 738 |
+
),
|
| 739 |
+
stacklevel=stacklevel,
|
| 740 |
+
)
|
| 741 |
+
|
| 742 |
+
# Fetch the code at the old location and compare it. If it is the
|
| 743 |
+
# same than the code store, we have a collision: the code in the
|
| 744 |
+
# file has not changed, but the name we have is pointing to a new
|
| 745 |
+
# code block.
|
| 746 |
+
if not old_first_line == first_line and source_file is not None:
|
| 747 |
+
if os.path.exists(source_file):
|
| 748 |
+
_, func_name = get_func_name(self.func, resolv_alias=False)
|
| 749 |
+
num_lines = len(func_code.split("\n"))
|
| 750 |
+
with tokenize.open(source_file) as f:
|
| 751 |
+
on_disk_func_code = f.readlines()[
|
| 752 |
+
old_first_line - 1 : old_first_line - 1 + num_lines - 1
|
| 753 |
+
]
|
| 754 |
+
on_disk_func_code = "".join(on_disk_func_code)
|
| 755 |
+
possible_collision = (
|
| 756 |
+
on_disk_func_code.rstrip() == old_func_code.rstrip()
|
| 757 |
+
)
|
| 758 |
+
else:
|
| 759 |
+
possible_collision = source_file.startswith("<doctest ")
|
| 760 |
+
if possible_collision:
|
| 761 |
+
warnings.warn(
|
| 762 |
+
JobLibCollisionWarning(
|
| 763 |
+
"Possible name collisions between functions "
|
| 764 |
+
"'%s' (%s:%i) and '%s' (%s:%i)"
|
| 765 |
+
% (
|
| 766 |
+
func_name,
|
| 767 |
+
source_file,
|
| 768 |
+
old_first_line,
|
| 769 |
+
func_name,
|
| 770 |
+
source_file,
|
| 771 |
+
first_line,
|
| 772 |
+
)
|
| 773 |
+
),
|
| 774 |
+
stacklevel=stacklevel,
|
| 775 |
+
)
|
| 776 |
+
|
| 777 |
+
# The function has changed, wipe the cache directory.
|
| 778 |
+
# XXX: Should be using warnings, and giving stacklevel
|
| 779 |
+
if self._verbose > 10:
|
| 780 |
+
_, func_name = get_func_name(self.func, resolv_alias=False)
|
| 781 |
+
self.warn(
|
| 782 |
+
"Function {0} (identified by {1}) has changed.".format(
|
| 783 |
+
func_name, self.func_id
|
| 784 |
+
)
|
| 785 |
+
)
|
| 786 |
+
self.clear(warn=True)
|
| 787 |
+
return False
|
| 788 |
+
|
| 789 |
+
def clear(self, warn=True):
|
| 790 |
+
"""Empty the function's cache."""
|
| 791 |
+
func_id = self.func_id
|
| 792 |
+
if self._verbose > 0 and warn:
|
| 793 |
+
self.warn("Clearing function cache identified by %s" % func_id)
|
| 794 |
+
self.store_backend.clear_path(
|
| 795 |
+
[
|
| 796 |
+
func_id,
|
| 797 |
+
]
|
| 798 |
+
)
|
| 799 |
+
|
| 800 |
+
func_code, _, first_line = self.func_code_info
|
| 801 |
+
self._write_func_code(func_code, first_line)
|
| 802 |
+
|
| 803 |
+
def call(self, *args, **kwargs):
|
| 804 |
+
"""Force the execution of the function with the given arguments.
|
| 805 |
+
|
| 806 |
+
The output values will be persisted, i.e., the cache will be updated
|
| 807 |
+
with any new values.
|
| 808 |
+
|
| 809 |
+
Parameters
|
| 810 |
+
----------
|
| 811 |
+
*args: arguments
|
| 812 |
+
The arguments.
|
| 813 |
+
**kwargs: keyword arguments
|
| 814 |
+
Keyword arguments.
|
| 815 |
+
|
| 816 |
+
Returns
|
| 817 |
+
-------
|
| 818 |
+
output : object
|
| 819 |
+
The output of the function call.
|
| 820 |
+
metadata : dict
|
| 821 |
+
The metadata associated with the call.
|
| 822 |
+
"""
|
| 823 |
+
call_id = (self.func_id, self._get_args_id(*args, **kwargs))
|
| 824 |
+
|
| 825 |
+
# Return the output and the metadata
|
| 826 |
+
return self._call(call_id, args, kwargs)
|
| 827 |
+
|
| 828 |
+
def _call(self, call_id, args, kwargs, shelving=False):
|
| 829 |
+
# Return the output and the metadata
|
| 830 |
+
self._before_call(args, kwargs)
|
| 831 |
+
start_time = time.time()
|
| 832 |
+
output = self.func(*args, **kwargs)
|
| 833 |
+
return self._after_call(call_id, args, kwargs, shelving, output, start_time)
|
| 834 |
+
|
| 835 |
+
def _before_call(self, args, kwargs):
|
| 836 |
+
if self._verbose > 0:
|
| 837 |
+
print(format_call(self.func, args, kwargs))
|
| 838 |
+
|
| 839 |
+
def _after_call(self, call_id, args, kwargs, shelving, output, start_time):
|
| 840 |
+
self.store_backend.dump_item(call_id, output, verbose=self._verbose)
|
| 841 |
+
duration = time.time() - start_time
|
| 842 |
+
if self._verbose > 0:
|
| 843 |
+
self._print_duration(duration)
|
| 844 |
+
metadata = self._persist_input(duration, call_id, args, kwargs)
|
| 845 |
+
if shelving:
|
| 846 |
+
return self._get_memorized_result(call_id, metadata), metadata
|
| 847 |
+
|
| 848 |
+
if self.mmap_mode is not None:
|
| 849 |
+
# Memmap the output at the first call to be consistent with
|
| 850 |
+
# later calls
|
| 851 |
+
output = self._load_item(call_id, metadata)
|
| 852 |
+
return output, metadata
|
| 853 |
+
|
| 854 |
+
def _persist_input(self, duration, call_id, args, kwargs, this_duration_limit=0.5):
|
| 855 |
+
"""Save a small summary of the call using json format in the
|
| 856 |
+
output directory.
|
| 857 |
+
|
| 858 |
+
output_dir: string
|
| 859 |
+
directory where to write metadata.
|
| 860 |
+
|
| 861 |
+
duration: float
|
| 862 |
+
time taken by hashing input arguments, calling the wrapped
|
| 863 |
+
function and persisting its output.
|
| 864 |
+
|
| 865 |
+
args, kwargs: list and dict
|
| 866 |
+
input arguments for wrapped function
|
| 867 |
+
|
| 868 |
+
this_duration_limit: float
|
| 869 |
+
Max execution time for this function before issuing a warning.
|
| 870 |
+
"""
|
| 871 |
+
start_time = time.time()
|
| 872 |
+
argument_dict = filter_args(self.func, self.ignore, args, kwargs)
|
| 873 |
+
|
| 874 |
+
input_repr = dict((k, repr(v)) for k, v in argument_dict.items())
|
| 875 |
+
# This can fail due to race-conditions with multiple
|
| 876 |
+
# concurrent joblibs removing the file or the directory
|
| 877 |
+
metadata = {
|
| 878 |
+
"duration": duration,
|
| 879 |
+
"input_args": input_repr,
|
| 880 |
+
"time": start_time,
|
| 881 |
+
}
|
| 882 |
+
|
| 883 |
+
self.store_backend.store_metadata(call_id, metadata)
|
| 884 |
+
|
| 885 |
+
this_duration = time.time() - start_time
|
| 886 |
+
if this_duration > this_duration_limit:
|
| 887 |
+
# This persistence should be fast. It will not be if repr() takes
|
| 888 |
+
# time and its output is large, because json.dump will have to
|
| 889 |
+
# write a large file. This should not be an issue with numpy arrays
|
| 890 |
+
# for which repr() always output a short representation, but can
|
| 891 |
+
# be with complex dictionaries. Fixing the problem should be a
|
| 892 |
+
# matter of replacing repr() above by something smarter.
|
| 893 |
+
warnings.warn(
|
| 894 |
+
"Persisting input arguments took %.2fs to run."
|
| 895 |
+
"If this happens often in your code, it can cause "
|
| 896 |
+
"performance problems "
|
| 897 |
+
"(results will be correct in all cases). "
|
| 898 |
+
"The reason for this is probably some large input "
|
| 899 |
+
"arguments for a wrapped function." % this_duration,
|
| 900 |
+
stacklevel=5,
|
| 901 |
+
)
|
| 902 |
+
return metadata
|
| 903 |
+
|
| 904 |
+
def _get_memorized_result(self, call_id, metadata=None):
|
| 905 |
+
return MemorizedResult(
|
| 906 |
+
self.store_backend,
|
| 907 |
+
call_id,
|
| 908 |
+
metadata=metadata,
|
| 909 |
+
timestamp=self.timestamp,
|
| 910 |
+
verbose=self._verbose - 1,
|
| 911 |
+
)
|
| 912 |
+
|
| 913 |
+
def _load_item(self, call_id, metadata=None):
|
| 914 |
+
return self.store_backend.load_item(
|
| 915 |
+
call_id, metadata=metadata, timestamp=self.timestamp, verbose=self._verbose
|
| 916 |
+
)
|
| 917 |
+
|
| 918 |
+
def _print_duration(self, duration, context=""):
|
| 919 |
+
_, name = get_func_name(self.func)
|
| 920 |
+
msg = f"{name} {context}- {format_time(duration)}"
|
| 921 |
+
print(max(0, (80 - len(msg))) * "_" + msg)
|
| 922 |
+
|
| 923 |
+
# ------------------------------------------------------------------------
|
| 924 |
+
# Private `object` interface
|
| 925 |
+
# ------------------------------------------------------------------------
|
| 926 |
+
|
| 927 |
+
def __repr__(self):
|
| 928 |
+
return "{class_name}(func={func}, location={location})".format(
|
| 929 |
+
class_name=self.__class__.__name__,
|
| 930 |
+
func=self.func,
|
| 931 |
+
location=self.store_backend.location,
|
| 932 |
+
)
|
| 933 |
+
|
| 934 |
+
|
| 935 |
+
###############################################################################
|
| 936 |
+
# class `AsyncMemorizedFunc`
|
| 937 |
+
###############################################################################
|
| 938 |
+
class AsyncMemorizedFunc(MemorizedFunc):
|
| 939 |
+
async def __call__(self, *args, **kwargs):
|
| 940 |
+
out = self._cached_call(args, kwargs, shelving=False)
|
| 941 |
+
out = await out if asyncio.iscoroutine(out) else out
|
| 942 |
+
return out[0] # Don't return metadata
|
| 943 |
+
|
| 944 |
+
async def call_and_shelve(self, *args, **kwargs):
|
| 945 |
+
out = self._cached_call(args, kwargs, shelving=True)
|
| 946 |
+
out = await out if asyncio.iscoroutine(out) else out
|
| 947 |
+
return out[0] # Don't return metadata
|
| 948 |
+
|
| 949 |
+
async def call(self, *args, **kwargs):
|
| 950 |
+
out = super().call(*args, **kwargs)
|
| 951 |
+
return await out if asyncio.iscoroutine(out) else out
|
| 952 |
+
|
| 953 |
+
async def _call(self, call_id, args, kwargs, shelving=False):
|
| 954 |
+
self._before_call(args, kwargs)
|
| 955 |
+
start_time = time.time()
|
| 956 |
+
output = await self.func(*args, **kwargs)
|
| 957 |
+
return self._after_call(call_id, args, kwargs, shelving, output, start_time)
|
| 958 |
+
|
| 959 |
+
|
| 960 |
+
###############################################################################
|
| 961 |
+
# class `Memory`
|
| 962 |
+
###############################################################################
|
| 963 |
+
class Memory(Logger):
|
| 964 |
+
"""A context object for caching a function's return value each time it
|
| 965 |
+
is called with the same input arguments.
|
| 966 |
+
|
| 967 |
+
All values are cached on the filesystem, in a deep directory
|
| 968 |
+
structure.
|
| 969 |
+
|
| 970 |
+
Read more in the :ref:`User Guide <memory>`.
|
| 971 |
+
|
| 972 |
+
Parameters
|
| 973 |
+
----------
|
| 974 |
+
location: str, pathlib.Path or None
|
| 975 |
+
The path of the base directory to use as a data store
|
| 976 |
+
or None. If None is given, no caching is done and
|
| 977 |
+
the Memory object is completely transparent. This option
|
| 978 |
+
replaces cachedir since version 0.12.
|
| 979 |
+
|
| 980 |
+
backend: str, optional, default='local'
|
| 981 |
+
Type of store backend for reading/writing cache files.
|
| 982 |
+
The 'local' backend is using regular filesystem operations to
|
| 983 |
+
manipulate data (open, mv, etc) in the backend.
|
| 984 |
+
|
| 985 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
|
| 986 |
+
The memmapping mode used when loading from cache
|
| 987 |
+
numpy arrays. See numpy.load for the meaning of the
|
| 988 |
+
arguments.
|
| 989 |
+
|
| 990 |
+
compress: boolean, or integer, optional
|
| 991 |
+
Whether to zip the stored data on disk. If an integer is
|
| 992 |
+
given, it should be between 1 and 9, and sets the amount
|
| 993 |
+
of compression. Note that compressed arrays cannot be
|
| 994 |
+
read by memmapping.
|
| 995 |
+
|
| 996 |
+
verbose: int, optional
|
| 997 |
+
Verbosity flag, controls the debug messages that are issued
|
| 998 |
+
as functions are evaluated.
|
| 999 |
+
|
| 1000 |
+
backend_options: dict, optional
|
| 1001 |
+
Contains a dictionary of named parameters used to configure
|
| 1002 |
+
the store backend.
|
| 1003 |
+
"""
|
| 1004 |
+
|
| 1005 |
+
# ------------------------------------------------------------------------
|
| 1006 |
+
# Public interface
|
| 1007 |
+
# ------------------------------------------------------------------------
|
| 1008 |
+
|
| 1009 |
+
def __init__(
|
| 1010 |
+
self,
|
| 1011 |
+
location=None,
|
| 1012 |
+
backend="local",
|
| 1013 |
+
mmap_mode=None,
|
| 1014 |
+
compress=False,
|
| 1015 |
+
verbose=1,
|
| 1016 |
+
backend_options=None,
|
| 1017 |
+
):
|
| 1018 |
+
Logger.__init__(self)
|
| 1019 |
+
self._verbose = verbose
|
| 1020 |
+
self.mmap_mode = mmap_mode
|
| 1021 |
+
self.timestamp = time.time()
|
| 1022 |
+
self.backend = backend
|
| 1023 |
+
self.compress = compress
|
| 1024 |
+
if backend_options is None:
|
| 1025 |
+
backend_options = {}
|
| 1026 |
+
self.backend_options = backend_options
|
| 1027 |
+
|
| 1028 |
+
if compress and mmap_mode is not None:
|
| 1029 |
+
warnings.warn("Compressed results cannot be memmapped", stacklevel=2)
|
| 1030 |
+
|
| 1031 |
+
self.location = location
|
| 1032 |
+
if isinstance(location, str):
|
| 1033 |
+
location = os.path.join(location, "joblib")
|
| 1034 |
+
|
| 1035 |
+
self.store_backend = _store_backend_factory(
|
| 1036 |
+
backend,
|
| 1037 |
+
location,
|
| 1038 |
+
verbose=self._verbose,
|
| 1039 |
+
backend_options=dict(
|
| 1040 |
+
compress=compress, mmap_mode=mmap_mode, **backend_options
|
| 1041 |
+
),
|
| 1042 |
+
)
|
| 1043 |
+
|
| 1044 |
+
def cache(
|
| 1045 |
+
self,
|
| 1046 |
+
func=None,
|
| 1047 |
+
ignore=None,
|
| 1048 |
+
verbose=None,
|
| 1049 |
+
mmap_mode=False,
|
| 1050 |
+
cache_validation_callback=None,
|
| 1051 |
+
):
|
| 1052 |
+
"""Decorates the given function func to only compute its return
|
| 1053 |
+
value for input arguments not cached on disk.
|
| 1054 |
+
|
| 1055 |
+
Parameters
|
| 1056 |
+
----------
|
| 1057 |
+
func: callable, optional
|
| 1058 |
+
The function to be decorated
|
| 1059 |
+
ignore: list of strings
|
| 1060 |
+
A list of arguments name to ignore in the hashing
|
| 1061 |
+
verbose: integer, optional
|
| 1062 |
+
The verbosity mode of the function. By default that
|
| 1063 |
+
of the memory object is used.
|
| 1064 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
|
| 1065 |
+
The memmapping mode used when loading from cache
|
| 1066 |
+
numpy arrays. See numpy.load for the meaning of the
|
| 1067 |
+
arguments. By default that of the memory object is used.
|
| 1068 |
+
cache_validation_callback: callable, optional
|
| 1069 |
+
Callable to validate whether or not the cache is valid. When
|
| 1070 |
+
the cached function is called with arguments for which a cache
|
| 1071 |
+
exists, this callable is called with the metadata of the cached
|
| 1072 |
+
result as its sole argument. If it returns True, then the
|
| 1073 |
+
cached result is returned, else the cache for these arguments
|
| 1074 |
+
is cleared and recomputed.
|
| 1075 |
+
|
| 1076 |
+
Returns
|
| 1077 |
+
-------
|
| 1078 |
+
decorated_func: MemorizedFunc object
|
| 1079 |
+
The returned object is a MemorizedFunc object, that is
|
| 1080 |
+
callable (behaves like a function), but offers extra
|
| 1081 |
+
methods for cache lookup and management. See the
|
| 1082 |
+
documentation for :class:`joblib.memory.MemorizedFunc`.
|
| 1083 |
+
"""
|
| 1084 |
+
if cache_validation_callback is not None and not callable(
|
| 1085 |
+
cache_validation_callback
|
| 1086 |
+
):
|
| 1087 |
+
raise ValueError(
|
| 1088 |
+
"cache_validation_callback needs to be callable. "
|
| 1089 |
+
f"Got {cache_validation_callback}."
|
| 1090 |
+
)
|
| 1091 |
+
if func is None:
|
| 1092 |
+
# Partial application, to be able to specify extra keyword
|
| 1093 |
+
# arguments in decorators
|
| 1094 |
+
return functools.partial(
|
| 1095 |
+
self.cache,
|
| 1096 |
+
ignore=ignore,
|
| 1097 |
+
mmap_mode=mmap_mode,
|
| 1098 |
+
verbose=verbose,
|
| 1099 |
+
cache_validation_callback=cache_validation_callback,
|
| 1100 |
+
)
|
| 1101 |
+
if self.store_backend is None:
|
| 1102 |
+
cls = (
|
| 1103 |
+
AsyncNotMemorizedFunc
|
| 1104 |
+
if inspect.iscoroutinefunction(func)
|
| 1105 |
+
else NotMemorizedFunc
|
| 1106 |
+
)
|
| 1107 |
+
return cls(func)
|
| 1108 |
+
if verbose is None:
|
| 1109 |
+
verbose = self._verbose
|
| 1110 |
+
if mmap_mode is False:
|
| 1111 |
+
mmap_mode = self.mmap_mode
|
| 1112 |
+
if isinstance(func, MemorizedFunc):
|
| 1113 |
+
func = func.func
|
| 1114 |
+
cls = AsyncMemorizedFunc if inspect.iscoroutinefunction(func) else MemorizedFunc
|
| 1115 |
+
return cls(
|
| 1116 |
+
func,
|
| 1117 |
+
location=self.store_backend,
|
| 1118 |
+
backend=self.backend,
|
| 1119 |
+
ignore=ignore,
|
| 1120 |
+
mmap_mode=mmap_mode,
|
| 1121 |
+
compress=self.compress,
|
| 1122 |
+
verbose=verbose,
|
| 1123 |
+
timestamp=self.timestamp,
|
| 1124 |
+
cache_validation_callback=cache_validation_callback,
|
| 1125 |
+
)
|
| 1126 |
+
|
| 1127 |
+
def clear(self, warn=True):
|
| 1128 |
+
"""Erase the complete cache directory."""
|
| 1129 |
+
if warn:
|
| 1130 |
+
self.warn("Flushing completely the cache")
|
| 1131 |
+
if self.store_backend is not None:
|
| 1132 |
+
self.store_backend.clear()
|
| 1133 |
+
|
| 1134 |
+
# As the cache is completely clear, make sure the _FUNCTION_HASHES
|
| 1135 |
+
# cache is also reset. Else, for a function that is present in this
|
| 1136 |
+
# table, results cached after this clear will be have cache miss
|
| 1137 |
+
# as the function code is not re-written.
|
| 1138 |
+
_FUNCTION_HASHES.clear()
|
| 1139 |
+
|
| 1140 |
+
def reduce_size(self, bytes_limit=None, items_limit=None, age_limit=None):
|
| 1141 |
+
"""Remove cache elements to make the cache fit its limits.
|
| 1142 |
+
|
| 1143 |
+
The limitation can impose that the cache size fits in ``bytes_limit``,
|
| 1144 |
+
that the number of cache items is no more than ``items_limit``, and
|
| 1145 |
+
that all files in cache are not older than ``age_limit``.
|
| 1146 |
+
|
| 1147 |
+
Parameters
|
| 1148 |
+
----------
|
| 1149 |
+
bytes_limit: int | str, optional
|
| 1150 |
+
Limit in bytes of the size of the cache. By default, the size of
|
| 1151 |
+
the cache is unlimited. When reducing the size of the cache,
|
| 1152 |
+
``joblib`` keeps the most recently accessed items first. If a
|
| 1153 |
+
str is passed, it is converted to a number of bytes using units
|
| 1154 |
+
{ K | M | G} for kilo, mega, giga.
|
| 1155 |
+
|
| 1156 |
+
items_limit: int, optional
|
| 1157 |
+
Number of items to limit the cache to. By default, the number of
|
| 1158 |
+
items in the cache is unlimited. When reducing the size of the
|
| 1159 |
+
cache, ``joblib`` keeps the most recently accessed items first.
|
| 1160 |
+
|
| 1161 |
+
age_limit: datetime.timedelta, optional
|
| 1162 |
+
Maximum age of items to limit the cache to. When reducing the size
|
| 1163 |
+
of the cache, any items last accessed more than the given length of
|
| 1164 |
+
time ago are deleted. Example: to remove files older than 5 days,
|
| 1165 |
+
use datetime.timedelta(days=5). Negative timedelta are not
|
| 1166 |
+
accepted.
|
| 1167 |
+
"""
|
| 1168 |
+
if self.store_backend is None:
|
| 1169 |
+
# No cached results, this function does nothing.
|
| 1170 |
+
return
|
| 1171 |
+
|
| 1172 |
+
if bytes_limit is None and items_limit is None and age_limit is None:
|
| 1173 |
+
# No limitation to impose, returning
|
| 1174 |
+
return
|
| 1175 |
+
|
| 1176 |
+
# Defers the actual limits enforcing to the store backend.
|
| 1177 |
+
self.store_backend.enforce_store_limits(bytes_limit, items_limit, age_limit)
|
| 1178 |
+
|
| 1179 |
+
def eval(self, func, *args, **kwargs):
|
| 1180 |
+
"""Eval function func with arguments `*args` and `**kwargs`,
|
| 1181 |
+
in the context of the memory.
|
| 1182 |
+
|
| 1183 |
+
This method works similarly to the builtin `apply`, except
|
| 1184 |
+
that the function is called only if the cache is not
|
| 1185 |
+
up to date.
|
| 1186 |
+
|
| 1187 |
+
"""
|
| 1188 |
+
if self.store_backend is None:
|
| 1189 |
+
return func(*args, **kwargs)
|
| 1190 |
+
return self.cache(func)(*args, **kwargs)
|
| 1191 |
+
|
| 1192 |
+
# ------------------------------------------------------------------------
|
| 1193 |
+
# Private `object` interface
|
| 1194 |
+
# ------------------------------------------------------------------------
|
| 1195 |
+
|
| 1196 |
+
def __repr__(self):
|
| 1197 |
+
return "{class_name}(location={location})".format(
|
| 1198 |
+
class_name=self.__class__.__name__,
|
| 1199 |
+
location=(
|
| 1200 |
+
None if self.store_backend is None else self.store_backend.location
|
| 1201 |
+
),
|
| 1202 |
+
)
|
| 1203 |
+
|
| 1204 |
+
def __getstate__(self):
|
| 1205 |
+
"""We don't store the timestamp when pickling, to avoid the hash
|
| 1206 |
+
depending from it.
|
| 1207 |
+
"""
|
| 1208 |
+
state = self.__dict__.copy()
|
| 1209 |
+
state["timestamp"] = None
|
| 1210 |
+
return state
|
| 1211 |
+
|
| 1212 |
+
|
| 1213 |
+
###############################################################################
|
| 1214 |
+
# cache_validation_callback helpers
|
| 1215 |
+
###############################################################################
|
| 1216 |
+
|
| 1217 |
+
|
| 1218 |
+
def expires_after(
|
| 1219 |
+
days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0
|
| 1220 |
+
):
|
| 1221 |
+
"""Helper cache_validation_callback to force recompute after a duration.
|
| 1222 |
+
|
| 1223 |
+
Parameters
|
| 1224 |
+
----------
|
| 1225 |
+
days, seconds, microseconds, milliseconds, minutes, hours, weeks: numbers
|
| 1226 |
+
argument passed to a timedelta.
|
| 1227 |
+
"""
|
| 1228 |
+
delta = datetime.timedelta(
|
| 1229 |
+
days=days,
|
| 1230 |
+
seconds=seconds,
|
| 1231 |
+
microseconds=microseconds,
|
| 1232 |
+
milliseconds=milliseconds,
|
| 1233 |
+
minutes=minutes,
|
| 1234 |
+
hours=hours,
|
| 1235 |
+
weeks=weeks,
|
| 1236 |
+
)
|
| 1237 |
+
|
| 1238 |
+
def cache_validation_callback(metadata):
|
| 1239 |
+
computation_age = time.time() - metadata["time"]
|
| 1240 |
+
return computation_age < delta.total_seconds()
|
| 1241 |
+
|
| 1242 |
+
return cache_validation_callback
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle.py
ADDED
|
@@ -0,0 +1,756 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Utilities for fast persistence of big data, with optional compression."""
|
| 2 |
+
|
| 3 |
+
# Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 4 |
+
# Copyright (c) 2009 Gael Varoquaux
|
| 5 |
+
# License: BSD Style, 3 clauses.
|
| 6 |
+
|
| 7 |
+
import io
|
| 8 |
+
import os
|
| 9 |
+
import pickle
|
| 10 |
+
import warnings
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
|
| 13 |
+
from .backports import make_memmap
|
| 14 |
+
from .compressor import (
|
| 15 |
+
_COMPRESSORS,
|
| 16 |
+
LZ4_NOT_INSTALLED_ERROR,
|
| 17 |
+
BinaryZlibFile,
|
| 18 |
+
BZ2CompressorWrapper,
|
| 19 |
+
GzipCompressorWrapper,
|
| 20 |
+
LZ4CompressorWrapper,
|
| 21 |
+
LZMACompressorWrapper,
|
| 22 |
+
XZCompressorWrapper,
|
| 23 |
+
ZlibCompressorWrapper,
|
| 24 |
+
lz4,
|
| 25 |
+
register_compressor,
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
# For compatibility with old versions of joblib, we need ZNDArrayWrapper
|
| 29 |
+
# to be visible in the current namespace.
|
| 30 |
+
from .numpy_pickle_compat import (
|
| 31 |
+
NDArrayWrapper,
|
| 32 |
+
ZNDArrayWrapper, # noqa: F401
|
| 33 |
+
load_compatibility,
|
| 34 |
+
)
|
| 35 |
+
from .numpy_pickle_utils import (
|
| 36 |
+
BUFFER_SIZE,
|
| 37 |
+
Pickler,
|
| 38 |
+
Unpickler,
|
| 39 |
+
_ensure_native_byte_order,
|
| 40 |
+
_read_bytes,
|
| 41 |
+
_reconstruct,
|
| 42 |
+
_validate_fileobject_and_memmap,
|
| 43 |
+
_write_fileobject,
|
| 44 |
+
)
|
| 45 |
+
|
| 46 |
+
# Register supported compressors
|
| 47 |
+
register_compressor("zlib", ZlibCompressorWrapper())
|
| 48 |
+
register_compressor("gzip", GzipCompressorWrapper())
|
| 49 |
+
register_compressor("bz2", BZ2CompressorWrapper())
|
| 50 |
+
register_compressor("lzma", LZMACompressorWrapper())
|
| 51 |
+
register_compressor("xz", XZCompressorWrapper())
|
| 52 |
+
register_compressor("lz4", LZ4CompressorWrapper())
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
###############################################################################
|
| 56 |
+
# Utility objects for persistence.
|
| 57 |
+
|
| 58 |
+
# For convenience, 16 bytes are used to be sure to cover all the possible
|
| 59 |
+
# dtypes' alignments. For reference, see:
|
| 60 |
+
# https://numpy.org/devdocs/dev/alignment.html
|
| 61 |
+
NUMPY_ARRAY_ALIGNMENT_BYTES = 16
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
class NumpyArrayWrapper(object):
|
| 65 |
+
"""An object to be persisted instead of numpy arrays.
|
| 66 |
+
|
| 67 |
+
This object is used to hack into the pickle machinery and read numpy
|
| 68 |
+
array data from our custom persistence format.
|
| 69 |
+
More precisely, this object is used for:
|
| 70 |
+
* carrying the information of the persisted array: subclass, shape, order,
|
| 71 |
+
dtype. Those ndarray metadata are used to correctly reconstruct the array
|
| 72 |
+
with low level numpy functions.
|
| 73 |
+
* determining if memmap is allowed on the array.
|
| 74 |
+
* reading the array bytes from a file.
|
| 75 |
+
* reading the array using memorymap from a file.
|
| 76 |
+
* writing the array bytes to a file.
|
| 77 |
+
|
| 78 |
+
Attributes
|
| 79 |
+
----------
|
| 80 |
+
subclass: numpy.ndarray subclass
|
| 81 |
+
Determine the subclass of the wrapped array.
|
| 82 |
+
shape: numpy.ndarray shape
|
| 83 |
+
Determine the shape of the wrapped array.
|
| 84 |
+
order: {'C', 'F'}
|
| 85 |
+
Determine the order of wrapped array data. 'C' is for C order, 'F' is
|
| 86 |
+
for fortran order.
|
| 87 |
+
dtype: numpy.ndarray dtype
|
| 88 |
+
Determine the data type of the wrapped array.
|
| 89 |
+
allow_mmap: bool
|
| 90 |
+
Determine if memory mapping is allowed on the wrapped array.
|
| 91 |
+
Default: False.
|
| 92 |
+
"""
|
| 93 |
+
|
| 94 |
+
def __init__(
|
| 95 |
+
self,
|
| 96 |
+
subclass,
|
| 97 |
+
shape,
|
| 98 |
+
order,
|
| 99 |
+
dtype,
|
| 100 |
+
allow_mmap=False,
|
| 101 |
+
numpy_array_alignment_bytes=NUMPY_ARRAY_ALIGNMENT_BYTES,
|
| 102 |
+
):
|
| 103 |
+
"""Constructor. Store the useful information for later."""
|
| 104 |
+
self.subclass = subclass
|
| 105 |
+
self.shape = shape
|
| 106 |
+
self.order = order
|
| 107 |
+
self.dtype = dtype
|
| 108 |
+
self.allow_mmap = allow_mmap
|
| 109 |
+
# We make numpy_array_alignment_bytes an instance attribute to allow us
|
| 110 |
+
# to change our mind about the default alignment and still load the old
|
| 111 |
+
# pickles (with the previous alignment) correctly
|
| 112 |
+
self.numpy_array_alignment_bytes = numpy_array_alignment_bytes
|
| 113 |
+
|
| 114 |
+
def safe_get_numpy_array_alignment_bytes(self):
|
| 115 |
+
# NumpyArrayWrapper instances loaded from joblib <= 1.1 pickles don't
|
| 116 |
+
# have an numpy_array_alignment_bytes attribute
|
| 117 |
+
return getattr(self, "numpy_array_alignment_bytes", None)
|
| 118 |
+
|
| 119 |
+
def write_array(self, array, pickler):
|
| 120 |
+
"""Write array bytes to pickler file handle.
|
| 121 |
+
|
| 122 |
+
This function is an adaptation of the numpy write_array function
|
| 123 |
+
available in version 1.10.1 in numpy/lib/format.py.
|
| 124 |
+
"""
|
| 125 |
+
# Set buffer size to 16 MiB to hide the Python loop overhead.
|
| 126 |
+
buffersize = max(16 * 1024**2 // array.itemsize, 1)
|
| 127 |
+
if array.dtype.hasobject:
|
| 128 |
+
# We contain Python objects so we cannot write out the data
|
| 129 |
+
# directly. Instead, we will pickle it out with version 5 of the
|
| 130 |
+
# pickle protocol.
|
| 131 |
+
pickle.dump(array, pickler.file_handle, protocol=5)
|
| 132 |
+
else:
|
| 133 |
+
numpy_array_alignment_bytes = self.safe_get_numpy_array_alignment_bytes()
|
| 134 |
+
if numpy_array_alignment_bytes is not None:
|
| 135 |
+
current_pos = pickler.file_handle.tell()
|
| 136 |
+
pos_after_padding_byte = current_pos + 1
|
| 137 |
+
padding_length = numpy_array_alignment_bytes - (
|
| 138 |
+
pos_after_padding_byte % numpy_array_alignment_bytes
|
| 139 |
+
)
|
| 140 |
+
# A single byte is written that contains the padding length in
|
| 141 |
+
# bytes
|
| 142 |
+
padding_length_byte = int.to_bytes(
|
| 143 |
+
padding_length, length=1, byteorder="little"
|
| 144 |
+
)
|
| 145 |
+
pickler.file_handle.write(padding_length_byte)
|
| 146 |
+
|
| 147 |
+
if padding_length != 0:
|
| 148 |
+
padding = b"\xff" * padding_length
|
| 149 |
+
pickler.file_handle.write(padding)
|
| 150 |
+
|
| 151 |
+
for chunk in pickler.np.nditer(
|
| 152 |
+
array,
|
| 153 |
+
flags=["external_loop", "buffered", "zerosize_ok"],
|
| 154 |
+
buffersize=buffersize,
|
| 155 |
+
order=self.order,
|
| 156 |
+
):
|
| 157 |
+
pickler.file_handle.write(chunk.tobytes("C"))
|
| 158 |
+
|
| 159 |
+
def read_array(self, unpickler, ensure_native_byte_order):
|
| 160 |
+
"""Read array from unpickler file handle.
|
| 161 |
+
|
| 162 |
+
This function is an adaptation of the numpy read_array function
|
| 163 |
+
available in version 1.10.1 in numpy/lib/format.py.
|
| 164 |
+
"""
|
| 165 |
+
if len(self.shape) == 0:
|
| 166 |
+
count = 1
|
| 167 |
+
else:
|
| 168 |
+
# joblib issue #859: we cast the elements of self.shape to int64 to
|
| 169 |
+
# prevent a potential overflow when computing their product.
|
| 170 |
+
shape_int64 = [unpickler.np.int64(x) for x in self.shape]
|
| 171 |
+
count = unpickler.np.multiply.reduce(shape_int64)
|
| 172 |
+
# Now read the actual data.
|
| 173 |
+
if self.dtype.hasobject:
|
| 174 |
+
# The array contained Python objects. We need to unpickle the data.
|
| 175 |
+
array = pickle.load(unpickler.file_handle)
|
| 176 |
+
else:
|
| 177 |
+
numpy_array_alignment_bytes = self.safe_get_numpy_array_alignment_bytes()
|
| 178 |
+
if numpy_array_alignment_bytes is not None:
|
| 179 |
+
padding_byte = unpickler.file_handle.read(1)
|
| 180 |
+
padding_length = int.from_bytes(padding_byte, byteorder="little")
|
| 181 |
+
if padding_length != 0:
|
| 182 |
+
unpickler.file_handle.read(padding_length)
|
| 183 |
+
|
| 184 |
+
# This is not a real file. We have to read it the
|
| 185 |
+
# memory-intensive way.
|
| 186 |
+
# crc32 module fails on reads greater than 2 ** 32 bytes,
|
| 187 |
+
# breaking large reads from gzip streams. Chunk reads to
|
| 188 |
+
# BUFFER_SIZE bytes to avoid issue and reduce memory overhead
|
| 189 |
+
# of the read. In non-chunked case count < max_read_count, so
|
| 190 |
+
# only one read is performed.
|
| 191 |
+
max_read_count = BUFFER_SIZE // min(BUFFER_SIZE, self.dtype.itemsize)
|
| 192 |
+
|
| 193 |
+
array = unpickler.np.empty(count, dtype=self.dtype)
|
| 194 |
+
for i in range(0, count, max_read_count):
|
| 195 |
+
read_count = min(max_read_count, count - i)
|
| 196 |
+
read_size = int(read_count * self.dtype.itemsize)
|
| 197 |
+
data = _read_bytes(unpickler.file_handle, read_size, "array data")
|
| 198 |
+
array[i : i + read_count] = unpickler.np.frombuffer(
|
| 199 |
+
data, dtype=self.dtype, count=read_count
|
| 200 |
+
)
|
| 201 |
+
del data
|
| 202 |
+
|
| 203 |
+
if self.order == "F":
|
| 204 |
+
array.shape = self.shape[::-1]
|
| 205 |
+
array = array.transpose()
|
| 206 |
+
else:
|
| 207 |
+
array.shape = self.shape
|
| 208 |
+
|
| 209 |
+
if ensure_native_byte_order:
|
| 210 |
+
# Detect byte order mismatch and swap as needed.
|
| 211 |
+
array = _ensure_native_byte_order(array)
|
| 212 |
+
|
| 213 |
+
return array
|
| 214 |
+
|
| 215 |
+
def read_mmap(self, unpickler):
|
| 216 |
+
"""Read an array using numpy memmap."""
|
| 217 |
+
current_pos = unpickler.file_handle.tell()
|
| 218 |
+
offset = current_pos
|
| 219 |
+
numpy_array_alignment_bytes = self.safe_get_numpy_array_alignment_bytes()
|
| 220 |
+
|
| 221 |
+
if numpy_array_alignment_bytes is not None:
|
| 222 |
+
padding_byte = unpickler.file_handle.read(1)
|
| 223 |
+
padding_length = int.from_bytes(padding_byte, byteorder="little")
|
| 224 |
+
# + 1 is for the padding byte
|
| 225 |
+
offset += padding_length + 1
|
| 226 |
+
|
| 227 |
+
if unpickler.mmap_mode == "w+":
|
| 228 |
+
unpickler.mmap_mode = "r+"
|
| 229 |
+
|
| 230 |
+
marray = make_memmap(
|
| 231 |
+
unpickler.filename,
|
| 232 |
+
dtype=self.dtype,
|
| 233 |
+
shape=self.shape,
|
| 234 |
+
order=self.order,
|
| 235 |
+
mode=unpickler.mmap_mode,
|
| 236 |
+
offset=offset,
|
| 237 |
+
)
|
| 238 |
+
# update the offset so that it corresponds to the end of the read array
|
| 239 |
+
unpickler.file_handle.seek(offset + marray.nbytes)
|
| 240 |
+
|
| 241 |
+
if (
|
| 242 |
+
numpy_array_alignment_bytes is None
|
| 243 |
+
and current_pos % NUMPY_ARRAY_ALIGNMENT_BYTES != 0
|
| 244 |
+
):
|
| 245 |
+
message = (
|
| 246 |
+
f"The memmapped array {marray} loaded from the file "
|
| 247 |
+
f"{unpickler.file_handle.name} is not byte aligned. "
|
| 248 |
+
"This may cause segmentation faults if this memmapped array "
|
| 249 |
+
"is used in some libraries like BLAS or PyTorch. "
|
| 250 |
+
"To get rid of this warning, regenerate your pickle file "
|
| 251 |
+
"with joblib >= 1.2.0. "
|
| 252 |
+
"See https://github.com/joblib/joblib/issues/563 "
|
| 253 |
+
"for more details"
|
| 254 |
+
)
|
| 255 |
+
warnings.warn(message)
|
| 256 |
+
|
| 257 |
+
return marray
|
| 258 |
+
|
| 259 |
+
def read(self, unpickler, ensure_native_byte_order):
|
| 260 |
+
"""Read the array corresponding to this wrapper.
|
| 261 |
+
|
| 262 |
+
Use the unpickler to get all information to correctly read the array.
|
| 263 |
+
|
| 264 |
+
Parameters
|
| 265 |
+
----------
|
| 266 |
+
unpickler: NumpyUnpickler
|
| 267 |
+
ensure_native_byte_order: bool
|
| 268 |
+
If true, coerce the array to use the native endianness of the
|
| 269 |
+
host system.
|
| 270 |
+
|
| 271 |
+
Returns
|
| 272 |
+
-------
|
| 273 |
+
array: numpy.ndarray
|
| 274 |
+
|
| 275 |
+
"""
|
| 276 |
+
# When requested, only use memmap mode if allowed.
|
| 277 |
+
if unpickler.mmap_mode is not None and self.allow_mmap:
|
| 278 |
+
assert not ensure_native_byte_order, (
|
| 279 |
+
"Memmaps cannot be coerced to a given byte order, "
|
| 280 |
+
"this code path is impossible."
|
| 281 |
+
)
|
| 282 |
+
array = self.read_mmap(unpickler)
|
| 283 |
+
else:
|
| 284 |
+
array = self.read_array(unpickler, ensure_native_byte_order)
|
| 285 |
+
|
| 286 |
+
# Manage array subclass case
|
| 287 |
+
if hasattr(array, "__array_prepare__") and self.subclass not in (
|
| 288 |
+
unpickler.np.ndarray,
|
| 289 |
+
unpickler.np.memmap,
|
| 290 |
+
):
|
| 291 |
+
# We need to reconstruct another subclass
|
| 292 |
+
new_array = _reconstruct(self.subclass, (0,), "b")
|
| 293 |
+
return new_array.__array_prepare__(array)
|
| 294 |
+
else:
|
| 295 |
+
return array
|
| 296 |
+
|
| 297 |
+
|
| 298 |
+
###############################################################################
|
| 299 |
+
# Pickler classes
|
| 300 |
+
|
| 301 |
+
|
| 302 |
+
class NumpyPickler(Pickler):
|
| 303 |
+
"""A pickler to persist big data efficiently.
|
| 304 |
+
|
| 305 |
+
The main features of this object are:
|
| 306 |
+
* persistence of numpy arrays in a single file.
|
| 307 |
+
* optional compression with a special care on avoiding memory copies.
|
| 308 |
+
|
| 309 |
+
Attributes
|
| 310 |
+
----------
|
| 311 |
+
fp: file
|
| 312 |
+
File object handle used for serializing the input object.
|
| 313 |
+
protocol: int, optional
|
| 314 |
+
Pickle protocol used. Default is pickle.DEFAULT_PROTOCOL.
|
| 315 |
+
"""
|
| 316 |
+
|
| 317 |
+
dispatch = Pickler.dispatch.copy()
|
| 318 |
+
|
| 319 |
+
def __init__(self, fp, protocol=None):
|
| 320 |
+
self.file_handle = fp
|
| 321 |
+
self.buffered = isinstance(self.file_handle, BinaryZlibFile)
|
| 322 |
+
|
| 323 |
+
# By default we want a pickle protocol that only changes with
|
| 324 |
+
# the major python version and not the minor one
|
| 325 |
+
if protocol is None:
|
| 326 |
+
protocol = pickle.DEFAULT_PROTOCOL
|
| 327 |
+
|
| 328 |
+
Pickler.__init__(self, self.file_handle, protocol=protocol)
|
| 329 |
+
# delayed import of numpy, to avoid tight coupling
|
| 330 |
+
try:
|
| 331 |
+
import numpy as np
|
| 332 |
+
except ImportError:
|
| 333 |
+
np = None
|
| 334 |
+
self.np = np
|
| 335 |
+
|
| 336 |
+
def _create_array_wrapper(self, array):
|
| 337 |
+
"""Create and returns a numpy array wrapper from a numpy array."""
|
| 338 |
+
order = (
|
| 339 |
+
"F" if (array.flags.f_contiguous and not array.flags.c_contiguous) else "C"
|
| 340 |
+
)
|
| 341 |
+
allow_mmap = not self.buffered and not array.dtype.hasobject
|
| 342 |
+
|
| 343 |
+
kwargs = {}
|
| 344 |
+
try:
|
| 345 |
+
self.file_handle.tell()
|
| 346 |
+
except io.UnsupportedOperation:
|
| 347 |
+
kwargs = {"numpy_array_alignment_bytes": None}
|
| 348 |
+
|
| 349 |
+
wrapper = NumpyArrayWrapper(
|
| 350 |
+
type(array),
|
| 351 |
+
array.shape,
|
| 352 |
+
order,
|
| 353 |
+
array.dtype,
|
| 354 |
+
allow_mmap=allow_mmap,
|
| 355 |
+
**kwargs,
|
| 356 |
+
)
|
| 357 |
+
|
| 358 |
+
return wrapper
|
| 359 |
+
|
| 360 |
+
def save(self, obj):
|
| 361 |
+
"""Subclass the Pickler `save` method.
|
| 362 |
+
|
| 363 |
+
This is a total abuse of the Pickler class in order to use the numpy
|
| 364 |
+
persistence function `save` instead of the default pickle
|
| 365 |
+
implementation. The numpy array is replaced by a custom wrapper in the
|
| 366 |
+
pickle persistence stack and the serialized array is written right
|
| 367 |
+
after in the file. Warning: the file produced does not follow the
|
| 368 |
+
pickle format. As such it can not be read with `pickle.load`.
|
| 369 |
+
"""
|
| 370 |
+
if self.np is not None and type(obj) in (
|
| 371 |
+
self.np.ndarray,
|
| 372 |
+
self.np.matrix,
|
| 373 |
+
self.np.memmap,
|
| 374 |
+
):
|
| 375 |
+
if type(obj) is self.np.memmap:
|
| 376 |
+
# Pickling doesn't work with memmapped arrays
|
| 377 |
+
obj = self.np.asanyarray(obj)
|
| 378 |
+
|
| 379 |
+
# The array wrapper is pickled instead of the real array.
|
| 380 |
+
wrapper = self._create_array_wrapper(obj)
|
| 381 |
+
Pickler.save(self, wrapper)
|
| 382 |
+
|
| 383 |
+
# A framer was introduced with pickle protocol 4 and we want to
|
| 384 |
+
# ensure the wrapper object is written before the numpy array
|
| 385 |
+
# buffer in the pickle file.
|
| 386 |
+
# See https://www.python.org/dev/peps/pep-3154/#framing to get
|
| 387 |
+
# more information on the framer behavior.
|
| 388 |
+
if self.proto >= 4:
|
| 389 |
+
self.framer.commit_frame(force=True)
|
| 390 |
+
|
| 391 |
+
# And then array bytes are written right after the wrapper.
|
| 392 |
+
wrapper.write_array(obj, self)
|
| 393 |
+
return
|
| 394 |
+
|
| 395 |
+
return Pickler.save(self, obj)
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
class NumpyUnpickler(Unpickler):
|
| 399 |
+
"""A subclass of the Unpickler to unpickle our numpy pickles.
|
| 400 |
+
|
| 401 |
+
Attributes
|
| 402 |
+
----------
|
| 403 |
+
mmap_mode: str
|
| 404 |
+
The memorymap mode to use for reading numpy arrays.
|
| 405 |
+
file_handle: file_like
|
| 406 |
+
File object to unpickle from.
|
| 407 |
+
ensure_native_byte_order: bool
|
| 408 |
+
If True, coerce the array to use the native endianness of the
|
| 409 |
+
host system.
|
| 410 |
+
filename: str
|
| 411 |
+
Name of the file to unpickle from. It should correspond to file_handle.
|
| 412 |
+
This parameter is required when using mmap_mode.
|
| 413 |
+
np: module
|
| 414 |
+
Reference to numpy module if numpy is installed else None.
|
| 415 |
+
|
| 416 |
+
"""
|
| 417 |
+
|
| 418 |
+
dispatch = Unpickler.dispatch.copy()
|
| 419 |
+
|
| 420 |
+
def __init__(self, filename, file_handle, ensure_native_byte_order, mmap_mode=None):
|
| 421 |
+
# The next line is for backward compatibility with pickle generated
|
| 422 |
+
# with joblib versions less than 0.10.
|
| 423 |
+
self._dirname = os.path.dirname(filename)
|
| 424 |
+
|
| 425 |
+
self.mmap_mode = mmap_mode
|
| 426 |
+
self.file_handle = file_handle
|
| 427 |
+
# filename is required for numpy mmap mode.
|
| 428 |
+
self.filename = filename
|
| 429 |
+
self.compat_mode = False
|
| 430 |
+
self.ensure_native_byte_order = ensure_native_byte_order
|
| 431 |
+
Unpickler.__init__(self, self.file_handle)
|
| 432 |
+
try:
|
| 433 |
+
import numpy as np
|
| 434 |
+
except ImportError:
|
| 435 |
+
np = None
|
| 436 |
+
self.np = np
|
| 437 |
+
|
| 438 |
+
def load_build(self):
|
| 439 |
+
"""Called to set the state of a newly created object.
|
| 440 |
+
|
| 441 |
+
We capture it to replace our place-holder objects, NDArrayWrapper or
|
| 442 |
+
NumpyArrayWrapper, by the array we are interested in. We
|
| 443 |
+
replace them directly in the stack of pickler.
|
| 444 |
+
NDArrayWrapper is used for backward compatibility with joblib <= 0.9.
|
| 445 |
+
"""
|
| 446 |
+
Unpickler.load_build(self)
|
| 447 |
+
|
| 448 |
+
# For backward compatibility, we support NDArrayWrapper objects.
|
| 449 |
+
if isinstance(self.stack[-1], (NDArrayWrapper, NumpyArrayWrapper)):
|
| 450 |
+
if self.np is None:
|
| 451 |
+
raise ImportError(
|
| 452 |
+
"Trying to unpickle an ndarray, but numpy didn't import correctly"
|
| 453 |
+
)
|
| 454 |
+
array_wrapper = self.stack.pop()
|
| 455 |
+
# If any NDArrayWrapper is found, we switch to compatibility mode,
|
| 456 |
+
# this will be used to raise a DeprecationWarning to the user at
|
| 457 |
+
# the end of the unpickling.
|
| 458 |
+
if isinstance(array_wrapper, NDArrayWrapper):
|
| 459 |
+
self.compat_mode = True
|
| 460 |
+
_array_payload = array_wrapper.read(self)
|
| 461 |
+
else:
|
| 462 |
+
_array_payload = array_wrapper.read(self, self.ensure_native_byte_order)
|
| 463 |
+
|
| 464 |
+
self.stack.append(_array_payload)
|
| 465 |
+
|
| 466 |
+
# Be careful to register our new method.
|
| 467 |
+
dispatch[pickle.BUILD[0]] = load_build
|
| 468 |
+
|
| 469 |
+
|
| 470 |
+
###############################################################################
|
| 471 |
+
# Utility functions
|
| 472 |
+
|
| 473 |
+
|
| 474 |
+
def dump(value, filename, compress=0, protocol=None):
|
| 475 |
+
"""Persist an arbitrary Python object into one file.
|
| 476 |
+
|
| 477 |
+
Read more in the :ref:`User Guide <persistence>`.
|
| 478 |
+
|
| 479 |
+
Parameters
|
| 480 |
+
----------
|
| 481 |
+
value: any Python object
|
| 482 |
+
The object to store to disk.
|
| 483 |
+
filename: str, pathlib.Path, or file object.
|
| 484 |
+
The file object or path of the file in which it is to be stored.
|
| 485 |
+
The compression method corresponding to one of the supported filename
|
| 486 |
+
extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used
|
| 487 |
+
automatically.
|
| 488 |
+
compress: int from 0 to 9 or bool or 2-tuple, optional
|
| 489 |
+
Optional compression level for the data. 0 or False is no compression.
|
| 490 |
+
Higher value means more compression, but also slower read and
|
| 491 |
+
write times. Using a value of 3 is often a good compromise.
|
| 492 |
+
See the notes for more details.
|
| 493 |
+
If compress is True, the compression level used is 3.
|
| 494 |
+
If compress is a 2-tuple, the first element must correspond to a string
|
| 495 |
+
between supported compressors (e.g 'zlib', 'gzip', 'bz2', 'lzma'
|
| 496 |
+
'xz'), the second element must be an integer from 0 to 9, corresponding
|
| 497 |
+
to the compression level.
|
| 498 |
+
protocol: int, optional
|
| 499 |
+
Pickle protocol, see pickle.dump documentation for more details.
|
| 500 |
+
|
| 501 |
+
Returns
|
| 502 |
+
-------
|
| 503 |
+
filenames: list of strings
|
| 504 |
+
The list of file names in which the data is stored. If
|
| 505 |
+
compress is false, each array is stored in a different file.
|
| 506 |
+
|
| 507 |
+
See Also
|
| 508 |
+
--------
|
| 509 |
+
joblib.load : corresponding loader
|
| 510 |
+
|
| 511 |
+
Notes
|
| 512 |
+
-----
|
| 513 |
+
Memmapping on load cannot be used for compressed files. Thus
|
| 514 |
+
using compression can significantly slow down loading. In
|
| 515 |
+
addition, compressed files take up extra memory during
|
| 516 |
+
dump and load.
|
| 517 |
+
|
| 518 |
+
"""
|
| 519 |
+
|
| 520 |
+
if Path is not None and isinstance(filename, Path):
|
| 521 |
+
filename = str(filename)
|
| 522 |
+
|
| 523 |
+
is_filename = isinstance(filename, str)
|
| 524 |
+
is_fileobj = hasattr(filename, "write")
|
| 525 |
+
|
| 526 |
+
compress_method = "zlib" # zlib is the default compression method.
|
| 527 |
+
if compress is True:
|
| 528 |
+
# By default, if compress is enabled, we want the default compress
|
| 529 |
+
# level of the compressor.
|
| 530 |
+
compress_level = None
|
| 531 |
+
elif isinstance(compress, tuple):
|
| 532 |
+
# a 2-tuple was set in compress
|
| 533 |
+
if len(compress) != 2:
|
| 534 |
+
raise ValueError(
|
| 535 |
+
"Compress argument tuple should contain exactly 2 elements: "
|
| 536 |
+
"(compress method, compress level), you passed {}".format(compress)
|
| 537 |
+
)
|
| 538 |
+
compress_method, compress_level = compress
|
| 539 |
+
elif isinstance(compress, str):
|
| 540 |
+
compress_method = compress
|
| 541 |
+
compress_level = None # Use default compress level
|
| 542 |
+
compress = (compress_method, compress_level)
|
| 543 |
+
else:
|
| 544 |
+
compress_level = compress
|
| 545 |
+
|
| 546 |
+
if compress_method == "lz4" and lz4 is None:
|
| 547 |
+
raise ValueError(LZ4_NOT_INSTALLED_ERROR)
|
| 548 |
+
|
| 549 |
+
if (
|
| 550 |
+
compress_level is not None
|
| 551 |
+
and compress_level is not False
|
| 552 |
+
and compress_level not in range(10)
|
| 553 |
+
):
|
| 554 |
+
# Raising an error if a non valid compress level is given.
|
| 555 |
+
raise ValueError(
|
| 556 |
+
'Non valid compress level given: "{}". Possible values are {}.'.format(
|
| 557 |
+
compress_level, list(range(10))
|
| 558 |
+
)
|
| 559 |
+
)
|
| 560 |
+
|
| 561 |
+
if compress_method not in _COMPRESSORS:
|
| 562 |
+
# Raising an error if an unsupported compression method is given.
|
| 563 |
+
raise ValueError(
|
| 564 |
+
'Non valid compression method given: "{}". Possible values are {}.'.format(
|
| 565 |
+
compress_method, _COMPRESSORS
|
| 566 |
+
)
|
| 567 |
+
)
|
| 568 |
+
|
| 569 |
+
if not is_filename and not is_fileobj:
|
| 570 |
+
# People keep inverting arguments, and the resulting error is
|
| 571 |
+
# incomprehensible
|
| 572 |
+
raise ValueError(
|
| 573 |
+
"Second argument should be a filename or a file-like object, "
|
| 574 |
+
"%s (type %s) was given." % (filename, type(filename))
|
| 575 |
+
)
|
| 576 |
+
|
| 577 |
+
if is_filename and not isinstance(compress, tuple):
|
| 578 |
+
# In case no explicit compression was requested using both compression
|
| 579 |
+
# method and level in a tuple and the filename has an explicit
|
| 580 |
+
# extension, we select the corresponding compressor.
|
| 581 |
+
|
| 582 |
+
# unset the variable to be sure no compression level is set afterwards.
|
| 583 |
+
compress_method = None
|
| 584 |
+
for name, compressor in _COMPRESSORS.items():
|
| 585 |
+
if filename.endswith(compressor.extension):
|
| 586 |
+
compress_method = name
|
| 587 |
+
|
| 588 |
+
if compress_method in _COMPRESSORS and compress_level == 0:
|
| 589 |
+
# we choose the default compress_level in case it was not given
|
| 590 |
+
# as an argument (using compress).
|
| 591 |
+
compress_level = None
|
| 592 |
+
|
| 593 |
+
if compress_level != 0:
|
| 594 |
+
with _write_fileobject(
|
| 595 |
+
filename, compress=(compress_method, compress_level)
|
| 596 |
+
) as f:
|
| 597 |
+
NumpyPickler(f, protocol=protocol).dump(value)
|
| 598 |
+
elif is_filename:
|
| 599 |
+
with open(filename, "wb") as f:
|
| 600 |
+
NumpyPickler(f, protocol=protocol).dump(value)
|
| 601 |
+
else:
|
| 602 |
+
NumpyPickler(filename, protocol=protocol).dump(value)
|
| 603 |
+
|
| 604 |
+
# If the target container is a file object, nothing is returned.
|
| 605 |
+
if is_fileobj:
|
| 606 |
+
return
|
| 607 |
+
|
| 608 |
+
# For compatibility, the list of created filenames (e.g with one element
|
| 609 |
+
# after 0.10.0) is returned by default.
|
| 610 |
+
return [filename]
|
| 611 |
+
|
| 612 |
+
|
| 613 |
+
def _unpickle(fobj, ensure_native_byte_order, filename="", mmap_mode=None):
|
| 614 |
+
"""Internal unpickling function."""
|
| 615 |
+
# We are careful to open the file handle early and keep it open to
|
| 616 |
+
# avoid race-conditions on renames.
|
| 617 |
+
# That said, if data is stored in companion files, which can be
|
| 618 |
+
# the case with the old persistence format, moving the directory
|
| 619 |
+
# will create a race when joblib tries to access the companion
|
| 620 |
+
# files.
|
| 621 |
+
unpickler = NumpyUnpickler(
|
| 622 |
+
filename, fobj, ensure_native_byte_order, mmap_mode=mmap_mode
|
| 623 |
+
)
|
| 624 |
+
obj = None
|
| 625 |
+
try:
|
| 626 |
+
obj = unpickler.load()
|
| 627 |
+
if unpickler.compat_mode:
|
| 628 |
+
warnings.warn(
|
| 629 |
+
"The file '%s' has been generated with a "
|
| 630 |
+
"joblib version less than 0.10. "
|
| 631 |
+
"Please regenerate this pickle file." % filename,
|
| 632 |
+
DeprecationWarning,
|
| 633 |
+
stacklevel=3,
|
| 634 |
+
)
|
| 635 |
+
except UnicodeDecodeError as exc:
|
| 636 |
+
# More user-friendly error message
|
| 637 |
+
new_exc = ValueError(
|
| 638 |
+
"You may be trying to read with "
|
| 639 |
+
"python 3 a joblib pickle generated with python 2. "
|
| 640 |
+
"This feature is not supported by joblib."
|
| 641 |
+
)
|
| 642 |
+
new_exc.__cause__ = exc
|
| 643 |
+
raise new_exc
|
| 644 |
+
return obj
|
| 645 |
+
|
| 646 |
+
|
| 647 |
+
def load_temporary_memmap(filename, mmap_mode, unlink_on_gc_collect):
|
| 648 |
+
from ._memmapping_reducer import JOBLIB_MMAPS, add_maybe_unlink_finalizer
|
| 649 |
+
|
| 650 |
+
with open(filename, "rb") as f:
|
| 651 |
+
with _validate_fileobject_and_memmap(f, filename, mmap_mode) as (
|
| 652 |
+
fobj,
|
| 653 |
+
validated_mmap_mode,
|
| 654 |
+
):
|
| 655 |
+
# Memmap are used for interprocess communication, which should
|
| 656 |
+
# keep the objects untouched. We pass `ensure_native_byte_order=False`
|
| 657 |
+
# to remain consistent with the loading behavior of non-memmaped arrays
|
| 658 |
+
# in workers, where the byte order is preserved.
|
| 659 |
+
# Note that we do not implement endianness change for memmaps, as this
|
| 660 |
+
# would result in inconsistent behavior.
|
| 661 |
+
obj = _unpickle(
|
| 662 |
+
fobj,
|
| 663 |
+
ensure_native_byte_order=False,
|
| 664 |
+
filename=filename,
|
| 665 |
+
mmap_mode=validated_mmap_mode,
|
| 666 |
+
)
|
| 667 |
+
|
| 668 |
+
JOBLIB_MMAPS.add(obj.filename)
|
| 669 |
+
if unlink_on_gc_collect:
|
| 670 |
+
add_maybe_unlink_finalizer(obj)
|
| 671 |
+
return obj
|
| 672 |
+
|
| 673 |
+
|
| 674 |
+
def load(filename, mmap_mode=None, ensure_native_byte_order="auto"):
|
| 675 |
+
"""Reconstruct a Python object from a file persisted with joblib.dump.
|
| 676 |
+
|
| 677 |
+
Read more in the :ref:`User Guide <persistence>`.
|
| 678 |
+
|
| 679 |
+
WARNING: joblib.load relies on the pickle module and can therefore
|
| 680 |
+
execute arbitrary Python code. It should therefore never be used
|
| 681 |
+
to load files from untrusted sources.
|
| 682 |
+
|
| 683 |
+
Parameters
|
| 684 |
+
----------
|
| 685 |
+
filename: str, pathlib.Path, or file object.
|
| 686 |
+
The file object or path of the file from which to load the object
|
| 687 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
|
| 688 |
+
If not None, the arrays are memory-mapped from the disk. This
|
| 689 |
+
mode has no effect for compressed files. Note that in this
|
| 690 |
+
case the reconstructed object might no longer match exactly
|
| 691 |
+
the originally pickled object.
|
| 692 |
+
ensure_native_byte_order: bool, or 'auto', default=='auto'
|
| 693 |
+
If True, ensures that the byte order of the loaded arrays matches the
|
| 694 |
+
native byte ordering (or _endianness_) of the host system. This is not
|
| 695 |
+
compatible with memory-mapped arrays and using non-null `mmap_mode`
|
| 696 |
+
parameter at the same time will raise an error. The default 'auto'
|
| 697 |
+
parameter is equivalent to True if `mmap_mode` is None, else False.
|
| 698 |
+
|
| 699 |
+
Returns
|
| 700 |
+
-------
|
| 701 |
+
result: any Python object
|
| 702 |
+
The object stored in the file.
|
| 703 |
+
|
| 704 |
+
See Also
|
| 705 |
+
--------
|
| 706 |
+
joblib.dump : function to save an object
|
| 707 |
+
|
| 708 |
+
Notes
|
| 709 |
+
-----
|
| 710 |
+
|
| 711 |
+
This function can load numpy array files saved separately during the
|
| 712 |
+
dump. If the mmap_mode argument is given, it is passed to np.load and
|
| 713 |
+
arrays are loaded as memmaps. As a consequence, the reconstructed
|
| 714 |
+
object might not match the original pickled object. Note that if the
|
| 715 |
+
file was saved with compression, the arrays cannot be memmapped.
|
| 716 |
+
"""
|
| 717 |
+
if ensure_native_byte_order == "auto":
|
| 718 |
+
ensure_native_byte_order = mmap_mode is None
|
| 719 |
+
|
| 720 |
+
if ensure_native_byte_order and mmap_mode is not None:
|
| 721 |
+
raise ValueError(
|
| 722 |
+
"Native byte ordering can only be enforced if 'mmap_mode' parameter "
|
| 723 |
+
f"is set to None, but got 'mmap_mode={mmap_mode}' instead."
|
| 724 |
+
)
|
| 725 |
+
|
| 726 |
+
if Path is not None and isinstance(filename, Path):
|
| 727 |
+
filename = str(filename)
|
| 728 |
+
|
| 729 |
+
if hasattr(filename, "read"):
|
| 730 |
+
fobj = filename
|
| 731 |
+
filename = getattr(fobj, "name", "")
|
| 732 |
+
with _validate_fileobject_and_memmap(fobj, filename, mmap_mode) as (fobj, _):
|
| 733 |
+
obj = _unpickle(fobj, ensure_native_byte_order=ensure_native_byte_order)
|
| 734 |
+
else:
|
| 735 |
+
with open(filename, "rb") as f:
|
| 736 |
+
with _validate_fileobject_and_memmap(f, filename, mmap_mode) as (
|
| 737 |
+
fobj,
|
| 738 |
+
validated_mmap_mode,
|
| 739 |
+
):
|
| 740 |
+
if isinstance(fobj, str):
|
| 741 |
+
# if the returned file object is a string, this means we
|
| 742 |
+
# try to load a pickle file generated with an version of
|
| 743 |
+
# Joblib so we load it with joblib compatibility function.
|
| 744 |
+
return load_compatibility(fobj)
|
| 745 |
+
|
| 746 |
+
# A memory-mapped array has to be mapped with the endianness
|
| 747 |
+
# it has been written with. Other arrays are coerced to the
|
| 748 |
+
# native endianness of the host system.
|
| 749 |
+
obj = _unpickle(
|
| 750 |
+
fobj,
|
| 751 |
+
ensure_native_byte_order=ensure_native_byte_order,
|
| 752 |
+
filename=filename,
|
| 753 |
+
mmap_mode=validated_mmap_mode,
|
| 754 |
+
)
|
| 755 |
+
|
| 756 |
+
return obj
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_compat.py
ADDED
|
@@ -0,0 +1,250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Numpy pickle compatibility functions."""
|
| 2 |
+
|
| 3 |
+
import inspect
|
| 4 |
+
import os
|
| 5 |
+
import pickle
|
| 6 |
+
import zlib
|
| 7 |
+
from io import BytesIO
|
| 8 |
+
|
| 9 |
+
from .numpy_pickle_utils import (
|
| 10 |
+
_ZFILE_PREFIX,
|
| 11 |
+
Unpickler,
|
| 12 |
+
_ensure_native_byte_order,
|
| 13 |
+
_reconstruct,
|
| 14 |
+
)
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def hex_str(an_int):
|
| 18 |
+
"""Convert an int to an hexadecimal string."""
|
| 19 |
+
return "{:#x}".format(an_int)
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def asbytes(s):
|
| 23 |
+
if isinstance(s, bytes):
|
| 24 |
+
return s
|
| 25 |
+
return s.encode("latin1")
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
_MAX_LEN = len(hex_str(2**64))
|
| 29 |
+
_CHUNK_SIZE = 64 * 1024
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def read_zfile(file_handle):
|
| 33 |
+
"""Read the z-file and return the content as a string.
|
| 34 |
+
|
| 35 |
+
Z-files are raw data compressed with zlib used internally by joblib
|
| 36 |
+
for persistence. Backward compatibility is not guaranteed. Do not
|
| 37 |
+
use for external purposes.
|
| 38 |
+
"""
|
| 39 |
+
file_handle.seek(0)
|
| 40 |
+
header_length = len(_ZFILE_PREFIX) + _MAX_LEN
|
| 41 |
+
length = file_handle.read(header_length)
|
| 42 |
+
length = length[len(_ZFILE_PREFIX) :]
|
| 43 |
+
length = int(length, 16)
|
| 44 |
+
|
| 45 |
+
# With python2 and joblib version <= 0.8.4 compressed pickle header is one
|
| 46 |
+
# character wider so we need to ignore an additional space if present.
|
| 47 |
+
# Note: the first byte of the zlib data is guaranteed not to be a
|
| 48 |
+
# space according to
|
| 49 |
+
# https://tools.ietf.org/html/rfc6713#section-2.1
|
| 50 |
+
next_byte = file_handle.read(1)
|
| 51 |
+
if next_byte != b" ":
|
| 52 |
+
# The zlib compressed data has started and we need to go back
|
| 53 |
+
# one byte
|
| 54 |
+
file_handle.seek(header_length)
|
| 55 |
+
|
| 56 |
+
# We use the known length of the data to tell Zlib the size of the
|
| 57 |
+
# buffer to allocate.
|
| 58 |
+
data = zlib.decompress(file_handle.read(), 15, length)
|
| 59 |
+
assert len(data) == length, (
|
| 60 |
+
"Incorrect data length while decompressing %s."
|
| 61 |
+
"The file could be corrupted." % file_handle
|
| 62 |
+
)
|
| 63 |
+
return data
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def write_zfile(file_handle, data, compress=1):
|
| 67 |
+
"""Write the data in the given file as a Z-file.
|
| 68 |
+
|
| 69 |
+
Z-files are raw data compressed with zlib used internally by joblib
|
| 70 |
+
for persistence. Backward compatibility is not guaranteed. Do not
|
| 71 |
+
use for external purposes.
|
| 72 |
+
"""
|
| 73 |
+
file_handle.write(_ZFILE_PREFIX)
|
| 74 |
+
length = hex_str(len(data))
|
| 75 |
+
# Store the length of the data
|
| 76 |
+
file_handle.write(asbytes(length.ljust(_MAX_LEN)))
|
| 77 |
+
file_handle.write(zlib.compress(asbytes(data), compress))
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
###############################################################################
|
| 81 |
+
# Utility objects for persistence.
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
class NDArrayWrapper(object):
|
| 85 |
+
"""An object to be persisted instead of numpy arrays.
|
| 86 |
+
|
| 87 |
+
The only thing this object does, is to carry the filename in which
|
| 88 |
+
the array has been persisted, and the array subclass.
|
| 89 |
+
"""
|
| 90 |
+
|
| 91 |
+
def __init__(self, filename, subclass, allow_mmap=True):
|
| 92 |
+
"""Constructor. Store the useful information for later."""
|
| 93 |
+
self.filename = filename
|
| 94 |
+
self.subclass = subclass
|
| 95 |
+
self.allow_mmap = allow_mmap
|
| 96 |
+
|
| 97 |
+
def read(self, unpickler):
|
| 98 |
+
"""Reconstruct the array."""
|
| 99 |
+
filename = os.path.join(unpickler._dirname, self.filename)
|
| 100 |
+
# Load the array from the disk
|
| 101 |
+
# use getattr instead of self.allow_mmap to ensure backward compat
|
| 102 |
+
# with NDArrayWrapper instances pickled with joblib < 0.9.0
|
| 103 |
+
allow_mmap = getattr(self, "allow_mmap", True)
|
| 104 |
+
kwargs = {}
|
| 105 |
+
if allow_mmap:
|
| 106 |
+
kwargs["mmap_mode"] = unpickler.mmap_mode
|
| 107 |
+
if "allow_pickle" in inspect.signature(unpickler.np.load).parameters:
|
| 108 |
+
# Required in numpy 1.16.3 and later to acknowledge the security
|
| 109 |
+
# risk.
|
| 110 |
+
kwargs["allow_pickle"] = True
|
| 111 |
+
array = unpickler.np.load(filename, **kwargs)
|
| 112 |
+
|
| 113 |
+
# Detect byte order mismatch and swap as needed.
|
| 114 |
+
array = _ensure_native_byte_order(array)
|
| 115 |
+
|
| 116 |
+
# Reconstruct subclasses. This does not work with old
|
| 117 |
+
# versions of numpy
|
| 118 |
+
if hasattr(array, "__array_prepare__") and self.subclass not in (
|
| 119 |
+
unpickler.np.ndarray,
|
| 120 |
+
unpickler.np.memmap,
|
| 121 |
+
):
|
| 122 |
+
# We need to reconstruct another subclass
|
| 123 |
+
new_array = _reconstruct(self.subclass, (0,), "b")
|
| 124 |
+
return new_array.__array_prepare__(array)
|
| 125 |
+
else:
|
| 126 |
+
return array
|
| 127 |
+
|
| 128 |
+
|
| 129 |
+
class ZNDArrayWrapper(NDArrayWrapper):
|
| 130 |
+
"""An object to be persisted instead of numpy arrays.
|
| 131 |
+
|
| 132 |
+
This object store the Zfile filename in which
|
| 133 |
+
the data array has been persisted, and the meta information to
|
| 134 |
+
retrieve it.
|
| 135 |
+
The reason that we store the raw buffer data of the array and
|
| 136 |
+
the meta information, rather than array representation routine
|
| 137 |
+
(tobytes) is that it enables us to use completely the strided
|
| 138 |
+
model to avoid memory copies (a and a.T store as fast). In
|
| 139 |
+
addition saving the heavy information separately can avoid
|
| 140 |
+
creating large temporary buffers when unpickling data with
|
| 141 |
+
large arrays.
|
| 142 |
+
"""
|
| 143 |
+
|
| 144 |
+
def __init__(self, filename, init_args, state):
|
| 145 |
+
"""Constructor. Store the useful information for later."""
|
| 146 |
+
self.filename = filename
|
| 147 |
+
self.state = state
|
| 148 |
+
self.init_args = init_args
|
| 149 |
+
|
| 150 |
+
def read(self, unpickler):
|
| 151 |
+
"""Reconstruct the array from the meta-information and the z-file."""
|
| 152 |
+
# Here we a simply reproducing the unpickling mechanism for numpy
|
| 153 |
+
# arrays
|
| 154 |
+
filename = os.path.join(unpickler._dirname, self.filename)
|
| 155 |
+
array = _reconstruct(*self.init_args)
|
| 156 |
+
with open(filename, "rb") as f:
|
| 157 |
+
data = read_zfile(f)
|
| 158 |
+
state = self.state + (data,)
|
| 159 |
+
array.__setstate__(state)
|
| 160 |
+
return array
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
class ZipNumpyUnpickler(Unpickler):
|
| 164 |
+
"""A subclass of the Unpickler to unpickle our numpy pickles."""
|
| 165 |
+
|
| 166 |
+
dispatch = Unpickler.dispatch.copy()
|
| 167 |
+
|
| 168 |
+
def __init__(self, filename, file_handle, mmap_mode=None):
|
| 169 |
+
"""Constructor."""
|
| 170 |
+
self._filename = os.path.basename(filename)
|
| 171 |
+
self._dirname = os.path.dirname(filename)
|
| 172 |
+
self.mmap_mode = mmap_mode
|
| 173 |
+
self.file_handle = self._open_pickle(file_handle)
|
| 174 |
+
Unpickler.__init__(self, self.file_handle)
|
| 175 |
+
try:
|
| 176 |
+
import numpy as np
|
| 177 |
+
except ImportError:
|
| 178 |
+
np = None
|
| 179 |
+
self.np = np
|
| 180 |
+
|
| 181 |
+
def _open_pickle(self, file_handle):
|
| 182 |
+
return BytesIO(read_zfile(file_handle))
|
| 183 |
+
|
| 184 |
+
def load_build(self):
|
| 185 |
+
"""Set the state of a newly created object.
|
| 186 |
+
|
| 187 |
+
We capture it to replace our place-holder objects,
|
| 188 |
+
NDArrayWrapper, by the array we are interested in. We
|
| 189 |
+
replace them directly in the stack of pickler.
|
| 190 |
+
"""
|
| 191 |
+
Unpickler.load_build(self)
|
| 192 |
+
if isinstance(self.stack[-1], NDArrayWrapper):
|
| 193 |
+
if self.np is None:
|
| 194 |
+
raise ImportError(
|
| 195 |
+
"Trying to unpickle an ndarray, but numpy didn't import correctly"
|
| 196 |
+
)
|
| 197 |
+
nd_array_wrapper = self.stack.pop()
|
| 198 |
+
array = nd_array_wrapper.read(self)
|
| 199 |
+
self.stack.append(array)
|
| 200 |
+
|
| 201 |
+
dispatch[pickle.BUILD[0]] = load_build
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
def load_compatibility(filename):
|
| 205 |
+
"""Reconstruct a Python object from a file persisted with joblib.dump.
|
| 206 |
+
|
| 207 |
+
This function ensures the compatibility with joblib old persistence format
|
| 208 |
+
(<= 0.9.3).
|
| 209 |
+
|
| 210 |
+
Parameters
|
| 211 |
+
----------
|
| 212 |
+
filename: string
|
| 213 |
+
The name of the file from which to load the object
|
| 214 |
+
|
| 215 |
+
Returns
|
| 216 |
+
-------
|
| 217 |
+
result: any Python object
|
| 218 |
+
The object stored in the file.
|
| 219 |
+
|
| 220 |
+
See Also
|
| 221 |
+
--------
|
| 222 |
+
joblib.dump : function to save an object
|
| 223 |
+
|
| 224 |
+
Notes
|
| 225 |
+
-----
|
| 226 |
+
|
| 227 |
+
This function can load numpy array files saved separately during the
|
| 228 |
+
dump.
|
| 229 |
+
"""
|
| 230 |
+
with open(filename, "rb") as file_handle:
|
| 231 |
+
# We are careful to open the file handle early and keep it open to
|
| 232 |
+
# avoid race-conditions on renames. That said, if data is stored in
|
| 233 |
+
# companion files, moving the directory will create a race when
|
| 234 |
+
# joblib tries to access the companion files.
|
| 235 |
+
unpickler = ZipNumpyUnpickler(filename, file_handle=file_handle)
|
| 236 |
+
try:
|
| 237 |
+
obj = unpickler.load()
|
| 238 |
+
except UnicodeDecodeError as exc:
|
| 239 |
+
# More user-friendly error message
|
| 240 |
+
new_exc = ValueError(
|
| 241 |
+
"You may be trying to read with "
|
| 242 |
+
"python 3 a joblib pickle generated with python 2. "
|
| 243 |
+
"This feature is not supported by joblib."
|
| 244 |
+
)
|
| 245 |
+
new_exc.__cause__ = exc
|
| 246 |
+
raise new_exc
|
| 247 |
+
finally:
|
| 248 |
+
if hasattr(unpickler, "file_handle"):
|
| 249 |
+
unpickler.file_handle.close()
|
| 250 |
+
return obj
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_utils.py
ADDED
|
@@ -0,0 +1,291 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Utilities for fast persistence of big data, with optional compression."""
|
| 2 |
+
|
| 3 |
+
# Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
|
| 4 |
+
# Copyright (c) 2009 Gael Varoquaux
|
| 5 |
+
# License: BSD Style, 3 clauses.
|
| 6 |
+
|
| 7 |
+
import contextlib
|
| 8 |
+
import io
|
| 9 |
+
import pickle
|
| 10 |
+
import sys
|
| 11 |
+
import warnings
|
| 12 |
+
|
| 13 |
+
from .compressor import _COMPRESSORS, _ZFILE_PREFIX
|
| 14 |
+
|
| 15 |
+
try:
|
| 16 |
+
import numpy as np
|
| 17 |
+
except ImportError:
|
| 18 |
+
np = None
|
| 19 |
+
|
| 20 |
+
Unpickler = pickle._Unpickler
|
| 21 |
+
Pickler = pickle._Pickler
|
| 22 |
+
xrange = range
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
try:
|
| 26 |
+
# The python standard library can be built without bz2 so we make bz2
|
| 27 |
+
# usage optional.
|
| 28 |
+
# see https://github.com/scikit-learn/scikit-learn/issues/7526 for more
|
| 29 |
+
# details.
|
| 30 |
+
import bz2
|
| 31 |
+
except ImportError:
|
| 32 |
+
bz2 = None
|
| 33 |
+
|
| 34 |
+
# Buffer size used in io.BufferedReader and io.BufferedWriter
|
| 35 |
+
_IO_BUFFER_SIZE = 1024**2
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _is_raw_file(fileobj):
|
| 39 |
+
"""Check if fileobj is a raw file object, e.g created with open."""
|
| 40 |
+
fileobj = getattr(fileobj, "raw", fileobj)
|
| 41 |
+
return isinstance(fileobj, io.FileIO)
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _get_prefixes_max_len():
|
| 45 |
+
# Compute the max prefix len of registered compressors.
|
| 46 |
+
prefixes = [len(compressor.prefix) for compressor in _COMPRESSORS.values()]
|
| 47 |
+
prefixes += [len(_ZFILE_PREFIX)]
|
| 48 |
+
return max(prefixes)
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _is_numpy_array_byte_order_mismatch(array):
|
| 52 |
+
"""Check if numpy array is having byte order mismatch"""
|
| 53 |
+
return (
|
| 54 |
+
sys.byteorder == "big"
|
| 55 |
+
and (
|
| 56 |
+
array.dtype.byteorder == "<"
|
| 57 |
+
or (
|
| 58 |
+
array.dtype.byteorder == "|"
|
| 59 |
+
and array.dtype.fields
|
| 60 |
+
and all(e[0].byteorder == "<" for e in array.dtype.fields.values())
|
| 61 |
+
)
|
| 62 |
+
)
|
| 63 |
+
) or (
|
| 64 |
+
sys.byteorder == "little"
|
| 65 |
+
and (
|
| 66 |
+
array.dtype.byteorder == ">"
|
| 67 |
+
or (
|
| 68 |
+
array.dtype.byteorder == "|"
|
| 69 |
+
and array.dtype.fields
|
| 70 |
+
and all(e[0].byteorder == ">" for e in array.dtype.fields.values())
|
| 71 |
+
)
|
| 72 |
+
)
|
| 73 |
+
)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def _ensure_native_byte_order(array):
|
| 77 |
+
"""Use the byte order of the host while preserving values
|
| 78 |
+
|
| 79 |
+
Does nothing if array already uses the system byte order.
|
| 80 |
+
"""
|
| 81 |
+
if _is_numpy_array_byte_order_mismatch(array):
|
| 82 |
+
array = array.byteswap().view(array.dtype.newbyteorder("="))
|
| 83 |
+
return array
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
###############################################################################
|
| 87 |
+
# Cache file utilities
|
| 88 |
+
def _detect_compressor(fileobj):
|
| 89 |
+
"""Return the compressor matching fileobj.
|
| 90 |
+
|
| 91 |
+
Parameters
|
| 92 |
+
----------
|
| 93 |
+
fileobj: file object
|
| 94 |
+
|
| 95 |
+
Returns
|
| 96 |
+
-------
|
| 97 |
+
str in {'zlib', 'gzip', 'bz2', 'lzma', 'xz', 'compat', 'not-compressed'}
|
| 98 |
+
"""
|
| 99 |
+
# Read the magic number in the first bytes of the file.
|
| 100 |
+
max_prefix_len = _get_prefixes_max_len()
|
| 101 |
+
if hasattr(fileobj, "peek"):
|
| 102 |
+
# Peek allows to read those bytes without moving the cursor in the
|
| 103 |
+
# file which.
|
| 104 |
+
first_bytes = fileobj.peek(max_prefix_len)
|
| 105 |
+
else:
|
| 106 |
+
# Fallback to seek if the fileobject is not peekable.
|
| 107 |
+
first_bytes = fileobj.read(max_prefix_len)
|
| 108 |
+
fileobj.seek(0)
|
| 109 |
+
|
| 110 |
+
if first_bytes.startswith(_ZFILE_PREFIX):
|
| 111 |
+
return "compat"
|
| 112 |
+
else:
|
| 113 |
+
for name, compressor in _COMPRESSORS.items():
|
| 114 |
+
if first_bytes.startswith(compressor.prefix):
|
| 115 |
+
return name
|
| 116 |
+
|
| 117 |
+
return "not-compressed"
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def _buffered_read_file(fobj):
|
| 121 |
+
"""Return a buffered version of a read file object."""
|
| 122 |
+
return io.BufferedReader(fobj, buffer_size=_IO_BUFFER_SIZE)
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def _buffered_write_file(fobj):
|
| 126 |
+
"""Return a buffered version of a write file object."""
|
| 127 |
+
return io.BufferedWriter(fobj, buffer_size=_IO_BUFFER_SIZE)
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
@contextlib.contextmanager
|
| 131 |
+
def _validate_fileobject_and_memmap(fileobj, filename, mmap_mode=None):
|
| 132 |
+
"""Utility function opening the right fileobject from a filename.
|
| 133 |
+
|
| 134 |
+
The magic number is used to choose between the type of file object to open:
|
| 135 |
+
* regular file object (default)
|
| 136 |
+
* zlib file object
|
| 137 |
+
* gzip file object
|
| 138 |
+
* bz2 file object
|
| 139 |
+
* lzma file object (for xz and lzma compressor)
|
| 140 |
+
|
| 141 |
+
Parameters
|
| 142 |
+
----------
|
| 143 |
+
fileobj: file object
|
| 144 |
+
filename: str
|
| 145 |
+
filename path corresponding to the fileobj parameter.
|
| 146 |
+
mmap_mode: str
|
| 147 |
+
memory map mode that should be used to open the pickle file. This
|
| 148 |
+
parameter is useful to verify that the user is not trying to one with
|
| 149 |
+
compression. Default: None.
|
| 150 |
+
|
| 151 |
+
Returns
|
| 152 |
+
-------
|
| 153 |
+
a tuple with a file like object, and the validated mmap_mode.
|
| 154 |
+
|
| 155 |
+
"""
|
| 156 |
+
# Detect if the fileobj contains compressed data.
|
| 157 |
+
compressor = _detect_compressor(fileobj)
|
| 158 |
+
validated_mmap_mode = mmap_mode
|
| 159 |
+
|
| 160 |
+
if compressor == "compat":
|
| 161 |
+
# Compatibility with old pickle mode: simply return the input
|
| 162 |
+
# filename "as-is" and let the compatibility function be called by the
|
| 163 |
+
# caller.
|
| 164 |
+
warnings.warn(
|
| 165 |
+
"The file '%s' has been generated with a joblib "
|
| 166 |
+
"version less than 0.10. "
|
| 167 |
+
"Please regenerate this pickle file." % filename,
|
| 168 |
+
DeprecationWarning,
|
| 169 |
+
stacklevel=2,
|
| 170 |
+
)
|
| 171 |
+
yield filename, validated_mmap_mode
|
| 172 |
+
else:
|
| 173 |
+
if compressor in _COMPRESSORS:
|
| 174 |
+
# based on the compressor detected in the file, we open the
|
| 175 |
+
# correct decompressor file object, wrapped in a buffer.
|
| 176 |
+
compressor_wrapper = _COMPRESSORS[compressor]
|
| 177 |
+
inst = compressor_wrapper.decompressor_file(fileobj)
|
| 178 |
+
fileobj = _buffered_read_file(inst)
|
| 179 |
+
|
| 180 |
+
# Checking if incompatible load parameters with the type of file:
|
| 181 |
+
# mmap_mode cannot be used with compressed file or in memory buffers
|
| 182 |
+
# such as io.BytesIO.
|
| 183 |
+
if mmap_mode is not None:
|
| 184 |
+
validated_mmap_mode = None
|
| 185 |
+
if isinstance(fileobj, io.BytesIO):
|
| 186 |
+
warnings.warn(
|
| 187 |
+
"In memory persistence is not compatible with "
|
| 188 |
+
'mmap_mode "%(mmap_mode)s" flag passed. '
|
| 189 |
+
"mmap_mode option will be ignored." % locals(),
|
| 190 |
+
stacklevel=2,
|
| 191 |
+
)
|
| 192 |
+
elif compressor != "not-compressed":
|
| 193 |
+
warnings.warn(
|
| 194 |
+
'mmap_mode "%(mmap_mode)s" is not compatible '
|
| 195 |
+
"with compressed file %(filename)s. "
|
| 196 |
+
'"%(mmap_mode)s" flag will be ignored.' % locals(),
|
| 197 |
+
stacklevel=2,
|
| 198 |
+
)
|
| 199 |
+
elif not _is_raw_file(fileobj):
|
| 200 |
+
warnings.warn(
|
| 201 |
+
'"%(fileobj)r" is not a raw file, mmap_mode '
|
| 202 |
+
'"%(mmap_mode)s" flag will be ignored.' % locals(),
|
| 203 |
+
stacklevel=2,
|
| 204 |
+
)
|
| 205 |
+
else:
|
| 206 |
+
validated_mmap_mode = mmap_mode
|
| 207 |
+
|
| 208 |
+
yield fileobj, validated_mmap_mode
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
def _write_fileobject(filename, compress=("zlib", 3)):
|
| 212 |
+
"""Return the right compressor file object in write mode."""
|
| 213 |
+
compressmethod = compress[0]
|
| 214 |
+
compresslevel = compress[1]
|
| 215 |
+
|
| 216 |
+
if compressmethod in _COMPRESSORS.keys():
|
| 217 |
+
file_instance = _COMPRESSORS[compressmethod].compressor_file(
|
| 218 |
+
filename, compresslevel=compresslevel
|
| 219 |
+
)
|
| 220 |
+
return _buffered_write_file(file_instance)
|
| 221 |
+
else:
|
| 222 |
+
file_instance = _COMPRESSORS["zlib"].compressor_file(
|
| 223 |
+
filename, compresslevel=compresslevel
|
| 224 |
+
)
|
| 225 |
+
return _buffered_write_file(file_instance)
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
# Utility functions/variables from numpy required for writing arrays.
|
| 229 |
+
# We need at least the functions introduced in version 1.9 of numpy. Here,
|
| 230 |
+
# we use the ones from numpy 1.10.2.
|
| 231 |
+
BUFFER_SIZE = 2**18 # size of buffer for reading npz files in bytes
|
| 232 |
+
|
| 233 |
+
|
| 234 |
+
def _read_bytes(fp, size, error_template="ran out of data"):
|
| 235 |
+
"""Read from file-like object until size bytes are read.
|
| 236 |
+
|
| 237 |
+
TODO python2_drop: is it still needed? The docstring mentions python 2.6
|
| 238 |
+
and it looks like this can be at least simplified ...
|
| 239 |
+
|
| 240 |
+
Raises ValueError if not EOF is encountered before size bytes are read.
|
| 241 |
+
Non-blocking objects only supported if they derive from io objects.
|
| 242 |
+
|
| 243 |
+
Required as e.g. ZipExtFile in python 2.6 can return less data than
|
| 244 |
+
requested.
|
| 245 |
+
|
| 246 |
+
This function was taken from numpy/lib/format.py in version 1.10.2.
|
| 247 |
+
|
| 248 |
+
Parameters
|
| 249 |
+
----------
|
| 250 |
+
fp: file-like object
|
| 251 |
+
size: int
|
| 252 |
+
error_template: str
|
| 253 |
+
|
| 254 |
+
Returns
|
| 255 |
+
-------
|
| 256 |
+
a bytes object
|
| 257 |
+
The data read in bytes.
|
| 258 |
+
|
| 259 |
+
"""
|
| 260 |
+
data = bytes()
|
| 261 |
+
while True:
|
| 262 |
+
# io files (default in python3) return None or raise on
|
| 263 |
+
# would-block, python2 file will truncate, probably nothing can be
|
| 264 |
+
# done about that. note that regular files can't be non-blocking
|
| 265 |
+
try:
|
| 266 |
+
r = fp.read(size - len(data))
|
| 267 |
+
data += r
|
| 268 |
+
if len(r) == 0 or len(data) == size:
|
| 269 |
+
break
|
| 270 |
+
except io.BlockingIOError:
|
| 271 |
+
pass
|
| 272 |
+
if len(data) != size:
|
| 273 |
+
msg = "EOF: reading %s, expected %d bytes got %d"
|
| 274 |
+
raise ValueError(msg % (error_template, size, len(data)))
|
| 275 |
+
else:
|
| 276 |
+
return data
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
def _reconstruct(*args, **kwargs):
|
| 280 |
+
# Wrapper for numpy._core.multiarray._reconstruct with backward compat
|
| 281 |
+
# for numpy 1.X
|
| 282 |
+
#
|
| 283 |
+
# XXX: Remove this function when numpy 1.X is not supported anymore
|
| 284 |
+
|
| 285 |
+
np_major_version = np.__version__[:2]
|
| 286 |
+
if np_major_version == "1.":
|
| 287 |
+
from numpy.core.multiarray import _reconstruct as np_reconstruct
|
| 288 |
+
elif np_major_version == "2.":
|
| 289 |
+
from numpy._core.multiarray import _reconstruct as np_reconstruct
|
| 290 |
+
|
| 291 |
+
return np_reconstruct(*args, **kwargs)
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/parallel.py
ADDED
|
@@ -0,0 +1,2075 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Helpers for embarrassingly parallel code.
|
| 3 |
+
"""
|
| 4 |
+
# Author: Gael Varoquaux < gael dot varoquaux at normalesup dot org >
|
| 5 |
+
# Copyright: 2010, Gael Varoquaux
|
| 6 |
+
# License: BSD 3 clause
|
| 7 |
+
|
| 8 |
+
from __future__ import division
|
| 9 |
+
|
| 10 |
+
import collections
|
| 11 |
+
import functools
|
| 12 |
+
import itertools
|
| 13 |
+
import os
|
| 14 |
+
import queue
|
| 15 |
+
import sys
|
| 16 |
+
import threading
|
| 17 |
+
import time
|
| 18 |
+
import warnings
|
| 19 |
+
import weakref
|
| 20 |
+
from contextlib import nullcontext
|
| 21 |
+
from math import floor, log10, sqrt
|
| 22 |
+
from multiprocessing import TimeoutError
|
| 23 |
+
from numbers import Integral
|
| 24 |
+
from uuid import uuid4
|
| 25 |
+
|
| 26 |
+
from ._multiprocessing_helpers import mp
|
| 27 |
+
|
| 28 |
+
# Make sure that those two classes are part of the public joblib.parallel API
|
| 29 |
+
# so that 3rd party backend implementers can import them from here.
|
| 30 |
+
from ._parallel_backends import (
|
| 31 |
+
AutoBatchingMixin, # noqa
|
| 32 |
+
FallbackToBackend,
|
| 33 |
+
LokyBackend,
|
| 34 |
+
MultiprocessingBackend,
|
| 35 |
+
ParallelBackendBase, # noqa
|
| 36 |
+
SequentialBackend,
|
| 37 |
+
ThreadingBackend,
|
| 38 |
+
)
|
| 39 |
+
from ._utils import _Sentinel, eval_expr
|
| 40 |
+
from .disk import memstr_to_bytes
|
| 41 |
+
from .logger import Logger, short_format_time
|
| 42 |
+
|
| 43 |
+
BACKENDS = {
|
| 44 |
+
"threading": ThreadingBackend,
|
| 45 |
+
"sequential": SequentialBackend,
|
| 46 |
+
}
|
| 47 |
+
# name of the backend used by default by Parallel outside of any context
|
| 48 |
+
# managed by ``parallel_config`` or ``parallel_backend``.
|
| 49 |
+
|
| 50 |
+
# threading is the only backend that is always everywhere
|
| 51 |
+
DEFAULT_BACKEND = "threading"
|
| 52 |
+
DEFAULT_THREAD_BACKEND = "threading"
|
| 53 |
+
DEFAULT_PROCESS_BACKEND = "threading"
|
| 54 |
+
|
| 55 |
+
MAYBE_AVAILABLE_BACKENDS = {"multiprocessing", "loky"}
|
| 56 |
+
|
| 57 |
+
# if multiprocessing is available, so is loky, we set it as the default
|
| 58 |
+
# backend
|
| 59 |
+
if mp is not None:
|
| 60 |
+
BACKENDS["multiprocessing"] = MultiprocessingBackend
|
| 61 |
+
from .externals import loky
|
| 62 |
+
|
| 63 |
+
BACKENDS["loky"] = LokyBackend
|
| 64 |
+
DEFAULT_BACKEND = "loky"
|
| 65 |
+
DEFAULT_PROCESS_BACKEND = "loky"
|
| 66 |
+
|
| 67 |
+
# Thread local value that can be overridden by the ``parallel_config`` context
|
| 68 |
+
# manager
|
| 69 |
+
_backend = threading.local()
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def _register_dask():
|
| 73 |
+
"""Register Dask Backend if called with parallel_config(backend="dask")"""
|
| 74 |
+
try:
|
| 75 |
+
from ._dask import DaskDistributedBackend
|
| 76 |
+
|
| 77 |
+
register_parallel_backend("dask", DaskDistributedBackend)
|
| 78 |
+
except ImportError as e:
|
| 79 |
+
msg = (
|
| 80 |
+
"To use the dask.distributed backend you must install both "
|
| 81 |
+
"the `dask` and distributed modules.\n\n"
|
| 82 |
+
"See https://dask.pydata.org/en/latest/install.html for more "
|
| 83 |
+
"information."
|
| 84 |
+
)
|
| 85 |
+
raise ImportError(msg) from e
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
EXTERNAL_BACKENDS = {
|
| 89 |
+
"dask": _register_dask,
|
| 90 |
+
}
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
# Sentinels for the default values of the Parallel constructor and
|
| 94 |
+
# the parallel_config and parallel_backend context managers
|
| 95 |
+
default_parallel_config = {
|
| 96 |
+
"backend": _Sentinel(default_value=None),
|
| 97 |
+
"n_jobs": _Sentinel(default_value=None),
|
| 98 |
+
"verbose": _Sentinel(default_value=0),
|
| 99 |
+
"temp_folder": _Sentinel(default_value=None),
|
| 100 |
+
"max_nbytes": _Sentinel(default_value="1M"),
|
| 101 |
+
"mmap_mode": _Sentinel(default_value="r"),
|
| 102 |
+
"prefer": _Sentinel(default_value=None),
|
| 103 |
+
"require": _Sentinel(default_value=None),
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
VALID_BACKEND_HINTS = ("processes", "threads", None)
|
| 108 |
+
VALID_BACKEND_CONSTRAINTS = ("sharedmem", None)
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _get_config_param(param, context_config, key):
|
| 112 |
+
"""Return the value of a parallel config parameter
|
| 113 |
+
|
| 114 |
+
Explicitly setting it in Parallel has priority over setting in a
|
| 115 |
+
parallel_(config/backend) context manager.
|
| 116 |
+
"""
|
| 117 |
+
if param is not default_parallel_config[key]:
|
| 118 |
+
# param is explicitly set, return it
|
| 119 |
+
return param
|
| 120 |
+
|
| 121 |
+
if context_config[key] is not default_parallel_config[key]:
|
| 122 |
+
# there's a context manager and the key is set, return it
|
| 123 |
+
return context_config[key]
|
| 124 |
+
|
| 125 |
+
# Otherwise, we are in the default_parallel_config,
|
| 126 |
+
# return the default value
|
| 127 |
+
return param.default_value
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def get_active_backend(
|
| 131 |
+
prefer=default_parallel_config["prefer"],
|
| 132 |
+
require=default_parallel_config["require"],
|
| 133 |
+
verbose=default_parallel_config["verbose"],
|
| 134 |
+
):
|
| 135 |
+
"""Return the active default backend"""
|
| 136 |
+
backend, config = _get_active_backend(prefer, require, verbose)
|
| 137 |
+
n_jobs = _get_config_param(default_parallel_config["n_jobs"], config, "n_jobs")
|
| 138 |
+
return backend, n_jobs
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
def _get_active_backend(
|
| 142 |
+
prefer=default_parallel_config["prefer"],
|
| 143 |
+
require=default_parallel_config["require"],
|
| 144 |
+
verbose=default_parallel_config["verbose"],
|
| 145 |
+
):
|
| 146 |
+
"""Return the active default backend"""
|
| 147 |
+
|
| 148 |
+
backend_config = getattr(_backend, "config", default_parallel_config)
|
| 149 |
+
|
| 150 |
+
backend = _get_config_param(
|
| 151 |
+
default_parallel_config["backend"], backend_config, "backend"
|
| 152 |
+
)
|
| 153 |
+
|
| 154 |
+
prefer = _get_config_param(prefer, backend_config, "prefer")
|
| 155 |
+
require = _get_config_param(require, backend_config, "require")
|
| 156 |
+
verbose = _get_config_param(verbose, backend_config, "verbose")
|
| 157 |
+
|
| 158 |
+
if prefer not in VALID_BACKEND_HINTS:
|
| 159 |
+
raise ValueError(
|
| 160 |
+
f"prefer={prefer} is not a valid backend hint, "
|
| 161 |
+
f"expected one of {VALID_BACKEND_HINTS}"
|
| 162 |
+
)
|
| 163 |
+
if require not in VALID_BACKEND_CONSTRAINTS:
|
| 164 |
+
raise ValueError(
|
| 165 |
+
f"require={require} is not a valid backend constraint, "
|
| 166 |
+
f"expected one of {VALID_BACKEND_CONSTRAINTS}"
|
| 167 |
+
)
|
| 168 |
+
if prefer == "processes" and require == "sharedmem":
|
| 169 |
+
raise ValueError(
|
| 170 |
+
"prefer == 'processes' and require == 'sharedmem' are inconsistent settings"
|
| 171 |
+
)
|
| 172 |
+
|
| 173 |
+
explicit_backend = True
|
| 174 |
+
if backend is None:
|
| 175 |
+
# We are either outside of the scope of any parallel_(config/backend)
|
| 176 |
+
# context manager or the context manager did not set a backend.
|
| 177 |
+
# create the default backend instance now.
|
| 178 |
+
backend = BACKENDS[DEFAULT_BACKEND](nesting_level=0)
|
| 179 |
+
explicit_backend = False
|
| 180 |
+
|
| 181 |
+
# Try to use the backend set by the user with the context manager.
|
| 182 |
+
|
| 183 |
+
nesting_level = backend.nesting_level
|
| 184 |
+
uses_threads = getattr(backend, "uses_threads", False)
|
| 185 |
+
supports_sharedmem = getattr(backend, "supports_sharedmem", False)
|
| 186 |
+
# Force to use thread-based backend if the provided backend does not
|
| 187 |
+
# match the shared memory constraint or if the backend is not explicitly
|
| 188 |
+
# given and threads are preferred.
|
| 189 |
+
force_threads = (require == "sharedmem" and not supports_sharedmem) or (
|
| 190 |
+
not explicit_backend and prefer == "threads" and not uses_threads
|
| 191 |
+
)
|
| 192 |
+
force_processes = not explicit_backend and prefer == "processes" and uses_threads
|
| 193 |
+
|
| 194 |
+
if force_threads:
|
| 195 |
+
# This backend does not match the shared memory constraint:
|
| 196 |
+
# fallback to the default thead-based backend.
|
| 197 |
+
sharedmem_backend = BACKENDS[DEFAULT_THREAD_BACKEND](
|
| 198 |
+
nesting_level=nesting_level
|
| 199 |
+
)
|
| 200 |
+
# Warn the user if we forced the backend to thread-based, while the
|
| 201 |
+
# user explicitly specified a non-thread-based backend.
|
| 202 |
+
if verbose >= 10 and explicit_backend:
|
| 203 |
+
print(
|
| 204 |
+
f"Using {sharedmem_backend.__class__.__name__} as "
|
| 205 |
+
f"joblib backend instead of {backend.__class__.__name__} "
|
| 206 |
+
"as the latter does not provide shared memory semantics."
|
| 207 |
+
)
|
| 208 |
+
# Force to n_jobs=1 by default
|
| 209 |
+
thread_config = backend_config.copy()
|
| 210 |
+
thread_config["n_jobs"] = 1
|
| 211 |
+
return sharedmem_backend, thread_config
|
| 212 |
+
|
| 213 |
+
if force_processes:
|
| 214 |
+
# This backend does not match the prefer="processes" constraint:
|
| 215 |
+
# fallback to the default process-based backend.
|
| 216 |
+
process_backend = BACKENDS[DEFAULT_PROCESS_BACKEND](nesting_level=nesting_level)
|
| 217 |
+
|
| 218 |
+
return process_backend, backend_config.copy()
|
| 219 |
+
|
| 220 |
+
return backend, backend_config
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
class parallel_config:
|
| 224 |
+
"""Set the default backend or configuration for :class:`~joblib.Parallel`.
|
| 225 |
+
|
| 226 |
+
This is an alternative to directly passing keyword arguments to the
|
| 227 |
+
:class:`~joblib.Parallel` class constructor. It is particularly useful when
|
| 228 |
+
calling into library code that uses joblib internally but does not expose
|
| 229 |
+
the various parallel configuration arguments in its own API.
|
| 230 |
+
|
| 231 |
+
Parameters
|
| 232 |
+
----------
|
| 233 |
+
backend: str or ParallelBackendBase instance, default=None
|
| 234 |
+
If ``backend`` is a string it must match a previously registered
|
| 235 |
+
implementation using the :func:`~register_parallel_backend` function.
|
| 236 |
+
|
| 237 |
+
By default the following backends are available:
|
| 238 |
+
|
| 239 |
+
- 'loky': single-host, process-based parallelism (used by default),
|
| 240 |
+
- 'threading': single-host, thread-based parallelism,
|
| 241 |
+
- 'multiprocessing': legacy single-host, process-based parallelism.
|
| 242 |
+
|
| 243 |
+
'loky' is recommended to run functions that manipulate Python objects.
|
| 244 |
+
'threading' is a low-overhead alternative that is most efficient for
|
| 245 |
+
functions that release the Global Interpreter Lock: e.g. I/O-bound
|
| 246 |
+
code or CPU-bound code in a few calls to native code that explicitly
|
| 247 |
+
releases the GIL. Note that on some rare systems (such as pyodide),
|
| 248 |
+
multiprocessing and loky may not be available, in which case joblib
|
| 249 |
+
defaults to threading.
|
| 250 |
+
|
| 251 |
+
In addition, if the ``dask`` and ``distributed`` Python packages are
|
| 252 |
+
installed, it is possible to use the 'dask' backend for better
|
| 253 |
+
scheduling of nested parallel calls without over-subscription and
|
| 254 |
+
potentially distribute parallel calls over a networked cluster of
|
| 255 |
+
several hosts.
|
| 256 |
+
|
| 257 |
+
It is also possible to use the distributed 'ray' backend for
|
| 258 |
+
distributing the workload to a cluster of nodes. See more details
|
| 259 |
+
in the Examples section below.
|
| 260 |
+
|
| 261 |
+
Alternatively the backend can be passed directly as an instance.
|
| 262 |
+
|
| 263 |
+
n_jobs: int, default=None
|
| 264 |
+
The maximum number of concurrently running jobs, such as the number
|
| 265 |
+
of Python worker processes when ``backend="loky"`` or the size of the
|
| 266 |
+
thread-pool when ``backend="threading"``.
|
| 267 |
+
This argument is converted to an integer, rounded below for float.
|
| 268 |
+
If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
|
| 269 |
+
``n_cpus`` is obtained with :func:`~cpu_count`.
|
| 270 |
+
For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
|
| 271 |
+
using ``n_jobs=-2`` will result in all CPUs but one being used.
|
| 272 |
+
This argument can also go above ``n_cpus``, which will cause
|
| 273 |
+
oversubscription. In some cases, slight oversubscription can be
|
| 274 |
+
beneficial, e.g., for tasks with large I/O operations.
|
| 275 |
+
If 1 is given, no parallel computing code is used at all, and the
|
| 276 |
+
behavior amounts to a simple python `for` loop. This mode is not
|
| 277 |
+
compatible with `timeout`.
|
| 278 |
+
None is a marker for 'unset' that will be interpreted as n_jobs=1
|
| 279 |
+
unless the call is performed under a :func:`~parallel_config`
|
| 280 |
+
context manager that sets another value for ``n_jobs``.
|
| 281 |
+
If n_jobs = 0 then a ValueError is raised.
|
| 282 |
+
|
| 283 |
+
verbose: int, default=0
|
| 284 |
+
The verbosity level: if non zero, progress messages are
|
| 285 |
+
printed. Above 50, the output is sent to stdout.
|
| 286 |
+
The frequency of the messages increases with the verbosity level.
|
| 287 |
+
If it more than 10, all iterations are reported.
|
| 288 |
+
|
| 289 |
+
temp_folder: str or None, default=None
|
| 290 |
+
Folder to be used by the pool for memmapping large arrays
|
| 291 |
+
for sharing memory with worker processes. If None, this will try in
|
| 292 |
+
order:
|
| 293 |
+
|
| 294 |
+
- a folder pointed by the ``JOBLIB_TEMP_FOLDER`` environment
|
| 295 |
+
variable,
|
| 296 |
+
- ``/dev/shm`` if the folder exists and is writable: this is a
|
| 297 |
+
RAM disk filesystem available by default on modern Linux
|
| 298 |
+
distributions,
|
| 299 |
+
- the default system temporary folder that can be
|
| 300 |
+
overridden with ``TMP``, ``TMPDIR`` or ``TEMP`` environment
|
| 301 |
+
variables, typically ``/tmp`` under Unix operating systems.
|
| 302 |
+
|
| 303 |
+
max_nbytes: int, str, or None, optional, default='1M'
|
| 304 |
+
Threshold on the size of arrays passed to the workers that
|
| 305 |
+
triggers automated memory mapping in temp_folder. Can be an int
|
| 306 |
+
in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
|
| 307 |
+
Use None to disable memmapping of large arrays.
|
| 308 |
+
|
| 309 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
|
| 310 |
+
Memmapping mode for numpy arrays passed to workers. None will
|
| 311 |
+
disable memmapping, other modes defined in the numpy.memmap doc:
|
| 312 |
+
https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
|
| 313 |
+
Also, see 'max_nbytes' parameter documentation for more details.
|
| 314 |
+
|
| 315 |
+
prefer: str in {'processes', 'threads'} or None, default=None
|
| 316 |
+
Soft hint to choose the default backend.
|
| 317 |
+
The default process-based backend is 'loky' and the default
|
| 318 |
+
thread-based backend is 'threading'. Ignored if the ``backend``
|
| 319 |
+
parameter is specified.
|
| 320 |
+
|
| 321 |
+
require: 'sharedmem' or None, default=None
|
| 322 |
+
Hard constraint to select the backend. If set to 'sharedmem',
|
| 323 |
+
the selected backend will be single-host and thread-based.
|
| 324 |
+
|
| 325 |
+
inner_max_num_threads: int, default=None
|
| 326 |
+
If not None, overwrites the limit set on the number of threads
|
| 327 |
+
usable in some third-party library threadpools like OpenBLAS,
|
| 328 |
+
MKL or OpenMP. This is only used with the ``loky`` backend.
|
| 329 |
+
|
| 330 |
+
backend_params: dict
|
| 331 |
+
Additional parameters to pass to the backend constructor when
|
| 332 |
+
backend is a string.
|
| 333 |
+
|
| 334 |
+
Notes
|
| 335 |
+
-----
|
| 336 |
+
Joblib tries to limit the oversubscription by limiting the number of
|
| 337 |
+
threads usable in some third-party library threadpools like OpenBLAS, MKL
|
| 338 |
+
or OpenMP. The default limit in each worker is set to
|
| 339 |
+
``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
|
| 340 |
+
overwritten with the ``inner_max_num_threads`` argument which will be used
|
| 341 |
+
to set this limit in the child processes.
|
| 342 |
+
|
| 343 |
+
.. versionadded:: 1.3
|
| 344 |
+
|
| 345 |
+
Examples
|
| 346 |
+
--------
|
| 347 |
+
>>> from operator import neg
|
| 348 |
+
>>> with parallel_config(backend='threading'):
|
| 349 |
+
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
|
| 350 |
+
...
|
| 351 |
+
[-1, -2, -3, -4, -5]
|
| 352 |
+
|
| 353 |
+
To use the 'ray' joblib backend add the following lines:
|
| 354 |
+
|
| 355 |
+
>>> from ray.util.joblib import register_ray # doctest: +SKIP
|
| 356 |
+
>>> register_ray() # doctest: +SKIP
|
| 357 |
+
>>> with parallel_config(backend="ray"): # doctest: +SKIP
|
| 358 |
+
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
|
| 359 |
+
[-1, -2, -3, -4, -5]
|
| 360 |
+
|
| 361 |
+
"""
|
| 362 |
+
|
| 363 |
+
def __init__(
|
| 364 |
+
self,
|
| 365 |
+
backend=default_parallel_config["backend"],
|
| 366 |
+
*,
|
| 367 |
+
n_jobs=default_parallel_config["n_jobs"],
|
| 368 |
+
verbose=default_parallel_config["verbose"],
|
| 369 |
+
temp_folder=default_parallel_config["temp_folder"],
|
| 370 |
+
max_nbytes=default_parallel_config["max_nbytes"],
|
| 371 |
+
mmap_mode=default_parallel_config["mmap_mode"],
|
| 372 |
+
prefer=default_parallel_config["prefer"],
|
| 373 |
+
require=default_parallel_config["require"],
|
| 374 |
+
inner_max_num_threads=None,
|
| 375 |
+
**backend_params,
|
| 376 |
+
):
|
| 377 |
+
# Save the parallel info and set the active parallel config
|
| 378 |
+
self.old_parallel_config = getattr(_backend, "config", default_parallel_config)
|
| 379 |
+
|
| 380 |
+
backend = self._check_backend(backend, inner_max_num_threads, **backend_params)
|
| 381 |
+
|
| 382 |
+
new_config = {
|
| 383 |
+
"n_jobs": n_jobs,
|
| 384 |
+
"verbose": verbose,
|
| 385 |
+
"temp_folder": temp_folder,
|
| 386 |
+
"max_nbytes": max_nbytes,
|
| 387 |
+
"mmap_mode": mmap_mode,
|
| 388 |
+
"prefer": prefer,
|
| 389 |
+
"require": require,
|
| 390 |
+
"backend": backend,
|
| 391 |
+
}
|
| 392 |
+
self.parallel_config = self.old_parallel_config.copy()
|
| 393 |
+
self.parallel_config.update(
|
| 394 |
+
{k: v for k, v in new_config.items() if not isinstance(v, _Sentinel)}
|
| 395 |
+
)
|
| 396 |
+
|
| 397 |
+
setattr(_backend, "config", self.parallel_config)
|
| 398 |
+
|
| 399 |
+
def _check_backend(self, backend, inner_max_num_threads, **backend_params):
|
| 400 |
+
if backend is default_parallel_config["backend"]:
|
| 401 |
+
if inner_max_num_threads is not None or len(backend_params) > 0:
|
| 402 |
+
raise ValueError(
|
| 403 |
+
"inner_max_num_threads and other constructor "
|
| 404 |
+
"parameters backend_params are only supported "
|
| 405 |
+
"when backend is not None."
|
| 406 |
+
)
|
| 407 |
+
return backend
|
| 408 |
+
|
| 409 |
+
if isinstance(backend, str):
|
| 410 |
+
# Handle non-registered or missing backends
|
| 411 |
+
if backend not in BACKENDS:
|
| 412 |
+
if backend in EXTERNAL_BACKENDS:
|
| 413 |
+
register = EXTERNAL_BACKENDS[backend]
|
| 414 |
+
register()
|
| 415 |
+
elif backend in MAYBE_AVAILABLE_BACKENDS:
|
| 416 |
+
warnings.warn(
|
| 417 |
+
f"joblib backend '{backend}' is not available on "
|
| 418 |
+
f"your system, falling back to {DEFAULT_BACKEND}.",
|
| 419 |
+
UserWarning,
|
| 420 |
+
stacklevel=2,
|
| 421 |
+
)
|
| 422 |
+
BACKENDS[backend] = BACKENDS[DEFAULT_BACKEND]
|
| 423 |
+
else:
|
| 424 |
+
raise ValueError(
|
| 425 |
+
f"Invalid backend: {backend}, expected one of "
|
| 426 |
+
f"{sorted(BACKENDS.keys())}"
|
| 427 |
+
)
|
| 428 |
+
|
| 429 |
+
backend = BACKENDS[backend](**backend_params)
|
| 430 |
+
else:
|
| 431 |
+
if len(backend_params) > 0:
|
| 432 |
+
raise ValueError(
|
| 433 |
+
"Constructor parameters backend_params are only "
|
| 434 |
+
"supported when backend is a string."
|
| 435 |
+
)
|
| 436 |
+
|
| 437 |
+
if inner_max_num_threads is not None:
|
| 438 |
+
msg = (
|
| 439 |
+
f"{backend.__class__.__name__} does not accept setting the "
|
| 440 |
+
"inner_max_num_threads argument."
|
| 441 |
+
)
|
| 442 |
+
assert backend.supports_inner_max_num_threads, msg
|
| 443 |
+
backend.inner_max_num_threads = inner_max_num_threads
|
| 444 |
+
|
| 445 |
+
# If the nesting_level of the backend is not set previously, use the
|
| 446 |
+
# nesting level from the previous active_backend to set it
|
| 447 |
+
if backend.nesting_level is None:
|
| 448 |
+
parent_backend = self.old_parallel_config["backend"]
|
| 449 |
+
if parent_backend is default_parallel_config["backend"]:
|
| 450 |
+
nesting_level = 0
|
| 451 |
+
else:
|
| 452 |
+
nesting_level = parent_backend.nesting_level
|
| 453 |
+
backend.nesting_level = nesting_level
|
| 454 |
+
|
| 455 |
+
return backend
|
| 456 |
+
|
| 457 |
+
def __enter__(self):
|
| 458 |
+
return self.parallel_config
|
| 459 |
+
|
| 460 |
+
def __exit__(self, type, value, traceback):
|
| 461 |
+
self.unregister()
|
| 462 |
+
|
| 463 |
+
def unregister(self):
|
| 464 |
+
setattr(_backend, "config", self.old_parallel_config)
|
| 465 |
+
|
| 466 |
+
|
| 467 |
+
class parallel_backend(parallel_config):
|
| 468 |
+
"""Change the default backend used by Parallel inside a with block.
|
| 469 |
+
|
| 470 |
+
.. warning::
|
| 471 |
+
It is advised to use the :class:`~joblib.parallel_config` context
|
| 472 |
+
manager instead, which allows more fine-grained control over the
|
| 473 |
+
backend configuration.
|
| 474 |
+
|
| 475 |
+
If ``backend`` is a string it must match a previously registered
|
| 476 |
+
implementation using the :func:`~register_parallel_backend` function.
|
| 477 |
+
|
| 478 |
+
By default the following backends are available:
|
| 479 |
+
|
| 480 |
+
- 'loky': single-host, process-based parallelism (used by default),
|
| 481 |
+
- 'threading': single-host, thread-based parallelism,
|
| 482 |
+
- 'multiprocessing': legacy single-host, process-based parallelism.
|
| 483 |
+
|
| 484 |
+
'loky' is recommended to run functions that manipulate Python objects.
|
| 485 |
+
'threading' is a low-overhead alternative that is most efficient for
|
| 486 |
+
functions that release the Global Interpreter Lock: e.g. I/O-bound code or
|
| 487 |
+
CPU-bound code in a few calls to native code that explicitly releases the
|
| 488 |
+
GIL. Note that on some rare systems (such as Pyodide),
|
| 489 |
+
multiprocessing and loky may not be available, in which case joblib
|
| 490 |
+
defaults to threading.
|
| 491 |
+
|
| 492 |
+
You can also use the `Dask <https://docs.dask.org/en/stable/>`_ joblib
|
| 493 |
+
backend to distribute work across machines. This works well with
|
| 494 |
+
scikit-learn estimators with the ``n_jobs`` parameter, for example::
|
| 495 |
+
|
| 496 |
+
>>> import joblib # doctest: +SKIP
|
| 497 |
+
>>> from sklearn.model_selection import GridSearchCV # doctest: +SKIP
|
| 498 |
+
>>> from dask.distributed import Client, LocalCluster # doctest: +SKIP
|
| 499 |
+
|
| 500 |
+
>>> # create a local Dask cluster
|
| 501 |
+
>>> cluster = LocalCluster() # doctest: +SKIP
|
| 502 |
+
>>> client = Client(cluster) # doctest: +SKIP
|
| 503 |
+
>>> grid_search = GridSearchCV(estimator, param_grid, n_jobs=-1)
|
| 504 |
+
... # doctest: +SKIP
|
| 505 |
+
>>> with joblib.parallel_backend("dask", scatter=[X, y]): # doctest: +SKIP
|
| 506 |
+
... grid_search.fit(X, y)
|
| 507 |
+
|
| 508 |
+
It is also possible to use the distributed 'ray' backend for distributing
|
| 509 |
+
the workload to a cluster of nodes. To use the 'ray' joblib backend add
|
| 510 |
+
the following lines::
|
| 511 |
+
|
| 512 |
+
>>> from ray.util.joblib import register_ray # doctest: +SKIP
|
| 513 |
+
>>> register_ray() # doctest: +SKIP
|
| 514 |
+
>>> with parallel_backend("ray"): # doctest: +SKIP
|
| 515 |
+
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
|
| 516 |
+
[-1, -2, -3, -4, -5]
|
| 517 |
+
|
| 518 |
+
Alternatively the backend can be passed directly as an instance.
|
| 519 |
+
|
| 520 |
+
By default all available workers will be used (``n_jobs=-1``) unless the
|
| 521 |
+
caller passes an explicit value for the ``n_jobs`` parameter.
|
| 522 |
+
|
| 523 |
+
This is an alternative to passing a ``backend='backend_name'`` argument to
|
| 524 |
+
the :class:`~Parallel` class constructor. It is particularly useful when
|
| 525 |
+
calling into library code that uses joblib internally but does not expose
|
| 526 |
+
the backend argument in its own API.
|
| 527 |
+
|
| 528 |
+
>>> from operator import neg
|
| 529 |
+
>>> with parallel_backend('threading'):
|
| 530 |
+
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
|
| 531 |
+
...
|
| 532 |
+
[-1, -2, -3, -4, -5]
|
| 533 |
+
|
| 534 |
+
Joblib also tries to limit the oversubscription by limiting the number of
|
| 535 |
+
threads usable in some third-party library threadpools like OpenBLAS, MKL
|
| 536 |
+
or OpenMP. The default limit in each worker is set to
|
| 537 |
+
``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
|
| 538 |
+
overwritten with the ``inner_max_num_threads`` argument which will be used
|
| 539 |
+
to set this limit in the child processes.
|
| 540 |
+
|
| 541 |
+
.. versionadded:: 0.10
|
| 542 |
+
|
| 543 |
+
See Also
|
| 544 |
+
--------
|
| 545 |
+
joblib.parallel_config: context manager to change the backend configuration.
|
| 546 |
+
"""
|
| 547 |
+
|
| 548 |
+
def __init__(
|
| 549 |
+
self, backend, n_jobs=-1, inner_max_num_threads=None, **backend_params
|
| 550 |
+
):
|
| 551 |
+
super().__init__(
|
| 552 |
+
backend=backend,
|
| 553 |
+
n_jobs=n_jobs,
|
| 554 |
+
inner_max_num_threads=inner_max_num_threads,
|
| 555 |
+
**backend_params,
|
| 556 |
+
)
|
| 557 |
+
|
| 558 |
+
if self.old_parallel_config is None:
|
| 559 |
+
self.old_backend_and_jobs = None
|
| 560 |
+
else:
|
| 561 |
+
self.old_backend_and_jobs = (
|
| 562 |
+
self.old_parallel_config["backend"],
|
| 563 |
+
self.old_parallel_config["n_jobs"],
|
| 564 |
+
)
|
| 565 |
+
self.new_backend_and_jobs = (
|
| 566 |
+
self.parallel_config["backend"],
|
| 567 |
+
self.parallel_config["n_jobs"],
|
| 568 |
+
)
|
| 569 |
+
|
| 570 |
+
def __enter__(self):
|
| 571 |
+
return self.new_backend_and_jobs
|
| 572 |
+
|
| 573 |
+
|
| 574 |
+
# Under Linux or OS X the default start method of multiprocessing
|
| 575 |
+
# can cause third party libraries to crash. Under Python 3.4+ it is possible
|
| 576 |
+
# to set an environment variable to switch the default start method from
|
| 577 |
+
# 'fork' to 'forkserver' or 'spawn' to avoid this issue albeit at the cost
|
| 578 |
+
# of causing semantic changes and some additional pool instantiation overhead.
|
| 579 |
+
DEFAULT_MP_CONTEXT = None
|
| 580 |
+
if hasattr(mp, "get_context"):
|
| 581 |
+
method = os.environ.get("JOBLIB_START_METHOD", "").strip() or None
|
| 582 |
+
if method is not None:
|
| 583 |
+
DEFAULT_MP_CONTEXT = mp.get_context(method=method)
|
| 584 |
+
|
| 585 |
+
|
| 586 |
+
class BatchedCalls(object):
|
| 587 |
+
"""Wrap a sequence of (func, args, kwargs) tuples as a single callable"""
|
| 588 |
+
|
| 589 |
+
def __init__(
|
| 590 |
+
self, iterator_slice, backend_and_jobs, reducer_callback=None, pickle_cache=None
|
| 591 |
+
):
|
| 592 |
+
self.items = list(iterator_slice)
|
| 593 |
+
self._size = len(self.items)
|
| 594 |
+
self._reducer_callback = reducer_callback
|
| 595 |
+
if isinstance(backend_and_jobs, tuple):
|
| 596 |
+
self._backend, self._n_jobs = backend_and_jobs
|
| 597 |
+
else:
|
| 598 |
+
# this is for backward compatibility purposes. Before 0.12.6,
|
| 599 |
+
# nested backends were returned without n_jobs indications.
|
| 600 |
+
self._backend, self._n_jobs = backend_and_jobs, None
|
| 601 |
+
self._pickle_cache = pickle_cache if pickle_cache is not None else {}
|
| 602 |
+
|
| 603 |
+
def __call__(self):
|
| 604 |
+
# Set the default nested backend to self._backend but do not set the
|
| 605 |
+
# change the default number of processes to -1
|
| 606 |
+
with parallel_config(backend=self._backend, n_jobs=self._n_jobs):
|
| 607 |
+
return [func(*args, **kwargs) for func, args, kwargs in self.items]
|
| 608 |
+
|
| 609 |
+
def __reduce__(self):
|
| 610 |
+
if self._reducer_callback is not None:
|
| 611 |
+
self._reducer_callback()
|
| 612 |
+
# no need to pickle the callback.
|
| 613 |
+
return (
|
| 614 |
+
BatchedCalls,
|
| 615 |
+
(self.items, (self._backend, self._n_jobs), None, self._pickle_cache),
|
| 616 |
+
)
|
| 617 |
+
|
| 618 |
+
def __len__(self):
|
| 619 |
+
return self._size
|
| 620 |
+
|
| 621 |
+
|
| 622 |
+
# Possible exit status for a task
|
| 623 |
+
TASK_DONE = "Done"
|
| 624 |
+
TASK_ERROR = "Error"
|
| 625 |
+
TASK_PENDING = "Pending"
|
| 626 |
+
|
| 627 |
+
|
| 628 |
+
###############################################################################
|
| 629 |
+
# CPU count that works also when multiprocessing has been disabled via
|
| 630 |
+
# the JOBLIB_MULTIPROCESSING environment variable
|
| 631 |
+
def cpu_count(only_physical_cores=False):
|
| 632 |
+
"""Return the number of CPUs.
|
| 633 |
+
|
| 634 |
+
This delegates to loky.cpu_count that takes into account additional
|
| 635 |
+
constraints such as Linux CFS scheduler quotas (typically set by container
|
| 636 |
+
runtimes such as docker) and CPU affinity (for instance using the taskset
|
| 637 |
+
command on Linux).
|
| 638 |
+
|
| 639 |
+
Parameters
|
| 640 |
+
----------
|
| 641 |
+
only_physical_cores : boolean, default=False
|
| 642 |
+
If True, does not take hyperthreading / SMT logical cores into account.
|
| 643 |
+
|
| 644 |
+
"""
|
| 645 |
+
if mp is None:
|
| 646 |
+
return 1
|
| 647 |
+
|
| 648 |
+
return loky.cpu_count(only_physical_cores=only_physical_cores)
|
| 649 |
+
|
| 650 |
+
|
| 651 |
+
###############################################################################
|
| 652 |
+
# For verbosity
|
| 653 |
+
|
| 654 |
+
|
| 655 |
+
def _verbosity_filter(index, verbose):
|
| 656 |
+
"""Returns False for indices increasingly apart, the distance
|
| 657 |
+
depending on the value of verbose.
|
| 658 |
+
|
| 659 |
+
We use a lag increasing as the square of index
|
| 660 |
+
"""
|
| 661 |
+
if not verbose:
|
| 662 |
+
return True
|
| 663 |
+
elif verbose > 10:
|
| 664 |
+
return False
|
| 665 |
+
if index == 0:
|
| 666 |
+
return False
|
| 667 |
+
verbose = 0.5 * (11 - verbose) ** 2
|
| 668 |
+
scale = sqrt(index / verbose)
|
| 669 |
+
next_scale = sqrt((index + 1) / verbose)
|
| 670 |
+
return int(next_scale) == int(scale)
|
| 671 |
+
|
| 672 |
+
|
| 673 |
+
###############################################################################
|
| 674 |
+
def delayed(function):
|
| 675 |
+
"""Decorator used to capture the arguments of a function."""
|
| 676 |
+
|
| 677 |
+
def delayed_function(*args, **kwargs):
|
| 678 |
+
return function, args, kwargs
|
| 679 |
+
|
| 680 |
+
try:
|
| 681 |
+
delayed_function = functools.wraps(function)(delayed_function)
|
| 682 |
+
except AttributeError:
|
| 683 |
+
" functools.wraps fails on some callable objects "
|
| 684 |
+
return delayed_function
|
| 685 |
+
|
| 686 |
+
|
| 687 |
+
###############################################################################
|
| 688 |
+
class BatchCompletionCallBack(object):
|
| 689 |
+
"""Callback to keep track of completed results and schedule the next tasks.
|
| 690 |
+
|
| 691 |
+
This callable is executed by the parent process whenever a worker process
|
| 692 |
+
has completed a batch of tasks.
|
| 693 |
+
|
| 694 |
+
It is used for progress reporting, to update estimate of the batch
|
| 695 |
+
processing duration and to schedule the next batch of tasks to be
|
| 696 |
+
processed.
|
| 697 |
+
|
| 698 |
+
It is assumed that this callback will always be triggered by the backend
|
| 699 |
+
right after the end of a task, in case of success as well as in case of
|
| 700 |
+
failure.
|
| 701 |
+
"""
|
| 702 |
+
|
| 703 |
+
##########################################################################
|
| 704 |
+
# METHODS CALLED BY THE MAIN THREAD #
|
| 705 |
+
##########################################################################
|
| 706 |
+
def __init__(self, dispatch_timestamp, batch_size, parallel):
|
| 707 |
+
self.dispatch_timestamp = dispatch_timestamp
|
| 708 |
+
self.batch_size = batch_size
|
| 709 |
+
self.parallel = parallel
|
| 710 |
+
self.parallel_call_id = parallel._call_id
|
| 711 |
+
self._completion_timeout_counter = None
|
| 712 |
+
|
| 713 |
+
# Internals to keep track of the status and outcome of the task.
|
| 714 |
+
|
| 715 |
+
# Used to hold a reference to the future-like object returned by the
|
| 716 |
+
# backend after launching this task
|
| 717 |
+
# This will be set later when calling `register_job`, as it is only
|
| 718 |
+
# created once the task has been submitted.
|
| 719 |
+
self.job = None
|
| 720 |
+
|
| 721 |
+
if not parallel._backend.supports_retrieve_callback:
|
| 722 |
+
# The status is only used for asynchronous result retrieval in the
|
| 723 |
+
# callback.
|
| 724 |
+
self.status = None
|
| 725 |
+
else:
|
| 726 |
+
# The initial status for the job is TASK_PENDING.
|
| 727 |
+
# Once it is done, it will be either TASK_DONE, or TASK_ERROR.
|
| 728 |
+
self.status = TASK_PENDING
|
| 729 |
+
|
| 730 |
+
def register_job(self, job):
|
| 731 |
+
"""Register the object returned by `submit`."""
|
| 732 |
+
self.job = job
|
| 733 |
+
|
| 734 |
+
def get_result(self, timeout):
|
| 735 |
+
"""Returns the raw result of the task that was submitted.
|
| 736 |
+
|
| 737 |
+
If the task raised an exception rather than returning, this same
|
| 738 |
+
exception will be raised instead.
|
| 739 |
+
|
| 740 |
+
If the backend supports the retrieval callback, it is assumed that this
|
| 741 |
+
method is only called after the result has been registered. It is
|
| 742 |
+
ensured by checking that `self.status(timeout)` does not return
|
| 743 |
+
TASK_PENDING. In this case, `get_result` directly returns the
|
| 744 |
+
registered result (or raise the registered exception).
|
| 745 |
+
|
| 746 |
+
For other backends, there are no such assumptions, but `get_result`
|
| 747 |
+
still needs to synchronously retrieve the result before it can
|
| 748 |
+
return it or raise. It will block at most `self.timeout` seconds
|
| 749 |
+
waiting for retrieval to complete, after that it raises a TimeoutError.
|
| 750 |
+
"""
|
| 751 |
+
|
| 752 |
+
backend = self.parallel._backend
|
| 753 |
+
|
| 754 |
+
if backend.supports_retrieve_callback:
|
| 755 |
+
# We assume that the result has already been retrieved by the
|
| 756 |
+
# callback thread, and is stored internally. It's just waiting to
|
| 757 |
+
# be returned.
|
| 758 |
+
return self._return_or_raise()
|
| 759 |
+
|
| 760 |
+
# For other backends, the main thread needs to run the retrieval step.
|
| 761 |
+
try:
|
| 762 |
+
result = backend.retrieve_result(self.job, timeout=timeout)
|
| 763 |
+
outcome = dict(result=result, status=TASK_DONE)
|
| 764 |
+
except BaseException as e:
|
| 765 |
+
outcome = dict(result=e, status=TASK_ERROR)
|
| 766 |
+
self._register_outcome(outcome)
|
| 767 |
+
|
| 768 |
+
return self._return_or_raise()
|
| 769 |
+
|
| 770 |
+
def _return_or_raise(self):
|
| 771 |
+
try:
|
| 772 |
+
if self.status == TASK_ERROR:
|
| 773 |
+
raise self._result
|
| 774 |
+
return self._result
|
| 775 |
+
finally:
|
| 776 |
+
del self._result
|
| 777 |
+
|
| 778 |
+
def get_status(self, timeout):
|
| 779 |
+
"""Get the status of the task.
|
| 780 |
+
|
| 781 |
+
This function also checks if the timeout has been reached and register
|
| 782 |
+
the TimeoutError outcome when it is the case.
|
| 783 |
+
"""
|
| 784 |
+
if timeout is None or self.status != TASK_PENDING:
|
| 785 |
+
return self.status
|
| 786 |
+
|
| 787 |
+
# The computation are running and the status is pending.
|
| 788 |
+
# Check that we did not wait for this jobs more than `timeout`.
|
| 789 |
+
now = time.time()
|
| 790 |
+
if self._completion_timeout_counter is None:
|
| 791 |
+
self._completion_timeout_counter = now
|
| 792 |
+
|
| 793 |
+
if (now - self._completion_timeout_counter) > timeout:
|
| 794 |
+
outcome = dict(result=TimeoutError(), status=TASK_ERROR)
|
| 795 |
+
self._register_outcome(outcome)
|
| 796 |
+
|
| 797 |
+
return self.status
|
| 798 |
+
|
| 799 |
+
##########################################################################
|
| 800 |
+
# METHODS CALLED BY CALLBACK THREADS #
|
| 801 |
+
##########################################################################
|
| 802 |
+
def __call__(self, *args, **kwargs):
|
| 803 |
+
"""Function called by the callback thread after a job is completed."""
|
| 804 |
+
|
| 805 |
+
# If the backend doesn't support callback retrievals, the next batch of
|
| 806 |
+
# tasks is dispatched regardless. The result will be retrieved by the
|
| 807 |
+
# main thread when calling `get_result`.
|
| 808 |
+
if not self.parallel._backend.supports_retrieve_callback:
|
| 809 |
+
self._dispatch_new()
|
| 810 |
+
return
|
| 811 |
+
|
| 812 |
+
# If the backend supports retrieving the result in the callback, it
|
| 813 |
+
# registers the task outcome (TASK_ERROR or TASK_DONE), and schedules
|
| 814 |
+
# the next batch if needed.
|
| 815 |
+
with self.parallel._lock:
|
| 816 |
+
# Edge case where while the task was processing, the `parallel`
|
| 817 |
+
# instance has been reset and a new call has been issued, but the
|
| 818 |
+
# worker managed to complete the task and trigger this callback
|
| 819 |
+
# call just before being aborted by the reset.
|
| 820 |
+
if self.parallel._call_id != self.parallel_call_id:
|
| 821 |
+
return
|
| 822 |
+
|
| 823 |
+
# When aborting, stop as fast as possible and do not retrieve the
|
| 824 |
+
# result as it won't be returned by the Parallel call.
|
| 825 |
+
if self.parallel._aborting:
|
| 826 |
+
return
|
| 827 |
+
|
| 828 |
+
# Retrieves the result of the task in the main process and dispatch
|
| 829 |
+
# a new batch if needed.
|
| 830 |
+
job_succeeded = self._retrieve_result(*args, **kwargs)
|
| 831 |
+
|
| 832 |
+
if job_succeeded:
|
| 833 |
+
self._dispatch_new()
|
| 834 |
+
|
| 835 |
+
def _dispatch_new(self):
|
| 836 |
+
"""Schedule the next batch of tasks to be processed."""
|
| 837 |
+
|
| 838 |
+
# This steps ensure that auto-batching works as expected.
|
| 839 |
+
this_batch_duration = time.time() - self.dispatch_timestamp
|
| 840 |
+
self.parallel._backend.batch_completed(self.batch_size, this_batch_duration)
|
| 841 |
+
|
| 842 |
+
# Schedule the next batch of tasks.
|
| 843 |
+
with self.parallel._lock:
|
| 844 |
+
self.parallel.n_completed_tasks += self.batch_size
|
| 845 |
+
self.parallel.print_progress()
|
| 846 |
+
if self.parallel._original_iterator is not None:
|
| 847 |
+
self.parallel.dispatch_next()
|
| 848 |
+
|
| 849 |
+
def _retrieve_result(self, out):
|
| 850 |
+
"""Fetch and register the outcome of a task.
|
| 851 |
+
|
| 852 |
+
Return True if the task succeeded, False otherwise.
|
| 853 |
+
This function is only called by backends that support retrieving
|
| 854 |
+
the task result in the callback thread.
|
| 855 |
+
"""
|
| 856 |
+
try:
|
| 857 |
+
result = self.parallel._backend.retrieve_result_callback(out)
|
| 858 |
+
outcome = dict(status=TASK_DONE, result=result)
|
| 859 |
+
except BaseException as e:
|
| 860 |
+
# Avoid keeping references to parallel in the error.
|
| 861 |
+
e.__traceback__ = None
|
| 862 |
+
outcome = dict(result=e, status=TASK_ERROR)
|
| 863 |
+
|
| 864 |
+
self._register_outcome(outcome)
|
| 865 |
+
return outcome["status"] != TASK_ERROR
|
| 866 |
+
|
| 867 |
+
##########################################################################
|
| 868 |
+
# This method can be called either in the main thread #
|
| 869 |
+
# or in the callback thread. #
|
| 870 |
+
##########################################################################
|
| 871 |
+
def _register_outcome(self, outcome):
|
| 872 |
+
"""Register the outcome of a task.
|
| 873 |
+
|
| 874 |
+
This method can be called only once, future calls will be ignored.
|
| 875 |
+
"""
|
| 876 |
+
# Covers the edge case where the main thread tries to register a
|
| 877 |
+
# `TimeoutError` while the callback thread tries to register a result
|
| 878 |
+
# at the same time.
|
| 879 |
+
with self.parallel._lock:
|
| 880 |
+
if self.status not in (TASK_PENDING, None):
|
| 881 |
+
return
|
| 882 |
+
self.status = outcome["status"]
|
| 883 |
+
|
| 884 |
+
self._result = outcome["result"]
|
| 885 |
+
|
| 886 |
+
# Once the result and the status are extracted, the last reference to
|
| 887 |
+
# the job can be deleted.
|
| 888 |
+
self.job = None
|
| 889 |
+
|
| 890 |
+
# As soon as an error as been spotted, early stopping flags are sent to
|
| 891 |
+
# the `parallel` instance.
|
| 892 |
+
if self.status == TASK_ERROR:
|
| 893 |
+
self.parallel._exception = True
|
| 894 |
+
self.parallel._aborting = True
|
| 895 |
+
|
| 896 |
+
if self.parallel.return_ordered:
|
| 897 |
+
return
|
| 898 |
+
|
| 899 |
+
with self.parallel._lock:
|
| 900 |
+
# For `return_as=generator_unordered`, append the job to the queue
|
| 901 |
+
# in the order of completion instead of submission.
|
| 902 |
+
self.parallel._jobs.append(self)
|
| 903 |
+
|
| 904 |
+
|
| 905 |
+
###############################################################################
|
| 906 |
+
def register_parallel_backend(name, factory, make_default=False):
|
| 907 |
+
"""Register a new Parallel backend factory.
|
| 908 |
+
|
| 909 |
+
The new backend can then be selected by passing its name as the backend
|
| 910 |
+
argument to the :class:`~Parallel` class. Moreover, the default backend can
|
| 911 |
+
be overwritten globally by setting make_default=True.
|
| 912 |
+
|
| 913 |
+
The factory can be any callable that takes no argument and return an
|
| 914 |
+
instance of ``ParallelBackendBase``.
|
| 915 |
+
|
| 916 |
+
Warning: this function is experimental and subject to change in a future
|
| 917 |
+
version of joblib.
|
| 918 |
+
|
| 919 |
+
.. versionadded:: 0.10
|
| 920 |
+
"""
|
| 921 |
+
BACKENDS[name] = factory
|
| 922 |
+
if make_default:
|
| 923 |
+
global DEFAULT_BACKEND
|
| 924 |
+
DEFAULT_BACKEND = name
|
| 925 |
+
|
| 926 |
+
|
| 927 |
+
def effective_n_jobs(n_jobs=-1):
|
| 928 |
+
"""Determine the number of jobs that can actually run in parallel
|
| 929 |
+
|
| 930 |
+
n_jobs is the number of workers requested by the callers. Passing n_jobs=-1
|
| 931 |
+
means requesting all available workers for instance matching the number of
|
| 932 |
+
CPU cores on the worker host(s).
|
| 933 |
+
|
| 934 |
+
This method should return a guesstimate of the number of workers that can
|
| 935 |
+
actually perform work concurrently with the currently enabled default
|
| 936 |
+
backend. The primary use case is to make it possible for the caller to know
|
| 937 |
+
in how many chunks to slice the work.
|
| 938 |
+
|
| 939 |
+
In general working on larger data chunks is more efficient (less scheduling
|
| 940 |
+
overhead and better use of CPU cache prefetching heuristics) as long as all
|
| 941 |
+
the workers have enough work to do.
|
| 942 |
+
|
| 943 |
+
Warning: this function is experimental and subject to change in a future
|
| 944 |
+
version of joblib.
|
| 945 |
+
|
| 946 |
+
.. versionadded:: 0.10
|
| 947 |
+
"""
|
| 948 |
+
if n_jobs == 1:
|
| 949 |
+
return 1
|
| 950 |
+
|
| 951 |
+
backend, backend_n_jobs = get_active_backend()
|
| 952 |
+
if n_jobs is None:
|
| 953 |
+
n_jobs = backend_n_jobs
|
| 954 |
+
return backend.effective_n_jobs(n_jobs=n_jobs)
|
| 955 |
+
|
| 956 |
+
|
| 957 |
+
###############################################################################
|
| 958 |
+
class Parallel(Logger):
|
| 959 |
+
"""Helper class for readable parallel mapping.
|
| 960 |
+
|
| 961 |
+
Read more in the :ref:`User Guide <parallel>`.
|
| 962 |
+
|
| 963 |
+
Parameters
|
| 964 |
+
----------
|
| 965 |
+
n_jobs: int, default=None
|
| 966 |
+
The maximum number of concurrently running jobs, such as the number
|
| 967 |
+
of Python worker processes when ``backend="loky"`` or the size of
|
| 968 |
+
the thread-pool when ``backend="threading"``.
|
| 969 |
+
This argument is converted to an integer, rounded below for float.
|
| 970 |
+
If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
|
| 971 |
+
``n_cpus`` is obtained with :func:`~cpu_count`.
|
| 972 |
+
For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
|
| 973 |
+
using ``n_jobs=-2`` will result in all CPUs but one being used.
|
| 974 |
+
This argument can also go above ``n_cpus``, which will cause
|
| 975 |
+
oversubscription. In some cases, slight oversubscription can be
|
| 976 |
+
beneficial, e.g., for tasks with large I/O operations.
|
| 977 |
+
If 1 is given, no parallel computing code is used at all, and the
|
| 978 |
+
behavior amounts to a simple python `for` loop. This mode is not
|
| 979 |
+
compatible with ``timeout``.
|
| 980 |
+
None is a marker for 'unset' that will be interpreted as n_jobs=1
|
| 981 |
+
unless the call is performed under a :func:`~parallel_config`
|
| 982 |
+
context manager that sets another value for ``n_jobs``.
|
| 983 |
+
If n_jobs = 0 then a ValueError is raised.
|
| 984 |
+
backend: str, ParallelBackendBase instance or None, default='loky'
|
| 985 |
+
Specify the parallelization backend implementation.
|
| 986 |
+
Supported backends are:
|
| 987 |
+
|
| 988 |
+
- "loky" used by default, can induce some
|
| 989 |
+
communication and memory overhead when exchanging input and
|
| 990 |
+
output data with the worker Python processes. On some rare
|
| 991 |
+
systems (such as Pyiodide), the loky backend may not be
|
| 992 |
+
available.
|
| 993 |
+
- "multiprocessing" previous process-based backend based on
|
| 994 |
+
`multiprocessing.Pool`. Less robust than `loky`.
|
| 995 |
+
- "threading" is a very low-overhead backend but it suffers
|
| 996 |
+
from the Python Global Interpreter Lock if the called function
|
| 997 |
+
relies a lot on Python objects. "threading" is mostly useful
|
| 998 |
+
when the execution bottleneck is a compiled extension that
|
| 999 |
+
explicitly releases the GIL (for instance a Cython loop wrapped
|
| 1000 |
+
in a "with nogil" block or an expensive call to a library such
|
| 1001 |
+
as NumPy).
|
| 1002 |
+
- finally, you can register backends by calling
|
| 1003 |
+
:func:`~register_parallel_backend`. This will allow you to
|
| 1004 |
+
implement a backend of your liking.
|
| 1005 |
+
|
| 1006 |
+
It is not recommended to hard-code the backend name in a call to
|
| 1007 |
+
:class:`~Parallel` in a library. Instead it is recommended to set
|
| 1008 |
+
soft hints (prefer) or hard constraints (require) so as to make it
|
| 1009 |
+
possible for library users to change the backend from the outside
|
| 1010 |
+
using the :func:`~parallel_config` context manager.
|
| 1011 |
+
return_as: str in {'list', 'generator', 'generator_unordered'}, default='list'
|
| 1012 |
+
If 'list', calls to this instance will return a list, only when
|
| 1013 |
+
all results have been processed and retrieved.
|
| 1014 |
+
If 'generator', it will return a generator that yields the results
|
| 1015 |
+
as soon as they are available, in the order the tasks have been
|
| 1016 |
+
submitted with.
|
| 1017 |
+
If 'generator_unordered', the generator will immediately yield
|
| 1018 |
+
available results independently of the submission order. The output
|
| 1019 |
+
order is not deterministic in this case because it depends on the
|
| 1020 |
+
concurrency of the workers.
|
| 1021 |
+
prefer: str in {'processes', 'threads'} or None, default=None
|
| 1022 |
+
Soft hint to choose the default backend if no specific backend
|
| 1023 |
+
was selected with the :func:`~parallel_config` context manager.
|
| 1024 |
+
The default process-based backend is 'loky' and the default
|
| 1025 |
+
thread-based backend is 'threading'. Ignored if the ``backend``
|
| 1026 |
+
parameter is specified.
|
| 1027 |
+
require: 'sharedmem' or None, default=None
|
| 1028 |
+
Hard constraint to select the backend. If set to 'sharedmem',
|
| 1029 |
+
the selected backend will be single-host and thread-based even
|
| 1030 |
+
if the user asked for a non-thread based backend with
|
| 1031 |
+
:func:`~joblib.parallel_config`.
|
| 1032 |
+
verbose: int, default=0
|
| 1033 |
+
The verbosity level: if non zero, progress messages are
|
| 1034 |
+
printed. Above 50, the output is sent to stdout.
|
| 1035 |
+
The frequency of the messages increases with the verbosity level.
|
| 1036 |
+
If it more than 10, all iterations are reported.
|
| 1037 |
+
timeout: float or None, default=None
|
| 1038 |
+
Timeout limit for each task to complete. If any task takes longer
|
| 1039 |
+
a TimeOutError will be raised. Only applied when n_jobs != 1
|
| 1040 |
+
pre_dispatch: {'all', integer, or expression, as in '3*n_jobs'}, default='2*n_jobs'
|
| 1041 |
+
The number of batches (of tasks) to be pre-dispatched.
|
| 1042 |
+
Default is '2*n_jobs'. When batch_size="auto" this is reasonable
|
| 1043 |
+
default and the workers should never starve. Note that only basic
|
| 1044 |
+
arithmetic are allowed here and no modules can be used in this
|
| 1045 |
+
expression.
|
| 1046 |
+
batch_size: int or 'auto', default='auto'
|
| 1047 |
+
The number of atomic tasks to dispatch at once to each
|
| 1048 |
+
worker. When individual evaluations are very fast, dispatching
|
| 1049 |
+
calls to workers can be slower than sequential computation because
|
| 1050 |
+
of the overhead. Batching fast computations together can mitigate
|
| 1051 |
+
this.
|
| 1052 |
+
The ``'auto'`` strategy keeps track of the time it takes for a
|
| 1053 |
+
batch to complete, and dynamically adjusts the batch size to keep
|
| 1054 |
+
the time on the order of half a second, using a heuristic. The
|
| 1055 |
+
initial batch size is 1.
|
| 1056 |
+
``batch_size="auto"`` with ``backend="threading"`` will dispatch
|
| 1057 |
+
batches of a single task at a time as the threading backend has
|
| 1058 |
+
very little overhead and using larger batch size has not proved to
|
| 1059 |
+
bring any gain in that case.
|
| 1060 |
+
temp_folder: str or None, default=None
|
| 1061 |
+
Folder to be used by the pool for memmapping large arrays
|
| 1062 |
+
for sharing memory with worker processes. If None, this will try in
|
| 1063 |
+
order:
|
| 1064 |
+
|
| 1065 |
+
- a folder pointed by the JOBLIB_TEMP_FOLDER environment
|
| 1066 |
+
variable,
|
| 1067 |
+
- /dev/shm if the folder exists and is writable: this is a
|
| 1068 |
+
RAM disk filesystem available by default on modern Linux
|
| 1069 |
+
distributions,
|
| 1070 |
+
- the default system temporary folder that can be
|
| 1071 |
+
overridden with TMP, TMPDIR or TEMP environment
|
| 1072 |
+
variables, typically /tmp under Unix operating systems.
|
| 1073 |
+
|
| 1074 |
+
Only active when ``backend="loky"`` or ``"multiprocessing"``.
|
| 1075 |
+
max_nbytes int, str, or None, optional, default='1M'
|
| 1076 |
+
Threshold on the size of arrays passed to the workers that
|
| 1077 |
+
triggers automated memory mapping in temp_folder. Can be an int
|
| 1078 |
+
in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
|
| 1079 |
+
Use None to disable memmapping of large arrays.
|
| 1080 |
+
Only active when ``backend="loky"`` or ``"multiprocessing"``.
|
| 1081 |
+
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
|
| 1082 |
+
Memmapping mode for numpy arrays passed to workers. None will
|
| 1083 |
+
disable memmapping, other modes defined in the numpy.memmap doc:
|
| 1084 |
+
https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
|
| 1085 |
+
Also, see 'max_nbytes' parameter documentation for more details.
|
| 1086 |
+
backend_kwargs: dict, optional
|
| 1087 |
+
Additional parameters to pass to the backend `configure` method.
|
| 1088 |
+
|
| 1089 |
+
Notes
|
| 1090 |
+
-----
|
| 1091 |
+
|
| 1092 |
+
This object uses workers to compute in parallel the application of a
|
| 1093 |
+
function to many different arguments. The main functionality it brings
|
| 1094 |
+
in addition to using the raw multiprocessing or concurrent.futures API
|
| 1095 |
+
are (see examples for details):
|
| 1096 |
+
|
| 1097 |
+
* More readable code, in particular since it avoids
|
| 1098 |
+
constructing list of arguments.
|
| 1099 |
+
|
| 1100 |
+
* Easier debugging:
|
| 1101 |
+
- informative tracebacks even when the error happens on
|
| 1102 |
+
the client side
|
| 1103 |
+
- using 'n_jobs=1' enables to turn off parallel computing
|
| 1104 |
+
for debugging without changing the codepath
|
| 1105 |
+
- early capture of pickling errors
|
| 1106 |
+
|
| 1107 |
+
* An optional progress meter.
|
| 1108 |
+
|
| 1109 |
+
* Interruption of multiprocesses jobs with 'Ctrl-C'
|
| 1110 |
+
|
| 1111 |
+
* Flexible pickling control for the communication to and from
|
| 1112 |
+
the worker processes.
|
| 1113 |
+
|
| 1114 |
+
* Ability to use shared memory efficiently with worker
|
| 1115 |
+
processes for large numpy-based datastructures.
|
| 1116 |
+
|
| 1117 |
+
Note that the intended usage is to run one call at a time. Multiple
|
| 1118 |
+
calls to the same Parallel object will result in a ``RuntimeError``
|
| 1119 |
+
|
| 1120 |
+
Examples
|
| 1121 |
+
--------
|
| 1122 |
+
|
| 1123 |
+
A simple example:
|
| 1124 |
+
|
| 1125 |
+
>>> from math import sqrt
|
| 1126 |
+
>>> from joblib import Parallel, delayed
|
| 1127 |
+
>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
|
| 1128 |
+
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
|
| 1129 |
+
|
| 1130 |
+
Reshaping the output when the function has several return
|
| 1131 |
+
values:
|
| 1132 |
+
|
| 1133 |
+
>>> from math import modf
|
| 1134 |
+
>>> from joblib import Parallel, delayed
|
| 1135 |
+
>>> r = Parallel(n_jobs=1)(delayed(modf)(i/2.) for i in range(10))
|
| 1136 |
+
>>> res, i = zip(*r)
|
| 1137 |
+
>>> res
|
| 1138 |
+
(0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5)
|
| 1139 |
+
>>> i
|
| 1140 |
+
(0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0)
|
| 1141 |
+
|
| 1142 |
+
The progress meter: the higher the value of `verbose`, the more
|
| 1143 |
+
messages:
|
| 1144 |
+
|
| 1145 |
+
>>> from time import sleep
|
| 1146 |
+
>>> from joblib import Parallel, delayed
|
| 1147 |
+
>>> r = Parallel(n_jobs=2, verbose=10)(
|
| 1148 |
+
... delayed(sleep)(.2) for _ in range(10)) #doctest: +SKIP
|
| 1149 |
+
[Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 0.6s
|
| 1150 |
+
[Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 0.8s
|
| 1151 |
+
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 1.4s finished
|
| 1152 |
+
|
| 1153 |
+
Traceback example, note how the line of the error is indicated
|
| 1154 |
+
as well as the values of the parameter passed to the function that
|
| 1155 |
+
triggered the exception, even though the traceback happens in the
|
| 1156 |
+
child process:
|
| 1157 |
+
|
| 1158 |
+
>>> from heapq import nlargest
|
| 1159 |
+
>>> from joblib import Parallel, delayed
|
| 1160 |
+
>>> Parallel(n_jobs=2)(
|
| 1161 |
+
... delayed(nlargest)(2, n) for n in (range(4), 'abcde', 3))
|
| 1162 |
+
... # doctest: +SKIP
|
| 1163 |
+
-----------------------------------------------------------------------
|
| 1164 |
+
Sub-process traceback:
|
| 1165 |
+
-----------------------------------------------------------------------
|
| 1166 |
+
TypeError Mon Nov 12 11:37:46 2012
|
| 1167 |
+
PID: 12934 Python 2.7.3: /usr/bin/python
|
| 1168 |
+
........................................................................
|
| 1169 |
+
/usr/lib/python2.7/heapq.pyc in nlargest(n=2, iterable=3, key=None)
|
| 1170 |
+
419 if n >= size:
|
| 1171 |
+
420 return sorted(iterable, key=key, reverse=True)[:n]
|
| 1172 |
+
421
|
| 1173 |
+
422 # When key is none, use simpler decoration
|
| 1174 |
+
423 if key is None:
|
| 1175 |
+
--> 424 it = izip(iterable, count(0,-1)) # decorate
|
| 1176 |
+
425 result = _nlargest(n, it)
|
| 1177 |
+
426 return map(itemgetter(0), result) # undecorate
|
| 1178 |
+
427
|
| 1179 |
+
428 # General case, slowest method
|
| 1180 |
+
TypeError: izip argument #1 must support iteration
|
| 1181 |
+
_______________________________________________________________________
|
| 1182 |
+
|
| 1183 |
+
|
| 1184 |
+
Using pre_dispatch in a producer/consumer situation, where the
|
| 1185 |
+
data is generated on the fly. Note how the producer is first
|
| 1186 |
+
called 3 times before the parallel loop is initiated, and then
|
| 1187 |
+
called to generate new data on the fly:
|
| 1188 |
+
|
| 1189 |
+
>>> from math import sqrt
|
| 1190 |
+
>>> from joblib import Parallel, delayed
|
| 1191 |
+
>>> def producer():
|
| 1192 |
+
... for i in range(6):
|
| 1193 |
+
... print('Produced %s' % i)
|
| 1194 |
+
... yield i
|
| 1195 |
+
>>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')(
|
| 1196 |
+
... delayed(sqrt)(i) for i in producer()) #doctest: +SKIP
|
| 1197 |
+
Produced 0
|
| 1198 |
+
Produced 1
|
| 1199 |
+
Produced 2
|
| 1200 |
+
[Parallel(n_jobs=2)]: Done 1 jobs | elapsed: 0.0s
|
| 1201 |
+
Produced 3
|
| 1202 |
+
[Parallel(n_jobs=2)]: Done 2 jobs | elapsed: 0.0s
|
| 1203 |
+
Produced 4
|
| 1204 |
+
[Parallel(n_jobs=2)]: Done 3 jobs | elapsed: 0.0s
|
| 1205 |
+
Produced 5
|
| 1206 |
+
[Parallel(n_jobs=2)]: Done 4 jobs | elapsed: 0.0s
|
| 1207 |
+
[Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s remaining: 0.0s
|
| 1208 |
+
[Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s finished
|
| 1209 |
+
|
| 1210 |
+
""" # noqa: E501
|
| 1211 |
+
|
| 1212 |
+
def __init__(
|
| 1213 |
+
self,
|
| 1214 |
+
n_jobs=default_parallel_config["n_jobs"],
|
| 1215 |
+
backend=default_parallel_config["backend"],
|
| 1216 |
+
return_as="list",
|
| 1217 |
+
verbose=default_parallel_config["verbose"],
|
| 1218 |
+
timeout=None,
|
| 1219 |
+
pre_dispatch="2 * n_jobs",
|
| 1220 |
+
batch_size="auto",
|
| 1221 |
+
temp_folder=default_parallel_config["temp_folder"],
|
| 1222 |
+
max_nbytes=default_parallel_config["max_nbytes"],
|
| 1223 |
+
mmap_mode=default_parallel_config["mmap_mode"],
|
| 1224 |
+
prefer=default_parallel_config["prefer"],
|
| 1225 |
+
require=default_parallel_config["require"],
|
| 1226 |
+
**backend_kwargs,
|
| 1227 |
+
):
|
| 1228 |
+
# Initiate parent Logger class state
|
| 1229 |
+
super().__init__()
|
| 1230 |
+
|
| 1231 |
+
# Interpret n_jobs=None as 'unset'
|
| 1232 |
+
if n_jobs is None:
|
| 1233 |
+
n_jobs = default_parallel_config["n_jobs"]
|
| 1234 |
+
|
| 1235 |
+
active_backend, context_config = _get_active_backend(
|
| 1236 |
+
prefer=prefer, require=require, verbose=verbose
|
| 1237 |
+
)
|
| 1238 |
+
|
| 1239 |
+
nesting_level = active_backend.nesting_level
|
| 1240 |
+
|
| 1241 |
+
self.verbose = _get_config_param(verbose, context_config, "verbose")
|
| 1242 |
+
self.timeout = timeout
|
| 1243 |
+
self.pre_dispatch = pre_dispatch
|
| 1244 |
+
|
| 1245 |
+
if return_as not in {"list", "generator", "generator_unordered"}:
|
| 1246 |
+
raise ValueError(
|
| 1247 |
+
'Expected `return_as` parameter to be a string equal to "list"'
|
| 1248 |
+
f',"generator" or "generator_unordered", but got {return_as} '
|
| 1249 |
+
"instead."
|
| 1250 |
+
)
|
| 1251 |
+
self.return_as = return_as
|
| 1252 |
+
self.return_generator = return_as != "list"
|
| 1253 |
+
self.return_ordered = return_as != "generator_unordered"
|
| 1254 |
+
|
| 1255 |
+
# Check if we are under a parallel_config or parallel_backend
|
| 1256 |
+
# context manager and use the config from the context manager
|
| 1257 |
+
# for arguments that are not explicitly set.
|
| 1258 |
+
self._backend_kwargs = {
|
| 1259 |
+
**backend_kwargs,
|
| 1260 |
+
**{
|
| 1261 |
+
k: _get_config_param(param, context_config, k)
|
| 1262 |
+
for param, k in [
|
| 1263 |
+
(max_nbytes, "max_nbytes"),
|
| 1264 |
+
(temp_folder, "temp_folder"),
|
| 1265 |
+
(mmap_mode, "mmap_mode"),
|
| 1266 |
+
(prefer, "prefer"),
|
| 1267 |
+
(require, "require"),
|
| 1268 |
+
(verbose, "verbose"),
|
| 1269 |
+
]
|
| 1270 |
+
},
|
| 1271 |
+
}
|
| 1272 |
+
|
| 1273 |
+
if isinstance(self._backend_kwargs["max_nbytes"], str):
|
| 1274 |
+
self._backend_kwargs["max_nbytes"] = memstr_to_bytes(
|
| 1275 |
+
self._backend_kwargs["max_nbytes"]
|
| 1276 |
+
)
|
| 1277 |
+
self._backend_kwargs["verbose"] = max(0, self._backend_kwargs["verbose"] - 50)
|
| 1278 |
+
|
| 1279 |
+
if DEFAULT_MP_CONTEXT is not None:
|
| 1280 |
+
self._backend_kwargs["context"] = DEFAULT_MP_CONTEXT
|
| 1281 |
+
elif hasattr(mp, "get_context"):
|
| 1282 |
+
self._backend_kwargs["context"] = mp.get_context()
|
| 1283 |
+
|
| 1284 |
+
if backend is default_parallel_config["backend"] or backend is None:
|
| 1285 |
+
backend = active_backend
|
| 1286 |
+
|
| 1287 |
+
elif isinstance(backend, ParallelBackendBase):
|
| 1288 |
+
# Use provided backend as is, with the current nesting_level if it
|
| 1289 |
+
# is not set yet.
|
| 1290 |
+
if backend.nesting_level is None:
|
| 1291 |
+
backend.nesting_level = nesting_level
|
| 1292 |
+
|
| 1293 |
+
elif hasattr(backend, "Pool") and hasattr(backend, "Lock"):
|
| 1294 |
+
# Make it possible to pass a custom multiprocessing context as
|
| 1295 |
+
# backend to change the start method to forkserver or spawn or
|
| 1296 |
+
# preload modules on the forkserver helper process.
|
| 1297 |
+
self._backend_kwargs["context"] = backend
|
| 1298 |
+
backend = MultiprocessingBackend(nesting_level=nesting_level)
|
| 1299 |
+
|
| 1300 |
+
elif backend not in BACKENDS and backend in MAYBE_AVAILABLE_BACKENDS:
|
| 1301 |
+
warnings.warn(
|
| 1302 |
+
f"joblib backend '{backend}' is not available on "
|
| 1303 |
+
f"your system, falling back to {DEFAULT_BACKEND}.",
|
| 1304 |
+
UserWarning,
|
| 1305 |
+
stacklevel=2,
|
| 1306 |
+
)
|
| 1307 |
+
BACKENDS[backend] = BACKENDS[DEFAULT_BACKEND]
|
| 1308 |
+
backend = BACKENDS[DEFAULT_BACKEND](nesting_level=nesting_level)
|
| 1309 |
+
|
| 1310 |
+
else:
|
| 1311 |
+
try:
|
| 1312 |
+
backend_factory = BACKENDS[backend]
|
| 1313 |
+
except KeyError as e:
|
| 1314 |
+
raise ValueError(
|
| 1315 |
+
"Invalid backend: %s, expected one of %r"
|
| 1316 |
+
% (backend, sorted(BACKENDS.keys()))
|
| 1317 |
+
) from e
|
| 1318 |
+
backend = backend_factory(nesting_level=nesting_level)
|
| 1319 |
+
|
| 1320 |
+
n_jobs = _get_config_param(n_jobs, context_config, "n_jobs")
|
| 1321 |
+
if n_jobs is None:
|
| 1322 |
+
# No specific context override and no specific value request:
|
| 1323 |
+
# default to the default of the backend.
|
| 1324 |
+
n_jobs = backend.default_n_jobs
|
| 1325 |
+
try:
|
| 1326 |
+
n_jobs = int(n_jobs)
|
| 1327 |
+
except ValueError:
|
| 1328 |
+
raise ValueError("n_jobs could not be converted to int")
|
| 1329 |
+
self.n_jobs = n_jobs
|
| 1330 |
+
|
| 1331 |
+
if require == "sharedmem" and not getattr(backend, "supports_sharedmem", False):
|
| 1332 |
+
raise ValueError("Backend %s does not support shared memory" % backend)
|
| 1333 |
+
|
| 1334 |
+
if batch_size == "auto" or isinstance(batch_size, Integral) and batch_size > 0:
|
| 1335 |
+
self.batch_size = batch_size
|
| 1336 |
+
else:
|
| 1337 |
+
raise ValueError(
|
| 1338 |
+
"batch_size must be 'auto' or a positive integer, got: %r" % batch_size
|
| 1339 |
+
)
|
| 1340 |
+
|
| 1341 |
+
if not isinstance(backend, SequentialBackend):
|
| 1342 |
+
if self.return_generator and not backend.supports_return_generator:
|
| 1343 |
+
raise ValueError(
|
| 1344 |
+
"Backend {} does not support return_as={}".format(
|
| 1345 |
+
backend, return_as
|
| 1346 |
+
)
|
| 1347 |
+
)
|
| 1348 |
+
# This lock is used to coordinate the main thread of this process
|
| 1349 |
+
# with the async callback thread of our the pool.
|
| 1350 |
+
self._lock = threading.RLock()
|
| 1351 |
+
self._jobs = collections.deque()
|
| 1352 |
+
self._jobs_set = set()
|
| 1353 |
+
self._pending_outputs = list()
|
| 1354 |
+
self._ready_batches = queue.Queue()
|
| 1355 |
+
self._reducer_callback = None
|
| 1356 |
+
|
| 1357 |
+
# Internal variables
|
| 1358 |
+
self._backend = backend
|
| 1359 |
+
self._running = False
|
| 1360 |
+
self._managed_backend = False
|
| 1361 |
+
self._id = uuid4().hex
|
| 1362 |
+
self._call_ref = None
|
| 1363 |
+
|
| 1364 |
+
def __enter__(self):
|
| 1365 |
+
self._managed_backend = True
|
| 1366 |
+
self._calling = False
|
| 1367 |
+
self._initialize_backend()
|
| 1368 |
+
return self
|
| 1369 |
+
|
| 1370 |
+
def __exit__(self, exc_type, exc_value, traceback):
|
| 1371 |
+
self._managed_backend = False
|
| 1372 |
+
if self.return_generator and self._calling:
|
| 1373 |
+
self._abort()
|
| 1374 |
+
self._terminate_and_reset()
|
| 1375 |
+
|
| 1376 |
+
def _initialize_backend(self):
|
| 1377 |
+
"""Build a process or thread pool and return the number of workers"""
|
| 1378 |
+
try:
|
| 1379 |
+
n_jobs = self._backend.configure(
|
| 1380 |
+
n_jobs=self.n_jobs, parallel=self, **self._backend_kwargs
|
| 1381 |
+
)
|
| 1382 |
+
if self.timeout is not None and not self._backend.supports_timeout:
|
| 1383 |
+
warnings.warn(
|
| 1384 |
+
"The backend class {!r} does not support timeout. "
|
| 1385 |
+
"You have set 'timeout={}' in Parallel but "
|
| 1386 |
+
"the 'timeout' parameter will not be used.".format(
|
| 1387 |
+
self._backend.__class__.__name__, self.timeout
|
| 1388 |
+
)
|
| 1389 |
+
)
|
| 1390 |
+
|
| 1391 |
+
except FallbackToBackend as e:
|
| 1392 |
+
# Recursively initialize the backend in case of requested fallback.
|
| 1393 |
+
self._backend = e.backend
|
| 1394 |
+
n_jobs = self._initialize_backend()
|
| 1395 |
+
|
| 1396 |
+
return n_jobs
|
| 1397 |
+
|
| 1398 |
+
def _effective_n_jobs(self):
|
| 1399 |
+
if self._backend:
|
| 1400 |
+
return self._backend.effective_n_jobs(self.n_jobs)
|
| 1401 |
+
return 1
|
| 1402 |
+
|
| 1403 |
+
def _terminate_and_reset(self):
|
| 1404 |
+
if hasattr(self._backend, "stop_call") and self._calling:
|
| 1405 |
+
self._backend.stop_call()
|
| 1406 |
+
self._calling = False
|
| 1407 |
+
if not self._managed_backend:
|
| 1408 |
+
self._backend.terminate()
|
| 1409 |
+
|
| 1410 |
+
def _dispatch(self, batch):
|
| 1411 |
+
"""Queue the batch for computing, with or without multiprocessing
|
| 1412 |
+
|
| 1413 |
+
WARNING: this method is not thread-safe: it should be only called
|
| 1414 |
+
indirectly via dispatch_one_batch.
|
| 1415 |
+
|
| 1416 |
+
"""
|
| 1417 |
+
# If job.get() catches an exception, it closes the queue:
|
| 1418 |
+
if self._aborting:
|
| 1419 |
+
return
|
| 1420 |
+
|
| 1421 |
+
batch_size = len(batch)
|
| 1422 |
+
|
| 1423 |
+
self.n_dispatched_tasks += batch_size
|
| 1424 |
+
self.n_dispatched_batches += 1
|
| 1425 |
+
|
| 1426 |
+
dispatch_timestamp = time.time()
|
| 1427 |
+
|
| 1428 |
+
batch_tracker = BatchCompletionCallBack(dispatch_timestamp, batch_size, self)
|
| 1429 |
+
|
| 1430 |
+
self._register_new_job(batch_tracker)
|
| 1431 |
+
|
| 1432 |
+
# If return_ordered is False, the batch_tracker is not stored in the
|
| 1433 |
+
# jobs queue at the time of submission. Instead, it will be appended to
|
| 1434 |
+
# the queue by itself as soon as the callback is triggered to be able
|
| 1435 |
+
# to return the results in the order of completion.
|
| 1436 |
+
|
| 1437 |
+
job = self._backend.submit(batch, callback=batch_tracker)
|
| 1438 |
+
batch_tracker.register_job(job)
|
| 1439 |
+
|
| 1440 |
+
def _register_new_job(self, batch_tracker):
|
| 1441 |
+
if self.return_ordered:
|
| 1442 |
+
self._jobs.append(batch_tracker)
|
| 1443 |
+
else:
|
| 1444 |
+
self._jobs_set.add(batch_tracker)
|
| 1445 |
+
|
| 1446 |
+
def dispatch_next(self):
|
| 1447 |
+
"""Dispatch more data for parallel processing
|
| 1448 |
+
|
| 1449 |
+
This method is meant to be called concurrently by the multiprocessing
|
| 1450 |
+
callback. We rely on the thread-safety of dispatch_one_batch to protect
|
| 1451 |
+
against concurrent consumption of the unprotected iterator.
|
| 1452 |
+
"""
|
| 1453 |
+
if not self.dispatch_one_batch(self._original_iterator):
|
| 1454 |
+
self._iterating = False
|
| 1455 |
+
self._original_iterator = None
|
| 1456 |
+
|
| 1457 |
+
def dispatch_one_batch(self, iterator):
|
| 1458 |
+
"""Prefetch the tasks for the next batch and dispatch them.
|
| 1459 |
+
|
| 1460 |
+
The effective size of the batch is computed here.
|
| 1461 |
+
If there are no more jobs to dispatch, return False, else return True.
|
| 1462 |
+
|
| 1463 |
+
The iterator consumption and dispatching is protected by the same
|
| 1464 |
+
lock so calling this function should be thread safe.
|
| 1465 |
+
|
| 1466 |
+
"""
|
| 1467 |
+
|
| 1468 |
+
if self._aborting:
|
| 1469 |
+
return False
|
| 1470 |
+
|
| 1471 |
+
batch_size = self._get_batch_size()
|
| 1472 |
+
|
| 1473 |
+
with self._lock:
|
| 1474 |
+
# to ensure an even distribution of the workload between workers,
|
| 1475 |
+
# we look ahead in the original iterators more than batch_size
|
| 1476 |
+
# tasks - However, we keep consuming only one batch at each
|
| 1477 |
+
# dispatch_one_batch call. The extra tasks are stored in a local
|
| 1478 |
+
# queue, _ready_batches, that is looked-up prior to re-consuming
|
| 1479 |
+
# tasks from the origal iterator.
|
| 1480 |
+
try:
|
| 1481 |
+
tasks = self._ready_batches.get(block=False)
|
| 1482 |
+
except queue.Empty:
|
| 1483 |
+
# slice the iterator n_jobs * batchsize items at a time. If the
|
| 1484 |
+
# slice returns less than that, then the current batchsize puts
|
| 1485 |
+
# too much weight on a subset of workers, while other may end
|
| 1486 |
+
# up starving. So in this case, re-scale the batch size
|
| 1487 |
+
# accordingly to distribute evenly the last items between all
|
| 1488 |
+
# workers.
|
| 1489 |
+
n_jobs = self._cached_effective_n_jobs
|
| 1490 |
+
big_batch_size = batch_size * n_jobs
|
| 1491 |
+
|
| 1492 |
+
try:
|
| 1493 |
+
islice = list(itertools.islice(iterator, big_batch_size))
|
| 1494 |
+
except Exception as e:
|
| 1495 |
+
# Handle the fact that the generator of task raised an
|
| 1496 |
+
# exception. As this part of the code can be executed in
|
| 1497 |
+
# a thread internal to the backend, register a task with
|
| 1498 |
+
# an error that will be raised in the user's thread.
|
| 1499 |
+
if isinstance(e.__context__, queue.Empty):
|
| 1500 |
+
# Suppress the cause of the exception if it is
|
| 1501 |
+
# queue.Empty to avoid cluttered traceback. Only do it
|
| 1502 |
+
# if the __context__ is really empty to avoid messing
|
| 1503 |
+
# with causes of the original error.
|
| 1504 |
+
e.__cause__ = None
|
| 1505 |
+
batch_tracker = BatchCompletionCallBack(0, batch_size, self)
|
| 1506 |
+
self._register_new_job(batch_tracker)
|
| 1507 |
+
batch_tracker._register_outcome(dict(result=e, status=TASK_ERROR))
|
| 1508 |
+
return True
|
| 1509 |
+
|
| 1510 |
+
if len(islice) == 0:
|
| 1511 |
+
return False
|
| 1512 |
+
elif (
|
| 1513 |
+
iterator is self._original_iterator and len(islice) < big_batch_size
|
| 1514 |
+
):
|
| 1515 |
+
# We reached the end of the original iterator (unless
|
| 1516 |
+
# iterator is the ``pre_dispatch``-long initial slice of
|
| 1517 |
+
# the original iterator) -- decrease the batch size to
|
| 1518 |
+
# account for potential variance in the batches running
|
| 1519 |
+
# time.
|
| 1520 |
+
final_batch_size = max(1, len(islice) // (10 * n_jobs))
|
| 1521 |
+
else:
|
| 1522 |
+
final_batch_size = max(1, len(islice) // n_jobs)
|
| 1523 |
+
|
| 1524 |
+
# enqueue n_jobs batches in a local queue
|
| 1525 |
+
for i in range(0, len(islice), final_batch_size):
|
| 1526 |
+
tasks = BatchedCalls(
|
| 1527 |
+
islice[i : i + final_batch_size],
|
| 1528 |
+
self._backend.get_nested_backend(),
|
| 1529 |
+
self._reducer_callback,
|
| 1530 |
+
self._pickle_cache,
|
| 1531 |
+
)
|
| 1532 |
+
self._ready_batches.put(tasks)
|
| 1533 |
+
|
| 1534 |
+
# finally, get one task.
|
| 1535 |
+
tasks = self._ready_batches.get(block=False)
|
| 1536 |
+
if len(tasks) == 0:
|
| 1537 |
+
# No more tasks available in the iterator: tell caller to stop.
|
| 1538 |
+
return False
|
| 1539 |
+
else:
|
| 1540 |
+
self._dispatch(tasks)
|
| 1541 |
+
return True
|
| 1542 |
+
|
| 1543 |
+
def _get_batch_size(self):
|
| 1544 |
+
"""Returns the effective batch size for dispatch"""
|
| 1545 |
+
if self.batch_size == "auto":
|
| 1546 |
+
return self._backend.compute_batch_size()
|
| 1547 |
+
else:
|
| 1548 |
+
# Fixed batch size strategy
|
| 1549 |
+
return self.batch_size
|
| 1550 |
+
|
| 1551 |
+
def _print(self, msg):
|
| 1552 |
+
"""Display the message on stout or stderr depending on verbosity"""
|
| 1553 |
+
# XXX: Not using the logger framework: need to
|
| 1554 |
+
# learn to use logger better.
|
| 1555 |
+
if not self.verbose:
|
| 1556 |
+
return
|
| 1557 |
+
if self.verbose < 50:
|
| 1558 |
+
writer = sys.stderr.write
|
| 1559 |
+
else:
|
| 1560 |
+
writer = sys.stdout.write
|
| 1561 |
+
writer(f"[{self}]: {msg}\n")
|
| 1562 |
+
|
| 1563 |
+
def _is_completed(self):
|
| 1564 |
+
"""Check if all tasks have been completed"""
|
| 1565 |
+
return self.n_completed_tasks == self.n_dispatched_tasks and not (
|
| 1566 |
+
self._iterating or self._aborting
|
| 1567 |
+
)
|
| 1568 |
+
|
| 1569 |
+
def print_progress(self):
|
| 1570 |
+
"""Display the process of the parallel execution only a fraction
|
| 1571 |
+
of time, controlled by self.verbose.
|
| 1572 |
+
"""
|
| 1573 |
+
|
| 1574 |
+
if not self.verbose:
|
| 1575 |
+
return
|
| 1576 |
+
|
| 1577 |
+
if self.n_tasks is not None and self.n_tasks > 0:
|
| 1578 |
+
width = floor(log10(self.n_tasks)) + 1
|
| 1579 |
+
else:
|
| 1580 |
+
width = 3
|
| 1581 |
+
elapsed_time = time.time() - self._start_time
|
| 1582 |
+
|
| 1583 |
+
if self._is_completed():
|
| 1584 |
+
# Make sure that we get a last message telling us we are done
|
| 1585 |
+
self._print(
|
| 1586 |
+
f"Done {self.n_completed_tasks:{width}d} out of "
|
| 1587 |
+
f"{self.n_completed_tasks:{width}d} | elapsed: "
|
| 1588 |
+
f"{short_format_time(elapsed_time)} finished"
|
| 1589 |
+
)
|
| 1590 |
+
return
|
| 1591 |
+
|
| 1592 |
+
# Original job iterator becomes None once it has been fully
|
| 1593 |
+
# consumed: at this point we know the total number of jobs and we are
|
| 1594 |
+
# able to display an estimation of the remaining time based on already
|
| 1595 |
+
# completed jobs. Otherwise, we simply display the number of completed
|
| 1596 |
+
# tasks.
|
| 1597 |
+
elif self._original_iterator is not None:
|
| 1598 |
+
if _verbosity_filter(self.n_dispatched_batches, self.verbose):
|
| 1599 |
+
return
|
| 1600 |
+
fmt_time = f"| elapsed: {short_format_time(elapsed_time)}"
|
| 1601 |
+
index = self.n_completed_tasks
|
| 1602 |
+
if self.n_tasks is not None:
|
| 1603 |
+
self._print(
|
| 1604 |
+
f"Done {index:{width}d} out of {self.n_tasks:{width}d} {fmt_time}"
|
| 1605 |
+
)
|
| 1606 |
+
else:
|
| 1607 |
+
pad = " " * (len("out of ") + width - len("tasks"))
|
| 1608 |
+
self._print(f"Done {index:{width}d} tasks {pad}{fmt_time}")
|
| 1609 |
+
else:
|
| 1610 |
+
index = self.n_completed_tasks
|
| 1611 |
+
# We are finished dispatching
|
| 1612 |
+
total_tasks = self.n_dispatched_tasks
|
| 1613 |
+
# We always display the first loop
|
| 1614 |
+
if index != 0:
|
| 1615 |
+
# Display depending on the number of remaining items
|
| 1616 |
+
# A message as soon as we finish dispatching, cursor is 0
|
| 1617 |
+
cursor = total_tasks - index + 1 - self._pre_dispatch_amount
|
| 1618 |
+
frequency = (total_tasks // self.verbose) + 1
|
| 1619 |
+
is_last_item = index + 1 == total_tasks
|
| 1620 |
+
if is_last_item or cursor % frequency:
|
| 1621 |
+
return
|
| 1622 |
+
remaining_time = (elapsed_time / max(index, 1)) * (
|
| 1623 |
+
self.n_dispatched_tasks - index
|
| 1624 |
+
)
|
| 1625 |
+
# only display status if remaining time is greater or equal to 0
|
| 1626 |
+
self._print(
|
| 1627 |
+
f"Done {index:{width}d} out of {total_tasks:{width}d} "
|
| 1628 |
+
f"| elapsed: {short_format_time(elapsed_time)} remaining: "
|
| 1629 |
+
f"{short_format_time(remaining_time)}"
|
| 1630 |
+
)
|
| 1631 |
+
|
| 1632 |
+
def _abort(self):
|
| 1633 |
+
# Stop dispatching new jobs in the async callback thread
|
| 1634 |
+
self._aborting = True
|
| 1635 |
+
|
| 1636 |
+
# If the backend allows it, cancel or kill remaining running
|
| 1637 |
+
# tasks without waiting for the results as we will raise
|
| 1638 |
+
# the exception we got back to the caller instead of returning
|
| 1639 |
+
# any result.
|
| 1640 |
+
backend = self._backend
|
| 1641 |
+
if not self._aborted and hasattr(backend, "abort_everything"):
|
| 1642 |
+
# If the backend is managed externally we need to make sure
|
| 1643 |
+
# to leave it in a working state to allow for future jobs
|
| 1644 |
+
# scheduling.
|
| 1645 |
+
ensure_ready = self._managed_backend
|
| 1646 |
+
backend.abort_everything(ensure_ready=ensure_ready)
|
| 1647 |
+
self._aborted = True
|
| 1648 |
+
|
| 1649 |
+
def _start(self, iterator, pre_dispatch):
|
| 1650 |
+
# Only set self._iterating to True if at least a batch
|
| 1651 |
+
# was dispatched. In particular this covers the edge
|
| 1652 |
+
# case of Parallel used with an exhausted iterator. If
|
| 1653 |
+
# self._original_iterator is None, then this means either
|
| 1654 |
+
# that pre_dispatch == "all", n_jobs == 1 or that the first batch
|
| 1655 |
+
# was very quick and its callback already dispatched all the
|
| 1656 |
+
# remaining jobs.
|
| 1657 |
+
self._iterating = False
|
| 1658 |
+
if self.dispatch_one_batch(iterator):
|
| 1659 |
+
self._iterating = self._original_iterator is not None
|
| 1660 |
+
|
| 1661 |
+
while self.dispatch_one_batch(iterator):
|
| 1662 |
+
pass
|
| 1663 |
+
|
| 1664 |
+
if pre_dispatch == "all":
|
| 1665 |
+
# The iterable was consumed all at once by the above for loop.
|
| 1666 |
+
# No need to wait for async callbacks to trigger to
|
| 1667 |
+
# consumption.
|
| 1668 |
+
self._iterating = False
|
| 1669 |
+
|
| 1670 |
+
def _get_outputs(self, iterator, pre_dispatch):
|
| 1671 |
+
"""Iterator returning the tasks' output as soon as they are ready."""
|
| 1672 |
+
dispatch_thread_id = threading.get_ident()
|
| 1673 |
+
detach_generator_exit = False
|
| 1674 |
+
try:
|
| 1675 |
+
self._start(iterator, pre_dispatch)
|
| 1676 |
+
# first yield returns None, for internal use only. This ensures
|
| 1677 |
+
# that we enter the try/except block and start dispatching the
|
| 1678 |
+
# tasks.
|
| 1679 |
+
yield
|
| 1680 |
+
|
| 1681 |
+
with self._backend.retrieval_context():
|
| 1682 |
+
yield from self._retrieve()
|
| 1683 |
+
|
| 1684 |
+
except GeneratorExit:
|
| 1685 |
+
# The generator has been garbage collected before being fully
|
| 1686 |
+
# consumed. This aborts the remaining tasks if possible and warn
|
| 1687 |
+
# the user if necessary.
|
| 1688 |
+
self._exception = True
|
| 1689 |
+
|
| 1690 |
+
# In some interpreters such as PyPy, GeneratorExit can be raised in
|
| 1691 |
+
# a different thread than the one used to start the dispatch of the
|
| 1692 |
+
# parallel tasks. This can lead to hang when a thread attempts to
|
| 1693 |
+
# join itself. As workaround, we detach the execution of the
|
| 1694 |
+
# aborting code to a dedicated thread. We then need to make sure
|
| 1695 |
+
# the rest of the function does not call `_terminate_and_reset`
|
| 1696 |
+
# in finally.
|
| 1697 |
+
if dispatch_thread_id != threading.get_ident():
|
| 1698 |
+
warnings.warn(
|
| 1699 |
+
"A generator produced by joblib.Parallel has been "
|
| 1700 |
+
"gc'ed in an unexpected thread. This behavior should "
|
| 1701 |
+
"not cause major -issues but to make sure, please "
|
| 1702 |
+
"report this warning and your use case at "
|
| 1703 |
+
"https://github.com/joblib/joblib/issues so it can "
|
| 1704 |
+
"be investigated."
|
| 1705 |
+
)
|
| 1706 |
+
|
| 1707 |
+
detach_generator_exit = True
|
| 1708 |
+
_parallel = self
|
| 1709 |
+
|
| 1710 |
+
class _GeneratorExitThread(threading.Thread):
|
| 1711 |
+
def run(self):
|
| 1712 |
+
_parallel._abort()
|
| 1713 |
+
if _parallel.return_generator:
|
| 1714 |
+
_parallel._warn_exit_early()
|
| 1715 |
+
_parallel._terminate_and_reset()
|
| 1716 |
+
|
| 1717 |
+
_GeneratorExitThread(name="GeneratorExitThread").start()
|
| 1718 |
+
return
|
| 1719 |
+
|
| 1720 |
+
# Otherwise, we are in the thread that started the dispatch: we can
|
| 1721 |
+
# safely abort the execution and warn the user.
|
| 1722 |
+
self._abort()
|
| 1723 |
+
if self.return_generator:
|
| 1724 |
+
self._warn_exit_early()
|
| 1725 |
+
|
| 1726 |
+
raise
|
| 1727 |
+
|
| 1728 |
+
# Note: we catch any BaseException instead of just Exception instances
|
| 1729 |
+
# to also include KeyboardInterrupt
|
| 1730 |
+
except BaseException:
|
| 1731 |
+
self._exception = True
|
| 1732 |
+
self._abort()
|
| 1733 |
+
raise
|
| 1734 |
+
finally:
|
| 1735 |
+
# Store the unconsumed tasks and terminate the workers if necessary
|
| 1736 |
+
_remaining_outputs = [] if self._exception else self._jobs
|
| 1737 |
+
self._jobs = collections.deque()
|
| 1738 |
+
self._jobs_set = set()
|
| 1739 |
+
self._running = False
|
| 1740 |
+
if not detach_generator_exit:
|
| 1741 |
+
self._terminate_and_reset()
|
| 1742 |
+
|
| 1743 |
+
while len(_remaining_outputs) > 0:
|
| 1744 |
+
batched_results = _remaining_outputs.popleft()
|
| 1745 |
+
batched_results = batched_results.get_result(self.timeout)
|
| 1746 |
+
for result in batched_results:
|
| 1747 |
+
yield result
|
| 1748 |
+
|
| 1749 |
+
def _wait_retrieval(self):
|
| 1750 |
+
"""Return True if we need to continue retrieving some tasks."""
|
| 1751 |
+
|
| 1752 |
+
# If the input load is still being iterated over, it means that tasks
|
| 1753 |
+
# are still on the dispatch waitlist and their results will need to
|
| 1754 |
+
# be retrieved later on.
|
| 1755 |
+
if self._iterating:
|
| 1756 |
+
return True
|
| 1757 |
+
|
| 1758 |
+
# If some of the dispatched tasks are still being processed by the
|
| 1759 |
+
# workers, wait for the compute to finish before starting retrieval
|
| 1760 |
+
if self.n_completed_tasks < self.n_dispatched_tasks:
|
| 1761 |
+
return True
|
| 1762 |
+
|
| 1763 |
+
# For backends that does not support retrieving asynchronously the
|
| 1764 |
+
# result to the main process, all results must be carefully retrieved
|
| 1765 |
+
# in the _retrieve loop in the main thread while the backend is alive.
|
| 1766 |
+
# For other backends, the actual retrieval is done asynchronously in
|
| 1767 |
+
# the callback thread, and we can terminate the backend before the
|
| 1768 |
+
# `self._jobs` result list has been emptied. The remaining results
|
| 1769 |
+
# will be collected in the `finally` step of the generator.
|
| 1770 |
+
if not self._backend.supports_retrieve_callback:
|
| 1771 |
+
if len(self._jobs) > 0:
|
| 1772 |
+
return True
|
| 1773 |
+
|
| 1774 |
+
return False
|
| 1775 |
+
|
| 1776 |
+
def _retrieve(self):
|
| 1777 |
+
timeout_control_job = None
|
| 1778 |
+
while self._wait_retrieval():
|
| 1779 |
+
# If the callback thread of a worker has signaled that its task
|
| 1780 |
+
# triggered an exception, or if the retrieval loop has raised an
|
| 1781 |
+
# exception (e.g. `GeneratorExit`), exit the loop and surface the
|
| 1782 |
+
# worker traceback.
|
| 1783 |
+
if self._aborting:
|
| 1784 |
+
self._raise_error_fast()
|
| 1785 |
+
break
|
| 1786 |
+
|
| 1787 |
+
nb_jobs = len(self._jobs)
|
| 1788 |
+
# Now wait for a job to be ready for retrieval.
|
| 1789 |
+
if self.return_ordered:
|
| 1790 |
+
# Case ordered: wait for completion (or error) of the next job
|
| 1791 |
+
# that have been dispatched and not retrieved yet. If no job
|
| 1792 |
+
# have been dispatched yet, wait for dispatch.
|
| 1793 |
+
# We assume that the time to wait for the next job to be
|
| 1794 |
+
# dispatched is always low, so that the timeout
|
| 1795 |
+
# control only have to be done on the amount of time the next
|
| 1796 |
+
# dispatched job is pending.
|
| 1797 |
+
if (nb_jobs == 0) or (
|
| 1798 |
+
self._jobs[0].get_status(timeout=self.timeout) == TASK_PENDING
|
| 1799 |
+
):
|
| 1800 |
+
time.sleep(0.01)
|
| 1801 |
+
continue
|
| 1802 |
+
|
| 1803 |
+
elif nb_jobs == 0:
|
| 1804 |
+
# Case unordered: jobs are added to the list of jobs to
|
| 1805 |
+
# retrieve `self._jobs` only once completed or in error, which
|
| 1806 |
+
# is too late to enable timeout control in the same way than in
|
| 1807 |
+
# the previous case.
|
| 1808 |
+
# Instead, if no job is ready to be retrieved yet, we
|
| 1809 |
+
# arbitrarily pick a dispatched job, and the timeout control is
|
| 1810 |
+
# done such that an error is raised if this control job
|
| 1811 |
+
# timeouts before any other dispatched job has completed and
|
| 1812 |
+
# been added to `self._jobs` to be retrieved.
|
| 1813 |
+
if timeout_control_job is None:
|
| 1814 |
+
timeout_control_job = next(iter(self._jobs_set), None)
|
| 1815 |
+
|
| 1816 |
+
# NB: it can be None if no job has been dispatched yet.
|
| 1817 |
+
if timeout_control_job is not None:
|
| 1818 |
+
timeout_control_job.get_status(timeout=self.timeout)
|
| 1819 |
+
|
| 1820 |
+
time.sleep(0.01)
|
| 1821 |
+
continue
|
| 1822 |
+
|
| 1823 |
+
elif timeout_control_job is not None:
|
| 1824 |
+
# Case unordered, when `nb_jobs > 0`:
|
| 1825 |
+
# It means that a job is ready to be retrieved, so no timeout
|
| 1826 |
+
# will occur during this iteration.
|
| 1827 |
+
# Before proceeding to retrieval of the next ready job, reset
|
| 1828 |
+
# the timeout control state to prepare the next iteration.
|
| 1829 |
+
timeout_control_job._completion_timeout_counter = None
|
| 1830 |
+
timeout_control_job = None
|
| 1831 |
+
|
| 1832 |
+
# We need to be careful: the job list can be filling up as
|
| 1833 |
+
# we empty it and Python list are not thread-safe by
|
| 1834 |
+
# default hence the use of the lock
|
| 1835 |
+
with self._lock:
|
| 1836 |
+
batched_results = self._jobs.popleft()
|
| 1837 |
+
if not self.return_ordered:
|
| 1838 |
+
self._jobs_set.remove(batched_results)
|
| 1839 |
+
|
| 1840 |
+
# Flatten the batched results to output one output at a time
|
| 1841 |
+
batched_results = batched_results.get_result(self.timeout)
|
| 1842 |
+
for result in batched_results:
|
| 1843 |
+
self._nb_consumed += 1
|
| 1844 |
+
yield result
|
| 1845 |
+
|
| 1846 |
+
def _raise_error_fast(self):
|
| 1847 |
+
"""If we are aborting, raise if a job caused an error."""
|
| 1848 |
+
|
| 1849 |
+
# Find the first job whose status is TASK_ERROR if it exists.
|
| 1850 |
+
with self._lock:
|
| 1851 |
+
error_job = next(
|
| 1852 |
+
(job for job in self._jobs if job.status == TASK_ERROR), None
|
| 1853 |
+
)
|
| 1854 |
+
|
| 1855 |
+
# If this error job exists, immediately raise the error by
|
| 1856 |
+
# calling get_result. This job might not exists if abort has been
|
| 1857 |
+
# called directly or if the generator is gc'ed.
|
| 1858 |
+
if error_job is not None:
|
| 1859 |
+
error_job.get_result(self.timeout)
|
| 1860 |
+
|
| 1861 |
+
def _warn_exit_early(self):
|
| 1862 |
+
"""Warn the user if the generator is gc'ed before being consumned."""
|
| 1863 |
+
ready_outputs = self.n_completed_tasks - self._nb_consumed
|
| 1864 |
+
is_completed = self._is_completed()
|
| 1865 |
+
msg = ""
|
| 1866 |
+
if ready_outputs:
|
| 1867 |
+
msg += (
|
| 1868 |
+
f"{ready_outputs} tasks have been successfully executed but not used."
|
| 1869 |
+
)
|
| 1870 |
+
if not is_completed:
|
| 1871 |
+
msg += " Additionally, "
|
| 1872 |
+
|
| 1873 |
+
if not is_completed:
|
| 1874 |
+
msg += (
|
| 1875 |
+
f"{self.n_dispatched_tasks - self.n_completed_tasks} tasks "
|
| 1876 |
+
"which were still being processed by the workers have been "
|
| 1877 |
+
"cancelled."
|
| 1878 |
+
)
|
| 1879 |
+
|
| 1880 |
+
if msg:
|
| 1881 |
+
msg += (
|
| 1882 |
+
" You could benefit from adjusting the input task "
|
| 1883 |
+
"iterator to limit unnecessary computation time."
|
| 1884 |
+
)
|
| 1885 |
+
|
| 1886 |
+
warnings.warn(msg)
|
| 1887 |
+
|
| 1888 |
+
def _get_sequential_output(self, iterable):
|
| 1889 |
+
"""Separate loop for sequential output.
|
| 1890 |
+
|
| 1891 |
+
This simplifies the traceback in case of errors and reduces the
|
| 1892 |
+
overhead of calling sequential tasks with `joblib`.
|
| 1893 |
+
"""
|
| 1894 |
+
try:
|
| 1895 |
+
self._iterating = True
|
| 1896 |
+
self._original_iterator = iterable
|
| 1897 |
+
batch_size = self._get_batch_size()
|
| 1898 |
+
|
| 1899 |
+
if batch_size != 1:
|
| 1900 |
+
it = iter(iterable)
|
| 1901 |
+
iterable_batched = iter(
|
| 1902 |
+
lambda: tuple(itertools.islice(it, batch_size)), ()
|
| 1903 |
+
)
|
| 1904 |
+
iterable = (task for batch in iterable_batched for task in batch)
|
| 1905 |
+
|
| 1906 |
+
# first yield returns None, for internal use only. This ensures
|
| 1907 |
+
# that we enter the try/except block and setup the generator.
|
| 1908 |
+
yield None
|
| 1909 |
+
|
| 1910 |
+
# Sequentially call the tasks and yield the results.
|
| 1911 |
+
for func, args, kwargs in iterable:
|
| 1912 |
+
self.n_dispatched_batches += 1
|
| 1913 |
+
self.n_dispatched_tasks += 1
|
| 1914 |
+
res = func(*args, **kwargs)
|
| 1915 |
+
self.n_completed_tasks += 1
|
| 1916 |
+
self.print_progress()
|
| 1917 |
+
yield res
|
| 1918 |
+
self._nb_consumed += 1
|
| 1919 |
+
except BaseException:
|
| 1920 |
+
self._exception = True
|
| 1921 |
+
self._aborting = True
|
| 1922 |
+
self._aborted = True
|
| 1923 |
+
raise
|
| 1924 |
+
finally:
|
| 1925 |
+
self._running = False
|
| 1926 |
+
self._iterating = False
|
| 1927 |
+
self._original_iterator = None
|
| 1928 |
+
self.print_progress()
|
| 1929 |
+
|
| 1930 |
+
def _reset_run_tracking(self):
|
| 1931 |
+
"""Reset the counters and flags used to track the execution."""
|
| 1932 |
+
|
| 1933 |
+
# Makes sur the parallel instance was not previously running in a
|
| 1934 |
+
# thread-safe way.
|
| 1935 |
+
with getattr(self, "_lock", nullcontext()):
|
| 1936 |
+
if self._running:
|
| 1937 |
+
msg = "This Parallel instance is already running !"
|
| 1938 |
+
if self.return_generator is True:
|
| 1939 |
+
msg += (
|
| 1940 |
+
" Before submitting new tasks, you must wait for the "
|
| 1941 |
+
"completion of all the previous tasks, or clean all "
|
| 1942 |
+
"references to the output generator."
|
| 1943 |
+
)
|
| 1944 |
+
raise RuntimeError(msg)
|
| 1945 |
+
self._running = True
|
| 1946 |
+
|
| 1947 |
+
# Counter to keep track of the task dispatched and completed.
|
| 1948 |
+
self.n_dispatched_batches = 0
|
| 1949 |
+
self.n_dispatched_tasks = 0
|
| 1950 |
+
self.n_completed_tasks = 0
|
| 1951 |
+
|
| 1952 |
+
# Following count is incremented by one each time the user iterates
|
| 1953 |
+
# on the output generator, it is used to prepare an informative
|
| 1954 |
+
# warning message in case the generator is deleted before all the
|
| 1955 |
+
# dispatched tasks have been consumed.
|
| 1956 |
+
self._nb_consumed = 0
|
| 1957 |
+
|
| 1958 |
+
# Following flags are used to synchronize the threads in case one of
|
| 1959 |
+
# the tasks error-out to ensure that all workers abort fast and that
|
| 1960 |
+
# the backend terminates properly.
|
| 1961 |
+
|
| 1962 |
+
# Set to True as soon as a worker signals that a task errors-out
|
| 1963 |
+
self._exception = False
|
| 1964 |
+
# Set to True in case of early termination following an incident
|
| 1965 |
+
self._aborting = False
|
| 1966 |
+
# Set to True after abortion is complete
|
| 1967 |
+
self._aborted = False
|
| 1968 |
+
|
| 1969 |
+
def __call__(self, iterable):
|
| 1970 |
+
"""Main function to dispatch parallel tasks."""
|
| 1971 |
+
|
| 1972 |
+
self._reset_run_tracking()
|
| 1973 |
+
self.n_tasks = len(iterable) if hasattr(iterable, "__len__") else None
|
| 1974 |
+
self._start_time = time.time()
|
| 1975 |
+
|
| 1976 |
+
if not self._managed_backend:
|
| 1977 |
+
n_jobs = self._initialize_backend()
|
| 1978 |
+
else:
|
| 1979 |
+
n_jobs = self._effective_n_jobs()
|
| 1980 |
+
|
| 1981 |
+
if n_jobs == 1:
|
| 1982 |
+
# If n_jobs==1, run the computation sequentially and return
|
| 1983 |
+
# immediately to avoid overheads.
|
| 1984 |
+
output = self._get_sequential_output(iterable)
|
| 1985 |
+
next(output)
|
| 1986 |
+
return output if self.return_generator else list(output)
|
| 1987 |
+
|
| 1988 |
+
# Let's create an ID that uniquely identifies the current call. If the
|
| 1989 |
+
# call is interrupted early and that the same instance is immediately
|
| 1990 |
+
# reused, this id will be used to prevent workers that were
|
| 1991 |
+
# concurrently finalizing a task from the previous call to run the
|
| 1992 |
+
# callback.
|
| 1993 |
+
with self._lock:
|
| 1994 |
+
self._call_id = uuid4().hex
|
| 1995 |
+
|
| 1996 |
+
# self._effective_n_jobs should be called in the Parallel.__call__
|
| 1997 |
+
# thread only -- store its value in an attribute for further queries.
|
| 1998 |
+
self._cached_effective_n_jobs = n_jobs
|
| 1999 |
+
|
| 2000 |
+
if isinstance(self._backend, LokyBackend):
|
| 2001 |
+
# For the loky backend, we add a callback executed when reducing
|
| 2002 |
+
# BatchCalls, that makes the loky executor use a temporary folder
|
| 2003 |
+
# specific to this Parallel object when pickling temporary memmaps.
|
| 2004 |
+
# This callback is necessary to ensure that several Parallel
|
| 2005 |
+
# objects using the same reusable executor don't use the same
|
| 2006 |
+
# temporary resources.
|
| 2007 |
+
|
| 2008 |
+
def _batched_calls_reducer_callback():
|
| 2009 |
+
# Relevant implementation detail: the following lines, called
|
| 2010 |
+
# when reducing BatchedCalls, are called in a thread-safe
|
| 2011 |
+
# situation, meaning that the context of the temporary folder
|
| 2012 |
+
# manager will not be changed in between the callback execution
|
| 2013 |
+
# and the end of the BatchedCalls pickling. The reason is that
|
| 2014 |
+
# pickling (the only place where set_current_context is used)
|
| 2015 |
+
# is done from a single thread (the queue_feeder_thread).
|
| 2016 |
+
self._backend._workers._temp_folder_manager.set_current_context( # noqa
|
| 2017 |
+
self._id
|
| 2018 |
+
)
|
| 2019 |
+
|
| 2020 |
+
self._reducer_callback = _batched_calls_reducer_callback
|
| 2021 |
+
|
| 2022 |
+
# self._effective_n_jobs should be called in the Parallel.__call__
|
| 2023 |
+
# thread only -- store its value in an attribute for further queries.
|
| 2024 |
+
self._cached_effective_n_jobs = n_jobs
|
| 2025 |
+
|
| 2026 |
+
backend_name = self._backend.__class__.__name__
|
| 2027 |
+
if n_jobs == 0:
|
| 2028 |
+
raise RuntimeError("%s has no active worker." % backend_name)
|
| 2029 |
+
|
| 2030 |
+
self._print(f"Using backend {backend_name} with {n_jobs} concurrent workers.")
|
| 2031 |
+
if hasattr(self._backend, "start_call"):
|
| 2032 |
+
self._backend.start_call()
|
| 2033 |
+
|
| 2034 |
+
# Following flag prevents double calls to `backend.stop_call`.
|
| 2035 |
+
self._calling = True
|
| 2036 |
+
|
| 2037 |
+
iterator = iter(iterable)
|
| 2038 |
+
pre_dispatch = self.pre_dispatch
|
| 2039 |
+
|
| 2040 |
+
if pre_dispatch == "all":
|
| 2041 |
+
# prevent further dispatch via multiprocessing callback thread
|
| 2042 |
+
self._original_iterator = None
|
| 2043 |
+
self._pre_dispatch_amount = 0
|
| 2044 |
+
else:
|
| 2045 |
+
self._original_iterator = iterator
|
| 2046 |
+
if hasattr(pre_dispatch, "endswith"):
|
| 2047 |
+
pre_dispatch = eval_expr(pre_dispatch.replace("n_jobs", str(n_jobs)))
|
| 2048 |
+
self._pre_dispatch_amount = pre_dispatch = int(pre_dispatch)
|
| 2049 |
+
|
| 2050 |
+
# The main thread will consume the first pre_dispatch items and
|
| 2051 |
+
# the remaining items will later be lazily dispatched by async
|
| 2052 |
+
# callbacks upon task completions.
|
| 2053 |
+
|
| 2054 |
+
# TODO: this iterator should be batch_size * n_jobs
|
| 2055 |
+
iterator = itertools.islice(iterator, self._pre_dispatch_amount)
|
| 2056 |
+
|
| 2057 |
+
# Use a caching dict for callables that are pickled with cloudpickle to
|
| 2058 |
+
# improve performances. This cache is used only in the case of
|
| 2059 |
+
# functions that are defined in the __main__ module, functions that
|
| 2060 |
+
# are defined locally (inside another function) and lambda expressions.
|
| 2061 |
+
self._pickle_cache = dict()
|
| 2062 |
+
|
| 2063 |
+
output = self._get_outputs(iterator, pre_dispatch)
|
| 2064 |
+
self._call_ref = weakref.ref(output)
|
| 2065 |
+
|
| 2066 |
+
# The first item from the output is blank, but it makes the interpreter
|
| 2067 |
+
# progress until it enters the Try/Except block of the generator and
|
| 2068 |
+
# reaches the first `yield` statement. This starts the asynchronous
|
| 2069 |
+
# dispatch of the tasks to the workers.
|
| 2070 |
+
next(output)
|
| 2071 |
+
|
| 2072 |
+
return output if self.return_generator else list(output)
|
| 2073 |
+
|
| 2074 |
+
def __repr__(self):
|
| 2075 |
+
return "%s(n_jobs=%s)" % (self.__class__.__name__, self.n_jobs)
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/pool.py
ADDED
|
@@ -0,0 +1,362 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Custom implementation of multiprocessing.Pool with custom pickler.
|
| 2 |
+
|
| 3 |
+
This module provides efficient ways of working with data stored in
|
| 4 |
+
shared memory with numpy.memmap arrays without inducing any memory
|
| 5 |
+
copy between the parent and child processes.
|
| 6 |
+
|
| 7 |
+
This module should not be imported if multiprocessing is not
|
| 8 |
+
available as it implements subclasses of multiprocessing Pool
|
| 9 |
+
that uses a custom alternative to SimpleQueue.
|
| 10 |
+
|
| 11 |
+
"""
|
| 12 |
+
# Author: Olivier Grisel <olivier.grisel@ensta.org>
|
| 13 |
+
# Copyright: 2012, Olivier Grisel
|
| 14 |
+
# License: BSD 3 clause
|
| 15 |
+
|
| 16 |
+
import copyreg
|
| 17 |
+
import sys
|
| 18 |
+
import warnings
|
| 19 |
+
from time import sleep
|
| 20 |
+
|
| 21 |
+
try:
|
| 22 |
+
WindowsError
|
| 23 |
+
except NameError:
|
| 24 |
+
WindowsError = type(None)
|
| 25 |
+
|
| 26 |
+
from io import BytesIO
|
| 27 |
+
|
| 28 |
+
# We need the class definition to derive from it, not the multiprocessing.Pool
|
| 29 |
+
# factory function
|
| 30 |
+
from multiprocessing.pool import Pool
|
| 31 |
+
from pickle import HIGHEST_PROTOCOL, Pickler
|
| 32 |
+
|
| 33 |
+
from ._memmapping_reducer import TemporaryResourcesManager, get_memmapping_reducers
|
| 34 |
+
from ._multiprocessing_helpers import assert_spawning, mp
|
| 35 |
+
|
| 36 |
+
try:
|
| 37 |
+
import numpy as np
|
| 38 |
+
except ImportError:
|
| 39 |
+
np = None
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
###############################################################################
|
| 43 |
+
# Enable custom pickling in Pool queues
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
class CustomizablePickler(Pickler):
|
| 47 |
+
"""Pickler that accepts custom reducers.
|
| 48 |
+
|
| 49 |
+
TODO python2_drop : can this be simplified ?
|
| 50 |
+
|
| 51 |
+
HIGHEST_PROTOCOL is selected by default as this pickler is used
|
| 52 |
+
to pickle ephemeral datastructures for interprocess communication
|
| 53 |
+
hence no backward compatibility is required.
|
| 54 |
+
|
| 55 |
+
`reducers` is expected to be a dictionary with key/values
|
| 56 |
+
being `(type, callable)` pairs where `callable` is a function that
|
| 57 |
+
give an instance of `type` will return a tuple `(constructor,
|
| 58 |
+
tuple_of_objects)` to rebuild an instance out of the pickled
|
| 59 |
+
`tuple_of_objects` as would return a `__reduce__` method. See the
|
| 60 |
+
standard library documentation on pickling for more details.
|
| 61 |
+
|
| 62 |
+
"""
|
| 63 |
+
|
| 64 |
+
# We override the pure Python pickler as its the only way to be able to
|
| 65 |
+
# customize the dispatch table without side effects in Python 2.7
|
| 66 |
+
# to 3.2. For Python 3.3+ leverage the new dispatch_table
|
| 67 |
+
# feature from https://bugs.python.org/issue14166 that makes it possible
|
| 68 |
+
# to use the C implementation of the Pickler which is faster.
|
| 69 |
+
|
| 70 |
+
def __init__(self, writer, reducers=None, protocol=HIGHEST_PROTOCOL):
|
| 71 |
+
Pickler.__init__(self, writer, protocol=protocol)
|
| 72 |
+
if reducers is None:
|
| 73 |
+
reducers = {}
|
| 74 |
+
if hasattr(Pickler, "dispatch"):
|
| 75 |
+
# Make the dispatch registry an instance level attribute instead of
|
| 76 |
+
# a reference to the class dictionary under Python 2
|
| 77 |
+
self.dispatch = Pickler.dispatch.copy()
|
| 78 |
+
else:
|
| 79 |
+
# Under Python 3 initialize the dispatch table with a copy of the
|
| 80 |
+
# default registry
|
| 81 |
+
self.dispatch_table = copyreg.dispatch_table.copy()
|
| 82 |
+
for type, reduce_func in reducers.items():
|
| 83 |
+
self.register(type, reduce_func)
|
| 84 |
+
|
| 85 |
+
def register(self, type, reduce_func):
|
| 86 |
+
"""Attach a reducer function to a given type in the dispatch table."""
|
| 87 |
+
if hasattr(Pickler, "dispatch"):
|
| 88 |
+
# Python 2 pickler dispatching is not explicitly customizable.
|
| 89 |
+
# Let us use a closure to workaround this limitation.
|
| 90 |
+
def dispatcher(self, obj):
|
| 91 |
+
reduced = reduce_func(obj)
|
| 92 |
+
self.save_reduce(obj=obj, *reduced)
|
| 93 |
+
|
| 94 |
+
self.dispatch[type] = dispatcher
|
| 95 |
+
else:
|
| 96 |
+
self.dispatch_table[type] = reduce_func
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
class CustomizablePicklingQueue(object):
|
| 100 |
+
"""Locked Pipe implementation that uses a customizable pickler.
|
| 101 |
+
|
| 102 |
+
This class is an alternative to the multiprocessing implementation
|
| 103 |
+
of SimpleQueue in order to make it possible to pass custom
|
| 104 |
+
pickling reducers, for instance to avoid memory copy when passing
|
| 105 |
+
memory mapped datastructures.
|
| 106 |
+
|
| 107 |
+
`reducers` is expected to be a dict with key / values being
|
| 108 |
+
`(type, callable)` pairs where `callable` is a function that, given an
|
| 109 |
+
instance of `type`, will return a tuple `(constructor, tuple_of_objects)`
|
| 110 |
+
to rebuild an instance out of the pickled `tuple_of_objects` as would
|
| 111 |
+
return a `__reduce__` method.
|
| 112 |
+
|
| 113 |
+
See the standard library documentation on pickling for more details.
|
| 114 |
+
"""
|
| 115 |
+
|
| 116 |
+
def __init__(self, context, reducers=None):
|
| 117 |
+
self._reducers = reducers
|
| 118 |
+
self._reader, self._writer = context.Pipe(duplex=False)
|
| 119 |
+
self._rlock = context.Lock()
|
| 120 |
+
if sys.platform == "win32":
|
| 121 |
+
self._wlock = None
|
| 122 |
+
else:
|
| 123 |
+
self._wlock = context.Lock()
|
| 124 |
+
self._make_methods()
|
| 125 |
+
|
| 126 |
+
def __getstate__(self):
|
| 127 |
+
assert_spawning(self)
|
| 128 |
+
return (self._reader, self._writer, self._rlock, self._wlock, self._reducers)
|
| 129 |
+
|
| 130 |
+
def __setstate__(self, state):
|
| 131 |
+
(self._reader, self._writer, self._rlock, self._wlock, self._reducers) = state
|
| 132 |
+
self._make_methods()
|
| 133 |
+
|
| 134 |
+
def empty(self):
|
| 135 |
+
return not self._reader.poll()
|
| 136 |
+
|
| 137 |
+
def _make_methods(self):
|
| 138 |
+
self._recv = recv = self._reader.recv
|
| 139 |
+
racquire, rrelease = self._rlock.acquire, self._rlock.release
|
| 140 |
+
|
| 141 |
+
def get():
|
| 142 |
+
racquire()
|
| 143 |
+
try:
|
| 144 |
+
return recv()
|
| 145 |
+
finally:
|
| 146 |
+
rrelease()
|
| 147 |
+
|
| 148 |
+
self.get = get
|
| 149 |
+
|
| 150 |
+
if self._reducers:
|
| 151 |
+
|
| 152 |
+
def send(obj):
|
| 153 |
+
buffer = BytesIO()
|
| 154 |
+
CustomizablePickler(buffer, self._reducers).dump(obj)
|
| 155 |
+
self._writer.send_bytes(buffer.getvalue())
|
| 156 |
+
|
| 157 |
+
self._send = send
|
| 158 |
+
else:
|
| 159 |
+
self._send = send = self._writer.send
|
| 160 |
+
if self._wlock is None:
|
| 161 |
+
# writes to a message oriented win32 pipe are atomic
|
| 162 |
+
self.put = send
|
| 163 |
+
else:
|
| 164 |
+
wlock_acquire, wlock_release = (self._wlock.acquire, self._wlock.release)
|
| 165 |
+
|
| 166 |
+
def put(obj):
|
| 167 |
+
wlock_acquire()
|
| 168 |
+
try:
|
| 169 |
+
return send(obj)
|
| 170 |
+
finally:
|
| 171 |
+
wlock_release()
|
| 172 |
+
|
| 173 |
+
self.put = put
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
class PicklingPool(Pool):
|
| 177 |
+
"""Pool implementation with customizable pickling reducers.
|
| 178 |
+
|
| 179 |
+
This is useful to control how data is shipped between processes
|
| 180 |
+
and makes it possible to use shared memory without useless
|
| 181 |
+
copies induces by the default pickling methods of the original
|
| 182 |
+
objects passed as arguments to dispatch.
|
| 183 |
+
|
| 184 |
+
`forward_reducers` and `backward_reducers` are expected to be
|
| 185 |
+
dictionaries with key/values being `(type, callable)` pairs where
|
| 186 |
+
`callable` is a function that, given an instance of `type`, will return a
|
| 187 |
+
tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the
|
| 188 |
+
pickled `tuple_of_objects` as would return a `__reduce__` method.
|
| 189 |
+
See the standard library documentation about pickling for more details.
|
| 190 |
+
|
| 191 |
+
"""
|
| 192 |
+
|
| 193 |
+
def __init__(
|
| 194 |
+
self, processes=None, forward_reducers=None, backward_reducers=None, **kwargs
|
| 195 |
+
):
|
| 196 |
+
if forward_reducers is None:
|
| 197 |
+
forward_reducers = dict()
|
| 198 |
+
if backward_reducers is None:
|
| 199 |
+
backward_reducers = dict()
|
| 200 |
+
self._forward_reducers = forward_reducers
|
| 201 |
+
self._backward_reducers = backward_reducers
|
| 202 |
+
poolargs = dict(processes=processes)
|
| 203 |
+
poolargs.update(kwargs)
|
| 204 |
+
super(PicklingPool, self).__init__(**poolargs)
|
| 205 |
+
|
| 206 |
+
def _setup_queues(self):
|
| 207 |
+
context = getattr(self, "_ctx", mp)
|
| 208 |
+
self._inqueue = CustomizablePicklingQueue(context, self._forward_reducers)
|
| 209 |
+
self._outqueue = CustomizablePicklingQueue(context, self._backward_reducers)
|
| 210 |
+
self._quick_put = self._inqueue._send
|
| 211 |
+
self._quick_get = self._outqueue._recv
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
class MemmappingPool(PicklingPool):
|
| 215 |
+
"""Process pool that shares large arrays to avoid memory copy.
|
| 216 |
+
|
| 217 |
+
This drop-in replacement for `multiprocessing.pool.Pool` makes
|
| 218 |
+
it possible to work efficiently with shared memory in a numpy
|
| 219 |
+
context.
|
| 220 |
+
|
| 221 |
+
Existing instances of numpy.memmap are preserved: the child
|
| 222 |
+
suprocesses will have access to the same shared memory in the
|
| 223 |
+
original mode except for the 'w+' mode that is automatically
|
| 224 |
+
transformed as 'r+' to avoid zeroing the original data upon
|
| 225 |
+
instantiation.
|
| 226 |
+
|
| 227 |
+
Furthermore large arrays from the parent process are automatically
|
| 228 |
+
dumped to a temporary folder on the filesystem such as child
|
| 229 |
+
processes to access their content via memmapping (file system
|
| 230 |
+
backed shared memory).
|
| 231 |
+
|
| 232 |
+
Note: it is important to call the terminate method to collect
|
| 233 |
+
the temporary folder used by the pool.
|
| 234 |
+
|
| 235 |
+
Parameters
|
| 236 |
+
----------
|
| 237 |
+
processes: int, optional
|
| 238 |
+
Number of worker processes running concurrently in the pool.
|
| 239 |
+
initializer: callable, optional
|
| 240 |
+
Callable executed on worker process creation.
|
| 241 |
+
initargs: tuple, optional
|
| 242 |
+
Arguments passed to the initializer callable.
|
| 243 |
+
temp_folder: (str, callable) optional
|
| 244 |
+
If str:
|
| 245 |
+
Folder to be used by the pool for memmapping large arrays
|
| 246 |
+
for sharing memory with worker processes. If None, this will try in
|
| 247 |
+
order:
|
| 248 |
+
- a folder pointed by the JOBLIB_TEMP_FOLDER environment variable,
|
| 249 |
+
- /dev/shm if the folder exists and is writable: this is a RAMdisk
|
| 250 |
+
filesystem available by default on modern Linux distributions,
|
| 251 |
+
- the default system temporary folder that can be overridden
|
| 252 |
+
with TMP, TMPDIR or TEMP environment variables, typically /tmp
|
| 253 |
+
under Unix operating systems.
|
| 254 |
+
if callable:
|
| 255 |
+
An callable in charge of dynamically resolving a temporary folder
|
| 256 |
+
for memmapping large arrays.
|
| 257 |
+
max_nbytes int or None, optional, 1e6 by default
|
| 258 |
+
Threshold on the size of arrays passed to the workers that
|
| 259 |
+
triggers automated memory mapping in temp_folder.
|
| 260 |
+
Use None to disable memmapping of large arrays.
|
| 261 |
+
mmap_mode: {'r+', 'r', 'w+', 'c'}
|
| 262 |
+
Memmapping mode for numpy arrays passed to workers.
|
| 263 |
+
See 'max_nbytes' parameter documentation for more details.
|
| 264 |
+
forward_reducers: dictionary, optional
|
| 265 |
+
Reducers used to pickle objects passed from main process to worker
|
| 266 |
+
processes: see below.
|
| 267 |
+
backward_reducers: dictionary, optional
|
| 268 |
+
Reducers used to pickle return values from workers back to the
|
| 269 |
+
main process.
|
| 270 |
+
verbose: int, optional
|
| 271 |
+
Make it possible to monitor how the communication of numpy arrays
|
| 272 |
+
with the subprocess is handled (pickling or memmapping)
|
| 273 |
+
prewarm: bool or str, optional, "auto" by default.
|
| 274 |
+
If True, force a read on newly memmapped array to make sure that OS
|
| 275 |
+
pre-cache it in memory. This can be useful to avoid concurrent disk
|
| 276 |
+
access when the same data array is passed to different worker
|
| 277 |
+
processes. If "auto" (by default), prewarm is set to True, unless the
|
| 278 |
+
Linux shared memory partition /dev/shm is available and used as temp
|
| 279 |
+
folder.
|
| 280 |
+
|
| 281 |
+
`forward_reducers` and `backward_reducers` are expected to be
|
| 282 |
+
dictionaries with key/values being `(type, callable)` pairs where
|
| 283 |
+
`callable` is a function that give an instance of `type` will return
|
| 284 |
+
a tuple `(constructor, tuple_of_objects)` to rebuild an instance out
|
| 285 |
+
of the pickled `tuple_of_objects` as would return a `__reduce__`
|
| 286 |
+
method. See the standard library documentation on pickling for more
|
| 287 |
+
details.
|
| 288 |
+
|
| 289 |
+
"""
|
| 290 |
+
|
| 291 |
+
def __init__(
|
| 292 |
+
self,
|
| 293 |
+
processes=None,
|
| 294 |
+
temp_folder=None,
|
| 295 |
+
max_nbytes=1e6,
|
| 296 |
+
mmap_mode="r",
|
| 297 |
+
forward_reducers=None,
|
| 298 |
+
backward_reducers=None,
|
| 299 |
+
verbose=0,
|
| 300 |
+
prewarm=False,
|
| 301 |
+
**kwargs,
|
| 302 |
+
):
|
| 303 |
+
manager = TemporaryResourcesManager(temp_folder)
|
| 304 |
+
self._temp_folder_manager = manager
|
| 305 |
+
|
| 306 |
+
# The usage of a temp_folder_resolver over a simple temp_folder is
|
| 307 |
+
# superfluous for multiprocessing pools, as they don't get reused, see
|
| 308 |
+
# get_memmapping_executor for more details. We still use it for code
|
| 309 |
+
# simplicity.
|
| 310 |
+
forward_reducers, backward_reducers = get_memmapping_reducers(
|
| 311 |
+
temp_folder_resolver=manager.resolve_temp_folder_name,
|
| 312 |
+
max_nbytes=max_nbytes,
|
| 313 |
+
mmap_mode=mmap_mode,
|
| 314 |
+
forward_reducers=forward_reducers,
|
| 315 |
+
backward_reducers=backward_reducers,
|
| 316 |
+
verbose=verbose,
|
| 317 |
+
unlink_on_gc_collect=False,
|
| 318 |
+
prewarm=prewarm,
|
| 319 |
+
)
|
| 320 |
+
|
| 321 |
+
poolargs = dict(
|
| 322 |
+
processes=processes,
|
| 323 |
+
forward_reducers=forward_reducers,
|
| 324 |
+
backward_reducers=backward_reducers,
|
| 325 |
+
)
|
| 326 |
+
poolargs.update(kwargs)
|
| 327 |
+
super(MemmappingPool, self).__init__(**poolargs)
|
| 328 |
+
|
| 329 |
+
def terminate(self):
|
| 330 |
+
n_retries = 10
|
| 331 |
+
for i in range(n_retries):
|
| 332 |
+
try:
|
| 333 |
+
super(MemmappingPool, self).terminate()
|
| 334 |
+
break
|
| 335 |
+
except OSError as e:
|
| 336 |
+
if isinstance(e, WindowsError):
|
| 337 |
+
# Workaround occasional "[Error 5] Access is denied" issue
|
| 338 |
+
# when trying to terminate a process under windows.
|
| 339 |
+
sleep(0.1)
|
| 340 |
+
if i + 1 == n_retries:
|
| 341 |
+
warnings.warn(
|
| 342 |
+
"Failed to terminate worker processes in"
|
| 343 |
+
" multiprocessing pool: %r" % e
|
| 344 |
+
)
|
| 345 |
+
|
| 346 |
+
# Clean up the temporary resources as the workers should now be off.
|
| 347 |
+
self._temp_folder_manager._clean_temporary_resources()
|
| 348 |
+
|
| 349 |
+
@property
|
| 350 |
+
def _temp_folder(self):
|
| 351 |
+
# Legacy property in tests. could be removed if we refactored the
|
| 352 |
+
# memmapping tests. SHOULD ONLY BE USED IN TESTS!
|
| 353 |
+
# We cache this property because it is called late in the tests - at
|
| 354 |
+
# this point, all context have been unregistered, and
|
| 355 |
+
# resolve_temp_folder_name raises an error.
|
| 356 |
+
if getattr(self, "_cached_temp_folder", None) is not None:
|
| 357 |
+
return self._cached_temp_folder
|
| 358 |
+
else:
|
| 359 |
+
self._cached_temp_folder = (
|
| 360 |
+
self._temp_folder_manager.resolve_temp_folder_name()
|
| 361 |
+
) # noqa
|
| 362 |
+
return self._cached_temp_folder
|
TTS/Kokoro/.venv/Lib/site-packages/joblib/testing.py
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Helper for testing.
|
| 3 |
+
"""
|
| 4 |
+
|
| 5 |
+
import os.path
|
| 6 |
+
import re
|
| 7 |
+
import subprocess
|
| 8 |
+
import sys
|
| 9 |
+
import threading
|
| 10 |
+
import warnings
|
| 11 |
+
|
| 12 |
+
import _pytest
|
| 13 |
+
import pytest
|
| 14 |
+
|
| 15 |
+
raises = pytest.raises
|
| 16 |
+
warns = pytest.warns
|
| 17 |
+
SkipTest = _pytest.runner.Skipped
|
| 18 |
+
skipif = pytest.mark.skipif
|
| 19 |
+
fixture = pytest.fixture
|
| 20 |
+
parametrize = pytest.mark.parametrize
|
| 21 |
+
timeout = pytest.mark.timeout
|
| 22 |
+
xfail = pytest.mark.xfail
|
| 23 |
+
param = pytest.param
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def warnings_to_stdout():
|
| 27 |
+
"""Redirect all warnings to stdout."""
|
| 28 |
+
showwarning_orig = warnings.showwarning
|
| 29 |
+
|
| 30 |
+
def showwarning(msg, cat, fname, lno, file=None, line=0):
|
| 31 |
+
showwarning_orig(msg, cat, os.path.basename(fname), line, sys.stdout)
|
| 32 |
+
|
| 33 |
+
warnings.showwarning = showwarning
|
| 34 |
+
# warnings.simplefilter('always')
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def check_subprocess_call(cmd, timeout=5, stdout_regex=None, stderr_regex=None):
|
| 38 |
+
"""Runs a command in a subprocess with timeout in seconds.
|
| 39 |
+
|
| 40 |
+
A SIGTERM is sent after `timeout` and if it does not terminate, a
|
| 41 |
+
SIGKILL is sent after `2 * timeout`.
|
| 42 |
+
|
| 43 |
+
Also checks returncode is zero, stdout if stdout_regex is set, and
|
| 44 |
+
stderr if stderr_regex is set.
|
| 45 |
+
"""
|
| 46 |
+
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
| 47 |
+
|
| 48 |
+
def terminate_process(): # pragma: no cover
|
| 49 |
+
"""
|
| 50 |
+
Attempt to terminate a leftover process spawned during test execution:
|
| 51 |
+
ideally this should not be needed but can help avoid clogging the CI
|
| 52 |
+
workers in case of deadlocks.
|
| 53 |
+
"""
|
| 54 |
+
warnings.warn(f"Timeout running {cmd}")
|
| 55 |
+
proc.terminate()
|
| 56 |
+
|
| 57 |
+
def kill_process(): # pragma: no cover
|
| 58 |
+
"""
|
| 59 |
+
Kill a leftover process spawned during test execution: ideally this
|
| 60 |
+
should not be needed but can help avoid clogging the CI workers in
|
| 61 |
+
case of deadlocks.
|
| 62 |
+
"""
|
| 63 |
+
warnings.warn(f"Timeout running {cmd}")
|
| 64 |
+
proc.kill()
|
| 65 |
+
|
| 66 |
+
try:
|
| 67 |
+
if timeout is not None:
|
| 68 |
+
terminate_timer = threading.Timer(timeout, terminate_process)
|
| 69 |
+
terminate_timer.start()
|
| 70 |
+
kill_timer = threading.Timer(2 * timeout, kill_process)
|
| 71 |
+
kill_timer.start()
|
| 72 |
+
stdout, stderr = proc.communicate()
|
| 73 |
+
stdout, stderr = stdout.decode(), stderr.decode()
|
| 74 |
+
if proc.returncode != 0:
|
| 75 |
+
message = ("Non-zero return code: {}.\nStdout:\n{}\nStderr:\n{}").format(
|
| 76 |
+
proc.returncode, stdout, stderr
|
| 77 |
+
)
|
| 78 |
+
raise ValueError(message)
|
| 79 |
+
|
| 80 |
+
if stdout_regex is not None and not re.search(stdout_regex, stdout):
|
| 81 |
+
raise ValueError(
|
| 82 |
+
"Unexpected stdout: {!r} does not match:\n{!r}".format(
|
| 83 |
+
stdout_regex, stdout
|
| 84 |
+
)
|
| 85 |
+
)
|
| 86 |
+
if stderr_regex is not None and not re.search(stderr_regex, stderr):
|
| 87 |
+
raise ValueError(
|
| 88 |
+
"Unexpected stderr: {!r} does not match:\n{!r}".format(
|
| 89 |
+
stderr_regex, stderr
|
| 90 |
+
)
|
| 91 |
+
)
|
| 92 |
+
|
| 93 |
+
finally:
|
| 94 |
+
if timeout is not None:
|
| 95 |
+
terminate_timer.cancel()
|
| 96 |
+
kill_timer.cancel()
|
TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/INSTALLER
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
pip
|
TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/METADATA
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Metadata-Version: 2.4
|
| 2 |
+
Name: jsonschema
|
| 3 |
+
Version: 4.26.0
|
| 4 |
+
Summary: An implementation of JSON Schema validation for Python
|
| 5 |
+
Project-URL: Homepage, https://github.com/python-jsonschema/jsonschema
|
| 6 |
+
Project-URL: Documentation, https://python-jsonschema.readthedocs.io/
|
| 7 |
+
Project-URL: Issues, https://github.com/python-jsonschema/jsonschema/issues/
|
| 8 |
+
Project-URL: Funding, https://github.com/sponsors/Julian
|
| 9 |
+
Project-URL: Tidelift, https://tidelift.com/subscription/pkg/pypi-jsonschema?utm_source=pypi-jsonschema&utm_medium=referral&utm_campaign=pypi-link
|
| 10 |
+
Project-URL: Changelog, https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst
|
| 11 |
+
Project-URL: Source, https://github.com/python-jsonschema/jsonschema
|
| 12 |
+
Author-email: Julian Berman <Julian+jsonschema@GrayVines.com>
|
| 13 |
+
License-Expression: MIT
|
| 14 |
+
License-File: COPYING
|
| 15 |
+
Keywords: data validation,json,json schema,jsonschema,validation
|
| 16 |
+
Classifier: Development Status :: 5 - Production/Stable
|
| 17 |
+
Classifier: Intended Audience :: Developers
|
| 18 |
+
Classifier: Operating System :: OS Independent
|
| 19 |
+
Classifier: Programming Language :: Python
|
| 20 |
+
Classifier: Programming Language :: Python :: 3.10
|
| 21 |
+
Classifier: Programming Language :: Python :: 3.11
|
| 22 |
+
Classifier: Programming Language :: Python :: 3.12
|
| 23 |
+
Classifier: Programming Language :: Python :: 3.13
|
| 24 |
+
Classifier: Programming Language :: Python :: 3.14
|
| 25 |
+
Classifier: Programming Language :: Python :: Implementation :: CPython
|
| 26 |
+
Classifier: Programming Language :: Python :: Implementation :: PyPy
|
| 27 |
+
Classifier: Topic :: File Formats :: JSON
|
| 28 |
+
Classifier: Topic :: File Formats :: JSON :: JSON Schema
|
| 29 |
+
Requires-Python: >=3.10
|
| 30 |
+
Requires-Dist: attrs>=22.2.0
|
| 31 |
+
Requires-Dist: jsonschema-specifications>=2023.03.6
|
| 32 |
+
Requires-Dist: referencing>=0.28.4
|
| 33 |
+
Requires-Dist: rpds-py>=0.25.0
|
| 34 |
+
Provides-Extra: format
|
| 35 |
+
Requires-Dist: fqdn; extra == 'format'
|
| 36 |
+
Requires-Dist: idna; extra == 'format'
|
| 37 |
+
Requires-Dist: isoduration; extra == 'format'
|
| 38 |
+
Requires-Dist: jsonpointer>1.13; extra == 'format'
|
| 39 |
+
Requires-Dist: rfc3339-validator; extra == 'format'
|
| 40 |
+
Requires-Dist: rfc3987; extra == 'format'
|
| 41 |
+
Requires-Dist: uri-template; extra == 'format'
|
| 42 |
+
Requires-Dist: webcolors>=1.11; extra == 'format'
|
| 43 |
+
Provides-Extra: format-nongpl
|
| 44 |
+
Requires-Dist: fqdn; extra == 'format-nongpl'
|
| 45 |
+
Requires-Dist: idna; extra == 'format-nongpl'
|
| 46 |
+
Requires-Dist: isoduration; extra == 'format-nongpl'
|
| 47 |
+
Requires-Dist: jsonpointer>1.13; extra == 'format-nongpl'
|
| 48 |
+
Requires-Dist: rfc3339-validator; extra == 'format-nongpl'
|
| 49 |
+
Requires-Dist: rfc3986-validator>0.1.0; extra == 'format-nongpl'
|
| 50 |
+
Requires-Dist: rfc3987-syntax>=1.1.0; extra == 'format-nongpl'
|
| 51 |
+
Requires-Dist: uri-template; extra == 'format-nongpl'
|
| 52 |
+
Requires-Dist: webcolors>=24.6.0; extra == 'format-nongpl'
|
| 53 |
+
Description-Content-Type: text/x-rst
|
| 54 |
+
|
| 55 |
+
==========
|
| 56 |
+
jsonschema
|
| 57 |
+
==========
|
| 58 |
+
|
| 59 |
+
|PyPI| |Pythons| |CI| |ReadTheDocs| |Precommit| |Zenodo|
|
| 60 |
+
|
| 61 |
+
.. |PyPI| image:: https://img.shields.io/pypi/v/jsonschema.svg
|
| 62 |
+
:alt: PyPI version
|
| 63 |
+
:target: https://pypi.org/project/jsonschema/
|
| 64 |
+
|
| 65 |
+
.. |Pythons| image:: https://img.shields.io/pypi/pyversions/jsonschema.svg
|
| 66 |
+
:alt: Supported Python versions
|
| 67 |
+
:target: https://pypi.org/project/jsonschema/
|
| 68 |
+
|
| 69 |
+
.. |CI| image:: https://github.com/python-jsonschema/jsonschema/workflows/CI/badge.svg
|
| 70 |
+
:alt: Build status
|
| 71 |
+
:target: https://github.com/python-jsonschema/jsonschema/actions?query=workflow%3ACI
|
| 72 |
+
|
| 73 |
+
.. |ReadTheDocs| image:: https://readthedocs.org/projects/python-jsonschema/badge/?version=stable&style=flat
|
| 74 |
+
:alt: ReadTheDocs status
|
| 75 |
+
:target: https://python-jsonschema.readthedocs.io/en/stable/
|
| 76 |
+
|
| 77 |
+
.. |Precommit| image:: https://results.pre-commit.ci/badge/github/python-jsonschema/jsonschema/main.svg
|
| 78 |
+
:alt: pre-commit.ci status
|
| 79 |
+
:target: https://results.pre-commit.ci/latest/github/python-jsonschema/jsonschema/main
|
| 80 |
+
|
| 81 |
+
.. |Zenodo| image:: https://zenodo.org/badge/3072629.svg
|
| 82 |
+
:alt: Zenodo DOI
|
| 83 |
+
:target: https://zenodo.org/badge/latestdoi/3072629
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
``jsonschema`` is an implementation of the `JSON Schema <https://json-schema.org>`_ specification for Python.
|
| 87 |
+
|
| 88 |
+
.. code:: python
|
| 89 |
+
|
| 90 |
+
>>> from jsonschema import validate
|
| 91 |
+
|
| 92 |
+
>>> # A sample schema, like what we'd get from json.load()
|
| 93 |
+
>>> schema = {
|
| 94 |
+
... "type" : "object",
|
| 95 |
+
... "properties" : {
|
| 96 |
+
... "price" : {"type" : "number"},
|
| 97 |
+
... "name" : {"type" : "string"},
|
| 98 |
+
... },
|
| 99 |
+
... }
|
| 100 |
+
|
| 101 |
+
>>> # If no exception is raised by validate(), the instance is valid.
|
| 102 |
+
>>> validate(instance={"name" : "Eggs", "price" : 34.99}, schema=schema)
|
| 103 |
+
|
| 104 |
+
>>> validate(
|
| 105 |
+
... instance={"name" : "Eggs", "price" : "Invalid"}, schema=schema,
|
| 106 |
+
... ) # doctest: +IGNORE_EXCEPTION_DETAIL
|
| 107 |
+
Traceback (most recent call last):
|
| 108 |
+
...
|
| 109 |
+
ValidationError: 'Invalid' is not of type 'number'
|
| 110 |
+
|
| 111 |
+
It can also be used from the command line by installing `check-jsonschema <https://github.com/python-jsonschema/check-jsonschema>`_.
|
| 112 |
+
|
| 113 |
+
Features
|
| 114 |
+
--------
|
| 115 |
+
|
| 116 |
+
* Full support for `Draft 2020-12 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft202012Validator>`_, `Draft 2019-09 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft201909Validator>`_, `Draft 7 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft7Validator>`_, `Draft 6 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft6Validator>`_, `Draft 4 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft4Validator>`_ and `Draft 3 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft3Validator>`_
|
| 117 |
+
|
| 118 |
+
* `Lazy validation <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/protocols/#jsonschema.protocols.Validator.iter_errors>`_ that can iteratively report *all* validation errors.
|
| 119 |
+
|
| 120 |
+
* `Programmatic querying <https://python-jsonschema.readthedocs.io/en/latest/errors/>`_ of which properties or items failed validation.
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
Installation
|
| 124 |
+
------------
|
| 125 |
+
|
| 126 |
+
``jsonschema`` is available on `PyPI <https://pypi.org/project/jsonschema/>`_. You can install using `pip <https://pip.pypa.io/en/stable/>`_:
|
| 127 |
+
|
| 128 |
+
.. code:: bash
|
| 129 |
+
|
| 130 |
+
$ pip install jsonschema
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
Extras
|
| 134 |
+
======
|
| 135 |
+
|
| 136 |
+
Two extras are available when installing the package, both currently related to ``format`` validation:
|
| 137 |
+
|
| 138 |
+
* ``format``
|
| 139 |
+
* ``format-nongpl``
|
| 140 |
+
|
| 141 |
+
They can be used when installing in order to include additional dependencies, e.g.:
|
| 142 |
+
|
| 143 |
+
.. code:: bash
|
| 144 |
+
|
| 145 |
+
$ pip install jsonschema'[format]'
|
| 146 |
+
|
| 147 |
+
Be aware that the mere presence of these dependencies – or even the specification of ``format`` checks in a schema – do *not* activate format checks (as per the specification).
|
| 148 |
+
Please read the `format validation documentation <https://python-jsonschema.readthedocs.io/en/latest/validate/#validating-formats>`_ for further details.
|
| 149 |
+
|
| 150 |
+
About
|
| 151 |
+
-----
|
| 152 |
+
|
| 153 |
+
I'm Julian Berman.
|
| 154 |
+
|
| 155 |
+
``jsonschema`` is on `GitHub <https://github.com/python-jsonschema/jsonschema>`_.
|
| 156 |
+
|
| 157 |
+
Get in touch, via GitHub or otherwise, if you've got something to contribute, it'd be most welcome!
|
| 158 |
+
|
| 159 |
+
If you feel overwhelmingly grateful, you can also `sponsor me <https://github.com/sponsors/Julian/>`_.
|
| 160 |
+
|
| 161 |
+
And for companies who appreciate ``jsonschema`` and its continued support and growth, ``jsonschema`` is also now supportable via `TideLift <https://tidelift.com/subscription/pkg/pypi-jsonschema?utm_source=pypi-jsonschema&utm_medium=referral&utm_campaign=readme>`_.
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
Release Information
|
| 165 |
+
-------------------
|
| 166 |
+
|
| 167 |
+
v4.26.0
|
| 168 |
+
=======
|
| 169 |
+
|
| 170 |
+
* Decrease import time by delaying importing of ``urllib.request`` (#1416).
|