giancap61 commited on
Commit
ec07f4a
·
verified ·
1 Parent(s): 1e54577

Add files using upload-large-folder tool

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/INSTALLER +1 -0
  2. TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/METADATA +84 -0
  3. TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/RECORD +57 -0
  4. TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/WHEEL +4 -0
  5. TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/entry_points.txt +3 -0
  6. TTS/Kokoro/.venv/Lib/site-packages/jinja2/__init__.py +38 -0
  7. TTS/Kokoro/.venv/Lib/site-packages/jinja2/_identifier.py +6 -0
  8. TTS/Kokoro/.venv/Lib/site-packages/jinja2/ext.py +870 -0
  9. TTS/Kokoro/.venv/Lib/site-packages/jinja2/filters.py +1873 -0
  10. TTS/Kokoro/.venv/Lib/site-packages/jinja2/loaders.py +693 -0
  11. TTS/Kokoro/.venv/Lib/site-packages/jinja2/meta.py +112 -0
  12. TTS/Kokoro/.venv/Lib/site-packages/jinja2/nativetypes.py +130 -0
  13. TTS/Kokoro/.venv/Lib/site-packages/jinja2/nodes.py +1206 -0
  14. TTS/Kokoro/.venv/Lib/site-packages/jinja2/optimizer.py +48 -0
  15. TTS/Kokoro/.venv/Lib/site-packages/jinja2/parser.py +1049 -0
  16. TTS/Kokoro/.venv/Lib/site-packages/jinja2/py.typed +0 -0
  17. TTS/Kokoro/.venv/Lib/site-packages/jinja2/runtime.py +1062 -0
  18. TTS/Kokoro/.venv/Lib/site-packages/jinja2/sandbox.py +436 -0
  19. TTS/Kokoro/.venv/Lib/site-packages/jinja2/tests.py +256 -0
  20. TTS/Kokoro/.venv/Lib/site-packages/jinja2/utils.py +766 -0
  21. TTS/Kokoro/.venv/Lib/site-packages/jinja2/visitor.py +92 -0
  22. TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/INSTALLER +1 -0
  23. TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/METADATA +172 -0
  24. TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/RECORD +217 -0
  25. TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/WHEEL +5 -0
  26. TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/top_level.txt +1 -0
  27. TTS/Kokoro/.venv/Lib/site-packages/joblib/__init__.py +163 -0
  28. TTS/Kokoro/.venv/Lib/site-packages/joblib/_cloudpickle_wrapper.py +18 -0
  29. TTS/Kokoro/.venv/Lib/site-packages/joblib/_dask.py +381 -0
  30. TTS/Kokoro/.venv/Lib/site-packages/joblib/_memmapping_reducer.py +715 -0
  31. TTS/Kokoro/.venv/Lib/site-packages/joblib/_multiprocessing_helpers.py +51 -0
  32. TTS/Kokoro/.venv/Lib/site-packages/joblib/_parallel_backends.py +753 -0
  33. TTS/Kokoro/.venv/Lib/site-packages/joblib/_store_backends.py +500 -0
  34. TTS/Kokoro/.venv/Lib/site-packages/joblib/_utils.py +120 -0
  35. TTS/Kokoro/.venv/Lib/site-packages/joblib/backports.py +195 -0
  36. TTS/Kokoro/.venv/Lib/site-packages/joblib/compressor.py +572 -0
  37. TTS/Kokoro/.venv/Lib/site-packages/joblib/disk.py +131 -0
  38. TTS/Kokoro/.venv/Lib/site-packages/joblib/executor.py +131 -0
  39. TTS/Kokoro/.venv/Lib/site-packages/joblib/func_inspect.py +379 -0
  40. TTS/Kokoro/.venv/Lib/site-packages/joblib/hashing.py +270 -0
  41. TTS/Kokoro/.venv/Lib/site-packages/joblib/logger.py +159 -0
  42. TTS/Kokoro/.venv/Lib/site-packages/joblib/memory.py +1242 -0
  43. TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle.py +756 -0
  44. TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_compat.py +250 -0
  45. TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_utils.py +291 -0
  46. TTS/Kokoro/.venv/Lib/site-packages/joblib/parallel.py +2075 -0
  47. TTS/Kokoro/.venv/Lib/site-packages/joblib/pool.py +362 -0
  48. TTS/Kokoro/.venv/Lib/site-packages/joblib/testing.py +96 -0
  49. TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/INSTALLER +1 -0
  50. TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/METADATA +170 -0
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/INSTALLER ADDED
@@ -0,0 +1 @@
 
 
1
+ pip
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/METADATA ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Metadata-Version: 2.4
2
+ Name: Jinja2
3
+ Version: 3.1.6
4
+ Summary: A very fast and expressive template engine.
5
+ Maintainer-email: Pallets <contact@palletsprojects.com>
6
+ Requires-Python: >=3.7
7
+ Description-Content-Type: text/markdown
8
+ Classifier: Development Status :: 5 - Production/Stable
9
+ Classifier: Environment :: Web Environment
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: License :: OSI Approved :: BSD License
12
+ Classifier: Operating System :: OS Independent
13
+ Classifier: Programming Language :: Python
14
+ Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
15
+ Classifier: Topic :: Text Processing :: Markup :: HTML
16
+ Classifier: Typing :: Typed
17
+ License-File: LICENSE.txt
18
+ Requires-Dist: MarkupSafe>=2.0
19
+ Requires-Dist: Babel>=2.7 ; extra == "i18n"
20
+ Project-URL: Changes, https://jinja.palletsprojects.com/changes/
21
+ Project-URL: Chat, https://discord.gg/pallets
22
+ Project-URL: Documentation, https://jinja.palletsprojects.com/
23
+ Project-URL: Donate, https://palletsprojects.com/donate
24
+ Project-URL: Source, https://github.com/pallets/jinja/
25
+ Provides-Extra: i18n
26
+
27
+ # Jinja
28
+
29
+ Jinja is a fast, expressive, extensible templating engine. Special
30
+ placeholders in the template allow writing code similar to Python
31
+ syntax. Then the template is passed data to render the final document.
32
+
33
+ It includes:
34
+
35
+ - Template inheritance and inclusion.
36
+ - Define and import macros within templates.
37
+ - HTML templates can use autoescaping to prevent XSS from untrusted
38
+ user input.
39
+ - A sandboxed environment can safely render untrusted templates.
40
+ - AsyncIO support for generating templates and calling async
41
+ functions.
42
+ - I18N support with Babel.
43
+ - Templates are compiled to optimized Python code just-in-time and
44
+ cached, or can be compiled ahead-of-time.
45
+ - Exceptions point to the correct line in templates to make debugging
46
+ easier.
47
+ - Extensible filters, tests, functions, and even syntax.
48
+
49
+ Jinja's philosophy is that while application logic belongs in Python if
50
+ possible, it shouldn't make the template designer's job difficult by
51
+ restricting functionality too much.
52
+
53
+
54
+ ## In A Nutshell
55
+
56
+ ```jinja
57
+ {% extends "base.html" %}
58
+ {% block title %}Members{% endblock %}
59
+ {% block content %}
60
+ <ul>
61
+ {% for user in users %}
62
+ <li><a href="{{ user.url }}">{{ user.username }}</a></li>
63
+ {% endfor %}
64
+ </ul>
65
+ {% endblock %}
66
+ ```
67
+
68
+ ## Donate
69
+
70
+ The Pallets organization develops and supports Jinja and other popular
71
+ packages. In order to grow the community of contributors and users, and
72
+ allow the maintainers to devote more time to the projects, [please
73
+ donate today][].
74
+
75
+ [please donate today]: https://palletsprojects.com/donate
76
+
77
+ ## Contributing
78
+
79
+ See our [detailed contributing documentation][contrib] for many ways to
80
+ contribute, including reporting issues, requesting features, asking or answering
81
+ questions, and making PRs.
82
+
83
+ [contrib]: https://palletsprojects.com/contributing/
84
+
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/RECORD ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ jinja2-3.1.6.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
2
+ jinja2-3.1.6.dist-info/METADATA,sha256=aMVUj7Z8QTKhOJjZsx7FDGvqKr3ZFdkh8hQ1XDpkmcg,2871
3
+ jinja2-3.1.6.dist-info/RECORD,,
4
+ jinja2-3.1.6.dist-info/WHEEL,sha256=_2ozNFCLWc93bK4WKHCO-eDUENDlo-dgc9cU3qokYO4,82
5
+ jinja2-3.1.6.dist-info/entry_points.txt,sha256=OL85gYU1eD8cuPlikifFngXpeBjaxl6rIJ8KkC_3r-I,58
6
+ jinja2-3.1.6.dist-info/licenses/LICENSE.txt,sha256=O0nc7kEF6ze6wQ-vG-JgQI_oXSUrjp3y4JefweCUQ3s,1475
7
+ jinja2/__init__.py,sha256=xxepO9i7DHsqkQrgBEduLtfoz2QCuT6_gbL4XSN1hbU,1928
8
+ jinja2/__pycache__/__init__.cpython-312.pyc,,
9
+ jinja2/__pycache__/_identifier.cpython-312.pyc,,
10
+ jinja2/__pycache__/async_utils.cpython-312.pyc,,
11
+ jinja2/__pycache__/bccache.cpython-312.pyc,,
12
+ jinja2/__pycache__/compiler.cpython-312.pyc,,
13
+ jinja2/__pycache__/constants.cpython-312.pyc,,
14
+ jinja2/__pycache__/debug.cpython-312.pyc,,
15
+ jinja2/__pycache__/defaults.cpython-312.pyc,,
16
+ jinja2/__pycache__/environment.cpython-312.pyc,,
17
+ jinja2/__pycache__/exceptions.cpython-312.pyc,,
18
+ jinja2/__pycache__/ext.cpython-312.pyc,,
19
+ jinja2/__pycache__/filters.cpython-312.pyc,,
20
+ jinja2/__pycache__/idtracking.cpython-312.pyc,,
21
+ jinja2/__pycache__/lexer.cpython-312.pyc,,
22
+ jinja2/__pycache__/loaders.cpython-312.pyc,,
23
+ jinja2/__pycache__/meta.cpython-312.pyc,,
24
+ jinja2/__pycache__/nativetypes.cpython-312.pyc,,
25
+ jinja2/__pycache__/nodes.cpython-312.pyc,,
26
+ jinja2/__pycache__/optimizer.cpython-312.pyc,,
27
+ jinja2/__pycache__/parser.cpython-312.pyc,,
28
+ jinja2/__pycache__/runtime.cpython-312.pyc,,
29
+ jinja2/__pycache__/sandbox.cpython-312.pyc,,
30
+ jinja2/__pycache__/tests.cpython-312.pyc,,
31
+ jinja2/__pycache__/utils.cpython-312.pyc,,
32
+ jinja2/__pycache__/visitor.cpython-312.pyc,,
33
+ jinja2/_identifier.py,sha256=_zYctNKzRqlk_murTNlzrju1FFJL7Va_Ijqqd7ii2lU,1958
34
+ jinja2/async_utils.py,sha256=vK-PdsuorOMnWSnEkT3iUJRIkTnYgO2T6MnGxDgHI5o,2834
35
+ jinja2/bccache.py,sha256=gh0qs9rulnXo0PhX5jTJy2UHzI8wFnQ63o_vw7nhzRg,14061
36
+ jinja2/compiler.py,sha256=9RpCQl5X88BHllJiPsHPh295Hh0uApvwFJNQuutULeM,74131
37
+ jinja2/constants.py,sha256=GMoFydBF_kdpaRKPoM5cl5MviquVRLVyZtfp5-16jg0,1433
38
+ jinja2/debug.py,sha256=CnHqCDHd-BVGvti_8ZsTolnXNhA3ECsY-6n_2pwU8Hw,6297
39
+ jinja2/defaults.py,sha256=boBcSw78h-lp20YbaXSJsqkAI2uN_mD_TtCydpeq5wU,1267
40
+ jinja2/environment.py,sha256=9nhrP7Ch-NbGX00wvyr4yy-uhNHq2OCc60ggGrni_fk,61513
41
+ jinja2/exceptions.py,sha256=ioHeHrWwCWNaXX1inHmHVblvc4haO7AXsjCp3GfWvx0,5071
42
+ jinja2/ext.py,sha256=5PF5eHfh8mXAIxXHHRB2xXbXohi8pE3nHSOxa66uS7E,31875
43
+ jinja2/filters.py,sha256=PQ_Egd9n9jSgtnGQYyF4K5j2nYwhUIulhPnyimkdr-k,55212
44
+ jinja2/idtracking.py,sha256=-ll5lIp73pML3ErUYiIJj7tdmWxcH_IlDv3yA_hiZYo,10555
45
+ jinja2/lexer.py,sha256=LYiYio6br-Tep9nPcupWXsPEtjluw3p1mU-lNBVRUfk,29786
46
+ jinja2/loaders.py,sha256=wIrnxjvcbqh5VwW28NSkfotiDq8qNCxIOSFbGUiSLB4,24055
47
+ jinja2/meta.py,sha256=OTDPkaFvU2Hgvx-6akz7154F8BIWaRmvJcBFvwopHww,4397
48
+ jinja2/nativetypes.py,sha256=7GIGALVJgdyL80oZJdQUaUfwSt5q2lSSZbXt0dNf_M4,4210
49
+ jinja2/nodes.py,sha256=m1Duzcr6qhZI8JQ6VyJgUNinjAf5bQzijSmDnMsvUx8,34579
50
+ jinja2/optimizer.py,sha256=rJnCRlQ7pZsEEmMhsQDgC_pKyDHxP5TPS6zVPGsgcu8,1651
51
+ jinja2/parser.py,sha256=lLOFy3sEmHc5IaEHRiH1sQVnId2moUQzhyeJZTtdY30,40383
52
+ jinja2/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
53
+ jinja2/runtime.py,sha256=gDk-GvdriJXqgsGbHgrcKTP0Yp6zPXzhzrIpCFH3jAU,34249
54
+ jinja2/sandbox.py,sha256=Mw2aitlY2I8la7FYhcX2YG9BtUYcLnD0Gh3d29cDWrY,15009
55
+ jinja2/tests.py,sha256=VLsBhVFnWg-PxSBz1MhRnNWgP1ovXk3neO1FLQMeC9Q,5926
56
+ jinja2/utils.py,sha256=rRp3o9e7ZKS4fyrWRbELyLcpuGVTFcnooaOa1qx_FIk,24129
57
+ jinja2/visitor.py,sha256=EcnL1PIwf_4RVCOMxsRNuR8AXHbS1qfAdMOE2ngKJz4,3557
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/WHEEL ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ Wheel-Version: 1.0
2
+ Generator: flit 3.11.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
TTS/Kokoro/.venv/Lib/site-packages/jinja2-3.1.6.dist-info/entry_points.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ [babel.extractors]
2
+ jinja2=jinja2.ext:babel_extract[i18n]
3
+
TTS/Kokoro/.venv/Lib/site-packages/jinja2/__init__.py ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Jinja is a template engine written in pure Python. It provides a
2
+ non-XML syntax that supports inline expressions and an optional
3
+ sandboxed environment.
4
+ """
5
+
6
+ from .bccache import BytecodeCache as BytecodeCache
7
+ from .bccache import FileSystemBytecodeCache as FileSystemBytecodeCache
8
+ from .bccache import MemcachedBytecodeCache as MemcachedBytecodeCache
9
+ from .environment import Environment as Environment
10
+ from .environment import Template as Template
11
+ from .exceptions import TemplateAssertionError as TemplateAssertionError
12
+ from .exceptions import TemplateError as TemplateError
13
+ from .exceptions import TemplateNotFound as TemplateNotFound
14
+ from .exceptions import TemplateRuntimeError as TemplateRuntimeError
15
+ from .exceptions import TemplatesNotFound as TemplatesNotFound
16
+ from .exceptions import TemplateSyntaxError as TemplateSyntaxError
17
+ from .exceptions import UndefinedError as UndefinedError
18
+ from .loaders import BaseLoader as BaseLoader
19
+ from .loaders import ChoiceLoader as ChoiceLoader
20
+ from .loaders import DictLoader as DictLoader
21
+ from .loaders import FileSystemLoader as FileSystemLoader
22
+ from .loaders import FunctionLoader as FunctionLoader
23
+ from .loaders import ModuleLoader as ModuleLoader
24
+ from .loaders import PackageLoader as PackageLoader
25
+ from .loaders import PrefixLoader as PrefixLoader
26
+ from .runtime import ChainableUndefined as ChainableUndefined
27
+ from .runtime import DebugUndefined as DebugUndefined
28
+ from .runtime import make_logging_undefined as make_logging_undefined
29
+ from .runtime import StrictUndefined as StrictUndefined
30
+ from .runtime import Undefined as Undefined
31
+ from .utils import clear_caches as clear_caches
32
+ from .utils import is_undefined as is_undefined
33
+ from .utils import pass_context as pass_context
34
+ from .utils import pass_environment as pass_environment
35
+ from .utils import pass_eval_context as pass_eval_context
36
+ from .utils import select_autoescape as select_autoescape
37
+
38
+ __version__ = "3.1.6"
TTS/Kokoro/.venv/Lib/site-packages/jinja2/_identifier.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ import re
2
+
3
+ # generated by scripts/generate_identifier_pattern.py
4
+ pattern = re.compile(
5
+ r"[\w·̀-ͯ·҃-֑҇-ׇֽֿׁׂׅׄؐ-ًؚ-ٰٟۖ-ۜ۟-۪ۤۧۨ-ܑۭܰ-݊ަ-ް߫-߽߳ࠖ-࠙ࠛ-ࠣࠥ-ࠧࠩ-࡙࠭-࡛࣓-ࣣ࣡-ःऺ-़ा-ॏ॑-ॗॢॣঁ-ঃ়া-ৄেৈো-্ৗৢৣ৾ਁ-ਃ਼ਾ-ੂੇੈੋ-੍ੑੰੱੵઁ-ઃ઼ા-ૅે-ૉો-્ૢૣૺ-૿ଁ-ଃ଼ା-ୄେୈୋ-୍ୖୗୢୣஂா-ூெ-ைொ-்ௗఀ-ఄా-ౄె-ైొ-్ౕౖౢౣಁ-ಃ಼ಾ-ೄೆ-ೈೊ-್ೕೖೢೣഀ-ഃ഻഼ാ-ൄെ-ൈൊ-്ൗൢൣංඃ්ා-ුූෘ-ෟෲෳัิ-ฺ็-๎ັິ-ູົຼ່-ໍ༹༘༙༵༷༾༿ཱ-྄྆྇ྍ-ྗྙ-ྼ࿆ါ-ှၖ-ၙၞ-ၠၢ-ၤၧ-ၭၱ-ၴႂ-ႍႏႚ-ႝ፝-፟ᜒ-᜔ᜲ-᜴ᝒᝓᝲᝳ឴-៓៝᠋-᠍ᢅᢆᢩᤠ-ᤫᤰ-᤻ᨗ-ᨛᩕ-ᩞ᩠-᩿᩼᪰-᪽ᬀ-ᬄ᬴-᭄᭫-᭳ᮀ-ᮂᮡ-ᮭ᯦-᯳ᰤ-᰷᳐-᳔᳒-᳨᳭ᳲ-᳴᳷-᳹᷀-᷹᷻-᷿‿⁀⁔⃐-⃥⃜⃡-⃰℘℮⳯-⵿⳱ⷠ-〪ⷿ-゙゚〯꙯ꙴ-꙽ꚞꚟ꛰꛱ꠂ꠆ꠋꠣ-ꠧꢀꢁꢴ-ꣅ꣠-꣱ꣿꤦ-꤭ꥇ-꥓ꦀ-ꦃ꦳-꧀ꧥꨩ-ꨶꩃꩌꩍꩻ-ꩽꪰꪲ-ꪴꪷꪸꪾ꪿꫁ꫫ-ꫯꫵ꫶ꯣ-ꯪ꯬꯭ﬞ︀-️︠-︯︳︴﹍-﹏_𐇽𐋠𐍶-𐍺𐨁-𐨃𐨅𐨆𐨌-𐨏𐨸-𐨿𐨺𐫦𐫥𐴤-𐽆𐴧-𐽐𑀀-𑀂𑀸-𑁆𑁿-𑂂𑂰-𑂺𑄀-𑄂𑄧-𑄴𑅅𑅆𑅳𑆀-𑆂𑆳-𑇀𑇉-𑇌𑈬-𑈷𑈾𑋟-𑋪𑌀-𑌃𑌻𑌼𑌾-𑍄𑍇𑍈𑍋-𑍍𑍗𑍢𑍣𑍦-𑍬𑍰-𑍴𑐵-𑑆𑑞𑒰-𑓃𑖯-𑖵𑖸-𑗀𑗜𑗝𑘰-𑙀𑚫-𑚷𑜝-𑜫𑠬-𑠺𑨁-𑨊𑨳-𑨹𑨻-𑨾𑩇𑩑-𑩛𑪊-𑪙𑰯-𑰶𑰸-𑰿𑲒-𑲧𑲩-𑲶𑴱-𑴶𑴺𑴼𑴽𑴿-𑵅𑵇𑶊-𑶎𑶐𑶑𑶓-𑶗𑻳-𑻶𖫰-𖫴𖬰-𖬶𖽑-𖽾𖾏-𖾒𛲝𛲞𝅥-𝅩𝅭-𝅲𝅻-𝆂𝆅-𝆋𝆪-𝆭𝉂-𝉄𝨀-𝨶𝨻-𝩬𝩵𝪄𝪛-𝪟𝪡-𝪯𞀀-𞀆𞀈-𞀘𞀛-𞀡𞀣𞀤𞀦-𞣐𞀪-𞣖𞥄-𞥊󠄀-󠇯]+" # noqa: B950
6
+ )
TTS/Kokoro/.venv/Lib/site-packages/jinja2/ext.py ADDED
@@ -0,0 +1,870 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Extension API for adding custom tags and behavior."""
2
+
3
+ import pprint
4
+ import re
5
+ import typing as t
6
+
7
+ from markupsafe import Markup
8
+
9
+ from . import defaults
10
+ from . import nodes
11
+ from .environment import Environment
12
+ from .exceptions import TemplateAssertionError
13
+ from .exceptions import TemplateSyntaxError
14
+ from .runtime import concat # type: ignore
15
+ from .runtime import Context
16
+ from .runtime import Undefined
17
+ from .utils import import_string
18
+ from .utils import pass_context
19
+
20
+ if t.TYPE_CHECKING:
21
+ import typing_extensions as te
22
+
23
+ from .lexer import Token
24
+ from .lexer import TokenStream
25
+ from .parser import Parser
26
+
27
+ class _TranslationsBasic(te.Protocol):
28
+ def gettext(self, message: str) -> str: ...
29
+
30
+ def ngettext(self, singular: str, plural: str, n: int) -> str:
31
+ pass
32
+
33
+ class _TranslationsContext(_TranslationsBasic):
34
+ def pgettext(self, context: str, message: str) -> str: ...
35
+
36
+ def npgettext(
37
+ self, context: str, singular: str, plural: str, n: int
38
+ ) -> str: ...
39
+
40
+ _SupportedTranslations = t.Union[_TranslationsBasic, _TranslationsContext]
41
+
42
+
43
+ # I18N functions available in Jinja templates. If the I18N library
44
+ # provides ugettext, it will be assigned to gettext.
45
+ GETTEXT_FUNCTIONS: t.Tuple[str, ...] = (
46
+ "_",
47
+ "gettext",
48
+ "ngettext",
49
+ "pgettext",
50
+ "npgettext",
51
+ )
52
+ _ws_re = re.compile(r"\s*\n\s*")
53
+
54
+
55
+ class Extension:
56
+ """Extensions can be used to add extra functionality to the Jinja template
57
+ system at the parser level. Custom extensions are bound to an environment
58
+ but may not store environment specific data on `self`. The reason for
59
+ this is that an extension can be bound to another environment (for
60
+ overlays) by creating a copy and reassigning the `environment` attribute.
61
+
62
+ As extensions are created by the environment they cannot accept any
63
+ arguments for configuration. One may want to work around that by using
64
+ a factory function, but that is not possible as extensions are identified
65
+ by their import name. The correct way to configure the extension is
66
+ storing the configuration values on the environment. Because this way the
67
+ environment ends up acting as central configuration storage the
68
+ attributes may clash which is why extensions have to ensure that the names
69
+ they choose for configuration are not too generic. ``prefix`` for example
70
+ is a terrible name, ``fragment_cache_prefix`` on the other hand is a good
71
+ name as includes the name of the extension (fragment cache).
72
+ """
73
+
74
+ identifier: t.ClassVar[str]
75
+
76
+ def __init_subclass__(cls) -> None:
77
+ cls.identifier = f"{cls.__module__}.{cls.__name__}"
78
+
79
+ #: if this extension parses this is the list of tags it's listening to.
80
+ tags: t.Set[str] = set()
81
+
82
+ #: the priority of that extension. This is especially useful for
83
+ #: extensions that preprocess values. A lower value means higher
84
+ #: priority.
85
+ #:
86
+ #: .. versionadded:: 2.4
87
+ priority = 100
88
+
89
+ def __init__(self, environment: Environment) -> None:
90
+ self.environment = environment
91
+
92
+ def bind(self, environment: Environment) -> "te.Self":
93
+ """Create a copy of this extension bound to another environment."""
94
+ rv = object.__new__(self.__class__)
95
+ rv.__dict__.update(self.__dict__)
96
+ rv.environment = environment
97
+ return rv
98
+
99
+ def preprocess(
100
+ self, source: str, name: t.Optional[str], filename: t.Optional[str] = None
101
+ ) -> str:
102
+ """This method is called before the actual lexing and can be used to
103
+ preprocess the source. The `filename` is optional. The return value
104
+ must be the preprocessed source.
105
+ """
106
+ return source
107
+
108
+ def filter_stream(
109
+ self, stream: "TokenStream"
110
+ ) -> t.Union["TokenStream", t.Iterable["Token"]]:
111
+ """It's passed a :class:`~jinja2.lexer.TokenStream` that can be used
112
+ to filter tokens returned. This method has to return an iterable of
113
+ :class:`~jinja2.lexer.Token`\\s, but it doesn't have to return a
114
+ :class:`~jinja2.lexer.TokenStream`.
115
+ """
116
+ return stream
117
+
118
+ def parse(self, parser: "Parser") -> t.Union[nodes.Node, t.List[nodes.Node]]:
119
+ """If any of the :attr:`tags` matched this method is called with the
120
+ parser as first argument. The token the parser stream is pointing at
121
+ is the name token that matched. This method has to return one or a
122
+ list of multiple nodes.
123
+ """
124
+ raise NotImplementedError()
125
+
126
+ def attr(
127
+ self, name: str, lineno: t.Optional[int] = None
128
+ ) -> nodes.ExtensionAttribute:
129
+ """Return an attribute node for the current extension. This is useful
130
+ to pass constants on extensions to generated template code.
131
+
132
+ ::
133
+
134
+ self.attr('_my_attribute', lineno=lineno)
135
+ """
136
+ return nodes.ExtensionAttribute(self.identifier, name, lineno=lineno)
137
+
138
+ def call_method(
139
+ self,
140
+ name: str,
141
+ args: t.Optional[t.List[nodes.Expr]] = None,
142
+ kwargs: t.Optional[t.List[nodes.Keyword]] = None,
143
+ dyn_args: t.Optional[nodes.Expr] = None,
144
+ dyn_kwargs: t.Optional[nodes.Expr] = None,
145
+ lineno: t.Optional[int] = None,
146
+ ) -> nodes.Call:
147
+ """Call a method of the extension. This is a shortcut for
148
+ :meth:`attr` + :class:`jinja2.nodes.Call`.
149
+ """
150
+ if args is None:
151
+ args = []
152
+ if kwargs is None:
153
+ kwargs = []
154
+ return nodes.Call(
155
+ self.attr(name, lineno=lineno),
156
+ args,
157
+ kwargs,
158
+ dyn_args,
159
+ dyn_kwargs,
160
+ lineno=lineno,
161
+ )
162
+
163
+
164
+ @pass_context
165
+ def _gettext_alias(
166
+ __context: Context, *args: t.Any, **kwargs: t.Any
167
+ ) -> t.Union[t.Any, Undefined]:
168
+ return __context.call(__context.resolve("gettext"), *args, **kwargs)
169
+
170
+
171
+ def _make_new_gettext(func: t.Callable[[str], str]) -> t.Callable[..., str]:
172
+ @pass_context
173
+ def gettext(__context: Context, __string: str, **variables: t.Any) -> str:
174
+ rv = __context.call(func, __string)
175
+ if __context.eval_ctx.autoescape:
176
+ rv = Markup(rv)
177
+ # Always treat as a format string, even if there are no
178
+ # variables. This makes translation strings more consistent
179
+ # and predictable. This requires escaping
180
+ return rv % variables # type: ignore
181
+
182
+ return gettext
183
+
184
+
185
+ def _make_new_ngettext(func: t.Callable[[str, str, int], str]) -> t.Callable[..., str]:
186
+ @pass_context
187
+ def ngettext(
188
+ __context: Context,
189
+ __singular: str,
190
+ __plural: str,
191
+ __num: int,
192
+ **variables: t.Any,
193
+ ) -> str:
194
+ variables.setdefault("num", __num)
195
+ rv = __context.call(func, __singular, __plural, __num)
196
+ if __context.eval_ctx.autoescape:
197
+ rv = Markup(rv)
198
+ # Always treat as a format string, see gettext comment above.
199
+ return rv % variables # type: ignore
200
+
201
+ return ngettext
202
+
203
+
204
+ def _make_new_pgettext(func: t.Callable[[str, str], str]) -> t.Callable[..., str]:
205
+ @pass_context
206
+ def pgettext(
207
+ __context: Context, __string_ctx: str, __string: str, **variables: t.Any
208
+ ) -> str:
209
+ variables.setdefault("context", __string_ctx)
210
+ rv = __context.call(func, __string_ctx, __string)
211
+
212
+ if __context.eval_ctx.autoescape:
213
+ rv = Markup(rv)
214
+
215
+ # Always treat as a format string, see gettext comment above.
216
+ return rv % variables # type: ignore
217
+
218
+ return pgettext
219
+
220
+
221
+ def _make_new_npgettext(
222
+ func: t.Callable[[str, str, str, int], str],
223
+ ) -> t.Callable[..., str]:
224
+ @pass_context
225
+ def npgettext(
226
+ __context: Context,
227
+ __string_ctx: str,
228
+ __singular: str,
229
+ __plural: str,
230
+ __num: int,
231
+ **variables: t.Any,
232
+ ) -> str:
233
+ variables.setdefault("context", __string_ctx)
234
+ variables.setdefault("num", __num)
235
+ rv = __context.call(func, __string_ctx, __singular, __plural, __num)
236
+
237
+ if __context.eval_ctx.autoescape:
238
+ rv = Markup(rv)
239
+
240
+ # Always treat as a format string, see gettext comment above.
241
+ return rv % variables # type: ignore
242
+
243
+ return npgettext
244
+
245
+
246
+ class InternationalizationExtension(Extension):
247
+ """This extension adds gettext support to Jinja."""
248
+
249
+ tags = {"trans"}
250
+
251
+ # TODO: the i18n extension is currently reevaluating values in a few
252
+ # situations. Take this example:
253
+ # {% trans count=something() %}{{ count }} foo{% pluralize
254
+ # %}{{ count }} fooss{% endtrans %}
255
+ # something is called twice here. One time for the gettext value and
256
+ # the other time for the n-parameter of the ngettext function.
257
+
258
+ def __init__(self, environment: Environment) -> None:
259
+ super().__init__(environment)
260
+ environment.globals["_"] = _gettext_alias
261
+ environment.extend(
262
+ install_gettext_translations=self._install,
263
+ install_null_translations=self._install_null,
264
+ install_gettext_callables=self._install_callables,
265
+ uninstall_gettext_translations=self._uninstall,
266
+ extract_translations=self._extract,
267
+ newstyle_gettext=False,
268
+ )
269
+
270
+ def _install(
271
+ self, translations: "_SupportedTranslations", newstyle: t.Optional[bool] = None
272
+ ) -> None:
273
+ # ugettext and ungettext are preferred in case the I18N library
274
+ # is providing compatibility with older Python versions.
275
+ gettext = getattr(translations, "ugettext", None)
276
+ if gettext is None:
277
+ gettext = translations.gettext
278
+ ngettext = getattr(translations, "ungettext", None)
279
+ if ngettext is None:
280
+ ngettext = translations.ngettext
281
+
282
+ pgettext = getattr(translations, "pgettext", None)
283
+ npgettext = getattr(translations, "npgettext", None)
284
+ self._install_callables(
285
+ gettext, ngettext, newstyle=newstyle, pgettext=pgettext, npgettext=npgettext
286
+ )
287
+
288
+ def _install_null(self, newstyle: t.Optional[bool] = None) -> None:
289
+ import gettext
290
+
291
+ translations = gettext.NullTranslations()
292
+
293
+ if hasattr(translations, "pgettext"):
294
+ # Python < 3.8
295
+ pgettext = translations.pgettext
296
+ else:
297
+
298
+ def pgettext(c: str, s: str) -> str: # type: ignore[misc]
299
+ return s
300
+
301
+ if hasattr(translations, "npgettext"):
302
+ npgettext = translations.npgettext
303
+ else:
304
+
305
+ def npgettext(c: str, s: str, p: str, n: int) -> str: # type: ignore[misc]
306
+ return s if n == 1 else p
307
+
308
+ self._install_callables(
309
+ gettext=translations.gettext,
310
+ ngettext=translations.ngettext,
311
+ newstyle=newstyle,
312
+ pgettext=pgettext,
313
+ npgettext=npgettext,
314
+ )
315
+
316
+ def _install_callables(
317
+ self,
318
+ gettext: t.Callable[[str], str],
319
+ ngettext: t.Callable[[str, str, int], str],
320
+ newstyle: t.Optional[bool] = None,
321
+ pgettext: t.Optional[t.Callable[[str, str], str]] = None,
322
+ npgettext: t.Optional[t.Callable[[str, str, str, int], str]] = None,
323
+ ) -> None:
324
+ if newstyle is not None:
325
+ self.environment.newstyle_gettext = newstyle # type: ignore
326
+ if self.environment.newstyle_gettext: # type: ignore
327
+ gettext = _make_new_gettext(gettext)
328
+ ngettext = _make_new_ngettext(ngettext)
329
+
330
+ if pgettext is not None:
331
+ pgettext = _make_new_pgettext(pgettext)
332
+
333
+ if npgettext is not None:
334
+ npgettext = _make_new_npgettext(npgettext)
335
+
336
+ self.environment.globals.update(
337
+ gettext=gettext, ngettext=ngettext, pgettext=pgettext, npgettext=npgettext
338
+ )
339
+
340
+ def _uninstall(self, translations: "_SupportedTranslations") -> None:
341
+ for key in ("gettext", "ngettext", "pgettext", "npgettext"):
342
+ self.environment.globals.pop(key, None)
343
+
344
+ def _extract(
345
+ self,
346
+ source: t.Union[str, nodes.Template],
347
+ gettext_functions: t.Sequence[str] = GETTEXT_FUNCTIONS,
348
+ ) -> t.Iterator[
349
+ t.Tuple[int, str, t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]]]
350
+ ]:
351
+ if isinstance(source, str):
352
+ source = self.environment.parse(source)
353
+ return extract_from_ast(source, gettext_functions)
354
+
355
+ def parse(self, parser: "Parser") -> t.Union[nodes.Node, t.List[nodes.Node]]:
356
+ """Parse a translatable tag."""
357
+ lineno = next(parser.stream).lineno
358
+
359
+ context = None
360
+ context_token = parser.stream.next_if("string")
361
+
362
+ if context_token is not None:
363
+ context = context_token.value
364
+
365
+ # find all the variables referenced. Additionally a variable can be
366
+ # defined in the body of the trans block too, but this is checked at
367
+ # a later state.
368
+ plural_expr: t.Optional[nodes.Expr] = None
369
+ plural_expr_assignment: t.Optional[nodes.Assign] = None
370
+ num_called_num = False
371
+ variables: t.Dict[str, nodes.Expr] = {}
372
+ trimmed = None
373
+ while parser.stream.current.type != "block_end":
374
+ if variables:
375
+ parser.stream.expect("comma")
376
+
377
+ # skip colon for python compatibility
378
+ if parser.stream.skip_if("colon"):
379
+ break
380
+
381
+ token = parser.stream.expect("name")
382
+ if token.value in variables:
383
+ parser.fail(
384
+ f"translatable variable {token.value!r} defined twice.",
385
+ token.lineno,
386
+ exc=TemplateAssertionError,
387
+ )
388
+
389
+ # expressions
390
+ if parser.stream.current.type == "assign":
391
+ next(parser.stream)
392
+ variables[token.value] = var = parser.parse_expression()
393
+ elif trimmed is None and token.value in ("trimmed", "notrimmed"):
394
+ trimmed = token.value == "trimmed"
395
+ continue
396
+ else:
397
+ variables[token.value] = var = nodes.Name(token.value, "load")
398
+
399
+ if plural_expr is None:
400
+ if isinstance(var, nodes.Call):
401
+ plural_expr = nodes.Name("_trans", "load")
402
+ variables[token.value] = plural_expr
403
+ plural_expr_assignment = nodes.Assign(
404
+ nodes.Name("_trans", "store"), var
405
+ )
406
+ else:
407
+ plural_expr = var
408
+ num_called_num = token.value == "num"
409
+
410
+ parser.stream.expect("block_end")
411
+
412
+ plural = None
413
+ have_plural = False
414
+ referenced = set()
415
+
416
+ # now parse until endtrans or pluralize
417
+ singular_names, singular = self._parse_block(parser, True)
418
+ if singular_names:
419
+ referenced.update(singular_names)
420
+ if plural_expr is None:
421
+ plural_expr = nodes.Name(singular_names[0], "load")
422
+ num_called_num = singular_names[0] == "num"
423
+
424
+ # if we have a pluralize block, we parse that too
425
+ if parser.stream.current.test("name:pluralize"):
426
+ have_plural = True
427
+ next(parser.stream)
428
+ if parser.stream.current.type != "block_end":
429
+ token = parser.stream.expect("name")
430
+ if token.value not in variables:
431
+ parser.fail(
432
+ f"unknown variable {token.value!r} for pluralization",
433
+ token.lineno,
434
+ exc=TemplateAssertionError,
435
+ )
436
+ plural_expr = variables[token.value]
437
+ num_called_num = token.value == "num"
438
+ parser.stream.expect("block_end")
439
+ plural_names, plural = self._parse_block(parser, False)
440
+ next(parser.stream)
441
+ referenced.update(plural_names)
442
+ else:
443
+ next(parser.stream)
444
+
445
+ # register free names as simple name expressions
446
+ for name in referenced:
447
+ if name not in variables:
448
+ variables[name] = nodes.Name(name, "load")
449
+
450
+ if not have_plural:
451
+ plural_expr = None
452
+ elif plural_expr is None:
453
+ parser.fail("pluralize without variables", lineno)
454
+
455
+ if trimmed is None:
456
+ trimmed = self.environment.policies["ext.i18n.trimmed"]
457
+ if trimmed:
458
+ singular = self._trim_whitespace(singular)
459
+ if plural:
460
+ plural = self._trim_whitespace(plural)
461
+
462
+ node = self._make_node(
463
+ singular,
464
+ plural,
465
+ context,
466
+ variables,
467
+ plural_expr,
468
+ bool(referenced),
469
+ num_called_num and have_plural,
470
+ )
471
+ node.set_lineno(lineno)
472
+ if plural_expr_assignment is not None:
473
+ return [plural_expr_assignment, node]
474
+ else:
475
+ return node
476
+
477
+ def _trim_whitespace(self, string: str, _ws_re: t.Pattern[str] = _ws_re) -> str:
478
+ return _ws_re.sub(" ", string.strip())
479
+
480
+ def _parse_block(
481
+ self, parser: "Parser", allow_pluralize: bool
482
+ ) -> t.Tuple[t.List[str], str]:
483
+ """Parse until the next block tag with a given name."""
484
+ referenced = []
485
+ buf = []
486
+
487
+ while True:
488
+ if parser.stream.current.type == "data":
489
+ buf.append(parser.stream.current.value.replace("%", "%%"))
490
+ next(parser.stream)
491
+ elif parser.stream.current.type == "variable_begin":
492
+ next(parser.stream)
493
+ name = parser.stream.expect("name").value
494
+ referenced.append(name)
495
+ buf.append(f"%({name})s")
496
+ parser.stream.expect("variable_end")
497
+ elif parser.stream.current.type == "block_begin":
498
+ next(parser.stream)
499
+ block_name = (
500
+ parser.stream.current.value
501
+ if parser.stream.current.type == "name"
502
+ else None
503
+ )
504
+ if block_name == "endtrans":
505
+ break
506
+ elif block_name == "pluralize":
507
+ if allow_pluralize:
508
+ break
509
+ parser.fail(
510
+ "a translatable section can have only one pluralize section"
511
+ )
512
+ elif block_name == "trans":
513
+ parser.fail(
514
+ "trans blocks can't be nested; did you mean `endtrans`?"
515
+ )
516
+ parser.fail(
517
+ f"control structures in translatable sections are not allowed; "
518
+ f"saw `{block_name}`"
519
+ )
520
+ elif parser.stream.eos:
521
+ parser.fail("unclosed translation block")
522
+ else:
523
+ raise RuntimeError("internal parser error")
524
+
525
+ return referenced, concat(buf)
526
+
527
+ def _make_node(
528
+ self,
529
+ singular: str,
530
+ plural: t.Optional[str],
531
+ context: t.Optional[str],
532
+ variables: t.Dict[str, nodes.Expr],
533
+ plural_expr: t.Optional[nodes.Expr],
534
+ vars_referenced: bool,
535
+ num_called_num: bool,
536
+ ) -> nodes.Output:
537
+ """Generates a useful node from the data provided."""
538
+ newstyle = self.environment.newstyle_gettext # type: ignore
539
+ node: nodes.Expr
540
+
541
+ # no variables referenced? no need to escape for old style
542
+ # gettext invocations only if there are vars.
543
+ if not vars_referenced and not newstyle:
544
+ singular = singular.replace("%%", "%")
545
+ if plural:
546
+ plural = plural.replace("%%", "%")
547
+
548
+ func_name = "gettext"
549
+ func_args: t.List[nodes.Expr] = [nodes.Const(singular)]
550
+
551
+ if context is not None:
552
+ func_args.insert(0, nodes.Const(context))
553
+ func_name = f"p{func_name}"
554
+
555
+ if plural_expr is not None:
556
+ func_name = f"n{func_name}"
557
+ func_args.extend((nodes.Const(plural), plural_expr))
558
+
559
+ node = nodes.Call(nodes.Name(func_name, "load"), func_args, [], None, None)
560
+
561
+ # in case newstyle gettext is used, the method is powerful
562
+ # enough to handle the variable expansion and autoescape
563
+ # handling itself
564
+ if newstyle:
565
+ for key, value in variables.items():
566
+ # the function adds that later anyways in case num was
567
+ # called num, so just skip it.
568
+ if num_called_num and key == "num":
569
+ continue
570
+ node.kwargs.append(nodes.Keyword(key, value))
571
+
572
+ # otherwise do that here
573
+ else:
574
+ # mark the return value as safe if we are in an
575
+ # environment with autoescaping turned on
576
+ node = nodes.MarkSafeIfAutoescape(node)
577
+ if variables:
578
+ node = nodes.Mod(
579
+ node,
580
+ nodes.Dict(
581
+ [
582
+ nodes.Pair(nodes.Const(key), value)
583
+ for key, value in variables.items()
584
+ ]
585
+ ),
586
+ )
587
+ return nodes.Output([node])
588
+
589
+
590
+ class ExprStmtExtension(Extension):
591
+ """Adds a `do` tag to Jinja that works like the print statement just
592
+ that it doesn't print the return value.
593
+ """
594
+
595
+ tags = {"do"}
596
+
597
+ def parse(self, parser: "Parser") -> nodes.ExprStmt:
598
+ node = nodes.ExprStmt(lineno=next(parser.stream).lineno)
599
+ node.node = parser.parse_tuple()
600
+ return node
601
+
602
+
603
+ class LoopControlExtension(Extension):
604
+ """Adds break and continue to the template engine."""
605
+
606
+ tags = {"break", "continue"}
607
+
608
+ def parse(self, parser: "Parser") -> t.Union[nodes.Break, nodes.Continue]:
609
+ token = next(parser.stream)
610
+ if token.value == "break":
611
+ return nodes.Break(lineno=token.lineno)
612
+ return nodes.Continue(lineno=token.lineno)
613
+
614
+
615
+ class DebugExtension(Extension):
616
+ """A ``{% debug %}`` tag that dumps the available variables,
617
+ filters, and tests.
618
+
619
+ .. code-block:: html+jinja
620
+
621
+ <pre>{% debug %}</pre>
622
+
623
+ .. code-block:: text
624
+
625
+ {'context': {'cycler': <class 'jinja2.utils.Cycler'>,
626
+ ...,
627
+ 'namespace': <class 'jinja2.utils.Namespace'>},
628
+ 'filters': ['abs', 'attr', 'batch', 'capitalize', 'center', 'count', 'd',
629
+ ..., 'urlencode', 'urlize', 'wordcount', 'wordwrap', 'xmlattr'],
630
+ 'tests': ['!=', '<', '<=', '==', '>', '>=', 'callable', 'defined',
631
+ ..., 'odd', 'sameas', 'sequence', 'string', 'undefined', 'upper']}
632
+
633
+ .. versionadded:: 2.11.0
634
+ """
635
+
636
+ tags = {"debug"}
637
+
638
+ def parse(self, parser: "Parser") -> nodes.Output:
639
+ lineno = parser.stream.expect("name:debug").lineno
640
+ context = nodes.ContextReference()
641
+ result = self.call_method("_render", [context], lineno=lineno)
642
+ return nodes.Output([result], lineno=lineno)
643
+
644
+ def _render(self, context: Context) -> str:
645
+ result = {
646
+ "context": context.get_all(),
647
+ "filters": sorted(self.environment.filters.keys()),
648
+ "tests": sorted(self.environment.tests.keys()),
649
+ }
650
+
651
+ # Set the depth since the intent is to show the top few names.
652
+ return pprint.pformat(result, depth=3, compact=True)
653
+
654
+
655
+ def extract_from_ast(
656
+ ast: nodes.Template,
657
+ gettext_functions: t.Sequence[str] = GETTEXT_FUNCTIONS,
658
+ babel_style: bool = True,
659
+ ) -> t.Iterator[
660
+ t.Tuple[int, str, t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]]]
661
+ ]:
662
+ """Extract localizable strings from the given template node. Per
663
+ default this function returns matches in babel style that means non string
664
+ parameters as well as keyword arguments are returned as `None`. This
665
+ allows Babel to figure out what you really meant if you are using
666
+ gettext functions that allow keyword arguments for placeholder expansion.
667
+ If you don't want that behavior set the `babel_style` parameter to `False`
668
+ which causes only strings to be returned and parameters are always stored
669
+ in tuples. As a consequence invalid gettext calls (calls without a single
670
+ string parameter or string parameters after non-string parameters) are
671
+ skipped.
672
+
673
+ This example explains the behavior:
674
+
675
+ >>> from jinja2 import Environment
676
+ >>> env = Environment()
677
+ >>> node = env.parse('{{ (_("foo"), _(), ngettext("foo", "bar", 42)) }}')
678
+ >>> list(extract_from_ast(node))
679
+ [(1, '_', 'foo'), (1, '_', ()), (1, 'ngettext', ('foo', 'bar', None))]
680
+ >>> list(extract_from_ast(node, babel_style=False))
681
+ [(1, '_', ('foo',)), (1, 'ngettext', ('foo', 'bar'))]
682
+
683
+ For every string found this function yields a ``(lineno, function,
684
+ message)`` tuple, where:
685
+
686
+ * ``lineno`` is the number of the line on which the string was found,
687
+ * ``function`` is the name of the ``gettext`` function used (if the
688
+ string was extracted from embedded Python code), and
689
+ * ``message`` is the string, or a tuple of strings for functions
690
+ with multiple string arguments.
691
+
692
+ This extraction function operates on the AST and is because of that unable
693
+ to extract any comments. For comment support you have to use the babel
694
+ extraction interface or extract comments yourself.
695
+ """
696
+ out: t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]]
697
+
698
+ for node in ast.find_all(nodes.Call):
699
+ if (
700
+ not isinstance(node.node, nodes.Name)
701
+ or node.node.name not in gettext_functions
702
+ ):
703
+ continue
704
+
705
+ strings: t.List[t.Optional[str]] = []
706
+
707
+ for arg in node.args:
708
+ if isinstance(arg, nodes.Const) and isinstance(arg.value, str):
709
+ strings.append(arg.value)
710
+ else:
711
+ strings.append(None)
712
+
713
+ for _ in node.kwargs:
714
+ strings.append(None)
715
+ if node.dyn_args is not None:
716
+ strings.append(None)
717
+ if node.dyn_kwargs is not None:
718
+ strings.append(None)
719
+
720
+ if not babel_style:
721
+ out = tuple(x for x in strings if x is not None)
722
+
723
+ if not out:
724
+ continue
725
+ else:
726
+ if len(strings) == 1:
727
+ out = strings[0]
728
+ else:
729
+ out = tuple(strings)
730
+
731
+ yield node.lineno, node.node.name, out
732
+
733
+
734
+ class _CommentFinder:
735
+ """Helper class to find comments in a token stream. Can only
736
+ find comments for gettext calls forwards. Once the comment
737
+ from line 4 is found, a comment for line 1 will not return a
738
+ usable value.
739
+ """
740
+
741
+ def __init__(
742
+ self, tokens: t.Sequence[t.Tuple[int, str, str]], comment_tags: t.Sequence[str]
743
+ ) -> None:
744
+ self.tokens = tokens
745
+ self.comment_tags = comment_tags
746
+ self.offset = 0
747
+ self.last_lineno = 0
748
+
749
+ def find_backwards(self, offset: int) -> t.List[str]:
750
+ try:
751
+ for _, token_type, token_value in reversed(
752
+ self.tokens[self.offset : offset]
753
+ ):
754
+ if token_type in ("comment", "linecomment"):
755
+ try:
756
+ prefix, comment = token_value.split(None, 1)
757
+ except ValueError:
758
+ continue
759
+ if prefix in self.comment_tags:
760
+ return [comment.rstrip()]
761
+ return []
762
+ finally:
763
+ self.offset = offset
764
+
765
+ def find_comments(self, lineno: int) -> t.List[str]:
766
+ if not self.comment_tags or self.last_lineno > lineno:
767
+ return []
768
+ for idx, (token_lineno, _, _) in enumerate(self.tokens[self.offset :]):
769
+ if token_lineno > lineno:
770
+ return self.find_backwards(self.offset + idx)
771
+ return self.find_backwards(len(self.tokens))
772
+
773
+
774
+ def babel_extract(
775
+ fileobj: t.BinaryIO,
776
+ keywords: t.Sequence[str],
777
+ comment_tags: t.Sequence[str],
778
+ options: t.Dict[str, t.Any],
779
+ ) -> t.Iterator[
780
+ t.Tuple[
781
+ int, str, t.Union[t.Optional[str], t.Tuple[t.Optional[str], ...]], t.List[str]
782
+ ]
783
+ ]:
784
+ """Babel extraction method for Jinja templates.
785
+
786
+ .. versionchanged:: 2.3
787
+ Basic support for translation comments was added. If `comment_tags`
788
+ is now set to a list of keywords for extraction, the extractor will
789
+ try to find the best preceding comment that begins with one of the
790
+ keywords. For best results, make sure to not have more than one
791
+ gettext call in one line of code and the matching comment in the
792
+ same line or the line before.
793
+
794
+ .. versionchanged:: 2.5.1
795
+ The `newstyle_gettext` flag can be set to `True` to enable newstyle
796
+ gettext calls.
797
+
798
+ .. versionchanged:: 2.7
799
+ A `silent` option can now be provided. If set to `False` template
800
+ syntax errors are propagated instead of being ignored.
801
+
802
+ :param fileobj: the file-like object the messages should be extracted from
803
+ :param keywords: a list of keywords (i.e. function names) that should be
804
+ recognized as translation functions
805
+ :param comment_tags: a list of translator tags to search for and include
806
+ in the results.
807
+ :param options: a dictionary of additional options (optional)
808
+ :return: an iterator over ``(lineno, funcname, message, comments)`` tuples.
809
+ (comments will be empty currently)
810
+ """
811
+ extensions: t.Dict[t.Type[Extension], None] = {}
812
+
813
+ for extension_name in options.get("extensions", "").split(","):
814
+ extension_name = extension_name.strip()
815
+
816
+ if not extension_name:
817
+ continue
818
+
819
+ extensions[import_string(extension_name)] = None
820
+
821
+ if InternationalizationExtension not in extensions:
822
+ extensions[InternationalizationExtension] = None
823
+
824
+ def getbool(options: t.Mapping[str, str], key: str, default: bool = False) -> bool:
825
+ return options.get(key, str(default)).lower() in {"1", "on", "yes", "true"}
826
+
827
+ silent = getbool(options, "silent", True)
828
+ environment = Environment(
829
+ options.get("block_start_string", defaults.BLOCK_START_STRING),
830
+ options.get("block_end_string", defaults.BLOCK_END_STRING),
831
+ options.get("variable_start_string", defaults.VARIABLE_START_STRING),
832
+ options.get("variable_end_string", defaults.VARIABLE_END_STRING),
833
+ options.get("comment_start_string", defaults.COMMENT_START_STRING),
834
+ options.get("comment_end_string", defaults.COMMENT_END_STRING),
835
+ options.get("line_statement_prefix") or defaults.LINE_STATEMENT_PREFIX,
836
+ options.get("line_comment_prefix") or defaults.LINE_COMMENT_PREFIX,
837
+ getbool(options, "trim_blocks", defaults.TRIM_BLOCKS),
838
+ getbool(options, "lstrip_blocks", defaults.LSTRIP_BLOCKS),
839
+ defaults.NEWLINE_SEQUENCE,
840
+ getbool(options, "keep_trailing_newline", defaults.KEEP_TRAILING_NEWLINE),
841
+ tuple(extensions),
842
+ cache_size=0,
843
+ auto_reload=False,
844
+ )
845
+
846
+ if getbool(options, "trimmed"):
847
+ environment.policies["ext.i18n.trimmed"] = True
848
+ if getbool(options, "newstyle_gettext"):
849
+ environment.newstyle_gettext = True # type: ignore
850
+
851
+ source = fileobj.read().decode(options.get("encoding", "utf-8"))
852
+ try:
853
+ node = environment.parse(source)
854
+ tokens = list(environment.lex(environment.preprocess(source)))
855
+ except TemplateSyntaxError:
856
+ if not silent:
857
+ raise
858
+ # skip templates with syntax errors
859
+ return
860
+
861
+ finder = _CommentFinder(tokens, comment_tags)
862
+ for lineno, func, message in extract_from_ast(node, keywords):
863
+ yield lineno, func, message, finder.find_comments(lineno)
864
+
865
+
866
+ #: nicer import names
867
+ i18n = InternationalizationExtension
868
+ do = ExprStmtExtension
869
+ loopcontrols = LoopControlExtension
870
+ debug = DebugExtension
TTS/Kokoro/.venv/Lib/site-packages/jinja2/filters.py ADDED
@@ -0,0 +1,1873 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Built-in template filters used with the ``|`` operator."""
2
+
3
+ import math
4
+ import random
5
+ import re
6
+ import typing
7
+ import typing as t
8
+ from collections import abc
9
+ from inspect import getattr_static
10
+ from itertools import chain
11
+ from itertools import groupby
12
+
13
+ from markupsafe import escape
14
+ from markupsafe import Markup
15
+ from markupsafe import soft_str
16
+
17
+ from .async_utils import async_variant
18
+ from .async_utils import auto_aiter
19
+ from .async_utils import auto_await
20
+ from .async_utils import auto_to_list
21
+ from .exceptions import FilterArgumentError
22
+ from .runtime import Undefined
23
+ from .utils import htmlsafe_json_dumps
24
+ from .utils import pass_context
25
+ from .utils import pass_environment
26
+ from .utils import pass_eval_context
27
+ from .utils import pformat
28
+ from .utils import url_quote
29
+ from .utils import urlize
30
+
31
+ if t.TYPE_CHECKING:
32
+ import typing_extensions as te
33
+
34
+ from .environment import Environment
35
+ from .nodes import EvalContext
36
+ from .runtime import Context
37
+ from .sandbox import SandboxedEnvironment # noqa: F401
38
+
39
+ class HasHTML(te.Protocol):
40
+ def __html__(self) -> str:
41
+ pass
42
+
43
+
44
+ F = t.TypeVar("F", bound=t.Callable[..., t.Any])
45
+ K = t.TypeVar("K")
46
+ V = t.TypeVar("V")
47
+
48
+
49
+ def ignore_case(value: V) -> V:
50
+ """For use as a postprocessor for :func:`make_attrgetter`. Converts strings
51
+ to lowercase and returns other types as-is."""
52
+ if isinstance(value, str):
53
+ return t.cast(V, value.lower())
54
+
55
+ return value
56
+
57
+
58
+ def make_attrgetter(
59
+ environment: "Environment",
60
+ attribute: t.Optional[t.Union[str, int]],
61
+ postprocess: t.Optional[t.Callable[[t.Any], t.Any]] = None,
62
+ default: t.Optional[t.Any] = None,
63
+ ) -> t.Callable[[t.Any], t.Any]:
64
+ """Returns a callable that looks up the given attribute from a
65
+ passed object with the rules of the environment. Dots are allowed
66
+ to access attributes of attributes. Integer parts in paths are
67
+ looked up as integers.
68
+ """
69
+ parts = _prepare_attribute_parts(attribute)
70
+
71
+ def attrgetter(item: t.Any) -> t.Any:
72
+ for part in parts:
73
+ item = environment.getitem(item, part)
74
+
75
+ if default is not None and isinstance(item, Undefined):
76
+ item = default
77
+
78
+ if postprocess is not None:
79
+ item = postprocess(item)
80
+
81
+ return item
82
+
83
+ return attrgetter
84
+
85
+
86
+ def make_multi_attrgetter(
87
+ environment: "Environment",
88
+ attribute: t.Optional[t.Union[str, int]],
89
+ postprocess: t.Optional[t.Callable[[t.Any], t.Any]] = None,
90
+ ) -> t.Callable[[t.Any], t.List[t.Any]]:
91
+ """Returns a callable that looks up the given comma separated
92
+ attributes from a passed object with the rules of the environment.
93
+ Dots are allowed to access attributes of each attribute. Integer
94
+ parts in paths are looked up as integers.
95
+
96
+ The value returned by the returned callable is a list of extracted
97
+ attribute values.
98
+
99
+ Examples of attribute: "attr1,attr2", "attr1.inner1.0,attr2.inner2.0", etc.
100
+ """
101
+ if isinstance(attribute, str):
102
+ split: t.Sequence[t.Union[str, int, None]] = attribute.split(",")
103
+ else:
104
+ split = [attribute]
105
+
106
+ parts = [_prepare_attribute_parts(item) for item in split]
107
+
108
+ def attrgetter(item: t.Any) -> t.List[t.Any]:
109
+ items = [None] * len(parts)
110
+
111
+ for i, attribute_part in enumerate(parts):
112
+ item_i = item
113
+
114
+ for part in attribute_part:
115
+ item_i = environment.getitem(item_i, part)
116
+
117
+ if postprocess is not None:
118
+ item_i = postprocess(item_i)
119
+
120
+ items[i] = item_i
121
+
122
+ return items
123
+
124
+ return attrgetter
125
+
126
+
127
+ def _prepare_attribute_parts(
128
+ attr: t.Optional[t.Union[str, int]],
129
+ ) -> t.List[t.Union[str, int]]:
130
+ if attr is None:
131
+ return []
132
+
133
+ if isinstance(attr, str):
134
+ return [int(x) if x.isdigit() else x for x in attr.split(".")]
135
+
136
+ return [attr]
137
+
138
+
139
+ def do_forceescape(value: "t.Union[str, HasHTML]") -> Markup:
140
+ """Enforce HTML escaping. This will probably double escape variables."""
141
+ if hasattr(value, "__html__"):
142
+ value = t.cast("HasHTML", value).__html__()
143
+
144
+ return escape(str(value))
145
+
146
+
147
+ def do_urlencode(
148
+ value: t.Union[str, t.Mapping[str, t.Any], t.Iterable[t.Tuple[str, t.Any]]],
149
+ ) -> str:
150
+ """Quote data for use in a URL path or query using UTF-8.
151
+
152
+ Basic wrapper around :func:`urllib.parse.quote` when given a
153
+ string, or :func:`urllib.parse.urlencode` for a dict or iterable.
154
+
155
+ :param value: Data to quote. A string will be quoted directly. A
156
+ dict or iterable of ``(key, value)`` pairs will be joined as a
157
+ query string.
158
+
159
+ When given a string, "/" is not quoted. HTTP servers treat "/" and
160
+ "%2F" equivalently in paths. If you need quoted slashes, use the
161
+ ``|replace("/", "%2F")`` filter.
162
+
163
+ .. versionadded:: 2.7
164
+ """
165
+ if isinstance(value, str) or not isinstance(value, abc.Iterable):
166
+ return url_quote(value)
167
+
168
+ if isinstance(value, dict):
169
+ items: t.Iterable[t.Tuple[str, t.Any]] = value.items()
170
+ else:
171
+ items = value # type: ignore
172
+
173
+ return "&".join(
174
+ f"{url_quote(k, for_qs=True)}={url_quote(v, for_qs=True)}" for k, v in items
175
+ )
176
+
177
+
178
+ @pass_eval_context
179
+ def do_replace(
180
+ eval_ctx: "EvalContext", s: str, old: str, new: str, count: t.Optional[int] = None
181
+ ) -> str:
182
+ """Return a copy of the value with all occurrences of a substring
183
+ replaced with a new one. The first argument is the substring
184
+ that should be replaced, the second is the replacement string.
185
+ If the optional third argument ``count`` is given, only the first
186
+ ``count`` occurrences are replaced:
187
+
188
+ .. sourcecode:: jinja
189
+
190
+ {{ "Hello World"|replace("Hello", "Goodbye") }}
191
+ -> Goodbye World
192
+
193
+ {{ "aaaaargh"|replace("a", "d'oh, ", 2) }}
194
+ -> d'oh, d'oh, aaargh
195
+ """
196
+ if count is None:
197
+ count = -1
198
+
199
+ if not eval_ctx.autoescape:
200
+ return str(s).replace(str(old), str(new), count)
201
+
202
+ if (
203
+ hasattr(old, "__html__")
204
+ or hasattr(new, "__html__")
205
+ and not hasattr(s, "__html__")
206
+ ):
207
+ s = escape(s)
208
+ else:
209
+ s = soft_str(s)
210
+
211
+ return s.replace(soft_str(old), soft_str(new), count)
212
+
213
+
214
+ def do_upper(s: str) -> str:
215
+ """Convert a value to uppercase."""
216
+ return soft_str(s).upper()
217
+
218
+
219
+ def do_lower(s: str) -> str:
220
+ """Convert a value to lowercase."""
221
+ return soft_str(s).lower()
222
+
223
+
224
+ def do_items(value: t.Union[t.Mapping[K, V], Undefined]) -> t.Iterator[t.Tuple[K, V]]:
225
+ """Return an iterator over the ``(key, value)`` items of a mapping.
226
+
227
+ ``x|items`` is the same as ``x.items()``, except if ``x`` is
228
+ undefined an empty iterator is returned.
229
+
230
+ This filter is useful if you expect the template to be rendered with
231
+ an implementation of Jinja in another programming language that does
232
+ not have a ``.items()`` method on its mapping type.
233
+
234
+ .. code-block:: html+jinja
235
+
236
+ <dl>
237
+ {% for key, value in my_dict|items %}
238
+ <dt>{{ key }}
239
+ <dd>{{ value }}
240
+ {% endfor %}
241
+ </dl>
242
+
243
+ .. versionadded:: 3.1
244
+ """
245
+ if isinstance(value, Undefined):
246
+ return
247
+
248
+ if not isinstance(value, abc.Mapping):
249
+ raise TypeError("Can only get item pairs from a mapping.")
250
+
251
+ yield from value.items()
252
+
253
+
254
+ # Check for characters that would move the parser state from key to value.
255
+ # https://html.spec.whatwg.org/#attribute-name-state
256
+ _attr_key_re = re.compile(r"[\s/>=]", flags=re.ASCII)
257
+
258
+
259
+ @pass_eval_context
260
+ def do_xmlattr(
261
+ eval_ctx: "EvalContext", d: t.Mapping[str, t.Any], autospace: bool = True
262
+ ) -> str:
263
+ """Create an SGML/XML attribute string based on the items in a dict.
264
+
265
+ **Values** that are neither ``none`` nor ``undefined`` are automatically
266
+ escaped, safely allowing untrusted user input.
267
+
268
+ User input should not be used as **keys** to this filter. If any key
269
+ contains a space, ``/`` solidus, ``>`` greater-than sign, or ``=`` equals
270
+ sign, this fails with a ``ValueError``. Regardless of this, user input
271
+ should never be used as keys to this filter, or must be separately validated
272
+ first.
273
+
274
+ .. sourcecode:: html+jinja
275
+
276
+ <ul{{ {'class': 'my_list', 'missing': none,
277
+ 'id': 'list-%d'|format(variable)}|xmlattr }}>
278
+ ...
279
+ </ul>
280
+
281
+ Results in something like this:
282
+
283
+ .. sourcecode:: html
284
+
285
+ <ul class="my_list" id="list-42">
286
+ ...
287
+ </ul>
288
+
289
+ As you can see it automatically prepends a space in front of the item
290
+ if the filter returned something unless the second parameter is false.
291
+
292
+ .. versionchanged:: 3.1.4
293
+ Keys with ``/`` solidus, ``>`` greater-than sign, or ``=`` equals sign
294
+ are not allowed.
295
+
296
+ .. versionchanged:: 3.1.3
297
+ Keys with spaces are not allowed.
298
+ """
299
+ items = []
300
+
301
+ for key, value in d.items():
302
+ if value is None or isinstance(value, Undefined):
303
+ continue
304
+
305
+ if _attr_key_re.search(key) is not None:
306
+ raise ValueError(f"Invalid character in attribute name: {key!r}")
307
+
308
+ items.append(f'{escape(key)}="{escape(value)}"')
309
+
310
+ rv = " ".join(items)
311
+
312
+ if autospace and rv:
313
+ rv = " " + rv
314
+
315
+ if eval_ctx.autoescape:
316
+ rv = Markup(rv)
317
+
318
+ return rv
319
+
320
+
321
+ def do_capitalize(s: str) -> str:
322
+ """Capitalize a value. The first character will be uppercase, all others
323
+ lowercase.
324
+ """
325
+ return soft_str(s).capitalize()
326
+
327
+
328
+ _word_beginning_split_re = re.compile(r"([-\s({\[<]+)")
329
+
330
+
331
+ def do_title(s: str) -> str:
332
+ """Return a titlecased version of the value. I.e. words will start with
333
+ uppercase letters, all remaining characters are lowercase.
334
+ """
335
+ return "".join(
336
+ [
337
+ item[0].upper() + item[1:].lower()
338
+ for item in _word_beginning_split_re.split(soft_str(s))
339
+ if item
340
+ ]
341
+ )
342
+
343
+
344
+ def do_dictsort(
345
+ value: t.Mapping[K, V],
346
+ case_sensitive: bool = False,
347
+ by: 'te.Literal["key", "value"]' = "key",
348
+ reverse: bool = False,
349
+ ) -> t.List[t.Tuple[K, V]]:
350
+ """Sort a dict and yield (key, value) pairs. Python dicts may not
351
+ be in the order you want to display them in, so sort them first.
352
+
353
+ .. sourcecode:: jinja
354
+
355
+ {% for key, value in mydict|dictsort %}
356
+ sort the dict by key, case insensitive
357
+
358
+ {% for key, value in mydict|dictsort(reverse=true) %}
359
+ sort the dict by key, case insensitive, reverse order
360
+
361
+ {% for key, value in mydict|dictsort(true) %}
362
+ sort the dict by key, case sensitive
363
+
364
+ {% for key, value in mydict|dictsort(false, 'value') %}
365
+ sort the dict by value, case insensitive
366
+ """
367
+ if by == "key":
368
+ pos = 0
369
+ elif by == "value":
370
+ pos = 1
371
+ else:
372
+ raise FilterArgumentError('You can only sort by either "key" or "value"')
373
+
374
+ def sort_func(item: t.Tuple[t.Any, t.Any]) -> t.Any:
375
+ value = item[pos]
376
+
377
+ if not case_sensitive:
378
+ value = ignore_case(value)
379
+
380
+ return value
381
+
382
+ return sorted(value.items(), key=sort_func, reverse=reverse)
383
+
384
+
385
+ @pass_environment
386
+ def do_sort(
387
+ environment: "Environment",
388
+ value: "t.Iterable[V]",
389
+ reverse: bool = False,
390
+ case_sensitive: bool = False,
391
+ attribute: t.Optional[t.Union[str, int]] = None,
392
+ ) -> "t.List[V]":
393
+ """Sort an iterable using Python's :func:`sorted`.
394
+
395
+ .. sourcecode:: jinja
396
+
397
+ {% for city in cities|sort %}
398
+ ...
399
+ {% endfor %}
400
+
401
+ :param reverse: Sort descending instead of ascending.
402
+ :param case_sensitive: When sorting strings, sort upper and lower
403
+ case separately.
404
+ :param attribute: When sorting objects or dicts, an attribute or
405
+ key to sort by. Can use dot notation like ``"address.city"``.
406
+ Can be a list of attributes like ``"age,name"``.
407
+
408
+ The sort is stable, it does not change the relative order of
409
+ elements that compare equal. This makes it is possible to chain
410
+ sorts on different attributes and ordering.
411
+
412
+ .. sourcecode:: jinja
413
+
414
+ {% for user in users|sort(attribute="name")
415
+ |sort(reverse=true, attribute="age") %}
416
+ ...
417
+ {% endfor %}
418
+
419
+ As a shortcut to chaining when the direction is the same for all
420
+ attributes, pass a comma separate list of attributes.
421
+
422
+ .. sourcecode:: jinja
423
+
424
+ {% for user in users|sort(attribute="age,name") %}
425
+ ...
426
+ {% endfor %}
427
+
428
+ .. versionchanged:: 2.11.0
429
+ The ``attribute`` parameter can be a comma separated list of
430
+ attributes, e.g. ``"age,name"``.
431
+
432
+ .. versionchanged:: 2.6
433
+ The ``attribute`` parameter was added.
434
+ """
435
+ key_func = make_multi_attrgetter(
436
+ environment, attribute, postprocess=ignore_case if not case_sensitive else None
437
+ )
438
+ return sorted(value, key=key_func, reverse=reverse)
439
+
440
+
441
+ @pass_environment
442
+ def sync_do_unique(
443
+ environment: "Environment",
444
+ value: "t.Iterable[V]",
445
+ case_sensitive: bool = False,
446
+ attribute: t.Optional[t.Union[str, int]] = None,
447
+ ) -> "t.Iterator[V]":
448
+ """Returns a list of unique items from the given iterable.
449
+
450
+ .. sourcecode:: jinja
451
+
452
+ {{ ['foo', 'bar', 'foobar', 'FooBar']|unique|list }}
453
+ -> ['foo', 'bar', 'foobar']
454
+
455
+ The unique items are yielded in the same order as their first occurrence in
456
+ the iterable passed to the filter.
457
+
458
+ :param case_sensitive: Treat upper and lower case strings as distinct.
459
+ :param attribute: Filter objects with unique values for this attribute.
460
+ """
461
+ getter = make_attrgetter(
462
+ environment, attribute, postprocess=ignore_case if not case_sensitive else None
463
+ )
464
+ seen = set()
465
+
466
+ for item in value:
467
+ key = getter(item)
468
+
469
+ if key not in seen:
470
+ seen.add(key)
471
+ yield item
472
+
473
+
474
+ @async_variant(sync_do_unique) # type: ignore
475
+ async def do_unique(
476
+ environment: "Environment",
477
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
478
+ case_sensitive: bool = False,
479
+ attribute: t.Optional[t.Union[str, int]] = None,
480
+ ) -> "t.Iterator[V]":
481
+ return sync_do_unique(
482
+ environment, await auto_to_list(value), case_sensitive, attribute
483
+ )
484
+
485
+
486
+ def _min_or_max(
487
+ environment: "Environment",
488
+ value: "t.Iterable[V]",
489
+ func: "t.Callable[..., V]",
490
+ case_sensitive: bool,
491
+ attribute: t.Optional[t.Union[str, int]],
492
+ ) -> "t.Union[V, Undefined]":
493
+ it = iter(value)
494
+
495
+ try:
496
+ first = next(it)
497
+ except StopIteration:
498
+ return environment.undefined("No aggregated item, sequence was empty.")
499
+
500
+ key_func = make_attrgetter(
501
+ environment, attribute, postprocess=ignore_case if not case_sensitive else None
502
+ )
503
+ return func(chain([first], it), key=key_func)
504
+
505
+
506
+ @pass_environment
507
+ def do_min(
508
+ environment: "Environment",
509
+ value: "t.Iterable[V]",
510
+ case_sensitive: bool = False,
511
+ attribute: t.Optional[t.Union[str, int]] = None,
512
+ ) -> "t.Union[V, Undefined]":
513
+ """Return the smallest item from the sequence.
514
+
515
+ .. sourcecode:: jinja
516
+
517
+ {{ [1, 2, 3]|min }}
518
+ -> 1
519
+
520
+ :param case_sensitive: Treat upper and lower case strings as distinct.
521
+ :param attribute: Get the object with the min value of this attribute.
522
+ """
523
+ return _min_or_max(environment, value, min, case_sensitive, attribute)
524
+
525
+
526
+ @pass_environment
527
+ def do_max(
528
+ environment: "Environment",
529
+ value: "t.Iterable[V]",
530
+ case_sensitive: bool = False,
531
+ attribute: t.Optional[t.Union[str, int]] = None,
532
+ ) -> "t.Union[V, Undefined]":
533
+ """Return the largest item from the sequence.
534
+
535
+ .. sourcecode:: jinja
536
+
537
+ {{ [1, 2, 3]|max }}
538
+ -> 3
539
+
540
+ :param case_sensitive: Treat upper and lower case strings as distinct.
541
+ :param attribute: Get the object with the max value of this attribute.
542
+ """
543
+ return _min_or_max(environment, value, max, case_sensitive, attribute)
544
+
545
+
546
+ def do_default(
547
+ value: V,
548
+ default_value: V = "", # type: ignore
549
+ boolean: bool = False,
550
+ ) -> V:
551
+ """If the value is undefined it will return the passed default value,
552
+ otherwise the value of the variable:
553
+
554
+ .. sourcecode:: jinja
555
+
556
+ {{ my_variable|default('my_variable is not defined') }}
557
+
558
+ This will output the value of ``my_variable`` if the variable was
559
+ defined, otherwise ``'my_variable is not defined'``. If you want
560
+ to use default with variables that evaluate to false you have to
561
+ set the second parameter to `true`:
562
+
563
+ .. sourcecode:: jinja
564
+
565
+ {{ ''|default('the string was empty', true) }}
566
+
567
+ .. versionchanged:: 2.11
568
+ It's now possible to configure the :class:`~jinja2.Environment` with
569
+ :class:`~jinja2.ChainableUndefined` to make the `default` filter work
570
+ on nested elements and attributes that may contain undefined values
571
+ in the chain without getting an :exc:`~jinja2.UndefinedError`.
572
+ """
573
+ if isinstance(value, Undefined) or (boolean and not value):
574
+ return default_value
575
+
576
+ return value
577
+
578
+
579
+ @pass_eval_context
580
+ def sync_do_join(
581
+ eval_ctx: "EvalContext",
582
+ value: t.Iterable[t.Any],
583
+ d: str = "",
584
+ attribute: t.Optional[t.Union[str, int]] = None,
585
+ ) -> str:
586
+ """Return a string which is the concatenation of the strings in the
587
+ sequence. The separator between elements is an empty string per
588
+ default, you can define it with the optional parameter:
589
+
590
+ .. sourcecode:: jinja
591
+
592
+ {{ [1, 2, 3]|join('|') }}
593
+ -> 1|2|3
594
+
595
+ {{ [1, 2, 3]|join }}
596
+ -> 123
597
+
598
+ It is also possible to join certain attributes of an object:
599
+
600
+ .. sourcecode:: jinja
601
+
602
+ {{ users|join(', ', attribute='username') }}
603
+
604
+ .. versionadded:: 2.6
605
+ The `attribute` parameter was added.
606
+ """
607
+ if attribute is not None:
608
+ value = map(make_attrgetter(eval_ctx.environment, attribute), value)
609
+
610
+ # no automatic escaping? joining is a lot easier then
611
+ if not eval_ctx.autoescape:
612
+ return str(d).join(map(str, value))
613
+
614
+ # if the delimiter doesn't have an html representation we check
615
+ # if any of the items has. If yes we do a coercion to Markup
616
+ if not hasattr(d, "__html__"):
617
+ value = list(value)
618
+ do_escape = False
619
+
620
+ for idx, item in enumerate(value):
621
+ if hasattr(item, "__html__"):
622
+ do_escape = True
623
+ else:
624
+ value[idx] = str(item)
625
+
626
+ if do_escape:
627
+ d = escape(d)
628
+ else:
629
+ d = str(d)
630
+
631
+ return d.join(value)
632
+
633
+ # no html involved, to normal joining
634
+ return soft_str(d).join(map(soft_str, value))
635
+
636
+
637
+ @async_variant(sync_do_join) # type: ignore
638
+ async def do_join(
639
+ eval_ctx: "EvalContext",
640
+ value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
641
+ d: str = "",
642
+ attribute: t.Optional[t.Union[str, int]] = None,
643
+ ) -> str:
644
+ return sync_do_join(eval_ctx, await auto_to_list(value), d, attribute)
645
+
646
+
647
+ def do_center(value: str, width: int = 80) -> str:
648
+ """Centers the value in a field of a given width."""
649
+ return soft_str(value).center(width)
650
+
651
+
652
+ @pass_environment
653
+ def sync_do_first(
654
+ environment: "Environment", seq: "t.Iterable[V]"
655
+ ) -> "t.Union[V, Undefined]":
656
+ """Return the first item of a sequence."""
657
+ try:
658
+ return next(iter(seq))
659
+ except StopIteration:
660
+ return environment.undefined("No first item, sequence was empty.")
661
+
662
+
663
+ @async_variant(sync_do_first) # type: ignore
664
+ async def do_first(
665
+ environment: "Environment", seq: "t.Union[t.AsyncIterable[V], t.Iterable[V]]"
666
+ ) -> "t.Union[V, Undefined]":
667
+ try:
668
+ return await auto_aiter(seq).__anext__()
669
+ except StopAsyncIteration:
670
+ return environment.undefined("No first item, sequence was empty.")
671
+
672
+
673
+ @pass_environment
674
+ def do_last(
675
+ environment: "Environment", seq: "t.Reversible[V]"
676
+ ) -> "t.Union[V, Undefined]":
677
+ """Return the last item of a sequence.
678
+
679
+ Note: Does not work with generators. You may want to explicitly
680
+ convert it to a list:
681
+
682
+ .. sourcecode:: jinja
683
+
684
+ {{ data | selectattr('name', '==', 'Jinja') | list | last }}
685
+ """
686
+ try:
687
+ return next(iter(reversed(seq)))
688
+ except StopIteration:
689
+ return environment.undefined("No last item, sequence was empty.")
690
+
691
+
692
+ # No async do_last, it may not be safe in async mode.
693
+
694
+
695
+ @pass_context
696
+ def do_random(context: "Context", seq: "t.Sequence[V]") -> "t.Union[V, Undefined]":
697
+ """Return a random item from the sequence."""
698
+ try:
699
+ return random.choice(seq)
700
+ except IndexError:
701
+ return context.environment.undefined("No random item, sequence was empty.")
702
+
703
+
704
+ def do_filesizeformat(value: t.Union[str, float, int], binary: bool = False) -> str:
705
+ """Format the value like a 'human-readable' file size (i.e. 13 kB,
706
+ 4.1 MB, 102 Bytes, etc). Per default decimal prefixes are used (Mega,
707
+ Giga, etc.), if the second parameter is set to `True` the binary
708
+ prefixes are used (Mebi, Gibi).
709
+ """
710
+ bytes = float(value)
711
+ base = 1024 if binary else 1000
712
+ prefixes = [
713
+ ("KiB" if binary else "kB"),
714
+ ("MiB" if binary else "MB"),
715
+ ("GiB" if binary else "GB"),
716
+ ("TiB" if binary else "TB"),
717
+ ("PiB" if binary else "PB"),
718
+ ("EiB" if binary else "EB"),
719
+ ("ZiB" if binary else "ZB"),
720
+ ("YiB" if binary else "YB"),
721
+ ]
722
+
723
+ if bytes == 1:
724
+ return "1 Byte"
725
+ elif bytes < base:
726
+ return f"{int(bytes)} Bytes"
727
+ else:
728
+ for i, prefix in enumerate(prefixes):
729
+ unit = base ** (i + 2)
730
+
731
+ if bytes < unit:
732
+ return f"{base * bytes / unit:.1f} {prefix}"
733
+
734
+ return f"{base * bytes / unit:.1f} {prefix}"
735
+
736
+
737
+ def do_pprint(value: t.Any) -> str:
738
+ """Pretty print a variable. Useful for debugging."""
739
+ return pformat(value)
740
+
741
+
742
+ _uri_scheme_re = re.compile(r"^([\w.+-]{2,}:(/){0,2})$")
743
+
744
+
745
+ @pass_eval_context
746
+ def do_urlize(
747
+ eval_ctx: "EvalContext",
748
+ value: str,
749
+ trim_url_limit: t.Optional[int] = None,
750
+ nofollow: bool = False,
751
+ target: t.Optional[str] = None,
752
+ rel: t.Optional[str] = None,
753
+ extra_schemes: t.Optional[t.Iterable[str]] = None,
754
+ ) -> str:
755
+ """Convert URLs in text into clickable links.
756
+
757
+ This may not recognize links in some situations. Usually, a more
758
+ comprehensive formatter, such as a Markdown library, is a better
759
+ choice.
760
+
761
+ Works on ``http://``, ``https://``, ``www.``, ``mailto:``, and email
762
+ addresses. Links with trailing punctuation (periods, commas, closing
763
+ parentheses) and leading punctuation (opening parentheses) are
764
+ recognized excluding the punctuation. Email addresses that include
765
+ header fields are not recognized (for example,
766
+ ``mailto:address@example.com?cc=copy@example.com``).
767
+
768
+ :param value: Original text containing URLs to link.
769
+ :param trim_url_limit: Shorten displayed URL values to this length.
770
+ :param nofollow: Add the ``rel=nofollow`` attribute to links.
771
+ :param target: Add the ``target`` attribute to links.
772
+ :param rel: Add the ``rel`` attribute to links.
773
+ :param extra_schemes: Recognize URLs that start with these schemes
774
+ in addition to the default behavior. Defaults to
775
+ ``env.policies["urlize.extra_schemes"]``, which defaults to no
776
+ extra schemes.
777
+
778
+ .. versionchanged:: 3.0
779
+ The ``extra_schemes`` parameter was added.
780
+
781
+ .. versionchanged:: 3.0
782
+ Generate ``https://`` links for URLs without a scheme.
783
+
784
+ .. versionchanged:: 3.0
785
+ The parsing rules were updated. Recognize email addresses with
786
+ or without the ``mailto:`` scheme. Validate IP addresses. Ignore
787
+ parentheses and brackets in more cases.
788
+
789
+ .. versionchanged:: 2.8
790
+ The ``target`` parameter was added.
791
+ """
792
+ policies = eval_ctx.environment.policies
793
+ rel_parts = set((rel or "").split())
794
+
795
+ if nofollow:
796
+ rel_parts.add("nofollow")
797
+
798
+ rel_parts.update((policies["urlize.rel"] or "").split())
799
+ rel = " ".join(sorted(rel_parts)) or None
800
+
801
+ if target is None:
802
+ target = policies["urlize.target"]
803
+
804
+ if extra_schemes is None:
805
+ extra_schemes = policies["urlize.extra_schemes"] or ()
806
+
807
+ for scheme in extra_schemes:
808
+ if _uri_scheme_re.fullmatch(scheme) is None:
809
+ raise FilterArgumentError(f"{scheme!r} is not a valid URI scheme prefix.")
810
+
811
+ rv = urlize(
812
+ value,
813
+ trim_url_limit=trim_url_limit,
814
+ rel=rel,
815
+ target=target,
816
+ extra_schemes=extra_schemes,
817
+ )
818
+
819
+ if eval_ctx.autoescape:
820
+ rv = Markup(rv)
821
+
822
+ return rv
823
+
824
+
825
+ def do_indent(
826
+ s: str, width: t.Union[int, str] = 4, first: bool = False, blank: bool = False
827
+ ) -> str:
828
+ """Return a copy of the string with each line indented by 4 spaces. The
829
+ first line and blank lines are not indented by default.
830
+
831
+ :param width: Number of spaces, or a string, to indent by.
832
+ :param first: Don't skip indenting the first line.
833
+ :param blank: Don't skip indenting empty lines.
834
+
835
+ .. versionchanged:: 3.0
836
+ ``width`` can be a string.
837
+
838
+ .. versionchanged:: 2.10
839
+ Blank lines are not indented by default.
840
+
841
+ Rename the ``indentfirst`` argument to ``first``.
842
+ """
843
+ if isinstance(width, str):
844
+ indention = width
845
+ else:
846
+ indention = " " * width
847
+
848
+ newline = "\n"
849
+
850
+ if isinstance(s, Markup):
851
+ indention = Markup(indention)
852
+ newline = Markup(newline)
853
+
854
+ s += newline # this quirk is necessary for splitlines method
855
+
856
+ if blank:
857
+ rv = (newline + indention).join(s.splitlines())
858
+ else:
859
+ lines = s.splitlines()
860
+ rv = lines.pop(0)
861
+
862
+ if lines:
863
+ rv += newline + newline.join(
864
+ indention + line if line else line for line in lines
865
+ )
866
+
867
+ if first:
868
+ rv = indention + rv
869
+
870
+ return rv
871
+
872
+
873
+ @pass_environment
874
+ def do_truncate(
875
+ env: "Environment",
876
+ s: str,
877
+ length: int = 255,
878
+ killwords: bool = False,
879
+ end: str = "...",
880
+ leeway: t.Optional[int] = None,
881
+ ) -> str:
882
+ """Return a truncated copy of the string. The length is specified
883
+ with the first parameter which defaults to ``255``. If the second
884
+ parameter is ``true`` the filter will cut the text at length. Otherwise
885
+ it will discard the last word. If the text was in fact
886
+ truncated it will append an ellipsis sign (``"..."``). If you want a
887
+ different ellipsis sign than ``"..."`` you can specify it using the
888
+ third parameter. Strings that only exceed the length by the tolerance
889
+ margin given in the fourth parameter will not be truncated.
890
+
891
+ .. sourcecode:: jinja
892
+
893
+ {{ "foo bar baz qux"|truncate(9) }}
894
+ -> "foo..."
895
+ {{ "foo bar baz qux"|truncate(9, True) }}
896
+ -> "foo ba..."
897
+ {{ "foo bar baz qux"|truncate(11) }}
898
+ -> "foo bar baz qux"
899
+ {{ "foo bar baz qux"|truncate(11, False, '...', 0) }}
900
+ -> "foo bar..."
901
+
902
+ The default leeway on newer Jinja versions is 5 and was 0 before but
903
+ can be reconfigured globally.
904
+ """
905
+ if leeway is None:
906
+ leeway = env.policies["truncate.leeway"]
907
+
908
+ assert length >= len(end), f"expected length >= {len(end)}, got {length}"
909
+ assert leeway >= 0, f"expected leeway >= 0, got {leeway}"
910
+
911
+ if len(s) <= length + leeway:
912
+ return s
913
+
914
+ if killwords:
915
+ return s[: length - len(end)] + end
916
+
917
+ result = s[: length - len(end)].rsplit(" ", 1)[0]
918
+ return result + end
919
+
920
+
921
+ @pass_environment
922
+ def do_wordwrap(
923
+ environment: "Environment",
924
+ s: str,
925
+ width: int = 79,
926
+ break_long_words: bool = True,
927
+ wrapstring: t.Optional[str] = None,
928
+ break_on_hyphens: bool = True,
929
+ ) -> str:
930
+ """Wrap a string to the given width. Existing newlines are treated
931
+ as paragraphs to be wrapped separately.
932
+
933
+ :param s: Original text to wrap.
934
+ :param width: Maximum length of wrapped lines.
935
+ :param break_long_words: If a word is longer than ``width``, break
936
+ it across lines.
937
+ :param break_on_hyphens: If a word contains hyphens, it may be split
938
+ across lines.
939
+ :param wrapstring: String to join each wrapped line. Defaults to
940
+ :attr:`Environment.newline_sequence`.
941
+
942
+ .. versionchanged:: 2.11
943
+ Existing newlines are treated as paragraphs wrapped separately.
944
+
945
+ .. versionchanged:: 2.11
946
+ Added the ``break_on_hyphens`` parameter.
947
+
948
+ .. versionchanged:: 2.7
949
+ Added the ``wrapstring`` parameter.
950
+ """
951
+ import textwrap
952
+
953
+ if wrapstring is None:
954
+ wrapstring = environment.newline_sequence
955
+
956
+ # textwrap.wrap doesn't consider existing newlines when wrapping.
957
+ # If the string has a newline before width, wrap will still insert
958
+ # a newline at width, resulting in a short line. Instead, split and
959
+ # wrap each paragraph individually.
960
+ return wrapstring.join(
961
+ [
962
+ wrapstring.join(
963
+ textwrap.wrap(
964
+ line,
965
+ width=width,
966
+ expand_tabs=False,
967
+ replace_whitespace=False,
968
+ break_long_words=break_long_words,
969
+ break_on_hyphens=break_on_hyphens,
970
+ )
971
+ )
972
+ for line in s.splitlines()
973
+ ]
974
+ )
975
+
976
+
977
+ _word_re = re.compile(r"\w+")
978
+
979
+
980
+ def do_wordcount(s: str) -> int:
981
+ """Count the words in that string."""
982
+ return len(_word_re.findall(soft_str(s)))
983
+
984
+
985
+ def do_int(value: t.Any, default: int = 0, base: int = 10) -> int:
986
+ """Convert the value into an integer. If the
987
+ conversion doesn't work it will return ``0``. You can
988
+ override this default using the first parameter. You
989
+ can also override the default base (10) in the second
990
+ parameter, which handles input with prefixes such as
991
+ 0b, 0o and 0x for bases 2, 8 and 16 respectively.
992
+ The base is ignored for decimal numbers and non-string values.
993
+ """
994
+ try:
995
+ if isinstance(value, str):
996
+ return int(value, base)
997
+
998
+ return int(value)
999
+ except (TypeError, ValueError):
1000
+ # this quirk is necessary so that "42.23"|int gives 42.
1001
+ try:
1002
+ return int(float(value))
1003
+ except (TypeError, ValueError, OverflowError):
1004
+ return default
1005
+
1006
+
1007
+ def do_float(value: t.Any, default: float = 0.0) -> float:
1008
+ """Convert the value into a floating point number. If the
1009
+ conversion doesn't work it will return ``0.0``. You can
1010
+ override this default using the first parameter.
1011
+ """
1012
+ try:
1013
+ return float(value)
1014
+ except (TypeError, ValueError):
1015
+ return default
1016
+
1017
+
1018
+ def do_format(value: str, *args: t.Any, **kwargs: t.Any) -> str:
1019
+ """Apply the given values to a `printf-style`_ format string, like
1020
+ ``string % values``.
1021
+
1022
+ .. sourcecode:: jinja
1023
+
1024
+ {{ "%s, %s!"|format(greeting, name) }}
1025
+ Hello, World!
1026
+
1027
+ In most cases it should be more convenient and efficient to use the
1028
+ ``%`` operator or :meth:`str.format`.
1029
+
1030
+ .. code-block:: text
1031
+
1032
+ {{ "%s, %s!" % (greeting, name) }}
1033
+ {{ "{}, {}!".format(greeting, name) }}
1034
+
1035
+ .. _printf-style: https://docs.python.org/library/stdtypes.html
1036
+ #printf-style-string-formatting
1037
+ """
1038
+ if args and kwargs:
1039
+ raise FilterArgumentError(
1040
+ "can't handle positional and keyword arguments at the same time"
1041
+ )
1042
+
1043
+ return soft_str(value) % (kwargs or args)
1044
+
1045
+
1046
+ def do_trim(value: str, chars: t.Optional[str] = None) -> str:
1047
+ """Strip leading and trailing characters, by default whitespace."""
1048
+ return soft_str(value).strip(chars)
1049
+
1050
+
1051
+ def do_striptags(value: "t.Union[str, HasHTML]") -> str:
1052
+ """Strip SGML/XML tags and replace adjacent whitespace by one space."""
1053
+ if hasattr(value, "__html__"):
1054
+ value = t.cast("HasHTML", value).__html__()
1055
+
1056
+ return Markup(str(value)).striptags()
1057
+
1058
+
1059
+ def sync_do_slice(
1060
+ value: "t.Collection[V]", slices: int, fill_with: "t.Optional[V]" = None
1061
+ ) -> "t.Iterator[t.List[V]]":
1062
+ """Slice an iterator and return a list of lists containing
1063
+ those items. Useful if you want to create a div containing
1064
+ three ul tags that represent columns:
1065
+
1066
+ .. sourcecode:: html+jinja
1067
+
1068
+ <div class="columnwrapper">
1069
+ {%- for column in items|slice(3) %}
1070
+ <ul class="column-{{ loop.index }}">
1071
+ {%- for item in column %}
1072
+ <li>{{ item }}</li>
1073
+ {%- endfor %}
1074
+ </ul>
1075
+ {%- endfor %}
1076
+ </div>
1077
+
1078
+ If you pass it a second argument it's used to fill missing
1079
+ values on the last iteration.
1080
+ """
1081
+ seq = list(value)
1082
+ length = len(seq)
1083
+ items_per_slice = length // slices
1084
+ slices_with_extra = length % slices
1085
+ offset = 0
1086
+
1087
+ for slice_number in range(slices):
1088
+ start = offset + slice_number * items_per_slice
1089
+
1090
+ if slice_number < slices_with_extra:
1091
+ offset += 1
1092
+
1093
+ end = offset + (slice_number + 1) * items_per_slice
1094
+ tmp = seq[start:end]
1095
+
1096
+ if fill_with is not None and slice_number >= slices_with_extra:
1097
+ tmp.append(fill_with)
1098
+
1099
+ yield tmp
1100
+
1101
+
1102
+ @async_variant(sync_do_slice) # type: ignore
1103
+ async def do_slice(
1104
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1105
+ slices: int,
1106
+ fill_with: t.Optional[t.Any] = None,
1107
+ ) -> "t.Iterator[t.List[V]]":
1108
+ return sync_do_slice(await auto_to_list(value), slices, fill_with)
1109
+
1110
+
1111
+ def do_batch(
1112
+ value: "t.Iterable[V]", linecount: int, fill_with: "t.Optional[V]" = None
1113
+ ) -> "t.Iterator[t.List[V]]":
1114
+ """
1115
+ A filter that batches items. It works pretty much like `slice`
1116
+ just the other way round. It returns a list of lists with the
1117
+ given number of items. If you provide a second parameter this
1118
+ is used to fill up missing items. See this example:
1119
+
1120
+ .. sourcecode:: html+jinja
1121
+
1122
+ <table>
1123
+ {%- for row in items|batch(3, '&nbsp;') %}
1124
+ <tr>
1125
+ {%- for column in row %}
1126
+ <td>{{ column }}</td>
1127
+ {%- endfor %}
1128
+ </tr>
1129
+ {%- endfor %}
1130
+ </table>
1131
+ """
1132
+ tmp: t.List[V] = []
1133
+
1134
+ for item in value:
1135
+ if len(tmp) == linecount:
1136
+ yield tmp
1137
+ tmp = []
1138
+
1139
+ tmp.append(item)
1140
+
1141
+ if tmp:
1142
+ if fill_with is not None and len(tmp) < linecount:
1143
+ tmp += [fill_with] * (linecount - len(tmp))
1144
+
1145
+ yield tmp
1146
+
1147
+
1148
+ def do_round(
1149
+ value: float,
1150
+ precision: int = 0,
1151
+ method: 'te.Literal["common", "ceil", "floor"]' = "common",
1152
+ ) -> float:
1153
+ """Round the number to a given precision. The first
1154
+ parameter specifies the precision (default is ``0``), the
1155
+ second the rounding method:
1156
+
1157
+ - ``'common'`` rounds either up or down
1158
+ - ``'ceil'`` always rounds up
1159
+ - ``'floor'`` always rounds down
1160
+
1161
+ If you don't specify a method ``'common'`` is used.
1162
+
1163
+ .. sourcecode:: jinja
1164
+
1165
+ {{ 42.55|round }}
1166
+ -> 43.0
1167
+ {{ 42.55|round(1, 'floor') }}
1168
+ -> 42.5
1169
+
1170
+ Note that even if rounded to 0 precision, a float is returned. If
1171
+ you need a real integer, pipe it through `int`:
1172
+
1173
+ .. sourcecode:: jinja
1174
+
1175
+ {{ 42.55|round|int }}
1176
+ -> 43
1177
+ """
1178
+ if method not in {"common", "ceil", "floor"}:
1179
+ raise FilterArgumentError("method must be common, ceil or floor")
1180
+
1181
+ if method == "common":
1182
+ return round(value, precision)
1183
+
1184
+ func = getattr(math, method)
1185
+ return t.cast(float, func(value * (10**precision)) / (10**precision))
1186
+
1187
+
1188
+ class _GroupTuple(t.NamedTuple):
1189
+ grouper: t.Any
1190
+ list: t.List[t.Any]
1191
+
1192
+ # Use the regular tuple repr to hide this subclass if users print
1193
+ # out the value during debugging.
1194
+ def __repr__(self) -> str:
1195
+ return tuple.__repr__(self)
1196
+
1197
+ def __str__(self) -> str:
1198
+ return tuple.__str__(self)
1199
+
1200
+
1201
+ @pass_environment
1202
+ def sync_do_groupby(
1203
+ environment: "Environment",
1204
+ value: "t.Iterable[V]",
1205
+ attribute: t.Union[str, int],
1206
+ default: t.Optional[t.Any] = None,
1207
+ case_sensitive: bool = False,
1208
+ ) -> "t.List[_GroupTuple]":
1209
+ """Group a sequence of objects by an attribute using Python's
1210
+ :func:`itertools.groupby`. The attribute can use dot notation for
1211
+ nested access, like ``"address.city"``. Unlike Python's ``groupby``,
1212
+ the values are sorted first so only one group is returned for each
1213
+ unique value.
1214
+
1215
+ For example, a list of ``User`` objects with a ``city`` attribute
1216
+ can be rendered in groups. In this example, ``grouper`` refers to
1217
+ the ``city`` value of the group.
1218
+
1219
+ .. sourcecode:: html+jinja
1220
+
1221
+ <ul>{% for city, items in users|groupby("city") %}
1222
+ <li>{{ city }}
1223
+ <ul>{% for user in items %}
1224
+ <li>{{ user.name }}
1225
+ {% endfor %}</ul>
1226
+ </li>
1227
+ {% endfor %}</ul>
1228
+
1229
+ ``groupby`` yields namedtuples of ``(grouper, list)``, which
1230
+ can be used instead of the tuple unpacking above. ``grouper`` is the
1231
+ value of the attribute, and ``list`` is the items with that value.
1232
+
1233
+ .. sourcecode:: html+jinja
1234
+
1235
+ <ul>{% for group in users|groupby("city") %}
1236
+ <li>{{ group.grouper }}: {{ group.list|join(", ") }}
1237
+ {% endfor %}</ul>
1238
+
1239
+ You can specify a ``default`` value to use if an object in the list
1240
+ does not have the given attribute.
1241
+
1242
+ .. sourcecode:: jinja
1243
+
1244
+ <ul>{% for city, items in users|groupby("city", default="NY") %}
1245
+ <li>{{ city }}: {{ items|map(attribute="name")|join(", ") }}</li>
1246
+ {% endfor %}</ul>
1247
+
1248
+ Like the :func:`~jinja-filters.sort` filter, sorting and grouping is
1249
+ case-insensitive by default. The ``key`` for each group will have
1250
+ the case of the first item in that group of values. For example, if
1251
+ a list of users has cities ``["CA", "NY", "ca"]``, the "CA" group
1252
+ will have two values. This can be disabled by passing
1253
+ ``case_sensitive=True``.
1254
+
1255
+ .. versionchanged:: 3.1
1256
+ Added the ``case_sensitive`` parameter. Sorting and grouping is
1257
+ case-insensitive by default, matching other filters that do
1258
+ comparisons.
1259
+
1260
+ .. versionchanged:: 3.0
1261
+ Added the ``default`` parameter.
1262
+
1263
+ .. versionchanged:: 2.6
1264
+ The attribute supports dot notation for nested access.
1265
+ """
1266
+ expr = make_attrgetter(
1267
+ environment,
1268
+ attribute,
1269
+ postprocess=ignore_case if not case_sensitive else None,
1270
+ default=default,
1271
+ )
1272
+ out = [
1273
+ _GroupTuple(key, list(values))
1274
+ for key, values in groupby(sorted(value, key=expr), expr)
1275
+ ]
1276
+
1277
+ if not case_sensitive:
1278
+ # Return the real key from the first value instead of the lowercase key.
1279
+ output_expr = make_attrgetter(environment, attribute, default=default)
1280
+ out = [_GroupTuple(output_expr(values[0]), values) for _, values in out]
1281
+
1282
+ return out
1283
+
1284
+
1285
+ @async_variant(sync_do_groupby) # type: ignore
1286
+ async def do_groupby(
1287
+ environment: "Environment",
1288
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1289
+ attribute: t.Union[str, int],
1290
+ default: t.Optional[t.Any] = None,
1291
+ case_sensitive: bool = False,
1292
+ ) -> "t.List[_GroupTuple]":
1293
+ expr = make_attrgetter(
1294
+ environment,
1295
+ attribute,
1296
+ postprocess=ignore_case if not case_sensitive else None,
1297
+ default=default,
1298
+ )
1299
+ out = [
1300
+ _GroupTuple(key, await auto_to_list(values))
1301
+ for key, values in groupby(sorted(await auto_to_list(value), key=expr), expr)
1302
+ ]
1303
+
1304
+ if not case_sensitive:
1305
+ # Return the real key from the first value instead of the lowercase key.
1306
+ output_expr = make_attrgetter(environment, attribute, default=default)
1307
+ out = [_GroupTuple(output_expr(values[0]), values) for _, values in out]
1308
+
1309
+ return out
1310
+
1311
+
1312
+ @pass_environment
1313
+ def sync_do_sum(
1314
+ environment: "Environment",
1315
+ iterable: "t.Iterable[V]",
1316
+ attribute: t.Optional[t.Union[str, int]] = None,
1317
+ start: V = 0, # type: ignore
1318
+ ) -> V:
1319
+ """Returns the sum of a sequence of numbers plus the value of parameter
1320
+ 'start' (which defaults to 0). When the sequence is empty it returns
1321
+ start.
1322
+
1323
+ It is also possible to sum up only certain attributes:
1324
+
1325
+ .. sourcecode:: jinja
1326
+
1327
+ Total: {{ items|sum(attribute='price') }}
1328
+
1329
+ .. versionchanged:: 2.6
1330
+ The ``attribute`` parameter was added to allow summing up over
1331
+ attributes. Also the ``start`` parameter was moved on to the right.
1332
+ """
1333
+ if attribute is not None:
1334
+ iterable = map(make_attrgetter(environment, attribute), iterable)
1335
+
1336
+ return sum(iterable, start) # type: ignore[no-any-return, call-overload]
1337
+
1338
+
1339
+ @async_variant(sync_do_sum) # type: ignore
1340
+ async def do_sum(
1341
+ environment: "Environment",
1342
+ iterable: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1343
+ attribute: t.Optional[t.Union[str, int]] = None,
1344
+ start: V = 0, # type: ignore
1345
+ ) -> V:
1346
+ rv = start
1347
+
1348
+ if attribute is not None:
1349
+ func = make_attrgetter(environment, attribute)
1350
+ else:
1351
+
1352
+ def func(x: V) -> V:
1353
+ return x
1354
+
1355
+ async for item in auto_aiter(iterable):
1356
+ rv += func(item)
1357
+
1358
+ return rv
1359
+
1360
+
1361
+ def sync_do_list(value: "t.Iterable[V]") -> "t.List[V]":
1362
+ """Convert the value into a list. If it was a string the returned list
1363
+ will be a list of characters.
1364
+ """
1365
+ return list(value)
1366
+
1367
+
1368
+ @async_variant(sync_do_list) # type: ignore
1369
+ async def do_list(value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]") -> "t.List[V]":
1370
+ return await auto_to_list(value)
1371
+
1372
+
1373
+ def do_mark_safe(value: str) -> Markup:
1374
+ """Mark the value as safe which means that in an environment with automatic
1375
+ escaping enabled this variable will not be escaped.
1376
+ """
1377
+ return Markup(value)
1378
+
1379
+
1380
+ def do_mark_unsafe(value: str) -> str:
1381
+ """Mark a value as unsafe. This is the reverse operation for :func:`safe`."""
1382
+ return str(value)
1383
+
1384
+
1385
+ @typing.overload
1386
+ def do_reverse(value: str) -> str: ...
1387
+
1388
+
1389
+ @typing.overload
1390
+ def do_reverse(value: "t.Iterable[V]") -> "t.Iterable[V]": ...
1391
+
1392
+
1393
+ def do_reverse(value: t.Union[str, t.Iterable[V]]) -> t.Union[str, t.Iterable[V]]:
1394
+ """Reverse the object or return an iterator that iterates over it the other
1395
+ way round.
1396
+ """
1397
+ if isinstance(value, str):
1398
+ return value[::-1]
1399
+
1400
+ try:
1401
+ return reversed(value) # type: ignore
1402
+ except TypeError:
1403
+ try:
1404
+ rv = list(value)
1405
+ rv.reverse()
1406
+ return rv
1407
+ except TypeError as e:
1408
+ raise FilterArgumentError("argument must be iterable") from e
1409
+
1410
+
1411
+ @pass_environment
1412
+ def do_attr(
1413
+ environment: "Environment", obj: t.Any, name: str
1414
+ ) -> t.Union[Undefined, t.Any]:
1415
+ """Get an attribute of an object. ``foo|attr("bar")`` works like
1416
+ ``foo.bar``, but returns undefined instead of falling back to ``foo["bar"]``
1417
+ if the attribute doesn't exist.
1418
+
1419
+ See :ref:`Notes on subscriptions <notes-on-subscriptions>` for more details.
1420
+ """
1421
+ # Environment.getattr will fall back to obj[name] if obj.name doesn't exist.
1422
+ # But we want to call env.getattr to get behavior such as sandboxing.
1423
+ # Determine if the attr exists first, so we know the fallback won't trigger.
1424
+ try:
1425
+ # This avoids executing properties/descriptors, but misses __getattr__
1426
+ # and __getattribute__ dynamic attrs.
1427
+ getattr_static(obj, name)
1428
+ except AttributeError:
1429
+ # This finds dynamic attrs, and we know it's not a descriptor at this point.
1430
+ if not hasattr(obj, name):
1431
+ return environment.undefined(obj=obj, name=name)
1432
+
1433
+ return environment.getattr(obj, name)
1434
+
1435
+
1436
+ @typing.overload
1437
+ def sync_do_map(
1438
+ context: "Context",
1439
+ value: t.Iterable[t.Any],
1440
+ name: str,
1441
+ *args: t.Any,
1442
+ **kwargs: t.Any,
1443
+ ) -> t.Iterable[t.Any]: ...
1444
+
1445
+
1446
+ @typing.overload
1447
+ def sync_do_map(
1448
+ context: "Context",
1449
+ value: t.Iterable[t.Any],
1450
+ *,
1451
+ attribute: str = ...,
1452
+ default: t.Optional[t.Any] = None,
1453
+ ) -> t.Iterable[t.Any]: ...
1454
+
1455
+
1456
+ @pass_context
1457
+ def sync_do_map(
1458
+ context: "Context", value: t.Iterable[t.Any], *args: t.Any, **kwargs: t.Any
1459
+ ) -> t.Iterable[t.Any]:
1460
+ """Applies a filter on a sequence of objects or looks up an attribute.
1461
+ This is useful when dealing with lists of objects but you are really
1462
+ only interested in a certain value of it.
1463
+
1464
+ The basic usage is mapping on an attribute. Imagine you have a list
1465
+ of users but you are only interested in a list of usernames:
1466
+
1467
+ .. sourcecode:: jinja
1468
+
1469
+ Users on this page: {{ users|map(attribute='username')|join(', ') }}
1470
+
1471
+ You can specify a ``default`` value to use if an object in the list
1472
+ does not have the given attribute.
1473
+
1474
+ .. sourcecode:: jinja
1475
+
1476
+ {{ users|map(attribute="username", default="Anonymous")|join(", ") }}
1477
+
1478
+ Alternatively you can let it invoke a filter by passing the name of the
1479
+ filter and the arguments afterwards. A good example would be applying a
1480
+ text conversion filter on a sequence:
1481
+
1482
+ .. sourcecode:: jinja
1483
+
1484
+ Users on this page: {{ titles|map('lower')|join(', ') }}
1485
+
1486
+ Similar to a generator comprehension such as:
1487
+
1488
+ .. code-block:: python
1489
+
1490
+ (u.username for u in users)
1491
+ (getattr(u, "username", "Anonymous") for u in users)
1492
+ (do_lower(x) for x in titles)
1493
+
1494
+ .. versionchanged:: 2.11.0
1495
+ Added the ``default`` parameter.
1496
+
1497
+ .. versionadded:: 2.7
1498
+ """
1499
+ if value:
1500
+ func = prepare_map(context, args, kwargs)
1501
+
1502
+ for item in value:
1503
+ yield func(item)
1504
+
1505
+
1506
+ @typing.overload
1507
+ def do_map(
1508
+ context: "Context",
1509
+ value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
1510
+ name: str,
1511
+ *args: t.Any,
1512
+ **kwargs: t.Any,
1513
+ ) -> t.Iterable[t.Any]: ...
1514
+
1515
+
1516
+ @typing.overload
1517
+ def do_map(
1518
+ context: "Context",
1519
+ value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
1520
+ *,
1521
+ attribute: str = ...,
1522
+ default: t.Optional[t.Any] = None,
1523
+ ) -> t.Iterable[t.Any]: ...
1524
+
1525
+
1526
+ @async_variant(sync_do_map) # type: ignore
1527
+ async def do_map(
1528
+ context: "Context",
1529
+ value: t.Union[t.AsyncIterable[t.Any], t.Iterable[t.Any]],
1530
+ *args: t.Any,
1531
+ **kwargs: t.Any,
1532
+ ) -> t.AsyncIterable[t.Any]:
1533
+ if value:
1534
+ func = prepare_map(context, args, kwargs)
1535
+
1536
+ async for item in auto_aiter(value):
1537
+ yield await auto_await(func(item))
1538
+
1539
+
1540
+ @pass_context
1541
+ def sync_do_select(
1542
+ context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
1543
+ ) -> "t.Iterator[V]":
1544
+ """Filters a sequence of objects by applying a test to each object,
1545
+ and only selecting the objects with the test succeeding.
1546
+
1547
+ If no test is specified, each object will be evaluated as a boolean.
1548
+
1549
+ Example usage:
1550
+
1551
+ .. sourcecode:: jinja
1552
+
1553
+ {{ numbers|select("odd") }}
1554
+ {{ numbers|select("odd") }}
1555
+ {{ numbers|select("divisibleby", 3) }}
1556
+ {{ numbers|select("lessthan", 42) }}
1557
+ {{ strings|select("equalto", "mystring") }}
1558
+
1559
+ Similar to a generator comprehension such as:
1560
+
1561
+ .. code-block:: python
1562
+
1563
+ (n for n in numbers if test_odd(n))
1564
+ (n for n in numbers if test_divisibleby(n, 3))
1565
+
1566
+ .. versionadded:: 2.7
1567
+ """
1568
+ return select_or_reject(context, value, args, kwargs, lambda x: x, False)
1569
+
1570
+
1571
+ @async_variant(sync_do_select) # type: ignore
1572
+ async def do_select(
1573
+ context: "Context",
1574
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1575
+ *args: t.Any,
1576
+ **kwargs: t.Any,
1577
+ ) -> "t.AsyncIterator[V]":
1578
+ return async_select_or_reject(context, value, args, kwargs, lambda x: x, False)
1579
+
1580
+
1581
+ @pass_context
1582
+ def sync_do_reject(
1583
+ context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
1584
+ ) -> "t.Iterator[V]":
1585
+ """Filters a sequence of objects by applying a test to each object,
1586
+ and rejecting the objects with the test succeeding.
1587
+
1588
+ If no test is specified, each object will be evaluated as a boolean.
1589
+
1590
+ Example usage:
1591
+
1592
+ .. sourcecode:: jinja
1593
+
1594
+ {{ numbers|reject("odd") }}
1595
+
1596
+ Similar to a generator comprehension such as:
1597
+
1598
+ .. code-block:: python
1599
+
1600
+ (n for n in numbers if not test_odd(n))
1601
+
1602
+ .. versionadded:: 2.7
1603
+ """
1604
+ return select_or_reject(context, value, args, kwargs, lambda x: not x, False)
1605
+
1606
+
1607
+ @async_variant(sync_do_reject) # type: ignore
1608
+ async def do_reject(
1609
+ context: "Context",
1610
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1611
+ *args: t.Any,
1612
+ **kwargs: t.Any,
1613
+ ) -> "t.AsyncIterator[V]":
1614
+ return async_select_or_reject(context, value, args, kwargs, lambda x: not x, False)
1615
+
1616
+
1617
+ @pass_context
1618
+ def sync_do_selectattr(
1619
+ context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
1620
+ ) -> "t.Iterator[V]":
1621
+ """Filters a sequence of objects by applying a test to the specified
1622
+ attribute of each object, and only selecting the objects with the
1623
+ test succeeding.
1624
+
1625
+ If no test is specified, the attribute's value will be evaluated as
1626
+ a boolean.
1627
+
1628
+ Example usage:
1629
+
1630
+ .. sourcecode:: jinja
1631
+
1632
+ {{ users|selectattr("is_active") }}
1633
+ {{ users|selectattr("email", "none") }}
1634
+
1635
+ Similar to a generator comprehension such as:
1636
+
1637
+ .. code-block:: python
1638
+
1639
+ (user for user in users if user.is_active)
1640
+ (user for user in users if test_none(user.email))
1641
+
1642
+ .. versionadded:: 2.7
1643
+ """
1644
+ return select_or_reject(context, value, args, kwargs, lambda x: x, True)
1645
+
1646
+
1647
+ @async_variant(sync_do_selectattr) # type: ignore
1648
+ async def do_selectattr(
1649
+ context: "Context",
1650
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1651
+ *args: t.Any,
1652
+ **kwargs: t.Any,
1653
+ ) -> "t.AsyncIterator[V]":
1654
+ return async_select_or_reject(context, value, args, kwargs, lambda x: x, True)
1655
+
1656
+
1657
+ @pass_context
1658
+ def sync_do_rejectattr(
1659
+ context: "Context", value: "t.Iterable[V]", *args: t.Any, **kwargs: t.Any
1660
+ ) -> "t.Iterator[V]":
1661
+ """Filters a sequence of objects by applying a test to the specified
1662
+ attribute of each object, and rejecting the objects with the test
1663
+ succeeding.
1664
+
1665
+ If no test is specified, the attribute's value will be evaluated as
1666
+ a boolean.
1667
+
1668
+ .. sourcecode:: jinja
1669
+
1670
+ {{ users|rejectattr("is_active") }}
1671
+ {{ users|rejectattr("email", "none") }}
1672
+
1673
+ Similar to a generator comprehension such as:
1674
+
1675
+ .. code-block:: python
1676
+
1677
+ (user for user in users if not user.is_active)
1678
+ (user for user in users if not test_none(user.email))
1679
+
1680
+ .. versionadded:: 2.7
1681
+ """
1682
+ return select_or_reject(context, value, args, kwargs, lambda x: not x, True)
1683
+
1684
+
1685
+ @async_variant(sync_do_rejectattr) # type: ignore
1686
+ async def do_rejectattr(
1687
+ context: "Context",
1688
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1689
+ *args: t.Any,
1690
+ **kwargs: t.Any,
1691
+ ) -> "t.AsyncIterator[V]":
1692
+ return async_select_or_reject(context, value, args, kwargs, lambda x: not x, True)
1693
+
1694
+
1695
+ @pass_eval_context
1696
+ def do_tojson(
1697
+ eval_ctx: "EvalContext", value: t.Any, indent: t.Optional[int] = None
1698
+ ) -> Markup:
1699
+ """Serialize an object to a string of JSON, and mark it safe to
1700
+ render in HTML. This filter is only for use in HTML documents.
1701
+
1702
+ The returned string is safe to render in HTML documents and
1703
+ ``<script>`` tags. The exception is in HTML attributes that are
1704
+ double quoted; either use single quotes or the ``|forceescape``
1705
+ filter.
1706
+
1707
+ :param value: The object to serialize to JSON.
1708
+ :param indent: The ``indent`` parameter passed to ``dumps``, for
1709
+ pretty-printing the value.
1710
+
1711
+ .. versionadded:: 2.9
1712
+ """
1713
+ policies = eval_ctx.environment.policies
1714
+ dumps = policies["json.dumps_function"]
1715
+ kwargs = policies["json.dumps_kwargs"]
1716
+
1717
+ if indent is not None:
1718
+ kwargs = kwargs.copy()
1719
+ kwargs["indent"] = indent
1720
+
1721
+ return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)
1722
+
1723
+
1724
+ def prepare_map(
1725
+ context: "Context", args: t.Tuple[t.Any, ...], kwargs: t.Dict[str, t.Any]
1726
+ ) -> t.Callable[[t.Any], t.Any]:
1727
+ if not args and "attribute" in kwargs:
1728
+ attribute = kwargs.pop("attribute")
1729
+ default = kwargs.pop("default", None)
1730
+
1731
+ if kwargs:
1732
+ raise FilterArgumentError(
1733
+ f"Unexpected keyword argument {next(iter(kwargs))!r}"
1734
+ )
1735
+
1736
+ func = make_attrgetter(context.environment, attribute, default=default)
1737
+ else:
1738
+ try:
1739
+ name = args[0]
1740
+ args = args[1:]
1741
+ except LookupError:
1742
+ raise FilterArgumentError("map requires a filter argument") from None
1743
+
1744
+ def func(item: t.Any) -> t.Any:
1745
+ return context.environment.call_filter(
1746
+ name, item, args, kwargs, context=context
1747
+ )
1748
+
1749
+ return func
1750
+
1751
+
1752
+ def prepare_select_or_reject(
1753
+ context: "Context",
1754
+ args: t.Tuple[t.Any, ...],
1755
+ kwargs: t.Dict[str, t.Any],
1756
+ modfunc: t.Callable[[t.Any], t.Any],
1757
+ lookup_attr: bool,
1758
+ ) -> t.Callable[[t.Any], t.Any]:
1759
+ if lookup_attr:
1760
+ try:
1761
+ attr = args[0]
1762
+ except LookupError:
1763
+ raise FilterArgumentError("Missing parameter for attribute name") from None
1764
+
1765
+ transfunc = make_attrgetter(context.environment, attr)
1766
+ off = 1
1767
+ else:
1768
+ off = 0
1769
+
1770
+ def transfunc(x: V) -> V:
1771
+ return x
1772
+
1773
+ try:
1774
+ name = args[off]
1775
+ args = args[1 + off :]
1776
+
1777
+ def func(item: t.Any) -> t.Any:
1778
+ return context.environment.call_test(name, item, args, kwargs, context)
1779
+
1780
+ except LookupError:
1781
+ func = bool # type: ignore
1782
+
1783
+ return lambda item: modfunc(func(transfunc(item)))
1784
+
1785
+
1786
+ def select_or_reject(
1787
+ context: "Context",
1788
+ value: "t.Iterable[V]",
1789
+ args: t.Tuple[t.Any, ...],
1790
+ kwargs: t.Dict[str, t.Any],
1791
+ modfunc: t.Callable[[t.Any], t.Any],
1792
+ lookup_attr: bool,
1793
+ ) -> "t.Iterator[V]":
1794
+ if value:
1795
+ func = prepare_select_or_reject(context, args, kwargs, modfunc, lookup_attr)
1796
+
1797
+ for item in value:
1798
+ if func(item):
1799
+ yield item
1800
+
1801
+
1802
+ async def async_select_or_reject(
1803
+ context: "Context",
1804
+ value: "t.Union[t.AsyncIterable[V], t.Iterable[V]]",
1805
+ args: t.Tuple[t.Any, ...],
1806
+ kwargs: t.Dict[str, t.Any],
1807
+ modfunc: t.Callable[[t.Any], t.Any],
1808
+ lookup_attr: bool,
1809
+ ) -> "t.AsyncIterator[V]":
1810
+ if value:
1811
+ func = prepare_select_or_reject(context, args, kwargs, modfunc, lookup_attr)
1812
+
1813
+ async for item in auto_aiter(value):
1814
+ if func(item):
1815
+ yield item
1816
+
1817
+
1818
+ FILTERS = {
1819
+ "abs": abs,
1820
+ "attr": do_attr,
1821
+ "batch": do_batch,
1822
+ "capitalize": do_capitalize,
1823
+ "center": do_center,
1824
+ "count": len,
1825
+ "d": do_default,
1826
+ "default": do_default,
1827
+ "dictsort": do_dictsort,
1828
+ "e": escape,
1829
+ "escape": escape,
1830
+ "filesizeformat": do_filesizeformat,
1831
+ "first": do_first,
1832
+ "float": do_float,
1833
+ "forceescape": do_forceescape,
1834
+ "format": do_format,
1835
+ "groupby": do_groupby,
1836
+ "indent": do_indent,
1837
+ "int": do_int,
1838
+ "join": do_join,
1839
+ "last": do_last,
1840
+ "length": len,
1841
+ "list": do_list,
1842
+ "lower": do_lower,
1843
+ "items": do_items,
1844
+ "map": do_map,
1845
+ "min": do_min,
1846
+ "max": do_max,
1847
+ "pprint": do_pprint,
1848
+ "random": do_random,
1849
+ "reject": do_reject,
1850
+ "rejectattr": do_rejectattr,
1851
+ "replace": do_replace,
1852
+ "reverse": do_reverse,
1853
+ "round": do_round,
1854
+ "safe": do_mark_safe,
1855
+ "select": do_select,
1856
+ "selectattr": do_selectattr,
1857
+ "slice": do_slice,
1858
+ "sort": do_sort,
1859
+ "string": soft_str,
1860
+ "striptags": do_striptags,
1861
+ "sum": do_sum,
1862
+ "title": do_title,
1863
+ "trim": do_trim,
1864
+ "truncate": do_truncate,
1865
+ "unique": do_unique,
1866
+ "upper": do_upper,
1867
+ "urlencode": do_urlencode,
1868
+ "urlize": do_urlize,
1869
+ "wordcount": do_wordcount,
1870
+ "wordwrap": do_wordwrap,
1871
+ "xmlattr": do_xmlattr,
1872
+ "tojson": do_tojson,
1873
+ }
TTS/Kokoro/.venv/Lib/site-packages/jinja2/loaders.py ADDED
@@ -0,0 +1,693 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """API and implementations for loading templates from different data
2
+ sources.
3
+ """
4
+
5
+ import importlib.util
6
+ import os
7
+ import posixpath
8
+ import sys
9
+ import typing as t
10
+ import weakref
11
+ import zipimport
12
+ from collections import abc
13
+ from hashlib import sha1
14
+ from importlib import import_module
15
+ from types import ModuleType
16
+
17
+ from .exceptions import TemplateNotFound
18
+ from .utils import internalcode
19
+
20
+ if t.TYPE_CHECKING:
21
+ from .environment import Environment
22
+ from .environment import Template
23
+
24
+
25
+ def split_template_path(template: str) -> t.List[str]:
26
+ """Split a path into segments and perform a sanity check. If it detects
27
+ '..' in the path it will raise a `TemplateNotFound` error.
28
+ """
29
+ pieces = []
30
+ for piece in template.split("/"):
31
+ if (
32
+ os.path.sep in piece
33
+ or (os.path.altsep and os.path.altsep in piece)
34
+ or piece == os.path.pardir
35
+ ):
36
+ raise TemplateNotFound(template)
37
+ elif piece and piece != ".":
38
+ pieces.append(piece)
39
+ return pieces
40
+
41
+
42
+ class BaseLoader:
43
+ """Baseclass for all loaders. Subclass this and override `get_source` to
44
+ implement a custom loading mechanism. The environment provides a
45
+ `get_template` method that calls the loader's `load` method to get the
46
+ :class:`Template` object.
47
+
48
+ A very basic example for a loader that looks up templates on the file
49
+ system could look like this::
50
+
51
+ from jinja2 import BaseLoader, TemplateNotFound
52
+ from os.path import join, exists, getmtime
53
+
54
+ class MyLoader(BaseLoader):
55
+
56
+ def __init__(self, path):
57
+ self.path = path
58
+
59
+ def get_source(self, environment, template):
60
+ path = join(self.path, template)
61
+ if not exists(path):
62
+ raise TemplateNotFound(template)
63
+ mtime = getmtime(path)
64
+ with open(path) as f:
65
+ source = f.read()
66
+ return source, path, lambda: mtime == getmtime(path)
67
+ """
68
+
69
+ #: if set to `False` it indicates that the loader cannot provide access
70
+ #: to the source of templates.
71
+ #:
72
+ #: .. versionadded:: 2.4
73
+ has_source_access = True
74
+
75
+ def get_source(
76
+ self, environment: "Environment", template: str
77
+ ) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
78
+ """Get the template source, filename and reload helper for a template.
79
+ It's passed the environment and template name and has to return a
80
+ tuple in the form ``(source, filename, uptodate)`` or raise a
81
+ `TemplateNotFound` error if it can't locate the template.
82
+
83
+ The source part of the returned tuple must be the source of the
84
+ template as a string. The filename should be the name of the
85
+ file on the filesystem if it was loaded from there, otherwise
86
+ ``None``. The filename is used by Python for the tracebacks
87
+ if no loader extension is used.
88
+
89
+ The last item in the tuple is the `uptodate` function. If auto
90
+ reloading is enabled it's always called to check if the template
91
+ changed. No arguments are passed so the function must store the
92
+ old state somewhere (for example in a closure). If it returns `False`
93
+ the template will be reloaded.
94
+ """
95
+ if not self.has_source_access:
96
+ raise RuntimeError(
97
+ f"{type(self).__name__} cannot provide access to the source"
98
+ )
99
+ raise TemplateNotFound(template)
100
+
101
+ def list_templates(self) -> t.List[str]:
102
+ """Iterates over all templates. If the loader does not support that
103
+ it should raise a :exc:`TypeError` which is the default behavior.
104
+ """
105
+ raise TypeError("this loader cannot iterate over all templates")
106
+
107
+ @internalcode
108
+ def load(
109
+ self,
110
+ environment: "Environment",
111
+ name: str,
112
+ globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
113
+ ) -> "Template":
114
+ """Loads a template. This method looks up the template in the cache
115
+ or loads one by calling :meth:`get_source`. Subclasses should not
116
+ override this method as loaders working on collections of other
117
+ loaders (such as :class:`PrefixLoader` or :class:`ChoiceLoader`)
118
+ will not call this method but `get_source` directly.
119
+ """
120
+ code = None
121
+ if globals is None:
122
+ globals = {}
123
+
124
+ # first we try to get the source for this template together
125
+ # with the filename and the uptodate function.
126
+ source, filename, uptodate = self.get_source(environment, name)
127
+
128
+ # try to load the code from the bytecode cache if there is a
129
+ # bytecode cache configured.
130
+ bcc = environment.bytecode_cache
131
+ if bcc is not None:
132
+ bucket = bcc.get_bucket(environment, name, filename, source)
133
+ code = bucket.code
134
+
135
+ # if we don't have code so far (not cached, no longer up to
136
+ # date) etc. we compile the template
137
+ if code is None:
138
+ code = environment.compile(source, name, filename)
139
+
140
+ # if the bytecode cache is available and the bucket doesn't
141
+ # have a code so far, we give the bucket the new code and put
142
+ # it back to the bytecode cache.
143
+ if bcc is not None and bucket.code is None:
144
+ bucket.code = code
145
+ bcc.set_bucket(bucket)
146
+
147
+ return environment.template_class.from_code(
148
+ environment, code, globals, uptodate
149
+ )
150
+
151
+
152
+ class FileSystemLoader(BaseLoader):
153
+ """Load templates from a directory in the file system.
154
+
155
+ The path can be relative or absolute. Relative paths are relative to
156
+ the current working directory.
157
+
158
+ .. code-block:: python
159
+
160
+ loader = FileSystemLoader("templates")
161
+
162
+ A list of paths can be given. The directories will be searched in
163
+ order, stopping at the first matching template.
164
+
165
+ .. code-block:: python
166
+
167
+ loader = FileSystemLoader(["/override/templates", "/default/templates"])
168
+
169
+ :param searchpath: A path, or list of paths, to the directory that
170
+ contains the templates.
171
+ :param encoding: Use this encoding to read the text from template
172
+ files.
173
+ :param followlinks: Follow symbolic links in the path.
174
+
175
+ .. versionchanged:: 2.8
176
+ Added the ``followlinks`` parameter.
177
+ """
178
+
179
+ def __init__(
180
+ self,
181
+ searchpath: t.Union[
182
+ str, "os.PathLike[str]", t.Sequence[t.Union[str, "os.PathLike[str]"]]
183
+ ],
184
+ encoding: str = "utf-8",
185
+ followlinks: bool = False,
186
+ ) -> None:
187
+ if not isinstance(searchpath, abc.Iterable) or isinstance(searchpath, str):
188
+ searchpath = [searchpath]
189
+
190
+ self.searchpath = [os.fspath(p) for p in searchpath]
191
+ self.encoding = encoding
192
+ self.followlinks = followlinks
193
+
194
+ def get_source(
195
+ self, environment: "Environment", template: str
196
+ ) -> t.Tuple[str, str, t.Callable[[], bool]]:
197
+ pieces = split_template_path(template)
198
+
199
+ for searchpath in self.searchpath:
200
+ # Use posixpath even on Windows to avoid "drive:" or UNC
201
+ # segments breaking out of the search directory.
202
+ filename = posixpath.join(searchpath, *pieces)
203
+
204
+ if os.path.isfile(filename):
205
+ break
206
+ else:
207
+ plural = "path" if len(self.searchpath) == 1 else "paths"
208
+ paths_str = ", ".join(repr(p) for p in self.searchpath)
209
+ raise TemplateNotFound(
210
+ template,
211
+ f"{template!r} not found in search {plural}: {paths_str}",
212
+ )
213
+
214
+ with open(filename, encoding=self.encoding) as f:
215
+ contents = f.read()
216
+
217
+ mtime = os.path.getmtime(filename)
218
+
219
+ def uptodate() -> bool:
220
+ try:
221
+ return os.path.getmtime(filename) == mtime
222
+ except OSError:
223
+ return False
224
+
225
+ # Use normpath to convert Windows altsep to sep.
226
+ return contents, os.path.normpath(filename), uptodate
227
+
228
+ def list_templates(self) -> t.List[str]:
229
+ found = set()
230
+ for searchpath in self.searchpath:
231
+ walk_dir = os.walk(searchpath, followlinks=self.followlinks)
232
+ for dirpath, _, filenames in walk_dir:
233
+ for filename in filenames:
234
+ template = (
235
+ os.path.join(dirpath, filename)[len(searchpath) :]
236
+ .strip(os.path.sep)
237
+ .replace(os.path.sep, "/")
238
+ )
239
+ if template[:2] == "./":
240
+ template = template[2:]
241
+ if template not in found:
242
+ found.add(template)
243
+ return sorted(found)
244
+
245
+
246
+ if sys.version_info >= (3, 13):
247
+
248
+ def _get_zipimporter_files(z: t.Any) -> t.Dict[str, object]:
249
+ try:
250
+ get_files = z._get_files
251
+ except AttributeError as e:
252
+ raise TypeError(
253
+ "This zip import does not have the required"
254
+ " metadata to list templates."
255
+ ) from e
256
+ return get_files()
257
+ else:
258
+
259
+ def _get_zipimporter_files(z: t.Any) -> t.Dict[str, object]:
260
+ try:
261
+ files = z._files
262
+ except AttributeError as e:
263
+ raise TypeError(
264
+ "This zip import does not have the required"
265
+ " metadata to list templates."
266
+ ) from e
267
+ return files # type: ignore[no-any-return]
268
+
269
+
270
+ class PackageLoader(BaseLoader):
271
+ """Load templates from a directory in a Python package.
272
+
273
+ :param package_name: Import name of the package that contains the
274
+ template directory.
275
+ :param package_path: Directory within the imported package that
276
+ contains the templates.
277
+ :param encoding: Encoding of template files.
278
+
279
+ The following example looks up templates in the ``pages`` directory
280
+ within the ``project.ui`` package.
281
+
282
+ .. code-block:: python
283
+
284
+ loader = PackageLoader("project.ui", "pages")
285
+
286
+ Only packages installed as directories (standard pip behavior) or
287
+ zip/egg files (less common) are supported. The Python API for
288
+ introspecting data in packages is too limited to support other
289
+ installation methods the way this loader requires.
290
+
291
+ There is limited support for :pep:`420` namespace packages. The
292
+ template directory is assumed to only be in one namespace
293
+ contributor. Zip files contributing to a namespace are not
294
+ supported.
295
+
296
+ .. versionchanged:: 3.0
297
+ No longer uses ``setuptools`` as a dependency.
298
+
299
+ .. versionchanged:: 3.0
300
+ Limited PEP 420 namespace package support.
301
+ """
302
+
303
+ def __init__(
304
+ self,
305
+ package_name: str,
306
+ package_path: "str" = "templates",
307
+ encoding: str = "utf-8",
308
+ ) -> None:
309
+ package_path = os.path.normpath(package_path).rstrip(os.path.sep)
310
+
311
+ # normpath preserves ".", which isn't valid in zip paths.
312
+ if package_path == os.path.curdir:
313
+ package_path = ""
314
+ elif package_path[:2] == os.path.curdir + os.path.sep:
315
+ package_path = package_path[2:]
316
+
317
+ self.package_path = package_path
318
+ self.package_name = package_name
319
+ self.encoding = encoding
320
+
321
+ # Make sure the package exists. This also makes namespace
322
+ # packages work, otherwise get_loader returns None.
323
+ import_module(package_name)
324
+ spec = importlib.util.find_spec(package_name)
325
+ assert spec is not None, "An import spec was not found for the package."
326
+ loader = spec.loader
327
+ assert loader is not None, "A loader was not found for the package."
328
+ self._loader = loader
329
+ self._archive = None
330
+
331
+ if isinstance(loader, zipimport.zipimporter):
332
+ self._archive = loader.archive
333
+ pkgdir = next(iter(spec.submodule_search_locations)) # type: ignore
334
+ template_root = os.path.join(pkgdir, package_path).rstrip(os.path.sep)
335
+ else:
336
+ roots: t.List[str] = []
337
+
338
+ # One element for regular packages, multiple for namespace
339
+ # packages, or None for single module file.
340
+ if spec.submodule_search_locations:
341
+ roots.extend(spec.submodule_search_locations)
342
+ # A single module file, use the parent directory instead.
343
+ elif spec.origin is not None:
344
+ roots.append(os.path.dirname(spec.origin))
345
+
346
+ if not roots:
347
+ raise ValueError(
348
+ f"The {package_name!r} package was not installed in a"
349
+ " way that PackageLoader understands."
350
+ )
351
+
352
+ for root in roots:
353
+ root = os.path.join(root, package_path)
354
+
355
+ if os.path.isdir(root):
356
+ template_root = root
357
+ break
358
+ else:
359
+ raise ValueError(
360
+ f"PackageLoader could not find a {package_path!r} directory"
361
+ f" in the {package_name!r} package."
362
+ )
363
+
364
+ self._template_root = template_root
365
+
366
+ def get_source(
367
+ self, environment: "Environment", template: str
368
+ ) -> t.Tuple[str, str, t.Optional[t.Callable[[], bool]]]:
369
+ # Use posixpath even on Windows to avoid "drive:" or UNC
370
+ # segments breaking out of the search directory. Use normpath to
371
+ # convert Windows altsep to sep.
372
+ p = os.path.normpath(
373
+ posixpath.join(self._template_root, *split_template_path(template))
374
+ )
375
+ up_to_date: t.Optional[t.Callable[[], bool]]
376
+
377
+ if self._archive is None:
378
+ # Package is a directory.
379
+ if not os.path.isfile(p):
380
+ raise TemplateNotFound(template)
381
+
382
+ with open(p, "rb") as f:
383
+ source = f.read()
384
+
385
+ mtime = os.path.getmtime(p)
386
+
387
+ def up_to_date() -> bool:
388
+ return os.path.isfile(p) and os.path.getmtime(p) == mtime
389
+
390
+ else:
391
+ # Package is a zip file.
392
+ try:
393
+ source = self._loader.get_data(p) # type: ignore
394
+ except OSError as e:
395
+ raise TemplateNotFound(template) from e
396
+
397
+ # Could use the zip's mtime for all template mtimes, but
398
+ # would need to safely reload the module if it's out of
399
+ # date, so just report it as always current.
400
+ up_to_date = None
401
+
402
+ return source.decode(self.encoding), p, up_to_date
403
+
404
+ def list_templates(self) -> t.List[str]:
405
+ results: t.List[str] = []
406
+
407
+ if self._archive is None:
408
+ # Package is a directory.
409
+ offset = len(self._template_root)
410
+
411
+ for dirpath, _, filenames in os.walk(self._template_root):
412
+ dirpath = dirpath[offset:].lstrip(os.path.sep)
413
+ results.extend(
414
+ os.path.join(dirpath, name).replace(os.path.sep, "/")
415
+ for name in filenames
416
+ )
417
+ else:
418
+ files = _get_zipimporter_files(self._loader)
419
+
420
+ # Package is a zip file.
421
+ prefix = (
422
+ self._template_root[len(self._archive) :].lstrip(os.path.sep)
423
+ + os.path.sep
424
+ )
425
+ offset = len(prefix)
426
+
427
+ for name in files:
428
+ # Find names under the templates directory that aren't directories.
429
+ if name.startswith(prefix) and name[-1] != os.path.sep:
430
+ results.append(name[offset:].replace(os.path.sep, "/"))
431
+
432
+ results.sort()
433
+ return results
434
+
435
+
436
+ class DictLoader(BaseLoader):
437
+ """Loads a template from a Python dict mapping template names to
438
+ template source. This loader is useful for unittesting:
439
+
440
+ >>> loader = DictLoader({'index.html': 'source here'})
441
+
442
+ Because auto reloading is rarely useful this is disabled by default.
443
+ """
444
+
445
+ def __init__(self, mapping: t.Mapping[str, str]) -> None:
446
+ self.mapping = mapping
447
+
448
+ def get_source(
449
+ self, environment: "Environment", template: str
450
+ ) -> t.Tuple[str, None, t.Callable[[], bool]]:
451
+ if template in self.mapping:
452
+ source = self.mapping[template]
453
+ return source, None, lambda: source == self.mapping.get(template)
454
+ raise TemplateNotFound(template)
455
+
456
+ def list_templates(self) -> t.List[str]:
457
+ return sorted(self.mapping)
458
+
459
+
460
+ class FunctionLoader(BaseLoader):
461
+ """A loader that is passed a function which does the loading. The
462
+ function receives the name of the template and has to return either
463
+ a string with the template source, a tuple in the form ``(source,
464
+ filename, uptodatefunc)`` or `None` if the template does not exist.
465
+
466
+ >>> def load_template(name):
467
+ ... if name == 'index.html':
468
+ ... return '...'
469
+ ...
470
+ >>> loader = FunctionLoader(load_template)
471
+
472
+ The `uptodatefunc` is a function that is called if autoreload is enabled
473
+ and has to return `True` if the template is still up to date. For more
474
+ details have a look at :meth:`BaseLoader.get_source` which has the same
475
+ return value.
476
+ """
477
+
478
+ def __init__(
479
+ self,
480
+ load_func: t.Callable[
481
+ [str],
482
+ t.Optional[
483
+ t.Union[
484
+ str, t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]
485
+ ]
486
+ ],
487
+ ],
488
+ ) -> None:
489
+ self.load_func = load_func
490
+
491
+ def get_source(
492
+ self, environment: "Environment", template: str
493
+ ) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
494
+ rv = self.load_func(template)
495
+
496
+ if rv is None:
497
+ raise TemplateNotFound(template)
498
+
499
+ if isinstance(rv, str):
500
+ return rv, None, None
501
+
502
+ return rv
503
+
504
+
505
+ class PrefixLoader(BaseLoader):
506
+ """A loader that is passed a dict of loaders where each loader is bound
507
+ to a prefix. The prefix is delimited from the template by a slash per
508
+ default, which can be changed by setting the `delimiter` argument to
509
+ something else::
510
+
511
+ loader = PrefixLoader({
512
+ 'app1': PackageLoader('mypackage.app1'),
513
+ 'app2': PackageLoader('mypackage.app2')
514
+ })
515
+
516
+ By loading ``'app1/index.html'`` the file from the app1 package is loaded,
517
+ by loading ``'app2/index.html'`` the file from the second.
518
+ """
519
+
520
+ def __init__(
521
+ self, mapping: t.Mapping[str, BaseLoader], delimiter: str = "/"
522
+ ) -> None:
523
+ self.mapping = mapping
524
+ self.delimiter = delimiter
525
+
526
+ def get_loader(self, template: str) -> t.Tuple[BaseLoader, str]:
527
+ try:
528
+ prefix, name = template.split(self.delimiter, 1)
529
+ loader = self.mapping[prefix]
530
+ except (ValueError, KeyError) as e:
531
+ raise TemplateNotFound(template) from e
532
+ return loader, name
533
+
534
+ def get_source(
535
+ self, environment: "Environment", template: str
536
+ ) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
537
+ loader, name = self.get_loader(template)
538
+ try:
539
+ return loader.get_source(environment, name)
540
+ except TemplateNotFound as e:
541
+ # re-raise the exception with the correct filename here.
542
+ # (the one that includes the prefix)
543
+ raise TemplateNotFound(template) from e
544
+
545
+ @internalcode
546
+ def load(
547
+ self,
548
+ environment: "Environment",
549
+ name: str,
550
+ globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
551
+ ) -> "Template":
552
+ loader, local_name = self.get_loader(name)
553
+ try:
554
+ return loader.load(environment, local_name, globals)
555
+ except TemplateNotFound as e:
556
+ # re-raise the exception with the correct filename here.
557
+ # (the one that includes the prefix)
558
+ raise TemplateNotFound(name) from e
559
+
560
+ def list_templates(self) -> t.List[str]:
561
+ result = []
562
+ for prefix, loader in self.mapping.items():
563
+ for template in loader.list_templates():
564
+ result.append(prefix + self.delimiter + template)
565
+ return result
566
+
567
+
568
+ class ChoiceLoader(BaseLoader):
569
+ """This loader works like the `PrefixLoader` just that no prefix is
570
+ specified. If a template could not be found by one loader the next one
571
+ is tried.
572
+
573
+ >>> loader = ChoiceLoader([
574
+ ... FileSystemLoader('/path/to/user/templates'),
575
+ ... FileSystemLoader('/path/to/system/templates')
576
+ ... ])
577
+
578
+ This is useful if you want to allow users to override builtin templates
579
+ from a different location.
580
+ """
581
+
582
+ def __init__(self, loaders: t.Sequence[BaseLoader]) -> None:
583
+ self.loaders = loaders
584
+
585
+ def get_source(
586
+ self, environment: "Environment", template: str
587
+ ) -> t.Tuple[str, t.Optional[str], t.Optional[t.Callable[[], bool]]]:
588
+ for loader in self.loaders:
589
+ try:
590
+ return loader.get_source(environment, template)
591
+ except TemplateNotFound:
592
+ pass
593
+ raise TemplateNotFound(template)
594
+
595
+ @internalcode
596
+ def load(
597
+ self,
598
+ environment: "Environment",
599
+ name: str,
600
+ globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
601
+ ) -> "Template":
602
+ for loader in self.loaders:
603
+ try:
604
+ return loader.load(environment, name, globals)
605
+ except TemplateNotFound:
606
+ pass
607
+ raise TemplateNotFound(name)
608
+
609
+ def list_templates(self) -> t.List[str]:
610
+ found = set()
611
+ for loader in self.loaders:
612
+ found.update(loader.list_templates())
613
+ return sorted(found)
614
+
615
+
616
+ class _TemplateModule(ModuleType):
617
+ """Like a normal module but with support for weak references"""
618
+
619
+
620
+ class ModuleLoader(BaseLoader):
621
+ """This loader loads templates from precompiled templates.
622
+
623
+ Example usage:
624
+
625
+ >>> loader = ModuleLoader('/path/to/compiled/templates')
626
+
627
+ Templates can be precompiled with :meth:`Environment.compile_templates`.
628
+ """
629
+
630
+ has_source_access = False
631
+
632
+ def __init__(
633
+ self,
634
+ path: t.Union[
635
+ str, "os.PathLike[str]", t.Sequence[t.Union[str, "os.PathLike[str]"]]
636
+ ],
637
+ ) -> None:
638
+ package_name = f"_jinja2_module_templates_{id(self):x}"
639
+
640
+ # create a fake module that looks for the templates in the
641
+ # path given.
642
+ mod = _TemplateModule(package_name)
643
+
644
+ if not isinstance(path, abc.Iterable) or isinstance(path, str):
645
+ path = [path]
646
+
647
+ mod.__path__ = [os.fspath(p) for p in path]
648
+
649
+ sys.modules[package_name] = weakref.proxy(
650
+ mod, lambda x: sys.modules.pop(package_name, None)
651
+ )
652
+
653
+ # the only strong reference, the sys.modules entry is weak
654
+ # so that the garbage collector can remove it once the
655
+ # loader that created it goes out of business.
656
+ self.module = mod
657
+ self.package_name = package_name
658
+
659
+ @staticmethod
660
+ def get_template_key(name: str) -> str:
661
+ return "tmpl_" + sha1(name.encode("utf-8")).hexdigest()
662
+
663
+ @staticmethod
664
+ def get_module_filename(name: str) -> str:
665
+ return ModuleLoader.get_template_key(name) + ".py"
666
+
667
+ @internalcode
668
+ def load(
669
+ self,
670
+ environment: "Environment",
671
+ name: str,
672
+ globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
673
+ ) -> "Template":
674
+ key = self.get_template_key(name)
675
+ module = f"{self.package_name}.{key}"
676
+ mod = getattr(self.module, module, None)
677
+
678
+ if mod is None:
679
+ try:
680
+ mod = __import__(module, None, None, ["root"])
681
+ except ImportError as e:
682
+ raise TemplateNotFound(name) from e
683
+
684
+ # remove the entry from sys.modules, we only want the attribute
685
+ # on the module object we have stored on the loader.
686
+ sys.modules.pop(module, None)
687
+
688
+ if globals is None:
689
+ globals = {}
690
+
691
+ return environment.template_class.from_module_dict(
692
+ environment, mod.__dict__, globals
693
+ )
TTS/Kokoro/.venv/Lib/site-packages/jinja2/meta.py ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Functions that expose information about templates that might be
2
+ interesting for introspection.
3
+ """
4
+
5
+ import typing as t
6
+
7
+ from . import nodes
8
+ from .compiler import CodeGenerator
9
+ from .compiler import Frame
10
+
11
+ if t.TYPE_CHECKING:
12
+ from .environment import Environment
13
+
14
+
15
+ class TrackingCodeGenerator(CodeGenerator):
16
+ """We abuse the code generator for introspection."""
17
+
18
+ def __init__(self, environment: "Environment") -> None:
19
+ super().__init__(environment, "<introspection>", "<introspection>")
20
+ self.undeclared_identifiers: t.Set[str] = set()
21
+
22
+ def write(self, x: str) -> None:
23
+ """Don't write."""
24
+
25
+ def enter_frame(self, frame: Frame) -> None:
26
+ """Remember all undeclared identifiers."""
27
+ super().enter_frame(frame)
28
+
29
+ for _, (action, param) in frame.symbols.loads.items():
30
+ if action == "resolve" and param not in self.environment.globals:
31
+ self.undeclared_identifiers.add(param)
32
+
33
+
34
+ def find_undeclared_variables(ast: nodes.Template) -> t.Set[str]:
35
+ """Returns a set of all variables in the AST that will be looked up from
36
+ the context at runtime. Because at compile time it's not known which
37
+ variables will be used depending on the path the execution takes at
38
+ runtime, all variables are returned.
39
+
40
+ >>> from jinja2 import Environment, meta
41
+ >>> env = Environment()
42
+ >>> ast = env.parse('{% set foo = 42 %}{{ bar + foo }}')
43
+ >>> meta.find_undeclared_variables(ast) == {'bar'}
44
+ True
45
+
46
+ .. admonition:: Implementation
47
+
48
+ Internally the code generator is used for finding undeclared variables.
49
+ This is good to know because the code generator might raise a
50
+ :exc:`TemplateAssertionError` during compilation and as a matter of
51
+ fact this function can currently raise that exception as well.
52
+ """
53
+ codegen = TrackingCodeGenerator(ast.environment) # type: ignore
54
+ codegen.visit(ast)
55
+ return codegen.undeclared_identifiers
56
+
57
+
58
+ _ref_types = (nodes.Extends, nodes.FromImport, nodes.Import, nodes.Include)
59
+ _RefType = t.Union[nodes.Extends, nodes.FromImport, nodes.Import, nodes.Include]
60
+
61
+
62
+ def find_referenced_templates(ast: nodes.Template) -> t.Iterator[t.Optional[str]]:
63
+ """Finds all the referenced templates from the AST. This will return an
64
+ iterator over all the hardcoded template extensions, inclusions and
65
+ imports. If dynamic inheritance or inclusion is used, `None` will be
66
+ yielded.
67
+
68
+ >>> from jinja2 import Environment, meta
69
+ >>> env = Environment()
70
+ >>> ast = env.parse('{% extends "layout.html" %}{% include helper %}')
71
+ >>> list(meta.find_referenced_templates(ast))
72
+ ['layout.html', None]
73
+
74
+ This function is useful for dependency tracking. For example if you want
75
+ to rebuild parts of the website after a layout template has changed.
76
+ """
77
+ template_name: t.Any
78
+
79
+ for node in ast.find_all(_ref_types):
80
+ template: nodes.Expr = node.template # type: ignore
81
+
82
+ if not isinstance(template, nodes.Const):
83
+ # a tuple with some non consts in there
84
+ if isinstance(template, (nodes.Tuple, nodes.List)):
85
+ for template_name in template.items:
86
+ # something const, only yield the strings and ignore
87
+ # non-string consts that really just make no sense
88
+ if isinstance(template_name, nodes.Const):
89
+ if isinstance(template_name.value, str):
90
+ yield template_name.value
91
+ # something dynamic in there
92
+ else:
93
+ yield None
94
+ # something dynamic we don't know about here
95
+ else:
96
+ yield None
97
+ continue
98
+ # constant is a basestring, direct template name
99
+ if isinstance(template.value, str):
100
+ yield template.value
101
+ # a tuple or list (latter *should* not happen) made of consts,
102
+ # yield the consts that are strings. We could warn here for
103
+ # non string values
104
+ elif isinstance(node, nodes.Include) and isinstance(
105
+ template.value, (tuple, list)
106
+ ):
107
+ for template_name in template.value:
108
+ if isinstance(template_name, str):
109
+ yield template_name
110
+ # something else we don't care about, we could warn here
111
+ else:
112
+ yield None
TTS/Kokoro/.venv/Lib/site-packages/jinja2/nativetypes.py ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import typing as t
2
+ from ast import literal_eval
3
+ from ast import parse
4
+ from itertools import chain
5
+ from itertools import islice
6
+ from types import GeneratorType
7
+
8
+ from . import nodes
9
+ from .compiler import CodeGenerator
10
+ from .compiler import Frame
11
+ from .compiler import has_safe_repr
12
+ from .environment import Environment
13
+ from .environment import Template
14
+
15
+
16
+ def native_concat(values: t.Iterable[t.Any]) -> t.Optional[t.Any]:
17
+ """Return a native Python type from the list of compiled nodes. If
18
+ the result is a single node, its value is returned. Otherwise, the
19
+ nodes are concatenated as strings. If the result can be parsed with
20
+ :func:`ast.literal_eval`, the parsed value is returned. Otherwise,
21
+ the string is returned.
22
+
23
+ :param values: Iterable of outputs to concatenate.
24
+ """
25
+ head = list(islice(values, 2))
26
+
27
+ if not head:
28
+ return None
29
+
30
+ if len(head) == 1:
31
+ raw = head[0]
32
+ if not isinstance(raw, str):
33
+ return raw
34
+ else:
35
+ if isinstance(values, GeneratorType):
36
+ values = chain(head, values)
37
+ raw = "".join([str(v) for v in values])
38
+
39
+ try:
40
+ return literal_eval(
41
+ # In Python 3.10+ ast.literal_eval removes leading spaces/tabs
42
+ # from the given string. For backwards compatibility we need to
43
+ # parse the string ourselves without removing leading spaces/tabs.
44
+ parse(raw, mode="eval")
45
+ )
46
+ except (ValueError, SyntaxError, MemoryError):
47
+ return raw
48
+
49
+
50
+ class NativeCodeGenerator(CodeGenerator):
51
+ """A code generator which renders Python types by not adding
52
+ ``str()`` around output nodes.
53
+ """
54
+
55
+ @staticmethod
56
+ def _default_finalize(value: t.Any) -> t.Any:
57
+ return value
58
+
59
+ def _output_const_repr(self, group: t.Iterable[t.Any]) -> str:
60
+ return repr("".join([str(v) for v in group]))
61
+
62
+ def _output_child_to_const(
63
+ self, node: nodes.Expr, frame: Frame, finalize: CodeGenerator._FinalizeInfo
64
+ ) -> t.Any:
65
+ const = node.as_const(frame.eval_ctx)
66
+
67
+ if not has_safe_repr(const):
68
+ raise nodes.Impossible()
69
+
70
+ if isinstance(node, nodes.TemplateData):
71
+ return const
72
+
73
+ return finalize.const(const) # type: ignore
74
+
75
+ def _output_child_pre(
76
+ self, node: nodes.Expr, frame: Frame, finalize: CodeGenerator._FinalizeInfo
77
+ ) -> None:
78
+ if finalize.src is not None:
79
+ self.write(finalize.src)
80
+
81
+ def _output_child_post(
82
+ self, node: nodes.Expr, frame: Frame, finalize: CodeGenerator._FinalizeInfo
83
+ ) -> None:
84
+ if finalize.src is not None:
85
+ self.write(")")
86
+
87
+
88
+ class NativeEnvironment(Environment):
89
+ """An environment that renders templates to native Python types."""
90
+
91
+ code_generator_class = NativeCodeGenerator
92
+ concat = staticmethod(native_concat) # type: ignore
93
+
94
+
95
+ class NativeTemplate(Template):
96
+ environment_class = NativeEnvironment
97
+
98
+ def render(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
99
+ """Render the template to produce a native Python type. If the
100
+ result is a single node, its value is returned. Otherwise, the
101
+ nodes are concatenated as strings. If the result can be parsed
102
+ with :func:`ast.literal_eval`, the parsed value is returned.
103
+ Otherwise, the string is returned.
104
+ """
105
+ ctx = self.new_context(dict(*args, **kwargs))
106
+
107
+ try:
108
+ return self.environment_class.concat( # type: ignore
109
+ self.root_render_func(ctx)
110
+ )
111
+ except Exception:
112
+ return self.environment.handle_exception()
113
+
114
+ async def render_async(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
115
+ if not self.environment.is_async:
116
+ raise RuntimeError(
117
+ "The environment was not created with async mode enabled."
118
+ )
119
+
120
+ ctx = self.new_context(dict(*args, **kwargs))
121
+
122
+ try:
123
+ return self.environment_class.concat( # type: ignore
124
+ [n async for n in self.root_render_func(ctx)] # type: ignore
125
+ )
126
+ except Exception:
127
+ return self.environment.handle_exception()
128
+
129
+
130
+ NativeEnvironment.template_class = NativeTemplate
TTS/Kokoro/.venv/Lib/site-packages/jinja2/nodes.py ADDED
@@ -0,0 +1,1206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """AST nodes generated by the parser for the compiler. Also provides
2
+ some node tree helper functions used by the parser and compiler in order
3
+ to normalize nodes.
4
+ """
5
+
6
+ import inspect
7
+ import operator
8
+ import typing as t
9
+ from collections import deque
10
+
11
+ from markupsafe import Markup
12
+
13
+ from .utils import _PassArg
14
+
15
+ if t.TYPE_CHECKING:
16
+ import typing_extensions as te
17
+
18
+ from .environment import Environment
19
+
20
+ _NodeBound = t.TypeVar("_NodeBound", bound="Node")
21
+
22
+ _binop_to_func: t.Dict[str, t.Callable[[t.Any, t.Any], t.Any]] = {
23
+ "*": operator.mul,
24
+ "/": operator.truediv,
25
+ "//": operator.floordiv,
26
+ "**": operator.pow,
27
+ "%": operator.mod,
28
+ "+": operator.add,
29
+ "-": operator.sub,
30
+ }
31
+
32
+ _uaop_to_func: t.Dict[str, t.Callable[[t.Any], t.Any]] = {
33
+ "not": operator.not_,
34
+ "+": operator.pos,
35
+ "-": operator.neg,
36
+ }
37
+
38
+ _cmpop_to_func: t.Dict[str, t.Callable[[t.Any, t.Any], t.Any]] = {
39
+ "eq": operator.eq,
40
+ "ne": operator.ne,
41
+ "gt": operator.gt,
42
+ "gteq": operator.ge,
43
+ "lt": operator.lt,
44
+ "lteq": operator.le,
45
+ "in": lambda a, b: a in b,
46
+ "notin": lambda a, b: a not in b,
47
+ }
48
+
49
+
50
+ class Impossible(Exception):
51
+ """Raised if the node could not perform a requested action."""
52
+
53
+
54
+ class NodeType(type):
55
+ """A metaclass for nodes that handles the field and attribute
56
+ inheritance. fields and attributes from the parent class are
57
+ automatically forwarded to the child."""
58
+
59
+ def __new__(mcs, name, bases, d): # type: ignore
60
+ for attr in "fields", "attributes":
61
+ storage: t.List[t.Tuple[str, ...]] = []
62
+ storage.extend(getattr(bases[0] if bases else object, attr, ()))
63
+ storage.extend(d.get(attr, ()))
64
+ assert len(bases) <= 1, "multiple inheritance not allowed"
65
+ assert len(storage) == len(set(storage)), "layout conflict"
66
+ d[attr] = tuple(storage)
67
+ d.setdefault("abstract", False)
68
+ return type.__new__(mcs, name, bases, d)
69
+
70
+
71
+ class EvalContext:
72
+ """Holds evaluation time information. Custom attributes can be attached
73
+ to it in extensions.
74
+ """
75
+
76
+ def __init__(
77
+ self, environment: "Environment", template_name: t.Optional[str] = None
78
+ ) -> None:
79
+ self.environment = environment
80
+ if callable(environment.autoescape):
81
+ self.autoescape = environment.autoescape(template_name)
82
+ else:
83
+ self.autoescape = environment.autoescape
84
+ self.volatile = False
85
+
86
+ def save(self) -> t.Mapping[str, t.Any]:
87
+ return self.__dict__.copy()
88
+
89
+ def revert(self, old: t.Mapping[str, t.Any]) -> None:
90
+ self.__dict__.clear()
91
+ self.__dict__.update(old)
92
+
93
+
94
+ def get_eval_context(node: "Node", ctx: t.Optional[EvalContext]) -> EvalContext:
95
+ if ctx is None:
96
+ if node.environment is None:
97
+ raise RuntimeError(
98
+ "if no eval context is passed, the node must have an"
99
+ " attached environment."
100
+ )
101
+ return EvalContext(node.environment)
102
+ return ctx
103
+
104
+
105
+ class Node(metaclass=NodeType):
106
+ """Baseclass for all Jinja nodes. There are a number of nodes available
107
+ of different types. There are four major types:
108
+
109
+ - :class:`Stmt`: statements
110
+ - :class:`Expr`: expressions
111
+ - :class:`Helper`: helper nodes
112
+ - :class:`Template`: the outermost wrapper node
113
+
114
+ All nodes have fields and attributes. Fields may be other nodes, lists,
115
+ or arbitrary values. Fields are passed to the constructor as regular
116
+ positional arguments, attributes as keyword arguments. Each node has
117
+ two attributes: `lineno` (the line number of the node) and `environment`.
118
+ The `environment` attribute is set at the end of the parsing process for
119
+ all nodes automatically.
120
+ """
121
+
122
+ fields: t.Tuple[str, ...] = ()
123
+ attributes: t.Tuple[str, ...] = ("lineno", "environment")
124
+ abstract = True
125
+
126
+ lineno: int
127
+ environment: t.Optional["Environment"]
128
+
129
+ def __init__(self, *fields: t.Any, **attributes: t.Any) -> None:
130
+ if self.abstract:
131
+ raise TypeError("abstract nodes are not instantiable")
132
+ if fields:
133
+ if len(fields) != len(self.fields):
134
+ if not self.fields:
135
+ raise TypeError(f"{type(self).__name__!r} takes 0 arguments")
136
+ raise TypeError(
137
+ f"{type(self).__name__!r} takes 0 or {len(self.fields)}"
138
+ f" argument{'s' if len(self.fields) != 1 else ''}"
139
+ )
140
+ for name, arg in zip(self.fields, fields):
141
+ setattr(self, name, arg)
142
+ for attr in self.attributes:
143
+ setattr(self, attr, attributes.pop(attr, None))
144
+ if attributes:
145
+ raise TypeError(f"unknown attribute {next(iter(attributes))!r}")
146
+
147
+ def iter_fields(
148
+ self,
149
+ exclude: t.Optional[t.Container[str]] = None,
150
+ only: t.Optional[t.Container[str]] = None,
151
+ ) -> t.Iterator[t.Tuple[str, t.Any]]:
152
+ """This method iterates over all fields that are defined and yields
153
+ ``(key, value)`` tuples. Per default all fields are returned, but
154
+ it's possible to limit that to some fields by providing the `only`
155
+ parameter or to exclude some using the `exclude` parameter. Both
156
+ should be sets or tuples of field names.
157
+ """
158
+ for name in self.fields:
159
+ if (
160
+ (exclude is None and only is None)
161
+ or (exclude is not None and name not in exclude)
162
+ or (only is not None and name in only)
163
+ ):
164
+ try:
165
+ yield name, getattr(self, name)
166
+ except AttributeError:
167
+ pass
168
+
169
+ def iter_child_nodes(
170
+ self,
171
+ exclude: t.Optional[t.Container[str]] = None,
172
+ only: t.Optional[t.Container[str]] = None,
173
+ ) -> t.Iterator["Node"]:
174
+ """Iterates over all direct child nodes of the node. This iterates
175
+ over all fields and yields the values of they are nodes. If the value
176
+ of a field is a list all the nodes in that list are returned.
177
+ """
178
+ for _, item in self.iter_fields(exclude, only):
179
+ if isinstance(item, list):
180
+ for n in item:
181
+ if isinstance(n, Node):
182
+ yield n
183
+ elif isinstance(item, Node):
184
+ yield item
185
+
186
+ def find(self, node_type: t.Type[_NodeBound]) -> t.Optional[_NodeBound]:
187
+ """Find the first node of a given type. If no such node exists the
188
+ return value is `None`.
189
+ """
190
+ for result in self.find_all(node_type):
191
+ return result
192
+
193
+ return None
194
+
195
+ def find_all(
196
+ self, node_type: t.Union[t.Type[_NodeBound], t.Tuple[t.Type[_NodeBound], ...]]
197
+ ) -> t.Iterator[_NodeBound]:
198
+ """Find all the nodes of a given type. If the type is a tuple,
199
+ the check is performed for any of the tuple items.
200
+ """
201
+ for child in self.iter_child_nodes():
202
+ if isinstance(child, node_type):
203
+ yield child # type: ignore
204
+ yield from child.find_all(node_type)
205
+
206
+ def set_ctx(self, ctx: str) -> "Node":
207
+ """Reset the context of a node and all child nodes. Per default the
208
+ parser will all generate nodes that have a 'load' context as it's the
209
+ most common one. This method is used in the parser to set assignment
210
+ targets and other nodes to a store context.
211
+ """
212
+ todo = deque([self])
213
+ while todo:
214
+ node = todo.popleft()
215
+ if "ctx" in node.fields:
216
+ node.ctx = ctx # type: ignore
217
+ todo.extend(node.iter_child_nodes())
218
+ return self
219
+
220
+ def set_lineno(self, lineno: int, override: bool = False) -> "Node":
221
+ """Set the line numbers of the node and children."""
222
+ todo = deque([self])
223
+ while todo:
224
+ node = todo.popleft()
225
+ if "lineno" in node.attributes:
226
+ if node.lineno is None or override:
227
+ node.lineno = lineno
228
+ todo.extend(node.iter_child_nodes())
229
+ return self
230
+
231
+ def set_environment(self, environment: "Environment") -> "Node":
232
+ """Set the environment for all nodes."""
233
+ todo = deque([self])
234
+ while todo:
235
+ node = todo.popleft()
236
+ node.environment = environment
237
+ todo.extend(node.iter_child_nodes())
238
+ return self
239
+
240
+ def __eq__(self, other: t.Any) -> bool:
241
+ if type(self) is not type(other):
242
+ return NotImplemented
243
+
244
+ return tuple(self.iter_fields()) == tuple(other.iter_fields())
245
+
246
+ __hash__ = object.__hash__
247
+
248
+ def __repr__(self) -> str:
249
+ args_str = ", ".join(f"{a}={getattr(self, a, None)!r}" for a in self.fields)
250
+ return f"{type(self).__name__}({args_str})"
251
+
252
+ def dump(self) -> str:
253
+ def _dump(node: t.Union[Node, t.Any]) -> None:
254
+ if not isinstance(node, Node):
255
+ buf.append(repr(node))
256
+ return
257
+
258
+ buf.append(f"nodes.{type(node).__name__}(")
259
+ if not node.fields:
260
+ buf.append(")")
261
+ return
262
+ for idx, field in enumerate(node.fields):
263
+ if idx:
264
+ buf.append(", ")
265
+ value = getattr(node, field)
266
+ if isinstance(value, list):
267
+ buf.append("[")
268
+ for idx, item in enumerate(value):
269
+ if idx:
270
+ buf.append(", ")
271
+ _dump(item)
272
+ buf.append("]")
273
+ else:
274
+ _dump(value)
275
+ buf.append(")")
276
+
277
+ buf: t.List[str] = []
278
+ _dump(self)
279
+ return "".join(buf)
280
+
281
+
282
+ class Stmt(Node):
283
+ """Base node for all statements."""
284
+
285
+ abstract = True
286
+
287
+
288
+ class Helper(Node):
289
+ """Nodes that exist in a specific context only."""
290
+
291
+ abstract = True
292
+
293
+
294
+ class Template(Node):
295
+ """Node that represents a template. This must be the outermost node that
296
+ is passed to the compiler.
297
+ """
298
+
299
+ fields = ("body",)
300
+ body: t.List[Node]
301
+
302
+
303
+ class Output(Stmt):
304
+ """A node that holds multiple expressions which are then printed out.
305
+ This is used both for the `print` statement and the regular template data.
306
+ """
307
+
308
+ fields = ("nodes",)
309
+ nodes: t.List["Expr"]
310
+
311
+
312
+ class Extends(Stmt):
313
+ """Represents an extends statement."""
314
+
315
+ fields = ("template",)
316
+ template: "Expr"
317
+
318
+
319
+ class For(Stmt):
320
+ """The for loop. `target` is the target for the iteration (usually a
321
+ :class:`Name` or :class:`Tuple`), `iter` the iterable. `body` is a list
322
+ of nodes that are used as loop-body, and `else_` a list of nodes for the
323
+ `else` block. If no else node exists it has to be an empty list.
324
+
325
+ For filtered nodes an expression can be stored as `test`, otherwise `None`.
326
+ """
327
+
328
+ fields = ("target", "iter", "body", "else_", "test", "recursive")
329
+ target: Node
330
+ iter: Node
331
+ body: t.List[Node]
332
+ else_: t.List[Node]
333
+ test: t.Optional[Node]
334
+ recursive: bool
335
+
336
+
337
+ class If(Stmt):
338
+ """If `test` is true, `body` is rendered, else `else_`."""
339
+
340
+ fields = ("test", "body", "elif_", "else_")
341
+ test: Node
342
+ body: t.List[Node]
343
+ elif_: t.List["If"]
344
+ else_: t.List[Node]
345
+
346
+
347
+ class Macro(Stmt):
348
+ """A macro definition. `name` is the name of the macro, `args` a list of
349
+ arguments and `defaults` a list of defaults if there are any. `body` is
350
+ a list of nodes for the macro body.
351
+ """
352
+
353
+ fields = ("name", "args", "defaults", "body")
354
+ name: str
355
+ args: t.List["Name"]
356
+ defaults: t.List["Expr"]
357
+ body: t.List[Node]
358
+
359
+
360
+ class CallBlock(Stmt):
361
+ """Like a macro without a name but a call instead. `call` is called with
362
+ the unnamed macro as `caller` argument this node holds.
363
+ """
364
+
365
+ fields = ("call", "args", "defaults", "body")
366
+ call: "Call"
367
+ args: t.List["Name"]
368
+ defaults: t.List["Expr"]
369
+ body: t.List[Node]
370
+
371
+
372
+ class FilterBlock(Stmt):
373
+ """Node for filter sections."""
374
+
375
+ fields = ("body", "filter")
376
+ body: t.List[Node]
377
+ filter: "Filter"
378
+
379
+
380
+ class With(Stmt):
381
+ """Specific node for with statements. In older versions of Jinja the
382
+ with statement was implemented on the base of the `Scope` node instead.
383
+
384
+ .. versionadded:: 2.9.3
385
+ """
386
+
387
+ fields = ("targets", "values", "body")
388
+ targets: t.List["Expr"]
389
+ values: t.List["Expr"]
390
+ body: t.List[Node]
391
+
392
+
393
+ class Block(Stmt):
394
+ """A node that represents a block.
395
+
396
+ .. versionchanged:: 3.0.0
397
+ the `required` field was added.
398
+ """
399
+
400
+ fields = ("name", "body", "scoped", "required")
401
+ name: str
402
+ body: t.List[Node]
403
+ scoped: bool
404
+ required: bool
405
+
406
+
407
+ class Include(Stmt):
408
+ """A node that represents the include tag."""
409
+
410
+ fields = ("template", "with_context", "ignore_missing")
411
+ template: "Expr"
412
+ with_context: bool
413
+ ignore_missing: bool
414
+
415
+
416
+ class Import(Stmt):
417
+ """A node that represents the import tag."""
418
+
419
+ fields = ("template", "target", "with_context")
420
+ template: "Expr"
421
+ target: str
422
+ with_context: bool
423
+
424
+
425
+ class FromImport(Stmt):
426
+ """A node that represents the from import tag. It's important to not
427
+ pass unsafe names to the name attribute. The compiler translates the
428
+ attribute lookups directly into getattr calls and does *not* use the
429
+ subscript callback of the interface. As exported variables may not
430
+ start with double underscores (which the parser asserts) this is not a
431
+ problem for regular Jinja code, but if this node is used in an extension
432
+ extra care must be taken.
433
+
434
+ The list of names may contain tuples if aliases are wanted.
435
+ """
436
+
437
+ fields = ("template", "names", "with_context")
438
+ template: "Expr"
439
+ names: t.List[t.Union[str, t.Tuple[str, str]]]
440
+ with_context: bool
441
+
442
+
443
+ class ExprStmt(Stmt):
444
+ """A statement that evaluates an expression and discards the result."""
445
+
446
+ fields = ("node",)
447
+ node: Node
448
+
449
+
450
+ class Assign(Stmt):
451
+ """Assigns an expression to a target."""
452
+
453
+ fields = ("target", "node")
454
+ target: "Expr"
455
+ node: Node
456
+
457
+
458
+ class AssignBlock(Stmt):
459
+ """Assigns a block to a target."""
460
+
461
+ fields = ("target", "filter", "body")
462
+ target: "Expr"
463
+ filter: t.Optional["Filter"]
464
+ body: t.List[Node]
465
+
466
+
467
+ class Expr(Node):
468
+ """Baseclass for all expressions."""
469
+
470
+ abstract = True
471
+
472
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
473
+ """Return the value of the expression as constant or raise
474
+ :exc:`Impossible` if this was not possible.
475
+
476
+ An :class:`EvalContext` can be provided, if none is given
477
+ a default context is created which requires the nodes to have
478
+ an attached environment.
479
+
480
+ .. versionchanged:: 2.4
481
+ the `eval_ctx` parameter was added.
482
+ """
483
+ raise Impossible()
484
+
485
+ def can_assign(self) -> bool:
486
+ """Check if it's possible to assign something to this node."""
487
+ return False
488
+
489
+
490
+ class BinExpr(Expr):
491
+ """Baseclass for all binary expressions."""
492
+
493
+ fields = ("left", "right")
494
+ left: Expr
495
+ right: Expr
496
+ operator: str
497
+ abstract = True
498
+
499
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
500
+ eval_ctx = get_eval_context(self, eval_ctx)
501
+
502
+ # intercepted operators cannot be folded at compile time
503
+ if (
504
+ eval_ctx.environment.sandboxed
505
+ and self.operator in eval_ctx.environment.intercepted_binops # type: ignore
506
+ ):
507
+ raise Impossible()
508
+ f = _binop_to_func[self.operator]
509
+ try:
510
+ return f(self.left.as_const(eval_ctx), self.right.as_const(eval_ctx))
511
+ except Exception as e:
512
+ raise Impossible() from e
513
+
514
+
515
+ class UnaryExpr(Expr):
516
+ """Baseclass for all unary expressions."""
517
+
518
+ fields = ("node",)
519
+ node: Expr
520
+ operator: str
521
+ abstract = True
522
+
523
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
524
+ eval_ctx = get_eval_context(self, eval_ctx)
525
+
526
+ # intercepted operators cannot be folded at compile time
527
+ if (
528
+ eval_ctx.environment.sandboxed
529
+ and self.operator in eval_ctx.environment.intercepted_unops # type: ignore
530
+ ):
531
+ raise Impossible()
532
+ f = _uaop_to_func[self.operator]
533
+ try:
534
+ return f(self.node.as_const(eval_ctx))
535
+ except Exception as e:
536
+ raise Impossible() from e
537
+
538
+
539
+ class Name(Expr):
540
+ """Looks up a name or stores a value in a name.
541
+ The `ctx` of the node can be one of the following values:
542
+
543
+ - `store`: store a value in the name
544
+ - `load`: load that name
545
+ - `param`: like `store` but if the name was defined as function parameter.
546
+ """
547
+
548
+ fields = ("name", "ctx")
549
+ name: str
550
+ ctx: str
551
+
552
+ def can_assign(self) -> bool:
553
+ return self.name not in {"true", "false", "none", "True", "False", "None"}
554
+
555
+
556
+ class NSRef(Expr):
557
+ """Reference to a namespace value assignment"""
558
+
559
+ fields = ("name", "attr")
560
+ name: str
561
+ attr: str
562
+
563
+ def can_assign(self) -> bool:
564
+ # We don't need any special checks here; NSRef assignments have a
565
+ # runtime check to ensure the target is a namespace object which will
566
+ # have been checked already as it is created using a normal assignment
567
+ # which goes through a `Name` node.
568
+ return True
569
+
570
+
571
+ class Literal(Expr):
572
+ """Baseclass for literals."""
573
+
574
+ abstract = True
575
+
576
+
577
+ class Const(Literal):
578
+ """All constant values. The parser will return this node for simple
579
+ constants such as ``42`` or ``"foo"`` but it can be used to store more
580
+ complex values such as lists too. Only constants with a safe
581
+ representation (objects where ``eval(repr(x)) == x`` is true).
582
+ """
583
+
584
+ fields = ("value",)
585
+ value: t.Any
586
+
587
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
588
+ return self.value
589
+
590
+ @classmethod
591
+ def from_untrusted(
592
+ cls,
593
+ value: t.Any,
594
+ lineno: t.Optional[int] = None,
595
+ environment: "t.Optional[Environment]" = None,
596
+ ) -> "Const":
597
+ """Return a const object if the value is representable as
598
+ constant value in the generated code, otherwise it will raise
599
+ an `Impossible` exception.
600
+ """
601
+ from .compiler import has_safe_repr
602
+
603
+ if not has_safe_repr(value):
604
+ raise Impossible()
605
+ return cls(value, lineno=lineno, environment=environment)
606
+
607
+
608
+ class TemplateData(Literal):
609
+ """A constant template string."""
610
+
611
+ fields = ("data",)
612
+ data: str
613
+
614
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> str:
615
+ eval_ctx = get_eval_context(self, eval_ctx)
616
+ if eval_ctx.volatile:
617
+ raise Impossible()
618
+ if eval_ctx.autoescape:
619
+ return Markup(self.data)
620
+ return self.data
621
+
622
+
623
+ class Tuple(Literal):
624
+ """For loop unpacking and some other things like multiple arguments
625
+ for subscripts. Like for :class:`Name` `ctx` specifies if the tuple
626
+ is used for loading the names or storing.
627
+ """
628
+
629
+ fields = ("items", "ctx")
630
+ items: t.List[Expr]
631
+ ctx: str
632
+
633
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Tuple[t.Any, ...]:
634
+ eval_ctx = get_eval_context(self, eval_ctx)
635
+ return tuple(x.as_const(eval_ctx) for x in self.items)
636
+
637
+ def can_assign(self) -> bool:
638
+ for item in self.items:
639
+ if not item.can_assign():
640
+ return False
641
+ return True
642
+
643
+
644
+ class List(Literal):
645
+ """Any list literal such as ``[1, 2, 3]``"""
646
+
647
+ fields = ("items",)
648
+ items: t.List[Expr]
649
+
650
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.List[t.Any]:
651
+ eval_ctx = get_eval_context(self, eval_ctx)
652
+ return [x.as_const(eval_ctx) for x in self.items]
653
+
654
+
655
+ class Dict(Literal):
656
+ """Any dict literal such as ``{1: 2, 3: 4}``. The items must be a list of
657
+ :class:`Pair` nodes.
658
+ """
659
+
660
+ fields = ("items",)
661
+ items: t.List["Pair"]
662
+
663
+ def as_const(
664
+ self, eval_ctx: t.Optional[EvalContext] = None
665
+ ) -> t.Dict[t.Any, t.Any]:
666
+ eval_ctx = get_eval_context(self, eval_ctx)
667
+ return dict(x.as_const(eval_ctx) for x in self.items)
668
+
669
+
670
+ class Pair(Helper):
671
+ """A key, value pair for dicts."""
672
+
673
+ fields = ("key", "value")
674
+ key: Expr
675
+ value: Expr
676
+
677
+ def as_const(
678
+ self, eval_ctx: t.Optional[EvalContext] = None
679
+ ) -> t.Tuple[t.Any, t.Any]:
680
+ eval_ctx = get_eval_context(self, eval_ctx)
681
+ return self.key.as_const(eval_ctx), self.value.as_const(eval_ctx)
682
+
683
+
684
+ class Keyword(Helper):
685
+ """A key, value pair for keyword arguments where key is a string."""
686
+
687
+ fields = ("key", "value")
688
+ key: str
689
+ value: Expr
690
+
691
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Tuple[str, t.Any]:
692
+ eval_ctx = get_eval_context(self, eval_ctx)
693
+ return self.key, self.value.as_const(eval_ctx)
694
+
695
+
696
+ class CondExpr(Expr):
697
+ """A conditional expression (inline if expression). (``{{
698
+ foo if bar else baz }}``)
699
+ """
700
+
701
+ fields = ("test", "expr1", "expr2")
702
+ test: Expr
703
+ expr1: Expr
704
+ expr2: t.Optional[Expr]
705
+
706
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
707
+ eval_ctx = get_eval_context(self, eval_ctx)
708
+ if self.test.as_const(eval_ctx):
709
+ return self.expr1.as_const(eval_ctx)
710
+
711
+ # if we evaluate to an undefined object, we better do that at runtime
712
+ if self.expr2 is None:
713
+ raise Impossible()
714
+
715
+ return self.expr2.as_const(eval_ctx)
716
+
717
+
718
+ def args_as_const(
719
+ node: t.Union["_FilterTestCommon", "Call"], eval_ctx: t.Optional[EvalContext]
720
+ ) -> t.Tuple[t.List[t.Any], t.Dict[t.Any, t.Any]]:
721
+ args = [x.as_const(eval_ctx) for x in node.args]
722
+ kwargs = dict(x.as_const(eval_ctx) for x in node.kwargs)
723
+
724
+ if node.dyn_args is not None:
725
+ try:
726
+ args.extend(node.dyn_args.as_const(eval_ctx))
727
+ except Exception as e:
728
+ raise Impossible() from e
729
+
730
+ if node.dyn_kwargs is not None:
731
+ try:
732
+ kwargs.update(node.dyn_kwargs.as_const(eval_ctx))
733
+ except Exception as e:
734
+ raise Impossible() from e
735
+
736
+ return args, kwargs
737
+
738
+
739
+ class _FilterTestCommon(Expr):
740
+ fields = ("node", "name", "args", "kwargs", "dyn_args", "dyn_kwargs")
741
+ node: Expr
742
+ name: str
743
+ args: t.List[Expr]
744
+ kwargs: t.List[Pair]
745
+ dyn_args: t.Optional[Expr]
746
+ dyn_kwargs: t.Optional[Expr]
747
+ abstract = True
748
+ _is_filter = True
749
+
750
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
751
+ eval_ctx = get_eval_context(self, eval_ctx)
752
+
753
+ if eval_ctx.volatile:
754
+ raise Impossible()
755
+
756
+ if self._is_filter:
757
+ env_map = eval_ctx.environment.filters
758
+ else:
759
+ env_map = eval_ctx.environment.tests
760
+
761
+ func = env_map.get(self.name)
762
+ pass_arg = _PassArg.from_obj(func) # type: ignore
763
+
764
+ if func is None or pass_arg is _PassArg.context:
765
+ raise Impossible()
766
+
767
+ if eval_ctx.environment.is_async and (
768
+ getattr(func, "jinja_async_variant", False) is True
769
+ or inspect.iscoroutinefunction(func)
770
+ ):
771
+ raise Impossible()
772
+
773
+ args, kwargs = args_as_const(self, eval_ctx)
774
+ args.insert(0, self.node.as_const(eval_ctx))
775
+
776
+ if pass_arg is _PassArg.eval_context:
777
+ args.insert(0, eval_ctx)
778
+ elif pass_arg is _PassArg.environment:
779
+ args.insert(0, eval_ctx.environment)
780
+
781
+ try:
782
+ return func(*args, **kwargs)
783
+ except Exception as e:
784
+ raise Impossible() from e
785
+
786
+
787
+ class Filter(_FilterTestCommon):
788
+ """Apply a filter to an expression. ``name`` is the name of the
789
+ filter, the other fields are the same as :class:`Call`.
790
+
791
+ If ``node`` is ``None``, the filter is being used in a filter block
792
+ and is applied to the content of the block.
793
+ """
794
+
795
+ node: t.Optional[Expr] # type: ignore
796
+
797
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
798
+ if self.node is None:
799
+ raise Impossible()
800
+
801
+ return super().as_const(eval_ctx=eval_ctx)
802
+
803
+
804
+ class Test(_FilterTestCommon):
805
+ """Apply a test to an expression. ``name`` is the name of the test,
806
+ the other field are the same as :class:`Call`.
807
+
808
+ .. versionchanged:: 3.0
809
+ ``as_const`` shares the same logic for filters and tests. Tests
810
+ check for volatile, async, and ``@pass_context`` etc.
811
+ decorators.
812
+ """
813
+
814
+ _is_filter = False
815
+
816
+
817
+ class Call(Expr):
818
+ """Calls an expression. `args` is a list of arguments, `kwargs` a list
819
+ of keyword arguments (list of :class:`Keyword` nodes), and `dyn_args`
820
+ and `dyn_kwargs` has to be either `None` or a node that is used as
821
+ node for dynamic positional (``*args``) or keyword (``**kwargs``)
822
+ arguments.
823
+ """
824
+
825
+ fields = ("node", "args", "kwargs", "dyn_args", "dyn_kwargs")
826
+ node: Expr
827
+ args: t.List[Expr]
828
+ kwargs: t.List[Keyword]
829
+ dyn_args: t.Optional[Expr]
830
+ dyn_kwargs: t.Optional[Expr]
831
+
832
+
833
+ class Getitem(Expr):
834
+ """Get an attribute or item from an expression and prefer the item."""
835
+
836
+ fields = ("node", "arg", "ctx")
837
+ node: Expr
838
+ arg: Expr
839
+ ctx: str
840
+
841
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
842
+ if self.ctx != "load":
843
+ raise Impossible()
844
+
845
+ eval_ctx = get_eval_context(self, eval_ctx)
846
+
847
+ try:
848
+ return eval_ctx.environment.getitem(
849
+ self.node.as_const(eval_ctx), self.arg.as_const(eval_ctx)
850
+ )
851
+ except Exception as e:
852
+ raise Impossible() from e
853
+
854
+
855
+ class Getattr(Expr):
856
+ """Get an attribute or item from an expression that is a ascii-only
857
+ bytestring and prefer the attribute.
858
+ """
859
+
860
+ fields = ("node", "attr", "ctx")
861
+ node: Expr
862
+ attr: str
863
+ ctx: str
864
+
865
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
866
+ if self.ctx != "load":
867
+ raise Impossible()
868
+
869
+ eval_ctx = get_eval_context(self, eval_ctx)
870
+
871
+ try:
872
+ return eval_ctx.environment.getattr(self.node.as_const(eval_ctx), self.attr)
873
+ except Exception as e:
874
+ raise Impossible() from e
875
+
876
+
877
+ class Slice(Expr):
878
+ """Represents a slice object. This must only be used as argument for
879
+ :class:`Subscript`.
880
+ """
881
+
882
+ fields = ("start", "stop", "step")
883
+ start: t.Optional[Expr]
884
+ stop: t.Optional[Expr]
885
+ step: t.Optional[Expr]
886
+
887
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> slice:
888
+ eval_ctx = get_eval_context(self, eval_ctx)
889
+
890
+ def const(obj: t.Optional[Expr]) -> t.Optional[t.Any]:
891
+ if obj is None:
892
+ return None
893
+ return obj.as_const(eval_ctx)
894
+
895
+ return slice(const(self.start), const(self.stop), const(self.step))
896
+
897
+
898
+ class Concat(Expr):
899
+ """Concatenates the list of expressions provided after converting
900
+ them to strings.
901
+ """
902
+
903
+ fields = ("nodes",)
904
+ nodes: t.List[Expr]
905
+
906
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> str:
907
+ eval_ctx = get_eval_context(self, eval_ctx)
908
+ return "".join(str(x.as_const(eval_ctx)) for x in self.nodes)
909
+
910
+
911
+ class Compare(Expr):
912
+ """Compares an expression with some other expressions. `ops` must be a
913
+ list of :class:`Operand`\\s.
914
+ """
915
+
916
+ fields = ("expr", "ops")
917
+ expr: Expr
918
+ ops: t.List["Operand"]
919
+
920
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
921
+ eval_ctx = get_eval_context(self, eval_ctx)
922
+ result = value = self.expr.as_const(eval_ctx)
923
+
924
+ try:
925
+ for op in self.ops:
926
+ new_value = op.expr.as_const(eval_ctx)
927
+ result = _cmpop_to_func[op.op](value, new_value)
928
+
929
+ if not result:
930
+ return False
931
+
932
+ value = new_value
933
+ except Exception as e:
934
+ raise Impossible() from e
935
+
936
+ return result
937
+
938
+
939
+ class Operand(Helper):
940
+ """Holds an operator and an expression."""
941
+
942
+ fields = ("op", "expr")
943
+ op: str
944
+ expr: Expr
945
+
946
+
947
+ class Mul(BinExpr):
948
+ """Multiplies the left with the right node."""
949
+
950
+ operator = "*"
951
+
952
+
953
+ class Div(BinExpr):
954
+ """Divides the left by the right node."""
955
+
956
+ operator = "/"
957
+
958
+
959
+ class FloorDiv(BinExpr):
960
+ """Divides the left by the right node and converts the
961
+ result into an integer by truncating.
962
+ """
963
+
964
+ operator = "//"
965
+
966
+
967
+ class Add(BinExpr):
968
+ """Add the left to the right node."""
969
+
970
+ operator = "+"
971
+
972
+
973
+ class Sub(BinExpr):
974
+ """Subtract the right from the left node."""
975
+
976
+ operator = "-"
977
+
978
+
979
+ class Mod(BinExpr):
980
+ """Left modulo right."""
981
+
982
+ operator = "%"
983
+
984
+
985
+ class Pow(BinExpr):
986
+ """Left to the power of right."""
987
+
988
+ operator = "**"
989
+
990
+
991
+ class And(BinExpr):
992
+ """Short circuited AND."""
993
+
994
+ operator = "and"
995
+
996
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
997
+ eval_ctx = get_eval_context(self, eval_ctx)
998
+ return self.left.as_const(eval_ctx) and self.right.as_const(eval_ctx)
999
+
1000
+
1001
+ class Or(BinExpr):
1002
+ """Short circuited OR."""
1003
+
1004
+ operator = "or"
1005
+
1006
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> t.Any:
1007
+ eval_ctx = get_eval_context(self, eval_ctx)
1008
+ return self.left.as_const(eval_ctx) or self.right.as_const(eval_ctx)
1009
+
1010
+
1011
+ class Not(UnaryExpr):
1012
+ """Negate the expression."""
1013
+
1014
+ operator = "not"
1015
+
1016
+
1017
+ class Neg(UnaryExpr):
1018
+ """Make the expression negative."""
1019
+
1020
+ operator = "-"
1021
+
1022
+
1023
+ class Pos(UnaryExpr):
1024
+ """Make the expression positive (noop for most expressions)"""
1025
+
1026
+ operator = "+"
1027
+
1028
+
1029
+ # Helpers for extensions
1030
+
1031
+
1032
+ class EnvironmentAttribute(Expr):
1033
+ """Loads an attribute from the environment object. This is useful for
1034
+ extensions that want to call a callback stored on the environment.
1035
+ """
1036
+
1037
+ fields = ("name",)
1038
+ name: str
1039
+
1040
+
1041
+ class ExtensionAttribute(Expr):
1042
+ """Returns the attribute of an extension bound to the environment.
1043
+ The identifier is the identifier of the :class:`Extension`.
1044
+
1045
+ This node is usually constructed by calling the
1046
+ :meth:`~jinja2.ext.Extension.attr` method on an extension.
1047
+ """
1048
+
1049
+ fields = ("identifier", "name")
1050
+ identifier: str
1051
+ name: str
1052
+
1053
+
1054
+ class ImportedName(Expr):
1055
+ """If created with an import name the import name is returned on node
1056
+ access. For example ``ImportedName('cgi.escape')`` returns the `escape`
1057
+ function from the cgi module on evaluation. Imports are optimized by the
1058
+ compiler so there is no need to assign them to local variables.
1059
+ """
1060
+
1061
+ fields = ("importname",)
1062
+ importname: str
1063
+
1064
+
1065
+ class InternalName(Expr):
1066
+ """An internal name in the compiler. You cannot create these nodes
1067
+ yourself but the parser provides a
1068
+ :meth:`~jinja2.parser.Parser.free_identifier` method that creates
1069
+ a new identifier for you. This identifier is not available from the
1070
+ template and is not treated specially by the compiler.
1071
+ """
1072
+
1073
+ fields = ("name",)
1074
+ name: str
1075
+
1076
+ def __init__(self) -> None:
1077
+ raise TypeError(
1078
+ "Can't create internal names. Use the "
1079
+ "`free_identifier` method on a parser."
1080
+ )
1081
+
1082
+
1083
+ class MarkSafe(Expr):
1084
+ """Mark the wrapped expression as safe (wrap it as `Markup`)."""
1085
+
1086
+ fields = ("expr",)
1087
+ expr: Expr
1088
+
1089
+ def as_const(self, eval_ctx: t.Optional[EvalContext] = None) -> Markup:
1090
+ eval_ctx = get_eval_context(self, eval_ctx)
1091
+ return Markup(self.expr.as_const(eval_ctx))
1092
+
1093
+
1094
+ class MarkSafeIfAutoescape(Expr):
1095
+ """Mark the wrapped expression as safe (wrap it as `Markup`) but
1096
+ only if autoescaping is active.
1097
+
1098
+ .. versionadded:: 2.5
1099
+ """
1100
+
1101
+ fields = ("expr",)
1102
+ expr: Expr
1103
+
1104
+ def as_const(
1105
+ self, eval_ctx: t.Optional[EvalContext] = None
1106
+ ) -> t.Union[Markup, t.Any]:
1107
+ eval_ctx = get_eval_context(self, eval_ctx)
1108
+ if eval_ctx.volatile:
1109
+ raise Impossible()
1110
+ expr = self.expr.as_const(eval_ctx)
1111
+ if eval_ctx.autoescape:
1112
+ return Markup(expr)
1113
+ return expr
1114
+
1115
+
1116
+ class ContextReference(Expr):
1117
+ """Returns the current template context. It can be used like a
1118
+ :class:`Name` node, with a ``'load'`` ctx and will return the
1119
+ current :class:`~jinja2.runtime.Context` object.
1120
+
1121
+ Here an example that assigns the current template name to a
1122
+ variable named `foo`::
1123
+
1124
+ Assign(Name('foo', ctx='store'),
1125
+ Getattr(ContextReference(), 'name'))
1126
+
1127
+ This is basically equivalent to using the
1128
+ :func:`~jinja2.pass_context` decorator when using the high-level
1129
+ API, which causes a reference to the context to be passed as the
1130
+ first argument to a function.
1131
+ """
1132
+
1133
+
1134
+ class DerivedContextReference(Expr):
1135
+ """Return the current template context including locals. Behaves
1136
+ exactly like :class:`ContextReference`, but includes local
1137
+ variables, such as from a ``for`` loop.
1138
+
1139
+ .. versionadded:: 2.11
1140
+ """
1141
+
1142
+
1143
+ class Continue(Stmt):
1144
+ """Continue a loop."""
1145
+
1146
+
1147
+ class Break(Stmt):
1148
+ """Break a loop."""
1149
+
1150
+
1151
+ class Scope(Stmt):
1152
+ """An artificial scope."""
1153
+
1154
+ fields = ("body",)
1155
+ body: t.List[Node]
1156
+
1157
+
1158
+ class OverlayScope(Stmt):
1159
+ """An overlay scope for extensions. This is a largely unoptimized scope
1160
+ that however can be used to introduce completely arbitrary variables into
1161
+ a sub scope from a dictionary or dictionary like object. The `context`
1162
+ field has to evaluate to a dictionary object.
1163
+
1164
+ Example usage::
1165
+
1166
+ OverlayScope(context=self.call_method('get_context'),
1167
+ body=[...])
1168
+
1169
+ .. versionadded:: 2.10
1170
+ """
1171
+
1172
+ fields = ("context", "body")
1173
+ context: Expr
1174
+ body: t.List[Node]
1175
+
1176
+
1177
+ class EvalContextModifier(Stmt):
1178
+ """Modifies the eval context. For each option that should be modified,
1179
+ a :class:`Keyword` has to be added to the :attr:`options` list.
1180
+
1181
+ Example to change the `autoescape` setting::
1182
+
1183
+ EvalContextModifier(options=[Keyword('autoescape', Const(True))])
1184
+ """
1185
+
1186
+ fields = ("options",)
1187
+ options: t.List[Keyword]
1188
+
1189
+
1190
+ class ScopedEvalContextModifier(EvalContextModifier):
1191
+ """Modifies the eval context and reverts it later. Works exactly like
1192
+ :class:`EvalContextModifier` but will only modify the
1193
+ :class:`~jinja2.nodes.EvalContext` for nodes in the :attr:`body`.
1194
+ """
1195
+
1196
+ fields = ("body",)
1197
+ body: t.List[Node]
1198
+
1199
+
1200
+ # make sure nobody creates custom nodes
1201
+ def _failing_new(*args: t.Any, **kwargs: t.Any) -> "te.NoReturn":
1202
+ raise TypeError("can't create custom node types")
1203
+
1204
+
1205
+ NodeType.__new__ = staticmethod(_failing_new) # type: ignore
1206
+ del _failing_new
TTS/Kokoro/.venv/Lib/site-packages/jinja2/optimizer.py ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The optimizer tries to constant fold expressions and modify the AST
2
+ in place so that it should be faster to evaluate.
3
+
4
+ Because the AST does not contain all the scoping information and the
5
+ compiler has to find that out, we cannot do all the optimizations we
6
+ want. For example, loop unrolling doesn't work because unrolled loops
7
+ would have a different scope. The solution would be a second syntax tree
8
+ that stored the scoping rules.
9
+ """
10
+
11
+ import typing as t
12
+
13
+ from . import nodes
14
+ from .visitor import NodeTransformer
15
+
16
+ if t.TYPE_CHECKING:
17
+ from .environment import Environment
18
+
19
+
20
+ def optimize(node: nodes.Node, environment: "Environment") -> nodes.Node:
21
+ """The context hint can be used to perform an static optimization
22
+ based on the context given."""
23
+ optimizer = Optimizer(environment)
24
+ return t.cast(nodes.Node, optimizer.visit(node))
25
+
26
+
27
+ class Optimizer(NodeTransformer):
28
+ def __init__(self, environment: "t.Optional[Environment]") -> None:
29
+ self.environment = environment
30
+
31
+ def generic_visit(
32
+ self, node: nodes.Node, *args: t.Any, **kwargs: t.Any
33
+ ) -> nodes.Node:
34
+ node = super().generic_visit(node, *args, **kwargs)
35
+
36
+ # Do constant folding. Some other nodes besides Expr have
37
+ # as_const, but folding them causes errors later on.
38
+ if isinstance(node, nodes.Expr):
39
+ try:
40
+ return nodes.Const.from_untrusted(
41
+ node.as_const(args[0] if args else None),
42
+ lineno=node.lineno,
43
+ environment=self.environment,
44
+ )
45
+ except nodes.Impossible:
46
+ pass
47
+
48
+ return node
TTS/Kokoro/.venv/Lib/site-packages/jinja2/parser.py ADDED
@@ -0,0 +1,1049 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Parse tokens from the lexer into nodes for the compiler."""
2
+
3
+ import typing
4
+ import typing as t
5
+
6
+ from . import nodes
7
+ from .exceptions import TemplateAssertionError
8
+ from .exceptions import TemplateSyntaxError
9
+ from .lexer import describe_token
10
+ from .lexer import describe_token_expr
11
+
12
+ if t.TYPE_CHECKING:
13
+ import typing_extensions as te
14
+
15
+ from .environment import Environment
16
+
17
+ _ImportInclude = t.TypeVar("_ImportInclude", nodes.Import, nodes.Include)
18
+ _MacroCall = t.TypeVar("_MacroCall", nodes.Macro, nodes.CallBlock)
19
+
20
+ _statement_keywords = frozenset(
21
+ [
22
+ "for",
23
+ "if",
24
+ "block",
25
+ "extends",
26
+ "print",
27
+ "macro",
28
+ "include",
29
+ "from",
30
+ "import",
31
+ "set",
32
+ "with",
33
+ "autoescape",
34
+ ]
35
+ )
36
+ _compare_operators = frozenset(["eq", "ne", "lt", "lteq", "gt", "gteq"])
37
+
38
+ _math_nodes: t.Dict[str, t.Type[nodes.Expr]] = {
39
+ "add": nodes.Add,
40
+ "sub": nodes.Sub,
41
+ "mul": nodes.Mul,
42
+ "div": nodes.Div,
43
+ "floordiv": nodes.FloorDiv,
44
+ "mod": nodes.Mod,
45
+ }
46
+
47
+
48
+ class Parser:
49
+ """This is the central parsing class Jinja uses. It's passed to
50
+ extensions and can be used to parse expressions or statements.
51
+ """
52
+
53
+ def __init__(
54
+ self,
55
+ environment: "Environment",
56
+ source: str,
57
+ name: t.Optional[str] = None,
58
+ filename: t.Optional[str] = None,
59
+ state: t.Optional[str] = None,
60
+ ) -> None:
61
+ self.environment = environment
62
+ self.stream = environment._tokenize(source, name, filename, state)
63
+ self.name = name
64
+ self.filename = filename
65
+ self.closed = False
66
+ self.extensions: t.Dict[
67
+ str, t.Callable[[Parser], t.Union[nodes.Node, t.List[nodes.Node]]]
68
+ ] = {}
69
+ for extension in environment.iter_extensions():
70
+ for tag in extension.tags:
71
+ self.extensions[tag] = extension.parse
72
+ self._last_identifier = 0
73
+ self._tag_stack: t.List[str] = []
74
+ self._end_token_stack: t.List[t.Tuple[str, ...]] = []
75
+
76
+ def fail(
77
+ self,
78
+ msg: str,
79
+ lineno: t.Optional[int] = None,
80
+ exc: t.Type[TemplateSyntaxError] = TemplateSyntaxError,
81
+ ) -> "te.NoReturn":
82
+ """Convenience method that raises `exc` with the message, passed
83
+ line number or last line number as well as the current name and
84
+ filename.
85
+ """
86
+ if lineno is None:
87
+ lineno = self.stream.current.lineno
88
+ raise exc(msg, lineno, self.name, self.filename)
89
+
90
+ def _fail_ut_eof(
91
+ self,
92
+ name: t.Optional[str],
93
+ end_token_stack: t.List[t.Tuple[str, ...]],
94
+ lineno: t.Optional[int],
95
+ ) -> "te.NoReturn":
96
+ expected: t.Set[str] = set()
97
+ for exprs in end_token_stack:
98
+ expected.update(map(describe_token_expr, exprs))
99
+ if end_token_stack:
100
+ currently_looking: t.Optional[str] = " or ".join(
101
+ map(repr, map(describe_token_expr, end_token_stack[-1]))
102
+ )
103
+ else:
104
+ currently_looking = None
105
+
106
+ if name is None:
107
+ message = ["Unexpected end of template."]
108
+ else:
109
+ message = [f"Encountered unknown tag {name!r}."]
110
+
111
+ if currently_looking:
112
+ if name is not None and name in expected:
113
+ message.append(
114
+ "You probably made a nesting mistake. Jinja is expecting this tag,"
115
+ f" but currently looking for {currently_looking}."
116
+ )
117
+ else:
118
+ message.append(
119
+ f"Jinja was looking for the following tags: {currently_looking}."
120
+ )
121
+
122
+ if self._tag_stack:
123
+ message.append(
124
+ "The innermost block that needs to be closed is"
125
+ f" {self._tag_stack[-1]!r}."
126
+ )
127
+
128
+ self.fail(" ".join(message), lineno)
129
+
130
+ def fail_unknown_tag(
131
+ self, name: str, lineno: t.Optional[int] = None
132
+ ) -> "te.NoReturn":
133
+ """Called if the parser encounters an unknown tag. Tries to fail
134
+ with a human readable error message that could help to identify
135
+ the problem.
136
+ """
137
+ self._fail_ut_eof(name, self._end_token_stack, lineno)
138
+
139
+ def fail_eof(
140
+ self,
141
+ end_tokens: t.Optional[t.Tuple[str, ...]] = None,
142
+ lineno: t.Optional[int] = None,
143
+ ) -> "te.NoReturn":
144
+ """Like fail_unknown_tag but for end of template situations."""
145
+ stack = list(self._end_token_stack)
146
+ if end_tokens is not None:
147
+ stack.append(end_tokens)
148
+ self._fail_ut_eof(None, stack, lineno)
149
+
150
+ def is_tuple_end(
151
+ self, extra_end_rules: t.Optional[t.Tuple[str, ...]] = None
152
+ ) -> bool:
153
+ """Are we at the end of a tuple?"""
154
+ if self.stream.current.type in ("variable_end", "block_end", "rparen"):
155
+ return True
156
+ elif extra_end_rules is not None:
157
+ return self.stream.current.test_any(extra_end_rules) # type: ignore
158
+ return False
159
+
160
+ def free_identifier(self, lineno: t.Optional[int] = None) -> nodes.InternalName:
161
+ """Return a new free identifier as :class:`~jinja2.nodes.InternalName`."""
162
+ self._last_identifier += 1
163
+ rv = object.__new__(nodes.InternalName)
164
+ nodes.Node.__init__(rv, f"fi{self._last_identifier}", lineno=lineno)
165
+ return rv
166
+
167
+ def parse_statement(self) -> t.Union[nodes.Node, t.List[nodes.Node]]:
168
+ """Parse a single statement."""
169
+ token = self.stream.current
170
+ if token.type != "name":
171
+ self.fail("tag name expected", token.lineno)
172
+ self._tag_stack.append(token.value)
173
+ pop_tag = True
174
+ try:
175
+ if token.value in _statement_keywords:
176
+ f = getattr(self, f"parse_{self.stream.current.value}")
177
+ return f() # type: ignore
178
+ if token.value == "call":
179
+ return self.parse_call_block()
180
+ if token.value == "filter":
181
+ return self.parse_filter_block()
182
+ ext = self.extensions.get(token.value)
183
+ if ext is not None:
184
+ return ext(self)
185
+
186
+ # did not work out, remove the token we pushed by accident
187
+ # from the stack so that the unknown tag fail function can
188
+ # produce a proper error message.
189
+ self._tag_stack.pop()
190
+ pop_tag = False
191
+ self.fail_unknown_tag(token.value, token.lineno)
192
+ finally:
193
+ if pop_tag:
194
+ self._tag_stack.pop()
195
+
196
+ def parse_statements(
197
+ self, end_tokens: t.Tuple[str, ...], drop_needle: bool = False
198
+ ) -> t.List[nodes.Node]:
199
+ """Parse multiple statements into a list until one of the end tokens
200
+ is reached. This is used to parse the body of statements as it also
201
+ parses template data if appropriate. The parser checks first if the
202
+ current token is a colon and skips it if there is one. Then it checks
203
+ for the block end and parses until if one of the `end_tokens` is
204
+ reached. Per default the active token in the stream at the end of
205
+ the call is the matched end token. If this is not wanted `drop_needle`
206
+ can be set to `True` and the end token is removed.
207
+ """
208
+ # the first token may be a colon for python compatibility
209
+ self.stream.skip_if("colon")
210
+
211
+ # in the future it would be possible to add whole code sections
212
+ # by adding some sort of end of statement token and parsing those here.
213
+ self.stream.expect("block_end")
214
+ result = self.subparse(end_tokens)
215
+
216
+ # we reached the end of the template too early, the subparser
217
+ # does not check for this, so we do that now
218
+ if self.stream.current.type == "eof":
219
+ self.fail_eof(end_tokens)
220
+
221
+ if drop_needle:
222
+ next(self.stream)
223
+ return result
224
+
225
+ def parse_set(self) -> t.Union[nodes.Assign, nodes.AssignBlock]:
226
+ """Parse an assign statement."""
227
+ lineno = next(self.stream).lineno
228
+ target = self.parse_assign_target(with_namespace=True)
229
+ if self.stream.skip_if("assign"):
230
+ expr = self.parse_tuple()
231
+ return nodes.Assign(target, expr, lineno=lineno)
232
+ filter_node = self.parse_filter(None)
233
+ body = self.parse_statements(("name:endset",), drop_needle=True)
234
+ return nodes.AssignBlock(target, filter_node, body, lineno=lineno)
235
+
236
+ def parse_for(self) -> nodes.For:
237
+ """Parse a for loop."""
238
+ lineno = self.stream.expect("name:for").lineno
239
+ target = self.parse_assign_target(extra_end_rules=("name:in",))
240
+ self.stream.expect("name:in")
241
+ iter = self.parse_tuple(
242
+ with_condexpr=False, extra_end_rules=("name:recursive",)
243
+ )
244
+ test = None
245
+ if self.stream.skip_if("name:if"):
246
+ test = self.parse_expression()
247
+ recursive = self.stream.skip_if("name:recursive")
248
+ body = self.parse_statements(("name:endfor", "name:else"))
249
+ if next(self.stream).value == "endfor":
250
+ else_ = []
251
+ else:
252
+ else_ = self.parse_statements(("name:endfor",), drop_needle=True)
253
+ return nodes.For(target, iter, body, else_, test, recursive, lineno=lineno)
254
+
255
+ def parse_if(self) -> nodes.If:
256
+ """Parse an if construct."""
257
+ node = result = nodes.If(lineno=self.stream.expect("name:if").lineno)
258
+ while True:
259
+ node.test = self.parse_tuple(with_condexpr=False)
260
+ node.body = self.parse_statements(("name:elif", "name:else", "name:endif"))
261
+ node.elif_ = []
262
+ node.else_ = []
263
+ token = next(self.stream)
264
+ if token.test("name:elif"):
265
+ node = nodes.If(lineno=self.stream.current.lineno)
266
+ result.elif_.append(node)
267
+ continue
268
+ elif token.test("name:else"):
269
+ result.else_ = self.parse_statements(("name:endif",), drop_needle=True)
270
+ break
271
+ return result
272
+
273
+ def parse_with(self) -> nodes.With:
274
+ node = nodes.With(lineno=next(self.stream).lineno)
275
+ targets: t.List[nodes.Expr] = []
276
+ values: t.List[nodes.Expr] = []
277
+ while self.stream.current.type != "block_end":
278
+ if targets:
279
+ self.stream.expect("comma")
280
+ target = self.parse_assign_target()
281
+ target.set_ctx("param")
282
+ targets.append(target)
283
+ self.stream.expect("assign")
284
+ values.append(self.parse_expression())
285
+ node.targets = targets
286
+ node.values = values
287
+ node.body = self.parse_statements(("name:endwith",), drop_needle=True)
288
+ return node
289
+
290
+ def parse_autoescape(self) -> nodes.Scope:
291
+ node = nodes.ScopedEvalContextModifier(lineno=next(self.stream).lineno)
292
+ node.options = [nodes.Keyword("autoescape", self.parse_expression())]
293
+ node.body = self.parse_statements(("name:endautoescape",), drop_needle=True)
294
+ return nodes.Scope([node])
295
+
296
+ def parse_block(self) -> nodes.Block:
297
+ node = nodes.Block(lineno=next(self.stream).lineno)
298
+ node.name = self.stream.expect("name").value
299
+ node.scoped = self.stream.skip_if("name:scoped")
300
+ node.required = self.stream.skip_if("name:required")
301
+
302
+ # common problem people encounter when switching from django
303
+ # to jinja. we do not support hyphens in block names, so let's
304
+ # raise a nicer error message in that case.
305
+ if self.stream.current.type == "sub":
306
+ self.fail(
307
+ "Block names in Jinja have to be valid Python identifiers and may not"
308
+ " contain hyphens, use an underscore instead."
309
+ )
310
+
311
+ node.body = self.parse_statements(("name:endblock",), drop_needle=True)
312
+
313
+ # enforce that required blocks only contain whitespace or comments
314
+ # by asserting that the body, if not empty, is just TemplateData nodes
315
+ # with whitespace data
316
+ if node.required:
317
+ for body_node in node.body:
318
+ if not isinstance(body_node, nodes.Output) or any(
319
+ not isinstance(output_node, nodes.TemplateData)
320
+ or not output_node.data.isspace()
321
+ for output_node in body_node.nodes
322
+ ):
323
+ self.fail("Required blocks can only contain comments or whitespace")
324
+
325
+ self.stream.skip_if("name:" + node.name)
326
+ return node
327
+
328
+ def parse_extends(self) -> nodes.Extends:
329
+ node = nodes.Extends(lineno=next(self.stream).lineno)
330
+ node.template = self.parse_expression()
331
+ return node
332
+
333
+ def parse_import_context(
334
+ self, node: _ImportInclude, default: bool
335
+ ) -> _ImportInclude:
336
+ if self.stream.current.test_any(
337
+ "name:with", "name:without"
338
+ ) and self.stream.look().test("name:context"):
339
+ node.with_context = next(self.stream).value == "with"
340
+ self.stream.skip()
341
+ else:
342
+ node.with_context = default
343
+ return node
344
+
345
+ def parse_include(self) -> nodes.Include:
346
+ node = nodes.Include(lineno=next(self.stream).lineno)
347
+ node.template = self.parse_expression()
348
+ if self.stream.current.test("name:ignore") and self.stream.look().test(
349
+ "name:missing"
350
+ ):
351
+ node.ignore_missing = True
352
+ self.stream.skip(2)
353
+ else:
354
+ node.ignore_missing = False
355
+ return self.parse_import_context(node, True)
356
+
357
+ def parse_import(self) -> nodes.Import:
358
+ node = nodes.Import(lineno=next(self.stream).lineno)
359
+ node.template = self.parse_expression()
360
+ self.stream.expect("name:as")
361
+ node.target = self.parse_assign_target(name_only=True).name
362
+ return self.parse_import_context(node, False)
363
+
364
+ def parse_from(self) -> nodes.FromImport:
365
+ node = nodes.FromImport(lineno=next(self.stream).lineno)
366
+ node.template = self.parse_expression()
367
+ self.stream.expect("name:import")
368
+ node.names = []
369
+
370
+ def parse_context() -> bool:
371
+ if self.stream.current.value in {
372
+ "with",
373
+ "without",
374
+ } and self.stream.look().test("name:context"):
375
+ node.with_context = next(self.stream).value == "with"
376
+ self.stream.skip()
377
+ return True
378
+ return False
379
+
380
+ while True:
381
+ if node.names:
382
+ self.stream.expect("comma")
383
+ if self.stream.current.type == "name":
384
+ if parse_context():
385
+ break
386
+ target = self.parse_assign_target(name_only=True)
387
+ if target.name.startswith("_"):
388
+ self.fail(
389
+ "names starting with an underline can not be imported",
390
+ target.lineno,
391
+ exc=TemplateAssertionError,
392
+ )
393
+ if self.stream.skip_if("name:as"):
394
+ alias = self.parse_assign_target(name_only=True)
395
+ node.names.append((target.name, alias.name))
396
+ else:
397
+ node.names.append(target.name)
398
+ if parse_context() or self.stream.current.type != "comma":
399
+ break
400
+ else:
401
+ self.stream.expect("name")
402
+ if not hasattr(node, "with_context"):
403
+ node.with_context = False
404
+ return node
405
+
406
+ def parse_signature(self, node: _MacroCall) -> None:
407
+ args = node.args = []
408
+ defaults = node.defaults = []
409
+ self.stream.expect("lparen")
410
+ while self.stream.current.type != "rparen":
411
+ if args:
412
+ self.stream.expect("comma")
413
+ arg = self.parse_assign_target(name_only=True)
414
+ arg.set_ctx("param")
415
+ if self.stream.skip_if("assign"):
416
+ defaults.append(self.parse_expression())
417
+ elif defaults:
418
+ self.fail("non-default argument follows default argument")
419
+ args.append(arg)
420
+ self.stream.expect("rparen")
421
+
422
+ def parse_call_block(self) -> nodes.CallBlock:
423
+ node = nodes.CallBlock(lineno=next(self.stream).lineno)
424
+ if self.stream.current.type == "lparen":
425
+ self.parse_signature(node)
426
+ else:
427
+ node.args = []
428
+ node.defaults = []
429
+
430
+ call_node = self.parse_expression()
431
+ if not isinstance(call_node, nodes.Call):
432
+ self.fail("expected call", node.lineno)
433
+ node.call = call_node
434
+ node.body = self.parse_statements(("name:endcall",), drop_needle=True)
435
+ return node
436
+
437
+ def parse_filter_block(self) -> nodes.FilterBlock:
438
+ node = nodes.FilterBlock(lineno=next(self.stream).lineno)
439
+ node.filter = self.parse_filter(None, start_inline=True) # type: ignore
440
+ node.body = self.parse_statements(("name:endfilter",), drop_needle=True)
441
+ return node
442
+
443
+ def parse_macro(self) -> nodes.Macro:
444
+ node = nodes.Macro(lineno=next(self.stream).lineno)
445
+ node.name = self.parse_assign_target(name_only=True).name
446
+ self.parse_signature(node)
447
+ node.body = self.parse_statements(("name:endmacro",), drop_needle=True)
448
+ return node
449
+
450
+ def parse_print(self) -> nodes.Output:
451
+ node = nodes.Output(lineno=next(self.stream).lineno)
452
+ node.nodes = []
453
+ while self.stream.current.type != "block_end":
454
+ if node.nodes:
455
+ self.stream.expect("comma")
456
+ node.nodes.append(self.parse_expression())
457
+ return node
458
+
459
+ @typing.overload
460
+ def parse_assign_target(
461
+ self, with_tuple: bool = ..., name_only: "te.Literal[True]" = ...
462
+ ) -> nodes.Name: ...
463
+
464
+ @typing.overload
465
+ def parse_assign_target(
466
+ self,
467
+ with_tuple: bool = True,
468
+ name_only: bool = False,
469
+ extra_end_rules: t.Optional[t.Tuple[str, ...]] = None,
470
+ with_namespace: bool = False,
471
+ ) -> t.Union[nodes.NSRef, nodes.Name, nodes.Tuple]: ...
472
+
473
+ def parse_assign_target(
474
+ self,
475
+ with_tuple: bool = True,
476
+ name_only: bool = False,
477
+ extra_end_rules: t.Optional[t.Tuple[str, ...]] = None,
478
+ with_namespace: bool = False,
479
+ ) -> t.Union[nodes.NSRef, nodes.Name, nodes.Tuple]:
480
+ """Parse an assignment target. As Jinja allows assignments to
481
+ tuples, this function can parse all allowed assignment targets. Per
482
+ default assignments to tuples are parsed, that can be disable however
483
+ by setting `with_tuple` to `False`. If only assignments to names are
484
+ wanted `name_only` can be set to `True`. The `extra_end_rules`
485
+ parameter is forwarded to the tuple parsing function. If
486
+ `with_namespace` is enabled, a namespace assignment may be parsed.
487
+ """
488
+ target: nodes.Expr
489
+
490
+ if name_only:
491
+ token = self.stream.expect("name")
492
+ target = nodes.Name(token.value, "store", lineno=token.lineno)
493
+ else:
494
+ if with_tuple:
495
+ target = self.parse_tuple(
496
+ simplified=True,
497
+ extra_end_rules=extra_end_rules,
498
+ with_namespace=with_namespace,
499
+ )
500
+ else:
501
+ target = self.parse_primary(with_namespace=with_namespace)
502
+
503
+ target.set_ctx("store")
504
+
505
+ if not target.can_assign():
506
+ self.fail(
507
+ f"can't assign to {type(target).__name__.lower()!r}", target.lineno
508
+ )
509
+
510
+ return target # type: ignore
511
+
512
+ def parse_expression(self, with_condexpr: bool = True) -> nodes.Expr:
513
+ """Parse an expression. Per default all expressions are parsed, if
514
+ the optional `with_condexpr` parameter is set to `False` conditional
515
+ expressions are not parsed.
516
+ """
517
+ if with_condexpr:
518
+ return self.parse_condexpr()
519
+ return self.parse_or()
520
+
521
+ def parse_condexpr(self) -> nodes.Expr:
522
+ lineno = self.stream.current.lineno
523
+ expr1 = self.parse_or()
524
+ expr3: t.Optional[nodes.Expr]
525
+
526
+ while self.stream.skip_if("name:if"):
527
+ expr2 = self.parse_or()
528
+ if self.stream.skip_if("name:else"):
529
+ expr3 = self.parse_condexpr()
530
+ else:
531
+ expr3 = None
532
+ expr1 = nodes.CondExpr(expr2, expr1, expr3, lineno=lineno)
533
+ lineno = self.stream.current.lineno
534
+ return expr1
535
+
536
+ def parse_or(self) -> nodes.Expr:
537
+ lineno = self.stream.current.lineno
538
+ left = self.parse_and()
539
+ while self.stream.skip_if("name:or"):
540
+ right = self.parse_and()
541
+ left = nodes.Or(left, right, lineno=lineno)
542
+ lineno = self.stream.current.lineno
543
+ return left
544
+
545
+ def parse_and(self) -> nodes.Expr:
546
+ lineno = self.stream.current.lineno
547
+ left = self.parse_not()
548
+ while self.stream.skip_if("name:and"):
549
+ right = self.parse_not()
550
+ left = nodes.And(left, right, lineno=lineno)
551
+ lineno = self.stream.current.lineno
552
+ return left
553
+
554
+ def parse_not(self) -> nodes.Expr:
555
+ if self.stream.current.test("name:not"):
556
+ lineno = next(self.stream).lineno
557
+ return nodes.Not(self.parse_not(), lineno=lineno)
558
+ return self.parse_compare()
559
+
560
+ def parse_compare(self) -> nodes.Expr:
561
+ lineno = self.stream.current.lineno
562
+ expr = self.parse_math1()
563
+ ops = []
564
+ while True:
565
+ token_type = self.stream.current.type
566
+ if token_type in _compare_operators:
567
+ next(self.stream)
568
+ ops.append(nodes.Operand(token_type, self.parse_math1()))
569
+ elif self.stream.skip_if("name:in"):
570
+ ops.append(nodes.Operand("in", self.parse_math1()))
571
+ elif self.stream.current.test("name:not") and self.stream.look().test(
572
+ "name:in"
573
+ ):
574
+ self.stream.skip(2)
575
+ ops.append(nodes.Operand("notin", self.parse_math1()))
576
+ else:
577
+ break
578
+ lineno = self.stream.current.lineno
579
+ if not ops:
580
+ return expr
581
+ return nodes.Compare(expr, ops, lineno=lineno)
582
+
583
+ def parse_math1(self) -> nodes.Expr:
584
+ lineno = self.stream.current.lineno
585
+ left = self.parse_concat()
586
+ while self.stream.current.type in ("add", "sub"):
587
+ cls = _math_nodes[self.stream.current.type]
588
+ next(self.stream)
589
+ right = self.parse_concat()
590
+ left = cls(left, right, lineno=lineno)
591
+ lineno = self.stream.current.lineno
592
+ return left
593
+
594
+ def parse_concat(self) -> nodes.Expr:
595
+ lineno = self.stream.current.lineno
596
+ args = [self.parse_math2()]
597
+ while self.stream.current.type == "tilde":
598
+ next(self.stream)
599
+ args.append(self.parse_math2())
600
+ if len(args) == 1:
601
+ return args[0]
602
+ return nodes.Concat(args, lineno=lineno)
603
+
604
+ def parse_math2(self) -> nodes.Expr:
605
+ lineno = self.stream.current.lineno
606
+ left = self.parse_pow()
607
+ while self.stream.current.type in ("mul", "div", "floordiv", "mod"):
608
+ cls = _math_nodes[self.stream.current.type]
609
+ next(self.stream)
610
+ right = self.parse_pow()
611
+ left = cls(left, right, lineno=lineno)
612
+ lineno = self.stream.current.lineno
613
+ return left
614
+
615
+ def parse_pow(self) -> nodes.Expr:
616
+ lineno = self.stream.current.lineno
617
+ left = self.parse_unary()
618
+ while self.stream.current.type == "pow":
619
+ next(self.stream)
620
+ right = self.parse_unary()
621
+ left = nodes.Pow(left, right, lineno=lineno)
622
+ lineno = self.stream.current.lineno
623
+ return left
624
+
625
+ def parse_unary(self, with_filter: bool = True) -> nodes.Expr:
626
+ token_type = self.stream.current.type
627
+ lineno = self.stream.current.lineno
628
+ node: nodes.Expr
629
+
630
+ if token_type == "sub":
631
+ next(self.stream)
632
+ node = nodes.Neg(self.parse_unary(False), lineno=lineno)
633
+ elif token_type == "add":
634
+ next(self.stream)
635
+ node = nodes.Pos(self.parse_unary(False), lineno=lineno)
636
+ else:
637
+ node = self.parse_primary()
638
+ node = self.parse_postfix(node)
639
+ if with_filter:
640
+ node = self.parse_filter_expr(node)
641
+ return node
642
+
643
+ def parse_primary(self, with_namespace: bool = False) -> nodes.Expr:
644
+ """Parse a name or literal value. If ``with_namespace`` is enabled, also
645
+ parse namespace attr refs, for use in assignments."""
646
+ token = self.stream.current
647
+ node: nodes.Expr
648
+ if token.type == "name":
649
+ next(self.stream)
650
+ if token.value in ("true", "false", "True", "False"):
651
+ node = nodes.Const(token.value in ("true", "True"), lineno=token.lineno)
652
+ elif token.value in ("none", "None"):
653
+ node = nodes.Const(None, lineno=token.lineno)
654
+ elif with_namespace and self.stream.current.type == "dot":
655
+ # If namespace attributes are allowed at this point, and the next
656
+ # token is a dot, produce a namespace reference.
657
+ next(self.stream)
658
+ attr = self.stream.expect("name")
659
+ node = nodes.NSRef(token.value, attr.value, lineno=token.lineno)
660
+ else:
661
+ node = nodes.Name(token.value, "load", lineno=token.lineno)
662
+ elif token.type == "string":
663
+ next(self.stream)
664
+ buf = [token.value]
665
+ lineno = token.lineno
666
+ while self.stream.current.type == "string":
667
+ buf.append(self.stream.current.value)
668
+ next(self.stream)
669
+ node = nodes.Const("".join(buf), lineno=lineno)
670
+ elif token.type in ("integer", "float"):
671
+ next(self.stream)
672
+ node = nodes.Const(token.value, lineno=token.lineno)
673
+ elif token.type == "lparen":
674
+ next(self.stream)
675
+ node = self.parse_tuple(explicit_parentheses=True)
676
+ self.stream.expect("rparen")
677
+ elif token.type == "lbracket":
678
+ node = self.parse_list()
679
+ elif token.type == "lbrace":
680
+ node = self.parse_dict()
681
+ else:
682
+ self.fail(f"unexpected {describe_token(token)!r}", token.lineno)
683
+ return node
684
+
685
+ def parse_tuple(
686
+ self,
687
+ simplified: bool = False,
688
+ with_condexpr: bool = True,
689
+ extra_end_rules: t.Optional[t.Tuple[str, ...]] = None,
690
+ explicit_parentheses: bool = False,
691
+ with_namespace: bool = False,
692
+ ) -> t.Union[nodes.Tuple, nodes.Expr]:
693
+ """Works like `parse_expression` but if multiple expressions are
694
+ delimited by a comma a :class:`~jinja2.nodes.Tuple` node is created.
695
+ This method could also return a regular expression instead of a tuple
696
+ if no commas where found.
697
+
698
+ The default parsing mode is a full tuple. If `simplified` is `True`
699
+ only names and literals are parsed; ``with_namespace`` allows namespace
700
+ attr refs as well. The `no_condexpr` parameter is forwarded to
701
+ :meth:`parse_expression`.
702
+
703
+ Because tuples do not require delimiters and may end in a bogus comma
704
+ an extra hint is needed that marks the end of a tuple. For example
705
+ for loops support tuples between `for` and `in`. In that case the
706
+ `extra_end_rules` is set to ``['name:in']``.
707
+
708
+ `explicit_parentheses` is true if the parsing was triggered by an
709
+ expression in parentheses. This is used to figure out if an empty
710
+ tuple is a valid expression or not.
711
+ """
712
+ lineno = self.stream.current.lineno
713
+ if simplified:
714
+
715
+ def parse() -> nodes.Expr:
716
+ return self.parse_primary(with_namespace=with_namespace)
717
+
718
+ else:
719
+
720
+ def parse() -> nodes.Expr:
721
+ return self.parse_expression(with_condexpr=with_condexpr)
722
+
723
+ args: t.List[nodes.Expr] = []
724
+ is_tuple = False
725
+
726
+ while True:
727
+ if args:
728
+ self.stream.expect("comma")
729
+ if self.is_tuple_end(extra_end_rules):
730
+ break
731
+ args.append(parse())
732
+ if self.stream.current.type == "comma":
733
+ is_tuple = True
734
+ else:
735
+ break
736
+ lineno = self.stream.current.lineno
737
+
738
+ if not is_tuple:
739
+ if args:
740
+ return args[0]
741
+
742
+ # if we don't have explicit parentheses, an empty tuple is
743
+ # not a valid expression. This would mean nothing (literally
744
+ # nothing) in the spot of an expression would be an empty
745
+ # tuple.
746
+ if not explicit_parentheses:
747
+ self.fail(
748
+ "Expected an expression,"
749
+ f" got {describe_token(self.stream.current)!r}"
750
+ )
751
+
752
+ return nodes.Tuple(args, "load", lineno=lineno)
753
+
754
+ def parse_list(self) -> nodes.List:
755
+ token = self.stream.expect("lbracket")
756
+ items: t.List[nodes.Expr] = []
757
+ while self.stream.current.type != "rbracket":
758
+ if items:
759
+ self.stream.expect("comma")
760
+ if self.stream.current.type == "rbracket":
761
+ break
762
+ items.append(self.parse_expression())
763
+ self.stream.expect("rbracket")
764
+ return nodes.List(items, lineno=token.lineno)
765
+
766
+ def parse_dict(self) -> nodes.Dict:
767
+ token = self.stream.expect("lbrace")
768
+ items: t.List[nodes.Pair] = []
769
+ while self.stream.current.type != "rbrace":
770
+ if items:
771
+ self.stream.expect("comma")
772
+ if self.stream.current.type == "rbrace":
773
+ break
774
+ key = self.parse_expression()
775
+ self.stream.expect("colon")
776
+ value = self.parse_expression()
777
+ items.append(nodes.Pair(key, value, lineno=key.lineno))
778
+ self.stream.expect("rbrace")
779
+ return nodes.Dict(items, lineno=token.lineno)
780
+
781
+ def parse_postfix(self, node: nodes.Expr) -> nodes.Expr:
782
+ while True:
783
+ token_type = self.stream.current.type
784
+ if token_type == "dot" or token_type == "lbracket":
785
+ node = self.parse_subscript(node)
786
+ # calls are valid both after postfix expressions (getattr
787
+ # and getitem) as well as filters and tests
788
+ elif token_type == "lparen":
789
+ node = self.parse_call(node)
790
+ else:
791
+ break
792
+ return node
793
+
794
+ def parse_filter_expr(self, node: nodes.Expr) -> nodes.Expr:
795
+ while True:
796
+ token_type = self.stream.current.type
797
+ if token_type == "pipe":
798
+ node = self.parse_filter(node) # type: ignore
799
+ elif token_type == "name" and self.stream.current.value == "is":
800
+ node = self.parse_test(node)
801
+ # calls are valid both after postfix expressions (getattr
802
+ # and getitem) as well as filters and tests
803
+ elif token_type == "lparen":
804
+ node = self.parse_call(node)
805
+ else:
806
+ break
807
+ return node
808
+
809
+ def parse_subscript(
810
+ self, node: nodes.Expr
811
+ ) -> t.Union[nodes.Getattr, nodes.Getitem]:
812
+ token = next(self.stream)
813
+ arg: nodes.Expr
814
+
815
+ if token.type == "dot":
816
+ attr_token = self.stream.current
817
+ next(self.stream)
818
+ if attr_token.type == "name":
819
+ return nodes.Getattr(
820
+ node, attr_token.value, "load", lineno=token.lineno
821
+ )
822
+ elif attr_token.type != "integer":
823
+ self.fail("expected name or number", attr_token.lineno)
824
+ arg = nodes.Const(attr_token.value, lineno=attr_token.lineno)
825
+ return nodes.Getitem(node, arg, "load", lineno=token.lineno)
826
+ if token.type == "lbracket":
827
+ args: t.List[nodes.Expr] = []
828
+ while self.stream.current.type != "rbracket":
829
+ if args:
830
+ self.stream.expect("comma")
831
+ args.append(self.parse_subscribed())
832
+ self.stream.expect("rbracket")
833
+ if len(args) == 1:
834
+ arg = args[0]
835
+ else:
836
+ arg = nodes.Tuple(args, "load", lineno=token.lineno)
837
+ return nodes.Getitem(node, arg, "load", lineno=token.lineno)
838
+ self.fail("expected subscript expression", token.lineno)
839
+
840
+ def parse_subscribed(self) -> nodes.Expr:
841
+ lineno = self.stream.current.lineno
842
+ args: t.List[t.Optional[nodes.Expr]]
843
+
844
+ if self.stream.current.type == "colon":
845
+ next(self.stream)
846
+ args = [None]
847
+ else:
848
+ node = self.parse_expression()
849
+ if self.stream.current.type != "colon":
850
+ return node
851
+ next(self.stream)
852
+ args = [node]
853
+
854
+ if self.stream.current.type == "colon":
855
+ args.append(None)
856
+ elif self.stream.current.type not in ("rbracket", "comma"):
857
+ args.append(self.parse_expression())
858
+ else:
859
+ args.append(None)
860
+
861
+ if self.stream.current.type == "colon":
862
+ next(self.stream)
863
+ if self.stream.current.type not in ("rbracket", "comma"):
864
+ args.append(self.parse_expression())
865
+ else:
866
+ args.append(None)
867
+ else:
868
+ args.append(None)
869
+
870
+ return nodes.Slice(lineno=lineno, *args) # noqa: B026
871
+
872
+ def parse_call_args(
873
+ self,
874
+ ) -> t.Tuple[
875
+ t.List[nodes.Expr],
876
+ t.List[nodes.Keyword],
877
+ t.Optional[nodes.Expr],
878
+ t.Optional[nodes.Expr],
879
+ ]:
880
+ token = self.stream.expect("lparen")
881
+ args = []
882
+ kwargs = []
883
+ dyn_args = None
884
+ dyn_kwargs = None
885
+ require_comma = False
886
+
887
+ def ensure(expr: bool) -> None:
888
+ if not expr:
889
+ self.fail("invalid syntax for function call expression", token.lineno)
890
+
891
+ while self.stream.current.type != "rparen":
892
+ if require_comma:
893
+ self.stream.expect("comma")
894
+
895
+ # support for trailing comma
896
+ if self.stream.current.type == "rparen":
897
+ break
898
+
899
+ if self.stream.current.type == "mul":
900
+ ensure(dyn_args is None and dyn_kwargs is None)
901
+ next(self.stream)
902
+ dyn_args = self.parse_expression()
903
+ elif self.stream.current.type == "pow":
904
+ ensure(dyn_kwargs is None)
905
+ next(self.stream)
906
+ dyn_kwargs = self.parse_expression()
907
+ else:
908
+ if (
909
+ self.stream.current.type == "name"
910
+ and self.stream.look().type == "assign"
911
+ ):
912
+ # Parsing a kwarg
913
+ ensure(dyn_kwargs is None)
914
+ key = self.stream.current.value
915
+ self.stream.skip(2)
916
+ value = self.parse_expression()
917
+ kwargs.append(nodes.Keyword(key, value, lineno=value.lineno))
918
+ else:
919
+ # Parsing an arg
920
+ ensure(dyn_args is None and dyn_kwargs is None and not kwargs)
921
+ args.append(self.parse_expression())
922
+
923
+ require_comma = True
924
+
925
+ self.stream.expect("rparen")
926
+ return args, kwargs, dyn_args, dyn_kwargs
927
+
928
+ def parse_call(self, node: nodes.Expr) -> nodes.Call:
929
+ # The lparen will be expected in parse_call_args, but the lineno
930
+ # needs to be recorded before the stream is advanced.
931
+ token = self.stream.current
932
+ args, kwargs, dyn_args, dyn_kwargs = self.parse_call_args()
933
+ return nodes.Call(node, args, kwargs, dyn_args, dyn_kwargs, lineno=token.lineno)
934
+
935
+ def parse_filter(
936
+ self, node: t.Optional[nodes.Expr], start_inline: bool = False
937
+ ) -> t.Optional[nodes.Expr]:
938
+ while self.stream.current.type == "pipe" or start_inline:
939
+ if not start_inline:
940
+ next(self.stream)
941
+ token = self.stream.expect("name")
942
+ name = token.value
943
+ while self.stream.current.type == "dot":
944
+ next(self.stream)
945
+ name += "." + self.stream.expect("name").value
946
+ if self.stream.current.type == "lparen":
947
+ args, kwargs, dyn_args, dyn_kwargs = self.parse_call_args()
948
+ else:
949
+ args = []
950
+ kwargs = []
951
+ dyn_args = dyn_kwargs = None
952
+ node = nodes.Filter(
953
+ node, name, args, kwargs, dyn_args, dyn_kwargs, lineno=token.lineno
954
+ )
955
+ start_inline = False
956
+ return node
957
+
958
+ def parse_test(self, node: nodes.Expr) -> nodes.Expr:
959
+ token = next(self.stream)
960
+ if self.stream.current.test("name:not"):
961
+ next(self.stream)
962
+ negated = True
963
+ else:
964
+ negated = False
965
+ name = self.stream.expect("name").value
966
+ while self.stream.current.type == "dot":
967
+ next(self.stream)
968
+ name += "." + self.stream.expect("name").value
969
+ dyn_args = dyn_kwargs = None
970
+ kwargs: t.List[nodes.Keyword] = []
971
+ if self.stream.current.type == "lparen":
972
+ args, kwargs, dyn_args, dyn_kwargs = self.parse_call_args()
973
+ elif self.stream.current.type in {
974
+ "name",
975
+ "string",
976
+ "integer",
977
+ "float",
978
+ "lparen",
979
+ "lbracket",
980
+ "lbrace",
981
+ } and not self.stream.current.test_any("name:else", "name:or", "name:and"):
982
+ if self.stream.current.test("name:is"):
983
+ self.fail("You cannot chain multiple tests with is")
984
+ arg_node = self.parse_primary()
985
+ arg_node = self.parse_postfix(arg_node)
986
+ args = [arg_node]
987
+ else:
988
+ args = []
989
+ node = nodes.Test(
990
+ node, name, args, kwargs, dyn_args, dyn_kwargs, lineno=token.lineno
991
+ )
992
+ if negated:
993
+ node = nodes.Not(node, lineno=token.lineno)
994
+ return node
995
+
996
+ def subparse(
997
+ self, end_tokens: t.Optional[t.Tuple[str, ...]] = None
998
+ ) -> t.List[nodes.Node]:
999
+ body: t.List[nodes.Node] = []
1000
+ data_buffer: t.List[nodes.Node] = []
1001
+ add_data = data_buffer.append
1002
+
1003
+ if end_tokens is not None:
1004
+ self._end_token_stack.append(end_tokens)
1005
+
1006
+ def flush_data() -> None:
1007
+ if data_buffer:
1008
+ lineno = data_buffer[0].lineno
1009
+ body.append(nodes.Output(data_buffer[:], lineno=lineno))
1010
+ del data_buffer[:]
1011
+
1012
+ try:
1013
+ while self.stream:
1014
+ token = self.stream.current
1015
+ if token.type == "data":
1016
+ if token.value:
1017
+ add_data(nodes.TemplateData(token.value, lineno=token.lineno))
1018
+ next(self.stream)
1019
+ elif token.type == "variable_begin":
1020
+ next(self.stream)
1021
+ add_data(self.parse_tuple(with_condexpr=True))
1022
+ self.stream.expect("variable_end")
1023
+ elif token.type == "block_begin":
1024
+ flush_data()
1025
+ next(self.stream)
1026
+ if end_tokens is not None and self.stream.current.test_any(
1027
+ *end_tokens
1028
+ ):
1029
+ return body
1030
+ rv = self.parse_statement()
1031
+ if isinstance(rv, list):
1032
+ body.extend(rv)
1033
+ else:
1034
+ body.append(rv)
1035
+ self.stream.expect("block_end")
1036
+ else:
1037
+ raise AssertionError("internal parsing error")
1038
+
1039
+ flush_data()
1040
+ finally:
1041
+ if end_tokens is not None:
1042
+ self._end_token_stack.pop()
1043
+ return body
1044
+
1045
+ def parse(self) -> nodes.Template:
1046
+ """Parse the whole template into a `Template` node."""
1047
+ result = nodes.Template(self.subparse(), lineno=1)
1048
+ result.set_environment(self.environment)
1049
+ return result
TTS/Kokoro/.venv/Lib/site-packages/jinja2/py.typed ADDED
File without changes
TTS/Kokoro/.venv/Lib/site-packages/jinja2/runtime.py ADDED
@@ -0,0 +1,1062 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The runtime functions and state used by compiled templates."""
2
+
3
+ import functools
4
+ import sys
5
+ import typing as t
6
+ from collections import abc
7
+ from itertools import chain
8
+
9
+ from markupsafe import escape # noqa: F401
10
+ from markupsafe import Markup
11
+ from markupsafe import soft_str
12
+
13
+ from .async_utils import auto_aiter
14
+ from .async_utils import auto_await # noqa: F401
15
+ from .exceptions import TemplateNotFound # noqa: F401
16
+ from .exceptions import TemplateRuntimeError # noqa: F401
17
+ from .exceptions import UndefinedError
18
+ from .nodes import EvalContext
19
+ from .utils import _PassArg
20
+ from .utils import concat
21
+ from .utils import internalcode
22
+ from .utils import missing
23
+ from .utils import Namespace # noqa: F401
24
+ from .utils import object_type_repr
25
+ from .utils import pass_eval_context
26
+
27
+ V = t.TypeVar("V")
28
+ F = t.TypeVar("F", bound=t.Callable[..., t.Any])
29
+
30
+ if t.TYPE_CHECKING:
31
+ import logging
32
+
33
+ import typing_extensions as te
34
+
35
+ from .environment import Environment
36
+
37
+ class LoopRenderFunc(te.Protocol):
38
+ def __call__(
39
+ self,
40
+ reciter: t.Iterable[V],
41
+ loop_render_func: "LoopRenderFunc",
42
+ depth: int = 0,
43
+ ) -> str: ...
44
+
45
+
46
+ # these variables are exported to the template runtime
47
+ exported = [
48
+ "LoopContext",
49
+ "TemplateReference",
50
+ "Macro",
51
+ "Markup",
52
+ "TemplateRuntimeError",
53
+ "missing",
54
+ "escape",
55
+ "markup_join",
56
+ "str_join",
57
+ "identity",
58
+ "TemplateNotFound",
59
+ "Namespace",
60
+ "Undefined",
61
+ "internalcode",
62
+ ]
63
+ async_exported = [
64
+ "AsyncLoopContext",
65
+ "auto_aiter",
66
+ "auto_await",
67
+ ]
68
+
69
+
70
+ def identity(x: V) -> V:
71
+ """Returns its argument. Useful for certain things in the
72
+ environment.
73
+ """
74
+ return x
75
+
76
+
77
+ def markup_join(seq: t.Iterable[t.Any]) -> str:
78
+ """Concatenation that escapes if necessary and converts to string."""
79
+ buf = []
80
+ iterator = map(soft_str, seq)
81
+ for arg in iterator:
82
+ buf.append(arg)
83
+ if hasattr(arg, "__html__"):
84
+ return Markup("").join(chain(buf, iterator))
85
+ return concat(buf)
86
+
87
+
88
+ def str_join(seq: t.Iterable[t.Any]) -> str:
89
+ """Simple args to string conversion and concatenation."""
90
+ return concat(map(str, seq))
91
+
92
+
93
+ def new_context(
94
+ environment: "Environment",
95
+ template_name: t.Optional[str],
96
+ blocks: t.Dict[str, t.Callable[["Context"], t.Iterator[str]]],
97
+ vars: t.Optional[t.Dict[str, t.Any]] = None,
98
+ shared: bool = False,
99
+ globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
100
+ locals: t.Optional[t.Mapping[str, t.Any]] = None,
101
+ ) -> "Context":
102
+ """Internal helper for context creation."""
103
+ if vars is None:
104
+ vars = {}
105
+ if shared:
106
+ parent = vars
107
+ else:
108
+ parent = dict(globals or (), **vars)
109
+ if locals:
110
+ # if the parent is shared a copy should be created because
111
+ # we don't want to modify the dict passed
112
+ if shared:
113
+ parent = dict(parent)
114
+ for key, value in locals.items():
115
+ if value is not missing:
116
+ parent[key] = value
117
+ return environment.context_class(
118
+ environment, parent, template_name, blocks, globals=globals
119
+ )
120
+
121
+
122
+ class TemplateReference:
123
+ """The `self` in templates."""
124
+
125
+ def __init__(self, context: "Context") -> None:
126
+ self.__context = context
127
+
128
+ def __getitem__(self, name: str) -> t.Any:
129
+ blocks = self.__context.blocks[name]
130
+ return BlockReference(name, self.__context, blocks, 0)
131
+
132
+ def __repr__(self) -> str:
133
+ return f"<{type(self).__name__} {self.__context.name!r}>"
134
+
135
+
136
+ def _dict_method_all(dict_method: F) -> F:
137
+ @functools.wraps(dict_method)
138
+ def f_all(self: "Context") -> t.Any:
139
+ return dict_method(self.get_all())
140
+
141
+ return t.cast(F, f_all)
142
+
143
+
144
+ @abc.Mapping.register
145
+ class Context:
146
+ """The template context holds the variables of a template. It stores the
147
+ values passed to the template and also the names the template exports.
148
+ Creating instances is neither supported nor useful as it's created
149
+ automatically at various stages of the template evaluation and should not
150
+ be created by hand.
151
+
152
+ The context is immutable. Modifications on :attr:`parent` **must not**
153
+ happen and modifications on :attr:`vars` are allowed from generated
154
+ template code only. Template filters and global functions marked as
155
+ :func:`pass_context` get the active context passed as first argument
156
+ and are allowed to access the context read-only.
157
+
158
+ The template context supports read only dict operations (`get`,
159
+ `keys`, `values`, `items`, `iterkeys`, `itervalues`, `iteritems`,
160
+ `__getitem__`, `__contains__`). Additionally there is a :meth:`resolve`
161
+ method that doesn't fail with a `KeyError` but returns an
162
+ :class:`Undefined` object for missing variables.
163
+ """
164
+
165
+ def __init__(
166
+ self,
167
+ environment: "Environment",
168
+ parent: t.Dict[str, t.Any],
169
+ name: t.Optional[str],
170
+ blocks: t.Dict[str, t.Callable[["Context"], t.Iterator[str]]],
171
+ globals: t.Optional[t.MutableMapping[str, t.Any]] = None,
172
+ ):
173
+ self.parent = parent
174
+ self.vars: t.Dict[str, t.Any] = {}
175
+ self.environment: Environment = environment
176
+ self.eval_ctx = EvalContext(self.environment, name)
177
+ self.exported_vars: t.Set[str] = set()
178
+ self.name = name
179
+ self.globals_keys = set() if globals is None else set(globals)
180
+
181
+ # create the initial mapping of blocks. Whenever template inheritance
182
+ # takes place the runtime will update this mapping with the new blocks
183
+ # from the template.
184
+ self.blocks = {k: [v] for k, v in blocks.items()}
185
+
186
+ def super(
187
+ self, name: str, current: t.Callable[["Context"], t.Iterator[str]]
188
+ ) -> t.Union["BlockReference", "Undefined"]:
189
+ """Render a parent block."""
190
+ try:
191
+ blocks = self.blocks[name]
192
+ index = blocks.index(current) + 1
193
+ blocks[index]
194
+ except LookupError:
195
+ return self.environment.undefined(
196
+ f"there is no parent block called {name!r}.", name="super"
197
+ )
198
+ return BlockReference(name, self, blocks, index)
199
+
200
+ def get(self, key: str, default: t.Any = None) -> t.Any:
201
+ """Look up a variable by name, or return a default if the key is
202
+ not found.
203
+
204
+ :param key: The variable name to look up.
205
+ :param default: The value to return if the key is not found.
206
+ """
207
+ try:
208
+ return self[key]
209
+ except KeyError:
210
+ return default
211
+
212
+ def resolve(self, key: str) -> t.Union[t.Any, "Undefined"]:
213
+ """Look up a variable by name, or return an :class:`Undefined`
214
+ object if the key is not found.
215
+
216
+ If you need to add custom behavior, override
217
+ :meth:`resolve_or_missing`, not this method. The various lookup
218
+ functions use that method, not this one.
219
+
220
+ :param key: The variable name to look up.
221
+ """
222
+ rv = self.resolve_or_missing(key)
223
+
224
+ if rv is missing:
225
+ return self.environment.undefined(name=key)
226
+
227
+ return rv
228
+
229
+ def resolve_or_missing(self, key: str) -> t.Any:
230
+ """Look up a variable by name, or return a ``missing`` sentinel
231
+ if the key is not found.
232
+
233
+ Override this method to add custom lookup behavior.
234
+ :meth:`resolve`, :meth:`get`, and :meth:`__getitem__` use this
235
+ method. Don't call this method directly.
236
+
237
+ :param key: The variable name to look up.
238
+ """
239
+ if key in self.vars:
240
+ return self.vars[key]
241
+
242
+ if key in self.parent:
243
+ return self.parent[key]
244
+
245
+ return missing
246
+
247
+ def get_exported(self) -> t.Dict[str, t.Any]:
248
+ """Get a new dict with the exported variables."""
249
+ return {k: self.vars[k] for k in self.exported_vars}
250
+
251
+ def get_all(self) -> t.Dict[str, t.Any]:
252
+ """Return the complete context as dict including the exported
253
+ variables. For optimizations reasons this might not return an
254
+ actual copy so be careful with using it.
255
+ """
256
+ if not self.vars:
257
+ return self.parent
258
+ if not self.parent:
259
+ return self.vars
260
+ return dict(self.parent, **self.vars)
261
+
262
+ @internalcode
263
+ def call(
264
+ __self,
265
+ __obj: t.Callable[..., t.Any],
266
+ *args: t.Any,
267
+ **kwargs: t.Any, # noqa: B902
268
+ ) -> t.Union[t.Any, "Undefined"]:
269
+ """Call the callable with the arguments and keyword arguments
270
+ provided but inject the active context or environment as first
271
+ argument if the callable has :func:`pass_context` or
272
+ :func:`pass_environment`.
273
+ """
274
+ if __debug__:
275
+ __traceback_hide__ = True # noqa
276
+
277
+ # Allow callable classes to take a context
278
+ if (
279
+ hasattr(__obj, "__call__") # noqa: B004
280
+ and _PassArg.from_obj(__obj.__call__) is not None
281
+ ):
282
+ __obj = __obj.__call__
283
+
284
+ pass_arg = _PassArg.from_obj(__obj)
285
+
286
+ if pass_arg is _PassArg.context:
287
+ # the active context should have access to variables set in
288
+ # loops and blocks without mutating the context itself
289
+ if kwargs.get("_loop_vars"):
290
+ __self = __self.derived(kwargs["_loop_vars"])
291
+ if kwargs.get("_block_vars"):
292
+ __self = __self.derived(kwargs["_block_vars"])
293
+ args = (__self,) + args
294
+ elif pass_arg is _PassArg.eval_context:
295
+ args = (__self.eval_ctx,) + args
296
+ elif pass_arg is _PassArg.environment:
297
+ args = (__self.environment,) + args
298
+
299
+ kwargs.pop("_block_vars", None)
300
+ kwargs.pop("_loop_vars", None)
301
+
302
+ try:
303
+ return __obj(*args, **kwargs)
304
+ except StopIteration:
305
+ return __self.environment.undefined(
306
+ "value was undefined because a callable raised a"
307
+ " StopIteration exception"
308
+ )
309
+
310
+ def derived(self, locals: t.Optional[t.Dict[str, t.Any]] = None) -> "Context":
311
+ """Internal helper function to create a derived context. This is
312
+ used in situations where the system needs a new context in the same
313
+ template that is independent.
314
+ """
315
+ context = new_context(
316
+ self.environment, self.name, {}, self.get_all(), True, None, locals
317
+ )
318
+ context.eval_ctx = self.eval_ctx
319
+ context.blocks.update((k, list(v)) for k, v in self.blocks.items())
320
+ return context
321
+
322
+ keys = _dict_method_all(dict.keys)
323
+ values = _dict_method_all(dict.values)
324
+ items = _dict_method_all(dict.items)
325
+
326
+ def __contains__(self, name: str) -> bool:
327
+ return name in self.vars or name in self.parent
328
+
329
+ def __getitem__(self, key: str) -> t.Any:
330
+ """Look up a variable by name with ``[]`` syntax, or raise a
331
+ ``KeyError`` if the key is not found.
332
+ """
333
+ item = self.resolve_or_missing(key)
334
+
335
+ if item is missing:
336
+ raise KeyError(key)
337
+
338
+ return item
339
+
340
+ def __repr__(self) -> str:
341
+ return f"<{type(self).__name__} {self.get_all()!r} of {self.name!r}>"
342
+
343
+
344
+ class BlockReference:
345
+ """One block on a template reference."""
346
+
347
+ def __init__(
348
+ self,
349
+ name: str,
350
+ context: "Context",
351
+ stack: t.List[t.Callable[["Context"], t.Iterator[str]]],
352
+ depth: int,
353
+ ) -> None:
354
+ self.name = name
355
+ self._context = context
356
+ self._stack = stack
357
+ self._depth = depth
358
+
359
+ @property
360
+ def super(self) -> t.Union["BlockReference", "Undefined"]:
361
+ """Super the block."""
362
+ if self._depth + 1 >= len(self._stack):
363
+ return self._context.environment.undefined(
364
+ f"there is no parent block called {self.name!r}.", name="super"
365
+ )
366
+ return BlockReference(self.name, self._context, self._stack, self._depth + 1)
367
+
368
+ @internalcode
369
+ async def _async_call(self) -> str:
370
+ rv = self._context.environment.concat( # type: ignore
371
+ [x async for x in self._stack[self._depth](self._context)] # type: ignore
372
+ )
373
+
374
+ if self._context.eval_ctx.autoescape:
375
+ return Markup(rv)
376
+
377
+ return rv
378
+
379
+ @internalcode
380
+ def __call__(self) -> str:
381
+ if self._context.environment.is_async:
382
+ return self._async_call() # type: ignore
383
+
384
+ rv = self._context.environment.concat( # type: ignore
385
+ self._stack[self._depth](self._context)
386
+ )
387
+
388
+ if self._context.eval_ctx.autoescape:
389
+ return Markup(rv)
390
+
391
+ return rv
392
+
393
+
394
+ class LoopContext:
395
+ """A wrapper iterable for dynamic ``for`` loops, with information
396
+ about the loop and iteration.
397
+ """
398
+
399
+ #: Current iteration of the loop, starting at 0.
400
+ index0 = -1
401
+
402
+ _length: t.Optional[int] = None
403
+ _after: t.Any = missing
404
+ _current: t.Any = missing
405
+ _before: t.Any = missing
406
+ _last_changed_value: t.Any = missing
407
+
408
+ def __init__(
409
+ self,
410
+ iterable: t.Iterable[V],
411
+ undefined: t.Type["Undefined"],
412
+ recurse: t.Optional["LoopRenderFunc"] = None,
413
+ depth0: int = 0,
414
+ ) -> None:
415
+ """
416
+ :param iterable: Iterable to wrap.
417
+ :param undefined: :class:`Undefined` class to use for next and
418
+ previous items.
419
+ :param recurse: The function to render the loop body when the
420
+ loop is marked recursive.
421
+ :param depth0: Incremented when looping recursively.
422
+ """
423
+ self._iterable = iterable
424
+ self._iterator = self._to_iterator(iterable)
425
+ self._undefined = undefined
426
+ self._recurse = recurse
427
+ #: How many levels deep a recursive loop currently is, starting at 0.
428
+ self.depth0 = depth0
429
+
430
+ @staticmethod
431
+ def _to_iterator(iterable: t.Iterable[V]) -> t.Iterator[V]:
432
+ return iter(iterable)
433
+
434
+ @property
435
+ def length(self) -> int:
436
+ """Length of the iterable.
437
+
438
+ If the iterable is a generator or otherwise does not have a
439
+ size, it is eagerly evaluated to get a size.
440
+ """
441
+ if self._length is not None:
442
+ return self._length
443
+
444
+ try:
445
+ self._length = len(self._iterable) # type: ignore
446
+ except TypeError:
447
+ iterable = list(self._iterator)
448
+ self._iterator = self._to_iterator(iterable)
449
+ self._length = len(iterable) + self.index + (self._after is not missing)
450
+
451
+ return self._length
452
+
453
+ def __len__(self) -> int:
454
+ return self.length
455
+
456
+ @property
457
+ def depth(self) -> int:
458
+ """How many levels deep a recursive loop currently is, starting at 1."""
459
+ return self.depth0 + 1
460
+
461
+ @property
462
+ def index(self) -> int:
463
+ """Current iteration of the loop, starting at 1."""
464
+ return self.index0 + 1
465
+
466
+ @property
467
+ def revindex0(self) -> int:
468
+ """Number of iterations from the end of the loop, ending at 0.
469
+
470
+ Requires calculating :attr:`length`.
471
+ """
472
+ return self.length - self.index
473
+
474
+ @property
475
+ def revindex(self) -> int:
476
+ """Number of iterations from the end of the loop, ending at 1.
477
+
478
+ Requires calculating :attr:`length`.
479
+ """
480
+ return self.length - self.index0
481
+
482
+ @property
483
+ def first(self) -> bool:
484
+ """Whether this is the first iteration of the loop."""
485
+ return self.index0 == 0
486
+
487
+ def _peek_next(self) -> t.Any:
488
+ """Return the next element in the iterable, or :data:`missing`
489
+ if the iterable is exhausted. Only peeks one item ahead, caching
490
+ the result in :attr:`_last` for use in subsequent checks. The
491
+ cache is reset when :meth:`__next__` is called.
492
+ """
493
+ if self._after is not missing:
494
+ return self._after
495
+
496
+ self._after = next(self._iterator, missing)
497
+ return self._after
498
+
499
+ @property
500
+ def last(self) -> bool:
501
+ """Whether this is the last iteration of the loop.
502
+
503
+ Causes the iterable to advance early. See
504
+ :func:`itertools.groupby` for issues this can cause.
505
+ The :func:`groupby` filter avoids that issue.
506
+ """
507
+ return self._peek_next() is missing
508
+
509
+ @property
510
+ def previtem(self) -> t.Union[t.Any, "Undefined"]:
511
+ """The item in the previous iteration. Undefined during the
512
+ first iteration.
513
+ """
514
+ if self.first:
515
+ return self._undefined("there is no previous item")
516
+
517
+ return self._before
518
+
519
+ @property
520
+ def nextitem(self) -> t.Union[t.Any, "Undefined"]:
521
+ """The item in the next iteration. Undefined during the last
522
+ iteration.
523
+
524
+ Causes the iterable to advance early. See
525
+ :func:`itertools.groupby` for issues this can cause.
526
+ The :func:`jinja-filters.groupby` filter avoids that issue.
527
+ """
528
+ rv = self._peek_next()
529
+
530
+ if rv is missing:
531
+ return self._undefined("there is no next item")
532
+
533
+ return rv
534
+
535
+ def cycle(self, *args: V) -> V:
536
+ """Return a value from the given args, cycling through based on
537
+ the current :attr:`index0`.
538
+
539
+ :param args: One or more values to cycle through.
540
+ """
541
+ if not args:
542
+ raise TypeError("no items for cycling given")
543
+
544
+ return args[self.index0 % len(args)]
545
+
546
+ def changed(self, *value: t.Any) -> bool:
547
+ """Return ``True`` if previously called with a different value
548
+ (including when called for the first time).
549
+
550
+ :param value: One or more values to compare to the last call.
551
+ """
552
+ if self._last_changed_value != value:
553
+ self._last_changed_value = value
554
+ return True
555
+
556
+ return False
557
+
558
+ def __iter__(self) -> "LoopContext":
559
+ return self
560
+
561
+ def __next__(self) -> t.Tuple[t.Any, "LoopContext"]:
562
+ if self._after is not missing:
563
+ rv = self._after
564
+ self._after = missing
565
+ else:
566
+ rv = next(self._iterator)
567
+
568
+ self.index0 += 1
569
+ self._before = self._current
570
+ self._current = rv
571
+ return rv, self
572
+
573
+ @internalcode
574
+ def __call__(self, iterable: t.Iterable[V]) -> str:
575
+ """When iterating over nested data, render the body of the loop
576
+ recursively with the given inner iterable data.
577
+
578
+ The loop must have the ``recursive`` marker for this to work.
579
+ """
580
+ if self._recurse is None:
581
+ raise TypeError(
582
+ "The loop must have the 'recursive' marker to be called recursively."
583
+ )
584
+
585
+ return self._recurse(iterable, self._recurse, depth=self.depth)
586
+
587
+ def __repr__(self) -> str:
588
+ return f"<{type(self).__name__} {self.index}/{self.length}>"
589
+
590
+
591
+ class AsyncLoopContext(LoopContext):
592
+ _iterator: t.AsyncIterator[t.Any] # type: ignore
593
+
594
+ @staticmethod
595
+ def _to_iterator( # type: ignore
596
+ iterable: t.Union[t.Iterable[V], t.AsyncIterable[V]],
597
+ ) -> t.AsyncIterator[V]:
598
+ return auto_aiter(iterable)
599
+
600
+ @property
601
+ async def length(self) -> int: # type: ignore
602
+ if self._length is not None:
603
+ return self._length
604
+
605
+ try:
606
+ self._length = len(self._iterable) # type: ignore
607
+ except TypeError:
608
+ iterable = [x async for x in self._iterator]
609
+ self._iterator = self._to_iterator(iterable)
610
+ self._length = len(iterable) + self.index + (self._after is not missing)
611
+
612
+ return self._length
613
+
614
+ @property
615
+ async def revindex0(self) -> int: # type: ignore
616
+ return await self.length - self.index
617
+
618
+ @property
619
+ async def revindex(self) -> int: # type: ignore
620
+ return await self.length - self.index0
621
+
622
+ async def _peek_next(self) -> t.Any:
623
+ if self._after is not missing:
624
+ return self._after
625
+
626
+ try:
627
+ self._after = await self._iterator.__anext__()
628
+ except StopAsyncIteration:
629
+ self._after = missing
630
+
631
+ return self._after
632
+
633
+ @property
634
+ async def last(self) -> bool: # type: ignore
635
+ return await self._peek_next() is missing
636
+
637
+ @property
638
+ async def nextitem(self) -> t.Union[t.Any, "Undefined"]:
639
+ rv = await self._peek_next()
640
+
641
+ if rv is missing:
642
+ return self._undefined("there is no next item")
643
+
644
+ return rv
645
+
646
+ def __aiter__(self) -> "AsyncLoopContext":
647
+ return self
648
+
649
+ async def __anext__(self) -> t.Tuple[t.Any, "AsyncLoopContext"]:
650
+ if self._after is not missing:
651
+ rv = self._after
652
+ self._after = missing
653
+ else:
654
+ rv = await self._iterator.__anext__()
655
+
656
+ self.index0 += 1
657
+ self._before = self._current
658
+ self._current = rv
659
+ return rv, self
660
+
661
+
662
+ class Macro:
663
+ """Wraps a macro function."""
664
+
665
+ def __init__(
666
+ self,
667
+ environment: "Environment",
668
+ func: t.Callable[..., str],
669
+ name: str,
670
+ arguments: t.List[str],
671
+ catch_kwargs: bool,
672
+ catch_varargs: bool,
673
+ caller: bool,
674
+ default_autoescape: t.Optional[bool] = None,
675
+ ):
676
+ self._environment = environment
677
+ self._func = func
678
+ self._argument_count = len(arguments)
679
+ self.name = name
680
+ self.arguments = arguments
681
+ self.catch_kwargs = catch_kwargs
682
+ self.catch_varargs = catch_varargs
683
+ self.caller = caller
684
+ self.explicit_caller = "caller" in arguments
685
+
686
+ if default_autoescape is None:
687
+ if callable(environment.autoescape):
688
+ default_autoescape = environment.autoescape(None)
689
+ else:
690
+ default_autoescape = environment.autoescape
691
+
692
+ self._default_autoescape = default_autoescape
693
+
694
+ @internalcode
695
+ @pass_eval_context
696
+ def __call__(self, *args: t.Any, **kwargs: t.Any) -> str:
697
+ # This requires a bit of explanation, In the past we used to
698
+ # decide largely based on compile-time information if a macro is
699
+ # safe or unsafe. While there was a volatile mode it was largely
700
+ # unused for deciding on escaping. This turns out to be
701
+ # problematic for macros because whether a macro is safe depends not
702
+ # on the escape mode when it was defined, but rather when it was used.
703
+ #
704
+ # Because however we export macros from the module system and
705
+ # there are historic callers that do not pass an eval context (and
706
+ # will continue to not pass one), we need to perform an instance
707
+ # check here.
708
+ #
709
+ # This is considered safe because an eval context is not a valid
710
+ # argument to callables otherwise anyway. Worst case here is
711
+ # that if no eval context is passed we fall back to the compile
712
+ # time autoescape flag.
713
+ if args and isinstance(args[0], EvalContext):
714
+ autoescape = args[0].autoescape
715
+ args = args[1:]
716
+ else:
717
+ autoescape = self._default_autoescape
718
+
719
+ # try to consume the positional arguments
720
+ arguments = list(args[: self._argument_count])
721
+ off = len(arguments)
722
+
723
+ # For information why this is necessary refer to the handling
724
+ # of caller in the `macro_body` handler in the compiler.
725
+ found_caller = False
726
+
727
+ # if the number of arguments consumed is not the number of
728
+ # arguments expected we start filling in keyword arguments
729
+ # and defaults.
730
+ if off != self._argument_count:
731
+ for name in self.arguments[len(arguments) :]:
732
+ try:
733
+ value = kwargs.pop(name)
734
+ except KeyError:
735
+ value = missing
736
+ if name == "caller":
737
+ found_caller = True
738
+ arguments.append(value)
739
+ else:
740
+ found_caller = self.explicit_caller
741
+
742
+ # it's important that the order of these arguments does not change
743
+ # if not also changed in the compiler's `function_scoping` method.
744
+ # the order is caller, keyword arguments, positional arguments!
745
+ if self.caller and not found_caller:
746
+ caller = kwargs.pop("caller", None)
747
+ if caller is None:
748
+ caller = self._environment.undefined("No caller defined", name="caller")
749
+ arguments.append(caller)
750
+
751
+ if self.catch_kwargs:
752
+ arguments.append(kwargs)
753
+ elif kwargs:
754
+ if "caller" in kwargs:
755
+ raise TypeError(
756
+ f"macro {self.name!r} was invoked with two values for the special"
757
+ " caller argument. This is most likely a bug."
758
+ )
759
+ raise TypeError(
760
+ f"macro {self.name!r} takes no keyword argument {next(iter(kwargs))!r}"
761
+ )
762
+ if self.catch_varargs:
763
+ arguments.append(args[self._argument_count :])
764
+ elif len(args) > self._argument_count:
765
+ raise TypeError(
766
+ f"macro {self.name!r} takes not more than"
767
+ f" {len(self.arguments)} argument(s)"
768
+ )
769
+
770
+ return self._invoke(arguments, autoescape)
771
+
772
+ async def _async_invoke(self, arguments: t.List[t.Any], autoescape: bool) -> str:
773
+ rv = await self._func(*arguments) # type: ignore
774
+
775
+ if autoescape:
776
+ return Markup(rv)
777
+
778
+ return rv # type: ignore
779
+
780
+ def _invoke(self, arguments: t.List[t.Any], autoescape: bool) -> str:
781
+ if self._environment.is_async:
782
+ return self._async_invoke(arguments, autoescape) # type: ignore
783
+
784
+ rv = self._func(*arguments)
785
+
786
+ if autoescape:
787
+ rv = Markup(rv)
788
+
789
+ return rv
790
+
791
+ def __repr__(self) -> str:
792
+ name = "anonymous" if self.name is None else repr(self.name)
793
+ return f"<{type(self).__name__} {name}>"
794
+
795
+
796
+ class Undefined:
797
+ """The default undefined type. This can be printed, iterated, and treated as
798
+ a boolean. Any other operation will raise an :exc:`UndefinedError`.
799
+
800
+ >>> foo = Undefined(name='foo')
801
+ >>> str(foo)
802
+ ''
803
+ >>> not foo
804
+ True
805
+ >>> foo + 42
806
+ Traceback (most recent call last):
807
+ ...
808
+ jinja2.exceptions.UndefinedError: 'foo' is undefined
809
+ """
810
+
811
+ __slots__ = (
812
+ "_undefined_hint",
813
+ "_undefined_obj",
814
+ "_undefined_name",
815
+ "_undefined_exception",
816
+ )
817
+
818
+ def __init__(
819
+ self,
820
+ hint: t.Optional[str] = None,
821
+ obj: t.Any = missing,
822
+ name: t.Optional[str] = None,
823
+ exc: t.Type[TemplateRuntimeError] = UndefinedError,
824
+ ) -> None:
825
+ self._undefined_hint = hint
826
+ self._undefined_obj = obj
827
+ self._undefined_name = name
828
+ self._undefined_exception = exc
829
+
830
+ @property
831
+ def _undefined_message(self) -> str:
832
+ """Build a message about the undefined value based on how it was
833
+ accessed.
834
+ """
835
+ if self._undefined_hint:
836
+ return self._undefined_hint
837
+
838
+ if self._undefined_obj is missing:
839
+ return f"{self._undefined_name!r} is undefined"
840
+
841
+ if not isinstance(self._undefined_name, str):
842
+ return (
843
+ f"{object_type_repr(self._undefined_obj)} has no"
844
+ f" element {self._undefined_name!r}"
845
+ )
846
+
847
+ return (
848
+ f"{object_type_repr(self._undefined_obj)!r} has no"
849
+ f" attribute {self._undefined_name!r}"
850
+ )
851
+
852
+ @internalcode
853
+ def _fail_with_undefined_error(
854
+ self, *args: t.Any, **kwargs: t.Any
855
+ ) -> "te.NoReturn":
856
+ """Raise an :exc:`UndefinedError` when operations are performed
857
+ on the undefined value.
858
+ """
859
+ raise self._undefined_exception(self._undefined_message)
860
+
861
+ @internalcode
862
+ def __getattr__(self, name: str) -> t.Any:
863
+ # Raise AttributeError on requests for names that appear to be unimplemented
864
+ # dunder methods to keep Python's internal protocol probing behaviors working
865
+ # properly in cases where another exception type could cause unexpected or
866
+ # difficult-to-diagnose failures.
867
+ if name[:2] == "__" and name[-2:] == "__":
868
+ raise AttributeError(name)
869
+
870
+ return self._fail_with_undefined_error()
871
+
872
+ __add__ = __radd__ = __sub__ = __rsub__ = _fail_with_undefined_error
873
+ __mul__ = __rmul__ = __div__ = __rdiv__ = _fail_with_undefined_error
874
+ __truediv__ = __rtruediv__ = _fail_with_undefined_error
875
+ __floordiv__ = __rfloordiv__ = _fail_with_undefined_error
876
+ __mod__ = __rmod__ = _fail_with_undefined_error
877
+ __pos__ = __neg__ = _fail_with_undefined_error
878
+ __call__ = __getitem__ = _fail_with_undefined_error
879
+ __lt__ = __le__ = __gt__ = __ge__ = _fail_with_undefined_error
880
+ __int__ = __float__ = __complex__ = _fail_with_undefined_error
881
+ __pow__ = __rpow__ = _fail_with_undefined_error
882
+
883
+ def __eq__(self, other: t.Any) -> bool:
884
+ return type(self) is type(other)
885
+
886
+ def __ne__(self, other: t.Any) -> bool:
887
+ return not self.__eq__(other)
888
+
889
+ def __hash__(self) -> int:
890
+ return id(type(self))
891
+
892
+ def __str__(self) -> str:
893
+ return ""
894
+
895
+ def __len__(self) -> int:
896
+ return 0
897
+
898
+ def __iter__(self) -> t.Iterator[t.Any]:
899
+ yield from ()
900
+
901
+ async def __aiter__(self) -> t.AsyncIterator[t.Any]:
902
+ for _ in ():
903
+ yield
904
+
905
+ def __bool__(self) -> bool:
906
+ return False
907
+
908
+ def __repr__(self) -> str:
909
+ return "Undefined"
910
+
911
+
912
+ def make_logging_undefined(
913
+ logger: t.Optional["logging.Logger"] = None, base: t.Type[Undefined] = Undefined
914
+ ) -> t.Type[Undefined]:
915
+ """Given a logger object this returns a new undefined class that will
916
+ log certain failures. It will log iterations and printing. If no
917
+ logger is given a default logger is created.
918
+
919
+ Example::
920
+
921
+ logger = logging.getLogger(__name__)
922
+ LoggingUndefined = make_logging_undefined(
923
+ logger=logger,
924
+ base=Undefined
925
+ )
926
+
927
+ .. versionadded:: 2.8
928
+
929
+ :param logger: the logger to use. If not provided, a default logger
930
+ is created.
931
+ :param base: the base class to add logging functionality to. This
932
+ defaults to :class:`Undefined`.
933
+ """
934
+ if logger is None:
935
+ import logging
936
+
937
+ logger = logging.getLogger(__name__)
938
+ logger.addHandler(logging.StreamHandler(sys.stderr))
939
+
940
+ def _log_message(undef: Undefined) -> None:
941
+ logger.warning("Template variable warning: %s", undef._undefined_message)
942
+
943
+ class LoggingUndefined(base): # type: ignore
944
+ __slots__ = ()
945
+
946
+ def _fail_with_undefined_error( # type: ignore
947
+ self, *args: t.Any, **kwargs: t.Any
948
+ ) -> "te.NoReturn":
949
+ try:
950
+ super()._fail_with_undefined_error(*args, **kwargs)
951
+ except self._undefined_exception as e:
952
+ logger.error("Template variable error: %s", e) # type: ignore
953
+ raise e
954
+
955
+ def __str__(self) -> str:
956
+ _log_message(self)
957
+ return super().__str__() # type: ignore
958
+
959
+ def __iter__(self) -> t.Iterator[t.Any]:
960
+ _log_message(self)
961
+ return super().__iter__() # type: ignore
962
+
963
+ def __bool__(self) -> bool:
964
+ _log_message(self)
965
+ return super().__bool__() # type: ignore
966
+
967
+ return LoggingUndefined
968
+
969
+
970
+ class ChainableUndefined(Undefined):
971
+ """An undefined that is chainable, where both ``__getattr__`` and
972
+ ``__getitem__`` return itself rather than raising an
973
+ :exc:`UndefinedError`.
974
+
975
+ >>> foo = ChainableUndefined(name='foo')
976
+ >>> str(foo.bar['baz'])
977
+ ''
978
+ >>> foo.bar['baz'] + 42
979
+ Traceback (most recent call last):
980
+ ...
981
+ jinja2.exceptions.UndefinedError: 'foo' is undefined
982
+
983
+ .. versionadded:: 2.11.0
984
+ """
985
+
986
+ __slots__ = ()
987
+
988
+ def __html__(self) -> str:
989
+ return str(self)
990
+
991
+ def __getattr__(self, name: str) -> "ChainableUndefined":
992
+ # Raise AttributeError on requests for names that appear to be unimplemented
993
+ # dunder methods to avoid confusing Python with truthy non-method objects that
994
+ # do not implement the protocol being probed for. e.g., copy.copy(Undefined())
995
+ # fails spectacularly if getattr(Undefined(), '__setstate__') returns an
996
+ # Undefined object instead of raising AttributeError to signal that it does not
997
+ # support that style of object initialization.
998
+ if name[:2] == "__" and name[-2:] == "__":
999
+ raise AttributeError(name)
1000
+
1001
+ return self
1002
+
1003
+ def __getitem__(self, _name: str) -> "ChainableUndefined": # type: ignore[override]
1004
+ return self
1005
+
1006
+
1007
+ class DebugUndefined(Undefined):
1008
+ """An undefined that returns the debug info when printed.
1009
+
1010
+ >>> foo = DebugUndefined(name='foo')
1011
+ >>> str(foo)
1012
+ '{{ foo }}'
1013
+ >>> not foo
1014
+ True
1015
+ >>> foo + 42
1016
+ Traceback (most recent call last):
1017
+ ...
1018
+ jinja2.exceptions.UndefinedError: 'foo' is undefined
1019
+ """
1020
+
1021
+ __slots__ = ()
1022
+
1023
+ def __str__(self) -> str:
1024
+ if self._undefined_hint:
1025
+ message = f"undefined value printed: {self._undefined_hint}"
1026
+
1027
+ elif self._undefined_obj is missing:
1028
+ message = self._undefined_name # type: ignore
1029
+
1030
+ else:
1031
+ message = (
1032
+ f"no such element: {object_type_repr(self._undefined_obj)}"
1033
+ f"[{self._undefined_name!r}]"
1034
+ )
1035
+
1036
+ return f"{{{{ {message} }}}}"
1037
+
1038
+
1039
+ class StrictUndefined(Undefined):
1040
+ """An undefined that barks on print and iteration as well as boolean
1041
+ tests and all kinds of comparisons. In other words: you can do nothing
1042
+ with it except checking if it's defined using the `defined` test.
1043
+
1044
+ >>> foo = StrictUndefined(name='foo')
1045
+ >>> str(foo)
1046
+ Traceback (most recent call last):
1047
+ ...
1048
+ jinja2.exceptions.UndefinedError: 'foo' is undefined
1049
+ >>> not foo
1050
+ Traceback (most recent call last):
1051
+ ...
1052
+ jinja2.exceptions.UndefinedError: 'foo' is undefined
1053
+ >>> foo + 42
1054
+ Traceback (most recent call last):
1055
+ ...
1056
+ jinja2.exceptions.UndefinedError: 'foo' is undefined
1057
+ """
1058
+
1059
+ __slots__ = ()
1060
+ __iter__ = __str__ = __len__ = Undefined._fail_with_undefined_error
1061
+ __eq__ = __ne__ = __bool__ = __hash__ = Undefined._fail_with_undefined_error
1062
+ __contains__ = Undefined._fail_with_undefined_error
TTS/Kokoro/.venv/Lib/site-packages/jinja2/sandbox.py ADDED
@@ -0,0 +1,436 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """A sandbox layer that ensures unsafe operations cannot be performed.
2
+ Useful when the template itself comes from an untrusted source.
3
+ """
4
+
5
+ import operator
6
+ import types
7
+ import typing as t
8
+ from _string import formatter_field_name_split # type: ignore
9
+ from collections import abc
10
+ from collections import deque
11
+ from functools import update_wrapper
12
+ from string import Formatter
13
+
14
+ from markupsafe import EscapeFormatter
15
+ from markupsafe import Markup
16
+
17
+ from .environment import Environment
18
+ from .exceptions import SecurityError
19
+ from .runtime import Context
20
+ from .runtime import Undefined
21
+
22
+ F = t.TypeVar("F", bound=t.Callable[..., t.Any])
23
+
24
+ #: maximum number of items a range may produce
25
+ MAX_RANGE = 100000
26
+
27
+ #: Unsafe function attributes.
28
+ UNSAFE_FUNCTION_ATTRIBUTES: t.Set[str] = set()
29
+
30
+ #: Unsafe method attributes. Function attributes are unsafe for methods too.
31
+ UNSAFE_METHOD_ATTRIBUTES: t.Set[str] = set()
32
+
33
+ #: unsafe generator attributes.
34
+ UNSAFE_GENERATOR_ATTRIBUTES = {"gi_frame", "gi_code"}
35
+
36
+ #: unsafe attributes on coroutines
37
+ UNSAFE_COROUTINE_ATTRIBUTES = {"cr_frame", "cr_code"}
38
+
39
+ #: unsafe attributes on async generators
40
+ UNSAFE_ASYNC_GENERATOR_ATTRIBUTES = {"ag_code", "ag_frame"}
41
+
42
+ _mutable_spec: t.Tuple[t.Tuple[t.Type[t.Any], t.FrozenSet[str]], ...] = (
43
+ (
44
+ abc.MutableSet,
45
+ frozenset(
46
+ [
47
+ "add",
48
+ "clear",
49
+ "difference_update",
50
+ "discard",
51
+ "pop",
52
+ "remove",
53
+ "symmetric_difference_update",
54
+ "update",
55
+ ]
56
+ ),
57
+ ),
58
+ (
59
+ abc.MutableMapping,
60
+ frozenset(["clear", "pop", "popitem", "setdefault", "update"]),
61
+ ),
62
+ (
63
+ abc.MutableSequence,
64
+ frozenset(
65
+ ["append", "clear", "pop", "reverse", "insert", "sort", "extend", "remove"]
66
+ ),
67
+ ),
68
+ (
69
+ deque,
70
+ frozenset(
71
+ [
72
+ "append",
73
+ "appendleft",
74
+ "clear",
75
+ "extend",
76
+ "extendleft",
77
+ "pop",
78
+ "popleft",
79
+ "remove",
80
+ "rotate",
81
+ ]
82
+ ),
83
+ ),
84
+ )
85
+
86
+
87
+ def safe_range(*args: int) -> range:
88
+ """A range that can't generate ranges with a length of more than
89
+ MAX_RANGE items.
90
+ """
91
+ rng = range(*args)
92
+
93
+ if len(rng) > MAX_RANGE:
94
+ raise OverflowError(
95
+ "Range too big. The sandbox blocks ranges larger than"
96
+ f" MAX_RANGE ({MAX_RANGE})."
97
+ )
98
+
99
+ return rng
100
+
101
+
102
+ def unsafe(f: F) -> F:
103
+ """Marks a function or method as unsafe.
104
+
105
+ .. code-block: python
106
+
107
+ @unsafe
108
+ def delete(self):
109
+ pass
110
+ """
111
+ f.unsafe_callable = True # type: ignore
112
+ return f
113
+
114
+
115
+ def is_internal_attribute(obj: t.Any, attr: str) -> bool:
116
+ """Test if the attribute given is an internal python attribute. For
117
+ example this function returns `True` for the `func_code` attribute of
118
+ python objects. This is useful if the environment method
119
+ :meth:`~SandboxedEnvironment.is_safe_attribute` is overridden.
120
+
121
+ >>> from jinja2.sandbox import is_internal_attribute
122
+ >>> is_internal_attribute(str, "mro")
123
+ True
124
+ >>> is_internal_attribute(str, "upper")
125
+ False
126
+ """
127
+ if isinstance(obj, types.FunctionType):
128
+ if attr in UNSAFE_FUNCTION_ATTRIBUTES:
129
+ return True
130
+ elif isinstance(obj, types.MethodType):
131
+ if attr in UNSAFE_FUNCTION_ATTRIBUTES or attr in UNSAFE_METHOD_ATTRIBUTES:
132
+ return True
133
+ elif isinstance(obj, type):
134
+ if attr == "mro":
135
+ return True
136
+ elif isinstance(obj, (types.CodeType, types.TracebackType, types.FrameType)):
137
+ return True
138
+ elif isinstance(obj, types.GeneratorType):
139
+ if attr in UNSAFE_GENERATOR_ATTRIBUTES:
140
+ return True
141
+ elif hasattr(types, "CoroutineType") and isinstance(obj, types.CoroutineType):
142
+ if attr in UNSAFE_COROUTINE_ATTRIBUTES:
143
+ return True
144
+ elif hasattr(types, "AsyncGeneratorType") and isinstance(
145
+ obj, types.AsyncGeneratorType
146
+ ):
147
+ if attr in UNSAFE_ASYNC_GENERATOR_ATTRIBUTES:
148
+ return True
149
+ return attr.startswith("__")
150
+
151
+
152
+ def modifies_known_mutable(obj: t.Any, attr: str) -> bool:
153
+ """This function checks if an attribute on a builtin mutable object
154
+ (list, dict, set or deque) or the corresponding ABCs would modify it
155
+ if called.
156
+
157
+ >>> modifies_known_mutable({}, "clear")
158
+ True
159
+ >>> modifies_known_mutable({}, "keys")
160
+ False
161
+ >>> modifies_known_mutable([], "append")
162
+ True
163
+ >>> modifies_known_mutable([], "index")
164
+ False
165
+
166
+ If called with an unsupported object, ``False`` is returned.
167
+
168
+ >>> modifies_known_mutable("foo", "upper")
169
+ False
170
+ """
171
+ for typespec, unsafe in _mutable_spec:
172
+ if isinstance(obj, typespec):
173
+ return attr in unsafe
174
+ return False
175
+
176
+
177
+ class SandboxedEnvironment(Environment):
178
+ """The sandboxed environment. It works like the regular environment but
179
+ tells the compiler to generate sandboxed code. Additionally subclasses of
180
+ this environment may override the methods that tell the runtime what
181
+ attributes or functions are safe to access.
182
+
183
+ If the template tries to access insecure code a :exc:`SecurityError` is
184
+ raised. However also other exceptions may occur during the rendering so
185
+ the caller has to ensure that all exceptions are caught.
186
+ """
187
+
188
+ sandboxed = True
189
+
190
+ #: default callback table for the binary operators. A copy of this is
191
+ #: available on each instance of a sandboxed environment as
192
+ #: :attr:`binop_table`
193
+ default_binop_table: t.Dict[str, t.Callable[[t.Any, t.Any], t.Any]] = {
194
+ "+": operator.add,
195
+ "-": operator.sub,
196
+ "*": operator.mul,
197
+ "/": operator.truediv,
198
+ "//": operator.floordiv,
199
+ "**": operator.pow,
200
+ "%": operator.mod,
201
+ }
202
+
203
+ #: default callback table for the unary operators. A copy of this is
204
+ #: available on each instance of a sandboxed environment as
205
+ #: :attr:`unop_table`
206
+ default_unop_table: t.Dict[str, t.Callable[[t.Any], t.Any]] = {
207
+ "+": operator.pos,
208
+ "-": operator.neg,
209
+ }
210
+
211
+ #: a set of binary operators that should be intercepted. Each operator
212
+ #: that is added to this set (empty by default) is delegated to the
213
+ #: :meth:`call_binop` method that will perform the operator. The default
214
+ #: operator callback is specified by :attr:`binop_table`.
215
+ #:
216
+ #: The following binary operators are interceptable:
217
+ #: ``//``, ``%``, ``+``, ``*``, ``-``, ``/``, and ``**``
218
+ #:
219
+ #: The default operation form the operator table corresponds to the
220
+ #: builtin function. Intercepted calls are always slower than the native
221
+ #: operator call, so make sure only to intercept the ones you are
222
+ #: interested in.
223
+ #:
224
+ #: .. versionadded:: 2.6
225
+ intercepted_binops: t.FrozenSet[str] = frozenset()
226
+
227
+ #: a set of unary operators that should be intercepted. Each operator
228
+ #: that is added to this set (empty by default) is delegated to the
229
+ #: :meth:`call_unop` method that will perform the operator. The default
230
+ #: operator callback is specified by :attr:`unop_table`.
231
+ #:
232
+ #: The following unary operators are interceptable: ``+``, ``-``
233
+ #:
234
+ #: The default operation form the operator table corresponds to the
235
+ #: builtin function. Intercepted calls are always slower than the native
236
+ #: operator call, so make sure only to intercept the ones you are
237
+ #: interested in.
238
+ #:
239
+ #: .. versionadded:: 2.6
240
+ intercepted_unops: t.FrozenSet[str] = frozenset()
241
+
242
+ def __init__(self, *args: t.Any, **kwargs: t.Any) -> None:
243
+ super().__init__(*args, **kwargs)
244
+ self.globals["range"] = safe_range
245
+ self.binop_table = self.default_binop_table.copy()
246
+ self.unop_table = self.default_unop_table.copy()
247
+
248
+ def is_safe_attribute(self, obj: t.Any, attr: str, value: t.Any) -> bool:
249
+ """The sandboxed environment will call this method to check if the
250
+ attribute of an object is safe to access. Per default all attributes
251
+ starting with an underscore are considered private as well as the
252
+ special attributes of internal python objects as returned by the
253
+ :func:`is_internal_attribute` function.
254
+ """
255
+ return not (attr.startswith("_") or is_internal_attribute(obj, attr))
256
+
257
+ def is_safe_callable(self, obj: t.Any) -> bool:
258
+ """Check if an object is safely callable. By default callables
259
+ are considered safe unless decorated with :func:`unsafe`.
260
+
261
+ This also recognizes the Django convention of setting
262
+ ``func.alters_data = True``.
263
+ """
264
+ return not (
265
+ getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
266
+ )
267
+
268
+ def call_binop(
269
+ self, context: Context, operator: str, left: t.Any, right: t.Any
270
+ ) -> t.Any:
271
+ """For intercepted binary operator calls (:meth:`intercepted_binops`)
272
+ this function is executed instead of the builtin operator. This can
273
+ be used to fine tune the behavior of certain operators.
274
+
275
+ .. versionadded:: 2.6
276
+ """
277
+ return self.binop_table[operator](left, right)
278
+
279
+ def call_unop(self, context: Context, operator: str, arg: t.Any) -> t.Any:
280
+ """For intercepted unary operator calls (:meth:`intercepted_unops`)
281
+ this function is executed instead of the builtin operator. This can
282
+ be used to fine tune the behavior of certain operators.
283
+
284
+ .. versionadded:: 2.6
285
+ """
286
+ return self.unop_table[operator](arg)
287
+
288
+ def getitem(
289
+ self, obj: t.Any, argument: t.Union[str, t.Any]
290
+ ) -> t.Union[t.Any, Undefined]:
291
+ """Subscribe an object from sandboxed code."""
292
+ try:
293
+ return obj[argument]
294
+ except (TypeError, LookupError):
295
+ if isinstance(argument, str):
296
+ try:
297
+ attr = str(argument)
298
+ except Exception:
299
+ pass
300
+ else:
301
+ try:
302
+ value = getattr(obj, attr)
303
+ except AttributeError:
304
+ pass
305
+ else:
306
+ fmt = self.wrap_str_format(value)
307
+ if fmt is not None:
308
+ return fmt
309
+ if self.is_safe_attribute(obj, argument, value):
310
+ return value
311
+ return self.unsafe_undefined(obj, argument)
312
+ return self.undefined(obj=obj, name=argument)
313
+
314
+ def getattr(self, obj: t.Any, attribute: str) -> t.Union[t.Any, Undefined]:
315
+ """Subscribe an object from sandboxed code and prefer the
316
+ attribute. The attribute passed *must* be a bytestring.
317
+ """
318
+ try:
319
+ value = getattr(obj, attribute)
320
+ except AttributeError:
321
+ try:
322
+ return obj[attribute]
323
+ except (TypeError, LookupError):
324
+ pass
325
+ else:
326
+ fmt = self.wrap_str_format(value)
327
+ if fmt is not None:
328
+ return fmt
329
+ if self.is_safe_attribute(obj, attribute, value):
330
+ return value
331
+ return self.unsafe_undefined(obj, attribute)
332
+ return self.undefined(obj=obj, name=attribute)
333
+
334
+ def unsafe_undefined(self, obj: t.Any, attribute: str) -> Undefined:
335
+ """Return an undefined object for unsafe attributes."""
336
+ return self.undefined(
337
+ f"access to attribute {attribute!r} of"
338
+ f" {type(obj).__name__!r} object is unsafe.",
339
+ name=attribute,
340
+ obj=obj,
341
+ exc=SecurityError,
342
+ )
343
+
344
+ def wrap_str_format(self, value: t.Any) -> t.Optional[t.Callable[..., str]]:
345
+ """If the given value is a ``str.format`` or ``str.format_map`` method,
346
+ return a new function than handles sandboxing. This is done at access
347
+ rather than in :meth:`call`, so that calls made without ``call`` are
348
+ also sandboxed.
349
+ """
350
+ if not isinstance(
351
+ value, (types.MethodType, types.BuiltinMethodType)
352
+ ) or value.__name__ not in ("format", "format_map"):
353
+ return None
354
+
355
+ f_self: t.Any = value.__self__
356
+
357
+ if not isinstance(f_self, str):
358
+ return None
359
+
360
+ str_type: t.Type[str] = type(f_self)
361
+ is_format_map = value.__name__ == "format_map"
362
+ formatter: SandboxedFormatter
363
+
364
+ if isinstance(f_self, Markup):
365
+ formatter = SandboxedEscapeFormatter(self, escape=f_self.escape)
366
+ else:
367
+ formatter = SandboxedFormatter(self)
368
+
369
+ vformat = formatter.vformat
370
+
371
+ def wrapper(*args: t.Any, **kwargs: t.Any) -> str:
372
+ if is_format_map:
373
+ if kwargs:
374
+ raise TypeError("format_map() takes no keyword arguments")
375
+
376
+ if len(args) != 1:
377
+ raise TypeError(
378
+ f"format_map() takes exactly one argument ({len(args)} given)"
379
+ )
380
+
381
+ kwargs = args[0]
382
+ args = ()
383
+
384
+ return str_type(vformat(f_self, args, kwargs))
385
+
386
+ return update_wrapper(wrapper, value)
387
+
388
+ def call(
389
+ __self, # noqa: B902
390
+ __context: Context,
391
+ __obj: t.Any,
392
+ *args: t.Any,
393
+ **kwargs: t.Any,
394
+ ) -> t.Any:
395
+ """Call an object from sandboxed code."""
396
+
397
+ # the double prefixes are to avoid double keyword argument
398
+ # errors when proxying the call.
399
+ if not __self.is_safe_callable(__obj):
400
+ raise SecurityError(f"{__obj!r} is not safely callable")
401
+ return __context.call(__obj, *args, **kwargs)
402
+
403
+
404
+ class ImmutableSandboxedEnvironment(SandboxedEnvironment):
405
+ """Works exactly like the regular `SandboxedEnvironment` but does not
406
+ permit modifications on the builtin mutable objects `list`, `set`, and
407
+ `dict` by using the :func:`modifies_known_mutable` function.
408
+ """
409
+
410
+ def is_safe_attribute(self, obj: t.Any, attr: str, value: t.Any) -> bool:
411
+ if not super().is_safe_attribute(obj, attr, value):
412
+ return False
413
+
414
+ return not modifies_known_mutable(obj, attr)
415
+
416
+
417
+ class SandboxedFormatter(Formatter):
418
+ def __init__(self, env: Environment, **kwargs: t.Any) -> None:
419
+ self._env = env
420
+ super().__init__(**kwargs)
421
+
422
+ def get_field(
423
+ self, field_name: str, args: t.Sequence[t.Any], kwargs: t.Mapping[str, t.Any]
424
+ ) -> t.Tuple[t.Any, str]:
425
+ first, rest = formatter_field_name_split(field_name)
426
+ obj = self.get_value(first, args, kwargs)
427
+ for is_attr, i in rest:
428
+ if is_attr:
429
+ obj = self._env.getattr(obj, i)
430
+ else:
431
+ obj = self._env.getitem(obj, i)
432
+ return obj, first
433
+
434
+
435
+ class SandboxedEscapeFormatter(SandboxedFormatter, EscapeFormatter):
436
+ pass
TTS/Kokoro/.venv/Lib/site-packages/jinja2/tests.py ADDED
@@ -0,0 +1,256 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Built-in template tests used with the ``is`` operator."""
2
+
3
+ import operator
4
+ import typing as t
5
+ from collections import abc
6
+ from numbers import Number
7
+
8
+ from .runtime import Undefined
9
+ from .utils import pass_environment
10
+
11
+ if t.TYPE_CHECKING:
12
+ from .environment import Environment
13
+
14
+
15
+ def test_odd(value: int) -> bool:
16
+ """Return true if the variable is odd."""
17
+ return value % 2 == 1
18
+
19
+
20
+ def test_even(value: int) -> bool:
21
+ """Return true if the variable is even."""
22
+ return value % 2 == 0
23
+
24
+
25
+ def test_divisibleby(value: int, num: int) -> bool:
26
+ """Check if a variable is divisible by a number."""
27
+ return value % num == 0
28
+
29
+
30
+ def test_defined(value: t.Any) -> bool:
31
+ """Return true if the variable is defined:
32
+
33
+ .. sourcecode:: jinja
34
+
35
+ {% if variable is defined %}
36
+ value of variable: {{ variable }}
37
+ {% else %}
38
+ variable is not defined
39
+ {% endif %}
40
+
41
+ See the :func:`default` filter for a simple way to set undefined
42
+ variables.
43
+ """
44
+ return not isinstance(value, Undefined)
45
+
46
+
47
+ def test_undefined(value: t.Any) -> bool:
48
+ """Like :func:`defined` but the other way round."""
49
+ return isinstance(value, Undefined)
50
+
51
+
52
+ @pass_environment
53
+ def test_filter(env: "Environment", value: str) -> bool:
54
+ """Check if a filter exists by name. Useful if a filter may be
55
+ optionally available.
56
+
57
+ .. code-block:: jinja
58
+
59
+ {% if 'markdown' is filter %}
60
+ {{ value | markdown }}
61
+ {% else %}
62
+ {{ value }}
63
+ {% endif %}
64
+
65
+ .. versionadded:: 3.0
66
+ """
67
+ return value in env.filters
68
+
69
+
70
+ @pass_environment
71
+ def test_test(env: "Environment", value: str) -> bool:
72
+ """Check if a test exists by name. Useful if a test may be
73
+ optionally available.
74
+
75
+ .. code-block:: jinja
76
+
77
+ {% if 'loud' is test %}
78
+ {% if value is loud %}
79
+ {{ value|upper }}
80
+ {% else %}
81
+ {{ value|lower }}
82
+ {% endif %}
83
+ {% else %}
84
+ {{ value }}
85
+ {% endif %}
86
+
87
+ .. versionadded:: 3.0
88
+ """
89
+ return value in env.tests
90
+
91
+
92
+ def test_none(value: t.Any) -> bool:
93
+ """Return true if the variable is none."""
94
+ return value is None
95
+
96
+
97
+ def test_boolean(value: t.Any) -> bool:
98
+ """Return true if the object is a boolean value.
99
+
100
+ .. versionadded:: 2.11
101
+ """
102
+ return value is True or value is False
103
+
104
+
105
+ def test_false(value: t.Any) -> bool:
106
+ """Return true if the object is False.
107
+
108
+ .. versionadded:: 2.11
109
+ """
110
+ return value is False
111
+
112
+
113
+ def test_true(value: t.Any) -> bool:
114
+ """Return true if the object is True.
115
+
116
+ .. versionadded:: 2.11
117
+ """
118
+ return value is True
119
+
120
+
121
+ # NOTE: The existing 'number' test matches booleans and floats
122
+ def test_integer(value: t.Any) -> bool:
123
+ """Return true if the object is an integer.
124
+
125
+ .. versionadded:: 2.11
126
+ """
127
+ return isinstance(value, int) and value is not True and value is not False
128
+
129
+
130
+ # NOTE: The existing 'number' test matches booleans and integers
131
+ def test_float(value: t.Any) -> bool:
132
+ """Return true if the object is a float.
133
+
134
+ .. versionadded:: 2.11
135
+ """
136
+ return isinstance(value, float)
137
+
138
+
139
+ def test_lower(value: str) -> bool:
140
+ """Return true if the variable is lowercased."""
141
+ return str(value).islower()
142
+
143
+
144
+ def test_upper(value: str) -> bool:
145
+ """Return true if the variable is uppercased."""
146
+ return str(value).isupper()
147
+
148
+
149
+ def test_string(value: t.Any) -> bool:
150
+ """Return true if the object is a string."""
151
+ return isinstance(value, str)
152
+
153
+
154
+ def test_mapping(value: t.Any) -> bool:
155
+ """Return true if the object is a mapping (dict etc.).
156
+
157
+ .. versionadded:: 2.6
158
+ """
159
+ return isinstance(value, abc.Mapping)
160
+
161
+
162
+ def test_number(value: t.Any) -> bool:
163
+ """Return true if the variable is a number."""
164
+ return isinstance(value, Number)
165
+
166
+
167
+ def test_sequence(value: t.Any) -> bool:
168
+ """Return true if the variable is a sequence. Sequences are variables
169
+ that are iterable.
170
+ """
171
+ try:
172
+ len(value)
173
+ value.__getitem__ # noqa B018
174
+ except Exception:
175
+ return False
176
+
177
+ return True
178
+
179
+
180
+ def test_sameas(value: t.Any, other: t.Any) -> bool:
181
+ """Check if an object points to the same memory address than another
182
+ object:
183
+
184
+ .. sourcecode:: jinja
185
+
186
+ {% if foo.attribute is sameas false %}
187
+ the foo attribute really is the `False` singleton
188
+ {% endif %}
189
+ """
190
+ return value is other
191
+
192
+
193
+ def test_iterable(value: t.Any) -> bool:
194
+ """Check if it's possible to iterate over an object."""
195
+ try:
196
+ iter(value)
197
+ except TypeError:
198
+ return False
199
+
200
+ return True
201
+
202
+
203
+ def test_escaped(value: t.Any) -> bool:
204
+ """Check if the value is escaped."""
205
+ return hasattr(value, "__html__")
206
+
207
+
208
+ def test_in(value: t.Any, seq: t.Container[t.Any]) -> bool:
209
+ """Check if value is in seq.
210
+
211
+ .. versionadded:: 2.10
212
+ """
213
+ return value in seq
214
+
215
+
216
+ TESTS = {
217
+ "odd": test_odd,
218
+ "even": test_even,
219
+ "divisibleby": test_divisibleby,
220
+ "defined": test_defined,
221
+ "undefined": test_undefined,
222
+ "filter": test_filter,
223
+ "test": test_test,
224
+ "none": test_none,
225
+ "boolean": test_boolean,
226
+ "false": test_false,
227
+ "true": test_true,
228
+ "integer": test_integer,
229
+ "float": test_float,
230
+ "lower": test_lower,
231
+ "upper": test_upper,
232
+ "string": test_string,
233
+ "mapping": test_mapping,
234
+ "number": test_number,
235
+ "sequence": test_sequence,
236
+ "iterable": test_iterable,
237
+ "callable": callable,
238
+ "sameas": test_sameas,
239
+ "escaped": test_escaped,
240
+ "in": test_in,
241
+ "==": operator.eq,
242
+ "eq": operator.eq,
243
+ "equalto": operator.eq,
244
+ "!=": operator.ne,
245
+ "ne": operator.ne,
246
+ ">": operator.gt,
247
+ "gt": operator.gt,
248
+ "greaterthan": operator.gt,
249
+ "ge": operator.ge,
250
+ ">=": operator.ge,
251
+ "<": operator.lt,
252
+ "lt": operator.lt,
253
+ "lessthan": operator.lt,
254
+ "<=": operator.le,
255
+ "le": operator.le,
256
+ }
TTS/Kokoro/.venv/Lib/site-packages/jinja2/utils.py ADDED
@@ -0,0 +1,766 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import enum
2
+ import json
3
+ import os
4
+ import re
5
+ import typing as t
6
+ from collections import abc
7
+ from collections import deque
8
+ from random import choice
9
+ from random import randrange
10
+ from threading import Lock
11
+ from types import CodeType
12
+ from urllib.parse import quote_from_bytes
13
+
14
+ import markupsafe
15
+
16
+ if t.TYPE_CHECKING:
17
+ import typing_extensions as te
18
+
19
+ F = t.TypeVar("F", bound=t.Callable[..., t.Any])
20
+
21
+
22
+ class _MissingType:
23
+ def __repr__(self) -> str:
24
+ return "missing"
25
+
26
+ def __reduce__(self) -> str:
27
+ return "missing"
28
+
29
+
30
+ missing: t.Any = _MissingType()
31
+ """Special singleton representing missing values for the runtime."""
32
+
33
+ internal_code: t.MutableSet[CodeType] = set()
34
+
35
+ concat = "".join
36
+
37
+
38
+ def pass_context(f: F) -> F:
39
+ """Pass the :class:`~jinja2.runtime.Context` as the first argument
40
+ to the decorated function when called while rendering a template.
41
+
42
+ Can be used on functions, filters, and tests.
43
+
44
+ If only ``Context.eval_context`` is needed, use
45
+ :func:`pass_eval_context`. If only ``Context.environment`` is
46
+ needed, use :func:`pass_environment`.
47
+
48
+ .. versionadded:: 3.0.0
49
+ Replaces ``contextfunction`` and ``contextfilter``.
50
+ """
51
+ f.jinja_pass_arg = _PassArg.context # type: ignore
52
+ return f
53
+
54
+
55
+ def pass_eval_context(f: F) -> F:
56
+ """Pass the :class:`~jinja2.nodes.EvalContext` as the first argument
57
+ to the decorated function when called while rendering a template.
58
+ See :ref:`eval-context`.
59
+
60
+ Can be used on functions, filters, and tests.
61
+
62
+ If only ``EvalContext.environment`` is needed, use
63
+ :func:`pass_environment`.
64
+
65
+ .. versionadded:: 3.0.0
66
+ Replaces ``evalcontextfunction`` and ``evalcontextfilter``.
67
+ """
68
+ f.jinja_pass_arg = _PassArg.eval_context # type: ignore
69
+ return f
70
+
71
+
72
+ def pass_environment(f: F) -> F:
73
+ """Pass the :class:`~jinja2.Environment` as the first argument to
74
+ the decorated function when called while rendering a template.
75
+
76
+ Can be used on functions, filters, and tests.
77
+
78
+ .. versionadded:: 3.0.0
79
+ Replaces ``environmentfunction`` and ``environmentfilter``.
80
+ """
81
+ f.jinja_pass_arg = _PassArg.environment # type: ignore
82
+ return f
83
+
84
+
85
+ class _PassArg(enum.Enum):
86
+ context = enum.auto()
87
+ eval_context = enum.auto()
88
+ environment = enum.auto()
89
+
90
+ @classmethod
91
+ def from_obj(cls, obj: F) -> t.Optional["_PassArg"]:
92
+ if hasattr(obj, "jinja_pass_arg"):
93
+ return obj.jinja_pass_arg # type: ignore
94
+
95
+ return None
96
+
97
+
98
+ def internalcode(f: F) -> F:
99
+ """Marks the function as internally used"""
100
+ internal_code.add(f.__code__)
101
+ return f
102
+
103
+
104
+ def is_undefined(obj: t.Any) -> bool:
105
+ """Check if the object passed is undefined. This does nothing more than
106
+ performing an instance check against :class:`Undefined` but looks nicer.
107
+ This can be used for custom filters or tests that want to react to
108
+ undefined variables. For example a custom default filter can look like
109
+ this::
110
+
111
+ def default(var, default=''):
112
+ if is_undefined(var):
113
+ return default
114
+ return var
115
+ """
116
+ from .runtime import Undefined
117
+
118
+ return isinstance(obj, Undefined)
119
+
120
+
121
+ def consume(iterable: t.Iterable[t.Any]) -> None:
122
+ """Consumes an iterable without doing anything with it."""
123
+ for _ in iterable:
124
+ pass
125
+
126
+
127
+ def clear_caches() -> None:
128
+ """Jinja keeps internal caches for environments and lexers. These are
129
+ used so that Jinja doesn't have to recreate environments and lexers all
130
+ the time. Normally you don't have to care about that but if you are
131
+ measuring memory consumption you may want to clean the caches.
132
+ """
133
+ from .environment import get_spontaneous_environment
134
+ from .lexer import _lexer_cache
135
+
136
+ get_spontaneous_environment.cache_clear()
137
+ _lexer_cache.clear()
138
+
139
+
140
+ def import_string(import_name: str, silent: bool = False) -> t.Any:
141
+ """Imports an object based on a string. This is useful if you want to
142
+ use import paths as endpoints or something similar. An import path can
143
+ be specified either in dotted notation (``xml.sax.saxutils.escape``)
144
+ or with a colon as object delimiter (``xml.sax.saxutils:escape``).
145
+
146
+ If the `silent` is True the return value will be `None` if the import
147
+ fails.
148
+
149
+ :return: imported object
150
+ """
151
+ try:
152
+ if ":" in import_name:
153
+ module, obj = import_name.split(":", 1)
154
+ elif "." in import_name:
155
+ module, _, obj = import_name.rpartition(".")
156
+ else:
157
+ return __import__(import_name)
158
+ return getattr(__import__(module, None, None, [obj]), obj)
159
+ except (ImportError, AttributeError):
160
+ if not silent:
161
+ raise
162
+
163
+
164
+ def open_if_exists(filename: str, mode: str = "rb") -> t.Optional[t.IO[t.Any]]:
165
+ """Returns a file descriptor for the filename if that file exists,
166
+ otherwise ``None``.
167
+ """
168
+ if not os.path.isfile(filename):
169
+ return None
170
+
171
+ return open(filename, mode)
172
+
173
+
174
+ def object_type_repr(obj: t.Any) -> str:
175
+ """Returns the name of the object's type. For some recognized
176
+ singletons the name of the object is returned instead. (For
177
+ example for `None` and `Ellipsis`).
178
+ """
179
+ if obj is None:
180
+ return "None"
181
+ elif obj is Ellipsis:
182
+ return "Ellipsis"
183
+
184
+ cls = type(obj)
185
+
186
+ if cls.__module__ == "builtins":
187
+ return f"{cls.__name__} object"
188
+
189
+ return f"{cls.__module__}.{cls.__name__} object"
190
+
191
+
192
+ def pformat(obj: t.Any) -> str:
193
+ """Format an object using :func:`pprint.pformat`."""
194
+ from pprint import pformat
195
+
196
+ return pformat(obj)
197
+
198
+
199
+ _http_re = re.compile(
200
+ r"""
201
+ ^
202
+ (
203
+ (https?://|www\.) # scheme or www
204
+ (([\w%-]+\.)+)? # subdomain
205
+ (
206
+ [a-z]{2,63} # basic tld
207
+ |
208
+ xn--[\w%]{2,59} # idna tld
209
+ )
210
+ |
211
+ ([\w%-]{2,63}\.)+ # basic domain
212
+ (com|net|int|edu|gov|org|info|mil) # basic tld
213
+ |
214
+ (https?://) # scheme
215
+ (
216
+ (([\d]{1,3})(\.[\d]{1,3}){3}) # IPv4
217
+ |
218
+ (\[([\da-f]{0,4}:){2}([\da-f]{0,4}:?){1,6}]) # IPv6
219
+ )
220
+ )
221
+ (?::[\d]{1,5})? # port
222
+ (?:[/?#]\S*)? # path, query, and fragment
223
+ $
224
+ """,
225
+ re.IGNORECASE | re.VERBOSE,
226
+ )
227
+ _email_re = re.compile(r"^\S+@\w[\w.-]*\.\w+$")
228
+
229
+
230
+ def urlize(
231
+ text: str,
232
+ trim_url_limit: t.Optional[int] = None,
233
+ rel: t.Optional[str] = None,
234
+ target: t.Optional[str] = None,
235
+ extra_schemes: t.Optional[t.Iterable[str]] = None,
236
+ ) -> str:
237
+ """Convert URLs in text into clickable links.
238
+
239
+ This may not recognize links in some situations. Usually, a more
240
+ comprehensive formatter, such as a Markdown library, is a better
241
+ choice.
242
+
243
+ Works on ``http://``, ``https://``, ``www.``, ``mailto:``, and email
244
+ addresses. Links with trailing punctuation (periods, commas, closing
245
+ parentheses) and leading punctuation (opening parentheses) are
246
+ recognized excluding the punctuation. Email addresses that include
247
+ header fields are not recognized (for example,
248
+ ``mailto:address@example.com?cc=copy@example.com``).
249
+
250
+ :param text: Original text containing URLs to link.
251
+ :param trim_url_limit: Shorten displayed URL values to this length.
252
+ :param target: Add the ``target`` attribute to links.
253
+ :param rel: Add the ``rel`` attribute to links.
254
+ :param extra_schemes: Recognize URLs that start with these schemes
255
+ in addition to the default behavior.
256
+
257
+ .. versionchanged:: 3.0
258
+ The ``extra_schemes`` parameter was added.
259
+
260
+ .. versionchanged:: 3.0
261
+ Generate ``https://`` links for URLs without a scheme.
262
+
263
+ .. versionchanged:: 3.0
264
+ The parsing rules were updated. Recognize email addresses with
265
+ or without the ``mailto:`` scheme. Validate IP addresses. Ignore
266
+ parentheses and brackets in more cases.
267
+ """
268
+ if trim_url_limit is not None:
269
+
270
+ def trim_url(x: str) -> str:
271
+ if len(x) > trim_url_limit:
272
+ return f"{x[:trim_url_limit]}..."
273
+
274
+ return x
275
+
276
+ else:
277
+
278
+ def trim_url(x: str) -> str:
279
+ return x
280
+
281
+ words = re.split(r"(\s+)", str(markupsafe.escape(text)))
282
+ rel_attr = f' rel="{markupsafe.escape(rel)}"' if rel else ""
283
+ target_attr = f' target="{markupsafe.escape(target)}"' if target else ""
284
+
285
+ for i, word in enumerate(words):
286
+ head, middle, tail = "", word, ""
287
+ match = re.match(r"^([(<]|&lt;)+", middle)
288
+
289
+ if match:
290
+ head = match.group()
291
+ middle = middle[match.end() :]
292
+
293
+ # Unlike lead, which is anchored to the start of the string,
294
+ # need to check that the string ends with any of the characters
295
+ # before trying to match all of them, to avoid backtracking.
296
+ if middle.endswith((")", ">", ".", ",", "\n", "&gt;")):
297
+ match = re.search(r"([)>.,\n]|&gt;)+$", middle)
298
+
299
+ if match:
300
+ tail = match.group()
301
+ middle = middle[: match.start()]
302
+
303
+ # Prefer balancing parentheses in URLs instead of ignoring a
304
+ # trailing character.
305
+ for start_char, end_char in ("(", ")"), ("<", ">"), ("&lt;", "&gt;"):
306
+ start_count = middle.count(start_char)
307
+
308
+ if start_count <= middle.count(end_char):
309
+ # Balanced, or lighter on the left
310
+ continue
311
+
312
+ # Move as many as possible from the tail to balance
313
+ for _ in range(min(start_count, tail.count(end_char))):
314
+ end_index = tail.index(end_char) + len(end_char)
315
+ # Move anything in the tail before the end char too
316
+ middle += tail[:end_index]
317
+ tail = tail[end_index:]
318
+
319
+ if _http_re.match(middle):
320
+ if middle.startswith("https://") or middle.startswith("http://"):
321
+ middle = (
322
+ f'<a href="{middle}"{rel_attr}{target_attr}>{trim_url(middle)}</a>'
323
+ )
324
+ else:
325
+ middle = (
326
+ f'<a href="https://{middle}"{rel_attr}{target_attr}>'
327
+ f"{trim_url(middle)}</a>"
328
+ )
329
+
330
+ elif middle.startswith("mailto:") and _email_re.match(middle[7:]):
331
+ middle = f'<a href="{middle}">{middle[7:]}</a>'
332
+
333
+ elif (
334
+ "@" in middle
335
+ and not middle.startswith("www.")
336
+ # ignore values like `@a@b`
337
+ and not middle.startswith("@")
338
+ and ":" not in middle
339
+ and _email_re.match(middle)
340
+ ):
341
+ middle = f'<a href="mailto:{middle}">{middle}</a>'
342
+
343
+ elif extra_schemes is not None:
344
+ for scheme in extra_schemes:
345
+ if middle != scheme and middle.startswith(scheme):
346
+ middle = f'<a href="{middle}"{rel_attr}{target_attr}>{middle}</a>'
347
+
348
+ words[i] = f"{head}{middle}{tail}"
349
+
350
+ return "".join(words)
351
+
352
+
353
+ def generate_lorem_ipsum(
354
+ n: int = 5, html: bool = True, min: int = 20, max: int = 100
355
+ ) -> str:
356
+ """Generate some lorem ipsum for the template."""
357
+ from .constants import LOREM_IPSUM_WORDS
358
+
359
+ words = LOREM_IPSUM_WORDS.split()
360
+ result = []
361
+
362
+ for _ in range(n):
363
+ next_capitalized = True
364
+ last_comma = last_fullstop = 0
365
+ word = None
366
+ last = None
367
+ p = []
368
+
369
+ # each paragraph contains out of 20 to 100 words.
370
+ for idx, _ in enumerate(range(randrange(min, max))):
371
+ while True:
372
+ word = choice(words)
373
+ if word != last:
374
+ last = word
375
+ break
376
+ if next_capitalized:
377
+ word = word.capitalize()
378
+ next_capitalized = False
379
+ # add commas
380
+ if idx - randrange(3, 8) > last_comma:
381
+ last_comma = idx
382
+ last_fullstop += 2
383
+ word += ","
384
+ # add end of sentences
385
+ if idx - randrange(10, 20) > last_fullstop:
386
+ last_comma = last_fullstop = idx
387
+ word += "."
388
+ next_capitalized = True
389
+ p.append(word)
390
+
391
+ # ensure that the paragraph ends with a dot.
392
+ p_str = " ".join(p)
393
+
394
+ if p_str.endswith(","):
395
+ p_str = p_str[:-1] + "."
396
+ elif not p_str.endswith("."):
397
+ p_str += "."
398
+
399
+ result.append(p_str)
400
+
401
+ if not html:
402
+ return "\n\n".join(result)
403
+ return markupsafe.Markup(
404
+ "\n".join(f"<p>{markupsafe.escape(x)}</p>" for x in result)
405
+ )
406
+
407
+
408
+ def url_quote(obj: t.Any, charset: str = "utf-8", for_qs: bool = False) -> str:
409
+ """Quote a string for use in a URL using the given charset.
410
+
411
+ :param obj: String or bytes to quote. Other types are converted to
412
+ string then encoded to bytes using the given charset.
413
+ :param charset: Encode text to bytes using this charset.
414
+ :param for_qs: Quote "/" and use "+" for spaces.
415
+ """
416
+ if not isinstance(obj, bytes):
417
+ if not isinstance(obj, str):
418
+ obj = str(obj)
419
+
420
+ obj = obj.encode(charset)
421
+
422
+ safe = b"" if for_qs else b"/"
423
+ rv = quote_from_bytes(obj, safe)
424
+
425
+ if for_qs:
426
+ rv = rv.replace("%20", "+")
427
+
428
+ return rv
429
+
430
+
431
+ @abc.MutableMapping.register
432
+ class LRUCache:
433
+ """A simple LRU Cache implementation."""
434
+
435
+ # this is fast for small capacities (something below 1000) but doesn't
436
+ # scale. But as long as it's only used as storage for templates this
437
+ # won't do any harm.
438
+
439
+ def __init__(self, capacity: int) -> None:
440
+ self.capacity = capacity
441
+ self._mapping: t.Dict[t.Any, t.Any] = {}
442
+ self._queue: te.Deque[t.Any] = deque()
443
+ self._postinit()
444
+
445
+ def _postinit(self) -> None:
446
+ # alias all queue methods for faster lookup
447
+ self._popleft = self._queue.popleft
448
+ self._pop = self._queue.pop
449
+ self._remove = self._queue.remove
450
+ self._wlock = Lock()
451
+ self._append = self._queue.append
452
+
453
+ def __getstate__(self) -> t.Mapping[str, t.Any]:
454
+ return {
455
+ "capacity": self.capacity,
456
+ "_mapping": self._mapping,
457
+ "_queue": self._queue,
458
+ }
459
+
460
+ def __setstate__(self, d: t.Mapping[str, t.Any]) -> None:
461
+ self.__dict__.update(d)
462
+ self._postinit()
463
+
464
+ def __getnewargs__(self) -> t.Tuple[t.Any, ...]:
465
+ return (self.capacity,)
466
+
467
+ def copy(self) -> "te.Self":
468
+ """Return a shallow copy of the instance."""
469
+ rv = self.__class__(self.capacity)
470
+ rv._mapping.update(self._mapping)
471
+ rv._queue.extend(self._queue)
472
+ return rv
473
+
474
+ def get(self, key: t.Any, default: t.Any = None) -> t.Any:
475
+ """Return an item from the cache dict or `default`"""
476
+ try:
477
+ return self[key]
478
+ except KeyError:
479
+ return default
480
+
481
+ def setdefault(self, key: t.Any, default: t.Any = None) -> t.Any:
482
+ """Set `default` if the key is not in the cache otherwise
483
+ leave unchanged. Return the value of this key.
484
+ """
485
+ try:
486
+ return self[key]
487
+ except KeyError:
488
+ self[key] = default
489
+ return default
490
+
491
+ def clear(self) -> None:
492
+ """Clear the cache."""
493
+ with self._wlock:
494
+ self._mapping.clear()
495
+ self._queue.clear()
496
+
497
+ def __contains__(self, key: t.Any) -> bool:
498
+ """Check if a key exists in this cache."""
499
+ return key in self._mapping
500
+
501
+ def __len__(self) -> int:
502
+ """Return the current size of the cache."""
503
+ return len(self._mapping)
504
+
505
+ def __repr__(self) -> str:
506
+ return f"<{type(self).__name__} {self._mapping!r}>"
507
+
508
+ def __getitem__(self, key: t.Any) -> t.Any:
509
+ """Get an item from the cache. Moves the item up so that it has the
510
+ highest priority then.
511
+
512
+ Raise a `KeyError` if it does not exist.
513
+ """
514
+ with self._wlock:
515
+ rv = self._mapping[key]
516
+
517
+ if self._queue[-1] != key:
518
+ try:
519
+ self._remove(key)
520
+ except ValueError:
521
+ # if something removed the key from the container
522
+ # when we read, ignore the ValueError that we would
523
+ # get otherwise.
524
+ pass
525
+
526
+ self._append(key)
527
+
528
+ return rv
529
+
530
+ def __setitem__(self, key: t.Any, value: t.Any) -> None:
531
+ """Sets the value for an item. Moves the item up so that it
532
+ has the highest priority then.
533
+ """
534
+ with self._wlock:
535
+ if key in self._mapping:
536
+ self._remove(key)
537
+ elif len(self._mapping) == self.capacity:
538
+ del self._mapping[self._popleft()]
539
+
540
+ self._append(key)
541
+ self._mapping[key] = value
542
+
543
+ def __delitem__(self, key: t.Any) -> None:
544
+ """Remove an item from the cache dict.
545
+ Raise a `KeyError` if it does not exist.
546
+ """
547
+ with self._wlock:
548
+ del self._mapping[key]
549
+
550
+ try:
551
+ self._remove(key)
552
+ except ValueError:
553
+ pass
554
+
555
+ def items(self) -> t.Iterable[t.Tuple[t.Any, t.Any]]:
556
+ """Return a list of items."""
557
+ result = [(key, self._mapping[key]) for key in list(self._queue)]
558
+ result.reverse()
559
+ return result
560
+
561
+ def values(self) -> t.Iterable[t.Any]:
562
+ """Return a list of all values."""
563
+ return [x[1] for x in self.items()]
564
+
565
+ def keys(self) -> t.Iterable[t.Any]:
566
+ """Return a list of all keys ordered by most recent usage."""
567
+ return list(self)
568
+
569
+ def __iter__(self) -> t.Iterator[t.Any]:
570
+ return reversed(tuple(self._queue))
571
+
572
+ def __reversed__(self) -> t.Iterator[t.Any]:
573
+ """Iterate over the keys in the cache dict, oldest items
574
+ coming first.
575
+ """
576
+ return iter(tuple(self._queue))
577
+
578
+ __copy__ = copy
579
+
580
+
581
+ def select_autoescape(
582
+ enabled_extensions: t.Collection[str] = ("html", "htm", "xml"),
583
+ disabled_extensions: t.Collection[str] = (),
584
+ default_for_string: bool = True,
585
+ default: bool = False,
586
+ ) -> t.Callable[[t.Optional[str]], bool]:
587
+ """Intelligently sets the initial value of autoescaping based on the
588
+ filename of the template. This is the recommended way to configure
589
+ autoescaping if you do not want to write a custom function yourself.
590
+
591
+ If you want to enable it for all templates created from strings or
592
+ for all templates with `.html` and `.xml` extensions::
593
+
594
+ from jinja2 import Environment, select_autoescape
595
+ env = Environment(autoescape=select_autoescape(
596
+ enabled_extensions=('html', 'xml'),
597
+ default_for_string=True,
598
+ ))
599
+
600
+ Example configuration to turn it on at all times except if the template
601
+ ends with `.txt`::
602
+
603
+ from jinja2 import Environment, select_autoescape
604
+ env = Environment(autoescape=select_autoescape(
605
+ disabled_extensions=('txt',),
606
+ default_for_string=True,
607
+ default=True,
608
+ ))
609
+
610
+ The `enabled_extensions` is an iterable of all the extensions that
611
+ autoescaping should be enabled for. Likewise `disabled_extensions` is
612
+ a list of all templates it should be disabled for. If a template is
613
+ loaded from a string then the default from `default_for_string` is used.
614
+ If nothing matches then the initial value of autoescaping is set to the
615
+ value of `default`.
616
+
617
+ For security reasons this function operates case insensitive.
618
+
619
+ .. versionadded:: 2.9
620
+ """
621
+ enabled_patterns = tuple(f".{x.lstrip('.').lower()}" for x in enabled_extensions)
622
+ disabled_patterns = tuple(f".{x.lstrip('.').lower()}" for x in disabled_extensions)
623
+
624
+ def autoescape(template_name: t.Optional[str]) -> bool:
625
+ if template_name is None:
626
+ return default_for_string
627
+ template_name = template_name.lower()
628
+ if template_name.endswith(enabled_patterns):
629
+ return True
630
+ if template_name.endswith(disabled_patterns):
631
+ return False
632
+ return default
633
+
634
+ return autoescape
635
+
636
+
637
+ def htmlsafe_json_dumps(
638
+ obj: t.Any, dumps: t.Optional[t.Callable[..., str]] = None, **kwargs: t.Any
639
+ ) -> markupsafe.Markup:
640
+ """Serialize an object to a string of JSON with :func:`json.dumps`,
641
+ then replace HTML-unsafe characters with Unicode escapes and mark
642
+ the result safe with :class:`~markupsafe.Markup`.
643
+
644
+ This is available in templates as the ``|tojson`` filter.
645
+
646
+ The following characters are escaped: ``<``, ``>``, ``&``, ``'``.
647
+
648
+ The returned string is safe to render in HTML documents and
649
+ ``<script>`` tags. The exception is in HTML attributes that are
650
+ double quoted; either use single quotes or the ``|forceescape``
651
+ filter.
652
+
653
+ :param obj: The object to serialize to JSON.
654
+ :param dumps: The ``dumps`` function to use. Defaults to
655
+ ``env.policies["json.dumps_function"]``, which defaults to
656
+ :func:`json.dumps`.
657
+ :param kwargs: Extra arguments to pass to ``dumps``. Merged onto
658
+ ``env.policies["json.dumps_kwargs"]``.
659
+
660
+ .. versionchanged:: 3.0
661
+ The ``dumper`` parameter is renamed to ``dumps``.
662
+
663
+ .. versionadded:: 2.9
664
+ """
665
+ if dumps is None:
666
+ dumps = json.dumps
667
+
668
+ return markupsafe.Markup(
669
+ dumps(obj, **kwargs)
670
+ .replace("<", "\\u003c")
671
+ .replace(">", "\\u003e")
672
+ .replace("&", "\\u0026")
673
+ .replace("'", "\\u0027")
674
+ )
675
+
676
+
677
+ class Cycler:
678
+ """Cycle through values by yield them one at a time, then restarting
679
+ once the end is reached. Available as ``cycler`` in templates.
680
+
681
+ Similar to ``loop.cycle``, but can be used outside loops or across
682
+ multiple loops. For example, render a list of folders and files in a
683
+ list, alternating giving them "odd" and "even" classes.
684
+
685
+ .. code-block:: html+jinja
686
+
687
+ {% set row_class = cycler("odd", "even") %}
688
+ <ul class="browser">
689
+ {% for folder in folders %}
690
+ <li class="folder {{ row_class.next() }}">{{ folder }}
691
+ {% endfor %}
692
+ {% for file in files %}
693
+ <li class="file {{ row_class.next() }}">{{ file }}
694
+ {% endfor %}
695
+ </ul>
696
+
697
+ :param items: Each positional argument will be yielded in the order
698
+ given for each cycle.
699
+
700
+ .. versionadded:: 2.1
701
+ """
702
+
703
+ def __init__(self, *items: t.Any) -> None:
704
+ if not items:
705
+ raise RuntimeError("at least one item has to be provided")
706
+ self.items = items
707
+ self.pos = 0
708
+
709
+ def reset(self) -> None:
710
+ """Resets the current item to the first item."""
711
+ self.pos = 0
712
+
713
+ @property
714
+ def current(self) -> t.Any:
715
+ """Return the current item. Equivalent to the item that will be
716
+ returned next time :meth:`next` is called.
717
+ """
718
+ return self.items[self.pos]
719
+
720
+ def next(self) -> t.Any:
721
+ """Return the current item, then advance :attr:`current` to the
722
+ next item.
723
+ """
724
+ rv = self.current
725
+ self.pos = (self.pos + 1) % len(self.items)
726
+ return rv
727
+
728
+ __next__ = next
729
+
730
+
731
+ class Joiner:
732
+ """A joining helper for templates."""
733
+
734
+ def __init__(self, sep: str = ", ") -> None:
735
+ self.sep = sep
736
+ self.used = False
737
+
738
+ def __call__(self) -> str:
739
+ if not self.used:
740
+ self.used = True
741
+ return ""
742
+ return self.sep
743
+
744
+
745
+ class Namespace:
746
+ """A namespace object that can hold arbitrary attributes. It may be
747
+ initialized from a dictionary or with keyword arguments."""
748
+
749
+ def __init__(*args: t.Any, **kwargs: t.Any) -> None: # noqa: B902
750
+ self, args = args[0], args[1:]
751
+ self.__attrs = dict(*args, **kwargs)
752
+
753
+ def __getattribute__(self, name: str) -> t.Any:
754
+ # __class__ is needed for the awaitable check in async mode
755
+ if name in {"_Namespace__attrs", "__class__"}:
756
+ return object.__getattribute__(self, name)
757
+ try:
758
+ return self.__attrs[name]
759
+ except KeyError:
760
+ raise AttributeError(name) from None
761
+
762
+ def __setitem__(self, name: str, value: t.Any) -> None:
763
+ self.__attrs[name] = value
764
+
765
+ def __repr__(self) -> str:
766
+ return f"<Namespace {self.__attrs!r}>"
TTS/Kokoro/.venv/Lib/site-packages/jinja2/visitor.py ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """API for traversing the AST nodes. Implemented by the compiler and
2
+ meta introspection.
3
+ """
4
+
5
+ import typing as t
6
+
7
+ from .nodes import Node
8
+
9
+ if t.TYPE_CHECKING:
10
+ import typing_extensions as te
11
+
12
+ class VisitCallable(te.Protocol):
13
+ def __call__(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.Any: ...
14
+
15
+
16
+ class NodeVisitor:
17
+ """Walks the abstract syntax tree and call visitor functions for every
18
+ node found. The visitor functions may return values which will be
19
+ forwarded by the `visit` method.
20
+
21
+ Per default the visitor functions for the nodes are ``'visit_'`` +
22
+ class name of the node. So a `TryFinally` node visit function would
23
+ be `visit_TryFinally`. This behavior can be changed by overriding
24
+ the `get_visitor` function. If no visitor function exists for a node
25
+ (return value `None`) the `generic_visit` visitor is used instead.
26
+ """
27
+
28
+ def get_visitor(self, node: Node) -> "t.Optional[VisitCallable]":
29
+ """Return the visitor function for this node or `None` if no visitor
30
+ exists for this node. In that case the generic visit function is
31
+ used instead.
32
+ """
33
+ return getattr(self, f"visit_{type(node).__name__}", None)
34
+
35
+ def visit(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.Any:
36
+ """Visit a node."""
37
+ f = self.get_visitor(node)
38
+
39
+ if f is not None:
40
+ return f(node, *args, **kwargs)
41
+
42
+ return self.generic_visit(node, *args, **kwargs)
43
+
44
+ def generic_visit(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.Any:
45
+ """Called if no explicit visitor function exists for a node."""
46
+ for child_node in node.iter_child_nodes():
47
+ self.visit(child_node, *args, **kwargs)
48
+
49
+
50
+ class NodeTransformer(NodeVisitor):
51
+ """Walks the abstract syntax tree and allows modifications of nodes.
52
+
53
+ The `NodeTransformer` will walk the AST and use the return value of the
54
+ visitor functions to replace or remove the old node. If the return
55
+ value of the visitor function is `None` the node will be removed
56
+ from the previous location otherwise it's replaced with the return
57
+ value. The return value may be the original node in which case no
58
+ replacement takes place.
59
+ """
60
+
61
+ def generic_visit(self, node: Node, *args: t.Any, **kwargs: t.Any) -> Node:
62
+ for field, old_value in node.iter_fields():
63
+ if isinstance(old_value, list):
64
+ new_values = []
65
+ for value in old_value:
66
+ if isinstance(value, Node):
67
+ value = self.visit(value, *args, **kwargs)
68
+ if value is None:
69
+ continue
70
+ elif not isinstance(value, Node):
71
+ new_values.extend(value)
72
+ continue
73
+ new_values.append(value)
74
+ old_value[:] = new_values
75
+ elif isinstance(old_value, Node):
76
+ new_node = self.visit(old_value, *args, **kwargs)
77
+ if new_node is None:
78
+ delattr(node, field)
79
+ else:
80
+ setattr(node, field, new_node)
81
+ return node
82
+
83
+ def visit_list(self, node: Node, *args: t.Any, **kwargs: t.Any) -> t.List[Node]:
84
+ """As transformers may return lists in some places this method
85
+ can be used to enforce a list as return value.
86
+ """
87
+ rv = self.visit(node, *args, **kwargs)
88
+
89
+ if not isinstance(rv, list):
90
+ return [rv]
91
+
92
+ return rv
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/INSTALLER ADDED
@@ -0,0 +1 @@
 
 
1
+ pip
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/METADATA ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Metadata-Version: 2.4
2
+ Name: joblib
3
+ Version: 1.5.3
4
+ Summary: Lightweight pipelining with Python functions
5
+ Author-email: Gael Varoquaux <gael.varoquaux@normalesup.org>
6
+ License-Expression: BSD-3-Clause
7
+ Project-URL: Homepage, https://joblib.readthedocs.io
8
+ Project-URL: Source, https://github.com/joblib/joblib
9
+ Platform: any
10
+ Classifier: Development Status :: 5 - Production/Stable
11
+ Classifier: Environment :: Console
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Intended Audience :: Science/Research
14
+ Classifier: Intended Audience :: Education
15
+ Classifier: Operating System :: OS Independent
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.9
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Topic :: Scientific/Engineering
23
+ Classifier: Topic :: Utilities
24
+ Classifier: Topic :: Software Development :: Libraries
25
+ Requires-Python: >=3.9
26
+ Description-Content-Type: text/x-rst
27
+ License-File: LICENSE.txt
28
+ Dynamic: license-file
29
+
30
+ |PyPi| |CIStatus| |ReadTheDocs| |Codecov|
31
+
32
+ .. |PyPi| image:: https://badge.fury.io/py/joblib.svg
33
+ :target: https://badge.fury.io/py/joblib
34
+ :alt: Joblib version
35
+
36
+ .. |CIStatus| image:: https://github.com/joblib/joblib/actions/workflows/test.yml/badge.svg
37
+ :target: https://github.com/joblib/joblib/actions/workflows/test.yml?query=branch%3Amain
38
+ :alt: CI status
39
+
40
+ .. |ReadTheDocs| image:: https://readthedocs.org/projects/joblib/badge/?version=latest
41
+ :target: https://joblib.readthedocs.io/en/latest/?badge=latest
42
+ :alt: Documentation Status
43
+
44
+ .. |Codecov| image:: https://codecov.io/gh/joblib/joblib/branch/main/graph/badge.svg
45
+ :target: https://codecov.io/gh/joblib/joblib
46
+ :alt: Codecov coverage
47
+
48
+
49
+ The homepage of joblib with user documentation is located on:
50
+
51
+ https://joblib.readthedocs.io
52
+
53
+ Getting the latest code
54
+ =======================
55
+
56
+ To get the latest code using git, simply type::
57
+
58
+ git clone https://github.com/joblib/joblib.git
59
+
60
+ If you don't have git installed, you can download a zip
61
+ of the latest code: https://github.com/joblib/joblib/archive/refs/heads/main.zip
62
+
63
+ Installing
64
+ ==========
65
+
66
+ You can use `pip` to install joblib from any directory::
67
+
68
+ pip install joblib
69
+
70
+ or install it in editable mode from the source directory::
71
+
72
+ pip install -e .
73
+
74
+ Dependencies
75
+ ============
76
+
77
+ - Joblib has no mandatory dependencies besides Python (supported versions are
78
+ 3.9+).
79
+ - Joblib has an optional dependency on Numpy (at least version 1.6.1) for array
80
+ manipulation.
81
+ - Joblib includes its own vendored copy of
82
+ `loky <https://github.com/tomMoral/loky>`_ for process management.
83
+ - Joblib can efficiently dump and load numpy arrays but does not require numpy
84
+ to be installed.
85
+ - Joblib has an optional dependency on
86
+ `python-lz4 <https://pypi.python.org/pypi/lz4>`_ as a faster alternative to
87
+ zlib and gzip for compressed serialization.
88
+ - Joblib has an optional dependency on psutil to mitigate memory leaks in
89
+ parallel worker processes.
90
+ - Some examples require external dependencies such as pandas. See the
91
+ instructions in the `Building the docs`_ section for details.
92
+
93
+ Workflow to contribute
94
+ ======================
95
+
96
+ To contribute to joblib, first create an account on `github
97
+ <https://github.com/>`_. Once this is done, fork the `joblib repository
98
+ <https://github.com/joblib/joblib>`_ to have your own repository,
99
+ clone it using ``git clone``. Make your changes in a branch of your clone, push
100
+ them to your github account, test them locally, and when you are happy with
101
+ them, send a pull request to the main repository.
102
+
103
+ You can use `pre-commit <https://pre-commit.com/#install>`_ to run code style checks
104
+ before each commit::
105
+
106
+ pip install pre-commit
107
+ pre-commit install
108
+
109
+ pre-commit checks can be disabled for a single commit with::
110
+
111
+ git commit -n
112
+
113
+ Running the test suite
114
+ ======================
115
+
116
+ To run the test suite, you need the pytest (version >= 3) and coverage modules.
117
+ Run the test suite using::
118
+
119
+ pytest joblib
120
+
121
+ from the root of the project.
122
+
123
+ Building the docs
124
+ =================
125
+
126
+ To build the docs you need to have sphinx (>=1.4) and some dependencies
127
+ installed::
128
+
129
+ pip install -U -r .readthedocs-requirements.txt
130
+
131
+ The docs can then be built with the following command::
132
+
133
+ make doc
134
+
135
+ The html docs are located in the ``doc/_build/html`` directory.
136
+
137
+
138
+ Making a source tarball
139
+ =======================
140
+
141
+ To create a source tarball, eg for packaging or distributing, run the
142
+ following command::
143
+
144
+ pip install build
145
+ python -m build --sdist
146
+
147
+ The tarball will be created in the `dist` directory. This command will create
148
+ the resulting tarball that can be installed with no extra dependencies than the
149
+ Python standard library.
150
+
151
+ Making a release and uploading it to PyPI
152
+ =========================================
153
+
154
+ This command is only run by project manager, to make a release, and
155
+ upload in to PyPI::
156
+
157
+ pip install build
158
+ python -m build --sdist --wheel
159
+ twine upload dist/*
160
+
161
+
162
+ Note that the documentation should automatically get updated at each git
163
+ push. If that is not the case, try building th doc locally and resolve
164
+ any doc build error (in particular when running the examples).
165
+
166
+ Updating the changelog
167
+ ======================
168
+
169
+ Changes are listed in the CHANGES.rst file. They must be manually updated
170
+ but, the following git command may be used to generate the lines::
171
+
172
+ git log --abbrev-commit --date=short --no-merges --sparse
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/RECORD ADDED
@@ -0,0 +1,217 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ joblib-1.5.3.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
2
+ joblib-1.5.3.dist-info/METADATA,sha256=zunGCJauTnqWWoQPveUNER0vgBj2EgHR7S6fw91Ig-Y,5542
3
+ joblib-1.5.3.dist-info/RECORD,,
4
+ joblib-1.5.3.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
5
+ joblib-1.5.3.dist-info/licenses/LICENSE.txt,sha256=QmEpEcGHLF5LQ_auDo7llGfNNQMyJBz3LOkGQCZPrmo,1527
6
+ joblib-1.5.3.dist-info/top_level.txt,sha256=P0LsoZ45gBL7ckL4lqQt7tdbrHD4xlVYhffmhHeeT_U,7
7
+ joblib/__init__.py,sha256=W-0Q8UNWm7rSJmZA9eq-ihckKM6JRKvOm1l8SN44piQ,5337
8
+ joblib/__pycache__/__init__.cpython-312.pyc,,
9
+ joblib/__pycache__/_cloudpickle_wrapper.cpython-312.pyc,,
10
+ joblib/__pycache__/_dask.cpython-312.pyc,,
11
+ joblib/__pycache__/_memmapping_reducer.cpython-312.pyc,,
12
+ joblib/__pycache__/_multiprocessing_helpers.cpython-312.pyc,,
13
+ joblib/__pycache__/_parallel_backends.cpython-312.pyc,,
14
+ joblib/__pycache__/_store_backends.cpython-312.pyc,,
15
+ joblib/__pycache__/_utils.cpython-312.pyc,,
16
+ joblib/__pycache__/backports.cpython-312.pyc,,
17
+ joblib/__pycache__/compressor.cpython-312.pyc,,
18
+ joblib/__pycache__/disk.cpython-312.pyc,,
19
+ joblib/__pycache__/executor.cpython-312.pyc,,
20
+ joblib/__pycache__/func_inspect.cpython-312.pyc,,
21
+ joblib/__pycache__/hashing.cpython-312.pyc,,
22
+ joblib/__pycache__/logger.cpython-312.pyc,,
23
+ joblib/__pycache__/memory.cpython-312.pyc,,
24
+ joblib/__pycache__/numpy_pickle.cpython-312.pyc,,
25
+ joblib/__pycache__/numpy_pickle_compat.cpython-312.pyc,,
26
+ joblib/__pycache__/numpy_pickle_utils.cpython-312.pyc,,
27
+ joblib/__pycache__/parallel.cpython-312.pyc,,
28
+ joblib/__pycache__/pool.cpython-312.pyc,,
29
+ joblib/__pycache__/testing.cpython-312.pyc,,
30
+ joblib/_cloudpickle_wrapper.py,sha256=HSFxIio3jiGnwVCstAa6obbxs4-5aRAIMDDUAA-cDPc,416
31
+ joblib/_dask.py,sha256=xUYA_2VVc0LvPavSiFy8M7TZc6KF0lIxcQhng5kPaXU,13217
32
+ joblib/_memmapping_reducer.py,sha256=AZ6dqA6fXlm4-ehBCf9m1nq43jUPKman4_2whrOButc,28553
33
+ joblib/_multiprocessing_helpers.py,sha256=f8-Vf_8ildmdg991eLz8xk4DJJFTS_bcrhj6CgQ4lxU,1878
34
+ joblib/_parallel_backends.py,sha256=fgy_FgZiKeNvTWr4wKbSX4kUNx2YD6m7p5O1J96xhb4,28766
35
+ joblib/_store_backends.py,sha256=xJWN4xvM1LU8kZXk3LEJwIjeEoj6zOmESsPHWmlUkPw,17918
36
+ joblib/_utils.py,sha256=bE4abMGL08HTLtVeisrww8QjvCllMLHVT2qbmYtAGaA,3248
37
+ joblib/backports.py,sha256=mITpG-yuEADimg89_LCdUY9QH9a5xQHsRNJnd7BmAMo,5450
38
+ joblib/compressor.py,sha256=GDDVJmeOBqftc6tMkDupryojHhk_jIV8WrNoMiTxTdQ,19281
39
+ joblib/disk.py,sha256=1J5hhMsCP5LDW65luTtArUxsMAJRrPB6wxSWf6GeBns,4332
40
+ joblib/executor.py,sha256=fbVmE_KKywjJcIKmHO9k8M3VkaMqZXEP4YXBRz_p6xU,5229
41
+ joblib/externals/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
42
+ joblib/externals/__pycache__/__init__.cpython-312.pyc,,
43
+ joblib/externals/cloudpickle/__init__.py,sha256=ErKyvqqAErOkbCowrizhGpN9DN-TaD22QtqI9CQ9sDw,308
44
+ joblib/externals/cloudpickle/__pycache__/__init__.cpython-312.pyc,,
45
+ joblib/externals/cloudpickle/__pycache__/cloudpickle.cpython-312.pyc,,
46
+ joblib/externals/cloudpickle/__pycache__/cloudpickle_fast.cpython-312.pyc,,
47
+ joblib/externals/cloudpickle/cloudpickle.py,sha256=KcLA_uAxp2KuMJb9vaTVve7FLFUQu7mYkD3dCscxeRc,58874
48
+ joblib/externals/cloudpickle/cloudpickle_fast.py,sha256=AI5ZKf2AbLNxD8lXyLDpKZyzeZ2ofFtdK1ZWFq_ec1c,323
49
+ joblib/externals/loky/__init__.py,sha256=8LzBTFpYfRFrjD1loIQpRF9QQ_8wwEkssJI6hYcGbfE,1105
50
+ joblib/externals/loky/__pycache__/__init__.cpython-312.pyc,,
51
+ joblib/externals/loky/__pycache__/_base.cpython-312.pyc,,
52
+ joblib/externals/loky/__pycache__/cloudpickle_wrapper.cpython-312.pyc,,
53
+ joblib/externals/loky/__pycache__/initializers.cpython-312.pyc,,
54
+ joblib/externals/loky/__pycache__/process_executor.cpython-312.pyc,,
55
+ joblib/externals/loky/__pycache__/reusable_executor.cpython-312.pyc,,
56
+ joblib/externals/loky/_base.py,sha256=LsQnEoKWKGhdeqGhMc68Aqwz4MrTnEs20KAYbFiUHzo,1057
57
+ joblib/externals/loky/backend/__init__.py,sha256=Ix9KThV1CYk7-M5OQnJ_A_JrrrWJ-Jowa-HMMeGbp18,312
58
+ joblib/externals/loky/backend/__pycache__/__init__.cpython-312.pyc,,
59
+ joblib/externals/loky/backend/__pycache__/_posix_reduction.cpython-312.pyc,,
60
+ joblib/externals/loky/backend/__pycache__/_win_reduction.cpython-312.pyc,,
61
+ joblib/externals/loky/backend/__pycache__/context.cpython-312.pyc,,
62
+ joblib/externals/loky/backend/__pycache__/fork_exec.cpython-312.pyc,,
63
+ joblib/externals/loky/backend/__pycache__/popen_loky_posix.cpython-312.pyc,,
64
+ joblib/externals/loky/backend/__pycache__/popen_loky_win32.cpython-312.pyc,,
65
+ joblib/externals/loky/backend/__pycache__/process.cpython-312.pyc,,
66
+ joblib/externals/loky/backend/__pycache__/queues.cpython-312.pyc,,
67
+ joblib/externals/loky/backend/__pycache__/reduction.cpython-312.pyc,,
68
+ joblib/externals/loky/backend/__pycache__/resource_tracker.cpython-312.pyc,,
69
+ joblib/externals/loky/backend/__pycache__/spawn.cpython-312.pyc,,
70
+ joblib/externals/loky/backend/__pycache__/synchronize.cpython-312.pyc,,
71
+ joblib/externals/loky/backend/__pycache__/utils.cpython-312.pyc,,
72
+ joblib/externals/loky/backend/_posix_reduction.py,sha256=xgCSrIaLI0k_MI0XNOBSp5e1ox1WN9idgrWbkWpMUr4,1776
73
+ joblib/externals/loky/backend/_win_reduction.py,sha256=WmNB0NXtyJ_o_WzfPUEGh5dPhXIeI6FkEnFNXUxO2ws,683
74
+ joblib/externals/loky/backend/context.py,sha256=RPdZvzkEk7iA0rtdAILSHNzl6wsHpm6XD6IL30owAPE,14284
75
+ joblib/externals/loky/backend/fork_exec.py,sha256=4DZ1iLBB-21rlg3Z4Kh9DTVZj35JPaWFE5rzWZaSDxk,2319
76
+ joblib/externals/loky/backend/popen_loky_posix.py,sha256=3G-2_-ovZtjWcHI-xSyW5zQjAZ-_Z9IGjzY1RrZH4nc,5541
77
+ joblib/externals/loky/backend/popen_loky_win32.py,sha256=bYkhRA0w8qUcYFwoezeGwcnlCocEdheWXc6SZ-_rVxo,5325
78
+ joblib/externals/loky/backend/process.py,sha256=4-Y94EoIrg4btsjTNxUBHAHhR96Nrugn_7_PGL6aU50,2018
79
+ joblib/externals/loky/backend/queues.py,sha256=eETFvbPHwKfdoYyOgNQCyKq_Zlm-lzH3fwwpUIh-_4U,7322
80
+ joblib/externals/loky/backend/reduction.py,sha256=861drQAefXTJjfFWAEWmYAS315d8lAyqWx0RgyxFw_0,6926
81
+ joblib/externals/loky/backend/resource_tracker.py,sha256=Jzbmb8otLR7etqhefKuZxAs1VvT1jV8d5Zev8vUcV6s,15403
82
+ joblib/externals/loky/backend/spawn.py,sha256=t4PzEJ3tjwoF9t8qnQUF9R7Q-LmBpDBIcHURWNznz8M,8626
83
+ joblib/externals/loky/backend/synchronize.py,sha256=nlDwBoLZB93m_l55qfZM_Ql-4L84PSYimoQqt5TzpDk,11768
84
+ joblib/externals/loky/backend/utils.py,sha256=RVsxqyET4TJdbjc9uUHJmfhlQ2v4Uq-fiT_5b5rfC0s,5757
85
+ joblib/externals/loky/cloudpickle_wrapper.py,sha256=jUnfhXI3qMXTlCeTUzpABQlv0VOLMJL1V7fpRlq2LgU,3609
86
+ joblib/externals/loky/initializers.py,sha256=dtKtRsJUmVwiJu0yZ-Ih0m8PvW_MxmouG7mShEcsStc,2567
87
+ joblib/externals/loky/process_executor.py,sha256=QPSKet0OCAWr6g_2fHwPt4yjQaAJsjfeJYFPiKhS9RE,52348
88
+ joblib/externals/loky/reusable_executor.py,sha256=d9ksrTnJS8549Oq50iG08u5pEhuMbhQ3oSYUSq0twNQ,10863
89
+ joblib/func_inspect.py,sha256=bhm_GpBe3H_Dmw5ripzP5BalA6wbq7ZFI3SEuAQbfek,14017
90
+ joblib/hashing.py,sha256=38MM0zRl0Ebk78Ij6cMdrQ8ibYZP0pCJxu3L4Yrw1sc,10694
91
+ joblib/logger.py,sha256=HK06qwNWJYInYIIXFYINAKCxjYxi0hoX45ckNKkogHQ,5342
92
+ joblib/memory.py,sha256=va7zWG9s_X6eE-Cm1junrH-QwKTnguin5cEJIhUXo98,45404
93
+ joblib/numpy_pickle.py,sha256=N_wQMf6_vgI71nRYLne0dc2kO6dfh0lkTaOZn8Tq5Hc,28791
94
+ joblib/numpy_pickle_compat.py,sha256=JOlSfMT1uDIztOyQ3dzYgp5fGVnzPVWBCqXjdIZsjLQ,8451
95
+ joblib/numpy_pickle_utils.py,sha256=j3GlI25QFvo-DTPn7uRptu-NtW16ztHM0DuglyQyEDI,9497
96
+ joblib/parallel.py,sha256=SkJYk-cTHC8oMvZU79SDXV61IZ10YIHbBYhrHB47yM8,86989
97
+ joblib/pool.py,sha256=JTc00PEAyPayo8mHdktmburp5OBsnNxwSQI4zzvtKYs,14134
98
+ joblib/test/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
99
+ joblib/test/__pycache__/__init__.cpython-312.pyc,,
100
+ joblib/test/__pycache__/common.cpython-312.pyc,,
101
+ joblib/test/__pycache__/test_backports.cpython-312.pyc,,
102
+ joblib/test/__pycache__/test_cloudpickle_wrapper.cpython-312.pyc,,
103
+ joblib/test/__pycache__/test_config.cpython-312.pyc,,
104
+ joblib/test/__pycache__/test_dask.cpython-312.pyc,,
105
+ joblib/test/__pycache__/test_disk.cpython-312.pyc,,
106
+ joblib/test/__pycache__/test_func_inspect.cpython-312.pyc,,
107
+ joblib/test/__pycache__/test_func_inspect_special_encoding.cpython-312.pyc,,
108
+ joblib/test/__pycache__/test_hashing.cpython-312.pyc,,
109
+ joblib/test/__pycache__/test_init.cpython-312.pyc,,
110
+ joblib/test/__pycache__/test_logger.cpython-312.pyc,,
111
+ joblib/test/__pycache__/test_memmapping.cpython-312.pyc,,
112
+ joblib/test/__pycache__/test_memory.cpython-312.pyc,,
113
+ joblib/test/__pycache__/test_memory_async.cpython-312.pyc,,
114
+ joblib/test/__pycache__/test_missing_multiprocessing.cpython-312.pyc,,
115
+ joblib/test/__pycache__/test_module.cpython-312.pyc,,
116
+ joblib/test/__pycache__/test_numpy_pickle.cpython-312.pyc,,
117
+ joblib/test/__pycache__/test_numpy_pickle_compat.cpython-312.pyc,,
118
+ joblib/test/__pycache__/test_numpy_pickle_utils.cpython-312.pyc,,
119
+ joblib/test/__pycache__/test_parallel.cpython-312.pyc,,
120
+ joblib/test/__pycache__/test_store_backends.cpython-312.pyc,,
121
+ joblib/test/__pycache__/test_testing.cpython-312.pyc,,
122
+ joblib/test/__pycache__/test_utils.cpython-312.pyc,,
123
+ joblib/test/__pycache__/testutils.cpython-312.pyc,,
124
+ joblib/test/common.py,sha256=vpjpcJgMbmr8H3skc3qsr_KC-u-ZnhVFRk2vAxmJqvA,2102
125
+ joblib/test/data/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
126
+ joblib/test/data/__pycache__/__init__.cpython-312.pyc,,
127
+ joblib/test/data/__pycache__/create_numpy_pickle.cpython-312.pyc,,
128
+ joblib/test/data/create_numpy_pickle.py,sha256=vZE7JNye9o0gYaxrn1555av6Igee0KeXacAWKNRhsu8,3334
129
+ joblib/test/data/joblib_0.10.0_compressed_pickle_py27_np16.gz,sha256=QYRH6Q2DSGVorjCSqWCxjTWCMOJKyew4Nl2qmfQVvQ8,769
130
+ joblib/test/data/joblib_0.10.0_compressed_pickle_py27_np17.gz,sha256=ofTozM_KlPJa50TR8FCwc09mMmO6OO0GQhgUBLNIsXs,757
131
+ joblib/test/data/joblib_0.10.0_compressed_pickle_py33_np18.gz,sha256=2eIVeA-XjOaT5IEQ6tI2UuHG3hwhiRciMmkBmPcIh4g,792
132
+ joblib/test/data/joblib_0.10.0_compressed_pickle_py34_np19.gz,sha256=Gr2z_1tVWDH1H3_wCVHmakknf8KqeHKT8Yz4d1vmUCM,794
133
+ joblib/test/data/joblib_0.10.0_compressed_pickle_py35_np19.gz,sha256=pWw_xuDbOkECqu1KGf1OFU7s2VbzC2v5F5iXhE7TwB4,790
134
+ joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl,sha256=icRQjj374B-AHk5znxre0T9oWUHokoHIBQ8MqKo8l-U,986
135
+ joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.bz2,sha256=oYQVIyMiUxyRgWSuBBSOvCWKzToA-kUpcoQWdV4UoV4,997
136
+ joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.gzip,sha256=Jpv3iGcDgKTv-O4nZsUreIbUK7qnt2cugZ-VMgNeEDQ,798
137
+ joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.lzma,sha256=c0wu0x8pPv4BcStj7pE61rZpf68FLG_pNzQZ4e82zH8,660
138
+ joblib/test/data/joblib_0.10.0_pickle_py27_np17.pkl.xz,sha256=77FG1FDG0GHQav-1bxc4Tn9ky6ubUW_MbE0_iGmz5wc,712
139
+ joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl,sha256=4GTC7s_cWNVShERn2nvVbspZYJgyK_0man4TEqvdVzU,1068
140
+ joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.bz2,sha256=6G1vbs_iYmz2kYJ6w4qB1k7D67UnxUMus0S4SWeBtFo,1000
141
+ joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.gzip,sha256=tlRUWeJS1BXmcwtLNSNK9L0hDHekFl07CqWxTShinmY,831
142
+ joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.lzma,sha256=CorPwnfv3rR5hjNtJI01-sEBMOnkSxNlRVaWTszMopA,694
143
+ joblib/test/data/joblib_0.10.0_pickle_py33_np18.pkl.xz,sha256=Dppj3MffOKsKETeptEtDaxPOv6MA6xnbpK5LzlDQ-oE,752
144
+ joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl,sha256=HL5Fb1uR9aPLjjhoOPJ2wwM1Qyo1FCZoYYd2HVw0Fos,1068
145
+ joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.bz2,sha256=Pyr2fqZnwfUxXdyrBr-kRwBYY8HA_Yi7fgSguKy5pUs,1021
146
+ joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.gzip,sha256=os8NJjQI9FhnlZM-Ay9dX_Uo35gZnoJCgQSIVvcBPfE,831
147
+ joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.lzma,sha256=Q_0y43qU7_GqAabJ8y3PWVhOisurnCAq3GzuCu04V58,697
148
+ joblib/test/data/joblib_0.10.0_pickle_py34_np19.pkl.xz,sha256=BNfmiQfpeLVpdfkwlJK4hJ5Cpgl0vreVyekyc5d_PNM,752
149
+ joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl,sha256=l7nvLolhBDIdPFznOz3lBHiMOPBPCMi1bXop1tFSCpY,1068
150
+ joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.bz2,sha256=pqGpuIS-ZU4uP8mkglHs8MaSDiVcPy7l3XHYJSppRgY,1005
151
+ joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.gzip,sha256=YRFXE6LEb6qK72yPqnXdqQVY8Ts8xKUS9PWQKhLxWvk,833
152
+ joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.lzma,sha256=Bf7gCUeTuTjCkbcIdyZYz69irblX4SAVQEzxCnMQhNU,701
153
+ joblib/test/data/joblib_0.10.0_pickle_py35_np19.pkl.xz,sha256=As8w2LGWwwNmKy3QNdKljK63Yq46gjRf_RJ0lh5_WqA,752
154
+ joblib/test/data/joblib_0.11.0_compressed_pickle_py36_np111.gz,sha256=1WrnXDqDoNEPYOZX1Q5Wr2463b8vVV6fw4Wm5S4bMt4,800
155
+ joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl,sha256=XmsOFxeC1f1aYdGETclG6yfF9rLoB11DayOAhDMULrw,1068
156
+ joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.bz2,sha256=vI2yWb50LKL_NgZyd_XkoD5teIg93uI42mWnx9ee-AQ,991
157
+ joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.gzip,sha256=1WrnXDqDoNEPYOZX1Q5Wr2463b8vVV6fw4Wm5S4bMt4,800
158
+ joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.lzma,sha256=IWA0JlZG2ur53HgTUDl1m7q79dcVq6b0VOq33gKoJU0,715
159
+ joblib/test/data/joblib_0.11.0_pickle_py36_np111.pkl.xz,sha256=3Xh_NbMZdBjYx7ynfJ3Fyke28izSRSSzzNB0z5D4k9Y,752
160
+ joblib/test/data/joblib_0.8.4_compressed_pickle_py27_np17.gz,sha256=Sp-ZT7i6pj5on2gbptszu7RarzJpOmHJ67UKOmCPQMg,659
161
+ joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np16.gz,sha256=NLtDrvo2XIH0KvUUAvhOqMeoXEjGW0IuTk_osu5XiDw,658
162
+ joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np17.gz,sha256=NLtDrvo2XIH0KvUUAvhOqMeoXEjGW0IuTk_osu5XiDw,658
163
+ joblib/test/data/joblib_0.9.2_compressed_pickle_py34_np19.gz,sha256=nzO9iiGkG3KbBdrF3usOho8higkrDj_lmICUzxZyF_Y,673
164
+ joblib/test/data/joblib_0.9.2_compressed_pickle_py35_np19.gz,sha256=nzO9iiGkG3KbBdrF3usOho8higkrDj_lmICUzxZyF_Y,673
165
+ joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl,sha256=naijdk2xIeKdIa3mfJw0JlmOdtiN6uRM1yOJg6-M73M,670
166
+ joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
167
+ joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
168
+ joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_03.npy,sha256=oMRa4qKJhBy-uiRDt-uqOzHAqencxzKUrKVynaAJJAU,236
169
+ joblib/test/data/joblib_0.9.2_pickle_py27_np16.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
170
+ joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl,sha256=LynX8dLOygfxDfFywOgm7wgWOhSxLG7z-oDsU6X83Dw,670
171
+ joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
172
+ joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
173
+ joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_03.npy,sha256=oMRa4qKJhBy-uiRDt-uqOzHAqencxzKUrKVynaAJJAU,236
174
+ joblib/test/data/joblib_0.9.2_pickle_py27_np17.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
175
+ joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl,sha256=w9TLxpDTzp5TI6cU6lRvMsAasXEChcQgGE9s30sm_CU,691
176
+ joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
177
+ joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
178
+ joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_03.npy,sha256=jt6aZKUrJdfbMJUJVsl47As5MrfRSs1avGMhbmS6vec,307
179
+ joblib/test/data/joblib_0.9.2_pickle_py33_np18.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
180
+ joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl,sha256=ilOBAOaulLFvKrD32S1NfnpiK-LfzA9rC3O2I7xROuI,691
181
+ joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
182
+ joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
183
+ joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_03.npy,sha256=jt6aZKUrJdfbMJUJVsl47As5MrfRSs1avGMhbmS6vec,307
184
+ joblib/test/data/joblib_0.9.2_pickle_py34_np19.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
185
+ joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl,sha256=WfDVIqKcMzzh1gSAshIfzBoIpdLdZQuG79yYf5kfpOo,691
186
+ joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_01.npy,sha256=DvvX2c5-7DpuCg20HnleA5bMo9awN9rWxhtGSEPSiAk,120
187
+ joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_02.npy,sha256=HBzzbLeB-8whuVO7CgtF3wktoOrg52WILlljzNcBBbE,120
188
+ joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_03.npy,sha256=jt6aZKUrJdfbMJUJVsl47As5MrfRSs1avGMhbmS6vec,307
189
+ joblib/test/data/joblib_0.9.2_pickle_py35_np19.pkl_04.npy,sha256=PsviRClLqT4IR5sWwbmpQR41af9mDtBFncodJBOB3wU,104
190
+ joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz,sha256=8jYfWJsx0oY2J-3LlmEigK5cClnJSW2J2rfeSTZw-Ts,802
191
+ joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_01.npy.z,sha256=YT9VvT3sEl2uWlOyvH2CkyE9Sok4od9O3kWtgeuUUqE,43
192
+ joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_02.npy.z,sha256=txA5RDI0PRuiU_UNKY8pGp-zQgQQ9vaVvMi60hOPaVs,43
193
+ joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz_03.npy.z,sha256=d3AwICvU2MpSNjh2aPIsdJeGZLlDjANAF1Soa6uM0Po,37
194
+ joblib/test/test_backports.py,sha256=ONt0JUPV1etZCO9DTLur1h84XmgHZYK_k73qmp4kRgg,1175
195
+ joblib/test/test_cloudpickle_wrapper.py,sha256=9jx3hqNVO9GXdVHCxr9mN-GiLR0XK-O5d6YPaaG8Y14,729
196
+ joblib/test/test_config.py,sha256=1Z102AO7Gb8Z8mHYahnZy2fxBA-9_vY0ZtWyNNk1cf4,5255
197
+ joblib/test/test_dask.py,sha256=X2MBEYvz5WQwzGZRN04JNgk_75iIHF96yA1F1t1sK_Y,22932
198
+ joblib/test/test_disk.py,sha256=0EaWGENlosrqwrSZvquPQw3jhqay1KD1NRlQ6YLHOOM,2223
199
+ joblib/test/test_func_inspect.py,sha256=RsORR-j48SfXrNBQbb5i-SdmfU7zk2Mr0IKvcu8m1tw,9314
200
+ joblib/test/test_func_inspect_special_encoding.py,sha256=5xILDjSO-xtjQAMLvMeVD-L7IG4ZURb2gvBiShaDE78,145
201
+ joblib/test/test_hashing.py,sha256=wZeTJMX8C8ua3fJsKAI7MKtperUfZf1fLt0ZaOjvSKw,15820
202
+ joblib/test/test_init.py,sha256=Y6y6Hcqa_cqwQ8S8ozUQ180y_RfkRajfZ_fDp2UXgbw,423
203
+ joblib/test/test_logger.py,sha256=FA9ohTNcqIFViQK60_rwZ5PEGL2zoYN5qBOrDwFqVzI,941
204
+ joblib/test/test_memmapping.py,sha256=z0aanbEs3yCDKShyW3IYlLkTARwdvqVTb4beTPRFmjk,43731
205
+ joblib/test/test_memory.py,sha256=yHoTaoYsWEaDo0CQLNHrBwY5cCpYtIFc3Hi5PArB0iM,51011
206
+ joblib/test/test_memory_async.py,sha256=tUoCI9dngR2AuJjAAKXElJIiz2Qm4AJGdXKn9c8lWaM,5245
207
+ joblib/test/test_missing_multiprocessing.py,sha256=FVoS91krFZogIoDFScyZuJPpaeiq6O-aLAxug0qCQyY,1171
208
+ joblib/test/test_module.py,sha256=IABzz5JmdeY_Adk_vZ0776JN94Ra7tWxDA7DPDNdJKI,1942
209
+ joblib/test/test_numpy_pickle.py,sha256=QExCnBSG-EXdVKnoDkJjNFk6kbX0FDeGeR50wtLHiso,42130
210
+ joblib/test/test_numpy_pickle_compat.py,sha256=paMz1G3Fr9SHdjFmKcG1ec6B5h_S-XE6WRtfHmX9r50,609
211
+ joblib/test/test_numpy_pickle_utils.py,sha256=iB2Ve1TYYUEN3DQiNB5qUxk_QxeIXl7Jpgv4TwkFWTY,382
212
+ joblib/test/test_parallel.py,sha256=_13kli8GYyclwh2QsxysXrRJa44o3gb3FEpSY61ag94,78095
213
+ joblib/test/test_store_backends.py,sha256=DyK1f7PTSPErzhk27gaRoMe2UQrstIz6fnvZh4hKIf0,3057
214
+ joblib/test/test_testing.py,sha256=jL-Ph5pzUJSXOgY2rqbjMRp2y3i3CCWmEi-Lbw4Wzr8,2520
215
+ joblib/test/test_utils.py,sha256=qSD3IJnIGyLiJlUSbEQ0ILVqwG18LW6u50BqTJ3Cn3o,1015
216
+ joblib/test/testutils.py,sha256=A1bm-A5Ydis2iZJVI2-r3aFKUufWR42NZ8Yttrp8mzg,252
217
+ joblib/testing.py,sha256=lK8HOBvrpXcTYUCSvpE-M2ede_dTVJzcmyw-9BrBsOc,3029
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/WHEEL ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ Wheel-Version: 1.0
2
+ Generator: setuptools (80.9.0)
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
5
+
TTS/Kokoro/.venv/Lib/site-packages/joblib-1.5.3.dist-info/top_level.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ joblib
TTS/Kokoro/.venv/Lib/site-packages/joblib/__init__.py ADDED
@@ -0,0 +1,163 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Joblib is a set of tools to provide **lightweight pipelining in
2
+ Python**. In particular:
3
+
4
+ 1. transparent disk-caching of functions and lazy re-evaluation
5
+ (memoize pattern)
6
+
7
+ 2. easy simple parallel computing
8
+
9
+ Joblib is optimized to be **fast** and **robust** on large
10
+ data in particular and has specific optimizations for `numpy` arrays. It is
11
+ **BSD-licensed**.
12
+
13
+
14
+ ==================== ===============================================
15
+ **Documentation:** https://joblib.readthedocs.io
16
+
17
+ **Download:** https://pypi.python.org/pypi/joblib#downloads
18
+
19
+ **Source code:** https://github.com/joblib/joblib
20
+
21
+ **Report issues:** https://github.com/joblib/joblib/issues
22
+ ==================== ===============================================
23
+
24
+
25
+ Vision
26
+ --------
27
+
28
+ The vision is to provide tools to easily achieve better performance and
29
+ reproducibility when working with long running jobs.
30
+
31
+ * **Avoid computing the same thing twice**: code is often rerun again and
32
+ again, for instance when prototyping computational-heavy jobs (as in
33
+ scientific development), but hand-crafted solutions to alleviate this
34
+ issue are error-prone and often lead to unreproducible results.
35
+
36
+ * **Persist to disk transparently**: efficiently persisting
37
+ arbitrary objects containing large data is hard. Using
38
+ joblib's caching mechanism avoids hand-written persistence and
39
+ implicitly links the file on disk to the execution context of
40
+ the original Python object. As a result, joblib's persistence is
41
+ good for resuming an application status or computational job, eg
42
+ after a crash.
43
+
44
+ Joblib addresses these problems while **leaving your code and your flow
45
+ control as unmodified as possible** (no framework, no new paradigms).
46
+
47
+ Main features
48
+ ------------------
49
+
50
+ 1) **Transparent and fast disk-caching of output value:** a memoize or
51
+ make-like functionality for Python functions that works well for
52
+ arbitrary Python objects, including very large numpy arrays. Separate
53
+ persistence and flow-execution logic from domain logic or algorithmic
54
+ code by writing the operations as a set of steps with well-defined
55
+ inputs and outputs: Python functions. Joblib can save their
56
+ computation to disk and rerun it only if necessary::
57
+
58
+ >>> from joblib import Memory
59
+ >>> location = 'your_cache_dir_goes_here'
60
+ >>> mem = Memory(location, verbose=1)
61
+ >>> import numpy as np
62
+ >>> a = np.vander(np.arange(3)).astype(float)
63
+ >>> square = mem.cache(np.square)
64
+ >>> b = square(a) # doctest: +ELLIPSIS
65
+ ______________________________________________________________________...
66
+ [Memory] Calling ...square...
67
+ square(array([[0., 0., 1.],
68
+ [1., 1., 1.],
69
+ [4., 2., 1.]]))
70
+ _________________________________________________...square - ...s, 0.0min
71
+
72
+ >>> c = square(a)
73
+ >>> # The above call did not trigger an evaluation
74
+
75
+ 2) **Embarrassingly parallel helper:** to make it easy to write readable
76
+ parallel code and debug it quickly::
77
+
78
+ >>> from joblib import Parallel, delayed
79
+ >>> from math import sqrt
80
+ >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
81
+ [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
82
+
83
+
84
+ 3) **Fast compressed Persistence**: a replacement for pickle to work
85
+ efficiently on Python objects containing large data (
86
+ *joblib.dump* & *joblib.load* ).
87
+
88
+ ..
89
+ >>> import shutil ; shutil.rmtree(location)
90
+
91
+ """
92
+
93
+ # PEP0440 compatible formatted version, see:
94
+ # https://www.python.org/dev/peps/pep-0440/
95
+ #
96
+ # Generic release markers:
97
+ # X.Y
98
+ # X.Y.Z # For bugfix releases
99
+ #
100
+ # Admissible pre-release markers:
101
+ # X.YaN # Alpha release
102
+ # X.YbN # Beta release
103
+ # X.YrcN # Release Candidate
104
+ # X.Y # Final release
105
+ #
106
+ # Dev branch marker is: 'X.Y.dev' or 'X.Y.devN' where N is an integer.
107
+ # 'X.Y.dev0' is the canonical version of 'X.Y.dev'
108
+ #
109
+ __version__ = "1.5.3"
110
+
111
+
112
+ import os
113
+
114
+ from ._cloudpickle_wrapper import wrap_non_picklable_objects
115
+ from ._parallel_backends import ParallelBackendBase
116
+ from ._store_backends import StoreBackendBase
117
+ from .compressor import register_compressor
118
+ from .hashing import hash
119
+ from .logger import Logger, PrintTime
120
+ from .memory import MemorizedResult, Memory, expires_after, register_store_backend
121
+ from .numpy_pickle import dump, load
122
+ from .parallel import (
123
+ Parallel,
124
+ cpu_count,
125
+ delayed,
126
+ effective_n_jobs,
127
+ parallel_backend,
128
+ parallel_config,
129
+ register_parallel_backend,
130
+ )
131
+
132
+ __all__ = [
133
+ # On-disk result caching
134
+ "Memory",
135
+ "MemorizedResult",
136
+ "expires_after",
137
+ # Parallel code execution
138
+ "Parallel",
139
+ "delayed",
140
+ "cpu_count",
141
+ "effective_n_jobs",
142
+ "wrap_non_picklable_objects",
143
+ # Context to change the backend globally
144
+ "parallel_config",
145
+ "parallel_backend",
146
+ # Helpers to define and register store/parallel backends
147
+ "ParallelBackendBase",
148
+ "StoreBackendBase",
149
+ "register_compressor",
150
+ "register_parallel_backend",
151
+ "register_store_backend",
152
+ # Helpers kept for backward compatibility
153
+ "PrintTime",
154
+ "Logger",
155
+ "hash",
156
+ "dump",
157
+ "load",
158
+ ]
159
+
160
+
161
+ # Workaround issue discovered in intel-openmp 2019.5:
162
+ # https://github.com/ContinuumIO/anaconda-issues/issues/11294
163
+ os.environ.setdefault("KMP_INIT_AT_FORK", "FALSE")
TTS/Kokoro/.venv/Lib/site-packages/joblib/_cloudpickle_wrapper.py ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Small shim of loky's cloudpickle_wrapper to avoid failure when
3
+ multiprocessing is not available.
4
+ """
5
+
6
+ from ._multiprocessing_helpers import mp
7
+
8
+
9
+ def _my_wrap_non_picklable_objects(obj, keep_wrapper=True):
10
+ return obj
11
+
12
+
13
+ if mp is not None:
14
+ from .externals.loky import wrap_non_picklable_objects
15
+ else:
16
+ wrap_non_picklable_objects = _my_wrap_non_picklable_objects
17
+
18
+ __all__ = ["wrap_non_picklable_objects"]
TTS/Kokoro/.venv/Lib/site-packages/joblib/_dask.py ADDED
@@ -0,0 +1,381 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import absolute_import, division, print_function
2
+
3
+ import asyncio
4
+ import concurrent.futures
5
+ import contextlib
6
+ import time
7
+ import weakref
8
+ from uuid import uuid4
9
+
10
+ from ._utils import (
11
+ _retrieve_traceback_capturing_wrapped_call,
12
+ _TracebackCapturingWrapper,
13
+ )
14
+ from .parallel import AutoBatchingMixin, ParallelBackendBase, parallel_config
15
+
16
+ try:
17
+ import dask
18
+ import distributed
19
+ except ImportError:
20
+ dask = None
21
+ distributed = None
22
+
23
+ if dask is not None and distributed is not None:
24
+ from dask.distributed import (
25
+ Client,
26
+ as_completed,
27
+ get_client,
28
+ rejoin,
29
+ secede,
30
+ )
31
+ from dask.sizeof import sizeof
32
+ from dask.utils import funcname
33
+ from distributed.utils import thread_state
34
+
35
+ try:
36
+ # asyncio.TimeoutError, Python3-only error thrown by recent versions of
37
+ # distributed
38
+ from distributed.utils import TimeoutError as _TimeoutError
39
+ except ImportError:
40
+ from tornado.gen import TimeoutError as _TimeoutError
41
+
42
+
43
+ def is_weakrefable(obj):
44
+ try:
45
+ weakref.ref(obj)
46
+ return True
47
+ except TypeError:
48
+ return False
49
+
50
+
51
+ class _WeakKeyDictionary:
52
+ """A variant of weakref.WeakKeyDictionary for unhashable objects.
53
+
54
+ This datastructure is used to store futures for broadcasted data objects
55
+ such as large numpy arrays or pandas dataframes that are not hashable and
56
+ therefore cannot be used as keys of traditional python dicts.
57
+
58
+ Furthermore using a dict with id(array) as key is not safe because the
59
+ Python is likely to reuse id of recently collected arrays.
60
+ """
61
+
62
+ def __init__(self):
63
+ self._data = {}
64
+
65
+ def __getitem__(self, obj):
66
+ ref, val = self._data[id(obj)]
67
+ if ref() is not obj:
68
+ # In case of a race condition with on_destroy.
69
+ raise KeyError(obj)
70
+ return val
71
+
72
+ def __setitem__(self, obj, value):
73
+ key = id(obj)
74
+ try:
75
+ ref, _ = self._data[key]
76
+ if ref() is not obj:
77
+ # In case of race condition with on_destroy.
78
+ raise KeyError(obj)
79
+ except KeyError:
80
+ # Insert the new entry in the mapping along with a weakref
81
+ # callback to automatically delete the entry from the mapping
82
+ # as soon as the object used as key is garbage collected.
83
+ def on_destroy(_):
84
+ del self._data[key]
85
+
86
+ ref = weakref.ref(obj, on_destroy)
87
+ self._data[key] = ref, value
88
+
89
+ def __len__(self):
90
+ return len(self._data)
91
+
92
+ def clear(self):
93
+ self._data.clear()
94
+
95
+
96
+ def _funcname(x):
97
+ try:
98
+ if isinstance(x, list):
99
+ x = x[0][0]
100
+ except Exception:
101
+ pass
102
+ return funcname(x)
103
+
104
+
105
+ def _make_tasks_summary(tasks):
106
+ """Summarize of list of (func, args, kwargs) function calls"""
107
+ unique_funcs = {func for func, args, kwargs in tasks}
108
+
109
+ if len(unique_funcs) == 1:
110
+ mixed = False
111
+ else:
112
+ mixed = True
113
+ return len(tasks), mixed, _funcname(tasks)
114
+
115
+
116
+ class Batch:
117
+ """dask-compatible wrapper that executes a batch of tasks"""
118
+
119
+ def __init__(self, tasks):
120
+ # collect some metadata from the tasks to ease Batch calls
121
+ # introspection when debugging
122
+ self._num_tasks, self._mixed, self._funcname = _make_tasks_summary(tasks)
123
+
124
+ def __call__(self, tasks=None):
125
+ results = []
126
+ with parallel_config(backend="dask"):
127
+ for func, args, kwargs in tasks:
128
+ results.append(func(*args, **kwargs))
129
+ return results
130
+
131
+ def __repr__(self):
132
+ descr = f"batch_of_{self._funcname}_{self._num_tasks}_calls"
133
+ if self._mixed:
134
+ descr = "mixed_" + descr
135
+ return descr
136
+
137
+
138
+ def _joblib_probe_task():
139
+ # Noop used by the joblib connector to probe when workers are ready.
140
+ pass
141
+
142
+
143
+ class DaskDistributedBackend(AutoBatchingMixin, ParallelBackendBase):
144
+ MIN_IDEAL_BATCH_DURATION = 0.2
145
+ MAX_IDEAL_BATCH_DURATION = 1.0
146
+ supports_retrieve_callback = True
147
+ default_n_jobs = -1
148
+
149
+ def __init__(
150
+ self,
151
+ scheduler_host=None,
152
+ scatter=None,
153
+ client=None,
154
+ loop=None,
155
+ wait_for_workers_timeout=10,
156
+ **submit_kwargs,
157
+ ):
158
+ super().__init__()
159
+
160
+ if distributed is None:
161
+ msg = (
162
+ "You are trying to use 'dask' as a joblib parallel backend "
163
+ "but dask is not installed. Please install dask "
164
+ "to fix this error."
165
+ )
166
+ raise ValueError(msg)
167
+
168
+ if client is None:
169
+ if scheduler_host:
170
+ client = Client(scheduler_host, loop=loop, set_as_default=False)
171
+ else:
172
+ try:
173
+ client = get_client()
174
+ except ValueError as e:
175
+ msg = (
176
+ "To use Joblib with Dask first create a Dask Client"
177
+ "\n\n"
178
+ " from dask.distributed import Client\n"
179
+ " client = Client()\n"
180
+ "or\n"
181
+ " client = Client('scheduler-address:8786')"
182
+ )
183
+ raise ValueError(msg) from e
184
+
185
+ self.client = client
186
+
187
+ if scatter is not None and not isinstance(scatter, (list, tuple)):
188
+ raise TypeError(
189
+ "scatter must be a list/tuple, got `%s`" % type(scatter).__name__
190
+ )
191
+
192
+ if scatter is not None and len(scatter) > 0:
193
+ # Keep a reference to the scattered data to keep the ids the same
194
+ self._scatter = list(scatter)
195
+ scattered = self.client.scatter(scatter, broadcast=True)
196
+ self.data_futures = {id(x): f for x, f in zip(scatter, scattered)}
197
+ else:
198
+ self._scatter = []
199
+ self.data_futures = {}
200
+ self.wait_for_workers_timeout = wait_for_workers_timeout
201
+ self.submit_kwargs = submit_kwargs
202
+ self.waiting_futures = as_completed(
203
+ [], loop=client.loop, with_results=True, raise_errors=False
204
+ )
205
+ self._results = {}
206
+ self._callbacks = {}
207
+
208
+ async def _collect(self):
209
+ while self._continue:
210
+ async for future, result in self.waiting_futures:
211
+ cf_future = self._results.pop(future)
212
+ callback = self._callbacks.pop(future)
213
+ if future.status == "error":
214
+ typ, exc, tb = result
215
+ cf_future.set_exception(exc)
216
+ else:
217
+ cf_future.set_result(result)
218
+ callback(result)
219
+ await asyncio.sleep(0.01)
220
+
221
+ def __reduce__(self):
222
+ return (DaskDistributedBackend, ())
223
+
224
+ def get_nested_backend(self):
225
+ return DaskDistributedBackend(client=self.client), -1
226
+
227
+ def configure(self, n_jobs=1, parallel=None, **backend_args):
228
+ self.parallel = parallel
229
+ return self.effective_n_jobs(n_jobs)
230
+
231
+ def start_call(self):
232
+ self._continue = True
233
+ self.client.loop.add_callback(self._collect)
234
+ self.call_data_futures = _WeakKeyDictionary()
235
+
236
+ def stop_call(self):
237
+ # The explicit call to clear is required to break a cycling reference
238
+ # to the futures.
239
+ self._continue = False
240
+ # wait for the future collection routine (self._backend._collect) to
241
+ # finish in order to limit asyncio warnings due to aborting _collect
242
+ # during a following backend termination call
243
+ time.sleep(0.01)
244
+ self.call_data_futures.clear()
245
+
246
+ def effective_n_jobs(self, n_jobs):
247
+ effective_n_jobs = sum(self.client.ncores().values())
248
+ if effective_n_jobs != 0 or not self.wait_for_workers_timeout:
249
+ return effective_n_jobs
250
+
251
+ # If there is no worker, schedule a probe task to wait for the workers
252
+ # to come up and be available. If the dask cluster is in adaptive mode
253
+ # task might cause the cluster to provision some workers.
254
+ try:
255
+ self.client.submit(_joblib_probe_task).result(
256
+ timeout=self.wait_for_workers_timeout
257
+ )
258
+ except _TimeoutError as e:
259
+ error_msg = (
260
+ "DaskDistributedBackend has no worker after {} seconds. "
261
+ "Make sure that workers are started and can properly connect "
262
+ "to the scheduler and increase the joblib/dask connection "
263
+ "timeout with:\n\n"
264
+ "parallel_config(backend='dask', wait_for_workers_timeout={})"
265
+ ).format(
266
+ self.wait_for_workers_timeout,
267
+ max(10, 2 * self.wait_for_workers_timeout),
268
+ )
269
+ raise TimeoutError(error_msg) from e
270
+ return sum(self.client.ncores().values())
271
+
272
+ async def _to_func_args(self, func):
273
+ itemgetters = dict()
274
+
275
+ # Futures that are dynamically generated during a single call to
276
+ # Parallel.__call__.
277
+ call_data_futures = getattr(self, "call_data_futures", None)
278
+
279
+ async def maybe_to_futures(args):
280
+ out = []
281
+ for arg in args:
282
+ arg_id = id(arg)
283
+ if arg_id in itemgetters:
284
+ out.append(itemgetters[arg_id])
285
+ continue
286
+
287
+ f = self.data_futures.get(arg_id, None)
288
+ if f is None and call_data_futures is not None:
289
+ try:
290
+ f = await call_data_futures[arg]
291
+ except KeyError:
292
+ pass
293
+ if f is None:
294
+ if is_weakrefable(arg) and sizeof(arg) > 1e3:
295
+ # Automatically scatter large objects to some of
296
+ # the workers to avoid duplicated data transfers.
297
+ # Rely on automated inter-worker data stealing if
298
+ # more workers need to reuse this data
299
+ # concurrently.
300
+ # set hash=False - nested scatter calls (i.e
301
+ # calling client.scatter inside a dask worker)
302
+ # using hash=True often raise CancelledError,
303
+ # see dask/distributed#3703
304
+ _coro = self.client.scatter(
305
+ arg, asynchronous=True, hash=False
306
+ )
307
+ # Centralize the scattering of identical arguments
308
+ # between concurrent apply_async callbacks by
309
+ # exposing the running coroutine in
310
+ # call_data_futures before it completes.
311
+ t = asyncio.Task(_coro)
312
+ call_data_futures[arg] = t
313
+
314
+ f = await t
315
+
316
+ if f is not None:
317
+ out.append(f)
318
+ else:
319
+ out.append(arg)
320
+ return out
321
+
322
+ tasks = []
323
+ for f, args, kwargs in func.items:
324
+ args = list(await maybe_to_futures(args))
325
+ kwargs = dict(zip(kwargs.keys(), await maybe_to_futures(kwargs.values())))
326
+ tasks.append((f, args, kwargs))
327
+
328
+ return (Batch(tasks), tasks)
329
+
330
+ def apply_async(self, func, callback=None):
331
+ cf_future = concurrent.futures.Future()
332
+ cf_future.get = cf_future.result # achieve AsyncResult API
333
+
334
+ async def f(func, callback):
335
+ batch, tasks = await self._to_func_args(func)
336
+ key = f"{repr(batch)}-{uuid4().hex}"
337
+
338
+ dask_future = self.client.submit(
339
+ _TracebackCapturingWrapper(batch),
340
+ tasks=tasks,
341
+ key=key,
342
+ **self.submit_kwargs,
343
+ )
344
+ self.waiting_futures.add(dask_future)
345
+ self._callbacks[dask_future] = callback
346
+ self._results[dask_future] = cf_future
347
+
348
+ self.client.loop.add_callback(f, func, callback)
349
+
350
+ return cf_future
351
+
352
+ def retrieve_result_callback(self, out):
353
+ return _retrieve_traceback_capturing_wrapped_call(out)
354
+
355
+ def abort_everything(self, ensure_ready=True):
356
+ """Tell the client to cancel any task submitted via this instance
357
+
358
+ joblib.Parallel will never access those results
359
+ """
360
+ with self.waiting_futures.lock:
361
+ self.waiting_futures.futures.clear()
362
+ while not self.waiting_futures.queue.empty():
363
+ self.waiting_futures.queue.get()
364
+
365
+ @contextlib.contextmanager
366
+ def retrieval_context(self):
367
+ """Override ParallelBackendBase.retrieval_context to avoid deadlocks.
368
+
369
+ This removes thread from the worker's thread pool (using 'secede').
370
+ Seceding avoids deadlock in nested parallelism settings.
371
+ """
372
+ # See 'joblib.Parallel.__call__' and 'joblib.Parallel.retrieve' for how
373
+ # this is used.
374
+ if hasattr(thread_state, "execution_state"):
375
+ # we are in a worker. Secede to avoid deadlock.
376
+ secede()
377
+
378
+ yield
379
+
380
+ if hasattr(thread_state, "execution_state"):
381
+ rejoin()
TTS/Kokoro/.venv/Lib/site-packages/joblib/_memmapping_reducer.py ADDED
@@ -0,0 +1,715 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Reducer using memory mapping for numpy arrays
3
+ """
4
+ # Author: Thomas Moreau <thomas.moreau.2010@gmail.com>
5
+ # Copyright: 2017, Thomas Moreau
6
+ # License: BSD 3 clause
7
+
8
+ import atexit
9
+ import errno
10
+ import os
11
+ import stat
12
+ import tempfile
13
+ import threading
14
+ import time
15
+ import warnings
16
+ import weakref
17
+ from mmap import mmap
18
+ from multiprocessing import util
19
+ from pickle import HIGHEST_PROTOCOL, PicklingError, dumps, loads, whichmodule
20
+ from uuid import uuid4
21
+
22
+ try:
23
+ WindowsError
24
+ except NameError:
25
+ WindowsError = type(None)
26
+
27
+ try:
28
+ import numpy as np
29
+ from numpy.lib.stride_tricks import as_strided
30
+ except ImportError:
31
+ np = None
32
+
33
+ from .backports import make_memmap
34
+ from .disk import delete_folder
35
+ from .externals.loky.backend import resource_tracker
36
+ from .numpy_pickle import dump, load, load_temporary_memmap
37
+
38
+ # Some system have a ramdisk mounted by default, we can use it instead of /tmp
39
+ # as the default folder to dump big arrays to share with subprocesses.
40
+ SYSTEM_SHARED_MEM_FS = "/dev/shm"
41
+
42
+ # Minimal number of bytes available on SYSTEM_SHARED_MEM_FS to consider using
43
+ # it as the default folder to dump big arrays to share with subprocesses.
44
+ SYSTEM_SHARED_MEM_FS_MIN_SIZE = int(2e9)
45
+
46
+ # Folder and file permissions to chmod temporary files generated by the
47
+ # memmapping pool. Only the owner of the Python process can access the
48
+ # temporary files and folder.
49
+ FOLDER_PERMISSIONS = stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
50
+ FILE_PERMISSIONS = stat.S_IRUSR | stat.S_IWUSR
51
+
52
+ # Set used in joblib workers, referencing the filenames of temporary memmaps
53
+ # created by joblib to speed up data communication. In child processes, we add
54
+ # a finalizer to these memmaps that sends a maybe_unlink call to the
55
+ # resource_tracker, in order to free main memory as fast as possible.
56
+ JOBLIB_MMAPS = set()
57
+
58
+
59
+ def _log_and_unlink(filename):
60
+ from .externals.loky.backend.resource_tracker import _resource_tracker
61
+
62
+ util.debug(
63
+ "[FINALIZER CALL] object mapping to {} about to be deleted,"
64
+ " decrementing the refcount of the file (pid: {})".format(
65
+ os.path.basename(filename), os.getpid()
66
+ )
67
+ )
68
+ _resource_tracker.maybe_unlink(filename, "file")
69
+
70
+
71
+ def add_maybe_unlink_finalizer(memmap):
72
+ util.debug(
73
+ "[FINALIZER ADD] adding finalizer to {} (id {}, filename {}, pid {})".format(
74
+ type(memmap), id(memmap), os.path.basename(memmap.filename), os.getpid()
75
+ )
76
+ )
77
+ weakref.finalize(memmap, _log_and_unlink, memmap.filename)
78
+
79
+
80
+ def unlink_file(filename):
81
+ """Wrapper around os.unlink with a retry mechanism.
82
+
83
+ The retry mechanism has been implemented primarily to overcome a race
84
+ condition happening during the finalizer of a np.memmap: when a process
85
+ holding the last reference to a mmap-backed np.memmap/np.array is about to
86
+ delete this array (and close the reference), it sends a maybe_unlink
87
+ request to the resource_tracker. This request can be processed faster than
88
+ it takes for the last reference of the memmap to be closed, yielding (on
89
+ Windows) a PermissionError in the resource_tracker loop.
90
+ """
91
+ NUM_RETRIES = 10
92
+ for retry_no in range(1, NUM_RETRIES + 1):
93
+ try:
94
+ os.unlink(filename)
95
+ break
96
+ except PermissionError:
97
+ util.debug(
98
+ "[ResourceTracker] tried to unlink {}, got PermissionError".format(
99
+ filename
100
+ )
101
+ )
102
+ if retry_no == NUM_RETRIES:
103
+ raise
104
+ else:
105
+ time.sleep(0.2)
106
+ except FileNotFoundError:
107
+ # In case of a race condition when deleting the temporary folder,
108
+ # avoid noisy FileNotFoundError exception in the resource tracker.
109
+ pass
110
+
111
+
112
+ resource_tracker._CLEANUP_FUNCS["file"] = unlink_file
113
+
114
+
115
+ class _WeakArrayKeyMap:
116
+ """A variant of weakref.WeakKeyDictionary for unhashable numpy arrays.
117
+
118
+ This datastructure will be used with numpy arrays as obj keys, therefore we
119
+ do not use the __get__ / __set__ methods to avoid any conflict with the
120
+ numpy fancy indexing syntax.
121
+ """
122
+
123
+ def __init__(self):
124
+ self._data = {}
125
+
126
+ def get(self, obj):
127
+ ref, val = self._data[id(obj)]
128
+ if ref() is not obj:
129
+ # In case of race condition with on_destroy: could never be
130
+ # triggered by the joblib tests with CPython.
131
+ raise KeyError(obj)
132
+ return val
133
+
134
+ def set(self, obj, value):
135
+ key = id(obj)
136
+ try:
137
+ ref, _ = self._data[key]
138
+ if ref() is not obj:
139
+ # In case of race condition with on_destroy: could never be
140
+ # triggered by the joblib tests with CPython.
141
+ raise KeyError(obj)
142
+ except KeyError:
143
+ # Insert the new entry in the mapping along with a weakref
144
+ # callback to automatically delete the entry from the mapping
145
+ # as soon as the object used as key is garbage collected.
146
+ def on_destroy(_):
147
+ del self._data[key]
148
+
149
+ ref = weakref.ref(obj, on_destroy)
150
+ self._data[key] = ref, value
151
+
152
+ def __getstate__(self):
153
+ raise PicklingError("_WeakArrayKeyMap is not pickleable")
154
+
155
+
156
+ ###############################################################################
157
+ # Support for efficient transient pickling of numpy data structures
158
+
159
+
160
+ def _get_backing_memmap(a):
161
+ """Recursively look up the original np.memmap instance base if any."""
162
+ b = getattr(a, "base", None)
163
+ if b is None:
164
+ # TODO: check scipy sparse datastructure if scipy is installed
165
+ # a nor its descendants do not have a memmap base
166
+ return None
167
+
168
+ elif isinstance(b, mmap):
169
+ # a is already a real memmap instance.
170
+ return a
171
+
172
+ else:
173
+ # Recursive exploration of the base ancestry
174
+ return _get_backing_memmap(b)
175
+
176
+
177
+ def _get_temp_dir(pool_folder_name, temp_folder=None):
178
+ """Get the full path to a subfolder inside the temporary folder.
179
+
180
+ Parameters
181
+ ----------
182
+ pool_folder_name : str
183
+ Sub-folder name used for the serialization of a pool instance.
184
+
185
+ temp_folder: str, optional
186
+ Folder to be used by the pool for memmapping large arrays
187
+ for sharing memory with worker processes. If None, this will try in
188
+ order:
189
+
190
+ - a folder pointed by the JOBLIB_TEMP_FOLDER environment
191
+ variable,
192
+ - /dev/shm if the folder exists and is writable: this is a
193
+ RAMdisk filesystem available by default on modern Linux
194
+ distributions,
195
+ - the default system temporary folder that can be
196
+ overridden with TMP, TMPDIR or TEMP environment
197
+ variables, typically /tmp under Unix operating systems.
198
+
199
+ Returns
200
+ -------
201
+ pool_folder : str
202
+ full path to the temporary folder
203
+ use_shared_mem : bool
204
+ whether the temporary folder is written to the system shared memory
205
+ folder or some other temporary folder.
206
+ """
207
+ use_shared_mem = False
208
+ if temp_folder is None:
209
+ temp_folder = os.environ.get("JOBLIB_TEMP_FOLDER", None)
210
+ if temp_folder is None:
211
+ if os.path.exists(SYSTEM_SHARED_MEM_FS) and hasattr(os, "statvfs"):
212
+ try:
213
+ shm_stats = os.statvfs(SYSTEM_SHARED_MEM_FS)
214
+ available_nbytes = shm_stats.f_bsize * shm_stats.f_bavail
215
+ if available_nbytes > SYSTEM_SHARED_MEM_FS_MIN_SIZE:
216
+ # Try to see if we have write access to the shared mem
217
+ # folder only if it is reasonably large (that is 2GB or
218
+ # more).
219
+ temp_folder = SYSTEM_SHARED_MEM_FS
220
+ pool_folder = os.path.join(temp_folder, pool_folder_name)
221
+ if not os.path.exists(pool_folder):
222
+ os.makedirs(pool_folder)
223
+ use_shared_mem = True
224
+ except (IOError, OSError):
225
+ # Missing rights in the /dev/shm partition, fallback to regular
226
+ # temp folder.
227
+ temp_folder = None
228
+ if temp_folder is None:
229
+ # Fallback to the default tmp folder, typically /tmp
230
+ temp_folder = tempfile.gettempdir()
231
+ temp_folder = os.path.abspath(os.path.expanduser(temp_folder))
232
+ pool_folder = os.path.join(temp_folder, pool_folder_name)
233
+ return pool_folder, use_shared_mem
234
+
235
+
236
+ def has_shareable_memory(a):
237
+ """Return True if a is backed by some mmap buffer directly or not."""
238
+ return _get_backing_memmap(a) is not None
239
+
240
+
241
+ def _strided_from_memmap(
242
+ filename,
243
+ dtype,
244
+ mode,
245
+ offset,
246
+ order,
247
+ shape,
248
+ strides,
249
+ total_buffer_len,
250
+ unlink_on_gc_collect,
251
+ ):
252
+ """Reconstruct an array view on a memory mapped file."""
253
+ if mode == "w+":
254
+ # Do not zero the original data when unpickling
255
+ mode = "r+"
256
+
257
+ if strides is None:
258
+ # Simple, contiguous memmap
259
+ return make_memmap(
260
+ filename,
261
+ dtype=dtype,
262
+ shape=shape,
263
+ mode=mode,
264
+ offset=offset,
265
+ order=order,
266
+ unlink_on_gc_collect=unlink_on_gc_collect,
267
+ )
268
+ else:
269
+ # For non-contiguous data, memmap the total enclosing buffer and then
270
+ # extract the non-contiguous view with the stride-tricks API
271
+ base = make_memmap(
272
+ filename,
273
+ dtype=dtype,
274
+ shape=total_buffer_len,
275
+ offset=offset,
276
+ mode=mode,
277
+ order=order,
278
+ unlink_on_gc_collect=unlink_on_gc_collect,
279
+ )
280
+ return as_strided(base, shape=shape, strides=strides)
281
+
282
+
283
+ def _reduce_memmap_backed(a, m):
284
+ """Pickling reduction for memmap backed arrays.
285
+
286
+ a is expected to be an instance of np.ndarray (or np.memmap)
287
+ m is expected to be an instance of np.memmap on the top of the ``base``
288
+ attribute ancestry of a. ``m.base`` should be the real python mmap object.
289
+ """
290
+ # offset that comes from the striding differences between a and m
291
+ util.debug(
292
+ "[MEMMAP REDUCE] reducing a memmap-backed array (shape, {}, pid: {})".format(
293
+ a.shape, os.getpid()
294
+ )
295
+ )
296
+ try:
297
+ from numpy.lib.array_utils import byte_bounds
298
+ except (ModuleNotFoundError, ImportError):
299
+ # Backward-compat for numpy < 2.0
300
+ from numpy import byte_bounds
301
+ a_start, a_end = byte_bounds(a)
302
+ m_start = byte_bounds(m)[0]
303
+ offset = a_start - m_start
304
+
305
+ # offset from the backing memmap
306
+ offset += m.offset
307
+
308
+ # 1D arrays are both F and C contiguous, so only set the flag in
309
+ # higher dimensions. See https://github.com/joblib/joblib/pull/1704.
310
+ if m.ndim > 1 and m.flags["F_CONTIGUOUS"]:
311
+ order = "F"
312
+ else:
313
+ # The backing memmap buffer is necessarily contiguous hence C if not
314
+ # Fortran
315
+ order = "C"
316
+
317
+ if a.flags["F_CONTIGUOUS"] or a.flags["C_CONTIGUOUS"]:
318
+ # If the array is a contiguous view, no need to pass the strides
319
+ strides = None
320
+ total_buffer_len = None
321
+ else:
322
+ # Compute the total number of items to map from which the strided
323
+ # view will be extracted.
324
+ strides = a.strides
325
+ total_buffer_len = (a_end - a_start) // a.itemsize
326
+
327
+ return (
328
+ _strided_from_memmap,
329
+ (
330
+ m.filename,
331
+ a.dtype,
332
+ m.mode,
333
+ offset,
334
+ order,
335
+ a.shape,
336
+ strides,
337
+ total_buffer_len,
338
+ False,
339
+ ),
340
+ )
341
+
342
+
343
+ def reduce_array_memmap_backward(a):
344
+ """reduce a np.array or a np.memmap from a child process"""
345
+ m = _get_backing_memmap(a)
346
+ if isinstance(m, np.memmap) and m.filename not in JOBLIB_MMAPS:
347
+ # if a is backed by a memmaped file, reconstruct a using the
348
+ # memmaped file.
349
+ return _reduce_memmap_backed(a, m)
350
+ else:
351
+ # a is either a regular (not memmap-backed) numpy array, or an array
352
+ # backed by a shared temporary file created by joblib. In the latter
353
+ # case, in order to limit the lifespan of these temporary files, we
354
+ # serialize the memmap as a regular numpy array, and decref the
355
+ # file backing the memmap (done implicitly in a previously registered
356
+ # finalizer, see ``unlink_on_gc_collect`` for more details)
357
+ return (loads, (dumps(np.asarray(a), protocol=HIGHEST_PROTOCOL),))
358
+
359
+
360
+ class ArrayMemmapForwardReducer(object):
361
+ """Reducer callable to dump large arrays to memmap files.
362
+
363
+ Parameters
364
+ ----------
365
+ max_nbytes: int
366
+ Threshold to trigger memmapping of large arrays to files created
367
+ a folder.
368
+ temp_folder_resolver: callable
369
+ An callable in charge of resolving a temporary folder name where files
370
+ for backing memmapped arrays are created.
371
+ mmap_mode: 'r', 'r+' or 'c'
372
+ Mode for the created memmap datastructure. See the documentation of
373
+ numpy.memmap for more details. Note: 'w+' is coerced to 'r+'
374
+ automatically to avoid zeroing the data on unpickling.
375
+ verbose: int, optional, 0 by default
376
+ If verbose > 0, memmap creations are logged.
377
+ If verbose > 1, both memmap creations, reuse and array pickling are
378
+ logged.
379
+ prewarm: bool, optional, False by default.
380
+ Force a read on newly memmapped array to make sure that OS pre-cache it
381
+ memory. This can be useful to avoid concurrent disk access when the
382
+ same data array is passed to different worker processes.
383
+ """
384
+
385
+ def __init__(
386
+ self,
387
+ max_nbytes,
388
+ temp_folder_resolver,
389
+ mmap_mode,
390
+ unlink_on_gc_collect,
391
+ verbose=0,
392
+ prewarm=True,
393
+ ):
394
+ self._max_nbytes = max_nbytes
395
+ self._temp_folder_resolver = temp_folder_resolver
396
+ self._mmap_mode = mmap_mode
397
+ self.verbose = int(verbose)
398
+ if prewarm == "auto":
399
+ self._prewarm = not self._temp_folder.startswith(SYSTEM_SHARED_MEM_FS)
400
+ else:
401
+ self._prewarm = prewarm
402
+ self._prewarm = prewarm
403
+ self._memmaped_arrays = _WeakArrayKeyMap()
404
+ self._temporary_memmaped_filenames = set()
405
+ self._unlink_on_gc_collect = unlink_on_gc_collect
406
+
407
+ @property
408
+ def _temp_folder(self):
409
+ return self._temp_folder_resolver()
410
+
411
+ def __reduce__(self):
412
+ # The ArrayMemmapForwardReducer is passed to the children processes: it
413
+ # needs to be pickled but the _WeakArrayKeyMap need to be skipped as
414
+ # it's only guaranteed to be consistent with the parent process memory
415
+ # garbage collection.
416
+ # Although this reducer is pickled, it is not needed in its destination
417
+ # process (child processes), as we only use this reducer to send
418
+ # memmaps from the parent process to the children processes. For this
419
+ # reason, we can afford skipping the resolver, (which would otherwise
420
+ # be unpicklable), and pass it as None instead.
421
+ args = (self._max_nbytes, None, self._mmap_mode, self._unlink_on_gc_collect)
422
+ kwargs = {
423
+ "verbose": self.verbose,
424
+ "prewarm": self._prewarm,
425
+ }
426
+ return ArrayMemmapForwardReducer, args, kwargs
427
+
428
+ def __call__(self, a):
429
+ m = _get_backing_memmap(a)
430
+ if m is not None and isinstance(m, np.memmap):
431
+ # a is already backed by a memmap file, let's reuse it directly
432
+ return _reduce_memmap_backed(a, m)
433
+
434
+ if (
435
+ not a.dtype.hasobject
436
+ and self._max_nbytes is not None
437
+ and a.nbytes > self._max_nbytes
438
+ ):
439
+ # check that the folder exists (lazily create the pool temp folder
440
+ # if required)
441
+ try:
442
+ os.makedirs(self._temp_folder)
443
+ os.chmod(self._temp_folder, FOLDER_PERMISSIONS)
444
+ except OSError as e:
445
+ if e.errno != errno.EEXIST:
446
+ raise e
447
+
448
+ try:
449
+ basename = self._memmaped_arrays.get(a)
450
+ except KeyError:
451
+ # Generate a new unique random filename. The process and thread
452
+ # ids are only useful for debugging purpose and to make it
453
+ # easier to cleanup orphaned files in case of hard process
454
+ # kill (e.g. by "kill -9" or segfault).
455
+ basename = "{}-{}-{}.pkl".format(
456
+ os.getpid(), id(threading.current_thread()), uuid4().hex
457
+ )
458
+ self._memmaped_arrays.set(a, basename)
459
+ filename = os.path.join(self._temp_folder, basename)
460
+
461
+ # In case the same array with the same content is passed several
462
+ # times to the pool subprocess children, serialize it only once
463
+
464
+ is_new_memmap = filename not in self._temporary_memmaped_filenames
465
+
466
+ # add the memmap to the list of temporary memmaps created by joblib
467
+ self._temporary_memmaped_filenames.add(filename)
468
+
469
+ if self._unlink_on_gc_collect:
470
+ # Bump reference count of the memmap by 1 to account for
471
+ # shared usage of the memmap by a child process. The
472
+ # corresponding decref call will be executed upon calling
473
+ # resource_tracker.maybe_unlink, registered as a finalizer in
474
+ # the child.
475
+ # the incref/decref calls here are only possible when the child
476
+ # and the parent share the same resource_tracker. It is not the
477
+ # case for the multiprocessing backend, but it does not matter
478
+ # because unlinking a memmap from a child process is only
479
+ # useful to control the memory usage of long-lasting child
480
+ # processes, while the multiprocessing-based pools terminate
481
+ # their workers at the end of a map() call.
482
+ resource_tracker.register(filename, "file")
483
+
484
+ if is_new_memmap:
485
+ # Incref each temporary memmap created by joblib one extra
486
+ # time. This means that these memmaps will only be deleted
487
+ # once an extra maybe_unlink() is called, which is done once
488
+ # all the jobs have completed (or been canceled) in the
489
+ # Parallel._terminate_backend() method.
490
+ resource_tracker.register(filename, "file")
491
+
492
+ if not os.path.exists(filename):
493
+ util.debug(
494
+ "[ARRAY DUMP] Pickling new array (shape={}, dtype={}) "
495
+ "creating a new memmap at {}".format(a.shape, a.dtype, filename)
496
+ )
497
+ for dumped_filename in dump(a, filename):
498
+ os.chmod(dumped_filename, FILE_PERMISSIONS)
499
+
500
+ if self._prewarm:
501
+ # Warm up the data by accessing it. This operation ensures
502
+ # that the disk access required to create the memmapping
503
+ # file are performed in the reducing process and avoids
504
+ # concurrent memmap creation in multiple children
505
+ # processes.
506
+ load(filename, mmap_mode=self._mmap_mode).max()
507
+
508
+ else:
509
+ util.debug(
510
+ "[ARRAY DUMP] Pickling known array (shape={}, dtype={}) "
511
+ "reusing memmap file: {}".format(
512
+ a.shape, a.dtype, os.path.basename(filename)
513
+ )
514
+ )
515
+
516
+ # The worker process will use joblib.load to memmap the data
517
+ return (
518
+ load_temporary_memmap,
519
+ (filename, self._mmap_mode, self._unlink_on_gc_collect),
520
+ )
521
+ else:
522
+ # do not convert a into memmap, let pickler do its usual copy with
523
+ # the default system pickler
524
+ util.debug(
525
+ "[ARRAY DUMP] Pickling array (NO MEMMAPPING) (shape={}, "
526
+ " dtype={}).".format(a.shape, a.dtype)
527
+ )
528
+ return (loads, (dumps(a, protocol=HIGHEST_PROTOCOL),))
529
+
530
+
531
+ def get_memmapping_reducers(
532
+ forward_reducers=None,
533
+ backward_reducers=None,
534
+ temp_folder_resolver=None,
535
+ max_nbytes=1e6,
536
+ mmap_mode="r",
537
+ verbose=0,
538
+ prewarm=False,
539
+ unlink_on_gc_collect=True,
540
+ **kwargs,
541
+ ):
542
+ """Construct a pair of memmapping reducer linked to a tmpdir.
543
+
544
+ This function manage the creation and the clean up of the temporary folders
545
+ underlying the memory maps and should be use to get the reducers necessary
546
+ to construct joblib pool or executor.
547
+ """
548
+ if forward_reducers is None:
549
+ forward_reducers = dict()
550
+ if backward_reducers is None:
551
+ backward_reducers = dict()
552
+
553
+ if np is not None:
554
+ # Register smart numpy.ndarray reducers that detects memmap backed
555
+ # arrays and that is also able to dump to memmap large in-memory
556
+ # arrays over the max_nbytes threshold
557
+ forward_reduce_ndarray = ArrayMemmapForwardReducer(
558
+ max_nbytes,
559
+ temp_folder_resolver,
560
+ mmap_mode,
561
+ unlink_on_gc_collect,
562
+ verbose,
563
+ prewarm=prewarm,
564
+ )
565
+ forward_reducers[np.ndarray] = forward_reduce_ndarray
566
+ forward_reducers[np.memmap] = forward_reduce_ndarray
567
+
568
+ # Communication from child process to the parent process always
569
+ # pickles in-memory numpy.ndarray without dumping them as memmap
570
+ # to avoid confusing the caller and make it tricky to collect the
571
+ # temporary folder
572
+ backward_reducers[np.ndarray] = reduce_array_memmap_backward
573
+ backward_reducers[np.memmap] = reduce_array_memmap_backward
574
+
575
+ return forward_reducers, backward_reducers
576
+
577
+
578
+ class TemporaryResourcesManager(object):
579
+ """Stateful object able to manage temporary folder and pickles
580
+
581
+ It exposes:
582
+ - a per-context folder name resolving API that memmap-based reducers will
583
+ rely on to know where to pickle the temporary memmaps
584
+ - a temporary file/folder management API that internally uses the
585
+ resource_tracker.
586
+ """
587
+
588
+ def __init__(self, temp_folder_root=None, context_id=None):
589
+ self._current_temp_folder = None
590
+ self._temp_folder_root = temp_folder_root
591
+ self._use_shared_mem = None
592
+ self._cached_temp_folders = dict()
593
+ self._id = uuid4().hex
594
+ self._finalizers = {}
595
+ if context_id is None:
596
+ # It would be safer to not assign a default context id (less silent
597
+ # bugs), but doing this while maintaining backward compatibility
598
+ # with the previous, context-unaware version get_memmaping_executor
599
+ # exposes too many low-level details.
600
+ context_id = uuid4().hex
601
+ self.set_current_context(context_id)
602
+
603
+ def set_current_context(self, context_id):
604
+ self._current_context_id = context_id
605
+ self.register_new_context(context_id)
606
+
607
+ def register_new_context(self, context_id):
608
+ # Prepare a sub-folder name specific to a context (usually a unique id
609
+ # generated by each instance of the Parallel class). Do not create in
610
+ # advance to spare FS write access if no array is to be dumped).
611
+ if context_id in self._cached_temp_folders:
612
+ return
613
+ else:
614
+ # During its lifecycle, one Parallel object can have several
615
+ # executors associated to it (for instance, if a loky worker raises
616
+ # an exception, joblib shutdowns the executor and instantly
617
+ # recreates a new one before raising the error - see
618
+ # ``ensure_ready``. Because we don't want two executors tied to
619
+ # the same Parallel object (and thus the same context id) to
620
+ # register/use/delete the same folder, we also add an id specific
621
+ # to the current Manager (and thus specific to its associated
622
+ # executor) to the folder name.
623
+ new_folder_name = "joblib_memmapping_folder_{}_{}_{}".format(
624
+ os.getpid(), self._id, context_id
625
+ )
626
+ new_folder_path, _ = _get_temp_dir(new_folder_name, self._temp_folder_root)
627
+ self.register_folder_finalizer(new_folder_path, context_id)
628
+ self._cached_temp_folders[context_id] = new_folder_path
629
+
630
+ def resolve_temp_folder_name(self):
631
+ """Return a folder name specific to the currently activated context"""
632
+ return self._cached_temp_folders[self._current_context_id]
633
+
634
+ # resource management API
635
+
636
+ def register_folder_finalizer(self, pool_subfolder, context_id):
637
+ # Register the garbage collector at program exit in case caller forgets
638
+ # to call terminate explicitly: note we do not pass any reference to
639
+ # ensure that this callback won't prevent garbage collection of
640
+ # parallel instance and related file handler resources such as POSIX
641
+ # semaphores and pipes
642
+ pool_module_name = whichmodule(delete_folder, "delete_folder")
643
+ resource_tracker.register(pool_subfolder, "folder")
644
+
645
+ def _cleanup():
646
+ # In some cases the Python runtime seems to set delete_folder to
647
+ # None just before exiting when accessing the delete_folder
648
+ # function from the closure namespace. So instead we reimport
649
+ # the delete_folder function explicitly.
650
+ # https://github.com/joblib/joblib/issues/328
651
+ # We cannot just use from 'joblib.pool import delete_folder'
652
+ # because joblib should only use relative imports to allow
653
+ # easy vendoring.
654
+ delete_folder = __import__(
655
+ pool_module_name, fromlist=["delete_folder"]
656
+ ).delete_folder
657
+ try:
658
+ delete_folder(pool_subfolder, allow_non_empty=True)
659
+ resource_tracker.unregister(pool_subfolder, "folder")
660
+ except OSError:
661
+ warnings.warn(
662
+ "Failed to delete temporary folder: {}".format(pool_subfolder)
663
+ )
664
+
665
+ self._finalizers[context_id] = atexit.register(_cleanup)
666
+
667
+ def _clean_temporary_resources(
668
+ self, context_id=None, force=False, allow_non_empty=False
669
+ ):
670
+ """Clean temporary resources created by a process-based pool"""
671
+ if context_id is None:
672
+ # Iterates over a copy of the cache keys to avoid Error due to
673
+ # iterating over a changing size dictionary.
674
+ for context_id in list(self._cached_temp_folders):
675
+ self._clean_temporary_resources(
676
+ context_id, force=force, allow_non_empty=allow_non_empty
677
+ )
678
+ else:
679
+ temp_folder = self._cached_temp_folders.get(context_id)
680
+ if temp_folder and os.path.exists(temp_folder):
681
+ for filename in os.listdir(temp_folder):
682
+ if force:
683
+ # Some workers have failed and the ref counted might
684
+ # be off. The workers should have shut down by this
685
+ # time so forcefully clean up the files.
686
+ resource_tracker.unregister(
687
+ os.path.join(temp_folder, filename), "file"
688
+ )
689
+ else:
690
+ resource_tracker.maybe_unlink(
691
+ os.path.join(temp_folder, filename), "file"
692
+ )
693
+
694
+ # When forcing clean-up, try to delete the folder even if some
695
+ # files are still in it. Otherwise, try to delete the folder
696
+ allow_non_empty |= force
697
+
698
+ # Clean up the folder if possible, either if it is empty or
699
+ # if none of the files in it are in used and allow_non_empty.
700
+ try:
701
+ delete_folder(temp_folder, allow_non_empty=allow_non_empty)
702
+ # Forget the folder once it has been deleted
703
+ self._cached_temp_folders.pop(context_id, None)
704
+ resource_tracker.unregister(temp_folder, "folder")
705
+
706
+ # Also cancel the finalizers that gets triggered at gc.
707
+ finalizer = self._finalizers.pop(context_id, None)
708
+ if finalizer is not None:
709
+ atexit.unregister(finalizer)
710
+
711
+ except OSError:
712
+ # Temporary folder cannot be deleted right now.
713
+ # This folder will be cleaned up by an atexit
714
+ # finalizer registered by the memmapping_reducer.
715
+ pass
TTS/Kokoro/.venv/Lib/site-packages/joblib/_multiprocessing_helpers.py ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Helper module to factorize the conditional multiprocessing import logic
2
+
3
+ We use a distinct module to simplify import statements and avoid introducing
4
+ circular dependencies (for instance for the assert_spawning name).
5
+ """
6
+
7
+ import os
8
+ import warnings
9
+
10
+ # Obtain possible configuration from the environment, assuming 1 (on)
11
+ # by default, upon 0 set to None. Should instructively fail if some non
12
+ # 0/1 value is set.
13
+ mp = int(os.environ.get("JOBLIB_MULTIPROCESSING", 1)) or None
14
+ if mp:
15
+ try:
16
+ import _multiprocessing # noqa
17
+ import multiprocessing as mp
18
+ except ImportError:
19
+ mp = None
20
+
21
+ # 2nd stage: validate that locking is available on the system and
22
+ # issue a warning if not
23
+ if mp is not None:
24
+ try:
25
+ # try to create a named semaphore using SemLock to make sure they are
26
+ # available on this platform. We use the low level object
27
+ # _multiprocessing.SemLock to avoid spawning a resource tracker on
28
+ # Unix system or changing the default backend.
29
+ import tempfile
30
+ from _multiprocessing import SemLock
31
+
32
+ _rand = tempfile._RandomNameSequence()
33
+ for i in range(100):
34
+ try:
35
+ name = "/joblib-{}-{}".format(os.getpid(), next(_rand))
36
+ _sem = SemLock(0, 0, 1, name=name, unlink=True)
37
+ del _sem # cleanup
38
+ break
39
+ except FileExistsError as e: # pragma: no cover
40
+ if i >= 99:
41
+ raise FileExistsError("cannot find name for semaphore") from e
42
+ except (FileExistsError, AttributeError, ImportError, OSError) as e:
43
+ mp = None
44
+ warnings.warn("%s. joblib will operate in serial mode" % (e,))
45
+
46
+
47
+ # 3rd stage: backward compat for the assert_spawning helper
48
+ if mp is not None:
49
+ from multiprocessing.context import assert_spawning
50
+ else:
51
+ assert_spawning = None
TTS/Kokoro/.venv/Lib/site-packages/joblib/_parallel_backends.py ADDED
@@ -0,0 +1,753 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Backends for embarrassingly parallel code.
3
+ """
4
+
5
+ import contextlib
6
+ import gc
7
+ import os
8
+ import threading
9
+ import warnings
10
+ from abc import ABCMeta, abstractmethod
11
+
12
+ from ._multiprocessing_helpers import mp
13
+ from ._utils import (
14
+ _retrieve_traceback_capturing_wrapped_call,
15
+ _TracebackCapturingWrapper,
16
+ )
17
+
18
+ if mp is not None:
19
+ from multiprocessing.pool import ThreadPool
20
+
21
+ from .executor import get_memmapping_executor
22
+
23
+ # Import loky only if multiprocessing is present
24
+ from .externals.loky import cpu_count, process_executor
25
+ from .externals.loky.process_executor import ShutdownExecutorError
26
+ from .pool import MemmappingPool
27
+
28
+
29
+ class ParallelBackendBase(metaclass=ABCMeta):
30
+ """Helper abc which defines all methods a ParallelBackend must implement"""
31
+
32
+ default_n_jobs = 1
33
+
34
+ supports_inner_max_num_threads = False
35
+
36
+ # This flag was introduced for backward compatibility reasons.
37
+ # New backends should always set it to True and implement the
38
+ # `retrieve_result_callback` method.
39
+ supports_retrieve_callback = False
40
+
41
+ @property
42
+ def supports_return_generator(self):
43
+ return self.supports_retrieve_callback
44
+
45
+ @property
46
+ def supports_timeout(self):
47
+ return self.supports_retrieve_callback
48
+
49
+ nesting_level = None
50
+
51
+ def __init__(
52
+ self, nesting_level=None, inner_max_num_threads=None, **backend_kwargs
53
+ ):
54
+ super().__init__()
55
+ self.nesting_level = nesting_level
56
+ self.inner_max_num_threads = inner_max_num_threads
57
+ self.backend_kwargs = backend_kwargs
58
+
59
+ MAX_NUM_THREADS_VARS = [
60
+ "OMP_NUM_THREADS",
61
+ "OPENBLAS_NUM_THREADS",
62
+ "MKL_NUM_THREADS",
63
+ "BLIS_NUM_THREADS",
64
+ "VECLIB_MAXIMUM_THREADS",
65
+ "NUMBA_NUM_THREADS",
66
+ "NUMEXPR_NUM_THREADS",
67
+ ]
68
+
69
+ TBB_ENABLE_IPC_VAR = "ENABLE_IPC"
70
+
71
+ @abstractmethod
72
+ def effective_n_jobs(self, n_jobs):
73
+ """Determine the number of jobs that can actually run in parallel
74
+
75
+ n_jobs is the number of workers requested by the callers. Passing
76
+ n_jobs=-1 means requesting all available workers for instance matching
77
+ the number of CPU cores on the worker host(s).
78
+
79
+ This method should return a guesstimate of the number of workers that
80
+ can actually perform work concurrently. The primary use case is to make
81
+ it possible for the caller to know in how many chunks to slice the
82
+ work.
83
+
84
+ In general working on larger data chunks is more efficient (less
85
+ scheduling overhead and better use of CPU cache prefetching heuristics)
86
+ as long as all the workers have enough work to do.
87
+ """
88
+
89
+ def apply_async(self, func, callback=None):
90
+ """Deprecated: implement `submit` instead."""
91
+ raise NotImplementedError("Implement `submit` instead.")
92
+
93
+ def submit(self, func, callback=None):
94
+ """Schedule a function to be run and return a future-like object.
95
+
96
+ This method should return a future-like object that allow tracking
97
+ the progress of the task.
98
+
99
+ If ``supports_retrieve_callback`` is False, the return value of this
100
+ method is passed to ``retrieve_result`` instead of calling
101
+ ``retrieve_result_callback``.
102
+
103
+ Parameters
104
+ ----------
105
+ func: callable
106
+ The function to be run in parallel.
107
+
108
+ callback: callable
109
+ A callable that will be called when the task is completed. This callable
110
+ is a wrapper around ``retrieve_result_callback``. This should be added
111
+ to the future-like object returned by this method, so that the callback
112
+ is called when the task is completed.
113
+
114
+ For future-like backends, this can be achieved with something like
115
+ ``future.add_done_callback(callback)``.
116
+
117
+ Returns
118
+ -------
119
+ future: future-like
120
+ A future-like object to track the execution of the submitted function.
121
+ """
122
+ warnings.warn(
123
+ "`apply_async` is deprecated, implement and use `submit` instead.",
124
+ DeprecationWarning,
125
+ )
126
+ return self.apply_async(func, callback)
127
+
128
+ def retrieve_result_callback(self, out):
129
+ """Called within the callback function passed to `submit`.
130
+
131
+ This method can customise how the result of the function is retrieved
132
+ from the future-like object.
133
+
134
+ Parameters
135
+ ----------
136
+ future: future-like
137
+ The future-like object returned by the `submit` method.
138
+
139
+ Returns
140
+ -------
141
+ result: object
142
+ The result of the function executed in parallel.
143
+ """
144
+
145
+ def retrieve_result(self, out, timeout=None):
146
+ """Hook to retrieve the result when support_retrieve_callback=False.
147
+
148
+ The argument `out` is the result of the `submit` call. This method
149
+ should return the result of the computation or raise an exception if
150
+ the computation failed.
151
+ """
152
+ if self.supports_timeout:
153
+ return out.get(timeout=timeout)
154
+ else:
155
+ return out.get()
156
+
157
+ def configure(
158
+ self, n_jobs=1, parallel=None, prefer=None, require=None, **backend_kwargs
159
+ ):
160
+ """Reconfigure the backend and return the number of workers.
161
+
162
+ This makes it possible to reuse an existing backend instance for
163
+ successive independent calls to Parallel with different parameters.
164
+ """
165
+ self.parallel = parallel
166
+ return self.effective_n_jobs(n_jobs)
167
+
168
+ def start_call(self):
169
+ """Call-back method called at the beginning of a Parallel call"""
170
+
171
+ def stop_call(self):
172
+ """Call-back method called at the end of a Parallel call"""
173
+
174
+ def terminate(self):
175
+ """Shutdown the workers and free the shared memory."""
176
+
177
+ def compute_batch_size(self):
178
+ """Determine the optimal batch size"""
179
+ return 1
180
+
181
+ def batch_completed(self, batch_size, duration):
182
+ """Callback indicate how long it took to run a batch"""
183
+
184
+ def abort_everything(self, ensure_ready=True):
185
+ """Abort any running tasks
186
+
187
+ This is called when an exception has been raised when executing a task
188
+ and all the remaining tasks will be ignored and can therefore be
189
+ aborted to spare computation resources.
190
+
191
+ If ensure_ready is True, the backend should be left in an operating
192
+ state as future tasks might be re-submitted via that same backend
193
+ instance.
194
+
195
+ If ensure_ready is False, the implementer of this method can decide
196
+ to leave the backend in a closed / terminated state as no new task
197
+ are expected to be submitted to this backend.
198
+
199
+ Setting ensure_ready to False is an optimization that can be leveraged
200
+ when aborting tasks via killing processes from a local process pool
201
+ managed by the backend it-self: if we expect no new tasks, there is no
202
+ point in re-creating new workers.
203
+ """
204
+ # Does nothing by default: to be overridden in subclasses when
205
+ # canceling tasks is possible.
206
+ pass
207
+
208
+ def get_nested_backend(self):
209
+ """Backend instance to be used by nested Parallel calls.
210
+
211
+ By default a thread-based backend is used for the first level of
212
+ nesting. Beyond, switch to sequential backend to avoid spawning too
213
+ many threads on the host.
214
+ """
215
+ nesting_level = getattr(self, "nesting_level", 0) + 1
216
+ if nesting_level > 1:
217
+ return SequentialBackend(nesting_level=nesting_level), None
218
+ else:
219
+ return ThreadingBackend(nesting_level=nesting_level), None
220
+
221
+ def _prepare_worker_env(self, n_jobs):
222
+ """Return environment variables limiting threadpools in external libs.
223
+
224
+ This function return a dict containing environment variables to pass
225
+ when creating a pool of process. These environment variables limit the
226
+ number of threads to `n_threads` for OpenMP, MKL, Accelerated and
227
+ OpenBLAS libraries in the child processes.
228
+ """
229
+ explicit_n_threads = self.inner_max_num_threads
230
+ default_n_threads = max(cpu_count() // n_jobs, 1)
231
+
232
+ # Set the inner environment variables to self.inner_max_num_threads if
233
+ # it is given. Else, default to cpu_count // n_jobs unless the variable
234
+ # is already present in the parent process environment.
235
+ env = {}
236
+ for var in self.MAX_NUM_THREADS_VARS:
237
+ if explicit_n_threads is None:
238
+ var_value = os.environ.get(var, default_n_threads)
239
+ else:
240
+ var_value = explicit_n_threads
241
+
242
+ env[var] = str(var_value)
243
+
244
+ if self.TBB_ENABLE_IPC_VAR not in os.environ:
245
+ # To avoid over-subscription when using TBB, let the TBB schedulers
246
+ # use Inter Process Communication to coordinate:
247
+ env[self.TBB_ENABLE_IPC_VAR] = "1"
248
+ return env
249
+
250
+ @contextlib.contextmanager
251
+ def retrieval_context(self):
252
+ """Context manager to manage an execution context.
253
+
254
+ Calls to Parallel.retrieve will be made inside this context.
255
+
256
+ By default, this does nothing. It may be useful for subclasses to
257
+ handle nested parallelism. In particular, it may be required to avoid
258
+ deadlocks if a backend manages a fixed number of workers, when those
259
+ workers may be asked to do nested Parallel calls. Without
260
+ 'retrieval_context' this could lead to deadlock, as all the workers
261
+ managed by the backend may be "busy" waiting for the nested parallel
262
+ calls to finish, but the backend has no free workers to execute those
263
+ tasks.
264
+ """
265
+ yield
266
+
267
+ @staticmethod
268
+ def in_main_thread():
269
+ return isinstance(threading.current_thread(), threading._MainThread)
270
+
271
+
272
+ class SequentialBackend(ParallelBackendBase):
273
+ """A ParallelBackend which will execute all batches sequentially.
274
+
275
+ Does not use/create any threading objects, and hence has minimal
276
+ overhead. Used when n_jobs == 1.
277
+ """
278
+
279
+ uses_threads = True
280
+ supports_timeout = False
281
+ supports_retrieve_callback = False
282
+ supports_sharedmem = True
283
+
284
+ def effective_n_jobs(self, n_jobs):
285
+ """Determine the number of jobs which are going to run in parallel"""
286
+ if n_jobs == 0:
287
+ raise ValueError("n_jobs == 0 in Parallel has no meaning")
288
+ return 1
289
+
290
+ def submit(self, func, callback=None):
291
+ """Schedule a func to be run"""
292
+ raise RuntimeError("Should never be called for SequentialBackend.")
293
+
294
+ def retrieve_result_callback(self, out):
295
+ raise RuntimeError("Should never be called for SequentialBackend.")
296
+
297
+ def get_nested_backend(self):
298
+ # import is not top level to avoid cyclic import errors.
299
+ from .parallel import get_active_backend
300
+
301
+ # SequentialBackend should neither change the nesting level, the
302
+ # default backend or the number of jobs. Just return the current one.
303
+ return get_active_backend()
304
+
305
+
306
+ class PoolManagerMixin(object):
307
+ """A helper class for managing pool of workers."""
308
+
309
+ _pool = None
310
+
311
+ def effective_n_jobs(self, n_jobs):
312
+ """Determine the number of jobs which are going to run in parallel"""
313
+ if n_jobs == 0:
314
+ raise ValueError("n_jobs == 0 in Parallel has no meaning")
315
+ elif mp is None or n_jobs is None:
316
+ # multiprocessing is not available or disabled, fallback
317
+ # to sequential mode
318
+ return 1
319
+ elif n_jobs < 0:
320
+ n_jobs = max(cpu_count() + 1 + n_jobs, 1)
321
+ return n_jobs
322
+
323
+ def terminate(self):
324
+ """Shutdown the process or thread pool"""
325
+ if self._pool is not None:
326
+ self._pool.close()
327
+ self._pool.terminate() # terminate does a join()
328
+ self._pool = None
329
+
330
+ def _get_pool(self):
331
+ """Used by `submit` to make it possible to implement lazy init"""
332
+ return self._pool
333
+
334
+ def submit(self, func, callback=None):
335
+ """Schedule a func to be run"""
336
+ # Here, we need a wrapper to avoid crashes on KeyboardInterruptErrors.
337
+ # We also call the callback on error, to make sure the pool does not
338
+ # wait on crashed jobs.
339
+ return self._get_pool().apply_async(
340
+ _TracebackCapturingWrapper(func),
341
+ (),
342
+ callback=callback,
343
+ error_callback=callback,
344
+ )
345
+
346
+ def retrieve_result_callback(self, result):
347
+ """Mimic concurrent.futures results, raising an error if needed."""
348
+ # In the multiprocessing Pool API, the callback are called with the
349
+ # result value as an argument so `result`(`out`) is the output of
350
+ # job.get(). It's either the result or the exception raised while
351
+ # collecting the result.
352
+ return _retrieve_traceback_capturing_wrapped_call(result)
353
+
354
+ def abort_everything(self, ensure_ready=True):
355
+ """Shutdown the pool and restart a new one with the same parameters"""
356
+ self.terminate()
357
+ if ensure_ready:
358
+ self.configure(
359
+ n_jobs=self.parallel.n_jobs,
360
+ parallel=self.parallel,
361
+ **self.parallel._backend_kwargs,
362
+ )
363
+
364
+
365
+ class AutoBatchingMixin(object):
366
+ """A helper class for automagically batching jobs."""
367
+
368
+ # In seconds, should be big enough to hide multiprocessing dispatching
369
+ # overhead.
370
+ # This settings was found by running benchmarks/bench_auto_batching.py
371
+ # with various parameters on various platforms.
372
+ MIN_IDEAL_BATCH_DURATION = 0.2
373
+
374
+ # Should not be too high to avoid stragglers: long jobs running alone
375
+ # on a single worker while other workers have no work to process any more.
376
+ MAX_IDEAL_BATCH_DURATION = 2
377
+
378
+ # Batching counters default values
379
+ _DEFAULT_EFFECTIVE_BATCH_SIZE = 1
380
+ _DEFAULT_SMOOTHED_BATCH_DURATION = 0.0
381
+
382
+ def __init__(self, **kwargs):
383
+ super().__init__(**kwargs)
384
+ self._effective_batch_size = self._DEFAULT_EFFECTIVE_BATCH_SIZE
385
+ self._smoothed_batch_duration = self._DEFAULT_SMOOTHED_BATCH_DURATION
386
+
387
+ def compute_batch_size(self):
388
+ """Determine the optimal batch size"""
389
+ old_batch_size = self._effective_batch_size
390
+ batch_duration = self._smoothed_batch_duration
391
+ if batch_duration > 0 and batch_duration < self.MIN_IDEAL_BATCH_DURATION:
392
+ # The current batch size is too small: the duration of the
393
+ # processing of a batch of task is not large enough to hide
394
+ # the scheduling overhead.
395
+ ideal_batch_size = int(
396
+ old_batch_size * self.MIN_IDEAL_BATCH_DURATION / batch_duration
397
+ )
398
+ # Multiply by two to limit oscilations between min and max.
399
+ ideal_batch_size *= 2
400
+
401
+ # dont increase the batch size too fast to limit huge batch sizes
402
+ # potentially leading to starving worker
403
+ batch_size = min(2 * old_batch_size, ideal_batch_size)
404
+
405
+ batch_size = max(batch_size, 1)
406
+
407
+ self._effective_batch_size = batch_size
408
+ if self.parallel.verbose >= 10:
409
+ self.parallel._print(
410
+ f"Batch computation too fast ({batch_duration}s.) "
411
+ f"Setting batch_size={batch_size}."
412
+ )
413
+ elif batch_duration > self.MAX_IDEAL_BATCH_DURATION and old_batch_size >= 2:
414
+ # The current batch size is too big. If we schedule overly long
415
+ # running batches some CPUs might wait with nothing left to do
416
+ # while a couple of CPUs a left processing a few long running
417
+ # batches. Better reduce the batch size a bit to limit the
418
+ # likelihood of scheduling such stragglers.
419
+
420
+ # decrease the batch size quickly to limit potential starving
421
+ ideal_batch_size = int(
422
+ old_batch_size * self.MIN_IDEAL_BATCH_DURATION / batch_duration
423
+ )
424
+ # Multiply by two to limit oscilations between min and max.
425
+ batch_size = max(2 * ideal_batch_size, 1)
426
+ self._effective_batch_size = batch_size
427
+ if self.parallel.verbose >= 10:
428
+ self.parallel._print(
429
+ f"Batch computation too slow ({batch_duration}s.) "
430
+ f"Setting batch_size={batch_size}."
431
+ )
432
+ else:
433
+ # No batch size adjustment
434
+ batch_size = old_batch_size
435
+
436
+ if batch_size != old_batch_size:
437
+ # Reset estimation of the smoothed mean batch duration: this
438
+ # estimate is updated in the multiprocessing apply_async
439
+ # CallBack as long as the batch_size is constant. Therefore
440
+ # we need to reset the estimate whenever we re-tune the batch
441
+ # size.
442
+ self._smoothed_batch_duration = self._DEFAULT_SMOOTHED_BATCH_DURATION
443
+
444
+ return batch_size
445
+
446
+ def batch_completed(self, batch_size, duration):
447
+ """Callback indicate how long it took to run a batch"""
448
+ if batch_size == self._effective_batch_size:
449
+ # Update the smoothed streaming estimate of the duration of a batch
450
+ # from dispatch to completion
451
+ old_duration = self._smoothed_batch_duration
452
+ if old_duration == self._DEFAULT_SMOOTHED_BATCH_DURATION:
453
+ # First record of duration for this batch size after the last
454
+ # reset.
455
+ new_duration = duration
456
+ else:
457
+ # Update the exponentially weighted average of the duration of
458
+ # batch for the current effective size.
459
+ new_duration = 0.8 * old_duration + 0.2 * duration
460
+ self._smoothed_batch_duration = new_duration
461
+
462
+ def reset_batch_stats(self):
463
+ """Reset batch statistics to default values.
464
+
465
+ This avoids interferences with future jobs.
466
+ """
467
+ self._effective_batch_size = self._DEFAULT_EFFECTIVE_BATCH_SIZE
468
+ self._smoothed_batch_duration = self._DEFAULT_SMOOTHED_BATCH_DURATION
469
+
470
+
471
+ class ThreadingBackend(PoolManagerMixin, ParallelBackendBase):
472
+ """A ParallelBackend which will use a thread pool to execute batches in.
473
+
474
+ This is a low-overhead backend but it suffers from the Python Global
475
+ Interpreter Lock if the called function relies a lot on Python objects.
476
+ Mostly useful when the execution bottleneck is a compiled extension that
477
+ explicitly releases the GIL (for instance a Cython loop wrapped in a "with
478
+ nogil" block or an expensive call to a library such as NumPy).
479
+
480
+ The actual thread pool is lazily initialized: the actual thread pool
481
+ construction is delayed to the first call to apply_async.
482
+
483
+ ThreadingBackend is used as the default backend for nested calls.
484
+ """
485
+
486
+ supports_retrieve_callback = True
487
+ uses_threads = True
488
+ supports_sharedmem = True
489
+
490
+ def configure(self, n_jobs=1, parallel=None, **backend_kwargs):
491
+ """Build a process or thread pool and return the number of workers"""
492
+ n_jobs = self.effective_n_jobs(n_jobs)
493
+ if n_jobs == 1:
494
+ # Avoid unnecessary overhead and use sequential backend instead.
495
+ raise FallbackToBackend(SequentialBackend(nesting_level=self.nesting_level))
496
+ self.parallel = parallel
497
+ self._n_jobs = n_jobs
498
+ return n_jobs
499
+
500
+ def _get_pool(self):
501
+ """Lazily initialize the thread pool
502
+
503
+ The actual pool of worker threads is only initialized at the first
504
+ call to apply_async.
505
+ """
506
+ if self._pool is None:
507
+ self._pool = ThreadPool(self._n_jobs)
508
+ return self._pool
509
+
510
+
511
+ class MultiprocessingBackend(PoolManagerMixin, AutoBatchingMixin, ParallelBackendBase):
512
+ """A ParallelBackend which will use a multiprocessing.Pool.
513
+
514
+ Will introduce some communication and memory overhead when exchanging
515
+ input and output data with the with the worker Python processes.
516
+ However, does not suffer from the Python Global Interpreter Lock.
517
+ """
518
+
519
+ supports_retrieve_callback = True
520
+ supports_return_generator = False
521
+
522
+ def effective_n_jobs(self, n_jobs):
523
+ """Determine the number of jobs which are going to run in parallel.
524
+
525
+ This also checks if we are attempting to create a nested parallel
526
+ loop.
527
+ """
528
+ if mp is None:
529
+ return 1
530
+
531
+ if mp.current_process().daemon:
532
+ # Daemonic processes cannot have children
533
+ if n_jobs != 1:
534
+ if inside_dask_worker():
535
+ msg = (
536
+ "Inside a Dask worker with daemon=True, "
537
+ "setting n_jobs=1.\nPossible work-arounds:\n"
538
+ "- dask.config.set("
539
+ "{'distributed.worker.daemon': False})"
540
+ "- set the environment variable "
541
+ "DASK_DISTRIBUTED__WORKER__DAEMON=False\n"
542
+ "before creating your Dask cluster."
543
+ )
544
+ else:
545
+ msg = (
546
+ "Multiprocessing-backed parallel loops "
547
+ "cannot be nested, setting n_jobs=1"
548
+ )
549
+ warnings.warn(msg, stacklevel=3)
550
+ return 1
551
+
552
+ if process_executor._CURRENT_DEPTH > 0:
553
+ # Mixing loky and multiprocessing in nested loop is not supported
554
+ if n_jobs != 1:
555
+ warnings.warn(
556
+ "Multiprocessing-backed parallel loops cannot be nested,"
557
+ " below loky, setting n_jobs=1",
558
+ stacklevel=3,
559
+ )
560
+ return 1
561
+
562
+ elif not (self.in_main_thread() or self.nesting_level == 0):
563
+ # Prevent posix fork inside in non-main posix threads
564
+ if n_jobs != 1:
565
+ warnings.warn(
566
+ "Multiprocessing-backed parallel loops cannot be nested"
567
+ " below threads, setting n_jobs=1",
568
+ stacklevel=3,
569
+ )
570
+ return 1
571
+
572
+ return super(MultiprocessingBackend, self).effective_n_jobs(n_jobs)
573
+
574
+ def configure(
575
+ self,
576
+ n_jobs=1,
577
+ parallel=None,
578
+ prefer=None,
579
+ require=None,
580
+ **memmapping_pool_kwargs,
581
+ ):
582
+ """Build a process or thread pool and return the number of workers"""
583
+ n_jobs = self.effective_n_jobs(n_jobs)
584
+ if n_jobs == 1:
585
+ raise FallbackToBackend(SequentialBackend(nesting_level=self.nesting_level))
586
+
587
+ memmapping_pool_kwargs = {
588
+ **self.backend_kwargs,
589
+ **memmapping_pool_kwargs,
590
+ }
591
+
592
+ # Make sure to free as much memory as possible before forking
593
+ gc.collect()
594
+ self._pool = MemmappingPool(n_jobs, **memmapping_pool_kwargs)
595
+ self.parallel = parallel
596
+ return n_jobs
597
+
598
+ def terminate(self):
599
+ """Shutdown the process or thread pool"""
600
+ super(MultiprocessingBackend, self).terminate()
601
+ self.reset_batch_stats()
602
+
603
+
604
+ class LokyBackend(AutoBatchingMixin, ParallelBackendBase):
605
+ """Managing pool of workers with loky instead of multiprocessing."""
606
+
607
+ supports_retrieve_callback = True
608
+ supports_inner_max_num_threads = True
609
+
610
+ def configure(
611
+ self,
612
+ n_jobs=1,
613
+ parallel=None,
614
+ prefer=None,
615
+ require=None,
616
+ idle_worker_timeout=None,
617
+ **memmapping_executor_kwargs,
618
+ ):
619
+ """Build a process executor and return the number of workers"""
620
+ n_jobs = self.effective_n_jobs(n_jobs)
621
+ if n_jobs == 1:
622
+ raise FallbackToBackend(SequentialBackend(nesting_level=self.nesting_level))
623
+
624
+ memmapping_executor_kwargs = {
625
+ **self.backend_kwargs,
626
+ **memmapping_executor_kwargs,
627
+ }
628
+
629
+ # Prohibit the use of 'timeout' in the LokyBackend, as 'idle_worker_timeout'
630
+ # better describes the backend's behavior.
631
+ if "timeout" in memmapping_executor_kwargs:
632
+ raise ValueError(
633
+ "The 'timeout' parameter is not supported by the LokyBackend. "
634
+ "Please use the `idle_worker_timeout` parameter instead."
635
+ )
636
+ if idle_worker_timeout is None:
637
+ idle_worker_timeout = self.backend_kwargs.get("idle_worker_timeout", 300)
638
+
639
+ self._workers = get_memmapping_executor(
640
+ n_jobs,
641
+ timeout=idle_worker_timeout,
642
+ env=self._prepare_worker_env(n_jobs=n_jobs),
643
+ context_id=parallel._id,
644
+ **memmapping_executor_kwargs,
645
+ )
646
+ self.parallel = parallel
647
+ return n_jobs
648
+
649
+ def effective_n_jobs(self, n_jobs):
650
+ """Determine the number of jobs which are going to run in parallel"""
651
+ if n_jobs == 0:
652
+ raise ValueError("n_jobs == 0 in Parallel has no meaning")
653
+ elif mp is None or n_jobs is None:
654
+ # multiprocessing is not available or disabled, fallback
655
+ # to sequential mode
656
+ return 1
657
+ elif mp.current_process().daemon:
658
+ # Daemonic processes cannot have children
659
+ if n_jobs != 1:
660
+ if inside_dask_worker():
661
+ msg = (
662
+ "Inside a Dask worker with daemon=True, "
663
+ "setting n_jobs=1.\nPossible work-arounds:\n"
664
+ "- dask.config.set("
665
+ "{'distributed.worker.daemon': False})\n"
666
+ "- set the environment variable "
667
+ "DASK_DISTRIBUTED__WORKER__DAEMON=False\n"
668
+ "before creating your Dask cluster."
669
+ )
670
+ else:
671
+ msg = (
672
+ "Loky-backed parallel loops cannot be called in a"
673
+ " multiprocessing, setting n_jobs=1"
674
+ )
675
+ warnings.warn(msg, stacklevel=3)
676
+
677
+ return 1
678
+ elif not (self.in_main_thread() or self.nesting_level == 0):
679
+ # Prevent posix fork inside in non-main posix threads
680
+ if n_jobs != 1:
681
+ warnings.warn(
682
+ "Loky-backed parallel loops cannot be nested below "
683
+ "threads, setting n_jobs=1",
684
+ stacklevel=3,
685
+ )
686
+ return 1
687
+ elif n_jobs < 0:
688
+ n_jobs = max(cpu_count() + 1 + n_jobs, 1)
689
+ return n_jobs
690
+
691
+ def submit(self, func, callback=None):
692
+ """Schedule a func to be run"""
693
+ future = self._workers.submit(func)
694
+ if callback is not None:
695
+ future.add_done_callback(callback)
696
+ return future
697
+
698
+ def retrieve_result_callback(self, future):
699
+ """Retrieve the result, here out is the future given by submit"""
700
+ try:
701
+ return future.result()
702
+ except ShutdownExecutorError:
703
+ raise RuntimeError(
704
+ "The executor underlying Parallel has been shutdown. "
705
+ "This is likely due to the garbage collection of a previous "
706
+ "generator from a call to Parallel with return_as='generator'."
707
+ " Make sure the generator is not garbage collected when "
708
+ "submitting a new job or that it is first properly exhausted."
709
+ )
710
+
711
+ def terminate(self):
712
+ if self._workers is not None:
713
+ # Don't terminate the workers as we want to reuse them in later
714
+ # calls, but cleanup the temporary resources that the Parallel call
715
+ # created. This 'hack' requires a private, low-level operation.
716
+ self._workers._temp_folder_manager._clean_temporary_resources(
717
+ context_id=self.parallel._id, force=False
718
+ )
719
+ self._workers = None
720
+
721
+ self.reset_batch_stats()
722
+
723
+ def abort_everything(self, ensure_ready=True):
724
+ """Shutdown the workers and restart a new one with the same parameters"""
725
+ self._workers.terminate(kill_workers=True)
726
+ self._workers = None
727
+
728
+ if ensure_ready:
729
+ self.configure(n_jobs=self.parallel.n_jobs, parallel=self.parallel)
730
+
731
+
732
+ class FallbackToBackend(Exception):
733
+ """Raised when configuration should fallback to another backend"""
734
+
735
+ def __init__(self, backend):
736
+ self.backend = backend
737
+
738
+
739
+ def inside_dask_worker():
740
+ """Check whether the current function is executed inside a Dask worker."""
741
+ # This function can not be in joblib._dask because there would be a
742
+ # circular import:
743
+ # _dask imports _parallel_backend that imports _dask ...
744
+ try:
745
+ from distributed import get_worker
746
+ except ImportError:
747
+ return False
748
+
749
+ try:
750
+ get_worker()
751
+ return True
752
+ except ValueError:
753
+ return False
TTS/Kokoro/.venv/Lib/site-packages/joblib/_store_backends.py ADDED
@@ -0,0 +1,500 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Storage providers backends for Memory caching."""
2
+
3
+ import collections
4
+ import datetime
5
+ import json
6
+ import operator
7
+ import os
8
+ import os.path
9
+ import re
10
+ import shutil
11
+ import threading
12
+ import time
13
+ import uuid
14
+ import warnings
15
+ from abc import ABCMeta, abstractmethod
16
+ from pickle import PicklingError
17
+
18
+ from . import numpy_pickle
19
+ from .backports import concurrency_safe_rename
20
+ from .disk import memstr_to_bytes, mkdirp, rm_subdirs
21
+ from .logger import format_time
22
+
23
+ CacheItemInfo = collections.namedtuple("CacheItemInfo", "path size last_access")
24
+
25
+
26
+ class CacheWarning(Warning):
27
+ """Warning to capture dump failures except for PicklingError."""
28
+
29
+ pass
30
+
31
+
32
+ def concurrency_safe_write(object_to_write, filename, write_func):
33
+ """Writes an object into a unique file in a concurrency-safe way."""
34
+ # Temporary name is composed of UUID, process_id and thread_id to avoid
35
+ # collisions due to concurrent write.
36
+ # UUID is unique across nodes and time and help avoid collisions, even if
37
+ # the cache folder is shared by several Python processes with the same pid and
38
+ # thread id on different nodes of a cluster for instance.
39
+ thread_id = id(threading.current_thread())
40
+ temporary_filename = f"{filename}.{uuid.uuid4().hex}-{os.getpid()}-{thread_id}"
41
+
42
+ write_func(object_to_write, temporary_filename)
43
+
44
+ return temporary_filename
45
+
46
+
47
+ class StoreBackendBase(metaclass=ABCMeta):
48
+ """Helper Abstract Base Class which defines all methods that
49
+ a StorageBackend must implement."""
50
+
51
+ location = None
52
+
53
+ @abstractmethod
54
+ def _open_item(self, f, mode):
55
+ """Opens an item on the store and return a file-like object.
56
+
57
+ This method is private and only used by the StoreBackendMixin object.
58
+
59
+ Parameters
60
+ ----------
61
+ f: a file-like object
62
+ The file-like object where an item is stored and retrieved
63
+ mode: string, optional
64
+ the mode in which the file-like object is opened allowed valued are
65
+ 'rb', 'wb'
66
+
67
+ Returns
68
+ -------
69
+ a file-like object
70
+ """
71
+
72
+ @abstractmethod
73
+ def _item_exists(self, location):
74
+ """Checks if an item location exists in the store.
75
+
76
+ This method is private and only used by the StoreBackendMixin object.
77
+
78
+ Parameters
79
+ ----------
80
+ location: string
81
+ The location of an item. On a filesystem, this corresponds to the
82
+ absolute path, including the filename, of a file.
83
+
84
+ Returns
85
+ -------
86
+ True if the item exists, False otherwise
87
+ """
88
+
89
+ @abstractmethod
90
+ def _move_item(self, src, dst):
91
+ """Moves an item from src to dst in the store.
92
+
93
+ This method is private and only used by the StoreBackendMixin object.
94
+
95
+ Parameters
96
+ ----------
97
+ src: string
98
+ The source location of an item
99
+ dst: string
100
+ The destination location of an item
101
+ """
102
+
103
+ @abstractmethod
104
+ def create_location(self, location):
105
+ """Creates a location on the store.
106
+
107
+ Parameters
108
+ ----------
109
+ location: string
110
+ The location in the store. On a filesystem, this corresponds to a
111
+ directory.
112
+ """
113
+
114
+ @abstractmethod
115
+ def clear_location(self, location):
116
+ """Clears a location on the store.
117
+
118
+ Parameters
119
+ ----------
120
+ location: string
121
+ The location in the store. On a filesystem, this corresponds to a
122
+ directory or a filename absolute path
123
+ """
124
+
125
+ @abstractmethod
126
+ def get_items(self):
127
+ """Returns the whole list of items available in the store.
128
+
129
+ Returns
130
+ -------
131
+ The list of items identified by their ids (e.g filename in a
132
+ filesystem).
133
+ """
134
+
135
+ @abstractmethod
136
+ def configure(self, location, verbose=0, backend_options=dict()):
137
+ """Configures the store.
138
+
139
+ Parameters
140
+ ----------
141
+ location: string
142
+ The base location used by the store. On a filesystem, this
143
+ corresponds to a directory.
144
+ verbose: int
145
+ The level of verbosity of the store
146
+ backend_options: dict
147
+ Contains a dictionary of named parameters used to configure the
148
+ store backend.
149
+ """
150
+
151
+
152
+ class StoreBackendMixin(object):
153
+ """Class providing all logic for managing the store in a generic way.
154
+
155
+ The StoreBackend subclass has to implement 3 methods: create_location,
156
+ clear_location and configure. The StoreBackend also has to provide
157
+ a private _open_item, _item_exists and _move_item methods. The _open_item
158
+ method has to have the same signature as the builtin open and return a
159
+ file-like object.
160
+ """
161
+
162
+ def load_item(self, call_id, verbose=1, timestamp=None, metadata=None):
163
+ """Load an item from the store given its id as a list of str."""
164
+ full_path = os.path.join(self.location, *call_id)
165
+
166
+ if verbose > 1:
167
+ ts_string = (
168
+ "{: <16}".format(format_time(time.time() - timestamp))
169
+ if timestamp is not None
170
+ else ""
171
+ )
172
+ signature = os.path.basename(call_id[0])
173
+ if metadata is not None and "input_args" in metadata:
174
+ kwargs = ", ".join(
175
+ "{}={}".format(*item) for item in metadata["input_args"].items()
176
+ )
177
+ signature += "({})".format(kwargs)
178
+ msg = "[Memory]{}: Loading {}".format(ts_string, signature)
179
+ if verbose < 10:
180
+ print("{0}...".format(msg))
181
+ else:
182
+ print("{0} from {1}".format(msg, full_path))
183
+
184
+ mmap_mode = None if not hasattr(self, "mmap_mode") else self.mmap_mode
185
+
186
+ filename = os.path.join(full_path, "output.pkl")
187
+ if not self._item_exists(filename):
188
+ raise KeyError(
189
+ "Non-existing item (may have been "
190
+ "cleared).\nFile %s does not exist" % filename
191
+ )
192
+
193
+ # file-like object cannot be used when mmap_mode is set
194
+ if mmap_mode is None:
195
+ with self._open_item(filename, "rb") as f:
196
+ item = numpy_pickle.load(f)
197
+ else:
198
+ item = numpy_pickle.load(filename, mmap_mode=mmap_mode)
199
+ return item
200
+
201
+ def dump_item(self, call_id, item, verbose=1):
202
+ """Dump an item in the store at the id given as a list of str."""
203
+ try:
204
+ item_path = os.path.join(self.location, *call_id)
205
+ if not self._item_exists(item_path):
206
+ self.create_location(item_path)
207
+ filename = os.path.join(item_path, "output.pkl")
208
+ if verbose > 10:
209
+ print("Persisting in %s" % item_path)
210
+
211
+ def write_func(to_write, dest_filename):
212
+ with self._open_item(dest_filename, "wb") as f:
213
+ try:
214
+ numpy_pickle.dump(to_write, f, compress=self.compress)
215
+ except PicklingError as e:
216
+ # TODO(1.5) turn into error
217
+ warnings.warn(
218
+ "Unable to cache to disk: failed to pickle "
219
+ "output. In version 1.5 this will raise an "
220
+ f"exception. Exception: {e}.",
221
+ FutureWarning,
222
+ )
223
+
224
+ self._concurrency_safe_write(item, filename, write_func)
225
+ except Exception as e: # noqa: E722
226
+ warnings.warn(
227
+ "Unable to cache to disk. Possibly a race condition in the "
228
+ f"creation of the directory. Exception: {e}.",
229
+ CacheWarning,
230
+ )
231
+
232
+ def clear_item(self, call_id):
233
+ """Clear the item at the id, given as a list of str."""
234
+ item_path = os.path.join(self.location, *call_id)
235
+ if self._item_exists(item_path):
236
+ self.clear_location(item_path)
237
+
238
+ def contains_item(self, call_id):
239
+ """Check if there is an item at the id, given as a list of str."""
240
+ item_path = os.path.join(self.location, *call_id)
241
+ filename = os.path.join(item_path, "output.pkl")
242
+
243
+ return self._item_exists(filename)
244
+
245
+ def get_item_info(self, call_id):
246
+ """Return information about item."""
247
+ return {"location": os.path.join(self.location, *call_id)}
248
+
249
+ def get_metadata(self, call_id):
250
+ """Return actual metadata of an item."""
251
+ try:
252
+ item_path = os.path.join(self.location, *call_id)
253
+ filename = os.path.join(item_path, "metadata.json")
254
+ with self._open_item(filename, "rb") as f:
255
+ return json.loads(f.read().decode("utf-8"))
256
+ except: # noqa: E722
257
+ return {}
258
+
259
+ def store_metadata(self, call_id, metadata):
260
+ """Store metadata of a computation."""
261
+ try:
262
+ item_path = os.path.join(self.location, *call_id)
263
+ self.create_location(item_path)
264
+ filename = os.path.join(item_path, "metadata.json")
265
+
266
+ def write_func(to_write, dest_filename):
267
+ with self._open_item(dest_filename, "wb") as f:
268
+ f.write(json.dumps(to_write).encode("utf-8"))
269
+
270
+ self._concurrency_safe_write(metadata, filename, write_func)
271
+ except: # noqa: E722
272
+ pass
273
+
274
+ def contains_path(self, call_id):
275
+ """Check cached function is available in store."""
276
+ func_path = os.path.join(self.location, *call_id)
277
+ return self.object_exists(func_path)
278
+
279
+ def clear_path(self, call_id):
280
+ """Clear all items with a common path in the store."""
281
+ func_path = os.path.join(self.location, *call_id)
282
+ if self._item_exists(func_path):
283
+ self.clear_location(func_path)
284
+
285
+ def store_cached_func_code(self, call_id, func_code=None):
286
+ """Store the code of the cached function."""
287
+ func_path = os.path.join(self.location, *call_id)
288
+ if not self._item_exists(func_path):
289
+ self.create_location(func_path)
290
+
291
+ if func_code is not None:
292
+ filename = os.path.join(func_path, "func_code.py")
293
+ with self._open_item(filename, "wb") as f:
294
+ f.write(func_code.encode("utf-8"))
295
+
296
+ def get_cached_func_code(self, call_id):
297
+ """Store the code of the cached function."""
298
+ filename = os.path.join(self.location, *call_id, "func_code.py")
299
+ try:
300
+ with self._open_item(filename, "rb") as f:
301
+ return f.read().decode("utf-8")
302
+ except: # noqa: E722
303
+ raise
304
+
305
+ def get_cached_func_info(self, call_id):
306
+ """Return information related to the cached function if it exists."""
307
+ return {"location": os.path.join(self.location, *call_id)}
308
+
309
+ def clear(self):
310
+ """Clear the whole store content."""
311
+ self.clear_location(self.location)
312
+
313
+ def enforce_store_limits(self, bytes_limit, items_limit=None, age_limit=None):
314
+ """
315
+ Remove the store's oldest files to enforce item, byte, and age limits.
316
+ """
317
+ items_to_delete = self._get_items_to_delete(bytes_limit, items_limit, age_limit)
318
+
319
+ for item in items_to_delete:
320
+ if self.verbose > 10:
321
+ print("Deleting item {0}".format(item))
322
+ try:
323
+ self.clear_location(item.path)
324
+ except OSError:
325
+ # Even with ignore_errors=True shutil.rmtree can raise OSError
326
+ # with:
327
+ # [Errno 116] Stale file handle if another process has deleted
328
+ # the folder already.
329
+ pass
330
+
331
+ def _get_items_to_delete(self, bytes_limit, items_limit=None, age_limit=None):
332
+ """
333
+ Get items to delete to keep the store under size, file, & age limits.
334
+ """
335
+ if isinstance(bytes_limit, str):
336
+ bytes_limit = memstr_to_bytes(bytes_limit)
337
+
338
+ items = self.get_items()
339
+ if not items:
340
+ return []
341
+
342
+ size = sum(item.size for item in items)
343
+
344
+ if bytes_limit is not None:
345
+ to_delete_size = size - bytes_limit
346
+ else:
347
+ to_delete_size = 0
348
+
349
+ if items_limit is not None:
350
+ to_delete_items = len(items) - items_limit
351
+ else:
352
+ to_delete_items = 0
353
+
354
+ if age_limit is not None:
355
+ older_item = min(item.last_access for item in items)
356
+ if age_limit.total_seconds() < 0:
357
+ raise ValueError("age_limit has to be a positive timedelta")
358
+ deadline = datetime.datetime.now() - age_limit
359
+ else:
360
+ deadline = None
361
+
362
+ if (
363
+ to_delete_size <= 0
364
+ and to_delete_items <= 0
365
+ and (deadline is None or older_item > deadline)
366
+ ):
367
+ return []
368
+
369
+ # We want to delete first the cache items that were accessed a
370
+ # long time ago
371
+ items.sort(key=operator.attrgetter("last_access"))
372
+
373
+ items_to_delete = []
374
+ size_so_far = 0
375
+ items_so_far = 0
376
+
377
+ for item in items:
378
+ if (
379
+ (size_so_far >= to_delete_size)
380
+ and items_so_far >= to_delete_items
381
+ and (deadline is None or deadline < item.last_access)
382
+ ):
383
+ break
384
+
385
+ items_to_delete.append(item)
386
+ size_so_far += item.size
387
+ items_so_far += 1
388
+
389
+ return items_to_delete
390
+
391
+ def _concurrency_safe_write(self, to_write, filename, write_func):
392
+ """Writes an object into a file in a concurrency-safe way."""
393
+ temporary_filename = concurrency_safe_write(to_write, filename, write_func)
394
+ self._move_item(temporary_filename, filename)
395
+
396
+ def __repr__(self):
397
+ """Printable representation of the store location."""
398
+ return '{class_name}(location="{location}")'.format(
399
+ class_name=self.__class__.__name__, location=self.location
400
+ )
401
+
402
+
403
+ class FileSystemStoreBackend(StoreBackendBase, StoreBackendMixin):
404
+ """A StoreBackend used with local or network file systems."""
405
+
406
+ _open_item = staticmethod(open)
407
+ _item_exists = staticmethod(os.path.exists)
408
+ _move_item = staticmethod(concurrency_safe_rename)
409
+
410
+ def clear_location(self, location):
411
+ """Delete location on store."""
412
+ if location == self.location:
413
+ rm_subdirs(location)
414
+ else:
415
+ shutil.rmtree(location, ignore_errors=True)
416
+
417
+ def create_location(self, location):
418
+ """Create object location on store"""
419
+ mkdirp(location)
420
+
421
+ def get_items(self):
422
+ """Returns the whole list of items available in the store."""
423
+ items = []
424
+
425
+ for dirpath, _, filenames in os.walk(self.location):
426
+ is_cache_hash_dir = re.match("[a-f0-9]{32}", os.path.basename(dirpath))
427
+
428
+ if is_cache_hash_dir:
429
+ output_filename = os.path.join(dirpath, "output.pkl")
430
+ try:
431
+ last_access = os.path.getatime(output_filename)
432
+ except OSError:
433
+ try:
434
+ last_access = os.path.getatime(dirpath)
435
+ except OSError:
436
+ # The directory has already been deleted
437
+ continue
438
+
439
+ last_access = datetime.datetime.fromtimestamp(last_access)
440
+ try:
441
+ full_filenames = [os.path.join(dirpath, fn) for fn in filenames]
442
+ dirsize = sum(os.path.getsize(fn) for fn in full_filenames)
443
+ except OSError:
444
+ # Either output_filename or one of the files in
445
+ # dirpath does not exist any more. We assume this
446
+ # directory is being cleaned by another process already
447
+ continue
448
+
449
+ items.append(CacheItemInfo(dirpath, dirsize, last_access))
450
+
451
+ return items
452
+
453
+ def configure(self, location, verbose=1, backend_options=None):
454
+ """Configure the store backend.
455
+
456
+ For this backend, valid store options are 'compress' and 'mmap_mode'
457
+ """
458
+ if backend_options is None:
459
+ backend_options = {}
460
+
461
+ # setup location directory
462
+ self.location = location
463
+ if not os.path.exists(self.location):
464
+ mkdirp(self.location)
465
+
466
+ # Automatically add `.gitignore` file to the cache folder.
467
+ # XXX: the condition is necessary because in `Memory.__init__`, the user
468
+ # passed `location` param is modified to be either `{location}` or
469
+ # `{location}/joblib` depending on input type (`pathlib.Path` vs `str`).
470
+ # The proper resolution of this inconsistency is tracked in:
471
+ # https://github.com/joblib/joblib/issues/1684
472
+ cache_directory = (
473
+ os.path.dirname(location)
474
+ if os.path.dirname(location) and os.path.basename(location) == "joblib"
475
+ else location
476
+ )
477
+ gitignore = os.path.join(cache_directory, ".gitignore")
478
+ if not os.path.exists(gitignore):
479
+ try:
480
+ with open(gitignore, "w") as file:
481
+ file.write("# Created by joblib automatically.\n")
482
+ file.write("*\n")
483
+ except OSError as e:
484
+ warnings.warn(f"Unable to write {gitignore}. Exception: {e}.")
485
+
486
+ # item can be stored compressed for faster I/O
487
+ self.compress = backend_options.get("compress", False)
488
+
489
+ # FileSystemStoreBackend can be used with mmap_mode options under
490
+ # certain conditions.
491
+ mmap_mode = backend_options.get("mmap_mode")
492
+ if self.compress and mmap_mode is not None:
493
+ warnings.warn(
494
+ "Compressed items cannot be memmapped in a "
495
+ "filesystem store. Option will be ignored.",
496
+ stacklevel=2,
497
+ )
498
+
499
+ self.mmap_mode = mmap_mode
500
+ self.verbose = verbose
TTS/Kokoro/.venv/Lib/site-packages/joblib/_utils.py ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Adapted from https://stackoverflow.com/a/9558001/2536294
2
+
3
+ import ast
4
+ import functools
5
+ import operator as op
6
+ from dataclasses import dataclass
7
+
8
+ from ._multiprocessing_helpers import mp
9
+
10
+ if mp is not None:
11
+ from .externals.loky.process_executor import _ExceptionWithTraceback
12
+
13
+
14
+ # supported operators
15
+ operators = {
16
+ ast.Add: op.add,
17
+ ast.Sub: op.sub,
18
+ ast.Mult: op.mul,
19
+ ast.Div: op.truediv,
20
+ ast.FloorDiv: op.floordiv,
21
+ ast.Mod: op.mod,
22
+ ast.Pow: op.pow,
23
+ ast.USub: op.neg,
24
+ }
25
+
26
+
27
+ def eval_expr(expr):
28
+ """Somewhat safely evaluate an arithmetic expression.
29
+
30
+ >>> eval_expr('2*6')
31
+ 12
32
+ >>> eval_expr('2**6')
33
+ 64
34
+ >>> eval_expr('1 + 2*3**(4) / (6 + -7)')
35
+ -161.0
36
+
37
+ Raises ValueError if the expression is invalid, too long
38
+ or its computation involves too large values.
39
+ """
40
+ # Restrict the length of the expression to avoid potential Python crashes
41
+ # as per the documentation of ast.parse.
42
+ max_length = 30
43
+ if len(expr) > max_length:
44
+ raise ValueError(
45
+ f"Expression {expr[:max_length]!r}... is too long. "
46
+ f"Max length is {max_length}, got {len(expr)}."
47
+ )
48
+ try:
49
+ return eval_(ast.parse(expr, mode="eval").body)
50
+ except (TypeError, SyntaxError, OverflowError, KeyError) as e:
51
+ raise ValueError(
52
+ f"{expr!r} is not a valid or supported arithmetic expression."
53
+ ) from e
54
+
55
+
56
+ def limit(max_=None):
57
+ """Return decorator that limits allowed returned values."""
58
+
59
+ def decorator(func):
60
+ @functools.wraps(func)
61
+ def wrapper(*args, **kwargs):
62
+ ret = func(*args, **kwargs)
63
+ try:
64
+ mag = abs(ret)
65
+ except TypeError:
66
+ pass # not applicable
67
+ else:
68
+ if mag > max_:
69
+ raise ValueError(
70
+ f"Numeric literal {ret} is too large, max is {max_}."
71
+ )
72
+ return ret
73
+
74
+ return wrapper
75
+
76
+ return decorator
77
+
78
+
79
+ @limit(max_=10**6)
80
+ def eval_(node):
81
+ if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
82
+ return node.value
83
+ elif isinstance(node, ast.BinOp): # <left> <operator> <right>
84
+ return operators[type(node.op)](eval_(node.left), eval_(node.right))
85
+ elif isinstance(node, ast.UnaryOp): # <operator> <operand> e.g., -1
86
+ return operators[type(node.op)](eval_(node.operand))
87
+ else:
88
+ raise TypeError(node)
89
+
90
+
91
+ @dataclass(frozen=True)
92
+ class _Sentinel:
93
+ """A sentinel to mark a parameter as not explicitly set"""
94
+
95
+ default_value: object
96
+
97
+ def __repr__(self):
98
+ return f"default({self.default_value!r})"
99
+
100
+
101
+ class _TracebackCapturingWrapper:
102
+ """Protect function call and return error with traceback."""
103
+
104
+ def __init__(self, func):
105
+ self.func = func
106
+
107
+ def __call__(self, **kwargs):
108
+ try:
109
+ return self.func(**kwargs)
110
+ except BaseException as e:
111
+ return _ExceptionWithTraceback(e)
112
+
113
+
114
+ def _retrieve_traceback_capturing_wrapped_call(out):
115
+ if isinstance(out, _ExceptionWithTraceback):
116
+ rebuild, args = out.__reduce__()
117
+ out = rebuild(*args)
118
+ if isinstance(out, BaseException):
119
+ raise out
120
+ return out
TTS/Kokoro/.venv/Lib/site-packages/joblib/backports.py ADDED
@@ -0,0 +1,195 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Backports of fixes for joblib dependencies
3
+ """
4
+
5
+ import os
6
+ import re
7
+ import time
8
+ from multiprocessing import util
9
+ from os.path import basename
10
+
11
+
12
+ class Version:
13
+ """Backport from deprecated distutils
14
+
15
+ We maintain this backport to avoid introducing a new dependency on
16
+ `packaging`.
17
+
18
+ We might rexplore this choice in the future if all major Python projects
19
+ introduce a dependency on packaging anyway.
20
+ """
21
+
22
+ def __init__(self, vstring=None):
23
+ if vstring:
24
+ self.parse(vstring)
25
+
26
+ def __repr__(self):
27
+ return "%s ('%s')" % (self.__class__.__name__, str(self))
28
+
29
+ def __eq__(self, other):
30
+ c = self._cmp(other)
31
+ if c is NotImplemented:
32
+ return c
33
+ return c == 0
34
+
35
+ def __lt__(self, other):
36
+ c = self._cmp(other)
37
+ if c is NotImplemented:
38
+ return c
39
+ return c < 0
40
+
41
+ def __le__(self, other):
42
+ c = self._cmp(other)
43
+ if c is NotImplemented:
44
+ return c
45
+ return c <= 0
46
+
47
+ def __gt__(self, other):
48
+ c = self._cmp(other)
49
+ if c is NotImplemented:
50
+ return c
51
+ return c > 0
52
+
53
+ def __ge__(self, other):
54
+ c = self._cmp(other)
55
+ if c is NotImplemented:
56
+ return c
57
+ return c >= 0
58
+
59
+
60
+ class LooseVersion(Version):
61
+ """Backport from deprecated distutils
62
+
63
+ We maintain this backport to avoid introducing a new dependency on
64
+ `packaging`.
65
+
66
+ We might rexplore this choice in the future if all major Python projects
67
+ introduce a dependency on packaging anyway.
68
+ """
69
+
70
+ component_re = re.compile(r"(\d+ | [a-z]+ | \.)", re.VERBOSE)
71
+
72
+ def __init__(self, vstring=None):
73
+ if vstring:
74
+ self.parse(vstring)
75
+
76
+ def parse(self, vstring):
77
+ # I've given up on thinking I can reconstruct the version string
78
+ # from the parsed tuple -- so I just store the string here for
79
+ # use by __str__
80
+ self.vstring = vstring
81
+ components = [x for x in self.component_re.split(vstring) if x and x != "."]
82
+ for i, obj in enumerate(components):
83
+ try:
84
+ components[i] = int(obj)
85
+ except ValueError:
86
+ pass
87
+
88
+ self.version = components
89
+
90
+ def __str__(self):
91
+ return self.vstring
92
+
93
+ def __repr__(self):
94
+ return "LooseVersion ('%s')" % str(self)
95
+
96
+ def _cmp(self, other):
97
+ if isinstance(other, str):
98
+ other = LooseVersion(other)
99
+ elif not isinstance(other, LooseVersion):
100
+ return NotImplemented
101
+
102
+ if self.version == other.version:
103
+ return 0
104
+ if self.version < other.version:
105
+ return -1
106
+ if self.version > other.version:
107
+ return 1
108
+
109
+
110
+ try:
111
+ import numpy as np
112
+
113
+ def make_memmap(
114
+ filename,
115
+ dtype="uint8",
116
+ mode="r+",
117
+ offset=0,
118
+ shape=None,
119
+ order="C",
120
+ unlink_on_gc_collect=False,
121
+ ):
122
+ """Custom memmap constructor compatible with numpy.memmap.
123
+
124
+ This function:
125
+ - is a backport the numpy memmap offset fix (See
126
+ https://github.com/numpy/numpy/pull/8443 for more details.
127
+ The numpy fix is available starting numpy 1.13)
128
+ - adds ``unlink_on_gc_collect``, which specifies explicitly whether
129
+ the process re-constructing the memmap owns a reference to the
130
+ underlying file. If set to True, it adds a finalizer to the
131
+ newly-created memmap that sends a maybe_unlink request for the
132
+ memmaped file to resource_tracker.
133
+ """
134
+ util.debug(
135
+ "[MEMMAP READ] creating a memmap (shape {}, filename {}, pid {})".format(
136
+ shape, basename(filename), os.getpid()
137
+ )
138
+ )
139
+
140
+ mm = np.memmap(
141
+ filename, dtype=dtype, mode=mode, offset=offset, shape=shape, order=order
142
+ )
143
+ if LooseVersion(np.__version__) < "1.13":
144
+ mm.offset = offset
145
+ if unlink_on_gc_collect:
146
+ from ._memmapping_reducer import add_maybe_unlink_finalizer
147
+
148
+ add_maybe_unlink_finalizer(mm)
149
+ return mm
150
+ except ImportError:
151
+
152
+ def make_memmap(
153
+ filename,
154
+ dtype="uint8",
155
+ mode="r+",
156
+ offset=0,
157
+ shape=None,
158
+ order="C",
159
+ unlink_on_gc_collect=False,
160
+ ):
161
+ raise NotImplementedError(
162
+ "'joblib.backports.make_memmap' should not be used "
163
+ "if numpy is not installed."
164
+ )
165
+
166
+
167
+ if os.name == "nt":
168
+ # https://github.com/joblib/joblib/issues/540
169
+ access_denied_errors = (5, 13)
170
+ from os import replace
171
+
172
+ def concurrency_safe_rename(src, dst):
173
+ """Renames ``src`` into ``dst`` overwriting ``dst`` if it exists.
174
+
175
+ On Windows os.replace can yield permission errors if executed by two
176
+ different processes.
177
+ """
178
+ max_sleep_time = 1
179
+ total_sleep_time = 0
180
+ sleep_time = 0.001
181
+ while total_sleep_time < max_sleep_time:
182
+ try:
183
+ replace(src, dst)
184
+ break
185
+ except Exception as exc:
186
+ if getattr(exc, "winerror", None) in access_denied_errors:
187
+ time.sleep(sleep_time)
188
+ total_sleep_time += sleep_time
189
+ sleep_time *= 2
190
+ else:
191
+ raise
192
+ else:
193
+ raise
194
+ else:
195
+ from os import replace as concurrency_safe_rename # noqa
TTS/Kokoro/.venv/Lib/site-packages/joblib/compressor.py ADDED
@@ -0,0 +1,572 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Classes and functions for managing compressors."""
2
+
3
+ import io
4
+ import zlib
5
+
6
+ from joblib.backports import LooseVersion
7
+
8
+ try:
9
+ from threading import RLock
10
+ except ImportError:
11
+ from dummy_threading import RLock
12
+
13
+ try:
14
+ import bz2
15
+ except ImportError:
16
+ bz2 = None
17
+
18
+ try:
19
+ import lz4
20
+ from lz4.frame import LZ4FrameFile
21
+ except ImportError:
22
+ lz4 = None
23
+
24
+ try:
25
+ import lzma
26
+ except ImportError:
27
+ lzma = None
28
+
29
+
30
+ LZ4_NOT_INSTALLED_ERROR = (
31
+ "LZ4 is not installed. Install it with pip: https://python-lz4.readthedocs.io/"
32
+ )
33
+
34
+ # Registered compressors
35
+ _COMPRESSORS = {}
36
+
37
+ # Magic numbers of supported compression file formats.
38
+ _ZFILE_PREFIX = b"ZF" # used with pickle files created before 0.9.3.
39
+ _ZLIB_PREFIX = b"\x78"
40
+ _GZIP_PREFIX = b"\x1f\x8b"
41
+ _BZ2_PREFIX = b"BZ"
42
+ _XZ_PREFIX = b"\xfd\x37\x7a\x58\x5a"
43
+ _LZMA_PREFIX = b"\x5d\x00"
44
+ _LZ4_PREFIX = b"\x04\x22\x4d\x18"
45
+
46
+
47
+ def register_compressor(compressor_name, compressor, force=False):
48
+ """Register a new compressor.
49
+
50
+ Parameters
51
+ ----------
52
+ compressor_name: str.
53
+ The name of the compressor.
54
+ compressor: CompressorWrapper
55
+ An instance of a 'CompressorWrapper'.
56
+ """
57
+ global _COMPRESSORS
58
+ if not isinstance(compressor_name, str):
59
+ raise ValueError(
60
+ "Compressor name should be a string, '{}' given.".format(compressor_name)
61
+ )
62
+
63
+ if not isinstance(compressor, CompressorWrapper):
64
+ raise ValueError(
65
+ "Compressor should implement the CompressorWrapper "
66
+ "interface, '{}' given.".format(compressor)
67
+ )
68
+
69
+ if compressor.fileobj_factory is not None and (
70
+ not hasattr(compressor.fileobj_factory, "read")
71
+ or not hasattr(compressor.fileobj_factory, "write")
72
+ or not hasattr(compressor.fileobj_factory, "seek")
73
+ or not hasattr(compressor.fileobj_factory, "tell")
74
+ ):
75
+ raise ValueError(
76
+ "Compressor 'fileobj_factory' attribute should "
77
+ "implement the file object interface, '{}' given.".format(
78
+ compressor.fileobj_factory
79
+ )
80
+ )
81
+
82
+ if compressor_name in _COMPRESSORS and not force:
83
+ raise ValueError("Compressor '{}' already registered.".format(compressor_name))
84
+
85
+ _COMPRESSORS[compressor_name] = compressor
86
+
87
+
88
+ class CompressorWrapper:
89
+ """A wrapper around a compressor file object.
90
+
91
+ Attributes
92
+ ----------
93
+ obj: a file-like object
94
+ The object must implement the buffer interface and will be used
95
+ internally to compress/decompress the data.
96
+ prefix: bytestring
97
+ A bytestring corresponding to the magic number that identifies the
98
+ file format associated to the compressor.
99
+ extension: str
100
+ The file extension used to automatically select this compressor during
101
+ a dump to a file.
102
+ """
103
+
104
+ def __init__(self, obj, prefix=b"", extension=""):
105
+ self.fileobj_factory = obj
106
+ self.prefix = prefix
107
+ self.extension = extension
108
+
109
+ def compressor_file(self, fileobj, compresslevel=None):
110
+ """Returns an instance of a compressor file object."""
111
+ if compresslevel is None:
112
+ return self.fileobj_factory(fileobj, "wb")
113
+ else:
114
+ return self.fileobj_factory(fileobj, "wb", compresslevel=compresslevel)
115
+
116
+ def decompressor_file(self, fileobj):
117
+ """Returns an instance of a decompressor file object."""
118
+ return self.fileobj_factory(fileobj, "rb")
119
+
120
+
121
+ class BZ2CompressorWrapper(CompressorWrapper):
122
+ prefix = _BZ2_PREFIX
123
+ extension = ".bz2"
124
+
125
+ def __init__(self):
126
+ if bz2 is not None:
127
+ self.fileobj_factory = bz2.BZ2File
128
+ else:
129
+ self.fileobj_factory = None
130
+
131
+ def _check_versions(self):
132
+ if bz2 is None:
133
+ raise ValueError(
134
+ "bz2 module is not compiled on your python standard library."
135
+ )
136
+
137
+ def compressor_file(self, fileobj, compresslevel=None):
138
+ """Returns an instance of a compressor file object."""
139
+ self._check_versions()
140
+ if compresslevel is None:
141
+ return self.fileobj_factory(fileobj, "wb")
142
+ else:
143
+ return self.fileobj_factory(fileobj, "wb", compresslevel=compresslevel)
144
+
145
+ def decompressor_file(self, fileobj):
146
+ """Returns an instance of a decompressor file object."""
147
+ self._check_versions()
148
+ fileobj = self.fileobj_factory(fileobj, "rb")
149
+ return fileobj
150
+
151
+
152
+ class LZMACompressorWrapper(CompressorWrapper):
153
+ prefix = _LZMA_PREFIX
154
+ extension = ".lzma"
155
+ _lzma_format_name = "FORMAT_ALONE"
156
+
157
+ def __init__(self):
158
+ if lzma is not None:
159
+ self.fileobj_factory = lzma.LZMAFile
160
+ self._lzma_format = getattr(lzma, self._lzma_format_name)
161
+ else:
162
+ self.fileobj_factory = None
163
+
164
+ def _check_versions(self):
165
+ if lzma is None:
166
+ raise ValueError(
167
+ "lzma module is not compiled on your python standard library."
168
+ )
169
+
170
+ def compressor_file(self, fileobj, compresslevel=None):
171
+ """Returns an instance of a compressor file object."""
172
+ if compresslevel is None:
173
+ return self.fileobj_factory(fileobj, "wb", format=self._lzma_format)
174
+ else:
175
+ return self.fileobj_factory(
176
+ fileobj, "wb", format=self._lzma_format, preset=compresslevel
177
+ )
178
+
179
+ def decompressor_file(self, fileobj):
180
+ """Returns an instance of a decompressor file object."""
181
+ return lzma.LZMAFile(fileobj, "rb")
182
+
183
+
184
+ class XZCompressorWrapper(LZMACompressorWrapper):
185
+ prefix = _XZ_PREFIX
186
+ extension = ".xz"
187
+ _lzma_format_name = "FORMAT_XZ"
188
+
189
+
190
+ class LZ4CompressorWrapper(CompressorWrapper):
191
+ prefix = _LZ4_PREFIX
192
+ extension = ".lz4"
193
+
194
+ def __init__(self):
195
+ if lz4 is not None:
196
+ self.fileobj_factory = LZ4FrameFile
197
+ else:
198
+ self.fileobj_factory = None
199
+
200
+ def _check_versions(self):
201
+ if lz4 is None:
202
+ raise ValueError(LZ4_NOT_INSTALLED_ERROR)
203
+ lz4_version = lz4.__version__
204
+ if lz4_version.startswith("v"):
205
+ lz4_version = lz4_version[1:]
206
+ if LooseVersion(lz4_version) < LooseVersion("0.19"):
207
+ raise ValueError(LZ4_NOT_INSTALLED_ERROR)
208
+
209
+ def compressor_file(self, fileobj, compresslevel=None):
210
+ """Returns an instance of a compressor file object."""
211
+ self._check_versions()
212
+ if compresslevel is None:
213
+ return self.fileobj_factory(fileobj, "wb")
214
+ else:
215
+ return self.fileobj_factory(fileobj, "wb", compression_level=compresslevel)
216
+
217
+ def decompressor_file(self, fileobj):
218
+ """Returns an instance of a decompressor file object."""
219
+ self._check_versions()
220
+ return self.fileobj_factory(fileobj, "rb")
221
+
222
+
223
+ ###############################################################################
224
+ # base file compression/decompression object definition
225
+ _MODE_CLOSED = 0
226
+ _MODE_READ = 1
227
+ _MODE_READ_EOF = 2
228
+ _MODE_WRITE = 3
229
+ _BUFFER_SIZE = 8192
230
+
231
+
232
+ class BinaryZlibFile(io.BufferedIOBase):
233
+ """A file object providing transparent zlib (de)compression.
234
+
235
+ TODO python2_drop: is it still needed since we dropped Python 2 support A
236
+ BinaryZlibFile can act as a wrapper for an existing file object, or refer
237
+ directly to a named file on disk.
238
+
239
+ Note that BinaryZlibFile provides only a *binary* file interface: data read
240
+ is returned as bytes, and data to be written should be given as bytes.
241
+
242
+ This object is an adaptation of the BZ2File object and is compatible with
243
+ versions of python >= 2.7.
244
+
245
+ If filename is a str or bytes object, it gives the name
246
+ of the file to be opened. Otherwise, it should be a file object,
247
+ which will be used to read or write the compressed data.
248
+
249
+ mode can be 'rb' for reading (default) or 'wb' for (over)writing
250
+
251
+ If mode is 'wb', compresslevel can be a number between 1
252
+ and 9 specifying the level of compression: 1 produces the least
253
+ compression, and 9 produces the most compression. 3 is the default.
254
+ """
255
+
256
+ wbits = zlib.MAX_WBITS
257
+
258
+ def __init__(self, filename, mode="rb", compresslevel=3):
259
+ # This lock must be recursive, so that BufferedIOBase's
260
+ # readline(), readlines() and writelines() don't deadlock.
261
+ self._lock = RLock()
262
+ self._fp = None
263
+ self._closefp = False
264
+ self._mode = _MODE_CLOSED
265
+ self._pos = 0
266
+ self._size = -1
267
+ self.compresslevel = compresslevel
268
+
269
+ if not isinstance(compresslevel, int) or not (1 <= compresslevel <= 9):
270
+ raise ValueError(
271
+ "'compresslevel' must be an integer "
272
+ "between 1 and 9. You provided 'compresslevel={}'".format(compresslevel)
273
+ )
274
+
275
+ if mode == "rb":
276
+ self._mode = _MODE_READ
277
+ self._decompressor = zlib.decompressobj(self.wbits)
278
+ self._buffer = b""
279
+ self._buffer_offset = 0
280
+ elif mode == "wb":
281
+ self._mode = _MODE_WRITE
282
+ self._compressor = zlib.compressobj(
283
+ self.compresslevel, zlib.DEFLATED, self.wbits, zlib.DEF_MEM_LEVEL, 0
284
+ )
285
+ else:
286
+ raise ValueError("Invalid mode: %r" % (mode,))
287
+
288
+ if isinstance(filename, str):
289
+ self._fp = io.open(filename, mode)
290
+ self._closefp = True
291
+ elif hasattr(filename, "read") or hasattr(filename, "write"):
292
+ self._fp = filename
293
+ else:
294
+ raise TypeError("filename must be a str or bytes object, or a file")
295
+
296
+ def close(self):
297
+ """Flush and close the file.
298
+
299
+ May be called more than once without error. Once the file is
300
+ closed, any other operation on it will raise a ValueError.
301
+ """
302
+ with self._lock:
303
+ if self._mode == _MODE_CLOSED:
304
+ return
305
+ try:
306
+ if self._mode in (_MODE_READ, _MODE_READ_EOF):
307
+ self._decompressor = None
308
+ elif self._mode == _MODE_WRITE:
309
+ self._fp.write(self._compressor.flush())
310
+ self._compressor = None
311
+ finally:
312
+ try:
313
+ if self._closefp:
314
+ self._fp.close()
315
+ finally:
316
+ self._fp = None
317
+ self._closefp = False
318
+ self._mode = _MODE_CLOSED
319
+ self._buffer = b""
320
+ self._buffer_offset = 0
321
+
322
+ @property
323
+ def closed(self):
324
+ """True if this file is closed."""
325
+ return self._mode == _MODE_CLOSED
326
+
327
+ def fileno(self):
328
+ """Return the file descriptor for the underlying file."""
329
+ self._check_not_closed()
330
+ return self._fp.fileno()
331
+
332
+ def seekable(self):
333
+ """Return whether the file supports seeking."""
334
+ return self.readable() and self._fp.seekable()
335
+
336
+ def readable(self):
337
+ """Return whether the file was opened for reading."""
338
+ self._check_not_closed()
339
+ return self._mode in (_MODE_READ, _MODE_READ_EOF)
340
+
341
+ def writable(self):
342
+ """Return whether the file was opened for writing."""
343
+ self._check_not_closed()
344
+ return self._mode == _MODE_WRITE
345
+
346
+ # Mode-checking helper functions.
347
+
348
+ def _check_not_closed(self):
349
+ if self.closed:
350
+ fname = getattr(self._fp, "name", None)
351
+ msg = "I/O operation on closed file"
352
+ if fname is not None:
353
+ msg += " {}".format(fname)
354
+ msg += "."
355
+ raise ValueError(msg)
356
+
357
+ def _check_can_read(self):
358
+ if self._mode not in (_MODE_READ, _MODE_READ_EOF):
359
+ self._check_not_closed()
360
+ raise io.UnsupportedOperation("File not open for reading")
361
+
362
+ def _check_can_write(self):
363
+ if self._mode != _MODE_WRITE:
364
+ self._check_not_closed()
365
+ raise io.UnsupportedOperation("File not open for writing")
366
+
367
+ def _check_can_seek(self):
368
+ if self._mode not in (_MODE_READ, _MODE_READ_EOF):
369
+ self._check_not_closed()
370
+ raise io.UnsupportedOperation(
371
+ "Seeking is only supported on files open for reading"
372
+ )
373
+ if not self._fp.seekable():
374
+ raise io.UnsupportedOperation(
375
+ "The underlying file object does not support seeking"
376
+ )
377
+
378
+ # Fill the readahead buffer if it is empty. Returns False on EOF.
379
+ def _fill_buffer(self):
380
+ if self._mode == _MODE_READ_EOF:
381
+ return False
382
+ # Depending on the input data, our call to the decompressor may not
383
+ # return any data. In this case, try again after reading another block.
384
+ while self._buffer_offset == len(self._buffer):
385
+ try:
386
+ rawblock = self._decompressor.unused_data or self._fp.read(_BUFFER_SIZE)
387
+ if not rawblock:
388
+ raise EOFError
389
+ except EOFError:
390
+ # End-of-stream marker and end of file. We're good.
391
+ self._mode = _MODE_READ_EOF
392
+ self._size = self._pos
393
+ return False
394
+ else:
395
+ self._buffer = self._decompressor.decompress(rawblock)
396
+ self._buffer_offset = 0
397
+ return True
398
+
399
+ # Read data until EOF.
400
+ # If return_data is false, consume the data without returning it.
401
+ def _read_all(self, return_data=True):
402
+ # The loop assumes that _buffer_offset is 0. Ensure that this is true.
403
+ self._buffer = self._buffer[self._buffer_offset :]
404
+ self._buffer_offset = 0
405
+
406
+ blocks = []
407
+ while self._fill_buffer():
408
+ if return_data:
409
+ blocks.append(self._buffer)
410
+ self._pos += len(self._buffer)
411
+ self._buffer = b""
412
+ if return_data:
413
+ return b"".join(blocks)
414
+
415
+ # Read a block of up to n bytes.
416
+ # If return_data is false, consume the data without returning it.
417
+ def _read_block(self, n_bytes, return_data=True):
418
+ # If we have enough data buffered, return immediately.
419
+ end = self._buffer_offset + n_bytes
420
+ if end <= len(self._buffer):
421
+ data = self._buffer[self._buffer_offset : end]
422
+ self._buffer_offset = end
423
+ self._pos += len(data)
424
+ return data if return_data else None
425
+
426
+ # The loop assumes that _buffer_offset is 0. Ensure that this is true.
427
+ self._buffer = self._buffer[self._buffer_offset :]
428
+ self._buffer_offset = 0
429
+
430
+ blocks = []
431
+ while n_bytes > 0 and self._fill_buffer():
432
+ if n_bytes < len(self._buffer):
433
+ data = self._buffer[:n_bytes]
434
+ self._buffer_offset = n_bytes
435
+ else:
436
+ data = self._buffer
437
+ self._buffer = b""
438
+ if return_data:
439
+ blocks.append(data)
440
+ self._pos += len(data)
441
+ n_bytes -= len(data)
442
+ if return_data:
443
+ return b"".join(blocks)
444
+
445
+ def read(self, size=-1):
446
+ """Read up to size uncompressed bytes from the file.
447
+
448
+ If size is negative or omitted, read until EOF is reached.
449
+ Returns b'' if the file is already at EOF.
450
+ """
451
+ with self._lock:
452
+ self._check_can_read()
453
+ if size == 0:
454
+ return b""
455
+ elif size < 0:
456
+ return self._read_all()
457
+ else:
458
+ return self._read_block(size)
459
+
460
+ def readinto(self, b):
461
+ """Read up to len(b) bytes into b.
462
+
463
+ Returns the number of bytes read (0 for EOF).
464
+ """
465
+ with self._lock:
466
+ return io.BufferedIOBase.readinto(self, b)
467
+
468
+ def write(self, data):
469
+ """Write a byte string to the file.
470
+
471
+ Returns the number of uncompressed bytes written, which is
472
+ always len(data). Note that due to buffering, the file on disk
473
+ may not reflect the data written until close() is called.
474
+ """
475
+ with self._lock:
476
+ self._check_can_write()
477
+ # Convert data type if called by io.BufferedWriter.
478
+ if isinstance(data, memoryview):
479
+ data = data.tobytes()
480
+
481
+ compressed = self._compressor.compress(data)
482
+ self._fp.write(compressed)
483
+ self._pos += len(data)
484
+ return len(data)
485
+
486
+ # Rewind the file to the beginning of the data stream.
487
+ def _rewind(self):
488
+ self._fp.seek(0, 0)
489
+ self._mode = _MODE_READ
490
+ self._pos = 0
491
+ self._decompressor = zlib.decompressobj(self.wbits)
492
+ self._buffer = b""
493
+ self._buffer_offset = 0
494
+
495
+ def seek(self, offset, whence=0):
496
+ """Change the file position.
497
+
498
+ The new position is specified by offset, relative to the
499
+ position indicated by whence. Values for whence are:
500
+
501
+ 0: start of stream (default); offset must not be negative
502
+ 1: current stream position
503
+ 2: end of stream; offset must not be positive
504
+
505
+ Returns the new file position.
506
+
507
+ Note that seeking is emulated, so depending on the parameters,
508
+ this operation may be extremely slow.
509
+ """
510
+ with self._lock:
511
+ self._check_can_seek()
512
+
513
+ # Recalculate offset as an absolute file position.
514
+ if whence == 0:
515
+ pass
516
+ elif whence == 1:
517
+ offset = self._pos + offset
518
+ elif whence == 2:
519
+ # Seeking relative to EOF - we need to know the file's size.
520
+ if self._size < 0:
521
+ self._read_all(return_data=False)
522
+ offset = self._size + offset
523
+ else:
524
+ raise ValueError("Invalid value for whence: %s" % (whence,))
525
+
526
+ # Make it so that offset is the number of bytes to skip forward.
527
+ if offset < self._pos:
528
+ self._rewind()
529
+ else:
530
+ offset -= self._pos
531
+
532
+ # Read and discard data until we reach the desired position.
533
+ self._read_block(offset, return_data=False)
534
+
535
+ return self._pos
536
+
537
+ def tell(self):
538
+ """Return the current file position."""
539
+ with self._lock:
540
+ self._check_not_closed()
541
+ return self._pos
542
+
543
+
544
+ class ZlibCompressorWrapper(CompressorWrapper):
545
+ def __init__(self):
546
+ CompressorWrapper.__init__(
547
+ self, obj=BinaryZlibFile, prefix=_ZLIB_PREFIX, extension=".z"
548
+ )
549
+
550
+
551
+ class BinaryGzipFile(BinaryZlibFile):
552
+ """A file object providing transparent gzip (de)compression.
553
+
554
+ If filename is a str or bytes object, it gives the name
555
+ of the file to be opened. Otherwise, it should be a file object,
556
+ which will be used to read or write the compressed data.
557
+
558
+ mode can be 'rb' for reading (default) or 'wb' for (over)writing
559
+
560
+ If mode is 'wb', compresslevel can be a number between 1
561
+ and 9 specifying the level of compression: 1 produces the least
562
+ compression, and 9 produces the most compression. 3 is the default.
563
+ """
564
+
565
+ wbits = 31 # zlib compressor/decompressor wbits value for gzip format.
566
+
567
+
568
+ class GzipCompressorWrapper(CompressorWrapper):
569
+ def __init__(self):
570
+ CompressorWrapper.__init__(
571
+ self, obj=BinaryGzipFile, prefix=_GZIP_PREFIX, extension=".gz"
572
+ )
TTS/Kokoro/.venv/Lib/site-packages/joblib/disk.py ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Disk management utilities.
3
+ """
4
+
5
+ # Authors: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
6
+ # Lars Buitinck
7
+ # Copyright (c) 2010 Gael Varoquaux
8
+ # License: BSD Style, 3 clauses.
9
+
10
+ import errno
11
+ import os
12
+ import shutil
13
+ import sys
14
+ import time
15
+ from multiprocessing import util
16
+
17
+ try:
18
+ WindowsError
19
+ except NameError:
20
+ WindowsError = OSError
21
+
22
+
23
+ def disk_used(path):
24
+ """Return the disk usage in a directory."""
25
+ size = 0
26
+ for file in os.listdir(path) + ["."]:
27
+ stat = os.stat(os.path.join(path, file))
28
+ if hasattr(stat, "st_blocks"):
29
+ size += stat.st_blocks * 512
30
+ else:
31
+ # on some platform st_blocks is not available (e.g., Windows)
32
+ # approximate by rounding to next multiple of 512
33
+ size += (stat.st_size // 512 + 1) * 512
34
+ # We need to convert to int to avoid having longs on some systems (we
35
+ # don't want longs to avoid problems we SQLite)
36
+ return int(size / 1024.0)
37
+
38
+
39
+ def memstr_to_bytes(text):
40
+ """Convert a memory text to its value in bytes."""
41
+ kilo = 1024
42
+ units = dict(K=kilo, M=kilo**2, G=kilo**3)
43
+ try:
44
+ size = int(units[text[-1]] * float(text[:-1]))
45
+ except (KeyError, ValueError) as e:
46
+ raise ValueError(
47
+ "Invalid literal for size give: %s (type %s) should be "
48
+ "alike '10G', '500M', '50K'." % (text, type(text))
49
+ ) from e
50
+ return size
51
+
52
+
53
+ def mkdirp(d):
54
+ """Ensure directory d exists (like mkdir -p on Unix)
55
+ No guarantee that the directory is writable.
56
+ """
57
+ try:
58
+ os.makedirs(d)
59
+ except OSError as e:
60
+ if e.errno != errno.EEXIST:
61
+ raise
62
+
63
+
64
+ # if a rmtree operation fails in rm_subdirs, wait for this much time (in secs),
65
+ # then retry up to RM_SUBDIRS_N_RETRY times. If it still fails, raise the
66
+ # exception. this mechanism ensures that the sub-process gc have the time to
67
+ # collect and close the memmaps before we fail.
68
+ RM_SUBDIRS_RETRY_TIME = 0.1
69
+ RM_SUBDIRS_N_RETRY = 10
70
+
71
+
72
+ def rm_subdirs(path, onerror=None):
73
+ """Remove all subdirectories in this path.
74
+
75
+ The directory indicated by `path` is left in place, and its subdirectories
76
+ are erased.
77
+
78
+ If onerror is set, it is called to handle the error with arguments (func,
79
+ path, exc_info) where func is os.listdir, os.remove, or os.rmdir;
80
+ path is the argument to that function that caused it to fail; and
81
+ exc_info is a tuple returned by sys.exc_info(). If onerror is None,
82
+ an exception is raised.
83
+ """
84
+
85
+ # NOTE this code is adapted from the one in shutil.rmtree, and is
86
+ # just as fast
87
+
88
+ names = []
89
+ try:
90
+ names = os.listdir(path)
91
+ except os.error:
92
+ if onerror is not None:
93
+ onerror(os.listdir, path, sys.exc_info())
94
+ else:
95
+ raise
96
+
97
+ for name in names:
98
+ fullname = os.path.join(path, name)
99
+ delete_folder(fullname, onerror=onerror)
100
+
101
+
102
+ def delete_folder(folder_path, onerror=None, allow_non_empty=True):
103
+ """Utility function to cleanup a temporary folder if it still exists."""
104
+ if os.path.isdir(folder_path):
105
+ if onerror is not None:
106
+ shutil.rmtree(folder_path, False, onerror)
107
+ else:
108
+ # allow the rmtree to fail once, wait and re-try.
109
+ # if the error is raised again, fail
110
+ err_count = 0
111
+ while True:
112
+ files = os.listdir(folder_path)
113
+ try:
114
+ if len(files) == 0 or allow_non_empty:
115
+ shutil.rmtree(folder_path, ignore_errors=False, onerror=None)
116
+ util.debug("Successfully deleted {}".format(folder_path))
117
+ break
118
+ else:
119
+ raise OSError(
120
+ "Expected empty folder {} but got {} files.".format(
121
+ folder_path, len(files)
122
+ )
123
+ )
124
+ except (OSError, WindowsError):
125
+ err_count += 1
126
+ if err_count > RM_SUBDIRS_N_RETRY:
127
+ # the folder cannot be deleted right now. It maybe
128
+ # because some temporary files have not been deleted
129
+ # yet.
130
+ raise
131
+ time.sleep(RM_SUBDIRS_RETRY_TIME)
TTS/Kokoro/.venv/Lib/site-packages/joblib/executor.py ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Utility function to construct a loky.ReusableExecutor with custom pickler.
2
+
3
+ This module provides efficient ways of working with data stored in
4
+ shared memory with numpy.memmap arrays without inducing any memory
5
+ copy between the parent and child processes.
6
+ """
7
+ # Author: Thomas Moreau <thomas.moreau.2010@gmail.com>
8
+ # Copyright: 2017, Thomas Moreau
9
+ # License: BSD 3 clause
10
+
11
+ from ._memmapping_reducer import TemporaryResourcesManager, get_memmapping_reducers
12
+ from .externals.loky.reusable_executor import _ReusablePoolExecutor
13
+
14
+ _executor_args = None
15
+
16
+
17
+ def get_memmapping_executor(n_jobs, **kwargs):
18
+ return MemmappingExecutor.get_memmapping_executor(n_jobs, **kwargs)
19
+
20
+
21
+ class MemmappingExecutor(_ReusablePoolExecutor):
22
+ @classmethod
23
+ def get_memmapping_executor(
24
+ cls,
25
+ n_jobs,
26
+ timeout=300,
27
+ initializer=None,
28
+ initargs=(),
29
+ env=None,
30
+ temp_folder=None,
31
+ context_id=None,
32
+ **backend_args,
33
+ ):
34
+ """Factory for ReusableExecutor with automatic memmapping for large
35
+ numpy arrays.
36
+ """
37
+ global _executor_args
38
+ # Check if we can reuse the executor here instead of deferring the test
39
+ # to loky as the reducers are objects that changes at each call.
40
+ executor_args = backend_args.copy()
41
+ executor_args.update(env if env else {})
42
+ executor_args.update(
43
+ dict(timeout=timeout, initializer=initializer, initargs=initargs)
44
+ )
45
+ reuse = _executor_args is None or _executor_args == executor_args
46
+ _executor_args = executor_args
47
+
48
+ manager = TemporaryResourcesManager(temp_folder)
49
+
50
+ # reducers access the temporary folder in which to store temporary
51
+ # pickles through a call to manager.resolve_temp_folder_name. resolving
52
+ # the folder name dynamically is useful to use different folders across
53
+ # calls of a same reusable executor
54
+ job_reducers, result_reducers = get_memmapping_reducers(
55
+ unlink_on_gc_collect=True,
56
+ temp_folder_resolver=manager.resolve_temp_folder_name,
57
+ **backend_args,
58
+ )
59
+ _executor, executor_is_reused = super().get_reusable_executor(
60
+ n_jobs,
61
+ job_reducers=job_reducers,
62
+ result_reducers=result_reducers,
63
+ reuse=reuse,
64
+ timeout=timeout,
65
+ initializer=initializer,
66
+ initargs=initargs,
67
+ env=env,
68
+ )
69
+
70
+ if not executor_is_reused:
71
+ # Only set a _temp_folder_manager for new executors. Reused
72
+ # executors already have a _temporary_folder_manager that must not
73
+ # be re-assigned like that because it is referenced in various
74
+ # places in the reducing machinery of the executor.
75
+ _executor._temp_folder_manager = manager
76
+
77
+ if context_id is not None:
78
+ # Only register the specified context once we know which manager
79
+ # the current executor is using, in order to not register an atexit
80
+ # finalizer twice for the same folder.
81
+ _executor._temp_folder_manager.register_new_context(context_id)
82
+
83
+ return _executor
84
+
85
+ def terminate(self, kill_workers=False):
86
+ self.shutdown(kill_workers=kill_workers)
87
+
88
+ # When workers are killed in a brutal manner, they cannot execute the
89
+ # finalizer of their shared memmaps. The refcount of those memmaps may
90
+ # be off by an unknown number, so instead of decref'ing them, we force
91
+ # delete the whole temporary folder, and unregister them. There is no
92
+ # risk of PermissionError at folder deletion because at this
93
+ # point, all child processes are dead, so all references to temporary
94
+ # memmaps are closed. Otherwise, just try to delete as much as possible
95
+ # with allow_non_empty=True but if we can't, it will be clean up later
96
+ # on by the resource_tracker.
97
+ with self._submit_resize_lock:
98
+ self._temp_folder_manager._clean_temporary_resources(
99
+ force=kill_workers, allow_non_empty=True
100
+ )
101
+
102
+ @property
103
+ def _temp_folder(self):
104
+ # Legacy property in tests. could be removed if we refactored the
105
+ # memmapping tests. SHOULD ONLY BE USED IN TESTS!
106
+ # We cache this property because it is called late in the tests - at
107
+ # this point, all context have been unregistered, and
108
+ # resolve_temp_folder_name raises an error.
109
+ if getattr(self, "_cached_temp_folder", None) is not None:
110
+ return self._cached_temp_folder
111
+ else:
112
+ self._cached_temp_folder = (
113
+ self._temp_folder_manager.resolve_temp_folder_name()
114
+ ) # noqa
115
+ return self._cached_temp_folder
116
+
117
+
118
+ class _TestingMemmappingExecutor(MemmappingExecutor):
119
+ """Wrapper around ReusableExecutor to ease memmapping testing with Pool
120
+ and Executor. This is only for testing purposes.
121
+
122
+ """
123
+
124
+ def apply_async(self, func, args):
125
+ """Schedule a func to be run"""
126
+ future = self.submit(func, *args)
127
+ future.get = future.result
128
+ return future
129
+
130
+ def map(self, f, *args):
131
+ return list(super().map(f, *args))
TTS/Kokoro/.venv/Lib/site-packages/joblib/func_inspect.py ADDED
@@ -0,0 +1,379 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ My own variation on function-specific inspect-like features.
3
+ """
4
+
5
+ # Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
6
+ # Copyright (c) 2009 Gael Varoquaux
7
+ # License: BSD Style, 3 clauses.
8
+
9
+ import collections
10
+ import inspect
11
+ import os
12
+ import re
13
+ import warnings
14
+ from itertools import islice
15
+ from tokenize import open as open_py_source
16
+
17
+ from .logger import pformat
18
+
19
+ full_argspec_fields = (
20
+ "args varargs varkw defaults kwonlyargs kwonlydefaults annotations"
21
+ )
22
+ full_argspec_type = collections.namedtuple("FullArgSpec", full_argspec_fields)
23
+
24
+
25
+ def get_func_code(func):
26
+ """Attempts to retrieve a reliable function code hash.
27
+
28
+ The reason we don't use inspect.getsource is that it caches the
29
+ source, whereas we want this to be modified on the fly when the
30
+ function is modified.
31
+
32
+ Returns
33
+ -------
34
+ func_code: string
35
+ The function code
36
+ source_file: string
37
+ The path to the file in which the function is defined.
38
+ first_line: int
39
+ The first line of the code in the source file.
40
+
41
+ Notes
42
+ ------
43
+ This function does a bit more magic than inspect, and is thus
44
+ more robust.
45
+ """
46
+ source_file = None
47
+ try:
48
+ code = func.__code__
49
+ source_file = code.co_filename
50
+ if not os.path.exists(source_file):
51
+ # Use inspect for lambda functions and functions defined in an
52
+ # interactive shell, or in doctests
53
+ source_code = "".join(inspect.getsourcelines(func)[0])
54
+ line_no = 1
55
+ if source_file.startswith("<doctest "):
56
+ source_file, line_no = re.match(
57
+ r"\<doctest (.*\.rst)\[(.*)\]\>", source_file
58
+ ).groups()
59
+ line_no = int(line_no)
60
+ source_file = "<doctest %s>" % source_file
61
+ return source_code, source_file, line_no
62
+ # Try to retrieve the source code.
63
+ with open_py_source(source_file) as source_file_obj:
64
+ first_line = code.co_firstlineno
65
+ # All the lines after the function definition:
66
+ source_lines = list(islice(source_file_obj, first_line - 1, None))
67
+ return "".join(inspect.getblock(source_lines)), source_file, first_line
68
+ except: # noqa: E722
69
+ # If the source code fails, we use the hash. This is fragile and
70
+ # might change from one session to another.
71
+ if hasattr(func, "__code__"):
72
+ # Python 3.X
73
+ return str(func.__code__.__hash__()), source_file, -1
74
+ else:
75
+ # Weird objects like numpy ufunc don't have __code__
76
+ # This is fragile, as quite often the id of the object is
77
+ # in the repr, so it might not persist across sessions,
78
+ # however it will work for ufuncs.
79
+ return repr(func), source_file, -1
80
+
81
+
82
+ def _clean_win_chars(string):
83
+ """Windows cannot encode some characters in filename."""
84
+ import urllib
85
+
86
+ if hasattr(urllib, "quote"):
87
+ quote = urllib.quote
88
+ else:
89
+ # In Python 3, quote is elsewhere
90
+ import urllib.parse
91
+
92
+ quote = urllib.parse.quote
93
+ for char in ("<", ">", "!", ":", "\\"):
94
+ string = string.replace(char, quote(char))
95
+ return string
96
+
97
+
98
+ def get_func_name(func, resolv_alias=True, win_characters=True):
99
+ """Return the function import path (as a list of module names), and
100
+ a name for the function.
101
+
102
+ Parameters
103
+ ----------
104
+ func: callable
105
+ The func to inspect
106
+ resolv_alias: boolean, optional
107
+ If true, possible local aliases are indicated.
108
+ win_characters: boolean, optional
109
+ If true, substitute special characters using urllib.quote
110
+ This is useful in Windows, as it cannot encode some filenames
111
+ """
112
+ if hasattr(func, "__module__"):
113
+ module = func.__module__
114
+ else:
115
+ try:
116
+ module = inspect.getmodule(func)
117
+ except TypeError:
118
+ if hasattr(func, "__class__"):
119
+ module = func.__class__.__module__
120
+ else:
121
+ module = "unknown"
122
+ if module is None:
123
+ # Happens in doctests, eg
124
+ module = ""
125
+ if module == "__main__":
126
+ try:
127
+ filename = os.path.abspath(inspect.getsourcefile(func))
128
+ except: # noqa: E722
129
+ filename = None
130
+ if filename is not None:
131
+ # mangling of full path to filename
132
+ parts = filename.split(os.sep)
133
+ if parts[-1].startswith("<ipython-input"):
134
+ # We're in a IPython (or notebook) session. parts[-1] comes
135
+ # from func.__code__.co_filename and is of the form
136
+ # <ipython-input-N-XYZ>, where:
137
+ # - N is the cell number where the function was defined
138
+ # - XYZ is a hash representing the function's code (and name).
139
+ # It will be consistent across sessions and kernel restarts,
140
+ # and will change if the function's code/name changes
141
+ # We remove N so that cache is properly hit if the cell where
142
+ # the func is defined is re-exectuted.
143
+ # The XYZ hash should avoid collisions between functions with
144
+ # the same name, both within the same notebook but also across
145
+ # notebooks
146
+ split = parts[-1].split("-")
147
+ parts[-1] = "-".join(split[:2] + split[3:])
148
+ elif len(parts) > 2 and parts[-2].startswith("ipykernel_"):
149
+ # In a notebook session (ipykernel). Filename seems to be 'xyz'
150
+ # of above. parts[-2] has the structure ipykernel_XXXXXX where
151
+ # XXXXXX is a six-digit number identifying the current run (?).
152
+ # If we split it off, the function again has the same
153
+ # identifier across runs.
154
+ parts[-2] = "ipykernel"
155
+ filename = "-".join(parts)
156
+ if filename.endswith(".py"):
157
+ filename = filename[:-3]
158
+ module = module + "-" + filename
159
+ module = module.split(".")
160
+ if hasattr(func, "func_name"):
161
+ name = func.func_name
162
+ elif hasattr(func, "__name__"):
163
+ name = func.__name__
164
+ else:
165
+ name = "unknown"
166
+ # Hack to detect functions not defined at the module-level
167
+ if resolv_alias:
168
+ # TODO: Maybe add a warning here?
169
+ if hasattr(func, "func_globals") and name in func.func_globals:
170
+ if func.func_globals[name] is not func:
171
+ name = "%s-alias" % name
172
+ if hasattr(func, "__qualname__") and func.__qualname__ != name:
173
+ # Extend the module name in case of nested functions to avoid
174
+ # (module, name) collisions
175
+ module.extend(func.__qualname__.split(".")[:-1])
176
+ if inspect.ismethod(func):
177
+ # We need to add the name of the class
178
+ if hasattr(func, "im_class"):
179
+ klass = func.im_class
180
+ module.append(klass.__name__)
181
+ if os.name == "nt" and win_characters:
182
+ # Windows can't encode certain characters in filenames
183
+ name = _clean_win_chars(name)
184
+ module = [_clean_win_chars(s) for s in module]
185
+ return module, name
186
+
187
+
188
+ def _signature_str(function_name, arg_sig):
189
+ """Helper function to output a function signature"""
190
+ return "{}{}".format(function_name, arg_sig)
191
+
192
+
193
+ def _function_called_str(function_name, args, kwargs):
194
+ """Helper function to output a function call"""
195
+ template_str = "{0}({1}, {2})"
196
+
197
+ args_str = repr(args)[1:-1]
198
+ kwargs_str = ", ".join("%s=%s" % (k, v) for k, v in kwargs.items())
199
+ return template_str.format(function_name, args_str, kwargs_str)
200
+
201
+
202
+ def filter_args(func, ignore_lst, args=(), kwargs=dict()):
203
+ """Filters the given args and kwargs using a list of arguments to
204
+ ignore, and a function specification.
205
+
206
+ Parameters
207
+ ----------
208
+ func: callable
209
+ Function giving the argument specification
210
+ ignore_lst: list of strings
211
+ List of arguments to ignore (either a name of an argument
212
+ in the function spec, or '*', or '**')
213
+ *args: list
214
+ Positional arguments passed to the function.
215
+ **kwargs: dict
216
+ Keyword arguments passed to the function
217
+
218
+ Returns
219
+ -------
220
+ filtered_args: list
221
+ List of filtered positional and keyword arguments.
222
+ """
223
+ args = list(args)
224
+ if isinstance(ignore_lst, str):
225
+ # Catch a common mistake
226
+ raise ValueError(
227
+ "ignore_lst must be a list of parameters to ignore "
228
+ "%s (type %s) was given" % (ignore_lst, type(ignore_lst))
229
+ )
230
+ # Special case for functools.partial objects
231
+ if not inspect.ismethod(func) and not inspect.isfunction(func):
232
+ if ignore_lst:
233
+ warnings.warn(
234
+ "Cannot inspect object %s, ignore list will not work." % func,
235
+ stacklevel=2,
236
+ )
237
+ return {"*": args, "**": kwargs}
238
+ arg_sig = inspect.signature(func)
239
+ arg_names = []
240
+ arg_defaults = []
241
+ arg_kwonlyargs = []
242
+ arg_varargs = None
243
+ arg_varkw = None
244
+ for param in arg_sig.parameters.values():
245
+ if param.kind is param.POSITIONAL_OR_KEYWORD:
246
+ arg_names.append(param.name)
247
+ elif param.kind is param.KEYWORD_ONLY:
248
+ arg_names.append(param.name)
249
+ arg_kwonlyargs.append(param.name)
250
+ elif param.kind is param.VAR_POSITIONAL:
251
+ arg_varargs = param.name
252
+ elif param.kind is param.VAR_KEYWORD:
253
+ arg_varkw = param.name
254
+ if param.default is not param.empty:
255
+ arg_defaults.append(param.default)
256
+ if inspect.ismethod(func):
257
+ # First argument is 'self', it has been removed by Python
258
+ # we need to add it back:
259
+ args = [
260
+ func.__self__,
261
+ ] + args
262
+ # func is an instance method, inspect.signature(func) does not
263
+ # include self, we need to fetch it from the class method, i.e
264
+ # func.__func__
265
+ class_method_sig = inspect.signature(func.__func__)
266
+ self_name = next(iter(class_method_sig.parameters))
267
+ arg_names = [self_name] + arg_names
268
+ # XXX: Maybe I need an inspect.isbuiltin to detect C-level methods, such
269
+ # as on ndarrays.
270
+
271
+ _, name = get_func_name(func, resolv_alias=False)
272
+ arg_dict = dict()
273
+ arg_position = -1
274
+ for arg_position, arg_name in enumerate(arg_names):
275
+ if arg_position < len(args):
276
+ # Positional argument or keyword argument given as positional
277
+ if arg_name not in arg_kwonlyargs:
278
+ arg_dict[arg_name] = args[arg_position]
279
+ else:
280
+ raise ValueError(
281
+ "Keyword-only parameter '%s' was passed as "
282
+ "positional parameter for %s:\n"
283
+ " %s was called."
284
+ % (
285
+ arg_name,
286
+ _signature_str(name, arg_sig),
287
+ _function_called_str(name, args, kwargs),
288
+ )
289
+ )
290
+
291
+ else:
292
+ position = arg_position - len(arg_names)
293
+ if arg_name in kwargs:
294
+ arg_dict[arg_name] = kwargs[arg_name]
295
+ else:
296
+ try:
297
+ arg_dict[arg_name] = arg_defaults[position]
298
+ except (IndexError, KeyError) as e:
299
+ # Missing argument
300
+ raise ValueError(
301
+ "Wrong number of arguments for %s:\n"
302
+ " %s was called."
303
+ % (
304
+ _signature_str(name, arg_sig),
305
+ _function_called_str(name, args, kwargs),
306
+ )
307
+ ) from e
308
+
309
+ varkwargs = dict()
310
+ for arg_name, arg_value in sorted(kwargs.items()):
311
+ if arg_name in arg_dict:
312
+ arg_dict[arg_name] = arg_value
313
+ elif arg_varkw is not None:
314
+ varkwargs[arg_name] = arg_value
315
+ else:
316
+ raise TypeError(
317
+ "Ignore list for %s() contains an unexpected "
318
+ "keyword argument '%s'" % (name, arg_name)
319
+ )
320
+
321
+ if arg_varkw is not None:
322
+ arg_dict["**"] = varkwargs
323
+ if arg_varargs is not None:
324
+ varargs = args[arg_position + 1 :]
325
+ arg_dict["*"] = varargs
326
+
327
+ # Now remove the arguments to be ignored
328
+ for item in ignore_lst:
329
+ if item in arg_dict:
330
+ arg_dict.pop(item)
331
+ else:
332
+ raise ValueError(
333
+ "Ignore list: argument '%s' is not defined for "
334
+ "function %s" % (item, _signature_str(name, arg_sig))
335
+ )
336
+ # XXX: Return a sorted list of pairs?
337
+ return arg_dict
338
+
339
+
340
+ def _format_arg(arg):
341
+ formatted_arg = pformat(arg, indent=2)
342
+ if len(formatted_arg) > 1500:
343
+ formatted_arg = "%s..." % formatted_arg[:700]
344
+ return formatted_arg
345
+
346
+
347
+ def format_signature(func, *args, **kwargs):
348
+ # XXX: Should this use inspect.formatargvalues/formatargspec?
349
+ module, name = get_func_name(func)
350
+ module = [m for m in module if m]
351
+ if module:
352
+ module.append(name)
353
+ module_path = ".".join(module)
354
+ else:
355
+ module_path = name
356
+ arg_str = list()
357
+ previous_length = 0
358
+ for arg in args:
359
+ formatted_arg = _format_arg(arg)
360
+ if previous_length > 80:
361
+ formatted_arg = "\n%s" % formatted_arg
362
+ previous_length = len(formatted_arg)
363
+ arg_str.append(formatted_arg)
364
+ arg_str.extend(["%s=%s" % (v, _format_arg(i)) for v, i in kwargs.items()])
365
+ arg_str = ", ".join(arg_str)
366
+
367
+ signature = "%s(%s)" % (name, arg_str)
368
+ return module_path, signature
369
+
370
+
371
+ def format_call(func, args, kwargs, object_name="Memory"):
372
+ """Returns a nicely formatted statement displaying the function
373
+ call with the given arguments.
374
+ """
375
+ path, signature = format_signature(func, *args, **kwargs)
376
+ msg = "%s\n[%s] Calling %s...\n%s" % (80 * "_", object_name, path, signature)
377
+ return msg
378
+ # XXX: Not using logging framework
379
+ # self.debug(msg)
TTS/Kokoro/.venv/Lib/site-packages/joblib/hashing.py ADDED
@@ -0,0 +1,270 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Fast cryptographic hash of Python objects, with a special case for fast
3
+ hashing of numpy arrays.
4
+ """
5
+
6
+ # Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
7
+ # Copyright (c) 2009 Gael Varoquaux
8
+ # License: BSD Style, 3 clauses.
9
+
10
+ import decimal
11
+ import hashlib
12
+ import io
13
+ import pickle
14
+ import struct
15
+ import sys
16
+ import types
17
+
18
+ Pickler = pickle._Pickler
19
+
20
+
21
+ class _ConsistentSet(object):
22
+ """Class used to ensure the hash of Sets is preserved
23
+ whatever the order of its items.
24
+ """
25
+
26
+ def __init__(self, set_sequence):
27
+ # Forces order of elements in set to ensure consistent hash.
28
+ try:
29
+ # Trying first to order the set assuming the type of elements is
30
+ # consistent and orderable.
31
+ # This fails on python 3 when elements are unorderable
32
+ # but we keep it in a try as it's faster.
33
+ self._sequence = sorted(set_sequence)
34
+ except (TypeError, decimal.InvalidOperation):
35
+ # If elements are unorderable, sorting them using their hash.
36
+ # This is slower but works in any case.
37
+ self._sequence = sorted((hash(e) for e in set_sequence))
38
+
39
+
40
+ class _MyHash(object):
41
+ """Class used to hash objects that won't normally pickle"""
42
+
43
+ def __init__(self, *args):
44
+ self.args = args
45
+
46
+
47
+ class Hasher(Pickler):
48
+ """A subclass of pickler, to do cryptographic hashing, rather than
49
+ pickling. This is used to produce a unique hash of the given
50
+ Python object that is not necessarily cryptographically secure.
51
+ """
52
+
53
+ def __init__(self, hash_name="md5"):
54
+ self.stream = io.BytesIO()
55
+ # By default we want a pickle protocol that only changes with
56
+ # the major python version and not the minor one
57
+ protocol = 3
58
+ Pickler.__init__(self, self.stream, protocol=protocol)
59
+ # Initialise the hash obj
60
+ self._hash = hashlib.new(hash_name, usedforsecurity=False)
61
+
62
+ def hash(self, obj, return_digest=True):
63
+ try:
64
+ self.dump(obj)
65
+ except pickle.PicklingError as e:
66
+ e.args += ("PicklingError while hashing %r: %r" % (obj, e),)
67
+ raise
68
+ dumps = self.stream.getvalue()
69
+ self._hash.update(dumps)
70
+ if return_digest:
71
+ return self._hash.hexdigest()
72
+
73
+ def save(self, obj):
74
+ if isinstance(obj, (types.MethodType, type({}.pop))):
75
+ # the Pickler cannot pickle instance methods; here we decompose
76
+ # them into components that make them uniquely identifiable
77
+ if hasattr(obj, "__func__"):
78
+ func_name = obj.__func__.__name__
79
+ else:
80
+ func_name = obj.__name__
81
+ inst = obj.__self__
82
+ if type(inst) is type(pickle):
83
+ obj = _MyHash(func_name, inst.__name__)
84
+ elif inst is None:
85
+ # type(None) or type(module) do not pickle
86
+ obj = _MyHash(func_name, inst)
87
+ else:
88
+ cls = obj.__self__.__class__
89
+ obj = _MyHash(func_name, inst, cls)
90
+ Pickler.save(self, obj)
91
+
92
+ def memoize(self, obj):
93
+ # We want hashing to be sensitive to value instead of reference.
94
+ # For example we want ['aa', 'aa'] and ['aa', 'aaZ'[:2]]
95
+ # to hash to the same value and that's why we disable memoization
96
+ # for strings
97
+ if isinstance(obj, (bytes, str)):
98
+ return
99
+ Pickler.memoize(self, obj)
100
+
101
+ # The dispatch table of the pickler is not accessible in Python
102
+ # 3, as these lines are only bugware for IPython, we skip them.
103
+ def save_global(self, obj, name=None, pack=struct.pack):
104
+ # We have to override this method in order to deal with objects
105
+ # defined interactively in IPython that are not injected in
106
+ # __main__
107
+ kwargs = dict(name=name, pack=pack)
108
+ del kwargs["pack"]
109
+ try:
110
+ Pickler.save_global(self, obj, **kwargs)
111
+ except pickle.PicklingError:
112
+ Pickler.save_global(self, obj, **kwargs)
113
+ module = getattr(obj, "__module__", None)
114
+ if module == "__main__":
115
+ my_name = name
116
+ if my_name is None:
117
+ my_name = obj.__name__
118
+ mod = sys.modules[module]
119
+ if not hasattr(mod, my_name):
120
+ # IPython doesn't inject the variables define
121
+ # interactively in __main__
122
+ setattr(mod, my_name, obj)
123
+
124
+ dispatch = Pickler.dispatch.copy()
125
+ # builtin
126
+ dispatch[type(len)] = save_global
127
+ # type
128
+ dispatch[type(object)] = save_global
129
+ # classobj
130
+ dispatch[type(Pickler)] = save_global
131
+ # function
132
+ dispatch[type(pickle.dump)] = save_global
133
+
134
+ # We use *args in _batch_setitems signature because _batch_setitems has an
135
+ # additional 'obj' argument in Python 3.14
136
+ def _batch_setitems(self, items, *args):
137
+ # forces order of keys in dict to ensure consistent hash.
138
+ try:
139
+ # Trying first to compare dict assuming the type of keys is
140
+ # consistent and orderable.
141
+ # This fails on python 3 when keys are unorderable
142
+ # but we keep it in a try as it's faster.
143
+ Pickler._batch_setitems(self, iter(sorted(items)), *args)
144
+ except TypeError:
145
+ # If keys are unorderable, sorting them using their hash. This is
146
+ # slower but works in any case.
147
+ Pickler._batch_setitems(
148
+ self, iter(sorted((hash(k), v) for k, v in items)), *args
149
+ )
150
+
151
+ def save_set(self, set_items):
152
+ # forces order of items in Set to ensure consistent hash
153
+ Pickler.save(self, _ConsistentSet(set_items))
154
+
155
+ dispatch[type(set())] = save_set
156
+
157
+
158
+ class NumpyHasher(Hasher):
159
+ """Special case the hasher for when numpy is loaded."""
160
+
161
+ def __init__(self, hash_name="md5", coerce_mmap=False):
162
+ """
163
+ Parameters
164
+ ----------
165
+ hash_name: string
166
+ The hash algorithm to be used
167
+ coerce_mmap: boolean
168
+ Make no difference between np.memmap and np.ndarray
169
+ objects.
170
+ """
171
+ self.coerce_mmap = coerce_mmap
172
+ Hasher.__init__(self, hash_name=hash_name)
173
+ # delayed import of numpy, to avoid tight coupling
174
+ import numpy as np
175
+
176
+ self.np = np
177
+ if hasattr(np, "getbuffer"):
178
+ self._getbuffer = np.getbuffer
179
+ else:
180
+ self._getbuffer = memoryview
181
+
182
+ def save(self, obj):
183
+ """Subclass the save method, to hash ndarray subclass, rather
184
+ than pickling them. Off course, this is a total abuse of
185
+ the Pickler class.
186
+ """
187
+ if isinstance(obj, self.np.ndarray) and not obj.dtype.hasobject:
188
+ # Compute a hash of the object
189
+ # The update function of the hash requires a c_contiguous buffer.
190
+ if obj.shape == ():
191
+ # 0d arrays need to be flattened because viewing them as bytes
192
+ # raises a ValueError exception.
193
+ obj_c_contiguous = obj.flatten()
194
+ elif obj.flags.c_contiguous:
195
+ obj_c_contiguous = obj
196
+ elif obj.flags.f_contiguous:
197
+ obj_c_contiguous = obj.T
198
+ else:
199
+ # Cater for non-single-segment arrays: this creates a
200
+ # copy, and thus alleviates this issue.
201
+ # XXX: There might be a more efficient way of doing this
202
+ obj_c_contiguous = obj.flatten()
203
+
204
+ # memoryview is not supported for some dtypes, e.g. datetime64, see
205
+ # https://github.com/numpy/numpy/issues/4983. The
206
+ # workaround is to view the array as bytes before
207
+ # taking the memoryview.
208
+ self._hash.update(self._getbuffer(obj_c_contiguous.view(self.np.uint8)))
209
+
210
+ # We store the class, to be able to distinguish between
211
+ # Objects with the same binary content, but different
212
+ # classes.
213
+ if self.coerce_mmap and isinstance(obj, self.np.memmap):
214
+ # We don't make the difference between memmap and
215
+ # normal ndarrays, to be able to reload previously
216
+ # computed results with memmap.
217
+ klass = self.np.ndarray
218
+ else:
219
+ klass = obj.__class__
220
+ # We also return the dtype and the shape, to distinguish
221
+ # different views on the same data with different dtypes.
222
+
223
+ # The object will be pickled by the pickler hashed at the end.
224
+ obj = (klass, ("HASHED", obj.dtype, obj.shape, obj.strides))
225
+ elif isinstance(obj, self.np.dtype):
226
+ # numpy.dtype consistent hashing is tricky to get right. This comes
227
+ # from the fact that atomic np.dtype objects are interned:
228
+ # ``np.dtype('f4') is np.dtype('f4')``. The situation is
229
+ # complicated by the fact that this interning does not resist a
230
+ # simple pickle.load/dump roundtrip:
231
+ # ``pickle.loads(pickle.dumps(np.dtype('f4'))) is not
232
+ # np.dtype('f4') Because pickle relies on memoization during
233
+ # pickling, it is easy to
234
+ # produce different hashes for seemingly identical objects, such as
235
+ # ``[np.dtype('f4'), np.dtype('f4')]``
236
+ # and ``[np.dtype('f4'), pickle.loads(pickle.dumps('f4'))]``.
237
+ # To prevent memoization from interfering with hashing, we isolate
238
+ # the serialization (and thus the pickle memoization) of each dtype
239
+ # using each time a different ``pickle.dumps`` call unrelated to
240
+ # the current Hasher instance.
241
+ self._hash.update("_HASHED_DTYPE".encode("utf-8"))
242
+ self._hash.update(pickle.dumps(obj))
243
+ return
244
+ Hasher.save(self, obj)
245
+
246
+
247
+ def hash(obj, hash_name="md5", coerce_mmap=False):
248
+ """Quick calculation of a hash to identify uniquely Python objects
249
+ containing numpy arrays.
250
+
251
+ Parameters
252
+ ----------
253
+ hash_name: 'md5' or 'sha1'
254
+ Hashing algorithm used. sha1 is supposedly safer, but md5 is
255
+ faster.
256
+ coerce_mmap: boolean
257
+ Make no difference between np.memmap and np.ndarray
258
+ """
259
+ valid_hash_names = ("md5", "sha1")
260
+ if hash_name not in valid_hash_names:
261
+ raise ValueError(
262
+ "Valid options for 'hash_name' are {}. Got hash_name={!r} instead.".format(
263
+ valid_hash_names, hash_name
264
+ )
265
+ )
266
+ if "numpy" in sys.modules:
267
+ hasher = NumpyHasher(hash_name=hash_name, coerce_mmap=coerce_mmap)
268
+ else:
269
+ hasher = Hasher(hash_name=hash_name)
270
+ return hasher.hash(obj)
TTS/Kokoro/.venv/Lib/site-packages/joblib/logger.py ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Helpers for logging.
3
+
4
+ This module needs much love to become useful.
5
+ """
6
+
7
+ # Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
8
+ # Copyright (c) 2008 Gael Varoquaux
9
+ # License: BSD Style, 3 clauses.
10
+
11
+ from __future__ import print_function
12
+
13
+ import logging
14
+ import os
15
+ import pprint
16
+ import shutil
17
+ import sys
18
+ import time
19
+
20
+ from .disk import mkdirp
21
+
22
+
23
+ def _squeeze_time(t):
24
+ """Remove .1s to the time under Windows: this is the time it take to
25
+ stat files. This is needed to make results similar to timings under
26
+ Unix, for tests
27
+ """
28
+ if sys.platform.startswith("win"):
29
+ return max(0, t - 0.1)
30
+ else:
31
+ return t
32
+
33
+
34
+ def format_time(t):
35
+ t = _squeeze_time(t)
36
+ return "%.1fs, %.1fmin" % (t, t / 60.0)
37
+
38
+
39
+ def short_format_time(t):
40
+ t = _squeeze_time(t)
41
+ if t > 60:
42
+ return "%4.1fmin" % (t / 60.0)
43
+ else:
44
+ return " %5.1fs" % (t)
45
+
46
+
47
+ def pformat(obj, indent=0, depth=3):
48
+ if "numpy" in sys.modules:
49
+ import numpy as np
50
+
51
+ print_options = np.get_printoptions()
52
+ np.set_printoptions(precision=6, threshold=64, edgeitems=1)
53
+ else:
54
+ print_options = None
55
+ out = pprint.pformat(obj, depth=depth, indent=indent)
56
+ if print_options:
57
+ np.set_printoptions(**print_options)
58
+ return out
59
+
60
+
61
+ ###############################################################################
62
+ # class `Logger`
63
+ ###############################################################################
64
+ class Logger(object):
65
+ """Base class for logging messages."""
66
+
67
+ def __init__(self, depth=3, name=None):
68
+ """
69
+ Parameters
70
+ ----------
71
+ depth: int, optional
72
+ The depth of objects printed.
73
+ name: str, optional
74
+ The namespace to log to. If None, defaults to joblib.
75
+ """
76
+ self.depth = depth
77
+ self._name = name if name else "joblib"
78
+
79
+ def warn(self, msg):
80
+ logging.getLogger(self._name).warning("[%s]: %s" % (self, msg))
81
+
82
+ def info(self, msg):
83
+ logging.info("[%s]: %s" % (self, msg))
84
+
85
+ def debug(self, msg):
86
+ # XXX: This conflicts with the debug flag used in children class
87
+ logging.getLogger(self._name).debug("[%s]: %s" % (self, msg))
88
+
89
+ def format(self, obj, indent=0):
90
+ """Return the formatted representation of the object."""
91
+ return pformat(obj, indent=indent, depth=self.depth)
92
+
93
+
94
+ ###############################################################################
95
+ # class `PrintTime`
96
+ ###############################################################################
97
+ class PrintTime(object):
98
+ """Print and log messages while keeping track of time."""
99
+
100
+ def __init__(self, logfile=None, logdir=None):
101
+ if logfile is not None and logdir is not None:
102
+ raise ValueError("Cannot specify both logfile and logdir")
103
+ # XXX: Need argument docstring
104
+ self.last_time = time.time()
105
+ self.start_time = self.last_time
106
+ if logdir is not None:
107
+ logfile = os.path.join(logdir, "joblib.log")
108
+ self.logfile = logfile
109
+ if logfile is not None:
110
+ mkdirp(os.path.dirname(logfile))
111
+ if os.path.exists(logfile):
112
+ # Rotate the logs
113
+ for i in range(1, 9):
114
+ try:
115
+ shutil.move(logfile + ".%i" % i, logfile + ".%i" % (i + 1))
116
+ except: # noqa: E722
117
+ "No reason failing here"
118
+ # Use a copy rather than a move, so that a process
119
+ # monitoring this file does not get lost.
120
+ try:
121
+ shutil.copy(logfile, logfile + ".1")
122
+ except: # noqa: E722
123
+ "No reason failing here"
124
+ try:
125
+ with open(logfile, "w") as logfile:
126
+ logfile.write("\nLogging joblib python script\n")
127
+ logfile.write("\n---%s---\n" % time.ctime(self.last_time))
128
+ except: # noqa: E722
129
+ """ Multiprocessing writing to files can create race
130
+ conditions. Rather fail silently than crash the
131
+ computation.
132
+ """
133
+ # XXX: We actually need a debug flag to disable this
134
+ # silent failure.
135
+
136
+ def __call__(self, msg="", total=False):
137
+ """Print the time elapsed between the last call and the current
138
+ call, with an optional message.
139
+ """
140
+ if not total:
141
+ time_lapse = time.time() - self.last_time
142
+ full_msg = "%s: %s" % (msg, format_time(time_lapse))
143
+ else:
144
+ # FIXME: Too much logic duplicated
145
+ time_lapse = time.time() - self.start_time
146
+ full_msg = "%s: %.2fs, %.1f min" % (msg, time_lapse, time_lapse / 60)
147
+ print(full_msg, file=sys.stderr)
148
+ if self.logfile is not None:
149
+ try:
150
+ with open(self.logfile, "a") as f:
151
+ print(full_msg, file=f)
152
+ except: # noqa: E722
153
+ """ Multiprocessing writing to files can create race
154
+ conditions. Rather fail silently than crash the
155
+ calculation.
156
+ """
157
+ # XXX: We actually need a debug flag to disable this
158
+ # silent failure.
159
+ self.last_time = time.time()
TTS/Kokoro/.venv/Lib/site-packages/joblib/memory.py ADDED
@@ -0,0 +1,1242 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ A context object for caching a function's return value each time it
3
+ is called with the same input arguments.
4
+
5
+ """
6
+
7
+ # Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
8
+ # Copyright (c) 2009 Gael Varoquaux
9
+ # License: BSD Style, 3 clauses.
10
+
11
+ import asyncio
12
+ import datetime
13
+ import functools
14
+ import inspect
15
+ import logging
16
+ import os
17
+ import pathlib
18
+ import pydoc
19
+ import re
20
+ import textwrap
21
+ import time
22
+ import tokenize
23
+ import traceback
24
+ import warnings
25
+ import weakref
26
+
27
+ from . import hashing
28
+ from ._store_backends import (
29
+ CacheWarning, # noqa
30
+ FileSystemStoreBackend,
31
+ StoreBackendBase,
32
+ )
33
+ from .func_inspect import (
34
+ filter_args,
35
+ format_call,
36
+ format_signature,
37
+ get_func_code,
38
+ get_func_name,
39
+ )
40
+ from .logger import Logger, format_time, pformat
41
+
42
+ FIRST_LINE_TEXT = "# first line:"
43
+
44
+ # TODO: The following object should have a data store object as a sub
45
+ # object, and the interface to persist and query should be separated in
46
+ # the data store.
47
+ #
48
+ # This would enable creating 'Memory' objects with a different logic for
49
+ # pickling that would simply span a MemorizedFunc with the same
50
+ # store (or do we want to copy it to avoid cross-talks?), for instance to
51
+ # implement HDF5 pickling.
52
+
53
+ # TODO: Same remark for the logger, and probably use the Python logging
54
+ # mechanism.
55
+
56
+
57
+ def extract_first_line(func_code):
58
+ """Extract the first line information from the function code
59
+ text if available.
60
+ """
61
+ if func_code.startswith(FIRST_LINE_TEXT):
62
+ func_code = func_code.split("\n")
63
+ first_line = int(func_code[0][len(FIRST_LINE_TEXT) :])
64
+ func_code = "\n".join(func_code[1:])
65
+ else:
66
+ first_line = -1
67
+ return func_code, first_line
68
+
69
+
70
+ class JobLibCollisionWarning(UserWarning):
71
+ """Warn that there might be a collision between names of functions."""
72
+
73
+
74
+ _STORE_BACKENDS = {"local": FileSystemStoreBackend}
75
+
76
+
77
+ def register_store_backend(backend_name, backend):
78
+ """Extend available store backends.
79
+
80
+ The Memory, MemorizeResult and MemorizeFunc objects are designed to be
81
+ agnostic to the type of store used behind. By default, the local file
82
+ system is used but this function gives the possibility to extend joblib's
83
+ memory pattern with other types of storage such as cloud storage (S3, GCS,
84
+ OpenStack, HadoopFS, etc) or blob DBs.
85
+
86
+ Parameters
87
+ ----------
88
+ backend_name: str
89
+ The name identifying the store backend being registered. For example,
90
+ 'local' is used with FileSystemStoreBackend.
91
+ backend: StoreBackendBase subclass
92
+ The name of a class that implements the StoreBackendBase interface.
93
+
94
+ """
95
+ if not isinstance(backend_name, str):
96
+ raise ValueError(
97
+ "Store backend name should be a string, '{0}' given.".format(backend_name)
98
+ )
99
+ if backend is None or not issubclass(backend, StoreBackendBase):
100
+ raise ValueError(
101
+ "Store backend should inherit StoreBackendBase, '{0}' given.".format(
102
+ backend
103
+ )
104
+ )
105
+
106
+ _STORE_BACKENDS[backend_name] = backend
107
+
108
+
109
+ def _store_backend_factory(backend, location, verbose=0, backend_options=None):
110
+ """Return the correct store object for the given location."""
111
+ if backend_options is None:
112
+ backend_options = {}
113
+
114
+ if isinstance(location, pathlib.Path):
115
+ location = str(location)
116
+
117
+ if isinstance(location, StoreBackendBase):
118
+ return location
119
+ elif isinstance(location, str):
120
+ obj = None
121
+ location = os.path.expanduser(location)
122
+ # The location is not a local file system, we look in the
123
+ # registered backends if there's one matching the given backend
124
+ # name.
125
+ for backend_key, backend_obj in _STORE_BACKENDS.items():
126
+ if backend == backend_key:
127
+ obj = backend_obj()
128
+
129
+ # By default, we assume the FileSystemStoreBackend can be used if no
130
+ # matching backend could be found.
131
+ if obj is None:
132
+ raise TypeError(
133
+ "Unknown location {0} or backend {1}".format(location, backend)
134
+ )
135
+
136
+ # The store backend is configured with the extra named parameters,
137
+ # some of them are specific to the underlying store backend.
138
+ obj.configure(location, verbose=verbose, backend_options=backend_options)
139
+ return obj
140
+ elif location is not None:
141
+ warnings.warn(
142
+ "Instantiating a backend using a {} as a location is not "
143
+ "supported by joblib. Returning None instead.".format(
144
+ location.__class__.__name__
145
+ ),
146
+ UserWarning,
147
+ )
148
+
149
+ return None
150
+
151
+
152
+ def _build_func_identifier(func):
153
+ """Build a roughly unique identifier for the cached function."""
154
+ modules, funcname = get_func_name(func)
155
+ # We reuse historical fs-like way of building a function identifier
156
+ return os.path.join(*modules, funcname)
157
+
158
+
159
+ # An in-memory store to avoid looking at the disk-based function
160
+ # source code to check if a function definition has changed
161
+ _FUNCTION_HASHES = weakref.WeakKeyDictionary()
162
+
163
+
164
+ ###############################################################################
165
+ # class `MemorizedResult`
166
+ ###############################################################################
167
+ class MemorizedResult(Logger):
168
+ """Object representing a cached value.
169
+
170
+ Attributes
171
+ ----------
172
+ location: str
173
+ The location of joblib cache. Depends on the store backend used.
174
+
175
+ func: function or str
176
+ function whose output is cached. The string case is intended only for
177
+ instantiation based on the output of repr() on another instance.
178
+ (namely eval(repr(memorized_instance)) works).
179
+
180
+ argument_hash: str
181
+ hash of the function arguments.
182
+
183
+ backend: str
184
+ Type of store backend for reading/writing cache files.
185
+ Default is 'local'.
186
+
187
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
188
+ The memmapping mode used when loading from cache numpy arrays. See
189
+ numpy.load for the meaning of the different values.
190
+
191
+ verbose: int
192
+ verbosity level (0 means no message).
193
+
194
+ timestamp, metadata: string
195
+ for internal use only.
196
+ """
197
+
198
+ def __init__(
199
+ self,
200
+ location,
201
+ call_id,
202
+ backend="local",
203
+ mmap_mode=None,
204
+ verbose=0,
205
+ timestamp=None,
206
+ metadata=None,
207
+ ):
208
+ Logger.__init__(self)
209
+ self._call_id = call_id
210
+ self.store_backend = _store_backend_factory(backend, location, verbose=verbose)
211
+ self.mmap_mode = mmap_mode
212
+
213
+ if metadata is not None:
214
+ self.metadata = metadata
215
+ else:
216
+ self.metadata = self.store_backend.get_metadata(self._call_id)
217
+
218
+ self.duration = self.metadata.get("duration", None)
219
+ self.verbose = verbose
220
+ self.timestamp = timestamp
221
+
222
+ @property
223
+ def func(self):
224
+ return self.func_id
225
+
226
+ @property
227
+ def func_id(self):
228
+ return self._call_id[0]
229
+
230
+ @property
231
+ def args_id(self):
232
+ return self._call_id[1]
233
+
234
+ def get(self):
235
+ """Read value from cache and return it."""
236
+ try:
237
+ return self.store_backend.load_item(
238
+ self._call_id,
239
+ timestamp=self.timestamp,
240
+ metadata=self.metadata,
241
+ verbose=self.verbose,
242
+ )
243
+ except ValueError as exc:
244
+ new_exc = KeyError(
245
+ "Error while trying to load a MemorizedResult's value. "
246
+ "It seems that this folder is corrupted : {}".format(
247
+ os.path.join(self.store_backend.location, *self._call_id)
248
+ )
249
+ )
250
+ raise new_exc from exc
251
+
252
+ def clear(self):
253
+ """Clear value from cache"""
254
+ self.store_backend.clear_item(self._call_id)
255
+
256
+ def __repr__(self):
257
+ return '{}(location="{}", func="{}", args_id="{}")'.format(
258
+ self.__class__.__name__, self.store_backend.location, *self._call_id
259
+ )
260
+
261
+ def __getstate__(self):
262
+ state = self.__dict__.copy()
263
+ state["timestamp"] = None
264
+ return state
265
+
266
+
267
+ class NotMemorizedResult(object):
268
+ """Class representing an arbitrary value.
269
+
270
+ This class is a replacement for MemorizedResult when there is no cache.
271
+ """
272
+
273
+ __slots__ = ("value", "valid")
274
+
275
+ def __init__(self, value):
276
+ self.value = value
277
+ self.valid = True
278
+
279
+ def get(self):
280
+ if self.valid:
281
+ return self.value
282
+ else:
283
+ raise KeyError("No value stored.")
284
+
285
+ def clear(self):
286
+ self.valid = False
287
+ self.value = None
288
+
289
+ def __repr__(self):
290
+ if self.valid:
291
+ return "{class_name}({value})".format(
292
+ class_name=self.__class__.__name__, value=pformat(self.value)
293
+ )
294
+ else:
295
+ return self.__class__.__name__ + " with no value"
296
+
297
+ # __getstate__ and __setstate__ are required because of __slots__
298
+ def __getstate__(self):
299
+ return {"valid": self.valid, "value": self.value}
300
+
301
+ def __setstate__(self, state):
302
+ self.valid = state["valid"]
303
+ self.value = state["value"]
304
+
305
+
306
+ ###############################################################################
307
+ # class `NotMemorizedFunc`
308
+ ###############################################################################
309
+ class NotMemorizedFunc(object):
310
+ """No-op object decorating a function.
311
+
312
+ This class replaces MemorizedFunc when there is no cache. It provides an
313
+ identical API but does not write anything on disk.
314
+
315
+ Attributes
316
+ ----------
317
+ func: callable
318
+ Original undecorated function.
319
+ """
320
+
321
+ # Should be a light as possible (for speed)
322
+ def __init__(self, func):
323
+ self.func = func
324
+
325
+ def __call__(self, *args, **kwargs):
326
+ return self.func(*args, **kwargs)
327
+
328
+ def call_and_shelve(self, *args, **kwargs):
329
+ return NotMemorizedResult(self.func(*args, **kwargs))
330
+
331
+ def __repr__(self):
332
+ return "{0}(func={1})".format(self.__class__.__name__, self.func)
333
+
334
+ def clear(self, warn=True):
335
+ # Argument "warn" is for compatibility with MemorizedFunc.clear
336
+ pass
337
+
338
+ def call(self, *args, **kwargs):
339
+ return self.func(*args, **kwargs), {}
340
+
341
+ def check_call_in_cache(self, *args, **kwargs):
342
+ return False
343
+
344
+
345
+ ###############################################################################
346
+ # class `AsyncNotMemorizedFunc`
347
+ ###############################################################################
348
+ class AsyncNotMemorizedFunc(NotMemorizedFunc):
349
+ async def call_and_shelve(self, *args, **kwargs):
350
+ return NotMemorizedResult(await self.func(*args, **kwargs))
351
+
352
+
353
+ ###############################################################################
354
+ # class `MemorizedFunc`
355
+ ###############################################################################
356
+ class MemorizedFunc(Logger):
357
+ """Callable object decorating a function for caching its return value
358
+ each time it is called.
359
+
360
+ Methods are provided to inspect the cache or clean it.
361
+
362
+ Attributes
363
+ ----------
364
+ func: callable
365
+ The original, undecorated, function.
366
+
367
+ location: string
368
+ The location of joblib cache. Depends on the store backend used.
369
+
370
+ backend: str
371
+ Type of store backend for reading/writing cache files.
372
+ Default is 'local', in which case the location is the path to a
373
+ disk storage.
374
+
375
+ ignore: list or None
376
+ List of variable names to ignore when choosing whether to
377
+ recompute.
378
+
379
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
380
+ The memmapping mode used when loading from cache
381
+ numpy arrays. See numpy.load for the meaning of the different
382
+ values.
383
+
384
+ compress: boolean, or integer
385
+ Whether to zip the stored data on disk. If an integer is
386
+ given, it should be between 1 and 9, and sets the amount
387
+ of compression. Note that compressed arrays cannot be
388
+ read by memmapping.
389
+
390
+ verbose: int, optional
391
+ The verbosity flag, controls messages that are issued as
392
+ the function is evaluated.
393
+
394
+ cache_validation_callback: callable, optional
395
+ Callable to check if a result in cache is valid or is to be recomputed.
396
+ When the function is called with arguments for which a cache exists,
397
+ the callback is called with the cache entry's metadata as its sole
398
+ argument. If it returns True, the cached result is returned, else the
399
+ cache for these arguments is cleared and the result is recomputed.
400
+ """
401
+
402
+ # ------------------------------------------------------------------------
403
+ # Public interface
404
+ # ------------------------------------------------------------------------
405
+
406
+ def __init__(
407
+ self,
408
+ func,
409
+ location,
410
+ backend="local",
411
+ ignore=None,
412
+ mmap_mode=None,
413
+ compress=False,
414
+ verbose=1,
415
+ timestamp=None,
416
+ cache_validation_callback=None,
417
+ ):
418
+ Logger.__init__(self)
419
+ self.mmap_mode = mmap_mode
420
+ self.compress = compress
421
+ self.func = func
422
+ self.cache_validation_callback = cache_validation_callback
423
+ self.func_id = _build_func_identifier(func)
424
+ self.ignore = ignore if ignore is not None else []
425
+ self._verbose = verbose
426
+
427
+ # retrieve store object from backend type and location.
428
+ self.store_backend = _store_backend_factory(
429
+ backend,
430
+ location,
431
+ verbose=verbose,
432
+ backend_options=dict(compress=compress, mmap_mode=mmap_mode),
433
+ )
434
+ if self.store_backend is not None:
435
+ # Create func directory on demand.
436
+ self.store_backend.store_cached_func_code([self.func_id])
437
+
438
+ self.timestamp = timestamp if timestamp is not None else time.time()
439
+ try:
440
+ functools.update_wrapper(self, func)
441
+ except Exception:
442
+ pass # Objects like ufunc don't like that
443
+ if inspect.isfunction(func):
444
+ doc = pydoc.TextDoc().document(func)
445
+ # Remove blank line
446
+ doc = doc.replace("\n", "\n\n", 1)
447
+ # Strip backspace-overprints for compatibility with autodoc
448
+ doc = re.sub("\x08.", "", doc)
449
+ else:
450
+ # Pydoc does a poor job on other objects
451
+ doc = func.__doc__
452
+ self.__doc__ = "Memoized version of %s" % doc
453
+
454
+ self._func_code_info = None
455
+ self._func_code_id = None
456
+
457
+ def _is_in_cache_and_valid(self, call_id):
458
+ """Check if the function call is cached and valid for given arguments.
459
+
460
+ - Compare the function code with the one from the cached function,
461
+ asserting if it has changed.
462
+ - Check if the function call is present in the cache.
463
+ - Call `cache_validation_callback` for user define cache validation.
464
+
465
+ Returns True if the function call is in cache and can be used, and
466
+ returns False otherwise.
467
+ """
468
+ # Check if the code of the function has changed
469
+ if not self._check_previous_func_code(stacklevel=4):
470
+ return False
471
+
472
+ # Check if this specific call is in the cache
473
+ if not self.store_backend.contains_item(call_id):
474
+ return False
475
+
476
+ # Call the user defined cache validation callback
477
+ metadata = self.store_backend.get_metadata(call_id)
478
+ if (
479
+ self.cache_validation_callback is not None
480
+ and not self.cache_validation_callback(metadata)
481
+ ):
482
+ self.store_backend.clear_item(call_id)
483
+ return False
484
+
485
+ return True
486
+
487
+ def _cached_call(self, args, kwargs, shelving):
488
+ """Call wrapped function and cache result, or read cache if available.
489
+
490
+ This function returns the wrapped function output or a reference to
491
+ the cached result.
492
+
493
+ Arguments:
494
+ ----------
495
+
496
+ args, kwargs: list and dict
497
+ input arguments for wrapped function
498
+
499
+ shelving: bool
500
+ True when called via the call_and_shelve function.
501
+
502
+
503
+ Returns
504
+ -------
505
+ output: Output of the wrapped function if shelving is false, or a
506
+ MemorizedResult reference to the value if shelving is true.
507
+ metadata: dict containing the metadata associated with the call.
508
+ """
509
+ args_id = self._get_args_id(*args, **kwargs)
510
+ call_id = (self.func_id, args_id)
511
+ _, func_name = get_func_name(self.func)
512
+ func_info = self.store_backend.get_cached_func_info([self.func_id])
513
+ location = func_info["location"]
514
+
515
+ if self._verbose >= 20:
516
+ logging.basicConfig(level=logging.INFO)
517
+ _, signature = format_signature(self.func, *args, **kwargs)
518
+ self.info(
519
+ textwrap.dedent(
520
+ f"""
521
+ Querying {func_name} with signature
522
+ {signature}.
523
+
524
+ (argument hash {args_id})
525
+
526
+ The store location is {location}.
527
+ """
528
+ )
529
+ )
530
+
531
+ # Compare the function code with the previous to see if the
532
+ # function code has changed and check if the results are present in
533
+ # the cache.
534
+ if self._is_in_cache_and_valid(call_id):
535
+ if shelving:
536
+ return self._get_memorized_result(call_id), {}
537
+
538
+ try:
539
+ start_time = time.time()
540
+ output = self._load_item(call_id)
541
+ if self._verbose > 4:
542
+ self._print_duration(
543
+ time.time() - start_time, context="cache loaded "
544
+ )
545
+ return output, {}
546
+ except Exception:
547
+ # XXX: Should use an exception logger
548
+ _, signature = format_signature(self.func, *args, **kwargs)
549
+ self.warn(
550
+ "Exception while loading results for {}\n {}".format(
551
+ signature, traceback.format_exc()
552
+ )
553
+ )
554
+
555
+ if self._verbose > 10:
556
+ self.warn(
557
+ f"Computing func {func_name}, argument hash {args_id} "
558
+ f"in location {location}"
559
+ )
560
+
561
+ # Returns the output but not the metadata
562
+ return self._call(call_id, args, kwargs, shelving)
563
+
564
+ @property
565
+ def func_code_info(self):
566
+ # 3-tuple property containing: the function source code, source file,
567
+ # and first line of the code inside the source file
568
+ if hasattr(self.func, "__code__"):
569
+ if self._func_code_id is None:
570
+ self._func_code_id = id(self.func.__code__)
571
+ elif id(self.func.__code__) != self._func_code_id:
572
+ # Be robust to dynamic reassignments of self.func.__code__
573
+ self._func_code_info = None
574
+
575
+ if self._func_code_info is None:
576
+ # Cache the source code of self.func . Provided that get_func_code
577
+ # (which should be called once on self) gets called in the process
578
+ # in which self.func was defined, this caching mechanism prevents
579
+ # undesired cache clearing when the cached function is called in
580
+ # an environment where the introspection utilities get_func_code
581
+ # relies on do not work (typically, in joblib child processes).
582
+ # See #1035 for more info
583
+ # TODO (pierreglaser): do the same with get_func_name?
584
+ self._func_code_info = get_func_code(self.func)
585
+ return self._func_code_info
586
+
587
+ def call_and_shelve(self, *args, **kwargs):
588
+ """Call wrapped function, cache result and return a reference.
589
+
590
+ This method returns a reference to the cached result instead of the
591
+ result itself. The reference object is small and picklable, allowing
592
+ to send or store it easily. Call .get() on reference object to get
593
+ result.
594
+
595
+ Returns
596
+ -------
597
+ cached_result: MemorizedResult or NotMemorizedResult
598
+ reference to the value returned by the wrapped function. The
599
+ class "NotMemorizedResult" is used when there is no cache
600
+ activated (e.g. location=None in Memory).
601
+ """
602
+ # Return the wrapped output, without the metadata
603
+ return self._cached_call(args, kwargs, shelving=True)[0]
604
+
605
+ def __call__(self, *args, **kwargs):
606
+ # Return the output, without the metadata
607
+ return self._cached_call(args, kwargs, shelving=False)[0]
608
+
609
+ def __getstate__(self):
610
+ # Make sure self.func's source is introspected prior to being pickled -
611
+ # code introspection utilities typically do not work inside child
612
+ # processes
613
+ _ = self.func_code_info
614
+
615
+ # We don't store the timestamp when pickling, to avoid the hash
616
+ # depending from it.
617
+ state = self.__dict__.copy()
618
+ state["timestamp"] = None
619
+
620
+ # Invalidate the code id as id(obj) will be different in the child
621
+ state["_func_code_id"] = None
622
+
623
+ return state
624
+
625
+ def check_call_in_cache(self, *args, **kwargs):
626
+ """Check if the function call is cached and valid for given arguments.
627
+
628
+ Does not call the function or do any work besides function inspection
629
+ and argument hashing.
630
+
631
+ - Compare the function code with the one from the cached function,
632
+ asserting if it has changed.
633
+ - Check if the function call is present in the cache.
634
+ - Call `cache_validation_callback` for user define cache validation.
635
+
636
+ Returns
637
+ -------
638
+ is_call_in_cache: bool
639
+ Whether or not the function call is in cache and can be used.
640
+ """
641
+ call_id = (self.func_id, self._get_args_id(*args, **kwargs))
642
+ return self._is_in_cache_and_valid(call_id)
643
+
644
+ # ------------------------------------------------------------------------
645
+ # Private interface
646
+ # ------------------------------------------------------------------------
647
+
648
+ def _get_args_id(self, *args, **kwargs):
649
+ """Return the input parameter hash of a result."""
650
+ return hashing.hash(
651
+ filter_args(self.func, self.ignore, args, kwargs),
652
+ coerce_mmap=self.mmap_mode is not None,
653
+ )
654
+
655
+ def _hash_func(self):
656
+ """Hash a function to key the online cache"""
657
+ func_code_h = hash(getattr(self.func, "__code__", None))
658
+ return id(self.func), hash(self.func), func_code_h
659
+
660
+ def _write_func_code(self, func_code, first_line):
661
+ """Write the function code and the filename to a file."""
662
+ # We store the first line because the filename and the function
663
+ # name is not always enough to identify a function: people
664
+ # sometimes have several functions named the same way in a
665
+ # file. This is bad practice, but joblib should be robust to bad
666
+ # practice.
667
+ func_code = "%s %i\n%s" % (FIRST_LINE_TEXT, first_line, func_code)
668
+ self.store_backend.store_cached_func_code([self.func_id], func_code)
669
+
670
+ # Also store in the in-memory store of function hashes
671
+ is_named_callable = (
672
+ hasattr(self.func, "__name__") and self.func.__name__ != "<lambda>"
673
+ )
674
+ if is_named_callable:
675
+ # Don't do this for lambda functions or strange callable
676
+ # objects, as it ends up being too fragile
677
+ func_hash = self._hash_func()
678
+ try:
679
+ _FUNCTION_HASHES[self.func] = func_hash
680
+ except TypeError:
681
+ # Some callable are not hashable
682
+ pass
683
+
684
+ def _check_previous_func_code(self, stacklevel=2):
685
+ """
686
+ stacklevel is the depth a which this function is called, to
687
+ issue useful warnings to the user.
688
+ """
689
+ # First check if our function is in the in-memory store.
690
+ # Using the in-memory store not only makes things faster, but it
691
+ # also renders us robust to variations of the files when the
692
+ # in-memory version of the code does not vary
693
+ try:
694
+ if self.func in _FUNCTION_HASHES:
695
+ # We use as an identifier the id of the function and its
696
+ # hash. This is more likely to falsely change than have hash
697
+ # collisions, thus we are on the safe side.
698
+ func_hash = self._hash_func()
699
+ if func_hash == _FUNCTION_HASHES[self.func]:
700
+ return True
701
+ except TypeError:
702
+ # Some callables are not hashable
703
+ pass
704
+
705
+ # Here, we go through some effort to be robust to dynamically
706
+ # changing code and collision. We cannot inspect.getsource
707
+ # because it is not reliable when using IPython's magic "%run".
708
+ func_code, source_file, first_line = self.func_code_info
709
+ try:
710
+ old_func_code, old_first_line = extract_first_line(
711
+ self.store_backend.get_cached_func_code([self.func_id])
712
+ )
713
+ except (IOError, OSError): # some backend can also raise OSError
714
+ self._write_func_code(func_code, first_line)
715
+ return False
716
+ if old_func_code == func_code:
717
+ return True
718
+
719
+ # We have differing code, is this because we are referring to
720
+ # different functions, or because the function we are referring to has
721
+ # changed?
722
+
723
+ _, func_name = get_func_name(
724
+ self.func, resolv_alias=False, win_characters=False
725
+ )
726
+ if old_first_line == first_line == -1 or func_name == "<lambda>":
727
+ if not first_line == -1:
728
+ func_description = "{0} ({1}:{2})".format(
729
+ func_name, source_file, first_line
730
+ )
731
+ else:
732
+ func_description = func_name
733
+ warnings.warn(
734
+ JobLibCollisionWarning(
735
+ "Cannot detect name collisions for function '{0}'".format(
736
+ func_description
737
+ )
738
+ ),
739
+ stacklevel=stacklevel,
740
+ )
741
+
742
+ # Fetch the code at the old location and compare it. If it is the
743
+ # same than the code store, we have a collision: the code in the
744
+ # file has not changed, but the name we have is pointing to a new
745
+ # code block.
746
+ if not old_first_line == first_line and source_file is not None:
747
+ if os.path.exists(source_file):
748
+ _, func_name = get_func_name(self.func, resolv_alias=False)
749
+ num_lines = len(func_code.split("\n"))
750
+ with tokenize.open(source_file) as f:
751
+ on_disk_func_code = f.readlines()[
752
+ old_first_line - 1 : old_first_line - 1 + num_lines - 1
753
+ ]
754
+ on_disk_func_code = "".join(on_disk_func_code)
755
+ possible_collision = (
756
+ on_disk_func_code.rstrip() == old_func_code.rstrip()
757
+ )
758
+ else:
759
+ possible_collision = source_file.startswith("<doctest ")
760
+ if possible_collision:
761
+ warnings.warn(
762
+ JobLibCollisionWarning(
763
+ "Possible name collisions between functions "
764
+ "'%s' (%s:%i) and '%s' (%s:%i)"
765
+ % (
766
+ func_name,
767
+ source_file,
768
+ old_first_line,
769
+ func_name,
770
+ source_file,
771
+ first_line,
772
+ )
773
+ ),
774
+ stacklevel=stacklevel,
775
+ )
776
+
777
+ # The function has changed, wipe the cache directory.
778
+ # XXX: Should be using warnings, and giving stacklevel
779
+ if self._verbose > 10:
780
+ _, func_name = get_func_name(self.func, resolv_alias=False)
781
+ self.warn(
782
+ "Function {0} (identified by {1}) has changed.".format(
783
+ func_name, self.func_id
784
+ )
785
+ )
786
+ self.clear(warn=True)
787
+ return False
788
+
789
+ def clear(self, warn=True):
790
+ """Empty the function's cache."""
791
+ func_id = self.func_id
792
+ if self._verbose > 0 and warn:
793
+ self.warn("Clearing function cache identified by %s" % func_id)
794
+ self.store_backend.clear_path(
795
+ [
796
+ func_id,
797
+ ]
798
+ )
799
+
800
+ func_code, _, first_line = self.func_code_info
801
+ self._write_func_code(func_code, first_line)
802
+
803
+ def call(self, *args, **kwargs):
804
+ """Force the execution of the function with the given arguments.
805
+
806
+ The output values will be persisted, i.e., the cache will be updated
807
+ with any new values.
808
+
809
+ Parameters
810
+ ----------
811
+ *args: arguments
812
+ The arguments.
813
+ **kwargs: keyword arguments
814
+ Keyword arguments.
815
+
816
+ Returns
817
+ -------
818
+ output : object
819
+ The output of the function call.
820
+ metadata : dict
821
+ The metadata associated with the call.
822
+ """
823
+ call_id = (self.func_id, self._get_args_id(*args, **kwargs))
824
+
825
+ # Return the output and the metadata
826
+ return self._call(call_id, args, kwargs)
827
+
828
+ def _call(self, call_id, args, kwargs, shelving=False):
829
+ # Return the output and the metadata
830
+ self._before_call(args, kwargs)
831
+ start_time = time.time()
832
+ output = self.func(*args, **kwargs)
833
+ return self._after_call(call_id, args, kwargs, shelving, output, start_time)
834
+
835
+ def _before_call(self, args, kwargs):
836
+ if self._verbose > 0:
837
+ print(format_call(self.func, args, kwargs))
838
+
839
+ def _after_call(self, call_id, args, kwargs, shelving, output, start_time):
840
+ self.store_backend.dump_item(call_id, output, verbose=self._verbose)
841
+ duration = time.time() - start_time
842
+ if self._verbose > 0:
843
+ self._print_duration(duration)
844
+ metadata = self._persist_input(duration, call_id, args, kwargs)
845
+ if shelving:
846
+ return self._get_memorized_result(call_id, metadata), metadata
847
+
848
+ if self.mmap_mode is not None:
849
+ # Memmap the output at the first call to be consistent with
850
+ # later calls
851
+ output = self._load_item(call_id, metadata)
852
+ return output, metadata
853
+
854
+ def _persist_input(self, duration, call_id, args, kwargs, this_duration_limit=0.5):
855
+ """Save a small summary of the call using json format in the
856
+ output directory.
857
+
858
+ output_dir: string
859
+ directory where to write metadata.
860
+
861
+ duration: float
862
+ time taken by hashing input arguments, calling the wrapped
863
+ function and persisting its output.
864
+
865
+ args, kwargs: list and dict
866
+ input arguments for wrapped function
867
+
868
+ this_duration_limit: float
869
+ Max execution time for this function before issuing a warning.
870
+ """
871
+ start_time = time.time()
872
+ argument_dict = filter_args(self.func, self.ignore, args, kwargs)
873
+
874
+ input_repr = dict((k, repr(v)) for k, v in argument_dict.items())
875
+ # This can fail due to race-conditions with multiple
876
+ # concurrent joblibs removing the file or the directory
877
+ metadata = {
878
+ "duration": duration,
879
+ "input_args": input_repr,
880
+ "time": start_time,
881
+ }
882
+
883
+ self.store_backend.store_metadata(call_id, metadata)
884
+
885
+ this_duration = time.time() - start_time
886
+ if this_duration > this_duration_limit:
887
+ # This persistence should be fast. It will not be if repr() takes
888
+ # time and its output is large, because json.dump will have to
889
+ # write a large file. This should not be an issue with numpy arrays
890
+ # for which repr() always output a short representation, but can
891
+ # be with complex dictionaries. Fixing the problem should be a
892
+ # matter of replacing repr() above by something smarter.
893
+ warnings.warn(
894
+ "Persisting input arguments took %.2fs to run."
895
+ "If this happens often in your code, it can cause "
896
+ "performance problems "
897
+ "(results will be correct in all cases). "
898
+ "The reason for this is probably some large input "
899
+ "arguments for a wrapped function." % this_duration,
900
+ stacklevel=5,
901
+ )
902
+ return metadata
903
+
904
+ def _get_memorized_result(self, call_id, metadata=None):
905
+ return MemorizedResult(
906
+ self.store_backend,
907
+ call_id,
908
+ metadata=metadata,
909
+ timestamp=self.timestamp,
910
+ verbose=self._verbose - 1,
911
+ )
912
+
913
+ def _load_item(self, call_id, metadata=None):
914
+ return self.store_backend.load_item(
915
+ call_id, metadata=metadata, timestamp=self.timestamp, verbose=self._verbose
916
+ )
917
+
918
+ def _print_duration(self, duration, context=""):
919
+ _, name = get_func_name(self.func)
920
+ msg = f"{name} {context}- {format_time(duration)}"
921
+ print(max(0, (80 - len(msg))) * "_" + msg)
922
+
923
+ # ------------------------------------------------------------------------
924
+ # Private `object` interface
925
+ # ------------------------------------------------------------------------
926
+
927
+ def __repr__(self):
928
+ return "{class_name}(func={func}, location={location})".format(
929
+ class_name=self.__class__.__name__,
930
+ func=self.func,
931
+ location=self.store_backend.location,
932
+ )
933
+
934
+
935
+ ###############################################################################
936
+ # class `AsyncMemorizedFunc`
937
+ ###############################################################################
938
+ class AsyncMemorizedFunc(MemorizedFunc):
939
+ async def __call__(self, *args, **kwargs):
940
+ out = self._cached_call(args, kwargs, shelving=False)
941
+ out = await out if asyncio.iscoroutine(out) else out
942
+ return out[0] # Don't return metadata
943
+
944
+ async def call_and_shelve(self, *args, **kwargs):
945
+ out = self._cached_call(args, kwargs, shelving=True)
946
+ out = await out if asyncio.iscoroutine(out) else out
947
+ return out[0] # Don't return metadata
948
+
949
+ async def call(self, *args, **kwargs):
950
+ out = super().call(*args, **kwargs)
951
+ return await out if asyncio.iscoroutine(out) else out
952
+
953
+ async def _call(self, call_id, args, kwargs, shelving=False):
954
+ self._before_call(args, kwargs)
955
+ start_time = time.time()
956
+ output = await self.func(*args, **kwargs)
957
+ return self._after_call(call_id, args, kwargs, shelving, output, start_time)
958
+
959
+
960
+ ###############################################################################
961
+ # class `Memory`
962
+ ###############################################################################
963
+ class Memory(Logger):
964
+ """A context object for caching a function's return value each time it
965
+ is called with the same input arguments.
966
+
967
+ All values are cached on the filesystem, in a deep directory
968
+ structure.
969
+
970
+ Read more in the :ref:`User Guide <memory>`.
971
+
972
+ Parameters
973
+ ----------
974
+ location: str, pathlib.Path or None
975
+ The path of the base directory to use as a data store
976
+ or None. If None is given, no caching is done and
977
+ the Memory object is completely transparent. This option
978
+ replaces cachedir since version 0.12.
979
+
980
+ backend: str, optional, default='local'
981
+ Type of store backend for reading/writing cache files.
982
+ The 'local' backend is using regular filesystem operations to
983
+ manipulate data (open, mv, etc) in the backend.
984
+
985
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
986
+ The memmapping mode used when loading from cache
987
+ numpy arrays. See numpy.load for the meaning of the
988
+ arguments.
989
+
990
+ compress: boolean, or integer, optional
991
+ Whether to zip the stored data on disk. If an integer is
992
+ given, it should be between 1 and 9, and sets the amount
993
+ of compression. Note that compressed arrays cannot be
994
+ read by memmapping.
995
+
996
+ verbose: int, optional
997
+ Verbosity flag, controls the debug messages that are issued
998
+ as functions are evaluated.
999
+
1000
+ backend_options: dict, optional
1001
+ Contains a dictionary of named parameters used to configure
1002
+ the store backend.
1003
+ """
1004
+
1005
+ # ------------------------------------------------------------------------
1006
+ # Public interface
1007
+ # ------------------------------------------------------------------------
1008
+
1009
+ def __init__(
1010
+ self,
1011
+ location=None,
1012
+ backend="local",
1013
+ mmap_mode=None,
1014
+ compress=False,
1015
+ verbose=1,
1016
+ backend_options=None,
1017
+ ):
1018
+ Logger.__init__(self)
1019
+ self._verbose = verbose
1020
+ self.mmap_mode = mmap_mode
1021
+ self.timestamp = time.time()
1022
+ self.backend = backend
1023
+ self.compress = compress
1024
+ if backend_options is None:
1025
+ backend_options = {}
1026
+ self.backend_options = backend_options
1027
+
1028
+ if compress and mmap_mode is not None:
1029
+ warnings.warn("Compressed results cannot be memmapped", stacklevel=2)
1030
+
1031
+ self.location = location
1032
+ if isinstance(location, str):
1033
+ location = os.path.join(location, "joblib")
1034
+
1035
+ self.store_backend = _store_backend_factory(
1036
+ backend,
1037
+ location,
1038
+ verbose=self._verbose,
1039
+ backend_options=dict(
1040
+ compress=compress, mmap_mode=mmap_mode, **backend_options
1041
+ ),
1042
+ )
1043
+
1044
+ def cache(
1045
+ self,
1046
+ func=None,
1047
+ ignore=None,
1048
+ verbose=None,
1049
+ mmap_mode=False,
1050
+ cache_validation_callback=None,
1051
+ ):
1052
+ """Decorates the given function func to only compute its return
1053
+ value for input arguments not cached on disk.
1054
+
1055
+ Parameters
1056
+ ----------
1057
+ func: callable, optional
1058
+ The function to be decorated
1059
+ ignore: list of strings
1060
+ A list of arguments name to ignore in the hashing
1061
+ verbose: integer, optional
1062
+ The verbosity mode of the function. By default that
1063
+ of the memory object is used.
1064
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
1065
+ The memmapping mode used when loading from cache
1066
+ numpy arrays. See numpy.load for the meaning of the
1067
+ arguments. By default that of the memory object is used.
1068
+ cache_validation_callback: callable, optional
1069
+ Callable to validate whether or not the cache is valid. When
1070
+ the cached function is called with arguments for which a cache
1071
+ exists, this callable is called with the metadata of the cached
1072
+ result as its sole argument. If it returns True, then the
1073
+ cached result is returned, else the cache for these arguments
1074
+ is cleared and recomputed.
1075
+
1076
+ Returns
1077
+ -------
1078
+ decorated_func: MemorizedFunc object
1079
+ The returned object is a MemorizedFunc object, that is
1080
+ callable (behaves like a function), but offers extra
1081
+ methods for cache lookup and management. See the
1082
+ documentation for :class:`joblib.memory.MemorizedFunc`.
1083
+ """
1084
+ if cache_validation_callback is not None and not callable(
1085
+ cache_validation_callback
1086
+ ):
1087
+ raise ValueError(
1088
+ "cache_validation_callback needs to be callable. "
1089
+ f"Got {cache_validation_callback}."
1090
+ )
1091
+ if func is None:
1092
+ # Partial application, to be able to specify extra keyword
1093
+ # arguments in decorators
1094
+ return functools.partial(
1095
+ self.cache,
1096
+ ignore=ignore,
1097
+ mmap_mode=mmap_mode,
1098
+ verbose=verbose,
1099
+ cache_validation_callback=cache_validation_callback,
1100
+ )
1101
+ if self.store_backend is None:
1102
+ cls = (
1103
+ AsyncNotMemorizedFunc
1104
+ if inspect.iscoroutinefunction(func)
1105
+ else NotMemorizedFunc
1106
+ )
1107
+ return cls(func)
1108
+ if verbose is None:
1109
+ verbose = self._verbose
1110
+ if mmap_mode is False:
1111
+ mmap_mode = self.mmap_mode
1112
+ if isinstance(func, MemorizedFunc):
1113
+ func = func.func
1114
+ cls = AsyncMemorizedFunc if inspect.iscoroutinefunction(func) else MemorizedFunc
1115
+ return cls(
1116
+ func,
1117
+ location=self.store_backend,
1118
+ backend=self.backend,
1119
+ ignore=ignore,
1120
+ mmap_mode=mmap_mode,
1121
+ compress=self.compress,
1122
+ verbose=verbose,
1123
+ timestamp=self.timestamp,
1124
+ cache_validation_callback=cache_validation_callback,
1125
+ )
1126
+
1127
+ def clear(self, warn=True):
1128
+ """Erase the complete cache directory."""
1129
+ if warn:
1130
+ self.warn("Flushing completely the cache")
1131
+ if self.store_backend is not None:
1132
+ self.store_backend.clear()
1133
+
1134
+ # As the cache is completely clear, make sure the _FUNCTION_HASHES
1135
+ # cache is also reset. Else, for a function that is present in this
1136
+ # table, results cached after this clear will be have cache miss
1137
+ # as the function code is not re-written.
1138
+ _FUNCTION_HASHES.clear()
1139
+
1140
+ def reduce_size(self, bytes_limit=None, items_limit=None, age_limit=None):
1141
+ """Remove cache elements to make the cache fit its limits.
1142
+
1143
+ The limitation can impose that the cache size fits in ``bytes_limit``,
1144
+ that the number of cache items is no more than ``items_limit``, and
1145
+ that all files in cache are not older than ``age_limit``.
1146
+
1147
+ Parameters
1148
+ ----------
1149
+ bytes_limit: int | str, optional
1150
+ Limit in bytes of the size of the cache. By default, the size of
1151
+ the cache is unlimited. When reducing the size of the cache,
1152
+ ``joblib`` keeps the most recently accessed items first. If a
1153
+ str is passed, it is converted to a number of bytes using units
1154
+ { K | M | G} for kilo, mega, giga.
1155
+
1156
+ items_limit: int, optional
1157
+ Number of items to limit the cache to. By default, the number of
1158
+ items in the cache is unlimited. When reducing the size of the
1159
+ cache, ``joblib`` keeps the most recently accessed items first.
1160
+
1161
+ age_limit: datetime.timedelta, optional
1162
+ Maximum age of items to limit the cache to. When reducing the size
1163
+ of the cache, any items last accessed more than the given length of
1164
+ time ago are deleted. Example: to remove files older than 5 days,
1165
+ use datetime.timedelta(days=5). Negative timedelta are not
1166
+ accepted.
1167
+ """
1168
+ if self.store_backend is None:
1169
+ # No cached results, this function does nothing.
1170
+ return
1171
+
1172
+ if bytes_limit is None and items_limit is None and age_limit is None:
1173
+ # No limitation to impose, returning
1174
+ return
1175
+
1176
+ # Defers the actual limits enforcing to the store backend.
1177
+ self.store_backend.enforce_store_limits(bytes_limit, items_limit, age_limit)
1178
+
1179
+ def eval(self, func, *args, **kwargs):
1180
+ """Eval function func with arguments `*args` and `**kwargs`,
1181
+ in the context of the memory.
1182
+
1183
+ This method works similarly to the builtin `apply`, except
1184
+ that the function is called only if the cache is not
1185
+ up to date.
1186
+
1187
+ """
1188
+ if self.store_backend is None:
1189
+ return func(*args, **kwargs)
1190
+ return self.cache(func)(*args, **kwargs)
1191
+
1192
+ # ------------------------------------------------------------------------
1193
+ # Private `object` interface
1194
+ # ------------------------------------------------------------------------
1195
+
1196
+ def __repr__(self):
1197
+ return "{class_name}(location={location})".format(
1198
+ class_name=self.__class__.__name__,
1199
+ location=(
1200
+ None if self.store_backend is None else self.store_backend.location
1201
+ ),
1202
+ )
1203
+
1204
+ def __getstate__(self):
1205
+ """We don't store the timestamp when pickling, to avoid the hash
1206
+ depending from it.
1207
+ """
1208
+ state = self.__dict__.copy()
1209
+ state["timestamp"] = None
1210
+ return state
1211
+
1212
+
1213
+ ###############################################################################
1214
+ # cache_validation_callback helpers
1215
+ ###############################################################################
1216
+
1217
+
1218
+ def expires_after(
1219
+ days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0
1220
+ ):
1221
+ """Helper cache_validation_callback to force recompute after a duration.
1222
+
1223
+ Parameters
1224
+ ----------
1225
+ days, seconds, microseconds, milliseconds, minutes, hours, weeks: numbers
1226
+ argument passed to a timedelta.
1227
+ """
1228
+ delta = datetime.timedelta(
1229
+ days=days,
1230
+ seconds=seconds,
1231
+ microseconds=microseconds,
1232
+ milliseconds=milliseconds,
1233
+ minutes=minutes,
1234
+ hours=hours,
1235
+ weeks=weeks,
1236
+ )
1237
+
1238
+ def cache_validation_callback(metadata):
1239
+ computation_age = time.time() - metadata["time"]
1240
+ return computation_age < delta.total_seconds()
1241
+
1242
+ return cache_validation_callback
TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle.py ADDED
@@ -0,0 +1,756 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Utilities for fast persistence of big data, with optional compression."""
2
+
3
+ # Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
4
+ # Copyright (c) 2009 Gael Varoquaux
5
+ # License: BSD Style, 3 clauses.
6
+
7
+ import io
8
+ import os
9
+ import pickle
10
+ import warnings
11
+ from pathlib import Path
12
+
13
+ from .backports import make_memmap
14
+ from .compressor import (
15
+ _COMPRESSORS,
16
+ LZ4_NOT_INSTALLED_ERROR,
17
+ BinaryZlibFile,
18
+ BZ2CompressorWrapper,
19
+ GzipCompressorWrapper,
20
+ LZ4CompressorWrapper,
21
+ LZMACompressorWrapper,
22
+ XZCompressorWrapper,
23
+ ZlibCompressorWrapper,
24
+ lz4,
25
+ register_compressor,
26
+ )
27
+
28
+ # For compatibility with old versions of joblib, we need ZNDArrayWrapper
29
+ # to be visible in the current namespace.
30
+ from .numpy_pickle_compat import (
31
+ NDArrayWrapper,
32
+ ZNDArrayWrapper, # noqa: F401
33
+ load_compatibility,
34
+ )
35
+ from .numpy_pickle_utils import (
36
+ BUFFER_SIZE,
37
+ Pickler,
38
+ Unpickler,
39
+ _ensure_native_byte_order,
40
+ _read_bytes,
41
+ _reconstruct,
42
+ _validate_fileobject_and_memmap,
43
+ _write_fileobject,
44
+ )
45
+
46
+ # Register supported compressors
47
+ register_compressor("zlib", ZlibCompressorWrapper())
48
+ register_compressor("gzip", GzipCompressorWrapper())
49
+ register_compressor("bz2", BZ2CompressorWrapper())
50
+ register_compressor("lzma", LZMACompressorWrapper())
51
+ register_compressor("xz", XZCompressorWrapper())
52
+ register_compressor("lz4", LZ4CompressorWrapper())
53
+
54
+
55
+ ###############################################################################
56
+ # Utility objects for persistence.
57
+
58
+ # For convenience, 16 bytes are used to be sure to cover all the possible
59
+ # dtypes' alignments. For reference, see:
60
+ # https://numpy.org/devdocs/dev/alignment.html
61
+ NUMPY_ARRAY_ALIGNMENT_BYTES = 16
62
+
63
+
64
+ class NumpyArrayWrapper(object):
65
+ """An object to be persisted instead of numpy arrays.
66
+
67
+ This object is used to hack into the pickle machinery and read numpy
68
+ array data from our custom persistence format.
69
+ More precisely, this object is used for:
70
+ * carrying the information of the persisted array: subclass, shape, order,
71
+ dtype. Those ndarray metadata are used to correctly reconstruct the array
72
+ with low level numpy functions.
73
+ * determining if memmap is allowed on the array.
74
+ * reading the array bytes from a file.
75
+ * reading the array using memorymap from a file.
76
+ * writing the array bytes to a file.
77
+
78
+ Attributes
79
+ ----------
80
+ subclass: numpy.ndarray subclass
81
+ Determine the subclass of the wrapped array.
82
+ shape: numpy.ndarray shape
83
+ Determine the shape of the wrapped array.
84
+ order: {'C', 'F'}
85
+ Determine the order of wrapped array data. 'C' is for C order, 'F' is
86
+ for fortran order.
87
+ dtype: numpy.ndarray dtype
88
+ Determine the data type of the wrapped array.
89
+ allow_mmap: bool
90
+ Determine if memory mapping is allowed on the wrapped array.
91
+ Default: False.
92
+ """
93
+
94
+ def __init__(
95
+ self,
96
+ subclass,
97
+ shape,
98
+ order,
99
+ dtype,
100
+ allow_mmap=False,
101
+ numpy_array_alignment_bytes=NUMPY_ARRAY_ALIGNMENT_BYTES,
102
+ ):
103
+ """Constructor. Store the useful information for later."""
104
+ self.subclass = subclass
105
+ self.shape = shape
106
+ self.order = order
107
+ self.dtype = dtype
108
+ self.allow_mmap = allow_mmap
109
+ # We make numpy_array_alignment_bytes an instance attribute to allow us
110
+ # to change our mind about the default alignment and still load the old
111
+ # pickles (with the previous alignment) correctly
112
+ self.numpy_array_alignment_bytes = numpy_array_alignment_bytes
113
+
114
+ def safe_get_numpy_array_alignment_bytes(self):
115
+ # NumpyArrayWrapper instances loaded from joblib <= 1.1 pickles don't
116
+ # have an numpy_array_alignment_bytes attribute
117
+ return getattr(self, "numpy_array_alignment_bytes", None)
118
+
119
+ def write_array(self, array, pickler):
120
+ """Write array bytes to pickler file handle.
121
+
122
+ This function is an adaptation of the numpy write_array function
123
+ available in version 1.10.1 in numpy/lib/format.py.
124
+ """
125
+ # Set buffer size to 16 MiB to hide the Python loop overhead.
126
+ buffersize = max(16 * 1024**2 // array.itemsize, 1)
127
+ if array.dtype.hasobject:
128
+ # We contain Python objects so we cannot write out the data
129
+ # directly. Instead, we will pickle it out with version 5 of the
130
+ # pickle protocol.
131
+ pickle.dump(array, pickler.file_handle, protocol=5)
132
+ else:
133
+ numpy_array_alignment_bytes = self.safe_get_numpy_array_alignment_bytes()
134
+ if numpy_array_alignment_bytes is not None:
135
+ current_pos = pickler.file_handle.tell()
136
+ pos_after_padding_byte = current_pos + 1
137
+ padding_length = numpy_array_alignment_bytes - (
138
+ pos_after_padding_byte % numpy_array_alignment_bytes
139
+ )
140
+ # A single byte is written that contains the padding length in
141
+ # bytes
142
+ padding_length_byte = int.to_bytes(
143
+ padding_length, length=1, byteorder="little"
144
+ )
145
+ pickler.file_handle.write(padding_length_byte)
146
+
147
+ if padding_length != 0:
148
+ padding = b"\xff" * padding_length
149
+ pickler.file_handle.write(padding)
150
+
151
+ for chunk in pickler.np.nditer(
152
+ array,
153
+ flags=["external_loop", "buffered", "zerosize_ok"],
154
+ buffersize=buffersize,
155
+ order=self.order,
156
+ ):
157
+ pickler.file_handle.write(chunk.tobytes("C"))
158
+
159
+ def read_array(self, unpickler, ensure_native_byte_order):
160
+ """Read array from unpickler file handle.
161
+
162
+ This function is an adaptation of the numpy read_array function
163
+ available in version 1.10.1 in numpy/lib/format.py.
164
+ """
165
+ if len(self.shape) == 0:
166
+ count = 1
167
+ else:
168
+ # joblib issue #859: we cast the elements of self.shape to int64 to
169
+ # prevent a potential overflow when computing their product.
170
+ shape_int64 = [unpickler.np.int64(x) for x in self.shape]
171
+ count = unpickler.np.multiply.reduce(shape_int64)
172
+ # Now read the actual data.
173
+ if self.dtype.hasobject:
174
+ # The array contained Python objects. We need to unpickle the data.
175
+ array = pickle.load(unpickler.file_handle)
176
+ else:
177
+ numpy_array_alignment_bytes = self.safe_get_numpy_array_alignment_bytes()
178
+ if numpy_array_alignment_bytes is not None:
179
+ padding_byte = unpickler.file_handle.read(1)
180
+ padding_length = int.from_bytes(padding_byte, byteorder="little")
181
+ if padding_length != 0:
182
+ unpickler.file_handle.read(padding_length)
183
+
184
+ # This is not a real file. We have to read it the
185
+ # memory-intensive way.
186
+ # crc32 module fails on reads greater than 2 ** 32 bytes,
187
+ # breaking large reads from gzip streams. Chunk reads to
188
+ # BUFFER_SIZE bytes to avoid issue and reduce memory overhead
189
+ # of the read. In non-chunked case count < max_read_count, so
190
+ # only one read is performed.
191
+ max_read_count = BUFFER_SIZE // min(BUFFER_SIZE, self.dtype.itemsize)
192
+
193
+ array = unpickler.np.empty(count, dtype=self.dtype)
194
+ for i in range(0, count, max_read_count):
195
+ read_count = min(max_read_count, count - i)
196
+ read_size = int(read_count * self.dtype.itemsize)
197
+ data = _read_bytes(unpickler.file_handle, read_size, "array data")
198
+ array[i : i + read_count] = unpickler.np.frombuffer(
199
+ data, dtype=self.dtype, count=read_count
200
+ )
201
+ del data
202
+
203
+ if self.order == "F":
204
+ array.shape = self.shape[::-1]
205
+ array = array.transpose()
206
+ else:
207
+ array.shape = self.shape
208
+
209
+ if ensure_native_byte_order:
210
+ # Detect byte order mismatch and swap as needed.
211
+ array = _ensure_native_byte_order(array)
212
+
213
+ return array
214
+
215
+ def read_mmap(self, unpickler):
216
+ """Read an array using numpy memmap."""
217
+ current_pos = unpickler.file_handle.tell()
218
+ offset = current_pos
219
+ numpy_array_alignment_bytes = self.safe_get_numpy_array_alignment_bytes()
220
+
221
+ if numpy_array_alignment_bytes is not None:
222
+ padding_byte = unpickler.file_handle.read(1)
223
+ padding_length = int.from_bytes(padding_byte, byteorder="little")
224
+ # + 1 is for the padding byte
225
+ offset += padding_length + 1
226
+
227
+ if unpickler.mmap_mode == "w+":
228
+ unpickler.mmap_mode = "r+"
229
+
230
+ marray = make_memmap(
231
+ unpickler.filename,
232
+ dtype=self.dtype,
233
+ shape=self.shape,
234
+ order=self.order,
235
+ mode=unpickler.mmap_mode,
236
+ offset=offset,
237
+ )
238
+ # update the offset so that it corresponds to the end of the read array
239
+ unpickler.file_handle.seek(offset + marray.nbytes)
240
+
241
+ if (
242
+ numpy_array_alignment_bytes is None
243
+ and current_pos % NUMPY_ARRAY_ALIGNMENT_BYTES != 0
244
+ ):
245
+ message = (
246
+ f"The memmapped array {marray} loaded from the file "
247
+ f"{unpickler.file_handle.name} is not byte aligned. "
248
+ "This may cause segmentation faults if this memmapped array "
249
+ "is used in some libraries like BLAS or PyTorch. "
250
+ "To get rid of this warning, regenerate your pickle file "
251
+ "with joblib >= 1.2.0. "
252
+ "See https://github.com/joblib/joblib/issues/563 "
253
+ "for more details"
254
+ )
255
+ warnings.warn(message)
256
+
257
+ return marray
258
+
259
+ def read(self, unpickler, ensure_native_byte_order):
260
+ """Read the array corresponding to this wrapper.
261
+
262
+ Use the unpickler to get all information to correctly read the array.
263
+
264
+ Parameters
265
+ ----------
266
+ unpickler: NumpyUnpickler
267
+ ensure_native_byte_order: bool
268
+ If true, coerce the array to use the native endianness of the
269
+ host system.
270
+
271
+ Returns
272
+ -------
273
+ array: numpy.ndarray
274
+
275
+ """
276
+ # When requested, only use memmap mode if allowed.
277
+ if unpickler.mmap_mode is not None and self.allow_mmap:
278
+ assert not ensure_native_byte_order, (
279
+ "Memmaps cannot be coerced to a given byte order, "
280
+ "this code path is impossible."
281
+ )
282
+ array = self.read_mmap(unpickler)
283
+ else:
284
+ array = self.read_array(unpickler, ensure_native_byte_order)
285
+
286
+ # Manage array subclass case
287
+ if hasattr(array, "__array_prepare__") and self.subclass not in (
288
+ unpickler.np.ndarray,
289
+ unpickler.np.memmap,
290
+ ):
291
+ # We need to reconstruct another subclass
292
+ new_array = _reconstruct(self.subclass, (0,), "b")
293
+ return new_array.__array_prepare__(array)
294
+ else:
295
+ return array
296
+
297
+
298
+ ###############################################################################
299
+ # Pickler classes
300
+
301
+
302
+ class NumpyPickler(Pickler):
303
+ """A pickler to persist big data efficiently.
304
+
305
+ The main features of this object are:
306
+ * persistence of numpy arrays in a single file.
307
+ * optional compression with a special care on avoiding memory copies.
308
+
309
+ Attributes
310
+ ----------
311
+ fp: file
312
+ File object handle used for serializing the input object.
313
+ protocol: int, optional
314
+ Pickle protocol used. Default is pickle.DEFAULT_PROTOCOL.
315
+ """
316
+
317
+ dispatch = Pickler.dispatch.copy()
318
+
319
+ def __init__(self, fp, protocol=None):
320
+ self.file_handle = fp
321
+ self.buffered = isinstance(self.file_handle, BinaryZlibFile)
322
+
323
+ # By default we want a pickle protocol that only changes with
324
+ # the major python version and not the minor one
325
+ if protocol is None:
326
+ protocol = pickle.DEFAULT_PROTOCOL
327
+
328
+ Pickler.__init__(self, self.file_handle, protocol=protocol)
329
+ # delayed import of numpy, to avoid tight coupling
330
+ try:
331
+ import numpy as np
332
+ except ImportError:
333
+ np = None
334
+ self.np = np
335
+
336
+ def _create_array_wrapper(self, array):
337
+ """Create and returns a numpy array wrapper from a numpy array."""
338
+ order = (
339
+ "F" if (array.flags.f_contiguous and not array.flags.c_contiguous) else "C"
340
+ )
341
+ allow_mmap = not self.buffered and not array.dtype.hasobject
342
+
343
+ kwargs = {}
344
+ try:
345
+ self.file_handle.tell()
346
+ except io.UnsupportedOperation:
347
+ kwargs = {"numpy_array_alignment_bytes": None}
348
+
349
+ wrapper = NumpyArrayWrapper(
350
+ type(array),
351
+ array.shape,
352
+ order,
353
+ array.dtype,
354
+ allow_mmap=allow_mmap,
355
+ **kwargs,
356
+ )
357
+
358
+ return wrapper
359
+
360
+ def save(self, obj):
361
+ """Subclass the Pickler `save` method.
362
+
363
+ This is a total abuse of the Pickler class in order to use the numpy
364
+ persistence function `save` instead of the default pickle
365
+ implementation. The numpy array is replaced by a custom wrapper in the
366
+ pickle persistence stack and the serialized array is written right
367
+ after in the file. Warning: the file produced does not follow the
368
+ pickle format. As such it can not be read with `pickle.load`.
369
+ """
370
+ if self.np is not None and type(obj) in (
371
+ self.np.ndarray,
372
+ self.np.matrix,
373
+ self.np.memmap,
374
+ ):
375
+ if type(obj) is self.np.memmap:
376
+ # Pickling doesn't work with memmapped arrays
377
+ obj = self.np.asanyarray(obj)
378
+
379
+ # The array wrapper is pickled instead of the real array.
380
+ wrapper = self._create_array_wrapper(obj)
381
+ Pickler.save(self, wrapper)
382
+
383
+ # A framer was introduced with pickle protocol 4 and we want to
384
+ # ensure the wrapper object is written before the numpy array
385
+ # buffer in the pickle file.
386
+ # See https://www.python.org/dev/peps/pep-3154/#framing to get
387
+ # more information on the framer behavior.
388
+ if self.proto >= 4:
389
+ self.framer.commit_frame(force=True)
390
+
391
+ # And then array bytes are written right after the wrapper.
392
+ wrapper.write_array(obj, self)
393
+ return
394
+
395
+ return Pickler.save(self, obj)
396
+
397
+
398
+ class NumpyUnpickler(Unpickler):
399
+ """A subclass of the Unpickler to unpickle our numpy pickles.
400
+
401
+ Attributes
402
+ ----------
403
+ mmap_mode: str
404
+ The memorymap mode to use for reading numpy arrays.
405
+ file_handle: file_like
406
+ File object to unpickle from.
407
+ ensure_native_byte_order: bool
408
+ If True, coerce the array to use the native endianness of the
409
+ host system.
410
+ filename: str
411
+ Name of the file to unpickle from. It should correspond to file_handle.
412
+ This parameter is required when using mmap_mode.
413
+ np: module
414
+ Reference to numpy module if numpy is installed else None.
415
+
416
+ """
417
+
418
+ dispatch = Unpickler.dispatch.copy()
419
+
420
+ def __init__(self, filename, file_handle, ensure_native_byte_order, mmap_mode=None):
421
+ # The next line is for backward compatibility with pickle generated
422
+ # with joblib versions less than 0.10.
423
+ self._dirname = os.path.dirname(filename)
424
+
425
+ self.mmap_mode = mmap_mode
426
+ self.file_handle = file_handle
427
+ # filename is required for numpy mmap mode.
428
+ self.filename = filename
429
+ self.compat_mode = False
430
+ self.ensure_native_byte_order = ensure_native_byte_order
431
+ Unpickler.__init__(self, self.file_handle)
432
+ try:
433
+ import numpy as np
434
+ except ImportError:
435
+ np = None
436
+ self.np = np
437
+
438
+ def load_build(self):
439
+ """Called to set the state of a newly created object.
440
+
441
+ We capture it to replace our place-holder objects, NDArrayWrapper or
442
+ NumpyArrayWrapper, by the array we are interested in. We
443
+ replace them directly in the stack of pickler.
444
+ NDArrayWrapper is used for backward compatibility with joblib <= 0.9.
445
+ """
446
+ Unpickler.load_build(self)
447
+
448
+ # For backward compatibility, we support NDArrayWrapper objects.
449
+ if isinstance(self.stack[-1], (NDArrayWrapper, NumpyArrayWrapper)):
450
+ if self.np is None:
451
+ raise ImportError(
452
+ "Trying to unpickle an ndarray, but numpy didn't import correctly"
453
+ )
454
+ array_wrapper = self.stack.pop()
455
+ # If any NDArrayWrapper is found, we switch to compatibility mode,
456
+ # this will be used to raise a DeprecationWarning to the user at
457
+ # the end of the unpickling.
458
+ if isinstance(array_wrapper, NDArrayWrapper):
459
+ self.compat_mode = True
460
+ _array_payload = array_wrapper.read(self)
461
+ else:
462
+ _array_payload = array_wrapper.read(self, self.ensure_native_byte_order)
463
+
464
+ self.stack.append(_array_payload)
465
+
466
+ # Be careful to register our new method.
467
+ dispatch[pickle.BUILD[0]] = load_build
468
+
469
+
470
+ ###############################################################################
471
+ # Utility functions
472
+
473
+
474
+ def dump(value, filename, compress=0, protocol=None):
475
+ """Persist an arbitrary Python object into one file.
476
+
477
+ Read more in the :ref:`User Guide <persistence>`.
478
+
479
+ Parameters
480
+ ----------
481
+ value: any Python object
482
+ The object to store to disk.
483
+ filename: str, pathlib.Path, or file object.
484
+ The file object or path of the file in which it is to be stored.
485
+ The compression method corresponding to one of the supported filename
486
+ extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used
487
+ automatically.
488
+ compress: int from 0 to 9 or bool or 2-tuple, optional
489
+ Optional compression level for the data. 0 or False is no compression.
490
+ Higher value means more compression, but also slower read and
491
+ write times. Using a value of 3 is often a good compromise.
492
+ See the notes for more details.
493
+ If compress is True, the compression level used is 3.
494
+ If compress is a 2-tuple, the first element must correspond to a string
495
+ between supported compressors (e.g 'zlib', 'gzip', 'bz2', 'lzma'
496
+ 'xz'), the second element must be an integer from 0 to 9, corresponding
497
+ to the compression level.
498
+ protocol: int, optional
499
+ Pickle protocol, see pickle.dump documentation for more details.
500
+
501
+ Returns
502
+ -------
503
+ filenames: list of strings
504
+ The list of file names in which the data is stored. If
505
+ compress is false, each array is stored in a different file.
506
+
507
+ See Also
508
+ --------
509
+ joblib.load : corresponding loader
510
+
511
+ Notes
512
+ -----
513
+ Memmapping on load cannot be used for compressed files. Thus
514
+ using compression can significantly slow down loading. In
515
+ addition, compressed files take up extra memory during
516
+ dump and load.
517
+
518
+ """
519
+
520
+ if Path is not None and isinstance(filename, Path):
521
+ filename = str(filename)
522
+
523
+ is_filename = isinstance(filename, str)
524
+ is_fileobj = hasattr(filename, "write")
525
+
526
+ compress_method = "zlib" # zlib is the default compression method.
527
+ if compress is True:
528
+ # By default, if compress is enabled, we want the default compress
529
+ # level of the compressor.
530
+ compress_level = None
531
+ elif isinstance(compress, tuple):
532
+ # a 2-tuple was set in compress
533
+ if len(compress) != 2:
534
+ raise ValueError(
535
+ "Compress argument tuple should contain exactly 2 elements: "
536
+ "(compress method, compress level), you passed {}".format(compress)
537
+ )
538
+ compress_method, compress_level = compress
539
+ elif isinstance(compress, str):
540
+ compress_method = compress
541
+ compress_level = None # Use default compress level
542
+ compress = (compress_method, compress_level)
543
+ else:
544
+ compress_level = compress
545
+
546
+ if compress_method == "lz4" and lz4 is None:
547
+ raise ValueError(LZ4_NOT_INSTALLED_ERROR)
548
+
549
+ if (
550
+ compress_level is not None
551
+ and compress_level is not False
552
+ and compress_level not in range(10)
553
+ ):
554
+ # Raising an error if a non valid compress level is given.
555
+ raise ValueError(
556
+ 'Non valid compress level given: "{}". Possible values are {}.'.format(
557
+ compress_level, list(range(10))
558
+ )
559
+ )
560
+
561
+ if compress_method not in _COMPRESSORS:
562
+ # Raising an error if an unsupported compression method is given.
563
+ raise ValueError(
564
+ 'Non valid compression method given: "{}". Possible values are {}.'.format(
565
+ compress_method, _COMPRESSORS
566
+ )
567
+ )
568
+
569
+ if not is_filename and not is_fileobj:
570
+ # People keep inverting arguments, and the resulting error is
571
+ # incomprehensible
572
+ raise ValueError(
573
+ "Second argument should be a filename or a file-like object, "
574
+ "%s (type %s) was given." % (filename, type(filename))
575
+ )
576
+
577
+ if is_filename and not isinstance(compress, tuple):
578
+ # In case no explicit compression was requested using both compression
579
+ # method and level in a tuple and the filename has an explicit
580
+ # extension, we select the corresponding compressor.
581
+
582
+ # unset the variable to be sure no compression level is set afterwards.
583
+ compress_method = None
584
+ for name, compressor in _COMPRESSORS.items():
585
+ if filename.endswith(compressor.extension):
586
+ compress_method = name
587
+
588
+ if compress_method in _COMPRESSORS and compress_level == 0:
589
+ # we choose the default compress_level in case it was not given
590
+ # as an argument (using compress).
591
+ compress_level = None
592
+
593
+ if compress_level != 0:
594
+ with _write_fileobject(
595
+ filename, compress=(compress_method, compress_level)
596
+ ) as f:
597
+ NumpyPickler(f, protocol=protocol).dump(value)
598
+ elif is_filename:
599
+ with open(filename, "wb") as f:
600
+ NumpyPickler(f, protocol=protocol).dump(value)
601
+ else:
602
+ NumpyPickler(filename, protocol=protocol).dump(value)
603
+
604
+ # If the target container is a file object, nothing is returned.
605
+ if is_fileobj:
606
+ return
607
+
608
+ # For compatibility, the list of created filenames (e.g with one element
609
+ # after 0.10.0) is returned by default.
610
+ return [filename]
611
+
612
+
613
+ def _unpickle(fobj, ensure_native_byte_order, filename="", mmap_mode=None):
614
+ """Internal unpickling function."""
615
+ # We are careful to open the file handle early and keep it open to
616
+ # avoid race-conditions on renames.
617
+ # That said, if data is stored in companion files, which can be
618
+ # the case with the old persistence format, moving the directory
619
+ # will create a race when joblib tries to access the companion
620
+ # files.
621
+ unpickler = NumpyUnpickler(
622
+ filename, fobj, ensure_native_byte_order, mmap_mode=mmap_mode
623
+ )
624
+ obj = None
625
+ try:
626
+ obj = unpickler.load()
627
+ if unpickler.compat_mode:
628
+ warnings.warn(
629
+ "The file '%s' has been generated with a "
630
+ "joblib version less than 0.10. "
631
+ "Please regenerate this pickle file." % filename,
632
+ DeprecationWarning,
633
+ stacklevel=3,
634
+ )
635
+ except UnicodeDecodeError as exc:
636
+ # More user-friendly error message
637
+ new_exc = ValueError(
638
+ "You may be trying to read with "
639
+ "python 3 a joblib pickle generated with python 2. "
640
+ "This feature is not supported by joblib."
641
+ )
642
+ new_exc.__cause__ = exc
643
+ raise new_exc
644
+ return obj
645
+
646
+
647
+ def load_temporary_memmap(filename, mmap_mode, unlink_on_gc_collect):
648
+ from ._memmapping_reducer import JOBLIB_MMAPS, add_maybe_unlink_finalizer
649
+
650
+ with open(filename, "rb") as f:
651
+ with _validate_fileobject_and_memmap(f, filename, mmap_mode) as (
652
+ fobj,
653
+ validated_mmap_mode,
654
+ ):
655
+ # Memmap are used for interprocess communication, which should
656
+ # keep the objects untouched. We pass `ensure_native_byte_order=False`
657
+ # to remain consistent with the loading behavior of non-memmaped arrays
658
+ # in workers, where the byte order is preserved.
659
+ # Note that we do not implement endianness change for memmaps, as this
660
+ # would result in inconsistent behavior.
661
+ obj = _unpickle(
662
+ fobj,
663
+ ensure_native_byte_order=False,
664
+ filename=filename,
665
+ mmap_mode=validated_mmap_mode,
666
+ )
667
+
668
+ JOBLIB_MMAPS.add(obj.filename)
669
+ if unlink_on_gc_collect:
670
+ add_maybe_unlink_finalizer(obj)
671
+ return obj
672
+
673
+
674
+ def load(filename, mmap_mode=None, ensure_native_byte_order="auto"):
675
+ """Reconstruct a Python object from a file persisted with joblib.dump.
676
+
677
+ Read more in the :ref:`User Guide <persistence>`.
678
+
679
+ WARNING: joblib.load relies on the pickle module and can therefore
680
+ execute arbitrary Python code. It should therefore never be used
681
+ to load files from untrusted sources.
682
+
683
+ Parameters
684
+ ----------
685
+ filename: str, pathlib.Path, or file object.
686
+ The file object or path of the file from which to load the object
687
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
688
+ If not None, the arrays are memory-mapped from the disk. This
689
+ mode has no effect for compressed files. Note that in this
690
+ case the reconstructed object might no longer match exactly
691
+ the originally pickled object.
692
+ ensure_native_byte_order: bool, or 'auto', default=='auto'
693
+ If True, ensures that the byte order of the loaded arrays matches the
694
+ native byte ordering (or _endianness_) of the host system. This is not
695
+ compatible with memory-mapped arrays and using non-null `mmap_mode`
696
+ parameter at the same time will raise an error. The default 'auto'
697
+ parameter is equivalent to True if `mmap_mode` is None, else False.
698
+
699
+ Returns
700
+ -------
701
+ result: any Python object
702
+ The object stored in the file.
703
+
704
+ See Also
705
+ --------
706
+ joblib.dump : function to save an object
707
+
708
+ Notes
709
+ -----
710
+
711
+ This function can load numpy array files saved separately during the
712
+ dump. If the mmap_mode argument is given, it is passed to np.load and
713
+ arrays are loaded as memmaps. As a consequence, the reconstructed
714
+ object might not match the original pickled object. Note that if the
715
+ file was saved with compression, the arrays cannot be memmapped.
716
+ """
717
+ if ensure_native_byte_order == "auto":
718
+ ensure_native_byte_order = mmap_mode is None
719
+
720
+ if ensure_native_byte_order and mmap_mode is not None:
721
+ raise ValueError(
722
+ "Native byte ordering can only be enforced if 'mmap_mode' parameter "
723
+ f"is set to None, but got 'mmap_mode={mmap_mode}' instead."
724
+ )
725
+
726
+ if Path is not None and isinstance(filename, Path):
727
+ filename = str(filename)
728
+
729
+ if hasattr(filename, "read"):
730
+ fobj = filename
731
+ filename = getattr(fobj, "name", "")
732
+ with _validate_fileobject_and_memmap(fobj, filename, mmap_mode) as (fobj, _):
733
+ obj = _unpickle(fobj, ensure_native_byte_order=ensure_native_byte_order)
734
+ else:
735
+ with open(filename, "rb") as f:
736
+ with _validate_fileobject_and_memmap(f, filename, mmap_mode) as (
737
+ fobj,
738
+ validated_mmap_mode,
739
+ ):
740
+ if isinstance(fobj, str):
741
+ # if the returned file object is a string, this means we
742
+ # try to load a pickle file generated with an version of
743
+ # Joblib so we load it with joblib compatibility function.
744
+ return load_compatibility(fobj)
745
+
746
+ # A memory-mapped array has to be mapped with the endianness
747
+ # it has been written with. Other arrays are coerced to the
748
+ # native endianness of the host system.
749
+ obj = _unpickle(
750
+ fobj,
751
+ ensure_native_byte_order=ensure_native_byte_order,
752
+ filename=filename,
753
+ mmap_mode=validated_mmap_mode,
754
+ )
755
+
756
+ return obj
TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_compat.py ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Numpy pickle compatibility functions."""
2
+
3
+ import inspect
4
+ import os
5
+ import pickle
6
+ import zlib
7
+ from io import BytesIO
8
+
9
+ from .numpy_pickle_utils import (
10
+ _ZFILE_PREFIX,
11
+ Unpickler,
12
+ _ensure_native_byte_order,
13
+ _reconstruct,
14
+ )
15
+
16
+
17
+ def hex_str(an_int):
18
+ """Convert an int to an hexadecimal string."""
19
+ return "{:#x}".format(an_int)
20
+
21
+
22
+ def asbytes(s):
23
+ if isinstance(s, bytes):
24
+ return s
25
+ return s.encode("latin1")
26
+
27
+
28
+ _MAX_LEN = len(hex_str(2**64))
29
+ _CHUNK_SIZE = 64 * 1024
30
+
31
+
32
+ def read_zfile(file_handle):
33
+ """Read the z-file and return the content as a string.
34
+
35
+ Z-files are raw data compressed with zlib used internally by joblib
36
+ for persistence. Backward compatibility is not guaranteed. Do not
37
+ use for external purposes.
38
+ """
39
+ file_handle.seek(0)
40
+ header_length = len(_ZFILE_PREFIX) + _MAX_LEN
41
+ length = file_handle.read(header_length)
42
+ length = length[len(_ZFILE_PREFIX) :]
43
+ length = int(length, 16)
44
+
45
+ # With python2 and joblib version <= 0.8.4 compressed pickle header is one
46
+ # character wider so we need to ignore an additional space if present.
47
+ # Note: the first byte of the zlib data is guaranteed not to be a
48
+ # space according to
49
+ # https://tools.ietf.org/html/rfc6713#section-2.1
50
+ next_byte = file_handle.read(1)
51
+ if next_byte != b" ":
52
+ # The zlib compressed data has started and we need to go back
53
+ # one byte
54
+ file_handle.seek(header_length)
55
+
56
+ # We use the known length of the data to tell Zlib the size of the
57
+ # buffer to allocate.
58
+ data = zlib.decompress(file_handle.read(), 15, length)
59
+ assert len(data) == length, (
60
+ "Incorrect data length while decompressing %s."
61
+ "The file could be corrupted." % file_handle
62
+ )
63
+ return data
64
+
65
+
66
+ def write_zfile(file_handle, data, compress=1):
67
+ """Write the data in the given file as a Z-file.
68
+
69
+ Z-files are raw data compressed with zlib used internally by joblib
70
+ for persistence. Backward compatibility is not guaranteed. Do not
71
+ use for external purposes.
72
+ """
73
+ file_handle.write(_ZFILE_PREFIX)
74
+ length = hex_str(len(data))
75
+ # Store the length of the data
76
+ file_handle.write(asbytes(length.ljust(_MAX_LEN)))
77
+ file_handle.write(zlib.compress(asbytes(data), compress))
78
+
79
+
80
+ ###############################################################################
81
+ # Utility objects for persistence.
82
+
83
+
84
+ class NDArrayWrapper(object):
85
+ """An object to be persisted instead of numpy arrays.
86
+
87
+ The only thing this object does, is to carry the filename in which
88
+ the array has been persisted, and the array subclass.
89
+ """
90
+
91
+ def __init__(self, filename, subclass, allow_mmap=True):
92
+ """Constructor. Store the useful information for later."""
93
+ self.filename = filename
94
+ self.subclass = subclass
95
+ self.allow_mmap = allow_mmap
96
+
97
+ def read(self, unpickler):
98
+ """Reconstruct the array."""
99
+ filename = os.path.join(unpickler._dirname, self.filename)
100
+ # Load the array from the disk
101
+ # use getattr instead of self.allow_mmap to ensure backward compat
102
+ # with NDArrayWrapper instances pickled with joblib < 0.9.0
103
+ allow_mmap = getattr(self, "allow_mmap", True)
104
+ kwargs = {}
105
+ if allow_mmap:
106
+ kwargs["mmap_mode"] = unpickler.mmap_mode
107
+ if "allow_pickle" in inspect.signature(unpickler.np.load).parameters:
108
+ # Required in numpy 1.16.3 and later to acknowledge the security
109
+ # risk.
110
+ kwargs["allow_pickle"] = True
111
+ array = unpickler.np.load(filename, **kwargs)
112
+
113
+ # Detect byte order mismatch and swap as needed.
114
+ array = _ensure_native_byte_order(array)
115
+
116
+ # Reconstruct subclasses. This does not work with old
117
+ # versions of numpy
118
+ if hasattr(array, "__array_prepare__") and self.subclass not in (
119
+ unpickler.np.ndarray,
120
+ unpickler.np.memmap,
121
+ ):
122
+ # We need to reconstruct another subclass
123
+ new_array = _reconstruct(self.subclass, (0,), "b")
124
+ return new_array.__array_prepare__(array)
125
+ else:
126
+ return array
127
+
128
+
129
+ class ZNDArrayWrapper(NDArrayWrapper):
130
+ """An object to be persisted instead of numpy arrays.
131
+
132
+ This object store the Zfile filename in which
133
+ the data array has been persisted, and the meta information to
134
+ retrieve it.
135
+ The reason that we store the raw buffer data of the array and
136
+ the meta information, rather than array representation routine
137
+ (tobytes) is that it enables us to use completely the strided
138
+ model to avoid memory copies (a and a.T store as fast). In
139
+ addition saving the heavy information separately can avoid
140
+ creating large temporary buffers when unpickling data with
141
+ large arrays.
142
+ """
143
+
144
+ def __init__(self, filename, init_args, state):
145
+ """Constructor. Store the useful information for later."""
146
+ self.filename = filename
147
+ self.state = state
148
+ self.init_args = init_args
149
+
150
+ def read(self, unpickler):
151
+ """Reconstruct the array from the meta-information and the z-file."""
152
+ # Here we a simply reproducing the unpickling mechanism for numpy
153
+ # arrays
154
+ filename = os.path.join(unpickler._dirname, self.filename)
155
+ array = _reconstruct(*self.init_args)
156
+ with open(filename, "rb") as f:
157
+ data = read_zfile(f)
158
+ state = self.state + (data,)
159
+ array.__setstate__(state)
160
+ return array
161
+
162
+
163
+ class ZipNumpyUnpickler(Unpickler):
164
+ """A subclass of the Unpickler to unpickle our numpy pickles."""
165
+
166
+ dispatch = Unpickler.dispatch.copy()
167
+
168
+ def __init__(self, filename, file_handle, mmap_mode=None):
169
+ """Constructor."""
170
+ self._filename = os.path.basename(filename)
171
+ self._dirname = os.path.dirname(filename)
172
+ self.mmap_mode = mmap_mode
173
+ self.file_handle = self._open_pickle(file_handle)
174
+ Unpickler.__init__(self, self.file_handle)
175
+ try:
176
+ import numpy as np
177
+ except ImportError:
178
+ np = None
179
+ self.np = np
180
+
181
+ def _open_pickle(self, file_handle):
182
+ return BytesIO(read_zfile(file_handle))
183
+
184
+ def load_build(self):
185
+ """Set the state of a newly created object.
186
+
187
+ We capture it to replace our place-holder objects,
188
+ NDArrayWrapper, by the array we are interested in. We
189
+ replace them directly in the stack of pickler.
190
+ """
191
+ Unpickler.load_build(self)
192
+ if isinstance(self.stack[-1], NDArrayWrapper):
193
+ if self.np is None:
194
+ raise ImportError(
195
+ "Trying to unpickle an ndarray, but numpy didn't import correctly"
196
+ )
197
+ nd_array_wrapper = self.stack.pop()
198
+ array = nd_array_wrapper.read(self)
199
+ self.stack.append(array)
200
+
201
+ dispatch[pickle.BUILD[0]] = load_build
202
+
203
+
204
+ def load_compatibility(filename):
205
+ """Reconstruct a Python object from a file persisted with joblib.dump.
206
+
207
+ This function ensures the compatibility with joblib old persistence format
208
+ (<= 0.9.3).
209
+
210
+ Parameters
211
+ ----------
212
+ filename: string
213
+ The name of the file from which to load the object
214
+
215
+ Returns
216
+ -------
217
+ result: any Python object
218
+ The object stored in the file.
219
+
220
+ See Also
221
+ --------
222
+ joblib.dump : function to save an object
223
+
224
+ Notes
225
+ -----
226
+
227
+ This function can load numpy array files saved separately during the
228
+ dump.
229
+ """
230
+ with open(filename, "rb") as file_handle:
231
+ # We are careful to open the file handle early and keep it open to
232
+ # avoid race-conditions on renames. That said, if data is stored in
233
+ # companion files, moving the directory will create a race when
234
+ # joblib tries to access the companion files.
235
+ unpickler = ZipNumpyUnpickler(filename, file_handle=file_handle)
236
+ try:
237
+ obj = unpickler.load()
238
+ except UnicodeDecodeError as exc:
239
+ # More user-friendly error message
240
+ new_exc = ValueError(
241
+ "You may be trying to read with "
242
+ "python 3 a joblib pickle generated with python 2. "
243
+ "This feature is not supported by joblib."
244
+ )
245
+ new_exc.__cause__ = exc
246
+ raise new_exc
247
+ finally:
248
+ if hasattr(unpickler, "file_handle"):
249
+ unpickler.file_handle.close()
250
+ return obj
TTS/Kokoro/.venv/Lib/site-packages/joblib/numpy_pickle_utils.py ADDED
@@ -0,0 +1,291 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Utilities for fast persistence of big data, with optional compression."""
2
+
3
+ # Author: Gael Varoquaux <gael dot varoquaux at normalesup dot org>
4
+ # Copyright (c) 2009 Gael Varoquaux
5
+ # License: BSD Style, 3 clauses.
6
+
7
+ import contextlib
8
+ import io
9
+ import pickle
10
+ import sys
11
+ import warnings
12
+
13
+ from .compressor import _COMPRESSORS, _ZFILE_PREFIX
14
+
15
+ try:
16
+ import numpy as np
17
+ except ImportError:
18
+ np = None
19
+
20
+ Unpickler = pickle._Unpickler
21
+ Pickler = pickle._Pickler
22
+ xrange = range
23
+
24
+
25
+ try:
26
+ # The python standard library can be built without bz2 so we make bz2
27
+ # usage optional.
28
+ # see https://github.com/scikit-learn/scikit-learn/issues/7526 for more
29
+ # details.
30
+ import bz2
31
+ except ImportError:
32
+ bz2 = None
33
+
34
+ # Buffer size used in io.BufferedReader and io.BufferedWriter
35
+ _IO_BUFFER_SIZE = 1024**2
36
+
37
+
38
+ def _is_raw_file(fileobj):
39
+ """Check if fileobj is a raw file object, e.g created with open."""
40
+ fileobj = getattr(fileobj, "raw", fileobj)
41
+ return isinstance(fileobj, io.FileIO)
42
+
43
+
44
+ def _get_prefixes_max_len():
45
+ # Compute the max prefix len of registered compressors.
46
+ prefixes = [len(compressor.prefix) for compressor in _COMPRESSORS.values()]
47
+ prefixes += [len(_ZFILE_PREFIX)]
48
+ return max(prefixes)
49
+
50
+
51
+ def _is_numpy_array_byte_order_mismatch(array):
52
+ """Check if numpy array is having byte order mismatch"""
53
+ return (
54
+ sys.byteorder == "big"
55
+ and (
56
+ array.dtype.byteorder == "<"
57
+ or (
58
+ array.dtype.byteorder == "|"
59
+ and array.dtype.fields
60
+ and all(e[0].byteorder == "<" for e in array.dtype.fields.values())
61
+ )
62
+ )
63
+ ) or (
64
+ sys.byteorder == "little"
65
+ and (
66
+ array.dtype.byteorder == ">"
67
+ or (
68
+ array.dtype.byteorder == "|"
69
+ and array.dtype.fields
70
+ and all(e[0].byteorder == ">" for e in array.dtype.fields.values())
71
+ )
72
+ )
73
+ )
74
+
75
+
76
+ def _ensure_native_byte_order(array):
77
+ """Use the byte order of the host while preserving values
78
+
79
+ Does nothing if array already uses the system byte order.
80
+ """
81
+ if _is_numpy_array_byte_order_mismatch(array):
82
+ array = array.byteswap().view(array.dtype.newbyteorder("="))
83
+ return array
84
+
85
+
86
+ ###############################################################################
87
+ # Cache file utilities
88
+ def _detect_compressor(fileobj):
89
+ """Return the compressor matching fileobj.
90
+
91
+ Parameters
92
+ ----------
93
+ fileobj: file object
94
+
95
+ Returns
96
+ -------
97
+ str in {'zlib', 'gzip', 'bz2', 'lzma', 'xz', 'compat', 'not-compressed'}
98
+ """
99
+ # Read the magic number in the first bytes of the file.
100
+ max_prefix_len = _get_prefixes_max_len()
101
+ if hasattr(fileobj, "peek"):
102
+ # Peek allows to read those bytes without moving the cursor in the
103
+ # file which.
104
+ first_bytes = fileobj.peek(max_prefix_len)
105
+ else:
106
+ # Fallback to seek if the fileobject is not peekable.
107
+ first_bytes = fileobj.read(max_prefix_len)
108
+ fileobj.seek(0)
109
+
110
+ if first_bytes.startswith(_ZFILE_PREFIX):
111
+ return "compat"
112
+ else:
113
+ for name, compressor in _COMPRESSORS.items():
114
+ if first_bytes.startswith(compressor.prefix):
115
+ return name
116
+
117
+ return "not-compressed"
118
+
119
+
120
+ def _buffered_read_file(fobj):
121
+ """Return a buffered version of a read file object."""
122
+ return io.BufferedReader(fobj, buffer_size=_IO_BUFFER_SIZE)
123
+
124
+
125
+ def _buffered_write_file(fobj):
126
+ """Return a buffered version of a write file object."""
127
+ return io.BufferedWriter(fobj, buffer_size=_IO_BUFFER_SIZE)
128
+
129
+
130
+ @contextlib.contextmanager
131
+ def _validate_fileobject_and_memmap(fileobj, filename, mmap_mode=None):
132
+ """Utility function opening the right fileobject from a filename.
133
+
134
+ The magic number is used to choose between the type of file object to open:
135
+ * regular file object (default)
136
+ * zlib file object
137
+ * gzip file object
138
+ * bz2 file object
139
+ * lzma file object (for xz and lzma compressor)
140
+
141
+ Parameters
142
+ ----------
143
+ fileobj: file object
144
+ filename: str
145
+ filename path corresponding to the fileobj parameter.
146
+ mmap_mode: str
147
+ memory map mode that should be used to open the pickle file. This
148
+ parameter is useful to verify that the user is not trying to one with
149
+ compression. Default: None.
150
+
151
+ Returns
152
+ -------
153
+ a tuple with a file like object, and the validated mmap_mode.
154
+
155
+ """
156
+ # Detect if the fileobj contains compressed data.
157
+ compressor = _detect_compressor(fileobj)
158
+ validated_mmap_mode = mmap_mode
159
+
160
+ if compressor == "compat":
161
+ # Compatibility with old pickle mode: simply return the input
162
+ # filename "as-is" and let the compatibility function be called by the
163
+ # caller.
164
+ warnings.warn(
165
+ "The file '%s' has been generated with a joblib "
166
+ "version less than 0.10. "
167
+ "Please regenerate this pickle file." % filename,
168
+ DeprecationWarning,
169
+ stacklevel=2,
170
+ )
171
+ yield filename, validated_mmap_mode
172
+ else:
173
+ if compressor in _COMPRESSORS:
174
+ # based on the compressor detected in the file, we open the
175
+ # correct decompressor file object, wrapped in a buffer.
176
+ compressor_wrapper = _COMPRESSORS[compressor]
177
+ inst = compressor_wrapper.decompressor_file(fileobj)
178
+ fileobj = _buffered_read_file(inst)
179
+
180
+ # Checking if incompatible load parameters with the type of file:
181
+ # mmap_mode cannot be used with compressed file or in memory buffers
182
+ # such as io.BytesIO.
183
+ if mmap_mode is not None:
184
+ validated_mmap_mode = None
185
+ if isinstance(fileobj, io.BytesIO):
186
+ warnings.warn(
187
+ "In memory persistence is not compatible with "
188
+ 'mmap_mode "%(mmap_mode)s" flag passed. '
189
+ "mmap_mode option will be ignored." % locals(),
190
+ stacklevel=2,
191
+ )
192
+ elif compressor != "not-compressed":
193
+ warnings.warn(
194
+ 'mmap_mode "%(mmap_mode)s" is not compatible '
195
+ "with compressed file %(filename)s. "
196
+ '"%(mmap_mode)s" flag will be ignored.' % locals(),
197
+ stacklevel=2,
198
+ )
199
+ elif not _is_raw_file(fileobj):
200
+ warnings.warn(
201
+ '"%(fileobj)r" is not a raw file, mmap_mode '
202
+ '"%(mmap_mode)s" flag will be ignored.' % locals(),
203
+ stacklevel=2,
204
+ )
205
+ else:
206
+ validated_mmap_mode = mmap_mode
207
+
208
+ yield fileobj, validated_mmap_mode
209
+
210
+
211
+ def _write_fileobject(filename, compress=("zlib", 3)):
212
+ """Return the right compressor file object in write mode."""
213
+ compressmethod = compress[0]
214
+ compresslevel = compress[1]
215
+
216
+ if compressmethod in _COMPRESSORS.keys():
217
+ file_instance = _COMPRESSORS[compressmethod].compressor_file(
218
+ filename, compresslevel=compresslevel
219
+ )
220
+ return _buffered_write_file(file_instance)
221
+ else:
222
+ file_instance = _COMPRESSORS["zlib"].compressor_file(
223
+ filename, compresslevel=compresslevel
224
+ )
225
+ return _buffered_write_file(file_instance)
226
+
227
+
228
+ # Utility functions/variables from numpy required for writing arrays.
229
+ # We need at least the functions introduced in version 1.9 of numpy. Here,
230
+ # we use the ones from numpy 1.10.2.
231
+ BUFFER_SIZE = 2**18 # size of buffer for reading npz files in bytes
232
+
233
+
234
+ def _read_bytes(fp, size, error_template="ran out of data"):
235
+ """Read from file-like object until size bytes are read.
236
+
237
+ TODO python2_drop: is it still needed? The docstring mentions python 2.6
238
+ and it looks like this can be at least simplified ...
239
+
240
+ Raises ValueError if not EOF is encountered before size bytes are read.
241
+ Non-blocking objects only supported if they derive from io objects.
242
+
243
+ Required as e.g. ZipExtFile in python 2.6 can return less data than
244
+ requested.
245
+
246
+ This function was taken from numpy/lib/format.py in version 1.10.2.
247
+
248
+ Parameters
249
+ ----------
250
+ fp: file-like object
251
+ size: int
252
+ error_template: str
253
+
254
+ Returns
255
+ -------
256
+ a bytes object
257
+ The data read in bytes.
258
+
259
+ """
260
+ data = bytes()
261
+ while True:
262
+ # io files (default in python3) return None or raise on
263
+ # would-block, python2 file will truncate, probably nothing can be
264
+ # done about that. note that regular files can't be non-blocking
265
+ try:
266
+ r = fp.read(size - len(data))
267
+ data += r
268
+ if len(r) == 0 or len(data) == size:
269
+ break
270
+ except io.BlockingIOError:
271
+ pass
272
+ if len(data) != size:
273
+ msg = "EOF: reading %s, expected %d bytes got %d"
274
+ raise ValueError(msg % (error_template, size, len(data)))
275
+ else:
276
+ return data
277
+
278
+
279
+ def _reconstruct(*args, **kwargs):
280
+ # Wrapper for numpy._core.multiarray._reconstruct with backward compat
281
+ # for numpy 1.X
282
+ #
283
+ # XXX: Remove this function when numpy 1.X is not supported anymore
284
+
285
+ np_major_version = np.__version__[:2]
286
+ if np_major_version == "1.":
287
+ from numpy.core.multiarray import _reconstruct as np_reconstruct
288
+ elif np_major_version == "2.":
289
+ from numpy._core.multiarray import _reconstruct as np_reconstruct
290
+
291
+ return np_reconstruct(*args, **kwargs)
TTS/Kokoro/.venv/Lib/site-packages/joblib/parallel.py ADDED
@@ -0,0 +1,2075 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Helpers for embarrassingly parallel code.
3
+ """
4
+ # Author: Gael Varoquaux < gael dot varoquaux at normalesup dot org >
5
+ # Copyright: 2010, Gael Varoquaux
6
+ # License: BSD 3 clause
7
+
8
+ from __future__ import division
9
+
10
+ import collections
11
+ import functools
12
+ import itertools
13
+ import os
14
+ import queue
15
+ import sys
16
+ import threading
17
+ import time
18
+ import warnings
19
+ import weakref
20
+ from contextlib import nullcontext
21
+ from math import floor, log10, sqrt
22
+ from multiprocessing import TimeoutError
23
+ from numbers import Integral
24
+ from uuid import uuid4
25
+
26
+ from ._multiprocessing_helpers import mp
27
+
28
+ # Make sure that those two classes are part of the public joblib.parallel API
29
+ # so that 3rd party backend implementers can import them from here.
30
+ from ._parallel_backends import (
31
+ AutoBatchingMixin, # noqa
32
+ FallbackToBackend,
33
+ LokyBackend,
34
+ MultiprocessingBackend,
35
+ ParallelBackendBase, # noqa
36
+ SequentialBackend,
37
+ ThreadingBackend,
38
+ )
39
+ from ._utils import _Sentinel, eval_expr
40
+ from .disk import memstr_to_bytes
41
+ from .logger import Logger, short_format_time
42
+
43
+ BACKENDS = {
44
+ "threading": ThreadingBackend,
45
+ "sequential": SequentialBackend,
46
+ }
47
+ # name of the backend used by default by Parallel outside of any context
48
+ # managed by ``parallel_config`` or ``parallel_backend``.
49
+
50
+ # threading is the only backend that is always everywhere
51
+ DEFAULT_BACKEND = "threading"
52
+ DEFAULT_THREAD_BACKEND = "threading"
53
+ DEFAULT_PROCESS_BACKEND = "threading"
54
+
55
+ MAYBE_AVAILABLE_BACKENDS = {"multiprocessing", "loky"}
56
+
57
+ # if multiprocessing is available, so is loky, we set it as the default
58
+ # backend
59
+ if mp is not None:
60
+ BACKENDS["multiprocessing"] = MultiprocessingBackend
61
+ from .externals import loky
62
+
63
+ BACKENDS["loky"] = LokyBackend
64
+ DEFAULT_BACKEND = "loky"
65
+ DEFAULT_PROCESS_BACKEND = "loky"
66
+
67
+ # Thread local value that can be overridden by the ``parallel_config`` context
68
+ # manager
69
+ _backend = threading.local()
70
+
71
+
72
+ def _register_dask():
73
+ """Register Dask Backend if called with parallel_config(backend="dask")"""
74
+ try:
75
+ from ._dask import DaskDistributedBackend
76
+
77
+ register_parallel_backend("dask", DaskDistributedBackend)
78
+ except ImportError as e:
79
+ msg = (
80
+ "To use the dask.distributed backend you must install both "
81
+ "the `dask` and distributed modules.\n\n"
82
+ "See https://dask.pydata.org/en/latest/install.html for more "
83
+ "information."
84
+ )
85
+ raise ImportError(msg) from e
86
+
87
+
88
+ EXTERNAL_BACKENDS = {
89
+ "dask": _register_dask,
90
+ }
91
+
92
+
93
+ # Sentinels for the default values of the Parallel constructor and
94
+ # the parallel_config and parallel_backend context managers
95
+ default_parallel_config = {
96
+ "backend": _Sentinel(default_value=None),
97
+ "n_jobs": _Sentinel(default_value=None),
98
+ "verbose": _Sentinel(default_value=0),
99
+ "temp_folder": _Sentinel(default_value=None),
100
+ "max_nbytes": _Sentinel(default_value="1M"),
101
+ "mmap_mode": _Sentinel(default_value="r"),
102
+ "prefer": _Sentinel(default_value=None),
103
+ "require": _Sentinel(default_value=None),
104
+ }
105
+
106
+
107
+ VALID_BACKEND_HINTS = ("processes", "threads", None)
108
+ VALID_BACKEND_CONSTRAINTS = ("sharedmem", None)
109
+
110
+
111
+ def _get_config_param(param, context_config, key):
112
+ """Return the value of a parallel config parameter
113
+
114
+ Explicitly setting it in Parallel has priority over setting in a
115
+ parallel_(config/backend) context manager.
116
+ """
117
+ if param is not default_parallel_config[key]:
118
+ # param is explicitly set, return it
119
+ return param
120
+
121
+ if context_config[key] is not default_parallel_config[key]:
122
+ # there's a context manager and the key is set, return it
123
+ return context_config[key]
124
+
125
+ # Otherwise, we are in the default_parallel_config,
126
+ # return the default value
127
+ return param.default_value
128
+
129
+
130
+ def get_active_backend(
131
+ prefer=default_parallel_config["prefer"],
132
+ require=default_parallel_config["require"],
133
+ verbose=default_parallel_config["verbose"],
134
+ ):
135
+ """Return the active default backend"""
136
+ backend, config = _get_active_backend(prefer, require, verbose)
137
+ n_jobs = _get_config_param(default_parallel_config["n_jobs"], config, "n_jobs")
138
+ return backend, n_jobs
139
+
140
+
141
+ def _get_active_backend(
142
+ prefer=default_parallel_config["prefer"],
143
+ require=default_parallel_config["require"],
144
+ verbose=default_parallel_config["verbose"],
145
+ ):
146
+ """Return the active default backend"""
147
+
148
+ backend_config = getattr(_backend, "config", default_parallel_config)
149
+
150
+ backend = _get_config_param(
151
+ default_parallel_config["backend"], backend_config, "backend"
152
+ )
153
+
154
+ prefer = _get_config_param(prefer, backend_config, "prefer")
155
+ require = _get_config_param(require, backend_config, "require")
156
+ verbose = _get_config_param(verbose, backend_config, "verbose")
157
+
158
+ if prefer not in VALID_BACKEND_HINTS:
159
+ raise ValueError(
160
+ f"prefer={prefer} is not a valid backend hint, "
161
+ f"expected one of {VALID_BACKEND_HINTS}"
162
+ )
163
+ if require not in VALID_BACKEND_CONSTRAINTS:
164
+ raise ValueError(
165
+ f"require={require} is not a valid backend constraint, "
166
+ f"expected one of {VALID_BACKEND_CONSTRAINTS}"
167
+ )
168
+ if prefer == "processes" and require == "sharedmem":
169
+ raise ValueError(
170
+ "prefer == 'processes' and require == 'sharedmem' are inconsistent settings"
171
+ )
172
+
173
+ explicit_backend = True
174
+ if backend is None:
175
+ # We are either outside of the scope of any parallel_(config/backend)
176
+ # context manager or the context manager did not set a backend.
177
+ # create the default backend instance now.
178
+ backend = BACKENDS[DEFAULT_BACKEND](nesting_level=0)
179
+ explicit_backend = False
180
+
181
+ # Try to use the backend set by the user with the context manager.
182
+
183
+ nesting_level = backend.nesting_level
184
+ uses_threads = getattr(backend, "uses_threads", False)
185
+ supports_sharedmem = getattr(backend, "supports_sharedmem", False)
186
+ # Force to use thread-based backend if the provided backend does not
187
+ # match the shared memory constraint or if the backend is not explicitly
188
+ # given and threads are preferred.
189
+ force_threads = (require == "sharedmem" and not supports_sharedmem) or (
190
+ not explicit_backend and prefer == "threads" and not uses_threads
191
+ )
192
+ force_processes = not explicit_backend and prefer == "processes" and uses_threads
193
+
194
+ if force_threads:
195
+ # This backend does not match the shared memory constraint:
196
+ # fallback to the default thead-based backend.
197
+ sharedmem_backend = BACKENDS[DEFAULT_THREAD_BACKEND](
198
+ nesting_level=nesting_level
199
+ )
200
+ # Warn the user if we forced the backend to thread-based, while the
201
+ # user explicitly specified a non-thread-based backend.
202
+ if verbose >= 10 and explicit_backend:
203
+ print(
204
+ f"Using {sharedmem_backend.__class__.__name__} as "
205
+ f"joblib backend instead of {backend.__class__.__name__} "
206
+ "as the latter does not provide shared memory semantics."
207
+ )
208
+ # Force to n_jobs=1 by default
209
+ thread_config = backend_config.copy()
210
+ thread_config["n_jobs"] = 1
211
+ return sharedmem_backend, thread_config
212
+
213
+ if force_processes:
214
+ # This backend does not match the prefer="processes" constraint:
215
+ # fallback to the default process-based backend.
216
+ process_backend = BACKENDS[DEFAULT_PROCESS_BACKEND](nesting_level=nesting_level)
217
+
218
+ return process_backend, backend_config.copy()
219
+
220
+ return backend, backend_config
221
+
222
+
223
+ class parallel_config:
224
+ """Set the default backend or configuration for :class:`~joblib.Parallel`.
225
+
226
+ This is an alternative to directly passing keyword arguments to the
227
+ :class:`~joblib.Parallel` class constructor. It is particularly useful when
228
+ calling into library code that uses joblib internally but does not expose
229
+ the various parallel configuration arguments in its own API.
230
+
231
+ Parameters
232
+ ----------
233
+ backend: str or ParallelBackendBase instance, default=None
234
+ If ``backend`` is a string it must match a previously registered
235
+ implementation using the :func:`~register_parallel_backend` function.
236
+
237
+ By default the following backends are available:
238
+
239
+ - 'loky': single-host, process-based parallelism (used by default),
240
+ - 'threading': single-host, thread-based parallelism,
241
+ - 'multiprocessing': legacy single-host, process-based parallelism.
242
+
243
+ 'loky' is recommended to run functions that manipulate Python objects.
244
+ 'threading' is a low-overhead alternative that is most efficient for
245
+ functions that release the Global Interpreter Lock: e.g. I/O-bound
246
+ code or CPU-bound code in a few calls to native code that explicitly
247
+ releases the GIL. Note that on some rare systems (such as pyodide),
248
+ multiprocessing and loky may not be available, in which case joblib
249
+ defaults to threading.
250
+
251
+ In addition, if the ``dask`` and ``distributed`` Python packages are
252
+ installed, it is possible to use the 'dask' backend for better
253
+ scheduling of nested parallel calls without over-subscription and
254
+ potentially distribute parallel calls over a networked cluster of
255
+ several hosts.
256
+
257
+ It is also possible to use the distributed 'ray' backend for
258
+ distributing the workload to a cluster of nodes. See more details
259
+ in the Examples section below.
260
+
261
+ Alternatively the backend can be passed directly as an instance.
262
+
263
+ n_jobs: int, default=None
264
+ The maximum number of concurrently running jobs, such as the number
265
+ of Python worker processes when ``backend="loky"`` or the size of the
266
+ thread-pool when ``backend="threading"``.
267
+ This argument is converted to an integer, rounded below for float.
268
+ If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
269
+ ``n_cpus`` is obtained with :func:`~cpu_count`.
270
+ For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
271
+ using ``n_jobs=-2`` will result in all CPUs but one being used.
272
+ This argument can also go above ``n_cpus``, which will cause
273
+ oversubscription. In some cases, slight oversubscription can be
274
+ beneficial, e.g., for tasks with large I/O operations.
275
+ If 1 is given, no parallel computing code is used at all, and the
276
+ behavior amounts to a simple python `for` loop. This mode is not
277
+ compatible with `timeout`.
278
+ None is a marker for 'unset' that will be interpreted as n_jobs=1
279
+ unless the call is performed under a :func:`~parallel_config`
280
+ context manager that sets another value for ``n_jobs``.
281
+ If n_jobs = 0 then a ValueError is raised.
282
+
283
+ verbose: int, default=0
284
+ The verbosity level: if non zero, progress messages are
285
+ printed. Above 50, the output is sent to stdout.
286
+ The frequency of the messages increases with the verbosity level.
287
+ If it more than 10, all iterations are reported.
288
+
289
+ temp_folder: str or None, default=None
290
+ Folder to be used by the pool for memmapping large arrays
291
+ for sharing memory with worker processes. If None, this will try in
292
+ order:
293
+
294
+ - a folder pointed by the ``JOBLIB_TEMP_FOLDER`` environment
295
+ variable,
296
+ - ``/dev/shm`` if the folder exists and is writable: this is a
297
+ RAM disk filesystem available by default on modern Linux
298
+ distributions,
299
+ - the default system temporary folder that can be
300
+ overridden with ``TMP``, ``TMPDIR`` or ``TEMP`` environment
301
+ variables, typically ``/tmp`` under Unix operating systems.
302
+
303
+ max_nbytes: int, str, or None, optional, default='1M'
304
+ Threshold on the size of arrays passed to the workers that
305
+ triggers automated memory mapping in temp_folder. Can be an int
306
+ in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
307
+ Use None to disable memmapping of large arrays.
308
+
309
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
310
+ Memmapping mode for numpy arrays passed to workers. None will
311
+ disable memmapping, other modes defined in the numpy.memmap doc:
312
+ https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
313
+ Also, see 'max_nbytes' parameter documentation for more details.
314
+
315
+ prefer: str in {'processes', 'threads'} or None, default=None
316
+ Soft hint to choose the default backend.
317
+ The default process-based backend is 'loky' and the default
318
+ thread-based backend is 'threading'. Ignored if the ``backend``
319
+ parameter is specified.
320
+
321
+ require: 'sharedmem' or None, default=None
322
+ Hard constraint to select the backend. If set to 'sharedmem',
323
+ the selected backend will be single-host and thread-based.
324
+
325
+ inner_max_num_threads: int, default=None
326
+ If not None, overwrites the limit set on the number of threads
327
+ usable in some third-party library threadpools like OpenBLAS,
328
+ MKL or OpenMP. This is only used with the ``loky`` backend.
329
+
330
+ backend_params: dict
331
+ Additional parameters to pass to the backend constructor when
332
+ backend is a string.
333
+
334
+ Notes
335
+ -----
336
+ Joblib tries to limit the oversubscription by limiting the number of
337
+ threads usable in some third-party library threadpools like OpenBLAS, MKL
338
+ or OpenMP. The default limit in each worker is set to
339
+ ``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
340
+ overwritten with the ``inner_max_num_threads`` argument which will be used
341
+ to set this limit in the child processes.
342
+
343
+ .. versionadded:: 1.3
344
+
345
+ Examples
346
+ --------
347
+ >>> from operator import neg
348
+ >>> with parallel_config(backend='threading'):
349
+ ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
350
+ ...
351
+ [-1, -2, -3, -4, -5]
352
+
353
+ To use the 'ray' joblib backend add the following lines:
354
+
355
+ >>> from ray.util.joblib import register_ray # doctest: +SKIP
356
+ >>> register_ray() # doctest: +SKIP
357
+ >>> with parallel_config(backend="ray"): # doctest: +SKIP
358
+ ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
359
+ [-1, -2, -3, -4, -5]
360
+
361
+ """
362
+
363
+ def __init__(
364
+ self,
365
+ backend=default_parallel_config["backend"],
366
+ *,
367
+ n_jobs=default_parallel_config["n_jobs"],
368
+ verbose=default_parallel_config["verbose"],
369
+ temp_folder=default_parallel_config["temp_folder"],
370
+ max_nbytes=default_parallel_config["max_nbytes"],
371
+ mmap_mode=default_parallel_config["mmap_mode"],
372
+ prefer=default_parallel_config["prefer"],
373
+ require=default_parallel_config["require"],
374
+ inner_max_num_threads=None,
375
+ **backend_params,
376
+ ):
377
+ # Save the parallel info and set the active parallel config
378
+ self.old_parallel_config = getattr(_backend, "config", default_parallel_config)
379
+
380
+ backend = self._check_backend(backend, inner_max_num_threads, **backend_params)
381
+
382
+ new_config = {
383
+ "n_jobs": n_jobs,
384
+ "verbose": verbose,
385
+ "temp_folder": temp_folder,
386
+ "max_nbytes": max_nbytes,
387
+ "mmap_mode": mmap_mode,
388
+ "prefer": prefer,
389
+ "require": require,
390
+ "backend": backend,
391
+ }
392
+ self.parallel_config = self.old_parallel_config.copy()
393
+ self.parallel_config.update(
394
+ {k: v for k, v in new_config.items() if not isinstance(v, _Sentinel)}
395
+ )
396
+
397
+ setattr(_backend, "config", self.parallel_config)
398
+
399
+ def _check_backend(self, backend, inner_max_num_threads, **backend_params):
400
+ if backend is default_parallel_config["backend"]:
401
+ if inner_max_num_threads is not None or len(backend_params) > 0:
402
+ raise ValueError(
403
+ "inner_max_num_threads and other constructor "
404
+ "parameters backend_params are only supported "
405
+ "when backend is not None."
406
+ )
407
+ return backend
408
+
409
+ if isinstance(backend, str):
410
+ # Handle non-registered or missing backends
411
+ if backend not in BACKENDS:
412
+ if backend in EXTERNAL_BACKENDS:
413
+ register = EXTERNAL_BACKENDS[backend]
414
+ register()
415
+ elif backend in MAYBE_AVAILABLE_BACKENDS:
416
+ warnings.warn(
417
+ f"joblib backend '{backend}' is not available on "
418
+ f"your system, falling back to {DEFAULT_BACKEND}.",
419
+ UserWarning,
420
+ stacklevel=2,
421
+ )
422
+ BACKENDS[backend] = BACKENDS[DEFAULT_BACKEND]
423
+ else:
424
+ raise ValueError(
425
+ f"Invalid backend: {backend}, expected one of "
426
+ f"{sorted(BACKENDS.keys())}"
427
+ )
428
+
429
+ backend = BACKENDS[backend](**backend_params)
430
+ else:
431
+ if len(backend_params) > 0:
432
+ raise ValueError(
433
+ "Constructor parameters backend_params are only "
434
+ "supported when backend is a string."
435
+ )
436
+
437
+ if inner_max_num_threads is not None:
438
+ msg = (
439
+ f"{backend.__class__.__name__} does not accept setting the "
440
+ "inner_max_num_threads argument."
441
+ )
442
+ assert backend.supports_inner_max_num_threads, msg
443
+ backend.inner_max_num_threads = inner_max_num_threads
444
+
445
+ # If the nesting_level of the backend is not set previously, use the
446
+ # nesting level from the previous active_backend to set it
447
+ if backend.nesting_level is None:
448
+ parent_backend = self.old_parallel_config["backend"]
449
+ if parent_backend is default_parallel_config["backend"]:
450
+ nesting_level = 0
451
+ else:
452
+ nesting_level = parent_backend.nesting_level
453
+ backend.nesting_level = nesting_level
454
+
455
+ return backend
456
+
457
+ def __enter__(self):
458
+ return self.parallel_config
459
+
460
+ def __exit__(self, type, value, traceback):
461
+ self.unregister()
462
+
463
+ def unregister(self):
464
+ setattr(_backend, "config", self.old_parallel_config)
465
+
466
+
467
+ class parallel_backend(parallel_config):
468
+ """Change the default backend used by Parallel inside a with block.
469
+
470
+ .. warning::
471
+ It is advised to use the :class:`~joblib.parallel_config` context
472
+ manager instead, which allows more fine-grained control over the
473
+ backend configuration.
474
+
475
+ If ``backend`` is a string it must match a previously registered
476
+ implementation using the :func:`~register_parallel_backend` function.
477
+
478
+ By default the following backends are available:
479
+
480
+ - 'loky': single-host, process-based parallelism (used by default),
481
+ - 'threading': single-host, thread-based parallelism,
482
+ - 'multiprocessing': legacy single-host, process-based parallelism.
483
+
484
+ 'loky' is recommended to run functions that manipulate Python objects.
485
+ 'threading' is a low-overhead alternative that is most efficient for
486
+ functions that release the Global Interpreter Lock: e.g. I/O-bound code or
487
+ CPU-bound code in a few calls to native code that explicitly releases the
488
+ GIL. Note that on some rare systems (such as Pyodide),
489
+ multiprocessing and loky may not be available, in which case joblib
490
+ defaults to threading.
491
+
492
+ You can also use the `Dask <https://docs.dask.org/en/stable/>`_ joblib
493
+ backend to distribute work across machines. This works well with
494
+ scikit-learn estimators with the ``n_jobs`` parameter, for example::
495
+
496
+ >>> import joblib # doctest: +SKIP
497
+ >>> from sklearn.model_selection import GridSearchCV # doctest: +SKIP
498
+ >>> from dask.distributed import Client, LocalCluster # doctest: +SKIP
499
+
500
+ >>> # create a local Dask cluster
501
+ >>> cluster = LocalCluster() # doctest: +SKIP
502
+ >>> client = Client(cluster) # doctest: +SKIP
503
+ >>> grid_search = GridSearchCV(estimator, param_grid, n_jobs=-1)
504
+ ... # doctest: +SKIP
505
+ >>> with joblib.parallel_backend("dask", scatter=[X, y]): # doctest: +SKIP
506
+ ... grid_search.fit(X, y)
507
+
508
+ It is also possible to use the distributed 'ray' backend for distributing
509
+ the workload to a cluster of nodes. To use the 'ray' joblib backend add
510
+ the following lines::
511
+
512
+ >>> from ray.util.joblib import register_ray # doctest: +SKIP
513
+ >>> register_ray() # doctest: +SKIP
514
+ >>> with parallel_backend("ray"): # doctest: +SKIP
515
+ ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
516
+ [-1, -2, -3, -4, -5]
517
+
518
+ Alternatively the backend can be passed directly as an instance.
519
+
520
+ By default all available workers will be used (``n_jobs=-1``) unless the
521
+ caller passes an explicit value for the ``n_jobs`` parameter.
522
+
523
+ This is an alternative to passing a ``backend='backend_name'`` argument to
524
+ the :class:`~Parallel` class constructor. It is particularly useful when
525
+ calling into library code that uses joblib internally but does not expose
526
+ the backend argument in its own API.
527
+
528
+ >>> from operator import neg
529
+ >>> with parallel_backend('threading'):
530
+ ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
531
+ ...
532
+ [-1, -2, -3, -4, -5]
533
+
534
+ Joblib also tries to limit the oversubscription by limiting the number of
535
+ threads usable in some third-party library threadpools like OpenBLAS, MKL
536
+ or OpenMP. The default limit in each worker is set to
537
+ ``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
538
+ overwritten with the ``inner_max_num_threads`` argument which will be used
539
+ to set this limit in the child processes.
540
+
541
+ .. versionadded:: 0.10
542
+
543
+ See Also
544
+ --------
545
+ joblib.parallel_config: context manager to change the backend configuration.
546
+ """
547
+
548
+ def __init__(
549
+ self, backend, n_jobs=-1, inner_max_num_threads=None, **backend_params
550
+ ):
551
+ super().__init__(
552
+ backend=backend,
553
+ n_jobs=n_jobs,
554
+ inner_max_num_threads=inner_max_num_threads,
555
+ **backend_params,
556
+ )
557
+
558
+ if self.old_parallel_config is None:
559
+ self.old_backend_and_jobs = None
560
+ else:
561
+ self.old_backend_and_jobs = (
562
+ self.old_parallel_config["backend"],
563
+ self.old_parallel_config["n_jobs"],
564
+ )
565
+ self.new_backend_and_jobs = (
566
+ self.parallel_config["backend"],
567
+ self.parallel_config["n_jobs"],
568
+ )
569
+
570
+ def __enter__(self):
571
+ return self.new_backend_and_jobs
572
+
573
+
574
+ # Under Linux or OS X the default start method of multiprocessing
575
+ # can cause third party libraries to crash. Under Python 3.4+ it is possible
576
+ # to set an environment variable to switch the default start method from
577
+ # 'fork' to 'forkserver' or 'spawn' to avoid this issue albeit at the cost
578
+ # of causing semantic changes and some additional pool instantiation overhead.
579
+ DEFAULT_MP_CONTEXT = None
580
+ if hasattr(mp, "get_context"):
581
+ method = os.environ.get("JOBLIB_START_METHOD", "").strip() or None
582
+ if method is not None:
583
+ DEFAULT_MP_CONTEXT = mp.get_context(method=method)
584
+
585
+
586
+ class BatchedCalls(object):
587
+ """Wrap a sequence of (func, args, kwargs) tuples as a single callable"""
588
+
589
+ def __init__(
590
+ self, iterator_slice, backend_and_jobs, reducer_callback=None, pickle_cache=None
591
+ ):
592
+ self.items = list(iterator_slice)
593
+ self._size = len(self.items)
594
+ self._reducer_callback = reducer_callback
595
+ if isinstance(backend_and_jobs, tuple):
596
+ self._backend, self._n_jobs = backend_and_jobs
597
+ else:
598
+ # this is for backward compatibility purposes. Before 0.12.6,
599
+ # nested backends were returned without n_jobs indications.
600
+ self._backend, self._n_jobs = backend_and_jobs, None
601
+ self._pickle_cache = pickle_cache if pickle_cache is not None else {}
602
+
603
+ def __call__(self):
604
+ # Set the default nested backend to self._backend but do not set the
605
+ # change the default number of processes to -1
606
+ with parallel_config(backend=self._backend, n_jobs=self._n_jobs):
607
+ return [func(*args, **kwargs) for func, args, kwargs in self.items]
608
+
609
+ def __reduce__(self):
610
+ if self._reducer_callback is not None:
611
+ self._reducer_callback()
612
+ # no need to pickle the callback.
613
+ return (
614
+ BatchedCalls,
615
+ (self.items, (self._backend, self._n_jobs), None, self._pickle_cache),
616
+ )
617
+
618
+ def __len__(self):
619
+ return self._size
620
+
621
+
622
+ # Possible exit status for a task
623
+ TASK_DONE = "Done"
624
+ TASK_ERROR = "Error"
625
+ TASK_PENDING = "Pending"
626
+
627
+
628
+ ###############################################################################
629
+ # CPU count that works also when multiprocessing has been disabled via
630
+ # the JOBLIB_MULTIPROCESSING environment variable
631
+ def cpu_count(only_physical_cores=False):
632
+ """Return the number of CPUs.
633
+
634
+ This delegates to loky.cpu_count that takes into account additional
635
+ constraints such as Linux CFS scheduler quotas (typically set by container
636
+ runtimes such as docker) and CPU affinity (for instance using the taskset
637
+ command on Linux).
638
+
639
+ Parameters
640
+ ----------
641
+ only_physical_cores : boolean, default=False
642
+ If True, does not take hyperthreading / SMT logical cores into account.
643
+
644
+ """
645
+ if mp is None:
646
+ return 1
647
+
648
+ return loky.cpu_count(only_physical_cores=only_physical_cores)
649
+
650
+
651
+ ###############################################################################
652
+ # For verbosity
653
+
654
+
655
+ def _verbosity_filter(index, verbose):
656
+ """Returns False for indices increasingly apart, the distance
657
+ depending on the value of verbose.
658
+
659
+ We use a lag increasing as the square of index
660
+ """
661
+ if not verbose:
662
+ return True
663
+ elif verbose > 10:
664
+ return False
665
+ if index == 0:
666
+ return False
667
+ verbose = 0.5 * (11 - verbose) ** 2
668
+ scale = sqrt(index / verbose)
669
+ next_scale = sqrt((index + 1) / verbose)
670
+ return int(next_scale) == int(scale)
671
+
672
+
673
+ ###############################################################################
674
+ def delayed(function):
675
+ """Decorator used to capture the arguments of a function."""
676
+
677
+ def delayed_function(*args, **kwargs):
678
+ return function, args, kwargs
679
+
680
+ try:
681
+ delayed_function = functools.wraps(function)(delayed_function)
682
+ except AttributeError:
683
+ " functools.wraps fails on some callable objects "
684
+ return delayed_function
685
+
686
+
687
+ ###############################################################################
688
+ class BatchCompletionCallBack(object):
689
+ """Callback to keep track of completed results and schedule the next tasks.
690
+
691
+ This callable is executed by the parent process whenever a worker process
692
+ has completed a batch of tasks.
693
+
694
+ It is used for progress reporting, to update estimate of the batch
695
+ processing duration and to schedule the next batch of tasks to be
696
+ processed.
697
+
698
+ It is assumed that this callback will always be triggered by the backend
699
+ right after the end of a task, in case of success as well as in case of
700
+ failure.
701
+ """
702
+
703
+ ##########################################################################
704
+ # METHODS CALLED BY THE MAIN THREAD #
705
+ ##########################################################################
706
+ def __init__(self, dispatch_timestamp, batch_size, parallel):
707
+ self.dispatch_timestamp = dispatch_timestamp
708
+ self.batch_size = batch_size
709
+ self.parallel = parallel
710
+ self.parallel_call_id = parallel._call_id
711
+ self._completion_timeout_counter = None
712
+
713
+ # Internals to keep track of the status and outcome of the task.
714
+
715
+ # Used to hold a reference to the future-like object returned by the
716
+ # backend after launching this task
717
+ # This will be set later when calling `register_job`, as it is only
718
+ # created once the task has been submitted.
719
+ self.job = None
720
+
721
+ if not parallel._backend.supports_retrieve_callback:
722
+ # The status is only used for asynchronous result retrieval in the
723
+ # callback.
724
+ self.status = None
725
+ else:
726
+ # The initial status for the job is TASK_PENDING.
727
+ # Once it is done, it will be either TASK_DONE, or TASK_ERROR.
728
+ self.status = TASK_PENDING
729
+
730
+ def register_job(self, job):
731
+ """Register the object returned by `submit`."""
732
+ self.job = job
733
+
734
+ def get_result(self, timeout):
735
+ """Returns the raw result of the task that was submitted.
736
+
737
+ If the task raised an exception rather than returning, this same
738
+ exception will be raised instead.
739
+
740
+ If the backend supports the retrieval callback, it is assumed that this
741
+ method is only called after the result has been registered. It is
742
+ ensured by checking that `self.status(timeout)` does not return
743
+ TASK_PENDING. In this case, `get_result` directly returns the
744
+ registered result (or raise the registered exception).
745
+
746
+ For other backends, there are no such assumptions, but `get_result`
747
+ still needs to synchronously retrieve the result before it can
748
+ return it or raise. It will block at most `self.timeout` seconds
749
+ waiting for retrieval to complete, after that it raises a TimeoutError.
750
+ """
751
+
752
+ backend = self.parallel._backend
753
+
754
+ if backend.supports_retrieve_callback:
755
+ # We assume that the result has already been retrieved by the
756
+ # callback thread, and is stored internally. It's just waiting to
757
+ # be returned.
758
+ return self._return_or_raise()
759
+
760
+ # For other backends, the main thread needs to run the retrieval step.
761
+ try:
762
+ result = backend.retrieve_result(self.job, timeout=timeout)
763
+ outcome = dict(result=result, status=TASK_DONE)
764
+ except BaseException as e:
765
+ outcome = dict(result=e, status=TASK_ERROR)
766
+ self._register_outcome(outcome)
767
+
768
+ return self._return_or_raise()
769
+
770
+ def _return_or_raise(self):
771
+ try:
772
+ if self.status == TASK_ERROR:
773
+ raise self._result
774
+ return self._result
775
+ finally:
776
+ del self._result
777
+
778
+ def get_status(self, timeout):
779
+ """Get the status of the task.
780
+
781
+ This function also checks if the timeout has been reached and register
782
+ the TimeoutError outcome when it is the case.
783
+ """
784
+ if timeout is None or self.status != TASK_PENDING:
785
+ return self.status
786
+
787
+ # The computation are running and the status is pending.
788
+ # Check that we did not wait for this jobs more than `timeout`.
789
+ now = time.time()
790
+ if self._completion_timeout_counter is None:
791
+ self._completion_timeout_counter = now
792
+
793
+ if (now - self._completion_timeout_counter) > timeout:
794
+ outcome = dict(result=TimeoutError(), status=TASK_ERROR)
795
+ self._register_outcome(outcome)
796
+
797
+ return self.status
798
+
799
+ ##########################################################################
800
+ # METHODS CALLED BY CALLBACK THREADS #
801
+ ##########################################################################
802
+ def __call__(self, *args, **kwargs):
803
+ """Function called by the callback thread after a job is completed."""
804
+
805
+ # If the backend doesn't support callback retrievals, the next batch of
806
+ # tasks is dispatched regardless. The result will be retrieved by the
807
+ # main thread when calling `get_result`.
808
+ if not self.parallel._backend.supports_retrieve_callback:
809
+ self._dispatch_new()
810
+ return
811
+
812
+ # If the backend supports retrieving the result in the callback, it
813
+ # registers the task outcome (TASK_ERROR or TASK_DONE), and schedules
814
+ # the next batch if needed.
815
+ with self.parallel._lock:
816
+ # Edge case where while the task was processing, the `parallel`
817
+ # instance has been reset and a new call has been issued, but the
818
+ # worker managed to complete the task and trigger this callback
819
+ # call just before being aborted by the reset.
820
+ if self.parallel._call_id != self.parallel_call_id:
821
+ return
822
+
823
+ # When aborting, stop as fast as possible and do not retrieve the
824
+ # result as it won't be returned by the Parallel call.
825
+ if self.parallel._aborting:
826
+ return
827
+
828
+ # Retrieves the result of the task in the main process and dispatch
829
+ # a new batch if needed.
830
+ job_succeeded = self._retrieve_result(*args, **kwargs)
831
+
832
+ if job_succeeded:
833
+ self._dispatch_new()
834
+
835
+ def _dispatch_new(self):
836
+ """Schedule the next batch of tasks to be processed."""
837
+
838
+ # This steps ensure that auto-batching works as expected.
839
+ this_batch_duration = time.time() - self.dispatch_timestamp
840
+ self.parallel._backend.batch_completed(self.batch_size, this_batch_duration)
841
+
842
+ # Schedule the next batch of tasks.
843
+ with self.parallel._lock:
844
+ self.parallel.n_completed_tasks += self.batch_size
845
+ self.parallel.print_progress()
846
+ if self.parallel._original_iterator is not None:
847
+ self.parallel.dispatch_next()
848
+
849
+ def _retrieve_result(self, out):
850
+ """Fetch and register the outcome of a task.
851
+
852
+ Return True if the task succeeded, False otherwise.
853
+ This function is only called by backends that support retrieving
854
+ the task result in the callback thread.
855
+ """
856
+ try:
857
+ result = self.parallel._backend.retrieve_result_callback(out)
858
+ outcome = dict(status=TASK_DONE, result=result)
859
+ except BaseException as e:
860
+ # Avoid keeping references to parallel in the error.
861
+ e.__traceback__ = None
862
+ outcome = dict(result=e, status=TASK_ERROR)
863
+
864
+ self._register_outcome(outcome)
865
+ return outcome["status"] != TASK_ERROR
866
+
867
+ ##########################################################################
868
+ # This method can be called either in the main thread #
869
+ # or in the callback thread. #
870
+ ##########################################################################
871
+ def _register_outcome(self, outcome):
872
+ """Register the outcome of a task.
873
+
874
+ This method can be called only once, future calls will be ignored.
875
+ """
876
+ # Covers the edge case where the main thread tries to register a
877
+ # `TimeoutError` while the callback thread tries to register a result
878
+ # at the same time.
879
+ with self.parallel._lock:
880
+ if self.status not in (TASK_PENDING, None):
881
+ return
882
+ self.status = outcome["status"]
883
+
884
+ self._result = outcome["result"]
885
+
886
+ # Once the result and the status are extracted, the last reference to
887
+ # the job can be deleted.
888
+ self.job = None
889
+
890
+ # As soon as an error as been spotted, early stopping flags are sent to
891
+ # the `parallel` instance.
892
+ if self.status == TASK_ERROR:
893
+ self.parallel._exception = True
894
+ self.parallel._aborting = True
895
+
896
+ if self.parallel.return_ordered:
897
+ return
898
+
899
+ with self.parallel._lock:
900
+ # For `return_as=generator_unordered`, append the job to the queue
901
+ # in the order of completion instead of submission.
902
+ self.parallel._jobs.append(self)
903
+
904
+
905
+ ###############################################################################
906
+ def register_parallel_backend(name, factory, make_default=False):
907
+ """Register a new Parallel backend factory.
908
+
909
+ The new backend can then be selected by passing its name as the backend
910
+ argument to the :class:`~Parallel` class. Moreover, the default backend can
911
+ be overwritten globally by setting make_default=True.
912
+
913
+ The factory can be any callable that takes no argument and return an
914
+ instance of ``ParallelBackendBase``.
915
+
916
+ Warning: this function is experimental and subject to change in a future
917
+ version of joblib.
918
+
919
+ .. versionadded:: 0.10
920
+ """
921
+ BACKENDS[name] = factory
922
+ if make_default:
923
+ global DEFAULT_BACKEND
924
+ DEFAULT_BACKEND = name
925
+
926
+
927
+ def effective_n_jobs(n_jobs=-1):
928
+ """Determine the number of jobs that can actually run in parallel
929
+
930
+ n_jobs is the number of workers requested by the callers. Passing n_jobs=-1
931
+ means requesting all available workers for instance matching the number of
932
+ CPU cores on the worker host(s).
933
+
934
+ This method should return a guesstimate of the number of workers that can
935
+ actually perform work concurrently with the currently enabled default
936
+ backend. The primary use case is to make it possible for the caller to know
937
+ in how many chunks to slice the work.
938
+
939
+ In general working on larger data chunks is more efficient (less scheduling
940
+ overhead and better use of CPU cache prefetching heuristics) as long as all
941
+ the workers have enough work to do.
942
+
943
+ Warning: this function is experimental and subject to change in a future
944
+ version of joblib.
945
+
946
+ .. versionadded:: 0.10
947
+ """
948
+ if n_jobs == 1:
949
+ return 1
950
+
951
+ backend, backend_n_jobs = get_active_backend()
952
+ if n_jobs is None:
953
+ n_jobs = backend_n_jobs
954
+ return backend.effective_n_jobs(n_jobs=n_jobs)
955
+
956
+
957
+ ###############################################################################
958
+ class Parallel(Logger):
959
+ """Helper class for readable parallel mapping.
960
+
961
+ Read more in the :ref:`User Guide <parallel>`.
962
+
963
+ Parameters
964
+ ----------
965
+ n_jobs: int, default=None
966
+ The maximum number of concurrently running jobs, such as the number
967
+ of Python worker processes when ``backend="loky"`` or the size of
968
+ the thread-pool when ``backend="threading"``.
969
+ This argument is converted to an integer, rounded below for float.
970
+ If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
971
+ ``n_cpus`` is obtained with :func:`~cpu_count`.
972
+ For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
973
+ using ``n_jobs=-2`` will result in all CPUs but one being used.
974
+ This argument can also go above ``n_cpus``, which will cause
975
+ oversubscription. In some cases, slight oversubscription can be
976
+ beneficial, e.g., for tasks with large I/O operations.
977
+ If 1 is given, no parallel computing code is used at all, and the
978
+ behavior amounts to a simple python `for` loop. This mode is not
979
+ compatible with ``timeout``.
980
+ None is a marker for 'unset' that will be interpreted as n_jobs=1
981
+ unless the call is performed under a :func:`~parallel_config`
982
+ context manager that sets another value for ``n_jobs``.
983
+ If n_jobs = 0 then a ValueError is raised.
984
+ backend: str, ParallelBackendBase instance or None, default='loky'
985
+ Specify the parallelization backend implementation.
986
+ Supported backends are:
987
+
988
+ - "loky" used by default, can induce some
989
+ communication and memory overhead when exchanging input and
990
+ output data with the worker Python processes. On some rare
991
+ systems (such as Pyiodide), the loky backend may not be
992
+ available.
993
+ - "multiprocessing" previous process-based backend based on
994
+ `multiprocessing.Pool`. Less robust than `loky`.
995
+ - "threading" is a very low-overhead backend but it suffers
996
+ from the Python Global Interpreter Lock if the called function
997
+ relies a lot on Python objects. "threading" is mostly useful
998
+ when the execution bottleneck is a compiled extension that
999
+ explicitly releases the GIL (for instance a Cython loop wrapped
1000
+ in a "with nogil" block or an expensive call to a library such
1001
+ as NumPy).
1002
+ - finally, you can register backends by calling
1003
+ :func:`~register_parallel_backend`. This will allow you to
1004
+ implement a backend of your liking.
1005
+
1006
+ It is not recommended to hard-code the backend name in a call to
1007
+ :class:`~Parallel` in a library. Instead it is recommended to set
1008
+ soft hints (prefer) or hard constraints (require) so as to make it
1009
+ possible for library users to change the backend from the outside
1010
+ using the :func:`~parallel_config` context manager.
1011
+ return_as: str in {'list', 'generator', 'generator_unordered'}, default='list'
1012
+ If 'list', calls to this instance will return a list, only when
1013
+ all results have been processed and retrieved.
1014
+ If 'generator', it will return a generator that yields the results
1015
+ as soon as they are available, in the order the tasks have been
1016
+ submitted with.
1017
+ If 'generator_unordered', the generator will immediately yield
1018
+ available results independently of the submission order. The output
1019
+ order is not deterministic in this case because it depends on the
1020
+ concurrency of the workers.
1021
+ prefer: str in {'processes', 'threads'} or None, default=None
1022
+ Soft hint to choose the default backend if no specific backend
1023
+ was selected with the :func:`~parallel_config` context manager.
1024
+ The default process-based backend is 'loky' and the default
1025
+ thread-based backend is 'threading'. Ignored if the ``backend``
1026
+ parameter is specified.
1027
+ require: 'sharedmem' or None, default=None
1028
+ Hard constraint to select the backend. If set to 'sharedmem',
1029
+ the selected backend will be single-host and thread-based even
1030
+ if the user asked for a non-thread based backend with
1031
+ :func:`~joblib.parallel_config`.
1032
+ verbose: int, default=0
1033
+ The verbosity level: if non zero, progress messages are
1034
+ printed. Above 50, the output is sent to stdout.
1035
+ The frequency of the messages increases with the verbosity level.
1036
+ If it more than 10, all iterations are reported.
1037
+ timeout: float or None, default=None
1038
+ Timeout limit for each task to complete. If any task takes longer
1039
+ a TimeOutError will be raised. Only applied when n_jobs != 1
1040
+ pre_dispatch: {'all', integer, or expression, as in '3*n_jobs'}, default='2*n_jobs'
1041
+ The number of batches (of tasks) to be pre-dispatched.
1042
+ Default is '2*n_jobs'. When batch_size="auto" this is reasonable
1043
+ default and the workers should never starve. Note that only basic
1044
+ arithmetic are allowed here and no modules can be used in this
1045
+ expression.
1046
+ batch_size: int or 'auto', default='auto'
1047
+ The number of atomic tasks to dispatch at once to each
1048
+ worker. When individual evaluations are very fast, dispatching
1049
+ calls to workers can be slower than sequential computation because
1050
+ of the overhead. Batching fast computations together can mitigate
1051
+ this.
1052
+ The ``'auto'`` strategy keeps track of the time it takes for a
1053
+ batch to complete, and dynamically adjusts the batch size to keep
1054
+ the time on the order of half a second, using a heuristic. The
1055
+ initial batch size is 1.
1056
+ ``batch_size="auto"`` with ``backend="threading"`` will dispatch
1057
+ batches of a single task at a time as the threading backend has
1058
+ very little overhead and using larger batch size has not proved to
1059
+ bring any gain in that case.
1060
+ temp_folder: str or None, default=None
1061
+ Folder to be used by the pool for memmapping large arrays
1062
+ for sharing memory with worker processes. If None, this will try in
1063
+ order:
1064
+
1065
+ - a folder pointed by the JOBLIB_TEMP_FOLDER environment
1066
+ variable,
1067
+ - /dev/shm if the folder exists and is writable: this is a
1068
+ RAM disk filesystem available by default on modern Linux
1069
+ distributions,
1070
+ - the default system temporary folder that can be
1071
+ overridden with TMP, TMPDIR or TEMP environment
1072
+ variables, typically /tmp under Unix operating systems.
1073
+
1074
+ Only active when ``backend="loky"`` or ``"multiprocessing"``.
1075
+ max_nbytes int, str, or None, optional, default='1M'
1076
+ Threshold on the size of arrays passed to the workers that
1077
+ triggers automated memory mapping in temp_folder. Can be an int
1078
+ in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
1079
+ Use None to disable memmapping of large arrays.
1080
+ Only active when ``backend="loky"`` or ``"multiprocessing"``.
1081
+ mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
1082
+ Memmapping mode for numpy arrays passed to workers. None will
1083
+ disable memmapping, other modes defined in the numpy.memmap doc:
1084
+ https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
1085
+ Also, see 'max_nbytes' parameter documentation for more details.
1086
+ backend_kwargs: dict, optional
1087
+ Additional parameters to pass to the backend `configure` method.
1088
+
1089
+ Notes
1090
+ -----
1091
+
1092
+ This object uses workers to compute in parallel the application of a
1093
+ function to many different arguments. The main functionality it brings
1094
+ in addition to using the raw multiprocessing or concurrent.futures API
1095
+ are (see examples for details):
1096
+
1097
+ * More readable code, in particular since it avoids
1098
+ constructing list of arguments.
1099
+
1100
+ * Easier debugging:
1101
+ - informative tracebacks even when the error happens on
1102
+ the client side
1103
+ - using 'n_jobs=1' enables to turn off parallel computing
1104
+ for debugging without changing the codepath
1105
+ - early capture of pickling errors
1106
+
1107
+ * An optional progress meter.
1108
+
1109
+ * Interruption of multiprocesses jobs with 'Ctrl-C'
1110
+
1111
+ * Flexible pickling control for the communication to and from
1112
+ the worker processes.
1113
+
1114
+ * Ability to use shared memory efficiently with worker
1115
+ processes for large numpy-based datastructures.
1116
+
1117
+ Note that the intended usage is to run one call at a time. Multiple
1118
+ calls to the same Parallel object will result in a ``RuntimeError``
1119
+
1120
+ Examples
1121
+ --------
1122
+
1123
+ A simple example:
1124
+
1125
+ >>> from math import sqrt
1126
+ >>> from joblib import Parallel, delayed
1127
+ >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
1128
+ [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
1129
+
1130
+ Reshaping the output when the function has several return
1131
+ values:
1132
+
1133
+ >>> from math import modf
1134
+ >>> from joblib import Parallel, delayed
1135
+ >>> r = Parallel(n_jobs=1)(delayed(modf)(i/2.) for i in range(10))
1136
+ >>> res, i = zip(*r)
1137
+ >>> res
1138
+ (0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5)
1139
+ >>> i
1140
+ (0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0)
1141
+
1142
+ The progress meter: the higher the value of `verbose`, the more
1143
+ messages:
1144
+
1145
+ >>> from time import sleep
1146
+ >>> from joblib import Parallel, delayed
1147
+ >>> r = Parallel(n_jobs=2, verbose=10)(
1148
+ ... delayed(sleep)(.2) for _ in range(10)) #doctest: +SKIP
1149
+ [Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 0.6s
1150
+ [Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 0.8s
1151
+ [Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 1.4s finished
1152
+
1153
+ Traceback example, note how the line of the error is indicated
1154
+ as well as the values of the parameter passed to the function that
1155
+ triggered the exception, even though the traceback happens in the
1156
+ child process:
1157
+
1158
+ >>> from heapq import nlargest
1159
+ >>> from joblib import Parallel, delayed
1160
+ >>> Parallel(n_jobs=2)(
1161
+ ... delayed(nlargest)(2, n) for n in (range(4), 'abcde', 3))
1162
+ ... # doctest: +SKIP
1163
+ -----------------------------------------------------------------------
1164
+ Sub-process traceback:
1165
+ -----------------------------------------------------------------------
1166
+ TypeError Mon Nov 12 11:37:46 2012
1167
+ PID: 12934 Python 2.7.3: /usr/bin/python
1168
+ ........................................................................
1169
+ /usr/lib/python2.7/heapq.pyc in nlargest(n=2, iterable=3, key=None)
1170
+ 419 if n >= size:
1171
+ 420 return sorted(iterable, key=key, reverse=True)[:n]
1172
+ 421
1173
+ 422 # When key is none, use simpler decoration
1174
+ 423 if key is None:
1175
+ --> 424 it = izip(iterable, count(0,-1)) # decorate
1176
+ 425 result = _nlargest(n, it)
1177
+ 426 return map(itemgetter(0), result) # undecorate
1178
+ 427
1179
+ 428 # General case, slowest method
1180
+ TypeError: izip argument #1 must support iteration
1181
+ _______________________________________________________________________
1182
+
1183
+
1184
+ Using pre_dispatch in a producer/consumer situation, where the
1185
+ data is generated on the fly. Note how the producer is first
1186
+ called 3 times before the parallel loop is initiated, and then
1187
+ called to generate new data on the fly:
1188
+
1189
+ >>> from math import sqrt
1190
+ >>> from joblib import Parallel, delayed
1191
+ >>> def producer():
1192
+ ... for i in range(6):
1193
+ ... print('Produced %s' % i)
1194
+ ... yield i
1195
+ >>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')(
1196
+ ... delayed(sqrt)(i) for i in producer()) #doctest: +SKIP
1197
+ Produced 0
1198
+ Produced 1
1199
+ Produced 2
1200
+ [Parallel(n_jobs=2)]: Done 1 jobs | elapsed: 0.0s
1201
+ Produced 3
1202
+ [Parallel(n_jobs=2)]: Done 2 jobs | elapsed: 0.0s
1203
+ Produced 4
1204
+ [Parallel(n_jobs=2)]: Done 3 jobs | elapsed: 0.0s
1205
+ Produced 5
1206
+ [Parallel(n_jobs=2)]: Done 4 jobs | elapsed: 0.0s
1207
+ [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s remaining: 0.0s
1208
+ [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s finished
1209
+
1210
+ """ # noqa: E501
1211
+
1212
+ def __init__(
1213
+ self,
1214
+ n_jobs=default_parallel_config["n_jobs"],
1215
+ backend=default_parallel_config["backend"],
1216
+ return_as="list",
1217
+ verbose=default_parallel_config["verbose"],
1218
+ timeout=None,
1219
+ pre_dispatch="2 * n_jobs",
1220
+ batch_size="auto",
1221
+ temp_folder=default_parallel_config["temp_folder"],
1222
+ max_nbytes=default_parallel_config["max_nbytes"],
1223
+ mmap_mode=default_parallel_config["mmap_mode"],
1224
+ prefer=default_parallel_config["prefer"],
1225
+ require=default_parallel_config["require"],
1226
+ **backend_kwargs,
1227
+ ):
1228
+ # Initiate parent Logger class state
1229
+ super().__init__()
1230
+
1231
+ # Interpret n_jobs=None as 'unset'
1232
+ if n_jobs is None:
1233
+ n_jobs = default_parallel_config["n_jobs"]
1234
+
1235
+ active_backend, context_config = _get_active_backend(
1236
+ prefer=prefer, require=require, verbose=verbose
1237
+ )
1238
+
1239
+ nesting_level = active_backend.nesting_level
1240
+
1241
+ self.verbose = _get_config_param(verbose, context_config, "verbose")
1242
+ self.timeout = timeout
1243
+ self.pre_dispatch = pre_dispatch
1244
+
1245
+ if return_as not in {"list", "generator", "generator_unordered"}:
1246
+ raise ValueError(
1247
+ 'Expected `return_as` parameter to be a string equal to "list"'
1248
+ f',"generator" or "generator_unordered", but got {return_as} '
1249
+ "instead."
1250
+ )
1251
+ self.return_as = return_as
1252
+ self.return_generator = return_as != "list"
1253
+ self.return_ordered = return_as != "generator_unordered"
1254
+
1255
+ # Check if we are under a parallel_config or parallel_backend
1256
+ # context manager and use the config from the context manager
1257
+ # for arguments that are not explicitly set.
1258
+ self._backend_kwargs = {
1259
+ **backend_kwargs,
1260
+ **{
1261
+ k: _get_config_param(param, context_config, k)
1262
+ for param, k in [
1263
+ (max_nbytes, "max_nbytes"),
1264
+ (temp_folder, "temp_folder"),
1265
+ (mmap_mode, "mmap_mode"),
1266
+ (prefer, "prefer"),
1267
+ (require, "require"),
1268
+ (verbose, "verbose"),
1269
+ ]
1270
+ },
1271
+ }
1272
+
1273
+ if isinstance(self._backend_kwargs["max_nbytes"], str):
1274
+ self._backend_kwargs["max_nbytes"] = memstr_to_bytes(
1275
+ self._backend_kwargs["max_nbytes"]
1276
+ )
1277
+ self._backend_kwargs["verbose"] = max(0, self._backend_kwargs["verbose"] - 50)
1278
+
1279
+ if DEFAULT_MP_CONTEXT is not None:
1280
+ self._backend_kwargs["context"] = DEFAULT_MP_CONTEXT
1281
+ elif hasattr(mp, "get_context"):
1282
+ self._backend_kwargs["context"] = mp.get_context()
1283
+
1284
+ if backend is default_parallel_config["backend"] or backend is None:
1285
+ backend = active_backend
1286
+
1287
+ elif isinstance(backend, ParallelBackendBase):
1288
+ # Use provided backend as is, with the current nesting_level if it
1289
+ # is not set yet.
1290
+ if backend.nesting_level is None:
1291
+ backend.nesting_level = nesting_level
1292
+
1293
+ elif hasattr(backend, "Pool") and hasattr(backend, "Lock"):
1294
+ # Make it possible to pass a custom multiprocessing context as
1295
+ # backend to change the start method to forkserver or spawn or
1296
+ # preload modules on the forkserver helper process.
1297
+ self._backend_kwargs["context"] = backend
1298
+ backend = MultiprocessingBackend(nesting_level=nesting_level)
1299
+
1300
+ elif backend not in BACKENDS and backend in MAYBE_AVAILABLE_BACKENDS:
1301
+ warnings.warn(
1302
+ f"joblib backend '{backend}' is not available on "
1303
+ f"your system, falling back to {DEFAULT_BACKEND}.",
1304
+ UserWarning,
1305
+ stacklevel=2,
1306
+ )
1307
+ BACKENDS[backend] = BACKENDS[DEFAULT_BACKEND]
1308
+ backend = BACKENDS[DEFAULT_BACKEND](nesting_level=nesting_level)
1309
+
1310
+ else:
1311
+ try:
1312
+ backend_factory = BACKENDS[backend]
1313
+ except KeyError as e:
1314
+ raise ValueError(
1315
+ "Invalid backend: %s, expected one of %r"
1316
+ % (backend, sorted(BACKENDS.keys()))
1317
+ ) from e
1318
+ backend = backend_factory(nesting_level=nesting_level)
1319
+
1320
+ n_jobs = _get_config_param(n_jobs, context_config, "n_jobs")
1321
+ if n_jobs is None:
1322
+ # No specific context override and no specific value request:
1323
+ # default to the default of the backend.
1324
+ n_jobs = backend.default_n_jobs
1325
+ try:
1326
+ n_jobs = int(n_jobs)
1327
+ except ValueError:
1328
+ raise ValueError("n_jobs could not be converted to int")
1329
+ self.n_jobs = n_jobs
1330
+
1331
+ if require == "sharedmem" and not getattr(backend, "supports_sharedmem", False):
1332
+ raise ValueError("Backend %s does not support shared memory" % backend)
1333
+
1334
+ if batch_size == "auto" or isinstance(batch_size, Integral) and batch_size > 0:
1335
+ self.batch_size = batch_size
1336
+ else:
1337
+ raise ValueError(
1338
+ "batch_size must be 'auto' or a positive integer, got: %r" % batch_size
1339
+ )
1340
+
1341
+ if not isinstance(backend, SequentialBackend):
1342
+ if self.return_generator and not backend.supports_return_generator:
1343
+ raise ValueError(
1344
+ "Backend {} does not support return_as={}".format(
1345
+ backend, return_as
1346
+ )
1347
+ )
1348
+ # This lock is used to coordinate the main thread of this process
1349
+ # with the async callback thread of our the pool.
1350
+ self._lock = threading.RLock()
1351
+ self._jobs = collections.deque()
1352
+ self._jobs_set = set()
1353
+ self._pending_outputs = list()
1354
+ self._ready_batches = queue.Queue()
1355
+ self._reducer_callback = None
1356
+
1357
+ # Internal variables
1358
+ self._backend = backend
1359
+ self._running = False
1360
+ self._managed_backend = False
1361
+ self._id = uuid4().hex
1362
+ self._call_ref = None
1363
+
1364
+ def __enter__(self):
1365
+ self._managed_backend = True
1366
+ self._calling = False
1367
+ self._initialize_backend()
1368
+ return self
1369
+
1370
+ def __exit__(self, exc_type, exc_value, traceback):
1371
+ self._managed_backend = False
1372
+ if self.return_generator and self._calling:
1373
+ self._abort()
1374
+ self._terminate_and_reset()
1375
+
1376
+ def _initialize_backend(self):
1377
+ """Build a process or thread pool and return the number of workers"""
1378
+ try:
1379
+ n_jobs = self._backend.configure(
1380
+ n_jobs=self.n_jobs, parallel=self, **self._backend_kwargs
1381
+ )
1382
+ if self.timeout is not None and not self._backend.supports_timeout:
1383
+ warnings.warn(
1384
+ "The backend class {!r} does not support timeout. "
1385
+ "You have set 'timeout={}' in Parallel but "
1386
+ "the 'timeout' parameter will not be used.".format(
1387
+ self._backend.__class__.__name__, self.timeout
1388
+ )
1389
+ )
1390
+
1391
+ except FallbackToBackend as e:
1392
+ # Recursively initialize the backend in case of requested fallback.
1393
+ self._backend = e.backend
1394
+ n_jobs = self._initialize_backend()
1395
+
1396
+ return n_jobs
1397
+
1398
+ def _effective_n_jobs(self):
1399
+ if self._backend:
1400
+ return self._backend.effective_n_jobs(self.n_jobs)
1401
+ return 1
1402
+
1403
+ def _terminate_and_reset(self):
1404
+ if hasattr(self._backend, "stop_call") and self._calling:
1405
+ self._backend.stop_call()
1406
+ self._calling = False
1407
+ if not self._managed_backend:
1408
+ self._backend.terminate()
1409
+
1410
+ def _dispatch(self, batch):
1411
+ """Queue the batch for computing, with or without multiprocessing
1412
+
1413
+ WARNING: this method is not thread-safe: it should be only called
1414
+ indirectly via dispatch_one_batch.
1415
+
1416
+ """
1417
+ # If job.get() catches an exception, it closes the queue:
1418
+ if self._aborting:
1419
+ return
1420
+
1421
+ batch_size = len(batch)
1422
+
1423
+ self.n_dispatched_tasks += batch_size
1424
+ self.n_dispatched_batches += 1
1425
+
1426
+ dispatch_timestamp = time.time()
1427
+
1428
+ batch_tracker = BatchCompletionCallBack(dispatch_timestamp, batch_size, self)
1429
+
1430
+ self._register_new_job(batch_tracker)
1431
+
1432
+ # If return_ordered is False, the batch_tracker is not stored in the
1433
+ # jobs queue at the time of submission. Instead, it will be appended to
1434
+ # the queue by itself as soon as the callback is triggered to be able
1435
+ # to return the results in the order of completion.
1436
+
1437
+ job = self._backend.submit(batch, callback=batch_tracker)
1438
+ batch_tracker.register_job(job)
1439
+
1440
+ def _register_new_job(self, batch_tracker):
1441
+ if self.return_ordered:
1442
+ self._jobs.append(batch_tracker)
1443
+ else:
1444
+ self._jobs_set.add(batch_tracker)
1445
+
1446
+ def dispatch_next(self):
1447
+ """Dispatch more data for parallel processing
1448
+
1449
+ This method is meant to be called concurrently by the multiprocessing
1450
+ callback. We rely on the thread-safety of dispatch_one_batch to protect
1451
+ against concurrent consumption of the unprotected iterator.
1452
+ """
1453
+ if not self.dispatch_one_batch(self._original_iterator):
1454
+ self._iterating = False
1455
+ self._original_iterator = None
1456
+
1457
+ def dispatch_one_batch(self, iterator):
1458
+ """Prefetch the tasks for the next batch and dispatch them.
1459
+
1460
+ The effective size of the batch is computed here.
1461
+ If there are no more jobs to dispatch, return False, else return True.
1462
+
1463
+ The iterator consumption and dispatching is protected by the same
1464
+ lock so calling this function should be thread safe.
1465
+
1466
+ """
1467
+
1468
+ if self._aborting:
1469
+ return False
1470
+
1471
+ batch_size = self._get_batch_size()
1472
+
1473
+ with self._lock:
1474
+ # to ensure an even distribution of the workload between workers,
1475
+ # we look ahead in the original iterators more than batch_size
1476
+ # tasks - However, we keep consuming only one batch at each
1477
+ # dispatch_one_batch call. The extra tasks are stored in a local
1478
+ # queue, _ready_batches, that is looked-up prior to re-consuming
1479
+ # tasks from the origal iterator.
1480
+ try:
1481
+ tasks = self._ready_batches.get(block=False)
1482
+ except queue.Empty:
1483
+ # slice the iterator n_jobs * batchsize items at a time. If the
1484
+ # slice returns less than that, then the current batchsize puts
1485
+ # too much weight on a subset of workers, while other may end
1486
+ # up starving. So in this case, re-scale the batch size
1487
+ # accordingly to distribute evenly the last items between all
1488
+ # workers.
1489
+ n_jobs = self._cached_effective_n_jobs
1490
+ big_batch_size = batch_size * n_jobs
1491
+
1492
+ try:
1493
+ islice = list(itertools.islice(iterator, big_batch_size))
1494
+ except Exception as e:
1495
+ # Handle the fact that the generator of task raised an
1496
+ # exception. As this part of the code can be executed in
1497
+ # a thread internal to the backend, register a task with
1498
+ # an error that will be raised in the user's thread.
1499
+ if isinstance(e.__context__, queue.Empty):
1500
+ # Suppress the cause of the exception if it is
1501
+ # queue.Empty to avoid cluttered traceback. Only do it
1502
+ # if the __context__ is really empty to avoid messing
1503
+ # with causes of the original error.
1504
+ e.__cause__ = None
1505
+ batch_tracker = BatchCompletionCallBack(0, batch_size, self)
1506
+ self._register_new_job(batch_tracker)
1507
+ batch_tracker._register_outcome(dict(result=e, status=TASK_ERROR))
1508
+ return True
1509
+
1510
+ if len(islice) == 0:
1511
+ return False
1512
+ elif (
1513
+ iterator is self._original_iterator and len(islice) < big_batch_size
1514
+ ):
1515
+ # We reached the end of the original iterator (unless
1516
+ # iterator is the ``pre_dispatch``-long initial slice of
1517
+ # the original iterator) -- decrease the batch size to
1518
+ # account for potential variance in the batches running
1519
+ # time.
1520
+ final_batch_size = max(1, len(islice) // (10 * n_jobs))
1521
+ else:
1522
+ final_batch_size = max(1, len(islice) // n_jobs)
1523
+
1524
+ # enqueue n_jobs batches in a local queue
1525
+ for i in range(0, len(islice), final_batch_size):
1526
+ tasks = BatchedCalls(
1527
+ islice[i : i + final_batch_size],
1528
+ self._backend.get_nested_backend(),
1529
+ self._reducer_callback,
1530
+ self._pickle_cache,
1531
+ )
1532
+ self._ready_batches.put(tasks)
1533
+
1534
+ # finally, get one task.
1535
+ tasks = self._ready_batches.get(block=False)
1536
+ if len(tasks) == 0:
1537
+ # No more tasks available in the iterator: tell caller to stop.
1538
+ return False
1539
+ else:
1540
+ self._dispatch(tasks)
1541
+ return True
1542
+
1543
+ def _get_batch_size(self):
1544
+ """Returns the effective batch size for dispatch"""
1545
+ if self.batch_size == "auto":
1546
+ return self._backend.compute_batch_size()
1547
+ else:
1548
+ # Fixed batch size strategy
1549
+ return self.batch_size
1550
+
1551
+ def _print(self, msg):
1552
+ """Display the message on stout or stderr depending on verbosity"""
1553
+ # XXX: Not using the logger framework: need to
1554
+ # learn to use logger better.
1555
+ if not self.verbose:
1556
+ return
1557
+ if self.verbose < 50:
1558
+ writer = sys.stderr.write
1559
+ else:
1560
+ writer = sys.stdout.write
1561
+ writer(f"[{self}]: {msg}\n")
1562
+
1563
+ def _is_completed(self):
1564
+ """Check if all tasks have been completed"""
1565
+ return self.n_completed_tasks == self.n_dispatched_tasks and not (
1566
+ self._iterating or self._aborting
1567
+ )
1568
+
1569
+ def print_progress(self):
1570
+ """Display the process of the parallel execution only a fraction
1571
+ of time, controlled by self.verbose.
1572
+ """
1573
+
1574
+ if not self.verbose:
1575
+ return
1576
+
1577
+ if self.n_tasks is not None and self.n_tasks > 0:
1578
+ width = floor(log10(self.n_tasks)) + 1
1579
+ else:
1580
+ width = 3
1581
+ elapsed_time = time.time() - self._start_time
1582
+
1583
+ if self._is_completed():
1584
+ # Make sure that we get a last message telling us we are done
1585
+ self._print(
1586
+ f"Done {self.n_completed_tasks:{width}d} out of "
1587
+ f"{self.n_completed_tasks:{width}d} | elapsed: "
1588
+ f"{short_format_time(elapsed_time)} finished"
1589
+ )
1590
+ return
1591
+
1592
+ # Original job iterator becomes None once it has been fully
1593
+ # consumed: at this point we know the total number of jobs and we are
1594
+ # able to display an estimation of the remaining time based on already
1595
+ # completed jobs. Otherwise, we simply display the number of completed
1596
+ # tasks.
1597
+ elif self._original_iterator is not None:
1598
+ if _verbosity_filter(self.n_dispatched_batches, self.verbose):
1599
+ return
1600
+ fmt_time = f"| elapsed: {short_format_time(elapsed_time)}"
1601
+ index = self.n_completed_tasks
1602
+ if self.n_tasks is not None:
1603
+ self._print(
1604
+ f"Done {index:{width}d} out of {self.n_tasks:{width}d} {fmt_time}"
1605
+ )
1606
+ else:
1607
+ pad = " " * (len("out of ") + width - len("tasks"))
1608
+ self._print(f"Done {index:{width}d} tasks {pad}{fmt_time}")
1609
+ else:
1610
+ index = self.n_completed_tasks
1611
+ # We are finished dispatching
1612
+ total_tasks = self.n_dispatched_tasks
1613
+ # We always display the first loop
1614
+ if index != 0:
1615
+ # Display depending on the number of remaining items
1616
+ # A message as soon as we finish dispatching, cursor is 0
1617
+ cursor = total_tasks - index + 1 - self._pre_dispatch_amount
1618
+ frequency = (total_tasks // self.verbose) + 1
1619
+ is_last_item = index + 1 == total_tasks
1620
+ if is_last_item or cursor % frequency:
1621
+ return
1622
+ remaining_time = (elapsed_time / max(index, 1)) * (
1623
+ self.n_dispatched_tasks - index
1624
+ )
1625
+ # only display status if remaining time is greater or equal to 0
1626
+ self._print(
1627
+ f"Done {index:{width}d} out of {total_tasks:{width}d} "
1628
+ f"| elapsed: {short_format_time(elapsed_time)} remaining: "
1629
+ f"{short_format_time(remaining_time)}"
1630
+ )
1631
+
1632
+ def _abort(self):
1633
+ # Stop dispatching new jobs in the async callback thread
1634
+ self._aborting = True
1635
+
1636
+ # If the backend allows it, cancel or kill remaining running
1637
+ # tasks without waiting for the results as we will raise
1638
+ # the exception we got back to the caller instead of returning
1639
+ # any result.
1640
+ backend = self._backend
1641
+ if not self._aborted and hasattr(backend, "abort_everything"):
1642
+ # If the backend is managed externally we need to make sure
1643
+ # to leave it in a working state to allow for future jobs
1644
+ # scheduling.
1645
+ ensure_ready = self._managed_backend
1646
+ backend.abort_everything(ensure_ready=ensure_ready)
1647
+ self._aborted = True
1648
+
1649
+ def _start(self, iterator, pre_dispatch):
1650
+ # Only set self._iterating to True if at least a batch
1651
+ # was dispatched. In particular this covers the edge
1652
+ # case of Parallel used with an exhausted iterator. If
1653
+ # self._original_iterator is None, then this means either
1654
+ # that pre_dispatch == "all", n_jobs == 1 or that the first batch
1655
+ # was very quick and its callback already dispatched all the
1656
+ # remaining jobs.
1657
+ self._iterating = False
1658
+ if self.dispatch_one_batch(iterator):
1659
+ self._iterating = self._original_iterator is not None
1660
+
1661
+ while self.dispatch_one_batch(iterator):
1662
+ pass
1663
+
1664
+ if pre_dispatch == "all":
1665
+ # The iterable was consumed all at once by the above for loop.
1666
+ # No need to wait for async callbacks to trigger to
1667
+ # consumption.
1668
+ self._iterating = False
1669
+
1670
+ def _get_outputs(self, iterator, pre_dispatch):
1671
+ """Iterator returning the tasks' output as soon as they are ready."""
1672
+ dispatch_thread_id = threading.get_ident()
1673
+ detach_generator_exit = False
1674
+ try:
1675
+ self._start(iterator, pre_dispatch)
1676
+ # first yield returns None, for internal use only. This ensures
1677
+ # that we enter the try/except block and start dispatching the
1678
+ # tasks.
1679
+ yield
1680
+
1681
+ with self._backend.retrieval_context():
1682
+ yield from self._retrieve()
1683
+
1684
+ except GeneratorExit:
1685
+ # The generator has been garbage collected before being fully
1686
+ # consumed. This aborts the remaining tasks if possible and warn
1687
+ # the user if necessary.
1688
+ self._exception = True
1689
+
1690
+ # In some interpreters such as PyPy, GeneratorExit can be raised in
1691
+ # a different thread than the one used to start the dispatch of the
1692
+ # parallel tasks. This can lead to hang when a thread attempts to
1693
+ # join itself. As workaround, we detach the execution of the
1694
+ # aborting code to a dedicated thread. We then need to make sure
1695
+ # the rest of the function does not call `_terminate_and_reset`
1696
+ # in finally.
1697
+ if dispatch_thread_id != threading.get_ident():
1698
+ warnings.warn(
1699
+ "A generator produced by joblib.Parallel has been "
1700
+ "gc'ed in an unexpected thread. This behavior should "
1701
+ "not cause major -issues but to make sure, please "
1702
+ "report this warning and your use case at "
1703
+ "https://github.com/joblib/joblib/issues so it can "
1704
+ "be investigated."
1705
+ )
1706
+
1707
+ detach_generator_exit = True
1708
+ _parallel = self
1709
+
1710
+ class _GeneratorExitThread(threading.Thread):
1711
+ def run(self):
1712
+ _parallel._abort()
1713
+ if _parallel.return_generator:
1714
+ _parallel._warn_exit_early()
1715
+ _parallel._terminate_and_reset()
1716
+
1717
+ _GeneratorExitThread(name="GeneratorExitThread").start()
1718
+ return
1719
+
1720
+ # Otherwise, we are in the thread that started the dispatch: we can
1721
+ # safely abort the execution and warn the user.
1722
+ self._abort()
1723
+ if self.return_generator:
1724
+ self._warn_exit_early()
1725
+
1726
+ raise
1727
+
1728
+ # Note: we catch any BaseException instead of just Exception instances
1729
+ # to also include KeyboardInterrupt
1730
+ except BaseException:
1731
+ self._exception = True
1732
+ self._abort()
1733
+ raise
1734
+ finally:
1735
+ # Store the unconsumed tasks and terminate the workers if necessary
1736
+ _remaining_outputs = [] if self._exception else self._jobs
1737
+ self._jobs = collections.deque()
1738
+ self._jobs_set = set()
1739
+ self._running = False
1740
+ if not detach_generator_exit:
1741
+ self._terminate_and_reset()
1742
+
1743
+ while len(_remaining_outputs) > 0:
1744
+ batched_results = _remaining_outputs.popleft()
1745
+ batched_results = batched_results.get_result(self.timeout)
1746
+ for result in batched_results:
1747
+ yield result
1748
+
1749
+ def _wait_retrieval(self):
1750
+ """Return True if we need to continue retrieving some tasks."""
1751
+
1752
+ # If the input load is still being iterated over, it means that tasks
1753
+ # are still on the dispatch waitlist and their results will need to
1754
+ # be retrieved later on.
1755
+ if self._iterating:
1756
+ return True
1757
+
1758
+ # If some of the dispatched tasks are still being processed by the
1759
+ # workers, wait for the compute to finish before starting retrieval
1760
+ if self.n_completed_tasks < self.n_dispatched_tasks:
1761
+ return True
1762
+
1763
+ # For backends that does not support retrieving asynchronously the
1764
+ # result to the main process, all results must be carefully retrieved
1765
+ # in the _retrieve loop in the main thread while the backend is alive.
1766
+ # For other backends, the actual retrieval is done asynchronously in
1767
+ # the callback thread, and we can terminate the backend before the
1768
+ # `self._jobs` result list has been emptied. The remaining results
1769
+ # will be collected in the `finally` step of the generator.
1770
+ if not self._backend.supports_retrieve_callback:
1771
+ if len(self._jobs) > 0:
1772
+ return True
1773
+
1774
+ return False
1775
+
1776
+ def _retrieve(self):
1777
+ timeout_control_job = None
1778
+ while self._wait_retrieval():
1779
+ # If the callback thread of a worker has signaled that its task
1780
+ # triggered an exception, or if the retrieval loop has raised an
1781
+ # exception (e.g. `GeneratorExit`), exit the loop and surface the
1782
+ # worker traceback.
1783
+ if self._aborting:
1784
+ self._raise_error_fast()
1785
+ break
1786
+
1787
+ nb_jobs = len(self._jobs)
1788
+ # Now wait for a job to be ready for retrieval.
1789
+ if self.return_ordered:
1790
+ # Case ordered: wait for completion (or error) of the next job
1791
+ # that have been dispatched and not retrieved yet. If no job
1792
+ # have been dispatched yet, wait for dispatch.
1793
+ # We assume that the time to wait for the next job to be
1794
+ # dispatched is always low, so that the timeout
1795
+ # control only have to be done on the amount of time the next
1796
+ # dispatched job is pending.
1797
+ if (nb_jobs == 0) or (
1798
+ self._jobs[0].get_status(timeout=self.timeout) == TASK_PENDING
1799
+ ):
1800
+ time.sleep(0.01)
1801
+ continue
1802
+
1803
+ elif nb_jobs == 0:
1804
+ # Case unordered: jobs are added to the list of jobs to
1805
+ # retrieve `self._jobs` only once completed or in error, which
1806
+ # is too late to enable timeout control in the same way than in
1807
+ # the previous case.
1808
+ # Instead, if no job is ready to be retrieved yet, we
1809
+ # arbitrarily pick a dispatched job, and the timeout control is
1810
+ # done such that an error is raised if this control job
1811
+ # timeouts before any other dispatched job has completed and
1812
+ # been added to `self._jobs` to be retrieved.
1813
+ if timeout_control_job is None:
1814
+ timeout_control_job = next(iter(self._jobs_set), None)
1815
+
1816
+ # NB: it can be None if no job has been dispatched yet.
1817
+ if timeout_control_job is not None:
1818
+ timeout_control_job.get_status(timeout=self.timeout)
1819
+
1820
+ time.sleep(0.01)
1821
+ continue
1822
+
1823
+ elif timeout_control_job is not None:
1824
+ # Case unordered, when `nb_jobs > 0`:
1825
+ # It means that a job is ready to be retrieved, so no timeout
1826
+ # will occur during this iteration.
1827
+ # Before proceeding to retrieval of the next ready job, reset
1828
+ # the timeout control state to prepare the next iteration.
1829
+ timeout_control_job._completion_timeout_counter = None
1830
+ timeout_control_job = None
1831
+
1832
+ # We need to be careful: the job list can be filling up as
1833
+ # we empty it and Python list are not thread-safe by
1834
+ # default hence the use of the lock
1835
+ with self._lock:
1836
+ batched_results = self._jobs.popleft()
1837
+ if not self.return_ordered:
1838
+ self._jobs_set.remove(batched_results)
1839
+
1840
+ # Flatten the batched results to output one output at a time
1841
+ batched_results = batched_results.get_result(self.timeout)
1842
+ for result in batched_results:
1843
+ self._nb_consumed += 1
1844
+ yield result
1845
+
1846
+ def _raise_error_fast(self):
1847
+ """If we are aborting, raise if a job caused an error."""
1848
+
1849
+ # Find the first job whose status is TASK_ERROR if it exists.
1850
+ with self._lock:
1851
+ error_job = next(
1852
+ (job for job in self._jobs if job.status == TASK_ERROR), None
1853
+ )
1854
+
1855
+ # If this error job exists, immediately raise the error by
1856
+ # calling get_result. This job might not exists if abort has been
1857
+ # called directly or if the generator is gc'ed.
1858
+ if error_job is not None:
1859
+ error_job.get_result(self.timeout)
1860
+
1861
+ def _warn_exit_early(self):
1862
+ """Warn the user if the generator is gc'ed before being consumned."""
1863
+ ready_outputs = self.n_completed_tasks - self._nb_consumed
1864
+ is_completed = self._is_completed()
1865
+ msg = ""
1866
+ if ready_outputs:
1867
+ msg += (
1868
+ f"{ready_outputs} tasks have been successfully executed but not used."
1869
+ )
1870
+ if not is_completed:
1871
+ msg += " Additionally, "
1872
+
1873
+ if not is_completed:
1874
+ msg += (
1875
+ f"{self.n_dispatched_tasks - self.n_completed_tasks} tasks "
1876
+ "which were still being processed by the workers have been "
1877
+ "cancelled."
1878
+ )
1879
+
1880
+ if msg:
1881
+ msg += (
1882
+ " You could benefit from adjusting the input task "
1883
+ "iterator to limit unnecessary computation time."
1884
+ )
1885
+
1886
+ warnings.warn(msg)
1887
+
1888
+ def _get_sequential_output(self, iterable):
1889
+ """Separate loop for sequential output.
1890
+
1891
+ This simplifies the traceback in case of errors and reduces the
1892
+ overhead of calling sequential tasks with `joblib`.
1893
+ """
1894
+ try:
1895
+ self._iterating = True
1896
+ self._original_iterator = iterable
1897
+ batch_size = self._get_batch_size()
1898
+
1899
+ if batch_size != 1:
1900
+ it = iter(iterable)
1901
+ iterable_batched = iter(
1902
+ lambda: tuple(itertools.islice(it, batch_size)), ()
1903
+ )
1904
+ iterable = (task for batch in iterable_batched for task in batch)
1905
+
1906
+ # first yield returns None, for internal use only. This ensures
1907
+ # that we enter the try/except block and setup the generator.
1908
+ yield None
1909
+
1910
+ # Sequentially call the tasks and yield the results.
1911
+ for func, args, kwargs in iterable:
1912
+ self.n_dispatched_batches += 1
1913
+ self.n_dispatched_tasks += 1
1914
+ res = func(*args, **kwargs)
1915
+ self.n_completed_tasks += 1
1916
+ self.print_progress()
1917
+ yield res
1918
+ self._nb_consumed += 1
1919
+ except BaseException:
1920
+ self._exception = True
1921
+ self._aborting = True
1922
+ self._aborted = True
1923
+ raise
1924
+ finally:
1925
+ self._running = False
1926
+ self._iterating = False
1927
+ self._original_iterator = None
1928
+ self.print_progress()
1929
+
1930
+ def _reset_run_tracking(self):
1931
+ """Reset the counters and flags used to track the execution."""
1932
+
1933
+ # Makes sur the parallel instance was not previously running in a
1934
+ # thread-safe way.
1935
+ with getattr(self, "_lock", nullcontext()):
1936
+ if self._running:
1937
+ msg = "This Parallel instance is already running !"
1938
+ if self.return_generator is True:
1939
+ msg += (
1940
+ " Before submitting new tasks, you must wait for the "
1941
+ "completion of all the previous tasks, or clean all "
1942
+ "references to the output generator."
1943
+ )
1944
+ raise RuntimeError(msg)
1945
+ self._running = True
1946
+
1947
+ # Counter to keep track of the task dispatched and completed.
1948
+ self.n_dispatched_batches = 0
1949
+ self.n_dispatched_tasks = 0
1950
+ self.n_completed_tasks = 0
1951
+
1952
+ # Following count is incremented by one each time the user iterates
1953
+ # on the output generator, it is used to prepare an informative
1954
+ # warning message in case the generator is deleted before all the
1955
+ # dispatched tasks have been consumed.
1956
+ self._nb_consumed = 0
1957
+
1958
+ # Following flags are used to synchronize the threads in case one of
1959
+ # the tasks error-out to ensure that all workers abort fast and that
1960
+ # the backend terminates properly.
1961
+
1962
+ # Set to True as soon as a worker signals that a task errors-out
1963
+ self._exception = False
1964
+ # Set to True in case of early termination following an incident
1965
+ self._aborting = False
1966
+ # Set to True after abortion is complete
1967
+ self._aborted = False
1968
+
1969
+ def __call__(self, iterable):
1970
+ """Main function to dispatch parallel tasks."""
1971
+
1972
+ self._reset_run_tracking()
1973
+ self.n_tasks = len(iterable) if hasattr(iterable, "__len__") else None
1974
+ self._start_time = time.time()
1975
+
1976
+ if not self._managed_backend:
1977
+ n_jobs = self._initialize_backend()
1978
+ else:
1979
+ n_jobs = self._effective_n_jobs()
1980
+
1981
+ if n_jobs == 1:
1982
+ # If n_jobs==1, run the computation sequentially and return
1983
+ # immediately to avoid overheads.
1984
+ output = self._get_sequential_output(iterable)
1985
+ next(output)
1986
+ return output if self.return_generator else list(output)
1987
+
1988
+ # Let's create an ID that uniquely identifies the current call. If the
1989
+ # call is interrupted early and that the same instance is immediately
1990
+ # reused, this id will be used to prevent workers that were
1991
+ # concurrently finalizing a task from the previous call to run the
1992
+ # callback.
1993
+ with self._lock:
1994
+ self._call_id = uuid4().hex
1995
+
1996
+ # self._effective_n_jobs should be called in the Parallel.__call__
1997
+ # thread only -- store its value in an attribute for further queries.
1998
+ self._cached_effective_n_jobs = n_jobs
1999
+
2000
+ if isinstance(self._backend, LokyBackend):
2001
+ # For the loky backend, we add a callback executed when reducing
2002
+ # BatchCalls, that makes the loky executor use a temporary folder
2003
+ # specific to this Parallel object when pickling temporary memmaps.
2004
+ # This callback is necessary to ensure that several Parallel
2005
+ # objects using the same reusable executor don't use the same
2006
+ # temporary resources.
2007
+
2008
+ def _batched_calls_reducer_callback():
2009
+ # Relevant implementation detail: the following lines, called
2010
+ # when reducing BatchedCalls, are called in a thread-safe
2011
+ # situation, meaning that the context of the temporary folder
2012
+ # manager will not be changed in between the callback execution
2013
+ # and the end of the BatchedCalls pickling. The reason is that
2014
+ # pickling (the only place where set_current_context is used)
2015
+ # is done from a single thread (the queue_feeder_thread).
2016
+ self._backend._workers._temp_folder_manager.set_current_context( # noqa
2017
+ self._id
2018
+ )
2019
+
2020
+ self._reducer_callback = _batched_calls_reducer_callback
2021
+
2022
+ # self._effective_n_jobs should be called in the Parallel.__call__
2023
+ # thread only -- store its value in an attribute for further queries.
2024
+ self._cached_effective_n_jobs = n_jobs
2025
+
2026
+ backend_name = self._backend.__class__.__name__
2027
+ if n_jobs == 0:
2028
+ raise RuntimeError("%s has no active worker." % backend_name)
2029
+
2030
+ self._print(f"Using backend {backend_name} with {n_jobs} concurrent workers.")
2031
+ if hasattr(self._backend, "start_call"):
2032
+ self._backend.start_call()
2033
+
2034
+ # Following flag prevents double calls to `backend.stop_call`.
2035
+ self._calling = True
2036
+
2037
+ iterator = iter(iterable)
2038
+ pre_dispatch = self.pre_dispatch
2039
+
2040
+ if pre_dispatch == "all":
2041
+ # prevent further dispatch via multiprocessing callback thread
2042
+ self._original_iterator = None
2043
+ self._pre_dispatch_amount = 0
2044
+ else:
2045
+ self._original_iterator = iterator
2046
+ if hasattr(pre_dispatch, "endswith"):
2047
+ pre_dispatch = eval_expr(pre_dispatch.replace("n_jobs", str(n_jobs)))
2048
+ self._pre_dispatch_amount = pre_dispatch = int(pre_dispatch)
2049
+
2050
+ # The main thread will consume the first pre_dispatch items and
2051
+ # the remaining items will later be lazily dispatched by async
2052
+ # callbacks upon task completions.
2053
+
2054
+ # TODO: this iterator should be batch_size * n_jobs
2055
+ iterator = itertools.islice(iterator, self._pre_dispatch_amount)
2056
+
2057
+ # Use a caching dict for callables that are pickled with cloudpickle to
2058
+ # improve performances. This cache is used only in the case of
2059
+ # functions that are defined in the __main__ module, functions that
2060
+ # are defined locally (inside another function) and lambda expressions.
2061
+ self._pickle_cache = dict()
2062
+
2063
+ output = self._get_outputs(iterator, pre_dispatch)
2064
+ self._call_ref = weakref.ref(output)
2065
+
2066
+ # The first item from the output is blank, but it makes the interpreter
2067
+ # progress until it enters the Try/Except block of the generator and
2068
+ # reaches the first `yield` statement. This starts the asynchronous
2069
+ # dispatch of the tasks to the workers.
2070
+ next(output)
2071
+
2072
+ return output if self.return_generator else list(output)
2073
+
2074
+ def __repr__(self):
2075
+ return "%s(n_jobs=%s)" % (self.__class__.__name__, self.n_jobs)
TTS/Kokoro/.venv/Lib/site-packages/joblib/pool.py ADDED
@@ -0,0 +1,362 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Custom implementation of multiprocessing.Pool with custom pickler.
2
+
3
+ This module provides efficient ways of working with data stored in
4
+ shared memory with numpy.memmap arrays without inducing any memory
5
+ copy between the parent and child processes.
6
+
7
+ This module should not be imported if multiprocessing is not
8
+ available as it implements subclasses of multiprocessing Pool
9
+ that uses a custom alternative to SimpleQueue.
10
+
11
+ """
12
+ # Author: Olivier Grisel <olivier.grisel@ensta.org>
13
+ # Copyright: 2012, Olivier Grisel
14
+ # License: BSD 3 clause
15
+
16
+ import copyreg
17
+ import sys
18
+ import warnings
19
+ from time import sleep
20
+
21
+ try:
22
+ WindowsError
23
+ except NameError:
24
+ WindowsError = type(None)
25
+
26
+ from io import BytesIO
27
+
28
+ # We need the class definition to derive from it, not the multiprocessing.Pool
29
+ # factory function
30
+ from multiprocessing.pool import Pool
31
+ from pickle import HIGHEST_PROTOCOL, Pickler
32
+
33
+ from ._memmapping_reducer import TemporaryResourcesManager, get_memmapping_reducers
34
+ from ._multiprocessing_helpers import assert_spawning, mp
35
+
36
+ try:
37
+ import numpy as np
38
+ except ImportError:
39
+ np = None
40
+
41
+
42
+ ###############################################################################
43
+ # Enable custom pickling in Pool queues
44
+
45
+
46
+ class CustomizablePickler(Pickler):
47
+ """Pickler that accepts custom reducers.
48
+
49
+ TODO python2_drop : can this be simplified ?
50
+
51
+ HIGHEST_PROTOCOL is selected by default as this pickler is used
52
+ to pickle ephemeral datastructures for interprocess communication
53
+ hence no backward compatibility is required.
54
+
55
+ `reducers` is expected to be a dictionary with key/values
56
+ being `(type, callable)` pairs where `callable` is a function that
57
+ give an instance of `type` will return a tuple `(constructor,
58
+ tuple_of_objects)` to rebuild an instance out of the pickled
59
+ `tuple_of_objects` as would return a `__reduce__` method. See the
60
+ standard library documentation on pickling for more details.
61
+
62
+ """
63
+
64
+ # We override the pure Python pickler as its the only way to be able to
65
+ # customize the dispatch table without side effects in Python 2.7
66
+ # to 3.2. For Python 3.3+ leverage the new dispatch_table
67
+ # feature from https://bugs.python.org/issue14166 that makes it possible
68
+ # to use the C implementation of the Pickler which is faster.
69
+
70
+ def __init__(self, writer, reducers=None, protocol=HIGHEST_PROTOCOL):
71
+ Pickler.__init__(self, writer, protocol=protocol)
72
+ if reducers is None:
73
+ reducers = {}
74
+ if hasattr(Pickler, "dispatch"):
75
+ # Make the dispatch registry an instance level attribute instead of
76
+ # a reference to the class dictionary under Python 2
77
+ self.dispatch = Pickler.dispatch.copy()
78
+ else:
79
+ # Under Python 3 initialize the dispatch table with a copy of the
80
+ # default registry
81
+ self.dispatch_table = copyreg.dispatch_table.copy()
82
+ for type, reduce_func in reducers.items():
83
+ self.register(type, reduce_func)
84
+
85
+ def register(self, type, reduce_func):
86
+ """Attach a reducer function to a given type in the dispatch table."""
87
+ if hasattr(Pickler, "dispatch"):
88
+ # Python 2 pickler dispatching is not explicitly customizable.
89
+ # Let us use a closure to workaround this limitation.
90
+ def dispatcher(self, obj):
91
+ reduced = reduce_func(obj)
92
+ self.save_reduce(obj=obj, *reduced)
93
+
94
+ self.dispatch[type] = dispatcher
95
+ else:
96
+ self.dispatch_table[type] = reduce_func
97
+
98
+
99
+ class CustomizablePicklingQueue(object):
100
+ """Locked Pipe implementation that uses a customizable pickler.
101
+
102
+ This class is an alternative to the multiprocessing implementation
103
+ of SimpleQueue in order to make it possible to pass custom
104
+ pickling reducers, for instance to avoid memory copy when passing
105
+ memory mapped datastructures.
106
+
107
+ `reducers` is expected to be a dict with key / values being
108
+ `(type, callable)` pairs where `callable` is a function that, given an
109
+ instance of `type`, will return a tuple `(constructor, tuple_of_objects)`
110
+ to rebuild an instance out of the pickled `tuple_of_objects` as would
111
+ return a `__reduce__` method.
112
+
113
+ See the standard library documentation on pickling for more details.
114
+ """
115
+
116
+ def __init__(self, context, reducers=None):
117
+ self._reducers = reducers
118
+ self._reader, self._writer = context.Pipe(duplex=False)
119
+ self._rlock = context.Lock()
120
+ if sys.platform == "win32":
121
+ self._wlock = None
122
+ else:
123
+ self._wlock = context.Lock()
124
+ self._make_methods()
125
+
126
+ def __getstate__(self):
127
+ assert_spawning(self)
128
+ return (self._reader, self._writer, self._rlock, self._wlock, self._reducers)
129
+
130
+ def __setstate__(self, state):
131
+ (self._reader, self._writer, self._rlock, self._wlock, self._reducers) = state
132
+ self._make_methods()
133
+
134
+ def empty(self):
135
+ return not self._reader.poll()
136
+
137
+ def _make_methods(self):
138
+ self._recv = recv = self._reader.recv
139
+ racquire, rrelease = self._rlock.acquire, self._rlock.release
140
+
141
+ def get():
142
+ racquire()
143
+ try:
144
+ return recv()
145
+ finally:
146
+ rrelease()
147
+
148
+ self.get = get
149
+
150
+ if self._reducers:
151
+
152
+ def send(obj):
153
+ buffer = BytesIO()
154
+ CustomizablePickler(buffer, self._reducers).dump(obj)
155
+ self._writer.send_bytes(buffer.getvalue())
156
+
157
+ self._send = send
158
+ else:
159
+ self._send = send = self._writer.send
160
+ if self._wlock is None:
161
+ # writes to a message oriented win32 pipe are atomic
162
+ self.put = send
163
+ else:
164
+ wlock_acquire, wlock_release = (self._wlock.acquire, self._wlock.release)
165
+
166
+ def put(obj):
167
+ wlock_acquire()
168
+ try:
169
+ return send(obj)
170
+ finally:
171
+ wlock_release()
172
+
173
+ self.put = put
174
+
175
+
176
+ class PicklingPool(Pool):
177
+ """Pool implementation with customizable pickling reducers.
178
+
179
+ This is useful to control how data is shipped between processes
180
+ and makes it possible to use shared memory without useless
181
+ copies induces by the default pickling methods of the original
182
+ objects passed as arguments to dispatch.
183
+
184
+ `forward_reducers` and `backward_reducers` are expected to be
185
+ dictionaries with key/values being `(type, callable)` pairs where
186
+ `callable` is a function that, given an instance of `type`, will return a
187
+ tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the
188
+ pickled `tuple_of_objects` as would return a `__reduce__` method.
189
+ See the standard library documentation about pickling for more details.
190
+
191
+ """
192
+
193
+ def __init__(
194
+ self, processes=None, forward_reducers=None, backward_reducers=None, **kwargs
195
+ ):
196
+ if forward_reducers is None:
197
+ forward_reducers = dict()
198
+ if backward_reducers is None:
199
+ backward_reducers = dict()
200
+ self._forward_reducers = forward_reducers
201
+ self._backward_reducers = backward_reducers
202
+ poolargs = dict(processes=processes)
203
+ poolargs.update(kwargs)
204
+ super(PicklingPool, self).__init__(**poolargs)
205
+
206
+ def _setup_queues(self):
207
+ context = getattr(self, "_ctx", mp)
208
+ self._inqueue = CustomizablePicklingQueue(context, self._forward_reducers)
209
+ self._outqueue = CustomizablePicklingQueue(context, self._backward_reducers)
210
+ self._quick_put = self._inqueue._send
211
+ self._quick_get = self._outqueue._recv
212
+
213
+
214
+ class MemmappingPool(PicklingPool):
215
+ """Process pool that shares large arrays to avoid memory copy.
216
+
217
+ This drop-in replacement for `multiprocessing.pool.Pool` makes
218
+ it possible to work efficiently with shared memory in a numpy
219
+ context.
220
+
221
+ Existing instances of numpy.memmap are preserved: the child
222
+ suprocesses will have access to the same shared memory in the
223
+ original mode except for the 'w+' mode that is automatically
224
+ transformed as 'r+' to avoid zeroing the original data upon
225
+ instantiation.
226
+
227
+ Furthermore large arrays from the parent process are automatically
228
+ dumped to a temporary folder on the filesystem such as child
229
+ processes to access their content via memmapping (file system
230
+ backed shared memory).
231
+
232
+ Note: it is important to call the terminate method to collect
233
+ the temporary folder used by the pool.
234
+
235
+ Parameters
236
+ ----------
237
+ processes: int, optional
238
+ Number of worker processes running concurrently in the pool.
239
+ initializer: callable, optional
240
+ Callable executed on worker process creation.
241
+ initargs: tuple, optional
242
+ Arguments passed to the initializer callable.
243
+ temp_folder: (str, callable) optional
244
+ If str:
245
+ Folder to be used by the pool for memmapping large arrays
246
+ for sharing memory with worker processes. If None, this will try in
247
+ order:
248
+ - a folder pointed by the JOBLIB_TEMP_FOLDER environment variable,
249
+ - /dev/shm if the folder exists and is writable: this is a RAMdisk
250
+ filesystem available by default on modern Linux distributions,
251
+ - the default system temporary folder that can be overridden
252
+ with TMP, TMPDIR or TEMP environment variables, typically /tmp
253
+ under Unix operating systems.
254
+ if callable:
255
+ An callable in charge of dynamically resolving a temporary folder
256
+ for memmapping large arrays.
257
+ max_nbytes int or None, optional, 1e6 by default
258
+ Threshold on the size of arrays passed to the workers that
259
+ triggers automated memory mapping in temp_folder.
260
+ Use None to disable memmapping of large arrays.
261
+ mmap_mode: {'r+', 'r', 'w+', 'c'}
262
+ Memmapping mode for numpy arrays passed to workers.
263
+ See 'max_nbytes' parameter documentation for more details.
264
+ forward_reducers: dictionary, optional
265
+ Reducers used to pickle objects passed from main process to worker
266
+ processes: see below.
267
+ backward_reducers: dictionary, optional
268
+ Reducers used to pickle return values from workers back to the
269
+ main process.
270
+ verbose: int, optional
271
+ Make it possible to monitor how the communication of numpy arrays
272
+ with the subprocess is handled (pickling or memmapping)
273
+ prewarm: bool or str, optional, "auto" by default.
274
+ If True, force a read on newly memmapped array to make sure that OS
275
+ pre-cache it in memory. This can be useful to avoid concurrent disk
276
+ access when the same data array is passed to different worker
277
+ processes. If "auto" (by default), prewarm is set to True, unless the
278
+ Linux shared memory partition /dev/shm is available and used as temp
279
+ folder.
280
+
281
+ `forward_reducers` and `backward_reducers` are expected to be
282
+ dictionaries with key/values being `(type, callable)` pairs where
283
+ `callable` is a function that give an instance of `type` will return
284
+ a tuple `(constructor, tuple_of_objects)` to rebuild an instance out
285
+ of the pickled `tuple_of_objects` as would return a `__reduce__`
286
+ method. See the standard library documentation on pickling for more
287
+ details.
288
+
289
+ """
290
+
291
+ def __init__(
292
+ self,
293
+ processes=None,
294
+ temp_folder=None,
295
+ max_nbytes=1e6,
296
+ mmap_mode="r",
297
+ forward_reducers=None,
298
+ backward_reducers=None,
299
+ verbose=0,
300
+ prewarm=False,
301
+ **kwargs,
302
+ ):
303
+ manager = TemporaryResourcesManager(temp_folder)
304
+ self._temp_folder_manager = manager
305
+
306
+ # The usage of a temp_folder_resolver over a simple temp_folder is
307
+ # superfluous for multiprocessing pools, as they don't get reused, see
308
+ # get_memmapping_executor for more details. We still use it for code
309
+ # simplicity.
310
+ forward_reducers, backward_reducers = get_memmapping_reducers(
311
+ temp_folder_resolver=manager.resolve_temp_folder_name,
312
+ max_nbytes=max_nbytes,
313
+ mmap_mode=mmap_mode,
314
+ forward_reducers=forward_reducers,
315
+ backward_reducers=backward_reducers,
316
+ verbose=verbose,
317
+ unlink_on_gc_collect=False,
318
+ prewarm=prewarm,
319
+ )
320
+
321
+ poolargs = dict(
322
+ processes=processes,
323
+ forward_reducers=forward_reducers,
324
+ backward_reducers=backward_reducers,
325
+ )
326
+ poolargs.update(kwargs)
327
+ super(MemmappingPool, self).__init__(**poolargs)
328
+
329
+ def terminate(self):
330
+ n_retries = 10
331
+ for i in range(n_retries):
332
+ try:
333
+ super(MemmappingPool, self).terminate()
334
+ break
335
+ except OSError as e:
336
+ if isinstance(e, WindowsError):
337
+ # Workaround occasional "[Error 5] Access is denied" issue
338
+ # when trying to terminate a process under windows.
339
+ sleep(0.1)
340
+ if i + 1 == n_retries:
341
+ warnings.warn(
342
+ "Failed to terminate worker processes in"
343
+ " multiprocessing pool: %r" % e
344
+ )
345
+
346
+ # Clean up the temporary resources as the workers should now be off.
347
+ self._temp_folder_manager._clean_temporary_resources()
348
+
349
+ @property
350
+ def _temp_folder(self):
351
+ # Legacy property in tests. could be removed if we refactored the
352
+ # memmapping tests. SHOULD ONLY BE USED IN TESTS!
353
+ # We cache this property because it is called late in the tests - at
354
+ # this point, all context have been unregistered, and
355
+ # resolve_temp_folder_name raises an error.
356
+ if getattr(self, "_cached_temp_folder", None) is not None:
357
+ return self._cached_temp_folder
358
+ else:
359
+ self._cached_temp_folder = (
360
+ self._temp_folder_manager.resolve_temp_folder_name()
361
+ ) # noqa
362
+ return self._cached_temp_folder
TTS/Kokoro/.venv/Lib/site-packages/joblib/testing.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Helper for testing.
3
+ """
4
+
5
+ import os.path
6
+ import re
7
+ import subprocess
8
+ import sys
9
+ import threading
10
+ import warnings
11
+
12
+ import _pytest
13
+ import pytest
14
+
15
+ raises = pytest.raises
16
+ warns = pytest.warns
17
+ SkipTest = _pytest.runner.Skipped
18
+ skipif = pytest.mark.skipif
19
+ fixture = pytest.fixture
20
+ parametrize = pytest.mark.parametrize
21
+ timeout = pytest.mark.timeout
22
+ xfail = pytest.mark.xfail
23
+ param = pytest.param
24
+
25
+
26
+ def warnings_to_stdout():
27
+ """Redirect all warnings to stdout."""
28
+ showwarning_orig = warnings.showwarning
29
+
30
+ def showwarning(msg, cat, fname, lno, file=None, line=0):
31
+ showwarning_orig(msg, cat, os.path.basename(fname), line, sys.stdout)
32
+
33
+ warnings.showwarning = showwarning
34
+ # warnings.simplefilter('always')
35
+
36
+
37
+ def check_subprocess_call(cmd, timeout=5, stdout_regex=None, stderr_regex=None):
38
+ """Runs a command in a subprocess with timeout in seconds.
39
+
40
+ A SIGTERM is sent after `timeout` and if it does not terminate, a
41
+ SIGKILL is sent after `2 * timeout`.
42
+
43
+ Also checks returncode is zero, stdout if stdout_regex is set, and
44
+ stderr if stderr_regex is set.
45
+ """
46
+ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
47
+
48
+ def terminate_process(): # pragma: no cover
49
+ """
50
+ Attempt to terminate a leftover process spawned during test execution:
51
+ ideally this should not be needed but can help avoid clogging the CI
52
+ workers in case of deadlocks.
53
+ """
54
+ warnings.warn(f"Timeout running {cmd}")
55
+ proc.terminate()
56
+
57
+ def kill_process(): # pragma: no cover
58
+ """
59
+ Kill a leftover process spawned during test execution: ideally this
60
+ should not be needed but can help avoid clogging the CI workers in
61
+ case of deadlocks.
62
+ """
63
+ warnings.warn(f"Timeout running {cmd}")
64
+ proc.kill()
65
+
66
+ try:
67
+ if timeout is not None:
68
+ terminate_timer = threading.Timer(timeout, terminate_process)
69
+ terminate_timer.start()
70
+ kill_timer = threading.Timer(2 * timeout, kill_process)
71
+ kill_timer.start()
72
+ stdout, stderr = proc.communicate()
73
+ stdout, stderr = stdout.decode(), stderr.decode()
74
+ if proc.returncode != 0:
75
+ message = ("Non-zero return code: {}.\nStdout:\n{}\nStderr:\n{}").format(
76
+ proc.returncode, stdout, stderr
77
+ )
78
+ raise ValueError(message)
79
+
80
+ if stdout_regex is not None and not re.search(stdout_regex, stdout):
81
+ raise ValueError(
82
+ "Unexpected stdout: {!r} does not match:\n{!r}".format(
83
+ stdout_regex, stdout
84
+ )
85
+ )
86
+ if stderr_regex is not None and not re.search(stderr_regex, stderr):
87
+ raise ValueError(
88
+ "Unexpected stderr: {!r} does not match:\n{!r}".format(
89
+ stderr_regex, stderr
90
+ )
91
+ )
92
+
93
+ finally:
94
+ if timeout is not None:
95
+ terminate_timer.cancel()
96
+ kill_timer.cancel()
TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/INSTALLER ADDED
@@ -0,0 +1 @@
 
 
1
+ pip
TTS/Kokoro/.venv/Lib/site-packages/jsonschema-4.26.0.dist-info/METADATA ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Metadata-Version: 2.4
2
+ Name: jsonschema
3
+ Version: 4.26.0
4
+ Summary: An implementation of JSON Schema validation for Python
5
+ Project-URL: Homepage, https://github.com/python-jsonschema/jsonschema
6
+ Project-URL: Documentation, https://python-jsonschema.readthedocs.io/
7
+ Project-URL: Issues, https://github.com/python-jsonschema/jsonschema/issues/
8
+ Project-URL: Funding, https://github.com/sponsors/Julian
9
+ Project-URL: Tidelift, https://tidelift.com/subscription/pkg/pypi-jsonschema?utm_source=pypi-jsonschema&utm_medium=referral&utm_campaign=pypi-link
10
+ Project-URL: Changelog, https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst
11
+ Project-URL: Source, https://github.com/python-jsonschema/jsonschema
12
+ Author-email: Julian Berman <Julian+jsonschema@GrayVines.com>
13
+ License-Expression: MIT
14
+ License-File: COPYING
15
+ Keywords: data validation,json,json schema,jsonschema,validation
16
+ Classifier: Development Status :: 5 - Production/Stable
17
+ Classifier: Intended Audience :: Developers
18
+ Classifier: Operating System :: OS Independent
19
+ Classifier: Programming Language :: Python
20
+ Classifier: Programming Language :: Python :: 3.10
21
+ Classifier: Programming Language :: Python :: 3.11
22
+ Classifier: Programming Language :: Python :: 3.12
23
+ Classifier: Programming Language :: Python :: 3.13
24
+ Classifier: Programming Language :: Python :: 3.14
25
+ Classifier: Programming Language :: Python :: Implementation :: CPython
26
+ Classifier: Programming Language :: Python :: Implementation :: PyPy
27
+ Classifier: Topic :: File Formats :: JSON
28
+ Classifier: Topic :: File Formats :: JSON :: JSON Schema
29
+ Requires-Python: >=3.10
30
+ Requires-Dist: attrs>=22.2.0
31
+ Requires-Dist: jsonschema-specifications>=2023.03.6
32
+ Requires-Dist: referencing>=0.28.4
33
+ Requires-Dist: rpds-py>=0.25.0
34
+ Provides-Extra: format
35
+ Requires-Dist: fqdn; extra == 'format'
36
+ Requires-Dist: idna; extra == 'format'
37
+ Requires-Dist: isoduration; extra == 'format'
38
+ Requires-Dist: jsonpointer>1.13; extra == 'format'
39
+ Requires-Dist: rfc3339-validator; extra == 'format'
40
+ Requires-Dist: rfc3987; extra == 'format'
41
+ Requires-Dist: uri-template; extra == 'format'
42
+ Requires-Dist: webcolors>=1.11; extra == 'format'
43
+ Provides-Extra: format-nongpl
44
+ Requires-Dist: fqdn; extra == 'format-nongpl'
45
+ Requires-Dist: idna; extra == 'format-nongpl'
46
+ Requires-Dist: isoduration; extra == 'format-nongpl'
47
+ Requires-Dist: jsonpointer>1.13; extra == 'format-nongpl'
48
+ Requires-Dist: rfc3339-validator; extra == 'format-nongpl'
49
+ Requires-Dist: rfc3986-validator>0.1.0; extra == 'format-nongpl'
50
+ Requires-Dist: rfc3987-syntax>=1.1.0; extra == 'format-nongpl'
51
+ Requires-Dist: uri-template; extra == 'format-nongpl'
52
+ Requires-Dist: webcolors>=24.6.0; extra == 'format-nongpl'
53
+ Description-Content-Type: text/x-rst
54
+
55
+ ==========
56
+ jsonschema
57
+ ==========
58
+
59
+ |PyPI| |Pythons| |CI| |ReadTheDocs| |Precommit| |Zenodo|
60
+
61
+ .. |PyPI| image:: https://img.shields.io/pypi/v/jsonschema.svg
62
+ :alt: PyPI version
63
+ :target: https://pypi.org/project/jsonschema/
64
+
65
+ .. |Pythons| image:: https://img.shields.io/pypi/pyversions/jsonschema.svg
66
+ :alt: Supported Python versions
67
+ :target: https://pypi.org/project/jsonschema/
68
+
69
+ .. |CI| image:: https://github.com/python-jsonschema/jsonschema/workflows/CI/badge.svg
70
+ :alt: Build status
71
+ :target: https://github.com/python-jsonschema/jsonschema/actions?query=workflow%3ACI
72
+
73
+ .. |ReadTheDocs| image:: https://readthedocs.org/projects/python-jsonschema/badge/?version=stable&style=flat
74
+ :alt: ReadTheDocs status
75
+ :target: https://python-jsonschema.readthedocs.io/en/stable/
76
+
77
+ .. |Precommit| image:: https://results.pre-commit.ci/badge/github/python-jsonschema/jsonschema/main.svg
78
+ :alt: pre-commit.ci status
79
+ :target: https://results.pre-commit.ci/latest/github/python-jsonschema/jsonschema/main
80
+
81
+ .. |Zenodo| image:: https://zenodo.org/badge/3072629.svg
82
+ :alt: Zenodo DOI
83
+ :target: https://zenodo.org/badge/latestdoi/3072629
84
+
85
+
86
+ ``jsonschema`` is an implementation of the `JSON Schema <https://json-schema.org>`_ specification for Python.
87
+
88
+ .. code:: python
89
+
90
+ >>> from jsonschema import validate
91
+
92
+ >>> # A sample schema, like what we'd get from json.load()
93
+ >>> schema = {
94
+ ... "type" : "object",
95
+ ... "properties" : {
96
+ ... "price" : {"type" : "number"},
97
+ ... "name" : {"type" : "string"},
98
+ ... },
99
+ ... }
100
+
101
+ >>> # If no exception is raised by validate(), the instance is valid.
102
+ >>> validate(instance={"name" : "Eggs", "price" : 34.99}, schema=schema)
103
+
104
+ >>> validate(
105
+ ... instance={"name" : "Eggs", "price" : "Invalid"}, schema=schema,
106
+ ... ) # doctest: +IGNORE_EXCEPTION_DETAIL
107
+ Traceback (most recent call last):
108
+ ...
109
+ ValidationError: 'Invalid' is not of type 'number'
110
+
111
+ It can also be used from the command line by installing `check-jsonschema <https://github.com/python-jsonschema/check-jsonschema>`_.
112
+
113
+ Features
114
+ --------
115
+
116
+ * Full support for `Draft 2020-12 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft202012Validator>`_, `Draft 2019-09 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft201909Validator>`_, `Draft 7 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft7Validator>`_, `Draft 6 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft6Validator>`_, `Draft 4 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft4Validator>`_ and `Draft 3 <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/validators/#jsonschema.validators.Draft3Validator>`_
117
+
118
+ * `Lazy validation <https://python-jsonschema.readthedocs.io/en/latest/api/jsonschema/protocols/#jsonschema.protocols.Validator.iter_errors>`_ that can iteratively report *all* validation errors.
119
+
120
+ * `Programmatic querying <https://python-jsonschema.readthedocs.io/en/latest/errors/>`_ of which properties or items failed validation.
121
+
122
+
123
+ Installation
124
+ ------------
125
+
126
+ ``jsonschema`` is available on `PyPI <https://pypi.org/project/jsonschema/>`_. You can install using `pip <https://pip.pypa.io/en/stable/>`_:
127
+
128
+ .. code:: bash
129
+
130
+ $ pip install jsonschema
131
+
132
+
133
+ Extras
134
+ ======
135
+
136
+ Two extras are available when installing the package, both currently related to ``format`` validation:
137
+
138
+ * ``format``
139
+ * ``format-nongpl``
140
+
141
+ They can be used when installing in order to include additional dependencies, e.g.:
142
+
143
+ .. code:: bash
144
+
145
+ $ pip install jsonschema'[format]'
146
+
147
+ Be aware that the mere presence of these dependencies – or even the specification of ``format`` checks in a schema – do *not* activate format checks (as per the specification).
148
+ Please read the `format validation documentation <https://python-jsonschema.readthedocs.io/en/latest/validate/#validating-formats>`_ for further details.
149
+
150
+ About
151
+ -----
152
+
153
+ I'm Julian Berman.
154
+
155
+ ``jsonschema`` is on `GitHub <https://github.com/python-jsonschema/jsonschema>`_.
156
+
157
+ Get in touch, via GitHub or otherwise, if you've got something to contribute, it'd be most welcome!
158
+
159
+ If you feel overwhelmingly grateful, you can also `sponsor me <https://github.com/sponsors/Julian/>`_.
160
+
161
+ And for companies who appreciate ``jsonschema`` and its continued support and growth, ``jsonschema`` is also now supportable via `TideLift <https://tidelift.com/subscription/pkg/pypi-jsonschema?utm_source=pypi-jsonschema&utm_medium=referral&utm_campaign=readme>`_.
162
+
163
+
164
+ Release Information
165
+ -------------------
166
+
167
+ v4.26.0
168
+ =======
169
+
170
+ * Decrease import time by delaying importing of ``urllib.request`` (#1416).