techprotrade commited on
Commit
23faf1f
·
verified ·
1 Parent(s): f4b63dd

Add loan-advisor-agent

Browse files
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/INSTALLER ADDED
@@ -0,0 +1 @@
 
 
1
+ uv
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/METADATA ADDED
@@ -0,0 +1,214 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Metadata-Version: 2.4
2
+ Name: tokenizers
3
+ Version: 0.22.2
4
+ Classifier: Development Status :: 5 - Production/Stable
5
+ Classifier: Intended Audience :: Developers
6
+ Classifier: Intended Audience :: Education
7
+ Classifier: Intended Audience :: Science/Research
8
+ Classifier: License :: OSI Approved :: Apache Software License
9
+ Classifier: Operating System :: OS Independent
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: Programming Language :: Python :: 3.9
12
+ Classifier: Programming Language :: Python :: 3.10
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Programming Language :: Python :: 3.13
16
+ Classifier: Programming Language :: Python :: 3 :: Only
17
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
18
+ Requires-Dist: huggingface-hub>=0.16.4,<2.0
19
+ Requires-Dist: pytest ; extra == 'testing'
20
+ Requires-Dist: pytest-asyncio ; extra == 'testing'
21
+ Requires-Dist: requests ; extra == 'testing'
22
+ Requires-Dist: numpy ; extra == 'testing'
23
+ Requires-Dist: datasets ; extra == 'testing'
24
+ Requires-Dist: ruff ; extra == 'testing'
25
+ Requires-Dist: ty ; extra == 'testing'
26
+ Requires-Dist: sphinx ; extra == 'docs'
27
+ Requires-Dist: sphinx-rtd-theme ; extra == 'docs'
28
+ Requires-Dist: setuptools-rust ; extra == 'docs'
29
+ Requires-Dist: tokenizers[testing] ; extra == 'dev'
30
+ Provides-Extra: testing
31
+ Provides-Extra: docs
32
+ Provides-Extra: dev
33
+ Keywords: NLP,tokenizer,BPE,transformer,deep learning
34
+ Author-email: Nicolas Patry <patry.nicolas@protonmail.com>, Anthony Moi <anthony@huggingface.co>
35
+ Requires-Python: >=3.9
36
+ Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
37
+ Project-URL: Homepage, https://github.com/huggingface/tokenizers
38
+ Project-URL: Source, https://github.com/huggingface/tokenizers
39
+
40
+ <p align="center">
41
+ <br>
42
+ <img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
43
+ <br>
44
+ <p>
45
+ <p align="center">
46
+ <a href="https://badge.fury.io/py/tokenizers">
47
+ <img alt="Build" src="https://badge.fury.io/py/tokenizers.svg">
48
+ </a>
49
+ <a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE">
50
+ <img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue">
51
+ </a>
52
+ </p>
53
+ <br>
54
+
55
+ # Tokenizers
56
+
57
+ Provides an implementation of today's most used tokenizers, with a focus on performance and
58
+ versatility.
59
+
60
+ Bindings over the [Rust](https://github.com/huggingface/tokenizers/tree/master/tokenizers) implementation.
61
+ If you are interested in the High-level design, you can go check it there.
62
+
63
+ Otherwise, let's dive in!
64
+
65
+ ## Main features:
66
+
67
+ - Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3
68
+ most common BPE versions).
69
+ - Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes
70
+ less than 20 seconds to tokenize a GB of text on a server's CPU.
71
+ - Easy to use, but also extremely versatile.
72
+ - Designed for research and production.
73
+ - Normalization comes with alignments tracking. It's always possible to get the part of the
74
+ original sentence that corresponds to a given token.
75
+ - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
76
+
77
+ ### Installation
78
+
79
+ #### With pip:
80
+
81
+ ```bash
82
+ pip install tokenizers
83
+ ```
84
+
85
+ #### From sources:
86
+
87
+ To use this method, you need to have the Rust installed:
88
+
89
+ ```bash
90
+ # Install with:
91
+ curl https://sh.rustup.rs -sSf | sh -s -- -y
92
+ export PATH="$HOME/.cargo/bin:$PATH"
93
+ ```
94
+
95
+ Once Rust is installed, you can compile doing the following
96
+
97
+ ```bash
98
+ git clone https://github.com/huggingface/tokenizers
99
+ cd tokenizers/bindings/python
100
+
101
+ # Create a virtual env (you can use yours as well)
102
+ python -m venv .env
103
+ source .env/bin/activate
104
+
105
+ # Install `tokenizers` in the current virtual env
106
+ pip install -e .
107
+ ```
108
+
109
+ ### Load a pretrained tokenizer from the Hub
110
+
111
+ ```python
112
+ from tokenizers import Tokenizer
113
+
114
+ tokenizer = Tokenizer.from_pretrained("bert-base-cased")
115
+ ```
116
+
117
+ ### Using the provided Tokenizers
118
+
119
+ We provide some pre-build tokenizers to cover the most common cases. You can easily load one of
120
+ these using some `vocab.json` and `merges.txt` files:
121
+
122
+ ```python
123
+ from tokenizers import CharBPETokenizer
124
+
125
+ # Initialize a tokenizer
126
+ vocab = "./path/to/vocab.json"
127
+ merges = "./path/to/merges.txt"
128
+ tokenizer = CharBPETokenizer(vocab, merges)
129
+
130
+ # And then encode:
131
+ encoded = tokenizer.encode("I can feel the magic, can you?")
132
+ print(encoded.ids)
133
+ print(encoded.tokens)
134
+ ```
135
+
136
+ And you can train them just as simply:
137
+
138
+ ```python
139
+ from tokenizers import CharBPETokenizer
140
+
141
+ # Initialize a tokenizer
142
+ tokenizer = CharBPETokenizer()
143
+
144
+ # Then train it!
145
+ tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])
146
+
147
+ # Now, let's use it:
148
+ encoded = tokenizer.encode("I can feel the magic, can you?")
149
+
150
+ # And finally save it somewhere
151
+ tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")
152
+ ```
153
+
154
+ #### Provided Tokenizers
155
+
156
+ - `CharBPETokenizer`: The original BPE
157
+ - `ByteLevelBPETokenizer`: The byte level version of the BPE
158
+ - `SentencePieceBPETokenizer`: A BPE implementation compatible with the one used by SentencePiece
159
+ - `BertWordPieceTokenizer`: The famous Bert tokenizer, using WordPiece
160
+
161
+ All of these can be used and trained as explained above!
162
+
163
+ ### Build your own
164
+
165
+ Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer,
166
+ by putting all the different parts you need together.
167
+ You can check how we implemented the [provided tokenizers](https://github.com/huggingface/tokenizers/tree/master/bindings/python/py_src/tokenizers/implementations) and adapt them easily to your own needs.
168
+
169
+ #### Building a byte-level BPE
170
+
171
+ Here is an example showing how to build your own byte-level BPE by putting all the different pieces
172
+ together, and then saving it to a single file:
173
+
174
+ ```python
175
+ from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors
176
+
177
+ # Initialize a tokenizer
178
+ tokenizer = Tokenizer(models.BPE())
179
+
180
+ # Customize pre-tokenization and decoding
181
+ tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
182
+ tokenizer.decoder = decoders.ByteLevel()
183
+ tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)
184
+
185
+ # And then train
186
+ trainer = trainers.BpeTrainer(
187
+ vocab_size=20000,
188
+ min_frequency=2,
189
+ initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
190
+ )
191
+ tokenizer.train([
192
+ "./path/to/dataset/1.txt",
193
+ "./path/to/dataset/2.txt",
194
+ "./path/to/dataset/3.txt"
195
+ ], trainer=trainer)
196
+
197
+ # And Save it
198
+ tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)
199
+ ```
200
+
201
+ Now, when you want to use this tokenizer, this is as simple as:
202
+
203
+ ```python
204
+ from tokenizers import Tokenizer
205
+
206
+ tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")
207
+
208
+ encoded = tokenizer.encode("I can feel the magic, can you?")
209
+ ```
210
+
211
+ ### Typing support and `stub.py`
212
+
213
+ The compiled PyO3 extension does not expose type annotations, so editors and type checkers would otherwise see most objects as `Any`. The `stub.py` helper walks the loaded extension modules, renders `.pyi` stub files (plus minimal forwarding `__init__.py` shims), and formats them so that tools like mypy/pyright can understand the public API. Run `python stub.py` whenever you change the Python-visible surface to keep the generated stubs in sync.
214
+
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/RECORD ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ tokenizers-0.22.2.dist-info/INSTALLER,sha256=5hhM4Q4mYTT9z6QB6PGpUAW81PGNFrYrdXMj4oM_6ak,2
2
+ tokenizers-0.22.2.dist-info/METADATA,sha256=Wtm8zyGuU9RfgScDyAx8A-zqMAgjZVopaPnmd4d5vbM,7428
3
+ tokenizers-0.22.2.dist-info/RECORD,,
4
+ tokenizers-0.22.2.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
5
+ tokenizers-0.22.2.dist-info/WHEEL,sha256=W8IbRrW6aFuwgkpVWKTtkIq8XwJIIpO5tHf4goK_jm8,95
6
+ tokenizers/__init__.py,sha256=UQ1eI0WGEkM9qfH-QwtQCfuIupBQjuQ0DSOXsywxkIA,2739
7
+ tokenizers/__init__.pyi,sha256=ubhr55w0uERGRitWoIi-HwmFn7dW9Vl6SHVlmC0ZJhc,57691
8
+ tokenizers/decoders/__init__.py,sha256=Qr1jbA2fvqwjgBBYOEyaDjLBwDwkBftP5jN3tRvT1jg,387
9
+ tokenizers/decoders/__init__.pyi,sha256=UJy1WXyQ7ogrw-Es6IJahSU6i04U7walI7bc_O86zPI,12588
10
+ tokenizers/implementations/__init__.py,sha256=9ZI2cJHPCUCZXP37GNmRHtBzIYiKWzyqky1rtcYdrPw,316
11
+ tokenizers/implementations/base_tokenizer.py,sha256=2RFMlsWFqS8AacXK2dbT9jCSVnOy86ERWwmaWH8vZFQ,16265
12
+ tokenizers/implementations/bert_wordpiece.py,sha256=jRUMSYU1C7nVEDZMKdyG3Vp7-CUongE1kGJhvOP_EHM,5671
13
+ tokenizers/implementations/byte_level_bpe.py,sha256=pWfbSeaew9atWKX8Nf2zIRia6qhAwXMfZgaeINInzo8,4394
14
+ tokenizers/implementations/char_level_bpe.py,sha256=Bt15rEcZRSzJ2RfEeq2YvOXZ8w_UYGq9i6O3y-41Aj4,5599
15
+ tokenizers/implementations/sentencepiece_bpe.py,sha256=4_dJFcNsyvC_F1wBnWDBquWssbUMV4kP-BEXeuHcxtI,3824
16
+ tokenizers/implementations/sentencepiece_unigram.py,sha256=KGjhIzXukwJkpao1QIe5XVrEIAMjKiGaHEzpAHL-1I8,7826
17
+ tokenizers/models/__init__.py,sha256=qE73qycPAKcey_pS8ov1FIz10L2Ybzy1E-1KGee27Qg,184
18
+ tokenizers/models/__init__.pyi,sha256=tO4jy0Qrq0emg7PoqmBXituSpe0dMxyChF4J9SinyDk,20261
19
+ tokenizers/normalizers/__init__.py,sha256=Pgb_wE1QQov3t_SW4vcsXY6lVBDYyGNfYGkJMk3wI7c,870
20
+ tokenizers/normalizers/__init__.pyi,sha256=fjuhSi1yirfLZSUsFGL9Vyx2azpANIjCz-VundvqakQ,26655
21
+ tokenizers/pre_tokenizers/__init__.py,sha256=QvNmMTtV0RGkaKSN9hTwEPimD1OLHRU6PbzVxXwM4zA,614
22
+ tokenizers/pre_tokenizers/__init__.pyi,sha256=VMERim4OPFGQaG8JUOTu5H3C0kgQCxv2hVYDfvPPztE,32581
23
+ tokenizers/processors/__init__.py,sha256=X1OKKtr3IUeede85mI1cLF0MqmDRlr7LNFMYdEy5KMU,316
24
+ tokenizers/processors/__init__.pyi,sha256=RElgDx8bbbZIq2MpxXy3G67KXSqDtYLj_7Vq0GPkwO0,14867
25
+ tokenizers/tokenizers.pyd,sha256=esuD9biRNll-DRS3iNgpF78ocN-UV1v3fhJzHE5JxN8,7395328
26
+ tokenizers/tokenizers.pyi,sha256=kTPgANU6E7wyKoBnR-a0YK6pVNiprgxjGhOonFvuRm0,485
27
+ tokenizers/tools/__init__.py,sha256=mJLVrHN1FP23Cf4_EJWoYUdh8yTb5BbJFNfa8Ec5vHM,56
28
+ tokenizers/tools/visualizer-styles.css,sha256=g98RJucUCjYnPLwXycfcV-dAbHMzqJpnZ7NoP_cCUBk,5020
29
+ tokenizers/tools/visualizer.py,sha256=jWcxhFTsyiQr-ZQmDJSjzEcSGZvZZoKDes2Ira1EVg8,15241
30
+ tokenizers/trainers/__init__.py,sha256=v9CdtoVauwD6b1wAA6R5goSLZslBbPE5vZHM2d8sgKM,256
31
+ tokenizers/trainers/__init__.pyi,sha256=WfXHAWSBQNPaqCym4sEREllLvqMoG76x0MFE7qSnTdA,11181
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/REQUESTED ADDED
File without changes
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/WHEEL ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ Wheel-Version: 1.0
2
+ Generator: maturin (1.10.2)
3
+ Root-Is-Purelib: false
4
+ Tag: cp39-abi3-win_amd64