Add loan-advisor-agent
Browse files- loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/INSTALLER +1 -0
- loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/METADATA +214 -0
- loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/RECORD +31 -0
- loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/REQUESTED +0 -0
- loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/WHEEL +4 -0
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/INSTALLER
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
uv
|
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/METADATA
ADDED
|
@@ -0,0 +1,214 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Metadata-Version: 2.4
|
| 2 |
+
Name: tokenizers
|
| 3 |
+
Version: 0.22.2
|
| 4 |
+
Classifier: Development Status :: 5 - Production/Stable
|
| 5 |
+
Classifier: Intended Audience :: Developers
|
| 6 |
+
Classifier: Intended Audience :: Education
|
| 7 |
+
Classifier: Intended Audience :: Science/Research
|
| 8 |
+
Classifier: License :: OSI Approved :: Apache Software License
|
| 9 |
+
Classifier: Operating System :: OS Independent
|
| 10 |
+
Classifier: Programming Language :: Python :: 3
|
| 11 |
+
Classifier: Programming Language :: Python :: 3.9
|
| 12 |
+
Classifier: Programming Language :: Python :: 3.10
|
| 13 |
+
Classifier: Programming Language :: Python :: 3.11
|
| 14 |
+
Classifier: Programming Language :: Python :: 3.12
|
| 15 |
+
Classifier: Programming Language :: Python :: 3.13
|
| 16 |
+
Classifier: Programming Language :: Python :: 3 :: Only
|
| 17 |
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
| 18 |
+
Requires-Dist: huggingface-hub>=0.16.4,<2.0
|
| 19 |
+
Requires-Dist: pytest ; extra == 'testing'
|
| 20 |
+
Requires-Dist: pytest-asyncio ; extra == 'testing'
|
| 21 |
+
Requires-Dist: requests ; extra == 'testing'
|
| 22 |
+
Requires-Dist: numpy ; extra == 'testing'
|
| 23 |
+
Requires-Dist: datasets ; extra == 'testing'
|
| 24 |
+
Requires-Dist: ruff ; extra == 'testing'
|
| 25 |
+
Requires-Dist: ty ; extra == 'testing'
|
| 26 |
+
Requires-Dist: sphinx ; extra == 'docs'
|
| 27 |
+
Requires-Dist: sphinx-rtd-theme ; extra == 'docs'
|
| 28 |
+
Requires-Dist: setuptools-rust ; extra == 'docs'
|
| 29 |
+
Requires-Dist: tokenizers[testing] ; extra == 'dev'
|
| 30 |
+
Provides-Extra: testing
|
| 31 |
+
Provides-Extra: docs
|
| 32 |
+
Provides-Extra: dev
|
| 33 |
+
Keywords: NLP,tokenizer,BPE,transformer,deep learning
|
| 34 |
+
Author-email: Nicolas Patry <patry.nicolas@protonmail.com>, Anthony Moi <anthony@huggingface.co>
|
| 35 |
+
Requires-Python: >=3.9
|
| 36 |
+
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
| 37 |
+
Project-URL: Homepage, https://github.com/huggingface/tokenizers
|
| 38 |
+
Project-URL: Source, https://github.com/huggingface/tokenizers
|
| 39 |
+
|
| 40 |
+
<p align="center">
|
| 41 |
+
<br>
|
| 42 |
+
<img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
|
| 43 |
+
<br>
|
| 44 |
+
<p>
|
| 45 |
+
<p align="center">
|
| 46 |
+
<a href="https://badge.fury.io/py/tokenizers">
|
| 47 |
+
<img alt="Build" src="https://badge.fury.io/py/tokenizers.svg">
|
| 48 |
+
</a>
|
| 49 |
+
<a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE">
|
| 50 |
+
<img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue">
|
| 51 |
+
</a>
|
| 52 |
+
</p>
|
| 53 |
+
<br>
|
| 54 |
+
|
| 55 |
+
# Tokenizers
|
| 56 |
+
|
| 57 |
+
Provides an implementation of today's most used tokenizers, with a focus on performance and
|
| 58 |
+
versatility.
|
| 59 |
+
|
| 60 |
+
Bindings over the [Rust](https://github.com/huggingface/tokenizers/tree/master/tokenizers) implementation.
|
| 61 |
+
If you are interested in the High-level design, you can go check it there.
|
| 62 |
+
|
| 63 |
+
Otherwise, let's dive in!
|
| 64 |
+
|
| 65 |
+
## Main features:
|
| 66 |
+
|
| 67 |
+
- Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3
|
| 68 |
+
most common BPE versions).
|
| 69 |
+
- Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes
|
| 70 |
+
less than 20 seconds to tokenize a GB of text on a server's CPU.
|
| 71 |
+
- Easy to use, but also extremely versatile.
|
| 72 |
+
- Designed for research and production.
|
| 73 |
+
- Normalization comes with alignments tracking. It's always possible to get the part of the
|
| 74 |
+
original sentence that corresponds to a given token.
|
| 75 |
+
- Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
|
| 76 |
+
|
| 77 |
+
### Installation
|
| 78 |
+
|
| 79 |
+
#### With pip:
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
pip install tokenizers
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
#### From sources:
|
| 86 |
+
|
| 87 |
+
To use this method, you need to have the Rust installed:
|
| 88 |
+
|
| 89 |
+
```bash
|
| 90 |
+
# Install with:
|
| 91 |
+
curl https://sh.rustup.rs -sSf | sh -s -- -y
|
| 92 |
+
export PATH="$HOME/.cargo/bin:$PATH"
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
Once Rust is installed, you can compile doing the following
|
| 96 |
+
|
| 97 |
+
```bash
|
| 98 |
+
git clone https://github.com/huggingface/tokenizers
|
| 99 |
+
cd tokenizers/bindings/python
|
| 100 |
+
|
| 101 |
+
# Create a virtual env (you can use yours as well)
|
| 102 |
+
python -m venv .env
|
| 103 |
+
source .env/bin/activate
|
| 104 |
+
|
| 105 |
+
# Install `tokenizers` in the current virtual env
|
| 106 |
+
pip install -e .
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
### Load a pretrained tokenizer from the Hub
|
| 110 |
+
|
| 111 |
+
```python
|
| 112 |
+
from tokenizers import Tokenizer
|
| 113 |
+
|
| 114 |
+
tokenizer = Tokenizer.from_pretrained("bert-base-cased")
|
| 115 |
+
```
|
| 116 |
+
|
| 117 |
+
### Using the provided Tokenizers
|
| 118 |
+
|
| 119 |
+
We provide some pre-build tokenizers to cover the most common cases. You can easily load one of
|
| 120 |
+
these using some `vocab.json` and `merges.txt` files:
|
| 121 |
+
|
| 122 |
+
```python
|
| 123 |
+
from tokenizers import CharBPETokenizer
|
| 124 |
+
|
| 125 |
+
# Initialize a tokenizer
|
| 126 |
+
vocab = "./path/to/vocab.json"
|
| 127 |
+
merges = "./path/to/merges.txt"
|
| 128 |
+
tokenizer = CharBPETokenizer(vocab, merges)
|
| 129 |
+
|
| 130 |
+
# And then encode:
|
| 131 |
+
encoded = tokenizer.encode("I can feel the magic, can you?")
|
| 132 |
+
print(encoded.ids)
|
| 133 |
+
print(encoded.tokens)
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
And you can train them just as simply:
|
| 137 |
+
|
| 138 |
+
```python
|
| 139 |
+
from tokenizers import CharBPETokenizer
|
| 140 |
+
|
| 141 |
+
# Initialize a tokenizer
|
| 142 |
+
tokenizer = CharBPETokenizer()
|
| 143 |
+
|
| 144 |
+
# Then train it!
|
| 145 |
+
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])
|
| 146 |
+
|
| 147 |
+
# Now, let's use it:
|
| 148 |
+
encoded = tokenizer.encode("I can feel the magic, can you?")
|
| 149 |
+
|
| 150 |
+
# And finally save it somewhere
|
| 151 |
+
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
#### Provided Tokenizers
|
| 155 |
+
|
| 156 |
+
- `CharBPETokenizer`: The original BPE
|
| 157 |
+
- `ByteLevelBPETokenizer`: The byte level version of the BPE
|
| 158 |
+
- `SentencePieceBPETokenizer`: A BPE implementation compatible with the one used by SentencePiece
|
| 159 |
+
- `BertWordPieceTokenizer`: The famous Bert tokenizer, using WordPiece
|
| 160 |
+
|
| 161 |
+
All of these can be used and trained as explained above!
|
| 162 |
+
|
| 163 |
+
### Build your own
|
| 164 |
+
|
| 165 |
+
Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer,
|
| 166 |
+
by putting all the different parts you need together.
|
| 167 |
+
You can check how we implemented the [provided tokenizers](https://github.com/huggingface/tokenizers/tree/master/bindings/python/py_src/tokenizers/implementations) and adapt them easily to your own needs.
|
| 168 |
+
|
| 169 |
+
#### Building a byte-level BPE
|
| 170 |
+
|
| 171 |
+
Here is an example showing how to build your own byte-level BPE by putting all the different pieces
|
| 172 |
+
together, and then saving it to a single file:
|
| 173 |
+
|
| 174 |
+
```python
|
| 175 |
+
from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors
|
| 176 |
+
|
| 177 |
+
# Initialize a tokenizer
|
| 178 |
+
tokenizer = Tokenizer(models.BPE())
|
| 179 |
+
|
| 180 |
+
# Customize pre-tokenization and decoding
|
| 181 |
+
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
|
| 182 |
+
tokenizer.decoder = decoders.ByteLevel()
|
| 183 |
+
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)
|
| 184 |
+
|
| 185 |
+
# And then train
|
| 186 |
+
trainer = trainers.BpeTrainer(
|
| 187 |
+
vocab_size=20000,
|
| 188 |
+
min_frequency=2,
|
| 189 |
+
initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
|
| 190 |
+
)
|
| 191 |
+
tokenizer.train([
|
| 192 |
+
"./path/to/dataset/1.txt",
|
| 193 |
+
"./path/to/dataset/2.txt",
|
| 194 |
+
"./path/to/dataset/3.txt"
|
| 195 |
+
], trainer=trainer)
|
| 196 |
+
|
| 197 |
+
# And Save it
|
| 198 |
+
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)
|
| 199 |
+
```
|
| 200 |
+
|
| 201 |
+
Now, when you want to use this tokenizer, this is as simple as:
|
| 202 |
+
|
| 203 |
+
```python
|
| 204 |
+
from tokenizers import Tokenizer
|
| 205 |
+
|
| 206 |
+
tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")
|
| 207 |
+
|
| 208 |
+
encoded = tokenizer.encode("I can feel the magic, can you?")
|
| 209 |
+
```
|
| 210 |
+
|
| 211 |
+
### Typing support and `stub.py`
|
| 212 |
+
|
| 213 |
+
The compiled PyO3 extension does not expose type annotations, so editors and type checkers would otherwise see most objects as `Any`. The `stub.py` helper walks the loaded extension modules, renders `.pyi` stub files (plus minimal forwarding `__init__.py` shims), and formats them so that tools like mypy/pyright can understand the public API. Run `python stub.py` whenever you change the Python-visible surface to keep the generated stubs in sync.
|
| 214 |
+
|
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/RECORD
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
tokenizers-0.22.2.dist-info/INSTALLER,sha256=5hhM4Q4mYTT9z6QB6PGpUAW81PGNFrYrdXMj4oM_6ak,2
|
| 2 |
+
tokenizers-0.22.2.dist-info/METADATA,sha256=Wtm8zyGuU9RfgScDyAx8A-zqMAgjZVopaPnmd4d5vbM,7428
|
| 3 |
+
tokenizers-0.22.2.dist-info/RECORD,,
|
| 4 |
+
tokenizers-0.22.2.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
| 5 |
+
tokenizers-0.22.2.dist-info/WHEEL,sha256=W8IbRrW6aFuwgkpVWKTtkIq8XwJIIpO5tHf4goK_jm8,95
|
| 6 |
+
tokenizers/__init__.py,sha256=UQ1eI0WGEkM9qfH-QwtQCfuIupBQjuQ0DSOXsywxkIA,2739
|
| 7 |
+
tokenizers/__init__.pyi,sha256=ubhr55w0uERGRitWoIi-HwmFn7dW9Vl6SHVlmC0ZJhc,57691
|
| 8 |
+
tokenizers/decoders/__init__.py,sha256=Qr1jbA2fvqwjgBBYOEyaDjLBwDwkBftP5jN3tRvT1jg,387
|
| 9 |
+
tokenizers/decoders/__init__.pyi,sha256=UJy1WXyQ7ogrw-Es6IJahSU6i04U7walI7bc_O86zPI,12588
|
| 10 |
+
tokenizers/implementations/__init__.py,sha256=9ZI2cJHPCUCZXP37GNmRHtBzIYiKWzyqky1rtcYdrPw,316
|
| 11 |
+
tokenizers/implementations/base_tokenizer.py,sha256=2RFMlsWFqS8AacXK2dbT9jCSVnOy86ERWwmaWH8vZFQ,16265
|
| 12 |
+
tokenizers/implementations/bert_wordpiece.py,sha256=jRUMSYU1C7nVEDZMKdyG3Vp7-CUongE1kGJhvOP_EHM,5671
|
| 13 |
+
tokenizers/implementations/byte_level_bpe.py,sha256=pWfbSeaew9atWKX8Nf2zIRia6qhAwXMfZgaeINInzo8,4394
|
| 14 |
+
tokenizers/implementations/char_level_bpe.py,sha256=Bt15rEcZRSzJ2RfEeq2YvOXZ8w_UYGq9i6O3y-41Aj4,5599
|
| 15 |
+
tokenizers/implementations/sentencepiece_bpe.py,sha256=4_dJFcNsyvC_F1wBnWDBquWssbUMV4kP-BEXeuHcxtI,3824
|
| 16 |
+
tokenizers/implementations/sentencepiece_unigram.py,sha256=KGjhIzXukwJkpao1QIe5XVrEIAMjKiGaHEzpAHL-1I8,7826
|
| 17 |
+
tokenizers/models/__init__.py,sha256=qE73qycPAKcey_pS8ov1FIz10L2Ybzy1E-1KGee27Qg,184
|
| 18 |
+
tokenizers/models/__init__.pyi,sha256=tO4jy0Qrq0emg7PoqmBXituSpe0dMxyChF4J9SinyDk,20261
|
| 19 |
+
tokenizers/normalizers/__init__.py,sha256=Pgb_wE1QQov3t_SW4vcsXY6lVBDYyGNfYGkJMk3wI7c,870
|
| 20 |
+
tokenizers/normalizers/__init__.pyi,sha256=fjuhSi1yirfLZSUsFGL9Vyx2azpANIjCz-VundvqakQ,26655
|
| 21 |
+
tokenizers/pre_tokenizers/__init__.py,sha256=QvNmMTtV0RGkaKSN9hTwEPimD1OLHRU6PbzVxXwM4zA,614
|
| 22 |
+
tokenizers/pre_tokenizers/__init__.pyi,sha256=VMERim4OPFGQaG8JUOTu5H3C0kgQCxv2hVYDfvPPztE,32581
|
| 23 |
+
tokenizers/processors/__init__.py,sha256=X1OKKtr3IUeede85mI1cLF0MqmDRlr7LNFMYdEy5KMU,316
|
| 24 |
+
tokenizers/processors/__init__.pyi,sha256=RElgDx8bbbZIq2MpxXy3G67KXSqDtYLj_7Vq0GPkwO0,14867
|
| 25 |
+
tokenizers/tokenizers.pyd,sha256=esuD9biRNll-DRS3iNgpF78ocN-UV1v3fhJzHE5JxN8,7395328
|
| 26 |
+
tokenizers/tokenizers.pyi,sha256=kTPgANU6E7wyKoBnR-a0YK6pVNiprgxjGhOonFvuRm0,485
|
| 27 |
+
tokenizers/tools/__init__.py,sha256=mJLVrHN1FP23Cf4_EJWoYUdh8yTb5BbJFNfa8Ec5vHM,56
|
| 28 |
+
tokenizers/tools/visualizer-styles.css,sha256=g98RJucUCjYnPLwXycfcV-dAbHMzqJpnZ7NoP_cCUBk,5020
|
| 29 |
+
tokenizers/tools/visualizer.py,sha256=jWcxhFTsyiQr-ZQmDJSjzEcSGZvZZoKDes2Ira1EVg8,15241
|
| 30 |
+
tokenizers/trainers/__init__.py,sha256=v9CdtoVauwD6b1wAA6R5goSLZslBbPE5vZHM2d8sgKM,256
|
| 31 |
+
tokenizers/trainers/__init__.pyi,sha256=WfXHAWSBQNPaqCym4sEREllLvqMoG76x0MFE7qSnTdA,11181
|
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/REQUESTED
ADDED
|
File without changes
|
loan-advisor-agent/.venv/Lib/site-packages/tokenizers-0.22.2.dist-info/WHEEL
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Wheel-Version: 1.0
|
| 2 |
+
Generator: maturin (1.10.2)
|
| 3 |
+
Root-Is-Purelib: false
|
| 4 |
+
Tag: cp39-abi3-win_amd64
|