Upload 6 files
Browse files- README.md +285 -13
- app.py +248 -0
- pinecli.py +670 -0
- requirements.txt +0 -0
- setup.py +23 -0
- sitemap_crawler.py +65 -0
README.md
CHANGED
|
@@ -1,13 +1,285 @@
|
|
| 1 |
-
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# pinecone-cli
|
| 2 |
+
Pinecode-cli is a command-line interface for control and data plane interfacing with [Pinecone](https://pinecone.io).
|
| 3 |
+
|
| 4 |
+
In addition to ALL of the Pinecone "actions/verbs", Pinecone-cli has several additional features that make Pinecone even more powerful including:
|
| 5 |
+
* Upload vectors from CSV files
|
| 6 |
+
* Upload embeddings of text from a given website URL. Embeddings generated by OpenAI embeddings API.
|
| 7 |
+
* New "head" command to peak into a given index, similar to "head" in linux/unix.
|
| 8 |
+
|
| 9 |
+
# Install
|
| 10 |
+
Feel free to use the tool directly from source here, or just
|
| 11 |
+
```console
|
| 12 |
+
pip install pinecone-cli
|
| 13 |
+
```
|
| 14 |
+
Pypi here: (https://pypi.org/project/pinecone-cli/) Note that the examples below assume you are using the source and have pulled the right deps. If you install from ```pip``` it gets installed as ```pinecli``` in your PATH.
|
| 15 |
+
|
| 16 |
+
# Usage
|
| 17 |
+
First off, let's simplify usage by setting the PINECONE_API_KEY env variable so we don't have to pass it every time in the commandline:
|
| 18 |
+
```console
|
| 19 |
+
% export PINECONE_API_KEY=1234-4567-abc
|
| 20 |
+
```
|
| 21 |
+
Otherwise you wind up having to pass the key as so:
|
| 22 |
+
```console
|
| 23 |
+
% ./pinecli.py query --apikey=1234 ....
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
The pattern for using the tool is to invoke 'pinecli' and then use a command. The list of commands appears with --help
|
| 28 |
+
|
| 29 |
+
```console
|
| 30 |
+
% ./pinecli.py --help
|
| 31 |
+
Usage: pinecli.py [OPTIONS] COMMAND [ARGS]...
|
| 32 |
+
|
| 33 |
+
A command line interface for working with Pinecone.
|
| 34 |
+
|
| 35 |
+
Options:
|
| 36 |
+
--help Show this message and exit.
|
| 37 |
+
|
| 38 |
+
Commands:
|
| 39 |
+
configure-index-pod-type Configures the given index to have a pod type.
|
| 40 |
+
configure-index-replicas Configures the number of replicas for a given
|
| 41 |
+
index.
|
| 42 |
+
create-collection Creates a Pinecone collection from the argument
|
| 43 |
+
'source_index'
|
| 44 |
+
create-index Creates a Pinecone Index.
|
| 45 |
+
delete-collection Deletes a collection.
|
| 46 |
+
delete-index Deletes an index. You will be prompted to
|
| 47 |
+
confirm.
|
| 48 |
+
describe-collection Describes a collection.
|
| 49 |
+
describe-index Describes an index.
|
| 50 |
+
describe-index-stats Prints out index stats to stdout.
|
| 51 |
+
fetch Fetches vectors from Pinecone specified by the
|
| 52 |
+
vectors' ids.
|
| 53 |
+
head Shows a preview of vectors in the
|
| 54 |
+
<PINECONE_INDEX_NAME>
|
| 55 |
+
list-collections Lists collections for the given apikey.
|
| 56 |
+
list-indexes Lists the indexes for your api key.
|
| 57 |
+
query Queries Pinecone with a given vector.
|
| 58 |
+
update Updates the index based on the given id passed in.
|
| 59 |
+
upsert Extracts text from url arg, vectorizes w/ openai
|
| 60 |
+
embedding api, and upserts to Pinecone.
|
| 61 |
+
upsert-file Upserts a file (csv) into the specified index.
|
| 62 |
+
upsert-random Upserts a vector(s) with random dimensions into
|
| 63 |
+
the specified vector.
|
| 64 |
+
upsert-webpage Extracts text from url arg, vectorizes w/ openai
|
| 65 |
+
embedding api, and upserts to Pinecone.
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
# Commands With Examples
|
| 69 |
+
Before you can use Pinecone an index is required. We can now do this on the commandline rather than in the UI: (Note not all of the cmdline options are required, they're shown here to demonstrate functionality and control)
|
| 70 |
+
```console
|
| 71 |
+
% ./pinecli.py create-index myindex --dims=1536 --metric=cosine --pods=2 --replicas=2 --shards=1 --pod-"type=p2.x1"
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
Note that for any command, if you want an exhasuive description of cmdline options, simply do something similar to the below, where "create-index" is replaced by one of the commands:
|
| 75 |
+
```console
|
| 76 |
+
% ./pinecli.py create-index --help
|
| 77 |
+
Usage: pinecli.py create-index [OPTIONS] PINECONE_INDEX_NAME
|
| 78 |
+
|
| 79 |
+
Creates the Pinecone index named <PINECONE_INDEX_NAME>
|
| 80 |
+
|
| 81 |
+
Options:
|
| 82 |
+
--apikey TEXT Pinecone API Key
|
| 83 |
+
--region TEXT Pinecone Index Region
|
| 84 |
+
--dims INTEGER Number of dimensions for this index [required]
|
| 85 |
+
--metric TEXT Distance metric to use. [required]
|
| 86 |
+
--pods INTEGER Number of pods [default: 1]
|
| 87 |
+
--replicas INTEGER Number of replicas [default: 1]
|
| 88 |
+
--shards INTEGER Number of shards [default: 1]
|
| 89 |
+
--pod-type TEXT Type of pods to create. [required]
|
| 90 |
+
--source_collection TEXT Source collection to create index from
|
| 91 |
+
--help Show this message and exit.
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
Let's try some commands showing two missing features I'd love to have had over the last year: a "head" command and a quick "stats" command:
|
| 95 |
+
|
| 96 |
+
## Index Stats Including Number of Vectors
|
| 97 |
+
``` console
|
| 98 |
+
% ./pinecli.py describe-index-stats myindex
|
| 99 |
+
Dimensions: 1536
|
| 100 |
+
Vectors: 7745
|
| 101 |
+
Index_Fullness: 0.0
|
| 102 |
+
Namespace data:
|
| 103 |
+
: 7745
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
## Head command to preview vectors
|
| 107 |
+
``` console
|
| 108 |
+
% ./pinecli.py head kids-facenet
|
| 109 |
+
{'matches': [{'id': 'bubba_50.jpg.vec',
|
| 110 |
+
'metadata': {},
|
| 111 |
+
'score': 12.182938,
|
| 112 |
+
'values': [-0.016061664,
|
| 113 |
+
-0.4495437,
|
| 114 |
+
-0.034082577,
|
| 115 |
+
.....
|
| 116 |
+
```
|
| 117 |
+
|
| 118 |
+
Now, let's query some nonsensical data from the index named 'upsertfile'
|
| 119 |
+
|
| 120 |
+
## Inserting a vector directly
|
| 121 |
+
*Note the double quites around the vector*
|
| 122 |
+
```console
|
| 123 |
+
% ./pinecli.py query myindex "[1.2, 1.0, 3.0]" --print-table --include-meta=True
|
| 124 |
+
🌲 upsertfile ns=() Index Results
|
| 125 |
+
┏━━━━━━┳━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
|
| 126 |
+
┃ ID ┃ NS ┃ Values ┃ Meta ┃ Score ┃
|
| 127 |
+
┡━━━━━━╇━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
|
| 128 |
+
│ vec1 │ │ 0.1,0.2,0.3 │ {'genre': 'drama'} │ 0.9640127 │
|
| 129 |
+
│ vec2 │ │ 0.2,0.3,0.4 │ {'genre': 'action'} │ 0.9552943 │
|
| 130 |
+
│ abc │ │ 0.23223,-1.333,0.2222222 │ {'foo': 'bar'} │ -0.083585836 │
|
| 131 |
+
│ ghi │ │ 0.23223,-1.333,0.2222222 │ {'bar': 'baz'} │ -0.083585836 │
|
| 132 |
+
└──────┴────┴──────────────────────────┴─────────────────────┴──────────────┘
|
| 133 |
+
```
|
| 134 |
+
|
| 135 |
+
Markdown of course does a great job of mangling great terminal output so here's a screenshot from using ```---print-table```:
|
| 136 |
+

|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
You can of course not output the pretty table by removing ```--print-table```:
|
| 140 |
+
```console
|
| 141 |
+
% ./pinecli.py query myindex "[1.2, 1.0, 3.0]" --include-meta=True
|
| 142 |
+
{'matches': [{'id': 'vec1',
|
| 143 |
+
'metadata': {'genre': 'drama'},
|
| 144 |
+
'score': 0.9640127,
|
| 145 |
+
'values': [0.1, 0.2, 0.3]},
|
| 146 |
+
...
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
## Upsert Vectors in Command Line Manually
|
| 150 |
+
Following the Pinecone vector format of the tuple formatted as:
|
| 151 |
+
```python
|
| 152 |
+
('vectorid', [vecdim1, vecdim2, vecdim3], {'metakey':'metaval'})
|
| 153 |
+
```
|
| 154 |
+
You can pass this in as a comma separated list of vectors on the command line:
|
| 155 |
+
```console
|
| 156 |
+
./pinecli.py upsert myindex "[('vec1', [0.1, 0.2, 0.3], {'genre': 'drama'}), ('vec2', [0.2, 0.3, 0.4], {'foo': 'bar'}),]"
|
| 157 |
+
```
|
| 158 |
+
|
| 159 |
+
## Upsert CSV file
|
| 160 |
+
Upserting a csv file is trivial. Simply create your csv file with any headings you have, but there must be at least a labeled id column and a labeled vector column for the vectors. Here's an example of a CSV file that is clearly a DataFrame dump due to the index column on the left which works great w/ pinecone-cli:
|
| 161 |
+
```console
|
| 162 |
+
index,my_id_column,my_vectors_column,Metadata
|
| 163 |
+
1,abc,"[0.23223, -1.333, 0.2222222]",{'foo':'bar'}
|
| 164 |
+
2,ghi,"[0.23223, -1.333, 0.2222222]",{'bar':'baz'}
|
| 165 |
+
```
|
| 166 |
+
The name of those columns in the header row can be arbitrary or you can name then "id", "vectors" and "metadata" which is our default assumption. If you have custom column names and don't want to change them, just pass in the ```--colmap``` argument which takes in a python dictionary mapping "id" and "vectors" to the naming you have in your csv. For example:
|
| 167 |
+
```"{"id":"my_id_column", "vectors":"my_vectors_column"}```
|
| 168 |
+
|
| 169 |
+
Note that as in other CSV file for Dataframes, we need an index column as in the example above.
|
| 170 |
+
Here's an example using the CSV headers and format above with the correct colmap argument:
|
| 171 |
+
```console
|
| 172 |
+
% ./pinecli.py upsert-file embeddings.csv myindex "{'id':'my_id_column', 'vectors':'my_vectors_column'}"
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
### More on CSV Formatting
|
| 176 |
+
For now you will need to manually provide an index column (we are using dataframes under the hood.)
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
## Upserting Vector Embeddings of Webpage Text!
|
| 180 |
+
pinecone-cli was built to make using Pinecone extremely easy and fast. We have integrated [OpenAI](https://openai.com/) (others coming) - using its [embedding APIs](https://platform.openai.com/docs/guides/embeddings) to fetch embeddings. We then upload them into your index for you, making uploading embeddings of an entire website's text - trivial.
|
| 181 |
+
```console
|
| 182 |
+
% ./pinecli.py upsert-webpage https://menlovc.com lpfactset --openaiapikey=12345-9876-abcdef
|
| 183 |
+
[nltk_data] Downloading package punkt to /Users/tim/nltk_data...
|
| 184 |
+
[nltk_data] Package punkt is already up-to-date!
|
| 185 |
+
100%|████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 61680.94it/s]
|
| 186 |
+
['About Us Our Promise Focus Areas Consumer Cloud Infrastructure Cybersecurity Fintech Healthcare SaaS Supply Chain and Automation Team Portfolio Perspective When we invest, we’re invested. Our promise to founders Building a business is a team sport. As investors, we don’t just sit on the sidelines but do whatever it takes to help our teams win. About Us The founders we back don’t limit themselves to what is, but relentlessly pursue what could be. We invest in transformative technology companies that are changing the way we live and work. Portfolio Menlo Labs starts companies. We work shoulder-to-shoulder
|
| 187 |
+
....
|
| 188 |
+
100%|█████████████████████████| 1/1 [00:00<00:00, 1.11it/s]
|
| 189 |
+
```
|
| 190 |
+
|
| 191 |
+
## Upsert Random Vectors
|
| 192 |
+
One of the more useful things we can do with pinecone-cli is insert random vectors, primarily for testing. Often we will create our index and the length of the vector will be 1,536 dimensions, for example. Instead of writing a bunch of code to go suddenly create those vectors somehow, we can use pinecone-cli to start generating vectors and upserting them:
|
| 193 |
+
```console
|
| 194 |
+
% ./pinecli.py upsert-random upsertfile --num_vector_dims=1536 --num_vectors=10 --debug
|
| 195 |
+
upserted_count: 10
|
| 196 |
+
1it [00:00, 4.36it/s]
|
| 197 |
+
```
|
| 198 |
+
The example above inserts 10 vectors that each have 1,536 random vectors in them. Note that the ```id``` for each vector is simply ```f'id-{i}'``` where is is the ith row (vector) inserted.
|
| 199 |
+
|
| 200 |
+
## Query Vectors
|
| 201 |
+
Querying can be done in two ways on the cmdline - pass in an actual vector string literal, or ask Pinecone to query randomly (maybe you want to just look at them or look at a TSNE). In the example below, the last argument (required) is either the string 'random' or an actual vector such as '[0.0, 1.0, 3.14569]'. Let's try random:
|
| 202 |
+
```console
|
| 203 |
+
% ./pinecli.py query myindex random
|
| 204 |
+
```
|
| 205 |
+
|
| 206 |
+
You can also plot a TSNE plot to view clustering of your vectors by using the ```-show-tsne=True``` flag. Note that this will pop up the plt plot by default.
|
| 207 |
+
|
| 208 |
+
```console
|
| 209 |
+
% ./pinecli.py query lpfactset random --show-tsne=true --topk=2500 --num-clusters=4
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+

|
| 213 |
+
|
| 214 |
+
## Fetching Vectors:
|
| 215 |
+
Fetching is simple - just pass in the vector id(s) of the vectors you're looking for as a comma separated list:
|
| 216 |
+
```console
|
| 217 |
+
% ./pinecli.py fetch myindex --vector_ids="05b4509ee655aacb10bfbb6ba212c65c,c626975ec096b9108f158a56a59b2fd6"
|
| 218 |
+
|
| 219 |
+
{'namespace': '',
|
| 220 |
+
'vectors': {'05b4509ee655aacb10bfbb6ba212c65c': {'id': '05b4509ee655aacb10bfbb6ba212c65c',
|
| 221 |
+
'metadata': {'content': 'Chime '
|
| 222 |
+
'Scholar '
|
| 223 |
+
'spotlight: '
|
| 224 |
+
```
|
| 225 |
+
|
| 226 |
+
## Updating Vectors
|
| 227 |
+
Updating vectors is simple - pass the id of the vector and the updating vector as below:
|
| 228 |
+
```console
|
| 229 |
+
% ./pinecli.py update "id-9" myindex "[0.0, 1.0, 3.0]"
|
| 230 |
+
```
|
| 231 |
+
## List Operations
|
| 232 |
+
pinecone-cli has all of the necessary 'list' operations as shown below:
|
| 233 |
+
|
| 234 |
+
### List Indexes
|
| 235 |
+
This gives you a list of all indexes under your api key:
|
| 236 |
+
```console
|
| 237 |
+
% ./pinecli.py list-indexes
|
| 238 |
+
cli
|
| 239 |
+
cli2
|
| 240 |
+
cli3
|
| 241 |
+
drivertest
|
| 242 |
+
```
|
| 243 |
+
### List Collections
|
| 244 |
+
This obviously lists the collections you've created:
|
| 245 |
+
```console
|
| 246 |
+
% ./pinecli.py list-collections
|
| 247 |
+
cli
|
| 248 |
+
cli2
|
| 249 |
+
cli3
|
| 250 |
+
drivertest
|
| 251 |
+
```
|
| 252 |
+
|
| 253 |
+
### Other Meta Operations
|
| 254 |
+
We showed the "describe-index-stats" command at the top of this page. There is also "describe-index" which provides the following:
|
| 255 |
+
```console
|
| 256 |
+
% ./pinecli.py describe-index lpfactset
|
| 257 |
+
Name: lpfactset
|
| 258 |
+
Dimensions: 1536
|
| 259 |
+
Metric: cosine
|
| 260 |
+
Pods: 1
|
| 261 |
+
PodType: p2.x1
|
| 262 |
+
Shards: 1
|
| 263 |
+
Replicas: 1
|
| 264 |
+
Ready: True
|
| 265 |
+
State: Ready
|
| 266 |
+
Metaconfig: None
|
| 267 |
+
Sourcecollection:
|
| 268 |
+
```
|
| 269 |
+
|
| 270 |
+
### Describe Collections
|
| 271 |
+
```console
|
| 272 |
+
% ./pinecli.py describe-collection testcoll
|
| 273 |
+
Name: testcoll
|
| 274 |
+
Dimensions: 1536
|
| 275 |
+
Vectors: 124
|
| 276 |
+
Status: Ready
|
| 277 |
+
Size: 3917544
|
| 278 |
+
```
|
| 279 |
+
|
| 280 |
+
## Deleting Indexes
|
| 281 |
+
Deleting an index is straightforward. To prevent catastrophic accidents, you'll be prompted to type in the name of the index backwards:
|
| 282 |
+
```console
|
| 283 |
+
./pinecli.py delete-index myindex2
|
| 284 |
+
Type name of index backwards to confirm: : 2xedniym
|
| 285 |
+
```
|
app.py
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
|
| 3 |
+
import plotly.express as px
|
| 4 |
+
import gradio as gr
|
| 5 |
+
import pandas as pd
|
| 6 |
+
from collections import defaultdict
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
from time import sleep
|
| 10 |
+
import openai
|
| 11 |
+
|
| 12 |
+
import json
|
| 13 |
+
|
| 14 |
+
import logging
|
| 15 |
+
from dotenv import load_dotenv
|
| 16 |
+
import pinecone
|
| 17 |
+
|
| 18 |
+
default_region = 'us-west1-gcp'
|
| 19 |
+
openai_embed_model = "text-embedding-ada-002"
|
| 20 |
+
|
| 21 |
+
def get_data(num_names_per_category=2, num_categories=3):
|
| 22 |
+
|
| 23 |
+
categories = ['Text', 'Bio', 'Audio', 'Legal', 'Chatbot/Conversational AI', 'Summarization', 'MLOps/Platform', 'Code', 'Avatars', 'Image', 'Semantic Search', 'Game', 'Video', 'Data', 'Sentiment Analysis']
|
| 24 |
+
num_categories = min(num_categories, len((categories)))
|
| 25 |
+
|
| 26 |
+
data = query_pinecone(categories, num_categories, num_names_per_category)
|
| 27 |
+
|
| 28 |
+
return data
|
| 29 |
+
|
| 30 |
+
def get_openai_embedding(apikey, data):
|
| 31 |
+
""" Fetch an embedding w/ given data """
|
| 32 |
+
openai.api_key = apikey
|
| 33 |
+
try:
|
| 34 |
+
res = openai.Embedding.create(input=data, engine=openai_embed_model)
|
| 35 |
+
except:
|
| 36 |
+
done = False
|
| 37 |
+
while not done:
|
| 38 |
+
sleep(5)
|
| 39 |
+
try:
|
| 40 |
+
res = openai.Embedding.create(
|
| 41 |
+
input=data, engine=openai_embed_model)
|
| 42 |
+
print('retrying')
|
| 43 |
+
done = True
|
| 44 |
+
except Exception as e:
|
| 45 |
+
logging.exception(e)
|
| 46 |
+
return res
|
| 47 |
+
|
| 48 |
+
def query_pinecone(categories, n_categories=3, top_k=4):
|
| 49 |
+
load_dotenv()
|
| 50 |
+
|
| 51 |
+
pinecone.init(
|
| 52 |
+
api_key=os.getenv('PINECONE_API_KEY'),
|
| 53 |
+
environment=os.getenv('PINECONE_ENVIRONMENT'),
|
| 54 |
+
)
|
| 55 |
+
index = pinecone.Index("vc-content-oracle-big")
|
| 56 |
+
|
| 57 |
+
out = defaultdict(list)
|
| 58 |
+
|
| 59 |
+
companies = set()
|
| 60 |
+
|
| 61 |
+
# loop thru all the categories
|
| 62 |
+
for category in categories[:n_categories]:
|
| 63 |
+
|
| 64 |
+
#get vector for each category
|
| 65 |
+
vector = get_openai_embedding(os.getenv('OPENAI_API_KEY'), category)['data'][0]['embedding']
|
| 66 |
+
|
| 67 |
+
res = index.query(
|
| 68 |
+
vector=vector,
|
| 69 |
+
top_k=top_k,
|
| 70 |
+
include_metadata=True,
|
| 71 |
+
include_values=False,
|
| 72 |
+
)
|
| 73 |
+
|
| 74 |
+
for match in res['matches']:
|
| 75 |
+
if 'Organization Name' not in match['metadata'] or 'Website' not in match['metadata']:
|
| 76 |
+
print('no Organization Name')
|
| 77 |
+
continue
|
| 78 |
+
|
| 79 |
+
#check if the match is already in the dictionary
|
| 80 |
+
company_link = '<a href="' + match['metadata']['Website'] + '" style="cursor: pointer; color:white; font-color:white " target="_blank" rel="noopener noreferrer">' + match['metadata']['Organization Name'] + '</a>'
|
| 81 |
+
|
| 82 |
+
print(out[category], company_link)
|
| 83 |
+
|
| 84 |
+
contains = False
|
| 85 |
+
if match['metadata']['Organization Name'] in companies:
|
| 86 |
+
contains = True
|
| 87 |
+
|
| 88 |
+
if not contains:
|
| 89 |
+
out[category].append(company_link)
|
| 90 |
+
else:
|
| 91 |
+
out[category].append(None)
|
| 92 |
+
|
| 93 |
+
companies.add(match['metadata']['Organization Name'])
|
| 94 |
+
|
| 95 |
+
return out
|
| 96 |
+
|
| 97 |
+
"""
|
| 98 |
+
returns { 'matches': [{
|
| 99 |
+
'id': 'A',
|
| 100 |
+
'metadata': {
|
| 101 |
+
'Organization Name': '',
|
| 102 |
+
'Organization Name URL': '',
|
| 103 |
+
'Last Funding Amount': '',
|
| 104 |
+
'Last Funding Amount Currency': '',
|
| 105 |
+
'Last Funding Amount Currency (in USD)': '',
|
| 106 |
+
'Website': '',
|
| 107 |
+
'Total Funding Amount': '',
|
| 108 |
+
'Total Funding Amount Currency': '',
|
| 109 |
+
'Total Funding Amount Currency (in USD)': '',
|
| 110 |
+
'Last Funding Date': '',
|
| 111 |
+
'Founded Date': '',
|
| 112 |
+
'Founded Date Precision': '',
|
| 113 |
+
'Last Funding Type': '',
|
| 114 |
+
'Number of Employees': '',
|
| 115 |
+
'Full Description': '',
|
| 116 |
+
'Top 5 Investors': '',
|
| 117 |
+
},
|
| 118 |
+
'score': 0.01,
|
| 119 |
+
'values':[]
|
| 120 |
+
}]}
|
| 121 |
+
"""
|
| 122 |
+
|
| 123 |
+
def make_plot(category, num_categories, num_names_per_category):
|
| 124 |
+
|
| 125 |
+
num_categories = int(num_categories)
|
| 126 |
+
num_names_per_category = int(num_names_per_category)
|
| 127 |
+
|
| 128 |
+
data = get_data(num_names_per_category, num_categories)
|
| 129 |
+
names = []
|
| 130 |
+
parents = []
|
| 131 |
+
|
| 132 |
+
for category, names_list in data.items():
|
| 133 |
+
names.extend(names_list)
|
| 134 |
+
parents.extend([category] * len(names_list))
|
| 135 |
+
|
| 136 |
+
unique_parents = list(set(parents))
|
| 137 |
+
names += unique_parents
|
| 138 |
+
parents += [''] * len(unique_parents)
|
| 139 |
+
|
| 140 |
+
fig = px.treemap(
|
| 141 |
+
names=names,
|
| 142 |
+
parents=parents,
|
| 143 |
+
|
| 144 |
+
)
|
| 145 |
+
|
| 146 |
+
fig.update_traces(root_color="lightgrey")
|
| 147 |
+
fig.update_traces(pathbar_textfont_color="white")
|
| 148 |
+
fig.update_layout(margin = dict(t=50, l=25, r=25, b=25))
|
| 149 |
+
return fig
|
| 150 |
+
|
| 151 |
+
def add_text(state, text):
|
| 152 |
+
|
| 153 |
+
#call open ai with the context from the state passed in
|
| 154 |
+
|
| 155 |
+
load_dotenv()
|
| 156 |
+
|
| 157 |
+
pinecone.init(
|
| 158 |
+
api_key=os.getenv('PINECONE_API_KEY'),
|
| 159 |
+
environment=os.getenv('PINECONE_ENVIRONMENT'),
|
| 160 |
+
)
|
| 161 |
+
index = pinecone.Index("vc-content-oracle-big")
|
| 162 |
+
|
| 163 |
+
vector = get_openai_embedding(os.getenv('OPENAI_API_KEY'), text)['data'][0]['embedding']
|
| 164 |
+
|
| 165 |
+
#make call to the vector store
|
| 166 |
+
|
| 167 |
+
res = index.query(
|
| 168 |
+
vector=vector,
|
| 169 |
+
top_k=3,
|
| 170 |
+
include_metadata=True,
|
| 171 |
+
include_values=False,
|
| 172 |
+
)
|
| 173 |
+
|
| 174 |
+
context = 'You are Andressen Horowitz, a Venture Capitalist. You know everything about startup investment, particularly in Generative AI. Everytime you are asked a question respond faithfully and consistently. Answer the question based on the context below \n\n'
|
| 175 |
+
|
| 176 |
+
# 'Organization Name': '',
|
| 177 |
+
# 'Organization Name URL': '',
|
| 178 |
+
# 'Last Funding Amount': '',
|
| 179 |
+
# 'Last Funding Amount Currency': '',
|
| 180 |
+
# 'Last Funding Amount Currency (in USD)': '',
|
| 181 |
+
# 'Website': '',
|
| 182 |
+
# 'Total Funding Amount': '',
|
| 183 |
+
# 'Total Funding Amount Currency': '',
|
| 184 |
+
# 'Total Funding Amount Currency (in USD)': '',
|
| 185 |
+
# 'Last Funding Date': '',
|
| 186 |
+
# 'Founded Date': '',
|
| 187 |
+
# 'Founded Date Precision': '',
|
| 188 |
+
# 'Last Funding Type': '',
|
| 189 |
+
# 'Number of Employees': '',
|
| 190 |
+
# 'Full Description': '',
|
| 191 |
+
# 'Top 5 Investors': '',
|
| 192 |
+
|
| 193 |
+
for match in res['matches'][:1]:
|
| 194 |
+
context += (
|
| 195 |
+
'Context\n: Name: ' +
|
| 196 |
+
str(match['metadata'].get('Organization Name', '')) +
|
| 197 |
+
'Funding: ' +
|
| 198 |
+
str(match['metadata'].get('Total Funding Amount', '')) +
|
| 199 |
+
'Founding Date: ' +
|
| 200 |
+
str(match['metadata'].get('Founded Date', '')) +
|
| 201 |
+
'Number of Employees: ' +
|
| 202 |
+
str(match['metadata'].get('Number of Employees', '')) +
|
| 203 |
+
'Description: ' +
|
| 204 |
+
str(match['metadata'].get('Full Description', '')) +
|
| 205 |
+
'Investors: ' +
|
| 206 |
+
str(match['metadata'].get('Top 5 Investors', ''))
|
| 207 |
+
)
|
| 208 |
+
|
| 209 |
+
context += 'Question: ' + text + '\nAnswer:'
|
| 210 |
+
|
| 211 |
+
response = openai.Completion.create(
|
| 212 |
+
model="text-davinci-003",
|
| 213 |
+
prompt=context,
|
| 214 |
+
max_tokens=200,
|
| 215 |
+
temperature=0,
|
| 216 |
+
)
|
| 217 |
+
|
| 218 |
+
state = state + [(text, response['choices'][0]['text'])]
|
| 219 |
+
|
| 220 |
+
return state, state
|
| 221 |
+
|
| 222 |
+
with gr.Blocks() as demo:
|
| 223 |
+
|
| 224 |
+
#make a title
|
| 225 |
+
title = gr.HTML('<h1 style="text-align: center; color: white; font-size: 50px; font-family: Helvetica">ChatGVC</h1>')
|
| 226 |
+
|
| 227 |
+
#make a subtitle
|
| 228 |
+
subtitle = gr.HTML('<h2 style="text-align: center; color: white; font-size: 20px; font-family: Helvetica">Auto-Generated VC Market Maps</h2>')
|
| 229 |
+
|
| 230 |
+
category = gr.Textbox(label='Market Category')
|
| 231 |
+
n_subcategories = gr.Number(label="Number of Subcategories")
|
| 232 |
+
n_companies = gr.Number(label="Max Number of Companies per Subcategory")
|
| 233 |
+
button = gr.Button("Generate")
|
| 234 |
+
plot = gr.Plot(label="Plot")
|
| 235 |
+
button.click(make_plot, inputs=[category, n_subcategories, n_companies], outputs=[plot])
|
| 236 |
+
# demo.load(make_plot, inputs=[button, category], outputs=[plot])
|
| 237 |
+
|
| 238 |
+
chatbot = gr.Chatbot(elem_id="chatbot")
|
| 239 |
+
state = gr.State([])
|
| 240 |
+
|
| 241 |
+
with gr.Row():
|
| 242 |
+
with gr.Column(scale=0.85):
|
| 243 |
+
txt = gr.Textbox(show_label=False, placeholder="Enter text and press enter, or upload an image").style(container=False)
|
| 244 |
+
|
| 245 |
+
txt.submit(add_text, [state, txt], [state, chatbot])
|
| 246 |
+
txt.submit(lambda :"", None, txt)
|
| 247 |
+
|
| 248 |
+
demo.launch(share=True)
|
pinecli.py
ADDED
|
@@ -0,0 +1,670 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# encoding: utf-8
|
| 3 |
+
# pylint: disable=line-too-long,too-many-arguments,invalid-name,no-member,missing-function-docstring,missing-module-docstring
|
| 4 |
+
|
| 5 |
+
import logging
|
| 6 |
+
import hashlib
|
| 7 |
+
import itertools
|
| 8 |
+
import os
|
| 9 |
+
import sys
|
| 10 |
+
|
| 11 |
+
from ast import literal_eval
|
| 12 |
+
from time import sleep
|
| 13 |
+
import requests
|
| 14 |
+
|
| 15 |
+
import click
|
| 16 |
+
import pandas as pd
|
| 17 |
+
import matplotlib.pyplot as plt
|
| 18 |
+
import numpy as np
|
| 19 |
+
import openai
|
| 20 |
+
import nltk.data
|
| 21 |
+
import pinecone
|
| 22 |
+
|
| 23 |
+
from bs4 import BeautifulSoup
|
| 24 |
+
from bs4.element import Comment
|
| 25 |
+
from numpy import random
|
| 26 |
+
from sklearn.manifold import TSNE
|
| 27 |
+
from sklearn.cluster import KMeans
|
| 28 |
+
from rich.console import Console
|
| 29 |
+
from rich.table import Table
|
| 30 |
+
from tqdm.auto import tqdm
|
| 31 |
+
|
| 32 |
+
default_region = 'us-west1-gcp'
|
| 33 |
+
openai_embed_model = "text-embedding-ada-002"
|
| 34 |
+
|
| 35 |
+
REGION_HELP = 'Pinecone cluster region'
|
| 36 |
+
|
| 37 |
+
def tag_visible(element):
|
| 38 |
+
""" Strip out undesirables """
|
| 39 |
+
if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
|
| 40 |
+
return False
|
| 41 |
+
if isinstance(element, Comment):
|
| 42 |
+
return False
|
| 43 |
+
return True
|
| 44 |
+
|
| 45 |
+
def text_from_html(body):
|
| 46 |
+
""" Obv pull text from doc with tag_visible filters """
|
| 47 |
+
soup = BeautifulSoup(body, 'html.parser')
|
| 48 |
+
texts = soup.findAll(string=True)
|
| 49 |
+
visible_texts = filter(tag_visible, texts)
|
| 50 |
+
return " ".join(t.strip() for t in visible_texts)
|
| 51 |
+
|
| 52 |
+
def get_openai_embedding(apikey, data):
|
| 53 |
+
""" Fetch an embedding w/ given data """
|
| 54 |
+
openai.api_key = apikey
|
| 55 |
+
try:
|
| 56 |
+
res = openai.Embedding.create(input=data, engine=openai_embed_model)
|
| 57 |
+
except:
|
| 58 |
+
done = False
|
| 59 |
+
while not done:
|
| 60 |
+
sleep(5)
|
| 61 |
+
try:
|
| 62 |
+
res = openai.Embedding.create(
|
| 63 |
+
input=data, engine=openai_embed_model)
|
| 64 |
+
done = True
|
| 65 |
+
except Exception as e:
|
| 66 |
+
click.echo(e)
|
| 67 |
+
return res
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
@click.group()
|
| 71 |
+
def cli():
|
| 72 |
+
"""
|
| 73 |
+
A command line interface for working with Pinecone.
|
| 74 |
+
"""
|
| 75 |
+
pass
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def exception_handler(exception_type, exception, traceback):
|
| 79 |
+
# All your trace are belong to us!
|
| 80 |
+
click.echo(f"Got exception: {exception_type.__name__} {exception}")
|
| 81 |
+
click.echo("Make sure PINECONE_API_KEY is correct.")
|
| 82 |
+
|
| 83 |
+
# sys.excepthook = exception_handler
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def _pinecone_init(apikey, environment, indexname=''):
|
| 87 |
+
apikey = os.environ.get('PINECONE_API_KEY', apikey)
|
| 88 |
+
if apikey == "":
|
| 89 |
+
sys.exit(
|
| 90 |
+
"No Pinecone API key set through PINECONE_API_KEY environment variable or --apikey\nExample: export PINECONE_API_KEY=1234-abc-9876")
|
| 91 |
+
environment = os.environ.get('PINECONE_ENVIRONMENT', environment)
|
| 92 |
+
pinecone.init(api_key=apikey, environment=environment)
|
| 93 |
+
index = None
|
| 94 |
+
if indexname:
|
| 95 |
+
try:
|
| 96 |
+
# index = pinecone.Index(indexname)
|
| 97 |
+
index = pinecone.GRPCIndex(indexname)
|
| 98 |
+
except Exception:
|
| 99 |
+
logging.exception("Unable to connect. Caught exception:")
|
| 100 |
+
sys.exit()
|
| 101 |
+
else:
|
| 102 |
+
return index
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def _format_values(vals):
|
| 106 |
+
return ",".join(str(x) for x in vals)[:30]
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def _print_table(res, pinecone_index_name, namespace, include_meta, include_values, expand_meta):
|
| 110 |
+
table = Table(
|
| 111 |
+
title=f"🌲 {pinecone_index_name} ns=({namespace}) Index Results")
|
| 112 |
+
table.add_column("ID", justify="right", style="cyan", no_wrap=True)
|
| 113 |
+
table.add_column("NS", justify="left", style="red", no_wrap=True)
|
| 114 |
+
if include_values:
|
| 115 |
+
table.add_column("Values", style="magenta")
|
| 116 |
+
if include_meta:
|
| 117 |
+
table.add_column("Meta", justify="right", style="green")
|
| 118 |
+
table.add_column("Score", justify="right", style="green")
|
| 119 |
+
|
| 120 |
+
ns = res['namespace'] if 'namespace' in res else ''
|
| 121 |
+
for row in res.matches:
|
| 122 |
+
metadata = ''
|
| 123 |
+
score = str(row['score'])
|
| 124 |
+
vecid = row['id']
|
| 125 |
+
if include_meta and 'metadata' in row:
|
| 126 |
+
metadata = str(row['metadata'])
|
| 127 |
+
metadata = metadata[:100] if not expand_meta else metadata
|
| 128 |
+
|
| 129 |
+
if include_values and include_meta:
|
| 130 |
+
table.add_row(vecid, ns, _format_values(
|
| 131 |
+
row['values']), metadata, score)
|
| 132 |
+
elif include_values and not include_meta:
|
| 133 |
+
table.add_row(vecid, ns, _format_values(
|
| 134 |
+
row['values']), score)
|
| 135 |
+
elif not include_values and include_meta:
|
| 136 |
+
table.add_row(vecid, ns, metadata, score)
|
| 137 |
+
elif not include_values and not include_meta:
|
| 138 |
+
table.add_row(vecid, ns, score)
|
| 139 |
+
|
| 140 |
+
console = Console()
|
| 141 |
+
console.print(table)
|
| 142 |
+
|
| 143 |
+
@click.command(short_help='Prints version number.')
|
| 144 |
+
def version():
|
| 145 |
+
if sys.version_info >= (3, 8):
|
| 146 |
+
from importlib import metadata
|
| 147 |
+
else:
|
| 148 |
+
import importlib_metadata as metadata
|
| 149 |
+
click.echo(metadata.version('pinecone_cli'))
|
| 150 |
+
|
| 151 |
+
@click.command(short_help='Queries Pinecone with a given vector.')
|
| 152 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 153 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 154 |
+
@click.option('--include_values', help='Should we return the vectors', show_default=True, default=True)
|
| 155 |
+
@click.option('--topk', '--numrows', 'topk', type=click.INT, show_default=True, default=10, help='Top K number to return')
|
| 156 |
+
@click.option('--namespace', default="", help='Namespace to select results from')
|
| 157 |
+
@click.option('--include-meta', help='Whether to include the metadata values', default=False, show_default=True)
|
| 158 |
+
@click.option('--expand-meta', help='Whether to fully expand the metadata returned.', is_flag=True, show_default=True, default=False)
|
| 159 |
+
@click.option('--meta_filter', help='Filter out metadata w/ the Pinecone filter syntax which is really JSON. Default is no filter.', default="{}")
|
| 160 |
+
@click.option('--print-table', help='Display the output as a pretty table.', is_flag=True, show_default=True, default=False)
|
| 161 |
+
@click.option('--num-clusters', help='Number of clusters in TSNE plot if --show-tsne is used.', type=click.INT, show_default=True, default=4)
|
| 162 |
+
@click.option('--perplexity', '--perp', help='The perplexity of the TSNE plot, if --show-tsne is used.', type=click.INT, default=15, show_default=True)
|
| 163 |
+
@click.option('--tsne-random-state', type=click.INT, default=42, show_default=True)
|
| 164 |
+
@click.option('--show-tsne', default=False)
|
| 165 |
+
@click.argument('pinecone_index_name')
|
| 166 |
+
@click.argument('query_vector')
|
| 167 |
+
def query(pinecone_index_name, apikey, query_vector, region, topk, include_values, include_meta, expand_meta, num_clusters, perplexity, tsne_random_state, namespace, show_tsne, meta_filter, print_table):
|
| 168 |
+
""" Queries Pinecone index named <PINECONE_INDEX_NAME> with the given <QUERY_VECTOR> and optional namespace.
|
| 169 |
+
|
| 170 |
+
\b
|
| 171 |
+
Example:
|
| 172 |
+
% ./pinecli.py query lpfactset "[0,0]"
|
| 173 |
+
|
| 174 |
+
\b
|
| 175 |
+
Example 2:
|
| 176 |
+
% ./pinecli.py query upsertfile "[1.2, 1.0, 3.0]" --print-table --include-meta=true --filter="{'genre':'drama'}"
|
| 177 |
+
|
| 178 |
+
\b
|
| 179 |
+
Example 3 [Query randomly]:
|
| 180 |
+
% ./pinecli.py query lpfactset random
|
| 181 |
+
|
| 182 |
+
For filter syntax see: https://docs.pinecone.io/docs/metadata-filtering
|
| 183 |
+
"""
|
| 184 |
+
pinecone_index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 185 |
+
|
| 186 |
+
if query_vector.lower() == "random":
|
| 187 |
+
res = pinecone_index.describe_index_stats()
|
| 188 |
+
num_vector_dims = res['dimension']
|
| 189 |
+
query_vector = [i for i in range(num_vector_dims)]
|
| 190 |
+
else:
|
| 191 |
+
query_vector = literal_eval(query_vector)
|
| 192 |
+
|
| 193 |
+
res = pinecone_index.query(vector=query_vector, top_k=topk, include_metadata=True,
|
| 194 |
+
include_values=include_values, namespace=namespace, filter=literal_eval(meta_filter))
|
| 195 |
+
if print_table:
|
| 196 |
+
_print_table(res, pinecone_index_name, namespace,
|
| 197 |
+
include_meta, include_values, expand_meta)
|
| 198 |
+
else:
|
| 199 |
+
click.echo(res)
|
| 200 |
+
|
| 201 |
+
if show_tsne:
|
| 202 |
+
show_tsne_plot(pinecone_index_name, res.matches,
|
| 203 |
+
num_clusters, perplexity, tsne_random_state)
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
def show_tsne_plot(pinecone_index_name, results, num_clusters, perplexity, random_state):
|
| 207 |
+
res2 = np.asarray([np.array(v['values']) for v in results])
|
| 208 |
+
df = pd.DataFrame(data=res2)
|
| 209 |
+
|
| 210 |
+
kmeans = KMeans(n_clusters=num_clusters, init="k-means++",
|
| 211 |
+
random_state=random_state, n_init='auto')
|
| 212 |
+
kmeans.fit(res2)
|
| 213 |
+
labels = kmeans.labels_
|
| 214 |
+
df["Cluster"] = labels
|
| 215 |
+
|
| 216 |
+
tsne = TSNE(n_components=2, perplexity=perplexity,
|
| 217 |
+
random_state=42, init='random', learning_rate=200)
|
| 218 |
+
embeddings2d = tsne.fit_transform(res2)
|
| 219 |
+
x = [x for x, y in embeddings2d]
|
| 220 |
+
y = [y for x, y in embeddings2d]
|
| 221 |
+
(_, ax) = plt.subplots(figsize=(9, 6)) # inches
|
| 222 |
+
plt.style.use('seaborn-whitegrid')
|
| 223 |
+
plt.grid(color='#EAEAEB', linewidth=0.5)
|
| 224 |
+
ax.spines['top'].set_color(None)
|
| 225 |
+
ax.spines['right'].set_color(None)
|
| 226 |
+
ax.spines['left'].set_color('#2B2F30')
|
| 227 |
+
ax.spines['bottom'].set_color('#2B2F30')
|
| 228 |
+
for category, color in enumerate(["purple", "green", "red", "blue", "brown", "gray", "olive", "cyan", "orange", "pink"]):
|
| 229 |
+
xs = np.array(x)[df.Cluster == category]
|
| 230 |
+
ys = np.array(y)[df.Cluster == category]
|
| 231 |
+
plt.scatter(xs, ys, color=color, alpha=0.3)
|
| 232 |
+
avg_x = xs.mean()
|
| 233 |
+
avg_y = ys.mean()
|
| 234 |
+
plt.scatter(avg_x, avg_y, marker="x", color=color, s=100)
|
| 235 |
+
|
| 236 |
+
plt.title(
|
| 237 |
+
f"Clustering of Pinecone Index {pinecone_index_name}", fontsize=16, fontweight='bold', pad=20)
|
| 238 |
+
plt.suptitle(
|
| 239 |
+
f't-SNE (perplexity={perplexity} clusters={num_clusters})', y=0.92, fontsize=13)
|
| 240 |
+
plt.legend(loc='best', frameon=True)
|
| 241 |
+
plt.show()
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
@click.command(short_help='Fetches vectors from Pinecone specified by the vectors\' ids.')
|
| 245 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 246 |
+
@click.option('--region', help='Pinecone Index Region', default=default_region)
|
| 247 |
+
@click.option('--vector_ids', help='Vector ids to fetch')
|
| 248 |
+
@click.option('--pretty', is_flag=True, help='Pretty print output.')
|
| 249 |
+
@click.option('--namespace', help='Namespace within the index to search.')
|
| 250 |
+
@click.argument('pinecone_index_name')
|
| 251 |
+
def fetch(pinecone_index_name, apikey, region, vector_ids, namespace, pretty):
|
| 252 |
+
""" Fetch queries from Pinecone by vector_ids
|
| 253 |
+
|
| 254 |
+
\b
|
| 255 |
+
Example:
|
| 256 |
+
% ./pinecli.py fetch lpfactset --vector_ids="05b4509ee655aacb10bfbb6ba212c65c"
|
| 257 |
+
"""
|
| 258 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 259 |
+
parsed_ids = [x.strip() for x in vector_ids.split(",")]
|
| 260 |
+
fetch_response = index.fetch(ids=parsed_ids, namespace=namespace)
|
| 261 |
+
if pretty:
|
| 262 |
+
click.echo(fetch_response)
|
| 263 |
+
else:
|
| 264 |
+
click.echo(fetch_response)
|
| 265 |
+
|
| 266 |
+
|
| 267 |
+
@click.command(short_help='Extracts text from url arg, vectorizes w/ openai embedding api, and upserts to Pinecone.')
|
| 268 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 269 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 270 |
+
@click.option('--namespace', help='Pinecone index namespace', default='', show_default=True)
|
| 271 |
+
@click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
|
| 272 |
+
@click.argument('pinecone_index_name')
|
| 273 |
+
@click.argument('vector_literal')
|
| 274 |
+
def upsert(pinecone_index_name, apikey, region, vector_literal, namespace, debug):
|
| 275 |
+
"""
|
| 276 |
+
Upserts vectors into the index <PINECONE_INDEX_NAME> by using the <VECTOR_LITERAL> which is a string representation of a list of tuples.
|
| 277 |
+
Note the literal is quoted.
|
| 278 |
+
|
| 279 |
+
\b
|
| 280 |
+
Example:
|
| 281 |
+
% ./pinecli.py upsert upsertfile "[('vec1', [0.1, 0.2, 0.3], {'genre': 'drama'}), ('vec2', [0.2, 0.3, 0.4], {'genre': 'action'}),]" --debug
|
| 282 |
+
"""
|
| 283 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 284 |
+
if debug:
|
| 285 |
+
click.echo(f"Will upload vectors as {literal_eval(vector_literal)}")
|
| 286 |
+
resp = index.upsert(vectors=literal_eval(
|
| 287 |
+
vector_literal), namespace=namespace)
|
| 288 |
+
if debug:
|
| 289 |
+
click.echo(resp)
|
| 290 |
+
|
| 291 |
+
|
| 292 |
+
@click.command(short_help='Updates the index based on the given id passed in.')
|
| 293 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 294 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 295 |
+
@click.option('--namespace', help='Pinecone index namespace', default='', show_default=True)
|
| 296 |
+
@click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
|
| 297 |
+
@click.option("--metadata", help="The update to the metadata values.", default='', show_default=True)
|
| 298 |
+
@click.argument("id")
|
| 299 |
+
@click.argument('pinecone_index_name')
|
| 300 |
+
@click.argument('vector_literal')
|
| 301 |
+
def update(pinecone_index_name, apikey, region, id, vector_literal, metadata, namespace, debug):
|
| 302 |
+
""" Updates the index <PINECONE_INDEX_NAME> with id <ID> and vector values <VECTOR_LITERAL> """
|
| 303 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 304 |
+
if metadata:
|
| 305 |
+
resp = index.update(id=id, values=literal_eval(
|
| 306 |
+
vector_literal), set_metadata=literal_eval(metadata), namespace=namespace)
|
| 307 |
+
else:
|
| 308 |
+
resp = index.update(id=id, values=literal_eval(
|
| 309 |
+
vector_literal), namespace=namespace)
|
| 310 |
+
if debug:
|
| 311 |
+
click.echo(resp)
|
| 312 |
+
|
| 313 |
+
|
| 314 |
+
@click.command(short_help='Extracts text from url arg, vectorizes w/ openai embedding api, and upserts to Pinecone.')
|
| 315 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 316 |
+
@click.option('--openaiapikey', required=True, help='OpenAI API Key')
|
| 317 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 318 |
+
@click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
|
| 319 |
+
@click.option("--namespace", help='Namespace to store the generated vectors', default='', show_default=True)
|
| 320 |
+
@click.option("--metadata_content_key", help="The key used to store the page content in the metadata with.", show_default=True, default="content")
|
| 321 |
+
@click.option("--window", help='Number of sentences to combine in one embedding (vector).', show_default=True, default=10)
|
| 322 |
+
@click.option("--stride", help='Number of sentences to stride over to create overlap', show_default=True, default=4)
|
| 323 |
+
@click.option('--url')
|
| 324 |
+
@click.option('--pinecone_index_name')
|
| 325 |
+
def upsert_webpage(pinecone_index_name, apikey, namespace, openaiapikey, metadata_content_key, region, url, window, stride, debug):
|
| 326 |
+
""" Upserts vectors into the index <PINECONE_INDEX_NAME> using the openai embeddings api. You will need your api key for openai and specify it using --openapikey """
|
| 327 |
+
pinecone_index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 328 |
+
|
| 329 |
+
# req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
|
| 330 |
+
# html = urllib.request.urlopen(req).read()
|
| 331 |
+
html = requests.get(url).text
|
| 332 |
+
html = text_from_html(html)
|
| 333 |
+
nltk.download('punkt')
|
| 334 |
+
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
|
| 335 |
+
sentences = tokenizer.tokenize(html)
|
| 336 |
+
sentences = list(filter(None, sentences))
|
| 337 |
+
|
| 338 |
+
new_data = []
|
| 339 |
+
|
| 340 |
+
text = ''
|
| 341 |
+
for i in tqdm(range(0, len(sentences), stride)): # TODO: Use trange
|
| 342 |
+
i_end = min(len(sentences)-1, i+window)
|
| 343 |
+
if sentences[i] == sentences[i_end]:
|
| 344 |
+
continue
|
| 345 |
+
text = ' '.join(sentences[i:i_end]).strip()
|
| 346 |
+
# create the new merged dataset
|
| 347 |
+
if debug:
|
| 348 |
+
click.echo(f"Text is: {text}")
|
| 349 |
+
if text != "":
|
| 350 |
+
new_data.append(text)
|
| 351 |
+
new_data.append(sentences[-1])
|
| 352 |
+
new_data = list(filter(None, new_data))
|
| 353 |
+
if debug:
|
| 354 |
+
click.echo(*new_data, sep="\n")
|
| 355 |
+
|
| 356 |
+
batch_size = 10 # how many embeddings we create and insert at once
|
| 357 |
+
|
| 358 |
+
for i in tqdm(range(0, len(new_data), batch_size)):
|
| 359 |
+
i_end = min(len(new_data), i+batch_size)
|
| 360 |
+
meta_batch = new_data[i:i_end]
|
| 361 |
+
ids_batch = [hashlib.md5(x.encode('utf-8')).hexdigest()
|
| 362 |
+
for x in meta_batch]
|
| 363 |
+
res = get_openai_embedding(openaiapikey, meta_batch)
|
| 364 |
+
embeds = [record['embedding'] for record in res['data']]
|
| 365 |
+
meta_batch = [{metadata_content_key: x} for x in meta_batch]
|
| 366 |
+
to_upsert = list(zip(ids_batch, embeds, meta_batch))
|
| 367 |
+
rv = pinecone_index.upsert(vectors=to_upsert, namespace=namespace)
|
| 368 |
+
if debug:
|
| 369 |
+
click.echo(rv)
|
| 370 |
+
|
| 371 |
+
|
| 372 |
+
@click.command(short_help='Shows a preview of vectors in the <PINECONE_INDEX_NAME>')
|
| 373 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 374 |
+
@click.option('--region', help='Pinecone Index Region', default=default_region)
|
| 375 |
+
@click.option('--topk', '--numrows', 'topk', default=10, help='The number of rows to show')
|
| 376 |
+
@click.option('--random_dims', is_flag=True, help='Flag to have query vector dims be random. Default will be 0.0')
|
| 377 |
+
@click.option('--include-values', help='Should we return the vectors', show_default=True, default=True)
|
| 378 |
+
@click.option('--namespace', default="", help='Namespace to select results from')
|
| 379 |
+
@click.option('--include-meta', help='Whether to include the metadata values', default=False, show_default=True)
|
| 380 |
+
@click.option('--expand-meta', help='Whether to fully expand the metadata returned.', is_flag=True, show_default=True, default=False)
|
| 381 |
+
@click.option('--print-table', help='Display the output as a pretty table.', is_flag=True, show_default=True, default=False)
|
| 382 |
+
@click.argument('pinecone_index_name')
|
| 383 |
+
def head(pinecone_index_name, apikey, region, topk, random_dims, namespace, include_meta, expand_meta, include_values, print_table):
|
| 384 |
+
""" Shows a preview of vectors in the <PINECONE_INDEX_NAME> with optional numrows (default 10)
|
| 385 |
+
|
| 386 |
+
\b
|
| 387 |
+
Example 1:
|
| 388 |
+
|
| 389 |
+
% ./pinecli.py head lpfactset --include-meta=true --include-values=true
|
| 390 |
+
{'matches': [{'id': 'ae23d7574c19cea0b3479c93858a3ee3',
|
| 391 |
+
'metadata': {'content': 'Oded K. R&D Group Lead Why '
|
| 392 |
+
'Pinecone Fast, fresh, and filtered '
|
| 393 |
+
'Python client. Scalable Scale from zero '
|
| 394 |
+
'to billions of items, with no downtime '
|
| 395 |
+
'and minimal latency impact.'},
|
| 396 |
+
'score': 0.0,
|
| 397 |
+
'sparseValues': {},
|
| 398 |
+
'values': [0.010676071,
|
| 399 |
+
|
| 400 |
+
\b
|
| 401 |
+
Example 2: (printing results with a table)
|
| 402 |
+
|
| 403 |
+
% tim@yoda pinecone-cli % ./pinecli.py head pageuploadtest --include-values=True --include-meta=True --namespace=test --print-table --topk=3
|
| 404 |
+
🌲 pageuploadtest ns=(test) Index Results
|
| 405 |
+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
|
| 406 |
+
┃ ID ┃ NS ┃ Values ┃ Meta ┃ Score ┃
|
| 407 |
+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
|
| 408 |
+
│ 25d4d1444f9b2942440ce22e026c2a06 │ test │ -0.00443430152,-0.0156992543,0 │ {'content': 'Sign Up for Free or contact us Use Cases What can you do with vector search ? Once you │ 0.0 │
|
| 409 |
+
│ d0bdfbee942fadf531b1feea5b909217 │ test │ 0.0214423928,-0.0283056349,0.0 │ {'content': "Easy to use Get started on the free plan with an easy-to-use API or the Python client. │ 0.0 │
|
| 410 |
+
│ ae23d7574c19cea0b3479c93858a3ee3 │ test │ 0.0106426,-0.000842177891,-0.0 │ {'content': "Oded K. R&D Group Lead Why Pinecone Fast, fresh, and filtered vector search. F │ 0.0 │
|
| 411 |
+
└──────────────────────────────────┴──────┴────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘
|
| 412 |
+
"""
|
| 413 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 414 |
+
res = index.describe_index_stats()
|
| 415 |
+
dims = res['dimension']
|
| 416 |
+
if random_dims:
|
| 417 |
+
dims = [random.random() for _ in range(dims)]
|
| 418 |
+
else:
|
| 419 |
+
dims = [0.0 for _ in range(dims)]
|
| 420 |
+
resp = index.query(vector=dims, top_k=topk, namespace=namespace,
|
| 421 |
+
include_metadata=include_meta, include_values=include_values)
|
| 422 |
+
if print_table:
|
| 423 |
+
_print_table(resp, pinecone_index_name, namespace,
|
| 424 |
+
include_meta, include_values, expand_meta)
|
| 425 |
+
else:
|
| 426 |
+
click.echo(resp)
|
| 427 |
+
|
| 428 |
+
|
| 429 |
+
@click.command(short_help='Creates a Pinecone Index.')
|
| 430 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 431 |
+
@click.option('--region', help='Pinecone Index Region', default=default_region)
|
| 432 |
+
@click.option('--dims', help='Number of dimensions for this index', type=click.INT, required=True)
|
| 433 |
+
@click.option('--metric', help='Distance metric to use.', required=True, default='cosine')
|
| 434 |
+
@click.option('--pods', help='Number of pods', default=1, show_default=True, type=click.INT)
|
| 435 |
+
@click.option('--replicas', help='Number of replicas', default=1, show_default=True, type=click.INT)
|
| 436 |
+
@click.option('--shards', help='Number of shards', default=1, show_default=True, type=click.INT)
|
| 437 |
+
@click.option('--pod-type', help='Type of pods to create.', required=True)
|
| 438 |
+
@click.option('--source_collection', help='Source collection to create index from')
|
| 439 |
+
@click.argument('pinecone_index_name')
|
| 440 |
+
def create_index(pinecone_index_name, apikey, region, dims, metric, pods, replicas, shards, pod_type, source_collection):
|
| 441 |
+
""" Creates the Pinecone index named <PINECONE_INDEX_NAME> """
|
| 442 |
+
_pinecone_init(apikey, region)
|
| 443 |
+
resp = pinecone.create_index(pinecone_index_name, dimension=dims, metric=metric,
|
| 444 |
+
pods=pods, replicas=replicas, shards=shards, pod_type=pod_type)
|
| 445 |
+
click.echo(resp)
|
| 446 |
+
|
| 447 |
+
|
| 448 |
+
def chunks(iterable, batch_size=100):
|
| 449 |
+
it = iter(iterable)
|
| 450 |
+
chunk = tuple(itertools.islice(it, batch_size))
|
| 451 |
+
while chunk:
|
| 452 |
+
yield chunk
|
| 453 |
+
chunk = tuple(itertools.islice(it, batch_size))
|
| 454 |
+
|
| 455 |
+
|
| 456 |
+
def chunks_df(df):
|
| 457 |
+
for row in df.itertuples(index=False):
|
| 458 |
+
yield row
|
| 459 |
+
|
| 460 |
+
|
| 461 |
+
@click.command(short_help='Upserts a vector(s) with random dimensions into the specified vector.')
|
| 462 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 463 |
+
@click.option('--region', help='Pinecone Index Region', default=default_region)
|
| 464 |
+
@click.argument('pinecone_index_name')
|
| 465 |
+
@click.option('--num_vectors', '--num_rows', '--numrows', type=click.INT)
|
| 466 |
+
@click.option('--debug', is_flag=True, default=False, show_default=True)
|
| 467 |
+
@click.option('--num_vector_dims', type=click.INT)
|
| 468 |
+
def upsert_random(pinecone_index_name, apikey, region, num_vectors, num_vector_dims, debug):
|
| 469 |
+
""" Upserts random vectors and dimension values using num_vectors (rows) and num_vector_dims (number of dims per vector). IDs for the example vectors will be of the form \'id-{rownum}\' """
|
| 470 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 471 |
+
|
| 472 |
+
# Example generator that generates many (id, vector) pairs
|
| 473 |
+
example_data_generator = map(lambda i: (
|
| 474 |
+
f'id-{i}', [random.random() for _ in range(num_vector_dims)]), range(num_vectors))
|
| 475 |
+
|
| 476 |
+
# Upsert data with 100 vectors per upsert request
|
| 477 |
+
batch_size = 100
|
| 478 |
+
for ids_vectors_chunk in tqdm(chunks(example_data_generator, batch_size=batch_size), total=num_vectors/batch_size):
|
| 479 |
+
rv = index.upsert(vectors=ids_vectors_chunk)
|
| 480 |
+
if debug:
|
| 481 |
+
click.echo(rv)
|
| 482 |
+
|
| 483 |
+
|
| 484 |
+
@click.command(short_help='Upserts a file (csv) into the specified index.')
|
| 485 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 486 |
+
@click.option('--region', help='Pinecone Index Region', default=default_region)
|
| 487 |
+
@click.option('--batch_size', help='Number vectors to upload per batch', default=100, type=click.INT)
|
| 488 |
+
@click.option('--namespace', default='')
|
| 489 |
+
@click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
|
| 490 |
+
@click.argument('vector-file', )
|
| 491 |
+
@click.argument('pinecone_index_name')
|
| 492 |
+
@click.argument('colmap')
|
| 493 |
+
def upsert_file(pinecone_index_name, apikey, region, vector_file, batch_size, colmap, namespace, debug):
|
| 494 |
+
colmap = literal_eval(colmap)
|
| 495 |
+
if (('id' not in colmap) or ('vectors' not in colmap)):
|
| 496 |
+
click.echo(
|
| 497 |
+
"Missing 'id' or 'vectors' keys in mapping of CSV file. Check header definitions.")
|
| 498 |
+
sys.exit(-1)
|
| 499 |
+
|
| 500 |
+
reverse_col_map = dict(reversed(list(colmap.items())))
|
| 501 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 502 |
+
|
| 503 |
+
def convert(item):
|
| 504 |
+
item = item.strip() # remove spaces at the end
|
| 505 |
+
item = item[1:-1] # remove `[ ]`
|
| 506 |
+
return list(map(float, item.split(',')))
|
| 507 |
+
|
| 508 |
+
usecols = [reverse_col_map['id'], reverse_col_map['vectors']]
|
| 509 |
+
converters = {reverse_col_map['vectors']: convert}
|
| 510 |
+
|
| 511 |
+
if 'metadata' in colmap:
|
| 512 |
+
csv_meta_col_name = reverse_col_map['metadata']
|
| 513 |
+
usecols.append(reverse_col_map['metadata'])
|
| 514 |
+
converters[csv_meta_col_name] = literal_eval
|
| 515 |
+
|
| 516 |
+
for chunk in pd.read_csv(vector_file, chunksize=batch_size, index_col=False, usecols=usecols, converters=converters):
|
| 517 |
+
v = chunk.to_records(index=False).tolist()
|
| 518 |
+
rv = index.upsert(vectors=v, namespace=namespace)
|
| 519 |
+
if debug:
|
| 520 |
+
click.echo(rv)
|
| 521 |
+
|
| 522 |
+
|
| 523 |
+
@click.command(short_help='Lists the indexes for your api key.')
|
| 524 |
+
@click.option('--apikey', help='API Key')
|
| 525 |
+
@click.argument('region', default=default_region)
|
| 526 |
+
def list_indexes(apikey, region):
|
| 527 |
+
""" List all Pinecone indexes for the given api key. """
|
| 528 |
+
_pinecone_init(apikey, region)
|
| 529 |
+
res = pinecone.list_indexes()
|
| 530 |
+
click.echo('\n'.join(res))
|
| 531 |
+
|
| 532 |
+
|
| 533 |
+
@click.command(short_help='Describes an index.')
|
| 534 |
+
@click.option('--apikey')
|
| 535 |
+
@click.argument('pinecone_index_name', required=True)
|
| 536 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 537 |
+
def describe_index(apikey, pinecone_index_name, region):
|
| 538 |
+
""" Describe a Pinecone index with given index_name. """
|
| 539 |
+
_pinecone_init(apikey, region, pinecone_index_name)
|
| 540 |
+
desc = pinecone.describe_index(pinecone_index_name)
|
| 541 |
+
click.echo("\n".join([f"Name: {desc.name}", f"Dimensions: {int(desc.dimension)}",
|
| 542 |
+
f"Metric: {desc.metric}", f"Pods: {desc.pods}", f"PodType: {desc.pod_type}", f"Shards: {desc.shards}",
|
| 543 |
+
f"Replicas: {desc.replicas}", f"Ready: {desc.status['ready']}", f"State: {desc.status['state']}",
|
| 544 |
+
f"Metaconfig: {desc.metadata_config}", f"Sourcecollection: {desc.source_collection}"]))
|
| 545 |
+
|
| 546 |
+
|
| 547 |
+
@click.command(short_help='Configures the given index to have a pod type.')
|
| 548 |
+
@click.option('--apikey', help='Pinecone API Key')
|
| 549 |
+
@click.argument('pinecone_index_name', required=True)
|
| 550 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 551 |
+
@click.option('--pod_type', required=True, help='Type of pod to use')
|
| 552 |
+
def configure_index_pod_type(apikey, pinecone_index_name, region, pod_type):
|
| 553 |
+
""" Configure the pod type for a given index_name. """
|
| 554 |
+
_pinecone_init(apikey, region)
|
| 555 |
+
pinecone.configure_index(pinecone_index_name, pod_type=pod_type)
|
| 556 |
+
|
| 557 |
+
|
| 558 |
+
@click.command(short_help='Configures the number of replicas for a given index.')
|
| 559 |
+
@click.option('--apikey')
|
| 560 |
+
@click.argument('pinecone_index_name', required=True)
|
| 561 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 562 |
+
@click.option('--num_replicas', required=True, help='Number of replicas to use.')
|
| 563 |
+
def configure_index_replicas(apikey, pinecone_index_name, region, num_replicas):
|
| 564 |
+
""" Configure the number of replicas for an index. """
|
| 565 |
+
_pinecone_init(apikey, region)
|
| 566 |
+
pinecone.configure_index(pinecone_index_name, replicas=num_replicas)
|
| 567 |
+
|
| 568 |
+
|
| 569 |
+
@click.command(short_help='Creates a Pinecone collection from the argument \'source_index\'')
|
| 570 |
+
@click.option('--apikey')
|
| 571 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 572 |
+
@click.option('--collection_name', help='The name of the collection to create.', required=True)
|
| 573 |
+
@click.option('--source_index', help='The name index to create collection from.', required=True)
|
| 574 |
+
def create_collection(apikey, region, collection_name, source_index):
|
| 575 |
+
""" Create a Pinecone collection with the given collection_name and source_index. """
|
| 576 |
+
_pinecone_init(apikey, region)
|
| 577 |
+
pinecone.create_collection(collection_name, source_index)
|
| 578 |
+
|
| 579 |
+
|
| 580 |
+
@click.command(short_help='Prints out index stats to stdout.')
|
| 581 |
+
@click.option('--apikey')
|
| 582 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 583 |
+
@click.argument('pinecone_index_name', required=True)
|
| 584 |
+
def describe_index_stats(apikey, region, pinecone_index_name):
|
| 585 |
+
""" Show the stats for index with name <PINECONE_INDEX_NAME>. Note that if the index has several namespaces, those will be broken out.
|
| 586 |
+
|
| 587 |
+
\b
|
| 588 |
+
Example:
|
| 589 |
+
% ./pinecli.py describe-index-stats lpfactset
|
| 590 |
+
"""
|
| 591 |
+
index = _pinecone_init(apikey, region, pinecone_index_name)
|
| 592 |
+
res = index.describe_index_stats()
|
| 593 |
+
console = Console()
|
| 594 |
+
click.echo(f"Dimensions: {res['dimension']}")
|
| 595 |
+
click.echo(f"Vectors: {res['total_vector_count']}")
|
| 596 |
+
click.echo(f"Index_Fullness: {res['index_fullness']}")
|
| 597 |
+
ns_data = res['namespaces']
|
| 598 |
+
console.print("Namespace data:", style="b")
|
| 599 |
+
for ns in ns_data.keys():
|
| 600 |
+
click.echo(f"\t{ns}: {ns_data[ns]['vector_count']}")
|
| 601 |
+
|
| 602 |
+
|
| 603 |
+
@click.command(short_help='Lists collections for the given apikey.')
|
| 604 |
+
@click.option('--apikey')
|
| 605 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 606 |
+
def list_collections(apikey, region):
|
| 607 |
+
""" List Pinecone collections with the given api key """
|
| 608 |
+
_pinecone_init(apikey, region)
|
| 609 |
+
res = pinecone.list_collections()
|
| 610 |
+
click.echo(*res, sep='\n')
|
| 611 |
+
|
| 612 |
+
|
| 613 |
+
@click.command(short_help='Describes a collection.')
|
| 614 |
+
@click.option('--apikey')
|
| 615 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 616 |
+
@click.argument('collection_name', required=True)
|
| 617 |
+
def describe_collection(apikey, region, collection_name):
|
| 618 |
+
""" Describe the collection described by <COLLECTION_NAME> """
|
| 619 |
+
_pinecone_init(apikey, region)
|
| 620 |
+
desc = pinecone.describe_collection(collection_name)
|
| 621 |
+
click.echo("\n".join([f"Name: {desc.name}", f"Dimensions: {int(desc.dimension)}",
|
| 622 |
+
f"Vectors: {int(desc.vector_count)}", f"Status: {desc.status}", f"Size: {desc.size}"]))
|
| 623 |
+
|
| 624 |
+
|
| 625 |
+
@click.command(short_help="Deletes a collection.")
|
| 626 |
+
@click.option('--apikey')
|
| 627 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 628 |
+
@click.option('--collection_name', help='The name of the collection to create.', required=True)
|
| 629 |
+
def delete_collection(apikey, region, collection_name):
|
| 630 |
+
""" Delete a collection with the given collection_name """
|
| 631 |
+
_pinecone_init(apikey, region)
|
| 632 |
+
pinecone.delete_collection(collection_name)
|
| 633 |
+
|
| 634 |
+
@click.command(short_help='Deletes an index. You will be prompted to confirm.')
|
| 635 |
+
@click.option('--apikey')
|
| 636 |
+
@click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
|
| 637 |
+
@click.argument('pinecone_index', required=True)
|
| 638 |
+
def delete_index(apikey, region, pinecone_index):
|
| 639 |
+
""" Delete an index with the given pinecone_index name """
|
| 640 |
+
_pinecone_init(apikey, region)
|
| 641 |
+
value = click.prompt('Type name of index backwards to confirm: ')
|
| 642 |
+
if value == pinecone_index[::-1]:
|
| 643 |
+
pinecone.delete_index(pinecone_index)
|
| 644 |
+
else:
|
| 645 |
+
click.echo("Index not deleted: reversed index name does not match.")
|
| 646 |
+
|
| 647 |
+
|
| 648 |
+
cli.add_command(query)
|
| 649 |
+
cli.add_command(upsert)
|
| 650 |
+
cli.add_command(upsert_file)
|
| 651 |
+
cli.add_command(upsert_random)
|
| 652 |
+
cli.add_command(update)
|
| 653 |
+
cli.add_command(list_indexes)
|
| 654 |
+
cli.add_command(delete_index)
|
| 655 |
+
cli.add_command(create_index)
|
| 656 |
+
cli.add_command(describe_index)
|
| 657 |
+
cli.add_command(upsert_webpage)
|
| 658 |
+
cli.add_command(configure_index_pod_type)
|
| 659 |
+
cli.add_command(configure_index_replicas)
|
| 660 |
+
cli.add_command(create_collection)
|
| 661 |
+
cli.add_command(list_collections)
|
| 662 |
+
cli.add_command(describe_collection)
|
| 663 |
+
cli.add_command(delete_collection)
|
| 664 |
+
cli.add_command(describe_index_stats)
|
| 665 |
+
cli.add_command(fetch)
|
| 666 |
+
cli.add_command(head)
|
| 667 |
+
cli.add_command(version)
|
| 668 |
+
|
| 669 |
+
if __name__ == "__main__":
|
| 670 |
+
cli()
|
requirements.txt
ADDED
|
File without changes
|
setup.py
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from setuptools import setup
|
| 2 |
+
|
| 3 |
+
setup(
|
| 4 |
+
name='pinecone-cli',
|
| 5 |
+
description='pinecone-cli is a command-line client for interacting with the pinecone vector embedding database.',
|
| 6 |
+
version='0.1.1',
|
| 7 |
+
url='https://github.com/tullytim/pinecone-cli',
|
| 8 |
+
author='Tim Tully',
|
| 9 |
+
author_email='tim@menlovc.com',
|
| 10 |
+
license='MIT',
|
| 11 |
+
keywords='pinecone vector vectors embeddings database transformers models',
|
| 12 |
+
python_requires='>=3',
|
| 13 |
+
py_modules=['pinecli'],
|
| 14 |
+
install_requires=[
|
| 15 |
+
'Click', 'pandas', 'numpy', 'openai', 'pinecone-client', 'matplotlib', \
|
| 16 |
+
'scikit-learn', 'beautifulsoup4', 'nltk', 'sklearn', 'rich'
|
| 17 |
+
],
|
| 18 |
+
entry_points={
|
| 19 |
+
'console_scripts': [
|
| 20 |
+
'pinecli = pinecli:cli',
|
| 21 |
+
],
|
| 22 |
+
},
|
| 23 |
+
)
|
sitemap_crawler.py
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import requests
|
| 2 |
+
from bs4 import BeautifulSoup as Soup
|
| 3 |
+
import pandas as pd
|
| 4 |
+
import hashlib
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
# Pass the headers you want to retrieve from the xml such as ["loc", "lastmod"]
|
| 8 |
+
def parse_sitemap( url,headers):
|
| 9 |
+
resp = requests.get(url)
|
| 10 |
+
# we didn't get a valid response, bail
|
| 11 |
+
if (200 != resp.status_code):
|
| 12 |
+
return False
|
| 13 |
+
|
| 14 |
+
# BeautifulSoup to parse the document
|
| 15 |
+
soup = Soup(resp.content, "html.parser")
|
| 16 |
+
|
| 17 |
+
# find all the <url> tags in the document
|
| 18 |
+
urls = soup.findAll('url')
|
| 19 |
+
sitemaps = soup.findAll('sitemap')
|
| 20 |
+
new_list = [] + headers
|
| 21 |
+
panda_out_total = pd.DataFrame([], columns=new_list)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
if not urls and not sitemaps:
|
| 25 |
+
return False
|
| 26 |
+
|
| 27 |
+
# Recursive call to the the function if sitemap contains sitemaps
|
| 28 |
+
if sitemaps:
|
| 29 |
+
for u in sitemaps:
|
| 30 |
+
test = u.find('loc').string
|
| 31 |
+
panda_recursive = parse_sitemap(test, headers)
|
| 32 |
+
panda_out_total = pd.concat([panda_out_total, panda_recursive], ignore_index=True)
|
| 33 |
+
|
| 34 |
+
out = []
|
| 35 |
+
|
| 36 |
+
# Extract the keys we want
|
| 37 |
+
for u in urls:
|
| 38 |
+
values = []
|
| 39 |
+
for head in headers:
|
| 40 |
+
loc = None
|
| 41 |
+
loc = u.find(head)
|
| 42 |
+
if not loc:
|
| 43 |
+
loc = "None"
|
| 44 |
+
else:
|
| 45 |
+
loc = loc.string
|
| 46 |
+
values.append(loc)
|
| 47 |
+
out.append(values)
|
| 48 |
+
|
| 49 |
+
# Create a dataframe
|
| 50 |
+
panda_out = pd.DataFrame(out, columns= new_list)
|
| 51 |
+
|
| 52 |
+
# If recursive then merge recursive dataframe
|
| 53 |
+
if not panda_out_total.empty:
|
| 54 |
+
panda_out = pd.concat([panda_out, panda_out_total], ignore_index=True)
|
| 55 |
+
|
| 56 |
+
#returns the dataframe
|
| 57 |
+
return panda_out
|
| 58 |
+
|
| 59 |
+
def main():
|
| 60 |
+
print(parse_sitemap("https://menlovc.com/sitemap_index.xml", ["loc"]))
|
| 61 |
+
#parse_sitemap("https://menlovc.com/sitemap_index.xml", ["loc"]).to_csv("menlo_links.csv")
|
| 62 |
+
|
| 63 |
+
if __name__ == "__main__":
|
| 64 |
+
main()
|
| 65 |
+
|