nicrso commited on
Commit
99fef8e
·
1 Parent(s): b7f07ab

Upload 6 files

Browse files
Files changed (6) hide show
  1. README.md +285 -13
  2. app.py +248 -0
  3. pinecli.py +670 -0
  4. requirements.txt +0 -0
  5. setup.py +23 -0
  6. sitemap_crawler.py +65 -0
README.md CHANGED
@@ -1,13 +1,285 @@
1
- ---
2
- title: ChatGVC
3
- emoji: 🐠
4
- colorFrom: pink
5
- colorTo: blue
6
- sdk: gradio
7
- sdk_version: 3.19.1
8
- app_file: app.py
9
- pinned: false
10
- license: mit
11
- ---
12
-
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # pinecone-cli
2
+ Pinecode-cli is a command-line interface for control and data plane interfacing with [Pinecone](https://pinecone.io).
3
+
4
+ In addition to ALL of the Pinecone "actions/verbs", Pinecone-cli has several additional features that make Pinecone even more powerful including:
5
+ * Upload vectors from CSV files
6
+ * Upload embeddings of text from a given website URL. Embeddings generated by OpenAI embeddings API.
7
+ * New "head" command to peak into a given index, similar to "head" in linux/unix.
8
+
9
+ # Install
10
+ Feel free to use the tool directly from source here, or just
11
+ ```console
12
+ pip install pinecone-cli
13
+ ```
14
+ Pypi here: (https://pypi.org/project/pinecone-cli/) Note that the examples below assume you are using the source and have pulled the right deps. If you install from ```pip``` it gets installed as ```pinecli``` in your PATH.
15
+
16
+ # Usage
17
+ First off, let's simplify usage by setting the PINECONE_API_KEY env variable so we don't have to pass it every time in the commandline:
18
+ ```console
19
+ % export PINECONE_API_KEY=1234-4567-abc
20
+ ```
21
+ Otherwise you wind up having to pass the key as so:
22
+ ```console
23
+ % ./pinecli.py query --apikey=1234 ....
24
+ ```
25
+
26
+
27
+ The pattern for using the tool is to invoke 'pinecli' and then use a command. The list of commands appears with --help
28
+
29
+ ```console
30
+ % ./pinecli.py --help
31
+ Usage: pinecli.py [OPTIONS] COMMAND [ARGS]...
32
+
33
+ A command line interface for working with Pinecone.
34
+
35
+ Options:
36
+ --help Show this message and exit.
37
+
38
+ Commands:
39
+ configure-index-pod-type Configures the given index to have a pod type.
40
+ configure-index-replicas Configures the number of replicas for a given
41
+ index.
42
+ create-collection Creates a Pinecone collection from the argument
43
+ 'source_index'
44
+ create-index Creates a Pinecone Index.
45
+ delete-collection Deletes a collection.
46
+ delete-index Deletes an index. You will be prompted to
47
+ confirm.
48
+ describe-collection Describes a collection.
49
+ describe-index Describes an index.
50
+ describe-index-stats Prints out index stats to stdout.
51
+ fetch Fetches vectors from Pinecone specified by the
52
+ vectors' ids.
53
+ head Shows a preview of vectors in the
54
+ <PINECONE_INDEX_NAME>
55
+ list-collections Lists collections for the given apikey.
56
+ list-indexes Lists the indexes for your api key.
57
+ query Queries Pinecone with a given vector.
58
+ update Updates the index based on the given id passed in.
59
+ upsert Extracts text from url arg, vectorizes w/ openai
60
+ embedding api, and upserts to Pinecone.
61
+ upsert-file Upserts a file (csv) into the specified index.
62
+ upsert-random Upserts a vector(s) with random dimensions into
63
+ the specified vector.
64
+ upsert-webpage Extracts text from url arg, vectorizes w/ openai
65
+ embedding api, and upserts to Pinecone.
66
+ ```
67
+
68
+ # Commands With Examples
69
+ Before you can use Pinecone an index is required. We can now do this on the commandline rather than in the UI: (Note not all of the cmdline options are required, they're shown here to demonstrate functionality and control)
70
+ ```console
71
+ % ./pinecli.py create-index myindex --dims=1536 --metric=cosine --pods=2 --replicas=2 --shards=1 --pod-"type=p2.x1"
72
+ ```
73
+
74
+ Note that for any command, if you want an exhasuive description of cmdline options, simply do something similar to the below, where "create-index" is replaced by one of the commands:
75
+ ```console
76
+ % ./pinecli.py create-index --help
77
+ Usage: pinecli.py create-index [OPTIONS] PINECONE_INDEX_NAME
78
+
79
+ Creates the Pinecone index named <PINECONE_INDEX_NAME>
80
+
81
+ Options:
82
+ --apikey TEXT Pinecone API Key
83
+ --region TEXT Pinecone Index Region
84
+ --dims INTEGER Number of dimensions for this index [required]
85
+ --metric TEXT Distance metric to use. [required]
86
+ --pods INTEGER Number of pods [default: 1]
87
+ --replicas INTEGER Number of replicas [default: 1]
88
+ --shards INTEGER Number of shards [default: 1]
89
+ --pod-type TEXT Type of pods to create. [required]
90
+ --source_collection TEXT Source collection to create index from
91
+ --help Show this message and exit.
92
+ ```
93
+
94
+ Let's try some commands showing two missing features I'd love to have had over the last year: a "head" command and a quick "stats" command:
95
+
96
+ ## Index Stats Including Number of Vectors
97
+ ``` console
98
+ % ./pinecli.py describe-index-stats myindex
99
+ Dimensions: 1536
100
+ Vectors: 7745
101
+ Index_Fullness: 0.0
102
+ Namespace data:
103
+ : 7745
104
+ ```
105
+
106
+ ## Head command to preview vectors
107
+ ``` console
108
+ % ./pinecli.py head kids-facenet
109
+ {'matches': [{'id': 'bubba_50.jpg.vec',
110
+ 'metadata': {},
111
+ 'score': 12.182938,
112
+ 'values': [-0.016061664,
113
+ -0.4495437,
114
+ -0.034082577,
115
+ .....
116
+ ```
117
+
118
+ Now, let's query some nonsensical data from the index named 'upsertfile'
119
+
120
+ ## Inserting a vector directly
121
+ *Note the double quites around the vector*
122
+ ```console
123
+ % ./pinecli.py query myindex "[1.2, 1.0, 3.0]" --print-table --include-meta=True
124
+ 🌲 upsertfile ns=() Index Results
125
+ ┏━━━━━━┳━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
126
+ ┃ ID ┃ NS ┃ Values ┃ Meta ┃ Score ┃
127
+ ┡━━━━━━╇━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
128
+ │ vec1 │ │ 0.1,0.2,0.3 │ {'genre': 'drama'} │ 0.9640127 │
129
+ │ vec2 │ │ 0.2,0.3,0.4 │ {'genre': 'action'} │ 0.9552943 │
130
+ │ abc │ │ 0.23223,-1.333,0.2222222 │ {'foo': 'bar'} │ -0.083585836 │
131
+ │ ghi │ │ 0.23223,-1.333,0.2222222 │ {'bar': 'baz'} │ -0.083585836 │
132
+ └──────┴────┴──────────────────────────┴─────────────────────┴──────────────┘
133
+ ```
134
+
135
+ Markdown of course does a great job of mangling great terminal output so here's a screenshot from using ```---print-table```:
136
+ ![alt](https://github.com/tullytim/pinecone-cli/blob/main/head-print.png?raw=true)
137
+
138
+
139
+ You can of course not output the pretty table by removing ```--print-table```:
140
+ ```console
141
+ % ./pinecli.py query myindex "[1.2, 1.0, 3.0]" --include-meta=True
142
+ {'matches': [{'id': 'vec1',
143
+ 'metadata': {'genre': 'drama'},
144
+ 'score': 0.9640127,
145
+ 'values': [0.1, 0.2, 0.3]},
146
+ ...
147
+ ```
148
+
149
+ ## Upsert Vectors in Command Line Manually
150
+ Following the Pinecone vector format of the tuple formatted as:
151
+ ```python
152
+ ('vectorid', [vecdim1, vecdim2, vecdim3], {'metakey':'metaval'})
153
+ ```
154
+ You can pass this in as a comma separated list of vectors on the command line:
155
+ ```console
156
+ ./pinecli.py upsert myindex "[('vec1', [0.1, 0.2, 0.3], {'genre': 'drama'}), ('vec2', [0.2, 0.3, 0.4], {'foo': 'bar'}),]"
157
+ ```
158
+
159
+ ## Upsert CSV file
160
+ Upserting a csv file is trivial. Simply create your csv file with any headings you have, but there must be at least a labeled id column and a labeled vector column for the vectors. Here's an example of a CSV file that is clearly a DataFrame dump due to the index column on the left which works great w/ pinecone-cli:
161
+ ```console
162
+ index,my_id_column,my_vectors_column,Metadata
163
+ 1,abc,"[0.23223, -1.333, 0.2222222]",{'foo':'bar'}
164
+ 2,ghi,"[0.23223, -1.333, 0.2222222]",{'bar':'baz'}
165
+ ```
166
+ The name of those columns in the header row can be arbitrary or you can name then "id", "vectors" and "metadata" which is our default assumption. If you have custom column names and don't want to change them, just pass in the ```--colmap``` argument which takes in a python dictionary mapping "id" and "vectors" to the naming you have in your csv. For example:
167
+ ```"{"id":"my_id_column", "vectors":"my_vectors_column"}```
168
+
169
+ Note that as in other CSV file for Dataframes, we need an index column as in the example above.
170
+ Here's an example using the CSV headers and format above with the correct colmap argument:
171
+ ```console
172
+ % ./pinecli.py upsert-file embeddings.csv myindex "{'id':'my_id_column', 'vectors':'my_vectors_column'}"
173
+ ```
174
+
175
+ ### More on CSV Formatting
176
+ For now you will need to manually provide an index column (we are using dataframes under the hood.)
177
+
178
+
179
+ ## Upserting Vector Embeddings of Webpage Text!
180
+ pinecone-cli was built to make using Pinecone extremely easy and fast. We have integrated [OpenAI](https://openai.com/) (others coming) - using its [embedding APIs](https://platform.openai.com/docs/guides/embeddings) to fetch embeddings. We then upload them into your index for you, making uploading embeddings of an entire website's text - trivial.
181
+ ```console
182
+ % ./pinecli.py upsert-webpage https://menlovc.com lpfactset --openaiapikey=12345-9876-abcdef
183
+ [nltk_data] Downloading package punkt to /Users/tim/nltk_data...
184
+ [nltk_data] Package punkt is already up-to-date!
185
+ 100%|████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 61680.94it/s]
186
+ ['About Us Our Promise Focus Areas Consumer Cloud Infrastructure Cybersecurity Fintech Healthcare SaaS Supply Chain and Automation Team Portfolio Perspective When we invest, we’re invested. Our promise to founders Building a business is a team sport. As investors, we don’t just sit on the sidelines but do whatever it takes to help our teams win. About Us The founders we back don’t limit themselves to what is, but relentlessly pursue what could be. We invest in transformative technology companies that are changing the way we live and work. Portfolio Menlo Labs starts companies. We work shoulder-to-shoulder
187
+ ....
188
+ 100%|█████████████████████████| 1/1 [00:00<00:00, 1.11it/s]
189
+ ```
190
+
191
+ ## Upsert Random Vectors
192
+ One of the more useful things we can do with pinecone-cli is insert random vectors, primarily for testing. Often we will create our index and the length of the vector will be 1,536 dimensions, for example. Instead of writing a bunch of code to go suddenly create those vectors somehow, we can use pinecone-cli to start generating vectors and upserting them:
193
+ ```console
194
+ % ./pinecli.py upsert-random upsertfile --num_vector_dims=1536 --num_vectors=10 --debug
195
+ upserted_count: 10
196
+ 1it [00:00, 4.36it/s]
197
+ ```
198
+ The example above inserts 10 vectors that each have 1,536 random vectors in them. Note that the ```id``` for each vector is simply ```f'id-{i}'``` where is is the ith row (vector) inserted.
199
+
200
+ ## Query Vectors
201
+ Querying can be done in two ways on the cmdline - pass in an actual vector string literal, or ask Pinecone to query randomly (maybe you want to just look at them or look at a TSNE). In the example below, the last argument (required) is either the string 'random' or an actual vector such as '[0.0, 1.0, 3.14569]'. Let's try random:
202
+ ```console
203
+ % ./pinecli.py query myindex random
204
+ ```
205
+
206
+ You can also plot a TSNE plot to view clustering of your vectors by using the ```-show-tsne=True``` flag. Note that this will pop up the plt plot by default.
207
+
208
+ ```console
209
+ % ./pinecli.py query lpfactset random --show-tsne=true --topk=2500 --num-clusters=4
210
+ ```
211
+
212
+ ![alt](https://github.com/tullytim/pinecone-cli/blob/main/tsne.png?raw=true)
213
+
214
+ ## Fetching Vectors:
215
+ Fetching is simple - just pass in the vector id(s) of the vectors you're looking for as a comma separated list:
216
+ ```console
217
+ % ./pinecli.py fetch myindex --vector_ids="05b4509ee655aacb10bfbb6ba212c65c,c626975ec096b9108f158a56a59b2fd6"
218
+
219
+ {'namespace': '',
220
+ 'vectors': {'05b4509ee655aacb10bfbb6ba212c65c': {'id': '05b4509ee655aacb10bfbb6ba212c65c',
221
+ 'metadata': {'content': 'Chime '
222
+ 'Scholar '
223
+ 'spotlight: '
224
+ ```
225
+
226
+ ## Updating Vectors
227
+ Updating vectors is simple - pass the id of the vector and the updating vector as below:
228
+ ```console
229
+ % ./pinecli.py update "id-9" myindex "[0.0, 1.0, 3.0]"
230
+ ```
231
+ ## List Operations
232
+ pinecone-cli has all of the necessary 'list' operations as shown below:
233
+
234
+ ### List Indexes
235
+ This gives you a list of all indexes under your api key:
236
+ ```console
237
+ % ./pinecli.py list-indexes
238
+ cli
239
+ cli2
240
+ cli3
241
+ drivertest
242
+ ```
243
+ ### List Collections
244
+ This obviously lists the collections you've created:
245
+ ```console
246
+ % ./pinecli.py list-collections
247
+ cli
248
+ cli2
249
+ cli3
250
+ drivertest
251
+ ```
252
+
253
+ ### Other Meta Operations
254
+ We showed the "describe-index-stats" command at the top of this page. There is also "describe-index" which provides the following:
255
+ ```console
256
+ % ./pinecli.py describe-index lpfactset
257
+ Name: lpfactset
258
+ Dimensions: 1536
259
+ Metric: cosine
260
+ Pods: 1
261
+ PodType: p2.x1
262
+ Shards: 1
263
+ Replicas: 1
264
+ Ready: True
265
+ State: Ready
266
+ Metaconfig: None
267
+ Sourcecollection:
268
+ ```
269
+
270
+ ### Describe Collections
271
+ ```console
272
+ % ./pinecli.py describe-collection testcoll
273
+ Name: testcoll
274
+ Dimensions: 1536
275
+ Vectors: 124
276
+ Status: Ready
277
+ Size: 3917544
278
+ ```
279
+
280
+ ## Deleting Indexes
281
+ Deleting an index is straightforward. To prevent catastrophic accidents, you'll be prompted to type in the name of the index backwards:
282
+ ```console
283
+ ./pinecli.py delete-index myindex2
284
+ Type name of index backwards to confirm: : 2xedniym
285
+ ```
app.py ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+
3
+ import plotly.express as px
4
+ import gradio as gr
5
+ import pandas as pd
6
+ from collections import defaultdict
7
+
8
+
9
+ from time import sleep
10
+ import openai
11
+
12
+ import json
13
+
14
+ import logging
15
+ from dotenv import load_dotenv
16
+ import pinecone
17
+
18
+ default_region = 'us-west1-gcp'
19
+ openai_embed_model = "text-embedding-ada-002"
20
+
21
+ def get_data(num_names_per_category=2, num_categories=3):
22
+
23
+ categories = ['Text', 'Bio', 'Audio', 'Legal', 'Chatbot/Conversational AI', 'Summarization', 'MLOps/Platform', 'Code', 'Avatars', 'Image', 'Semantic Search', 'Game', 'Video', 'Data', 'Sentiment Analysis']
24
+ num_categories = min(num_categories, len((categories)))
25
+
26
+ data = query_pinecone(categories, num_categories, num_names_per_category)
27
+
28
+ return data
29
+
30
+ def get_openai_embedding(apikey, data):
31
+ """ Fetch an embedding w/ given data """
32
+ openai.api_key = apikey
33
+ try:
34
+ res = openai.Embedding.create(input=data, engine=openai_embed_model)
35
+ except:
36
+ done = False
37
+ while not done:
38
+ sleep(5)
39
+ try:
40
+ res = openai.Embedding.create(
41
+ input=data, engine=openai_embed_model)
42
+ print('retrying')
43
+ done = True
44
+ except Exception as e:
45
+ logging.exception(e)
46
+ return res
47
+
48
+ def query_pinecone(categories, n_categories=3, top_k=4):
49
+ load_dotenv()
50
+
51
+ pinecone.init(
52
+ api_key=os.getenv('PINECONE_API_KEY'),
53
+ environment=os.getenv('PINECONE_ENVIRONMENT'),
54
+ )
55
+ index = pinecone.Index("vc-content-oracle-big")
56
+
57
+ out = defaultdict(list)
58
+
59
+ companies = set()
60
+
61
+ # loop thru all the categories
62
+ for category in categories[:n_categories]:
63
+
64
+ #get vector for each category
65
+ vector = get_openai_embedding(os.getenv('OPENAI_API_KEY'), category)['data'][0]['embedding']
66
+
67
+ res = index.query(
68
+ vector=vector,
69
+ top_k=top_k,
70
+ include_metadata=True,
71
+ include_values=False,
72
+ )
73
+
74
+ for match in res['matches']:
75
+ if 'Organization Name' not in match['metadata'] or 'Website' not in match['metadata']:
76
+ print('no Organization Name')
77
+ continue
78
+
79
+ #check if the match is already in the dictionary
80
+ company_link = '<a href="' + match['metadata']['Website'] + '" style="cursor: pointer; color:white; font-color:white " target="_blank" rel="noopener noreferrer">' + match['metadata']['Organization Name'] + '</a>'
81
+
82
+ print(out[category], company_link)
83
+
84
+ contains = False
85
+ if match['metadata']['Organization Name'] in companies:
86
+ contains = True
87
+
88
+ if not contains:
89
+ out[category].append(company_link)
90
+ else:
91
+ out[category].append(None)
92
+
93
+ companies.add(match['metadata']['Organization Name'])
94
+
95
+ return out
96
+
97
+ """
98
+ returns { 'matches': [{
99
+ 'id': 'A',
100
+ 'metadata': {
101
+ 'Organization Name': '',
102
+ 'Organization Name URL': '',
103
+ 'Last Funding Amount': '',
104
+ 'Last Funding Amount Currency': '',
105
+ 'Last Funding Amount Currency (in USD)': '',
106
+ 'Website': '',
107
+ 'Total Funding Amount': '',
108
+ 'Total Funding Amount Currency': '',
109
+ 'Total Funding Amount Currency (in USD)': '',
110
+ 'Last Funding Date': '',
111
+ 'Founded Date': '',
112
+ 'Founded Date Precision': '',
113
+ 'Last Funding Type': '',
114
+ 'Number of Employees': '',
115
+ 'Full Description': '',
116
+ 'Top 5 Investors': '',
117
+ },
118
+ 'score': 0.01,
119
+ 'values':[]
120
+ }]}
121
+ """
122
+
123
+ def make_plot(category, num_categories, num_names_per_category):
124
+
125
+ num_categories = int(num_categories)
126
+ num_names_per_category = int(num_names_per_category)
127
+
128
+ data = get_data(num_names_per_category, num_categories)
129
+ names = []
130
+ parents = []
131
+
132
+ for category, names_list in data.items():
133
+ names.extend(names_list)
134
+ parents.extend([category] * len(names_list))
135
+
136
+ unique_parents = list(set(parents))
137
+ names += unique_parents
138
+ parents += [''] * len(unique_parents)
139
+
140
+ fig = px.treemap(
141
+ names=names,
142
+ parents=parents,
143
+
144
+ )
145
+
146
+ fig.update_traces(root_color="lightgrey")
147
+ fig.update_traces(pathbar_textfont_color="white")
148
+ fig.update_layout(margin = dict(t=50, l=25, r=25, b=25))
149
+ return fig
150
+
151
+ def add_text(state, text):
152
+
153
+ #call open ai with the context from the state passed in
154
+
155
+ load_dotenv()
156
+
157
+ pinecone.init(
158
+ api_key=os.getenv('PINECONE_API_KEY'),
159
+ environment=os.getenv('PINECONE_ENVIRONMENT'),
160
+ )
161
+ index = pinecone.Index("vc-content-oracle-big")
162
+
163
+ vector = get_openai_embedding(os.getenv('OPENAI_API_KEY'), text)['data'][0]['embedding']
164
+
165
+ #make call to the vector store
166
+
167
+ res = index.query(
168
+ vector=vector,
169
+ top_k=3,
170
+ include_metadata=True,
171
+ include_values=False,
172
+ )
173
+
174
+ context = 'You are Andressen Horowitz, a Venture Capitalist. You know everything about startup investment, particularly in Generative AI. Everytime you are asked a question respond faithfully and consistently. Answer the question based on the context below \n\n'
175
+
176
+ # 'Organization Name': '',
177
+ # 'Organization Name URL': '',
178
+ # 'Last Funding Amount': '',
179
+ # 'Last Funding Amount Currency': '',
180
+ # 'Last Funding Amount Currency (in USD)': '',
181
+ # 'Website': '',
182
+ # 'Total Funding Amount': '',
183
+ # 'Total Funding Amount Currency': '',
184
+ # 'Total Funding Amount Currency (in USD)': '',
185
+ # 'Last Funding Date': '',
186
+ # 'Founded Date': '',
187
+ # 'Founded Date Precision': '',
188
+ # 'Last Funding Type': '',
189
+ # 'Number of Employees': '',
190
+ # 'Full Description': '',
191
+ # 'Top 5 Investors': '',
192
+
193
+ for match in res['matches'][:1]:
194
+ context += (
195
+ 'Context\n: Name: ' +
196
+ str(match['metadata'].get('Organization Name', '')) +
197
+ 'Funding: ' +
198
+ str(match['metadata'].get('Total Funding Amount', '')) +
199
+ 'Founding Date: ' +
200
+ str(match['metadata'].get('Founded Date', '')) +
201
+ 'Number of Employees: ' +
202
+ str(match['metadata'].get('Number of Employees', '')) +
203
+ 'Description: ' +
204
+ str(match['metadata'].get('Full Description', '')) +
205
+ 'Investors: ' +
206
+ str(match['metadata'].get('Top 5 Investors', ''))
207
+ )
208
+
209
+ context += 'Question: ' + text + '\nAnswer:'
210
+
211
+ response = openai.Completion.create(
212
+ model="text-davinci-003",
213
+ prompt=context,
214
+ max_tokens=200,
215
+ temperature=0,
216
+ )
217
+
218
+ state = state + [(text, response['choices'][0]['text'])]
219
+
220
+ return state, state
221
+
222
+ with gr.Blocks() as demo:
223
+
224
+ #make a title
225
+ title = gr.HTML('<h1 style="text-align: center; color: white; font-size: 50px; font-family: Helvetica">ChatGVC</h1>')
226
+
227
+ #make a subtitle
228
+ subtitle = gr.HTML('<h2 style="text-align: center; color: white; font-size: 20px; font-family: Helvetica">Auto-Generated VC Market Maps</h2>')
229
+
230
+ category = gr.Textbox(label='Market Category')
231
+ n_subcategories = gr.Number(label="Number of Subcategories")
232
+ n_companies = gr.Number(label="Max Number of Companies per Subcategory")
233
+ button = gr.Button("Generate")
234
+ plot = gr.Plot(label="Plot")
235
+ button.click(make_plot, inputs=[category, n_subcategories, n_companies], outputs=[plot])
236
+ # demo.load(make_plot, inputs=[button, category], outputs=[plot])
237
+
238
+ chatbot = gr.Chatbot(elem_id="chatbot")
239
+ state = gr.State([])
240
+
241
+ with gr.Row():
242
+ with gr.Column(scale=0.85):
243
+ txt = gr.Textbox(show_label=False, placeholder="Enter text and press enter, or upload an image").style(container=False)
244
+
245
+ txt.submit(add_text, [state, txt], [state, chatbot])
246
+ txt.submit(lambda :"", None, txt)
247
+
248
+ demo.launch(share=True)
pinecli.py ADDED
@@ -0,0 +1,670 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # encoding: utf-8
3
+ # pylint: disable=line-too-long,too-many-arguments,invalid-name,no-member,missing-function-docstring,missing-module-docstring
4
+
5
+ import logging
6
+ import hashlib
7
+ import itertools
8
+ import os
9
+ import sys
10
+
11
+ from ast import literal_eval
12
+ from time import sleep
13
+ import requests
14
+
15
+ import click
16
+ import pandas as pd
17
+ import matplotlib.pyplot as plt
18
+ import numpy as np
19
+ import openai
20
+ import nltk.data
21
+ import pinecone
22
+
23
+ from bs4 import BeautifulSoup
24
+ from bs4.element import Comment
25
+ from numpy import random
26
+ from sklearn.manifold import TSNE
27
+ from sklearn.cluster import KMeans
28
+ from rich.console import Console
29
+ from rich.table import Table
30
+ from tqdm.auto import tqdm
31
+
32
+ default_region = 'us-west1-gcp'
33
+ openai_embed_model = "text-embedding-ada-002"
34
+
35
+ REGION_HELP = 'Pinecone cluster region'
36
+
37
+ def tag_visible(element):
38
+ """ Strip out undesirables """
39
+ if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
40
+ return False
41
+ if isinstance(element, Comment):
42
+ return False
43
+ return True
44
+
45
+ def text_from_html(body):
46
+ """ Obv pull text from doc with tag_visible filters """
47
+ soup = BeautifulSoup(body, 'html.parser')
48
+ texts = soup.findAll(string=True)
49
+ visible_texts = filter(tag_visible, texts)
50
+ return " ".join(t.strip() for t in visible_texts)
51
+
52
+ def get_openai_embedding(apikey, data):
53
+ """ Fetch an embedding w/ given data """
54
+ openai.api_key = apikey
55
+ try:
56
+ res = openai.Embedding.create(input=data, engine=openai_embed_model)
57
+ except:
58
+ done = False
59
+ while not done:
60
+ sleep(5)
61
+ try:
62
+ res = openai.Embedding.create(
63
+ input=data, engine=openai_embed_model)
64
+ done = True
65
+ except Exception as e:
66
+ click.echo(e)
67
+ return res
68
+
69
+
70
+ @click.group()
71
+ def cli():
72
+ """
73
+ A command line interface for working with Pinecone.
74
+ """
75
+ pass
76
+
77
+
78
+ def exception_handler(exception_type, exception, traceback):
79
+ # All your trace are belong to us!
80
+ click.echo(f"Got exception: {exception_type.__name__} {exception}")
81
+ click.echo("Make sure PINECONE_API_KEY is correct.")
82
+
83
+ # sys.excepthook = exception_handler
84
+
85
+
86
+ def _pinecone_init(apikey, environment, indexname=''):
87
+ apikey = os.environ.get('PINECONE_API_KEY', apikey)
88
+ if apikey == "":
89
+ sys.exit(
90
+ "No Pinecone API key set through PINECONE_API_KEY environment variable or --apikey\nExample: export PINECONE_API_KEY=1234-abc-9876")
91
+ environment = os.environ.get('PINECONE_ENVIRONMENT', environment)
92
+ pinecone.init(api_key=apikey, environment=environment)
93
+ index = None
94
+ if indexname:
95
+ try:
96
+ # index = pinecone.Index(indexname)
97
+ index = pinecone.GRPCIndex(indexname)
98
+ except Exception:
99
+ logging.exception("Unable to connect. Caught exception:")
100
+ sys.exit()
101
+ else:
102
+ return index
103
+
104
+
105
+ def _format_values(vals):
106
+ return ",".join(str(x) for x in vals)[:30]
107
+
108
+
109
+ def _print_table(res, pinecone_index_name, namespace, include_meta, include_values, expand_meta):
110
+ table = Table(
111
+ title=f"🌲 {pinecone_index_name} ns=({namespace}) Index Results")
112
+ table.add_column("ID", justify="right", style="cyan", no_wrap=True)
113
+ table.add_column("NS", justify="left", style="red", no_wrap=True)
114
+ if include_values:
115
+ table.add_column("Values", style="magenta")
116
+ if include_meta:
117
+ table.add_column("Meta", justify="right", style="green")
118
+ table.add_column("Score", justify="right", style="green")
119
+
120
+ ns = res['namespace'] if 'namespace' in res else ''
121
+ for row in res.matches:
122
+ metadata = ''
123
+ score = str(row['score'])
124
+ vecid = row['id']
125
+ if include_meta and 'metadata' in row:
126
+ metadata = str(row['metadata'])
127
+ metadata = metadata[:100] if not expand_meta else metadata
128
+
129
+ if include_values and include_meta:
130
+ table.add_row(vecid, ns, _format_values(
131
+ row['values']), metadata, score)
132
+ elif include_values and not include_meta:
133
+ table.add_row(vecid, ns, _format_values(
134
+ row['values']), score)
135
+ elif not include_values and include_meta:
136
+ table.add_row(vecid, ns, metadata, score)
137
+ elif not include_values and not include_meta:
138
+ table.add_row(vecid, ns, score)
139
+
140
+ console = Console()
141
+ console.print(table)
142
+
143
+ @click.command(short_help='Prints version number.')
144
+ def version():
145
+ if sys.version_info >= (3, 8):
146
+ from importlib import metadata
147
+ else:
148
+ import importlib_metadata as metadata
149
+ click.echo(metadata.version('pinecone_cli'))
150
+
151
+ @click.command(short_help='Queries Pinecone with a given vector.')
152
+ @click.option('--apikey', help='Pinecone API Key')
153
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
154
+ @click.option('--include_values', help='Should we return the vectors', show_default=True, default=True)
155
+ @click.option('--topk', '--numrows', 'topk', type=click.INT, show_default=True, default=10, help='Top K number to return')
156
+ @click.option('--namespace', default="", help='Namespace to select results from')
157
+ @click.option('--include-meta', help='Whether to include the metadata values', default=False, show_default=True)
158
+ @click.option('--expand-meta', help='Whether to fully expand the metadata returned.', is_flag=True, show_default=True, default=False)
159
+ @click.option('--meta_filter', help='Filter out metadata w/ the Pinecone filter syntax which is really JSON. Default is no filter.', default="{}")
160
+ @click.option('--print-table', help='Display the output as a pretty table.', is_flag=True, show_default=True, default=False)
161
+ @click.option('--num-clusters', help='Number of clusters in TSNE plot if --show-tsne is used.', type=click.INT, show_default=True, default=4)
162
+ @click.option('--perplexity', '--perp', help='The perplexity of the TSNE plot, if --show-tsne is used.', type=click.INT, default=15, show_default=True)
163
+ @click.option('--tsne-random-state', type=click.INT, default=42, show_default=True)
164
+ @click.option('--show-tsne', default=False)
165
+ @click.argument('pinecone_index_name')
166
+ @click.argument('query_vector')
167
+ def query(pinecone_index_name, apikey, query_vector, region, topk, include_values, include_meta, expand_meta, num_clusters, perplexity, tsne_random_state, namespace, show_tsne, meta_filter, print_table):
168
+ """ Queries Pinecone index named <PINECONE_INDEX_NAME> with the given <QUERY_VECTOR> and optional namespace.
169
+
170
+ \b
171
+ Example:
172
+ % ./pinecli.py query lpfactset "[0,0]"
173
+
174
+ \b
175
+ Example 2:
176
+ % ./pinecli.py query upsertfile "[1.2, 1.0, 3.0]" --print-table --include-meta=true --filter="{'genre':'drama'}"
177
+
178
+ \b
179
+ Example 3 [Query randomly]:
180
+ % ./pinecli.py query lpfactset random
181
+
182
+ For filter syntax see: https://docs.pinecone.io/docs/metadata-filtering
183
+ """
184
+ pinecone_index = _pinecone_init(apikey, region, pinecone_index_name)
185
+
186
+ if query_vector.lower() == "random":
187
+ res = pinecone_index.describe_index_stats()
188
+ num_vector_dims = res['dimension']
189
+ query_vector = [i for i in range(num_vector_dims)]
190
+ else:
191
+ query_vector = literal_eval(query_vector)
192
+
193
+ res = pinecone_index.query(vector=query_vector, top_k=topk, include_metadata=True,
194
+ include_values=include_values, namespace=namespace, filter=literal_eval(meta_filter))
195
+ if print_table:
196
+ _print_table(res, pinecone_index_name, namespace,
197
+ include_meta, include_values, expand_meta)
198
+ else:
199
+ click.echo(res)
200
+
201
+ if show_tsne:
202
+ show_tsne_plot(pinecone_index_name, res.matches,
203
+ num_clusters, perplexity, tsne_random_state)
204
+
205
+
206
+ def show_tsne_plot(pinecone_index_name, results, num_clusters, perplexity, random_state):
207
+ res2 = np.asarray([np.array(v['values']) for v in results])
208
+ df = pd.DataFrame(data=res2)
209
+
210
+ kmeans = KMeans(n_clusters=num_clusters, init="k-means++",
211
+ random_state=random_state, n_init='auto')
212
+ kmeans.fit(res2)
213
+ labels = kmeans.labels_
214
+ df["Cluster"] = labels
215
+
216
+ tsne = TSNE(n_components=2, perplexity=perplexity,
217
+ random_state=42, init='random', learning_rate=200)
218
+ embeddings2d = tsne.fit_transform(res2)
219
+ x = [x for x, y in embeddings2d]
220
+ y = [y for x, y in embeddings2d]
221
+ (_, ax) = plt.subplots(figsize=(9, 6)) # inches
222
+ plt.style.use('seaborn-whitegrid')
223
+ plt.grid(color='#EAEAEB', linewidth=0.5)
224
+ ax.spines['top'].set_color(None)
225
+ ax.spines['right'].set_color(None)
226
+ ax.spines['left'].set_color('#2B2F30')
227
+ ax.spines['bottom'].set_color('#2B2F30')
228
+ for category, color in enumerate(["purple", "green", "red", "blue", "brown", "gray", "olive", "cyan", "orange", "pink"]):
229
+ xs = np.array(x)[df.Cluster == category]
230
+ ys = np.array(y)[df.Cluster == category]
231
+ plt.scatter(xs, ys, color=color, alpha=0.3)
232
+ avg_x = xs.mean()
233
+ avg_y = ys.mean()
234
+ plt.scatter(avg_x, avg_y, marker="x", color=color, s=100)
235
+
236
+ plt.title(
237
+ f"Clustering of Pinecone Index {pinecone_index_name}", fontsize=16, fontweight='bold', pad=20)
238
+ plt.suptitle(
239
+ f't-SNE (perplexity={perplexity} clusters={num_clusters})', y=0.92, fontsize=13)
240
+ plt.legend(loc='best', frameon=True)
241
+ plt.show()
242
+
243
+
244
+ @click.command(short_help='Fetches vectors from Pinecone specified by the vectors\' ids.')
245
+ @click.option('--apikey', help='Pinecone API Key')
246
+ @click.option('--region', help='Pinecone Index Region', default=default_region)
247
+ @click.option('--vector_ids', help='Vector ids to fetch')
248
+ @click.option('--pretty', is_flag=True, help='Pretty print output.')
249
+ @click.option('--namespace', help='Namespace within the index to search.')
250
+ @click.argument('pinecone_index_name')
251
+ def fetch(pinecone_index_name, apikey, region, vector_ids, namespace, pretty):
252
+ """ Fetch queries from Pinecone by vector_ids
253
+
254
+ \b
255
+ Example:
256
+ % ./pinecli.py fetch lpfactset --vector_ids="05b4509ee655aacb10bfbb6ba212c65c"
257
+ """
258
+ index = _pinecone_init(apikey, region, pinecone_index_name)
259
+ parsed_ids = [x.strip() for x in vector_ids.split(",")]
260
+ fetch_response = index.fetch(ids=parsed_ids, namespace=namespace)
261
+ if pretty:
262
+ click.echo(fetch_response)
263
+ else:
264
+ click.echo(fetch_response)
265
+
266
+
267
+ @click.command(short_help='Extracts text from url arg, vectorizes w/ openai embedding api, and upserts to Pinecone.')
268
+ @click.option('--apikey', help='Pinecone API Key')
269
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
270
+ @click.option('--namespace', help='Pinecone index namespace', default='', show_default=True)
271
+ @click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
272
+ @click.argument('pinecone_index_name')
273
+ @click.argument('vector_literal')
274
+ def upsert(pinecone_index_name, apikey, region, vector_literal, namespace, debug):
275
+ """
276
+ Upserts vectors into the index <PINECONE_INDEX_NAME> by using the <VECTOR_LITERAL> which is a string representation of a list of tuples.
277
+ Note the literal is quoted.
278
+
279
+ \b
280
+ Example:
281
+ % ./pinecli.py upsert upsertfile "[('vec1', [0.1, 0.2, 0.3], {'genre': 'drama'}), ('vec2', [0.2, 0.3, 0.4], {'genre': 'action'}),]" --debug
282
+ """
283
+ index = _pinecone_init(apikey, region, pinecone_index_name)
284
+ if debug:
285
+ click.echo(f"Will upload vectors as {literal_eval(vector_literal)}")
286
+ resp = index.upsert(vectors=literal_eval(
287
+ vector_literal), namespace=namespace)
288
+ if debug:
289
+ click.echo(resp)
290
+
291
+
292
+ @click.command(short_help='Updates the index based on the given id passed in.')
293
+ @click.option('--apikey', help='Pinecone API Key')
294
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
295
+ @click.option('--namespace', help='Pinecone index namespace', default='', show_default=True)
296
+ @click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
297
+ @click.option("--metadata", help="The update to the metadata values.", default='', show_default=True)
298
+ @click.argument("id")
299
+ @click.argument('pinecone_index_name')
300
+ @click.argument('vector_literal')
301
+ def update(pinecone_index_name, apikey, region, id, vector_literal, metadata, namespace, debug):
302
+ """ Updates the index <PINECONE_INDEX_NAME> with id <ID> and vector values <VECTOR_LITERAL> """
303
+ index = _pinecone_init(apikey, region, pinecone_index_name)
304
+ if metadata:
305
+ resp = index.update(id=id, values=literal_eval(
306
+ vector_literal), set_metadata=literal_eval(metadata), namespace=namespace)
307
+ else:
308
+ resp = index.update(id=id, values=literal_eval(
309
+ vector_literal), namespace=namespace)
310
+ if debug:
311
+ click.echo(resp)
312
+
313
+
314
+ @click.command(short_help='Extracts text from url arg, vectorizes w/ openai embedding api, and upserts to Pinecone.')
315
+ @click.option('--apikey', help='Pinecone API Key')
316
+ @click.option('--openaiapikey', required=True, help='OpenAI API Key')
317
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
318
+ @click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
319
+ @click.option("--namespace", help='Namespace to store the generated vectors', default='', show_default=True)
320
+ @click.option("--metadata_content_key", help="The key used to store the page content in the metadata with.", show_default=True, default="content")
321
+ @click.option("--window", help='Number of sentences to combine in one embedding (vector).', show_default=True, default=10)
322
+ @click.option("--stride", help='Number of sentences to stride over to create overlap', show_default=True, default=4)
323
+ @click.option('--url')
324
+ @click.option('--pinecone_index_name')
325
+ def upsert_webpage(pinecone_index_name, apikey, namespace, openaiapikey, metadata_content_key, region, url, window, stride, debug):
326
+ """ Upserts vectors into the index <PINECONE_INDEX_NAME> using the openai embeddings api. You will need your api key for openai and specify it using --openapikey """
327
+ pinecone_index = _pinecone_init(apikey, region, pinecone_index_name)
328
+
329
+ # req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
330
+ # html = urllib.request.urlopen(req).read()
331
+ html = requests.get(url).text
332
+ html = text_from_html(html)
333
+ nltk.download('punkt')
334
+ tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
335
+ sentences = tokenizer.tokenize(html)
336
+ sentences = list(filter(None, sentences))
337
+
338
+ new_data = []
339
+
340
+ text = ''
341
+ for i in tqdm(range(0, len(sentences), stride)): # TODO: Use trange
342
+ i_end = min(len(sentences)-1, i+window)
343
+ if sentences[i] == sentences[i_end]:
344
+ continue
345
+ text = ' '.join(sentences[i:i_end]).strip()
346
+ # create the new merged dataset
347
+ if debug:
348
+ click.echo(f"Text is: {text}")
349
+ if text != "":
350
+ new_data.append(text)
351
+ new_data.append(sentences[-1])
352
+ new_data = list(filter(None, new_data))
353
+ if debug:
354
+ click.echo(*new_data, sep="\n")
355
+
356
+ batch_size = 10 # how many embeddings we create and insert at once
357
+
358
+ for i in tqdm(range(0, len(new_data), batch_size)):
359
+ i_end = min(len(new_data), i+batch_size)
360
+ meta_batch = new_data[i:i_end]
361
+ ids_batch = [hashlib.md5(x.encode('utf-8')).hexdigest()
362
+ for x in meta_batch]
363
+ res = get_openai_embedding(openaiapikey, meta_batch)
364
+ embeds = [record['embedding'] for record in res['data']]
365
+ meta_batch = [{metadata_content_key: x} for x in meta_batch]
366
+ to_upsert = list(zip(ids_batch, embeds, meta_batch))
367
+ rv = pinecone_index.upsert(vectors=to_upsert, namespace=namespace)
368
+ if debug:
369
+ click.echo(rv)
370
+
371
+
372
+ @click.command(short_help='Shows a preview of vectors in the <PINECONE_INDEX_NAME>')
373
+ @click.option('--apikey', help='Pinecone API Key')
374
+ @click.option('--region', help='Pinecone Index Region', default=default_region)
375
+ @click.option('--topk', '--numrows', 'topk', default=10, help='The number of rows to show')
376
+ @click.option('--random_dims', is_flag=True, help='Flag to have query vector dims be random. Default will be 0.0')
377
+ @click.option('--include-values', help='Should we return the vectors', show_default=True, default=True)
378
+ @click.option('--namespace', default="", help='Namespace to select results from')
379
+ @click.option('--include-meta', help='Whether to include the metadata values', default=False, show_default=True)
380
+ @click.option('--expand-meta', help='Whether to fully expand the metadata returned.', is_flag=True, show_default=True, default=False)
381
+ @click.option('--print-table', help='Display the output as a pretty table.', is_flag=True, show_default=True, default=False)
382
+ @click.argument('pinecone_index_name')
383
+ def head(pinecone_index_name, apikey, region, topk, random_dims, namespace, include_meta, expand_meta, include_values, print_table):
384
+ """ Shows a preview of vectors in the <PINECONE_INDEX_NAME> with optional numrows (default 10)
385
+
386
+ \b
387
+ Example 1:
388
+
389
+ % ./pinecli.py head lpfactset --include-meta=true --include-values=true
390
+ {'matches': [{'id': 'ae23d7574c19cea0b3479c93858a3ee3',
391
+ 'metadata': {'content': 'Oded K. R&D Group Lead Why '
392
+ 'Pinecone Fast, fresh, and filtered '
393
+ 'Python client. Scalable Scale from zero '
394
+ 'to billions of items, with no downtime '
395
+ 'and minimal latency impact.'},
396
+ 'score': 0.0,
397
+ 'sparseValues': {},
398
+ 'values': [0.010676071,
399
+
400
+ \b
401
+ Example 2: (printing results with a table)
402
+
403
+ % tim@yoda pinecone-cli % ./pinecli.py head pageuploadtest --include-values=True --include-meta=True --namespace=test --print-table --topk=3
404
+ 🌲 pageuploadtest ns=(test) Index Results
405
+ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
406
+ ┃ ID ┃ NS ┃ Values ┃ Meta ┃ Score ┃
407
+ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
408
+ │ 25d4d1444f9b2942440ce22e026c2a06 │ test │ -0.00443430152,-0.0156992543,0 │ {'content': 'Sign Up for Free or contact us Use Cases What can you do with vector search ? Once you │ 0.0 │
409
+ │ d0bdfbee942fadf531b1feea5b909217 │ test │ 0.0214423928,-0.0283056349,0.0 │ {'content': "Easy to use Get started on the free plan with an easy-to-use API or the Python client. │ 0.0 │
410
+ │ ae23d7574c19cea0b3479c93858a3ee3 │ test │ 0.0106426,-0.000842177891,-0.0 │ {'content': "Oded K. R&D Group Lead Why Pinecone Fast, fresh, and filtered vector search. F │ 0.0 │
411
+ └──────────────────────────────────┴──────┴────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘
412
+ """
413
+ index = _pinecone_init(apikey, region, pinecone_index_name)
414
+ res = index.describe_index_stats()
415
+ dims = res['dimension']
416
+ if random_dims:
417
+ dims = [random.random() for _ in range(dims)]
418
+ else:
419
+ dims = [0.0 for _ in range(dims)]
420
+ resp = index.query(vector=dims, top_k=topk, namespace=namespace,
421
+ include_metadata=include_meta, include_values=include_values)
422
+ if print_table:
423
+ _print_table(resp, pinecone_index_name, namespace,
424
+ include_meta, include_values, expand_meta)
425
+ else:
426
+ click.echo(resp)
427
+
428
+
429
+ @click.command(short_help='Creates a Pinecone Index.')
430
+ @click.option('--apikey', help='Pinecone API Key')
431
+ @click.option('--region', help='Pinecone Index Region', default=default_region)
432
+ @click.option('--dims', help='Number of dimensions for this index', type=click.INT, required=True)
433
+ @click.option('--metric', help='Distance metric to use.', required=True, default='cosine')
434
+ @click.option('--pods', help='Number of pods', default=1, show_default=True, type=click.INT)
435
+ @click.option('--replicas', help='Number of replicas', default=1, show_default=True, type=click.INT)
436
+ @click.option('--shards', help='Number of shards', default=1, show_default=True, type=click.INT)
437
+ @click.option('--pod-type', help='Type of pods to create.', required=True)
438
+ @click.option('--source_collection', help='Source collection to create index from')
439
+ @click.argument('pinecone_index_name')
440
+ def create_index(pinecone_index_name, apikey, region, dims, metric, pods, replicas, shards, pod_type, source_collection):
441
+ """ Creates the Pinecone index named <PINECONE_INDEX_NAME> """
442
+ _pinecone_init(apikey, region)
443
+ resp = pinecone.create_index(pinecone_index_name, dimension=dims, metric=metric,
444
+ pods=pods, replicas=replicas, shards=shards, pod_type=pod_type)
445
+ click.echo(resp)
446
+
447
+
448
+ def chunks(iterable, batch_size=100):
449
+ it = iter(iterable)
450
+ chunk = tuple(itertools.islice(it, batch_size))
451
+ while chunk:
452
+ yield chunk
453
+ chunk = tuple(itertools.islice(it, batch_size))
454
+
455
+
456
+ def chunks_df(df):
457
+ for row in df.itertuples(index=False):
458
+ yield row
459
+
460
+
461
+ @click.command(short_help='Upserts a vector(s) with random dimensions into the specified vector.')
462
+ @click.option('--apikey', help='Pinecone API Key')
463
+ @click.option('--region', help='Pinecone Index Region', default=default_region)
464
+ @click.argument('pinecone_index_name')
465
+ @click.option('--num_vectors', '--num_rows', '--numrows', type=click.INT)
466
+ @click.option('--debug', is_flag=True, default=False, show_default=True)
467
+ @click.option('--num_vector_dims', type=click.INT)
468
+ def upsert_random(pinecone_index_name, apikey, region, num_vectors, num_vector_dims, debug):
469
+ """ Upserts random vectors and dimension values using num_vectors (rows) and num_vector_dims (number of dims per vector). IDs for the example vectors will be of the form \'id-{rownum}\' """
470
+ index = _pinecone_init(apikey, region, pinecone_index_name)
471
+
472
+ # Example generator that generates many (id, vector) pairs
473
+ example_data_generator = map(lambda i: (
474
+ f'id-{i}', [random.random() for _ in range(num_vector_dims)]), range(num_vectors))
475
+
476
+ # Upsert data with 100 vectors per upsert request
477
+ batch_size = 100
478
+ for ids_vectors_chunk in tqdm(chunks(example_data_generator, batch_size=batch_size), total=num_vectors/batch_size):
479
+ rv = index.upsert(vectors=ids_vectors_chunk)
480
+ if debug:
481
+ click.echo(rv)
482
+
483
+
484
+ @click.command(short_help='Upserts a file (csv) into the specified index.')
485
+ @click.option('--apikey', help='Pinecone API Key')
486
+ @click.option('--region', help='Pinecone Index Region', default=default_region)
487
+ @click.option('--batch_size', help='Number vectors to upload per batch', default=100, type=click.INT)
488
+ @click.option('--namespace', default='')
489
+ @click.option("--debug", is_flag=True, show_default=True, default=False, help="Output debug to stdout.")
490
+ @click.argument('vector-file', )
491
+ @click.argument('pinecone_index_name')
492
+ @click.argument('colmap')
493
+ def upsert_file(pinecone_index_name, apikey, region, vector_file, batch_size, colmap, namespace, debug):
494
+ colmap = literal_eval(colmap)
495
+ if (('id' not in colmap) or ('vectors' not in colmap)):
496
+ click.echo(
497
+ "Missing 'id' or 'vectors' keys in mapping of CSV file. Check header definitions.")
498
+ sys.exit(-1)
499
+
500
+ reverse_col_map = dict(reversed(list(colmap.items())))
501
+ index = _pinecone_init(apikey, region, pinecone_index_name)
502
+
503
+ def convert(item):
504
+ item = item.strip() # remove spaces at the end
505
+ item = item[1:-1] # remove `[ ]`
506
+ return list(map(float, item.split(',')))
507
+
508
+ usecols = [reverse_col_map['id'], reverse_col_map['vectors']]
509
+ converters = {reverse_col_map['vectors']: convert}
510
+
511
+ if 'metadata' in colmap:
512
+ csv_meta_col_name = reverse_col_map['metadata']
513
+ usecols.append(reverse_col_map['metadata'])
514
+ converters[csv_meta_col_name] = literal_eval
515
+
516
+ for chunk in pd.read_csv(vector_file, chunksize=batch_size, index_col=False, usecols=usecols, converters=converters):
517
+ v = chunk.to_records(index=False).tolist()
518
+ rv = index.upsert(vectors=v, namespace=namespace)
519
+ if debug:
520
+ click.echo(rv)
521
+
522
+
523
+ @click.command(short_help='Lists the indexes for your api key.')
524
+ @click.option('--apikey', help='API Key')
525
+ @click.argument('region', default=default_region)
526
+ def list_indexes(apikey, region):
527
+ """ List all Pinecone indexes for the given api key. """
528
+ _pinecone_init(apikey, region)
529
+ res = pinecone.list_indexes()
530
+ click.echo('\n'.join(res))
531
+
532
+
533
+ @click.command(short_help='Describes an index.')
534
+ @click.option('--apikey')
535
+ @click.argument('pinecone_index_name', required=True)
536
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
537
+ def describe_index(apikey, pinecone_index_name, region):
538
+ """ Describe a Pinecone index with given index_name. """
539
+ _pinecone_init(apikey, region, pinecone_index_name)
540
+ desc = pinecone.describe_index(pinecone_index_name)
541
+ click.echo("\n".join([f"Name: {desc.name}", f"Dimensions: {int(desc.dimension)}",
542
+ f"Metric: {desc.metric}", f"Pods: {desc.pods}", f"PodType: {desc.pod_type}", f"Shards: {desc.shards}",
543
+ f"Replicas: {desc.replicas}", f"Ready: {desc.status['ready']}", f"State: {desc.status['state']}",
544
+ f"Metaconfig: {desc.metadata_config}", f"Sourcecollection: {desc.source_collection}"]))
545
+
546
+
547
+ @click.command(short_help='Configures the given index to have a pod type.')
548
+ @click.option('--apikey', help='Pinecone API Key')
549
+ @click.argument('pinecone_index_name', required=True)
550
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
551
+ @click.option('--pod_type', required=True, help='Type of pod to use')
552
+ def configure_index_pod_type(apikey, pinecone_index_name, region, pod_type):
553
+ """ Configure the pod type for a given index_name. """
554
+ _pinecone_init(apikey, region)
555
+ pinecone.configure_index(pinecone_index_name, pod_type=pod_type)
556
+
557
+
558
+ @click.command(short_help='Configures the number of replicas for a given index.')
559
+ @click.option('--apikey')
560
+ @click.argument('pinecone_index_name', required=True)
561
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
562
+ @click.option('--num_replicas', required=True, help='Number of replicas to use.')
563
+ def configure_index_replicas(apikey, pinecone_index_name, region, num_replicas):
564
+ """ Configure the number of replicas for an index. """
565
+ _pinecone_init(apikey, region)
566
+ pinecone.configure_index(pinecone_index_name, replicas=num_replicas)
567
+
568
+
569
+ @click.command(short_help='Creates a Pinecone collection from the argument \'source_index\'')
570
+ @click.option('--apikey')
571
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
572
+ @click.option('--collection_name', help='The name of the collection to create.', required=True)
573
+ @click.option('--source_index', help='The name index to create collection from.', required=True)
574
+ def create_collection(apikey, region, collection_name, source_index):
575
+ """ Create a Pinecone collection with the given collection_name and source_index. """
576
+ _pinecone_init(apikey, region)
577
+ pinecone.create_collection(collection_name, source_index)
578
+
579
+
580
+ @click.command(short_help='Prints out index stats to stdout.')
581
+ @click.option('--apikey')
582
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
583
+ @click.argument('pinecone_index_name', required=True)
584
+ def describe_index_stats(apikey, region, pinecone_index_name):
585
+ """ Show the stats for index with name <PINECONE_INDEX_NAME>. Note that if the index has several namespaces, those will be broken out.
586
+
587
+ \b
588
+ Example:
589
+ % ./pinecli.py describe-index-stats lpfactset
590
+ """
591
+ index = _pinecone_init(apikey, region, pinecone_index_name)
592
+ res = index.describe_index_stats()
593
+ console = Console()
594
+ click.echo(f"Dimensions: {res['dimension']}")
595
+ click.echo(f"Vectors: {res['total_vector_count']}")
596
+ click.echo(f"Index_Fullness: {res['index_fullness']}")
597
+ ns_data = res['namespaces']
598
+ console.print("Namespace data:", style="b")
599
+ for ns in ns_data.keys():
600
+ click.echo(f"\t{ns}: {ns_data[ns]['vector_count']}")
601
+
602
+
603
+ @click.command(short_help='Lists collections for the given apikey.')
604
+ @click.option('--apikey')
605
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
606
+ def list_collections(apikey, region):
607
+ """ List Pinecone collections with the given api key """
608
+ _pinecone_init(apikey, region)
609
+ res = pinecone.list_collections()
610
+ click.echo(*res, sep='\n')
611
+
612
+
613
+ @click.command(short_help='Describes a collection.')
614
+ @click.option('--apikey')
615
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
616
+ @click.argument('collection_name', required=True)
617
+ def describe_collection(apikey, region, collection_name):
618
+ """ Describe the collection described by <COLLECTION_NAME> """
619
+ _pinecone_init(apikey, region)
620
+ desc = pinecone.describe_collection(collection_name)
621
+ click.echo("\n".join([f"Name: {desc.name}", f"Dimensions: {int(desc.dimension)}",
622
+ f"Vectors: {int(desc.vector_count)}", f"Status: {desc.status}", f"Size: {desc.size}"]))
623
+
624
+
625
+ @click.command(short_help="Deletes a collection.")
626
+ @click.option('--apikey')
627
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
628
+ @click.option('--collection_name', help='The name of the collection to create.', required=True)
629
+ def delete_collection(apikey, region, collection_name):
630
+ """ Delete a collection with the given collection_name """
631
+ _pinecone_init(apikey, region)
632
+ pinecone.delete_collection(collection_name)
633
+
634
+ @click.command(short_help='Deletes an index. You will be prompted to confirm.')
635
+ @click.option('--apikey')
636
+ @click.option('--region', help='Pinecone Index Region', show_default=True, default=default_region)
637
+ @click.argument('pinecone_index', required=True)
638
+ def delete_index(apikey, region, pinecone_index):
639
+ """ Delete an index with the given pinecone_index name """
640
+ _pinecone_init(apikey, region)
641
+ value = click.prompt('Type name of index backwards to confirm: ')
642
+ if value == pinecone_index[::-1]:
643
+ pinecone.delete_index(pinecone_index)
644
+ else:
645
+ click.echo("Index not deleted: reversed index name does not match.")
646
+
647
+
648
+ cli.add_command(query)
649
+ cli.add_command(upsert)
650
+ cli.add_command(upsert_file)
651
+ cli.add_command(upsert_random)
652
+ cli.add_command(update)
653
+ cli.add_command(list_indexes)
654
+ cli.add_command(delete_index)
655
+ cli.add_command(create_index)
656
+ cli.add_command(describe_index)
657
+ cli.add_command(upsert_webpage)
658
+ cli.add_command(configure_index_pod_type)
659
+ cli.add_command(configure_index_replicas)
660
+ cli.add_command(create_collection)
661
+ cli.add_command(list_collections)
662
+ cli.add_command(describe_collection)
663
+ cli.add_command(delete_collection)
664
+ cli.add_command(describe_index_stats)
665
+ cli.add_command(fetch)
666
+ cli.add_command(head)
667
+ cli.add_command(version)
668
+
669
+ if __name__ == "__main__":
670
+ cli()
requirements.txt ADDED
File without changes
setup.py ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from setuptools import setup
2
+
3
+ setup(
4
+ name='pinecone-cli',
5
+ description='pinecone-cli is a command-line client for interacting with the pinecone vector embedding database.',
6
+ version='0.1.1',
7
+ url='https://github.com/tullytim/pinecone-cli',
8
+ author='Tim Tully',
9
+ author_email='tim@menlovc.com',
10
+ license='MIT',
11
+ keywords='pinecone vector vectors embeddings database transformers models',
12
+ python_requires='>=3',
13
+ py_modules=['pinecli'],
14
+ install_requires=[
15
+ 'Click', 'pandas', 'numpy', 'openai', 'pinecone-client', 'matplotlib', \
16
+ 'scikit-learn', 'beautifulsoup4', 'nltk', 'sklearn', 'rich'
17
+ ],
18
+ entry_points={
19
+ 'console_scripts': [
20
+ 'pinecli = pinecli:cli',
21
+ ],
22
+ },
23
+ )
sitemap_crawler.py ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import requests
2
+ from bs4 import BeautifulSoup as Soup
3
+ import pandas as pd
4
+ import hashlib
5
+
6
+
7
+ # Pass the headers you want to retrieve from the xml such as ["loc", "lastmod"]
8
+ def parse_sitemap( url,headers):
9
+ resp = requests.get(url)
10
+ # we didn't get a valid response, bail
11
+ if (200 != resp.status_code):
12
+ return False
13
+
14
+ # BeautifulSoup to parse the document
15
+ soup = Soup(resp.content, "html.parser")
16
+
17
+ # find all the <url> tags in the document
18
+ urls = soup.findAll('url')
19
+ sitemaps = soup.findAll('sitemap')
20
+ new_list = [] + headers
21
+ panda_out_total = pd.DataFrame([], columns=new_list)
22
+
23
+
24
+ if not urls and not sitemaps:
25
+ return False
26
+
27
+ # Recursive call to the the function if sitemap contains sitemaps
28
+ if sitemaps:
29
+ for u in sitemaps:
30
+ test = u.find('loc').string
31
+ panda_recursive = parse_sitemap(test, headers)
32
+ panda_out_total = pd.concat([panda_out_total, panda_recursive], ignore_index=True)
33
+
34
+ out = []
35
+
36
+ # Extract the keys we want
37
+ for u in urls:
38
+ values = []
39
+ for head in headers:
40
+ loc = None
41
+ loc = u.find(head)
42
+ if not loc:
43
+ loc = "None"
44
+ else:
45
+ loc = loc.string
46
+ values.append(loc)
47
+ out.append(values)
48
+
49
+ # Create a dataframe
50
+ panda_out = pd.DataFrame(out, columns= new_list)
51
+
52
+ # If recursive then merge recursive dataframe
53
+ if not panda_out_total.empty:
54
+ panda_out = pd.concat([panda_out, panda_out_total], ignore_index=True)
55
+
56
+ #returns the dataframe
57
+ return panda_out
58
+
59
+ def main():
60
+ print(parse_sitemap("https://menlovc.com/sitemap_index.xml", ["loc"]))
61
+ #parse_sitemap("https://menlovc.com/sitemap_index.xml", ["loc"]).to_csv("menlo_links.csv")
62
+
63
+ if __name__ == "__main__":
64
+ main()
65
+