stackConsult commited on
Commit
ce2bee1
·
verified ·
1 Parent(s): ff2aaef

Add comprehensive API Rate Limits & Throttling documentation

Browse files

Added comprehensive API rate limiting documentation covering all major integration points:

**LinkedIn API:**
- Connection/messaging limits (100-200/week, 150 messages/day)
- API request limits by scale (7,500-30,000 requests/day)
- Exponential backoff implementation (15-30 min delays)

**Email APIs (SendGrid, Mailgun):**
- Request rate limits (300-600 requests/minute)
- Daily email quotas by plan tier
- Automatic throttling behavior

**Phone/SMS (Twilio):**
- 1 CPS limit per account/phone number
- Conversation limits (30 messages/30 seconds)
- Error codes and handling

**Data Enrichment (ZoomInfo, Clearbit, Hunter.io, Apollo.io):**
- Per-second and per-minute limits
- Daily quotas by plan
- Batch enrichment constraints
- Credit-based pricing models

**CRM (Salesforce):**
- 100,000+ daily requests (Enterprise)
- Concurrent request limits (25 max)
- Bulk API batch limits (15,000/day)
- Rolling 24-hour windows

**Platform Implementation:**
- Rate limit detection and monitoring
- Circuit breaker patterns
- Queue management
- Live Diagnostics panel integration
- Configuration file references

**Additional Resources:**
- Links to official API documentation for each service
- Best practices and recommended actions
- Monitoring and alerting guidelines

This documentation addresses the rate limit warning observed in Live Diagnostics and provides operational guidance for managing API quotas across all integrated services.

Files changed (1) hide show
  1. README.md +162 -0
README.md CHANGED
@@ -49,3 +49,165 @@ Example construction is provided in app.py showing default data structures and r
49
  ---
50
  (legacy platform and API docs retained below)
51
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  ---
50
  (legacy platform and API docs retained below)
51
 
52
+ ---
53
+
54
+ ## API Rate Limits & Throttling
55
+
56
+ ### Overview
57
+
58
+ This platform integrates with multiple external APIs for B2B outreach automation. Each service enforces rate limits to prevent abuse and ensure service stability. Understanding these limits is critical for:
59
+ - **Preventing service disruptions** (429 errors, temporary blocks)
60
+ - **Optimizing batch operations** (cadence scheduling, bulk enrichment)
61
+ - **Compliance monitoring** (audit trail accuracy)
62
+ - **Cost management** (avoiding overage charges)
63
+
64
+ **Rate Limit Reset Timing:** Most APIs reset limits at **midnight UTC** daily, with some using rolling windows.
65
+
66
+ ---
67
+
68
+ ### LinkedIn API Rate Limits
69
+
70
+ **Connection & Messaging Limits:**
71
+ - **Connection Requests:** 100-200 per week (varies by account reputation)
72
+ - New accounts: ~50-75/week
73
+ - Established accounts: ~100/week
74
+ - Trusted accounts (high SSI): up to 200/week
75
+ - **Messages:** 150 per day (premium accounts)
76
+ - **Profile Views:** 80/day (standard), up to 1,000/day (premium)
77
+
78
+ **API Request Limits (Scale-Dependent):**
79
+ - **1,000-5,000 members:** 7,500 requests/day, 1,500 requests/hour
80
+ - **5,001-10,000 members:** 15,000 requests/day, 15,000 requests/hour
81
+ - **10,001-20,000 members:** 30,000 requests/day, 30,000 requests/hour
82
+ - **Rate limit errors:** HTTP 429 "Too Many Requests"
83
+ - **Alert threshold:** Email notification at 75% of quota
84
+
85
+ **Best Practices:**
86
+ - Implement **exponential backoff** (15-30 minute delays) when hitting limits
87
+ - Monitor `X-RateLimit-Remaining` response headers
88
+ - Limits reset at midnight UTC daily
89
+
90
+ ---
91
+
92
+ ### Email API Rate Limits
93
+
94
+ #### SendGrid
95
+ - **API Rate Limit:** 600 requests/minute (varies by endpoint)
96
+ - **Free Plan:** 100 emails/day
97
+ - **Paid Plans:** Up to 1,500 emails/day, max 400 emails/hour
98
+ - **Enterprise:** 1.5M emails/month with dedicated IP
99
+
100
+ #### Mailgun
101
+ - **API Rate Limit:** 300 requests/minute
102
+ - **Free Plan:** 5,000 emails/month, 300 emails/day
103
+ - **Paid Plans:** 50,000-100,000 emails/month (Foundation/Growth/Scale)
104
+ - **Throttling:** Automatic rate limiting based on sender behavior
105
+
106
+ ---
107
+
108
+ ### Phone/SMS API Rate Limits (Twilio)
109
+
110
+ - **Outbound Calls:** 1 call per second (CPS) per account
111
+ - **SMS Messages:** 1 message per second per phone number
112
+ - **Conversation Limit:** 30 outbound replies between two numbers in 30 seconds
113
+ - **Error Code:** 14107 when SMS rate limit exceeded
114
+ - **Webhook Limits:** Service-specific (see Twilio Verify documentation)
115
+
116
+ **Important:** The 500th user in a batch will receive their call/SMS after ~500 seconds due to 1 CPS limit.
117
+
118
+ ---
119
+
120
+ ### Data Enrichment API Rate Limits
121
+
122
+ #### ZoomInfo
123
+ - **Standard APIs:** 25 requests/second (1,500 requests/minute)
124
+ - **Scaling APIs:** 33 requests/second
125
+ - **Daily Limits:**
126
+ - Pro Plan: ~1,000 API calls/day
127
+ - Enterprise Plan: up to 10,000 API calls/day
128
+ - **Batch Enrichment:** 100,000 records max per batch
129
+ - **Pricing:** Credit-based system (~$15,000/year for 5,000 credits)
130
+
131
+ #### Clearbit (Breeze Intelligence)
132
+ - **Rate Limits:** Not publicly specified (HubSpot integration)
133
+ - **Batch Enrichment:** Limited to 100,000 records at once
134
+ - **Pricing:** Credit-based, starts at $45-50/month (100 credits)
135
+ - **Real-time Updates:** API-first approach for live data
136
+
137
+ #### Hunter.io
138
+ - **Domain Search & Email Finder:** 15 requests/second, 500 requests/minute
139
+ - **Email Verifier:** 10 requests/second, 300 requests/minute
140
+ - **Error Response:** HTTP 403 when limit exceeded
141
+ - **Plans:**
142
+ - Free: Limited credits
143
+ - Growth: 10,000 credits/month
144
+ - Scale: 25,000 credits/month
145
+
146
+ #### Apollo.io
147
+ - **Free Plan:** 50 API calls/minute, 600 calls/day
148
+ - **Paid Plans:** 200 API calls/minute, 2,000 calls/day
149
+ - **Bulk Enrichment:** 1/10 of standard rate limit (but 10 inputs per call)
150
+ - **Error:** HTTP 429 after exceeding limit
151
+
152
+ ---
153
+
154
+ ### CRM API Rate Limits (Salesforce)
155
+
156
+ - **Daily Request Limit:** 100,000 base requests/24 hours (Enterprise Edition)
157
+ - Plus 1,000 additional requests per user license
158
+ - Example: 50 users = 150,000 daily requests
159
+ - **Concurrent Requests:** Max 25 long-running requests (20+ seconds)
160
+ - Developer orgs: 5 concurrent requests
161
+ - **API Timeout:** Hard 10-minute maximum per call
162
+ - **Bulk API:** 15,000 batches/day, 10,000 records per batch
163
+ - **Report API:** 1,200 async requests/hour
164
+ - **Measurement Window:** Rolling 24-hour window (not calendar days)
165
+
166
+ ---
167
+
168
+ ### Platform Implementation
169
+
170
+ **Our Rate Limit Handling Strategy:**
171
+
172
+ 1. **Detection:** Monitor response headers (`X-RateLimit-*`, HTTP 429 status)
173
+ 2. **Exponential Backoff:** 15-minute initial delay, doubling on repeated failures
174
+ 3. **Circuit Breaker:** Temporary pause (15-30 min) when threshold reached
175
+ 4. **Queue Management:** Batch operations respect per-service limits
176
+ 5. **Alerting:** Live Diagnostics panel logs rate limit events
177
+
178
+ **Configuration Files:**
179
+ - Rate limit configs: `/docs/configs/rate_limits.json`
180
+ - Throttling logic: See `cadence_agent.py` backoff implementation
181
+ - Monitoring: Live Diagnostics Panel tracks limit-related warnings
182
+
183
+ **Example from Live Diagnostics:**
184
+ ```
185
+ [2025-11-06T06:33:00Z] warning: Rate limit reached for LinkedIn API;
186
+ implementing exponential backoff for 15 minutes
187
+ ```
188
+
189
+ ---
190
+
191
+ ### Monitoring & Alerts
192
+
193
+ **Dashboard Integration:**
194
+ - **Live Diagnostics Panel:** Real-time rate limit warnings
195
+ - **Audit Trail Export:** Historical rate limit incidents (CSV export)
196
+ - **Compliance Tracking:** Flags batches affected by rate limiting
197
+
198
+ **Recommended Actions:**
199
+ - Review Live Diagnostics panel regularly for rate limit patterns
200
+ - Adjust batch sizes and cadence timing based on observed limits
201
+ - Consider upgrading API plans if consistently hitting free tier limits
202
+ - Implement staggered scheduling to distribute load throughout the day
203
+
204
+ ---
205
+
206
+ ### Additional Resources
207
+
208
+ - LinkedIn API Documentation: [Microsoft Learn - Rate Limits](https://learn.microsoft.com/en-us/linkedin/shared/api-guide/concepts/rate-limits)
209
+ - Twilio Rate Limits: [Understanding Twilio Rate Limits](https://help.twilio.com/articles/115002943027)
210
+ - Salesforce API Limits: [API Request Limits and Allocations](https://developer.salesforce.com/docs/atlas.en-us.salesforce_app_limits_cheatsheet.meta)
211
+ - Hunter.io API: [Rate Limits Documentation](https://help.hunter.io/en/articles/1970956-hunter-api)
212
+ - Apollo.io API: [View API Usage Stats](https://docs.apollo.io/reference/view-api-usage-stats)
213
+