File size: 11,389 Bytes
097eaf0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
# Firecrawl Java SDK

Java SDK for [Firecrawl](https://firecrawl.dev) — search, scrape, and interact with the web.

## Prerequisites

Before using the Java SDK, ensure you have the following installed:

### Java Development Kit (JDK)

- **Required:** Java 11 or later
- **Installation (macOS):**
  ```bash
  brew install openjdk
  ```
  
  Then add Java to your PATH:
  ```bash
  echo 'export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"' >> ~/.zshrc
  source ~/.zshrc
  ```

- **Installation (Linux):**
  ```bash
  # Ubuntu/Debian
  sudo apt-get update
  sudo apt-get install openjdk-11-jdk
  
  # Fedora/RHEL
  sudo dnf install java-11-openjdk-devel
  ```

- **Verify Installation:**
  ```bash
  java --version
  ```

### Gradle (for building from source)

- **Required:** Gradle 8+ 
- **Installation (macOS):**
  ```bash
  brew install gradle
  ```

- **Installation (Linux):**
  ```bash
  # Ubuntu/Debian
  sudo apt-get install gradle
  
  # Or use SDKMAN
  curl -s "https://get.sdkman.io" | bash
  sdk install gradle
  ```

- **Verify Installation:**
  ```bash
  gradle --version
  ```

### API Key Setup

1. Get your API key from [Firecrawl Dashboard](https://firecrawl.dev)
2. Set it as an environment variable:
   ```bash
   export FIRECRAWL_API_KEY="fc-your-api-key-here"
   ```

3. **Or** add it to your shell profile for persistence:
   ```bash
   # For Zsh (macOS/Linux)
   echo 'export FIRECRAWL_API_KEY="fc-your-api-key-here"' >> ~/.zshrc
   source ~/.zshrc
   
   # For Bash
   echo 'export FIRECRAWL_API_KEY="fc-your-api-key-here"' >> ~/.bashrc
   source ~/.bashrc
   ```

## Installation

### Gradle (Kotlin DSL)

```kotlin
implementation("com.firecrawl:firecrawl-java:1.6.0")
```

### Gradle (Groovy)

```groovy
implementation 'com.firecrawl:firecrawl-java:1.6.0'
```

### Maven

```xml
<dependency>
    <groupId>com.firecrawl</groupId>
    <artifactId>firecrawl-java</artifactId>
    <version>1.6.0</version>
</dependency>
```

## Quick Start

```java
import com.firecrawl.client.FirecrawlClient;
import com.firecrawl.models.*;
import java.util.List;

// Create client with explicit API key
FirecrawlClient client = FirecrawlClient.builder()
    .apiKey("fc-your-api-key")
    .build();

// Scrape a page
Document doc = client.scrape("https://example.com",
    ScrapeOptions.builder()
        .formats(List.of("markdown"))
        .build());

System.out.println(doc.getMarkdown());
```

Or create a client from the environment variable:

```java
// export FIRECRAWL_API_KEY=fc-your-api-key
FirecrawlClient client = FirecrawlClient.fromEnv();
```

## API Reference

### Scrape

Scrape a single URL and get the content in various formats.

```java
Document doc = client.scrape("https://example.com",
    ScrapeOptions.builder()
        .formats(List.of("markdown", "html"))
        .onlyMainContent(true)
        .waitFor(5000)
        .build());

System.out.println(doc.getMarkdown());
System.out.println(doc.getMetadata().get("title"));
```

### Video Extraction

Use the `video` format on supported video URLs, including YouTube and TikTok. The returned `video` field is a signed URL to the extracted video file.

```java
Document doc = client.scrape("https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    ScrapeOptions.builder()
        .formats(List.of("video"))
        .build());

System.out.println(doc.getVideo());
```

### Parse Uploaded Files

Upload local files (`html`, `pdf`, `docx`, etc.) via multipart form data and parse them synchronously.
Parse options intentionally exclude browser-only features like change tracking, screenshot, branding, audio, video, actions, waitFor, location, and mobile.

```java
ParseFile file = ParseFile.builder()
    .filename("upload.html")
    .content("<!DOCTYPE html><html><body><h1>Java Parse</h1></body></html>".getBytes())
    .contentType("text/html")
    .build();

Document parsed = client.parse(file,
    ParseOptions.builder()
        .formats(List.of("markdown"))
        .build());

System.out.println(parsed.getMarkdown());
```

#### JSON Extraction

```java
import com.firecrawl.models.JsonFormat;

JsonFormat jsonFmt = JsonFormat.builder()
    .prompt("Extract the product name and price")
    .schema(Map.of(
        "type", "object",
        "properties", Map.of(
            "name", Map.of("type", "string"),
            "price", Map.of("type", "number")
        )
    ))
    .build();

Document doc = client.scrape("https://example.com/product",
    ScrapeOptions.builder()
        .formats(List.of(jsonFmt))
        .build());

System.out.println(doc.getJson());
```

#### Scrape-Bound Interactive Session

Run browser automation against the page context captured by a scrape job:

```java
Document doc = client.scrape("https://example.com");
String scrapeId = String.valueOf(doc.getMetadata().get("scrapeId"));

BrowserExecuteResponse exec = client.interact(
    scrapeId,
    "console.log(await page.title());",
    "node",
    30
);

System.out.println(exec.getStdout());

BrowserDeleteResponse deleted = client.stopInteractiveBrowser(scrapeId);
System.out.println("Deleted: " + deleted.isSuccess());
```

### Crawl

Crawl an entire website. The `crawl()` method polls until completion.

```java
// Convenience method — polls until done
CrawlJob job = client.crawl("https://example.com",
    CrawlOptions.builder()
        .limit(50)
        .maxDiscoveryDepth(3)
        .scrapeOptions(ScrapeOptions.builder()
            .formats(List.of("markdown"))
            .build())
        .build());

for (Document doc : job.getData()) {
    System.out.println(doc.getMetadata().get("sourceURL"));
}
```

#### Async Crawl (manual polling)

```java
CrawlResponse start = client.startCrawl("https://example.com",
    CrawlOptions.builder().limit(100).build());

System.out.println("Job started: " + start.getId());

// Poll manually
CrawlJob status;
do {
    try { Thread.sleep(2000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); break; }
    status = client.getCrawlStatus(start.getId());
    System.out.println(status.getCompleted() + "/" + status.getTotal());
} while (!status.isDone());
```

### Batch Scrape

Scrape multiple URLs in parallel.

```java
BatchScrapeJob job = client.batchScrape(
    List.of("https://example.com", "https://example.org"),
    BatchScrapeOptions.builder()
        .options(ScrapeOptions.builder()
            .formats(List.of("markdown"))
            .build())
        .build());

for (Document doc : job.getData()) {
    System.out.println(doc.getMarkdown());
}
```

### Map

Discover all URLs on a website.

```java
MapData data = client.map("https://example.com",
    MapOptions.builder()
        .limit(100)
        .search("blog")
        .build());

for (Map<String, Object> link : data.getLinks()) {
    System.out.println(link.get("url") + " - " + link.get("title"));
}
```

### Search

Search the web and optionally scrape results.

```java
SearchData results = client.search("firecrawl",
    SearchOptions.builder()
        .limit(10)
        .build());

if (results.getWeb() != null) {
    for (Map<String, Object> result : results.getWeb()) {
        System.out.println(result.get("title") + " — " + result.get("url"));
    }
}
```

### Agent

Run an AI-powered agent to research and extract data from the web.

```java
AgentStatusResponse result = client.agent(
    AgentOptions.builder()
        .prompt("Find the pricing plans for Firecrawl and compare them")
        .build());

System.out.println(result.getData());
```

### Usage & Metrics

```java
ConcurrencyCheck conc = client.getConcurrency();
System.out.println("Concurrency: " + conc.getConcurrency() + "/" + conc.getMaxConcurrency());

CreditUsage credits = client.getCreditUsage();
System.out.println("Remaining credits: " + credits.getRemainingCredits());
```

## Async Support

All methods have async variants that return `CompletableFuture`:

```java
import java.util.concurrent.CompletableFuture;

CompletableFuture<Document> future = client.scrapeAsync(
    "https://example.com",
    ScrapeOptions.builder().formats(List.of("markdown")).build());

future.thenAccept(doc -> System.out.println(doc.getMarkdown()));
```

## Error Handling

The SDK throws unchecked exceptions:

```java
import com.firecrawl.errors.*;

try {
    Document doc = client.scrape("https://example.com");
} catch (AuthenticationException e) {
    // 401 — invalid API key
    System.err.println("Auth failed: " + e.getMessage());
} catch (RateLimitException e) {
    // 429 — too many requests
    System.err.println("Rate limited: " + e.getMessage());
} catch (JobTimeoutException e) {
    // Async job timed out
    System.err.println("Job " + e.getJobId() + " timed out after " + e.getTimeoutSeconds() + "s");
} catch (FirecrawlException e) {
    // All other API errors
    System.err.println("Error " + e.getStatusCode() + ": " + e.getMessage());
}
```

## Configuration

```java
FirecrawlClient client = FirecrawlClient.builder()
    .apiKey("fc-your-api-key")            // Required (or set FIRECRAWL_API_KEY env var)
    .apiUrl("https://api.firecrawl.dev")  // Optional (or set FIRECRAWL_API_URL env var)
    .timeoutMs(300_000)                   // HTTP timeout: 5 min default
    .maxRetries(3)                        // Auto-retries for transient failures
    .backoffFactor(0.5)                   // Exponential backoff factor (seconds)
    .asyncExecutor(myExecutor)            // Custom executor for async methods
    .build();
```

## Building from Source

### Clone and Build

```bash
# Clone the repository (if you haven't already)
git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl/apps/java-sdk

# Build the project
gradle build
```

### Generate JAR

```bash
gradle jar
# Output: build/libs/firecrawl-java-1.6.0.jar
```

### Install Locally

```bash
gradle publishToMavenLocal
# Now available as: com.firecrawl:firecrawl-java:1.6.0 in local Maven repository
```

## Running Tests

The SDK includes both unit tests and E2E integration tests.

### Unit Tests (No API Key Required)

Unit tests verify SDK functionality without making actual API calls:

```bash
gradle test
```

### E2E Integration Tests (API Key Required)

E2E tests make real API calls and require a valid API key. These tests will be **skipped** if `FIRECRAWL_API_KEY` is not set:

```bash
# Set your API key
export FIRECRAWL_API_KEY="fc-your-api-key-here"

# Run all tests including E2E
gradle test
```

### Run Specific Tests

```bash
# Run only scrape tests
gradle test --tests "*testScrape*"

# Run only E2E tests
gradle test --tests "*E2E"

# Run specific test class
gradle test --tests "com.firecrawl.FirecrawlClientTest"
```

### View Test Results

After running tests, view the detailed report:

```bash
open build/reports/tests/test/index.html  # macOS
xdg-open build/reports/tests/test/index.html  # Linux
```

## Development Setup

If you're contributing to the SDK or testing local changes:

1. **Install Prerequisites** (see Prerequisites section above)

2. **Set Environment Variables:**
   ```bash
   export FIRECRAWL_API_KEY="fc-your-api-key"
   # Optional: use local API server
   export FIRECRAWL_API_URL="http://localhost:3002"
   ```

3. **Build and Test:**
   ```bash
   gradle clean build test
   ```

4. **Make Changes and Retest:**
   ```bash
   # Quick compilation check
   gradle compileJava
   
   # Run tests
   gradle test --tests "*testYourFeature*"
   ```