Batch API

Batch API

The Batch API allows you to submit multiple chat requests for asynchronous processing. Batch requests are processed within 24 hours and cost 50% less than synchronous requests, making them ideal for bulk processing, data pipelines, and offline workflows.

Why Batch Processing?

• 50% cost savings: Batch requests are half the price of synchronous requests
• Higher throughput: No rate limiting for batch submissions
• Reliability: Automatic retries and fault tolerance
• Scalability: Process thousands of requests without managing concurrency

When to Use Batch API

Good use cases:

• Bulk content generation (product descriptions, summaries)
• Data enrichment pipelines
• Offline analysis and classification
• Translation at scale
• Evaluation and testing across many inputs

Not ideal for:

• Real-time applications requiring immediate responses
• Interactive chat interfaces
• Time-sensitive operations

Note

Batch processing can take up to 24 hours to complete. For time-sensitive work, use synchronous requests with appropriate rate limiting.

Basic Usage

Check Provider Support

Not all providers support the Batch API. Use core.AsBatchProvider() to check:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/petal-labs/iris/core"
    "github.com/petal-labs/iris/providers/openai"
)

func main() {
    provider := openai.New(os.Getenv("OPENAI_API_KEY"))

    // Check if provider supports batch API
    bp, ok := core.AsBatchProvider(provider)
    if !ok {
        fmt.Println("Provider does not support Batch API")
        return
    }

    fmt.Printf("Batch API available for: %s\n", provider.ID())
    // Use bp for batch operations...
}

Submit a Batch

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/petal-labs/iris/core"
    "github.com/petal-labs/iris/providers/openai"
)

func main() {
    provider := openai.New(os.Getenv("OPENAI_API_KEY"))
    ctx := context.Background()

    bp, ok := core.AsBatchProvider(provider)
    if !ok {
        panic("provider does not support batch API")
    }

    // Create batch requests
    requests := []core.BatchRequest{
        {
            CustomID: "translate-1",
            Request: core.ChatRequest{
                Model: "gpt-4o-mini",
                Messages: []core.Message{
                    {Role: core.RoleSystem, Content: "Translate to French."},
                    {Role: core.RoleUser, Content: "Hello, how are you?"},
                },
            },
        },
        {
            CustomID: "translate-2",
            Request: core.ChatRequest{
                Model: "gpt-4o-mini",
                Messages: []core.Message{
                    {Role: core.RoleSystem, Content: "Translate to Spanish."},
                    {Role: core.RoleUser, Content: "Good morning!"},
                },
            },
        },
        {
            CustomID: "translate-3",
            Request: core.ChatRequest{
                Model: "gpt-4o-mini",
                Messages: []core.Message{
                    {Role: core.RoleSystem, Content: "Translate to German."},
                    {Role: core.RoleUser, Content: "Thank you very much."},
                },
            },
        },
    }

    // Submit the batch
    batchID, err := bp.CreateBatch(ctx, requests)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Batch submitted: %s\n", batchID)
    fmt.Println("Check status with: bp.GetBatchStatus(ctx, batchID)")
}

Check Batch Status

info, err := bp.GetBatchStatus(ctx, batchID)
if err != nil {
    panic(err)
}

fmt.Printf("Status: %s\n", info.Status)
fmt.Printf("Progress: %d/%d completed, %d failed\n",
    info.Completed, info.Total, info.Failed)

// Check if complete
if info.IsComplete() {
    fmt.Println("Batch is complete!")
}

Retrieve Results

if info.Status == core.BatchStatusCompleted {
    results, err := bp.GetBatchResults(ctx, batchID)
    if err != nil {
        panic(err)
    }

    for _, result := range results {
        if result.IsSuccess() {
            fmt.Printf("%s: %s\n", result.CustomID, result.Response.Output)
        } else {
            fmt.Printf("%s: ERROR - %s\n", result.CustomID, result.Error.Message)
        }
    }
}

BatchWaiter Utility

For convenience, use BatchWaiter to automatically poll for completion:

package main

import (
    "context"
    "fmt"
    "os"
    "time"

    "github.com/petal-labs/iris/core"
    "github.com/petal-labs/iris/providers/openai"
)

func main() {
    provider := openai.New(os.Getenv("OPENAI_API_KEY"))
    ctx := context.Background()

    bp, _ := core.AsBatchProvider(provider)

    // Submit batch
    requests := []core.BatchRequest{
        // ... your requests
    }
    batchID, _ := bp.CreateBatch(ctx, requests)

    // Create waiter with custom poll interval
    waiter := core.NewBatchWaiter(bp).
        WithPollInterval(30 * time.Second).
        WithMaxWait(2 * time.Hour)

    // Wait for completion
    info, err := waiter.Wait(ctx, batchID)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Batch completed: %d/%d succeeded\n", info.Completed, info.Total)

    // Get results
    results, _ := bp.GetBatchResults(ctx, batchID)
    for _, r := range results {
        fmt.Printf("%s: %s\n", r.CustomID, r.Response.Output)
    }
}

WaitAndCollect Shortcut

For the common pattern of waiting and then collecting results:

waiter := core.NewBatchWaiter(bp).WithPollInterval(30 * time.Second)

// Wait and collect results in one call
results, err := waiter.WaitAndCollect(ctx, batchID)
if err != nil {
    panic(err)
}

for _, result := range results {
    fmt.Printf("%s: %s\n", result.CustomID, result.Response.Output)
}

Batch Status Values

Status	Description
BatchStatusPending	Batch submitted, waiting to start
BatchStatusInProgress	Batch is being processed
BatchStatusCompleted	All requests completed successfully
BatchStatusFailed	Batch failed (check error details)
BatchStatusCancelled	Batch was cancelled
BatchStatusExpired	Batch expired before completion

Managing Batches

List Recent Batches

batches, err := bp.ListBatches(ctx, 10) // Get last 10 batches
if err != nil {
    panic(err)
}

for _, b := range batches {
    fmt.Printf("%s: %s (%d/%d completed)\n",
        b.ID, b.Status, b.Completed, b.Total)
}

Cancel a Batch

err := bp.CancelBatch(ctx, batchID)
if err != nil {
    fmt.Printf("Failed to cancel: %v\n", err)
} else {
    fmt.Println("Batch cancelled")
}

Error Handling

Batch-Level Errors

info, err := bp.GetBatchStatus(ctx, batchID)
if err != nil {
    var notFoundErr *core.NotFoundError
    if errors.As(err, &notFoundErr) {
        fmt.Println("Batch not found - it may have expired")
    }
    return
}

if info.Status == core.BatchStatusFailed {
    fmt.Printf("Batch failed: %s\n", info.Error)
}

Request-Level Errors

Individual requests in a batch can fail while others succeed:

results, _ := bp.GetBatchResults(ctx, batchID)

var succeeded, failed int
for _, result := range results {
    if result.IsSuccess() {
        succeeded++
        // Process successful result
        processResult(result.CustomID, result.Response)
    } else {
        failed++
        // Log failed request for retry
        logFailure(result.CustomID, result.Error)
    }
}

fmt.Printf("Results: %d succeeded, %d failed\n", succeeded, failed)

Best Practices

1. Use Meaningful Custom IDs

Custom IDs help you match results to your source data:

requests := []core.BatchRequest{
    {
        CustomID: fmt.Sprintf("doc-%s-summary", documentID),
        Request:  buildSummaryRequest(document),
    },
}

2. Handle Partial Failures

Always check individual result status, even for completed batches:

results, _ := bp.GetBatchResults(ctx, batchID)

var retryRequests []core.BatchRequest
for _, result := range results {
    if !result.IsSuccess() {
        // Collect failed requests for retry
        retryRequests = append(retryRequests, originalRequests[result.CustomID])
    }
}

if len(retryRequests) > 0 {
    retryBatchID, _ := bp.CreateBatch(ctx, retryRequests)
    fmt.Printf("Retrying %d failed requests: %s\n", len(retryRequests), retryBatchID)
}

3. Set Appropriate Timeouts

For long-running batches, use context with timeout:

// Wait up to 4 hours for batch completion
ctx, cancel := context.WithTimeout(context.Background(), 4*time.Hour)
defer cancel()

waiter := core.NewBatchWaiter(bp).WithPollInterval(1 * time.Minute)
results, err := waiter.WaitAndCollect(ctx, batchID)
if errors.Is(err, context.DeadlineExceeded) {
    fmt.Println("Batch is still processing, check back later")
    fmt.Printf("Batch ID: %s\n", batchID)
}

4. Store Batch IDs

For production systems, persist batch IDs to handle restarts:

// Submit batch and store ID
batchID, err := bp.CreateBatch(ctx, requests)
if err != nil {
    return err
}

// Store in database for later retrieval
db.SaveBatchJob(batchID, "processing", time.Now())

// Later, in a separate process or after restart
batchID := db.GetPendingBatchID()
results, err := waiter.WaitAndCollect(ctx, batchID)

5. Batch Size Recommendations

Scenario	Recommended Batch Size
Simple completions	1,000 - 10,000
Complex reasoning	100 - 1,000
With tools	100 - 500
Long context	50 - 200

Provider Support

Provider	Batch API
OpenAI	✓
Anthropic	-
Gemini	-
xAI	-
Ollama	-

Note

Batch API support varies by provider. Check provider documentation for specific limitations and quotas.

Next Steps

Structured Output

Combine batching with JSON Schema. Structured Output →

Testing Guide

Mock batch operations in tests. Testing →

Examples

See batch processing examples. Examples →

API Reference

Full BatchProvider API. API →