REST Integration

Overview

Connect to REST APIs as sources or transformations in your pipelines.

REST Source Node

{
  "id": "rest-source",
  "type": "rest",
  "data": {
    "url": "https://api.example.com/users",
    "method": "GET",
    "headers": {
      "Authorization": "Bearer {{secrets.API_KEY}}",
      "Content-Type": "application/json"
    },
    "params": {
      "page": 1,
      "limit": 100
    }
  }
}

REST Transform Node

{
  "id": "rest-transform",
  "type": "rest",
  "data": {
    "url": "https://api.example.com/enrich",
    "method": "POST",
    "headers": {
      "Content-Type": "application/json"
    },
    "request_format": "json",
    "input_field": "data"
  }
}

Configuration Options

Option	Type	Description
url	string	API endpoint URL
method	string	HTTP method (GET, POST, PUT, DELETE)
headers	object	Request headers
params	object	Query parameters
body	object	Request body (for POST/PUT)
timeout	number	Request timeout in seconds (default: 30)
retry_count	number	Number of retries on failure (default: 3)

Authentication

Bearer Token

{
  "headers": {
    "Authorization": "Bearer {{secrets.TOKEN}}"
  }
}

API Key

{
  "headers": {
    "X-API-Key": "{{secrets.API_KEY}}"
  }
}

Basic Auth

{
  "headers": {
    "Authorization": "Basic {{secrets.BASIC_AUTH}}"
  }
}

Pagination

Offset-based

{
  "pagination": {
    "type": "offset",
    "limit_param": "limit",
    "offset_param": "offset",
    "max_pages": 10
  }
}

Cursor-based

{
  "pagination": {
    "type": "cursor",
    "cursor_param": "cursor",
    "next_cursor_path": "response.next_cursor"
  }
}

Error Handling

{
  "error_handling": {
    "on_error": "continue",  // or "fail"
    "retry_status_codes": [429, 500, 502, 503, 504],
    "timeout_seconds": 30
  }
}