Skip to main content

Node Types

Source Nodes​

  • Static JSON: Inline JSON data
  • CSV: File upload
  • REST Source: HTTP API calls
  • Database: SQL queries from connections
  • Map Input: Special node for map-reduce sub-pipelines

Transform Nodes​

  • Python: Custom Python code (transform or process functions)
  • SQL: DuckDB SQL queries
  • Filter: Row filtering with conditions
  • Reduce: Aggregations (sum, avg, count, min, max)
  • Map: Map-reduce pattern with sub-pipelines
  • DBT: DBT model execution
  • REST Transform: REST API calls with data

Destination Nodes​

  • File: Export to CSV/JSON/YAML
  • Database: Write to database tables
  • Staging: DuckDB staging tables

Node Configuration​

Python Node​

{
  "id": "python-1",
  "type": "python",
  "data": {
    "code": "def transform(df):\n    return df[df['amount'] > 100]",
    "function_name": "transform"
  }
}

SQL Node​

{
  "id": "sql-1",
  "type": "sql",
  "data": {
    "query": "SELECT * FROM input WHERE amount > 100"
  }
}

Filter Node​

{
  "id": "filter-1",
  "type": "filter",
  "data": {
    "conditions": [
      {"column": "status", "operator": "equals", "value": "active"}
    ]
  }
}

Reduce Node​

{
  "id": "reduce-1",
  "type": "reduce",
  "data": {
    "operations": [
      {"type": "sum", "column": "amount"},
      {"type": "avg", "column": "quantity"}
    ],
    "group_by": ["category"]
  }
}

REST Source Node​

{
  "id": "rest-1",
  "type": "rest",
  "data": {
    "url": "https://api.example.com/data",
    "method": "GET",
    "headers": {
      "Authorization": "Bearer {{secrets.API_KEY}}"
    }
  }
}

Database Source Node​

{
  "id": "db-1",
  "type": "database",
  "data": {
    "connection_id": "uuid-of-connection",
    "query": "SELECT * FROM orders WHERE date > '2024-01-01'"
  }
}

Database Destination Node​

{
  "id": "db-dest-1",
  "type": "database",
  "data": {
    "connection_id": "uuid-of-connection",
    "table": "orders_staging",
    "mode": "append"  // or "replace"
  }
}