Skip to main content

Airbyte Integration

Overview​

Airbyte connectors enable data ingestion from 300+ sources and destinations.

Architecture​

  • Managers: AirbyteConnectorManager handles connector lifecycle
  • Specs: Dynamic generation of configuration forms based on Airbyte specifications
  • Secrets: Integrated with Supabase for secure credential storage

Configuration​

Add Connection​

  1. Go to Settings > Airbyte Connections
  2. Click "Add Source" or "Add Destination"
  3. Select connector type
  4. Enter configuration
  5. Test connection
  6. Save

Using in Pipelines​

{
  "id": "airbyte-source",
  "type": "airbyte",
  "data": {
    "connection_id": "uuid-of-airbyte-connection",
    "stream": "users",
    "sync_mode": "full_refresh",
    "destination": "internal_staging"
  }
}

Supported Connectors​

Sources​

  • Database (PostgreSQL, MySQL, Oracle)
  • SaaS (Salesforce, HubSpot, Stripe)
  • File (SFTP, Google Sheets)
  • CDC (Debezium-based)

Destinations​

  • Database (PostgreSQL, Snowflake, BigQuery)
  • Data Warehouse (Redshift, Databricks)
  • File (S3, GCS)

Sync Modes​

ModeDescription
full_refreshRead entire dataset
incrementalOnly new/updated records
cdcChange data capture

Troubleshooting​

Connection fails​

  1. Check credentials are correct
  2. Verify network access
  3. Check Airbyte logs

Sync stalls​

  1. Check connector state
  2. Verify destination has space
  3. Review rate limits