Skip to main content

Kubernetes Deployment

Overview​

Deploy Sensyze Dataflow to Kubernetes for production workloads.

Prerequisites​

  • Kubernetes cluster (1.24+)
  • Helm 3
  • kubectl configured

Quick Start​

Install Helm Chart​

helm repo add sensyze https://charts.sensyze.com
helm install dataflow sensyze/dataflow

Configuration​

# values.yaml
replicaCount: 3

env:
  SUPABASE_DATABASE_URL: postgresql://...
  TEMPORAL_HOST: temporal:7233
  REDIS_HOST: redis:6379

ingress:
  enabled: true
  host: dataflow.yourdomain.com

Components​

Backend Deployment​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dataflow-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: dataflow-server
  template:
    spec:
      containers:
        - name: server
          image: sensyze/dataflow:latest
          ports:
            - containerPort: 8000
          env:
            - name: SUPABASE_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: sensyze-secrets
                  key: supabase-url

Temporal Deployment​

# Install Temporal operator
kubectl apply -f https://github.com/temporalio/operator/releases/latest/download/crds.yaml
kubectl apply -f https://github.com/temporalio/operator/releases/latest/download/temporal.yaml

Redis Cache​

apiVersion: v1
kind: Pod
metadata:
  name: redis
spec:
  containers:
    - name: redis
      image: redis:7-alpine
      ports:
        - containerPort: 6379

Scaling​

Horizontal Pod Autoscaling​

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: dataflow-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: dataflow-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Monitoring​

Prometheus scraping​

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8000"
  prometheus.io/path: "/metrics"

Log aggregation​

kubectl logs -f deployment/dataflow-server

Troubleshooting​

Pods not starting​

kubectl describe pod <pod-name>
kubectl logs <pod-name>

Connection issues​

  • Check services: kubectl get svc
  • Check endpoints: kubectl get endpoints
  • Test connectivity: kubectl exec -it <pod> -- curl temporal:7233