Logging Configuration

Overview

Sensyze Dataflow uses a centralized logging configuration with service-specific controls.

Environment Variables

Global Logging

LOG_LEVEL=INFO                           # Global level

Service-Specific Logging

LOG_LEVEL_API=INFO                       # API-specific
LOG_LEVEL_PIPELINE_RUNNER=INFO           # Pipeline-specific
LOG_LEVEL_WORKER=INFO                    # Temporal worker

Log Levels

Level	Description
DEBUG	Detailed information for debugging
INFO	General informational messages
WARNING	Warning messages
ERROR	Error messages
CRITICAL	Critical issues

Log Directory

Logs are written to /app/logs/ (mounted volume).

Structured Logging

The system uses structured JSON logging for better parsing:

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "INFO",
  "logger": "pipeline_runner",
  "message": "Pipeline execution started",
  "run_id": "run-abc-123",
  "pipeline_id": "pipe-xyz-456"
}

Best Practices

1. Use DEBUG for development/troubleshooting
2. Use INFO for production
3. Use WARNING or higher for specific noisy components

Integration with Observability

Pipeline execution logs are integrated with the observability system:

• Use obs_logger for pipeline-related logging
• Use standard logging for API/infrastructure