Windsurf prompt

Windsurf: Python ETL Data Pipeline

Extract, transform, and load pipeline with pandas, data validation, error handling, and scheduling.

Build a Python ETL data pipeline for {{pipeline_name}}.

## Source: {{data_source}}
## Destination: {{data_destination}}
## Schedule: {{schedule}}
## Data volume: {{data_volume}}

### Pipeline Architecture

**Extract**
Connect to {{data_source}}: {{connection_method}}.
Handle pagination or incremental extraction (only fetch new/changed records).
Retry logic: exponential backoff on connection failures (3 attempts, 1s/4s/16s).
Log all extraction statistics: records fetched, duration, errors.

**Transform (pandas)**
Load raw data into DataFrame.
Data validation: schema check (column names + types), null checks on required fields, range/format validation.
Cleaning: strip whitespace, normalize case, parse dates with explicit format, handle nulls.
Transformations specific to {{pipeline_name}}: {{transformations}}.
Create audit columns: processed_at, source, pipeline_version.

**Load**
Write to {{data_destination}}.
Upsert logic: update existing records, insert new ones (use merge/ON CONFLICT).
Transaction wrapping: all-or-nothing per batch.
Batch size: {{batch_size}} records per write.

**Error Handling**
Failed records go to dead letter queue (separate table or file).
Log error details: record, error type, timestamp.
Send alert on failure: {{alert_method}}.
Pipeline continues processing remaining records after single-record failures.

**Scheduling and Orchestration**
{{schedule}} schedule using APScheduler or Airflow DAG.
Idempotent runs: safe to re-run without duplicates.
Run log with: start time, end time, records processed, status.

**Configuration**
All connection strings and secrets via environment variables.
Pipeline config in YAML file (not code).
Logging to stdout in JSON format for log aggregation.

Fill in: {{pipeline_name}}, {{data_source}}, {{data_destination}}, {{schedule}}, {{data_volume}}, {{connection_method}}, {{transformations}}, {{batch_size}}, {{alert_method}}. In Loopy you fill these in a form and copy the finished prompt in one click.