Windsurf prompt

Windsurf: Respectful Web Scraper

Web scraper with rate limiting, retry logic, robots.txt compliance, and structured data export.

Build a web scraper for {{scraper_name}} that collects {{data_to_collect}} from {{target_site}}.

## Target: {{target_site}}
## Data to collect: {{data_to_collect}}
## Output format: {{output_format}}

### Ethical Scraping Principles
1. Read and respect robots.txt
2. Identify yourself in User-Agent header
3. Rate limit requests: {{requests_per_second}} req/s max
4. Cache pages to avoid re-fetching
5. Do not overload the server

### Tech Stack
- HTTP: httpx (async) or requests
- Parsing: BeautifulSoup4 or Playwright (for JS-rendered pages)
- Storage: SQLite for intermediate, {{output_format}} for final export
- Scheduling: simple script for one-time, APScheduler for recurring

### Implementation

**Configuration**
```python
BASE_URL = "{{target_site}}"
RATE_LIMIT = {{requests_per_second}}  # requests per second
RETRY_MAX = 3
RETRY_BACKOFF = 2  # seconds between retries
USER_AGENT = "{{scraper_name}}/1.0 (+contact@yoursite.com)"
```

**robots.txt Check**
Use urllib.robotparser to check before scraping each URL.
Skip disallowed paths entirely.

**Request Handler**
Rate limiting via asyncio sleep or token bucket.
Retry on: 429, 503, connection errors.
Cache responses in SQLite to avoid re-fetching.

**Parsing**
CSS selectors or XPath for each {{data_to_collect}} field.
Handle: missing fields (null), encoding issues, pagination.

**Storage**
Intermediate: SQLite table with scraped fields + url + scraped_at.
Final export: {{output_format}} with clean, deduplicated data.

**Output**
Progress bar (tqdm).
Summary: pages scraped, records collected, errors.

Fill in: {{scraper_name}}, {{data_to_collect}}, {{target_site}}, {{output_format}}, {{requests_per_second}}. In Loopy you fill these in a form and copy the finished prompt in one click.