Windsurf prompt
Windsurf: Respectful Web Scraper
Web scraper with rate limiting, retry logic, robots.txt compliance, and structured data export.
Build a web scraper for {{scraper_name}} that collects {{data_to_collect}} from {{target_site}}. ## Target: {{target_site}} ## Data to collect: {{data_to_collect}} ## Output format: {{output_format}} ### Ethical Scraping Principles 1. Read and respect robots.txt 2. Identify yourself in User-Agent header 3. Rate limit requests: {{requests_per_second}} req/s max 4. Cache pages to avoid re-fetching 5. Do not overload the server ### Tech Stack - HTTP: httpx (async) or requests - Parsing: BeautifulSoup4 or Playwright (for JS-rendered pages) - Storage: SQLite for intermediate, {{output_format}} for final export - Scheduling: simple script for one-time, APScheduler for recurring ### Implementation **Configuration** ```python BASE_URL = "{{target_site}}" RATE_LIMIT = {{requests_per_second}} # requests per second RETRY_MAX = 3 RETRY_BACKOFF = 2 # seconds between retries USER_AGENT = "{{scraper_name}}/1.0 (+contact@yoursite.com)" ``` **robots.txt Check** Use urllib.robotparser to check before scraping each URL. Skip disallowed paths entirely. **Request Handler** Rate limiting via asyncio sleep or token bucket. Retry on: 429, 503, connection errors. Cache responses in SQLite to avoid re-fetching. **Parsing** CSS selectors or XPath for each {{data_to_collect}} field. Handle: missing fields (null), encoding issues, pagination. **Storage** Intermediate: SQLite table with scraped fields + url + scraped_at. Final export: {{output_format}} with clean, deduplicated data. **Output** Progress bar (tqdm). Summary: pages scraped, records collected, errors.
Fill in: {{scraper_name}}, {{data_to_collect}}, {{target_site}}, {{output_format}}, {{requests_per_second}}. In Loopy you fill these in a form and copy the finished prompt in one click.
More automation prompts
- CursorCursor: Create GitHub Actions CI PipelineComplete GitHub Actions CI/CD pipeline with parallel jobs for testing, linting, type checking, security scanning, and deployment.
- n8nn8n: Content Publishing PipelineWrite → review → approve → publish → announce automation across platforms.
- n8nn8n: Customer Onboarding SequenceSign up → welcome → email sequence → task creation → check-in automation workflow.
- n8nn8n: Error Alerting SystemMonitor → detect → alert → create ticket workflow for system error management.
- n8nn8n: Lead Capture to CRM AutomationWebhook → lead enrichment → CRM creation → team notification workflow.
- n8nn8n: Social Media SchedulerContent calendar → format per platform → schedule → post → track engagement workflow.

