Scraping

​

Overview

• Profile data (handle, followers, bio)
• Post/video lists
• Engagement metrics
• Media URLs

​

Cost model

​

Facebook billing

• Each post = 1 request
• High-volume creators = higher cost

​

Sync schedule

​

Nightly sync

• Time: 10pm ET daily
• Scope: Full roster (~1,149 accounts)
• Job: Metrics::SyncJob

​

Manual sync

• 24-hour throttle per account
• Prevents excessive API usage

​

Configuration

​

Environment variables

​

API client

client = Integrations::ScrapeCreatorsClient.new
posts = client.fetch_posts(handle: "@creator", platform: "tt")

​

Methods

​

Sync flow

Metrics::SyncJob
       │
       ▼
CreatorSocial.syncable
       │
       ▼
ScrapeCreatorsClient.fetch_posts
       │
       ▼
Metrics::Ingestor.ingest_for_social
       │
       ▼
CampaignSubmission records updated

​

Health monitoring

​

SyncWatchdogJob

• Checks for stuck syncs
• Monitors completion rates
• Alerts on failures

​

CreditsAlert

• Alerts when credits low
• Re-arms on recovery
• Pings Rafael + Brandon

​

Slack reporting

• Sync start notification
• Completion summary
• Error counts

​

Error handling

​

Transient errors

• Network timeouts: Auto-retry
• Rate limits: Backoff and retry
• API errors: Log and continue

​

Persistent errors

• Account banned: Mark sync_failed
• Invalid handle: Mark sync_failed
• Auth issues: Require re-auth

​

Kill switch

# Set environment variable
SCRAPE_SYNC_PAUSED=1

• API outage detected
• Cost overrun risk
• Data quality issues

​

Troubleshooting

​

Common issues

​

Debug sync

# Console commands
social = CreatorSocial.find(id)
client = Integrations::ScrapeCreatorsClient.new
posts = client.fetch_posts(handle: social.handle, platform: social.platform)

​

Backend implementation

• app/services/integrations/scrape_creators_client.rb
• app/jobs/metrics/sync_job.rb
• app/services/metrics/sync_slack_reporter.rb
• app/services/metrics/credits_alert.rb