> ## Documentation Index
> Fetch the complete documentation index at: https://docs.evomarketing.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Scraping

> ScrapeCreators integration for social media data

EVO Dialed uses **ScrapeCreators** as the primary service for scraping social media data.

## Overview

ScrapeCreators provides:

* Profile data (handle, followers, bio)
* Post/video lists
* Engagement metrics
* Media URLs

## Cost model

ScrapeCreators bills per request:

| Pack       | Cost    | Credit cost        |
| ---------- | ------- | ------------------ |
| \$47 pack  | Default | \$1.88/1K requests |
| \$497 pack | Bulk    | \$0.99/1K requests |

### Facebook billing

Facebook charges per-post (not per-profile):

* Each post = 1 request
* High-volume creators = higher cost

## Sync schedule

### Nightly sync

* **Time:** 10pm ET daily
* **Scope:** Full roster (\~1,149 accounts)
* **Job:** `Metrics::SyncJob`

<Warning>
  Avoid deploying between 10pm-10:15pm ET to prevent sync interruption.
</Warning>

### Manual sync

Individual accounts can be synced manually:

* 24-hour throttle per account
* Prevents excessive API usage

## Configuration

### Environment variables

| Variable                        | Purpose                           |
| ------------------------------- | --------------------------------- |
| `SCRAPE_CREATORS_API_KEY`       | API key (single key)              |
| `SCRAPE_SYNC_PAUSED`            | Kill switch (set to `1` to pause) |
| `SCRAPE_SYNC_SLACK_WEBHOOK_URL` | Slack alerts webhook              |

<Note>
  Both `SCRAPE_CREATORS_API_KEY` vars hold the same key since July 2026. The old primary key was retired.
</Note>

## API client

**File:** `app/services/integrations/scrape_creators_client.rb`

```ruby theme={null}
client = Integrations::ScrapeCreatorsClient.new
posts = client.fetch_posts(handle: "@creator", platform: "tt")
```

### Methods

| Method          | Purpose          |
| --------------- | ---------------- |
| `fetch_profile` | Get profile data |
| `fetch_posts`   | Get recent posts |
| `fetch_metrics` | Get post metrics |

## Sync flow

```
Metrics::SyncJob
       │
       ▼
CreatorSocial.syncable
       │
       ▼
ScrapeCreatorsClient.fetch_posts
       │
       ▼
Metrics::Ingestor.ingest_for_social
       │
       ▼
CampaignSubmission records updated
```

## Health monitoring

### SyncWatchdogJob

Runs every 15 minutes:

* Checks for stuck syncs
* Monitors completion rates
* Alerts on failures

### CreditsAlert

Monitors API credit balance:

* Alerts when credits low
* Re-arms on recovery
* Pings Rafael + Brandon

### Slack reporting

`Metrics::SyncSlackReporter` posts to Slack:

* Sync start notification
* Completion summary
* Error counts

## Error handling

### Transient errors

* Network timeouts: Auto-retry
* Rate limits: Backoff and retry
* API errors: Log and continue

### Persistent errors

* Account banned: Mark `sync_failed`
* Invalid handle: Mark `sync_failed`
* Auth issues: Require re-auth

## Kill switch

Emergency pause all syncs:

```bash theme={null}
# Set environment variable
SCRAPE_SYNC_PAUSED=1
```

Use when:

* API outage detected
* Cost overrun risk
* Data quality issues

## Troubleshooting

### Common issues

| Issue           | Check                   |
| --------------- | ----------------------- |
| No posts synced | Brand links exist?      |
| Sync failed     | API key valid?          |
| Missing metrics | Post within date range? |
| High costs      | FB post volume?         |

### Debug sync

```ruby theme={null}
# Console commands
social = CreatorSocial.find(id)
client = Integrations::ScrapeCreatorsClient.new
posts = client.fetch_posts(handle: social.handle, platform: social.platform)
```

## Backend implementation

**Key files:**

* `app/services/integrations/scrape_creators_client.rb`
* `app/jobs/metrics/sync_job.rb`
* `app/services/metrics/sync_slack_reporter.rb`
* `app/services/metrics/credits_alert.rb`