40 hours of manual CRM reporting a month, replaced by one pipeline
A marketing team was running 500+ campaigns a month on WebEngage — a platform with no API for the content and reports they needed. We reverse-engineered the dashboard's own endpoints, intercepted its report emails, and built a scheduled pipeline that lands clean, categorized data in a warehouse. What took a skilled analyst two working days a week now runs unattended.
Arham Mirkar
DataLayer — Enterprise Data Infrastructure
A reporting process that ran on a human
WebEngage is a powerful CRM, but it exposes no public API for campaign content or for the bulk stats export the team lived in. So every reporting cycle, someone repeated the same manual loop by hand — across email, push, WhatsApp, and web push:
- 1Log into the WebEngage dashboard and open the campaign stats report builder
- 2Request an “All Campaigns” export for the date range and wait for it to generate
- 3Watch an inbox for the report-ready email from noreply@webengage.com
- 4Click the tracking link, download the ZIP, and unzip the CSV
- 5Open each campaign one-by-one to read subject lines, push copy, and CTAs
- 6Hand-map every journey name to a business category in a spreadsheet
- 7Stitch it all into the weekly and monthly reports leadership expects
Documented at roughly 40–50 analyst hours a month across 500+ campaigns — slow, error-prone, and impossible to trust for month-over-month analysis.
One trigger, seven stages, zero clicks
The pipeline reproduces every step a human took — then removes the human. The hardest part isn't any single call; it's that WebEngage hands the report back over email, so the system has to bridge an API, an inbox, and cloud storage without ever breaking.
Trigger
SchedulerWindows Task Scheduler fires the pipeline weekly (Monday 09:00) and monthly (1st, 09:00). The same entrypoint is exposed to a CLI, a Flask endpoint, an optional n8n bridge, and the AI agent — one code path, four ways to start it.
Authenticate
Session cookiesWebEngage has no public content API, so the pipeline authenticates the way the dashboard does — with session cookies (WebKlipperAuth, _we_a_ssid, WeAuth, GCILB). Cookies are acquired four ways and stored encrypted.
Generate report
Dashboard APIA POST to the reverse-engineered /campaigns/stats/generate-report endpoint enqueues an “All Campaigns / DAILY” export server-side — the exact call the dashboard makes when a human clicks the button.
Intercept the email
IMAP side-channelWebEngage delivers the finished report by email, not by API response. The pipeline polls an inbox over IMAP every 30s (up to 15 min), gated on freshness so it never grabs a stale report.
Unwrap the download
Base64 + GCSThe email only contains a click-tracking link. The pipeline decodes its base64 payload to recover the real Google Cloud Storage URL, then streams and unzips the CSV — no browser, no clicking.
Transform
Categorize + parseEvery journey is normalized and mapped to a business category via a 79-rule taxonomy. Email HTML is parsed for body copy and CTAs; push payloads are flattened across Android and iOS.
Load
WarehouseRows land in BigQuery with a delete-then-append window so re-runs never double-count, and campaign content is upserted into a Supabase repository that the whole stack can query.
API archaeology
With no public API to call, the endpoints had to be recovered from the dashboard's own network traffic and turned into a private, stable SDK. Each call sends the browser-like headers and session cookies WebEngage expects, so the server can't tell the difference between the pipeline and a logged-in analyst.
| Method | Reverse-engineered endpoint | Purpose |
|---|---|---|
| POST | /campaigns/stats/generate-report | Enqueue an All-Campaigns export for a date range |
| GET | /emails/{id}?fetchAllVar=true | Pull full email content and every variation |
| GET | /push-notifications/{id} | Pull push copy, deep links, and payloads |
| GET | /whatsapp-messages/{id} | Pull WhatsApp template content |
| GET | /journeys/{id}/conversion-stats | Pull journey-level conversion performance |
| GET | /api/v2/accounts | Discover which projects a session can access |
The exact payload that enqueues a report — recovered from the dashboard and issued directly:
POST /api/v2/accounts/{account}/campaigns/stats/generate-report
{
"reportConfig": {
"channel": "ALL",
"from": "2026-06-01",
"to": "2026-06-30",
"reportFrequency": "DAILY",
"splitByVariation": false
}
}Auth that survives the real world
Session cookies expire, CAPTCHAs appear, and OTPs interrupt. Instead of one brittle login, the system has four ways to obtain a valid session — and stores every credential encrypted at rest.
A validity probe checks cookies before every run and silently re-authenticates when they lapse, then syncs the fresh session into the scheduled pipeline's config — so unattended runs keep working for weeks without anyone logging in.
The email side-channel
WebEngage doesn't return the report in the API response — it emails a link when the export is ready. So the pipeline treats the inbox as part of the API: it polls over IMAP, verifies the message is fresh, then unwraps a click-tracking link to recover the real download URL.
# Recover the real download URL hidden inside the tracking link
p_value = unquote(p_value)
p_value += "=" * (4 - len(p_value) % 4) % 4 # fix base64 padding
payload = json.loads(base64.b64decode(p_value))
download_url = payload["toURL"] # → GCS report ZIPFrom raw CSV to trustworthy warehouse
Raw exports aren't analysis-ready. The transform layer turns messy CRM naming into clean BI dimensions, and the load layer is built so a re-run is always safe.
From a script to a platform
The automation grew into an internal product — an ops console the team runs day-to-day, a searchable content repository, campaign drafting from a spreadsheet, and an AI assistant that queries the whole warehouse in plain English.
Technology
| Layer | Technologies |
|---|---|
| Automation | Python, requests, Playwright, BeautifulSoup, concurrent.futures |
| Auth & Security | Session-cookie auth, Windows DPAPI + AES-GCM decrypt, Fernet at-rest encryption |
| Email side-channel | imaplib (IMAP SSL), base64 tracking-URL decode, streamed GCS ZIP download |
| Transform | pandas, openpyxl, 79-rule journey taxonomy, Unicode/Arabic header normalization |
| Warehouse | Google BigQuery (pandas-gbq), Supabase / PostgREST content + stats repo |
| Orchestration | Windows Task Scheduler, Flask ops console, optional n8n bridge, ReAct LLM agent |
Arham Mirkar
Founder & Data Engineer, DataLayer
“This one started as a favor to a marketing team drowning in manual exports. The fun part wasn't any single API call — it was making an inbox, a dashboard, and a cloud bucket behave like one reliable system that runs while everyone sleeps. I designed and built the whole thing myself, end to end.”
Is a person your team's reporting bottleneck?
If a workflow has clear rules and eats hours every week, it can almost certainly run unattended. That's what we build.