Case Study · Data EngineeringFintech / Market Data

Every bank card deal in Pakistan, kept fresh every day

The card offers that power CardWise and CardIntel change constantly — new merchants appear, discounts shift, deals expire. This is the engine that keeps up: an incremental sync that reverse-engineers the deals platform seven banks embed, and lands thousands of clean, current offers in a Supabase warehouse without re-fetching the whole country every night.

Arham Mirkar

DataLayer — Enterprise Data Infrastructure

Banks synced from one pipeline

9,500+

Live card offers maintained

1,200+

Merchants normalized

Cities covered nationwide

The Problem

Fresh data is the whole product

A deals app is only as good as its last sync. Show a discount that expired last week and you've broken trust; miss a merchant a competitor already lists and you look incomplete. Across seven banks and thirty-three cities, that's tens of thousands of offers that each have their own lifecycle.

The naive fix — re-download everything, every day — is slow, wasteful, and hammers the source. The real problem was building a sync that's incremental: fetch only what changed, update in place, expire what vanished, and never lose the thread if a long run gets interrupted halfway through.

An earlier version of this project scraped bank pages with an LLM-powered browser extension. It was replaced: reading the structured API the portals already speak is faster, cheaper, and far more reliable than asking a model to read a webpage.

The System

Six stages, run every day

The design principle is simple: do the least work that keeps the data correct. Almost every stage exists to avoid re-doing work that hasn't changed.

Discover

SDK reverse-engineering

Each bank embeds Peekaboo Guru's deals portal in an iframe. The pipeline talks to that same private SDK API directly — carrying the per-bank owner key and IFRAME headers so every request looks exactly like the bank's own embed.

Fetch

Paginated JSON

For every bank × city, it pages through the merchant list 100 at a time from one obfuscated endpoint, then pulls each merchant's full deal set from a second — a clean JSON tree instead of scraped HTML.

Detect changes

Signature cache

A local cache fingerprints each merchant (logo, cover, gallery, deal count). If nothing moved since yesterday, the deal fetch is skipped entirely — so a daily run only touches what actually changed.

Normalize

Structure

Roughly a hundred Peekaboo tags collapse into 9 clean categories; merchants and cards are de-duplicated, terms are split out, and branches flattened — turning one vendor blob into related rows.

Load

Upsert on deal URL

Every offer's deal URL is its natural key. New offers insert, changed ones patch in place, and offers that disappeared from the source get expired rather than deleted — so history stays intact.

Refresh intelligence

CardIntel RPCs

Once the raw data lands, the sync fires the CardIntel refresh chain — rankings, merchant intelligence, and dashboard snapshots — so the analytics layer is current the moment the sync finishes.

Deep Dive 01

Speaking the portal's own language

The banks don't publish an API. They embed a third-party deals portal in an iframe, and that portal talks to a private SDK endpoint. The sync becomes a well-behaved client of that same endpoint — carrying each bank's owner key and the exact headers its iframe sends.

A request that looks native

Obfuscated endpoints, a per-bank owner key, and an IFRAMEmedium make each call indistinguishable from the bank's own embed — no headless browser required.

POST secure-sdk.peekaboo.guru
     /uljin2s3nitoi89njkhklgkj5   # merchants
ownerkey: <per-bank secret>
medium:   IFRAME
origin:   https://hbl-web.peekaboo.guru
referer:  https://hbl-web.peekaboo.guru/

One client, seven banks

A single bank registry drives the whole run. Each entry maps a bank to its subdomain, owner key, and API base — including one bank that lives on a different host entirely.

MeezanAlfalahBank Al HabibFaysalHBLMCBUBL

This is a reverse-engineered client of a private SDK, not an official partner feed — the same discipline used elsewhere in the portfolio, applied to keep a consumer product's core data fresh.

Deep Dive 02

Only sync what actually moved

The difference between a five-minute sync and a five-hour one is knowing what to skip. Three mechanisms keep the daily run cheap and safe to interrupt.

Signature cache

Each merchant is fingerprinted by its logo, cover, gallery, and deal count. Unchanged since yesterday? The deal fetch is skipped and its offers are left untouched.

Resumable runs

Progress is checkpointed by city and page. If a multi-bank run is interrupted, it picks up where it stopped instead of restarting the whole country.

Expire, don't delete

On a full scan, offers whose deal URL no longer appears at the source are marked expired and closed — the data stays for history instead of vanishing.

The payoff: a daily run re-fetches only the merchants that changed, typically a few hundred out of well over a thousand — the rest are served from cache in seconds.

Deep Dive 03

One JSON blob, ten related tables

The source returns a nested blob per merchant. That gets decomposed into a relational schema so the app can filter offers by bank, city, category, and card — and so CardIntel can rank them.

Relational model

bankscitiescategoriescardsmerchantsmerchant_categoriesmerchant_citiestermsoffersoffer_cards

Merchants link to categories and cities through junction tables; offers hang off a merchant, bank, city, and terms; and each offer connects to the specific cards it applies to. Around a hundred raw source tags map down to nine clean, queryable categories.

Upsert by URL

Each offer's deal URL is a unique key. Re-runs patch existing offers rather than duplicating them.

CREATE UNIQUE INDEX
  idx_offers_card_deal_url
  ON offers (card_deal_url);

-- seen  → PATCH
-- new   → INSERT
-- gone  → is_expired = true

Downstream

What this pipeline feeds

This sync is the quiet layer underneath two products. Everything they show a user traces back to a row this pipeline kept fresh.

CardWise

The consumer app surfaces these offers to shoppers by city, bank, and card — served from the clean tables this pipeline maintains, so users never see an expired deal.

See the case study

CardIntel

The B2B intelligence layer ranks cards, benchmarks discounts, and finds merchant gaps — all rebuilt automatically at the end of every sync from the same fresh data.

Explore CardIntel

Arham Mirkar

Founder & Data Engineer, DataLayer

“My first version of this had an LLM read each bank's webpage and guess the structure. It worked in a demo and broke in production. The rewrite does the opposite — it speaks the structured API the portal already uses, and spends all its cleverness on notre-doing work. Boring, incremental, and it just keeps running.”

Under the Hood

Technology

Layer	Approach
Source	Reverse-engineered Peekaboo Guru bank SDK API, per-bank owner-key auth, IFRAME-medium headers
Ingestion	Python 3, requests, city-paginated entity + deal fetch, 3× retry with exponential backoff
Change detection	File-based signature cache, resumable run state, full-scan stale-offer expiry
Normalization	~100-tag → 9-category map, merchant + card de-duplication, terms dedup, branch flattening
Storage	Supabase (PostgreSQL) via PostgREST, unique-URL upserts, many-to-many junction tables
Analytics refresh	Post-sync CardIntel RPCs — rankings, merchant intel, dashboard snapshots

Need data that stays current on its own?

A dataset is a liability the day it goes stale. We build the incremental syncs that keep it alive — fetch what changed, update in place, and never fall over on a re-run.

Build a data pipeline More case studies