A small data service for a UAE fintech client. It collects the EIBOR fixings the UAE central bank publishes each business day, keeps the full history in SQLite and serves it to the client's CRM as a keyed JSON API. It can also push new rates to a webhook as soon as they appear.
The Brief
EIBOR, the Emirates Interbank Offered Rate, is published by the UAE central bank on its website as an interactive chart, not as an API. The client wanted every daily fixing inside their CRM automatically, including the full history, without anyone copying numbers off a web page.
The obstacle was access. The data endpoint sits behind Cloudflare, which returns 403 to plain HTTP clients and to headless browsers alike. The service also had to handle the dull realities of a daily feed: weekends and holidays with no fixing, rates the bank later corrects, and a source page that can change without warning.
What We Built
-
✦
The source behind the chart Instead of scraping rendered HTML, the service reads the endpoint the central bank's EIBOR page loads its own chart from. That endpoint returns every daily fixing since June 2018 across six tenors (overnight, one week, and one, three, six and twelve months) along with value dates. A referer is set so the "last updated" stamp comes back in English.
-
✦
A real browser, out of sight Playwright opens a normal installed Chrome (or Edge) window positioned off-screen, reads the data and closes the window. A full scrape takes about two seconds. A headless mode remains available behind a setting in case the source's protection relaxes.
-
✦
Corrections, not just appends Every run upserts by date: new fixings are stored, and any the bank has corrected are updated in place. Each run is logged with its result. SQLite runs in WAL mode, so the API keeps reading while the scraper writes.
-
✦
A keyed API shaped for a CRM Endpoints cover the latest fixing, date ranges of up to 5,000 rows, a single date with a fallback to the last fixing on or before it for weekends and holidays, whole months and years with per-tenor averages, scrape status and an on-demand refresh. The API key is accepted as a header or a query parameter and compared in constant time, and interactive docs come built in.
-
✦
Push as well as pull An optional webhook posts an update event carrying the latest fixing plus every new and changed one, signed with a shared-secret header and retried three times on failure. The first run loads the whole history and deliberately does not fire it.
-
✦
Runs itself The API process scrapes once on startup and then every 30 minutes by default, which comfortably covers the bank's usual midday publication on business days. The scraper also runs standalone, so it can be scheduled from Windows Task Scheduler without the API.
The Stack
Python throughout, in four small modules for config, database, scraper and API. Playwright drives the browser, FastAPI on Uvicorn serves the API, and SQLite holds the history, with settings read from the environment. The built-in scheduler runs each scrape in a child process with a timeout, one at a time behind a lock, so a hung or crashed browser can never take the API down with it and a manual refresh never collides with a timed run. Because Chrome runs as a real window, the service needs a desktop session: a Windows PC or VM with a logged-in user, or a Linux server under a virtual display. It reaches a cloud CRM through a server or a tunnel, over HTTPS with an API key.
The Outcome
Delivered with a CRM integration guide. To pull, the CRM calls the latest endpoint once a day after the publication window and upserts by date, which is unique, and backfills history year by year from 2018. To push, the client sets a webhook URL instead. Failures surface instead of going quiet: if the source page changes, the status endpoint reports the error and its message, and every rate already stored stays available. The deployment notes also flag that Cloudflare treats data-centre IP addresses more strictly than office connections, with a check to run after any move.
Need something similar? Reach out — we turn data that only exists on a web page into a clean, reliable feed your systems can use.