Designing a Scalable Dashboard Architecture for High‑Volume Data

A dashboard that works with 10 routes will collapse under 500. Here's how we architected the Ztoolx Analytics platform to stay snappy even with years of historical data.

Layer 1: The Aggregation Service

Never query raw transaction data from the dashboard. We built a Node.js aggregation worker that runs nightly, computing daily summaries by route, by driver, and by cost center. Those summaries are stored in a simple PostgreSQL table. The dashboard only ever queries pre‑aggregated data.

Layer 2: Materialized Views for Speed

For common queries like "total deliveries this month," we use database materialized views. They're essentially cached query results that refresh every hour. The dashboard hits these views directly, returning results in under 10ms.

Layer 3: Client‑Side Data Limiting

Even with pre‑aggregated data, don't load everything into the browser's memory. Our dashboards implement pagination and "load‑on‑scroll" for tables. Charts use data sampling: if a user requests 5 years of daily data, the server returns weekly averages instead, because 5 years × 365 points is impossible to render smoothly.

Layer 4: Caching with CDN

If the aggregated data doesn't change until the next nightly run, why recalculate it for every user? We cache API responses in a CDN (via Vercel's Edge Cache) with a TTL of 1 hour. This means that after the first user loads the dashboard, every subsequent user gets a near‑instant response.