Using Python for API Data Extraction and Transformation

While our frontend tools run in the browser, our internal analytics rely heavily on Python. It's the glue that pulls data from 15 different APIs, cleans it, and feeds it to our dashboards.

Fetching Data with `requests` and Sessions

We always use a `requests.Session` object with a custom retry adapter. Logistics APIs can be flaky; a three‑retry backoff policy has saved hundreds of manual re‑runs.

Pandas for Transformation

Once we have JSON, it goes straight into a Pandas DataFrame. Operations that would be painful in JavaScript—like merging two DataFrames on an associate ID, then pivoting to show hours per week—are one‑liners in Pandas. We output the final result as a CSV that our Node.js frontend can consume.

Building a Lightweight API with FastAPI

For our Python‑native tools, we expose a FastAPI endpoint that accepts a file upload, runs the transformation, and returns a downloadable Excel file. FastAPI's automatic OpenAPI docs mean our frontend team doesn't even need to ask me what the endpoint expects—they just read the interactive `/docs` page.

Scheduling with Cron and GitHub Actions

Our daily data pipeline runs on a GitHub Actions cron schedule. Every morning at 5 AM UTC, the workflow spins up, pulls the latest data from four APIs, processes it, and commits the cleaned CSV back to a private repo. Then our Vercel deployment redeploys the dashboard with fresh data. Total cost: $0.