Privacy-by-Design in a Healthcare Data Pipeline

A client was sitting on a patient management system full of genuinely useful data, and almost none of it was reachable. PDFs everywhere, no easy way to aggregate anything, and a director who needed regular summary reports that currently meant hours of opening files and copying numbers by hand. They also wanted to use the historical data for research — but you can't hand raw patient records to researchers, and rightly so.

So I built a pipeline that pulled data from their system, parsed the PDFs, generated the director's reports automatically, and produced a de-identified dataset the research side could actually touch. The technically interesting parts are the PDF wrangling and the de-identification. The part that mattered most to the client was getting hours of manual work back every week.

The starting point

The patient management system had an API — not a good one, but a functional one. It returned records as structured data and gave download links for PDF reports. The PDFs were the problem. They held the clinical detail — treatment notes, outcomes — in a layout designed for a human to read, not a machine to parse.

They followed a template, loosely. Labels were consistent; their positions weren't. Some sections appeared on some reports and not others. Tables had variable row counts. That's just the nature of PDFs: the format preserves how something looks, not what it means, and you spend your time reconstructing the meaning.

When ecosystem beats language preference

I started in the language I'd been enjoying most for backend work — single-binary output, good concurrency, a type system that catches a class of mistakes early. The plan was a small tool running on a schedule, and the API integration went exactly as hoped.

Then I hit the PDFs, and that ecosystem's libraries for them were thin. The options I tried extracted lossy text — tables came out as scrambled strings, layout gone — and I noticed I was spending more time fighting the library than solving the problem. That's always the signal to stop and reassess.

The honest options were: push on with a limited library, write better bindings to a mature native PDF library myself, or move to an ecosystem where this was already solved. Writing PDF parsing from scratch wasn't realistic — it's a genuinely nasty format, with multiple text encodings, embedded fonts, and layout constructs. So I moved the pipeline to the ecosystem whose PDF tooling had been beaten into shape against millions of real documents. I had text extraction working within a day and table parsing within two.

Was the language I started with the "better" one in the abstract? In some ways, yes. But I finished in the one that had already solved my hardest sub-problem, and finishing is the job. Ecosystem is an architectural constraint, not a detail — the language you'd prefer matters less than the libraries that already handle the ugliest part of your problem. That's not a compromise I make reluctantly; it's one of the more reliable senior instincts I've developed.

Parsing the PDFs

PDF parsing is more art than science. The tooling hands you text with coordinates; rebuilding the structure is on you. A table isn't marked as a table — it's just text sitting in a suspiciously grid-like pattern. My approach:

1. Extract every text element with its x/y coordinates
2. Group elements into lines by y-position, with tolerance for slight drift
3. Identify section headers by font size or weight
4. Detect table columns by finding x-positions that stay consistent across rows
5. Map the values into a structured schema using the headers and field labels

The messy reality was inconsistency between reporting periods — "Patient Name:" here, "Name:" there, date formats that wandered. I ended up with a small stack of heuristics and fallbacks: try the expected shape first, then the known alternatives. Multi-page reports added another wrinkle — y-coordinates reset each page, so I had to track which table I was inside and keep appending rows across the break. The output was a clean structured record per report — demographics, clinical dates, treatments, outcome codes — and everything downstream built on that.

The director's reports

The director's ask was simple: recent activity, aggregate stats, and anything that looked like an outlier. Previously that meant opening dozens of PDFs and tallying by hand. The generator queries the latest extracted data, aggregates it (counts by treatment type, average durations, outcome distributions), flags outliers (unusual wait times, incomplete records, pending follow-ups), and renders it into a clean report that goes out on a schedule.

Nobody has to remember to run it, and the formatting is identical every time. If something needs a closer look, they click through to the source record.

One small touch I'm quietly proud of: every report carries a "data freshness" timestamp showing when the source last synced. If the pipeline ever fails silently, the stale date is right there at the top screaming about it — instead of everyone trusting a report that quietly stopped updating three weeks ago. Cheap to add, and exactly the kind of thing that saves you an awkward conversation later.

De-identification for research

This is where the real care went. Australian privacy regulation — and plain ethics — is strict about using patient data for secondary purposes, so the research dataset had to be de-identified properly.

And "properly" is the operative word, because de-identification is not "delete the name column". It's a spectrum. Direct identifiers — name, address, Medicare number — obviously go. But combinations of indirect identifiers can re-identify someone just as effectively: if a record is the only 87-year-old man in a given postcode who had a specific rare procedure on a specific date, that combination is a name in all but spelling. What I implemented:

• Removal: names, addresses, contact details, Medicare numbers — gone entirely
• Generalisation: exact dates collapsed to year-quarter, ages into ranges, postcodes truncated toward state level
• Pseudonymisation: a consistent hash of the patient ID, so researchers can follow one patient across records without ever knowing who they are

The generalisation rules followed OAIC (Office of the Australian Information Commissioner) guidance, aiming for k-anonymity — any combination of quasi-identifiers should match at least k people in the set, not one. The de-identified records lived in a separate database, access gated behind ethics approval and logged for audit. Researchers could query aggregates and anonymised individuals; the raw data was never in reach.

Automating the whole thing

The pipeline runs on a schedule: sync new records from the API, download and parse any new PDFs, store the structured data, de-identify and populate the research dataset, and — on report day — generate and send the director's summary. Simple state tracking stops it reprocessing anything it's already seen; each run only looks at what's new since the last clean sync.

Error handling got real attention, because healthcare data can't just vanish into a failed job. If a specific document won't parse, the system logs it, marks that record for manual review, and carries on with the rest — and the director's report includes a count of anything that didn't process. Failing loudly and partially beats failing silently and completely, every time.

What made this work

The headline win was boring and enormous: hours of manual work, gone. The director gets better information because it's consistent and timely, and the research dataset grows continuously instead of via periodic hand-cranked exports.

The ecosystem switch stung and was still correct. I lost a day rewriting what I'd built and earned it straight back the moment I had libraries that actually worked. A good reminder that the tooling around a language is a first-order concern, not a footnote.

But the real lesson is about when privacy gets designed in. De-identification wasn't a stage bolted onto the end — it shaped which fields I kept, how I structured the data, and how the databases were segmented, from the first commit. That's the whole idea behind privacy-by-design, and in a regulated domain it's not a nice-to-have: get it right early and compliance is a property of the system; get it wrong and it's a rebuild you'll be explaining to people far less forgiving than a code reviewer.