Somewhere in government today, a team is trying to do something genuinely useful for people and can’t get the data to do it. They’ve made what feels like a reasonable data request – but it comes back “no”. Or, worse, no clear answer at all – a slow drift of emails, forms and meetings that never quite resolve, until pressing on feels more time-consuming than it’s worth, and they give up.
And here is the price we pay. When data won’t move, we end up designing policy and services around the data we have, rather than the data that would do more to improve outcomes for those who need it most. And that gap shows up somewhere very real: in the family we don’t reach early enough, or the problem that escalates because we didn’t see it coming.
I have spent much of the last year on the inside of this problem: helping a central government department work out what data it really needed, what it could and couldn’t access, and developing practical ways to unblock its highest priorities. I have sat in the rooms where organisations set out the barriers, and compare notes on what gets in their way. I have come away convinced that the barriers standing between a citizen and a better, data-enabled service are rarely insurmountable. The biggest factor is usually the absence of anyone whose job it is to broker a way through.
I’ve started to call that brokering data diplomacy, and the person who does it a data diplomat.
The role of the diplomat
Diplomats negotiate agreements. They come to the table recognising different interests, different risks, and the real limits on what each side can agree to. They don’t win by force; they build trust, and find the outcome both sides can live with, where the balance of risk and reward sits right for each. They learn each side’s constraints as well as their own. And they protect the relationship, because they know they will be back at that table one day.
A data diplomat does the same job: building a bridge between a team that needs data and the organisation that holds it. Understanding where the real issues are, convening the right conversations and creating the conditions for trust that enable sharing to happen.
Above all, they work quietly. An ambassador who reaches for the threat of force has usually already failed at the job. The same is true here – the skills that enable data to move between organisations turn out to be far more diplomatic than they are technical or legal.
Why caution runs deep
To do this well, you have to understand why data sharing stalls, and take it seriously rather than resent it.
Some of it is resource. The team you’re asking for data from is often stretched. Every request is work, and it has no budget and no time to deal with yours. Even in organisations with a dedicated data-sharing front door, the effort to extract the data and serve it up to you may simply not be funded — even the most compelling public service case has to be paid for.
Some of it is technical: the data you are after may not exist in the form you imagine, or be genuinely hard to extract.
And some of it is legal risk, which is far more intimidating than most people realise.
The colleague saying no to you may be carrying real, personal, legal risk. Disclose HMRC information about an individual outside the law and the official responsible can be prosecuted and face up to two years in prison. Nor is the risk only personal – the organisation can be liable too, and the reputational damage often stings more than any fine. So the cautious official isn’t being awkward; it’s a fear response – and, like most fears, larger than the thing itself. I’ve never come across an actual prosecution for these offences; if they happen at all, they are vanishingly rare.
The real obstacle isn’t that the law forbids sharing – it’s that the web of legislation governing it is so tangled that finding the lawful route is hard and off-putting, and stopping can feel safer than pressing on. The law is read differently in different buildings, and even the lawyers disagree. The Digital Economy Act was meant to ease this, and in places it has, but it hasn’t plugged every gap, and some areas, income data among them, are still governed by law that pulls in different directions.
When stuck cases are actually worked through, the law is rarely what’s in the way. The Data Sharing Network of Experts spent two years unpicking 30 blocked data shares across government; the recurring obstacles weren’t legal prohibitions, but awareness, process, and how the request had been framed and understood.
The regulator has helped too: the Information Commissioner has eased the threat of fines on public bodies, and has said plainly that data protection is not a barrier to sharing. The gap that remains is simply that most people aren’t confident about what the law allows.
Lessons from the front line
After a year of this, here is what I have found works, whether you are the one asking for data or the one being asked.
Lead with the why. The strongest requests don’t begin with a dataset; they begin with an objective. Don’t say “we’d like everyone’s debt records”, do say “we want to reach residents sliding into unmanageable debt early enough to help, before it tips into enforcement”. Bring in policy and data experts on both sides at this point, and two things happen: the holder can help you reach yes because they understand what you’re trying to do, and you often find the data is easier to get at than you feared. A one-off check of who meets a threshold is a world away from a standing feed of everyone’s records: far less risky for the holder, and far easier to agree.
Be clear on your priorities. Few things lose a data holder’s goodwill faster than three teams from the same organisation asking for three overlapping datasets, in three different ways, with no sense of which matters most. Work out your priorities at home, decide what comes first, and approach the other body with one voice rather than as a crowd. This takes central co-ordination and an ongoing internal conversation as priorities shift. It is rare, it is welcomed, and it changes how you’re received.
See the “ask” from their side. Do your homework on the law — including understanding both your lawful basis and your legal gateway — then put yourself in their shoes. The other body may, perfectly legitimately, read the same legal framework differently, usually because it carries a different risk appetite shaped by different duties, not because they are being obstructive. Remember, too, that the person opposite may be the one personally on the hook if it goes wrong, so what looks like caution is often someone protecting themselves as much as the data. Your job is to make saying yes feel safe: to show you understand their exposure and exactly how you’ll look after what they share.
Patience beats pressure. When you hit friction, the instinct can be to escalate, to find a senior sponsor (or even Minister) to lean on them — especially when you’re under pressure to act with urgency. It occasionally works, but more often it hardens the very position you were trying to soften, and expends trust that would have been worth far more later. Keep talking instead. Be honest about your pressures, and work out together what is doable in the time you have.
Stay open to a different shape of yes. The lawful, affordable, workable yes is often a slightly different shape from the ask you started with: narrower, delivered another way, or phased so confidence can build before the data does. Data minimisation is often critical to keeping a share within the law, and there’s usually room to reach an agreed position on it. A good diplomat doesn’t cling to the opening position; they find the version everyone can sign. That flexibility is usually the difference between a project that moves and one still “in discussion” a year later.
The quiet part
None of this is glamorous. There is no transformation programme, no new platform, no launch. At heart it is about treating data sharing as a conversation rather than a transaction: patient, well-prepared, a little persistent, between people who each have good reasons to be careful but who are, in my experience, genuinely committed to public service.
That is easiest where both sides have a dedicated team (a data-sharing front door, or centre of excellence) with the time to help shape a request and a response. Often that isn’t the case. Many data holders, especially small or under-resourced bodies sitting on valuable data, have no one with the capacity to engage at all, and the work falls to whoever is asking to make sharing as near-effortless as possible. That is where an experienced pair of hands earns its keep.
And most of the difference is made early. A few hours spent shaping the ask well at the outset — clarifying the purpose, finding the right legal route, settling on the lightest data that will do the job — saves weeks of pain later, and heads off the panic and escalation that set in when a deadline looms and the data still hasn’t moved.
Most of what’s stuck isn’t forbidden; it’s waiting for someone to work across the legal, financial and practical questions and find the version of yes everyone can live with.
Working through data challenges like this is a growing part of the wider data-strategy work I do, and one I really enjoy. So, if it’s the wall you keep hitting, do get in touch.