When political donations data was released earlier this year there were, as expected, plenty of people calling for an overhaul of the system. There is widespread recognition that the system is inadequate for properly scrutinising the influence of political donors. There are three separate bills currently before parliament aimed at improving the system.
As inadequate as the current system is, it does have some level of utility. Unfortunately, the Australian Electoral Commission (AEC)—the agency tasked with administration of the system—deliberately frustrates independent analysis of donations by refusing to release the data in the form that is most useful for such a task.
Recently, the AEC effectively refused my freedom of information request for the political donation disclosures database (even after internal review), so it looks like we're stuck scraping AEC websites to analyse political donations.
The AEC's hostility to improving transparency—especially around political donations—is baffling to me.
Its reasoning for not releasing the complete database in a useful form boils down to "it's already available online." This is (almost) technically true, but makes a mockery of transparency.
Unfortunately, the online version is in a form that is all but useless for any serious analysis. For example, it's impossible to link people or organisations across all types of disclosures with the data already made available online.
So data journalists have been writing scrapers to convert the website into usable data. Special mention here goes to Nick Evershed at the Guardian. See, for example, this scraper which gets associated entity donations in a more useful format than the AEC provides.
This freedom of information (FOI) refusal continues a history of the AEC fighting against public scrutiny of data and systems it maintains. Notably it has strongly opposed access to source code for vote counting software. The detailed case notes on an attempt by Michael Cordover to have the source code released under FOI are well worth a read.
As I mentioned at the outset, Australian political donations laws are a bit of a mess. While this is by no means the fault of the AEC, why exactly it is further frustrating efforts at transparency around donations, where there's no reason to, must be questioned.
Refusing access to donations data in the form requested in my FOI—especially the way it was refused—would have been more work than simply granting access to data which is intended to be public.
A database export is quick and I explicitly suggested my request be processed as one for administrative access. Processing requests via administrative access is usually faster and easier for government agencies.
So, this whole thing could have been a fast and cheap way to provide good access to data—which, again, is intended to be open for public scrutiny—and a benefit to Australian tax payers via greater transparency.
Instead a lot more time (and money) was spent finding ways to validly refuse giving me the data in a usable form.
In the end, the AEC did provide a database export, but only of very specific fields which are—in AEC's words—"historical, redundant data fields dating back to 1998 when the database was first created". Exporting only this specific data would have been more work than simply exporting the whole thing.