Breaking News – NSW Parliament Adopts our Proposal, go to our Campaign page.
Evidence Based Policy Research Project
(a not-for-profit incorporated association)
Our Story
In November 2017 a Symposium convened by the newDemocracy Foundation agreed to challenge opinion leaders by asking them what practical, measurable steps could be taken to improve public trust in government decisions. The objective was to move from complaining about problems to offering potential solutions. The Evidence Based Policy Review Project was one of the major initiatives to emerge from the 2017 Symposium event.
Many participants – company directors, the advocacy sector, journalists and even former MPs – lamented that “evidenced based policy making” had become an empty phrase which everyone claimed to pursue but no one knew how to quantify.
Former Secretary of the NSW Treasury, Percy Allan, championed a proposal to draw on the “good practice” policy making criteria developed by Prof. Kenneth Wiltshire AO from the University of Queensland which had been used in assessing the quality of federal legislation by the Institute of Public Administration Australia (IPAA) in 2012 when Percy was its national president.
With the support of newDemocracy, it was agreed to undertake an independent benchmarking assessment of legislative initiatives from the Commonwealth and three states – New South Wales, Victoria and Queensland – rating them against the “Wiltshire criteria”.
As a result, the Institute of Public Affairs (IPA), a self-described ‘free-market’ think tank identified with the Right of politics, and Per Capita Australia, a self-labelled ‘progressive’ think tank identified with the Left, were commissioned in 2018, 2019, 2020 and 2021, in a ‘double blind’ exercise which each year saw them assess 20 policies drawn from around the country: eight from the Commonwealth and four from each of the three states.
Though these think tanks often have opposing policy outlooks, their benchmarking results against the Wiltshire criteria were remarkably similar, across what is now a total of 80 legislative initiatives analysed. For four years running, the think tanks have found that basic standards of evidence and consultation-based policy making are inadequately met by Australian federal and state governments.
Our Findings
The results of the 80 case studies undertaken so far over the last four years suggest a solid process was followed in twenty-seven of them by the governments involved. In nineteen cases the ratings were well below par. In the balance of cases the process quality was mediocre. See table below.
Overall Findings, 2018-2021
| Policy Decision-Making Process |
Think Tanks’ Average Score out of 10 Test Criteria
|
2018 Case Studies Number |
2019 Case Studies Number |
2020 Case Studies Number |
2021 Case Studies Number |
2018-21 Total Case Studies No. & % Share |
|
Acceptable, |
7 – 10 criteria satisfied
|
6 | 6 | 9 |
6
|
27 (33.8%) |
| Mediocre | 5 – 6.5 criteria satisfied |
10 | 6 | 9 | 9 |
34 (42.5%)
|
| Unacceptable | Under 5 criteria satisfied |
4 | 8 | 2 |
5
|
19 (23.7%)
|
|
Total
|
20 | 20 |
20
|
20
|
80 (100%)
|
The think tanks’ total scores on the ten Wiltshire criteria for the eighty case studies to date were remarkably similar in sixty cases (either identical or only one-point difference). Of the remaining twenty cases, seventeen had differences in scoring of two-points. See table below.
| Total Score Differences | 2018 Case Studies Number |
2019 Case Studies Number |
2020 Case Studies Number |
2021 Case Studies Number |
2018-21 Total Case Studies No. & % Share |
|
None
|
7
|
8 |
7
|
3 |
25 (31.3%)
|
|
1 point
|
10 | 7 | 9 | 9 |
35 (43.8%)
|
|
2 point
|
3 | 5 | 4 | 5 |
17 (21.2%)
|
|
3 point
|
0 | 0 | 0 | 3 |
3 (3.7%)
|
|
Total
|
20 | 20 | 20 | 20 |
80 (100%)
|
Findings by Jurisdictions
Averaging the two think tanks’ total scores (out of 10) for each of the 80 cases done over the last four years (20 each in 2018, 2019, 2020 and 2021) gives the following average score for each jurisdiction:
Fed: 5.4 (32 cases)
NSW: 5.5 (16 cases)
Vic: 6.1 (16 cases)
Qld: 6.2 (16 cases)
Weighted Mean Average: 5.7 (80 cases)
Case Study Score Ratings
- Excellent: 9.0 – 10.0
- Sound: 8.0 – 8.5
- Acceptable: 7.0 – 7.5
- Mediocre: 5.0 – 6.5
- Unacceptable: below 5.0
Our Proposal
The remedy is for a more rigorous process of explanation, justification and consultation (such as the traditional but now-rarely used, two-stage Green and White paper approach) to apply to any legislation that deals with a contentious public policy issue. What that involves is explained here.
At the very least, all bills introduced in a parliament should be accompanied by a Statement of Public Interest (SPI) that answers six fundamental questions at the core of the Wiltshire criteria, namely:
- Need
Why is the policy needed based on factual evidence and stakeholder input? - Objectives
What is the policy’s objective couched in terms of the public interest? - Options
What alternative policies and mechanisms were considered in advance of the bill? - Analysis
What were the pros/cons and benefits/costs of each option considered? - Pathway
What are the timetable and steps for the policy’s rollout and who will administer it? - Consultation
Were the views of affected stakeholders sought and considered in making the policy?
An SPI questionnaire would take only a few pages for a bill’s proponent to answer in the affirmative or negative so would not be onerous to prepare. Every member of Parliament and interested citizen is entitled to have this information before a bill is considered.
An SPI would encourage public servants, ministers and private members who develop and submit bills to address the fundamental steps of good policy making.
Also, it would assist a parliament to decide whether a bill should be debated and voted upon without further action or referred to a standing committee of the parliament for closer scrutiny.