The State Integrity Investigation assesses the existence, effectiveness, and accessibility (i.e. citizen access) of key governance and anti-corruption mechanisms through a qualitative and indicator-based research process.
A first round of inquiry was conducted in 2011 and published in 2012, assessing the state of integrity in all 50 states. In 2014, Global Integrity again partnered with the Center for Public Integrity to embark on a second round of research. The indicators diagnose the strengths and weaknesses of the institutional safeguards applied against corruption in each state – mechanisms of openness, transparency, and accountability – rather than corruption itself. The indicators comprise 13 sections divided into 92 de jure indicators, 153 de facto indicators. Categories assessed include:
- Section 1: Public Access to Information
- Section 2: Political Finance
- Section 3: Electoral Oversight
- Section 4: Executive Accountability
- Section 5: Legislative Accountability
- Section 6: Judicial Accountability
- Section 7: State Budget Processes
- Section 8: State Civil Service Management
- Section 9: Procurement
- Section 10: Internal Auditing
- Section 11: Lobbying Disclosure
- Section 12: Ethics Enforcement Agencies
- Section 13: State Pension Fund Management
To identify the project’s indicators, staff from Global Integrity and the Center for Public Integrity contacted nearly 100 state-level organizations and experts working in the areas of good government and public sector reform around the country during 2011. We asked what issues mattered most in their state when it came to the risk of significant corruption occurring in the public sector. The outcome was a list of more than 300 questions, rooted in the inner workings of state government operations. Based on these discussions, Global Integrity and the Center for Public Integrity also selected and included relevant indicators from previous projects in the overall indicator set.
In order to further strengthen the overall set of questions — and building on the lessons learned from the project’s first round — Global Integrity reduced the overall number of indicators while maintaining the project’s scope. This decision was taken in light of feedback from various stakeholders, with an aim to assessing the most salient corruption risks in each state. GI also revamped and refined the scoring conditions of select indicators in order to increase the reliability and validity of the assessment. As part of this process, the investigation dropped two sections from the previous round of research: “State Insurance Commissions” and “Redistricting,” while adding a section on “Electoral Oversight.” As a result, the total number of indicators dropped from 330 to 245.
Evidence-based expert assessment methodology
Typical Global Integrity assessments provide detailed data and reporting on the mechanisms in place to prevent abuses of power and promote public integrity at the national, sub-national and at sector levels, both in the U.S. and abroad. Using a blend of social science and journalism, local teams of independent researchers and journalists report on the de jure as well as de facto reality of corruption and governance. Measuring the scope of the legal framework and what occurs in practice is key in our effort to produce actionable governance data that help governments, citizens and civil society understand and evaluate the status quo and identify intervention points for possible reform efforts.
Global Integrity’s evidence-based expert assessments require researchers (or journalists, in the case of the State Integrity Investigation) to compile and document evidence to inform and support their score choices for each of the indicators. Rather than relying on experiences or pre-existing perceptions by experts, the strength of Global Integrity’s methodology is that it requires a variety of information sources to be reviewed and documented (including legal and scholarly reviews, the text of government laws and procedures, interviews with experts, and reviews of media stories) to substantiate the score choice and triangulate the data points identified.
Various factors can affect the interpretation of a particular indicator and the score assigned to it. To minimize this effect and to maximize inter-coder reliability, Global Integrity provides researchers and peer reviewers with scoring criteria for every indicator. The scoring criteria anchor each indicator to a predefined set of conditions. In essence, the scoring criteria guide the researcher and the peer reviewer by suggesting, "If you see X on the ground, score this indicator in the following way."
For "in law" indicators, scoring criteria are provided for "Yes (100)" "Moderate (50)"and "No (0)" responses. For “in practice” indicators, scoring criteria are provided for 100, 50 and 0 – but researchers also have the option to score 75 or 25, whenever the higher or lower defined criteria don’t accurately represent the researcher’s findings.
Researchers must provide a fact-based explanatory comment to substantiate the score chosen for each indicator. They must also provide at least three references to substantiate each comment for any “in practice” indicator. Evidence can take on the form of interviews conducted with knowledgeable domain experts, a website link to information provided by government or civil society, a relevant report produced within the period of study, published news articles, or a widely acknowledged and regularly-read blog produced by a subject matter expert.
In the case of de jure indicators, the laws or regulations and a reference to relevant articles or legislative/regulatory sections serve as evidence. In some cases, more than just one law is needed as a valid reference, and in those cases any number of laws or other aspects of the legal framework (including court decisions) may serve as evidence.
Global Integrity staffers rigorously verify sources and perform fact-checking and quality control together with the researcher prior to declaring the research as final. After completion of the draft research, the indicators follow a double-blind peer review process, in which a peer reviewer (a similarly qualified in-state domain expert) carefully scrutinizes and amends, supplements and enhances the research. The double-blind nature of the peer review process guarantees feedback free of considerations associated with who collected the data and scored the indicators, and avoids a peer-influenced consensus. Until the public release of the indicators, the researchers and peer reviewers are – by default – unaware of the identities of other members of the team.
Along with the scoring criteria, researchers and peer reviewers are also provided with a series of guidelines and definitions they must adhere to when performing their research and coding or reviewing the data. These guidelines also provide guidance about appropriate sources and about how to score complex indicators in certain scenarios. For example, we ask about political parties and individual candidates in one question – offering scoring options to intelligently and consistently score cases in which only one element is applicable.
For the purpose of producing a state’s aggregate report card, a simple aggregation method was used. As described above, the reporter assigned a score to each indicator. Each score was then averaged within its parent subcategory, which produced a subcategory score. The subcategory score was in turn averaged with the other subcategory scores into a parent category score. Category scores were averaged to produce an overall state score. The State Integrity Index groups states into the following 11 performance "tiers" according to a state's overall aggregated score:
• 90 and above: A
• 84–86: B
• 74–76: C
• 64–66: D
• 59 and below: F