Building and Using Evidence

Data is a key tool for performance management but it can also play a valuable role in building longer term evidence about interventions that achieve the best outcomes. In order to take full advantage of data, it is important to have the evaluation infrastructure that allows empirical evidence to inform an administration’s budget, management, and policy decisions.

“Knowing what works” in a particular context (and what to fund) requires research and evaluation capacity both within and outside of government. Research and evaluation offices can conduct evaluations, build lasting collaborative research partnerships, and provide expertise on high-quality evidence-building activities. At their best, these evaluation offices engage residents to identify key priorities, especially how governments can build and use evidence to improve equity. Funding these capacities is essential and can be done through set asides, carve-outs in program funds, and even budget line items. Embedding research infrastructure in government that is aligned with budgeting processes allows states to move towards results-focused spending.

Blueprint Strategies for Delivering Results

Actions:

Hire a Chief Evaluation Officer or Chief Learning Officer to provide leadership on an overall approach to evaluation and learning
People
Build cross-department and cross-functional teams to ensure the right breadth of perspective is leveraged to address top administration priorities
PolicyPeople

Provide the appropriate level of authority, responsibility, and resources to these leaders and teams so they can properly execute

MASSACHUSETTS

Chief Strategy and Research Officer

The Massachusetts Department of Elementary and Secondary Education’s Office of Planning and Research is led by a Chief Strategy and Research Officer who is responsible for improving the use of data and research findings to influence policy and program decision-making.

More information: 2019 State Standard of Excellence (Criteria 6).

OHIO

Office of Research, Evaluation and Advanced Analytics

The Director of the Ohio Department of Education’s Office of Research, Evaluation and Advanced Analytics helps educational leaders across the state use evidence through resources and a clearinghouse.

More information: 2019 State Standard of Excellence (Criteria 6).

TENNESSEE

Office of Evidence and Impact

Tennessee’s Office of Evidence and Impact, founded in 2019 supports the state’s evidence-based budgeting and program inventories through evidence reviews and technical assistance on developing evidence-building evaluations and activities.

Actions:

Develop policies which promote and guide the use of data and evidence in program development and assessment, including the public release of reports
Policy

Consider developing an explicit “learning agenda” for the capture of data across programs, so that performance management efforts go beyond just tracking outcomes to focus more broadly on learning and analysis; identify ways to seek community input on the learning agenda so that it reflects the needs of a diverse group of stakeholders

Build a program inventory of all efforts by area (e.g. criminal justice, mental health, child welfare) to enable better comparison and evaluation
Infrastructure
Create clear definitions and tiers of evidence (e.g. evidence-based vs. data supported) to better communicate within departments, the public, and the legislature about what is working and what is not
Policy
Engage outside experts, such as university-based researchers, to enhance internal capacity for research and evaluation of programs
People

MINNESOTA

Evaluation Policy, Evidence Definitions, and Clearinghouse

In partnership with The Pew Results First Initiative, Minnesota Management and Budget inventoried nearly 500 publicly-funded interventions and rated their evidence of effectiveness in a clearinghouse. To support the ongoing focus on funding evidence-based programs and interventions, Minnesota Management and Budget released an evaluation policy to govern evaluations, which specifically mandates the release of all completed evaluation reports regardless of findings. Minnesota Management and Budget also defines evidence for research, evaluation, and funding purposes.

More information: 2019 State Standard of Excellence (Criteria 7).

CALIFORNIA

Evidence-Based Clearinghouse for Child Welfare

The California Department of Social Services created an Evidence-Based Clearinghouse for Child Welfare, which allows child welfare providers and professionals to identify, select, and implement evidence-based child welfare practices. In addition, the Clearinghouse provides guidance and support for program implementation. The Clearinghouse’s numerical rating scale categorizes programs into six tiers of evidence and uses a relevance scale as a complement to the scientific rating scale to demonstrate applicability for client populations.

More information: 2019 State Standard of Excellence (Criteria 10).

TENNESSEE

Education Research Alliance

The Tennessee Education Research Alliance (TERA) is a formal research partnership between the Tennessee Department of Education and Vanderbilt University’s Peabody College of Education. Led by seven full-time staff and guided by a steering committee and advisory council, the Department and the University have co-developed a research agenda that allows the University to conduct research on a variety of issues.

More information: 2019 State Standard of Excellence (Criteria 7).

Actions:

Develop a budget decision-making process that preferences requests with evidence and/or data about quantifiable outcomes (i.e. not just counting inputs and outputs)
Policy

Consider a rubric that increases the priority of a specific request based on the level of evidence for the program, while also creating space for new programs that do not yet have an evidence base, but do have strong data that demonstrates quantifiable outcomes

Create a standard funding allocation to build the state’s capacity to evaluate programs (e.g. 1% with a cap on the total amount); this is especially relevant for new programs where it is important to design the evaluations and metrics that will be used to measure the success of the initiative
Policy
Ensure the new budget process is integrated into the broader systematic approach to managing results, so that priorities and learnings from different parts of the governor’s office and agency leadership are not siloed
Infrastructure
Engage with the legislature to refine the new budget process
People

Clarify terms such as “evidence-based” vs. “data informed” to ensure consistency and clear communication

COLORADO

Budget Instructions

Colorado’s FY 2020-2021 budget development instructions (pp. 10-12) prioritize new program requests “based on the evidence and body of research supporting the program’s effect on desired outcomes and proposed implementation plan.” In the FY 2020-2021 budget cycle, the state applied an evidence continuum to budget requests and used that criteria to inform resource allocation decisions.

MINNESOTA

Guidance for New and Expanded Programs

Minnesota Management and Budget issued guidance on how to report outcome data. In addition, the state also used evidence to inform funding decisions resulting in $87 million in new or expanded evidence-based programming in the FY 2020-2021 budget.

More information: 2019 State Standard of Excellence (Criteria 10).

NORTH CAROLINA

Budget Office Reform

In 2018, North Carolina redesigned its budget process to increase the use of evidence-based decision-making. As a result, the 2019-2021 budget development instructions require that agency requests “for new or expanded programs or services must include evidence and research supporting the program’s effect on desired outcomes.”

More information: 2019 State Standard of Excellence (Criteria 9).

What Could This Look Like Over Time?

Building and Using Evidence

Strategy G:

Create leadership structures to effectively build and use evidence across state programs and departments.

Beginning

Establish cross-agency teams to address specific results-focused projects
Designate/hire a Chief Evaluation Officer
Ensure clear coordination and authority across data and evidence-related efforts

Building

Assess and revise the agenda for Chief Evaluation Officer and cross-agency teams
Address capacity gaps

Sustaining

Engage legislature to build support and identify potential legislative opportunities

Strategy H:

Build the infrastructure to evaluate state programs through evaluation policies, evidence definitions, program inventories and funding to support state leaders.

Beginning

Assess current policy practices, identify gaps, and compare to leading practices
Develop clear definitions and tiers of evidence for use across the state
Identify immediate and near term actions to support cross-agency team projects
Begin a program inventory by topic area

Building

Establish learning agenda tied to broader performance management system
Engage outside experts to bolster capacity
Ensure the evidence definitions are used properly in practice
Build out program inventory

Sustaining

Assess data and evidence infrastructure and address gaps

Strategy I:

Position the budget as a key tool to build state evaluation capacity and apply that capacity to focus resources on evidence-based investments.

Beginning

Create and pilot a new budget process using a data and evidence decision-making rubric

Building

Refine and expand new budget process and provide support for agencies to use it
Integrate new process with performance management
Engage the legislature in new budget process finalization

Sustaining

Consider standard resource allocations, scaled by program size and with a cap, to support new and existing program evaluation

Wins

By following the actions above and building over time, states should aim to achieve wins along the way to galvanize internal and external support. Here are some sample wins, though there are many other types of achievements that states could use to mark their progress.

Short-term Wins

Broadly adopted evidence definitions across state government (such as in Colorado)
Designate/hire a chief evaluation officer to lead and coordinate evaluation activities (such as in Minnesota)
Develop a clear structure for coordinating evaluation and evidence activities across agencies, which could include the state legislature (See Alabama’s Commission on Evaluation Services)

Mid-term Wins

Create learning agendas (statewide and at agencies)
Develop an evaluation policy aligned with evidence definitions (see Minnesota Management and Budget’s evaluation policy)
Build relationships with institutions and organizations that can provide external research capacity (such as the research-practice partnership model of the Tennessee Education Research Alliance)
Make evidence and evaluations publicly available through a clearinghouse or inventory (such as the California Evidence-Based Clearinghouse for Child Welfare)

Long-term Wins

Create a strong state evaluation culture with centralized, agency, and external capacity (see State Education Agencies in Mississippi, Ohio, and Massachusetts, which have created research and evaluation offices to lead and coordinate evaluation activities)
Use tiered evidence structures that build evidence and encourage innovation (see the Education Innovation and Research program from the U.S Department of Education)
Require evidence of effectiveness in budget proposals, especially for newly proposed programs (see Minnesota Management and Budget guidance on making evidence-based budget proposals)

Back to Blueprint

Download Blueprint Download This Blueprint Theme Only