Our partners at the Kettering Foundation recently published an insightful interview with Prof. Katie Knobloch of the Center for Public Deliberation – an NCDD organizational member – that we wanted to share here. There’s a lot to learn from Katie’s reflection on the challenges of evaluating and legitimizing D&D work, so we encourage you to read the interview below or find the original piece here.

Does Our Work Really Matter? Deliberative Practitioners Reflect on the Impact of Their Work

As attention to public deliberation has increased, one core interest of researchers has been evaluating the impact of deliberative processes. Researchers, practitioners, elected officials, and participants themselves want to know if what they’re doing matters. Does public deliberation impact policy? Does it change our attitude toward issues? Does it adhere to democratic ideals?

Professor Katherine R. Knobloch has been intimately involved in evaluation work, refining our understanding of these questions. Former research assistant Jack Becker sat down with her to talk about her work around evaluation, as well as her work with the Oregon Citizens’ Initiative Review.

Katherine R. Knobloch is an assistant professor and the associate director of the Center for Public Deliberation in the Department of Communication Studies at Colorado State University. Her research and teaching focus on political communication and civic engagement, specifically exploring how deliberative public processes can create a more informed and engaged citizenry. For this work, she has received a grant from the National Science Foundation to study the expansion of a new governing institution, the Citizens’ Initiative Review. Her work has appeared in The Journal of Applied Communication Research, Politics, and the International Journal of Communication.

Jack Becker: Your work explores the development, evaluation, and impact of deliberative public processes. How do you compartmentalize each of these in your research?

Katherine R. Knobloch: The central element of interest is figuring out how to implement deliberative practices in ways that matter. To look at the development of public deliberation, I talk with people about what goes into running organizations, how they work with public officials to implement their processes, and how they got involved in public deliberation. I do a lot of fieldwork and observations to examine this.

For evaluation, I have worked alongside a number of scholars to develop a coding scheme that allows us to break the deliberative process out into segments. We then use that scheme to judge the deliberations against goals that practitioners identified and goals and definitions that we as researchers have developed to analyze if processes are fulfilling democratic and deliberative standards.

For example, we have used an updated definition from John Gastil’s Political Communication and Deliberation (2008), that deliberation is an analytic information gathering process, a democratic discussion process, and a decision-making process. I will also spend time observing participants and getting feedback from them directly, asking, for example, did they reach their goals? Did they uphold deliberative criteria? I will also do a pre- and post-survey of participants to examine a variety of factors, such as attitudinal changes.

To look at impact, with the Citizens’ Initiative Review (CIR) process, for example, we are looking at whether the process has an impact in how voters make their decisions. Do people read the CIR statement? Do they find the information valuable?

You have a chapter in Democracy in Motion: Evaluating the Practice and Impact of Deliberative Civic Engagement (2012) in which you and your coauthors lay out criteria for evaluating deliberative public processes. What is it we learn from evaluating deliberation, and what are our challenges?

I think we’re looking to refine our methods. I’m concerned that we do evaluation in an inefficient way. Much of my own work in evaluating deliberation relies on grants, and that’s not sustainable, particularly for small organizations that lack the capacity to get large grants and do the evaluative work. So we need to figure out what survey methods are best and how they can be refined to make it easier for practitioners to regularly evaluate their work.

For the CIR, we wanted to start a coding scheme that would be applicable across deliberative events. Deliberative processes are dynamic, and that’s another challenge to the work of evaluation. During deliberative processes, the agenda may change in real time, and in the past, we’ve changed coding schemes, but now we’re trying to use the same coding scheme and develop one that will work in other deliberative processes. The goal of evaluation is to be able to look back and say what the most valuable results from a process are.

Are we seeing more practitioners evaluate their own work?

I think that’s been a trend in recent years. More people want to know if their work is doing what they say it’s doing. Also, they want to know if it is effective in impacting communities, organizations, and people.

I attended a session at the National Conference on Dialogue and Deliberation this year that was focused on practitioners and academics getting on the same page with evaluation. One of the challenges is that everyone is working off of different frameworks. Josh Lerner with Participatory Budgeting (PB) pointed out at the conference how many different teams are evaluating PB processes. So they are trying to create at least a funnel point to gather this info and synthesize this.

I’ve been talking about civic infrastructure with people for the past year. How do innovations such as the Citizens’ Initiative Review (CIR) fit into ideas of civic infrastructure?

One of the most important breakthroughs for the CIR in Oregon is that it is a legitimate and formal part of the governing process. I think effective public engagement matters. It’s important for participants to come away from deliberative processes feeling like their participation was purposeful and that it could have a real impact on public decision making. I think that’s the legitimizing part of the CIR. It legitimizes deliberation as part of governance. Ideally, we would like to see more processes like these become embedded in government as ways to improve the quality of our civic infrastructure.

Organizations, practitioners, and theorizers are taking this process seriously. As a field, deliberation faces the challenge of implementing decisions that publics make at deliberative events. So people make decisions through deliberative processes, but then decision makers decide whether to use it. So the CIR specifically addresses that problem, in that recommendations go right to voters in the voters guide for their consideration. The CIR finds a way to make those decisions matter at the policymaking level.

Participatory Budgeting is a wonderful example of making things matter for people as well. City councils and city governments are handing over portions of their budget to citizen decision making, showing that citizens have the capacity to make these decisions.

So part of the success I hear there is that they are creating connections to the decision-making process by working with decision makers. Are elected and appointed officials into this?

I think there are more city officials who are into deliberation. It may be wishful thinking, but I see city officials taking citizen voice more seriously. I think they want to understand what citizens want and why. Even President Obama making the call for a discussion on mental health is a good example. And models like the Colorado State University Center for Public Deliberation are great examples of linking deliberative practice more directly with city officials and providing recommendations to city councils in ways that are impactful.

Hawaii state senator Les Ihara Jr. stressed to me in a recent interview the importance of meeting elected officials where they are. Does this resonate as a productive approach to growing deliberative practice?

Legislators are often wary of the initiative process since the policy or legislation is created without a connection to the resources allocation process. So it creates a misalignment in the policymaking process. Legislators are open to how to improve the initiative process. And so in Oregon, officials were interested in how to improve that process and saw that the CIR could potentially bring more alignment to the initiative process.

So in developing the Citizens’ Initiative Review, to what extent was the process driven by government officials in demanding these changes?

It was really driven by the founders of the CIR who were not a part of government. When they first proposed the CIR, they had a conversation with the Oregon Secretary of State who asked them to run a pilot. The founders of the process drove it. But they worked closely alongside legislators and public officials to identify what they thought would be useful to improve the process and to make sure it met the needs for Oregon as a whole. And the legislators of Oregon asked for a thorough evaluation of the process during the pilot, exploratory phase. So it really comes back to the importance of evaluation in growing deliberative practice.

The original version of this Kettering Foundation interview can be found at http://kettering.org/kfnews/does-our-work-matter.