Introduction
Developing Process Questions
Developing Outcome Questions
Outcome Chart
Summary
References

Related Factsheets

Local Program Evaluation
Evaluation Measures
Data Analysis & Presentation

Introduction

The evaluation focus will determine the questions asked. An evaluation can occur at different points in the program cycle. A program engages in very concrete sets of services, which may, or may not, lead to outcomes for the consumer, family of the consumer, or the community at large. In addition, the outcomes may occur immediately during or after the program is offered, or some time later. All of this can be summarized in the following chart.

Developing Process Questions

Process evaluation helps answer questions about whether or not the ongoing program is functioning as it was intended. It is also the most helpful tool for program managers in operating the program on a day to day basis. Process evaluation addresses the ongoing program structure and activities. Questions such as those on the following list are important for process evaluation.

• Is the appropriate structure in place to maintain the program?
• Is the appropriate staff in place to offer the intended program?
• Are the people receiving the service those who were identified as most in need?
• Are the services which are being provided those which were intended to be provided?
• Are consumers receiving the amount of service they need?

Process evaluation can be divided into structural questions and activity questions. Structural questions relate to resources and organization, whereas activity questions relate to staff and consumer behaviors.

Structural Questions

Structural questions are similar to standards for licensing or accreditation. It is generally believed that some organization and resources are needed to properly run a program. Some structural questions are:

• Is there a written statement of the philosophy and approach to services, a mission statement?
• Is there an administrative structure which would support a program (organizational chart, board of directors, personnel policy, hiring practices which are consistent with philosophy and program intent, etc.)?
• Does the physical setting of the program conform to the program's intent?
• Are there proper procedures for consumer involvement in program planning and evaluation; recruitment and enrollment; and routine and emergency program operations?
• Are there mechanisms for accountability?

Activity Questions

Activity questions are aimed at actual staff and consumer behavior. Staff activity relates to:

• What kinds of activities did staff engage in?
• How much did each staff member do?
• To which consumers did the staff give service?
• Where did the staff provide the service?

Staff activity should imply consumer activity, but it may not. Another set of questions, similar to staff questions, can be asked about consumers:

• What kinds and how many consumers were served?
• What kinds of services were offered and in what intensity?
• At what times and in what settings were the services received?

Being able to document that there is an active program is an important accountability activity. It is also important for internal program management to know who is engaged in what activity.

Developing Outcome Questions

In order to develop short or long term outcome questions, it is important to know clearly how the program activities relate to changes in consumer, family, and community behavior, attitudes, knowledge, skills or intentions. The "model" which relates the mundane activities of the program to the short and long term outcomes and impacts in the consumers, families and community is called a theory of action.

The theory of action provides links between current activity and future intended goals. To develop a theory of action just keep asking, "Why do we do this?"

What is expected to be the immediate result of a program action? By repeating this question for each step of the theory of action, you develop a linkage between program activities and future changes in consumer, family or community behavior, attitudes, knowledge, skills or intentions. The outcome of one activity is usually the next "activity" so that a chain develops linking the program's services with longer term outcomes (see Outcome Chart below).

Outcome Chart

With this set of linkages, choices can be made about what is an appropriate outcome, when it is likely to occur, and what should be the focus of an evaluation effort.

*Note: measuring this as an outcome
requires a sophisticated evaluation design.

Summary

Evaluation questions should follow from a thorough understanding of the program's operations, intentions, and consumers. When developing a theory of action, choices can be made about what is an appropriate outcome, when it is likely to occur, and what should be the focus of an evaluation effort.

References

Fitz-Gibbons, Carol Tayler (1987). How to Design a Program Evaluation. Newbury Part, CA; Sage Publication.

King, Jean A. (1987). How to Assess Program Implementation. Newbury Park, CA; Sage Publication.

Patton, Michael Quinn (1987). How to Use Qualitative Methods in Evaluation. Newbury Park, CA; Sage Publications.

Shadish, William R., Cook, Thomas D., and Leviton, Laura C. (1991) Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA; Sage Publication.

Outcome Evaluation Manual

Evaluating and Reporting Outcomes: a Guide for Respite and Crisis Care Program Managers [Note: this volume is currently out of print and is being revised. The 2nd Edition will be available in Spring, 2003.]

About the Authors: David B. Langmeyer, Ph.D., is an Evaluation Consultant. He was Chief of Evaluation and Research in the NC Division of MH/DD/SA Services for 13 years. Gail S. Huntington, Ph.D., is the former Director of Evaluation and Research for ARCH. She is a Research Investigator at the Frank Porter Graham Child Development Center at the University of North Carolina at Chapel Hill

ARCH Fact Sheet Number 13, August, 1992