Evaluation Measures

Contents

Introduction
What is to be measured?
How will it be measured?
Who will complete the measure?
Confidentiality
Choosing a measure
Practical considerations
Summary
Resources

Related Factsheets

Local Program Evaluation
Developing Evaluation Questions
Data Analysis & Presentation

Outcome Evaluation Manual

Evaluating and Reporting Outcomes: a Guide for Respite and Crisis Care Program Managers

Introduction

In selecting or developing measures for evaluation questions or objectives, three questions always have to be answered:

What is to be measured?
How will it be measured?
Who will complete the measure?

What is to be measured?

Evaluation measures may focus on the following: behaviors, consumer satisfaction, knowledge, skills, attitudes, beliefs, or events.

In focusing on the process evaluation of program activities and short term outcomes, what is measured primarily includes staff and program events as well as consumer changes in knowledge, attitudes and beliefs. The real payoff for most programs is changes in behavior, which might take place at a later time and in a different setting.

How will it be measured?

A complete list of measurement strategies is impossible to formulate because any activity which yields information can be a measurement method. The number of such activities is limited only by the imagination and resourcefulness of the person planning the evaluation. Common measurement strategies can be summarized into the following five categories:

Program Records: existing records, documents, intake and discharge forms. Program records are most useful for evaluations of program activities. Commonly, staff activities and other program activities are routinely captured. In addition, intake information is often routinely recorded. Using this information can be inexpensive and non-intrusive since it already exists. If the information is not suitable, it may be possible to make small changes in what is recorded, how often it is recorded, or some other aspect of the program record to make the information useful for evaluation purposes.

Public Domain Records: government statistics and documents. Public domain records can be helpful in knowing how a program is affecting the community at large. A crisis nursery program may be intended to reduce the incidence of child abuse, and the public record of the number of reported incidents might be used as an outcome indicator. In addition, individuals in the program can be matched with public records. This might include driving records, arrest and conviction records, or school records.

Behavioral Observations: observations, simulations, demonstrations, and physical evidence. Behavioral observations can be helpful if the program intends to teach particular skills or behaviors. Observing parents interacting with their child provides an opportunity for programs to demonstrate alternative and non-abusive methods of discipline. Observing parents at play with their child could demonstrate how parents are integrating physical therapy activities along with the play activities.

Interviews: structured interviews, and clinical examinations. Interviews are a rich source of information. Aside from the informant responding to formal questions, it also gives participants a chance to talk about the program and the individual's participation in the program.

Paper and Pencil Instruments: questionnaires, ratings, diaries, surveys. Paper and pencil tests will be discussed further in the section on "choosing a measure."

Who will complete the measure?

The source of the information is not always a matter of choice; but often the "same" information can be gathered from a number of sources, e.g., belief about the stress level of a family can come from the mother, the father, other family members, program staff, or professional consultants. In general, the choices for source of information are: program records; program recipient (client); family members, friends, co-workers; program staff; or another observer or professional.

Confidentiality

Whatever the source of information, attention must be paid to confidentiality. There are two factors in confidentiality: (a) doing no harm, (b) informed consent.

Doing No Harm

The way in which you collect and report information must not do harm to consumers, staff, or others associated with the program. Obviously, personal identifying information must not be included on data collection forms or reports. In addition, the manner in which information is gathered should not, in and of itself, pose a physical, psychological, or social threat to an individual. In some cases, just revealing that an individual is receiving services is damaging.

Informed Consent

Informed consent is a strong protection for the participant in an evaluation. Under this principle, an individual participates in the evaluation only after being fully told what would be expected of them in participating, and any conceivable risks of participating. This would include the methods that will be employed to preserve the anonymity of the individual. Depending on the source of the information, and the purposes for which the information is to be used, federal guidelines do not always require informed consent, but it is always the surest way of protecting the program and the consumers who participate in an evaluation.

Choosing a Measure

Measuring knowledge, attitudes, beliefs, skills and even observing behaviors is a deceptively simple task. It is easy to do incorrectly. Whether dealt with openly or not, any measure is subject to issues of (a) reliability and (b) validity.

Reliability

Reliability is the consistency of the measure. Would you get the same response if taken again under similar conditions? How different would the measure be under alternative conditions, such as a different time or setting?

Validity

Validity is the extent to which an instrument measures what it is intended to measure. Often the most common form of validity is "nominal validity," that is, the name of the test is the same as the indicator of interest; but this is not really adequate.

Fortunately, most measures used in program evaluation are obvious. They involve program, staff and participant event information which is easily observed and recorded. Any differences between what is recorded as happening and what actually happened are usually trivial. For example, the sign-in sheet used to count the number of participants may be "off" by one or two, but that may not be a big enough error to warrant abandoning the use of the sign-in sheet.

When it is necessary to assess something other than program events or records, the choice is between using or adapting an existing measure, or constructing a new one. It is worth a fair amount of searching to locate measures that have already proven themselves. Instruments with a history of use usually have information about reliability, validity, and which populations are appropriate for their use.

Finding previously used measures is not always easy. A literature search is a good place to start. Contact with other crisis nursery or respite care providers can help, as can contact with a national clearinghouse or resource center. Wherever you find possible measures, the following list of information may be helpful in deciding if a measure can be useful.

General Reference Information

What is the name of the measure?
Who is the author?
Who is the publisher?
What is the date of publication?
What is the cost?
How long does it take to administer?

Validity Information

With what sort of groups has the measure been employed?
With what other measures has this measure been correlated?
With what later outcomes has this measure been correlated?
How accurate a prediction does the measure give of significant outside criteria?
How closely do the measures' reported objectives correspond to your objectives?
What have reviewers and critics said about the measure?

Reliability Information

Do the authors indicate the size and nature of groups for which data is reported?
Do they indicate the type of reliability coefficient computed?
Do they give the mean and standard deviation for the groups?
Do they report reliability for each type of group that may be included in the evaluation?

Practical Considerations

Are the instructions and procedures suited to your population?
Are the time requirements reasonable for your purposes?
Is the form to be used legible, attractive and convenient?
How much time is required in scoring the measure?
Do you or your staff have the skills to administer and score the instrument. If not, are there funds available to hire someone?
Were the norms for the measure developed on a similar population?
How much does the measure cost when employed in your situation?

Summary

Selecting the evaluation measures is one of the more critical activities when developing an evaluation program. Whatever a program's intentions are in providing a service, or in changing the lives of consumers, what is chosen to be measured will be the way the program is judged.

Resources

Patton, Michael Quinn (1986). Utilization-Focused Evaluation. Newbury Park, CA; Sage Publication.

Stecher, Brian M. (1987). How to Focus an Evaluation. Newbury Park, CA; Sage Publication.

Weiss, Heather B., Jacobs, Francine H. (Eds.) (1988). Evaluating Family Programs. New York; de Gruyter.

About the Authors: David B. Langmeyer, Ph.D., is an Evaluation Consultant to ARCH. He was Chief of Evaluation and Research in the NC Division of MH/DD/SA Services for 13 years. Gail S. Huntington, Ph.D., is Director of Evaluation and Research for ARCH. She is a Research Investigator at the Frank Porter Graham Child Development Center at the University of North Carolina at Chapel Hill.

ARCH Fact Sheet Number 14, August, 1992