Centre for Expertise in Family Literacy -Evaluating your Program

Evaluating Your Current Program

	What is Family Literacy?
	Research
	Current Family Literacy Programs
	Types of Family Literacy Programs
	Overview of Funding
	Resources/Materials
	Training
	Evaluating Your Current Program
	Getting Started
	Links
	Home

Evaluation is a key component to the development of quality family literacy programming. The evaluation plan needs to reflect the stated program goals and outcomes, so it is essential that these goals and outcomes are clearly stated and realistic. New tools may have to be developed so that all the data needed for the evaluation is collected. Good evaluation helps clarify what works and what doesn’t. It is vital to good decision making in terms of further program development and the deployment of resources.

The following information was abridged from Basic Guide to Program Evaluation written by Carter McNamara. For a more in-depth discussion of evaluation see this website.

Another site with an excellent overview of evaluation can be found at the "A" is for Assessment: A Primer on Program Evaluation website.

Major Types of Evaluation

There are three types of evaluation which are common in organizations, goal-based evaluation, process–based evaluation and outcome-based evaluation.

1. Goal Based Evaluation

	Often programs are established to meet one or more specific goals. These goals are often described in the original program plans.
	Goal-based evaluations are evaluating the extent to which programs are meeting predetermined goals of objectives.

2. Process-Based Evaluation

	Process-based evaluations are geared to fully understand how your program really works—how does it produce the result it does.
	These evaluations are useful if programs are long standing and have changed over the years, employees or customers report a large number of complaints about the program, and /or there appear to be large inefficiencies in delivering program services.
	If your program is new, you may need to evaluate the process in the program to verify that the program is indeed carried out according to your original plans.
	They are also useful for accurately portraying to outside parties how a program truly operates (e.g. for replication elsewhere).

3. Outcome-Based Evaluation

	Program Evaluation with an outcomes focus is increasingly important for non-profits and asked for by funders.
	Outcomes are benefits to clients from participating in the program.
	Outcomes are usually in terms of enhanced learning (knowledge, perceptions/attitudes or skills) or conditions e.g. increased literacy, self reliance.
	Outcomes are often confused with program outputs or units of service e.g. the number of clients who went through a program.
	An outcome-based evaluation facilitates you asking if the organization is really doing the right program activities to bring about the outcome you believe (or better yet verified) to be needed by your clients (rather than just engaging in busy activities which seem reasonable to do at the time).

Overview of Methods to Collect Information

The following table provides an overview of the major methods used for collecting data during evaluations.

Method	Overall Purpose	Advantages	Challenges
questionnaires, surveys, checklists	when need to quickly and/or easily get lots of information from people in a non threatening way	-can complete anonymously -inexpensive to administer -easy to compare and analyze -administer to many people -can get lots of data -many sample questionnaires already exist	-might not get careful feedback -wording can bias client's responses -are impersonal -in surveys, may need sampling expert - doesn't get full story
interviews	when want to fully understand someone's impressions or experiences, or learn more about their answers to questionnaires	-get full range and depth of information -develops relationship with client -can be flexible with client	-can take much time -can be hard to analyze and compare -can be costly -interviewer can bias client's responses
documentation review	when want impression of how program operates without interrupting the program; is from review of applications, finances, memos, minutes, etc.	-get comprehensive and historical information -doesn't interrupt program or client's routine in program -information already exists -few biases about information	-often takes much time -info may be incomplete -need to be quite clear about what looking for -not flexible means to get data; data restricted to what already exists
observation	to gather accurate information about how a program actually operates, particularly about processes	-view operations of a program as they are actually occurring -can adapt to events as they occur	-can be difficult to interpret seen behaviors -can be complex to categorize observations -can influence behaviors of program participants -can be expensive
focus groups	explore a topic in depth through group discussion, e.g., about reactions to an experience or suggestion, understanding common complaints, etc.; useful in evaluation and marketing	-quickly and reliably get common impressions -can be efficient way to get much range and depth of information in short time - can convey key information about programs	-can be hard to analyze responses -need good facilitator for safety and closure -difficult to schedule 6-8 people together
case studies	to fully understand or depict client's experiences in a program, and conduct comprehensive examination through cross comparison of cases	-fully depicts client's experience in program input, process and results -powerful means to portray program to outsiders	-usually quite time consuming to collect, organize and describe -represents depth of information, rather than breadth

The following links will allow you to download or view PDF files. You will need Adobe Reader to view them. You can get Adobe reader by clicking the graphic to the right.

The following files are samples of questionnaires used in other programs:

Questionnaire 1

Questionnaire 2

If you would like to submit a questionnaire you have used successfully in a program email it for consideration.

Tools for assessing emergent or early literacy

Sure Start Language Measure (SSLM)
Used with 2 year olds to measure language skills
Is an interview tool where parents can report on their child’s development, especially their language skills.

Peabody Picture Vocabulary (PPVT III)
Is a quick and easy diagnostic tool that provides normative data
For ages 2.5 to 40, take 10 to 20 minutes
$270.00 US (without software, $370.00 with) available from AGS publishing

Slosson Oral Reading Test
Targeted at preschool to adult, take 3 to 5 minutes
Designed to assess level of oral word recognition, word calling or reading level (not a diagnostic tool)
$70.00 US available through www.slosson.com
Expressive Vocabulary Test (ETV)
For ages 2 to 6 to adult (uses age based standard scores)
Takes about 15 minutes
$160.00 US available through AGS Publishing

Nipissing District Developmental Screen
For children 1 month to 6 years to identify those who may need early intervention
Can be administered by parent or caregiver

Resources for Evaluation of Family Literacy Programs

	Outcomes and Measures for Family Literacy Programs, 2nd Edition $19.00 US, available through the National Centre for Family Literacy -an evaluation and assessment resource for family literacy programs
	Guide to Evaluation for Family Literacy Projects in Alberta $20.00 CAN, available through The Centre for Family Literacy – provides evaluation models and methods