Program Evaluation - Validity ?

Dear students - Try to be a devils' advocate!

I am interested to know your understanding about the different validities ? How implementation evaluation and process evaluation can be effected by these validities ?

Though, we have discussed about the threats to validity ? Please discuss how these threats can support in terms of implementation and process evaluation ?

How to do an Outcome Evaluation

Usually Outcome Evaluation is based on one of the following designs.

1. Randomized Controlled Trial

2. Comparison Group

3. Pre-post Comparison

Designs are usually chosen considering the available resources for conducting evaluation. Even though Randomized Controlled Trial design is considered to be the strongest among the three, it is complicated and more resource intensive as compared to pre-post comparison.

Randomized Controlled Trial

In this design, we use two or more groups. Both of them undergo the same pre and post-tests, however, one group is assigned a treatment, the other might be on a viable alternative, or even placebo.

Comparison Group

The difference that Comparison Group design has with Randomized Control Trial is that here the comparison group is arbitrarily and/deliberately chosen, rather then randomly. They are chosen so as to keep the two groups as much alike before the treatment as possible.

Pre-Post Design

Here, target audience is assessed on the same indicators and over the same time period before and after the treatment. They are less statistically rigorous and can produce useful results. Follow-up in these time-series designs is a drawback, however. Also, there is not much room to identify confounding factors that might have affected the program.

Solomon Four Group Design

Taking further the discussion of pre-test post-test control group design, and analyzing its problems, "Solomon Four Group Design" must be considered.

This design has two basic benefits;

1. It checks many of the internal validity threats, by providing more control over variables. And,
2. Due to the inclusion of two extra groups, the confounding variables influence can be checked, if any.

Explanation of Solomon Four Group Design

We will first take four randomized groups, Group A, Group B | Group C and Group D.
Pretest will be applied to the first two groups only.
Treatment will be provided to Group A and Group C.
Followed by Post Test of all groups.

First the post-test of Group C and D will be compared with that of Group A and Group B, if the results are considerably different, it is right to assume that the pre-test has had an impact on the final result.
Second, comparison between Group B's pretest and Group D's post-test will establish if any external factor has caused the change.
Third, Group A and Group C post-tests will be compared. If the results differ, which should ideally not, we will know that the pre-test caused the change.
And finally, Group B and Group D post-test comparison will let us know, if without the "treatment", only pre-test has caused the change.

This method is not very common in usage due to its high cost, technicalities involved, and usually higher time constraint of dealing with four random groups.

I will appreciate if Sir can add further to it.

Six Steps to Useful Outcome Evaluation

So, I came across a presentation which outlines six steps to useful outcome evaluation. I will briefly share:

Step # 1: Mission and Goal Statements
Mission Statement: Are the problems that the said organization wants to address properly identified? Along with target population and desired results?
Goals: Broad statement of desired outcome(s).

Step # 2: Outcome Objectives
As we studied in the formulation of problem, this has to be either an increase or decrease of something. In terms of change. Long term outcomes should be related to short and medium term milestones necessary for the achievement of the goal.
Example: As in my report, CSF plans to increase Pakistan's competitiveness rank to 55 from 101.

Step # 3: Performance Standards
How much change is expected, how many targets will be affected, and by when.

Step # 4: Measures and Indicators
Defining the yardstick through which you will measure the expected change.

Step # 5: Data Collection and Analysis
Pretty self-explanatory.

Step # 6: Dissemination and Program Modification
Interpreting the degree to which goals have been achieved, what changes should be considered, if any? for improvement. And finally, how these deductions will be shared.

What are the different Criterion for Policy Analysis ?

Dear Policy Analysis Students

Please tell me the different criteria which you have encountered in Eugene Bardach book. What are the trade-off between these criterion ?

An insightful response will be appreciated. However, if you want to narrate different criterion for sake of explanation for others class-fellows. It will be greatly appreciated.

Policy Analysis - How you will come up with policy alternative ?

Dear Students

Now you are at stage where you are working on your policy alternatives. Please tell me how you will design your alternatives - what special consideration you will have, when you will be suggesting your alternatives ? What should be the total number of alternatives ? and Why ?

What are the Problems with Pre-Test Post-Test control group design

Dear Students

We discussed about design consideration and focused on Pre-Test Post-Test control group design. What are the problems with this design ? Please mention case where you would not like to use for your studies.

Also, there will be great appreciation in terms of grade for those who can initiate a new thread!