Evaluation+Philosophy+Paper



Philosophy of Program Evaluation Statement

Introduction
Education in the United States is often a public endeavor, supported by taxpayers, and as such, the various programs within "education" ought to be regularly evaluated for effectiveness. It is important that practicing program evaluators develop their own philosophy of evaluation. In particular, they ought to determine the purpose of their work (to what end?), how they achieve it (by what means?), to what degree they do it, and why. In this paper, I offer my philosophy of evaluation with the full understanding that as I work in this field, this philosophy would evolve.

To What End?
The prompt for this evaluation philosophy paper has me assuming that I am an evaluator. So, if I am an evaluator, what are my objectives as an evaluator? I believe that the methods of an evaluation are dictated by the situation.
 * In the contract, clearly outline to the client what I am able to offer, and procure necessary resources to successfully complete a given evaluation. (Resources)
 * Identify my audience (client and stakeholders), and determine the appropriate methods for evaluation. (Audience)
 * Be fair in recording and representing the data. (Fairness of Reporting)

My evaluation questions for my spring 2011 evaluation of the dissertation process of the School Improvement Program were generated as a result of two conversations with the client. During the first conversation, I came in with some ideas for what I wanted to do, but as our conversation continued, I heard a different need emerging from my client. I took notes during both conversations, and then I crafted initial questions. I sent these questions to my client several times for feedback, and he did not recommend any changes. As a result of sharing with the members of my group and with my professor, the questions evolved to what they are now. The table below shows the initial questions and the final questions.

Table 1 Evolution of evaluation questions Initial Questions 1. What should be the dissertation process as defined by the program administrators? 2. What is the current dissertation process for dissertation chairs throughout the process? 3. What is the current dissertation process for students before, during, and after the dissertation process?

Final Questions 1. What is the official dissertation process as defined by the program director, administrative support staff, and official records? 2. What are dissertation committee chair perceptions and experiences of the dissertation process? 3. What are doctoral student’s perceptions and experiences of the dissertation process? 4. How can the dissertation process be improved?

The questions did not change a great deal, but I added the fourth question about whether or not the dissertation process could be improved. My professor recommended that I include a yes/no question, but the closest thing I could come up with that would fit my study was Can the dissertation process be improved? I felt the question would be better answered as a more open-ended question, which is why it is phrased as above.

By What Means?
In the section above, I shared four philosophical objectives by which I would function as an evaluator. I would now like to take a closer look at each objective and explain how I would achieve it.

Resources The first objective is this: In the contract, clearly outline to the client what I am able to offer, and procure necessary resources to successfully complete a given evaluation. If I cannot procure the necessary resources, then I need not conduct the evaluation. I am currently in "graduate student mode," so part of my job is to do the work without pay. But as I grow more competent in my skills--in this case, my skills related to evaluation--I have to become more confident in my right and responsibility to make my own reasonable demands. If I am to do a competent job at conducting a program evaluation in the future, a clear contract with boundaries and responsibilites and compensation established is an important tool that I will use to help the process of communication, to manage expectations on both sides, and ultimately ensure that the evaluation is of decent quality.

Audience As I work on evaluation projects, I need to have a keen understanding of my audience because each audience brings certain values to bear in the evaluation process. When dealing with high-level decision makers, for example, the value of "accountability" may be paramount. Such an audience may need to know what the outcomes of a given program are. To measure outcomes, I may want to look at quantitative tools and measures to answer the questions of interest. On the other hand, my client and stakeholder audience may consist of powerless groups. I may be asked to evaluate existing inequities in a program or system. The methods needed to conduct such an evaluation may have to be more qualitative in nature. Each audience brings particular values to the evaluation situation, and these in turn must dictate the methods. For my part, I hope to become competent with both quantitative and qualitative methods so that I am able to respond with the appropriate evaluation approach, regardless of client.

Fairness of Reporting It may seem obvious that as evaluators we ought to "be fair in recording and representing the data." However, given that evaluations tend to cause anxieties in the people and groups under study, it is easy for the evaluator--whether novice or experienced--to succumb to political pressures to change the way in which the final story is told. It would not be difficult to omit some data and highlight other data. Few people desire to look at the primary sources of data collected and even fewer are authorized to do so. It is up to each individual researcher to maintain high ethical standards and report on no more or less than that which has been observed. I am not sure what tools and strategies I would use to do this. I personally believe that either you do this or you do not. I feel that fairness of reporting is one of my strengths to a fault, which is why I take so long to do anything. Rather than gloss over things, I agonize over the details, wanting to make sure nothing is left out. I will continue to do this, but my challenge will be to become more efficient if I am to survive data collection and reporting. I believe that technology tools such as SPSS, Excel, and Dedoose will help me to become more efficient while keeping honest with what the data is saying. I have used SPSS this past semester as well as three years ago, and I will use it again this fall for quantitative data analysis. For qualitative data analysis, I have used Microsoft Word, and, with greater success, Microsoft Excel. I believe that Dedoose will be an excellent tool for mixed methods research.

To What Degree?
I will know that I am effective if I derive satisfaction from my evaluation projects. If I do not enjoy this work, I will not last long doing it (unless it is the only way to earn my daily bread in life; then, I will make myself love it). If I do not enjoy the work, I may begin to "slip" on the "Fairness of Reporting" data. Or, I may cave into some political pressures from some member of my audience. This, in turn, may erode my enjoyment of this work.

I also will know that I am effective based on whether or not I get hired to continue conducting evaluations. My simple logic here is that if I continue to be hired, the world of organization improvement is saying my work is good. I do not wish to naively suggest that just because one is repeatedly hired to conduct evaluations that the quality of these evaluations must be excellent. It is possible that an evaluator is just good at working with people. On the other hand, I believe it possible to conduct and excellent and sound evaluation, only to have undesirable consequences follow (lack of future work, for example). For instance, the evaluation could be completely ignored by the client. Perhaps the evaluation revealed problems too big for the client to solve. Another result of an excellent evaluation could be that the client or others in the organization are fired. This could make others in positions of power distrust the evaluator, blaming her for these upsetting workplace changes, when the real issue may be organizational culture. This may ultimately result in the evaluator getting a "bad reputation" and not being hired for future work. Still, this measurement of effectiveness based on whether or not I continue to acquire work is one of the best I have for now.

So, again, I would rely on 1) how I feel about my work, and 2) whether or not I am able to find regular employment conducting evaluations. I am sure that if I were to become an evaluator, I would develop more detailed self-evaluation criteria.

Why?
Program evaluation is important because it keeps everyone mindful of the organization as an entity that serves a purpose. It is very easy for the individuals in an organization to become comfortable in their role, feeling entitled to being employed. It is easy to forget that the organization has a purpose outside of individual needs and interests, and these constantly evolve. Through regular evaluation, an organization can ensure that it is doing what it is supposed to do and serving the audiences it is intended to serve. If it is not doing this, then the program needs to be discontinued or changed so that it can meet the needs of its constituents.

Conclusion
Personally, over the past five years, I have become increasingly interested in online and blended learning environments, which are increasingly being used as a method of "technology integration" in public classrooms. The innovation is here, but is it being used to meet real learning needs? Are students benefiting from it? As I continue to study the fields that are involved with "K-12 online course quality," I hope that I am able to frame my conversations and interventions with this evaluative mindset in mind. And, I hope to become proficient with the tools and methods necessary to conduct such evaluations.