EdTech 505: Week 12 - Critique of an Evaluation Report

Assignment: Critique an Evaluation Report ("The Maine Experience")

Discussion Posts:

#1: Program Evaluator Job Ads: Discuss what you find, or don’t find, in these ads. What do you think of the jobs as described? How do you stack up to the qualifications and requirements? Think you could make “program evaluation” a PT or FT business? Other thoughts on the job ads?

These jobs certainly do reflect the skills we've overviewed and practiced in this course. Their descriptions include most of the components we included in our Response to RFP (meetings with stakeholders, designing various eval methods, traveling and overviewing the administration of the evaluations, collecting and analyzing data, offering a report and making suggestions if requested).  Upon first glance, they don't seem TOO out my league after working through this course (and from my previous experience & educational coursework).  I know a job poster is going to err on the side of the VERY specific! However, there are some things that made me feel a little ill-prepared to take on one of these positions.

First of all, many jobs ask for the applicant to specify their evaluation skills (e.g. "experience employing social science empirical research methods, theories and analytical approaches" or "experience using a mixed-method approach").  Outside of learning the theory and ideal techniques of program evaluation and completing one formal evaluation, I simply don't have these kinds of extensive and varied experiences! In addition, very few of the jobs are seeking teachers or educational technologists.  A large amount of positions also request expertise in statistical/data analysis, including special software packages.  This is one area that is definitely still quite out of my league!  The words "metrics", "audits" and "IT" come up quite often. I would have a big learning curve with those aspects.  Many also are looking for people with experience in project management, which I just don't possess. I haven't pursued any administration coursework or experiences in my career so far. It makes sense to me that many request experience with research or grant writing.  (I do have some of that.)

Overall, it seems that the ideal candidate being sought after is one who not only has experience in the specific field (healthcare, social services, etc.), but also has experience being on the other side of the table (a "leadership" stakeholder -- instructional designer, grant-writier, program manager, etc.)  They also want an excellent writer (many requesting sample of one's work), mathematician, designer, coordinator, etc.  These jobs seem pretty intense! I would've thought they would pay more than they do.  (At least in the case of the ones that post their salary...)  A lot of work is involved and a lot of expertise is expected. (Where are the entry-level positions?!)  For now, for me, I'm content with using program evaluation skills within the context of my current teaching career.

#2: Discussion of end of book

Yes, the more resources (and exemplars) the merrier.  One big difference, I feel, between Appendix B's example report and the one we're required to to for this course is that there is greater treatment of "recommendations" than we are required to provide in our own report's discussion. Appendix B also reports results according to each program objective, however I don't believe that is expected of us in this course's final report (although, if it makes sense, I can see how that would help organize and frame the discussion of data).  Appendix B's report also includes a lot of data from focus groups and interviews, which I understand that we were discouraged from using in our own report for this course.

Chapter 10 was a good pep talk for evaluators looking to permeate the professional field. Being proactive in the search for work seems like a daunting task!  At least it ensures you are evaluating something you're interested and experienced in.  The discussion of "emergency resources" was interesting, as it got me to consider the ethical dilemma of evaluating -- is the relationship with the organization or the evaluation results more important? Definitely a tricky scenario!  The discussion of contract negotiation was definitely something I'd never experienced before. Program evaluation seems quite entrepreneurial in nature. A lot of self-direction, seeking out clients, and building a reputation is involved -- not to mention the evaluation processes themselves!  The first few years must be tricky.