One of the keys of being successful in evaluation is keeping the existing clients coming back for more. I recently came across a blog post emphasizing the importance of having a follow-up meeting immediately after a project is over. The article points out the following:

“A follow-up meeting helps to enhance value, fortify trust and cement the relationship. It helps you do a better job next time around. It differentiates you from the other guys whose policy is: Do it, bill it and forget it.”

The full blog post can be found here: http://freelanceswitch.com/the-business-of-freelancing/follow-up-meeting/

Image: jannoon028 / FreeDigitalPhotos.net

AEA and CDC are hosting their annual Summer Evaluation Institute June 3-6, 2012 in Atlanta, GA.

On June 5, 2012, famed educational researcher Robert Marzano, PhD will be speaking at Fremont High School in Fremont, Nebraska.

* Image: on Flickr.com by mezzoblue

I decided to survey six of my peers.  Yes I know that is a puny sample, but it’s a start!  I heard some of the sentiments below.

It means a critique of whatever is being evaluated.

It means I may be not be doing something as well as I should and I am about to hear it.

It means that another budget cut is on the way, it’s all about money anyway.

It means some consultant is coming to tell us what we already know.

I don’t really worry too much about it since I am not going to do research. I have taken the necessary classes and I am done for now.

It’s too confusing with all these talk of internal, external, outcome and what have you. Who knows what they mean anyway?

I also thought about some of these and as someone interested in evaluation,  I couldn’t pick out sentiments or acknowledgment of the great value that evaluation has in informing what is valuable and effective.  I notice that evaluation terminologies were a constant source of confusion.  I realize that this is a social work program and the emphasis is not on evaluation as a career goal.  However, graduate social workers will often end up becoming program directors, agency coordinators etc.  It is important that we are aware of the value of evaluation but we need to understand what it means.  So what should evaluation look like to the novice?  How should it be understood?

* Image: on Flickr.com by Jordanhill School D&T Dept

What do you want or need from our next evaluation conference?  Please comment with exactly what would get you excited about attending our next conference.  And when/where/how would you like that conference to be held?

Look forward to reading your comments!

* Image: on Flickr.com by boellstiftung

Where are we as a profession? The Occupational Outlook Handbook provides labor statistics for the United States Department of Labor. It summarizes employment projections for the 2010-20 decade (http://www.bls.gov/ooh/).  If you search for ‘Program Evaluator’ on their site, you find yourself in an interesting mix of Political Scientist, Home Health and Personal Care Aides, Industrial Engineers, and Quality Control Inspectors.  The closest I could find to my profession, as I see it, was survey researcher. If I instead search for ‘Evaluator’ I find sales manager, financial analyst, and actuary, insurance underwriter, and market research analyst. Hmmm…do you ever feel like others don’t understand what you do for a living?

My family and I traveled to Florida over spring break. As I was in line for a ride—an opportunity for extended discourse with strangers you’ll never meet again—I struck up a conversation with a nice couple from Canada. He was a businessman and she a nurse. When I tried to explain what I did, she said, “So you’re a judge?” No, not exactly.

My introduction to evaluation came just 11 years ago. I had not heard of it prior to 2001. As I worked as a research assistant under a program evaluator, and completed course work in program evaluation, I began to understand its role and potential impact on services to children and families. My life had been spent in education—teaching and administration. Now I saw a way to impact more broadly the community and state I serve through quality program evaluation services. Services that I hoped would continuously improve the quality of services funded on behalf of children and families.

So my question to each of you is this:  What is your elevator speech summary to describe what you do as an evaluator? An “elevator speech” means it has to be short—very short—and capture only the most importance essence of what you do, enough so that the person can begin to understand your message in just a few seconds.

I’m eager to begin a dialogue about what matters most in our profession. Perhaps for the next decade, 2020-30, we’ll actually see ourselves in the Occupational Outlook Handbook?

A couple weeks ago, she came home excited to show me a phonics website that she learned about in her 1^st grade computer class. The website allowed her to pick interactive stories to read. At the end of each story, she was asked for her opinion.

My brilliant daughter was participating in evaluation! Her excitement about being asked for her experience and knowing her opinion mattered was clear. This simple and fun evaluation intended for early readers made me stop and think about my own work with evaluating child and family service programs. As evaluators, do we make our work harder or more complicated than it needs to be at times? Do we think enough about assuring that participating in evaluation is a positive experience?

I have a primary interest in interagency coordination and have spent a lot of time exploring measures of collaboration. It’s often complicated and unclear as we define units of analysis and operationalize the concepts. I work largely with projects that are focused on culture change – creating paradigm shifts in their communities – which are difficult to isolate and measure. Recently, the multi-disciplinary team that I am a part of submitted a grant application requiring a fairly sophisticated cost analysis as part of the evaluation plan – a new evaluation approach for us that we spent countless hours struggling to understand and apply.

In contrast to these challenges at work, when I come home I am greeted by my daughter who, since discovering the happy face evaluation, is routinely presenting me with cards, notes, and pictures with hand-drawn faces to gauge my opinion. She moved from engaging in evaluation as a participant to being an internal evaluator! Now, in addition to knowing her experience matters, she is seeking clear feedback from others to guide her projects. Again, I was struck by her excitement and sense of satisfaction. Each time I complete one of these happy face evaluations for her, I’m reminded of two important points: 1) Evaluation can be simple. There is value in keeping measures clear and focused. And 2) Evaluation can be exciting and satisfying for all parties – evaluator and clients (both programs and ultimate consumers). It is essential to think about the experience of participating in evaluation and monitor that experience. The relationship between evaluator and client is an important component of that experience.

I’m reminded of a presenter that I heard several years ago – Scott Miller, PhD, a well-known counseling psychologist and founder of the International Center for Clinical Excellence (http://scottdmiller.com/), who developed tools for therapeutic work with clients that use measuring experience and relationship to improve outcomes. The Session Rating Scale 3.0 (SRS) and the Outcome Rating Scale (ORS) allow clinicians (of all types) and their clients to monitor status related to the therapeutic relationship and intervention (the “who” and “how” of change) on a session-by-session basis. These simple tools – 4 items, even using different faces! – have been connected to improved retention and outcomes. Frequent monitoring of the “who” and “how” communicates that the client’s perspective is valued and serves to open communication about progress or any issues. This idea could be easily translated to evaluation work – a frequent but simple assessment of the “who” and “how” of evaluation with the goal of achieving and maintaining at least some of the excitement and satisfaction my daughter experiences engaging in her happy face evaluations.

So, as our evaluations grow in understanding  and measuring complex concepts and activities, how do you keep it simple? How do you build into your evaluation the essence of the happy face evaluation? How do you monitor the “who” and “how” of evaluation on a regular basis and keep the excitement alive?

While every job that brings together diverse stakeholders can use positive ways to open up conversation, I love this activity even more for distinctly cross-cultural or international settings.

CFCA Program Evaluation Workshop in Guatemala

I open the activity talking about the necessity of creativity and communication in designing, implementing and using evaluation. But we also talk about language and culture in general, how we can better understand each other through the course of the training or work and how evaluation has its own language which brings unique concepts and connotations to all the language backgrounds we are negotiating. I find that metaphor and simile are perhaps even more common in other cultures where language is not as direct or abrupt in meaning as my American English.

 I generally bring an array of objects with me to the meeting, but also ask some of the participants to bring a variety of items from their world (without yet knowing their intended use). This produces another visual metaphor of the cultures of stakeholders coming together. Some of my favorite metaphors have risen from objects brought in by participants.

 A machete is a tool that everyone can use in my community. It can be used to tear down and destroy. It is also used to bring in the harvest and to clear out brush to make room for new growth. Evaluation also can damage or give growth; it all depends on how you use it.

 For the participants, evaluation no longer seems like a foreign concept. It is something they can define. And while the activity doesn’t erase negative feelings or apprehensions about what is to come, it certainly brings them to the surface with the positive excitement and allows us to address them together. The metaphors also make great strides in helping me better understand the culture and experiences of those that I will be working with. The descriptions used give me language to carry through my training that is alive and relevant to the participants.

 So, while I may have to periodically explain to the TSA employees why I am traveling with a flower pot and screwdriver stuffed into my suitcase, the benefits of building metaphors and similes will keep them as central part of how I communicate and train on evaluation concepts.

What metaphors resonate with you or give you new perspective on evaluation?

At IgniteAEA, the Ignite event put on by the Data Visualization and Reporting TIG at EVAL2011 in Anaheim, Chris delivered five minutes of thoughts on why evaluators should be blogging. You can watch his presentation below.

“Blogs are great for putting everything in context” and allow you to “bring your experience and what you know as an evaluator” to the world.

I know what some of you are thinking: blogging is hard. Actually, it’s not. As Chris notes, “if you can email, you can blog.”

How much time do you spend writing emails to clients, colleagues, and others about your opinions, beliefs and thoughts on the topic of evaluation? Untold hours, I’d imagine.  Do you imagine you could donate one hour of your time one time per month to share your experience with the world?  Of course you could. Would you consider taking just one of the emails you craft each month and turn it into a sharable post? I bet you could. What a great service that would be to the field of evaluation. And imagine the feedback you could receive on your ideas from others.

You’re welcome to contribute right here in this blog.  We’d love to hear from you!

Infographics are one of the growing trends in presenting data and information. The idea is to take data and information and make it bright, colorful and pictorial. Often it is trying to bridge the fields of data analysts, marketing and visual graphics. By bridging these fields, they try to produce products that catch people’s attention, make them want to engage with the information and often help them make decisions or come to conclusions. Overall, they want to make the data approachable and usable.

In many respects, the goals of these infographics are the same as what we try to do as evaluators. We too are trying to produce data products that our funders, clients and stakeholders can use. We want our evaluation findings to catch their attention. We want them to engage with the evaluation results. We want them make decisions and changes based on the information that come from our work. Overall, we too want to make the evaluation data approachable and usable.

I think that in many respects the use of infographics, if done well, can further extend the reach and influence of our evaluations. For instance, I have been in charge of doing a large scale student survey in Wyoming for several years now. In 2008, I created for each county and school district a two-page, graphical fact sheet along with a larger and more comprehensive survey report. After releasing these products, I had the opportunity to interact with community members, prevention specialist and stakeholders about the survey results. Almost invariably when I did so, the data product that the community members pulled out were the factsheets and not the report. They liked the factsheets because they were short, and simple. They also had graphs that were relatively easy to understand. I cannot say that they were as fancy and graphical as most infographics, but it did show the power of accessible summaries.

The problem comes in the fact that infographics often want to emphasize the attention getting features of their presentation over and above what is considered good practices of displaying data. For instance, someone obviously spent a large amount of time creating this infographic:

The silhouettes of the different monuments and the amount of data packed into this chart are impressive. I know for myself it draws me in and makes me want to learn more.

However, as soon as I start looking at the graph I get lost. The graph presents the percentage rise and percentage fall in prices. It also displays the average hotel room cost in each city. When I try to track a given “ray” back to its “axis,” I invariably get lost and cannot tell what value the ray actually represents. It also raises questions like, how do I interpret the fact that the width of each ray gets bigger the farther it is away from the center point? Is there any interpretation of the height the different monuments? How did they choose the values to emphasize at the bottom of the graphic?

It quickly occurred to me that splitting this information into two bar graphs, one for the percentage change and one for the prices would vastly simplify and clarify the message. It would also help, in an electronic version to link to a table containing the values of this graph so that user can find the exact values. If it was a print version, the table could be printed on the back of the paper.

In the preparation of this article, I looked at many different infographics. I found that there some infographics that are better than others. However, what I also noticed is that it often took me much longer to understand many of the infographics than it would if the data were presented in a simple graph or table. A particularly notable trend is the use of circles to portray the data. This is despite the fact that Pie Charts and Ray Charts are so often misunderstood and misinterpreted. Almost invariably, if something could be portrayed as a bar chart, line chart or rectangular area chart, they would change it to something that is curved or even triangular shaped.

There may be many other concerns about infographics. For instance, many infographics are biased and are simply being used for marketing purposes. They also often do not identify their sources and it is next to impossible to make a judgment about how reliable and valid are the data.

Setting those legitimate concerns behind, I must admit that I also have my concerns about how usable they may appear at first glance against how usable they actually are. At first they seem to be a major step forward in making our evaluation results accessible and usable. However in practice, I am afraid that the most current practices obscure and distort a lot more than they illuminate and explain.

Let me know what you think of infographics? Are there any techniques, and presentations that your find particularly effective and straightforward when trying to create infographics or other similar summary materials? How do you balance the need to grab people’s attention with the need to present the data in a clear and unambiguous form?

Our post-conference poll was responded to by 75% of attendees, with 100% reporting as “satisfied” or “very satisfied” overall as well as with the panel session.  The breakout session yielded some dissatisfaction, and comments suggested it may have been improved with a better plan for input as well as a more tangible product or outcome to share with larger group – possibly via website.High satisfaction with the facility (University of Nebraska Omaha’s Thompson Alumni Center) and the conference location (Omaha) stand out.  However, comments suggest EN4MRB find ways to collaborate with the Research and Evaluation Exchange launched in Kansas City this past year.

100% of attendees were very likely or likely to attend again, and 25 out of 27 were just as likely to recommend membership to a colleague.  It should be noted that postconference, 41% were only somewhat likely to participate on the website.  Networking with people and learning of practice in other locations and disciplines were cited as “take aways” from the conference.  Finally, a number of comments noted the challenges to sustain personal interaction beyond the conference and to maintain an “intimate nature” as size of conference grows.  Good luck to new Program Committee.  I hope to see many of you at EN4MRB 2012!