PRDV009: Writing Grant Proposals

This section is about setting up an evaluation process, i.e., choosing the evaluators to carry it out and planning what it will look like. The two are closely connected, but it probably makes sense to choose your evaluators first. Then, consult with the organization, community, and target population to devise a plan to help you get the information you need to become or continue being effective.

 

Why Pay Attention to the Selection of Evaluators?

An evaluation is not simply looking at your organization and saying, "It is doing okay." It should examine what you are doing from several angles and perspectives, give you a clear sense of whether you are accomplishing your goals, where your strengths and weaknesses lie, how to improve your work, and offer new directions to explore. No matter how well your evaluation is planned – and it should be well planned – you need evaluators with the skills and knowledge to look at your situation accurately.

The evaluation should help you improve your organization. Accurate information and analysis will help you identify and build on your organization's strengths and pinpoint the areas that need work. A well-conceived evaluation can help you adjust to changing community or target population needs, learn more about an issue's implications, provide a base for advocacy, and help with fundraising.

You can hire evaluators or choose volunteers from your staff or the community. You may also create a planning team to select evaluators. In any case, the individuals you choose will influence the shape and results of the evaluation.

Although we are primarily talking about professional and non-professional evaluators, most organizations conduct in-house evaluations (i.e., using staff members, participants, and/or organizational records to gather and analyze information), use an individual or team from the community, or find a professional to help with an evaluation without charge.

The same is true for community and other planning teams. A planning team is one possibility, depending on an organization's resources, which include time. An imperfect evaluation is useful and better than no evaluation. A planning team is a great idea, but not if putting it together and working with it prevents an organization from attempting an evaluation.

If you hire professionals, such as a consulting firm or university group, they will have particular research methods, prejudices about what an evaluation should cover, and particular interests, which may or may not coincide with what you or your community wants. Depending on the individuals involved and the groups they represent, community and in-house evaluators also have prejudices, interests, and needs that may influence how they view the evaluation. The same goes for graduate students or others who may donate their time to conduct or assist with an evaluation. It is extremely important to choose people whose agenda matches your own or who will put their agendas aside and respond directly to the needs and desires of the organization or initiative and the community. You will wind up with an evaluation that best serves your purposes.

 

When Should You Choose Evaluators?

The short answer is "as soon as possible." Evaluations should not start at the end of a project. They should be ongoing so you can continually use the information to improve your work. A planning team should probably be chosen when the organization or initiative begins, or even before so that an evaluation can examine what the organization has done and how it changed over the evaluation period.

Many funders require evaluation from the start of their grants. For instance, the grant for an adult literacy program run by a Massachusetts employment training agency stipulated that 10% of the money be used for evaluation in the program's first year. A University of Massachusetts graduate student was hired at the start of the program, and he conducted an evaluation that involved visiting classrooms several times during the year, meeting regularly with the program staff, and interviewing participants. By the end of the year, he had gained a complete picture of the program and a sense of its movement and growth over time. The evaluation he presented was extremely helpful in guiding the program over the next year.

As will become apparent below and in succeeding sections, It is important to know your community and its needs, as well as the needs of your organization or initiative, before you set out to evaluate. If your organization or initiative is just starting, it may sometimes make more sense to wait a while until you have a clear understanding of the context of your evaluation before beginning to pick evaluators.

 

Context of the Evaluation

An organization with the resources may or may not decide to hire professional evaluators. The context refers to the unique situation of the evaluated organization or initiative and its community, the combination of geographical, historical, political, social, cultural, and other factors that form that situation. Understanding that context is basic to understanding how to approach this particular community to evaluate this organization.

An example: An evaluation of an attempt to establish a rural adult literacy program initially showed that the program failed to attract many students. It looked like a failure and risked losing its funding. But the program staff knew two things the evaluators and funders did not: the area's inhabitants were suspicious and slow to accept "outsiders" and that other programs only lasted a year because they did not attract enough students. The staff convinced the funders that time was the missing element in earlier efforts. The programs that left confirmed the belief that the outsiders did not care about the community's problems and were not serious about helping. The program was thriving by the end of its second year because the local people believed it would stay. Now, 15 years later, the program has become a fixture in the community and is always full.

 

How Should You Decide Between Professional and Community or Other Volunteer Evaluators?

Evaluation is a type of research that entails systematically gathering information and drawing conclusions. Students, academics, and consultants engage in several types of research for a living. So, hiring trained researchers to conduct evaluations makes sense, right?

Well, yes and no. In many cases, a professional researcher will do the best job. But in other situations, that may not be true for various reasons.

 

Before you Decide Who is Going to do your Evaluation, There are Several Things to Consider

Money

This is usually important, especially for new and small organizations. Generally, the more grassroots your base is, the less money you have. Hiring professionals is an option if you have a funder who requires or is willing to pay for an evaluation as part of a grant. You may try turning to the community or other volunteers if money is unavailable.

If you believe you must have professional evaluators or guidance, you might try fundraising what you need (a difficult alternative unless you have a lot of lead time and a great community base) or finding a volunteer mentor at a local college or university. Graduate students may be willing to consult or conduct the evaluation without charge for the experience or because the research aligns with their dissertations or works they are already doing. Other possible sources include other health or human service agencies, local government, or a fundraising drive or proposal (such as to a local foundation, for instance).

For example, a graduate program evaluation course instructor solicited proposals from community agencies and organizations to conduct their evaluations. The organizations with successful proposals received a team of grad students who worked on evaluations with them over the semester. The agencies were invited to a public event where they learned the results and received program evaluation tips written by the grad students. This process produced useful evaluations for the agencies, invaluable hands-on experience for the grad students, and a strengthened connection between the university and the community.

If you can consider professionals, the issues below come into play.

 

The Complexity of the Evaluation

For some organizations or initiatives, an evaluation may be relatively straightforward. They simply want to measure whether their specific goals are being met. Are more children being vaccinated this year after a public education initiative? Are learners in a literacy program gaining proficiency in reading, writing, and math? What percentage of participants in a substance abuse treatment program are substance-free after a given period? The evaluation can be fairly simple for these types of questions.

But what if the questions asked are more complex? Did the initiative, some other factor, or a combination increase vaccination rates? What environmental factors influenced student literacy? What do the substance abusers who backslide have in common? The methods for answering these questions may also need to be more complex. This issue is muddied by the fact that deep knowledge of the community may do more to answer complex questions than knowledge of research methods. You must decide whether it is the best idea to use professionals.

 

The Type of Information Desired and How it Needs to be Analyzed

Concluding quantitative data (i.e., numbers) may be impossible. Researchers use statistical procedures to tell you what the numbers mean, their significance (i.e., the odds that what you are measuring caused the differences or changes), and what they imply about other operating methods. If you need this information or funders specifically ask for it, you may need to hire professional researchers or find some volunteer professional guidance.

Statistical procedures may tell you important things you never expected to find. For example, a researcher trying to determine what prompted a group of women to return to school in their thirties and forties found that a major factor was being co-opted to care for younger siblings when they were teenagers. He barely noticed this trend in interviews since it was overshadowed by more dramatic factors such as divorce, substance abuse, and domestic violence. However, computer analysis showed this factor was decisive. When he returned to talk to several participants, they acknowledged that abandoning their education at such a young age still loomed large in their lives and decision-making.

 

What Are You Using the Evaluation For?

Are you evaluating to help you improve what you do or increase your credibility in the community? Perhaps your continued funding depends on it? Is it a combination of these or other factors? Your answer will help determine the questions you should ask during the evaluation, the form the answers must take, the complexity of the evaluation, and whether you must hire researchers. The choice of professionals or community volunteers will depend on these factors and your situation. Professionals may analyze data better, but they may not have access to the same information as people from the community (or they may have better access because they are perceived as neutral). Once again, you need to determine what is best for your situation.

 

How You Want the Evaluation to Be Perceived

Some organizations or initiatives may want to demonstrate their community-based or grassroots orientation by ensuring their evaluation is community-based. Others may want to show their evaluation is objective by hiring evaluators without a community connection. Others may want to involve those they serve or benefit from the evaluation process for philosophical or practical reasons. There are many variations and possibilities: legitimate, right for some organizations, and wrong for others. As with anything you do, it is important to be consistent with your mission and principles when choosing evaluators.

 

What Should You Look For in Choosing Evaluators?

You want certain characteristics in professionals and a community planning or evaluation team.

An evaluation can be structured in many different ways. The split between professional and non-professional can blur many possibilities. For example, you might get a professional evaluator for free due to their interests or commitment to your organization or the community. While a professional, community or in-house group could perform the evaluation, many variations on these themes exist.

A paid evaluator or team will probably work with a group from the organization or the community. A paid evaluator might work with a community or organization planning team to help devise the evaluation and conduct it without further input from the planning team. Or they may withdraw entirely to have the planning team take over arranging and conducting the evaluation. A local planning team might independently plan the evaluation and hire a professional to carry it out or just analyze the data. Professionals might work alongside a community team throughout the whole process. Whether and how you use professionals, community, or in-house planning evaluators depends on what makes sense for your organization.

For example, one evaluator says, "I was hired to help community members evaluate HIV/AIDS services in the community. The community members here were a mixed group, mostly agency folks but some nonagency people living with HIV /AIDS. I asked for some volunteers to plan the evaluation with me and was pleased when about half a dozen came forward.

The evaluator then conducted the evaluation and analyzed the data independently before presenting it to the community.

 

Characteristics to Look for in All Evaluators

Willingness to leave one's agenda at the door

Often, researchers, particularly those attached to universities, may have reasons for embarking on an evaluation. It may fit into a doctoral dissertation, a book a professor is writing, or a piece of long-term research that will eventually be published. They may also have strong prejudices about the kind of research they want to use or what they expect to find and a need to prove their prejudices correct. They may have issues about power and their standing in relation to that of members of a planning team or organizational staff. If the researchers' needs mesh perfectly with yours, then there is no problem. But if they do not quite fit, there can be a serious problem. If you are paying for a service, you should get what you are paying for and not simply what the researcher wants to give you. It is important to be clear about this on all sides at the beginning and to make sure by writing it into a contract or through some other formal mechanism that the professionals are willing to do what meets your needs, not theirs.

Just as professionals may have research priorities that have nothing to do with the evaluation, community members may have personal or political priorities that have little to do with the business at hand but can affect their performance on an evaluation team. As much as possible, people should put these priorities aside while engaged in the evaluation process or screen out those who will try to impose their ways of thinking on everyone or use the process to further their ends.

Ability to communicate with a broad range of people

Evaluators must deal with people from all walks of life, including all political, religious, philosophical persuasions, and many ethnic, language, and racial groups. To gather accurate information, these groups must perceive them as trustworthy with a certain level of comfort.

Have they worked with groups that included a broad spectrum of community members? Do they have the verbal and interpersonal skills and the patience to explain their methods clearly to people who have never had contact with research before (especially if the evaluators are in that same category)? Can they speak some of the language of the community?

Interview potential evaluators before hiring them. Evaluators must gather the information for the evaluation and couch the results in terms the community can understand and use. The best evaluation plan is worthless if the evaluators' approach means people cannot or will not respond to them or cannot use the results they have gathered.

Cultural sensitivity

Especially in a community that contains residents of many cultures (depending on how you define "culture," that includes virtually every community), mutual respect and some understanding and acceptance of how others see the world is crucial to the functioning of evaluators. Do they understand, or are they willing to learn to understand, the cultures of those in the community? An urban, largely working-class community is culturally very different from an upper-middle-class suburb; a Haitian neighborhood differs from a Vietnamese or a Puerto Rican one. Evaluators need to respect the cultures of the communities they work with and not violate them, intentionally or unintentionally.

Ability to treat everyone with the same degree of respect

How evaluators approach people reflects on the organization. If they do not treat everyone respectfully, they will not get accurate – or any – information, and they will complicate the organization's relationship with its staff, the target population, and the community.

Absolute commitment to keeping all individuals' information confidential

Whether evaluators are paid professionals or not, for ethical, practical, and legal reasons, it is almost always necessary to guarantee that any information gathered in the course of an evaluation will be kept confidential (i.e., used only for the evaluation and not connected to the individual) and that people will not be identified either by name or by other factors that could lead to them. In the case, for instance, of the evaluation of a domestic violence prevention program or women's shelter, confidentiality could be a matter of personal safety. In most instances, it will help evaluators obtain more accurate data. It will protect the evaluators and the organization from lawsuits that could be brought by individuals injured in some way by the information they provided.

Professional or university researchers often ask informants to read and sign an "informed consent" form that explains exactly what the researchers are doing, what any information will be used for, and how the researchers will protect confidentiality and anonymity. If the researcher violates the terms of the form, the informant might have grounds for a lawsuit, and the researcher's findings might be questioned. An evaluator, professional or not, who asks for responses from members of the target population or the community might consider using a similar form because it helps explain what the evaluation is about and demonstrates the evaluator's commitment to confidentiality.

Commitment to the evaluation process

For planners and evaluators, whether professionals or otherwise, this means trying to do the best evaluation possible, with an eye toward its usefulness for the organization. For the organization and other community members, commitment means believing enough in the process to take the evaluation seriously and use it to adjust and improve the program, service, or activity. No matter how elegant and informative, an evaluation is worthless if it is not used.

An important part of a first evaluation is a plan for continued evaluation. To be truly useful, evaluation must be ongoing throughout the organization's life. If evaluators can provide a guide for setting up and conducting evaluations on a regular basis, they will have performed a service to the organization far beyond the value of the evaluation itself.

 

Community or In-House Planners or Evaluators

A word about planning vs. evaluation teams: A planning team is meant to plan the evaluation process. If the organization can hire professional evaluators, a community or staff planning team may decide whether or not to do so and may serve as the hiring committee for those evaluators. Its main job (which it may do alone or with the help of a hired evaluator) will probably be to examine the possibilities and come up with an actual evaluation plan, detailing what the organization needs to find out, where it can find that information, who will do the information gathering and analysis, and how the information will be used once It is gathered and analyzed.

A community evaluation team is a volunteer team set up to conduct the evaluation. In practice, this may be the same group as the planning team, whose function is to accomplish the task once the planning is done. It is helpful if the evaluation team includes someone with enough understanding of the research so their evaluation will be valid and its results can be used with some confidence.

You can choose all the members individually with a community planning or evaluation team. There may be some people who, because of their position in the organization or the community, have to be on the team. By the same token, some others may not be able to be included. In general, the team should be small enough so that it will not be too difficult for members to meet and contact one another and large enough so that there are enough people to do the work. And, in general, the team members should represent the interested stakeholders in the organization or initiative: staff and Board, target population, and community members at large. Characteristics you might look for in those on your planning or evaluation team include:

Representation

The team could include any number of the following possibilities (but, for most organizations, would probably number only four or five):

• A range of organizational representatives (e.g., the director, one or two line staff members [those who work directly with the target population], and a Board member)
• Members of the target population, ideally individuals who understand and have participated in the organization or initiative
• Community members with a stake in the organization or initiative (business people, health and human service agency representatives, school and local government officials, and other citizens with an interest in the issue)
• Youth, if appropriate
• Relevant language and other minority groups
• Someone with research skills and experience, perhaps from a local college or university.

Ability to understand the purpose of the evaluation

It is important to understand what the evaluation is supposed to measure and how its form and purpose are related.

Willingness to listen and learn

This covers a wide spectrum of people and behaviors. It includes the local high school principal being able to listen to and take the opinions of a high school dropout seriously and that dropout's belief that they can learn what they need to know to contribute to the team. It also includes everyone being aware that they have things to learn about the process, one another, and evaluation in general, and, by the same token, they have valuable knowledge, skills, and information to contribute.

Ability to work in a group

This can be sticky in a group encompassing several cultures and classes. Welfare recipients or migrant workers may not have been exposed to the "meeting skills" that most educated people have been unconsciously learning and practicing since junior high. They may lack the confidence to speak up or act in ways others deem inappropriate. People without meeting skills need support and encouragement in an intimidating situation. You might pair all team members so those needing support have a mentor. You might start by explaining group dynamics and meeting skills so no one is singled out and the ground rules are clear for all.

 

Professionals

Unless you work with an individual consultant, a team will likely come assembled when you work with professionals. Thus, you are looking at the whole package: who the team members are, what they know how to do, and how they are likely to interact with your planning team, if you have one, and with your organization and the community. As mentioned above, most professional teams will be private consultants or university researchers. Probably the most important features to look for are their professional experience and skills and their expertise and style in working with community groups:

Knowledge of different kinds of evaluation techniques

Can they use quantitative (numbers and statistics) and qualitative (facts, stories, anecdotes, analysis of situations and events, etc.) research, and do they know when each is appropriate? Have they used them in situations like yours, or do they have good ideas about how to do that?

If your team is from higher education, you should consider carefully what department (s) of a university to contact to find evaluators whose professional and scholarly specialties are a good fit for the work you do. Depending upon what your organization or initiative does and what information you are looking for, some possibilities are:

• Public Health
• Medicine
• Education
• Sociology
• Psychology
• Environmental Sciences
• Urban Studies

Past performance

Have they done evaluations before? What techniques have they used? Do they have good references? Can you see examples of past evaluations they have done?

You would probably not hire someone without checking their references, so you should contact previous employers. An interview and resume will probably tell you what you need to know, but it can be disastrous when they do not, costing you money, time, results, and relationships. Ask to see the results of previous evaluations. Are they readable and understandable? How did the candidate explain them to the organization and the community? Knowing what you are getting can help you choose the right evaluator.

Experience working with community groups

Have they worked in partnership with community groups before? Do they understand the roles in this situation, and can they handle theirs well? Have they been able to work with the community to address the appropriate goals? Are they responsive to suggestions, to the community's needs, and the context of the evaluation? Do they collaborate, in other words, rather than telling the community group what it needs?

Ability to listen

The evaluator should be able to listen, respect others, and not assume the "expert" role unless it is appropriate (you are hiring them for their expertise). If the evaluation is community-based, the evaluators must listen to the planning group about what the organization or initiative needs to know and then to the community to gather that information. Real listening (listening not only for the meaning of what is being said but for the meaning behind the meaning) is an indispensable skill for all good evaluators.

 

In Summary

The individual or team you select to plan and/or conduct the evaluation of your organization or initiative will do much to determine the character and usefulness of the evaluation itself. If you work with professionals, you need to consider their credentials, experience, and level of expertise. Whether you choose professional evaluators, community volunteers, or some combination of the two, you need to think carefully about how their needs and interests fit in with those of the organization and the community, about the range and quality of their communication skills, about their willingness to work as partners with the organization and the community, and about their fit with one another. Once you have put together a good team, you are well on your way to carrying out an accurate and valuable evaluation.

---

Source: University of Kansas, https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/choose-evaluators/main
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.