PRDV009: Writing Grant Proposals

Evaluation Feedback Is a Gift

I am Christopher Lutz, a chemistry faculty member at Anoka-Ramsey Community College. When our project was initially awarded, I was a first-time National Science Foundation (NSF) principal investigator. I understood external evaluation was required for grants but saw it as an administrative hurdle in the grant process. I viewed evaluation as proof for the NSF that we did the project and as an outcome metric. While both aspects are important, I learned evaluation is also an opportunity to monitor and improve your process and grant. Working with our excellent external evaluators, we built a stronger program in our grant project. You can, too, if you are open to evaluation feedback.

Our evaluation team was composed of an excellent evaluator and a technical expert. I started working with both about halfway through the proposal development process (a few months before submission) to ensure they could contribute to the project. I recommend contacting evaluators during the initial stages of proposal development and checking in several times before submission. This gives adequate time for your evaluators to develop a quality evaluation plan and gives you time to understand how to incorporate your evaluator's advice. Our funded project yielded great successes, but we could have saved time and achieved more if we had involved our evaluators earlier in the process.

After receiving funding, we convened grant personnel and evaluators for a face-to-face meeting to avoid wasted effort at the project start. Meeting in person allowed us to collaborate on a deep level quickly. For example, our project evaluator made real-time adjustments to the evaluation plan as our academic team and technical evaluator worked to plan our project videos and training tools. Include evaluator travel funds in your budget and possibly select an evaluator nearby. We did not designate travel funds for our Kansas-based evaluator, but his ties to Minnesota and understanding of the value of face-to-face collaboration led him to use some of his evaluation salary to travel and meet with our team.

Here are three ways we used evaluation feedback to strengthen our project:

Example 1: The first-year evaluation report showed a perceived deficiency in the project's provision of hands-on experience with MALDI-MS instrumentation. In response, we had students make small quantities of liquid solution instead of giving pre-mixed solutions and let them analyze more lab samples. This change required minimal time but led students to regard the project's hands-on nature as a strength in the second-year evaluation.

Example 2: Another area for improvement was students' lack of confidence in analyzing data. In response to this feedback, project staff create Excel data analysis tools and a new training activity for students to practice with literature data before analyzing their own. The subsequent year's evaluation report indicated increased student confidence.

Example 3: Input from our technical evaluator allowed us to create videos used in academic institutions in at least three U.S. states, the U.K.'s Open University system, and Iceland.

Provided here are some overall tips:

1. Work with your evaluator(s) early in the proposal process to avoid wasted effort.
2. Build in at least one face-to-face meeting with your evaluator(s).

Review evaluation data and reports with the goal of improving your project in the next year.

Consider external evaluators as critical friends who are there to help improve your project. This will help move your project forward and help you have a greater impact for all.

Three Tips for a Strong NSF Proposal Evaluation Plan

I am Leslie Goodyear, an evaluator who also served as a program officer for three years at the National Science Foundation in the Division of Research on Learning in the Education and Human Resources Directorate. While there, I oversaw evaluation activities in the division and reviewed many grant proposals with evaluation sections.

In May 2016, I had the pleasure of participating in the webinar "Meeting Requirements, Exceeding Expectations: Understanding the Role of Evaluation in Federal Grants." This webinar was hosted by Lori Wingate at EvaluATE and Ann Beheler at the Centers Collaborative for Technical Assistance. It covered evaluation fundamentals, requirements and expectations, and evaluation staffing, budgeting, and utilization.

In the webinar, I shared my perspective on the role of evaluation at NSF, the strengths and weaknesses of evaluation plans in proposals, and how reviewers assess Results from Prior NSF Support sections of proposals, among other topics. In this blog, I will briefly overview some important takeaways from the webinar.

First, if you propose education or outreach programs, you will likely need to include some form of project evaluation. Be sure to read the program solicitation carefully to know the specific requirements for that program. There are no agency-wide evaluation requirements – instead, they are specified in each solicitation. Lori had a great suggestion on the webinar: Search the solicitation for "eval" to ensure you find all the evaluation-related details.

Second, you must ensure your evaluation plan is tailored to your proposed activities and outcomes. NSF reviewers and program officers can smell a "cookie-cutter" evaluation plan, so make sure you have talked with your evaluator while developing your proposal and that they have had the chance to read the goals and objectives of your proposed work before drafting the plan. You want the plan to be incorporated into the proposal to appear seamless.

Third, indicators of a strong evaluation plan include carefully crafted, relevant overall evaluation questions, a thoughtful project logic model, a detailed data collection plan coordinated with project activities, and a plan for reporting and disseminating findings. You will also want to include a bio for your evaluator so that the reviewers know who is on your team and what makes them uniquely qualified to evaluate your project.

Additions that can make your plan "pop" include:

• A table that maps out the evaluation questions to the data collection plans can save space by conveying lots of information in a table instead of a narrative.
• Combining the evaluation and project timelines allows the reviewers to see how the evaluation will be coordinated with the project and offer timely feedback.

Some programs allow the use of the Supplemental Documents section for additional evaluation information. Remember that reviewers are not required to read these supplemental documents, so ensure the important information is still in the 15-page proposal.

You want to be brief and outcome-focused for the Results of Prior NSF Support section. Use this space to describe what resulted from the prior work, not what you did. Be sure to be clear how that work informs the proposed work by suggesting, for example, that these outcomes set up the questions you are pursuing in this proposal.

Source: Christopher Lutz and Leslie Goodyear, https://www.evalu-ate.org/blog/goodyear-aug2016/
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.