Northeast SARE program evaluation, Sustainable Community Grants

View the project final report

Commodities

Practices

• Sustainable Communities: sustainability measures

This award was made in response to the following call for program evaluation: Overview The Northeast Sustainable Agriculture Research and Education (SARE) program seeks proposals for the evaluation of its Sustainable Community Grant program. Sustainable Community Grants address economic and social issues in the farm community and support appropriate growth, improved farm profits, a better quality of life, a cleaner environment, or improved farm stewardship. We seek evaluation of the impact of these grants, including whether and how these grants advance the Northeast SARE mission. We expect 30 projects to be completed in time for this evaluation and we seek quantitative analysis and qualitative synthesis about grant program effectiveness. Background SARE is a national USDA grant program with four regions; the Northeast region serves Connecticut, Delaware, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, Vermont, West Virginia, and Washington, D.C. The SARE mission is to advance agriculture that is profitable, good for the environment, and has a positive effect on the community. The primary tool for achieving the mission is a competitive grants effort. The Sustainable Community grant was funded in 2006 with each grant capped at $10,000. Since 2006, the criteria for awards has been refined somewhat, with an increasing emphasis on specific aspects of community and economic development. All projects are expected to break new ground and offer fresh approaches to enhancing sustainable commercial agriculture and community development. Scope of the inquiry There are about 30 completed Sustainable Community projects to be reviewed. The evaluator will survey each project manager and at least one project collaborator to see what specific accomplishments can be assigned to the grant award, to what extent the project achieved its written objectives, and to what extent subsequent benefits resulted from the seed money. The initial survey pool will be about 75 individuals. Gathering survey data electronically is an acceptable and efficient strategy. Follow-up by e-mail, telephone, and U.S. mail will likely be necessary. Minimally, 25 projects of the 30 should be represented in the response pool. Northeast SARE with contact grantees subject to this evaluation to let them know the evaluation is pending, assure them of anonymity, and enhance their receptiveness to the survey. Consistency across regions In 2005, the Western SARE region conducted an evaluation of its Farmer Grant program, which awards funds to commercial farmers to explore promising new production and marketing efforts that are likely to improve profits, advance good stewardship, and strengthen the community. This survey, which was done via the Internet, is a potential model for developing an instrument specific to this type of inquiry. The evaluator is free to use materials from that survey, which are available on request by sending e-mail to nesare@uvm.edu. That said, the content of the Sustainable Community projects overlaps the content of the Farmer Grant projects in a rather limited way. One of the key tasks for the evaluator will be to revise the previous Western SARE survey materials so that the inquiry will be specific to the goals of the Sustainable Community Grant program. Northeast SARE has issued a separate call for evaluation of its Farmer and Partnership grants that specifically requires the use of survey materials consistent with this previous evaluation. In this call for evaluation, Northeast SARE asks only that the selected evaluator use these prior materials as part of a context, and to look for ways to yield data that may be generally useful to this other, parallel, evaluation. Also, in October of 2007, the Southern SARE region evaluated a community grant program very similar to this one but using different measurement techniques that relied on face-to-face interviews with a smaller pool of project participants. That effort yielded some data but also had a broad narrative focus. The evaluation report from that effort is available (send a request to nesare@uvm.edu), but Northeast SARE explicitly seeks a more survey-based and data-driven evaluation, with techniques modeled chiefly on the Western SARE Farmer Grant effort. The primary questions to be investigated Northeast SARE wants to know how and whether this grant program contributes to and is consistent with the Northeast SARE outcome statement, which says that “Agriculture in the Northeast will be diversified and profitable, providing healthful products to its customers; it will be conducted by farmers who manage resources wisely, who are satisfied with their lifestyles, and have a positive influence on their communities and the environment.” More specifically, Northeast SARE wants to determine: • Whether Sustainable Community Grant projects had a positive influence on the economic position of commercial farmers; • Whether Sustainable Community Grant projects had a positive influence on the economic position of the community; • Whether Sustainable Community Grant projects had a positive influence on quality of life, a cleaner environment, or improved farm stewardship; • Whether grantees and project cooperators agree about the value of project results; • Whether the constituencies being surveyed are satisfied with SARE staff, program policies, and program delivery from initial proposal to final reporting, and whether SARE grantees have any specific recommendations for improved SARE program management; • Whether the SARE grants had any surprising or unintended results, either good or bad, that affected the sustainability of farms and farming in the Northeast; • Whether and how many projects resulted in ongoing activity or impacts after the grant period ended, including the leveraging of other funds to continue the project; and • Whether this grant program has been effective in addressing Northeast SARE’s outcome statement, based on the cumulative impact of these individual projects. Access The evaluator will have access to Northeast SARE internal lists of contact information for the grantees and their cooperators, as well as annual and final reports for the projects being evaluated. Northeast SARE will make a good-faith effort to ensure that contact information and reports are as current as possible, with the understanding that some of this information has degraded over time. An online database (www.sare.org/projects/index.htm) is also a resource for understanding project contents and results. These online reports are text only—no graphs, charts, or photos. Still, this database is an inventory of awards and can give the outlines of each project and its key reported results. The evaluator will also have access to prior calls for proposals, instructional and outreach materials, and documents specific to this grant program, can assume discussion and coordination with Northeast SARE staff and leadership. Work products Products in draft: A draft survey instrument should be submitted to Northeast SARE to make sure the survey content is directly linked to the primary questions to be investigated and offers appropriate consistency with prior survey work done for SARE in the West. A draft of the evaluator’s report is due 60 days before the close of the contracted performance period. This draft report will allow Northeast SARE to provide feedback and to make sure the report is addressing the primary questions to be investigated and are appropriately consistent with the prior survey work referenced above. Products in final form: A final report from the evaluator should present survey results and use those results to assess program delivery, impacts, strengths, and flaws. We suggest strongly that both the draft and final reports use a structure that follows the sequence outlined under “primary questions to be investigated,” above, and that each draft and final report also include summary findings and recommendations to Northeast SARE based on the evaluator’s interpretation of the data. These summary recommendations should specifically address the effectiveness of the grant program and give an overview of program impacts based on survey findings. On completion of the evaluation, Northeast SARE will need 30 hard copies and one soft copy of the final two program evaluation reports. Timeline Northeast SARE anticipates awarding a contract for evaluation in the spring of 2009. We further anticipate that the evaluator will review project reports and develop the draft survey by June 1, 2009. Survey work should proceed over the summer of 2009, with a draft report ready on or before October 31, and a final report submitted to Northeast SARE on or before December 31, 2009. Selection Criteria The following will be the criteria used to select an evaluator: • Documented previous experience with similar program evaluation • Clear evidence of familiarity with sustainable agriculture • A specific and effective plan to address the scope of work described above • Appropriate cost of the evaluation based on time needed to complete each step in the plan • Feedback from references you provide for whom you have performed similar work Specifics on the proposed budget, plan of work, and reporting products may be negotiated between the selected evaluator and Northeast SARE prior to entering into a contract. If a project agreement cannot be reached between the finalist and Northeast SARE, an alternate evaluator will be selected. Proposal format Respond on separate sheets of 8 ½” x 11” plain paper. Use standard fonts such as Times Roman or Arial, no smaller than 11 points. 1. Describe the professional context of your evaluation work (company or organization name and mission, support staff, typical clients, and years of experience). 2. Describe your previous experience with program evaluation, including the types of programs, the depth of evaluation required, and techniques deployed. In particular, describe any previous evaluation or similar work you have done within the farm community, with Cooperative Extension, farmers, and agricultural consultants. 3. Describe your background and familiarity with the key issues and themes in sustainable agriculture. 4. Provide a detailed description of your approach to addressing the scope of work as described above. 5. Estimate the cost of the evaluation and time needed to complete the survey, evaluation, and reports. 6. Provide the names and contact information of three references for whom you have completed similar evaluations. How to submit your proposal Send your proposal via e-mail to Northeast SARE, at nesare@uvm.edu. Include a cover letter and any relevant appendices. Proposals should be received no later than December 31, 2008. Questions about proposal format and content should be directed to the Northeast SARE office at 802/656-0471 or via e-mail to nesare@uvm.edu.

There are no specific performance targets for this evaluation. The final report is available as a download from the final report.