Final report for SNE20-013-VT
Project Information
This project, a continuation of work started in 2018, built evaluation capacity among a cadre of 14 Extension educators and non-profit organization and public agency staff. It also conducted a needs assessment to identify topics where development of knowledge and skill among agricultural service providers in Vermont is needed to address ongoing sustainable agriculture issues.
Evaluation Capacity: While the majority of Vermont's sustainable agriculture programming is well-received and well-attended by the farming community, a 2017 review of professional development priorities among Extension, NGO and agency personnel identified evaluation as a priority topic. Many service providers indicated that they struggle to develop effective and efficient evaluation processes and tools that both meet funders' reporting requirements and that can help identify approaches and components of programs that are most effective at supporting on-farm changes. UVM Extension has been without an evaluation specialist since 2005, leaving evaluation planning and curriculum development the responsibility of the faculty and staff delivering the programs.
Through this final year of the project, a cohort of 14 educators and service providers continued to build evaluation skills and apply them identify and amplify components of their farming programming that are effective at supporting tangible changes on the farm. Through distance education approaches, participants continued to expand their evaluation knowledge base, and received support to use and adapt techniques, resources and tools that had practical applications in their sustainable agriculture programs. Throughout the year, project activities were modified support participants’ needs to adjust and adapt to their own program changes resulting from the coronavirus pandemic. At the project's close, eight members of the cohort reported using what they learned to improve design and delivery of farmer programs related to water quality protection and farmer development programs reaching over 1290 farmers.
Needs Assessment: The needs assessment resulted in the identification of three, interrelated topic areas where development of knowledge and skills among service providers can have a significant impact on sustainable agriculture in Vermont. These topic areas have been incorporated into the next three-year professional development project. They are: building and updating information resources and referral networks for beginning farmers; assisting established farmers with farm business succession and transfer planning; and incorporating strategies to address social justice in their programming for farmers, farm workers and farm businesses.
1) 14 Extension educators and non-profit personnel will use increased outcome evaluation knowledge and skills to improve programs designed to help 120 beginning farmers launch and grow farm enterprises that meet their business, stewardship and lifestyle goals, and help 120 established producers adopt nutrient management, cover crop and other production practices that support Vermont's new water quality goals.
2) Additionally, the project director will conduct a stakeholder needs assessment in preparation for the 2021-2024 Vermont professional development project.
Component 1: Building Evaluation Capacity
This project continues to focus on building evaluation knowledge and skills service providers can use to assess how the sustainable agriculture training and support they provide to farmers positively affects the three “Ps” of sustainable agriculture: profit, people and protection of natural resources. Effective evaluation will contribute value to sustainable agricultural programming beyond bean counting (outputs) and measuring outcomes. Used properly, evaluation becomes an iterative process that serves to improve programming while better achieving measurable outcomes. The most important function of evaluation, according to Cornelia Flora (2003) is learning, particularly for programs that are working for systems changes. "The learning and evaluation process allows contributors to examine and alter what they do and how they do it, which motivates system change.”
In Vermont, the majority our agricultural educators in Extension—from our agronomy and pasture teams that are helping farmers improve soil health through cover cropping, reduced tillage, management intensive grazing, to our farm business viability team that assists farm owners with financial management and business and transition planning—currently provide education and technical assistance that contributes to the goals of the SARE outcome statement. While the majority of this programming is well received and well attended by the farming community, a review of needs assessments of Extension educators conducted annually at the Extension Professional Improvement Conference revealed that program evaluation and reporting has been a top professional development need in Vermont for at least the past four years. Similarly, NGO and agency partners indicated interest in professional development on evaluation, and have expressed challenges accurately assessing impacts of their programs.
Extension educators and our NGO partners are committed to evaluating their farmer programming but many struggle to develop effective and efficient evaluation that both meets funders requirements and that help us identify the approaches and components of our programs that are most effective at supporting on-farm change. UVM Extension has not had an evaluation faculty specialist or staff coordinator since 2005, leaving evaluation planning and implementation as the responsibility of the faculty and staff delivering programs. While some states have Extension evaluation and curriculum development specialists with whom educators and service providers can consult when developing an evaluation plan, developing survey questions, or analyzing data, Vermont agricultural service providers are generally undertaking these activities on their own, sometimes looking to each other for feedback, guidance and suggestions.
Therefore, this project is training a cadre of Extension educators and non-profit organization staff to apply evaluation concepts and approaches in their work with farmers to help us all identify and amplify components that are effective at supporting tangible changes on the farm. Using a combination of in-person and distance education approaches, the project is providing information about a variety of evaluation approaches, techniques, resources and tools, allowing participants to use and adapt those that make sense for their programs. The project has also been working to develop a community of practice so through which participants can learn from and support one and other beyond the life of the project. In this final year, we will continue to offer resources, tools and individual support to help participants integrate these processes in their work.
Component 2: Needs Assessment
To ensure that Vermont's 2021-2024 project focuses on issues of high relevance to farmers and agricultural service providers, the project coordinator will conduct a needs assessment to identify topics where development of knowledge and skills among ag service providers is needed to address three key sustainable agriculture issues in Vermont: a) farm succession planning; b) beginning farmer programming; and, c) labor management. The project director will use a variety of needs assessment tools, including document review, surveys and interviews to identify both content and delivery approach for the next three-year project. The needs assessment will identify farmer perspectives on topics for farmer-oriented education/technical assistance that could help them achieve greater sustainability, service provider interest in the topic area, and the professional development within the agricultural service provider community needed to effectively deliver that education/services. The needs assessment will incorporate findings from recent strategic planning efforts conducted by UVM Extension and needs assessments conducted by NGOs in Vermont.
Advisors/Cooperators
- - Technical Advisor
- - Technical Advisor (Educator and Researcher)
- - Technical Advisor
- - Producer
- - Technical Advisor (Educator and Researcher)
Educational Approach
Recruitment
Evaluation Capacity Building: In this final year of the project, we continued engage participants via email correspondence and individual consultations via phone and video meetings to continue to support their individual and organizational implementation needs. The project worked with an established cohort that selected through an application process. The application process included a self-assessment of their evaluation-related learning priorities, and required them to make commitments to attend trainings and participate in peer/partner learning circles, and to report the impacts of their work back to the project director. Several, citing pandemic-related interruptions and changes to their work, requested additional time (and resources and consultations) to implement their learning in their programs and services.
Stakeholder Needs Assessment: The project invited individuals representing farmer, Extension, nonprofit and public agency perspectives to participate in the needs assessment. The project director offered individuals a variety of mechanisms -- surveys, interviews (phone and video meeting) and document sharing – through which to provide input, with the goal of accommodating schedules and communication preferences. Perspectives reflected variety of Vermont crop and livestock industries, and addressed environmental, economic and social dimensions of sustainability.
Education
Evaluation Capacity Building
Using primarily distance education approaches, the project continued to provide information and education about variety of evaluation approaches, techniques, resources and tools, allowing participants to use and adapt those that made sense in the context of their programs.
All group learning sessions (whether in-person or online) will include action planning sessions, where service provider participants discuss ideas implementing evaluation activities in their programs, with at least one action step for doing so. Action plans may include steps such as planning (and conducting) needs assessment interviews; identifying key indicators; developing survey instruments; analyzing data from document reviews, interviews, focus groups and/or surveys; and creating reader-friendly reports and data visualizations.
Needs Assessment
While the needs assessment does not involve a formal educational program, the project director will share results with advisors and facilitate discussions through which advisors will provide feedback and guidance.
Verification
Verification activities followed the plan outlined in the original project proposal with some minor adjustments to respond to participant preferences for providing feedback. We used online questionnaires and registration forms to assess baseline knowledge and skill levels. Follow-up surveys, interviews and action plans verified participants’ learning and helped to identify additional needs for learning and support. Participants created individual action plans that identified their intention to use what they learn to teach and advise farmers, and shared these plans with the project director. In response to participant preference, the project director used more informal, individual check-ins both to support participants and to learn about progress. The project director used a final evaluation survey and interviews at the end of the project verify the ways participants followed-through on their action plans to use what they learned to improve and assess the performance of their education programs for beginning farmer development and water quality improvement practices. These final evaluation activities also explored ways that participants needed to modify their plans in response to pandemic-related changes to their work and farmer needs.
Needs Assessment
The project director conducted the bulk of the needs assessment activities in the first and second quarters of the project year, to gain understanding of agricultural service provider needs and interests in the following general content area:
- farm transfer and transition planning;
- diversity, equity and inclusion as it relates to agricultural education;
- collaboration and networks to deliver beginning farmer services; and
- labor management education and technical assistance.
The selection of these topics was informed both by UVM Extension's strategic planning process, and by responses to farmer, farm employee and aspiring farmer interviews, questionnaires and surveys conducted between 2018 and 2021. Discussions with project advisors and potential program participants in the third quarter of the project year (April-June), identified three topic areas as being of higher priority and more directly relevant to emerging needs of a broad cross section of Vermont’s agricultural service providers and educators. These topic areas were: information resources and referrals for beginning farmers; assisting farmers the business lifecycle with farm business succession and transfer planning information; and incorporating strategies to address social justice in their programming for farmers, farm workers and farm businesses.
Milestones
Agricultural service providers engage in ongoing discussion and evaluation development with at least one peer (matches made by the program coordinator) and/or consulting evaluator as they integrate new knowledge and skills into their programming. (Starting May 2018 and continuing through August 2021.) Participants use peer learning template and action planning documents to communicate progress and needs to the project coordinator.
20
14
September 30, 2021
Completed
September 30, 2021
During the project year, 14 members of the core cohort group engaged in ongoing discussion and program development with at least one peer and/or the project director. Based on their needs for specific evaluation knowledge gain, the project supported six individuals to attend online workshops offered by nationally recognized evaluation specialists.
Agricultural service providers incorporate new evaluation approaches/strategies in their 2021 farmer education programs, and document changes through peer learning and action planning documents.
14
8
September 30, 2021
Completed
September 30, 2021
Eight individuals provided specific information about how they incorporated new evaluation strategies and approaches into their education and services.
Agricultural service providers use evaluation data from the last 1-3 years to modify (or plan to modify) some aspect of program delivery/content in 2021 or 2022.
8
8
September 30, 2021
Completed
September 30, 2021
Eight individuals provided detailed information about how they used (or planned to use) evaluation data to improve their water quality and/or beginning farmer programs. Results are discussed in the educational and performance target outcome sections below.
Agricultural service providers use new skills and knowledge they gain from workshops, meetings and consultations to improve data collection, data analysis, data presentation and evaluation communication with their stakeholders.
14
8
September 30, 2021
Completed
September 30, 2021
ASPs reported integrating skills and knowledge they gained over the four years of the project to make changes in data collection and analysis, presentation and reporting, and communications with stakeholders and funders.
Agricultural service providers complete a follow-up self assessment survey to report on actions taken thus far to incorporate evaluation skills and knowledge from this project in their water quality or beginning farmer programs.
14
8
September 30, 2021
Completed
September 30, 2021
Participants reviewed their work/accomplishments and shared reflections via a survey and individual interviews with the project coordinator.
Agricultural service providers share in-depth information about how their expanded evaluation capacity improved their programs and impacts which the project team uses to write case study examples. These participants are selected by the project coordinator, based on the information they provide in the July 2021 survey. The case studies are published on the Vermont SARE website.
6
3
September 30, 2021
Completed
September 30, 2021
Three individuals provided detailed information about their expanded evaluation capacity. Summaries of their work were added to the resource materials available to Evaluation Works participants. To round out stories showing different approaches to evaluating sustainable agriculture work, the project director also added links to resources aggregated by the Gaining Results through Evaluation USDA-Beginning Farmer and Rancher Development program project.
Needs Assessment - Agricultural educators and service providers from Extension, nonprofit organizations and public agencies provide input into sustainable agriculture professional development needs and priorities for the next 3 years through individual interviews, and discussion sessions.
12
2
12
July 31, 2021
Completed
July 30, 2021
The project coordinator engaged in interviews and group discussions with service providers engaged in a variety of programs and service to identify high priority topics for professional development over the next 3+ years. The project coordinator also considered:
- data and analysis that was done as part of the UVM Extension's recent strategic planning process;
- responses farmers, farm employees and aspiring farmers provided between 2018 and 2021 (in interviews, focus groups and surveys) to questions about the kinds of support and education they anticipate needing in the future;
- data and analysis shared through the Vermont Farm to Plate initiative.
Four umbrella topics emerged as ones that had broad-based interest and need:
- farm transfer and transition planning
- diversity equity and inclusion as it relates to agricultural education
- collaboration, resources and networks to deliver services for beginning farmers
- labor management education and technical assistance
The project director then had subsequent discussions with a subset of the original stakeholder group and project advisors to further prioritize and narrow the focus of the proposed 2021-2024 program. Among the gaps that were identified as priorities to address:
- UVM Extension has a large collection of beginning farmer resources. However, many of these materials are now 10+ years old. While many of the concepts remain relevant to supporting aspiring, new and beginning farmers, these need to be updated to reflect changes in laws and regulations, new production knowledge and approaches, changes in market conditions and available support services. Service providers from Extension, nonprofit organizations, and state and federal agencies all indicated there is a need both among service providers and the aspiring, new and beginning farmers they work with, for updated and expanded materials.
- Extension educators and service providers who specialize in production, marketing and food safety topics, reported that farm transfer and business succession issues often emerge as an issue during consultations or farm visits focusing on their area of expertise. These service providers articulated an interest in professional development and supporting resources that will help them provide basic farm transfer information and appropriate referrals to more specialized service providers and consultants.
- Finally, service providers and educators expressed strong interest in diversity, equity and inclusion (DEI) learning opportunities that will help them provide more inclusive, accessible and effective services to more diverse farmer audiences.
These three topics have been integrated into the 2021-2024 Vermont State SARE work plan.
Milestone Activities and Participation Summary
2-Stakeholder needs assessment interviews
3-website development
Participation Summary:
Learning Outcomes
The project coordinator verified changes in knowledge, attitudes, skills and/or awareness in all 14 of the individuals who participated in the activities supported by this one-year concurrent extension of the Evaluation Works project (also funded by NEVT17-001). Verification tools included: review of participants' work product, interviews and surveys both during the previous project and at the end of this project. In the final evaluation survey, all eight respondents reported increased knowledge, skill and confidence related to using evaluation to improve programming. All eight also reported that their involvement in the project changed the way they approach evaluation activities and evaluation's role in program delivery.
The survey instrument asked respondents to rate changes in their knowledge, confidence and skills on a five point scale from "No change" to "Significant increase."
- With regard to confidence, all eight reported increases with three reporting "moderate" increases and five reporting "significant" increases.
- With regard to knowledge gain, seven people reported moderate (5) or (2) significant increases
- with regard to skills, five people reported moderate (4) or significant (1) improvements in their evaluation skills.
Performance Target Outcomes
Performance Target Outcomes - Service Providers
Target #1
Extension educators, nonprofit and agency personnel will use increased evaluation knowledge and skills to improve programs designed to help a) beginning farmers launch and grow farm enterprises that meet their business, stewardship and lifestyle goals and b) established producers adopt nutrient management, cover crop and other production practices that support Vermont's water quality goals.
N/A
| Year 1 | Year 2 | Year 3 |
|---|---|---|
| 8 |
Participants in Evaluation Works improved evaluation practices and used evaluation results to improve design and delivery of farmer education programs related to water quality protection and farm business development. All eight made changes to their evaluation planning, assessment of outcomes for program participants, data collection and analysis practices, and the way they communicate with funders and stakeholders about program activities and accomplishment.
Of the eight people who completed the final evaluation survey, three reported improvements to water quality programs, one reported changes to a farm business development program, and four reported making changes to both farmer development and water quality education programs.
| Year 1 | Year 2 | Year 3 |
|---|---|---|
| 1290 | 0 | 0 |
| Year 1 | Year 2 | Year 3 |
|---|---|---|
NA |
NA |
NA |
| Activity | Year 1 | Year 2 | Year 3 | Total |
|---|---|---|---|---|
| Stakeholder assessment and feedback data from farmers and aspiring farmers via registration, intake and feedback forms (8) farmer feedback and assessment via interviews and focus groups (2) modifications to grant application, review and administrative processes (4); online conference planning and decision-making (2); communication and program opportunities, activities and accomplishments via written reports (5) | 18 | 18 |
Eight participants completed a final evaluation survey. Following are changes they reported making as a result of participation in the project:
- Techniques to assess outcomes for participants (8)
- Techniques to assess participant satisfaction with programs/services (6)
- Planning new programs/services (5)
- Decisions about whether to continue programs/services
All survey respondents said their involvement in the project influenced the kinds of data they now collect. Additionally, they said it also influenced the performance measures they use (7) how much data they collect (6), and how they involve stakeholders in developing program goals, activities and evaluations (6).
All respondents said they have changed the wording they use on registration forms and/or evaluation surveys. Participants also reported improving the overall design of their forms/surveys (4), acquiring new and better data analysis practices (4) and using new methods of data collection such as in-session polling (3) and use of observation, focus group/interview questions.
All respondents said that as a result of participating in the project they changed the way they are communicating with funders and stakeholders about program activities. Specific areas of change/improvement included the content they include in written reports (8), data/evidence included in grant proposals (8); information shared with partners and collaborators (6) and the quality of information provided in their reporting (6).
Combined, these ASPs estimated that over 1290 farmers participated in the programs in which they applied learning from the Evaluation Works project. The minimum number of farmer program participants was 20 and the maximum was 500.
Performance Target Outcomes - Farmers
NA
Additional Project Outcomes
As a continuation of NEVT17-001 this phase of the Evaluation Works project was affected by the COVID-19 pandemic and particularly the number of resignations and position changes within several of the key organizations where participants are/were employed. Six of the original cohort of 22 participants withdrew from Evaluation works prior to the project's end due to changes in their employment. Additionally, covid-related changes in work plans, staffing shortages and pressures to address farmer needs and project deliverables, the supervisor of two other participants restricted the from participating in the project's final year.
Seven of the eight survey respondents said that the pandemic's impact on delivery of farmer education, as well as the pressures on their farmer clients, affected their ability to incorporate Evaluation Works learning in their programs, with six of them rating the impact as "moderate" or "significant." However, six of them also reported that Evaluation works helped them to adapt and respond to the pandemic. (See the NEVT-017 for additional information and anecdotes from participants.)
Please see NEVT17-001, which provides some stories from participants illustrating the benefits they gained from participating over multiple years in the Evaluation Works project.
Please see NEVT17-001, which provides participants insights into the value of the multi-year approach.
SARE Outreach
Vermont SARE outreach efforts continued to help Vermont's farmers, agricultural service providers, policy makers and others better understand Northeast SARE, its grants programs and project results. However, due to pandemic-related restrictions on in-person gatherings, most of it was conducted virtually during the project period through direct emails, e-newsletters, social media posts, conducted through presentations. Based on website analytics, e-newsletter opens, and numbers of individual emails and phone calls, outreach reached about 2500 people but in a different way than when when tabling at events when we can respond to individuals' questions and interests, and put SARE bulletins and books, and grant program brochures in their hands.
The following were planned but not possible because of pandemic restrictions.
1. Presentations:
- Info session on Graduate Student Grants (March 2021 - total of 10 students)
- Info sessions on Farmer Grants with VTC students (fall 2020, spring 2021 - total of 20 students)
2. Exhibits: We will collaborate with partners to bring Northeast SARE’s exhibit and materials to at least four of these annual events in the winter of 2021, provided that in-person conferences are occurring. Selection of events will balance maintaining visibility/support at key sustainable agriculture events with reaching out to new audiences. Aggregated, these exhibits will be at events attended by at least 500 Vermonters, including approximately 425 farmers and 75 agricultural service providers and educators.)
- Vermont Vegetable and Berry Growers Association Annual Meeting (January)
- Vermont Pasture Conference (January)
- Vermont Association of Landscape
- Northeast Organic Farming Association of Vt Winter Conference (February);
- Vermont No-Till and Cover Crop Conference (February)
- Vermont Industrial Hemp Conference (February)
- Vermont Hop Conference (February)
- Vermont Grain Conference (March)
3. Responding to inquiries about SARE and Northeast SARE Grant Programs, including referring applicants to relevant information, resources and SARE staff. Based on prior years, experience, the project coordinate will respond to approximately 20 individual, specific inquiries about SARE and its grant programs during the year.
4. Distributing announcements about Northeast SARE grant programs events and information resources via e-newsletters, listservs, and social media. In aggregate, these channels will reach an estimated 2000 individuals, including approximately 150 agricultural service providers and 850 farmers.
5. Participate in SARE professional development programs (Summer and winter meetings)
6. Serve as a reviewer for at least one of SARE's grant programs.
Recieved information about SARE grant programs and information resouces:
| Audience | Year 1 | Year 2 | Year 3 | Total |
|---|---|---|---|---|
| Service providers | 250 | 0 | 0 | 250 |
| Farmers | 2250 | 0 | 0 | 2250 |