Progress report for NEVT17-001
Enhancing Evaluation Capacity to Improve Sustainable Agriculture Programs and Outcomes in Vermont
Project Type: PDP State Program
Funds awarded in 2017: $46,111.00
Projected End Date: 10/31/2020
Grant Recipient:
University of Vermont
Region: Northeast
State: Vermont
State Coordinator:
Project Information
Performance Target:
20 Extension educators and non-profit personnel will use increased outcome evaluation knowledge and skills to improve programs designed to help 150 beginning farmers launch and grow farm enterprises that meet their business, stewardship and lifestyle goals, and help 150 established producers adopt nutrient management, cover crop, and other production practices that support Vermont’s new water quality goals.
Introduction:
The goal of this plan is increase the ability of agricultural service providers in Vermont to apply evaluation concepts and techniques to improve their sustainable agriculture programs in ways that increase impact at the farm level.
The project will focus on evaluation in the context of two transdisciplinary areas of high relevance to NESARE’s outcome statement: programs that a) help new farmers establish and grow sustainable farm businesses and b) those that help farmers implement practices that reduce agricultural nonpoint water pollution. As such, it will engage a diverse group of agricultural service providers from Extension, nonprofit and government agencies who work across a variety of crop and livestock production, marketing and business development content areas. Participants will gain both a conceptual foundation and practical skills that they will use to strengthen their nutrient management, cover cropping, grazing management, enterprise planning, business planning, marketing, financial management, apprenticeship, and farmer mentoring programs.
Through the application of what they learn, participants will: gather more meaningful needs assessment data; develop more powerful learning objectives for their programs; keep better track of participant learning and behavior change; increase their understanding of factors that support or inhibit farm level adoption; use data generated through internal reviews and client feedback to improve service planning and delivery; and more effectively communicate the value of their sustainable agriculture programs to agricultural audiences, funders, and the general public.
In Vermont, the majority our agricultural educators in Extension—from our agronomy and pasture teams that are helping farmers improve soil health through cover cropping, reduced tillage, management intensive grazing, etc. to our farm business viability team that assists farm owners with financial management and business and transition planning—currently provide education and technical assistance that contributes to the goals of the SARE outcome statement.
While the majority of this programming is well received and well attended by the farming community, a review of needs assessments of Extension educators conducted annually at the Extension Professional Improvement Conference revealed that program evaluation and reporting has been a top professional development need in Vermont for at least the past four years. Similarly, NGO and agency partners have indicated interest in professional development on evaluation, and have expressed challenges accurately assessing impacts of their programs.
Extension educators and our NGO partners are committed to evaluating their farmer programming but many struggle to develop effective and efficient evaluation that both meets funders requirements and that help us identify the approaches and components of our programs that are most effective at supporting on-farm change. UVM Extension has not had an evaluation faculty specialist or staff coordinator since 2005, leaving evaluation planning and implementation as the responsibility of the faculty and staff delivering programs. While some larger states have Extension evaluation and curriculum development specialists with whom educators and service providers can consult when developing an evaluation plan, developing survey questions, or analyzing data, Vermont agricultural service providers are generally undertaking these activities on their own, sometimes looking to each other for feedback, guidance and suggestions.
Therefore, this project seeks to train a cadre of Extension educators and non-profit organization staff to apply evaluation concepts and approaches in their work with farmers to help us all identify and amplify components that are effective at supporting tangible changes on the farm.
We will draw on Northeast SARE’s outcomes funding framework (Williams et al., 1996) and the logic model approach used in other USDA programs. Penna and Philips (2005) suggest that these outcome evaluation frameworks allow educators to move beyond project outputs to measure meaningful outcomes. The project will also draw upon “Theory of Change,”a methodology used for planning, participation, and evaluation to promote social change (Taplin and Rasic, 2012).
Advisors/Cooperators
Educational Approach
Educational approach:
Through an online application, the project will administer a competitive process to recruit 14 Extension educators and 6 non-profit and government agency staff to participate in the three-year project. We will select participants from this pool based on supervisor support, commitment to participate, and the relevance of their area of expertise/programming to water quality and or beginning farmer topics.
The project will employ a variety of educational delivery methods, including workshops, webinars, and individual technical assistance and mentoring. The project will supplement in-person and distance trainings with small (2-person) team peer-review exercises and web conference check-ins. The check-ins will include time for progress updates among participants as well as supplemental content delivery by outside professionals as needed (identified by participants). To coalesce a co-learning community of participants, we will create an email listserv to support ongoing dialogue and resource exchange. Participants will select one of their sustainable agriculture education projects to use for their evaluation work over the course of the PD project.
The project will provide modest incentives for participation, including evaluation publications, mileage reimbursement to trainings and follow-up activities, and access to online evaluation and reporting tools. Through the creation of a project lending library and purchase of reprints, participants will have access to both scholarly and practical articles, books and guides on evaluation approaches and techniques.
All group learning sessions (whether in-person or online) will include action planning sessions, where service provider participants discuss ideas implementing evaluation activities in their programs, with at least one action step for doing so. Action plans may include steps such as planning (and conducting) needs assessment interviews; identifying key indicators; developing survey instruments; analyzing data from document reviews, interviews, focus groups and/or surveys; and creating reader-friendly reports and data visualizations.
Milestones for year 1
What beneficiaries do and learn:
1. 12 Extension, nonprofit and public agency key informants participate in interviews and focus groups to identify and prioritize their learning needs and priorities related to evaluation of beginning farmer development and water quality programming practices.
Proposed number of agriculture service provider beneficiaries who will participate:
12
Actual number of agriculture service provider beneficiaries who participated:
11
Actual number of farmer beneficiaries who participated:
1
Proposed Completion Date:
January 31, 2018
Status:
Completed
Date Completed:
May 15, 2018
Accomplishments:
Eleven agricultural service providers and one farmer participated in interviews about their learning needs related to beginning farmer development and water quality programs. The individuals represented Extension, nonprofit organizations, state government agencies and private consultants. Half of the individuals were people whose responsibilities include program leadership and coordination of multi-organizational projects, so they were able to provide insights into needs across the organizations they represent. The information they shared confirmed that building evaluation capacity and expertise is a need across organizations working on water quality and new farmer development. The needs assessment also gave us new appreciation for the amount customization and that will be needed to meet the variety of needs, and to support implementation within their teams and projects. For that reason, we revised our target for the number of participants from 24 to 12-15.
What beneficiaries do and learn:
2. 120 agricultural service providers receive an announcement describing the new SARE evaluation education project; the announcement includes an invitation to enroll as a participant and complete an anonymous baseline assessment of their knowledge and skills.
Proposed number of agriculture service provider beneficiaries who will participate:
120
Actual number of agriculture service provider beneficiaries who participated:
80
Proposed Completion Date:
July 31, 2018
Status:
Completed
Date Completed:
November 30, 2018
Accomplishments:
We conducted outreach to 80 agricultural educators and service providers across Vermont. Outreach was conducted by direct, individual email to individuals who work for UVM Extension, the Vermont Agency of Agriculture, Vermont conservation districts, nonprofit organizations and USDA FSA and NRCS. Within those groups, we targeted our outreach to people whose work has a significant focus either on helping new producers grow sustainable farm businesses or helping farmers implement practice that reduce agricultural non-point pollution. Emails pointed interested people to the new VT-SARE website, which provided an overview of the project. We also offered an interactive introductory webinar to potential participants a taste of the kinds of activities they would engage in and opportunities to ask questions. A recording of the webinar was posted on the project website so people who could not attend “live” could view it. 18 people attended or viewed the webinar.
What beneficiaries do and learn:
3. 24 agricultural service providers complete the self-assessment and online application, and commit to participating in the online, in-person and peer-to-peer components of the project. (end of August 2018). The project team uses the information from registration forms to match participants with a “peer learning partner” from the group and notifies all participants of the match.
Proposed number of agriculture service provider beneficiaries who will participate:
24
Actual number of agriculture service provider beneficiaries who participated:
15
Proposed Completion Date:
August 31, 2018
Status:
In Progress
Accomplishments:
Seventeen individuals completed an online application and were accepted into the program. Subsequently, 3 individuals withdrew. Two of the individuals withdrew because of changes in their job. The third withdrew because the program was not a good match for that individual’s priorities at this point in their career.
The individuals current in the program are employed by Extension programs (6), nonprofit organizations (4), local or state government agencies (3) and a land grant university. They work across a variety of programs involved in nutrient management, cover cropping, grazing management, speciality crop, enterprise planning, business planning and apprenticeship and farmer mentoring programs. As expected, the cohort is diverse in terms of their prior knowledge and experience with evaluation topics, and in the specific kinds of information and skill building they hope to gain through the project. Participants indicated they anticipate using what they learn to improve current programming (13 people) to plan new programs (12 people) to build their network (12 people) and to make decisions about whether to continue programs (9 people). We decided to wait until the cohort has more interaction with each other to assign peer learning partners.
What beneficiaries do and learn:
4. 14 service providers complete an online form verifying that they have completed a series of pre-workshop assignments. These assignments reviewing informational materials about theory of change, developing learning goals, and methods for assessing program performance. They also share the indicators their programs and/or projects are currently using to measure progress toward outcome goals. (September 2018)
Proposed number of agriculture service provider beneficiaries who will participate:
24
Actual number of agriculture service provider beneficiaries who participated:
14
Proposed Completion Date:
September 30, 2018
Status:
In Progress
Accomplishments:
All participants completed an in-depth self assessment about their current evaluation practices and providing us with information about the approaches/frameworks and measures they are using in the current programming.
Participants have also completed work sheets outlining their own program’s theory of change and assumptions, and 12 of the 14 have completed and shared with the project coordinator their personal action and learning plan through the summer of 2019.
What beneficiaries do and learn:
5. 22 agricultural service providers attend a full-day workshop to increase their knowledge and skills in the targeted evaluation topic areas. The focus of this session will be on framing evaluation questions and outcome indicators and approaches for collecting reliable, valid data. During the session, service providers will engage in individual reflections and small group discussions to generate ideas for implementing what they learn in their work with farmers, and planning at least one action step to improve programming related to new farmer development or water quality protection. They also obtain an orientation to the peer learning and expert consulting components of the project. (October 2018)
Proposed number of agriculture service provider beneficiaries who will participate:
20
Actual number of agriculture service provider beneficiaries who participated:
14
Proposed Completion Date:
October 31, 2018
Status:
Completed
Date Completed:
April 12, 2019
Accomplishments:
We held two sessions of the in-person workshop in April 2019. The same content was covered in both sessions. Two sessions were needed because schedules did not align for a single session.
What beneficiaries do and learn:
6. 12 agricultural service providers use an online application form to request individual technical assistance from one of the consulting expert evaluators. (By the end of October 2018)
Proposed number of agriculture service provider beneficiaries who will participate:
20
Actual number of agriculture service provider beneficiaries who participated:
12
Proposed Completion Date:
October 31, 2019
Status:
In Progress
Accomplishments:
Requests for technical assistance were incorporated into the action planning document. Two individuals will be completing theirs in May. The project coordinator is currently working to identify the consulting evaluators to provide specific technical assistance/information individuals have requested.
Because 6-8 individuals are looking for additional training in the methods of Results Based Accountability, we are exploring offering a multi-session training for this group this spring/summer.
Milestones for year 2
What beneficiaries do and learn:
1. 14 agricultural service providers receive monthly email updates about the project and invitations to participate in bi-monthly distance meetings (via Zoom). (Starting December 2018 and continuing through the second and third year of the project.) Each distance meeting will combine a presentation by an expert on a technical topic with opportunities for participants ask more general questions, and share challenges, successes and progress towards action steps. The monthly emails will provide links to related resources. While we will fine-tune the topics based on the priorities of the cohort, anticipated year 2 distance session topics include:
• writing strong survey questions and reporting survey results;
• technology to automate, streamline and strengthen data collection in farmer education settings;
• Beyond the survey: observation, documents, interviews, and other data collection methods
• Uses and benefits of qualitative data.
Proposed number of agriculture service provider beneficiaries who will participate:
14
Actual number of agriculture service provider beneficiaries who participated:
14
Proposed Completion Date:
September 30, 2019
Status:
In Progress
Accomplishments:
Fourteen agricultural service providers have are receiving regular communications from the project director.
We have held 3 distance meetings (via Zoom, and recorded), and two in-person workshops. The topics covered during the three sessions were:
- Orientation to the Evaluation Works program
- Learning Organizations and Theory of Change
- Change Management
Supplementary resources and follow-up activities were provided for the second and third sessions.
Two workshops (covering the same content) were necessary because it was not possible to bring all the participants together for a single day session. Participants at both sessions indicated strong satisfaction with the opportunity to learn in a highly interactive, small group session.
What beneficiaries do and learn:
2. 2. 14 agricultural service providers launch a peer learning circles process. Learning circles groups adopt a meeting and recording template (provided by the team) to outline the format, ground rules and expectations for their collaborative work and to document progress, accomplishments and challenges they encounter. (April 2018).
Proposed number of agriculture service provider beneficiaries who will participate:
14
Proposed number of farmer beneficiaries who will participate:
14
Actual number of agriculture service provider beneficiaries who participated:
10
Proposed Completion Date:
May 2, 2019
Status:
In Progress
Accomplishments:
Participants have been assigned to five learning circles. The project director decided the membership of the learning circles based on the interest areas of the individuals (outlined in their self assessment). All of the learning circles are using the same meeting and recording template to outline the format, ground rules and expectations for their collaborative work. The template also serves as a way to document progress, accomplishments and challenges they they encounter, and to communicate with the project director about their progress and needs.
What beneficiaries do and learn:
3. 20 agricultural service providers engage in ongoing discussion and program development with at least one peer (matches made by the program coordinator) and a consulting evaluator as they integrate new knowledge and skills into their programming. (Starting October 2018 and continuing through September 2020.) The consulting evaluator documents the questions/challenges participants are addressing and shares that information with the project coordinator.
Proposed number of agriculture service provider beneficiaries who will participate:
14
Actual number of agriculture service provider beneficiaries who participated:
14
Proposed Completion Date:
September 30, 2020
Status:
In Progress
Accomplishments:
Four of the five learning circles have had at least their first meeting and the fifth learning circle will formally launch on May 2.
Two learning circles each have two members, two have three members and one has four.
What beneficiaries do and learn:
4. 20 agricultural service providers complete a follow-up self-assessment survey from the Project Director and report on actions taken thus far to use evaluation skills and knowledge from this project in their water quality and/or beginning farmer programs, and give input about future training needs. (by May 2019)
Proposed number of agriculture service provider beneficiaries who will participate:
20
Proposed Completion Date:
May 31, 2019
Status:
In Progress
What beneficiaries do and learn:
5. Through the survey, 6 agricultural service providers share tools and templates that they developed/used in their work, which the project team shares on the Vermont SARE website. (July 2019)
Proposed number of agriculture service provider beneficiaries who will participate:
6
Proposed Completion Date:
July 31, 2019
Status:
In Progress
What beneficiaries do and learn:
6. 20 agricultural service providers attend the project’s Year 2 Workshop, which builds on the skill base developed through the first year’s professional development activities. The focus of the year 2 workshop will shift from designing their evaluation plan and collecting data to analyzing their data and implementing changes in their programming. While we plan to fine tune content based on the priorities identified in the May 2019 survey, anticipated topics will include practices for assessing program performance (at the input, output, outcome and impact levels); using that performance data to modify programming to better respond to client needs and emerging conditions; and communicating results to stakeholders. As with the first workshop, the session will be highly interactive, with opportunities for both group discussion and individual reflection. By the end of the day, all participants will plan (and share with the project director) at least one new action step to improve programming related to new farmer development or water quality protection. (September 2019).
Proposed number of agriculture service provider beneficiaries who will participate:
20
Proposed Completion Date:
September 30, 2019
Status:
In Progress
Milestone Activities and Participation Summary
Educational activities and events conducted by the project team:
| Activity | Year 1 | Year 2 | Year 3 | Total |
|---|---|---|---|---|
| Curricula, factsheets or educational tools | 1 | 1 | ||
| Webinars, talks and presentations | 1 | 1 | ||
| 1- Needs assessment - conducted interviews with 12 individuals 2- New Vt-SARE website with section focusing on the Evaluation Works project 3- Application form and self assessment instrument to collect baseline information about project participants | 3 | 3 |
Beneficiaries who particpated in the project’s educational activities and events:
| Audience | Year 1 | Year 2 | Year 3 | Total Individuals |
|---|---|---|---|---|
| Extension | 12 | 0 | 0 | 12 |
| Researchers | 3 | 0 | 0 | 3 |
| Nonprofit | 7 | 0 | 0 | 7 |
| Agency | 4 | 0 | 0 | 4 |
| Service providers (other or unspecified) | 26 | 0 | 0 | 26 |
| Farmers / ranchers | 1 | 0 | 0 | 1 |
| Others | 2 | 0 | 0 | 2 |
Participation Summary
29 Ag service providers who participated in the project
Learning Outcomes
15 Ag service providers intend to use knowledge, attititudes, skills and/or awareness learned through this project in their educational activities and services for farmers
Key areas in which the service providers (and farmers if indicated above) reported a change in knowledge, attitudes, skills and/or awareness::
We have not gotten to that point in the project yet.
Performance Target Outcomes
Performance Target Outcomes - Service Providers
Target #1
Target: Number of service providers who will take action to educate/advise farmers:
20
Target: The educational action(s) they will take:
20 Extension educators and non-profit personnel will use increased outcome evaluation knowledge and skills to improve programs designed to help 150 beginning farmers launch and grow farm enterprises that meet their business, stewardship and lifestyle goals, and help 150 established producers adopt nutrient management, cover crop, and other production practices that support Vermont’s new water quality goals.
Target: The number of farmers who will be educated/advised by the service providers:
300
Target: Total size/scale of the farms these farmers manage (e.g. total acres or animal units managed, gross sales or production volume, etc.):
SARE Outreach
Outreach about SARE:
In 2018, the Vermont SARE Program conducted outreach about SARE grant opportunities and results of SARE research in the following ways:
- Bringing the Northeast SARE exhibit and materials to five major Vermont agricultural conferences:
- The Vermont Farm Show (3 days, January 2018)
- Northeast Organic Farming Association of Vt Winter Conference (2 days, February 2018);
- Vermont No-Till and Cover Crop Conference (1 day, February 2018)
- Vermont Hop Conference (1 day, February 2018)
- Vermont Grain Conference (1 day, March 2018)
- Coordinating and delivering a presentation at a 75-minute information session on SARE’s graduate student grant program (April 2019).
- Building a new Vt-SARE website (http://blog.uvm.edu/vtsare/)
- Continue to build social media presence of SARE projects on Twitter, Facebook, and partner blogs.
Recieved information about SARE grant programs and information resouces:
| Audience | Year 1 | Year 2 | Year 3 | Total |
|---|---|---|---|---|
| Service providers | 250 | 0 | 0 | 250 |
| Farmers | 1800 | 0 | 0 | 1800 |
Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the U.S. Department of Agriculture or SARE.
