Progress report for NEVT17-001
Project Information
20 Extension educators and non-profit personnel will use increased outcome evaluation knowledge and skills to improve programs designed to help 150 beginning farmers launch and grow farm enterprises that meet their business, stewardship and lifestyle goals, and help 150 established producers adopt nutrient management, cover crop, and other production practices that support Vermont’s new water quality goals.
The goal of this plan is increase the ability of agricultural service providers in Vermont to apply evaluation concepts and techniques to improve their sustainable agriculture programs in ways that increase impact at the farm level.
The project will focus on evaluation in the context of two transdisciplinary areas of high relevance to NESARE’s outcome statement: programs that a) help new farmers establish and grow sustainable farm businesses and b) those that help farmers implement practices that reduce agricultural nonpoint water pollution. As such, it will engage a diverse group of agricultural service providers from Extension, nonprofit and government agencies who work across a variety of crop and livestock production, marketing and business development content areas. Participants will gain both a conceptual foundation and practical skills that they will use to strengthen their nutrient management, cover cropping, grazing management, enterprise planning, business planning, marketing, financial management, apprenticeship, and farmer mentoring programs.
Through the application of what they learn, participants will: gather more meaningful needs assessment data; develop more powerful learning objectives for their programs; keep better track of participant learning and behavior change; increase their understanding of factors that support or inhibit farm level adoption; use data generated through internal reviews and client feedback to improve service planning and delivery; and more effectively communicate the value of their sustainable agriculture programs to agricultural audiences, funders, and the general public.
In Vermont, the majority our agricultural educators in Extension—from our agronomy and pasture teams that are helping farmers improve soil health through cover cropping, reduced tillage, management intensive grazing, etc. to our farm business viability team that assists farm owners with financial management and business and transition planning—currently provide education and technical assistance that contributes to the goals of the SARE outcome statement.
While the majority of this programming is well received and well attended by the farming community, a review of needs assessments of Extension educators conducted annually at the Extension Professional Improvement Conference revealed that program evaluation and reporting has been a top professional development need in Vermont for at least the past four years. Similarly, NGO and agency partners have indicated interest in professional development on evaluation, and have expressed challenges accurately assessing impacts of their programs.
Extension educators and our NGO partners are committed to evaluating their farmer programming but many struggle to develop effective and efficient evaluation that both meets funders requirements and that help us identify the approaches and components of our programs that are most effective at supporting on-farm change. UVM Extension has not had an evaluation faculty specialist or staff coordinator since 2005, leaving evaluation planning and implementation as the responsibility of the faculty and staff delivering programs. While some larger states have Extension evaluation and curriculum development specialists with whom educators and service providers can consult when developing an evaluation plan, developing survey questions, or analyzing data, Vermont agricultural service providers are generally undertaking these activities on their own, sometimes looking to each other for feedback, guidance and suggestions.
Therefore, this project seeks to train a cadre of Extension educators and non-profit organization staff to apply evaluation concepts and approaches in their work with farmers to help us all identify and amplify components that are effective at supporting tangible changes on the farm.
We will draw on Northeast SARE’s outcomes funding framework (Williams et al., 1996) and the logic model approach used in other USDA programs. Penna and Philips (2005) suggest that these outcome evaluation frameworks allow educators to move beyond project outputs to measure meaningful outcomes. The project will also draw upon “Theory of Change,”a methodology used for planning, participation, and evaluation to promote social change (Taplin and Rasic, 2012).
Advisors/Cooperators
Educational Approach
Through an online application, the project will administer a competitive process to recruit 14 Extension educators and 6 non-profit and government agency staff to participate in the three-year project. We will select participants from this pool based on supervisor support, commitment to participate, and the relevance of their area of expertise/programming to water quality and or beginning farmer topics.
The project will employ a variety of educational delivery methods, including workshops, webinars, and individual technical assistance and mentoring. The project will supplement in-person and distance trainings with small (2-person) team peer-review exercises and web conference check-ins. The check-ins will include time for progress updates among participants as well as supplemental content delivery by outside professionals as needed (identified by participants). To coalesce a co-learning community of participants, we will create an email listserv to support ongoing dialogue and resource exchange. Participants will select one of their sustainable agriculture education projects to use for their evaluation work over the course of the PD project.
The project will provide modest incentives for participation, including evaluation publications, mileage reimbursement to trainings and follow-up activities, and access to online evaluation and reporting tools. Through the creation of a project lending library and purchase of reprints, participants will have access to both scholarly and practical articles, books and guides on evaluation approaches and techniques.
All group learning sessions (whether in-person or online) will include action planning sessions, where service provider participants discuss ideas implementing evaluation activities in their programs, with at least one action step for doing so. Action plans may include steps such as planning (and conducting) needs assessment interviews; identifying key indicators; developing survey instruments; analyzing data from document reviews, interviews, focus groups and/or surveys; and creating reader-friendly reports and data visualizations.
Milestones
1. 12 Extension, nonprofit and public agency key informants participate in interviews and focus groups to identify and prioritize their learning needs and priorities related to evaluation of beginning farmer development and water quality programming practices.
Eleven agricultural service providers and one farmer participated in interviews about their learning needs related to beginning farmer development and water quality programs. The individuals represented Extension, nonprofit organizations, state government agencies and private consultants. Half of the individuals were people whose responsibilities include program leadership and coordination of multi-organizational projects, so they were able to provide insights into needs across the organizations they represent. The information they shared confirmed that building evaluation capacity and expertise is a need across organizations working on water quality and new farmer development. The needs assessment also gave us new appreciation for the amount customization and that will be needed to meet the variety of needs, and to support implementation within their teams and projects. For that reason, we revised our target for the number of participants from 24 to 12-15.
2. 120 agricultural service providers receive an announcement describing the new SARE evaluation education project; the announcement includes an invitation to enroll as a participant and complete an anonymous baseline assessment of their knowledge and skills.
We conducted outreach to 80 agricultural educators and service providers across Vermont. Outreach was conducted by direct, individual email to individuals who work for UVM Extension, the Vermont Agency of Agriculture, Vermont conservation districts, nonprofit organizations and USDA FSA and NRCS. Within those groups, we targeted our outreach to people whose work has a significant focus either on helping new producers grow sustainable farm businesses or helping farmers implement practice that reduce agricultural non-point pollution. Emails pointed interested people to the new VT-SARE website, which provided an overview of the project. We also offered an interactive introductory webinar to potential participants a taste of the kinds of activities they would engage in and opportunities to ask questions. A recording of the webinar was posted on the project website so people who could not attend "live" could view it. 18 people attended or viewed the webinar.
3. 24 agricultural service providers complete the self-assessment and online application, and commit to participating in the online, in-person and peer-to-peer components of the project. (end of August 2018). The project team uses the information from registration forms to match participants with a “peer learning partner” from the group and notifies all participants of the match.
Seventeen individuals completed an online application and were accepted into the program. Subsequently, 3 individuals withdrew. Two of the individuals withdrew because of changes in their job. The third withdrew because the program was not a good match for that individual's priorities at this point in their career.
The individuals current in the program are employed by Extension programs (6), nonprofit organizations (4), local or state government agencies (3) and a land grant university. They work across a variety of programs involved in nutrient management, cover cropping, grazing management, speciality crop, enterprise planning, business planning and apprenticeship and farmer mentoring programs. As expected, the cohort is diverse in terms of their prior knowledge and experience with evaluation topics, and in the specific kinds of information and skill building they hope to gain through the project. Participants indicated they anticipate using what they learn to improve current programming (13 people) to plan new programs (12 people) to build their network (12 people) and to make decisions about whether to continue programs (9 people). We decided to wait until the cohort has more interaction with each other to assign peer learning partners.
4. 14 service providers complete an online form verifying that they have completed a series of pre-workshop assignments. These assignments reviewing informational materials about theory of change, developing learning goals, and methods for assessing program performance. They also share the indicators their programs and/or projects are currently using to measure progress toward outcome goals. (September 2018)
All participants completed an in-depth self assessment about their current evaluation practices and providing us with information about the approaches/frameworks and measures they are using in the current programming.
Participants have also completed work sheets outlining their own program's theory of change and assumptions, and 12 of the 14 have completed and shared with the project coordinator their personal action and learning plan through the summer of 2019.
5. 22 agricultural service providers attend a full-day workshop to increase their knowledge and skills in the targeted evaluation topic areas. The focus of this session will be on framing evaluation questions and outcome indicators and approaches for collecting reliable, valid data. During the session, service providers will engage in individual reflections and small group discussions to generate ideas for implementing what they learn in their work with farmers, and planning at least one action step to improve programming related to new farmer development or water quality protection. They also obtain an orientation to the peer learning and expert consulting components of the project. (October 2018)
We held two sessions of the in-person workshop in April 2019. The same content was covered in both sessions. Two sessions were needed because schedules did not align for a single session.
6. 12 agricultural service providers use an online application form to request individual technical assistance from one of the consulting expert evaluators. (By the end of October 2018)
Requests for technical assistance were incorporated into the action planning document. Two individuals will be completing theirs in May 2019. The project coordinator is currently working to identify the consulting evaluators to provide specific technical assistance/information individuals have requested.
Because 6-8 individuals are looking for additional training in the methods of Results Based Accountability, we are exploring offering a multi-session training for this group this spring/summer.
1. 14 agricultural service providers receive monthly email updates about the project and invitations to participate in bi-monthly distance meetings (via Zoom). (Starting December 2018 and continuing through the second and third year of the project.) Each distance meeting will combine a presentation by an expert on a technical topic with opportunities for participants ask more general questions, and share challenges, successes and progress towards action steps. Emails will provide links to related resources. While we will fine-tune the topics based on the priorities of the cohort, anticipated year 2 distance session topics include:
• writing strong survey questions and reporting survey results;
• technology to automate, streamline and strengthen data collection in farmer education settings;
• Beyond the survey: observation, documents, interviews, and other data collection methods
• Uses and benefits of qualitative data.
In year 2 of the project, 21 agricultural educators and service providers engaged with the program, attending a combination of online and in-person educational workshops.
In the first half of the project year we held 3 distance meetings (via Zoom, and recorded), and two in-person workshops.
The topics covered during the three sessions were:
- Orientation to the Evaluation Works program
- Learning Organizations and Theory of Change
- Change Management
Supplementary resources and follow-up activities were provided for the second and third sessions.
Two workshops (covering the same content) were necessary because it was not possible to bring all the participants together for a single day session. Participants at both sessions indicated strong satisfaction with the opportunity to learn in a highly interactive, small group session. Content covered during the sessions included:
- Evaluation Frameworks: Participants learned about three approaches to evaluation frameworks -- logic models, rubrics and the Results Based Accountability framework -- with an emphasis on understanding similarities and differences, and uses and limitations of each of the frameworks models. Through individual and small group work, participants applied these concepts to determining an evaluation framework for their work, and began to consider which model(s) make the most sense for their program and/or projects.
- Learning Goals & Evaluation: Participants learned about and practiced writing learning goals that articulate changes in client behavior in specific measurable ways. Participants worked individually and in pairs or trios giving each other feedback and working together to develop learning goals that are both meaningful and practical.
- Data Collection & Survey Design: After discussion of different methods of collecting evaluation data and on survey and survey question design, participants worked individually to apply the concepts discussed to their work. We asked all participants to bring a the theory of change sheet they developed earlier in the year and recent (or planned) survey or other data collection tool (such as a application form) from their project. Each participant then marked up their survey/tool, coding it to identify the kind of performance measures it collects and what components of their theory of change it addresses. Participants were asked to consider whether they are collecting the right data, enough data or perhaps too much data given the goals of their program and the resources/capacity they have. Each participant also made a "to do" list about priority changes they want to pursue. Then in discussion in small groups, participants summarized what they discovered about their own instrument/theory of change, and provided feedback to each other. Based on feedback, participants updated their to-do list.
- Introduction to the Learning Circles and Technical Assistance Components of the Project: Participants learned about how the learning circles and technical assistance components would be structured.
- Action Planning: All participants completed an action plan with three next steps that they want to pursue. We took photos of their action plans and are using that to guide upcoming content for webinars and individual consulting.
- Reflections and take-aways: We concluded the session by getting feedback from participants about what worked well for them during the session and what they would do differently if we offered it again.
In the second half of the project year, the project director conducted an in-person, 75-minute workshop on Using Technology to Streamline Reporting at a UVM Extension in-service training. The session was attended by 13 Extension employees, both individuals who are part of the Evaluation Works cohort (4) and additional Extension faculty and program staff (9). The session provided information about ways technological tools can help agricultural service providers reduce the time they spend entering, managing, and report data and that can help make it easier to analyze what the data means for program performance and impact. At the end of the session, participants identified at least one approach to streamlining data collection that they would pursue in the coming months.
During the second half of the year the project also supported five people to attend a full-day, Stakeholder Engagement Training, with Curtis Ogden of the Interaction Institute for Social Change. The training took participants through the entire stakeholder engagement process, familiarizing them with a variety of helpful frameworks and assessment tools and, most importantly, providing hands on opportunities for them to apply the frameworks and tools to their own programs and needs assessment priorities, and give and receive feedback from other attendees. Participants found that it was valuable spend an entire day "thinking deeply about why, how, when, and with whom we should be engaging in our work.”
2. 14 agricultural service providers launch a peer learning circles process. Learning circles groups adopt a meeting and recording template (provided by the team) to outline the format, ground rules and expectations for their collaborative work and to document progress, accomplishments and challenges they encounter. (April 2018).
Participants have been assigned to five learning circles. The project director decided the membership of the learning circles based on the interest areas of the individuals (outlined in their self assessment). All of the learning circles are using the same meeting and recording template to outline the format, ground rules and expectations for their collaborative work. The template also serves as a way to document progress, accomplishments and challenges they they encounter, and to communicate with the project director about their progress and needs.
3. 20 agricultural service providers engage in ongoing discussion and program development with at least one peer (matches made by the program coordinator) and a consulting evaluator as they integrate new knowledge and skills into their programming. (Starting October 2018 and continuing through September 2020.) The consulting evaluator documents the questions/challenges participants are addressing and shares that information with the project coordinator.
Two of the learning circles have been meeting consistently and independently in the 8 months since they launched, with each member moving forward with their goals. The other three learning circles met for the first two or three months after they were launched, but encountered scheduling conflicts during the growing season and did not start back up independently. For the third year of the project, the project director is reorganizing the learning circles into pairs/triads to try reduce scheduling conflicts while still incorporating a peer-to-peer learning and support aspect to the project.
The project director and consulting evaluator have provided assistance and support to Evaluation Works participants based on their requests. Seven individuals have requested assistance, some on more than one topic. Assistance has been provided on the following topics
- survey design (2 people)
- survey questions (3 people)
- analyzing and interpreting survey data (1 person)
- developing an evaluation framework for their program (2 people)
- communicating findings to stakeholders and the general public (1 person).
Because a significant component of the cohort was interested in additional training on the Results Based Accountability framework and process, the project director and consulting evaluator developed a hybrid (in person and online) six-session "short course" for the cohort. RBA is a process for evaluating and improving public sector programs that has been adopted by Vermont state government. We invited Evaluation Works participants to invite colleagues from their organizations to participate in the training. As a result, six additional individuals are now receiving training through the project. We are now half way through the short course and will report in full on outcomes and impacts in year 3. The project director has developed a section of the Evaluation Works website for the RBA course, which includes recordings of sessions, supplementary materials, and assignments.
4. 20 agricultural service providers complete a follow-up self-assessment survey from the Project Director and report on actions taken thus far to use evaluation skills and knowledge from this project in their water quality and/or beginning farmer programs, and give input about future training needs. (by May 2019)
Of the 20 individuals who have participated in Evaluation Works education and training to date, 14 have provided feedback about changes in their knowledge, skillset, attitude and/or behavior via surveys, self assessments, learning circle notes and/or individual interviews. All 14 report they are in the process of integrating new knowledge and skills in their work.
Significantly, 10 participants report working collaboratively with other Evaluation Works participants to modify, improve and align their measures and data collection approaches across project, program, and organizations/agencies.
Additionally, four have used feedback from the project director, their learning circle partners and/or the consulting evaluator to make changes to surveys and other evaluation instruments.
For example, an Extension employee who routinely collaborates with a nonprofit on a variety of farmer education initiatives, reported that over the last year she and the executive director of the nonprofit (also an Evaluation Works participant) have had multiple conversations between themselves and with advisors and the nonprofit's board aligning data collection in the two organizations. Their goal -- to reduce survey fatigue among clients while focusing more on key measures and collecting "really substantive information to communicate to funders and policy makers." As a result, they have dropped some survey questions that they had been asking for a number of years, and modified or added others so that their instruments align.
Following are additional quotes from participants about the impact of participation thus far.
"I am really finding the information useful, and it has enabled our team to think more broadly about our program evaluation."
"The elements of RBA are helping me to consider new ways of measuring success."
"I'm gaining new understanding of the major areas of measurement, engaging with colleagues, and understanding how to lead my team through the process of developing these metrics."
"I am gaining tools to help facilitate discussions with a large team around finding commonalities for multi-programatic alignment."
"This course has really opened my eyes and provided me with a new area of excitement in my work."
"I keep thinking there is a magic formula to this -- like there is going to be an absolute right. It seems like there are wrongs but not a perfect right. So I'd like to get more comfortable with that."
"Its so great to have this toolkit because it is so relevant to the work I'm doing . . . I'll be having a metrics meeting with [my board] to really look at how can we expedite data collection and at the same time make sure we are collecting the information that can really help us make our program better. And I don't think it would have happened without this program."
The project director will follow-up in year 3 with all Evaluation Works participants and report on their progress incorporating new knowledge and skills in their program evaluation work.
5. Through the survey, 6 agricultural service providers share tools and templates that they developed/used in their work, which the project team shares on the Vermont SARE website. (July 2019)
Formal sharing of tools and templates (between participants) has been pushed back to year 3.
6. 20 agricultural service providers attend the project’s Year 2 Workshop, which builds on the skill base developed through the first year’s professional development activities. The focus of the year 2 workshop will shift from designing their evaluation plan and collecting data to analyzing their data and implementing changes in their programming. While we plan to fine tune content based on the priorities identified in the May 2019 survey, anticipated topics will include practices for assessing program performance (at the input, output, outcome and impact levels); using that performance data to modify programming to better respond to client needs and emerging conditions; and communicating results to stakeholders. As with the first workshop, the session will be highly interactive, with opportunities for both group discussion and individual reflection. By the end of the day, all participants will plan (and share with the project director) at least one new action step to improve programming related to new farmer development or water quality protection. (September 2019).
Responding to participants priorities, we substituted six-session RBA Short Course (which spans years 2 and 3) for a single day workshop. We will report on it in year 3.
The project’s core group of 14 agricultural service provider participants attend the
project’s second in-person workshop, which builds on the skill base developed thus far
in the project. The focus of this workshop will shift from designing their evaluation plan
and collecting data to analyzing their data and implementing changes in their
programming.
14 agricultural service providers engage in ongoing discussion and program
development with at least one peer (matches made by the program coordinator) and a
consulting evaluator as requested as they integrate new knowledge and skills into their
programming. (Starting May 2018 and continuing through August 2020.) The consulting
evaluator documents the questions/challenges participants are addressing and shares
that information with the project coordinator.
14 agricultural service providers identify least one new action step to improve programming related to new farmer development or water quality protection and incorporate it into their programming
14 agricultural service providers continue to receive monthly email updates from the
project and invitations participate in bi-monthly distance meetings (via Zoom). (through
September 2020).
14 agricultural service providers complete a follow-up survey from the Project Director
and report on actions taken thus far to use evaluation skills and knowledge from this
project in in their water quality and/or beginning farmer programs. (August 2020)
6 agricultural service providers share in-depth information about how their expanded
evaluation capacity improved their programs and impacts which the project team uses
to write case study examples. These participants are selected by the project
coordinator, based on the information they provide in the August 2020 survey. The case
studies are published on the Vermont SARE website. (September 2020).
30 agricultural service providers (core group of participants joined by a broader group of
agricultural service providers from the organizations and agencies they represent)
attend a final in-person meeting to share information about how they have used
evaluation to improve programs and to map out next steps at the individual,
organizational and inter-organizational levels. The budget for this summit would be in
the 2020-2021 state plan.
Milestone Activities and Participation Summary
Educational activities conducted by the project team:
| Activity | Year 1 | Year 2 | Year 3 | Total |
|---|---|---|---|---|
| Consultations | 0 | 10 | 10 | |
| Curricula, factsheets or educational tools | 1 | 3 | 4 | |
| Online trainings | 0 | 3 | 3 | |
| Study circle / focus groups | 5 | 5 | ||
| Webinars, talks and presentations | 1 | 3 | 4 | |
| Workshop / field days | 2 | 2 | ||
| Other educational activities: 1- Needs assessment - conducted interviews with 12 individuals 2- New Vt-SARE website with section focusing on the Evaluation Works project 3- Application form and self assessment instrument to collect baseline information about project participants | 3 | 3 |
Beneficiaries who particpated in the project’s educational activities and events:
| Audience | Year 1 | Year 2 | Year 3 | Total Individuals |
|---|---|---|---|---|
| Extension | 12 | 18 | 0 | 12 |
| Researchers | 3 | 2 | 0 | 3 |
| Nonprofit | 7 | 5 | 0 | 7 |
| Agency | 4 | 4 | 0 | 4 |
| Service providers (other or unspecified) | 26 | 0 | 0 | 26 |
| Farmers / ranchers | 1 | 0 | 0 | 1 |
| Others | 2 | 0 | 0 | 2 |
Participation Summary:
Learning Outcomes
We have not gotten to that point in the project yet.
Performance Target Outcomes
Performance Target Outcomes - Service Providers
Target #1
20 Extension educators and non-profit personnel will use increased outcome evaluation knowledge and skills to improve programs designed to help 150 beginning farmers launch and grow farm enterprises that meet their business, stewardship and lifestyle goals, and help 150 established producers adopt nutrient management, cover crop, and other production practices that support Vermont’s new water quality goals.
SARE Outreach
In 2019, the Vermont SARE Program conducted outreach about SARE grant opportunities and results of SARE research in the following ways:
- Bringing the Northeast SARE exhibit and materials to four major Vermont agricultural conferences:
- The Vermont Farm Show (3 days, January 2019)
- Northeast Organic Farming Association of Vt Winter Conference (2 days, February 2019);
- Vermont Industrial Hop Conference (1 day, February 2019)
- Vermont Grain Conference (1 day, March 2019)
- Coordinating and delivering a presentation at a 75-minute information session on SARE's graduate student grant program (April 2019).
- Maintaining and expanding the website for the Vermont SARE state program and the Evaluation Works professional development project (http://blog.uvm.edu/vtsare/).
- Continue to build social media presence of SARE projects on Twitter, Facebook, and partner blogs.
- Responding to individual telephone and email inquiries (approximately 20 over the year) from farmers and service about SARE funding opportunities.
In 2018, the Vermont SARE Program conducted outreach about SARE grant opportunities and results of SARE research in the following ways:
- Bringing the Northeast SARE exhibit and materials to five major Vermont agricultural conferences:
- The Vermont Farm Show (3 days, January 2018)
- Northeast Organic Farming Association of Vt Winter Conference (2 days, February 2018);
- Vermont No-Till and Cover Crop Conference (1 day, February 2018)
- Vermont Hop Conference (1 day, February 2018)
- Vermont Grain Conference (1 day, March 2018)
- Coordinating and delivering a presentation at a 75-minute information session on SARE's graduate student grant program (April 2019).
- Building a new website for the Vermont State SARE Program and state professional development project (http://blog.uvm.edu/vtsare/).
- Continue to build social media presence of SARE projects on Twitter, Facebook, and partner blogs.
Recieved information about SARE grant programs and information resouces:
| Audience | Year 1 | Year 2 | Year 3 | Total |
|---|---|---|---|---|
| Service providers | 250 | 250 | 0 | 500 |
| Farmers | 1800 | 1800 | 0 | 3600 |