Progress report for NEVT17-001

1. 12 Extension, nonprofit and public agency key informants participate in interviews and focus groups to identify and prioritize their learning needs and priorities related to evaluation of beginning farmer development and water quality programming practices.

Eleven agricultural service providers and one farmer participated in interviews about their learning needs related to beginning farmer development and water quality programs. The individuals represented Extension, nonprofit organizations, state government agencies and private consultants. Half of the individuals were people whose responsibilities include program leadership and coordination of multi-organizational projects, so they were able to provide insights into needs across the organizations they represent. The information they shared confirmed that building evaluation capacity and expertise is a need across organizations working on water quality and new farmer development. The needs assessment also gave us new appreciation for the amount customization and that will be needed to meet the variety of needs, and to support implementation within their teams and projects. For that reason, we revised our target for the number of participants from 24 to 12-15.

2. 120 agricultural service providers receive an announcement describing the new SARE evaluation education project; the announcement includes an invitation to enroll as a participant and complete an anonymous baseline assessment of their knowledge and skills.

We conducted outreach to 80 agricultural educators and service providers across Vermont. Outreach was conducted by direct, individual email to individuals who work for UVM Extension, the Vermont Agency of Agriculture, Vermont conservation districts, nonprofit organizations and USDA FSA and NRCS.  Within those groups, we targeted our outreach to people whose work has a significant focus either on helping new producers grow sustainable farm businesses or helping farmers implement practice that reduce agricultural non-point pollution. Emails pointed interested people to the new VT-SARE website, which provided an overview of the project. We also offered an interactive introductory webinar to potential participants a taste of the kinds of activities they would engage in and opportunities to ask questions. A recording of the webinar was posted on the project website so people who could not attend “live” could view it. 18 people attended or viewed the webinar.

3. 24 agricultural service providers complete the self-assessment and online application, and commit to participating in the online, in-person and peer-to-peer components of the project. (end of August 2018). The project team uses the information from registration forms to match participants with a “peer learning partner” from the group and notifies all participants of the match.

Seventeen individuals completed an online application and were accepted into the program. Subsequently, 3 individuals withdrew. Two of the individuals withdrew because of changes in their job. The third withdrew because the program was not a good match for that individual’s priorities at this point in their career.

The individuals current in the program are employed by Extension programs (6), nonprofit organizations (4), local or state government agencies (3) and a land grant university. They work across a variety of programs involved in nutrient management, cover cropping, grazing management, speciality crop, enterprise planning, business planning and apprenticeship and farmer mentoring programs. As expected, the cohort is diverse in terms of their prior knowledge and experience with evaluation topics, and in the specific kinds of information and skill building they hope to gain through the project. Participants indicated they anticipate using what they learn to improve current programming (13 people) to plan new programs (12 people) to build their network (12 people) and to make decisions about whether to continue programs (9 people). We decided to wait until the cohort has more interaction with each other to assign peer learning partners.

4. 14 service providers complete an online form verifying that they have completed a series of pre-workshop assignments. These assignments reviewing informational materials about theory of change, developing learning goals, and methods for assessing program performance. They also share the indicators their programs and/or projects are currently using to measure progress toward outcome goals. (September 2018)

All participants completed an in-depth self assessment about their current evaluation practices and providing us with information about the approaches/frameworks and measures they are using in the current programming. 

Participants have also completed work sheets outlining their own program’s theory of change and assumptions, and 12 of the 14 have completed and shared with the project coordinator their personal action and learning plan through the summer of 2019.

5. 22 agricultural service providers attend a full-day workshop to increase their knowledge and skills in the targeted evaluation topic areas. The focus of this session will be on framing evaluation questions and outcome indicators and approaches for collecting reliable, valid data. During the session, service providers will engage in individual reflections and small group discussions to generate ideas for implementing what they learn in their work with farmers, and planning at least one action step to improve programming related to new farmer development or water quality protection. They also obtain an orientation to the peer learning and expert consulting components of the project. (October 2018)

We held two sessions of the in-person workshop in April 2019. The same content was covered in both sessions. Two sessions were needed because schedules did not align for a single session.

6. 12 agricultural service providers use an online application form to request individual technical assistance from one of the consulting expert evaluators. (By the end of October 2018)

Requests for technical assistance were incorporated into the action planning document. Two individuals will be completing theirs in May. The project coordinator is currently working to identify the consulting evaluators to provide specific technical assistance/information individuals have requested. 

Because 6-8 individuals are looking for additional training in the methods of Results Based Accountability, we are exploring offering a multi-session training for this group this spring/summer.

1. 14 agricultural service providers receive monthly email updates about the project and invitations to participate in bi-monthly distance meetings (via Zoom). (Starting December 2018 and continuing through the second and third year of the project.) Each distance meeting will combine a presentation by an expert on a technical topic with opportunities for participants ask more general questions, and share challenges, successes and progress towards action steps. The monthly emails will provide links to related resources. While we will fine-tune the topics based on the priorities of the cohort, anticipated year 2 distance session topics include:
• writing strong survey questions and reporting survey results;
• technology to automate, streamline and strengthen data collection in farmer education settings;
• Beyond the survey: observation, documents, interviews, and other data collection methods
• Uses and benefits of qualitative data.

Fourteen agricultural service providers have are receiving regular communications from the project director.

We have held 3 distance meetings (via Zoom, and recorded), and two in-person workshops. The topics covered during the three sessions were:

1. Orientation to the Evaluation Works program
2. Learning Organizations and Theory of Change
3. Change Management 

Supplementary resources and follow-up activities were provided for the second and third sessions.

Two workshops (covering the same content) were necessary because it was not possible to bring all the participants together for a single day session. Participants at both sessions indicated strong satisfaction with the opportunity to learn in a highly interactive, small group session. Content covered during the sessions included:

Evaluation Frameworks: Participants learned about three approaches to evaluation frameworks — logic models, rubrics and the Results Based Accountability framework — with an emphasis on understanding similarities and differences, and uses and limitations of each of the frameworks models. Through individual and small group work, participants applied these concepts to determining an evaluation framework for their work, and began to consider which model(s) make the most sense for their program and/or projects.

Learning Goals & Evaluation: Participants learned about and practiced writing learning goals that articulate changes in client behavior in specific measurable ways. Participants worked individually and in pairs or trios giving each other feedback and working together to develop learning goals that are both meaningful and practical.

Data Collection & Survey Design: After discussion of different methods of collecting evaluation data and on survey and survey question design, participants worked individually to apply the concepts discussed to their work. We asked all participants to bring a the theory of change sheet they developed earlier in the year and recent (or planned) survey or other data collection tool (such as a application form) from their project. Each participant then marked up their survey/tool, coding it to identify the kind of performance measures it collects and what components of their theory of change it addresses. Participants were asked to consider whether they are collecting the right data, enough data or perhaps too much data given the goals of their program and the resources/capacity they have. Each participant also made a “to do” list about priority changes they want to pursue. Then in discussion in small groups, participants summarized what they discovered about their own instrument/theory of change, and provided feedback to each other. Based on feedback, participants updated their to-do list.

Introduction to the Learning Circles and Technical Assistance Components of the Project: Participants learned about how the learning circles and technical assistance components would be structured.

Action Planning: All participants completed an action plan with three next steps that they want to pursue. We took photos of their action plans and are using that to guide upcoming content for webinars and individual consulting.

Reflections and take-aways: We concluded the session by getting feedback from participants about what worked well for them during the session and what they would do differently if we offered it again.

2. 2. 14 agricultural service providers launch a peer learning circles process. Learning circles groups adopt a meeting and recording template (provided by the team) to outline the format, ground rules and expectations for their collaborative work and to document progress, accomplishments and challenges they encounter. (April 2018).

Participants have been assigned to five learning circles. The project director decided the membership of the learning circles based on the interest areas of the individuals (outlined in their self assessment). All of the learning circles are using the same meeting and recording template to outline the format, ground rules and expectations for their collaborative work. The template also serves as a way to document progress, accomplishments and challenges they they encounter, and to communicate with the project director about their progress and needs.

3. 20 agricultural service providers engage in ongoing discussion and program development with at least one peer (matches made by the program coordinator) and a consulting evaluator as they integrate new knowledge and skills into their programming. (Starting October 2018 and continuing through September 2020.) The consulting evaluator documents the questions/challenges participants are addressing and shares that information with the project coordinator.

Four of the five learning circles have had at least their first meeting and the fifth learning circle will formally launch on May 2.

Two learning circles each have two members, two have three members and one has four.

4. 20 agricultural service providers complete a follow-up self-assessment survey from the Project Director and report on actions taken thus far to use evaluation skills and knowledge from this project in their water quality and/or beginning farmer programs, and give input about future training needs. (by May 2019)

5. Through the survey, 6 agricultural service providers share tools and templates that they developed/used in their work, which the project team shares on the Vermont SARE website. (July 2019)

6. 20 agricultural service providers attend the project’s Year 2 Workshop, which builds on the skill base developed through the first year’s professional development activities. The focus of the year 2 workshop will shift from designing their evaluation plan and collecting data to analyzing their data and implementing changes in their programming. While we plan to fine tune content based on the priorities identified in the May 2019 survey, anticipated topics will include practices for assessing program performance (at the input, output, outcome and impact levels); using that performance data to modify programming to better respond to client needs and emerging conditions; and communicating results to stakeholders. As with the first workshop, the session will be highly interactive, with opportunities for both group discussion and individual reflection. By the end of the day, all participants will plan (and share with the project director) at least one new action step to improve programming related to new farmer development or water quality protection. (September 2019).