Educational Materials and Training that Foster Implementation of Ecologically Based Pest Management Decision-Making in Great Lakes Apple Production

The process of producing the DVD is fully discussed under the accomplishments/milestones section of this report.

Appendix 2.
Evaluation Report Prepared by Jean Haley

Introduction
After receiving funding through USDA’s North Central Region Sustainable Agriculture Research & Education grant program, Michigan State University contracted Haley Consulting Services LLC (HCS) to provide evaluation services for the project titled Educational Materials and Training that Foster Implementation of Ecologically Based Pest Management Decision-Making in Great Lakes Apple Production. Short-term project outcomes were described in the original grant proposal as:

• Videos will be made available through extension libraries, organized workshops, and colleges.
• Video and CD-ROM will reach an audience of 1000+ viewers, including growers, educators, crop consultants, scouts, input supplier field representatives, university extension specialists, college students, potential scouts, and residential fruit growers.
• Crop consultants, agricultural extension specialists, and input suppliers will use the materials to recruit and train new scouts.

This evaluation report provides a summary of evaluation activities that were conducted to measure the above outcomes. In addition to measuring the above outcomes, HCS and project management made the decision to look more closely at the process that was undertaken to produce the video. Providing an understanding of what went into the production of the video and how project participants were able to actually achieve their desired outcomes offers participants and funders an opportunity to document some critical “lessons learned” that can be applied to future, similar efforts.

Report Organization
Following a section on methods used to collect evaluation data is a brief description of the project background. The background serves to set the stage for the rest of the report, which is primarily made up of the results section. A conclusion is provided after the results. The appendices contain all data collection instruments and an interim report on focus group results.

Methods
HCS undertook three main evaluation activities for the Apple Scout Training DVD project:
1. Grower / Consultant / Scout Focus Group. Conducted in December 2003, a three-hour focus group of 11 apple growers, consultants and scouts provided constructive feedback on an early working draft of the training video. The meeting started with a brief overview and history of the project, given by David Epstein, principal investigator for the project. After his presentation, David was asked to leave the room so that participants would be encouraged to more freely express their comments and feedback. David’s presentation was followed by a viewing of the DVD in its entirety. Participants used a workbook to take notes and rate the clarity of the various sections during their viewing. They were given some guidance as to what could be changed and what could not be changed at that stage of production. For example, it was no longer possible to get additional video footage, but it would be possible to insert still photographs and graphics. At the end of the viewing, the meeting participants discussed each section, the rating they gave it, and suggestions they had to improve the product.

Workbooks were collected from participants at the end of the focus group, ratings for each section of the DVD were entered into a statistical package, and distribution statistics were calculated. Any additional comments that were provided in their workbooks that were not discussed during the focus group were added to the database. Comments from the group discussion that were recorded on flip chart paper were transcribed and summarized. See Appendices E1 and E2 for a complete report of the focus group results and a copy of the workbook.

2. Workshop Participant Feedback. During the winter of 2003-2004, project representatives attended seven regional grower meetings throughout Michigan where apple production was a focus. At these meetings the DVD was presented to audiences that numbered between 15 and 100 participants. After two of these sessions where the DVD was presented and discussed, participants were asked to complete a very brief feedback form. The total number of completed forms was 45 out of approximately 123 people in attendance at the two meetings surveyed (37 percent response rate). Data from these feedback forms were entered into a statistical package and frequencies were run. Open-ended questions were coded by hand and summarized. See Appendix E3 for a copy of the feedback form used at these meetings.

3. Key Player Interviews. As the focus group and workshops were being prepared for evaluating the Apple Scout Training DVD, it became clear that a lot of valuable evaluation information would not be collected if we were to focus solely on measuring desired outcomes. While it is extremely important to remain results-driven while implementing a project, it is often equally important to focus on process during the evaluation. Without the closer look at process, valuable lessons-learned remain undocumented and wheels then need to be reinvented by others attempting to accomplish similar goals.

During the spring of 2004, HCS conducted eight interviews with key players from the project. Each person answered the same seven questions and conversations lasted between 10 and 25 minutes, depending on the level of detail of responses and the level of involvement in the project. Results from these interviews contribute substantially to the results presented in this report as well as the structure for presenting results. Please see Appendix E4 for a copy of the interview guide that was used to conduct the interviews and a list of interviewees. Those who participated in interviews were guaranteed anonymity to encourage more freedom to express their views. As such, no names are linked to specific comments in the report.

Project Background
While there was one individual who stands out as the leader and coordinator of this project, from its very inception, what has come to be known as the Apple Scout Training DVD was a team effort. The idea for an educational video that could be used to train apple production field scouts was conceived during a Michigan farm tour in the summer of 2000. John Bakker, a scout for more than 30 years, and at the time the director of Westcentral Michigan Crop Management Association, was speaking to participants about the difficulties in training and retaining good field scouts. He mentioned during his presentation that extension publications are helpful, but that actual field experience is the most effective training method. Unfortunately, learning the vast complex of pests found in apple orchards cannot be accomplished in one, two, or even several trips through an orchard. John dreamed of the day when there was some sort of video program that had actual film footage of the various pests and diseases found in the orchards throughout the growing season that he could use to train his scouts.

Also participating in the tour that day were David Epstein, the Tree Fruit Specialist with the MSU Integrated Pest Management Program, and Jean Haley, an evaluation specialist. While still on the tour, these three people put their heads together to figure out a way to give John’s vision shape and make it a reality. Funding was secured in 2001 from the North Central Region of Sustainable Agriculture Research & Education through the diligent efforts of David Epstein who spearheaded the grant writing effort. David brought together key players at Michigan State University and the industry to design the project and build a budget. Once funding was secured, work on the project began in earnest. The final version of the DVD was produced in the spring of 2004, some three years later. Along the road were trials and triumphs and many lessons learned. The following section on Results provides the story behind the DVD, a kind of “making of” for the project. Within the story, results from the focus group and workshop feedback are provided as appropriate.

Results
Key player interviews provided an overview of the major steps involved in producing the final product for this project. Figure 1 on the next page illustrates these steps, which provides a straightforward chronological outline of the project. Following the figure is a brief description of each of the steps, the challenges faced by the project in achieving them, and lessons learned.

Project Coordination • Idea
• Recruit Key Players
o Flesh out project design
• Secure Funding
o Grant research
o Grant writing
o Contracts
• Establish Project Team
o Growers
o Consultants & Scouts
o University & Extension specialists
o Professional video crew
• Write video script
o Training from professionals on what makes a good video
o Outline content modules
o Assign writing to teams & individuals
o Compile, review & edit scripts
• Identify film shots
• Gather film footage
o Arrange for professional film crew to get shots in the field
o Collect shots with narrator in the field
o Set up network of consultants and agents to inform team when specific shots (e.g., pest populations) are in orchards that they work in
o Buy high quality portable camera to take advantage of opportunities of shots while working in the field
• Edit film footage
o Put together video based on script and shots
o Record and insert voice-over narration
o Insert still shots & graphics
• Review & edit rough cut
o Script writers’ review & edit
o Focus group with end-users
• Final edits
• Produce, market & distribute
o Presentations at all relevant meetings
o Publication in University Bulletin
o Advertising in relevant trade publications
o Use as course material for training workshops

Figure 1. Key Steps in Video Production.

Once funding has been secured for the project, the first order of business is establishing the project team. It was very important for this project that the team was made up of people from industry as well as the university. As one interviewee put it, getting “folks on the battle lines to contribute” was “critical to the quality of the product.” The other important team member for the project was a professional video company. There were, however, some difficulties with having the professional video company participate, not the least of which was the dissolution of the partnership between the two video company owners. In addition, the amount of time necessary to coordinate all the people involved and set up professional equipment in the field to gather footage was grossly underestimated. The team improved over time at scoping and prepping the site for the professionals, but they still ran into unforeseen problems such as rain and gusting winds. One other difficulty that arose was that the professional video crew was not available at a moment’s notice to run to a field where there might be a particularly good shot of an orchard pest. Shots with the professionals had to be scheduled in advance, which proved to be very difficult because of the relative unpredictability of pest outbreaks and weather. More on gathering footage will be discussed below.

During the early negotiations and meetings with the professional video company, discussions on the media format took place. The decision was made to create a DVD instead of a VHS video, which was the original format proposed. The professionals made strong arguments for the DVD format, which is more versatile than VHS, which was described as “going the way of the 8-track tape player.”

After putting together the project team and deciding on the media format, the next order of business is getting the script written. It was emphasized by most of those interviewed that this was one of the most important and most challenging aspects of the project. The professional video crew trained project members on what makes a good video and the basic elements of writing a script. Script writers for the project participated in full day meetings to hash out together the content of the video and outline the modules. Once modules, or topic sections, were agreed upon, writing assignments were given to small teams and individuals. Further meetings were held to review and edit draft scripts. The project coordinator, David Epstein, compiled the scripts into one document so that specific film shots could be identified. More than one participant mentioned that identifying shots prior to heading into the field was of critical importance. Having the script written first dictates exactly what shots need to be collected and just how long they should last. There were a few instances where the script was edited after the shots were collected and the team found themselves wishing they had had a few more seconds of a specific shot to better fit the running time of the narration.

Three interviewed participants offered the suggestion of having a video camera set up during these writing and editing sessions so that they could record drafts of the script and review how they looked and sounded on screen. Having never done such a project before, they did not really know how some things looked or sounded when projected on a television. One participant described some of the early drafts of portions of the script as text written for a thesis, “just plain boring!” Knowing that they really wanted to engage the viewer, highly academic writing was not the ideal. Having a camera on the spot to record and review may have helped with this earlier in the process. In addition to the sound of the text, it would have been beneficial for the narrator to have an opportunity to practice and review what he looked like on film prior to having to take final shots in the field. Having the benefit of seeing what he looked like on screen in advance would have helped him better understand and execute some of the direction given by the professionals during the filming.

Gathering the film footage was far more complicated than project organizers thought it would be. At the beginning of the project, the team anticipated that the professional film crew would be collecting all of the footage. It became quickly evident that this would not be possible. Using the professional film crew was cumbersome and time consuming. The first problem was the need to schedule shoot dates and times with the professional crew in advance. When dealing with a natural system, this was not going to be possible for all situations. In addition, the amount of time to set up a shoot was considerable. To deal with the issue of timing, the project invested in a high quality camera that the coordinator kept in his vehicle at all times. During the season, when he was out in the field, he could then take advantage of opportunities to film specific pests and orchard conditions. Team members who are field scouts, consultants and agents alerted the coordinator when they came across any of the pests or field conditions on the shot list. Operating as a team in this fashion, they were able to capture some excellent examples of pests and diseases for the final product. It should be noted that despite difficulties with the specific company that was hired, the general consensus among interviewed participants is that a professional video company should be involved in this process because their knowledge and expertise in what makes a good video is invaluable and something that no one else in the project had.

Once all the footage was collected from the field, the video was put together based on the script. The number of hours involved in this process was again grossly underestimated by the project team. The project coordinator estimated that for every minute of video in the final product, he alone put in approximately 52 hours of work. This estimate included all hours spent from beginning to end, from funding to marketing the final product. The hours spent with the professional video crew were considerable as they had to edit down two years worth of film footage, insert voice-over narration, locate and insert still photographs, create and insert graphics, and make sure that the whole thing flowed and had continuity from start to finish.

After a rough cut, or working draft, was completed, the project team reviewed the video in its entirety as a group. During this viewing they looked for typographical errors in any of the graphics that were added, and they listened to the voice-over narration to make sure that it was clear and engaging. Some edits were made to this version before it was taken to a focus group of potential end users. The focus group as conducted in December 2003 with 11 apple growers, consultants and scouts. Overall, focus group participants were very impressed with the DVD and expressed that they thought it would be a useful tool for them and the industry. Growers, scouts and consultants alike could envision using the DVD to train newly hired scouts. While they did notice some repetitiveness between sections of video, they did not find this to be much of a problem since they would likely use the DVD as it was intended—a resource to be viewed at specific points during the season.

Three major themes emerged during the discussion period, two of which became the slogans for each section:
• Name it – show it! The group felt that if a particular pest or beneficial was important enough to be discussed in the narrative, then it should be shown on the screen.
• Show it – label it! In the same way, if a pest or beneficial is going to show up on the screen, then it should be labeled. The group rallied together with these two battle cries whenever they came across something that had been named and not shown or something that had been shown and not labeled.
• The final theme that emerged during the discussion called for including one last brief summary at the end of each section to highlight key points. Participants suggested that a bulleted list would work well for this.

It should be noted that the focus group participants gave each of the 21 modules (including the introductory module on how to use the video) consistently high scores. On a scale from 1 to 10, where 1 is not clear and 10 is extremely clear, averaged scores ranged from 8.21 to 9.39, indicating that any additional changes made would further enhance a product that was already very good. For a complete copy of the focus group report, see Appendix E1.

Considerable time and effort was taken to incorporate the “show it” and “label it” suggestions wherever possible. This included tracking down and inserting photographs, creating and inserting captions, improving and creating graphics, and where possible, additional video footage. The suggestion of adding a summary at the end of each module was not incorporated due to time and budget constraints. The video was already at 95 minutes, 35 minutes longer than originally planned, and the addition of a summary for each section would have made it even longer still. Additionally, the video was never intended to be viewed in its entirety, as it was for the focus group. The intent is for it to be used as a resource during specific times of the season, so that viewers could pull up specific modules to study. When used in this manner, the need for a summary at the end of every section becomes unnecessary.

During the final editing process, the project took advantage of regional grower meetings that take place during the winter months to present and market the training DVD. At two of these meetings, feedback was collected from audience members to gauge how useful they thought the video would be for their industry and whether or not they personally would be able to make use of it in their work. The video was also promoted at three of these regional meetings by having the DVD running continuously on a monitor set up at an educational booth for visitors to view.

Feedback from the two meetings was highly favorable with nearly three-quarters of respondents (74 percent) stating that they felt the DVD would be extremely useful for the apple industry. Only three percent said they felt it would be not useful. Additionally, 76 percent expressed that they would be able to personally use the DVD in their work. When asked how they would use the DVD, the 43 percent described using it in some form of training or education for others, 33 percent described using it to educate or further train themselves, 19 percent mentioned using it as a reference tool and one person wrote, “use as information for decision making.”

Conclusion
The project” Educational Materials and Training that Foster Implementation of Ecologically Based Pest Management Decision-Making in Great Lakes Apple Production” was successful on many levels, including achieving the short-term goals outlined at the beginning of the project. The DVD is being made available through the Extension Bulletin, workshops and the University. It has already reached an audience of nearly 1,000 viewers, including growers, educators, crop consultants, scouts, input supplier field representatives, university extension specialists, college students, and potential scouts. In addition, regional meeting participants who were surveyed stated that they would be able to use the video to train current and new scouts.

In addition to achieving the outlined goals, the project paved the way for more training videos of this caliber to be developed for other crops throughout the country. The lessons leaned by the dedicated project team members will help others who set out to produce a similar product. One of the final interview questions was, “If someone else were to produce a similar DVD in another crop, what three pieces of advice would you give them?” The top three responses were that the project must have a strong, dedicated coordinator who is capable of organizing and facilitating groups, a professional filming organization should be involved in the project, and enough time and money should be budgeted to complete the project. One this last point, project members estimated that they underestimated the amount of time necessary to complete the product by as much as 100 times, and the budget was underestimated by approximately $16,000. Additional money had to be found elsewhere in order to complete the project with the level of quality desired. One other strong piece of advice offered was that similar projects should involve both university and industry specialists working together. Both informed the process and the product in unique and important ways. All team members who were interviewed mentioned at some point during the interview that, despite many difficulties, they all enjoyed the project very much.

Focus Group

To test the beta-version of the DVD, a three-hour focus group was conducted on December 11, 2003 in Grand Rapids, Michigan at the Great Lakes Fruit and Vegetable Expo. To encourage participants to attend a meeting being held at the end of a long week of meetings, the project provided bag lunches and drinks. (One of the participants brought his own apple cider to share with the group.) Eleven potential end-users were invited to participate in a focus group. Prior to the meeting, seven of the 11 were identified apple growers, and four were consultants and scouts. The meeting was attended by nine people who self-identified as:

• 5 growers
• 2 grower / scouts
• 1 consultant / scout
• 1 unknown (did not specify)
Complete anonymity was guaranteed to the meeting participants so that they would be encouraged to comment freely on the content and usefulness of the DVD.

The meeting started with a brief overview and history of the project, given by David Epstein, principal investigator for the project. After his presentation, David was asked to leave the room so that participants would be encouraged to more freely express their comments and feedback. David’s presentation was followed by a viewing of the DVD in its entirety. Participants had a workbook to use to take notes and rate the various sections during their viewing. They were given some guidance as to what could be changed and what could not be changed at that stage of production. For example, it was no longer possible to get additional video footage, but it would be possible to insert still photographs and graphics. At the end of the viewing, the meeting participants discussed each section, the rating they gave it, and suggestions they had to improve the product.

Initially, there were some difficulties getting the room set up; facilities management had not provided the extension cords and tables that were needed. These items were scavenged from other rooms and hallways, and participants were good sports to help acquire tables and set them up.

The DVD was run from a laptop computer that was hooked up to an LCD panel. The image was projected onto a large screen at the front of the room so that all meeting participants could easily see the picture. During the viewing, I sat in the back of the room and took notes. During the discussion, I took notes using the flip chart in the front of the room. Figure 1 shows the how the meeting room was set up during the DVD viewing.

Results Highlights
Overall, focus group participants were very impressed with the DVD and expressed that they thought it would be a useful tool for them and the industry. Growers, scouts and consultants alike could envision using the DVD to train newly hired scouts. While they did notice some repetitiveness between sections of video, they did not find this to be much of a problem since they would likely use the DVD as it was intended—a resource to be viewed at specific points during the season.

When asked how they thought it could best be distributed, everyone agreed that it should be advertised in the Fruit Grower News and extension bulletins. They mentioned that the distribution of the IPM scouting guide had worked well and recommended using the same strategy for the DVD.

Three major themes emerged during the discussion period, two of which became the slogans for each section:
• Name it – show it! The group felt that if a particular pest or beneficial was important enough to be discussed in the narrative, then it should be shown on the screen. Specific examples are given below in the section by section discussion.
• Show it – label it! In the same way, if a pest or beneficial is going to show up on the screen, then it should be labeled. The group rallied together with these two battle cries whenever they came across something that had been named and not shown or something that had been shown and not labeled.
• The final theme that emerged during the discussion called for including one last brief summary at the end of each section to highlight key points. Participants suggested that a bulleted list would work well for this. The drawback to this suggestion is that for 21 sections of the DVD, this could potentially add significant time to the overall production.

Conclusions and Recommendations

With a range of average clarity scores from 8.21 to 9.39, the production team should be commended on creating such a clear training DVD. As stated earlier, the overall impression of the DVD is that it is very professional and that it will be useful to growers, consultants, and scouts alike.

The overall mantras of the focus group participants should be taken to heart: If it’s important enough to mention, it’s important enough to show; and if it’s shown, it should be labeled. Some specific examples are given in the individual sections above, but the entire video should be reviewed to identify other examples where thumbnails, stills and labels would be helpful.

Bulleted reviews at the end of each of the modules should be incorporated into the video. Despite the amount of time this might add to the overall length of the DVD, the group felt this would be very helpful.

Among the individual modules, the Growing Degree Day Modeling section received the most enthusiastic conversation. The first few paragraphs of the script should be reviewed and clarified so that viewers come away from this section feeling that GDD modeling was explained. Refer to the specific comments above for more in-depth comments on how to improve this section.

Future workshops with growers, consultants and scouts should either be lengthened to four to five hours or truncated to cover only portions of the DVD so that ample time is available for discussion.

The short-term project outcomes were described in the original grant proposal as:

• Videos will be made available through extension libraries, organized workshops, and colleges.
• Video and CD-ROM will reach an audience of 1,000+ viewers, including growers, educators, crop consultants, scouts, input supplier field representatives, university extension specialists, college students, potential scouts, and residential fruit growers.
• Crop consultants, agricultural extension specialists, and input suppliers will use the materials to recruit and train new scouts.

If the production team is able to incorporate these suggested improvements to the DVD and produce a final product by early 2004, then it is anticipated that the short-term project outcomes will be achieved by early summer 2004.