Final Report for ES03-067
Project Information
Objective 1: Develop Multi-Disciplinary, Multi-Institutional Training Teams. Our first objective was to develop a team of trainers in each state. These teams included state Extension specialists, representatives of certifying agencies and experienced organic producers. These teams took responsibility for developing and implementing training for field personnel in each state. The desired outcome was that each state has a functional team of experienced and knowledgeable trainers. These teams provide leadership for training programs in organic production in each state.
Objective 2: Develop a Two-Day Comprehensive Training Program for Field Personnel. Our second objective was for the three teams to work together to develop a comprehensive two-day training program in the regulations covering organic production and certification for field personnel. The desired outcome was a training program for field personnel from the Cooperative Extension Service, other USDA field agencies, collaborating state agencies, and organic certifying agencies (public or private). The desired behavior was to incorporate this training program into the formal in-service training programs provided by Cooperative Extension Service, other USDA field agencies, collaborating state agencies and training for organic certifiers offered through both the public and private sector.
Objective 3: Deliver Training to Field Personnel. Our third objective was to deliver the training program to at least 125 individuals who work directly with farmers. The desired outcome was for field service providers to understand and be able to communicate the regulations regarding organic production in the United States to agricultural producers. The desired behavior was that field service providers develop training programs about the requirements for organic production for farmers and that they provide timely, accurate information to growers about the requirements for organic production on an individual consultation basis.
Objective 4: Finalization of Training Curriculum Distributed. We develop a comprehensive training program that can be used by other land grant institutions or other agriculturally related parties in organic regulations and certification. The materials are available through the University of Florida’s Electronic Dissemination Information System (EDIS) free of charge to all users. They will also be available as a for sale publication for those who prefer a CD ROM delivery venue.
Timeline of project and trainings:
Initial Planning Meeting: January 26-27 2004, Gainesville, FL. Representatives from OMRI, ATTRA, Florida Organic Growers, University of Florida, University of Virgin Islands and Kentucky State University were present at the meeting At this meeting we determined the training strategies and the content of the modules. A timeline for the trainings was also set. The responsibilities were allocated for the development of the materials.
Pilot Workshop: July 24- 26 2004, Ft. Pierce, FL. The modules were pilot tested. 50 total trainees attended including farmers, Extension faculty from Florida. Representatives from NCAT, OMRI, FOG and the University of Virgin Islands and Kentucky State University also participated.
Major Module Revisions: July 2004 – March 2005. Modules were modified and the livestock module was added.
Training 1: April 14-15 2005, University of the Virgin Islands, St Croix. Trainers from Florida Organic Growers, OMRI, NCAT, and the University of Virgin Islands participated. 28 total trainees attended including extension faculty from the Virgin Islands and Puerto Rico, heads of farming organizations, state employees (territorial of USDA), employees of the Department Licensing and Consumer Affairs, and employees of the Department of Natural Resources at the University of Virgin Islands.
Training 2: July 6-7 2005, Quincy, FL. 22 total trainees attended including extension faculty, farmers, University of Florida graduate students in the Doctoral program in Plant Medicine and University of Florida faculty. Trainers included University of Florida faculty and representatives of OMRI, FOG and NCAT.
Training 3: September 7-9 2005, University of the Virgin Islands, St.Thomas. Trainers included University and Virgin Islands and University of Florida faculty and representatives of Florida Organic Growers and NCAT. 16 trainees attended including extension faculty from the Virgin Islands and Puerto Rico, heads of farming organizations, state employees (territorial of USDA), and employees of the Department Licensing and Consumer Affairs and employees of the Department of Natural Resources at the University of Virgin Islands.
Training 4: March 8-9 2006, Gainesville, FL. 20 total trainees attended including University of Florida faculty and graduate students, Florida and South Carolina Extension faculty, and Extension faculty from Auburn University, Alabama. Trainers included University of Florida faculty and representatives of OMRI, NCAT and FOG.
Trainings 5 & 6: March 2006 Lake Barkley, Kentucky with 16 total trainees and 26 total trainees at Kentucky State University, Frankfurt. Trainers included Kentucky State University faculty and representatives of OMRI, NCAT and FOG.
**Minor revisions of the modules were made as needed after some of the trainings.
Education & Outreach Initiatives
Methods
Measurement of Knowledge
We evaluated changes in knowledge of trainers through a pre- and post-test. The pre-post test method is commonly used for measuring change in knowledge, which is calculated as a percentage change in correct responses to a standardized set of questions. Ms. Kendall Sanderson, as part of her M.S. thesis research (2004), developed a standardized self-completion test based on the key components of the National Organic Standards and the Organic Production System Plan required for organic certification. She ensured high content validity by developing questions directly from the training modules developed by the project implementation team. The team members approved the final test.
A meeting was held in January 2004 for the tri-state team members. Here the training team decided the content of the training and ranked the criticality and frequency of the topics to be covered. Sanderson grouped the questions they suggested for the pre- and posttest of knowledge by topic or module, eliminating repetitive questions and adding more when needed. Eighty percent of the questions were divided into the categories of the Organic Production System Plan, Water Quality, Soil Quality and Crop Fertility, Crop Management, and Organic Integrity. Six percent of the content was devoted to the Overview of Organic Production, Planting Stock- Seeds and Transplants, and Livestock. Four percent was devoted to Handling and Processing and the remaining 10% to Organic Resources.
She developed the questions for the standardized test based on four cognitive levels of learning that were targeted in the training: knowledge, comprehension, application and analysis. Knowledge is the lowest level of cognition and only reflects basic retention of the subject matter. An example from the test is, “The transition period from conventional to organic agriculture is __ years”. The second level, comprehension, is the lowest level of understanding and shows if the participant actually understands the meaning of the material. The third level is application, allowing the respondent to show if she or he can use what was learned. An example from the test is “List three disease control methods that organic farmers can use”. Analysis is the final level addressed in the pretest and posttest and requires an understanding of the content and structure of the learned material. An example is, “True or False. A storage box or bin originally used for conventional crops can be reused for organic crops as long as both crops are not stored together in it at the same time”. Discriminatory power, or the ability to detect who really knows the answer from those who do not, is increased, as more and higher levels of testing are included.
The preliminary test was administered to people knowledgeable about the subject matter, but not necessarily experts. From these tests she eliminated invalid questions. Invalid questions were ones that fewer than ten percent answered correctly or more than ninety percent answered correctly. This test was then administered to a class of college students learning about organic agriculture. The same process was used to determine invalid questions. Backup questions were substituted at this time for questions that either too few people answered correctly, or too many people answered correctly, taking care to retain the balance of cognitive levels. The questions were weighted for higher cognitive levels since that was the aim of the training.
Determining a scoring system and answer key before administering the test increases precision and reliability and is also required protocol in the development of a standardized test. This test is based on a fifty point scoring system with each correct answer worth one point. She also calculated a weighted score for each test by multiplying each correct answer by the cognitive level of the question. For example, knowledge level questions were worth one point, comprehension questions were worth two points, application questions were worth three points and analysis questions worth four. Thus a correct answer to an analysis level question is worth four points in the weighted score. Each pre- and posttest was given two different scores, a raw score out of 50 points and a weighted score. It took eight months to develop the standardized pre- and posttest of knowledge.
Our test may overestimate change in knowledge because we administered the post-test immediately after the training event when recall is at a peak. We are aware of the bias that this introduces into our data, but feel that the advantage of being able to acquire data points for all participants outweighs the importance of the bias. If we had waited to administer the post-test for the recommended 10 to 14 days after training, experience shows that the response rate is apt to be low. The bias is probably negligible because our test emphasized higher cognitive levels that are relatively unaffected by short term recall and because the actual final weighted score gives a much higher weight to the higher level questions.
Measurement of Attitude
Sanderson also developed a scale to measure the attitudes of local service providers about organic production. Attitudes are psychological constructs or ways of conceptualizing intangible elements. Attitude has been defined in many ways. In plain terms, attitude is the extent of liking or disliking something. A scale is designed to evaluate the intensity and direction of the subject’s feelings about a concept or practice. The scale is constructed so that all items address only one specific issue or concept.
For this project a uni-dimensional Likert scale was developed. Likert scales present a range of statements about a topic and subjects rate how respondents feel about the statement on a scale of 1 to 5 where 1 indicates strong disagreement and 5 shows strong agreement. Statements in a Likert scale are not neutral. They are meant to elicit an opinion from the respondent who indicates to what extent she or he agrees with the statement. Statements used in this study range from “Extension should do more to help organic farmers” to “We need to pay attention to our mainstream clientele, not waste time with organic hobby farmers”. The frequency of positive and negative statements is balanced to minimize bias.
Coding is simple with scales because the value of each answer is determined when the scale is constructed. For example, a respondent answering “Disagree” to the statement, “Organic food is the only safe food” receives 2 points. Answers range in value from 1 to 5 corresponding to the answer choice, strongly disagree (1 point), disagree (2 points), indifferent (3 points), agree (4 points), or strongly agree (5 points). It is necessary to reverse the score on questions containing negatives in them, such as “Learning about organic farming is a waste of time”. Someone who answers “strongly agree” with that statement actually has a negative opinion of organic agriculture and is awarded a score of 1 (the reverse score of 5, strongly agree). Likert scales use a summative scoring procedure. Therefore, when the sum of the scores for each individual response is tallied, a higher score indicates greater approval of organic agriculture. Attitude rating scales are easy to use because they provide a single score that indicates both the direction (positive or negative) and the intensity (very positive or very negative) of a person’s attitude (Henerson, et al., 1987).
Precision is the exactness of a tool. By assigning one number value to a respondent’s scale, precision is increased, especially when compared to other more subjective research methods. Higher internal consistency, measured by inter-item consistency, increases precision.
The validity of a scale is the degree to which it measures the specific attitude of interest (Sommer & Sommer, 2002) and its appropriateness. The main argument against attitudinal scales is that people’s attitudes are complex and may not be measurable along a single dimension. According to Mueller (1986) validity is the most serious weakness in attitudinal scales. Responses can be faked or adjusted, especially if self-scoring. This is a universal problem in affective measurement.
Reliability of a scale indicates its consistency and accuracy in measurement. Originally Sanderson generated 134 statements regarding organic agriculture. She asked colleagues to rank the strength of the statements on a seven-point scale. In this way she eliminated the statements where the judges disagreed on the strength, if it was strong, weak, or any degree in between. This step helps ensure the reliability of the instrument. This reduced the number of statements to 60.
She then conducted a preliminary test to further reduce the number of statements to be used in the final scale. Groups of people were asked to indicate what their general opinion of organic agriculture was (negative, indifferent, favorable) before they began the scale. These participants were asked to complete the scale, indicating how each statement made them feel, ranking each question from 1-5 as described above (strongly disagree to strongly agree). It was necessary to obtain a balance of respondents who felt negatively, indifferently and favorably about organic agriculture in order to determine how well each item under consideration for inclusion in the scale differentiated among subjects.
Sanderson eliminated all items that elicited a neutral response because they did not discriminate between the respondents who claimed they were in favor of organic agriculture and those who felt negatively about it. She wanted to find statements that elicited a strong response, either for or against organic agriculture. She therefore ran tests to determine the inter-item correlation, Cronbach’s alpha, and the standardized alpha for each statement. Any statement with low scores was removed. To discriminate those whose opinions were favorable about organic agriculture and those who felt negatively towards it, a t-test was run using the top 10% and bottom 10% of responses. Normally 25% is used to compare responses in a t-test but in our pretest, the results showed strong positive and negative opinions. It was difficult to discriminate the average positive answer from the very strong positive answer using the traditional 25% approach. Using the top and bottom 10% did allow us to discriminate the strength of the answer. The statements were reduced to 20.
Cronbach’s alpha was 0.96 for the final 20 items selected for the scale as a result of preliminary testing. After the first training workshop was completed, Sanderson ran another internal consistency test using the raw data from the respondents’ pretest and posttest and the data from a quasi-control group’s pretest to determine if any additional items should be rejected. Cronbach’s alpha was 0.84, the standardized alpha was 0.84 and the inter-item consistency was 0.22. She decided to drop two more items from the final scale because both statements had low inter-item consistency scores and brought the average down considerably. She also chose to eliminate one other item, “People who buy organic food are gullible,” because the word gullible had to be explained to some participants who did not speak English as a first language. The final instrument had a Cronbach’s alpha of 0.85, a standardized alpha of 0.85 and an average inter-item consistency of 0.26. This process of developing the Likert scale for attitude took eight months.
Measurement of Confidence
Sanderson also developed an index to measure the confidence of participants to perform certain job related behaviors prior to and following the training. The index was developed with the help of Cooperative Extension faculty who suggested behaviors related to organic agriculture that one might expect to perform on the job. They also suggested what weighted score to assign each behavior, depending upon the frequency and difficulty of performance. Those behaviors that were mentioned by at least 80% of the respondents were included in the index. The index scores were weighted to reflect the difficulty levels of performing certain tasks. The mode of the weights suggested by the different respondents for each item was used as the weighting factor. Tasks ranged from answering questions from homeowners and consumers about organic products and options, to including organic farms on field tours. A task such as highlighting organic farms as a source for fresh food for consumers would not be as difficult a task for Extension faculty as making field visits to an organic farm for troubleshooting. Participants were asked to rate their confidence with a scalar response indicating how confident they felt about performing certain tasks related to organic agriculture.
Measurement of Intention
One goal of the training, What Service Providers Must Know About the Organic Rules and Regulations, was to enable agricultural service providers to assist farmers who want to meet National Organic Standards. The training proposed to increase the real number of agricultural service providers in the field who understand the National Organic Standards and who can advise organic farmers and farmers interested in transitioning to or beginning organic production. The workshop emphasized increasing participants’ ability to interpret the National Organic Standards and therefore their ability to advise farmers. It also provided them with training manuals and lesson plans they can use to develop their own training sessions for farmers.
Ajzen and Fishbein’s (1980) theory of reasoned action states that intentions are an accurate measure of behavior, so long as certain criteria are met. Namely, there must be a high correspondence between the intention and the behavior in four areas: action, target, context and time. A shorter time period between the measurement of the intention and the behavior is more accurate than a longer period, largely due to the consequences of unforeseen circumstances. Following these prescriptions and with the assistance of the grant team members, Sanderson developed an index with scalar response questions to measure participants’ intentions to perform job related behaviors. These behaviors are the same ones that were used in the index of confidence, and the generation of these behaviors and their validation has already been described.
Provide organic advice and options to homeowners.
Answer questions from homeowners and consumers about organic products.
Highlight organic farms as a source of farm fresh food for consumers.
Seek out organic producers in your county.
Include organic farm tours on field days.
Include organic techniques in demonstrations.
Make field visits to organic farms for troubleshooting.
Educate yourself about organic production (i.e. attend other trainings, seek out useful sources of information, etc.)
Include organic farming as an alternative for farmers who call you for advice.
Respond to organic producers questions about production practices and the National Organic Program.
Add organic producers to your advisory council.
Hold a training workshop about organic practices and standards.
Advise producers about where to get organic supplies.
Include information about organic production and standards in media such as a website, newsletter, radio or TV communication.
Workshop participants were asked to indicate how likely they were to conduct these activities in the next six months. For example, the statement, “I intend to hold a training workshop about organic practices and standards”, fits the four requirements for an accurate measure of intention to predict behavior. The desired action is that the agricultural service provider conducts a specified target, a training workshop. Other agricultural service providers or farmers interested in organic agriculture are the context for the target. A short time frame, six months, is given in order to make a more accurate prediction, and also, to motivate the agricultural service providers to take action sooner. Test participants checked their answer along a continuum of answers from probable to improbable. This measure of intent was included in the posttest and was only given to those test subjects who had completed the two-day workshop. This data was not collected from the quasi-control comparison group.
Measurement of Behavior
Follow-up evaluations were conducted approximately one year after the trainings to determine job related behavior of training participants. The index included the same list of job related tasks as the index for intentions. The tasks require the participants to interpret and advise to farmers about the national organic rules and regulation (see list of index items above). The possible answers for the behavior index were never, rarely, sometimes, often and almost always on a scale from 1 to 5, respectively. The items in the index were weighted to reflect the difficulty levels of performing certain tasks. All training participants were contacted by phone or email for a follow up evaluation. The number of respondents for the measurement of behavior was approximately 25% of the original training participants.
Outreach and Publications
The training materials generated from the grant are available online through the University of Florida's EDIS publications. The links are:
http://edis.ifas.ufl.edu/FY819
http://edis.ifas.ufl.edu/FY820
Analysis
We conducted tests of central tendency for the pre- and post-tests to determine whether training participants demonstrated increased knowledge, more positive attitudes and higher self-efficacy after the training. We used paired t-tests to examine differences in pre- and post-training scores for attitude, knowledge and confidence among workshop participants. For the Likert scale measuring attitudes about organic farmers and farming, the t-tests were performed for the mean response for each participant. The mean response was easier to interpret and understand. Similarly, for the standardized test of knowledge and the index of confidence, we performed t-tests for weighted responses.
We used multiple regression models to determine the relationships between the three predictor variables (attitude, knowledge, and confidence) and the outcome variable, intent to change practice. We conducted the multiple regression models to determine which of the variables contributed most significantly to the training participant’s intentions to apply the new rules and regulation for organic agriculture standards in their professions.
We analyzed the job related behavior of the participants after the training by calculating weighted summative index scores. We then collapsed the index scores to a limited variable of 0/1 to reflect low (0) and high (1) level of job related behavior that requires them to interpret and advise farmers about national organic rules and regulations. We also calculated the proportions of participants performing some of the key job related tasks.
Results
Results of this training were outstanding and far exceeded the project team’s expectations. After the training the participants demonstrated significantly greater knowledge and higher self-efficacy to interpret and advise farmers about the national organic agriculture rules and regulations than prior to the training. Also, after the training self-efficacy remained significant in its ability to predict the participant’s intentions to interpret and advise farmers while attitudes and knowledge became less predictive of their intentions.
Significant differences exist between before and after the training for the participant’s knowledge and self-efficacy but there was no significant difference in participant’s attitude before and after the training. Matched-paired t-tests for pre- and post-tests indicate that the training significantly increased participant’s knowledge (p-value <0.0001) and self-efficacy (p-value<0.0001) in regards to interpreting the new national rules and regulations for organic agriculture and advising farmers.
Table 1: Results of t-test for scores on pre- and post-test indices and scales of knowledge, attitudes, self-efficacy and intentions of training participants with regard to national organic agriculture rules and regulations 2005-2006 Florida, Virgin Islands and Kentucky
Variable Test Mean score Standard deviation t-score p-value*
Knowledge Pre 90.45 18.39 -12.23 <0.0001
Post 113.81 10.86
Attitude Pre 68.49 10.79 1.58 0.12
Post 66.78 9.51
Self-efficacy Pre 147.24 43.19 -5.85 <0.0001
Post 178.14 49.66
* Compare to alpha value of 0.05
Multiple regression analyses demonstrate that self-efficacy consistently is the most significant variable contributing to the training participant’s intentions to interpret and advise farmers about the new national rules and regulations for organic agriculture. Knowledge contributed less significantly to predicting the participant’s intentions after the training than before the training.
Table 2: Multiple regression analysis for scores on pre- and post-test indices and scales of knowledge, attitudes, and self-efficacy of participants to interpret and advise farmers about the national organic agriculture rules and regulations 2005-2006 Florida, Virgin Islands and Kentucky
Variable p-value pre-test p-value post-test
Knowledge 0.15 0.97
Attitude 0.34 0.10
Self-efficacy <0.0001 <0.0001
Pre-test multiple r-squared= 0.54, p<0.0001
Post test multiple r-squared= 0.62, p<0.0001
Analysis of the follow-up evaluations conducted after the training indicates that 57% of the sampled participants were performing a high-level of job related tasks that require them to interpret and advise farmers about the national organic rules and regulations. Sixty percent of the sampled participants made field visits to organic farms for troubleshooting since their participation in the training. Nearly half (48%) of the participants had included information about organic production and standards in media such as a website, newsletter, or radio or TV show since the trainings.
Table 3: Proportions of participants that perform job related tasks that require them to interpret and advise farmers about the national organic agriculture rules and regulations 2005-2006 Florida, Virgin Islands and Kentucky
Percent of sampled participants
High level of job related tasks 57%
Add organic producers to advisory council 30%
Hold training workshop about organic practices and standards 17%
Make field visits to organic farms for troubleshooting 60%
Include information about organic production and standards in media such as a website, newsletter, or radio or TV show 48%
A sample of participants was asked to comment about the training. Some of the participants said they specifically liked the hands-on approach of the training. Others noted that the training was useful and that it improved their ability to help their clientele. Some participants commented on specific activities that they thought addressed their concerns. Many participants noted that the training was very informative and they feel better educated about the national organic rules and regulations. Many participants also expressed interest in future advanced trainings.
Project Outcomes
The two main accomplishments of this project are: 1) design a training that increased self-efficacy of participants to interpret and advise farmers about national organic rules and regulations, and 2) demonstrate the importance of addressing self-efficacy in training extension professionals.
Training development often focuses on the desired result, overlooking the required behavioral changes needed to achieve those results. The first step to address self-efficacy in professional training is the structure of the program. Three main steps should be taken in the training development: identify the behaviors of interest, clearly define the concrete set of behaviors, and train professionals “how” to do tasks not just “what” to do.
The goal of professional trainings should be the change in behavior that results in the professional performing the job related tasks that require them to interpret and advise farmers about the new national organic rules and regulations. The training development team should first identify a concrete set of behaviors that will need to change for the job related tasks to occur. Trainings can focus on specific goals based on clear definitions of the desired behavioral change. Most importantly, program trainings need to focus on how professionals can use their knowledge in the face of challenges. The training team should train service providers about how to address the challenging tasks required in interpreting and advising farmers about the national organic rules and regulations, not just what tasks to do. Providing information is not enough to raise professional’s self-efficacy about interpreting the regulations and providing advice to farmers, they need the training to know how to modify their behavior. The training team can develop specific activities that address these defined behavioral changes.
This training project was explicitly designed to enhance self-efficacy and was successful in increasing participant self-efficacy to interpret and advise farmers about the national organic agriculture rules and regulations. The approach taken in this training is based on the concept that an individual’s confidence will increase when they know how to find needed information, how to interpret and apply information to complex situations, and when to seek outside help. The training aimed at the higher cognitive levels to enhance the participant’s understanding of the tools they have to interpret and advise, not just the information. The goal of the training was to increase the participant’s comfort in interpreting their understanding of the material.
The activities of the training were specifically designed to address self-efficacy not just to provide information. The fact that in a two-day training only 20 minutes were spent lecturing is demonstrative of the approach taken in this training. Participants were provided with OMRI, ATTRA and NOP materials but at no time in the training did trainers or participants have to read the materials instead they were prompted to participate in activities that required them to seek out the appropriate information and make an appropriate interpretation. The training focused on the difficult portions of the materials to interpret and never reiterated the basic information.
Activities were designed so that participants felt confident to 1) find where in the materials to find an answer to farmer’s questions, 2) feel comfortable when it is appropriate for them to interpret the materials and 3) most importantly, know when not to interpret the materials, recognizing when the certifying agencies will need to make the judgment call on-farm.
The activities included hands-on role-playing where the participants had to interpret materials in groups. The participants all completed an organic agricultural system plan based on realistic farm scenarios.
Knowledge contributed less significantly in the post-test multiple regression model highlighting the training’s impacts on and importance of self-efficacy in professional performance. An important result of this training is how knowledge became less predictive of participant’s intentions after the training. Prior to the training knowledge was a significant predictor of their intentions. After the training, although the participants acquired significantly more knowledge (matched t-test), their level of knowledge became less predictive of their intentions. Both knowledge and attitudes became less important while self-efficacy remained significantly predictive of intentions. This is reflective of the training’s focus on interpretation not knowledge. Self-efficacy being the most predictive variable of intentions supports the training’s success at improving participant’s ability to interpret and advise. Knowledge is less important when all the information is available to the public; it becomes much more important to be confident in your ability to interpret the information and know when a judgment call will need to be made by the certifying agencies.
le below.
Future Recommendations
The project team recommends the use of these materials for in service training of agricultural service providers who are interested in learning more about the National Organic Program regulation. These modules provide a framework for developing a general understanding of the regulation and how to identify and use resource materials currently available.