Final Report for LNC01-185
Project Information
This study evaluates three SARE programs from inception through 2002. 171 former SARE Principal Investigators (PIs) were surveyed, and 33 interviewed, representing three grant “families”–researchers, producers, and educators – to: 1) evaluate cascade impacts of SARE funding; 2) explore ways in which SARE influences knowledge, attitudes, and behaviors; 3) understand participants’ perception of a “successful” SARE project; 4) provide recommendations; and 5) compare responses between groups. Quantitative results indicated moderate impacts; however, interviews revealed considerable support for each program’s value. Producers favored more niche production research and rated the value of the final SARE reports significantly higher than the other two groups.
Introduction:
Evaluation of government-funded programs is necessary in order to help stakeholders understand the ways in which programs add value and where they can be improved to better serve their intended populations (Henry et al., 1998; SARE, 1997; Rog & Fournier, 1997; Rogers et al., 2000; Stufflebeam, 2001). This article provides selected findings from a recent evaluation of the North Central Region Sustainable Agriculture Research and Education (NCR-SARE) program conducted as a part of the program’s commitment to evaluation and program improvement (SARE, 1997).
In 1988, the USDA established the Low-Input Sustainable Agriculture (LISA) program to fund research and adoption of sustainable agricultural (SA) practices. Later the program was revised and renamed the Sustainable Agriculture Research and Education program, and it continues under that banner (SARE, 1997). While scholars and advocates continue to offer variations on the definition of the term, sustainable agriculture (Allen, 1993; Francis et al., 1990; Hall & Kuepper, 1999; Harwood, 1990; Kroma & Flora, 2001; MacRae et al., 1990; Pretty, 1998; SARE, 1997), sustainable agricultural practices are generally considered to include those that are economically viable, environmentally sound, and socially just. Appropriate methods for measuring “it” continue to be debated among the disciplines represented under the umbrella concept of sustainable agriculture (AAE, 1998; Francis et al., 1995; Grieshop & Raj, 1990; Lightfoot & Noble, 2001; Morse et al., 2002; SARE, 1997).
The initial SARE program provided $3.9 million to fund researchers conducting collaborative, on-farm trials through Research and Education (R & E) competitive grants (SARE, 1997). In 1992, a second program was initiated in the North Central Region under the SARE umbrella, which provided funds directly to producers to conduct their own applied, on-farm research (AEE, 1998). In 1994, Congress approved a third SARE program, Professional Development Program (PDP) grants, to support sustainable agricultural education and outreach activities for Cooperative Extension system educators and specialists, Natural Resources Conservation Service staff, and others who provide educational programming to farmers and ranchers (SARE, 1997).
From inception through implementation, all funded projects include a strong outreach component and significant involvement from their farmer/rancher audience or others they are intended to reach. Research and Education grants are led by universities or non-profit organizations that use an interdisciplinary approach to fund projects up to $150,000. Successful Producer grant applicants receive funding for on-farm research that typically costs between $6000 (individuals) and $18,000 (groups). The size of PDP grants varies, depending on the number of states involved and the overall scope of the education projects. By 2004, Congress had increased funding for SARE to $16.6 million, annually. That totals $176 million for sustainable agriculture programming to fund close to 3,000 projects since the program’s inception (Elaine Hauhn, SARE Program Analyst, personal communication, May 11, 2004).
Funded projects are selected from among applications that survive a rigorous review process unique to each respective grant “family.” The process involves review by various committees made up of industry experts from universities, non-profit and for-profit organizations and producers. Grants typically range from two-year to three-year duration. Final reports are required at the conclusion of all projects.
In 1997, the USDA the report, “Ten Years of SARE,” to highlight accomplishments of the SARE funding, nationwide, during the program’s first decade. The case study report provided brief descriptions of one project in each of the four regions (North Central, Northeast, South and West) that represented each of ten categories: Crop Production, Animal Production, Natural Resource Protection, Marketing, Community Development, Education, Pest Management, Horticulture, Professional Development, and Integrated Farm/Ranch Systems.
The next year, den Biggelar and Suvedi (AEE, 1998) published “An Evaluation of the North-Central Region SARE Producer Grant Program.” This report was the result of an extensive study by the AEE Center for Evaluative Studies at Michigan State University, which examined successful and unsuccessful Producer Grant applicants on a variety of research questions. Questions included opinions about the grant program, perceptions of problems with the program, and program needs for increasing growth of sustainable agriculture. Participants were asked about their preferred sources of information on conventional and sustainable agriculture, and the economic, environmental and social impacts of the grant program. To date, there has been no parallel evaluation to assess the impacts of either the R & E or the PDP grants.
To strengthen its understanding of SARE’s impacts and effectiveness, the North Central Region SARE program has identified evaluation as an on-going process that will receive priority (Bauer, 1998). Despite a lack of formal program evaluation, we have witnessed, over the years, numerous anecdotal examples of ways in which SARE funding has led to increased knowledge, shifts in attitudes, and the embracing of farming or ranching practices that demonstrate the presence of a sustainable agricultural paradigm. In 2001, Poincelot used the SARE program as an indicator that, indeed, sustainable agriculture had become accepted in mainstream American agriculture (Poincelot, 2001).
With indicators that sustainable agriculture is now on the radar screens of administrators and researchers in the U.S. (Allen, 1993; Francis, 2000; Francis et al., 1995; Poincelot, 2001), we became curious about ways to adequately evaluate the effectiveness of the SARE program. We wanted to look not only at the short-term impacts provided by the final reports, but also the longer-term, direct and indirect effects – herein referred to as the “cascade effects” – of the SARE funding over the course of the program’s existence. In addition, we were curious about whether or not perceived differences in attitudes and behavioral outcomes toward the SARE program existed among the funded groups (researchers, producers and educators). To date, no evaluation had attempted to look at common data across the spectrum of the grant programs.
A thorough evaluation of the North Central SARE program will help decision-makers (e.g., committees and advisory boards, policy makers, lobbyists, USDA), SARE administrators (e.g., program directors, regional directors), educators (e.g., Extension, Natural Resource and Conservation Service, Natural Resources Districts, environmental educators), and potential grant applicants make the most informed decisions regarding programming, funding, and future focus. The purpose of this study was to evaluate the North Central Region’s Sustainable Agricultural Research and Education (SARE) grants operated through the U.S. Department of Agriculture, from their inception through 2002, and to compare responses between producer grant and R & E and PDP grant recipients.
The present study was heavily influenced by the Henry et al. (1998) discussion of emergent realist evaluation (ERE). Emergent realist evaluation is “an approach to evaluation based both on a philosophy called neo-realism and on an integration of lessons drawn by many evaluators from practice, theory, and research (Henry et al., 1998, p. 4). Evaluators embracing ERE recognize that an evaluation of a social program can rarely measure all of the possible consequences that each stakeholder group and the public value. In this study, ERE is an appropriate theoretical base because it recognizes that different individuals and groups assign varying levels of importance to different values, and that social programs should include evaluation of these value positions in context of the program being studied and the populations in which the program exists.
The study employed a decision/accountability-oriented evaluation approach (Stufflebeam, 2001). The purpose of this approach is to provide a knowledge and value base for making and being accountable for decisions that result in developing, delivering, and making informed use of cost-effective services. According to Stufflebeam, the decision/accountability-oriented evaluation approach should be used proactively to improve a program, as well as retroactively to judge its merit and worth. The most important purpose of the decision/accountability-oriented evaluation approach is “not to prove, but to improve" (Stufflebeam, 2001, p. 56).
The decision/accountability-oriented approach is particularly effective for addressing questions such as: Has an appropriate beneficiary population been determined? What beneficiary needs should be addressed? What are the available alternative ways to address these needs and what are the comparative merits and costs? Is the program staff sufficiently qualified and credible? Are the participants effectively carrying out their assignments? Is the program working and should it be revised in any way? Is the program reaching all targeted beneficiaries? Is the program meeting participants’ needs? Is the program better than competing alternatives? Is it sustainable? Is it transportable? (Stufflebeam, 2001).
This study provides an assessment of the SARE funding that allows grant recipients to respond to relevant survey questions, and also to give their own attributions to the SARE experience. In designing the present evaluation, we were especially interested in discovering and uncovering cascade effects, “emergent properties” (Gliessman, 1998; Henry et al., 1998), or unexpected qualities or outcomes that result from trying new practices or designing new systems on the farm, in one’s field of study, or from one’s programming.
Goals of the study were: 1) to evaluate the perceived cascade social, economic and environmental impacts of SARE funding, from inception through 2002; 2) to explore ways in which the SARE experience influenced the knowledge, attitudes, and behaviors of past grant recipients; 3) to understand participants’ perception of a “successful” SARE project; 4) to provide recommendations for strengthening the SARE program and processes; and 5) to compare responses between groups in order to discern differing attitudes and experiences that might influence adoption and engagement in sustainable agricultural activities. This evaluation will help decision-makers understand the perceived strengths and weaknesses of the regional programs from the grant recipients’ perspective. This information is necessary in order to maintain areas of strength, modify areas identified as needing improvement, attract increased participation and funding, and provide guidelines potentially helpful to other SARE regions.
This study uses the earlier den Biggelar and Suvedi (AEE, 1998) evaluation of the producer program as a reference point for developing the evaluation plan. Their comprehensive treatment in the earlier evaluation was invaluable as we took this next step toward more fully understanding the extent to which SARE funding is perceive to lead to increased awareness and adoption of sustainable practices.
References
AEE Center for Evaluative Studies (1998). An Evaluation of the North-Central Region SARE Producer Grant Program. C. den Biggelaar & M. Suvedi (Eds.), East Lansing, MI.
Allen, P. (1993). Food for the Future: Conditions and Contradictions of Sustainability. New York, Wiley & Sons.
Bauer, L. (1998). North Central Region SARE 1998 Annual Report. North Central Region SARE. Lincoln, NE: NCR-SARE.
den Biggelaar, C., & Suvedi, M. (2000). Farmers' definitions, goals, and bottlenecks of sustainable agriculture in the North Central Region. Agriculture and Human Values, 17, 347-358.
Francis, C.A. (2000). How sustainable agriculture programs impact U.S. land grant universities: an update. Sustainable Agriculture Education Workshop, Univ. Wisconsin. Madison, Wisconsin: Center for Integrated Agricultural Systems.
Francis, C. A., Edwards, C., Gerber, J., Harwood, R., Keeney, D., Liebhardt, W., & Liebman, M. (1995). Impact of sustainable agriculture programs on U.S. landgrant universities. Journal of Sustainable Agriculture, 5, 19-33.
Francis, C.A., Flora, C.B., & King, L.D. (1990). Sustainable Agriculture in Temperate Zones. New York: Wiley and Sons.
Gliessman, S.R. (1998). Agroecology: Ecological Processes in Sustainable Agriculture. Chelsea, Michigan: Ann Arbor Press.
Grieshop, J. I., & Raj, A.K. (1990). Are California's farmers headed toward sustainable agriculture? California Agriculture, 46(2): 4-7.
Hall, B., & Kuepper, G. 1999. Making the Transition to Sustainable Farming. Appropriate Technology Transfer for Rural Areas (ATTRA).
Harwood, R. R. (1990). A history of sustainable agriculture. In C. A. Edwards, R. Lal, P. Madden, R. H. Miller, & G. House (Eds.), Sustainable Agricultural Systems. Pp. 3-19. Ankeny, IA: Soil and Water Conservation Society.
Henry, G.T., Julnes, G., & Mark, M.M. (1998). Realist evaluation: An emerging theory in support of practice. San Francisco: Jossey-Bass.
Kroma, M. M., & Flora, C.B. (2001). An assessment of SARE-funded farmer research on sustainable agriculture in the north central U.S. American Journal of Alternative Agriculture, 16(2): 73-80.
Lightfoot, C., & Noble, R. (2001). Tracking the ecological soundness of farming systems: Instruments and indicators. Journal of Sustainable Agriculture, 19(1): 9-30.
MacRae, R. J., Hill, S. B., Henning, J., & Bentley, A. J. (1990). Policies, programs, and regulations to support the transition to sustainable agriculture in Canada. American Journal of Alternative Agriculture, 5(2), 76-92.
Morse, S., McNamara, N., & Acholo, M. (2002). Agricultural sustainability: Comparing external and internal perspectives. Journal of Sustainable Agriculture, 20(4): 29 - 59.
Poincelot, R. P. (2001). Editorial. Journal of Sustainable Agriculture, 17(4): 1-5.
Pretty, J. N. (1998). Supporting policies and practice for scaling up sustainable agriculture. In N. G. Röling, & M.A.E. Wagemakers (Eds.) Facilitating Sustainable Agriculture. Pp. 23-45. New York: Cambridge University Press.
Rog, D.J. & Fournier, D. (1997). Progress and Future Directions in Evaluation: Perspectives on Theory, Practice, and Methods. San Francisco: Jossey-Bass.
Rogers, P.J., Hacsi, T.A., Petrosino, A., & Huebner, T.A. (2000). Program Theory in Evaluation: Challenges and Opportunities. San Francisco: Jossey-Bass.
SARE (1997). Ten Years of SARE.
Stufflebeam, D.L. (2001). Evaluation Models. San Francisco: Jossey-Bass.
1. Assess the local and regional impacts of SARE grants on productivity, economic returns, environmental and social viability of farming in agricultural landscapes and rural communities.
2. Evaluate the cascade (indirect) impacts of SARE grants on research in universities and tests by farmers, and on the current research priorities identified by investigators and farmers.
3. Develop indicators of sustainability as well as return on investment of research funds invested in SARE projects to date, and involve a graduate student in the research process.
Cooperators
Research
The present study examined and explored identifiable and perceived cascade effects of funding among SARE grant participants, from the inception of each respective grant (R & E grants, 1988; Producer grants, 1992; PDP grants, 1994) to the latest reports available at the time of data collection (2002). We were especially interested in those types of effects that may not be gleaned from final project reports. Surveys and interview guides were developed that would allow us to gather both qualitative and quantitative data, consistent with the theoretical (Henry et al., 1998) and procedural (Stufflebeam, 2001) orientations of this study. Permission to collect both survey and audiotaped data was granted by the Institutional Review Board of the University of Nebraska-Lincoln.
To accomplish our goals, we first interviewed stakeholders to focus the evaluation so it would address their most important questions. We then prepared a survey that garnered parallel information from all three grant families. At approximately the same time as the quantitative data were being gathered, qualitative data were collected through audiotaped personal interviews with a subset of former grant recipients from throughout the North Central Region.
Sample
All participants had served as Principal Investigators (PIs) on at least one SARE project granted prior to 2002. All PIs were recruited for participation in the survey, with the following number of respondents: R & E Grants: 51 of 160 possible (32% response rate); Producer Grants: 113 of 170 possible (66% response rate); PDP: 16 of 32 possible (50% response rate). In situations where individuals had served as a PI on both R & E and PDP grants, they were assigned to one grant family or the other and were approached only within context of that specific grant assignment. This bears mentioning because the wording was changed in the various versions of the survey to present each question within the context of that grant family, only. The total possible responses, above, do not reflect the purging of these participants’ names from the database from which they were removed. Of the returned surveys, 170 yielded usable data for our analysis.
Respondents ranged in age from 27 to 76, with a mean age of 50. Ninety percent were male. Experience in their respective fields ranged between two and 49 years. Producers had been farming or ranching an average of 25 years, while researchers reported being in their work for an average of 19 years, and educators, on average, 14 years.
Interviews were conducted with 33 participants (12 Producer Grant, 21 R & E/PDP Grant) at or near their home or office or at a mutual gathering site (e.g., conference, professional meeting). Purposeful selection (Creswell, 2005) was used to achieve representation from all 12 states in the North Central Region, and from a range of project types. In order to allow as much representation as possible, we deliberately did not seek “good examples” of completed projects. Rather, we recruited participants directly off rosters of PIs and who were available for interviews during pre-planned travel opportunities.
Quantitative Data
Instrument. Following four initial interviews to determine direction for the areas to be evaluated (2 researcher, 1 PDP, 1 producer), the survey was critiqued by an advisory committee (2 producers, 2 Extension educators) to determine question value and revise the instrument. The survey that resulted contained four sections. The self-report questions were general in nature, rather then specific, measured outcomes, such as dollar amounts or specific proof of environmental impacts. The demographics section included indicators that helped identify participants’ engagement in sustainable behaviors, areas of study, or educational programming.
Thirteen to seventeen items (depending on which grant family) asked about SARE impacts (e.g., Because of my involvement in SARE funding, I have become involved in a formal sustainable ag organization or network. My SARE-funded grant(s) has led to further non-SARE funding for me.) Five items were included to assess respondents’ opinions as to what determines a “successful” SARE project (e.g., The success of SARE-funded projects should be judged primarily on positive financial impacts to producers. The most important outcome of a SARE-funded project is that a producer learn something from it.) Nine items measured opinions about suggested changes or adaptations to the SARE program or process (e.g., I needed more SARE technical assistance to help support me during the grant period. I prefer limiting grant recipients to those who have proven that they are committed to SA practices and research.). Electronic versions provided room for comments at the end of each section. Tables and their related survey questions are included in the results section of this report.
Approximately one week before launching the surveys, a letter from the second author was distributed (electronically to R & E and PDP; first-class mail to Producers), announcing the launch of this study and encouraging participation. Requests to not participate resulting from this pre-survey invitation were acknowledged and their names purged from the database prior to data collection.
Procedure: R& E and PDP. Electronic surveys were distributed to R & E and PDP participants via email messages, which contained an overview of the study’s purpose and anticipated completion time, informed consent, and a link to the web-based surveys. In a few cases (5 R & E, 3 PDP), persons who had initially been contacted electronically requested and completed hardcopy versions of the survey. Only one follow-up reminder, which also contained the web link to the survey, was e-mailed one week after the initial launch of the survey. Some correspondence was initiated by survey recipients over the course of the three weeks the survey remained live. All persons e-mailing comments received personal responses from the investigators.
Procedure: Producers. Packets containing the Producer Grant surveys were distributed by first-class mail. Packets included a cover letter explaining the study, an invitation to participate, and informed consent statements. Packets also contained the survey and self-addressed stamped envelope. Incentives were included in the form of a brief, humorous, illustrated story of the first author’s childhood experiences on the farm and an attached 50-cent-piece. The coins were included at the personal expense of the first author, and were not paid for by SARE or any other public funds. Reminder postcards were sent to non-respondents approximately three weeks after the initial mailing. Two weeks after the postcard reminders, a final follow-up request and second survey were sent to non-respondents.
Grouping adjustment. Some questions on the survey and within the interview protocol were changed to allow wording most appropriate for the respondent’s particular involvement with the SARE program. Perhaps because many PDP grants were received by active researchers and representatives of non-government organizations, analysis revealed little difference between R & E and PDP sample populations. For the purpose of this report, these two sample sets are combined and reported as “R & E,” and findings cross-tabulated to determine significant differences in responses between two groups, Producer and R & E.
Qualitative data
Thirty-three personal interviews were conducted in locations that were mutually convenient to the participants and researchers. All interviewees signed an Informed Consent document, which remains on file with this study’s principal investigator for two years beyond the conclusion of this study.
The semi-structured interviews designed to address the first two research objectives (assess local and regional impacts …; evaluate cascade impacts on research universities and tests by farmers and on current research priorities….) explored the questions: 1) How did your original SARE grant work out related to economic, social, and/or environmental impacts? 2) Did your SARE experience lead to other impacts? If so, what were they? 3) How do you define success in terms of SARE grant experiences? 4) What suggestions do you have for improving the SARE program or procedures?
The first two authors conducted all of the interviews, with a few early sessions conducted in tandem until the interview protocol was finalized. Interviews were audiotaped and transcribed, verbatim, and several (8) of the transcripts were returned to interviewees for member checking and clarification of inaudible phrases. Responses were used here to clarify or interpret some of the survey results in this report.
This section provides an analysis of the data by research question. Following a brief discussion of the quantitative results, a representation of qualitative responses related to that question helps add to an understanding of the responses, in context.
Impacts of Funding
Subjects were asked to respond to 16 questions designed to assess the impacts of their SARE funding. Questions were clustered for analysis on five evaluation dimensions: financial, social, knowledge, affect/attitude, and behaviors. Independent t-tests comparing group means revealed favorable, but near-average, impacts (mean range from 4.1 to 5.3 on a 1 [“entirely disagree”] -to-7 [entirely agree”] Likert scale) across both groups. Only financial impacts demonstrated a significant difference (p<.028) between groups, with producers indicating significantly less financial impact from funded projects than the R & E group (see Table 1).
QUESTIONS: Financial Impacts (7-point Likert scale)
1. My SARE-funded grant(s) has led to further, non-SARE funding for me.
2. My SARE-funded project demonstrated favorable economic viability.
QUESTIONS: Social Impacts (7-point Likert scale)
1. My network of others involved in SA has expanded as a result of my being involved in a SARE-funded project.
2. Because of my SARE project(s), I know more [producers] [researchers] [educators] who are engaged in sustainable agriculture.
3. Because of my involvement in SARE funding, I have become involved in a formal sustainable ag organization or network.
QUESTIONS: Knowledge Impacts (7-point Likert scale)
1. I have a better understanding of what works and what doesn’t work as a result of having worked on a SARE-funded project.
2. My SARE-funded grant helped me learn more about SA options and practices.
3. By participating in a SARE-funded project, I learned more about how to put a financial value on sustainability.
QUESTIONS: Attitudinal Impacts (7-point Likert scale)
1. I feel more strongly committed to sustainable agriculture as a result of having worked on a SARE-funded project.
2. I support principles of sustainable agriculture even when I don’t have grant funding to study them. [reverse-coded during analysis]
3. Being involved in a SARE project was worth it to me, even if I didn’t see a great financial reward.
QUESTIONS: Behavioral Impacts (7-point Likert scale)
1. My SARE-funded grant(s) has led to my involvement in additional sustainable ag-oriented activities.
2. I would have conducted the SA research even if SARE had not provided the funding. [reverse-coded during analysis]
Qualitative data related to the impacts of funding suggest that the difference in financial impacts may be because having a SARE-funded project served as a catalyst for additional funding more often among R & E respondents than among producers. Forty-eight percent of the researchers and educators had received more than one SARE grant, compared to twenty-five percent of the producer grant principal investigators. Interviewed researchers could easily discuss ways in which their SARE-funded project led them to new economic opportunities. Some of these opportunities lay in additional grants, while others were seen in employment opportunities that emerged as a result of the SARE grant experience.
In one case, a social science researcher had become disillusioned with SARE’s lack of commitment to social science research, and had left the field. An agronomy-based researcher’s work had drifted from examining environmental concerns because there were such limited financial resources available to sustain studies as required in his area of expertise. In most cases, however, researchers continued to include some involvement in studying their area of expertise in terms of sustainability, although most had those projects going as a minor portion of their research agenda.
Despite the imbalance in repeat funding, producer interviews typically revealed identifiable examples of economic impact, as can be seen in this producer’s explanation:
“The biggest [economic impact was] qualifying for that [contract hog buyer] market. That’s been a godsend to us actually, here. Um, we started selling to them in ’98 [as one of the few] and … there’s over 200 farmers selling to them now…. So in ‘98, they bought half of our pigs and we went through that price debacle like everybody else did. But the half of our pigs that we were selling to [the contract buyer] really helped us pull through there. They have a 40-cent floor on the pigs. Otherwise, it’s a 6-cent premium. [I]t helps me sleep better at night just knowing that we got a secure price. And the grants that we get from SARE help us qualify our buildings [for that buyer]….”
While producers often made comments, such as, “[W]ithout the grant, it wouldn’t have been feasible,” many participants acknowledged that the funding “didn’t make the make-or-break difference in my bottom-line.” For producers, funding was often a one-time “contribution” to their operation that allowed the producer to try something new or expand something already initiated, regardless of whether or not the funding was significant enough for them to see it as a bona fide financial boon. “We still owe money. But I/we haven’t been as traditional as a lot of people…. For instance, a couple years we didn’t run cow/calf, we ran yearlings…” This producer went on to say, “We actually made money when our neighbors were losing money.” So, whereas the SARE funds themselves did not necessarily contribute a huge sum of money, the impact was, at least in the cases we interviewed, perceived to result in an economic value that far exceeded the dollars received.
Cascade effects were often discussed in terms of getting the opportunity (“money,” “reason,” “excuse”) to take a risk to try something new. “… [T]he thing it got us to do was to do some of this marketing work that we’ve always talked about, but didn’t have the time/ --or didn’t take the time and effort -- to do. And so it … basically got us started….”
Frequent statements were offered that expressed considerable appreciation for other non-financial contributions provided by the funding. “It helped me see a new way of doing things” was the point of several such comments. Another knowledge-based impact involved the opportunity to try an older approach they had heard about, but had not been able to explore until they received funding. "Oh, the technology’s been around for hundreds of years…. My dad always talked about a farm that [used the idea]. So I knew that it could do it. I just never had seen one work until I had my own.” In general, producers’ comments referred to the reduction of risk so they could afford to experiment with an idea. “It’s a learning process and we didn’t do very good that first year [before getting funded] when we tried a few things ourselves.”
In describing examples of cascade effects, interviewees identified ways in which their projects taught them more about what nature could do to help solve a production dilemma. “[Y]’know, grass was grass when we started this whole thing.…[A] lot of our land looks a lot better now – more cover for wildlife and just a more natural, esthetic look or beauty.”
Many participants referred to cascade impacts that included meeting new people. “…[I]n our group it’s mostly veterans and we have a couple of young guys and we really encourage them….” “[Our organization] was able to convene, um, a very diverse collaborative of university, non-profit, private, agent steering committee members and the farmers.” “[W]e were able … even the other four producers on the grant – we … didn’t really know them well, and so it gave us a chance to work more with them.” “Some of our neighbors would come over and … look at the project….” “We must have had 40 or 50 farmers there that first field day and they all wanted to know about this market.” “Plus, by going to some different meetings, we’ve met some other people …. which has been kind of fun. Every once in a while, we talk to ‘em. Just sort of touch base…. It’s really interesting just to see how other people’s lives are progressing and the different things they’re doing.”
In several interviews, participants credited their SARE involvement with positioning them in the eyes of others as a leader in sustainable agriculture. “…[B]y the fact that I already had a project going with SARE it kind of allowed me to build on that with [a target group interested in knowing more].”
Positive results from SARE projects had helped participants deal with the social issue of peer pressure (“what the neighbors think”) when changing their operation to include less-conventional practices. “Ya, we don’t put our [grazed corn field] out by the road [chuckles].” One farmer, whose project involved year-round cover crops, explained:
“[The neighbors] used to ask if I’m still doing that [using cover crops]. And now, they know that I am…. Ya, they didn’t think it’d last… Of course, most of the comment is, ‘Well, we know when we’ve come around your territory, cause it’s always green. You have something growing there all the time.’”
One interviewed researcher, despite having had several recent pre-proposals turned down, had found ways to continue the experiment he had started following earlier LISA and SARE projects. His now-20-year study has yielded results that have changed the attitudes of stakeholders so he is now allowed to put more pressure on the test acres than originally allowed. The researcher’s extension of his SARE work has led to a “side project” that has been able to examine additional issues and continue over an extended period of time. “And so, this [LISA and SARE funding] started this project that has just continued on and on and on, and led to other things.”
Another researcher used the illustration of a producers’ evaluation tool to describe the way their SARE-funded assessment instrument motivated them to continue their work: “We no longer had any official ‘funding’…, but we certainly had a groundswell of interests that birthed that project in the first place. That’s the thing, isn’t it? Your funding goes away, but your ideas and inspiration don’t.”
Definition of “successful” SARE studies
Five survey questions were asked to measure what participants considered as identifiers of “success” of any given SARE project (see Table 2). Questions were developed to examine the commitment to actual producer-level outcomes, compared to respondents’ own, discipline-specific outcome as a researcher or educator. One item contained confounding language between versions, so was discarded from this analysis. Again using a 7-point Likert scale, participants were asked to what extent they agreed (7) or disagreed (1) with each statement.
The statement, “The success of SARE-funded projects should be judged primarily on positive financial impacts to producers,” resulted in relative consensus, with 59% indicating they moderately (3) to strongly (1) disagreed with this statement.
Both groups responded in a similar way to the statement, I believe a SARE project has been successful when a producer continues the activity even once the funding has ended. Considerable consensus was demonstrated in the responses, where most people (88%) indicated somewhat agree (5) to strongly agree (7).
Respondents were asked three questions that would help determine the most important outcome of a SARE grant – whether that would be the respondent making a commitment to studying SA issues, the study adding to the SA literature, or a producer learning something from the grant (regardless of what grant type). Regarding the statement, The most important outcome of a [participant-relevant] grant is that [farmers/ranchers, researchers, educators] become more committed to studying SA issues, the wording was modified for each group to strengthen the personal relevance to that respondent’s particular application. A Mann-Whitney U test was conducted and the results of the test were significant, z=-4.576, p<.000. Researchers and educators (mean = 4.83) provided a greater range of responses from moderately disagree (2) (8%) to strongly agree (7) (11%). Producers, however, indicated a stronger consensus of opinion about this question with 82% of their responses indicating agreement to strong agreement (5 to 7 on the survey), with a mean of 5.80; in fact 35.5% of producers strongly agreed (7) with this statement.
A final item was included to assess respondents’ commitment to producer-level learning, specifically, with the statement, The most important outcome of a SARE-funded project is that a producer learn something from it. Between-group disparity in responses (p<.06) was near the p<.05 threshold for significance. No producers moderately (2) or strongly (1) disagreed with this statement, while 87% of their responses indicated they somewhat (5) to strongly (7) agreed and resulted in a mean of 5.90. Researcher and educator responses, however, fell across the entire 7-point range, with 82% of responses falling within the same 5-to-7 range and demonstrating a mean of 5.52. (See Table 2.)
Successful Project statements:
SS1. The success of SARE-funded projects should be judged primarily on positive financial impacts to producers.
SS2. I believe a SARE project has been successful when a producer continues the activity even once the funding has ended.
SS3. The most important outcome of a [participant-specific] grant is that a [researcher/educator/producer] becomes more committed to studying SA issues.
SS5. The most important outcome of a SARE-funded project is that a producer learn something from it.
Qualitative data were supportive of these findings – especially in terms of the project purpose of building knowledge in producers. Interviewers consistently heard comments such as, “A producer should learn something from it,” or “It should make you see different ways to do things – show you different options.” Several R & E interviewees indicated a “hope” that projects would lead to more-sustainable practices, and several expressed a need for PDP grants, in particular, to lead to more recruitment of advocates at the researcher and educator level. “We still have lots of educators suggesting pesticides as the first method of pest control,” one PDP respondent said. “I wish there was some way we could get more SARE funding to influence experts to recommend other options.” One outlying response came from a producer participant who identified his key indicator of a project’s success simply as, “It’s gotta be fun! If you can’t have fun, you shouldn’t be doing it.” While this, admittedly, did not appear to be a recurring opinion, many producers did allude to their appreciation for the opportunity to “learn something new” and “keep life interesting” as a result of their funded project.
Suggestions for changes or adaptations
A set of 10 questions allowed respondents to indicate their level of support for several SARE program features or procedures. These questions reflected areas of concern identified through qualitative data including written project reports and preliminary personal interviews conducted prior to data collection.
Results showed that 79% of all respondents felt they did not need more technical assistance from SARE during the grant period. Eighty one percent of all respondents indicated neutral-to-strong agreement that the pre-proposal/proposal process works well. A significant difference (p<.003) was demonstrated regarding the value of the SARE reports, with producers seeing greater value in these products than researchers. Because submitting final reports is more unique in the world of producers than that of researchers or educators, the statement, the reporting process for completed SARE projects is reasonable, was presented only on the Producer version of the survey. Eighty five percent of respondents somewhat agreed (5) to strongly agreed (7) with this statement.
Both groups appeared relatively neutral about the ease of access to SARE project results, and several producers offered the opinion that they would like some system to be devised that would allow grant recipients to have better access to each other and to final reports. The two groups were significantly different (p<.000), with producers more favorable about the two-year grant duration and researchers more strongly in favor of projects running for longer than two years.
Producers were significantly (p<.002) more in favor of funding more niche production research than were R & E respondents, while the two groups demonstrated strong consensus that they would like to see SARE fund more projects that will encourage conventional producers to experiment with sustainable practices. Regarding this latter statement, 81% of all responses received a score of 5 to 7. In an attempt to understand these two statements further, the R & E and PDP versions of the surveys included an additional question asking about limiting grant recipients to those who have already proven they are committed to sustainable agricultural practices and research. Sixty-eight percent of respondents disagreed (3) to strongly disagreed (1) with this statement.
Suggested Changes statements:
C1. I’d like SARE to support more niche production research.
C2. I needed more SARE technical assistance to help support me during the grant period.
C3. Results of other SARE-funded projects are easy to access.
C4. Two years is about the right amount of time to run projects and evaluate outcomes of SA projects.
C5. The pre-proposal/proposal process works well.
C6. Final SARE reports are a valuable part of the process.
C7. I’d like to see SARE fund more projects that will encourage conventional producers to experiment with sustainable practices.
C8. I prefer limiting grant recipients to those who have proven that they are committed to sustainable agricultural practices and research.
C9. SARE projects should run for more than two years.
C10. The reporting process for completed SARE projects is reasonable.
Timeliness of receiving the producer grant funds emerged as an issue from interviews with both producers and researchers. “One of the things I don’t like about the way the process is, is you get some of the money, but it’s a long time before you get the rest of the money and.… [I]t complicates cash flow, especially in small businesses where you don’t need complications in cash flow.”
Researchers interviewed tended to be the most critical of both the procedures and the priorities and distribution of the grants, with arguments made both for and against supporting established advocates with additional grants. Overall, however, participants seemed well aware of the dilemma SARE decision-makers face when trying to decide between continuing support for proven advocates and recruiting persons new to the field.
“[I]t’s been a hard area to get grant funding for,” was a common expression about the reality that exists for seasoned veterans. “Some of the traditional granting groups were the ones that want you to spend money, and they give you grant money to develop ways to encourage people to spend more money.” “[Granting to already-identified advocates] really limits the number of dollars available for new researchers who are showing interest, but on the other hand, there aren’t many funding sources available for sustainable ag research. So if they can’t get this money, where are they going to turn to further their work?” “If we had more money available, what would be best to support both. Maybe there could be some kind of ‘invitation’ grants without having to shut down the ongoing research of the ‘old guard.’”
Displeasure at having several recent pre-proposals turned down was apparent in the voice of this researcher who had received former SARE grants:
“I’ve kind of lost some enthusiasm…. See, sustainable agriculture, almost by definition … is long-term. But for a funding agency like this [shifts his approach]. You see, if we did two years of it, [they say], ‘Well, they’ve done that research.’ They [SARE] can’t see what might be the next step….”
A few researchers discussed another dilemma -- whether SARE should support niche production research or “move conventional research more toward the sustainability end.” No one who addressed this issue offered an absolute solution, but rather, acknowledged it as an on-going dilemma:
“On the one hand, I don’t much support niche research – I don’t think that’s ever going to be where farmers and those guys can make a real living. But on the other hand, some of our most innovative ideas are coming from people who haven’t grown up in the conventional ag world. They’re making it work for them in niche products, so it’s a problem.”
This study revealed that Sustainable Agricultural Research and Education Programs do lead to cascade impacts that are easily identified by grant recipients. While no one indicated that their SARE funding provided a significant portion of their research, education, or production dollars, participants were able to easily discuss ways in which their funding had led to further economic returns (e.g., additional grants, higher prices, financial savings, greater yields).
We conclude from this research project that it is extremely difficult to assess the economic returns on investment in research through the SARE program by using conventional, quantitative economic analysis, alone. Although it is obvious that there are economic impacts for the farmers involved in specific initiatives that lead to changes in their operations, and that the long-term economic effects of the research conducted by universities are vital to agriculture, it is close to impossible to pin down the unique financial contributions of a given project. When there are studies of rotational grazing systems, field days and tours supplemented by brochures or technical articles, the interviews in this study help us remain somewhat confident that there is a large economic impact on those who are touched by the new information from the project. Unlike counting acres planted to a new wheat variety and measuring and comparing yield with input cost data, to put a monetary value on this contribution would require economic science research dedicated specifically to this objective.
As for environmental impacts, this study supports the extensive body of literature that exists concerning ways in which sustainable agriculture reduces inputs and reinstates environmental ecosystems. This experience was confirmed consistently throughout the interviews with participants of all three grant families.
Perhaps the most important contribution of this study is to strengthen what is known about the social impacts of this funding. When participants speak in terms of having their neighbors inquire, tease, or challenge their unconventional ways of doing things, they know the message is being spread. Educational experiences such as field days are consistently identified as turning points for curious producers to gain insight into alternative ways to use land or raise animals. And researchers, once engaged, often find innovative ways to maintain their research plots, even beyond their original funding, often because they see something emerging and becoming intellectually engaged at a personal level.
Economic Analysis
Evaluation is an important part of any funded program, and in this way, the present study is considered an efficient use of public funds. It is a program that offers a thorough description of its methods, the instrument used, and the analysis conducted. This allows successive studies to build on, replicate, or refine the study and its instruments and methods used. The study has resulted in a number of publications, and offers additional publication opportunities, within ethical boundaries, because of the depth and breadth of the data collected.
Farmer Adoption
As stated, above, this study did not serve a purpose of impacting farmer adoption. As addressed in the study implications, above, however, this study will indirectly impact farmer adoption by informing SARE decision-makers so they can adapt program structure and activities in ways that will better accommodate farmer adoption.
Results from this study should help program decision-makers answer a number of questions appropriate from the Decision/Accountability-oriented approach (Stufflebeam, 2001) used here. Stakeholders involved with supporting and expanding the SARE program should use the results of this study to strengthen and improve the program so it can continue to make an impact on producers – directly and indirectly -- and in return, impact the sustainability of our food and fiber systems, worldwide. Results from this study seem to support the continued promotion and expansion of the SARE program.
Direction must be given that will resolve the dilemma between continuing the work of advocates with a proven track record and encouraging conventional farmers, researchers, and educators new to sustainable agriculture. Educators and others who provide technical support for the SARE producer grants should be aware of the social pressures on mainstream producers and should help design strategies that will allow experimentation to be done in ways that will respect their social position within their given community. Additional attention should be given to technology transfer and the ways in which networks among past grant recipients can be improved.
Impacts on farmer adoption will be further strengthened when the results of this study are combined with a partner study by these authors that examined motivation sources among the participants in this study, and comparing those results with parallel non-SARE populations.
Educational & Outreach Activities
Participation Summary:
To date, we have produced a number of academic publications, with additional publications being planned. In this way, the results of this study will continue to reach out to inform a variety of applied audiences. Following is a list of publications and presentations, as of the submission of this report:
Dissertation
Trout, S.K. (2004). Motivation as an antecedent to positive environmental behaviors of agricultural leaders. Dissertation Abstracts International.
Publications
Trout, S.K., Barbuto, J.E., Jr., & Francis, C.A. (in press). Comparing motivation sources in two agricultural populations. Journal of Sustainable Agriculture.
Trout, S.K., Francis, C.A., & Barbuto, J.E., Jr. (in press). Evaluation and Perceived Impacts of the North-Central Region SARE Grants, 1988 – 2002. Journal of Sustainable Agriculture.
Book Chapter
Trout, S.K., Francis, C.A., & Barbuto, J.E., Jr. (in press). Motivation for the conversion to sustainable agricultural system. In C. Francis, R. Poincelot, and G. Bird (Eds.), A New Social Contract: Developing and Extending Sustainable Agriculture. Ch. 14.
Project Outcomes
Areas needing additional study
Study Limitations
This study was limited by including only a population of principal investigators. Additional perspectives would provide important data that could allow for a more comprehensive understanding of ways in which SARE funding does and does not lead to valuable cascade effects. Additional questions can strengthen future studies. Also, developing or identifying a more stringent assessment tool regarding economic impacts will help satisfy policymakers’ need to have even stronger quantitative justification for the continuation and possible expansion of the program.
Areas Needing Additional Study
Admittedly, the design of this evaluation targeted friendly critics because it engaged feedback only from persons who had benefited directly from the funding. Additional studies are needed to further explore SARE impacts from additional stakeholder points of view. Additional effort should be given to identify the most useful questions to ask and to continue to ask them in ways that can be both generalized to a population and understood within context. By continuing this evaluation process, stakeholders will be better informed so they might proactively improve the SARE program, as they retroactively judge its merit and worth. In this way, SARE funding can be expanded to make even greater impacts on U.S. food and fiber production systems and the persons who depend on and believe in them.