Welcome

Brochure

Current Projects

Research Reports
and Bulletins

Newsletters

Vol.1 No.1
Vol.1 No.2
Vol.1 No.3
Vol.1 No.4
Vol.2 No.1
Vol.2 No.2
Vol.2 No.3
Vol.2 No.4
Vol.3 No.1
Vol.3 No.2
Vol.3 No.3
Vol.3 No.4

Newsletter--Volume 3, Number 1

Now that Congress has mandated that commodity promotion programs be independently evaluated "no less than once every five years," it is useful to review the principles that govern quality evaluation. Evaluation, as specified in the new farm bill, means determining whether the promotion program is indeed effective at maintaining or increasing the demand for the agricultural commodity and thereby improving farm income. Quality is defined as an evaluation that meets accepted standards of scientific rigor while at the same time addressing the question "does promotion pay?" in a manner consistent with economic theory.

Because cost will be a major consideration in meeting the evaluation requirement, especially for smaller programs with limited budgets, I will endeavor to identify practical things commodity promotion organizations can do to facilitate the evaluation process and thereby lower costs. Chief among these is getting an early start on the evaluation process. Although the first evaluation is not due for five years, putting it off until the last minute is the surest way to compromise quality and increase cost.

Most evaluations take at least six months from the time the researcher gets the data until a final report can be given. Add to this the time needed to develop an evaluation proposal, select a researcher, and have the completed evaluation critiqued, and the time frame lengthens to one year at the very least. And this assumes that the necessary data are available and in the proper form for economic analysis. Unfortunately, this is rarely the case, so a portion of this article is devoted to describing the data needs of researchers.

The principles and data needs described in this article are distilled from 16 years of personal experience in evaluating commodity promotion programs in the United States, Canada, Korea, and Australia. This experience also includes working with a very small commodity promotion organization, namely catfish, so I am cognizant of the unique problems faced by smaller organizations.

EVALUATION PRINCIPLES
The three pillars of sound evaluation are the scientific method, reproducibility, and peer review. Eliminate any pillar, and the edifice upon which evaluation rests is undermined. Lest the pillars sound abstract or unworkable, I will describe each briefly, using examples to illustrate what producers or promotion board managers should look for when assessing an evaluation proposal.

The Scientific Method. The scientific method consists of four interrelated components: theory, hypotheses, data, and empirical testing. Most promotion evaluations make ample use of data and empirical tests; the real problem is the underutilization of economic theory and the consequent overly-narrow framing (or misframing) of hypotheses. For example, we know from economic theory that program profitability depends on a few key structural elements as follows: supply response, cross-commodity substitution, middlemen markup behavior, processor technology, market structure (e.g., imperfect competition beyond the farm gate), farm programs, tax shifting (the incidence of promotion checkoff), and trade status (e.g., whether promotion occurs in a large or small open-economy setting). Yet few studies address these structural elements in any systematic fashion, and most ignore all but one or two.

Theory is essential for sound evaluation because it directs attention to appropriate variables to be included in the analysis, it tells how the variables are related to one another, and it informs the data collection process. A telltale sign of whether an evaluation proposal gives adequate attention to theory is the sophistication of the proposal's economic model. An economic model consisting of only demand equations, for example, is almost surely to be untenable from the standpoint of economic theory. Any analysis flowing from such a model is likely to be misleading and thus of questionable value because key structural elements (e.g., supply response and markup behavior) are ignored.

Reproducibility. The second principle of sound evaluation is reproducibility. Reproducibility means that the methods used to arrive at a study's conclusions must be clearly spelled out so that another researcher competent in the field can replicate the study without undue guesswork. Reproducibility also means the data and programs used in the analysis should be made available, if not in the study's appendix, then at least on a diskette so other researchers have access to the information.

Reproducibility is important because advertising effects are known to be fragile. That is, estimated advertising elasticities, a key component of any economic evaluation of advertising impact, tend to be sensitive to model specification, estimation procedure, measurement error, and sample period. For example, in a recently completed study, we found that adding just seven observations to the sample period altered the estimated coefficients for beef advertising from a positive number to zero (Kinnucan, Xiao, Hsia, and Jackson). Thus, the ability to replicate studies is essential in making sure results are trustworthy.

Peer Review. The third principle of sound evaluation is peer review. Promotion evaluation is complex. And each evaluation presents unique challenges requiring creative solutions. For these reasons, it is essential that others with expertise in promotion evaluation be given the opportunity to critique the research, to raise questions, and to suggest changes that would strengthen the analysis. Peer review, properly done, is a win-win situation. The evaluator wins because peer-reviewed research generally has a higher level of rigor and scholarship, which reflects well on the researcher. The promotion board wins because peer-reviewed research is more likely to be accepted by the scientific community, which means it will carry greater weight in policy discussions, in communications with producers, and as evidence in legal proceedings. The best possible peer review is to have the study published in a reputable journal, such as the American Journal of Agricultural Economics.

Peer review can extend to the proposal phase as well. By enlisting the proper expertise in the evaluation's design phase, costly mistakes can often be avoided and a better proposal can be tendered to the research community. As indicated earlier, a common pitfall of many evaluations is overinvestment in data analysis and underinvestment in economic theory. An ex ante peer review, properly done and with the right expertise, can prevent this problem.

DATA NEEDS
Evaluations that adhere to the foregoing principles are likely to be expensive, so what can be done to keep costs within reasonable limits? Besides getting an early start on the evaluation process, probably the most important thing a commodity promotion board can do to lower evaluation costs is to focus on data. Data have to be in a particular form if they are to be amenable to economic analysis. By paying advance attention to the data needs of researchers, commodity boards can save time and money when results are needed. What follows is a suggested format for making data available to researchers. The format is based on my own experience working with commodity boards, in addition to suggestions provided by research colleagues.

Data reporting has three critical elements: accuracy, periodicity, and categorization. Accuracy means the data must reflect actual program activity. For example, in reporting advertising expenditures, it is essential that actual outlays, not budgeted or planned amounts, be given. Accuracy also means that sufficient detail be given so researchers can interpret the numbers intelligently. For example, instances of significant change in target audiences, media strategy, creative appeals, and so forth should be reported along with the numbers. This permits researchers to test whether or not qualitative or strategic aspects of the campaign have any detectable effects on response coefficients.

Periodicity refers to reporting interval. The best reporting interval, at least for promotion evaluation, is monthly. Monthly data are best because advertising lag structures can be estimated more accurately, degrees of freedom are less of a problem, and the analysis can be carried out over a shorter time frame. For example, if only annual data were available, the researcher would need to go back 20-30 years to get sufficient data to estimate the model. A model estimated over such a long time period is likely to suffer from specification error, unless the industry is stagnant or advertising effects are stable, which is not likely to be the case. With monthly data, a time frame of four to five years is usually sufficient to estimate advertising responses.

Categorization refers to how the data are to be separated. A desirable categorization is much harder to define than a desirable periodicity. A lot will depend on the complexity of the board's activities. For a relatively simple program like catfish, where the major emphasis is on media advertising in the domestic market, it would be sufficient to report advertising expenditures per month broken down by medium (television, radio, print). For a highly complex program like dairy, where extensive investments are made in research and nutrition education as well as promotion in domestic and export markets, a much more detailed categorization of expenditures would be required.

An example categorization that might be suitable for the "average" commodity promotion organization is given in Exhibit 1. In this example, the commodity organization would provide monthly expenditure data for six program activities: media advertising, merchandising (e.g., point-of-purchase promotions), public relations (including consumer information programs), research on new product development and production (or agronomic research), and administration. In reporting the data, care should be taken that the numbers add up across the categories to assure accuracy and completeness.

The categories provided in Exhibit 1 (note: please contact us to get a copy of this chart), of course, could be refined or expanded as needed to more clearly represent the organization's total promotion program. For example, if the board participates in USDA's export promotion programs, columns would need to be added to indicate target markets and expenditures in each market, including monies provided by government and foreign third-party cooperators. In addition, if the commodity organization has price or quantity data pertaining to the commodity, this information could also be included in the data matrix. In general, however, the researcher can obtain this information from public sources.

Returning to my earlier point about the need to start the evaluation process early, note that carrying out an econometric analysis based on the data given in Exhibit 1, the researcher would need at least four years of observations. Thus, to meet the time line set by Congress, boards need to begin thinking about data collection procedures now. Delay will increase cost and lower quality.

CONCLUDING COMMENTS
A basic theme of this article is that sound evaluation follows certain immutable principles. Armed with this knowledge, promotion boards are in the best position to serve the interest of producers in terms of providing accurate and reliable information about program performance. There is nothing less relevant, or more costly, than a poor evaluation, i.e., one that does not address the question (does promotion pay?) or that attempts to cut corners.

A side benefit of making data available is that it opens up the possibility for evaluation to be undertaken at a university at little or no cost to the promotion program, as has been the case for catfish. University researchers have an incentive to publish, promotion evaluation lends itself to the "tools-of-the-trade," and academic interest in the subject matter is growing. A drawback, of course, to making data generally available is that the commodity organization loses control of the evaluation process. Still, if objective, low-cost, sound evaluation is desired, the single best thing a commodity promotion organization can do is publicize the relevant data.

REFERENCES
Kinnucan, H.W., H. Xiao, C.-J. Hsia, and J.D. Jackson. "Effects of Health Information and Generic Advertising on U.S. Meat Demand." American Journal of Agricultural Economics 79(1997): forthcoming.

Editor's Notes
Jennifer L. Ferrero

Those of you in the commodity promotion business will want to store this issue of the NICPRE Quarterly for future reference because of Henry Kinnucan's excellent overview of the data needs required for economic evaluation of promotion programs.

Bill Allen, the president of The Catfish Institute, offers a perspective on evaluating limited budget promotion organizations in the Manager's Viewpoint. Although his group is not required by law to evaluate promotion program results, he explains why monitoring promotion allocation is so important for TCI and other limited budget organizations.

In his Director's Column, Harry Kaiser briefly describes how current allocations of checkoff investment and various promotion expenditures can be reallocated for optimal benefit to promotion organizations.

This issue of the Quarterly marks my first as official newsletter editor--I'm confident you'll find the same high level of quality as that of our previous two years in this and future issues. I invite you to contact us with any comments or suggestions you may have about the newsletter.

Manager's Viewpoint
Bill Allen, President
The Catfish Institute

Evaluating Results of a Limited Budget Promotion Organization

The Catfish Institute (TCI) qualifies as a “limited budget promotion organization” with total annual revenues varying from around $2 million when TCI was organized in 1986 to $3.4 million in 1996. The goals of The Catfish Institute are to increase awareness, improve attitudes, and increase demand for U.S. farm-raised catfish. TCI differs from most U.S. commodity promotion organizations in several ways. First and most notably, we are an independent, nonprofit corporation, not created or regulated by any sort of federal or state mandated “checkoff” legislation.

The Catfish Institute executes its various programs through arrangements with agencies and independent contractors with annual review and renewal options, rather than through staffing. We have found this to be a very satisfactory method of retaining the specific expertise that we need to address vastly different target audiences and still retain the flexibility to change targets and tactics without upsetting staff positions. When we start “evaluating,” we are essentially evaluating agency or contractor performance. We routinely build a number of specific measurement vehicles into each agency or contractor budget. My job is that of a “manager,” overseeing and coordinating the activities of the various agencies and contractors while reporting directly to a board of directors appointed by member feed mills.

While no specific program of evaluation is required of The Catfish Institute, evaluation is probably more important to us than to other commodity organizations. We must constantly evaluate the results of our expenditures and report on our activities to various industry groups in order to keep the support of the catfish farmers and processors in addition to the voluntary financial backing of member feed mills.

The Catfish Institute has been generally viewed as “successful” by various industry groups and outside observers. Total round weight sales volume of catfish from farmers to processors has grown along with growth revenues at the processor level. Best of all, this growth has occurred with relatively stable pricing to farmers and processors compared to the recent explosive growth in U.S. salmon consumption accompanied by disastrous pricing consequences for producers.

It’s my personal opinion that generic promotional organizations usually get too much credit when things are good and likewise take too much of blame when their industries are experiencing problems. Realistically, an organization like The Catfish Institute, spending from 0.5 percent to 1.0 percent of industry gross revenues on promotion, cannot be expected to dramatically impact short-term supply and demand cycles or pricing. What we can do is develop carefully targeted programs to help allocate limited revenues to activities that will improve consumer attitudes toward the product over the long term and create a more favorable selling environment for our processors. Our current spending mix is approximately 65 percent consumer advertising; 15 percent public relations activities; 7 percent targeted food service and culinary school promotion and training programs; 7 percent targeted export promotions to Germany (net of USDA/MAP funds); and 6 percent overhead and administrative costs.

For advertising, we conduct consumer and trade focus group interviews from time to time to see how various audiences view our messages and what they take home from our advertisements. We also conduct “attitude, awareness, and usage surveys” by telephone on a biannual basis to track demographics of catfish users, measure changes in awareness and usage of the product, and monitor their agreement or disagreement with certain specific product attributes. We have also used coupons extensively in print ads to provide performance comparisons between various publications within a media schedule. One final measurement for advertising is econometric modeling. This may be the best possible way to isolate the effectiveness of various media advertising tactics when sufficient data is available to evaluate.

For public relations, the most tangible measurement comes from clipping services. We spend lots of time and money each year disseminating information to various food and news editors, and this is the best way to determine whether they use or discard it. We go a step beyond clipping services by sending our clips to an independent evaluation agency to audit them based on the total number of catfish stories or spots, estimated total audience, and estimated dollar value of equivalent advertising. More importantly, we audit clips for specific messages that we’ve included in our press materials. We don’t want to take credit for random catfish articles, but we are especially pleased when we can document our specific messages being picked up and reported by editors.

Evaluation of food service promotion programs might include “mystery shopper” visits to various food service outlets to confirm that they are, in fact, running TCI promotions and visibly displaying materials sent to them as well as evaluation of pre and post promotion sales volume data provided by food service distributors. Evaluation of culinary school training programs might consider whether major culinary schools actually use catfish in their seafood training curriculum, whether or not they allow TCI representatives to conduct on-site training workshops for students, and whether or not they incorporate videotapes, diskettes, or other training aids sent by TCI into their teaching programs.

Our German programs are different from our U.S. programs since there is a very limited catfish distribution in Germany. We support our German efforts with MAP funding administered by USDA/FAS. TCI has a trade representative and a public relations agency in Hamburg, and initial evaluations of their efforts include such measures as the number of top German seafood importers brought on U.S. trade missions, number of German distributors carrying U.S. catfish, total export volume, number of restaurants, office cafeterias, and retail supermarkets conducting catfish promotions, and the number and quality of catfish articles written in German seafood trade and consumer publications. Also, with USDA/FAS funding, we have been required to conduct an independent evaluation of our German activities for each of the past two years.

The preceding paragraphs have detailed some of the specific measures that we use from time to time to evaluate our separate program elements, agencies, and contractors. In addition, we have to look at broader measures that relate to our effectiveness. We can’t ignore sales volume, pricing, and general industry profitability even though these are really beyond the scope of TCI activities in the short run. We also have to evaluate the relevancy of our programs to processors who are actually selling the product. I think it's very important for promotional organizations and their everchanging staff and agency personnel to be in tune with the people who are actually growing, processing, marketing, distributing, and selling the product. This takes a little effort in communication and analysis of personal feedback from the industry.

In summary, I think that managers and directors of promotional organizations should consider as many different types of evaluation vehicles as possible when making decisions. Realize that none of them are infallible and that no one has invented a magic formula that will accurately consider all factors that affect sales of our products and give absolute quantitative analysis of promotional programs. Don’t panic when certain measures indicate problems or poor performance. Work closely with the staff, agency, or contractor responsible when things don’t turn out as expected. If they are worth their hire, they will help to honestly evaluate the situation and identify problems and possible solutions for future programs. If they resist criticism and change, find new people.

In managing a “limited budge promotional organization,” I think that the key is to carefully evaluate all programs on an ongoing basis, try to strike a balance between total inflexibility and extreme knee-jerk reactions to various evaluation methods, and seek to carry out a few well-planned and executed programs over a long period of time.

Director's Corner
Harry M. Kaiser

Although most people think of evaluation when they think of economists' roles in examining promotion programs, another equally important analysis economists can provide promotion organizations with is evaluating whether or not the current allocation of funds is optimal.

Is the current allocation of advertising dollars across markets optimal? Could producers get a bigger bang for their buck by allocating advertising dollars among media types differently? Is the current mix of advertising, promotion, research, new product development, and educational programming maximizing producer investment? Can efficiencies be gained by reallocating advertising among products, e.g., more money on fluid milk and less on butter advertising? These are just a few of the many important questions economists can answer if given appropriate data.

The answers to these questions are important because they provide the potential for promotion organizations to increase profits to farmers at virtually no cost. For example, if an economic analysis revealed that taking 20 percent of television advertising expenditures and reallocating them to radio and print advertising would result in higher producer revenues, then this would represent a no cost strategy that could achieve higher profits to farmers. As it turns out, this is not just a hypothetical example, but rather an actual empirical result of a forthcoming study on optimal media advertising for generic dairy promotion which will appear in a future issue of the NICPRE Quarterly. In some cases, economists have found that reallocating producer checkoff money in some fashion can result in significant returns.

The point is that many people in promotion organizations view economists as adversaries because they evaluate how good or bad a job is being done promoting commodities. While this may be true to some degree, economists can also help managers of commodity promotion groups by providing them with advice on how to better manage checkoff investment activities. Perhaps if economists are used more for the latter task, a greater trust and appreciation of the economist's role in commodity promotion will occur.

NEC-63 FALL 1997 MEETING

The Research Committee on Commodity Promotion (NEC-63) will sponsor a symposium to examine the effectiveness of different nutrition education programs and messages in changing consumers' nutrition knowledge, awareness of diet-health linkages, and attitudes about healthy eating. The meeting will be held in Washington, DC, on October 6 and 7, 1997. If you are interested in presenting a paper or attending, please contact Jim Blaylock, ERS, (202) 219-0900, jblayloc@econ.ag.gov or Karen Ackerman, ERS, (202) 501-8511, ackerman@econ.ag.gov.