CONTENTS

Evaluation Principles and Data Needs

Manager's Viewpoint

Editor's Notes

Director’s Corner

Next Meeting

NEC-63 Fall 1997 October 6-7, 1997 Washington, DC A Symposium to Examine the Effectiveness of Different Nutrition Education Programs

Evaluation Principles and Data Needs

by Henry W. Kinnucan

Now that Congress has mandated that commodity promotion programs be independently evaluated "no less than once every five years," it is useful to review the principles that govern quality evaluation. Evaluation, as specified in the new farm bill, means determining whether the promotion program is indeed effective at maintaining or increasing the demand for the agricultural commodity and thereby improving farm income. Quality is defined as an evaluation that meets accepted standards of scientific rigor while at the same time addressing the question "does promotion pay?" in a manner consistent with economic theory.

Because cost will be a major consideration in meeting the evaluation requirement, especially for smaller programs with limited budgets, I will endeavor to identify practical things commodity promotion organizations can do to facilitate the evaluation process and thereby lower costs. Chief among these is getting an early start on the evaluation process. Although the first evaluation is not due for five years, putting it off until the last minute is the surest way to compromise quality and increase cost.

Most evaluations take at least six months from the time the researcher gets the data until a final report can be given. Add to this the time needed to develop an evaluation proposal, select a researcher, and have the completed evaluation critiqued, and the time frame lengthens to one year at the very least. And this assumes that the necessary data are available and in the proper form for economic analysis. Unfortunately, this is rarely the case, so a portion of this article is devoted to describing the data needs of researchers.

The principles and data needs described in this article are distilled from 16 years of personal experience in evaluating commodity promotion programs in the United States, Canada, Korea, and Australia. This experience also includes working with a very small commodity promotion organization, namely catfish, so I am cognizant of the unique problems faced by smaller organizations.

Evaluation Principles
The three pillars of sound evaluation are the scientific method, reproducibility, and peer review. Eliminate any pillar, and the edifice upon which evaluation rests is undermined. Lest the pillars sound abstract or unworkable, I will describe each briefly, using examples to illustrate what producers or promotion board managers should look for when assessing an evaluation proposal.

The Scientific Method. The scientific method consists of four interrelated components: theory, hypotheses, data, and empirical testing. Most promotion evaluations make ample use of data and empirical tests; the real problem is the underutilization of economic theory and the consequent overly-narrow framing (or misframing) of hypotheses. For example, we know from economic theory that program profitability depends on a few key structural elements as follows: supply response, cross-commodity substitution, middlemen markup behavior, processor technology, market structure (e.g., imperfect competition beyond the farm gate), farm programs, tax shifting (the incidence of promotion checkoff), and trade status (e.g., whether promotion occurs in a large or small open-economy setting). Yet few studies address these structural elements in any systematic fashion, and most ignore all but one or two.

Theory is essential for sound evaluation because it directs attention to appropriate variables to be included in the analysis, it tells how the variables are related to one another, and it informs the data collection process. A telltale sign of whether an evaluation proposal gives adequate attention to theory is the sophistication of the proposal's economic model. An economic model consisting of only demand equations, for example, is almost surely to be untenable from the standpoint of economic theory. Any analysis flowing from such a model is likely to be misleading and thus of questionable value because key structural elements (e.g., supply response and markup behavior) are ignored.

Reproducibility. The second principle of sound evaluation is reproducibility. Reproducibility means that the methods used to arrive at a study's conclusions must be clearly spelled out so that another researcher competent in the field can replicate the study without undue guesswork. Reproducibility also means the data and programs used in the analysis should be made available, if not in the study's appendix, then at least on a diskette so other researchers have access to the information.

Reproducibility is important because advertising effects are known to be fragile. That is, estimated advertising elasticities, a key component of any economic evaluation of advertising impact, tend to be sensitive to model specification, estimation procedure, measurement error, and sample period. For example, in a recently completed study, we found that adding just seven observations to the sample period altered the estimated coefficients for beef advertising from a positive number to zero (Kinnucan, Xiao, Hsia, and Jackson). Thus, the ability to replicate studies is essential in making sure results are trustworthy.

Peer Review. The third principle of sound evaluation is peer review. Promotion evaluation is complex. And each evaluation presents unique challenges requiring creative solutions. For these reasons, it is essential that others with expertise in promotion evaluation be given the opportunity to critique the research, to raise questions, and to suggest changes that would strengthen the analysis. Peer review, properly done, is a win-win situation. The evaluator wins because peer-reviewed research generally has a higher level of rigor and scholarship, which reflects well on the researcher. The promotion board wins because peer-reviewed research is more likely to be accepted by the scientific community, which means it will carry greater weight in policy discussions, in communications with producers, and as evidence in legal proceedings. The best possible peer review is to have the study published in a reputable journal, such as the American Journal of Agricultural Economics.

Peer review can extend to the proposal phase as well. By enlisting the proper expertise in the evaluation's design phase, costly mistakes can often be avoided and a better proposal can be tendered to the research community. As indicated earlier, a common pitfall of many evaluations is overinvestment in data analysis and underinvestment in economic theory. An ex ante peer review, properly done and with the right expertise, can prevent this problem.

Data Needs
Evaluations that adhere to the foregoing principles are likely to be expensive, so what can be done to keep costs within reasonable limits? Besides getting an early start on the evaluation process, probably the most important thing a commodity promotion board can do to lower evaluation costs is to focus on data. Data have to be in a particular form if they are to be amenable to economic analysis. By paying advance attention to the data needs of researchers, commodity boards can save time and money when results are needed. What follows is a suggested format for making data available to researchers. The format is based on my own experience working with commodity boards, in addition to suggestions provided by research colleagues.

Data reporting has three critical elements: accuracy, periodicity, and categorization. Accuracy means the data must reflect actual program activity. For example, in reporting advertising expenditures, it is essential that actual outlays, not budgeted or planned amounts, be given. Accuracy also means that sufficient detail be given so researchers can interpret the numbers intelligently. For example, instances of significant change in target audiences, media strategy, creative appeals, and so forth should be reported along with the numbers. This permits researchers to test whether or not qualitative or strategic aspects of the campaign have any detectable effects on response coefficients.

Periodicity refers to reporting interval. The best reporting interval, at least for promotion evaluation, is monthly. Monthly data are best because advertising lag structures can be estimated more accurately, degrees of freedom are less of a problem, and the analysis can be carried out over a shorter time frame. For example, if only annual data were available, the researcher would need to go back 20-30 years to get sufficient data to estimate the model. A model estimated over such a long time period is likely to suffer from specification error, unless the industry is stagnant or advertising effects are stable, which is not likely to be the case. With monthly data, a time frame of four to five years is usually sufficient to estimate advertising responses.

Categorization refers to how the data are to be separated. A desirable categorization is much harder to define than a desirable periodicity. A lot will depend on the complexity of the board's activities. For a relatively simple program like catfish, where the major emphasis is on media advertising in the domestic market, it would be sufficient to report advertising expenditures per month broken down by medium (television, radio, print). For a highly complex program like dairy, where extensive investments are made in research and nutrition education as well as promotion in domestic and export markets, a much more detailed categorization of expenditures would be required.

An example categorization that might be suitable for the "average" commodity promotion organization is given in Exhibit 1. In this example, the commodity organization would provide monthly expenditure data for six program activities: media advertising, merchandising (e.g., point-of-purchase promotions), public relations (including consumer information programs), research on new product development and production (or agronomic research), and administration. In reporting the data, care should be taken that the numbers add up across the categories to assure accuracy and completeness.

The categories provided in Exhibit 1, of course, could be refined or expanded as needed to more clearly represent the organization's total promotion program. For example, if the board participates in USDA's export promotion programs, columns would need to be added to indicate target markets and expenditures in each market, including monies provided by government and foreign third-party cooperators. In addition, if the commodity organization has price or quantity data pertaining to the commodity, this information could also be included in the data matrix. In general, however, the researcher can obtain this information from public sources.

Returning to my earlier point about the need to start the evaluation process early, note that carrying out an econometric analysis based on the data given in Exhibit 1, the researcher would need at least four years of observations. Thus, to meet the time line set by Congress, boards need to begin thinking about data collection procedures now. Delay will increase cost and lower quality.

Concluding Comments
A basic theme of this article is that sound evaluation follows certain immutable principles. Armed with this knowledge, promotion boards are in the best position to serve the interest of producers in terms of providing accurate and reliable information about program performance. There is nothing less relevant, or more costly, than a poor evaluation, i.e., one that does not address the question (does promotion pay?) or that attempts to cut corners.

A side benefit of making data available is that it opens up the possibility for evaluation to be undertaken at a university at little or no cost to the promotion program, as has been the case for catfish. University researchers have an incentive to publish, promotion evaluation lends itself to the "tools-of-the-trade," and academic interest in the subject matter is growing. A drawback, of course, to making data generally available is that the commodity organization loses control of the evaluation process. Still, if objective, low-cost, sound evaluation is desired, the single best thing a commodity promotion organization can do is publicize the relevant data.

References:
Kinnucan, H.W., H. Xiao, C.-J. Hsia, and J.D. Jackson. "Effects of Health Information and Generic Advertising on U.S. Meat Demand." American Journal of Agricultural Economics 79(1997): forthcoming.

[ return to text ] [ top ]

Exhibit 1: A Prototypical Data Reporting Matrix for a Commodity Promotion Organization

Year/Month	Media Advertising	Merchandising	Public Relations	Research		Administration
				Product Development	Agronomic
19xx	- - - - - - - - - Actual Dollar Outlays - - - - - - - - -
January
February
. . .
December
Annual Total
19xy
January
February
. . .
December
Annual Total
19xz
. . .