Critique : The Validity and Reliability of Cross-National Surveys Analysis

The purpose of this review paper is to provide a guideline “on how to review and comment on published scientific paper”. The original article is well-written, based on secondary data. This paper was written about a decade ago, though still very relevant and useful for the students and scholars of social and environmental sciences. This article deals with the validity and reliability of cross national survey research primarily based on secondary information. The author opens the question about the on-going validity and reliability debate and issues of post modernism on global environmentalism, pioneered by Steven R. Brechin and Willet Kempton in 1994. Eric Neumayer (2002) has effectively introduced the themes of “Global Environmentalism: A challenge to the Post-Materialism Thesis”, a paper by Brechin and Kempton (1994) which challenged the conventional wisdom that people in developing countries lack environmental values. Brechin and Kempton (1994) rejected the established notion of the western world’s view of developing nations as having little concern for environmental conservation, which was until recently guided by a colonial mentality. The article by Brechin and Kempton advocates in favor of global environment concern by people of the developing world. They have used the data from Dunlap, Riley E., George H. Gallup, Jr., and Alec M. Gallup (1993a), where they show that people from the developing world are equally concerned about the environmental problems. Neumayer (2002) did not adequately describe the core concept of Brechin and Kempton in examining the validity and reliability of cross-national surveys.

The purpose of this review paper is to provide a guideline "on how to review and comment on published scientific paper".The original article is well-written, based on secondary data.This paper was written about a decade ago, though still very relevant and useful for the students and scholars of social and environmental sciences.This article deals with the validity and reliability of cross national survey research primarily based on secondary information.The author opens the question about the on-going validity and reliability debate and issues of post modernism on global environmentalism, pioneered by Steven R. Brechin andWillet Kempton in 1994. Eric Neumayer (2002) has effectively introduced the themes of "Global Environmentalism: A challenge to the Post-Materialism Thesis", a paper by Brechin and Kempton (1994) which challenged the conventional wisdom that people in developing countries lack environmental values.Brechin and Kempton (1994) rejected the established notion of the western world's view of developing nations as having little concern for environmental conservation, which was until recently guided by a colonial mentality.The article by Brechin and Kempton advocates in favor of global environment concern by people of the developing world.They have used the data from Dunlap, Riley E., George H. Gallup, Jr., and Alec M. Gallup (1993a), where they show that people from the developing world are equally concerned about the environmental problems.Neumayer (2002) did not adequately describe the core concept of Brechin and Kempton in examining the validity and reliability of cross-national surveys.
In this review, I will first present a snap-shot of the article in term of research design and testing of validity and reliability.And secondly, I will examine to what extent the author has succeeded in bringing to light empirical evidence on cross-national research design (see my final section on the strengths and weaknesses of the article).

Research design:
Neumayer's research is based on secondary data from Dunlap, Gallup and Gallup (1993a) and his empirical evidence is primarily drawn from Brechin and Kempton (1994) and Inglehart (1995) Neumayer's article is silent about the research design, though its logical order and argumentation shows that the author used descriptive research design.Neumayer's article does not discuss any significant cross-national or cross-cultural research, which has been found in most papers of the discipline (bio-physical to socioeconomic).Therefore, I think his article is only based on logic and opinion.Neumayer uses the comparative method within the quantitative research frame, utilizing the published data; however, he does not provide the raw data of Dunlap et al. (1993a), which makes it difficult to test the validity of the conclusions.He also does not include the specific data from the World Bank and other agencies.Again, going back to the title of the article, "Do We Trust the Data"?Neumayer therefore leaves me room for a question: does this article itself demonstrate a valid approach?The reliability test was conducted with variables used by Gallup et al. (1993a) and others.The first variables were Higher Prices (percentage who say they are "willing" to pay higher prices to protect the environment) and data from Frizzell and Pammett (1997), Much Higher Taxes, (percentage who say they are "very willing" or "fairly willing" to pay much higher taxes in order to protect the environment).The second variables were Much Higher Prices, data from Frizzell and Pammett (1997) and Higher Taxes (percentage who "strongly agree" or "agree" to an increase in taxes if the extra money is used to prevent environmental pollution) from Inglehart, Basanez and Moreno (1998).

Data Selection:
For the validity check, the author has illustrated four major international surveys concerning the environment from Louis Harris and Associates (1989), Dunlap et al. (1993a), the World Values Survey (Inglehart, Basanez and Moreno, 1998), and the International Social Survey Program (Frizzell and Pammett, 1998).The data and their authorship were difficult to match up on first reading.There is no explanation about the survey tools of the above-mentioned surveys; however the author states that he rejected all of them and only used the Gallup et al. data.He states that three surveys mentioned above did not ask sufficient questions regarding environmental concern.He concentrates on Gallup et al's data for the validity checks.Among the variables used are World Bank (1996) and UNDP (1995) data, but he does not illustrate them in explaining his validity checks.My understanding is that he uses his own value judgment to qualify or disqualify the survey data.For example, he could have used data collected with similar methods.There is no indication of how these data were collected.I do not see a valid method for data selection.

Data analysis:
For the validity checks, variables denoting environmental concern were used, such as people's perspectives about the environment in terms of seriousness of inadequate sanitation, air pollution, water pollution, noise pollution and quality of the nation's environment, and were compared with another variable such as "percentage of population with access to sanitation and "real GDP per capita in purchasing power parity", or overpopulation and the effect of over population, population growth, effect of lack of education, effect of government, educational attainment and index of political freedom, all using a correlation matrix.For the reliability test, variables such as higher prices, much higher prices and higher taxes were used to measure the willingness to pay for better environment, with the application of the same statistical tool, correlation analysis, without considering other perhaps more suitable statistical treatment for these very different kinds of data.

Results and Discussion:
Neumayer's article claims to provide evidence for the validity and reliability of cross-national environmental surveys, which is not common.It focuses on validity, which can be checked in examining cross-national differences in environmental attitudes.This notion is not contrary to agreement with theoretical hypotheses.This study can claim that cross-national environmental surveys can be valid, but not complete.The results show inadequate sanitation is negatively correlated to the actual extent of access to sanitation in the surveyed country.It also shows that from the published surveys several validity checks could be undertaken.The author has used both Pearson and Spearman rank correlations for similar questions from differing studies.Neumayer (2002) has presented three correlation matrixes to show the validity of the Gallup et al. (1993) survey.The use of Gallup et al.'s (1993) survey to check validity can be accepted as a good example of data for the validity tests.However, there is no raw data appended, and Neumayer has only analyzed a few sets of the data, therefore I do not see that it is appropriate to claim on the basis of this scant information that the whole survey is valid.
In the first validity test (Table 1, Neumayer 2002): the first variables Seriousness of inadequate sanitation (Pearson Correlation -0.544), Seriousness of air pollution (-0.530), Seriousness of water pollution (-0.666), Seriousness of noise pollution (-0.619) and Bad quality of nations environment (-0.505), to which respondents replied very serious and very bad, were strongly correlated with the second variables "Percentage of population with access to sanitation" and "Real GDP per capita in purchasing power parity" respectively.Likewise, in the second validity test (Neumayer's Table 2), the first variables Seriousness for overpopulation (0.710) and Effect of over population (0.791) were also strongly correlated with population growth rates (a second variable).In the third validity test (Neumayer's Table 3): Effect of lack of education (-0.528),Effect of government (-0.590) and Effect of government (-0.777)were well correlated with Index of educational attainment, Index of political freedom and Index of perceived corruption respectively.
On another point, the reliability test (Neumayer's Table 4) was not as significant as the validity tests.The first variables (i) higher prices (ii) higher prices, (iii) much higher taxes were correlated only with significance of 0.467, 0.448 and 0.303 to much higher prices, higher taxes and higher taxes in terms of willingness to pay.In general, the last-mentioned correlations are low and statistically insignificant.On the basis of these low correlations, Neumayer has questions about the reliability of the empirical studies based on cross-national survey data, because the wording of the questions may differ, but it is not clear from the information he provides whether the questions were similarly worded or otherwise.

Strengths and limitations of the study:
To achieve the appropriate quality standard of any research, I think the questions and scales of measurement should be examined with many possible methods.Research should undergo an evaluation using standard quality criteria and tools and techniques such as reliability and validity testing.The researcher needs to apply all possible options to evaluate the previously used questions in general surveys and particularly in crossnational surveys.If it is a new case of new questions, evaluations should be based on "predictions" that take into account their respective properties with the strongest empirical support.Reliability and validity are commonly used criteria for evaluation of research; however, these are not the only criteria that matter.The researcher also needs to give attention to other options (situation, spatial and temporal elements, bio-physical and socio-economic criteria) and also needs to examine the comparability of items over time and space, expected item non-response, social desirability and other potential biases, and the avoidance of ambiguity, vagueness and double-barreled questions.Neumayer appears not to consider any points of researchers' ethics but just uses a set of data and rejects other data without proper examination.
Neumayer states "validity refers here to the degree to which the empirical data genuinely or truly measure the theoretical construct of interest (internal validity).Given that we do not know the "true" attitude of respondents the only way to check validity is to examine whether the answers given are consistent with strong theoretical hypotheses.Reliability refers here to the consistency of data coming from different survey (comparative reliability)".Neumayer states that validity and reliability issues are not considered as seriously as they should be in the cross-national surveys.He uses the empirical evidence from Brechin and Kempton (1994)'s and Dunlap et al. (1993a).Neumayer presents their arguments, briefly examines the nature of some of the data (but not comprehensively) and reviews the results in terms of validity and reliability.In a one sentence conclusion Neumayer concludes that the several studies he extracted data from were valid but is not clear about their reliability.
I have some concern with his presentation, explanation and conclusion as well.The basic dilemma for me is his narrow focus on the issues.The cross-national survey which Neumayer presents from Dunlap et al. (1993a) is not the first work of this type.Neumayer primarily ignores the information on cross national surveys which were conducted prior to Dunlap et al. 1993a (in business, economics, medical sciences and other fields), for example Zanna, ed.(1961) has evaluated Trans-cultural variables and conceptual equivalence in studying Personality in cross cultural settings, Serpell (1961) has examined cross-culture and cross national life journey in An African Society, van (1978) In general, I have no disagreement with the author's point that is the title itself "Do We Trust the Data"?The article does not give the answer, however, because in the case of cross-national surveys it depends on the nature of the research.If it is health related research such as cancer, AIDs or well-documented epidemic then there is less doubt and questions.Likewise, if we are examining the environmental concerns, yes, it is an established rule of thumb that the nature of environment problems is similar within the North (developed world) and similar within the South (developing world) but not similar between North and South.However, according to Kohn (1987) "As with any research strategy, cross-national research comes at a price: it is costly in time and money; it is difficult to do; it often seems to raise more interpretive problems than it solves" (Comparative Research).In the present context, cross-national survey based data collection procedures are becoming common standards from individual level and household levels to community, nations and crossnational level.The problem of such research is related to the nature of research.If research is macro level and using broader comparisons with standardized data it is easy to generalize, but if it is micro level research such as Brechin and Kempton's (1994)

Variables and measures :
In the first of Neumayer's tests (Validity Test for Sanitation and Aspects of Environmental Pollution); the variables used are derived from an environmental health survey carried out by Dunlap et al. (1993a).As first variables, he uses Seriousness of Inadequate Sanitation, Seriousness of Air Pollution, Seriousness of Water Pollution, Seriousness of Noise Pollution and Bad Quality of Nation's Environment, (respondent's responses were mainly "very serious/bad").The second variables were "Percentage of Population with Access to Sanitation in 1995" (World Bank, 1998) and "Real GDP per capita in Purchasing Power Parity in 1992" (UNDP, 1995).In the second test (Validity Test for Overpopulation), the first variables used were Seriousness of Overpopulation and Effect of Overpopulation, while Population Growth Rates were the second variable (data from Gallup et al. (1993a, b) and World Bank (1996) respectively).The third validity test (Validity Test for Factors Contributing to a Nation's Environmental Problems) was carried out using the same source, Gallup et al. (1993a), for Effect of Lack of Education and Effect of Government as first variables.Index of Educational Attainment from a UNDP (1995) data source, Index of Political Freedom from Freedom House (1999) and Index of Perceived Corruption from Transparency International (1998) were the second variables.
, has done research on "Cross-cultural Research Methodology as a Case of Construct Validity".Similarly, Arbor, Werner, Oscar and Campbell (1970), have studied the problem of decentering and Naroll and Cohen, eds (1970) has shown the evidences of cross-national survey problem in examining the historical artifact.Likewise, Shimp and Sharma (1987) have shown the use of cross national and cross-cultural situation while examining on Consumer Ethnocentrism: Construct Validation of the CETSCALE".In the same niche Schwartz (1992) have gathered and discussed the importance of cross-national and cross-cultural studies while examining the universals in the content and structure of values, with the application of twenty countries' empirical tests.These are just a few examples of cross national survey research that provide great detail about the data for such surveys.Neumayer lacks the theoretical ground of the crossnational survey research by not incorporating the large number of empirical evidences.Furthermore, Neumayer has not incorporated the very similar research by Brechin (1999), which has dealt with the objective problems, subjective values, and global environmentalism using same secondary source of data from Dunlap at el (1993a).Ignoring the established theoretical ground of cross national survey design Neumayer (2002) leaves the ground for the question of whether his arguments are valid.
study of post modern theses or Dunlap et al.'s (1993a) environmental health research it is challenging.The author has not evaluated such problems of research while developing his logic based upon them.The main article: Neumayer Eric (2002), Do We Trust the Data?On the Validity and Reliability of Cross-National Environmental Surveys, Social Science Quarterly, Volume 83, Number 1, March 2002, pp.332-340(9/eprints.lse.ac.uk/17880/1/Do%20we%20trust%20the%20data(lsero).pdf(downloaded on 1/24/2018) . Neumayer's article uses a type of Meta analysis (because it is based on published articles), though it is an incomplete Meta analysis design.Correlation is the only statistical tool used.It is unclear whether Dunlap et al.'s paper Health of the Planet: Results of a 1992 International Environmental Opinion Survey of Citizens in 24 Nations had already used the same correlation analysis or not.