What is Sound Science?

by Frederick W. Pontius, P.E.

Soon after a regulatory compliance seminar ended, an attendee said, “That EPA rule is not based on sound science!” When I asked what constituted sound science, he returned a rather puzzled look. Finding weaknesses in a scientific study often is not difficult given the nature of science. But when should a study be considered sound?

To say “I don’t know what sound science is, but I know it’s not that study!” or “This study is obviously sound science, because it concludes what I already believe is true,” simply confuses the matter.

Since earliest days, science and engineering have formed the foundation of public drinking water supply. The 1996 Safe Drinking Water Act (SDWA) amendments specifically require USEPA, to the degree that an agency action is based on science, to use the best available, peer‑reviewed science and supporting studies conducted in accordance with sound and objective scientific practices. Data used must be collected by accepted or best available methods, if the reliability of the method and the nature of the decision justifies use of the data. Given the importance of science to the rulemaking endeavor, sharp disagreement often exists as to what constitutes sound science.

A Quest for Scientific Truth
Ultimately, our attempt to define sound science is a search for truth within a scientific context. One of the earliest attempts is credited to Zeno (c. 475 B.C.), who defined truth by reducing alternative positions to the absurd, known as the reductio ad absurdum argument. The law of noncontradiction is a fundamental principle of logical thought. No position that generates contradictions can be considered true. At best, however, this is a negative test for truth, demonstrating that some positions are false, but failing to determine which ones are true.

Inductive and deductive reasoning play an important role in the quest for scientific truth. Though he accepted both forms of reasoning, Aristotle (384-322 B.C.) was the first Western philosopher to elaborate rules for deduction. Deductive reasoning is simply arguing from the general to the particular. Formal rules of logic are applied in a series of propositions, called a syllogism. A valid syllogism has a particular structural form, as described in introductory textbooks on philosophy and logic. But not all valid syllogisms are true.

For a syllogism to be considered “sound,” the premises must be true. This is the principal limitation of deductive reasoning. Often in scientific and regulatory policy debates, the premises used to form arguments are actually assertions or statements based on prior assumptions, the truth of which are assumed but unknown.

For example, consider the major premise “drinking water meeting USEPA regulations is safe to drink” (the general). And the premise “the drinking water of town x meets current USEPA regulations” (the particular). One might argue, then, that “town x’s drinking water is safe to drink” (the conclusion). Now suppose town x’s water also contained 48 mg/l of arsenic. The town’s water meets the arsenic MCL that is currently in effect, but not the new USEPA standard for arsenic of 10 mg/l expected to take effect five years hence. Should the soundness of our original argument be called into question?

The Inductive Method
Advancement of the inductive method of discovering scientific truth is credited to Francis Bacon (1561-1626). He formulated the basic rules of induction, which became the forerunner of Cannons of Inductive Logic, by John Stuart Mill (1806-1873). Mill’s inductive method is summarized by these rules:
(1) The Method of Agreement – The one factor common to all antecedent situations where an effect occurs is probably the cause of the effect.
(2) The Method of Difference – Whenever an effect occurs when A is present but not when it is absent, then A is probably the cause of the effect.
(3) The Joint Method – The first two methods are combined when one method alone does not yield a definite result.
(4) The Method of Concomitant Variations--When an antecedent factor
varies concomitantly with a consequent factor, then the former is probably the cause of the latter.

The principal limitation of the inductive method is that one cannot be absolutely sure of the conclusion without complete or universal observation or knowledge, which is impossible. Nevertheless, the inductive method has played an important role in the development of public water supply.

In one of the first epidemiological investigations, Dr. John Snow plotted the location of deaths from cholera on a map of central London in 1854. The area’s eleven water pumps were also located on the map. Snow observed a correlation – that cholera occurred almost entirely among those who lived near and drank from the Broad Street water pump. Snow’s hypothesis was that the disease was caused by an unknown microorganism in the drinking water contaminated by fecal material from diseased persons. This was a radical belief in that day, not within the accepted scientific mainstream of thought. Snow had to argue his findings and theory before the local authorities. He prevailed, and the handle of the pump was removed, ending the neighborhood epidemic that had taken more than 500 lives.

Snow’s good, educated, scientific guess was later proved true as the body of science expanded. In 1883, almost thirty years later, Koch finally isolated vibrio cholerae as the organism responsible for causing this particular disease. If Snow had argued that the disease was actually caused by a little invisible green man from Mars that only sits on pump handles, what would have happened? When they removed the pump handle and the disease outbreak ended, one might have been enticed to believe in the little green man hypothesis. But removal of the pump handle alone would not prove the little green man hypothesis to be true. Nor did removal of the handle prove Snow’s hypothesis that microorganisms were responsible. This episode does illustrate how scientific knowledge is applied in real world decision making.

The Scientific Method
The scientific method uses both forms of reasoning: deductive and inductive. Truth claims are made by appeal to factual evidence, based on observation and experience. Beliefs are treated as hypotheses subject to repeated testing and open to public confirmation or refutation. Society, not the individual’s experience, is the final court of appeal for a scientific theory or explanation.

In general, the scientific method involves four basic steps:
(1) Formulating a statement carefully, clearly
(2) Predicting the implications of such a belief
(3) Performing controlled experiments to confirm or refute these implications and observing the consequences
(4) Accepting, rejecting, or modifying the statement as a result

But experimentation is not always possible. This is the principal limitation of the scientific method. In a sense, scientific hypotheses are intuitive leaps in the dark. But they need not be blind leaps, if based on the current body of observable scientific fact. Formulation of a problem in a clear and concise way that can be subjected to experimentation and observation is not always possible. Hence, at times, even the scientific method must rely on good, educated guesses.

All rational disciplines, including the scientific method, take certain things for granted, called presuppositions. Presuppositions are things considered true even though they cannot be proven within that particular discipline, or proven at all. All knowledge has certain beginning points that are simply to be accepted and are impervious to scientific methods. Hence, differences regarding whether a particular study constitutes sound science can arise simply because investigators view the study through a different perspective, reflecting differing basic assumptions.

Observations do not interpret themselves, but are interpreted by a human mind. Scientists can use several techniques to limit the influence of bias in research:

Experimental design. The experimental design should be such that the number of factors that could affect the outcome are limited and should include appropriate controls. Experiments should be structured in advance to limit or eliminate bias to the degree possible.

Replication. Scientists should disclose not only what was found in a research study but how it was found. Experiments should be described in detail, along with a description of the controls used. If other scientists can replicate the results, it provides further evidence that the results are real and objective.

Peer Review. Scientists submit findings to scientific journals which are reviewed by peers for possible errors or leaps in logic. The methods of experimentation should be reviewed to determine whether the experimental designs were valid and show what the author claims. Only when scientists from different areas agree the research is valid should it be published. Peer-review is not simply agreement from other scientists who already agree with the study findings, like a popular vote. Properly conducted, peer review minimizes bias. Over time, accumulated data tends to overthrow erroneous theories and expose fraud.

Falsifiability. A well-conceived hypothesis is falsifiable; that is, it can be proven wrong or untrue. Within the constraints of the scientific method, a scientific claim should not be considered true unless it can be proven or disproven.

Reason Rejected Altogether
For most of the 20th century, a Modern worldview dominated Western culture, claiming that the sciences are rational. Logic is used to outline problems, interpret observations, and formulate and study hypotheses. The postmodern worldview, first articulated in 1979 by Lyotard, has become increasingly influential and pervasive in Western culture, especially over the last decade. Post modernism argues that there is no fixed vantage point beyond our own structuring of the world. There is no such thing as objective reality; there is nothing out there that is true, regardless of whether someone believes it or not.

Objective reasoning is considered a myth. Objective understanding and the power of reason are rejected, making scientific truth relative. What we think is scientific knowledge, what we think is a firm grasp of truth and reality, are viewed only as an opinion or narrative of our own unique view. Within this context, a scientific fact can be true for one person and not true for someone else. Arguments over what is scientifically true can quickly become a warfare of ideas and presuppositions, rather than a thoughtful exchange of scientific thought. Once scientific truth becomes whatever anyone wants it to be, the common ground necessary of meaningful exchange of ideas is eroded, the power of logic and persuasion is diminished or eliminated altogether, and relationships become fractured. Frustration increases between those convinced by existing facts and reasoning that something is ‘true’ and those convinced that the opposite is “true” in spite of existing facts. Like Koch arguing for removal of the pump handle, knowing which set of arguments is ultimately correct can be elusive when scientific knowledge is incomplete. Assertion of power over others to believe as we do and force law or policy changes is the next step.

A key premise of scientific relativism is that one theory is simply replaced by another, without any logical connection, because of paradigm shifts, yet neither is closer to any objective truth. The scientific method is attacked, as objective truth is believed not to exist, even in what is known as the “hard” sciences (chemistry, biology, etc.). Proponents of one scientific theory develop their own language and scientific viewpoint, with no common language between those with other scientific theories.

This postmodern philosophy has serious implications for public water supply. A concept considered true by an epidemiologist would not necessarily be considered true by a toxicologist or engineer. No common ground upon which to exchange ideas would be considered to exist between these disciplines. Each would develop his own worldview apart from the other. Nothing is true or false in an absolute sense, nor is there need for discussion between scientists to integrate differing disciplines into a unified scientific theory or construct to describe objective reality. Something is considered scientifically true simply because it is believed, regardless of whether it is antithetical to scientific or observable “fact.” The little green man hypothesis regarding cholera deaths in London would hold equal footing as truth to Koch’s microorganism hypothesis. Neither would be considered closer to any objective truth.

But, consider the waterborne disease outbreak in Walkerton, Ontario. Contaminated ground water infected more than 1,000 people in a population of 5,000 and caused 11 deaths. The citizens’ belief that the water was safe to drink did not invalidate the reality of disease-causing microorganisms in the water.

The postmodern worldview and scientific relativism contain a serious contradiction – observation and logical inference are used to argue that observation and logical inference tell us nothing. By using the tools of science to argue its views, the postmodern view demonstrates the belief that these tools actually work, rather than these tools being worthless. Arguing that scientific relativism is universally true is internally inconsistent and contradictory.

Although diversity of informed scientific opinion is desirable in the advancement and testing of hypotheses, scientific relativism undermines the very concept of sound science. Although imperfect, the fact remains that science attains the goal of discovering objective, rational truth much of the time. Modern science still represents the closest thing we have to finding objective, international, “universal” knowledge, and no better alternative yet exists.

Epidemiology Versus Toxicology
Defining sound science is especially difficult in the area of assessing potential health risks to drinking water contaminants. Epidemiologists and toxicologists (as well as engineers, chemists, microbiologists, and water plant operators) use differing techniques and are trained in their respective disciplines to think differently from one another. They often start with differing fundamental assumptions. Hence, debates over the interpretation and meaning of epidemiological and toxicological studies can be sharp.

Unlike physicians who study disease in individuals, epidemiology is the study of patterns of disease in groups of people or populations. Epidemiologists focus on what is common and general about members of populations. Unlike laboratory studies of the toxicologist or chemist in which the investigator determines the conditions under which observations are made, epidemiologists observe the world as it is and must draw inferences that accommodate the study subject’s particular habits (such as smoking, drinking, and poor diet).The validity of the association observed between exposure and disease in an epidemiological study must be assured before causality can be evaluated. Epidemiology is observational and inductive; therefore, no single study can provide a definite answer about causality even if systematic bias is minimal. A body of evidence is necessary from studies conducted in different geographic areas and populations.

Epidemiologists can apply certain guidelines to assess the possible causality of associations observed in well-designed and conducted studies. Epidemiological data must be interpreted with caution and within the greater context of available scientific information. The following are typically applied to assess evidence about causality:
Biological Plausibility. An inference of causality is strengthened if the association is supported by evidence from clinical research or toxicology about biological behavior or mechanisms.

Temporal Association. Exposure must precede the disease. This can be inferred in most epidemiological studies. However, when exposure and disease are measured simultaneously, it is possible that the exposure has been modified by the presence of disease.

Study Precision and Validity. Individual studies that provide evidence of an association must be well designed with an adequate number of study participants (good precision) and must be conducted well with valid results (i.e., the association is not likely due to systematic bias).

Strength of Association. The larger the Relative Risk (RR) or Odds Ratio (OR), the less likely the association is to be spurious or due to confounding bias. A causal association cannot be ruled out only when weak association is observed.

Consistency. Repeated observation of an association under different study conditions supports an inference of causality, but the absence of consistency does not rule out causality.

Specificity. A putative cause or exposure leads to a specific effect. The presence of specificity argues causality, but its absence does not rule it out.

Dose-Response Relationship. A causal interpretation is more plausible when a risk gradient is found (e.g., higher risk is associated with larger exposures).

Reversibility. An observed association leads to some preventative action, and removal of the possible cause leads to reduction of disease or risk of disease.

In general, toxicology is the study of poisons, their effects, and antidotes. Human epidemiological data alone usually cannot define cause-and-effect relationships and human health effects data, for many contaminants do not exist. Thus, a contaminant’s potential risk to humans is typically estimated based on the response of laboratory animals to the contaminant. The assumption is that effects observed in animals may occur in humans, but this is not always true.

Use of controlled animal exposure studies for assessing human health effects has been the subject of ongoing discussion. In general, health effects observed in animals are considered applicable to humans, if properly qualified. In addition, exposing animals to high doses of toxic agents is still considered a valid scientific method of discovering possible carcinogenic hazards in humans. But one limitation of toxicological studies is that they can be difficult to interpret in terms of human health risks, assumptions, modeling, and application of uncertainty factors that may or may not be reflective of reality.

Truth Claims Must Be Evaluated
Throughout the regulatory development process, deliberations occur over scientific data with arguments for and against positions, policies, and regulatory options. Disagreement among experts as to what is true can create confusion as to the best way to approach a regulatory issue.

Disagreement regarding sound science is to be expected as experts move beyond facts that can be proven using a scientific protocol or logical reasoning to express beliefs that cannot be scientifically documented. Scientific integrity depends on institutions that maintain competition between scientists, and scientific groups who are numerous, dispersed, and independent. Scientific claims should be subject to replication and verification. Informed opinions of scientific and engineering experts are needed throughout the regulatory process, both in terms of what is scientifically factual as well as what may be possible, probable, and improbable.

Scientists and engineers also have opinions, and like Koch, make informed, educated guesses. Formulation and defense of different hypotheses that extend beyond known facts helps the regulatory process, as long as a clear distinction is made between what we can show to be true scientifically from what we believe or suspect to be true. The criteria for what counts as knowledge, the epistemology of science, must remain high. Emphasis should be on the quality of the science, not who performed or funded the study. Peer-review is important in the evaluation and acceptance of scientific findings, but it must be carefully conducted to ensure reviews are fair and not unduly biased.

Replacing scientific standards and methods with feelings of sincerity shifts the focus from searching for universal truths on which to base good decisions to the expression of personal testimonies. What is true becomes far less important or even irrelevant, as in the case of the postmodern view, compared to what is personally authentic. Although science cannot answer every question definitively, replacing science with sincere feelings undermines the scientific underpinnings of good decision-making. Otherwise sound science becomes defined by an eminent authority, or by whoever is in power or can overpower everyone else.

Alternative views and truth claims in the “marketplace” of ideas must be carefully evaluated. Typically, a spectrum of beliefs and data must be deciphered and evaluated to extract those scientific facts, assumptions, and beliefs that will stand the test of time. Determining what constitutes sound science requires time to gather factual information, consideration of opposing views and alternative hypotheses, application of good cross-discipline integrative thinking habits, and a clear objective. For informed public policy and regulatory policy discussions, context, source, presumptions, and bias in scientific studies all must be evaluated to determine when sound science is truly sound.

Fred Pontius, Pontius Water Consultants, Inc., specializes in Regulatory Affairs and Compliance. He can be reached at fredp@pontiuswater.com