Tuesday, July 24, 2012

How Should Science be Done?

Lately I keep running into the idea that the proper way to do science is to continually strive to disprove a hypothesis, rather than support it*.  According to these writers, this is what scientists are supposed to aspire to, but I've never actually heard a scientist say this.  The latest example was recently published in the Wall Street Journal (1).  This evokes an image of the Super Scientist, one who is so skeptical that he never believes his own ideas and is constantly trying to tear them down.  I'm no philosopher of science, but this idea never sat well with me, and it's contrary to how science is practiced. 

I could spend my entire career trying to disprove Pasteur's germ theory, and it would be a waste of time.  I could spend my career trying to disprove the idea that DNA contains genetic material, and I would also be wasting my time.  Why did we ever move on from testing these hypotheses?  Because the evidence supporting them is overwhelming.  At some level of evidence, one has to conclude that a hypothesis is sufficiently supported, stop testing it, and move on. 

The scientific method is just a formalized version of common sense.  If you were to try to eat five rocks, and break your teeth each time, you'd conclude that rocks aren't good food and stop trying to eat them.  You wouldn't conclude that you failed to disprove the idea that rocks aren't good food, and keep trying to eat them. 

To decide whether or not a hypothesis (i.e., an idea or model) is supported by evidence, a critical element is the use of a "hypothesis test".  Hypothesis tests are based on probability.  The techniques that allow us to do this are called statistics.  These hypothesis tests are fundamental to quantitative science, because they are what allow you to say that your results are "statistically significant" rather than arising by chance, and this is an essential element of being able to claim that your hypothesis is supported rather than unsupported.

Basically, a hypothesis test is set up by pitting one hypothesis against another.  Hypothesis #1 is the effect you're looking for, for example that tall people on average have bigger feet than short people.  Hypothesis #2 is called the "null hypothesis", and it is what would be observed if hypothesis #1 were not correct, i.e. there is no difference in the foot size of tall and short people. 

If we take our measurements and find, using the appropriate statistical test, that there is a difference in foot size between groups, and that this difference is unlikely to have arisen by chance, then we reject the null hypothesis.  Therefore, the experimental hypothesis is supported and tall people probably do have bigger feet on average.

This is important to understand.  In this case, the hypothesis test rejects the null hypothesis, supporting the experimental hypothesis.  We don't say "our results fail to reject the hypothesis that tall people have bigger feet", as we would if every experiment were designed to try to reject our idea.  We say "our results support the hypothesis that tall people have bigger feet", because the null hypothesis, that foot size is the same, has been rejected.  Next, we have to decide if the effect size is large enough to be important, and how it fits in with the rest of the scientific literature.  Ideally, other groups will independently do the same experiment and find the same result, otherwise we have to question our conclusions.

Experiments support hypotheses, they do not fail to reject them.  This is good science.  It is true that we will never be able to weed out all subjectivity from scientific research, that some scientists hold irrational beliefs in regard to their own research, and that these irrational beliefs are often due to social factors and self-serving motivations, because after all scientists are humans too.  But the scientific method is nevertheless the best tool we have for minimizing subjectivity in the pursuit of information, and the way we are using it currently is pretty darn effective. 


* As an aside, in many cases it is literally impossible to disprove or falsify a hypothesis using conventional statistics methods.  Going back to the foot size example, if we find that there is no statistically significant difference in foot size between short and tall people, technically speaking we do not reject the hypothesis that tall people have bigger feet.  We have not disproven it, what we have done is failed to support it because we couldn't reject the null hypothesis.  Our test could not rule out the possibility that in the population at large (as opposed to the random sample of people in our experiment), there is a real difference in foot size that was too small to detect in our experiment.

The goal of most experiments is not to try to falsify or disprove a hypothesis (which in any case is often impossible), it is to test a hypothesis by pitting it against the null hypothesis.  In other words, does the model accurately predict reality when it is tested?  This is how it should be.  The outcome of many experiments is either a) the hypothesis is supported, or b) the null hypothesis is not rejected, i.e. there is not sufficient evidence in support of the hypothesis.  There is often no option "c", the hypothesis is falsified.

It is often said that an idea must be falsifiable to be scientific.  Given the fact that hypotheses often cannot be falsified using our current methods, I think a better way to convey this idea is to say that an idea must be testable to be scientific. We can fudge this a little bit and say that an idea has been falsified if we test it several different ways and none of them support it, or if it's clear that even if the effect exists, it's too small to be important.

56 comments:

Todd Hargrove said...

I am no philosopher of science either, but I totally agree with your statement that "the scientific method is just a formalized version of common sense." Well said.

Justin said...

I think what is meant by people saying that scientists should be trying to disprove their hypothesis is that a good, valid hypothesis test is one that CAN disprove their hypothesis.

If they know the results going in, their test isn't exactly a test, is it?

David L said...

A key area of reasoning that people are less familiar with is the strong role of Bayesian statistics in formming and tetsing hypotheses. Most of the Type 1/Type 2 reasoning is presented in the form of objectivist, aka frequentist, probabilities.

However, the nature of most scientific reasoning is that a hypothesis is developed in the context of limited information, and "subjective"judgements have to be applied in order to make any positvistic intepretation of the data.

One key person in this regard is Edwin Jaynes, epsecially when related to the maximum entropy hypothesis. A poor book (IMO) is "The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy," by Sharon Bertsch McGrayne .

Suffice it to say, it is my contention that scientific discovery can only be made via application of Bayesian reasoning, and that the use "objective" statistics will get you nowhere. A deep appreciation of entropy (and dis-entropy, aka information) is most useful when examining these issues.

Anonymous said...

I have to disagree that the scientific process is common sense to most people.

Change the hypothesis to "people are tall because they have bigger feet."

I'm pretty sure most/all the progress in understanding comes from the disproving/disambiguation of common ideologies.

Jim said...

Stephan, I think you are an intellectually disciplined and humble individual who has trained his common sense to be closer to ideal than most people have.

I also think you’re right that people misuse Popper’s dictum.

With that said, scientists should be aware of confirmation bias and work extra hard to try to set up fair tests that could disconfirm their hypothesis, and not do the more natural thing and set up tests their hypothesis is likely to pass.

Most good scientists have been trained to try to set aside bias, but not many can do it perfectly every time. The good thing is that the scientific community serves to correct for the natural oversights of bias. If you set up a biased experiment, or overlook likely confounders in data, and so on, it will be noticed by other scientists -- especially those who champion a different theory.

Science is not glorified common sense. Science is the result of noticing all the ways we have of fooling ourselves using common sense, and developing rigorous methodologies that allow us to avoid fooling ourselves.

Elliot said...

"If you were to try to eat five rocks, and break your teeth each time, you'd conclude that rocks aren't good food and stop trying to eat them. You wouldn't conclude that you failed to reject the idea that rocks aren't good food, and keep trying to eat them."

This makes no sense.

If your original hypothesis is that "rocks aren't good food" you clearly would conclude that you failed to reject this hypothesis. Because it would be true!

The idea that you would keep trying to eat them is a complete non sequitur-- Why would you? Your hypothesis to be falsified was that they weren't good food in the first place!

Chemical Blender said...

I'm pretty sure that once a hypothesis reaches a level of acceptance (i.e. scientist can no longer think of a way to disprove the hypothesis) it becomes a theory. Scientist call it a theory at the highest level of acceptance to signify that there can always be something that has not been thought of that will cause the theory to be false, which will open up a new paradigm. When you propose a new hypothesis, your experiments should be set up to prove that it is correct. This includes testing for things that would disprove your hypothesis. If you are not doing that, I would say your hypothesis has not been validated. See any epidemiology study these days. There work, at best, can only suggest a starting hypothesis for testing yet their conclusion is that their hypothesis is correct. That is improper science and it is being done all the time. I see it everywhere. During my time as a young tech. working for a member of the NAS, he would constantly bring this issue up. If you have a novel hypothesis, test it, try to disprove it, set up every control experiment you can think of systematically and logically and test them.
I personally agree that you should not waste your time trying to disprove a theory, but a hypothesis should never be published without rigorously trying to disprove it. You also need to prove it, but a both and not an either.
Sadly, this post reminds me of everything that is going wrong in science. An hypothesis is a starting point, not an end result.

Puddleg said...

Karl Popper was a philosopher who never invented anything. His formulation is hardly unassailable.
Others have also made formulas; Richard Feynman is often quoted
http://en.wikiquote.org/wiki/Richard_Feynman
"Science is the belief in the ignorance of experts."

"There is one feature I notice that is generally missing in "cargo cult science." It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it; other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked — to make sure the other fellow can tell they have been eliminated.
Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can — if you know anything at all wrong, or possibly wrong — to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.
In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another."

"A great deal more is known than has been proved."

the whole Wikiquotes page on Feynman, which is very long, is worth reading.

Jane said...

In the years when I was doing experimental science, I never thought in terms of hypotheses. It just didn't seem relevant. I remember my shock when I gave my first talk as a post-doc and the head of the lab said 'how are you going to test your model?'

I was speechless. I had no model. I was simply describing my results. His question meant he had not understood what I was talking about. I did not (and still don't) see how I could have been clearer.

My results were in fact very strange and (to my mind) very interesting. Many years later other people discovered what I had discovered, and there were two articles in Nature with a picture on the cover. Even then, it was not termed a hypothesis, a model, a theory, or anything of that kind. It was just an interesting finding.

David Moss said...

It might be helpful to distinguish between the best way to test a particular hypothesis and other pragmatic recommendations. If you wanted to ensure that you had good reasons for believing Pasteur's germ theory, a good way of doing this would be to try very hard to find evidence that would disprove it. However, given that there are lots of hypotheses and limited resources, it might not be pragmatically advisable to investigate Pasteur's germ theory, because it seems unlikely to produce interesting results and so on. What you ought to investigate overall and how you ought to investigate particular questions are distinct questions.

Also as the WSJ article you cite says, there's also good pragmatic reason to seek very hard to disprove one's hypotheses, to counteract the effect of one's own confirmation bias (and other instances of motivated reasoning)- that's independent of what would be the best way to investigate if one were not in fact biased.

Also it's worth noting that falsificationists don't conclude that if you bite 5 rocks and break your teeth every time, you have failed to reject the hypothesis that that rocks aren't [sic] good food. (For one thing, biting 5 rocks would be a way to test the hypothesis that rocks are good food.) The idea you describe just seems to be that one can never find evidence for anything, ever.

“To decide whether or not a hypothesis (i.e., an idea or model) is correct, we use a "hypothesis test".  Hypothesis tests are based on probability.  The techniques that allow us to do this are called statistics.”
This doesn't actually say anything. The first and last sentences just tell us the names for things. Asserting that hypotheses are tested through “probability”, by itself, is like saying that hypotheses are tested using evidence- not very informative. I think the point you really want to make is that you can support hypotheses through induction (and so: inductive probability), rather than just falsifying hypotheses- which places you in the company of most philosophers of science.

The “tall people, big feet” argument wouldn't have impressed Popper. This is like, if Popper were to ask how we know that all swans are white, answering that we counted up all these swans and 100% were white and that this is statistically significant.

“the hypothesis test rejects the null hypothesis,supporting the experimental hypothesis. We don't say "our results fail to reject the hypothesis that tall people have bigger feet", as we would if every experiment were designed to try to reject our idea. We say "our results support the hypothesis that tall people have bigger feet", because the null hypothesis, that foot size is the same, has been rejected.”

In your mind, what's the difference between “rejecting the null hypothesis” and “falsifying a hypothesis”?
You don't seem to have described an experimental procedure at all distinct from trying to falsify your hypothesis. The only difference is that you insist on adding, “...and this supports our hypothesis” at the end. You can't argue against the WSJ article with examples that equally serve to falsify hypotheses. Simply saying “ah, but I was trying to support my hypothesis, not to falsify the other hypothesis” doesn't achieve anything if these are the same thing.

FrankG said...

You pose a very big question "How Should Science be Done?" and then answer it by relating how, in your own limited experience, you see it being done... are we to assume (as you clearly have) that by default you are doing it right?

Of course the idea is not that you set out to disprove your own hypothesis BUT that does not excuse you from exercising due diligence to test and rule out other possible interpretations of your observations -- ignoring or dismissing these because they are inconvenient, or do not fit in with your own pre-conceived outcome is not the way that science should be done.

My understanding of Karl Popper's Black Swan is not that you CAN disprove you own hypothesis BUT that it at least has the potential
for being disproven.

The Theory of Evolution by Natural Selection is almost universally accepted as "fact" by anyone with a critical mind who does not accept things based on blind faith. When first put forward, there were large gaps in the evidence -- we didn't even know about DNA yet. This despite decades of effort by Darwin (and others) prior to publishing, trying to rule out other possible interpretations of the observations. Nowadays many of those gaps have been filled BUT that is not to say it is irrefutable fact, which could not be turned over by a new discovery tomorrow.

We could say the same about something as substantial as the Newton/Einstein Theory of Gravity... yes it makes sense based on our current understanding of the physical universe BUT do we really lack the humility to question that we just might not truly understand the physical universe?

This is what separates science from faith. YOU need to keep an open-mind to where the evidence leads you, rather than pushing the evidence in the direction YOU want it to go.

Having a PhD does not automatically make you or anyone else a scientist. That requires s certain way of thinking about the world around us.

Unknown said...

Formalized common sense? I am not sure I like that definition because if the counterintuitive nature and nuance of many things in nature. Also, a lot journal articles are simply not that good, scientifically or otherwise. And finally, innovation and results often go against accepted wisdom. People observe a phenomena or develop a technique that has to be explained with new science that challenges old ideas, that may be incomplete or just plain wrong.

David Moss said...

@doctor_ostric

Stephan says that the scientific method is formalised common sense, not that the results of science are common sense. Taking Stephan's example- it is common sense that if you 'experiment' and keep trying to eat rocks and it never works, then rocks aren't good to eat. That's not to say that the results of your experiment were revealed to you directly by common sense, rather than as a result of the experimentation.

This 'science is a rigorous extension of ordinary capacities' line is taken by Susan Haack. The first chapter of her 'Defending Science: Within Reason' is subtitled 'A Critical Common-Sensist Manifesto.'

Interestingly, by taking this line, Stephan is also echoing none other than Karl Popper himself! (http://128.40.111.250/evidence/content/haack.pdf)

FrankG said...

Common sense tells me that the earth is flat and stationary, with the sun circling us overhead. It is only by asking what Jacob Bronowski called "impertinent questions" that we get to see the more complete picture.

These questions might seem childlike in their inquisitiveness "but WHY?" yet unlike a child I will no longer meekly accept the authority of a an adult (or anyone who relies solely on their position) telling me "because I said so... that's why!". Show me the evidence and show me why I should agree only with your interpretation of it.

Björn Axén said...
This comment has been removed by the author.
Björn Axén said...

Stephan, great blog!

Popper's ideas on how science works are most often misunderstood en simplified into nonsens like the dictum of falsification. For a better understanding of the scientific method I recommend this essay by Imre Lakatos:
http://philosophy.ru/edu/ref/sci/lakatos.html

Jimmy Gee said...

Stephan,

Your analogy is inaccurate. Hypothesis testing is typically done by stating a null (Ho) and an alternate (Ha or H1) hypothesis. Statisticians prefer to state "failure to accept or reject the null hypothesis". Acceptance implies that the null hypothesis is true. Failure to reject implies that the data are not sufficiently persuasive for us to prefer the alternative hypothesis over the null hypothesis. This reasoning, in-part stems from the notion that hypothesis testing commonly involves a sample of a population and not every possible constituent. Therefore, it is not appropriate to state the hypothesis is true since there is a probability that your sample is not sufficient.

CarbSane said...

Jimmy Gee,

Statisticians NEVER "accept" a hypothesis. They either reject or fail-to-reject.

JBG said...

It's been forty years since I read Popper, but my recollection is that the main point of the Conjectures and Refutations doctrine was the importance of the generation of fruitful conjectures. The role of refutations is to clear away wrong conjectures promptly so as to open the way for fresh conjectures. Thus, as Juistin said, a hypothesis test should be designed to present as rigorous a challenge to the hypothesis as possible.

At any given time, "science" consists of the hypotheses that remain standing, from extremely secure ones, like the atomic theory of matter, to extremely tentative ones, like the notion that fragrance in bar soap can relieve nocturnal leg cramps (see Peoples Pharmacy).

Note that even longstanding hypotheses, with convincing theories underlying them, can turn out to be wrong. "Inert" gases are a case in point. After some impertinent person produced the first compound involving an "inert" gas a few decades ago, hundreds more were quickly found.

Stephan Guyenet said...

Hi Justin,

I think this is another place where there is a lot of confusion. In many instances, we are literally unable to falsify a hypothesis regardless of experimental design and data. The common statistical tests generally cannot say "X hypothesis is false", all they can say is that we have not identified evidence to support it. So personally, I prefer to think in terms of testability rather than falsifiability.

Hi Jim,

I don't know how I would set up an experiment that would make a hypothesis likely to be supported if it were incorrect, without seriously botching my experimental design. The only way I can think of to do this would be to repeat it 20 times, and only report the one instance where it was significant due to chance.

Hi Elliot,

Yes, you could conclude both, but you WOULD only conclude the former because it's the only useful conclusion, practically speaking. That's my point. My second point is that we don't need to keep testing a hypothesis once we have enough certainty about it, as some of these people seem to be advocating.

Hi Chemical,

You said "This includes testing for things that would disprove your hypothesis. If you are not doing that, I would say your hypothesis has not been validated." Again, in most instances at least in my field it is statistically impossible to disprove a hypothesis. All you can do is determine that it's not supported by pitting it against the null and seeing if it's significantly different from it. That's one of the reasons why I object to this idea that we need to go around trying to disprove hypotheses-- what we do is pit them against a null hypothesis and see which one comes out on top-- not quite the same thing. If the null comes out on top, the hypothesis has not been falsified technically speaking.

JBG said...

Addendum: For the whole thing to work, conjectures need to be framed in such a way that rigorous tests of them can be constructed.

Stephan Guyenet said...

Hi David,

Again, I don't think you could disprove Pasteur's germ theory even if it were false; at most you could conclude that you can't identify evidence to support it because your experimental data are not statistically different from your null hypothesis. In many instances you literally can not falsify a hypothesis, all you can do is fail to support it.

I agree with some of the points you made (including your comments on the rock example), and I'm not making any claims about what true falsificationists would say. What I'm commenting on is how these ideas are abused in the public sphere.

You said "This doesn't actually say anything. The first and last sentences just tell us the names for things (sic)." Come on now David, you're nitpicking me. This is a public blog, and saying the names of things is useful for people who are not familiar with statistics.

Hi FrankG,

Yes, again I am not really criticizing the philosophers of science here, I am criticizing the people who misapply their ideas.

Hi Jimmy,

It is not inaccurate. It is acceptable to conclude that your hypothesis is supported when the null is rejected. You cannot accept the null hypothesis, but you can accept the experimental hypothesis. I understand that some hard core statisticians will say that you reject the null not accept the alternative, but most will tell you that it is acceptable to say that the alternative hypothesis is supported. Otherwise, what would be the point of doing the experiment?

RLL said...

A lot of the bad science I see on blogs relates to the following misunderstandings.

In classical logic, If A implies B, and if B implies C. Therefore A implies C. Non scientists read abstracts as though what was proven fits into the classic logic.

It would help for people to visualise these studies in terms of a Venn diagram.

Under certain circumstances parts of A affect certain things in B. And likewise under certain circumstances parts of B affect parts of C. Obviously seeing it this way one would not say that A implies C.

Jimmy Gee said...

CarbSane:

My issue with quick typing and not editing before posting - yes and actually we reject or fail to reject null hypothesis.

Stephan:

My point is really similar to yours. The point of doing research is to explore, learn, improve etc... However, the inaccuracy was more in reference to your statement "We don't say our results fail to reject the hypothesis..."". Actually, you are failing to reject the null hypothesis and this may or may not support your line of research.

Anonymous said...

I would argue that Bayes' Theorem (not science) is literally a mathematically rigorous formalized version of common sense, and that the scientific method as typically practiced is a special case which is suitable for some specific experiments.

It would be a huge advance to science, and clear up a lot of confusion if scientists would use Bayes' Theorem directly, and avoid the concepts of statistical significance and null hypotheses, which often aren't rigorously applicable to the experiments for which they're used.

Puddleg said...

My hypothesis is that a bridge of a certain length, made of certain materials to a certain design, will span a river and allow traffic to cross
I build the bridge, traffic crosses
my hypothesis has been proven
if the bridge failed it would have been falsified

There is a certainty here unknown by epidemiology
the example of DNA is closer to the bridge
if you can use it to get to other places, reliable, to far away, and for a long time, it is probably true

LeonRover said...
This comment has been removed by the author.
LeonRover said...
This comment has been removed by the author.
LeonRover said...

Well, George, I guess most of us empirical types would opine that nutritional epidemiology is "A Bridge Too Far".

Slainte

Murph said...

I'm having a hard time following this one.

Are you saying that concluding "x hypothesis is false" is bad science, but saying "x hypothesis is incorrect" is ok?

Haven't you been spending an awful lot of time as of late "disproving" the carbohydrate-insulin hypothesis? Or have you been proving that it is unsupported by the evidence?

Sounds like you are just playing word games here.

Bagi Ilmu said...

good...i give you 1000 likes..heheh

David Moss said...

Thanks for your reply Stephan.

Sorry, I assumed that you said that it would be a waste trying to falsify germ theory, "because the evidence supporting [it] is overwhelming" and it is true. The idea that germ theory just can't be falsified is more odd. (Seems straightforward: just look and see that there aren't any micro-organisms...) It had better be falsifiable in principle, if it's to be an empirical hypothesis. If it is, but is simply difficult to actually test/falsify at present, then the considerations against seeking to falsify it are pragmatic ones.

Sorry, but nit-picking is essential in philosophy! The reason for my 'nitpicking' is that it sounds like you're giving an explanation of how science can be done other than by just falsifying hypotheses, but 2 of your 3 statements are just stipulative definitions and so non-explanatory. For example, if I say ““evidence” is that which supports scientific conclusions” then I've not explained how evidence supports scientific conclusions. That sentence would only be informative if we already know what evidence is and can see that that's the thing that supports conclusions. Your other statement, that hypothesis-testing is based on probability, just asserts that falsificationism is wrong: but the very debate is between whether science proceeds through falsification or through inductive probability. So it would be completely appropriate for you to say “I assume that science can proceed through probabilistic induction”- and most scientists and philosophers of science, including myself, agree with you, but it would be misleading to think that anything you've said (including your practical examples) is any evidence against falsificationism.

Anonymous said...

I think this is a great post on a very difficult subject. I do wish you had not used the words "common sense," though. So many things seem to be common sense and yet they are not. If Galileo had accepted common sense he would never have discovered the strange, counter-intuitive phenomenon of heavy and light objects both falling at the same rate. And Einstein would perhaps have not revealed to us how strange time is. I would like to hope that science is something much more than formalized common sense.

general semanticist said...

"The goal of most experiments is not to try to falsify or disprove a hypothesis (which in any case is often impossible), it is to test a hypothesis by pitting it against the null hypothesis. In other words, does the model accurately predict reality when it is tested? This is how it should be. The outcome of many experiments is either a) the hypothesis is supported, or b) the null hypothesis is not rejected, i.e. there is not sufficient evidence in support of the hypothesis. There is often no option "c", the hypothesis is falsified."

I would say the goal of most experiments is to see if empirical results agree with theoretical predictions. Statistical methods are often required to establish the strength of this agreement. The days of 2-valued determinism are gone!

FrankG said...

@general semanticist: "I would say the goal of most experiments is to see if empirical results agree with theoretical predictions."

So I theorize that the earth is flat and stationary, with the sun revolving around it...

I go outside and observe that it does indeed appear to be overall flat -- even allowing for mountains and valleys, there is always a measurable and consistent "up" and "down".. a vertical vs. horizontal orientation. I might use a spirit or water level as my guide and indeed it seems to be so.

I am not being thrown about, nor is there a constant wind; as one would expect to feel on the surface of a rapidly moving object. I observe that, if left untouched, the water in a glass settles to a flat, mirror-like surface. So empirically, the earth (on which I stand) is stationary.

If I closely observe the Sun I will see it rise from one horizon, move across the sky over my head and disappear over the other horizon, with all this taking place in a predictable period time that varies little over days, weeks or months -- I have charts for this stretching back centuries in fact. So an overall circular path for the sun makes the most sense of what is seen.


Job done? Or do I need to consider alternate explanations for what I have observed?

As Richard Feinman (the other) in a recent blog post noted "In nutrition we have recommendation-driven research rather than the other way around. Recommendations are made and then people do the research to support them. Peer review is provided by the people who made the recommendations in the first place."

http://rdfeinman.wordpress.com/2012/07/11/reading-the-scientific-literature-a-guide-to-flawed-studies/

general semanticist said...

@FrankG

Not sure what you are getting at. The proposition that the earth is spherical goes all the way back to the ancient Greeks - and measurements agree with this proposition.

Armi Legge said...

Hey Stephan, great article. Have you read the book "Why People Believe Weird Things" by Michael Shermer? He asserts that skepticism is a method for determining the validity of claims, not necessarily to disprove them. His definition of skepticism seems to fit with the premise of your article. Do you agree?

Thanks for the great work, it means a lot.

- Armi

Anonymous said...

Hi, Stephan :)



This is the best thing I have found on the Internet that clears up all the misconceptions abuot science. I will post it below its from the University of California at Berkely.

http://undsci.berkeley.edu/etaching/misconceptions.php#b11


*There is a MYTH that scientific laws are absolute and not able to be modified, revised or shwon to be wrong

*There is a MYTH that scientific laws and scientifci theories have a heirarchy and that a scientific law is "above" or "better than " a scientifc theory . This is completely false.


*Some poeple falsely believe that science "proves" things.

*Some people falsely believe that science can address the Supernatural and/or the issue of whether or not God exists . It cannot at all.


*There is a myth that " all science can do is disprove."


Karl Popper's falsification is not a very accurate picture of how scientific knowloedge is built.

I link to it at my blog.It's from the Univeristy of California at Berkeley and it is fabulous:

http://undsci.berkeley.edu/teaching/misconceptions.php#b11

Science is great for certain things but it has its VERY considerable limitations.

I hope this link helps your readers. The Internet gurus should read it. It would give them an education that they depserately need.

If more people were scientifically literate the Internet salesmen woukld be out of business for good.

LeonRover said...

R A Z Z Honey,

" depserately " ??

I can empirically see evidence you need to finish your education.

Slainte

Anonymous said...

Leon Rover

You must understand: I have typed the same thing SO many times, I do not care if I have typing mistakes anymore.

Plus, my keys are busted from arguing with morons.

Add the fact I do not have my contact lenses in, as I am giving my eyes a much needed rest from wearing them for 13 hours straight.

I am in the business of putting the Internet gurus out of business by showing the public what they promote is NON- science and that they do not even have a grasp of the fundamentals of what science is.

Jane said...

R A Z Z, I imagine you are the same person as Razwell?

You will never put the internet gurus out of business, and that's as it should be because all of them, in my experience, talk a lot of sense. Perhaps they talk nonsense at times, but if they do, it's just because they haven't read enough of the literature. How can they read it all? Their ideas evolve, just like yours do.

Stephan's ideas have evolved over the past 4 years since I started reading WHS, and I have observed this with utter fascination. This is how things are supposed to be. Anyone with fixed ideas is simply not reading the literature.

Anonymous said...

Stephan is a genuine researcher. He is not included in my Internet guru list at all. My buddy ( very scientifically literate) Urgelt respects Stephan a lot. Urgelt is the guy who set me straight about the scammers out there. Most of the common sites asre loaded with junk non-science.


The salesmen sites are not valid sorces of information. I have already shwon how most SALESMEN Internet gurus have misrepresented the genuine science about obesity.

My information is of the highest quality with detailed lectures by Dr. Jeffrey Friedman who is the real deal.

Anybody smart can clearly see how he Internet gurus have been discredited by the work of Friedman, Leibel, Coleman and Rosenbaum, among others. These gurus do not even accurately relay the information.

Look into this more. Every single biologist and physicist I have spoken with have said the same thing to me about these Internet salesmen selling fat loss books etc.It's unanimous. Their information is junk.Stick to real researchers.

You're correct, however, that I probably will not put them out of business ( even though they clearly have been discredited as their information is completely at odds with the world's obesity scientists) because they appeal to a gullible public who lacks scientific literacy.

have a look at all of my links. That information is far closer to the truth than the Internet gurus. The information is "less wrong."Those scientists are the real dea and excellent.

Don S said...

Another vote against science as common sense, even with the "formalized" caveat. Albeit I understand the desire for a sound bite that captures it enough.

Science instead is a social endeavor of trying to create models that increasingly better predict what will be observed in the future, creating tools to make observations to test those predictions along the way.

Each model is held provisionally, with minds hopefully always open to the model being revised by enough additional observations that do not fit the predictions, or even replaced by a model that both fits past and future observations better. Because science does not accept revealed truths it manages uncertain futures by regulating the degree of doubt we hold in future predictions. That doubt can, in a limit function sort of way, approach zero, be infinitely close to zero, but it never is zero.

Those who make interesting observations, those who theorize and make models, those who make astute criticisms, are all part of that process of doing science. Together they create something that can never be "the truth" but can instead be something better at predicting the future and having control over it than anything else currently available.

Don S said...

I am also not so sure why that WSJ article set you off. It's point is a valid one: confirmation bias is a cognitive trap that scientists can fall prey to just like anyone else. Many of your posting here have been good examples of how to avoid that trap as you provide a very good model to your readers of keeping an open mind to where the data leads whether it agrees with your extant belief system or not ... but many who respond here demonstrate the exact problem that the WSJ writer discusses, believing studies that agree with their extant position and not ones that do not.

Stephen said...

I just read the WSJ article. I don't understand why that got your back up. If you're wary of potential confirmation bias in your own research, and strive to avoid it, then why get so upset.

Unless, of course, the whole point of the post was to attack Gary Taubes. Can't let that one go can you.

Dragan said...

"The scientific method is just a formalized version of common sense."

Frame that quote and sell it, because it's catchy and it resonates. The closes a philosopher of science came to capturing what scientists do is Feyerabend who said "anything goes." Anything as long as it holds up, of course.

Blake said...

Although it is not my current profession, my degree emphasis was in Philosophy of Science.

The one thing that I would like to say about the scientific method is that it assumes that human perception and reasoning is necessarily flawed. Therefore the goal is to take something that seems reasonable and formulate a test to test that reasonableness. This first test will tell you whether or not it is reasonable. Further testing must be conducted to determine if it is not only reasonable but true.

In our society, especially in the area of nutrition, we read the NYT and see that a proposition passed the reasonable test and believe that that's as far as it goes. "See! Meat causes cancer!" Slow down, chief. We need to test this for truth not just reasonableness.

Anonymous said...

The myth of the singular scientific method. This is a pervasive myth. There is really no such thingas the singular scientific method and what gets taught to us in 6th grade is a complete misrepresentation of the process of science This is a link to a reputable site from the University of California at Berkeley. This is the best link on the entire nternet as to how the REAL processes of science work in reality :

http://razzwell.blogspot.com/2012/07/the-myth-of-scientific-method.html

Puddleg said...

The great Rupert Sheldrake (in "7 experiments" and elsewhere) shows that "fundamental constants" have actually varied over time.

http://www.sheldrake.org/experiments/constants/

"In practice, then, the values of the constants change. But in theory they are supposed to be changeless. The conflict between theory and empirical reality is usually brushed aside without discussion, because all variations are assumed to be due to experimental errors, and the latest values are assumed to be the best.

But what if the constants really change? What if the underlying nature of nature changes? Before this subject can even be discussed, it is necessary to think about one of the most fundamental assumptions of science as we know it: faith in the uniformity of nature. For the committed believer, these questions are nonsensical. Constants must be constant.

Most constants have been measured only in this small region of the universe for a few decades, and the actual measurements have varied erratically. The idea that all constants are the same everywhere and always is not an extrapolations from the data. If it were an extrapolation it would be outrageous. The values of the constants as actually measured on earth have changed considerably over the last fifty years. To assume they had not changed for fifteen billion years anywhere in the universe goes far beyond the meager evidence. The fact that this assumption is so little questioned, so readily taken for granted, shows the strength of scientific faith in eternal truths."

Puddleg said...

Anyway, gathering data while watching a hypothesis grow, all by itself, is 99% of the fun of science.
Even if all the glory is in testing it.
Inductive reasoning; just steep in the data long enough and you will see the truth in its fumes.

http://hopefulgeranium.blogspot.co.nz/

michael said...

It's humbling to hear the greats being evoked. It's important to frequently reflect back on the science one does. That being said, there's certainly poor science being done, some of it tied into the design and perspective of the hypothesis. What are your thoughts on skepticism from a priori assumptions and bias in interpreting data Stephen Gould wrote about in Mismeasure of Man? Thanks!

Sanjeev said...

> The great Rupert Sheldrake (in "7 experiments"

good setup.

... still waiting for the punch line.

Anonymous said...

Speaking of science and entropy , thermodynamics etc. - here is a quote from the great physicist Richard Feynman ( he is one of the greatest in history) and something to consider strongly.


"So far as we know all the fundamental laws of physics, like Newton's Equations, are reversible. Then were does irreversibility come from? It comes from order going to disorder, but we do not udnerstand this until we know the origin of the order"

- Richard P. Feynman

This is from "Feynman Lectures On Physics."

I strongly recommend this rather than listening to Internet salesmen speak about thermodynamics and entropy etc.

None of these Internet "gurus" are nowhere near as qualified or as intelligent as Feynman was to speak on matters of physics.


Feynman could TRULY speak on thermodynamics and entropy because he was a genius on the topic of physics.Learn from the best.

P.S. Feynman figured out what happened with Space Shuttle Challenger The guy was extraordinary.

Victor Venema said...

The question Karl Popper wanted to answer is "what is the difference between scientific and non-scientific ideas". The answer is that scientific ideas are formulated so precisely that you can falsify them. This was a very important idea, but it does not tell you how to do science, nor does it say the non-scientific ideas are not important in their own right.

A few comments back, Rupert Sheldrake was mentioned. His ideas on morphic fields are a clear example of ideas that are not sufficiently precise to be science. There is no experiment that could refute Sheldrakes ideas. He could always claim that the effect was just a bit weaker as could be detected by the experiment, because he does not quantify anything. Maybe one day, Sheldrake or someone else will improve these ideas and make them more precise and turn them into a scientific theory.

How to do science? That is hard to say. Best look at how a good scientist works. That is one of the reasons why the master apprentice relation is so important in science. Being critical of your own work is not necessary for science; also someone else can refute your scientific theory (if it was at least falsifiable). Being critical of your own work is good for your career. Being wrong is okay, but you should at least be wrong for an interesting reason. Being stubborn and sticking to an idea when meeting some opposition is often a good idea. Being attached to an idea and having a stimulating argument with colleagues can bring the best out of someone and may lead to proofs one would otherwise never have thought of. This doesn't mean that you should be in denial. Having a good intuition for where this boundary lies, is one of the things which makes a great scientist.

I think the post gives much too much emphasis on statistical testing, which is probably the main problem in nutritional science. Ideally you design experiments such that you do not need statistics. In nutritional science this is difficult as it would require very large groups of test persons, which is expensive. And also understanding the mechanisms is important in deciding between two theories, not just one statistical test on one specific relationship. Then you quickly leave the realm of nutritional science and venture into medicine, biology or bio-chemistry.

Some more name dropping. Thomas Kuhn wrote a good book about what scientist do when the do science, especially about the difficult times when a major shift in scientific thinking occurs.

Imre Lakatos wrote a good book about what scientists do when the do not do science.

Also Feyerabend was already mentioned. "Anything goes" when it comes to coming up with scientific ideas or when one in not superior to another one in all respects. Creativity and intuition are very important in this phase.

general semanticist said...

Very nice post Victor. I would also recommend Alfred Korzybski who wrote Science and Sanity:An Introduction To Non-Aristotelian Systems And General Semantics. He characterizes knowledge as "similarity of structure" between verbal structure (symbolism etc.) and objective structure (experience etc.) A scientist's job is to make this "map" as accurate as possible. One immediately sees that this is a never ending process.

Anonymous said...

Hello Everybody:

Even though this thread is old, I think I can add something of value to it from what I have learned over the years from top sources:




*Most scientific ideas are wrong.

*Most experiments are wrong, too- the first time they are done.

This is very important to understand but it is not conveyed to the public often enough.

(Einstein himself once humorously admitted that a fair share of the papers he authored were wrong, so his colleague should "not worry so much" when they were working together).

Richard P. Feynman stressed we should be very careful to present all the evidence that agrees and disagrees with our idea or theory. To have enormous bend over backwards scientific integrity.

"Science is about figuring out our mistakes."

Saul Perlmutter


Science should be done by thoroughly challenging our most deeply held views and beliefs every single day. In science, we test it. And we keep on testing it until it does not work anymore. And when it does not work anymore, THEN we will know that there is something FUNDAMENTAL we are missing about the science. This principle is powerful and can be used in many areas.

Some principles of General Relativity are tested every single day in undergraduate laboratories.It holds wonderfully.

In science, we force our views to conform to the evidence of reality. We do not force the evidence from reality to conform to our beliefs. This is how many Internet sites are set up etc.- 30 bananas a day , vegans, and what have you etc. These types of salesmen are PRESUMING the answers BEFORE asking the question. This is a huge no-no.Professor Filippenko would have a lot to say about this.

There are no scared truths in science. One of the GREATEST gifts of science is that it makes us very uncomfortable.


We must be especially careful.To spend more time trying to disconfirm our idea or theory than to confirm it... That we are ourselves are the easiest person to fool. Taking tremendous care NOT to fool ourselves is the first principle. Enormous care must be taken.

In science, it helps to "think like a Martian." Feynman talked about thinking about a problem completely anew as if we had never seen it before.

There is lots and lots of junk published every day- even in peer reviewed journals. Considerable amounts of junk get by referees. This system has considerable problems and can be abused and sometimes is. Gate keepers etc. It is not by any means the arbiter of truth. Just a basic check of scientific work meeting the minimal standards.

As Plank once said, "Science proceeds one funeral at a time." Gate keepers, stuffy journal editors do exist. Shenanigans go on in science as Paul Davies points out. All the political crap, back stabbing, ego etc goes on.

In science, what really matters is this:

When somebody else finds your work interesting... And they take it up.... And they perform A1 quality experiments.... And it WORKS.... And then it gets to be done more and more and more and more. Then, and only then, does something become part of the "cannon of science."

Replicated "A number 1 quality experiments "from many different research groups around the world over and over preferably using several different testing methods- all getting the same result represents top quality work. Such is the case with the discovery of Dark Matter. As such, our confidence in the result grew and grew.

Quantum mechanics represents a TOP QUALITY scientific theory. Replicated experiments of A number 1 quality over and over throughout the world.



It is worth noting Isaac Newton's "Principia Mathematica "
was not peer reviewed. Yet, it is some of the very finest quality scientific work ever done. All scientific theories are only approximations with varying degrees of certainty- NONE absolute. :)

Take care, everybody.

Raz