Expected Preparations:

  Inquiry:
The scientific method; Evidence based reasoning; How to design, execute and document an experiment; Conjecture, hypothesis and theory.
  
  If you are not already familiar with the prior knowledge listed above, you need to prepare yourself from other information sources.  
Keywords: Cargo Cult science; Cargo Cult bioinformatics
Objectives:

This unit will …

  • … introduce the metaphor of a “Cargo Cult”, as applied to bioinformatics and science in general;

  • … illustrate with examples;

  • … discuss principles to avoid the problem.

 Outcomes:

After working through this unit you …

  • … can identify “Cargo Cult”-type issues;

  • … have contributed an example to our collection, and/or;

  • … have contributed to the discussion in our collection;

  • … are able to critically evaluate projects and activities regarding whether they can contribute to thier claimed value;

  • … are able to propose improvements.

Deliverables:

Time management: Before you begin, estimate how long it will take you to complete this unit. Then, record in your course journal: the number of hours you estimated, the number of hours you worked on the unit, and the amount of time that passed between start and completion of this unit.

Journal: Document your progress in your Course Journal. Some tasks may ask you to include specific items in your journal. Don’t overlook these.

Insights: If you find something particularly noteworthy about this unit, make a note in your insights! page.

Evaluation:

NA: This unit is not evaluated for course marks.

Contents

Not all activities lead to valuable outcomes and “Cargo Cult Science” is an important metaphor for a class of conceptual problems that are the hallmark of “poor science”. These are activities that are not causally connected to their claimed beneficial outcomes.

 

The concept of Cargo cult science(W) was popularized by Richard Feynman(W) in his 1974 Caltech Commencement address. In a nutshell, Feynman pointed out how scientific practices that lack “scientific integrity” are similar to the activities of a premodern spiritual cult in the South Sea islands that developed rituals for attracting goods-bearing supply airplanes by building mock airports.

The essence of Cargo Cult is not merely poor science. What makes it “Cargo Cult” is a disconnect between form and contents: the form is compelling, but there can’t be a rational expectation of a benefit from the activity because there is no causal connection between the activity and the claimed outcome. This is often, but not always due to logical fallacies(W).

The topic is interesting for bioinformatics because the deficiencies are often subtle, and hard for the non-expert to spot. To guard against Cargo Cult takes integrity, and practice. A structured approach may be helpful that first clearly identifies the hoped-for outcome, then defines the proposed activities, then asks in specific detail how the outcome would be caused by the activity. Causation is key here - many examples of Cargo Cult behaviour are based on a mistaken belief in causation, where actually merely a correlation was observed. But you have to be careful: the fact that causation has not been demonstrated does not prove it is absent. And even if causation is absent, that does not automatically make the behaviour invalid: sometimes you are right for the wrong reason. Both cases are not Cargo Cult. Rather, it is characteristic for situations we should label as Cargo Cult that there “is no cargo in the system”: you are looking in the wrong place, you don’t have a control or reference value, you don’t understand your data - or similar problems.

 

Examples1

In 2018, an anti-discrimination lawsuit claimed that Harvard University discriminated against Asian-American student admissions. Impossible, argued a talk-show host: Asian-Americans make up only 5.8% of the US population, yet contributed 22.1% to the Harvard admissions class of 2021 - it would rather seem that Harvard favours Asian-Americans. Right?

Not so. Many factors contribute to admission decisions, but most fundamentally, one would need to consider the percentage of Asian-Americans among the applicants, not in the general population. As an extreme example: if 99% of applicants were Asian-American, but only 22% had been admitted, that would be hard to explain.

The Cargo Cult here is to conduct a statistical analysis - complete with barplots and percentage counts - with an irrelevant control group. That control group contains no information about applications, and nothing about applications can be learned from its size.

(contributed by Rachel Silverstein, BCB420-2018)

In my highschool, we had a metal detector at the entrance door. However apparently, it was not properly set up, since it would make a sound every time someone passed through it. Additionally, there was no security guard to control this detector and check people entering and exiting school. Finally, there was a lot of space between the walls and the detector, so anyone could go in-between them (what most people did to avoid the sound), which killed the very purpose of having a metal detector in the first place.

(contributed by Anastasia Minenkova, BCH441-2017)

[…]James Priess was an embryologist who came up with a clever experiment that shifted a scientific paradigm of his time, and unpacked some of the cargo.

Left to their own devices, nematode embryos will develop in much the same way every time. There are a few asymmetrical divisions, resulting in “founder cells”, and these go on to become all the different and interesting parts of a worm. In particular there is an early cell division that results in two sister cells, one anterior and one posterior, named ABa and ABp respectively. Priess found that if the sister cells were physically rearranged (swapped spots) with a blunt needle before they differentiated further, each had the potential to become ABa or ABp. The orientation of the embryonic axes would be reversed, but the embryo would give rise to an entirely normal worm.

This was interesting because at the time, it was thought that these cells “knew” their fates, and developed accordingly. There was very little attention paid to the importance of cell signalling in determining cell fate. What Priess showed was that the sister AB cells were identical, and the differences between them were due to the signalling with other cells they were in contact with. This resulted in deeper investigation into cellular context as a factor in determining cell fate.

The previous cargo cult belief that every cell has its fate programmed into it likely arose because every wild type nematode embryo has a very consistent development. Each cell seems to be arranged in the same way no matter what. However, this doesn’t take into consideration that the arrangement of the cells determines their identity.

(contributed by Caitlin Harrigan, BCH441-2017)

An elderly relative of one of our students read about a Japanese village, where residents have pumpkin, not rice, as the main source of carbohydrate. Indeed, a research study was able to find an anti-diabetic effect of the pumpkin metabolites Trigenolline and Nicotinic acid. So the elderly gentleman decided to eat lots of pumpkin with every meal - and promptly found his blood sugar levels completely out of control.

Even if pumpkin is less diabetogenic than rice, eating too much of it will elevate your blood sugar. Pumpkin is not a magical, sugar-nullifying compound. Many things that are supposedly “good for you”, when inspected carefully, are promoted based on similar pre-modern misunderstandings that don’t properly understand cause and effect.

(contributed by June Cai, BCH441-2017)

There was a widespread belief in Korea that eating mackerel - especially the dark meat - would make children smart. Mackerel contains a high amount of DHA (Docosahexaenoic acid). The argument at the time that was aired by several news stations was that because approximately 50% of our brains is made up of fat and its majority content is DHA, eating more DHA would make our brains “healthier”. I was only 6 years old at that time so I didn’t know what this all meant obviously so I used to eat 3 mackerels every day for many years.

As it turns out, DHA consumption predominantly affects the developing brain of a fetus, not so much the already (relatively) developed 6 year old child. And indeed children who were breastfed with a high content in DHA had a higher IQ than those who were fed with a low amount of DHA.

The news stations in Korea 15 years ago reported a valid method for becoming smarter (consuming a lot of DHA by eating a lot of mackerel) but they did not understand how or who this method applies to. What they stated was not wrong - DHA does affect brain development. However, it was targeted mostly towards the wrong people. Because of the lack of information, I was forced to eat a lifetime’s worth of mackerels and my parents were not able to reproduce the result (which was making their children smarter).

(contributed by Juliana Lee, BCH441-2017)

Further Reading

Wikipedia: List of Logical Fallacies is a very comprehensive resource. One would wish that the presence of such a list itself would have a beneficial effect on science.

Questions, comments

If in doubt, ask! If anything about this contents is not clear to you, do not proceed but ask for clarification. If you have ideas about how to make this material better, let’s hear them. We are aiming to compile a list of FAQs for all learning units, and your contributions will count towards your participation marks.

Improve this page! If you have questions or comments, please post them on the Quercus Discussion board with a subject line that includes the name of the unit.

References

Page ID: FND-Cargo_Cult

Author:
Boris Steipe ( <boris.steipe@utoronto.ca> )
Created:
2017-08-05
Last modified:
2022-10-11
Version:
1.3
Version History:
–  Removed Student Wiki task
–  1.2 Maintenance
–  1.1 Add references to logical fallacies; review older submissions and move some into Not (quite) Cargo Cult page.
–  1.0 Completed to first live version
–  0.1 Material collected from previous tutorial
Tagged with:
–  Unit
–  Live
–  Has lecture slides
–  Has further reading

 

[END]

  1. These examples were compiled and commented on by previous classes. If you have come across an interesting example of your own, please share it on the Forum.↩︎