Is Significance Significant?
Mark Pottenger
I recently read a few articles on questions about statistical significance in a psychological journal Mom loaned to me (Theory & Psychology Volume 8 Number 3, June 1998). They raise some interesting questions, which I believe astrologers should also pay attention to, especially since many attempts at astrological research take psychological research as a model.
The set of fairly short articles by several authors contains only partial agreement, but the articles and their many citations make it very clear that this is a longstanding question for researchers and philosophers of science. The reading is a little slow for people like me who are not statistical experts, but I think it is worth the effort for anyone with access to a good University library that would carry the journal.
The key thought I carried away from the reading was that statistically significant results do not, in and of themselves, have any predictive value regarding what results you should expect in a replication study. This is because all statistical calculations that give measures of significance include infinity somewhere along the way, e.g., this number is what we would expect after an infinite number of replications. Once you bring infinity into the picture, any level of significance could still be “by chance”.
Significance in the direction of a theory’s prediction is useful, though there are apparently wide variations in how useful it is thought to be. Some very interesting distinctions come into the discussion, such as: are you trying to measure the significance of the data given the hypothesis or the significance of the hypothesis given the data?
Statistical significance without a theoretical framework is clearly meaningless. Since conventional science refuses to accept the premises of astrology as a valid theoretical framework, this helps explain how some scientists and/or skeptics can keep ignoring astrological studies that produce massive statistical significance. Single significant results have much less meaning than multiple results based on the same hypothesis or theory. This is one way pseudo-skeptics try to trivialize the Gauquelin work—by calling it “The Mars Effect” as if it were only a single study instead of dozens of studies with different subject groups and highlighted planets.
The issue also reinforces how important it is for any would-be astrological researcher to have a hypothesis before looking at any data. Just fishing for “significant” numbers without a theoretical reason to expect them is close to scientifically useless.
All of us who would like to study astrology “scientifically” need to learn a lot about what “science” is.