Caveat lector

A recent discussion on a friend's wall involved a 2014 survey finding that 92% of Saudi Arabians believe "Islamic State conforms to the values of Islam and Islamic law."

Wei Hong: As you know, study results depend not only on reality but also on methodology. It is therefore important to examine how questions were phrased and percentages calculated. Further details of the Al-Hayat survey do not appear to be readily available, other than that it was conducted via social media, a method known to introduce large selection bias. For now, the extraordinary claim of 92% does not appear to have commensurately extraordinary evidence.

My friends argued that it is acceptable to use suboptimal evidence as long as one is aware of the limitations, correctly pointing out that this is often a practical necessity.

Wei Hong: While there does exist a necessary epistemological continuum, it does not actually follow that poor evidence must be better than no evidence. Medical researchers recently demonstrated this truth by intentionally using poor methodology to find that patient outcomes in a cardiology trial (coincidentally named ISIS-2 too) varied significantly according to astrological star sign. Even without studies showing the contrary, it does not follow that doctors should pay more than zero heed to astrology. While a poorly designed study can show essentially any result, we must remember that such result may not only fail to show truth, but also succeed in showing anti-truth, such that poor evidence can also be worse than no evidence. (Those who suspect that a large proportion of Saudi Arabians supported Islamic State in 2014 should also consider that Saudi Arabia has participated in air strikes against Islamic State since 2014.)