Critical Eye: An example of hype in science publishing and reporting
Critical Eye is a semi-regular feature where RealKM analyses and discusses the methodology and science behind claims made in publications.
Previous articles in RealKM Magazine have revealed the disturbing trend of the hyping of language used in science abstracts, and also looked at the low quality of science reporting in the mainstream media. This follow-on article uses a paper published in the highly respected journal Science to illustrate these issues.
The paper, titled Little Emperors: Behavioral Impacts of China’s One-Child Policy1, was published online on 10 January 2013, ahead of its publication in Volume 339, Issue 6122 of Science on 22 February 2013.
The “Little Emperors” paper received considerable media coverage at the time of its online release. For example, there were articles in BBC News, The Independent, Fox News, LA Times, and The Atlantic. This suggests that a media release was issued to coincide with the online publication of the paper.
Following the 10 January 2013 online release of the “Little Emperors” paper, concerns were raised about its findings in a letter published in the same 22 February 2013 issue of Science as the paper, and in an article published in The Conversation on 18 March 2013.
The letter published in Science, titled China’s Little Emperors Show Signs of Success2, discusses other research with different findings to those of the “Little Emperors” paper, and cautions “against overgeneralization from the economic experiments and personality surveys conducted in this study.”
The article published in The Conversation, titled Is China’s one-child policy really to blame for personality changes?, questions the strong claims made in the paper on the basis of “merely suggestive evidence from a single study on a narrow sample.”
How the “Little Emperors” paper is hyped
Is China a land of “little emperors” whose parents dote on them exclusively?
The main text of the Little Emperors: Behavioral Impacts of China’s One-Child Policy paper begins with the two sentences:
China’s One-Child Policy (OCP) restricts the number of children that urban couples can have to one, with exceptions for those from ethnic minorities or with a severely disabled child. The policy has given rise to a land of “little emperors” whose parents dote on them exclusively.
The first sentence indicates that the one-child policy (OCP) applies only to urban couples, but no information on the relative proportions of urban versus rural couples in China is given, and the second sentence then hypes this by indicating that the OCP has affected children right across the land, turning them into “little emperors”.
The English language term “little emperor” is a literal translation of the Chinese term 小皇帝 (xiǎohuángdì), which is used as a slang label for a spoilt child. As well as being used in the opening sentences, this slang label is used at the beginning of the title of the paper. If a research paper was to similarly use a slang label from another country, for example the “redneck” label from the United States or “bogan” label from Australia, there would be understandable outrage. Appropriate terms for use in a research paper would be “only child” or “singleton“, but they are of course less likely to be appealing to the media than the hyped emotive term “little emperors”.
The hyped emotive phrase “a land of “little emperors” whose parents dote on them exclusively” creates a mental picture of children not mixing with other children, but rather spending every waking hour that they are not at school being tirelessly waited on by their father and mother.
But there’s just one problem with that. It’s not what actually happens in reality. As well as many couples having two children due to exceptions allowed or granted under the OCP, China is a highly collectivist culture where the concept of family extends beyond parents and children to encompass grandparents, aunts and uncles and their children, and neighbours and friends and their children. Children who don’t have siblings still socialise extensively with their cousins and the children of neighbours and friends.
Further, because of the intense study demands of preparing for 高考 (gāokǎo), the highly competitive national higher education entrance examination, Chinese teenagers board at their school for the three years of senior high school (years 10, 11, and 12). This includes students living nearby, who will only go home to be with their parents for one or two days on some weekends. Vocational high school students also board at their schools, and all students will continue to board through their university or college years including students who live nearby. The students live in dormitories with between 4 and 8 students in each room, and spend all of their time together – in class, studying, dining, and recreating.
So, rather than being at home and being doted on by their parents, older teenagers and young adults actually spend most of their time an environment where they are being highly socialised. This aligns with the findings of other research discussed in the letter published in Science that had raised concerns about the findings of the “Little Emperors” paper:
In longitudinal studies among the first post-OCP cohort – the same cohort as that studied by Cameron et al. [in the “Little Emperors” paper] – some singletons showed more behavioral problems and less independence in childhood. Yet, by adolescence, differences in behavioral problems disappeared and independence levels reversed.
The letter concludes with this advice:
In such complex systems as that of human psychology and behavior, one must move beyond linear notions of causality. Circular processes of self-correction at the individual, family, and social levels often provide surprising compensatory responses to initial conditions.
Then there’s the situation of left-behind children (留守儿童). These children, located mainly in rural areas, are cared for by just one parent or their grandparents while one or both parents works in the city:
According to a survey of the National Women’s Federation, the number of such children now stands around 58 million, accounting for 28.29 percent of all children in rural areas.
Rather than being doted on exclusively by their parents, left-behind children will only see their parents once or twice a year. The negative impacts of this are potentially far more serious than any potential behavioural impacts of the OCP:
The left-behind children are going through a critical period of growth and development. The lack of parental attention and care will lead to various problems, even posing potential threats to their safety and psychological health. Worse still is this may propel them towards a life of crime.
Are the so-called “little emperors” of significant concern to Chinese people?
After introducing the notion of “little emperors”, the main text of the paper then puts forward evidence to support “the observation that these children tend to be more self-centred and less cooperative”, stating that:
This can be seen in developments such as employers including phrases like “no single children” in job advertisements. In March 2007, 30 delegates in the Chinese People’s Political Consultative Conference (CPPCC) called on the government to abolish the policy. Their concerns centred on “social problems and personality disorders in young people.”
However, these statements are also hyped. No information is given in regard to what proportion of job advertisements include phrases like “no single children” – is this practice widespread, or are such occurrences just isolated incidents? This would need to be determined from a detailed analysis of job advertisements across China over a period of time, but the number of only children who are gainfully employed across China tends to suggest that it’s not as significant an issue as indicated by the statement in the “Little Emperors” paper.
Further, no information is given about what proportion 30 delegates constitutes of the CPPCC membership, or in regard to what the government did or didn’t do in response to the call by these delegates for the abolition of the OCP on the basis of it causing social and personality problems. Information on the number of CPPCC members participating in the March 2007 meeting is readily available, so could have been included in the paper. There were 2,267 registered CPPCC National Committee members for the March 2007 meeting, meaning that 30 delegates constitutes just 1.3% of CPPCC National Committee members. This suggests that public concern about the so-called “little emperors” is nowhere near as significant an issue as indicated by the statements in the “Little Emperors” paper.
The Chinese government has since moved to abandon the OCP, but because of imbalances in the population structure and not as the result of any perceived behavioral issues associated with children born under the policy. As a result of the OCP, China is faced with the challenge of a decreasing labour force supporting an increasingly aging population.
Do the conclusions of the paper match the evidence?
The “Little Emperors” paper uses data from economic games conducted with people born just before and after the introduction of the OCP in 1979 to conclude that:
China’s One-Child Policy has produced less trusting, less trustworthy, more risk-averse, less competitive, more pessimistic and less conscientious individuals.
However, the article published in The Conversation finds a gap between this emphatic conclusion and the evidence. The article alerts that in the “Little Emperors” paper:
The term “causal” is used five times and “impact” ten – these are bold terms, describing a relation that is much more precise than mere association.
But the article finds that the results are actually not this definitive. Firstly, the significance of the results is marginal:
…when controlling for relevant variables to rule out alternative explanations, four of the five differences in behavioural outcomes (trust, trustworthiness, risk-taking, competitiveness, but not altruism) are labelled “significant”, but using a criterion of 10%.
Had the widely-accepted 5% cut-off been used, two of the outcomes (trustworthiness and competitiveness) would not have been labelled significant.
Secondly, the OCP is not the only potential explanation for the concluded changes in behaviour. The introduction of the OCP in 1979 occurred at a time of widespread and significant change in China. The Cultural Revolution had ended in 1976, and from 1978 China embarked on a program of significant reform and opening up.
The article concludes that the hype does not match the reality:
Strong claims based on merely suggestive evidence from a single study on a narrow sample will convince nobody with influence, and perhaps only offend a population of young Chinese-born individuals who must wear the “little emperor” label.
Did the media practice investigative journalism?
Rather than investigate and critically analyse the conclusions in the “Little Emperors” paper, the mainstream media has simply spread the hype that would have arrived by media release in their inboxes. For example, the article in The Independent states that:
…now, scientists have produced the first convincing evidence to suggest that the one-child generation of China has indeed become a rather maladjusted lot.
Some media agencies furthered the hype through the use of emotive words. For example, in the article in the LA Times:
In China, the legions of children who commanded the undivided attention and resources of their parents have long been viewed with suspicion. But as they matured into adulthood, the “Little Emperors” could afford to roll their eyes at their fretting elders: Western research has consistently shown that only-children — singletons, as demographers call them — are no more selfish, lazy or maladjusted than their peers with siblings.
But China’s elders, apparently, were right.
Other media agencies furthered the hype through the use of emotive imagery. For example, the article in The Atlantic:
Beware of the hype in science publishing and reporting
Most people reading any of the media articles about the Little Emperors: Behavioral Impacts of China’s One-Child Policy paper or the paper itself would have formed the view that there is strong and definitive evidence that China’s one-child policy has produced significant negative behavioural changes. These readers would also now incorrectly believe that China is “a land of “little emperors” whose parents dote on them exclusively.”
However, while the “Little Emperors” paper has identified behavioural indications worthy of further investigation, “The conclusions … are not yet proportional to the evidence,” as stated in the article published in The Conversation.
As previously discussed in RealKM Magazine, the hyping of science in this way undercuts trust, and the tendency for science and the media to be sensational with what they publish potentially contributes to science denial.
When considering research findings reported in the media and scientific journals, beware of the hype. To assist, here are twenty tips for interpreting scientific claims.
Also published on Medium.
I totally agree with your reservations about the article “Little Emperors: Behavioral Impacts of China’s One-Child Policy”. From my own private and professional experience in China, I can only confirm your analysis. Unfortunately, I am a little more pessimistic in my conclusion. You write that “the hyping of science in this way undercuts trust”. My opinion would be that with most readers, such articles which are based on an extremely biased use of available data are extremely popular, because they arrive at clear-cut conclusions to which the reader can relate from his own immediate experience.
In Western countries, only children have got the reputation of being more egocentric and less social than children with siblings. In China, for the reasons you describe very will above, things are far from being that simple (and they are probably not that simple in Western countries either). However, if a research project pretends that it can confirm such simplistic preconceptions, most readers will trust it much more than if it reaches a more nuanced conclusion or even challenges the views of the reader.
In addition, each research project which “shows” that the Chinese development model is heading for disaster is always highly welcome. That’s the reason why for so many decades now, the experts whcih predict the imminent breakdown of China get massive media attention, whereas those with a more differentiated analysis tend to get sidelined.
In the short term at least, such articles will therefore allow many readers to get a more personal and direct relationship to academic research. Of course, in the long term, or with readers who have got the opportunity to verify the situation on the ground, things are different. Such hypes might very well be responsible for the decline in trust in the mainstream media which has become alarming in recent years. It is therefore important to put in place mechanisms to limit such cases of hyped or biased research.
Some articles on my blog show that in the field of the Chinese socio-economic development, extremely biased research is even more frequent than with regards to social phenomena:
Many thanks Otto for your insights. This raises the issue that beyond hyping in general, there are indications of bias. It would be interesting to know if this spreads beyond China, and to explore its basis and determinants: Is it a developed country bias against developing countries, a political bias, both, or are there other factors at play? To what extent is in intentional or unintentional?
In a previous comment on the subject I proved euphemistic when saying that unless we adhere to ethos, pathos and logos we present pseudoscience as science and this is a disservice to society. I should have been more accurate to point out that dishonest scientists and institutions that champion them (colleges, journals, and “science ” writers ) compound the problem and degrade the quality of science. It is not the figures that lie, but the liars that figure.
Honest research that points out insignificant findings does not lead to academic promotions or to publications in prestigious journals.