“Half of negative tweets about the Last Jedi were the works of Russian trolls”. It’s the breaking news of the week since researcher Morten Bay released his preprint paper. But aren’t we a little quick in jumping to conclusions?
I’m a Star Wars fan since the Empire Strikes Back (yes, I’m that old). So when I read, this week, that research showed that half of the negative tweets on the movie The Last Jedi were the works of Russian trolls, I was all worked up. Then I started to think it over…
The dark side of research and controversy
The paper is a pre-version of a study to come by Morten Bay. It reads: Weaponizing the haters: The Last Jedi and the strategic politicization of pop culture through social media manipulation.
Why study The Last Jedi?
Because it brought about a massive dispute between Star Wars fans. Many thinking it was a disgrace to the franchise and sharing that thought loudly. Obviously, we will not discuss here the movie itself but rather the conclusions of the study.
In the paper, one can discover the method used by the author. To begin with, only tweets addressed directly to Last Jedi director Rian Johnson were studied. Needless to say, this represents a very small fraction of all tweets on the subject. Over a seven months period, this added up to 1273 tweets. After getting rid of irrelevant ones, the volume was down to 967 tweets.
To get a sense of scale, last week alone the hashtag #theLastJedi was used over 750 times (this excludes retweets).
Based on feeling (!), the author identified 206 negative tweets. 61 of which were posted by real people with a political agenda. 11 tweets were traced to bots. And of the 33 trolls who took part in the discussion, 16 were Russians.
The jump to lightspeed conclusions
Both the method and the conclusion are hazy. The amount of actual Russian trolls in the debate is only 3,4% (33 out of 967). So how did it end up on the news as 50%?
Let’s ponder on the abstract of the paper:
The results of the study show that among those who address The Last Jedi director Rian Johnson directly on Twitter to express their dissatisfaction, more than half are bots, trolls/sock puppets or political activists using the debate to propagate political messages supporting extreme right-wing causes and the discrimination of gender, race or sexuality. A number of these users appear to be Russian trolls.
This abstract does reflect the data above. Indeed, over half of the 206 negative tweets were the works of – in order of importance – political activists, bots and trolls. And only some of them are Russian trolls.
But many must have stopped at this abstract and not confronted the data. They only kept “half” and “Russian”. It was far more selling than “sample” and “insufficient”…
A lesson for Padawans
This episode and Morten’s research teach us – Internet Padawans – an important thing.
Scientific research is often used to prove a point or to write a selling headline. Not only can science be biased but stopping at the abstract can lead to vast misconceptions. Retweets do the rest.
We are now experiencing the effets of the Internet’s resonant chamber on a study which may not deserve to be taken at face value. This should be a lesson for all (myself included) who are not immune to sharing incomplete data.
May the Force to resist catchy headlines be with you all