Twitter Defends Fake Account Metrics After Latest Queries from Elon Musk

So, how’s Elon Musk’s Twitter takeover going?

Well, not ideal.

Today, in response to public criticism from Musk that Twitter’s methodology in counting spam and fake accounts is clearly wrong, current Twitter CEO Parag Agrawal tweeted out a long explanation of the company’s current fake profile detection and removal process.

Over the last week, Musk has questioned whether Twitter’s estimation that 5% of all Twitter accounts are fake could possibly be correct, and has invited his 90m+ followers to conduct their own rudimentary experiments to see what they find.

Of course, any anecdotal test like this is massively flawed, and will ultimately prove nothing.

But as with many aspects of the platform, Musk is seemingly learning about these processes in real time, in the midst of a $44 billion takeover, despite all of his various questions being raised and assessed, in various form, in the past.

So, with chatter around this specific element rising, Agrawal addressed Elon’s criticisms directly.

First off, Agrawal explained that detection of bot accounts is not easy, and likely can’t be done by regular users:

“The most advanced spam campaigns use combinations of coordinated humans + automation. They also compromise real accounts, and then use them to advance their campaign. […] The hard challenge is that many accounts which look fake superficially, are actually real people. And some of the spam accounts which are actually the most dangerous – and cause the most harm to our users – can look totally legitimate on the surface. Our team updates our systems and rules constantly to remove as much spam as possible, without inadvertently suspending real people or adding unnecessary friction for real people when they use Twitter: none of us want to solve a captcha every time we use Twitter.”

So randomly counting the accounts that you suspect are fake in your own following list won’t be accurate. Which, really, seems obvious.

Note, too, the reference to performing a CAPTCHA verification process every time you use the app is a direct reference to another suggestion by Musk.

“Each human review is based on Twitter rules that define spam and platform manipulation, and uses both public and private data (eg, IP address, phone number, geolocation, client/browser signatures, what the account does when it’s active…) to make a determination on each account. The use of private data is particularly important to avoid misclassifying users who are actually real. FirstnameBunchOfNumbers with no profile pic and odd tweets might seem like a bot or spam to you, but behind the scenes we often see multiple indicators that it’s a real person.”

In other words, there are many elements that go into detecting and determining fake profiles and spam, not just gut feel, or account info that makes a regular user narrow their eyes. Twitter has the back-end insight to support its investigations, and it’s more in-depth than a simple eye test.

“Every quarter, we have estimated that <5% of reported mDAU for the quarter are spam accounts. Our estimate is based on multiple human reviews (in replicate) of thousands of accounts, that are sampled at random, consistently over time, from accounts we count as mDAUs. We do this every quarter, and we have been doing this for many years.”

That’s interesting, because according to Musk, Twitter only samples 100 random accounts for this figure.

As you can see, Musk then further claims that Twitter contacted him over violating the terms of his agreement on this element – so either Musk is making this up, or Agrawal is exaggerating Twitter’s process.

Again, there’s no smooth sailing in the Elon takeover push.

In summing up, Agrawal says that, actually, based on its most recent reporting, its fake account figure is less than 5%.

“Our actual internal estimates for the last four quarters were all well under 5% – based on the methodology outlined above. The error margins on our estimates give us confidence in our public statements each quarter. Unfortunately, we don’t believe that this specific estimation can be performed externally, given the critical need to use both public and private information (which we can’t share). Externally, it’s not even possible to know which accounts are counted as mDAUs on any given day. There are LOTS of details that are very important underneath this high-level description.”

Musk’s response?

Right. Seems like that’s going to go over well – and again, it’s fascinating to see the internal machinations of a multi-billion dollar deal come down to misinterpretations, and technical interpretations versus an individual’s personal view.

At best, the back and forth would have to trigger some level of concern over Musk’s capacity to transform Twitter into a money-making machine, as per his projections, given that he seems unable to grasp even some of the most basic social media metrics.

But then again, many will also say that Musk is right, that it’s not possible for Twitter’s fake profile count to be so low, based on their own, personal experience, but also, based on published external studies which would clearly suggest otherwise.

Indeed, back in 2017, a joint study conducted by researchers from the University of Southern California and Indiana University found that around 15% of Twitter’s user base were bots rather than people, while in 2018, Twitter suspended or removed more than 70 million accounts which it had determined were fake. At that time, Twitter had 330 million monthly active users, which would suggest that around 20% of its user count were not actual, real people.

In response to this latest exchange, SparkToro conducted a fresh analysis of 44,058 active Twitter accounts and determined that 19.42% ‘fit a conservative definition of fake or spam accounts’.

The key may lie, then, in this element identified in Agrawal’s explanation:

What, exactly, is an mDAU according to Twitter?

“We define monetizable daily active usage or users (mDAU) as Twitter users who logged in or were otherwise authenticated and accessed Twitter on any given day through Twitter.com or Twitter applications that are able to show ads.”

So there could be other users that fall outside of this count – maybe they’re not active every day so they miss the assessment, or they’re not eligible to be shown ads, due to varying system logic. There may be, due to varying parameters, some technical explanations that would address some level of variance between these third-party findings and Twitter’s own data.

But even then, Twitter’s 5% count, on balance, seems like a stretch, and a bending of the actual truth in respect to what people are experiencing in the app.

And as Musk says:

Which, of course, is a fundamental question that’s been raised by every advertiser and every business in digital marketing circles since the inception of the medium, so it’s not like this is breaking new ground, or will come as startling new insight for those in the sector.

But it is, seemingly, fairly new news for Musk himself. Which, again, could raise questions about his massive growth plans for the app.

So what happens now? Can Musk back out of the Twitter deal if, somehow, he can prove Agrawal wrong, and show that Twitter’s fake profile account is actually much higher than it says?

Well, probably not.

Musk, for better or worse, waived a lot of due diligence measures in his Twitter takeover proposal in order to push the deal through faster, which means that, as it stands right now, he’s pretty much locked in, and it would take a significant legal argument for him to walk away from the deal.

It’s feasible, with at least some legal precedent, that if Musk could prove definitively that Twitter had deliberately mislead the SEC in its regular financial statements, via its misreporting of fake profiles, that he could possibly get out of these requirements, with the argument being that the platform is simply not worth what believed, based on its financials. But that seems like a stretch, especially given Agrawal’s explanation.

Maybe, then, Musk is simply pushing to reduce his offer price, in order to get a better deal.

It’s not 100% clear what outcome Musk is seeking, but as it stands, it still looks like Musk will have to go through with the Twitter deal, even if he and his followers think these stats are wrong.

Will we ever find out what Twitter’s actual fake profile count is? Based on Agrawal’s explanation, that also seems unlikely, though the technicalities here seem important, in specifying what’s counted, and what’s not, in Twitter’s user stats.

Either way, the public debate can only be a negative for Twitter, in either reducing advertiser trust in its metrics, or highlighting a long-running issue with its systems.

Is that a good thing for Musk, the future owner of the platform, to be pushing in public? It seems like a flawed approach, especially given the various other options he would have available, in terms of private audits, discussing with Twitter internally, verifying through independent researchers, etc.

Maybe this is just how Musk plans to do things, but it’s not exactly an inspiring origin story for the next stage of the app.