The Offline Network: Maybe permanent uptime isn’t a Web3 thing after all

Crypto and blockchain inherit a lot from the big-tech centric Web2, including semi-regular downtime.

On Oct. 4, 2021, Facebook went offline for about seven hours — the same amount of time that the Telegram-linked blockchain TON just spent offline.

TON was apparently crashed by a memecoin airdrop, DOGS, with a canine mascot that’s said to have been created by the arrested-but-not-yet-released Telegram CEO Pavel Durov. 

Facebook’s October 2021 outage was arguably its most high profile to date. No Facebook, no WhatsApp, no Instagram — Web2’s flagship apps were dark to the outside world. 

On the ground, devs and other problem solvers were even locked out of tools they needed to work and communicate. Some employees couldn’t even enter company buildings or conference rooms, as the authentication systems for scanning badges had also been knocked out.

Read more: Solana price is getting better at shaking off major outages

Facebook had somehow disconnected itself from the internet. Even though it had worked to decentralize its physical operations from the broader internet — a single bug still brought it all tumbling down.

Tech giants like Google, Amazon and, of course, Facebook famously build their own internets. An internet within an internet, so that they can stay online without relying on anyone else. 

They also spread their data centers all over the world, making it possible to reroute traffic to different destinations at times of high load in particular areas (and protect against natural disaster swallowing all our personal data).

With this in mind, Facebook, WhatsApp and Instagram were still there — technically — it’s just the internet had no way of knowing how to reach them.

The culprit was a configuration error in software used to route traffic to and from Facebook data centers, Border Gateway Protocol v4, also described as the “postal service of the internet.”

TON’s recent outage could be a similar situation. But without a technical post-mortem, we’re mostly in the dark as to what caused it.

We know that the DOGS airdrop triggered a spike in onchain activity which knocked validators out of consensus. That meant they couldn’t agree which block should be added to the chain next.

When the airdrop was activated, transactions per second surged to 280 at most, which seems to be around the typical breaking point for the network — much lower than the 100,000-plus marketed by TON. 

The flatlining orange area shows when the TON outage occurred (Source)

A prior incident in December occurred at around 300 transactions per second resulting from inscriptions spam.

In this instance, validators coordinated Wednesday morning to restart the chain, as was the case with Solana’s multiple reboots over the past few years. 

It’s unclear whether centralization was a factor in the TON outage. Say, a breakdown in logic buried somewhere in the validator client, hypothetically fixed if there were more validator clients built by different teams — particularly ones that can handle higher volumes.

“The short answer is ‘tech is shitty, could be better,’ Mikko Ohtamaa, early Bitcoin developer and cofounder of Trading Strategy, told me this morning. He’s been tracking these sorts of snafus and posting about them on X. 

Read more: Latest DeFi exploits show audits are no guarantee

“The root cause is poor developer culture — distribution is more important than tech in the short-term. TON has had several security audits, none published. The first step would be to fix this,” he said.

Decentralization is such a vague term that it almost says nothing about the state of blockchain tech stacks. It’s all a spectrum, and something can only be decentralized in comparison to another thing.

At the same time, there are blockchains out there much more resilient to downtime than TON and even Solana (the big two being Bitcoin and Ethereum). 

That TON dumped only marginally in response to its downtime, crossed with the fact that negative reactions to Solana’s own crashes are becoming less intense over time, suggests the market trusts in those teams’ abilities to eventually fix the problems.

In the meantime, decentralization for much of the Web3 space is mostly about control. Whether the network or protocol can be hijacked by any one particular entity or group — not whether it’s immune to going dark. 

Maybe in Web4.

A modified version of this article first appeared in the daily Empire newsletter. Subscribe here so you don’t miss tomorrow’s edition.

Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.

Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. Subscribe to the On the Margin newsletter.

The Lightspeed newsletter is all things Solana, in your inbox, every day. Subscribe to daily Solana news from Jack Kubinec and Jeff Albus.