Facebook global outage

[SirDice]

I have a whole bunch of social media accounts. Don't really use them though, I just created them so my "name" is reserved and nobody could impersonate me. I do use Facebook but mostly to keep in touch with my parents and some friends. The only thing I've been recently posting are pictures of my cats (you're supposed to post those on the internet ? ). And I couldn't post new ones for 6 whole hours! My whole world collapsed! Seriously though, I barely noticed it. Did read a lot of news stories about it, that was hard to miss, it got plastered all over.

 

[teo]

The worst thing knowingly those applications or systems are intrument of mass espionage as (facebook, whatsapp, instagram, google, microsoft) that collect personal data, trade personal data at will, we still continue to use. It is time for our relatives to use other secure applications such as Telegram or Signal, and thus safeguard something, and are not at the mercy of hackers, because oligopolies do not give us a penny on the dollar.

 

[ralphbsz]

The United States are a really scary place...

It's not 100% clear that Texas is really in the United States. I mean, legally it technically is ... but emotionally, it doesn't quite feel like it.

It's somewhat like Bavaria and Germany: When in Bavaria, you know that you're primarily in the "Free state of Bavaria" (Freistaat Bayern), and only secondarily in Germany.

Now, seriously about the power grid: People from Europe have to recognize that some of the isolation is also simply due to the distances involved: From Houston, Texas to New York is about as far as from Madrid, Spain to Warsaw, Poland. Also about the same direction (northeast).

 

[eternal_noob]

It's not 100% clear that Texas is really in the United States. I mean, legally it technically is ... but emotionally, it doesn't quite feel like it.

Yup.

I have a whole bunch of social media accounts. Don't really use them though, I just created them so my "name" is reserved and nobody could impersonate me.

I use a different name on every platform i am active on (3 forums, 1 Github). So that one can't link my identity to my interests.

On topic: I find it hilarious / alarming / disturbing that Mr. Zuckerberg lost $7 billion just because his services weren't working for 6 hours. Society is rotten.

 

[Geezer]

I use a different name on every platform... So that one can't link my identity to my interests.

How many hybrid duck-squid do you think there are in the world?

 

[eternal_noob]

I am unique. In every way.

s/squid/Cthulhu/g

 

[jbo@]

Now, seriously about the power grid: People from Europe have to recognize that some of the isolation is also simply due to the distances involved: From Houston, Texas to New York is about as far as from Madrid, Spain to Warsaw, Poland. Also about the same direction (northeast).

I appreciate that input. I do tend to remind myself of that but whenever I read something like this internally I still go "uuuh". I live in a small land-locked country in central Europe. I believe the longest border-to-border distance is something just above 300km.

I use a different name on every platform i am active on (3 forums, 1 Github). So that one can't link my identity to my interests.

Given that you're a FreeBSD user I take it that you know that profiling does not stop at comparing user names

 

[Deleted member 30996]

Not FUD. Search for "Carrington Event".

If it happened now it would take years to replace all the transformers that it would fry. It's supposed to be something that happens in a cycle and that not supposed to the first time it occured

"We will have lost nearly 40% of of our magnetic field strength by 2030. If earth continues to lose 5%+ per decade of magnetic field strength during this solar maximum, it will be less able to hold energy from a large solar CME away from the surface mantle and core of earth. Things will melt. A Carrington sized solar CME (coronal mass ejection), which melted the electric wires and components of 1859, will have an exponentially greater impact on earth with our currently depleted, still weakening field strength.​

The Carrington event was not the largest ever recorded and is in the middle range of historic extreme CME events​

The Carrington event is a documented, once in every 150 to 200 year cyclic solar event, of that large size, or magnitude for you science terminology fans. Do the math as we approach the 200 year mark. There is currently no developed theoretical or practical shielding to protect electrical components, if solar electrical forcing is powerful enough to reach into earths core, sending a wave of energy then back through the mantle into the crust outward."​

Solar Kill Shot? Cyclic 200 Year Super Flare, To Hit Our Exhausted Magnetic Field

IRUUR1 writes for insiders. Globalists and individualists alike, who are aware of the fundamental underpinnings of the emerging centralized system, practical options for necessary, evolutionary decentralization, and are curious about the natural envi...

www.publish0x.com

Col. Ed Dames, a remote viewer for the CIA, predicted "The Killshot" in 2012. Then new age noodle noggins got involved and started talking nonsense. Ed Snowden said the CIA was aware of the cycle.

NASA and NOAA Space Weather Satellite Data Updated Daily:
Solar Storm Monitor

 

[hitest]

Agree. I had a fleeting interest in social media but then I saw what it was/is. I have had a Twitter account 2x and rage quit both times because frankly I get obsessed with it and there are so many trolls. At least I had the good sense to only make fake burner accounts.

At the moment I have no social media accounts. I have a love/hate relationship with social media (I mostly hate it). I find that when I use social media, like you, I become too obsessive. I deleted my FB account 2 weeks ago, it was the last social media platform to go. I deleted FB once before and then foolishly re-engaged. I don't miss the white noise of social media.

 

[astyle]

I don't have a Facebook account, and proud of that. I do have a Twitter account, but that was acquired back before 2010, when I was trying to do my research on how to jailbreak an Ipod Touch 2nd gen. I haven't touched that account since, so I would not be surprised if it got weeded out for being inactive.

But man, Facebook is more fuss than it's worth - Not only there's pressure to curate what you've got, if you do something stupid, it can snowball into something beyond your control in a hurry. Even a simple like on Heineken's page can, after awhile, get you labeled an alcoholic, that will pull in devel/violence as a dependency, and then armed police will show up at your doorstep, charge you with a crime committed in a country you never even heard of, and good luck trying to completely untangle all that in court. All from a simple like on Heineken's page.

 

[Beastie7]

Wasn't Facebook originally a DARPA funded project called LifeLog?

 

[eternal_noob]

Nope. Zuckerberg stole it from the Winklevoss twins.

 

[astyle]

Wasn't Facebook originally a DARPA funded project called LifeLog?

I didn't even know LifeLog existed, until Beastie7 mentioned it here. But reading about it on Wikipedia - one would think that Zuck stole the idea of such extensive tracking/logging from DARPA, rather than the Winklevoss twins. The Winklevoss lawsuit centered around source code and mis-representing the company valuation duiring negotiations, which amounted to securities fraud. DARPA's ambitions were fulfilled by Zuck perfectly, to a T, with the money that the Winklevoss twins were cheated out of, allegedly. That would be my tongue-in-cheek conclusion.

 

[jammied]

One thing I do wonder is, would it not make sense when operating a project as large as Facebook to use static routes to some degree to make it harder for routing failures to occur? Thing is, dynamic protocols such as BGP are innately more at risk of failure. I feel like the whole outage is evidence of some degree of poor network management. Not that I am greatly surprised for a company whose product is in fact written in PHP.

 

[zirias@]

One thing I do wonder is, would it not make sense when operating a project as large as Facebook to use static routes to some degree to make it harder for routing failures to occur? Thing is, dynamic protocols such as BGP are innately more at risk of failure. I feel like the whole outage is evidence of some degree of poor network management. Not that I am greatly surprised for a company whose product is in fact written in PHP.

AFAIK, there's no way not to use BGP. As I understand it, you can configure routes statically that are announced by BGP, but if you mess up these routes when changing something in your infrastructure, you will end up with the same problem.

 

[jammied]

AFAIK, there's no way not to use BGP. As I understand it, you can configure routes statically that are announced by BGP, but if you mess up these routes when changing something in your infrastructure, you will end up with the same problem.

Yes and no. So, to make your routes globally available, BGP has to be involved somewhere along the time. Key points are:

• Individual ISPs could setup static routes to your network without relying on BGP. This is relevant as it would provide some level of backup in event of routing failure.
• You certainly do not need to use BGP on your internal networks, at all. You can either use other routing protocols (of which there are many), entirely rely on static routes, or have a mix of dynamically routed IPs and static routes. It is worth noting, it is fairly straightforward for BGP to obtain route information that is either declared by static routes or discovered by other routing protocols.
• The most important role for BGP in some respects is to discover where other IP blocks are on the internet as opposed to annoucing your own routes. You could if you wanted implement a structure where your network never announces BGP routes to other peers but where you rely on the carriers you use to annouce for you that they have routes to your network. That is not to say you would actually want to do this, but it is technically possible.

On the last point, I think some techniques would make sense for adding a backup in case of BGP failures:

1. Seek to have a small handful of ISPs setup static routes to a restricted portion of your address space so that if your BGP does fail, you have some form of failover.
2. Have some network equipment that has external IP addresses assigned by carriers and not yourself.
3. Setup VPN access to network equipment with external IP addresses assigned by other carriers from their own IP blocks and use the VPN as an alternative way for getting external access to your full address block in event of BGP failure.

 

[zirias@]

So, to make your routes globally available, BGP has to be involved somewhere along the time.

And that's the main point. And we have seen in the past how fragile this can be ?

Individual ISPs could setup static routes to your network without relying on BGP. This is relevant as it would provide some level of backup in event of routing failure.

Sure. I have no experience about that but: would they be even willing to do so?

You certainly do not need to use BGP on your internal networks, at all.

Of course not. And I have no idea how facebook handled that…

The most important role for BGP in some respects is to discover where other IP blocks are on the internet as opposed to annoucing your own routes. You could if you wanted implement a structure where your network never announces BGP routes to other peers but where you rely on the carriers you use to annouce for you that they have routes to your network.

Well again, I kind of doubt your peers would be willing to do that. But again, I don't have any experience in that field…

 

[jammied]

And that's the main point. And we have seen in the past how fragile this can be ?

Sure. I have no experience about that but: would they be even willing to do so?

Of course not. And I have no idea how facebook handled that…

Well again, I kind of doubt your peers would be willing to do that. But again, I don't have any experience in that field…

I think for a company as large as Facebook, I think the peers would have a strong incentive to agree to that. Also, want to highlight specifically, with the notion I suggested of getting IP addresses assigned from carrier IP blocks, you wouldn't need to a big company to get that, I could easily get that from broadband provider on my home internet package.

 

[sko]

You couldn't reliable set up e.g. multiple uplinks with static routes. If one goes down the route would still be there and the ISP would try to send traffic over that (inactive) link. The whole point of BGP is to make routing dynamic and resilient - There is (usually) never only a single path to a network, but multiple over different paths. So if one goes down, there are still others available - they might be slower, longer or have a low preference set, but they are still there so the prefixes are still available.

BGP works; it has been there since the dawn of time and has proven to be reliable and has been refined and hardened over the years (ROA/RPKI). Human error will always haunt _every_ service, no matter how well designed the protocol/service is; this has nothing to do with BGP. With properly configured BGP it is much harder to bring a network down by accident than by mangling static routes and (huge) routing tables by hand or somehow automated/scripted. (been there, had to do that for a crappy uplink with LTE-backup, would never want to touch such bullsh*t again)
AWS and Microsoft brought down their clouds (multiple times) without any involvement of BGP, purely by other bad design decisions... (e.g. MS placing all servers for an essential background service in a single DC...)

In the case of FB, they made the assumption their backbone from their "edge datacenters" to the main DC would never go down. To prevent malfunctioning servers from being available to the world it seems they thought it would be best to just let them revoke their routes if they can't reach their backend - but as their backbone went dark, all DNS servers revoked their routes; effectively killing all DNS for their zones, hence everything else that relies on DNS (hint: almost everything) also stopped working and even more routes were revorked.
Some static routes *might* have worked here to some extent, but static routes are a nightmare to maintain even on smaller networks, let alone in multi-homed AS or huge networks consisting of multiple AS. You absolutely have to use a dynamic routing protocol here, and BGP is the best and agreed-upon standard for inter-network routing.

Internally you can always set up multiple routing protocols; e.g. OSPF; and you can even exchange routing information between different protocols - but then you again start to make it complex and errors or unforeseeable events will propagate and might again cause a chain reaction.

 

[jammied]

You couldn't reliable set up e.g. multiple uplinks with static routes. If one goes down the route would still be there and the ISP would try to send traffic over that (inactive) link. The whole point of BGP is to make routing dynamic and resilient

Well, you could. Just would be questionable how much point there would be in doing it through a method other than through BGP directly except for a small part of your address space which intended to receive limited traffic. Essentially though, you just need the connection to peers to be made in a way that would result in the route dropping from the foreign peers routing table if the connection was lost. An obvious way of doing this would be maintaining the connection through CHAP in such a way that the existence of the route in the routing table was inherently tied to the existence of the CHAP connection.

Either way, if you know a certain connection is always going to provide access to a certain set of routes, you can simply ask the remote peer to create static routes to your network on the edge routers working on the basis that the remote peers internal routing protocols will then propagate the routes across their network using their internal routing protocols. Essentially, the existence of those routes on the remote peers network is not tied to state of your own routing protocols but only that of the remote peers network. Wouldn't not be difficult to maintain a small number of static routes in such a manner. If however you try to setup static routes on every router, yes, I would agree, that would become more of a nuisance.