Planned and Unplanned Outages

#1

It seems that on June 12, 2020, there was a planned outage to the Eufy servers serving the security camera line and the result was users were unable to access recorded footage. Suggestion is to provide advanced notice of planned outages and real-time updates on current outages via email to active users.

1 Like
#2

A heads up email would be helpful

#3

Agree, especially for planned outages but they should send something for unplanned as well so we don’t go around checking our setup looking for the problem when it’s a eufy server problem.

#4

I have significant geeky IT knowledge. Possibly more than anyone else here?

I warn you heads-up, if a planned outage takes hours, unplanned outages will take days.

Here is two parallel scenarios for planned outages, then I’ll replay it for unplanned outage. So first planned.

Scenario A: two shared-nothing async mirror High Availability server pairs.

  1. background replication from HA pair, I’ll name HA1 HA2.
  2. Outage begins. Stop updates to data on HA1 from users.
  3. sync the changes from HA1 to HA2
  4. switch users (typically via DNS) from HA1 to HA2.
  5. Outage ends, users on HA2. Outage duration of the order of 5-15 minutes.
  6. Upgrade HA1, it takes however long it takes (hours, days, but HA2 is serving users so no long outage).
  7. Another outage begins. Stop user updates on HA2. Sync changes from HA2 to HA1.
  8. Switch users from HA2 to HA1. Outage ends. Total outage of the order of 5 minutes.
  9. Resume sync of changes from HA1 to HA2.

So here you had two short planned outages. Typically you’d do these a week apart at known low-usage periods as the sync of two async mirrors takes least time during least change / activity.

Scenario B: no HA2 exists.

  1. begin outage.
  2. takes however long it takes, hours.
  3. outage ends.

Then let’s replay this for unplanned.

  1. HA1 goes down
  2. panic, oh, panic, wow. Where’s our backup? Oh.
  3. days pass
  4. outage ends.

So my concern here is if users had hours of issues in a planned outage, Eufy is inherent under-architecting their server architecture and so is implicitly vulnerable for very long unplanned outages.

1 Like
#5

How do you have so much “geeky IT knowledge” like how do I get on your level because that is truly something to strive for?

1 Like
#6

a) Math
b) time.

1 Like
#7

I love math so hopefully I’ll get to your level of knowledge someday!

2 Likes