Safe-mail.net is down for critical maintainance (maintenance)

teris3 · Oct 30, 2015

Once it's back up (if it will ever be) delete all mail and leave Safe-mail!
I ll do the same once I'm able to grab some contact information out of Safe-mail.

Gindan · Oct 30, 2015

Any idea what does it mean?

Fri Oct 30 06:50 EDT 2015

We are sorry! Safe-mail is not ready yet.
Safe-mail.net is down for critical maintainance.

The Safe-mail.net system had a major hardware storage crash.
About 80% of user's data is not affected by the crash, as we already moved it to a new system during the last few months.

The rest is either to be salvaged from the crashed storage, or will be restored from the Backup system.

At this time we will permanently remove about 250,000 user accounts that were involved in activity which violates the Safe-mail.net Agreement and policy.

Free web based registration to Safe-mail.net will not be available.

Free accounts will be opened by sending a request to Safe-mail.net.

In the near future, separate subsystems will be used for Free and Premium accounts.
This will enable better service and even enable shutdown of the Free subsystem without affecting the Premium users.

Further information will be published.

rradam · Oct 30, 2015

They have migrated 80% of users to a new storage system prior to this crash (reason of the migration could be speculated as the old hardware was failing). They are restoring the other 20% from the failed storage, or more likely, backups. They also seem to be taking this opportunity to clean up users that violated the ToS. This could speed up the restore if those accounts are simply ignored during the restore. They are going to put free users on a separate system and put premium users on a more HA system, to dampen impact on paying users for any further outages. Free accounts will no longer be able to be opened automatically. There is likely no way of knowing at this time if you were already migrated or not. If you were in the unlucky 20%, I would wager to say you've lost all of you email in the past month at least.

EQ Admin · Oct 30, 2015

Gindan said:
Any idea what does it mean?

This is pure speculation. I'll take a shot at it.

It's strange to me that they mixed a storage issue with accounts that violate their TOS. It makes sense that they are trying to speed up their recovery time by restoring all premium accounts and free accounts that are active. It would make sense if they didn't restore, or restored last, the inactive accounts. Maybe their design makes that harder to do than it's worth.

They stated accounts that violate their agreement and policy will not be restored, not inactive accounts, which leads me to think it's spammers and/or accounts identified to be involved in illegal activity that will not be restored.

I suspect that Safe-Mail is a "medium" size email service. They likely have some redundancy built into the systems, but they're all located in a single location. It's not a single server, but they also didn't appear to exist in multiple geographic locations.

If that's right, to help visualize what's going on, their network might look something like this:

Users and email traffic flow into their data center from the top. What I suspect is broken is some flavor of highly redundant storage that many/all of the email and web servers depend on.

I marked the storage with a red X.

In this scenario they could lose lots of things such as a web server, a mail server, a firewall, a load balancer, probably with minimum impact to the user experience.

The storage I'm describing has a lot of redundancy within it. If a hard drive within it failed that shouldn't cause a problem. A catastrophic failure would have to happen to it. Multiple shelves impacted, several disk groups are no longer available.

teris3 · Oct 30, 2015

I really think the problem is that they grew too fast too soon and left the system virtually unchecked until critical mass was reached and then needed to back peddle.

EQ Admin · Oct 30, 2015

@rradam got in with a good reply while I was drawing pictures

They have migrated 80% of users to a new storage system prior to this crash (reason of the migration could be speculated as the old hardware was failing). They are restoring the other 20% from the failed storage, or more likely, backups.

That makes sense if an old storage failed and they were working in migrating to new storage.

Picture a second storage next to the old one, and that unfortunately, some of the user data was only on the old storage (and hopefully backed up!) before the old storage went offline.

rradam · Oct 30, 2015

They have squeezed in "Only users violating their policy will be deleted, most users will not be affected"

JackJones · Oct 30, 2015

any idea what the Safe-mail.net Agreement and policy was as after all who reads these things? Be interesting to know as I didn't read it.

JackJones · Oct 30, 2015

and this problem has been going on for years so why hadn't they fixed it before it got to a critical stage. Be nice to get rid of the spammers as most of my emails seem to be spam these days most of it is ludicrous but I am intrigued as to who writes this stuff as they cant spell and have no idea of English grammar

EQ Admin · Oct 30, 2015

JackJones said:
any idea what the Safe-mail.net Agreement and policy was as after all who reads these things?

I found this information using archive.org ( Safe-mail Information Center ):

Agreement

Dear Potential User,
Safe-mail is pleased to offer you the use of our Safe-mail e-mail system. You may use Safe-mail in ANY legal way for your personal, business or other needs. Safe-mail may terminate your access to Safe-mail upon violation of our terms. Our terms for using the system are set out below. By registering with us, a step that is necessary to gain access to the system, you indicate your agreement to these terms. Please read them carefully.

USE: Safe-mail grants you a right to use the Safe-mail system to send and receive e-mail, on the terms stated here. This is a license to use the system. There is no fee for this license.

OUR RIGHTS: All rights, title, and interest in the Safe-mail system, including related copyrights, trademark and other intellectual property rights, belong to Safe-mail This agreement confers no license or right, expressly, by implication, or otherwise, other than the licensed right of use.

PRIVACY: Safe-mail will not disclose information about you or your use of the Safe-mail system, unless Safe-mail believes that such action is necessary to comply with its legal requirements or process; enforce these terms; or protect the interests of Safe-mail, its members or others. You agree that Safe-mail may access your account, including its contents, for these reasons or for service or technical reasons. Please note that your Internet Protocol address is transmitted with each message sent from your account.

NO REPRESENTATIONS: The Safe-mail e-mail system is provided "as is," without any warranty or representation of any kind, either express or implied. We do not promise that the Safe-mail system will be uninterrupted, impenetrable, problem or virus-free. Safe-mail is not responsible for user generated content.

NO DAMAGES: In no event will we be liable to you or any third party for damages of any kind arising out of this agreement, or the use, inability to use, or termination of access to Safe-mail, even if we have been advised of the possibility of such damages. This includes with respect to problems in sending or receiving, storage, or unauthorized access to or alteration of, transmissions or data. You agree that your sole remedy for any damage, is to discontinue using the Safe-mail system.

LIMITS AND LAW: You may not use Safe-mail in a way that is threatening, harmful, or invasive of the rights of other; for spamming, chain letters, pyramid schemes, junk mail, unsolicited advertising or bulk e-mail; or otherwise in a way that is damaging, offensive, or that creates a nuisance. Disguising the origin of transmitted content is prohibited. You agree to abide by all laws and regulations applicable to this agreement and use of the e-mail system. This agreement is made under and shall be construed according to the laws of the State of Israel and Israel's courts will have exclusive jurisdiction over any dispute related to the system or this agreement.

ACCEPTANCE OF TERMS: Use of Safe-mail is offered to you conditioned on your acceptance, without modification, of these terms. You may not sublicense or otherwise provide to any third party, the right granted in this license. This agreement does not entitle you to support or service, updates or future versions, or other Safe-mail products. Safe-mail, may terminate your access to Safe-mail at any time, without notice, for any reason. If you violate these terms, your right to use the e-mail system terminates automatically, and this is in addition to any other remedies available to us.

ENTIRE AGREEMENT: This is the complete agreement between us and it supersedes any information or representations you received relating to the Safe-mail system. All rights not specifically granted to you under this agreement, are reserved by Safe-mail

Signed,

Safe-mail

JackJones · Oct 30, 2015

Many thanks for that so that means I hopefully wont get all the spam I have been getting when they delete the misused email accounts - and I can buy prescription drugs from a guy in India still who is on safe mail - my B12 injections cost so much where I am and so getting from a reputable seller in India is so much more cost effective but as he runs a legitimate business he won't be affected by any of this

rradam · Oct 30, 2015

tl;dr don't spam.

izimbra · Oct 31, 2015

Any guess as to when service will be restored? This has got me in a tough situation. I have no idea as to what kind of time and effort is required to fix something like this. I certainly won't be holding my breath since it took six days to communicate the nature of the problem. Educated guesses are appreciated.

EQ Admin · Oct 31, 2015

It's impossible to know for sure. Without a sign that they seem to be close to done, I'd expect it to be days/weeks and not a few more hours.

Gindan · Oct 31, 2015

popowich said:
It's impossible to know for sure. Without a sign that they seem to be close to done, I'd expect it to be days/weeks and not a few more hours.

I am afraid the service will not be restored at all in worst case or full data loss happened in best case. One week downtime means total failure.

rradam · Nov 1, 2015

I'm a bit bored, curious (and tired) myself. taking a guess here, but assuming 10million accounts (most of probably which are free, so limits on mailbox space), 20% loss, guessing no data-dedup in place in their backup/restore strategy, maybe 1PB of data, using LTO6 tapes and two tape drives, ~41 days straight if working non-stop. Many variables to account for though (4 tape drives would almost half the restore time for example, while LTO4 tapes would nearly take twice as long ).

JackJones · Nov 1, 2015

nice to see people thinking clearly about the magnitude of the problem and aren't saying that safe mail has gone for good like so many are. I will put my faith in the knowledge of such as rradam and popowich as I would prefer to remain positive and not dismiss safe mail out of hand.

izimbra · Nov 1, 2015

JackJones said:
nice to see people thinking clearly about the magnitude of the problem and aren't saying that safe mail has gone for good like so many are. I will put my faith in the knowledge of such as rradam and popowich as I would prefer to remain positive and not dismiss safe mail out of hand.

Agreed and thanks to everyone for their input.
I do think they should give us a rough estimate as to when they expect to be back up. That would certainly help inform me as to how I want to proceed here.

teris3 · Nov 2, 2015

The site is now down for me.

JackJones · Nov 2, 2015

teris3 said:
The site is now down for me.

same here tried it earlier this morning and nada all I have is "This website is not available" liked it far better when I actually had a notice up from safe mail saying it was down for critical maintenance. Maybe there is just another problem that has cropped up

Safe-mail.net is down for critical maintainance (maintenance)

New Email

Valued Member

Valued Member

EQ Forum Admin

New Email

EQ Forum Admin

Valued Member

Valued Member

Valued Member

EQ Forum Admin

Valued Member

Valued Member

New Email

EQ Forum Admin

Valued Member

Valued Member

Valued Member

New Email

New Email

Valued Member

Similar threads

Privacy & Transparency

Privacy & Transparency