When dealing with security, boring is not that bad an attribute as it may otherwise sounds.
Clipperz has been running prettly smoothly lately, with some degradation of server-side performances we haven’t solved yet, but that is not so bad to negatively impact regular use of our application.
Nonetheless, from time to time some hiccups happen also to our smoothly running service; and last Sunday morning was such a time.
We do monitor our service (as you can tell from the graph above), but we do not have automatic alerts to notify us about issues. Last Sunday I was made aware of Clipperz not working by this tweet:
@Clipperz Are there any issues with Clipperz right now? Can’t seem to login. “Login Failed”
Admittedly, it is not very funny to receive a message about your service not working while having breakfast on a Sunday morning.
Even if Clipperz is a free service and we do provide a way for our users to access their own data even when our service down, we don’t like to let anyone down; expecially becasue we know how important the data save on Clipperz may be, as we use Clipperz daily ourselves and we feel the pain when it lets you down.
I quickly finished my breakfast and opened up my computer. I had to “eat our own dog food”, as I had to resort to an offline-copy of Clipperz own account in order to retrieve all the credentials required to enter into our systems. Fortunately the problem was fixed by a simple reboot; half an hour after the first tweet was sent, the service was regularly online again.
After having verified that the service was fully working, I looked into Clipperz’s email accounts and online forum to check if other people had been affected by the outage. Indeed I found a few messages by people concerned about the integrity of their own accounts, as the application was not completely offline, but was instead misbehaving reporting any login attempt as failed.
This was the first message I stumbled upon:
Dear Clipperz developers,
I’ve been using Clipperz successfully in the last six years. I simply love it!
It’s quite amazing that you can offer such a great service with nearly zero technical issues for free. I never tried to contact you in the past since there was simply no need to. It always worked magically. […]
Now, last Sunday was off to a bad start, but messages like this are a great way to restore the mood.
Designing Clipperz we have always been very careful to avoid leaking any information about our users identity; we don’t even know (nor can verify) what usernames are being used. This means that we can’t have any interaction with our users, unless someone decides to write us; but this usually only happens when something goes wrong and –being the service quite reliable– this is not very often.