Hi all, as many of you already know yesterday February 13th was really a tough day for me and also for a lot of you. I decided to write this post to give a quick recap of what happened and what the consequences will be.

What happened? around at 9.30pm SLT the MD Labs server started to perform an automatic (and not requested) software update, trying to update several modules that are vital for the server’s functioning. Unfortunately a few of those automatic updates failed and caused the complete breakdown of the basic server’s features, most of all the ability to handle requests from SL. As I was not aware of those automatic updates, I woke up and found the server completely down having no clue of what happened. Since I am not a server technician it took almost the whole day and the help of a real server wizard to fix the issue and restore the server’s operation.

When happened? the breakdown took place between February 12th 9.30pm SLT and February 13th 3.30pm SLT.

What does it means? since almost all my products rely on the server for several features, this means that during the breakdown period none of my product has been working correctly; some of them kept working with basic functions only in-world, while other got stuck completely.

Which are the products affected? the products that were mostly affected from the server breakdown are:

• MD Vendor System: this product relies completely on the server and for this reason each vendor stopped working completely, causing these main issues:
  • each customer who paid any vendor did not receive any product and also got no refund;
  • no transaction performed during the breakdown period was stored and for this reason those customers will not be able to request a redelivery for the product purchased (and not delivered);
  • it was impossible for customers to use any redelivery terminal and request product redelivery;
  • it was impossible for customers to use gift cards;
  • Any other operation such as mass redelivery or manual redelivery failed;
• MD Gacha Machine:
  • existing devices kept working in-world, delivering the prizes;
  • no transaction performed during the breakdown period was stored online;
  • no special item (if present) was given;
• MD Visitors Tracker:
  • existing devices kept working in-world, greeting visitors;
  • no visit performed during the breakdown period was stored online;
• MD Security System:
  • existing devices kept working in-world;
  • no event was stored online during the breakdown period;
  • no user add or removal was stored online during the breakdown period;
• MD Group Gift Giver Script:
  • existing devices kept working in-world;
  • no transaction performed during the breakdown period was stored online;
• MD Bid Board:
  • existing devices kept working in-world;
  • no bid nor bid session performed during the breakdown period was stored online;

What happens now? since February 13th 3.30pm SLT the MD Labs server is up and running again, so everything will work as expected now. However this severe issue brings a couple of conseguences:

• Any new vendor/gacha machine/visitors tracker/security system/group gift giver/bid board that was rezzed during the breakdown period must be resetted and re-initialized, as this will force the device to say “hi, I am here” to the server and sync with it. So please, if you rezzed any new device during the breakdown reset it.
• While all the other product should now work correctly the MD Vendor System – being deeply integrated with the server – will need a few more steps:
  • Reset your redelivery terminals & product servers and set them up again;
  • if you have vendors connected as client, make sure they are working correctly by making them “ping” the vendor server or terminal server they are connected to. In order to do this, simply click on the vendor>owner menu>plugins>remote control>ping server.

Anything else? yes, and again on MD Vendor System. As already stated any purchase performed during the breakdown resulted into an error, which means that your customers paid the vendor and did not get any product nor any refund. Moreover, the transaction was not stored online so they cannot request any redelivery for the failed transaction. The only way I can think of to fix this issue is having store owners using their vendors to gift the products purchased and not delivered to their customers. This will both deliver the item now and also create a transaction which will be stored giving the customer access to redelivery. I know this is a hell of a work and I am very sorry for this.

Needless to say, I am very, deeply sorry for what happened, the stress that caused to all of you and all the lost sales you have had. This issue was really hard to be foreseen and even harder to fix once it happened. Even tho I am not expecting this to happen again, I will definitely take into consideration some sort of server snapshot / server cloning in order to be able to recover quickly from this kind of disasters, without interrupting the service for too long.

My last words for this post are sentimental and dedicated to all of you whom contacted me when nothing was working and instead of raging against me tossed me a lot of nice words. Your sweetness, patience and understanding really surprised me and helped me to work on the issue with a less nervous approach. I am perfectly aware of the complains you were getting and the lost of sales, so I feel to thank you all for the unexpected love you all showed me. The next time I will be in need of some love, I will break something else again.