Sunday at midnight, we had scheduled some downtime for Wufoo to transfer our current setup to some new servers in another facility. We, unfortunately, ran into some unexpected problems and the downtime spilled over into this Monday morning. For that, we offer the most sincere apologies. The purpose of this message is to let you know that all accounts are now up and functioning properly, and that we are a bit embarrassed at the level of service we provided this morning.
For those of you interested in the technical details, here’s what happened. Wufoo is unique in that each form created is essentially it’s own database setup. Because of this, we have some complicated and non-traditional procedures that need to be followed when we make changes like we did today to our servers. When we moved the accounts to the new servers, the actual estimated transfer time was about 55 minutes. The first transfer completed on time, but due to our setup, certain configuration files were corrupted in the process. After that first transfer, it took us 3 additional transfers plus some digging time to isolate and fix the problem.
We’ll be the first to tell you that we have no excuses to justify all the things that went wrong today, and we have learned quite a few things from the experience. In the future, we will be running maintenance on Friday / Saturday night to prevent any chance of the work week being affected, and we will always test as much as we can ahead of time given our resources. Additionally, we’re working on a way for you to be better notified in these situations. We have to present proper error messages on your sites to keep not just you, but your users informed as well — there will be no cryptic database messages in the future from Wufoo.
Thanks to everyone who was patient and understanding while we worked on this. You guys were great. We, obviously, have a lot of work to do now, but we assure you that this isn’t a common thing for us. For our recent crop of users and customers, we hope you don’t judge us by these hiccups.
Again, we can’t thank you enough for your understanding.