One of my favorite movie lines is from the 2001 movie Bandits, starring Bruce Willis and Billy Bob Thornton. Billy Bob plays a neurotic genius who, at one point in the movie, states “You know the hardest thing about being smart? I always pretty much know what’s going to happen next. There’s no suspense.”

So at the risk of sounding less-than-humble, this is an “I knew what was going to happen next and I told you so” post.

Last month, I wrote a product review for my new ecobee Thermostat. I still love the product, but in that review I expressed my primary concern with the unit: remote access to the thermostat, and remote control of the thermostat, is completely dependent upon ecobee’s servers remaining online. And, as of today, ecobee’s servers have been offline for at least two days.

I first noticed the problem yesterday when I decided to check on the thermostat in my Utah house. ecobee’s main website (http://ecobee.com/) was still online, but when I logged in with my username and password I got this:

On a positive note, that’s an attractive and well-designed failure notice. It’s a bit less attractive in the iPhone’s web browser:

And as for ecobee’s dedicated iPhone app, it just sits there “Authenticating…” for a while until a pop-up tells me “Error connecting to the server. Data could not be retrieved from the server. Please check your network settings.”

LOL. My network settings are fine and dandy, thank you very much! I can access my email, my blog, my Facebook, my Amazon Web Services account, all of my web servers, and the routers in my Utah house, cabin, parents’ and inlaws’ houses. I just can’t access my ecobee. 🙂

Compounding this issue is that it’s the week between Christmas and New Year’s, so I suspect that more than a few of ecobee’s technicians are likely on vacation. I’ll be keeping an eye on their remote access servers to see how long it takes to get them back online and I’ll post an update here. Maybe we could get a pool going in the comments! 🙂

The “good” news is that I already know when my next visit to Utah is, so I had already placed my ecobee thermostat in Vacation Mode so that it would return to normal programming the day before I got there (I like the furnace to kick in a full day in advance so that hard surfaces such as counter-tops and wood floors have a chance to warm up to air temperature). But if my trip happened to be tomorrow instead, and had I been planning on simply remotely connecting to my ecobee the day before my flight and manually adjusting the settings, I’d be arriving to a cold house… which is precisely what a WiFi-enabled thermostat like the ecobee is designed to prevent.

Please, please, please, ecobee! Please listen to your customers. Or maybe just listen to this one. I just happen to have a wee bit of experience with home automation, remote access, networking, web servers, hosted applications, and technology in general. Don’t get me wrong – I really like your product. It’s a good product, and because you were smart enough to build it with hardware that allows its embedded software to be remotely upgraded and improved, it has the potential to be a great product. I also really like the well-designed and powerful interface of your remote access website. However, as I stated in my earlier review, the power of your product all hinges on a single weak point: the uptime of your servers. And as of right now, they’ve been offline for at least two days that I know of (I don’t know if they were offline earlier when I wasn’t checking my thermostat).

I still want to be able to use your hosted web interface, but I also need to be able to connect to my thermostat directly in situations where your server crashes, or your network connection goes down, or your DNS records get messed up, or a tornado takes out your building, or Jimmy in tech support accidentally trips over a power cord and takes your hosted application offline. Unless you’re willing to invest in the type of infrastructure that can ensure 99.99999999% uptime to your remote access servers, please give us direct access to our thermostats.

I don’t mean to be overly harsh, ecobee. I like you guys. And I wish I’d been wrong. But, at the risk of sounding less-than-humble again, that just doesn’t happen very often. 🙂 Server downtime is a reality that all true geeks must accept. Therefore, taking whatever steps are necessary to ensure that customers can access a device whose primary purpose is to be remotely accessible, even if something goes wrong on your end, should be what keeps everyone at your company up at night – from the CEO to the marketing managers to the engineers to the web designers. Heck, I’d get the janitor on board with the concept, too.

Please allow me to quote my earlier review:

…remote access methods rely on ecobee’s private servers to provide remote access to your thermostat. If ecobee ever disappears, or their web server dies, so does your ability to use this thermostat for its intended purpose. Any device this smart that sits on your network and has an IP address should have a user-accessible web interface. But the ecobee doesn’t, which is a big drawback for me. Just as I can with my router, modem, wireless printer, and a host of other network-enabled devices in my house (including my Yamaha piano), I should be able to access and control my ecobee thermostat directly from any web browser on the local network, as well as set up a port forwarding rule on my router to access my ecobee from any remote browser using a dynamic DNS service.

I stand by those words – and I know you guys want to stand by your product. You’ve got a smart thermostat. It should come as no surprise that smart customers buy it. So if you have smart customers, the smart thing to do is listen to them.

————————

12/30 @ 4:30PM PST: Now when I attempt to login, I’m redirected to this page, which says:

Our Webportal is currently under maintenance. We are sorry for the inconvenience.

That’s different than the previous error message, so I’ll cross my fingers and hope that means they’re making progress on a solution!

12/30 @ 7:19PM PST: I’ve received two emails from ecobee so far today thanking me for this blog post and assuring me their development team is feverishly working on a fix. I’ve been in their shoes in the past, and I know it’s not fun.

12/30 @ 9:37PM PST: Ecobee’s MAIN site (http://ecobee.com/), which didn’t seem to have any issues before now, now looks hosed when viewed in my primary browser, Google Chrome. I checked and it looks fine in Firefox and IE (even after clearing my cache), so either Chrome is connecting to an out-of-date version of the site on a cloud-based edge server, or their server is sniffing for browser strings and feeding the wrong info to Chrome (that’s my guess). Chrome is seeing a version of their site where all the URLs (including links to CSS) are the wrong local hostname (just “web2”) with no TLD string. I’m assuming web2 is an intranet or dev site hostname, which probably works inside ecobee’s network, but breaks for the rest of us. Their stylesheet, for example, is trying to pull from https://web2/wp-content/themes/ecobee/style.css. Bummer. But fortunately, that’s an easy fix (and I just emailed them to alert them).

The maintenance message after attempting to login to the remote management interface is prettier now, and says:

We are currently experiencing issues with the ecobee servers and our Development team is working on the resolution. We apologize for this inconvenience and if you would like to be sent updates on our progress please email [email protected] and we will add you to our notification list. Thank you for your understanding. The ecobee Team.

I emailed and asked to be put on the notify list.

12/30 @ 9:43PM PST: ecobee just sent this announcement to the customers on their notify list:

ecobee Server Update – December 31, 2010, 1:00 am EST

Thank you to all of our customers who have shown patience through this period, we appreciate your understanding and support. The Development team has been working on the issue and has isolated the problem which is great news. We are now bringing a select group of thermostats online as part of our due diligence and we will send another email informing you of when this process is complete. In response to some concerns that have been raised, your ecobee Smart Thermostat will continue to run the program you have created during this period of limited web accessibility.

12/31 @ 6:09AM PST: ecobee just sent this announcement to the customers on their notify list:

ecobee Server Update – December 31^st, 2010, 9:15 am.

Good Morning;

The Development is continuing to make progress on bringing a limited number of test units online but we have are not able to reinstate full web portal accessibility for all customers at this time. The entire team is continuing to work on the issue and will do so until there is a resolution.

Thank you to those who have offered their support and encouragement and we are sorry for the inconvenience that this is causing to our customers.

12/31 @ 9:45AM PST: ecobee’s main website works in Google Chrome again. The maintenance message on their website has been updated to say:

We are currently experiencing issues with the ecobee servers but we have isolated the issue and the Development Team is running their final due diligence. We expect to have all of our customer’s web portal access available in the next 2 hours.

they also just sent this announcement to the customers on their notify list:

ecobee Server Update – December 31, 12:45 pm EST

Good afternoon,

We have isolated the issue and the Development Team is running their final due diligence. We expect to have all of our customer’s web portal access available in the next 2 hours. Thank you for your patience during this period.

Happy New Year.

That sounds encouraging! However, I can’t help but think that during a technical outage (of which my companies have had their fair share), I would never include a specific deadline on a website or email message (such as “in the next 2 hours”), no matter how certain I was that we would hit it. Optimism is great, but I’ve said stuff like that in the past, and then ended up with egg on my face 3 hours later. Until everything is actually up and running again, I prefer to stick with “as soon as possible.”

12/31 @ 11:52AM PST: ecobee just sent this announcement to the customers on their notify list:

ecobee Server Update – December 31, 3:00pm EST

The issue we have been experiencing with the ecobee server has now been resolved. All customers should have access to their web portals. Please note that during this interruption some of the reports data may not have been captured resulting in the reports graph having no data at certain times.

Thank you for your patience during this period and a Happy New Year.

Hooray! I have verified that I can login and access my thermostat and reports. They were less than 15 mins late on their 2 hour promise, so that’s commendable. Surprisingly, I’m only missing reporting data from around 2:45AM to 6:30AM yesterday morning. All of my programming data and other settings appear to be intact, and everything appears to be back to normal. I really didn’t want to have to write a tongue-in-cheek blog post entitled “My ecobee has been offline since last year!”

12/31 @ 2:06PM PST: ecobee Founder and CEO Stuart Lombard (who had commented earlier on this post… see below) sent the following open letter via email to all ecobee customers:

An Open Letter to ecobee Customers:

I am a founder and President of ecobee. Please accept my sincere apologies for the issues we have experienced over the last two days with our web servers. I believe that, up until Wednesday, we have had an excellent uptime record (over 99.9%). Unfortunately, on Wednesday we ran into a performance-related issue and the resolution was not straightforward or easy to isolate.

We had a dedicated team working around the clock to resolve the issues, and we are now operational again. We will review our processes thoroughly to determine what went wrong and how we can prevent this situation from happening in the future.

We are extremely upset with the downtime. We are committed to delivering excellent uptime and reliability, and we expect that to continue in the future. We appreciate your patience and ongoing support.

If you have any questions or comments, please do not hesitate to contact me directly.

Best regards,
Stuart

A well-written letter saying exactly what it should for the average customer. Of course, über-geek customers want to know the gory details of the outage, but an open letter isn’t the right place for that. The comments section of an über-geek’s blog is the right place for that (hint, hint).

Good job getting it sorted out, guys! Look on the bright side: tomorrow, you can say you’ve had 100% uptime in 2011! 🙂

Go enjoy the New Year’s holiday. You’ve earned it. But when you get back in the office, I’m still dying to know exactly what went wrong, and what you’re planning on doing (in geek talk, not marketing talk) to decrease the chances of it happening again.

And, yes, I still wish we had direct access to our units. 🙂

10/23/2011 Update:

Due to repeated requests about an Ecobee users discussion forum, I’ve started one myself. If you’re an Ecobee owner, enthusiast, or would just like to join the conversation, you can subscribe here: