Let me get this straight…

Over the past couple of months, everyone seems to have become an armchair engineer. A good friend of mine, Jim Moore (who writes a lot of fairly popular stuff, including a best selling political book) has been saying “All they had to do was install a $500,000 kill switch, and this disaster never would have happened”, while he waxes poetic about Turtles in the Gulf.

Well, hindsight is 20/20. I recently finished reading Malcolm Gladwell’s “What the Dog Saw”, a collection of short essays. One of the topics he covers relates to the causes of failure of complex systems. The two examples he gives are Three Mile Island and the Space Shuttle Challenger Explosion (the Columbia disaster had not yet happened when he first wrote the essay). The Space Shuttle explosion, in particular, is one that I’ve spent some time investigating, having a personal interest in human space travel. He talks about how NASA keeps a record of all the possible failures – the O rings (that failed) being one of them. The list, however, is 6 volumes thick. Sure, they could have fixed the O rings before hand, but it was considered a very slight risk, and was known. If they had known the O ring was going to fail, disastrously, of course they would not have tried to launch with them on that fateful day. Interestingly, I remember years ago, reading an article that talked about the probabilities of failure on the space shuttle. While each component is designed with extraordinarily tight tolerances, each component having a stunningly small chance of failure, statistics and the real world multiply all those failure percentages together to come up with what, in retrospect, has proven to be an amazingly accurate number. Statistically, a space shuttle should fail about once in every 72 launches. There have been a total of 132 shuttle launches. We’ve lost two shuttles. NASA was not surprised.

Complex machines fail, it’s the nature of complexity. Perfection is impossible – so the goal is to minimize the probability of failure within reasonable economic parameters. Like the $500,000 kill switch – sure, we’ve heard about it now. Just like we knew every single one of the 9/11 terrorists by 11 am on September 11, 2001. Knowing what would have helped retrospectively gives us this false sense of knowledge – we can see who the terrorists were after seeing that they committed an act of terror. But remember, they did not break any laws (with the exception of planning a terrorist attack) prior to the attack. How many people boarded airplanes with pocket knives that day? If, on September 10, 2001, the president announced that every airport in the country was going to require body scanners, gross violations of privacy, add an extra hour to every flight, all at a cost of at least $8 Billion per year to the taxpayers, most Americans would grab their pitchfork and head to the White House. The fact is, there’s a lot of pieces moving, and only after a disaster can you find out what the key piece would have been for this disaster.

An offshore oil rig is an incredibly complex machine. Parts of The Deepwater Horizon experienced an environment more hostile than even the space shuttle endures. Hell, just look at how Wikipedia describes the thing: “Deepwater Horizon was a fifth-generation, RBS-8D design, ultra-deepwater, dynamically positioned, column-stabilized, semi-submersible mobile offshore drilling unit (MODU)”. Wow. It even sounds complex. Could they have made the explosion impossible? Nope. Could they have made it less likely? Possibly.

If the president announced, on April 19, 2010, that he had mandated the reduction of the probability of disastrous failure of each offshore oil rig from 1 in 1000 to 1 in 10,000 (no, those aren’t the actual numbers, but they are representative), and that gas would cost at least $1 more per gallon, then we’d all be joining the Tea Party.

S3

…Most things are shades of right.

Recently, I was reading about a pilot who heard an explosion as he was flying, and his engine started vibrating uncontrollably. He managed not only to get his airplane down safely (by adding a little power, rather than cutting the engine), but even managed to land it at an airport, and ultimately got it fixed. Turns out one of the connecting rods had actually freed itself from the engine and blew right out of the engine compartment. I started reading a forum discussion about this event, and the guy was on the thread, responding to people’s questions. One person asked about a parachute (like the Cirrus CAPS parachute) to which the guy commented that “…a parachute … would have been the WRONG conclusion…” (his airplane did not have a parachute, his comment was basically saying that this was a good thing, because he was not tempted to deploy it) Naturally, I challenged this. And this is the point of my post. He is, in a word, wrong. Pulling the parachute would probably not have been the wrong conclusion. It’s very simple – when you’re in an airplane, and the engine explodes, you have one priority (just like with everything else) – get down safely. Did he do that? Yes. Did he make a risky decision to do so (adding power rather than cutting the engine)? Yes. If he had deployed the parachute, would he had done that? The data suggests he would have. In other words, what he did was right. But what he did not do was probably not wrong. The only wrong decision would be one where he failed to survive. And, quite honestly, a lesser pilot would have been more likely to survive with a parachute pull than with what he did.

This doesn’t only apply to airplanes and survival. People often ask me “if you could go back, and re-start S3, what would you do differently?” You know my answer? No, it isn’t “fire people faster” (although I am a fan of that). My answer is nothing. Why? Because whatever we did has worked. Could we have done something more right? Possibly, but what we’ve done so far has worked. Sure, we’ve had learning experiences – and we use them as such – to learn how to address things in the future. We don’t get stuck in the past – we learn from the past.

My advice? Learn from what you do, make good decisions, and don’t get hung up on what’s wrong.

Facebook is an incredible tool. It lets you connect with people you haven’t seen for decades, and see how your lives have diverged. But it also helps you find new business contacts, or new friends, or even new love. Facebook, as a company, has succeeded. I read a quote recently (sadly, I cannot remember the source – if you know the source, please let me know so I can give proper credit) that said (approximately) “not being on facebook today is like not having a cell phone – by doing so, you are making a statement, whether you intend to or not”.

They are estimated to have roughly $300 million in revenue. The company is worth somewhere around $6.5 Billion. They have 900+ employees. So why the hell do they keep changing things that people get comfortable with? The front page of facebook has changed significantly at least three times in the past year. Unlike cell phones, where people have an option to select a new one, Facebook has a single primary interface.

Facebook appears to be following the disastrous trend of allowing developers to create their software. No, I don’t mean actually write the software, of course they do that – but create the experience that a customer will see. There are a couple of reasons for this, but none of them relate to the kind of things a $300 million (not to mention $6.5 billion market cap) company should have. Developers are amongst the most necessary people at the inception of a software (or web) company. you can make a product with a developer – it may look like crap, but it’ll work. You can’t do that with anyone else. It’s the same reason QA is always an afterthought – no one ever started a company with a QA person and no developer (ok, ok – outsourced testing companies might – but if you learn about S3′s Production System, you’ll realize that outsourcing QA is a terrible decision, but that’s for another post).

So when you start a company, you hire some developers. And they do a good job. But over time, your product matures. And your developers become less critical. So you have several choices – put them on things that matter to you, but aren’t very technically interesting, fire them, or keep them designing the critical stuff, ignoring all the input you get from your clients (and probably many people within the company) telling you that they’re making bad decisions. Each of these has its downfalls – the first makes the developers bored, and disenfranchised. That sentiment will probably spread through your organization. And then they’ll probably quit anyway. as for the second, firing them feels wrong. These are the people who helped you get off the ground – these people are the reason your company exists. The third makes your clients disenfranchised, and frustrated.

So, if you want to continue to grow, and keep a strong culture, and keep your clients content, this is when you have to fire your developers. It’ll feel bad. And there are days you’ll regret doing it. But if you don’t, you can watch your company erode, and instead of losing some people, you may well lose them all.

Sure, if you happen to be the sort of monopolistic company that Facebook has managed to become, the last option is survivable, but probably only in the short term – some day, someone will figure out a way to overthrow you over time – maybe with just 140 characters.

Oh yeah? You do? Let me guess – you donated money to the earthquake relief effort. Or you even decided to go to Haiti to help “rebuild”. See, here’s the thing – there’s nothing to rebuild.

The country is the poorest country in the western hemisphere. The annual GDP per person is probably less than you make in a week (unless you’re making minimum wage, in which case less than you make in three weeks). And that’s an average. The vast majority of them make less than that.

In the late 90′s, my wife (fiancee at the time) and I went to The Dominican Republic. It was a good, cheap, Caribbean destination. We enjoyed an all inclusive resort, with excess of food and drink. On the drive from the airport to the resort, we were amazed at how poor the people were. We had both traveled a good deal, but never before to anywhere poorer than the G8 nations. The houses were tiny concrete huts. Large families crammed into these small houses. we were shocked. We were told that the average worker was making $200 a month. And that we would be perfectly safe in the resort – the waiting list for a job there was two years – people would not do something that might get them fired.

While we were there, we went on a tour. The owner of the tour company (a German) joined us on the tour – he was testing a new stop on the tour – a sugar cane plantation. What we saw shocked us. The sugar cane was tended by Haitians. The Haitians moved to the Dominican because the standard of living was so much higher. They lived in mud huts. No doors (just holes), no windows. There was no infrastructure at all – they were surrounded by miles of sugar cane fields. The children were naked, the adults clothed in scraps. As we left, we gave them a bottle of water. the children all surged for it, hoping for a sip of the pure spring water we all take for granted. And these were the migrant workers. The ones who were fortunate enough to escape from the poverty of Haiti.

Where were you on Monday, January 11? Were you praying for the Haitians? Sending them money? Taking a week off work to go help them grow their economy, or to give them the medicine they need? No. You were sipping fine wine at your favorite restaurant. You were driving around in a vehicle that cost more than most Haitians will see in a lifetime.

You say you care about Haiti? Well, prove it. Quit talking about texting for $5. Or taking time off work to help them. They don’t need that kind of help. If you really care, and aren’t just doing this to show the world how good you are, wait till all this earthquake hullabaloo dies down. Start buying Fair Trade Haitian products. Or invest in companies that invest in Haiti. At S3, if any employee donates money to Haiti, we’re matching, but not with cash – we’re matching donations by investing in Haitian entrepreneurs on Kiva. If you really care, teach the Haitians how to fish.

One of the dumbest phrases I’ve ever heard is “is there anything we can do to make you stay?” It’s pathetic. begging people to stay is completely unproductive. They don’t want to be there. Let them go.

But, you say, maybe we could do something to make them happy. Nope. It’s too late. you may well have had a chance a while ago, back when they were starting to get disenfranchised. And I suspect you did have that chance. You heard rumors. they seemed less interested. You could have talked to them. But you didn’t. What’s different now? they’ve made the decision to leave. Let them go.

I know, it’s not your fault – they didn’t come talk to you. That’s a good point. Why didn’t they? Well, did you invite them to? Did you say “Tell me how I’m failing you”? and not just to the people who seem unhappy, but to everyone? sounds like it was your fault.

So get out there, talk to your employees – if they have a problem, they’ll talk to you before they quit. So you never have to beg pathetcally again.

One of our former employees would often say to me “Mark… you know what your problem is? YOU HAVE NO CHARISMA!!!”. Why? Well, you’ve read my blog. I’m not exactly Dale Carnegie. In fact, Jim Moore has been known to say “Mark, you’re the only person Dale Carnegie would punch”.

One of the founders of Southwest Airlines, Herb Kelleher has long been touted as an incredibly charismatic leader. He managed not only to convince governments to allow his struggling airline to get off the ground, but (far more importantly), he managed to convince an entire team of people working for him to do everything they could to get that airline off the ground, despite the fact that they were literally grounded.

So what’s my point? Yes, you saw it coming, I’m going to compare myself to Herb. One of the often touted examples of Herb’s character is his willingness to listen to his employees. But in reading The Southwest Airlines Way, I read an interesting quote from one of the pilots: ‘…You don’t just call and say there’s a problem. He’ll say “think about it and tell me the solution that you think will work”…’ And this is exactly what I do… only I say “Quit your damn complaining. Come to me with a solution!” And that is my advice today. Quit complaining. And don’t let people complain to you.

Complaints are downers – no one likes complaints. And no one likes complainers. But it seems people like to complain. Don’t stand for it. If you have a complaint, make sure you have a solution to it. And if someone comes to you with a complaint, challenge them to find the solution.

One other thing – If someone comes to you with a problem, and a solution – figure out how to address the problem. And at least some of the time, use their solution. Far worse than complaining would be to listen to someone’s problem, listen to their solution, and ignore it (and even worse would be to ignore the problem as well). If you do that, not only will they revert to complaining, but they’ll also add a new complaint – that you don’t ever listen.

If you’re in the software industry, you’ve probably heard a lot of buzzwords about making software fast, good and error free. for a long time, people have been solving problems using some pretty standard techniques. One of these has manifested itself in Software as “Waterfall” development. Waterfall is basically the same idea that everything else in the world is developed by – first, you figure out what the problem is, then you figure out how to solve it, then you write the program that solves it, then you test it, then you deploy it. Seems like a pretty sensible approach – for every dollar you spend in each step, you can expect to spend 10x that on the next step. so if you spend ten dollars to make a design decision, making that same decision after deployment could cost many thousands of dollars. It’s really just like in construction – moving a line on a blue print is a lot cheaper than moving a wall in a building.

But software isn’t really like everything else. Partially completed software can actually be used. A bridge with a few feet missing is entirely useless. But a program with a few features missing tends to be pretty useful. Because of this, people figured out that they could do other development methodologies with software. And so was born the agile development methnodologies. Lean, Scrum, XP – all sorts of different names were given to them. They compressed the time from analysis to production down from what could sometimes be years to months. Were they a good thing? Sure, to an extent. They’ve been criticized for various things by the Waterfallers – missing requirements, not able to address all concerns, etc. But that’s not my approach.

Agile Software methodologies fail not because they are too lean, but because they are not lean enough. That is to say, they absolutely not lean at all. They’re just a name given to short waterfall cycle. “Lean” manufacturing was invented by Toyota. The idea of lean is to eliminate waste. Waste from everywhere, but notably in inventory. But software doesn’t have inventory, you say. Of course it does – inventory is any piece of functionality that is not delivered to the customer.

Reread that – that’s the crux of S3′s Production System. Inventory is any requirement functionality that is not delivered to the customer. If you have a requirement that is sitting in a queue anywhere, that’s inventory. Have software designed but not coded? Inventory. Have software coded but not tested? Inventory. Software tested but not deployed? Inventory.

Agile (like waterfall) groups features together. Sometimes related, sometimes not. But it groups features into releases. Why? because that’s how it’s always been done. Nobody deploys a single feature. It’s always grouped. Which means there’s always inventory. If a software methodology is going to claim to be lean, truly lean, it has to get rid of the inventory. Why do you have three developers working on three different features, then that whole package goes to a QA department (possibly overseas)? S3 does true one piece flow. One piece of functionality – from inception to production with no wait points. No inventory. Design, Development, QA, production in the same room – a piece of functionality never has a chance to be wrong. Quality suffers? No way – just like Toyota, by having everything there together, quality skyrockets. Reworks disappear. Bugs are caught before they are even written.

There is a lot more to come on how S3 truly adapted lean manufacturing to software – this is just something to get you started thinking. Challenge me, please. We’re always looking for better ways. But if it’s something that exists today, we’ve already determined that it’s too slow.

Ok, so that’s not entirely true. Part of it isn’t wrong.

What the hell am I talking about? Simple – the US constitution is, primarily, a specification. It is a specification that describes how to perform a certain set of requirements (which, coincidentally, is the part that isn’t wrong). So why is that wrong? It’s not actually wrong – it simply doesn’t take into account all available information. It takes into account merely the information available to the creators of the specification. There are numerous examples of how these specifications may have applied to a world eleven score and two years ago, but not today. As illustration, I’ll use a non-controversial example in the seventh amendment: “In suits at common law, where the value in controversy shall exceed twenty dollars, the right of trial by jury shall be preserved….” Obviously, if you’re suing someone over $20 today, you have too much time and/or money on your hands, and are doing it for sport. From my research, that $20 most likely represented somewhere around $8,000 today. Presumably, the reason for putting a price on it was to limit the invocation of a civil trial by jury, but that limit is effectively meaningless today. We have no idea why they worded the specs the way they did – we probably understand their world a little better than they do ours (history is easier to understand than future). But we do have something valuable to use.

The US constitution includes a very clear requirements section – i.e. the justification for the specs. What are these requirements? “…in order to form a more perfect union, establish justice, insure domestic tranquility, provide for the common defense, promote the general welfare, and secure the blessings of liberty to ourselves and our posterity…”. Everyone hears a lot of bickering of what the intent of the founding fathers was. But there is no need to bicker. It’s right there, very clear. Their intent was not to make it so people could have guns, or to have state run health care. Their intent was not to make it so people could have abortions, or pray in school. Their intent was to form a more perfect union. To establish justice. to insure domestic tranquility. To provide for the common defense. To promote the general welfare, and finally, to secure the blessings of liberty themselves and their posterity.

So why am I writing about the constitution? Because it’s exactly the kind of thing we do at S3. Do you have a problem? I bet you have an idea for a solution. Many companies would love to talk to you about that solution, and give you what you’re asking for. But S3 doesn’t. We’re not in the business of implementing someone else’s solution. We don’t have complete knowledge of the problem you are facing. You probably don’t have full knowledge of what the system is capable of, even if you’ve used it in the past. Just like we today have knowledge that the founding fathers never did (like the fact that $20 is an inconsequential amount of money compared to legal fees), and they knew clearly what they were trying to do, you and S3 have complementary knowledge.

With constitutional arguments, don’t try to convince me that your assault rifles are legal because the constitution says they are. Convince me that it will help provide for the common defense, or insure domestic tranquility. Don’t tell me that the government should disallow prayer in schools because the constitution guarantees freedom of religion. Tell me why that will promote the general welfare or secure the blessings of liberty. With software – tell me your problem. I’d love to hear your solution. But be assured I’m going to challenge it. Because I’m going to make sure your requirements are met. My goal is not to implement your specs. The more we understand your problem, the better we can implement a solution that solves it. My goal is to give you a more perfect solution to your problem.

And hopefully, we will all continue to strive for that more perfect union.