Why no one wants software: a case study

No one wants software.

Really, no one.

What they want is documents … pictures … loan applications … insurance claims … Software is just another tool they can use to maybe produce more of the things they really want, better or cheaper.

What this means to legions of unhappy, cynical programmers is that no one cares about the quality of the code. Nope. They don’t. And odds are, they shouldn’t.

Here’s a little story to illustrate why. (By the way, this is the kind of thing you’ll see in an MBA course. If you don’t already do this kind of thinking, you should stop telling yourself there’s no value in getting an MBA.)

The pitch

I’m in charge at an insurance company. I have a manual, paper-based process that requires 100 people working full time to process all the claims.

Someone comes in and offers to build me a system to automate much of the process. He projects the new system could reduce headcount by half. It will take six months for a team of four people to build.

If you’re a programmer, and you think this probably sounds like a winner, try looking at the real numbers.

The direct cost

100 claims processors working at $8/hour. $1.6M per year in salaries. (Let’s leave benefits out of it. The insurance company probably does.)

Four people on the project:

  • architect/dev lead, $100/hour
  • junior dev, $60/hour
  • DBA, $80/hour
  • analyst/UI designer, $75/hour

Total $325/hour, or about $325k for six months’ work.

Still sounds like a winner, right? $325k for an $800k/year savings!

Except the savings doesn’t start for six months. So my first year savings are at best $400k. Damn, now it’s barely breaking even in the first year. That’s OK though, it’ll start paying off nicely in year two.

The hidden costs

Oh wait, then I need to include training costs for the new system. Let’s figure four weeks of training before processors are back to their current efficiency. Maybe a short-term 20% bump in headcount through a temp agency to maintain current throughput during the conversion and training. Add the agency cut and you’re paying $15/hour for the temps. 20 temps x $15/hour x 40 hours x 4 weeks = $48k one-time cost. Now my first-year cost is up to $373k.

And don’t forget to add the cost of hiring a trainer. Say two weeks to create the training materials plus the four weeks of on-site training. Since this is a high-skill, short-term gig (possibly with travel) I’ll be paying probably $150/hour or more. $36k for the trainer.

So if everything goes perfectly, I’ll be paying $409k in the first year. And actually, I don’t get even the $400k savings. I can’t start cutting headcount until efficiency actually doubles. Generously assume that will be three months after the training finishes. Now I’ve got three months of gradually declining headcount, and only two months of full headcount reduction. Maybe $200k in reduced salary.

Of course you need to add a percentage for profit for the development company. Let’s go with 30%. So …

The balance sheet

Software $325k + 30% = $422.5k
Trainer $36k
Training (temps) $48k
Total Y1 cost $506.5k
Projected Y1 savings $200k
Shortfall $306.5k
Y2 savings $64k/month

The project breaks even near the end of the fifth month of year 2. And that’s if NOTHING GOES WRONG! The code works on time, it does exactly what it’s supposed to, I don’t lose all my senior processors as they see the layoffs starting, etc. etc. etc.

The other pitch

Then a lone consultant comes in and offers to build me a little Access database app. A simple data-entry form to replace the paper version, a printable claim form, and a couple quick management reports. Two months’ work, and I’ll see a 10% headcount reduction. The consultant will do the training, which will only take a week because the new app will duplicate their current workflow.

Software $200/hour x 8 weeks = $64k
Training $200/hour x 1 week = $8k
Total Y1 cost $72k
Savings $12.8k/month (starting in the fourth month)

The project breaks even six months after the project is done, so early in the ninth month of Y1. Since the scope was much less ambitious, the risk is also lower.

The obvious choice

Which sales pitch do you think I will go with? Does that mean I don’t respect “proper” software development practices? And, the bottom line: should I spend more money on the “better” solution? And why?

The difference between innovation and sloppiness

I took a couple of art classes in college. In Life Drawing we were supposed to look at the arrangement or person in front of us and put it on paper. In pencil, then charcoal, then ink … watercolor … oil … etc.

I was always good at photorealism. I might not have been fast, but I had some pencil drawings that could have passed (at a distance) for black-and-white photos.

There was another student, an art major who fancied himself an “artiste”. His work spanned the range from abstract to really abstract. He looked down on my mere technical facility.

But my grades were as good as his, sometimes better. It seems when he talked about his rejection of formal rules, it really meant he wasn’t able to do realism. He didn’t have the command of the tools, the mere technical facility. So everything he did owed as much to chance as to intent.

He may have been right, that I didn’t have the creativity to do modern art. I’ll admit that I appreciate paintings that simply look good, without high-flying pretension or socio-political overtones. I guess I’ll never have a SOHO gallery showing. C’est la vie.

But with all his fervor, and whatever glorious visions he had in his head, he couldn’t reliably get them onto the paper. He couldn’t create something specific, on purpose.

[Cue un-subtle segue … ]

But what does this have to do with the business world? Scott Berkun wrote recently about how constraints can help creative thinking. When a large corporation does “blue sky” thinking, they can wander aimlessly and never produce. Constraints set a direction.

But I think there’s another problem with “blue sky” thinking that goes beyond a lack of direction. It’s summed up by Voltaire’s famous maxim:

The perfect is the enemy of the good.

When a company tries to “blue sky” a problem, they are implicitly seeking perfection: With no limits, what would be possible?

But there are always limits. They may be self-imposed, inconsequential, misunderstood, overblown, or in any number of other ways not real limits. And it helps to know the difference.

When you start out by asking people what they would do if there were no constraints, don’t be surprised when they come back with a solution that can’t possibly work. And then convince themselves that theirs is the only possible solution. By then, you’ve already lost.

Hate the game if you want, don’t pretend it isn’t being played

When I was in college I worked at a bar that had a pool room. I played a lot when it was slow and after hours. I got pretty good on those tables. Only “pretty good” and only on those tables … I knew where the dead spots were in the rails.

If I bet anything on the game, it was usually who bought the next round. Sometimes we couldn’t drink as fast as we played, and we’d go for a dollar a game. We were all friends, and it was just to make the game more interesting.

But every so often someone would come in who none of us recognized. You could usually tell really fast who was better than just a casual player. Sometimes they’d see if they could get a game for $5. I’d always take them up on it. And I always won the first game.

Then they’d ask for a rematch … let them win their money back. I’d take that game, too. Sometimes I’d win, sometimes not. But I was breaking even so I didn’t care. It was usually after the second game that they’d look at their watch and realize they had somewhere they needed to be. But they had time for one more game.

How about one last round for $50?

No.

How about $30?

No.

Come on, give me a chance to win it back!

No, you’re a better pool player than I am. But you suck at reading people. You thought I didn’t know you were throwing the first two games.

That’s when they’d get pissed. It wasn’t fair that I took their money even though they could beat me without trying hard. I was just a punk-ass bitch that couldn’t carry their stick.

Yup. But I had their money, and they were leaving.

If I wanted to, I could have spent the equivalent of a full-time job becoming a professional level pool player. I would have run into diminishing returns as I was going up against ever stronger competition. It would have dominated my life, and the only way to make a steady living would be constant travel.

Plus there’d be no retirement plan. Your earnings stop the moment you stop playing. The skills don’t translate to anything else worthwhile.

So I stayed in school and learned to be a programmer, which doesn’t suffer from any of those negatives. </sarcasm>

Hmm, that came out a lot longer (and a lot faster) than I expected. It all started from one idea, though: You don’t win by being better, you win by playing better. And you start by knowing what game you’re playing.

When you’re in an interview, you’re playing the “get the job” game. Once you have the job you’re in the “impress the decision-makers” game. If you go the uISV route you’re in the “sell the most product” game.

Being better at coding is one of the plays in each of those playbooks. But it’s not the one they keep score with.

Is your resume hot or not?

Getting a job is like getting a date.

  1. There may or may not be rules to it.
  2. It may or may not be fair.
  3. They may be so desperate it doesn’t matter what you say. But do you want to hook up with someone who’s desperate?
  4. Or maybe it doesn’t matter what you say because they’ve decided the answer is “no” before you opened your mouth.
  5. The more in-demand you are, the more choosy you can afford to be. But if you want to get this job — or this date — then you’re probably going to have to say what they want to hear. Good luck figuring out what that is.

Drug Testing as a Job Prerequisite

This topic comes up often enough online that I figured it would be more convenient to just lay out my thoughts on the subject once and point people to it. So here goes:

  • Most companies use pre-employment drug testing to filter people out because they don’t have any good performance-based measurements they can use. If they knew how to directly evaluate a potential employee, they wouldn’t need artificial criteria.
  • Some companies have a legitimate need for pre-employment testing: hospitals, pharmacies, chemical labs. Jobs where people will have unusual access to drugs or their components.
  • There are some jobs where you clearly don’t want someone working under the influence: bus drivers, teachers, etc. But if you’re going to make that argument, doesn’t it make more sense to do random screenings after they’re employed?
  • Some jobs have an inherent conflict with drug use: law enforcement. Enough said on that one.
  • As a job candidate, I would prefer that a company’s policies aligned with my own. But I recognize that most HR policies are about balancing various legal liabilities, and have little to do with the day-to-day business culture. So yes, I’ll take a test if they want one.
  • As a potential employer or hiring manager, I prefer not to know what an employee does when they’re away from work. But if my employer’s HR department requires a test, that’s the policy I’ll tell any candidates.

Please don’t believe anything in this post

In Yet Another Internet Forum Discussion About Offshoring (I hereby claim authorship of the acronym YAIFDAO) someone wrote:

A lot of decisions should NOT be left to developers to make. imho, the time to think out of the box is gone by the time it is TIME TO CODE. It’s not time to think about alternatives to what to do.

That is absolutely right. You never want developers talking to end users. They might suggest some other plan than what was painstakingly shepherded through four levels of approvals.

And let’s just squash the notion right now that sometimes there are trade offs to consider. Just because the analyst’s solution will take three weeks of coding effort and a new application server, while the programmer knows of a reusable component that will take one hour and no increased hardware, is no reason to institute the Change Control Process.

Alternatives should always be considered in isolation from the impact they cause. Implementation issues should never be allowed to intrude into the world of business decisions.

Next thing you know someone’s going to suggest that maybe mere programmers could have a meaningful contribution to make to the business process. What rubbish.

How to Kill a Project by Accident

In my last post I talked about perverse incentives in project management. Something I mentioned in passing was what happens when you don’t have positive incentives: the cases where there is simply no incentive to do the right thing. I realized there’s actually another way to get this wrong by trying too hard to get it right.

Let’s say you have just finished a project, gone into production, and it blew up. Data corruption, security problems, too slow, everything that can go wrong with a product. First you do an emergency project to fix it, then you do the after-action review to see what went wrong.

What you find is that you didn’t have good test coverage. There were whole modules that were never reviewed before going into production. It’s painfully obvious that there was a complete breakdown in the QA process.

Fixing yesterday’s problem

You’re not going to make that mistake again. You write up policies for demonstrating test coverage. You create reports to track test execution and results. You re-write employee performance expectations to align with the new methodology. (If you’ve read my stuff before, you should hear the alarm bells start ringing when you see the “m word”.)

Your next project is going exactly according to plan. Test coverage is at 90% overall, with 100% of high priority use cases covered. You’re on schedule to execute all test cases before acceptance testing starts. Defects are identified and corrected.

Then the users get it. They hate it. It doesn’t do anything the way they wanted it to. Not only that, it doesn’t even do what they asked for. How could you be so far off?

You don’t get what you don’t measure

Look back at what incentives you created. You reward doing the methodology, following the checklist. Test coverage is great, but it’s not the goal of the project. The goal is to provide something of value to the users … or at least it should be. Did you include a line in the new process that explicitly says, “Check with the users that it’s doing what they need”?

So how do you create the right incentives? Just flip the emphasis. Instead of saying an employee’s performance evaluation is 80% following the methodology and 20% client satisfaction, turn the numbers around. Your users don’t care that you followed “best practices.” They care that the product does what they need. Where is that measured in your methodology?

How to Fail by Succeeding

Dave Christiansen over at Information Technology Dark Side is talking about perverse incentives in project management, which he defines as:

Any policy, practice, cultural value, or behavior that creates perceived or real obstacles to acting in the best interest of the organization.

One class of these perverse incentives comes from the methodology police. These departments exist to turn all processes into checklists. If only you would follow the checklist, everything will work. But more importantly, if you follow the checklist you can’t be blamed if you fail.

How’s that for perverse?

A great example is that there is rarely any incentive to not spend money. Before you decide I’m out of touch with reality, notice I didn’t say “save money.” I said “not spend money.” Here’s the difference.

IT by the numbers

Let’s say you do internal IT for a company that produces widgets. Someone from Operations says that they need a new application to track defects. If you follow the checklist, you:

  • engage a business analyst, who
  • documents the business requirements, including
  • calculating the Quantifiable Business Objective, then
  • writes a specification, which is
  • inspected for completeness and format, then
  • passed on to an architect, who
  • determines if there is an off-the-shelf solution, or if it needs custom development.

At this point you’re probably several weeks and tens of thousands of dollars into the Analysis Phase of your Software Development Lifecycle. Whose job is it to step in and point out that all Ops needs is a spreadsheet with an input form and some formulas to spit out a weekly report?

Let’s put some numbers to this thing.

Assume the new reporting system will identify production problems. With this new information, Operations can save $100,000 per month. A standard ROI calculation says the project should cost no more than $2.4-million, so that it will pay for itself within two years.

Take 25% of that for hardware costs, and 25% for first-year licensing, you’ve got $1.2-million for labor costs. If people are billed out at $100/hour – and contractors can easily go three to four times that for niche industries – that’s 300 man-weeks of labor. Get ten people on the project – a project manager, two business analysts, four programmers, two testers, one sysadmin – and that’s about seven months.

If everything goes exactly to plan, seven months after the initial request you’re $2.4-million in the hole and you start saving $100,000 per month in reduced production costs. Everyone gets their bonus.

And 31 months after the initial request, assuming nothing has changed, you break even on the investment. Assuming the new system had $0 support costs.

But what if …

Way back at the first step, you gave a programmer a week to come up with a spreadsheet. Maybe the reports aren’t as good as what the large project would have produced. You only enable $50,000 per month in savings. That week to produce it costs you $4,000 in labor, and $0 in hardware and licensing.

You are only able to show half the operational savings, so you don’t get a bonus. You don’t get to put “brought multi-million dollar project in on time and on budget” on your resume.

And 31 months after the initial request, the spreadsheet has enabled over $1.5-million in operational savings.

How to Finish IT Projects Faster with Less Documentation

If you’re responsible for running an IT project you want things to be done on time and within budget. So how do you set your schedule and budget? Hopefully you define what you want to accomplish, and then ask the developers how long it’s going to take. If you’re putting out a request for proposal (RFP) you’ll have several different answers to that question. Typically the highest consideration in the selection is the total proposed cost. But really, the total time is a better choice.

Why that’s true is based not on ideas about processes, but on ideas about people.

Consultants live and die by billable hours. In the short term, they don’t have any incentive to finish their current project any faster. But in the long term, finishing faster should lead to more work as clients come to respect their ability to meet a deadline. If project managers and clients learn to value that behavior, that is.

How it Could Be

Let’s look at a production support issue for an example. Production support is completely different from most project work in one very important way: the problem is well defined. Something worked on Monday, it doesn’t work on Tuesday. Make it work just like Monday again.

For something with six-figure impact per hour of downtime – and if you think that’s an artificially-high number you’ve never worked with credit card processing – you don’t want a programmer with an impressive resume, dozens of certifications, and decades of experience with your primary programming language. You want Bob, the guy who wrote the system from scratch and demands $1k per hour with a four-hour minimum.

Once you’ve got Bob and his hand-picked team of support people, you get out of their way and let them work. Status reports might be no more than, “We’ve found the problem … We’ve identified the solution … We’re ready to test the fix … It’s live.”

How it Is

But when it comes to new development, companies play it “safe” and look for the best qualifications on paper. They hire based on keyword matching and offer rates based on industry standards for a given skill set. They require specific processes and deliverables (pet peeve: when did “deliverable” become a noun?) and status reporting becomes a significant percentage of the total budget.

Why the Difference?

There are two reasons expert teams get away with less formal process than is typical, but I can only prove one of them. The public answer business sponsors tell themselves to justify the exception to “official methodology” is that the experts have worked the methodology for so long that they can follow the same procedures without exhaustively documenting all the steps. And there is some truth to that.

But I suspect the larger reason is that experts get the work done so much faster there just isn’t enough time for documentation to build up.

The best athletes make things look easy that most people could not even do. A high jumper might clear six feet without even trying hard. Most people would never come close even with months to try.

The best IT people do the same thing. They complete projects in weeks that other people could never do. As “safe” projects drag on specifications are refined, status reports are produced, contracts are negotiated, updates are requested and provided. Meanwhile the expert team has just released to production – so it must have been a small project.

The hard part for the client is to recognize the difference between a project that went smoothly because it was easy, and one that went smoothly because the team made it look easy. But here’s the secret: You don’t really need to recognize the difference.

How to Do Better

The reason you hired someone else to do the work is because you couldn’t do it yourself. Which means you can’t accurately judge which projects really are hard, and which ones just look hard. So don’t judge the project, judge the people.

The people who seem to always be working on small, simple projects – after all, they always go quickly with no major problems – are better at execution. They will be better no matter what the project is.

A Tale of Two Techies

You’ve just finished learning MS SQL 4.2 and VB 5 in school. You get a job with a company that has just upgraded to those languages. You get to learn the ins and outs of the languages along with the rest of your co-workers.

Two years later you want to upgrade from your entry-level salary. You know that big raises only come from job hopping, and you see that most of the job ads are for VB6, so you start studying it on your own. You find some cool new things you think would help in your current job and start pushing people to upgrade.

Your tech lead, Bob — who has been with the company for seven years — says the current applications are stable and it’s not worth the cost to upgrade. The boss listens to Bob instead of you. You say bad things about Bob on an internet forum.

You find a new contract gig working with VB6, and making almost as much as Bob does. Boy, Bob sure is dumb. If he had any balls he’d have jumped already. You start racking up frequent-flyer miles chasing the next gig. Bob evaluates and recommends a new third-party tool that uses VB6. He gets a VB6 class for his whole staff included in the project cost.

Five years later, you’re a .NET hired gun and you know which airports have the best frequent-flyer clubs. You’ve got a fat bank account and all the best buzzwords on your resume.

Bob is still with the same company, but now he’s the IT Director. He’s not making as much as you, but he’s vested and his 401 is looking pretty good. He hasn’t touched any code in a couple of years, but has a few long-term employees working for him whose opinions he trusts. He also hasn’t answered an after-hours page for a few years.

You meet Bob on a street corner one day and talk about old times, catch up on what’s been happening. Suddenly a car jumps the curb and puts you both in the hospital. Oops.

You were smart enough to get good medical insurance, but your income stops since you’re not billing hours any more. Bob goes on medical leave. His wife takes his two children out of junior high and comes in to visit. Your pregnant wife comes in to visit. (You spent your twenties traveling, so are just starting your family.)

Three months later, Bob goes back to work part-time while you sit at home, surfing the net, searching for a gig that will give you the flexibility you need to work around your physical therapy.

By the end of the year, your savings are gone. Microsoft has released the Next Big Thing after .NET, and you don’t have any work experience with it on your resume. You’re applying for maintenance gigs on “legacy” apps — two-year-old .NET apps written by guys straight out of school, who just left for their first not-entry-level jobs. Maybe in another year or two you’ll be able to climb back onto the leading edge.

Bob just accepted an internal transfer to run the division he’s been supporting for the last decade. He recommends as his replacement the long-term employee who filled in for him during his absence. The last division head held the job for 15 years until his retirement. Bob could do the same, and retire with a decent pension when he’s 60.