This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new!

Feature: Open Source

How useful are 'proprietary vs. open source' TCO studies?

By Robin 'Roblimo' Miller on April 07, 2004 (8:00:00 AM)

Share    Print    Comments   

We've all seen Microsoft's latest barrage of studies that say their products are more cost-effective than the competition. Naturally, many accuse Microsoft -- and other companies that publish outside studies that support their corporate goals -- of bias, if not outright deception. To get a little more insight into the value of TCO studies, we turned (via email) to David A. Wheeler, who has done more than a few software cost-comparisons himself, for advice about how much we should (and shouldn't) trust "sponsored" research.
NewsForge: Microsoft has recently been claiming -- loudly -- that the total cost of ownership (TCO) for its Windows server products are lower than the cost of equivalent Linux and open source software. Is this true?

Wheeler: Not in the way Microsoft wishes. TCO is extremely sensitive to a specific circumstance, so a TCO for one situation doesn't usually apply to other cases. I'm sure that there are cases where Microsoft's approach has a lower TCO than alternatives, so in those specific cases it's true. However, there are also cases where open source software or Linux-based solutions have a lower TCO. You really have to consider all the costs for your specific situation, and your results may differ.

That point is made by some of the papers that Microsoft is referencing, but it's not restated in Microsoft's "Get the Facts" Web pages.

Also, note that almost all of their "independent" studies were actually funded by Microsoft. You should consider suspect any study of a vendor that's funded by that vendor -- especially if that was the only funding.

NewsForge: Why are self-funded studies so suspect?

Wheeler: The short answer is, "because organizations self-fund public studies to give them good press, not necessarily to give customers a full understanding." And I say exactly the same thing about IBM, Apple, Red Hat, or anyone else who funded a public study reviewing their own products. Companies have a shareholder obligation to maximize profit, not to provide truly independent assessments to potential customers.

I doubt that these studies just made up their figures, but the problem with self-funded studies is that it's so easy to skew studies in more subtle ways:

  1. A funder can control the study's setup. For example, a funder make itself look good by asking an evaluator to only look at a few specific factors (ignoring others), or only look at specific environments and situations. In the old 1999 Mindcraft studies, for example, Microsoft chose to only evaluate an extremely unrealistic environment favorable to itself.
  2. A funder can control exactly how the study measures its results. That can make a significant difference, since different measurement approaches can produce wildly different results. If the study uses samples, it's easy to bias a sample to produce biased results.
  3. A funder can also control the study outputs. For example, maybe many factors were measured, or many separate studies were made, and only the favorable ones were reported. Conflicting results could have been suppressed. Or perhaps some of the key controlling variables weren't explained or controlled. The results can even be correctly described in a misleading way (for a humorous example, see the information about dihydrogen monoxide).

But let's also give credit where credit is due -- at least most of the studies acknowledge that they were funded by the vendor. Back in 1999, when Microsoft funded the original Mindcraft study, early reports didn't acknowledge Microsoft's funding at all. Yet that study was funded by Microsoft, the Microsoft systems were specially optimized by Microsoft engineers for the test, and the tests (including those of Microsoft's competitor) were even performed at Microsoft. There was an understandable outcry!

In contrast, if you look at the current crop of studies carefully, all but one of the "independent" studies referenced by Microsoft acknowledge that Microsoft funded the study (I didn't find any such statement from Embedded Market Forecasters; perhaps it was truly independent). IDC, to its credit, places the statement "Sponsored by Microsoft Corporation" in bold letters right under the author names, so it's hard to miss, but its report isn't even in the independent list (though it's in another list that Microsoft provides). So I commend those study authors for acknowledging this potential conflict of interest. In a few places, Microsoft's "Get the Facts" page even acknowledges when a study was funded by the company, but it really should specifically identify every self-funded study (not just some of them).

There may be useful information in the self-funded studies, but I don't have any way to be confident with them. There may have been no manipulation at all, but the money flow creates a strong incentive for it, and there's no way to know otherwise. The Object Watch study does claim that there was no editorial control, and suggests that the funding wasn't total -- that's very encouraging, but it's also very hard for someone like me to verify. Most of the other studies don't even say that. The problem is that self-funded studies have a built-in conflict-of-interest that an independent observer can't really examine. Even indirect funding can be a problem ("give me a good report, and I'll give you some/more money later for something else").

What's really needed is more independent studies that are clearly independent, and not funded directly or indirectly by a vendor.

NewsForge: You often come across as an ardent Linux partisan. Aren't your studies suspect because of that perceived bias?

Wheeler: Actually, I'm not a Linux advocate. I'm an advocate for considering the use of open source software / free software (OSS/FS). As I clearly state in my "Why OSS/FS? Look at the Numbers!" paper, I think it's a serious problem that "many people fail to even consider OSS/FS products." In fact, my paper's goal is to "show that you should consider using OSS/FS when you're looking for software" (and many more consider OSS/FS now than when I first wrote the paper). But as I also note in the paper, "I use both proprietary and OSS/FS products myself."

I work hard to be unbiased. In particular, I wasn't paid by either side (proprietary or OSS/FS) for writing my papers contrasting them. You can (and should) "follow the money," but in my case, you'll find I have no incentive to be generous to either side.

Do I perceive some advantages for OSS/FS? Sure, there's no point in considering an option if it has no advantages. OSS/FS tends to be more flexible (since you can modify the code), and the openness of the code has fundamental advantages for security. Mature OSS/FS tends to have a lower initial purchase cost, though total cost calculations are more complicated. Most importantly, OSS/FS frees users from the control of any particular vendor; a user can later self-support or switch to a different supplier of that same software, options unavailable to proprietary users. I believe in the value of competition, and anything that introduces competition into a market (as OSS/FS is doing) usually has a very positive impact. But a particular proprietary program can have key advantages over a particular OSS/FS program, and that's the sort of comparison you have to consider on a case by case basis.

NewsForge: Who can we trust to do independent studies? Is anyone truly independent and unbiased?

Wheeler: In the end, the only way to be really sure that you have unbiased results is to do the comparison yourself -- which you have to do anyway, because some measures like total cost of ownership (TCO) and performance are incredibly sensitive to specific environments.

Before you do your own measures, you can certainly try to gain insight from other reports. I highly recommend trying to identify how a given report was funded, and giving more weight to reports that were clearly not paid for by any side.

But even potentially biased reports can give you some useful data, as long as you're careful with them. A report paid to review a vendor's own product will often raise issues that vendor thinks are to the vendor's advantage -- but those issues might be very important to you, and thus worth thinking about (and examining the competitor for that attribute). Also, these vendor-sponsored papers often identify who that vendor thinks is valid competition -- so make sure you include that other vendor in your evaluation! For example, Microsoft has information comparing to Microsoft Office (previously noted in Slashdot). So as an acquirer, that's a tip-off that if I'm thinking of buying/upgrading Microsoft Office, I'd better also consider

NewsForge: You say, "What's really needed is more independent studies that are clearly independent, and not funded directly or indirectly by a vendor." Who will do these studies? Who will pay for them? And do you know of any already out there we should look at?

Wheeler: If that were easy to answer, there wouldn't be a need for more independent studies . But I think part of the answer is in groups and organizations that are funded by potential customers, not vendors. Consumer Reports is a good example of this (though they don't focus on software reviews). Magazines can sometimes play this role, though magazine funding is often dominated by vendor advertising, making it difficult to stay objective. And I can imagine organizations banding together (each offering a certain amount of money for a particular review) until they can actually fund a particular review.

Often some of the most interesting and objective studies are from people who are really interested in investigating something else, and through their investigations find interesting new information. The Fuzz studies were like that; here were academicians who devised a new method for measuring reliability, and decided to use it to measure both proprietary and open source software. There was no monetary reason to report one way or another; they simply needed results to demonstrate the method. Reasoning, Inc., has used its tools to examine the source code of proprietary and open source software; their goals are to market the value of their tools and services, and don't care which software is "better."

Another great source is previous customers who have already done the analysis themselves. After all, if you're looking at the alternatives, others have probably done so before you. Please, please, please -- if you've done an in-depth analysis of products on a particular subject, post them on the Web, or at least offer them for sale! You'll get free advertising for your organization, and you'll get free useful corrections and clarifications.

As far as what studies should be looked at, my Numbers paper tries to identify any cases where I suspected a potential conflict of interest for the ones claiming an OSS/FS advantage.

But in the end, as I said before, the best independent study is the one you do yourself.

Share    Print    Comments   


on How useful are 'proprietary vs. open source' TCO studies?

Note: Comments are owned by the poster. We are not responsible for their content.

Link to story broken

Posted by: Anonymous Coward on April 07, 2004 08:43 PM
Infinite re-direct.


Man-hours is where the cost is.

Posted by: Anonymous Coward on April 07, 2004 11:28 PM
The costs for software-licenses are in most cases quite low today, atleast if you buy microsoft software and not some overpriced IBM software.

Say for example that you need some simple entry-level SQL database and will do with a $1500 SQL server from microsoft. You only get a person setting it up for a couple of days for this price. The costs for man-hours will most likely be a lot higher than the license-cost.

To sum it up: Easy of use means less man-hours = in most cases lowest cost.


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 07, 2004 11:56 PM
That would depend on the application, which is the entire point of the article. Says so at the top.


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 01:33 AM
"Say for example that you need some simple entry-level SQL database and will do with a $1500 SQL server from microsoft." Then why not a use MYSql for the database. It's free to use. You would only have to pay for a commercial license is if you don't want to distribute your code under the GPL...


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 07:06 PM
Because it typically takes more manhours to use it.


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 01:55 AM
The parent post gave an example of a software license fo $1,500 and then stated that it would only pay for a few days of effort.

I believe that this poster is assuming that the need is in a company in the "Western world". If the database is needed by a small community project (like for a church group or a class project in a school), then $1,500 would be prohibatively expensive, regardless of how easy it is to set up. Similarly, if the need is for a commercial entity in a "developing nation", $1,500 could well represent many month's salary of an IT professional (and would require a "hard currency" which can be difficult to come by). In either situation, the Open Source database has the transparancy required to be implemented successfully by the volunteer/student/low-wage worker.

Furthermore, the $1,500 price tag will grow rapidly when one needs to purchase a copy for every user, server, and/or project that will use the database. If, instead, an Open Source database were selected, then the knowledge gained in setting it up would apply to all subsequent installations; amortizing the labor cost accross *all* such installations could eliminate the difference in the two systems, restoring license fees as the primary diferentiator between the two solutions.

Such is the entire point of the article... one can never make blanket statements about which is the "correct" solution. One must always evaluate the product with respect to the particulars of the situation.


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 04:12 AM
MySQL is easy to set up. In other words, just because something is open source, it does not necessarily follow that it's hard to set up.

Also, your Microsoft SQLServer solution doesn't seem so cheap when you factor in that the initial $1500 cost is only for 5 database connections, additional connection slots cost more and an appropriate Windows server licence is required.


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 06:09 AM
Last I heard, setting up a Windows server AND setting up SQL Server properly took a couple of days and the help of Microsoft support as well IT personnel. May have changed in the last couple releases, of course.

OTOH, if you look at Windows 2000 Server Group Policy and what a mess that was, which has now been addressed by making a special Management Console for it in Windows 2003 Server, clearly ease of use is not always present in Windows products either.

Yes, man-hours always cost more than initial purchase price (and usually even more than maintenance contracts) - but the point of OSS is that there frequently IS NO initial purchase price . And that means less money. Only IF the OSS product costs more man-hours to maintain and use is the TCO better with commercial. And it is common knowledge and backed by studies that a well-trained UNIX sys admin can support more boxes and users than a well-trained MCSE.

Script monkies tend to be more efficient than mouse monkies. (Although they tend to underestimate and under-report the time to develop and maintain those scripts.)


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 06:26 AM

Say for example that you need some simple entry-level SQL database and will do with a $1500 SQL server from microsoft. You only get a person setting it up for a couple of days for this price.

This brings up several points about such costs:

First, I've never spent more than a couple of days setting up PostgreSQL and pgaccess, or some similar set of programs that constitute an "entry level" database. I've done this, BTW, on several different platforms.

Second, there are startup costs with any DBMS, no matter how easy it is to install or use. Calculating these accurately is, at best, a dubious proposition.

Third, as you've alluded, man-hours aren't always the principal cost. If you have to deliver 100 of these systems, programming labor is much less important.

Like the man said, you have to calculate costs based on your situations. Even then, you'll be guessing a lot.


Re:Man-hours is where the cost is.

Posted by: smurfnsanta on April 08, 2004 11:32 AM
Under ease of use, I've had at least triple the problems with developers and users on MS servers as I've ever had with linux.

What MS seems to ignore are things like syslog, logwatch, clamd, tripwire, and a miriad of other apps that help maintain security and data integrity. $1,500 doesn't even begin to cover the utility's cost in MS expenses for the apps needed maintain what I consider to be a stable, secure server. In many cases MS just doesn't have the functionality, which seems crazy considering renewing licensing fees (which seems to be completely ignored by MS's lit).

New admins are surprised when they are emailed that users attempt to access / change files they don't have permissions to, or when databases, tables, partitions grow to prohibitive sizes or fill up. Correct setup removes a lot of the guess work and tedium from admin'ing large systems. As long as a competent admin is setting up those servers, I'll take linux every time. The admins may cost more, but guess what? Their work is worth several multiples what MCSE's get paid to do on similar MS systems.

Note: I wouldn't bother to post this, but I've received dozens of "Wow!" and "Thanks for the Linux server, transactions are 4 times faster!" messages from MS admins from dozens of clients whose ass has been saved by well setup Linux servers. And none of those linux clients has failed to ask for a linux equivalent when installing additional servers. That says something.


Re:Man-hours is where the cost is.

Posted by: Anonymous Coward on April 08, 2004 10:44 PM
I can't remember taking more than 6 or 8 hours to set up either postgres or mysql -- even when I didn't know what a database was.

The last time I did it -- on RedHat 9 I installed webmin, and set up the database all in under an hour. That still leaves most of the posters 'couple of days' to spend on other things.

Now lets look at something else. To pay for the $1500 in a couple of days would imply that the cost was going to run $93.75/hour. The big companies may be willing to pay that much for contract labor to some of the big contract houses, but where I live (a small town in Texas, USA) $50/hour would be considered kind of high. at $50/hour that $1500 would buy 30 hours, or just short of 4 days of setup time. The reality in this area is closer to $30/hour which changes the picture to more like 50 hours or 6 days and a couple of hours.

So, lets say that I'm going to help a customer switch their database to an open source database.
1) we will probably need another machine $600
2) we will need to install linux (2 hours) $60
3) we will need to install webmin(1/2 hour) $15
4) we will need to set up the database and users(1 hour) $30) -- the time here depends on the complexity of the database, tables and user list.

So the cost for a Linux based solution would be $705 in this case. What would it be for windows? would they need the aditional machine anyway? if they need the aditional machine, then the install time will be there. and your not going to get out of the database, table and user setup.

I don't understand how windows is less expensive than linux, unless you insist on paying Red Hat for tech support that you never use, and compare that with Microsoft's purchase price. If your not going to use Red Hat's tech support then why not go with White Box Enterprise Linux or Lineox Enterprise linux( which both provide what is essentially RHEL 3.0 with patches for a free download(legally).


So why do these ads appear on Newsforge ?

Posted by: Anonymous Coward on April 08, 2004 02:21 PM
It is quite strange that Newsforge gets sponsored by
these very ads?


This story has been archived. Comments can no longer be posted.

Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya