Learn about Linux
Download Linux
Get Linux helpSpecial Offers
Feature
Michlmayr advocates time-based release management for FOSS projects
Share
Print
CommentsMichlmayr says that FOSS projects have two characteristics that make them unique in software development. First, their members are geographically dispersed, which delays communication. Second, many members are volunteers, and, although highly motivated, may not always be available at crucial times. These features have some advantages -- for example, members' dispersal means that all communication is recorded -- and are certainly preferable to centralized, paid workers who lack motivation. However, "taken together," Michlmayr says, "these aspects can make coordination and release management quite difficult."
In particular, these characteristics often result in a lack of planning. "Quite a few large and complex projects adhere to a 'release when it's ready' policy, but this strategy is disastrous in large projects where there is always something to be done," Michlmayr says. "It can lead to delays, out-of-date software, and frustration, and it also means that users and vendors cannot plan, because nobody knows when the software will actually be released." In the draft of his thesis currently online, Michlmayr singles out Debian as a prime example of these problems.
Left unsolved, these problems can lead to "loss of credibility and fewer contributors," Michlmayr says. "Why would volunteers want to contribute to a project when it takes years for their contributions to be shipped to actual users? Loss of credibility is particularly problematic because it's a vicious circle. If you repeatedly face delays, nobody will believe that the project can actually release on time, and further delays are inevitable."
The solution to this situation, Michlmayr says, are regularly scheduled releases. He concedes that converting to a schedule requires "considerable work," but he points to GNOME as proof that it can be done. In fact, he considers it a necessity when a project begins to have trouble with its release strategy.
"Many problems with release management occur because processes that worked well when the project was small are not suited to a large project," Michlmayr says. "In a small project, it's possible to say, 'release when it's ready.' In a large project, such an unorganized approach will lead to significant problems."
The advantages of time-based releases
According to Michlmayr, a move to time-based releases benefits everyone involved. For large organizations using the software produced by the project, time-based releases provide the predictability they need for their own business and infrastructure planning. Similarly, commercial distributors can plan their marketing, while individual users can have increased confidence in the project.
However, the largest benefits of time-based releases are for the project's own development community. Over time, Michlmayr sees regular releases as increasing the discipline in the community, allowing its members to coordinate their efforts more efficiently. Regular releases also allow better planning in both the short and the long term. In addition, as part of the cycle, user feedback becomes more timely and can be incorporated into planning with less effort.
By contrast, Michlmayr says that "so far, no clear disadvantage has been found" for regular releases. He notes that it is uncertain "whether you can make major changes while still following a time-based release strategy," adding that GNOME has debated this question in talking about how to develop GNOME 3.0. He says that "this question cannot fully be answered yet," and suggests that "it's possible for features to be developed on a branch over more than one release interval," which would make major changes compatible with a regular release strategy.
Knowing when to move to time-based releases
As with addiction treatment, the first step in moving to time-based releases is to acknowledge that your current strategy doesn't work, Michlmayr says. He gives four criteria for knowing when the solution might be appropriate:
- Significant new fixes and features are added with each release. When a project is small enough that releases are incremental, time-based release is probably an unnecessary discipline.
- Distribution is cheap. In other words, time-based release is not practical for commercial shrinkware, but is for free software downloaded over the Internet.
- The next release does not require specific new functionality or major changes. That excludes work done for a specific client, or for a commercial product that needs new features to convince users to upgrade.
- The code is modular, so that individual components can be developed and released independently of other ones. This requirement helps to provide flexibility that makes meeting deadlines easier.
Once a project has decided to move to regular releases, it needs to choose an appropriate release interval and to develop detailed schedules, policies, and infrastructure.
For Michlmayr, a particularly important part of the transition is the appointment of an effective release manager. He defines such a manager as someone with both management skills and the ability to build communities, develop a strong vision of where the project is heading, and to stay on target and refuse requests when necessary.
He also emphasizes that release managers must "actually perform hands-on management. Some projects believe that simply publishing a release date or schedule is enough and that project participants will automatically do their work. In reality, it's important to constantly remind people of the schedule and of looming deadlines."
Applying the findings
Michlmayr declined to suggest a detailed set of best practices for release management in FOSS projects. "That would be a whole Ph.D. (or three) on its own," he says.
However, Michlmayr did say that his research was not only potentially useful for FOSS projects, but for any volunteer effort.
"With the Internet, there will be more collaborative, volunteer projects in the future," he says. "The insights we've now gained with free software can be applied to these projects. My research on release management in free software projects can be seen as a way to approach the question as to how products can be released by projects with little control over their contributors."
Bruce Byfield is a computer journalist who writes regularly for NewsForge, Linux.com, and IT Manager's Journal.
Comments
on Michlmayr advocates time-based release management for FOSS projectsNote: Comments are owned by the poster. We are not responsible for their content.
What is need is a preferred value for project to work to at least achive.
Is this 2 weeks 1 month 6 months 12 months.
Something that has never been talked about in OSS to standardize.
It all depends on the type of the software project
Posted by: Anonymous Coward on March 28, 2007 11:27 PMSomething like Wine may do well with a relatively rapid release schedule while something like 2 years / releasemight suit, for example, Debian/GNU Linux.
I can tell you that if I accept the ones for X and Kernel, my system will not boot. But I can accept the others.
My other experience has been with RPM / YUM failing because of a faulty update to some component. I have learned that these two critical softwares use shared libraries, and one of the non-mainstream applications updated one or more of these, and that update caused the YUM failure.
Automated testing did not have find this problem, as neither YUM or RPM was modified, I will not conjecture why.
So, automated testing or no automated testing, as the number of applications / modules increases, the testing time / schedule should also increase, particularly if we want to maintain quality.
Any loss in quality/reliablity is wonderful ammunition to the two major competitors.
I am very much in favor of moving to a 9-12 month release schedule. To accomodate the must have individuals, the test versions could be "forked" so that they may enjoy their daily updates.
Any plan to fix the problems with OSS, must take this deliberate sabotage, into account.
Jade @ <a href="http://linuxhelp.150m.com/" title="150m.com">http://linuxhelp.150m.com/</a 150m.com> (<a href="http://m.domaindlx.com/LinuxHelp/" title="domaindlx.com">http://m.domaindlx.com/LinuxHelp/</a domaindlx.com> mirror) where there are HOWTOs on:
1) cloning your windows XP/2000 installations using Linux (back-ups),
2) installing windows XP/2000 on a spare partition with Linux,
3) accessing and writing to Windows XP (formatted with the NTFS) from Linux,
4) a script to walk you through a Gentoo Linux installation,
5) remix those 14 Debian installation CDs as 2 DVDs,
6) the entire book "Linux Device Drivers 3" as a single web-page (ie in HTML format),
7) 3D acceleration for ATI cards (simple procedure, works for SuSE and Mandriva and Debian),
8) some discussion on the GPL and non-free third party kernel modules,
9) compiling the worlds best DVD/Movie/Video/MP3 Player and Encoder (MPlayer and MEncoder),
10) some politics, eg: Israel Fakes a Provocation for War (the "kidnapping" of Cpl Shalit),
11) and a detailed comparison of many common filesystems.
I am not in the testing business but this cartesian product tells me all the possible testing scenarios (in terms of loaded modules) I can have when I limit myself to exactly 2 modules loaded at one time [this includes loading the same module twice].
To consider all the possible ways to have any number of any of the modules, assuming that the order of loading is irrelevant (which it isn't), compute 2 raised to the 4k power. That is a huge number, and if we consider order of loading to be meaningful, then we have 4k factorial plus 4k-1 factorial plus 4k-2 factorial plus<nobr> <wbr></nobr>... which is even larger though we are already in the realm of meaninglessness.
I got the 2**(4k) because that is the sum of the number of combinations of of 4k taken 1 at a time plus 4k taken 2 at a time plus<nobr> <wbr></nobr>... 4k taken 4k at a time [use combinations (not permutations/factorials) if assume loading order is irrelevant and sum all of these since we might load 4k modules but we might load only 10 modules or 4k-1 or 954 etc. And the sum of all of these combinations is 2**4k (sum of the (4k+1)th row in pascal's triangle)]
Thus we will always be short of human beings and computers to do a thorough testing. Fortunately, if the modules can be determined to be independent of each other (as much as this can be the case) then we just have to test 4k cases. Anyway, the reality is somewhere in between, but a thorough testing of just one of these gazillion combinations is impossible in most cases anyway. And if you think I am crazy with the calculation above (which assumes that a test can be done to absolutely determine if a combination of loaded modules works.. such a test is only theoretical anyway so we have 2**4k theoretical tests)... if you think I am crazy, then consider that each module has at least one bit of state. We are thus looking at 4k bits as a lower bound. The total number of such different states (assuming each different set of bits is a different condition which is a good assumption if the modules aren't identical), we get 2**4k different states once again. And this is basically a lower bound because each module has more than 1 bit of state. To recap, we have something like 2**4k module scenarios on which we must do a test so comprehensive as to definitely ascertain if there is any flaw or not. Alternatively, we can create a checklist of all the possible correct configurations for the states. This table would be 2**4k in size if each module had only 1 bit of state. In reality the table would be much much larger (eg, if 10 bits of state per module => 2**40k).
In short, 1 day or 100 years is almost insignificant if we wanted to be comprehensive and not rely on any shortcuts. In practice, we can probably study the code for correctness (even if nonrigorously), then test boundary cases (as errors are flagged or as interesting situations are discovered), use debugging tools, do unit tests, etc. So the design of Linux (making modules as independent as possible) is the most important element for coming up with a testing framework that is manageable and that we can hope will be able to catch a nonnegligible fraction of possible errors. To repeat, if we can approximate module independence (and this was all there was to it), we would only need 4k theoretical tests instead of the much much larger 2**(4k) such tests.
In real life, we are doomed to be imperfect by far, but we can survive.
[Notice that if a single application (more complex than a much smaller limited module), can use up a gig of memory, and we wanted to look at the full state, we need to look at at least 2**1gig as these are all the ways to stuff 1 gig with 0s or 1s and presumably each of these states would be independent of the rest in terms of determining correctness. As an example, consider only the pixel memory used by Firefox. We would look at all the ways to pixilize a monitor and some would be correct and others would be off on some pixel value. In reality though, not only can we ignore many of these cases (a pixel being off is frequently not noticeable to the eye), but we have in practice much less than 2**1gig degrees of freedom (eg, some bitmaps will always be rendered identically.. at least we may come to that conclusion by studying the correctness of the relevant functions and thereby eliminate zillions of possible states from consideration). Also, note that we are not just looking at a full screen of bits. Each full screen is in the context of more state. If I go to google.com, I don't want to see microsoft.com or any other webpage. In short, thorough testing of a complete system involves doing an infinite number of tests, for all intents and purposes. Being smart about it though means we will strive for independence of modules and basically ignore combinations except in special circumstances. This is linear growth (if modules are roughly equally complex to build as their numbers grow). Then it makes sense to talk about 9 months vs. 3 months or a week, but Linux devs are growing too. In essence let one or several people be responsible for a module, which is what happens anyway. Naturally, in real life, not all things are equal: combinations of apps/modules must be considered; human management/communication and other logistics become issues; etc.]
Sorry for overdoing this post. Just think that the cartezian discussion was thrown in there without a base and it would lead to wrong conclusions. It would make sense, for example, if after considering everything, we estimated that testing 2 modules at one time provided fairly decent test coverage of likely errors.
In my view, I feel that there is a drop in quality in the linux deliverables during the past two years, and I attribute that to inadequate testing. By using the cartesian product, it was my attempt to indicate that the number of potential tests is growing at a rate that recommends a longer period between releases, and a longer settling in time.
Any study better jive with reality. Above are two with experience advancing successful projects (successful in my opinion at least) giving their two cents.
Here is my argument as to why we should slow down.
Two years ago there were about 6000 modules in linux, with around 4000 being critical. The cartesian product of 4k by 4k is what we should have as critical tests to confirm correctness of execution. In these tests, when a bug is found, partial retesting is required. Suppose there were really 4k*4k or 16k tests to assure that release.
Today (two years later), with enhancments to X, to the kernel, virtual operating system execution, to the applications, and growth in number of new applications (modules), we have over 8,0000 modules of which 6000 are easily installed (a count I derived from my own linux desktop system).
Drawing on the same logic, we should have 6k*6k or 36k tests to perform to assure a clean distribution.
The qualitive ratio of module counts from two years ago to today is 36/16, or more than double the number or required tests to maintain the same quality.
But we do not have double the number of test people, and we have not increased the time between releases. Nor have we automated the testing procedures.
My take on this is that I have definitely experienced a very much larger number of bugs in the maintenance updates now,from what I experienced two years ago. In fact, I would say that the number of bugs has gone up by the square of the number of bugs reported two years ago, or more than the ratio of 36/16.
My conclusion is that we require a kind of Moores law for deciding when to schedule/time a new release.
As the number of modules increases, so must the testing times increase exponentially.
My experience over 40 years in IT/development leads me to recommend fixed releases, but not necessarily on a periodic (example 6 monthly) schedule, but rather that they be based on the number of program changes that occur in a 3 month period following the previous release.
Leslie Satenstein
From October to today, my favourite linux distribution has listed over 325 bug fix patches. There should only be not more than 2 per week or 20.
[..] the time based release strategy can be considered as an important means of quality improvement in FOSS projects.
Kudos to Martin to honestly have highlighted that there are problems in Open Source projects, he also stressed the importance of Regularity and the Use of schedule. As a matter of fact the use of schedule claims a project management function (release manager), reducing somehow the degree of independence among contributors. Our research in this respect stated that:
[..] a pure modular structure - that is one lacking of hierarchy, such as a market - embeds flexibility, but it lacks coherence, the ability to coevolve after adapting to change.(cfr. Langlois Richard “Do firm plan?” 1995)
A hierarchy is a must, then, when you need to manage a complex activity coordinating many contributors, either volunteers or employees. Martin makes clear that policies and infrastructures are needed to support his release strategy.
Reading the paragraph “Limitations and Future Research” I would suggest another question:
Introducing time-based release management could move developers’ focus from software’s effectiveness to meeting release targets? How to balance the trade-off between time and quality?
Eclipse
Posted by: Anonymous Coward on March 28, 2007 03:26 AM#