This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature: Linux

Replacing init with Upstart

By Scott James Remnant on October 04, 2006 (8:00:00 AM)

Share    Print    Comments   

For years, most Linux distributions have been using an init daemon based on the one found in Unix System V. The init daemon is spawned by the kernel itself, and tasked with booting the rest of the system, starting all other processes, and taking care of them when they need to be stopped or when they die. While the System V init setup has worked well for Linux in the past, it hasn't aged well -- which is why we're replacing the aging init system with Upstart in Ubuntu 6.10, codenamed Edgy Eft.

Upstart is designed to be a replacement for System V Init (sysvinit), with a different purpose than other init replacements that have been developed. The problem that has been facing us as we've attempted to make things "just work" in Ubuntu is that modern desktop computers and servers are very different beasts from those in use ten years ago.

In the past, boot-time operations, such as checking and mounting the filesystem, were relatively simple. Only a limited arrangement of hardware was supported, and connection or disconnection of devices while the power was on was not possible. The kernel always knew the number and type of disks, so the user-space boot process could simply check and mount them without difficulty.

Dealing with modern hardware

Today we have to deal with a very different world. Hardware, such as disk drives and input devices, can be connected to buses that support a near-unlimited number of devices in arbitrarily complicated topologies.

To make matters more interesting, these devices can be plugged in and removed at any time, and are not required to power up until they are going to be used. While this means that the kernel now receives interrupts when hardware changes, the initial discovery of connected hardware is more problematic; a probe can be sent, but there's no point at which we know we have received all the responses.

When we bring network filesystems into the equation, the problem gets even more difficult. Not only do we have to wait for responses from the server, the network interfaces themselves need to be configured; this may involve uploading firmware into the card, negotiating with a wireless access point, requesting an IP address from a server on the network, and logging in to a Web page to gain Internet access.

Some solutions do exist for these problems. For example, Ubuntu probes for likely hardware on which it will find the root filesystem, and then sits in a tight loop until the expected hardware appears. These kinds of tricks rely both on waiting in the boot process and knowing the hardware that we're waiting for. As users demand that their computer boots ever faster, waiting in the boot process is simply untenable. Knowing in advance the hardware we're looking for might seem prescribed, yet a lot of the more interesting features we'd like to introduce break this assumption.

We would like certain daemons, such the HP Linux Printing and Imaging System, to be only started then the associated hardware is actually available. This way users who have the hardware benefit, and users who don't are not penalized. More interesting ideas include only running podcast daemons while an iPod is connected, and so on.

The Linux kernel has become increasingly better at supporting hot-pluggable hardware, and with the introduction of udev and the kernel driver-core, user-space is now provided with information about the connected hardware, as well as notification of connection and disconnection. Yet, despite this, we still rely on a static set of shell scripts for the boot process which make assumptions about what hardware is available at what point in the boot process.

Other attempts to modernize the init system

Previous attempts at replacing the init system such as the LSB specification and initNG have largely been focused on automatically determining the order of these scripts, both to ease the task of the developer, and to allow parallel start-up of scripts to reduce boot speed.

Other systems such as Solaris SMF and runit were instead designed to tackle service management problems, allowing a system administrator to query the state of, and change the set of, running services. Finally, Apple's launchd solves neither of these use cases and instead provides a single daemon that replaces the traditional Unix services of init, cron, inetd, and so on.

None of these systems tackled the problems we wanted to solve. We wanted an init daemon that allowed the selection and order of scripts to be determined not just by information in the scripts themselves, but by events coming from outside the init system, in particular udev. In fact, what we wanted was an init sequence driven entirely by these events and those of its own making.

To avoid reinventing the wheel, we first looked at how much effort it would be to use of modify the existing replacements to be able to do this. Sun SMF and Apple launchd were immediately ruled out due to licence issues. It was important for us that the solution be uncontroversially free so that other distributions might adopt it; many had already rejected these for GPL incompatibility reasons.

InitNG received a lot of study, including wondering whether our problems could be solved by dependencies on scripts that waited until released by udev events, however we felt that it was no better a solution than what we had already. The same solutions could be implemented with our existing init scripts using the features introduced in the LSB specification without replacing the init daemon itself.

What we wanted was a system that would allow filesystems listed in /etc/fstab to be checked and mounted when the kernel detected the block device. Likewise, it would configure network interfaces when the kernel detected the hardware, and if an IP address is obtained, attempt to mount remote filesystems. The more we thought about this, the more we realized that the chain of events did not need to stop at hardware either. If the scripts themselves generated events when they were finished, and any daemons started when they were running, other scripts and daemons could be started by those events.

An event-based init daemon

The term we adopted for this design was an event-based init daemon. Jobs to be managed would be split into two basic categories; scripts or binaries to be run when an event occurs, and services to be both started and stopped when events occur. Events would be generated by the init daemon at particular points, such as startup and shutdown, as well as whenever jobs changed state; other events would be received from other parts of the system such as udev.

An interesting comparison can be drawn to the dependency-based init daemons such as initNG; they work by giving each job a list of dependencies, and starting the dependencies when they are required by something else. This means that an ultimate set of goals is required to ensure that anything is started at all, normally given as the configuration for the different runlevels.

A dependency-based init daemon would start networking because it's a dependency of the Apache goal, and would mount the filesystems because they are a dependency of both the Apache and gdm goals. If either gdm or Apache fail to start, this means that networking won't be available unless it itself is a goal.

An event-based init daemon works the other way around; it starts off with a single event such as "startup" and everything else will be run by that or subsequent events. An event-based init daemon has no need for goals or runlevels, the system will boot as far as it can get with the available hardware; for a distribution, this means that the default installation can be far more flexible.

Networking will always be started if networking hardware is available, assuming the default configuration is for DHCP to be attempted. As with the dependency-based system, if no hardware is connected at boot time, Apache still won't start. However, with an event-based system, if the network card is plugged in a few minutes later, once it's been retrieved from the back of the sofa, Apache would be started automatically.

Our example tasks of mounting filesystems might work as follows:

  1. A "startup" event causes the udev daemon to be started.
  2. The udev daemon is then configured to send a "block-device-added" event for each new block device.
  3. A "block-device-added" event causes the device to be checked and mounted if listed in /etc/fstab.
  4. When all FHS-specified filesystems listed in /etc/fstab have been mounted, the "fhs-filesystem" event is sent.

Along side this, we can also be configuring network interfaces and mounting remote filesystems:

  1. The udev daemon is configured to send a "network-interface-added" event for each new network interface.
  2. Then a "network-interface-added" event causes the interface to be brought up and configured if listed in /etc/network/interfaces.
  3. Acquisition of an IP address causes the "network-interface-up" event to be sent.
  4. Setting of the default route causes the "default-route-up" event to be sent.
  5. Either of these events causes an attempt to mount remote network filesystems, which can therefore cause the "fhs-filesystem" event.

These chains of events, along with others, all cause different parts of the system to come up no matter how the user has configured it. There is no need for dependencies of the jobs to be adjusted if the system doesn't rely on any network filesystems, the event is automatically issued earlier.

Defining events

Events themselves are simple strings, with optional environment variables to convey more detailed information to the jobs handling the event; though we are considering adding arguments or an origin for the purposes of identifying which device was added, or which job was stopped, etc. Any part of the system can sent an event by using the "initctl" tool, or by communicating with init directly over a Unix domain socket.

Jobs are defined by files in the /etc/event.d directory, and the simplest can be just the name of a binary, or some shell code to run. The more complex can include shell code to be executed before the job is started and other shell code for after the job has stopped, as well as resource limits and environment settings. Jobs can be started when any of a list of named events occur, with multiple running instances either permitted or disallowed. They can be allowed to terminate normally themselves, stopped when another list of named events occur, or respawned on termination until any event in that list occurs.

A goal of the project is to simplify how the system is configured. Right now, one has to configure jobs to be run on startup, hourly, when a network device is up, when the AC power is reconnected, on shutdown, or when the system is suspended in several different places.

Because events can be received from any other process on the system, we intend to modify other daemons so that instead of running directories of scripts themselves, they send an event to init so all such jobs can be configured in /etc/event.d. Potential plans for the future include running jobs on behalf of users, time-based events so that daemons such as cron, anacron, and atd could be replaced and even, potentially, the ability to replace the inetd daemon.

It's our hope that with Upstart we can solve a lot of the problems that prevent Linux from "just working," without any ugly workarounds or race conditions, as well as make the job of sysadmins much easier. To find out more, visit the upstart Website, which has packages for Ubuntu and Debian, source code, mailing lists, and bug tracking for upstart.

Upstart is suitable for deployment in any Linux distribution as a replacement for sysvinit. One should be aware that, like other replacements, it has its own native configuration and will not run your existing init scripts unless configured to do so. The recommended migration plan, which Ubuntu is following, is to run the existing System V rc script and let that take care of the legacy init scripts.

A set of example jobs is available from the Website to do this, though they will require modification for other distributions. Once up and running, you can then migrate init scripts to upstart jobs one by one, as time permits, without dropping compatibility with other applications.

The Edgy Eft repositories include Upstart, so you can install preview releases of Edgy, starting with Knot 3, to get a hands-on look at Upstart right now.

Share    Print    Comments   

Comments

on Replacing init with Upstart

Note: Comments are owned by the poster. We are not responsible for their content.

is he replacing /sbin/init

Posted by: Anonymous Coward on October 05, 2006 01:06 AM
or the scripts called by it? I'm afraid that if he can't be clear about what he's doing, he's not touching my system and I pity the unbuntu users.

#

Re:is he replacing /sbin/init

Posted by: Anonymous Coward on October 05, 2006 01:32 AM
From the upstart link referenced in the article:
"Upstart is an event-based replacement for the<nobr> <wbr></nobr>/sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running."

#

Re:is he replacing /sbin/init

Posted by: Anonymous Coward on October 05, 2006 02:04 AM
he's not touching my system and I pity the unbuntu (sic) users.

Your loss. Upstart has been a wonderful replacement for the SysV init. It works, is hardly noticeable and it shuts down like the wind. Even the boot-up proces is slightly faster on my hardware.

#

Re:is he replacing /sbin/init

Posted by: Anonymous Coward on October 05, 2006 06:22 PM
/sbin/init has been replaced. At the moment (for Edgy) upstart still uses the old init scripts to boot the system.

For Edgy+1, these scripts will be replaced with upstart scripts.

#

Edgy Eft "UUID's" in fstab

Posted by: Anonymous Coward on October 05, 2006 01:33 AM
Are the UUID strings that are used to name devices in Edgy Eft's fstab, instead of the common (simpler)<nobr> <wbr></nobr>/dev/hda etc.. physical location naming method, a product of this (outlined) new init method?

If so, explain why these UUID's are necessary as part of this new method? It seems to smell of Microsoft's intrusive UUID hardware-hash check nonsense.

If it isn't an absolute requirement. Lose It!

#

Re:Edgy Eft "UUID's" in fstab

Posted by: Anonymous Coward on October 05, 2006 01:54 AM
No, they're entirely unrelated.

The UUIDs are used in<nobr> <wbr></nobr>/etc/fstab because of the upcoming hda->sda migration (edgy+1, 2.6.19, etc.), and to make your system more resiliant against hardware order changes.

#

Re:Edgy Eft "UUID's" in fstab

Posted by: Anonymous Coward on October 05, 2006 02:59 AM
The length and alpha numeric make-up of these UUID's suggests that it is an extreme overkill if used only for the sole purpose of implementing the hda-> sda migration. Unless, of course, you can foresee at some point, the need to connect some number of billion (plus) physical devices to your local hardware. Not likely. Any reasonable analysis of these UUID's would logically conclude that the (as yet un-divulged) purpose of these ID's are far more than for the uses you cite. Suggesting 'Global Unique Identifiers' for hard drives and other hardware. Maybe then used as part of Cononical's commercial support (disk drive specific) verification mechanism.

Privacy Advocates will likely take Ubuntu to task for this. FOSS is about freedom. Intruding on users privacy goes against this end.

#

Re:Edgy Eft "UUID's" in fstab

Posted by: Anonymous Coward on October 05, 2006 05:24 AM
My uninformed guess is that the UUIDs are useful if you want to boot off of a USB HD or some other device which may have a changing<nobr> <wbr></nobr>/dev/ name.

#

Re:Edgy Eft "UUID's" in fstab

Posted by: Anonymous Coward on October 05, 2006 05:46 AM
That analysis is rather paranoid.

Most filesystems, with only a few exceptions, include a UUID in them when formatted. This has been the case for years.

The only other common identifying piece of information is the LABEL.

You can mount filesystems (in<nobr> <wbr></nobr>/etc/fstab) with either UUID=... or LABEL=..., one could have

LABEL=users<nobr> <wbr></nobr>/home ext3 defaults 0 0

in there to mount any filesytem with the LABEL "users" on<nobr> <wbr></nobr>/home.

Obviously LABELs aren't that unique, and are often blank -- so for the automatic conversion they weren't that useful. To make sure the automatic conversion was reliable, and didn't suddenly cause people's drives to dance, we used the filesystem UUID instead of the LABEL.

You can change it to LABEL= if you like; or even just to<nobr> <wbr></nobr>/dev/hd* again -- but then don't file bugs when the device name changes

#

Re:Edgy Eft "UUID's" in fstab

Posted by: Administrator on October 05, 2006 01:08 PM
Yes, UUIDs are somewhat annoying, but they're the best solution we have:


  • Disk labels aren't guaranteed to be unique.


  • The kernel's device paths aren't constant enough -- move your SCSI interface card to a different slot and your system doesn't boot any more..?


  • Device names aren't constant either -- they depend on module load order. "Worse": the kernel is migrating to scanning its hardware in parallel. Therefore, which device ends up as<nobr> <wbr></nobr>/dev/sda will be entirely random Real Soon.


Besides, there's a nice tool at<nobr> <wbr></nobr>/sbin/blkid which will happily print a list of device names, filesystem labels and UUIDs for you. Even if you aren't root.


In other words, this (a) Just Works, and (b) is *way* better than anything you get with Windows. So stop complaining.<nobr> <wbr></nobr>;-)

#

elsewhere

Posted by: Anonymous Coward on October 05, 2006 05:22 AM
Hm, that apache dependency on network interface isn't really nice example. Reminds of a friend some 6 or 7 years ago who was studying warezesque MSSQL at home, and who had to get a 8029 NIC for it to even start. I've been studying BIND and (gasp) Sendmail at home with only 486 and lo.<nobr> <wbr></nobr>:)

But guess you knew that already. Thanks for the project and the article; and you might still pick something interesting off one other development.

Within ALT Linux, we've been thinking over similar set of problems for several years. There's whitepaper in Russian and a conclusion that several uninterrupted man-months are needed, which wasn't really possible for those who could implement it.

Just in case someone near you can read Russian or someone can translate (or skim and decide) for you, here's an URL:

<a href="http://samba.org/~ab/initscripts-replacement-proposal.txt" title="samba.org">http://samba.org/~ab/initscripts-replacement-prop<nobr>o<wbr></nobr> sal.txt</a samba.org>

--
Michael Shigorin
shigorin/gmail

#

How does one boot into "rescue" mode with upstart?

Posted by: Anonymous Coward on October 05, 2006 07:01 AM
If network services will ALWAYS be started, how do I boot into a rescue mode if the network driver is causing the system to panic? This system doesn't sound like it is fully cooked yet.

#

Re:How does one boot into "rescue" mode with upsta

Posted by: Anonymous Coward on October 05, 2006 02:10 PM
Rescue mode would be redundant as upstart will start what it can when it can. If the network wont start and the rest of the system will the you will get a system with everything working but the network.

If you then, for example plug in the network cable the network event will then trigger the start of the network. Or if you want to explicitly start (or stop) the network you would simply use the start command (eg "start network").

#

Re:How does one boot into "rescue" mode with upsta

Posted by: Anonymous Coward on October 05, 2006 02:13 PM
If the network wont start and the rest of the system will the you will get a system with everything working but the network AND THOSE THINGS THAT DEPEND ON NETWORK.

#

Re:How does one boot into "rescue" mode with upsta

Posted by: Anonymous Coward on October 05, 2006 04:43 PM
"Rescue mode will be redundant"

Wrong answer.

Rescue mode will always still be needed -- imagine that you end up with an fscked network driver module that causes a panic+crash every time it gets loaded. You need to be able to prevent the system from loading that module so you can get in and unbreak things. That's what rescue mode is for.

#

Re:How does one boot into "rescue" mode with upsta

Posted by: Anonymous Coward on October 23, 2006 08:50 PM
Maybe rescue mode should be like that, but it's debatable because people who are able to fix broken ethernet drivers should also be able to add "init=/bin/sh" at the boot prompt.

#

when to start apache

Posted by: Anonymous Coward on October 05, 2006 07:33 AM
The fact that there is no network card does not mean Apache should not be started. I test stuff all the time when I am not connected -- like a wiki that is hosted on my laptop and replicated to another machine when I have a chance to do so.

#

Re:when to start apache

Posted by: Anonymous Coward on October 05, 2006 08:26 AM
I'm sure the author knows this and was just using Apache as an example. Sometimes when a programmer writes documentation like this, they assume what is obvious to them is also obvious to the audience.

But even in the system he describes, it seems to me that it would be a simple procedure to cause Apache to be started when another lower-level event is fired, such as "Startup" or "/usr mounted" or some such thing.

#

Parallel

Posted by: Anonymous Coward on October 05, 2006 09:23 AM
If in an event based startup... services only start up when another particular event happens, does this mean that the boot time can not be done in parallel, and therefore will never be as fast as initNG ?

#

Re:Parallel

Posted by: Anonymous Coward on October 05, 2006 01:33 PM
According to this review
<a href="http://www.tectonic.co.za/view.php?id=1193" title="tectonic.co.za">http://www.tectonic.co.za/view.php?id=1193</a tectonic.co.za>
Upstart boots WAY fast.

#

Re:Parallel

Posted by: Anonymous Coward on October 05, 2006 08:19 PM
ote that previous boot was also *very* slow, a stock Kubuntu 6.06TS take 1min to boot on my computer (and then ~10+ seconds for KDE), so the previous situation was very bad, no surprise that he felt an improvement.

BeOS on a much older computer than the articles (Celeron333+128Mo of RAM) booted in 14s (Kernel+GUI) so booting fast is possible, just not currently with Linux (KDE itself takes nearly the same time to start that BeOS did for the whole boot).

#

Re:Parallel

Posted by: Anonymous Coward on October 05, 2006 02:32 PM
One event could trigger the starting of multiple processes/demons which would start in parallel.

Also events would occur asynchronously. That is new events triggering new process startups could occur before other processes have finished starting.

If you had wireless and ethernet cards and udev saw the ethernet card and triggered the network DHCP startup on that udev might then see the wireless adapter and trigger the network start up on it even if the ethernet has only just started initialising. this would leave both network adapters starting in parallel

#

Plan 9?

Posted by: Anonymous Coward on October 05, 2006 04:47 PM
Apache can be useful even when you're not connected to the Internet. Example when your network is done, or you do local web developement, etc.

Also, I wonder what system of startup does Plan 9 use?
Plan 9 usually seem to have alot of smart solutions, maybe they have a really smart solution for startup too?

Also, I heard that BeOS was heavily multi-threaded and so, maybe it had a good startup?

Eitherway, Upstart sounds quite interesting...

#

Edgy Eft "UUID's" in fstab-cont.

Posted by: Anonymous Coward on October 06, 2006 12:39 AM
UUID=285d696a-e778-41a5-905c-e3fab41f200d

Above is an example of the UUID's Ubuntu is using in Edgy Eft's<nobr> <wbr></nobr>/etc/fstab to identify hardware. Why they are needed is no longer the issue.

The vast number of available addresses in alpha numeric strings this long, far exceed their stated purpose, and could number every harddrive on the planet many times over.

It can be meant as nothing other than a "Global Unique Identifier". Not just for local hardware.
As yet, none of the posters have addressed the need for strings this long. At best, it's unneeded code bloat.

#

Re:Edgy Eft "UUID's" in fstab-cont.

Posted by: Anonymous Coward on October 06, 2006 04:01 AM
That point is that the UUID is virtually guaranteed to be unique and since it is randomly generated it needs to be that long to ensure that the chance of there being another filesystem with the same UUID is so small as to be insignificant.

As to the code bloat from an overly long UUID, the difference in the current length and one half the length would probably be measured in tens of bytes, which isn't really much to worry about.

#

Re:Edgy Eft "UUID's" in fstab-cont.

Posted by: Anonymous Coward on February 01, 2007 10:06 PM
At one time people thought IPv4 gave enough unique addresses and now we have IPv6. Considering that even randomly generated UUIDs reserve space to identify the variant, you need a string of this size in order to provide some degree of certainty of generating a unique value.

#

what does all this mean?

Posted by: Anonymous Coward on October 06, 2006 02:14 AM
Excuse my ignorance but what does all of this mean for the average linux user (who does not program)?

#

Re:what does all this mean?

Posted by: Anonymous Coward on October 06, 2006 03:39 AM
I'm sure more knowlegable people could come up with more benefits to non-devlopers, but as as a start, you should see faster start times and faster power down times. I'm seeing both of these already with Ubuntu (Edgy) 6.10.

#

Re:what does all this mean?

Posted by: Anonymous Coward on October 07, 2006 01:03 AM
I hope it means, finally an understandable init scheme for Debian/Ubuntu (and I hope Fedora and other sysvinit distros will follow).

The sysvinit scheme has been really awfull... and is one reason I've been using Slackware and Archlinux and experimenting with initng.

#

Re:what does all this mean?

Posted by: Anonymous Coward on January 01, 2007 06:10 AM
Initng is good.but it lacks scripts.i for users like me who wants xfs and xfstt to be started...no scripts are available esp for xfstt-talking about debian/ubuntu..

#

Replacing init with Upstart

Posted by: Anonymous [ip: 216.98.179.10] on February 29, 2008 12:24 AM
What came first Solaris Service Management Facility or Linux Upstart?

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya