This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature

GoboLinux's recipe for delicious package management

By Mayank Sharma on February 14, 2007 (8:00:00 AM)

Share    Print    Comments   

GoboLinux is a unique distribution in many ways. It's built from scratch following the Linux From Scratch procedure and uses custom boot scripts, personalized directory structure, and a simple yet comprehensive source-based dependency-resolving package management system.

GoboLinux is perhaps best known for its alternate filesystem hierarchy. Unlike Linux's traditional Filesystem hierarchy, where a program has bits and pieces scattered in several places like /etc, /usr/bin, and /usr/share, each program gets its own directory tree under GoboLinux.

Needless to say, this type of hierarchy aids in package management. Users can install and use multiple versions of the same program or system libraries. In fact, when GoboLinux switched over to the GNU Compiler Collection (GCC) version 3, they still kept older programs running since the filesystem allows for multiple libraries to exist together in peace. Also, uninstallation is just a matter of removing the program directory.

But how does one install applications under such a radical directory structure?

Source is good enough

From the start, GoboLinux's developers had no intentions of adding another package format like RPM or Debian packages. Furthermore, depending on the popularity of an application it might or might not be available in the RPM or Debian package formats. But all applications will be available as a compressed source tarball.

Hisham H. Muhammad, who developed GoboLinux along with André Detsch, explains that a tarball can simply be unpacked, and then three commands, "configure, make, make install", should install it.

Dependencies are checked by configure (though not resolved) and the program is linked to the correct libraries present on the system, with no incompatibilities. The developers reasoned that if a source tarball is decently put together, and follows a few GNU conventions, there is no reason why it couldn't work in any Linux distribution. Therefore, ideally, no distribution formats would be needed at all and a tarball should be good enough. Additionally, if an application is put together using standard build systems, like the GNU buildsystem its installation can be automated.

But not everyone uses the GNU buildsystem to put together their source tarball. This is where GoboLinux's Compile tool comes into play. Compile works on the basis that there a few standard ways that sources are distributed, and that there are lots of minor variations that can be declaratively specified. Each of these ways is an "execution model", and support for them is implemented separately. There are three main execution models: 'compileprogram' (which covers configure-based files), 'makefile' (which covers configure-less make/make-install projects) and 'xmkmf' (which covers applications based on the X imake tools).

It's the job of Compile to download source tarballs, unpack, compile and install them, with a single command. Compile uses a simple file called a "recipe" that describes how an application is to be compiled.

So how does Compile work?

The best thing about Compile is that it uses every project's own download site to fetch sources. The GoboLinux repository is just used for downloading recipes. Recipes can be downloaded explicitly, using the GetRecipe command, or on-the-fly by Compile, when a compilation is requested.

To download the Joe text editor using Compile, simply do Compile joe. This will first do a case-insensitive search for a recipe named joe. If you want a particular version, you could do a Compile joe 3.1, which will look for a recipe for the 3.1 version of the application. The recipes are available as compressed tarballs and the one for joe is named Joe--3.1-r2--recipe.tar.bz2.

Upon uncompressing the tarballs, Compile finds a file called Recipe and a directory called Resources. The recipe file looks something like this:

# Recipe (MakeRecipe) for Joe by roko, <ro.ko@mcnon.com>, on Wed Oct 27 03:01:32 BRST 2004
# Recipe for version 3.1 by roko, <ro.ko@mcnon.com>, on Wed Oct 27 03:01:32 BRST 2004
compile_version=1.8.0
url="$httpSourceforge/joe-editor/joe-3.1.tar.gz"
file_size=381201
file_md5=2a6ef018870fca9b7df85401994fb0e0
recipe_type=configure

Compile follows the URL and downloads the source tarball from the project's home. The recipe_type option tells Compile the kind of compilation a particular program needs, in this case, the common ./configure, make and make install procedure. The file size and MD5 signature help Compile verify the package after downloading.

Next, Compile turns its attention to the Resources directory. This directory contains two files, one called Description which briefly describes the application and one called Dependencies, which lists all the libraries Joe depends upon. If the Dependencies file lists a library not available on the system, Compile gives you the option to fetch and install the missing library as well.

After downloading and verifying an application and satisfying its dependencies, Compile configures and install the application. Compile has a special flag "--configure-options" where one can supply extra parameters that Compile passes on to the configure script. For example, Compile --configure-options="--enable-debug" joe.

Once the program is compiled, it is placed under its own directory under /Programs. In the case of Joe it's under /Program/Joe/3.1/. Compile then copies the recipe's Resources directory inside the target directory at /Programs/Joe/3.1/, and generates four new files inside this directory.

The Architecture file tells which architecture the package was compiled for, BuildInformation stores which dependencies were tracked by ldd (an application that lists dynamic dependencies) after the package was compiled, FileHash stores md5sum for each file in the package and FileHash.sig contains FileHash's PGP signature. Lucas Villa, one of the main developers of GoboLinux explains that the idea is to have all contributors listed in a "ring" of public keys, so that one can identify who created a given package at the time of its installation.

When all the files and directories have been created and populated, Compile creates all the necessary links throughout the system and the application is ready to use. Villa says that currently GoboLinux's repositories contain 5,376 recipes for 1,709 different programs.

Write your own recipe

Sitting and watching Compile go about installing Joe, I wondered what it would take to create a recipe. Since Joe's recipe is for version 3.1, I decided to create one for the latest version, 3.5. I was surprised to find that creating a recipe is pretty straightforward.

There are two methods of creating recipes. The MakeRecipe method, explained later, is for creating recipes for applications that don't yet have a recipe. The NewVersion method helps create a new recipe based on an existing older version recipe.

NewVersion takes a package's name and its version number as parameters. NewVersion joe 3.5 will create a template for the new recipe based on the existing recipe which is kept under /Files/Compile/Recipes/. It also modifies the download URL, since generally it only involves replacing the old version with the new one in the URL's path. In case the location of the application has changed, you can pass the new URL as a parameter when calling NewVersion. For example, NewVersion joe 3.5 http://jaist.dl.sourceforge.net/sourceforge/joe-editor/joe-3.5.tar.gz will use this download URL instead of the existing one in the original recipe. All locally created recipes are kept under /File/Compile/LocalRecipes/.

The new recipe looks like this:

# Recipe for version 3.5 by Mayank Sharma, on Mon Jan 29 16:49:14 IST 2007
# Recipe (MakeRecipe) for Joe by roko, <ro.ko@mcnon.com>, on Wed Oct 27 03:01:32 BRST 2004
compile_version=1.7.1
url="$httpSourceforge/joe-editor/joe-3.5.tar.gz"
recipe_type=configure

There's no better way to check a recipe than installing from it. Once you have the new recipe under /File/Compile/LocalRecipes/, ask compile to install Joe again. This time around, Compile will use the locally available recipe for version 3.5, download the new tarball, which verifies the URL, and then configure and install Joe, in a sandbox. A sandbox allows for testing the application without affecting the base system. After verifying that everything is in order, that all executables are available and all paths map properly, Compile copies the files under the respective /Programs directory and updates system paths. Finally, it also creates a compressed tarball of the new recipe that's kept under /File/Compile/Store, ready for submission to the GoboLinux repository.

If you want a faster way to verify your recipe, test it with the GenRecipeStore script. This script checks the recipe's structure, verifies URLs, does basic syntax validation, generates a temporary report under /System/Variable/tmp and creates the compressed recipe tarball in the Store.

Now let's suppose, that you want to create a recipe for an application Foo, which doesn't have a recipe. MakeRecipe http://unc.dl.sourceforge.net/sourceforge/foo-app/foo-1.0.tar.gz will not only download the application, but also extract its name and version from the download URL and create a new recipe.-

MakeRecipe will then let you know whether the source was assembled using standard toolchains, like Autoconf, or not. If it's built with standard tools, it can be compiled with Compile, which will install the application and prepare the compressed recipe.

Resolving dependencies and creating binary packages

If the application Foo, depends on libraries not on the system, Compile will fail to install them. In this case, one will have to create the Dependencies file by hand. To overcome this hurdle, Villa says that the GoboLinux developers have lately been working with a tool called ChrootCompile. "It's basically a chroot environment which is constructed only by "clean" packages, taken from the recipe store. The base set of packages, which is assumed to exist in all GoboLinux installations, are automatically merged in the chroot, and after the script is finished with its creation, Compile is called from inside that root filesystem. All other dependencies not covered by these basic set of packages needs to be explicitly listed in the Dependencies file, which is parsed by the script responsible of creating the chroot environment."

Villa also points to the fact that Dependencies can now be specified in a range. For example, GTK+ > 2.8.0 specifies that the application depends upon a version of GTK released after 2.8 and GLib < 1.2.10 means a GLib version earlier than 1.2.10 is required.

They've also added support for cross-compilation, to clearly specify which dependencies are only needed when cross-compiling. So Autoconf 2.59 [cross], means that Autoconf needs to be installed in order to cross-compile the package and Xorg [!cross] means that Xorg is only needed if not cross-compiling.

Once you have successfully installed a package, either from an existing recipe or from your own, you can also create a binary package for it. Binary packages install quickly since they are pre-compiled. You can create binary packages for your own use, or distribute them to help users that don't want to install from source.

To create a binary package, you need to have that application installed on your system. Since we have two versions of the Joe text editor installed, CreatePackage joe 3.5 will create a binary package of version 3.5.

CreatePackage simply compresses the /Programs/Joe/3.5 directory, after scrapping off the settings sub-directory /Programs/Joe/Settings. These settings are replaced by the default settings stored inside /Programs/Joe/3.5/Resources, which is copied from the Recipe tarball and contains the four additional files mentioned earlier.

Installing Binary packages is covered in detail on GoboLinux's wiki. No matter how you installed a package, either from source or binary, it can be uninstalled using the RemoveProgram tool.

Conclusion

GoboLinux is as much an educational tool as it is an interesting distribution. It took me some time to understand its layout and the various package management tools and how they operate. Once past that hurdle, it hardly took me anytime to contribute my first recipe.

GoboLinux doesn't have as many packages as other popular distributions, but this will change dramatically as more and more applications are built using standard toolchains. Package management on GoboLinux will become a non-issue as creating recipes will only be a matter of executing the MakeRecipe script.

Share    Print    Comments   

Comments

on GoboLinux's recipe for delicious package management

Note: Comments are owned by the poster. We are not responsible for their content.

very Windows-like

Posted by: Anonymous Coward on February 15, 2007 04:46 AM
It's an interesting idea, although I'm not really sure what real advantages this layout could bring. It looks a lot like the Windows file-system hierarchy to me.

Actually, what I think would be much better is use standard packaging management such as dpkg to manage core applications (basic tools, but also gcc, xserver, and other critical packages) and common programs (i.e. what's usually in a basic desktop or server install) and then be able to add extra packages using the method described here in a special folder (such as Programs, if you like the Windows way of naming things).

This would leave you with the best of the two world: A solid system for core packages including security support etc. and an easy way to add not yet packaged software without breaking the system or having to do major packaging work.

#

Re:very Windows-like

Posted by: Anonymous Coward on February 15, 2007 05:11 AM
"I'm not really sure what real advantages this layout could bring. It looks a lot like the Windows file-system hierarchy to me."

Do not forget Apple. They use a very similar approach too. You dont want to imply that an idea becomes bad if MS uses it? (Btw they dont stick to that a lot..<nobr> <wbr></nobr>.they use a very user unfriendly Registry<nobr> <wbr></nobr>... they have a problem to cope with programs that install not in<nobr> <wbr></nobr>/program files but instead insist on installing to "/Programme", which happened for me several times...)

#

Re:very Windows-like

Posted by: Anonymous Coward on February 15, 2007 05:13 AM
"Actually, what I think would be much better is use standard packaging management such as dpkg "

I dont think the dpkg guys are ever interested to support another philosophy. Its old, it works, it wont change.

Personally I think uptakers like Paludis or Pacman3 would be much more likely to support alternate ideas than ancient tools such as dpkg (maybe i am biased, I found debian devs to be very conservative in regards to suggestions)

#

Re:very Windows-like

Posted by: Anonymous Coward on February 15, 2007 06:01 AM
I was just talking about using dpkg for packages that are vital for the system.

I also wasn't implying that something becomes bad because Windows uses it. I was just speculating on the background of the designers of the distribution.

Your point on using images the way Apple does it is valid, I just don't think this is the best approach for programs in the core system. As I said, I would consider to use the best of both worlds. I wouldn't mind to install 5 different versions of Firefox in different locations, but I just don't think this would be the right approach for other applications such as the x-server, gcc or most types of libraries.

#

Re:very Windows-like

Posted by: Anonymous Coward on February 15, 2007 09:26 AM
"It looks a lot like the Windows file-system hierarchy to me."
It's not at all like the Windows file-system (at least not meant to be), just that the name are spelled out instead of beeing three letter abreviations. The object of the directory layout is just to avoid having a packaging management tool. To uninstall a package all you need to do is remove its directory from the<nobr> <wbr></nobr>/Programs folder, you don't even need a tool for that. The advantage is just that - you don't _need_ any tools for management. They are just there to help.

I don't see why a packaging management tool would help, as there's no problem with the core packages. The system is already solid using this "packaging" technique.

#

Re:very Windows-like

Posted by: Anonymous Coward on February 15, 2007 11:03 PM
Where does it put the configuration files? In the package directory too? What about libraries? One way or another, you either have files outside the package directory or every time you install a package in a different location you have to manually copy over the configuration files (except those in your home, obviously). Having one directory per program was actually what I meant when I said it was very Windows-like.

Actually, this makes me think of something else: What about the config files in home? How are those handled if you install multiple (possibly incompatible, otherwise there wouldn't be much sense in installing two versions) versions of a program?

#

Re:very Windows-like

Posted by: Anonymous Coward on February 16, 2007 09:06 AM
ln -s.

The real files are in the package directory, but there are a couple of directories where symlinks to all executables/libraries/configs in the system reside. This way you can find them whereever they are; multiple versions are handled in the usual way, with multiple symlinks for different versions and libtools-scripts.

#

Directory hierarchy

Posted by: Anonymous Coward on February 15, 2007 11:20 AM
I kinda like the alternative directory hierarchy.

It makes sense to have all application files in example<nobr> <wbr></nobr>/Programs/Gimp/ instead of having them scattered all over the system.

#

A good idea. But -

Posted by: Anonymous Coward on February 15, 2007 03:08 PM

This is the first significant re-think of the Unix filesystem hierarchy in over 30 years. I've been doing programming in the Unix environment since 1977 (and for "Linux" zealots, the GNU environment pretty much duplicates the Unix environment), and I wish this re-think had happened 20 years ago.


The only problems I see are conservatism and inertia. Unfortunately, those are the strongest forces known to sociology.

#

Re:Bad Idea

Posted by: Anonymous Coward on February 15, 2007 03:45 PM
You miss the fact that even though libraries may be in different directories, it's still possible to get access them for every application wanting to load them. There is no need to have multiple versions of one library, why do you think that?

If you compare to legacy file system you can have libraries in<nobr> <wbr></nobr>/lib, usr/lib,<nobr> <wbr></nobr>/usr/local/lib,<nobr> <wbr></nobr>/opt/lib etc, what is the difference?

One advantage of having the way GoboLinux has it is that it is possible to actually have two versions of one library installed at the same time, if that is required, e.g. two applications require two incompatible versions of one library.

#

Re:Bad Idea

Posted by: Anonymous Coward on February 15, 2007 03:57 PM
--snip--
>There is no need to have multiple versions of one
>library, why do you think that?
--snip--

--snip--
>One advantage of having the way GoboLinux has it
>is that it is possible to actually have two
>versions of one library installed at the same time
--snip--

HU???

I think what you wanted to say was you don't need to have multiple copies of the same version of the same library? But i'm just guessing. If so you didn't read what i wrote. I'm saying you need multiple _versions_ of the same library. If you app needs a different version of a library thats already installed, fix your app.

I know Gobo doesn't write apps, just puts them all together. But its going in the wrong direction, many versions of the same stuff. Where as a perfect Distro would have no duplication. Thats not possible now but thats where we should be trying to get to.

Eli Criffield

#

Re:Bad Idea

Posted by: Anonymous Coward on February 15, 2007 04:26 PM

HU???

I think what you wanted to say was you don't need to have multiple copies of the same version of the same library? But i'm just guessing. If so you didn't read what i wrote. I'm saying you need multiple _versions_ of the same library. If you app needs a different version of a library thats already installed, fix your app.

No Eli, he meant exactly what he said. For example, let's say you install a package with a library that is commonly used in<nobr> <wbr></nobr>/Programs/SomeCoolLib/2.1.3, there is nothing to stop that being share by several other programs.

But there is also the advantage, and it definitely is an advantage, of being able to have<nobr> <wbr></nobr>/Programs/SomeCoolLib/1.0.0 installed and co-existing at the same time should some other application be dependent on it.

I know Gobo doesn't write apps, just puts them all together. But its going in the wrong direction, many versions of the same stuff. Where as a perfect Distro would have no duplication. Thats not possible now but thats where we should be trying to get to.
Eli Criffield

It is not encouraging many versions but it is a really nice side effect of the way it operates that you can elegantly have multiple versions of libraries and applications installed at the same time.

#

Re:Bad Idea

Posted by: Anonymous Coward on February 15, 2007 05:38 PM
I meant exactly what I wrote: "There is no need to have multiple versions of one library". Why do you think I meant multiple _copies_? And you still haven't answered why you think that multiple _versions_ are needed.

It is actually possible to have no dupliation on a GoboLinux system, i.e. it's a perfect distro according to you.

#

Re:Bad Idea

Posted by: Anonymous Coward on February 16, 2007 08:44 AM
Really, this thing about wishing that only one version of a given library were ever needed is a nice sentiment, but it's not very farsighted. Let's face the facts: libraries must evolve (like all other software), lest they rot. Also, not all application programmers will jump through hoops to update their apps whenever a new version of one of their many dependent libraries is created. Ergo, multiple versions of the same library will have to be supported.

It's very easy for a system to only support one version of a library, but its users would curse it to hell: what good is a computer if I can't run what I want on it? If I want to use two applications which use different versions of the same library, well, the system better damn well support both versions! Or else I will look for a system that does -- there's plenty around. Any of those "imperfect" Linux distributions can.

#

Re:Bad Idea

Posted by: Administrator on February 16, 2007 04:51 PM
You are missing the point. The thread started with someone thought they couldn't share libraries because they were in different directories and there fore you had to have multiple copies/versions of each library (this is the way I understood the first post).

Then I replied that one doesn't _need_ to have this (multiple version) but one _could_ have this. There was no arguing that only one version of a library was supported, rather that GoboLinux actually supports multiple versions better than most other distros out there (it does).

What I meant in my last reply was that it's actually possible to only have one version of library installed at any one time on GoboLinux (if one install a new version of a library it's "only" needed to recompile all apps depending on that library and then remove the old one), but then I didn't care about API incompability (which is another issue).

#

Re:Nice idea

Posted by: Anonymous Coward on February 15, 2007 07:10 PM
Who cares about popularity as long as you're happy with the distro of your choice?

There are also other good source-based distros besides GoboLinux and Gentoo -- at least CRUX, Lunar Linux and Source Mage GNU/Linux.

Source Mage is my personal favourite. It has one unified, easy-to-use interface (ncurses dialog or gtk xdialog) for configuring all the aspects of package management (optimization flags, integrity checks, notification sounds, etc.). It asks which optional dependencies you want and it lets you tweak the make options before the compile starts -- and it remembers your answers the next time you compile the same app.

Source Mage and Lunar are forks of Sorcerer GNU/Linux (which later on became non-GPL). An early review of Sorcery at DistroWatch offers a surprisingly nice overview to what Source Mage is like.
<a href="http://distrowatch.com/dwres.php?resource=review-sorcerer" title="distrowatch.com">http://distrowatch.com/dwres.php?resource=review-<nobr>s<wbr></nobr> orcerer</a distrowatch.com>
<a href="http://distrowatch.com/dwres.php?resource=review-sourcemage" title="distrowatch.com">http://distrowatch.com/dwres.php?resource=review-<nobr>s<wbr></nobr> ourcemage</a distrowatch.com>

And here's a comparison of Gentoo's and Source Mage's package managers (and also some other aspects of these distros).
<a href="http://www.sourcemage.org/Gentoo" title="sourcemage.org">http://www.sourcemage.org/Gentoo</a sourcemage.org>

One final tip: It's been a while since Source Mage 0.9.6 was released (2006-03-15) and if someone wants to try Source Mage now, it's useful to update smgl-archspecs first before compiling glibc.

#

Re:Nice idea

Posted by: Administrator on February 15, 2007 09:40 PM
Who cares about popularity as long as you're happy "with the distro of your choice?

Good point. I realized that after re-reading what I wrote.

#

Yes, but...

Posted by: Anonymous Coward on February 16, 2007 01:02 PM
I agree, libraries should not be in<nobr> <wbr></nobr>/programs/lib/
But software such as GIMP, Inkscape, KDE, GNOME, etc should be in<nobr> <wbr></nobr>/programs/, I think.

#

Re:Yes, but...

Posted by: Anonymous Coward on February 22, 2007 07:49 AM
You have never read Gobolinux wiki.

Please do. Its a shame that people who never read it, post here without knowing the
Gobolinux way at all. People, PLEASE read it before posting!<nobr> <wbr></nobr>:(

#

Re:Bad Idea

Posted by: Administrator on February 15, 2007 02:53 PM
Most valid point. And I am curious as to how Gobo linux achieves the same given the filesystem structure they have adopted..

#

Nice idea

Posted by: Administrator on February 15, 2007 04:12 AM
If only it would get as popular as <a href="http://www.gentoo.org/" title="gentoo.org">http://www.gentoo.org/</a gentoo.org>

#

Re:Bad Idea

Posted by: Administrator on February 17, 2007 06:42 AM
1. Read docs: <a href="http://www.gobolinux.org/?page=at_a_glance" title="gobolinux.org">http://www.gobolinux.org/?page=at_a_glance</a gobolinux.org>
2.<nobr> <wbr></nobr>...
3. profit!

#

Bad Idea

Posted by: Administrator on February 15, 2007 11:36 AM
Here is why:


The reason there is a<nobr> <wbr></nobr>/lib directory and not a<nobr> <wbr></nobr>/programs//lib/ directory is because libraries are shared. I'm sure Gobo uses shared libraries like everything else, but a distribution should work very hard to minimize multiple version of any one library. Not to maximize different versions. Its a waste of resources.



For example on my modest laptop i have about 70 executable in the process table, I'm using a grand total of about 100Megs of memory. Many of those programs show a size of 10 or even 20 megs of memory. If add up the size of all the running process its well over 300Meg. I don't even have 300 Megs of real memory. That's the magic of shared libraries. (and to a lesser extent copy on write, and allocating memory at use and not on request)



By not duplicating libraries we save disk space and memory, a perfect distribution to me is one that doesn't have any duplication in libraries, not only versions but functionality (KDE/GNOME anyone?). Then maybe i could get down to only using 64M of memory.


Eli Criffield

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya