This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new!

Feature: BSD

Implementing Linux emulation on NetBSD

By Peter Seebach on May 13, 2004 (8:00:00 AM)

Share    Print    Comments   

NetBSD's Linux emulation doesn't run a Linux kernel on a virtual machine; it runs Linux binaries on a NetBSD kernel. Linux emulation let you run plenty of useful programs that won't run natively under NetBSD, such as Sun's 1.4 Java Runtime Environment and JDK.

Setting up Linux emulation requires a kernel option (COMPAT_LINUX) in the NetBSD kernel, and some local files. The kernel option is specified in the default GENERIC configuration; if you aren't using that, just make sure you have a line reading "options COMPAT_LINUX" in your kernel configuration file. As for the Linux programs, most are dynamically linked and thus need copies of Linux shared libraries, which are installed under /emul/linux.

The easiest way to set up the necessary libraries and support files is to use the packages found in pkgsrc to install the various binaries correctly. The key package is called suse_base; there are a number of others. You can install them all at once by installing the suse_linux package, which gives you all the most common libraries, including X Window support. As of this writing, this package is based on SUSE Linux version 7.3. This isn't an install of SUSE; it's just unpacking a handful of shared libraries and support files.

You may need to mount the /proc filesystem, which gives filesystem access to a number of system data structures and process-specific data structures. If you already have it in your fstab, add the "linux" option to the flags field. If not, a sample line would look like this:

/proc /proc procfs rw,linux 0 0

You need to create the /proc directory first, with owner root and mode 755 privileges.

Once you've got these steps done, most Linux binaries should just run out of the box. When you start a Linux executable, the NetBSD kernel flags it as needing Linux emulation. This changes the behavior of some system calls to provide behavior compatible with Linux kernel behavior.

Some packages may give strange error messages in Linux emulation mode; for instance, StarOffice 7 gives hundreds of repetitions of "file_image_pagein: Function not implemented" during startup, but it works fine anyway. Packages such as Sun's JRE and JDK work without complaint, as does the Linux build of Netscape. To find programs that won't work you have to look a little farther afield. Video games are more likely to have problems, as are some programs that use system calls that are harder to emulate; a Linux build of a program such as WINE, for instance, is very unlikely to work without effort and may not work at all.

The Linux emulation code is actively maintained by the NetBSD project staff, so support is improving. Things that were unstable or unreliable a while back may work better today. Also, sometimes an apparent problem with emulation is actually a problem with the program running under emulation; for instance, until Sun fixed a bug where Sun's Java runtime would hang in multithreaded programs, I thought I was seeing an emulation problem, but it was actually a bug in the Java runtime.

How does it work?

The NetBSD kernel has a number of execsw structures that describe different types of executable programs. One feature of the execsw structure is a struct emul object, which is attached to the process and which contains various tables used to translate system calls, error codes, and other such communication between systems. When trying to execute a binary, the kernel iterates through the available execsw structures, testing each to see whether it thinks it can handle the executable. For instance, the Linux emulator checks the file for an ELF header that claims to be a Linux binary. If the kernel finds one that can execute the file it uses it. If execsw structure has a struct emul associated with it, then that emulation layer is used for that process.

Let's use an example to see how this works. If you try to start a Linux ELF binary, the NetBSD execsw structure's probe routine looks at the file, concludes it can't handle it, and rejects it. The exec system call then tries other structures (perhaps other emulators, or the default "interpreter" structure used with shell scripts), until the Linux emulation execsw structure's probe routine confirms that the one being tested can run the file. The corresponding struct emul is then attached to the process, which is run using the Linux emulator's code from that point on. The emulator translates system calls, errno values, and anything else used in the API between the kernel and user code.

Filesystem magic

One obvious problem is that, since most programs are dynamically linked, they need to be able to find the shared C library, which means you need to have a shared C library loaded. Linux and BSD systems have very different libraries, so there needs to be an additional shared library for the Linux executables.

To facilitate loading the "right" versions of various system files, the kernel intercepts all file lookups from emulated binaries. They are first looked up in a "shadow root" -- a directory tree in /emul/linux containing files that differ between Linux and BSD systems. If the file in question is not there, the kernel looks in the regular filesystem tree. Thus, is loaded from the shadow root, but user home directories are found in the regular filesystem tree. Looking in the /emul/linux tree to see which files are loaded there helps a lot in understanding the Linux emulation code.

This system isn't quite flawless -- for instance, symbolic links don't use the shadow root -- but in practice, it works fine.

Linux and NetBSD systems use somewhat different formats for the /proc filesystem. NetBSD's mount_procfs command takes an optional -o linux option to direct it to use more Linux-compatible formats and enable a couple of Linux features normally not present in NetBSD's /proc filesystem (for instance, a file called "stat" which holds data similar to, but differently formatted than, the file called "status" that NetBSD normally uses). Not all programs need this, but many do.

Programs that don't work

Not every application works in emulation mode. The most common problem is that the system may lack a shared library that a program needs. The default install from pkgsrc installs the most commonly used libraries, but you could end up needing a library that's not installed. Getting the replacement library isn't always easy. One useful strategy is to install the program on a Linux box (ideally, the same distro and version from which you're getting all your shared libraries), and execute the ldd command to get a list of libraries it's linking with. Once you have such a list, you can copy the files over.

One thing to watch out for: Some libraries may have multiple versions that are intended to work on different systems. A rendering library that uses direct rendering support on a Linux system is unlikely to work under the very different memory architecture of a NetBSD system. If you have both hardware and software versions of a rendering library, the software one is more likely to work, but it may be quite a bit slower.

Only rarely will a program actually cause the operating system to crash. More likely is for a program to fail, possibly with a confusing error message. However, I've seen one program (a video game) actually hang a NetBSD system, so make sure you try a program a couple of times before you run it on a heavily loaded system doing other work. Sync your disks and stop any important or resource-intensive activities before testing a new application for the first time.

A more subtle kind of error comes from programs whose install scripts start with #!/bin/sh, but which are actually bash scripts. This used to be more common than it is now, but it still happens often enough to be annoying. Merely having the right /bin/sh in the /emul/linux "shadow root" isn't enough; the script may be run without the Linux emulation flag when spawned from other programs, since it apparently runs just fine as a standard shell script. One workaround is to temporarily replace /bin/sh with a link to some version of bash. Another is to modify the script's #! line to refer to something other than /bin/sh, such as /bin/bash.

When a program isn't working, another useful technique is to run it under ktrace to see what output it produces. This command logs all the system calls the program makes and writes them to a file called, by default, ktrace.out. You can review the chain of system calls with kdump. You might see output like this (taken from the command ls /foo):

  4653 ls       CALL  __stat13(0x80510c0,0xbfbff4e0)
  4653 ls       NAMI  "/foo"
  4653 ls       RET   __stat13 -1 errno 2 No such file or directory 

What this tells you is that namei (a kernel internal function that converts names to inodes) tried to look up "/foo" and got "No such file or directory." If a Linux program aborted and a look through kdump reveals that it couldn't find some file, check a Linux system to see whether that file is part of a standard installation, and if so, copy it over into the /emul/linux tree.

Linux device drivers won't work at all under NetBSD. Maybe someday someone will write code to make them work, but for now, there's just no way to use device drivers or other kernel modules from Linux systems on a BSD system.

Nominal performance

The majority of real-world applications exhibit normal performance in Linux emulation mode. People who have gotten used to PC emulators on a Mac, such as VirtualPC, often expect emulation to imply slow operation. Not so. While it's difficult to isolate emulation cost from other OS overhead or differences in the performance of C libraries, in practice, if a program doesn't rely heavily on direct rendering and is usable under Linux, it'll be just as usable under Linux emulation on comparable hardware. StarOffice 7 is slick and responsive. Netscape 7 runs as reliably as a native browser, and runs Flash animations and the like quite nicely. You probably wouldn't want to use Linux emulation code for a high-performance project, but for getting access to a specific application or module, it's pretty useful.

Linux executables can be run in emulation mode on Alpha, ARM, i386, m68k, and PowerPC systems. Both a.out and ELF executables are supported. If you use the pkgsrc installer, it's pretty easy to set up all the important files, and the kernel options you need are included in the GENERIC kernel, so it Linux emulation should run out of the box. You do need root privileges to configure a system to run Linux binaries, but once the emulator and support files are set up, the applications work fine for ordinary users. Linux emulation code lets you run most standard applications without a great deal of effort, and could help provide a standardized environment for users who need to run different OSs, or use proprietary code. It won't do everything, but it does enough to be a practical and useful tool.

Peter Seebach has been running Linux binaries on NetBSD for a few years, and no one has arrested him yet, so it's probably allowed.

- Write for us - and get paid! -

Share    Print    Comments   


on Implementing Linux emulation on NetBSD

Note: Comments are owned by the poster. We are not responsible for their content.

Compare between other BSD's

Posted by: Anonymous Coward on May 14, 2004 12:12 AM
Well, how is the emulation when comparing to other BSD's? A NetBSD or BSD user would rather know this; given most (new) people (here) would be non-BSD (Linux, Windows) users who haven't decided yet which BSD they'd like to run...


Re:Compare between other BSD's

Posted by: Anonymous Coward on May 14, 2004 01:35 AM
I am running the Linux version of the TextMaker word processor in NetBSD 1.6.2, using emulation. It works quite well. No hangs or slowdowns. It prints hardcopy just fine.


This story has been archived. Comments can no longer be posted.

Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya