One of the key technologies of enterprise server market is Storage Area Network: an infrastructure that abstracts storage resources. When Linux companies started to compete in server market, Linux had support for accessing SAN storages (Fibrechannel and iSCSI drivers), advanced disk partitioning support (LVM and EVMS), but no free shared-storage filesystem. So RedHat acquired Sistina’s GFS, a shared-storage filesystem, imported some work from OpenGFS developers, released it under Open Source license and evolved it to GFS2.

In the meanwhile, Novell looked around and found that Oracle had an ongoing open source project named OCFS2. It was a general purpose refactoring of original OCFS filesystem, which Oracle had developed years before to deal with clustering of its database product. So Novell decided to integrate OCFS2 into its Suse Linux Enterprise Server platform and advertise it as their top-notch mature filesystem for clustering in SAN environment.

Unluckily, what Novell marketing dept didn’t actually know is that OCFS2 has never been production-ready, yet.

In the last two years, I’ve deployed a number of OCFS2 filesystems with Novell SLES 10 SP2 and experienced the following troubles:

1. In certain situations, filesystem reports “Not enough disk space” even if df reports 50-60% usage, due to a bug in inode allocation when disk is very fragmented. This bug was reported over two years ago and is still “in the wild”!
2. If a node crashes, it has no support for intelligent fencing like RedHat’s, so if your cluster has several nodes, you may need to restore quorum manually.
3. There are several racing conditions in file locking that lead to corruption in shared bdb databases or similar faults.
4. Sharing OCFS2 folders with Samba on the nodes crashes the kernel, due to a bug in distributed locking routines. This bug was posted over one year ago and is still marked as “NEW” in Oracle’s bugzilla.
5. In the event of a system crash, OCFS2 may not recover automatically and needs a fsck. In this case, fsck takes forever, may report critical errors and finally fail, leaving the filesystem unusable and unrecoverable.
6. Restoring from backup a SAN filesystem of several Terabytes on OCFS2 takes longer. How longer? More.

Any attempt to fix these problems using Novell rpm packages, Oracle-released source packages, Linux stock kernels, Linux experimental branches and patches found on bugzilla failed miserably. Furthermore, it’s pretty clear that Oracle treats users as if they were beta-testers.

Buyers beware: OCFS2 sucks.

Is GFS2 any better? Yes (it’s really designed as an enterprise product and integrated with RedHat clustering suite), but it’s still too slow for enterprise applications.

Bottom line: Don’t believe the marketing vapor, Linux on a SAN in 2010 is still a no-go.

This post deals with two system-wide alternatives, sysinitv and Mac OS X’s launchd: the first represents the tradition, while the latter represent innovation. Feel free to use comments to share your tips.

System V init

Also known as sysinitv, it’s the historical choice for most unices. Every OS has customized its own version so there’s no common standard. First process, typically /sbin/init reads config in /etc/inittab and executes a series of commands, depending on which runlevel is currently set.

In typical configurations, init launches a main rc script on runlevel change (boot sequence runlevel, default multiuser runlevel, etc), which in turn runs all the initialization scripts in a runlevel-specific folder. For example, in Redhat Linux each system service has an init script in /etc/init.d/. These init scripts respond to a number of actions that can be passed as parameter, for example /etc/init.d/sendmail start runs a function that starts sendmail daemon. Typical actions include: start, stop, restart, status. Some scripts implement specific actions: for instance apache service implements reload action, which does a graceful signal-based reload. Furthermore, most OSes include additional dependency information in those scripts and import script configuration directives from external files (/etc/sysconfig/ on Debian, SuSE, etc, /etc/conf.d/ in Gentoo).

When rc scripts is ready to process init scripts, it looks in appropriate runlevel folder (in RedHat Linux and derivatives its /etc/rc.d/rc<runlevel>.d/,  Gentoo has /etc/runlevels/<runlevel>/, etc) and executes entries which are symlinks to the actual /etc/init.d scripts.

Most of these systems use a specific naming convention for these symlinks, to define order of execution (scripts are run in alphabetic order) and action to be passed as parameter. For example, on RedHat-style systems, scripts are named as S<NN><service> or K<NN><service>, where NN is a number from 00 to 99. S means start action (ie. /etc/rc.d/rc3.d/S60sendmail start), while K means stop action. In other systems such as Gentoo, order of execution is determined dynamically by dependency information (requires additional computing and dependency caching).

Operating systems typically implement command-line, ncurses-based or X11-based tools to manage adding and removing items from runlevels: chkconfig under RedHat/CentOS, update-rc.d under Debian, rc-update under Gentoo, boot up manager (bum) under Ubuntu.

Pros: well known, easy for unix admins, easy to add new custom scripts.
Cons: no common standard, no parallel execution (except for Gentoo), no support process monitoring (ie. restarting daemons on crash).

launchd

It’s installed by default on Mac OS X 10.4 (Tiger) and later. Once launchd is run by the kernel, it runs /etc/rc (a BSD-style system wide initialization script) and scans/System/Library/LaunchAgents and /Library/LaunchDaemons for plist files. Each plist contains a series of properties that define behaviour of launchd and customize run setup (command parameters, working directory, process owner, resource limits, etc). Launchd monitors processes and dynamically acts to adapt process status to requirements defined by configuration. This includes: running process automatically at boot, respawning process on unexpected quit, starting and stopping process upon several conditions (cpu load, network status, filesystem mounts, file presence, time schedule, shutdown, sleep, etc).

Finally, launchd companion is user tool launchctl, which interacts with launchd through a socket and allows administrator to send in requests like start, stop, list, limit, shutdown, etc.

Pro: amazing configuration directives, powerful extension mechanism, post-spawn process monitoring and usage management, scheduled actions (as in cron), stdio-to-inet support as inetd, can add processes without configuration files.
Cons: configuration plists are an Apple-world standard, many great features are Mac OS X specific, strict process launching requisites (see launchd.plist manpage), no support for configurable dependencies, no support for custom process signaling, complex logic may lead to unexpected results.

(continues)