What evil lurks in OCFS2

In the beginning, Linux was a free general purpose OS and it was not clear how Linux companies would generate profits out of it. In 1999 RedHat company went public and started to develop a real business plan. After a few years, in 2003, one of its main competitors, SuSE Linux, was acquired by Novell. Since then, both companies worked hard to reduce their involvement in desktop solutions and develop a segment known as “server market”.

One of the key technologies of enterprise server market is Storage Area Network: an infrastructure that abstracts storage resources. When Linux companies started to compete in server market, Linux had support for accessing SAN storages (Fibrechannel and iSCSI drivers), advanced disk partitioning support (LVM and EVMS), but no free shared-storage filesystem. So RedHat acquired Sistina’s GFS, a shared-storage filesystem, imported some work from OpenGFS developers, released it under Open Source license and evolved it to GFS2.

In the meanwhile, Novell looked around and found that Oracle had an ongoing open source project named OCFS2. It was a general purpose refactoring of original OCFS filesystem, which Oracle had developed years before to deal with clustering of its database product. So Novell decided to integrate OCFS2 into its Suse Linux Enterprise Server platform and advertise it as their top-notch mature filesystem for clustering in SAN environment.

Unluckily, what Novell marketing dept didn’t actually know is that OCFS2 has never been production-ready, yet.

In the last two years, I’ve deployed a number of OCFS2 filesystems with Novell SLES 10 SP2 and experienced the following troubles:

  1. In certain situations, filesystem reports “Not enough disk space” even if df reports 50-60% usage, due to a bug in inode allocation when disk is very fragmented. This bug was reported over two years ago and is still “in the wild”!
  2. If a node crashes, it has no support for intelligent fencing like RedHat’s, so if your cluster has several nodes, you may need to restore quorum manually.
  3. There are several racing conditions in file locking that lead to corruption in shared bdb databases or similar faults.
  4. Sharing OCFS2 folders with Samba on the nodes crashes the kernel, due to a bug in distributed locking routines. This bug was posted over one year ago and is still marked as “NEW” in Oracle’s bugzilla.
  5. In the event of a system crash, OCFS2 may not recover automatically and needs a fsck. In this case, fsck takes forever, may report critical errors and finally fail, leaving the filesystem unusable and unrecoverable.
  6. Restoring from backup a SAN filesystem of several Terabytes on OCFS2 takes longer. How longer? More.

Any attempt to fix these problems using Novell rpm packages, Oracle-released source packages, Linux stock kernels, Linux experimental branches and patches found on bugzilla failed miserably. Furthermore, it’s pretty clear that Oracle treats users as if they were beta-testers.

Buyers beware: OCFS2 sucks.

Is GFS2 any better? Yes (it’s really designed as an enterprise product and integrated with RedHat clustering suite), but it’s still too slow for enterprise applications.

Bottom line: Don’t believe the marketing vapor, Linux on a SAN in 2010 is still a no-go.

This entry was posted in Rants, Sysadmin and tagged , , , , , , , , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

7 Comments

  1. Posted January 29, 2010 at 1:11 PM | Permalink

    Do you know any benchmark between GFS2 and OCFS2?

  2. Posted January 29, 2010 at 1:48 PM | Permalink

    Currently, there are no reliable benchmarks, due to the fact that several nasty performance bugs have been fixed in OCFS2 in the last 2 years.

    From my experience, current OCFS2 is quite faster (1.5x to 2x) in metadata handling, so overall performance difference between OCFS2 and GFS2 is more noticeable in filesystems with lots of small files and lots of files per directory (for instance a mail server cluster with Maildir+ storage).

    About 6 months ago, I did some testing on a Gentoo 3-nodes cluster with Fibrechannel-connected storage with most current releases. A typical performance test was extraction of a Gentoo portage snapshot with tar and an rsync from our local mirror. GFS2 took several minutes longer than OCFS2 and load peak was 3 times higher.

  3. Craig
    Posted January 30, 2010 at 2:54 PM | Permalink

    Do you have bug-numbers or a link to them in the tracker? Thanks.

  4. Posted January 30, 2010 at 4:11 PM | Permalink

    @Craig: links in the post lead to the bugs in the oracle's tracker (#915 and #1058).

  5. blernsball
    Posted January 30, 2010 at 10:41 PM | Permalink

    As far as I know, Oracle has abandoned OCFS2 for newer development efforts. It's kinda surprising that Novell picked it up

  6. Posted January 31, 2010 at 2:09 AM | Permalink

    @blernsball: could you please report your source? There's no mention of dropping support or stopping development for OCFS2 in Oracle's website or in their OCFS2 roadmap published at http://oss.oracle.com/osswiki/OCFS2/Roadmap

  7. Marc
    Posted June 29, 2010 at 1:48 PM | Permalink

    Any news on these issues being solved in ocfs2 1.4 as released May 2010

One Trackback

  1. [...] of Novell and Oracle, here is a new post that speaks about both. It’s negative news: Novell looked around and found that Oracle had [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>