Here are the steps to setup multiple uplinks through policy routing on Gentoo:

1. First of all, to access multiple networks, either you have multiple physical NICs or you need to configure your network uplink to let your network ports access multiple VLANs. For more information on VLANs configurations under Gentoo, you can check Gentoo Handbook section on VLANs.
2. On Linux kernel, you need to enable CONFIG_IP_MULTIPLE_TABLES option (in Linux kernel menuconfig, you find it under Networking support => Networking options => IP: policy routing). Recompile and install kernel.
3. Next, you need to install iproute2 package, which allows editing multiple routing tables:

   # emerge -av sys-apps/iproute2

4. Edit /etc/iproute2/rt_tables and add the following route table lines:

   100        T0
   101        T1

5. Edit your /etc/conf.d/net file to enable network startup configuration. First add the following lines, modifying addresses and interface names to suit your needs:

   config_eth0=( "192.168.0.100/24" )
   routes_eth0=(
       "192.168.0.0/24 src 192.168.0.100 table T0"
       "default via 192.168.0.1 table T0"
       "default nexthop via 192.168.0.1 weight 1"
   )
   rules_eth1=("from 192.168.1.1/32 table T0 priority 100" )
   
   config_eth1=( "192.168.1.200/24" )
   routes_eth0=(
       "192.168.1.0/24 src 192.168.1.200 table T1"
       "default via 192.168.1.1 table T1"
       "default nexthop via 192.168.0.1 weight 1"
   )
   rules_eth1=("from 192.168.1.100/32 table T1 priority 101" )

   Then uncomment the following functions (if you copied your /etc/conf.d/net from /etc/conf.d/net.example, they should be already there in comments):

   postup() {
          local x="rules_${IFVAR}[@]"
          local -a rules=( "${!x}" )
          if [[ -n ${rules} ]] ; then
                  einfo "Adding IP policy routing rules"
                  eindent
                  # Ensure that the kernel supports policy routing
                  if ! ip rule list | grep -q "^" ; then
                          eerror "You need to enable IP Policy Routing (CONFIG_IP_MULTIPLE_TABLES)"
                          eerror "in your kernel to use ip rules"
                  else
                          for x in "${rules[@]}" ; do
                                  ebegin "${x}"
                                  ip rule add ${x} dev "${IFACE}"
                                  eend $?
                          done
                  fi
                  eoutdent
                  # Flush the cache
                  ip route flush cache dev "${IFACE}"
          fi
   }
   
   postdown() {
          # Automatically erase any ip rules created in the example postup above
          if interface_exists "${IFACE}" ; then
                  # Remove any rules for this interface
                  local rule
                  ip rule list | grep " iif ${IFACE}[ ]*" | {
                          while read rule ; do
                                  rule="${rule#*:}"
                                  ip rule del ${rule}
                          done
                  }
                  # Flush the route cache
                  ip route flush cache dev "${IFACE}"
          fi
   
          # Return 0 always
          return 0
   }

6. Finally, reboot with your new kernel. My advice is to proceed with this step while you can access your machine locally, just in case anything goes wrong.

Some in-depth on what I described above: with policy routing you can insert additional routing tables and configure your system to use a set of rules to decide which table to apply for each IP packet. So if you create T0 and T1 tables, you can set your host to respond to requests from each interface back to the same interface and load balance routes going to outer network by giving the same weight to both gateways in generic route table.

If you use this setup to publish your server on multiple public networks, you will probably need to configure multiple DNS A records in round-robin over your IPs.

If you’re interested in more Gentoo tips, just subscribe to my feed or follow me on Twitter.

Here’s the top feeds you should be subscribed to (CVE tags are reported in brackets):

1. NIST Vulnerability Database.
2. US Cert Technical Security Alerts [CERT].
3. SecurityFocus Vulnerabilities [SF-INCIDENTS].
4. Open Source Vulnerability Database [OSVDB].
5. IBM Internet Security Systems Threats [ISS].
6. Vupen Security Advisories [VUPEN].
7. Secunia Latest Security Advisories (Unofficial) [SECUNIA].
8. eEye Security Advisories [EEYE].

The above list is also available as OPML file you can import into your feed reader.

Furthermore, you should subscribe to Operating Systems product-centric vulnerability feeds to ensure you receive timely information regarding updated packages and suggested workarounds for your infrastructure. Here’s a comprehensive list, sorted alphabetically:

1. Apple Security Announce (Mac OS X, iPhone, etc) [APPLE].
2. Checkpoint’s SmartDefense Service [CHECKPOINT].
3. Cisco’s Product & Service Security Advisories [CISCO].
4. Debian Security Advisories [DEBIAN].
5. Fedora Security Updates [FEDORA].
6. FreeBSD Security Advisories [FREEBSD].
7. Gentoo Linux Security Advisories (GLSA) [GENTOO].
8. Mandriva Security Advisories [MANDRIVA].
9. Microsoft’s Security Notification Service Comprehensive Edition [MS].
10. NetBSD Security Advisories [NETBSD].
11. OpenPKG Security Advisories [OPENPKG].
12. OpenBSD Errata [OPENBSD].
13. Red Hat Security Advisories [REDHAT].
14. Slackware Linux Security Advisories [SLACKWARE].
15. Solaris SunSolve Alerts [SUNALERT].
16. SUSE Linux Enterprise Security Advisories (also contains OpenSUSE advisories) [SUSE].
17. Ubuntu Security Notices [UBUNTU].

OS security advisory feeds are available as OPML file as well.

Have I missed anything? Please report if you find some advisory feed I accidentally missed. Also, if you’re into an Operating System security team and you don’t offer a security announcement feed, please consider making it available.

The problem with their approach is simple: they asked how to fix their setup, but forgot to ask what they’re trying to protect. Having your root filesystem on an encrypted disk doesn’t protect you from remote exploitation or credential leaks. It just protects you from the risk of someone being able to access your machine locally and steal your data, or just steal the whole machine altogether. Now, if I were an attacker having access to your hardware locally, I could easily setup a trap for you in less than 5 minutes:

1. Shut down your machine and open it.
2. Connect your machine’s root disk to an external USB interface connected to my laptop.
3. Copy your initramfs file from boot partition (which is clear-text, remember?), access internal files and extract SSH server keys.
4. Bring up an interface with a fake ssh server running on my laptop which runs your initrd script, slightly modified to tap passwords.
5. Just wait for you to notice your machine went down and connect via ssh to bring it back up. Ta-da.

Depending on the scenario, some additional step covering may be necessary, but the theory is there: if you can’t check your hardware personally, disk encryption is useless (and even then, human stupidity is the weak link).

In 2004, Autistici/Inventati hacking group was running a server at Aruba server farm and hosting several mailboxes and websites. During a police inquiry on one of those mailboxes, law officers wanted to obtain TLS encryption keys to tap users messages, so they unplugged the server and copied data from the server volumes. When server admins asked Aruba about the downtime, Aruba told them it was an electrical fault, so it took one year to find out about the crackdown. If disks had been encrypted with LUKS and set up for remote unlocking, it would have been quite easy for law officers to trick server admins into typing unlock key over the wire, since ISP employees were under their control.

Bottom line: if you’re paranoid enough to setup encrypted disks, you shouldn’t really trust remote unlocking anyway.

How many times did you start a long-running task like gcc compilation on a remote server and then suddenly needed to disconnect from your shell? Maybe you just needed to move to some other place with your laptop, but if you disconnected from your LAN, your ssh connection would go down. How many times you thought “Damn, if I had launched screen before this…”?

The trick to save your compile time and not break your schedule is simple: just have your shell .profile script run screen at startup on your remote server. For bash, the syntax is simple, just add the following line at the end of your ˜/.profile script:

if [ ${SHLVL} -eq 1 ]; then
    ((SHLVL+=1)); export SHLVL
    exec screen -R -e "^Ee" ${SHELL} -l
fi

Quick implementation notes:

1. Parameter -R reattaches to an existing detached session, if it exists, otherwise creates a new one.
2. Parameter -e sets a non-standard escape character. This is useful since you don’t want login screen to interfere with other screens you may spawn during your activity. I chose Ctrl-E as it’s not used by other well-known keyboard shortcuts and works on most OSes.

To detach from your server type Ctrl-E d or just close your terminal window. Running processes will remain active in background, without detaching from your shell. When you connect to your remote shell again, you’ll get back to your session.

Do you like Unix tips like this? Follow me on Twitter or subscribe to my RSS feed for more.

What I found out was that machines were configured to access netlogon share and run a VBScript script upon logon, to set a few things like printing shares and stuff like that. Unluckily, Windows 2000 has a problem with this, because if VBScript instantiates a WMI object and uses it to read registry keys, then WMI object is not released correctly and this locks the registry hive and therefore Windows logoff sync aborts after a long wait.

The problem is known to Microsoft and a report on the issue is published at KB 819536. Now since this lab is located in Italy and and Windows 2000 is localized in italian, Microsoft Knowledge Base site opened in Italian and the page with the bug report mentioned that an hotfix was available but one should call Microsoft support service at no charge to receive it, no link to download the file or anything. So my collegue looked up Microsoft Italia phone number and called up:

Sysadmin: «Hello, I have a problem with a Windows 2000 workstation and I need a hotfix.»
Microsoft female voice: «Ok, so I need to open a support case and have someone from the staff call you back.»
Sysadmin: «Ok.»
Microsoft female voice: «Is your workstation stand-alone or in a domain?»
Sysadmin: «Domain.» (it was a samba domain, but better not tell.)
Microsoft female voice: «Ok, listen, your machine is NOT in a domain.»
Sysadmin: «Uh?»
Microsoft female voice: «Look, if I open a enterprise support request for machines in a domain, you have to pay € 299, while if you request a stand-alone support call, it costs you just € 79.»
Sysadmin: «Erm…Your website states that hotfixes are free, aren’t they?»
Microsoft female voice: «Possibly, but just in case you need further help, your fee will be lower.»
Sysadmin: «Err…ok…nice.» (these Microsoft employees must be really fed up working for the devil himself.)
Microsoft female voice: «So, now I need your phone number and your credit card details…»

After one hour we received a call from a guy at their support team. We explained him what we needed and he told us that hotfix was free and we would not pay anything. Then he sent the hotfix over via e-mail. The hotfix worked perfectly and replaced C:\WINDOWS\system32\wbem\stdprov.dll version 1.50.1085.104 with version 1.50.1085.105.

In the hotfix e-mail he also dropped a note saying that Windows 2000 is not supported anymore and therefore hotfixes that are not security-related are not available anymore to customers (he sent the hotfix by mistake). He also said that we would not pay the hotfix he already sent, but that we should consider this information for future requests. In a subsequent phone call from him, he told me that his boss made clear that this is the official position of the company regarding hotfixes for Windows 2000.

Now the crazy thing: when I looked back on Microsoft’s knowledge base in English (USA), I noticed that on all country versions except Italy, hotfixes for all languages and all Operating Systems are available for download directly at their site at no charge (a download request link is rendered just under the title).

Moral of the story: proprietary software can actually let you down in ways you don’t imagine, so if you’re near the end of support cycle, download all the patches and hotfixes you can and make backups.

# mdadm --zero-superblock devicepath

If you need to apply this fix on a system that doesn’t boot up (for instance when your root volume is on RAID), remember that mdadm and other disk administration utilities are available in Gentoo minimal installation disk.

UPDATE: Rav asked for the gory details so here it is: when you initially create a Linux RAID array, mkraid writes a signature to the disk called superblock, which contains a unique UUID code for the array and a description of the array (size, raid level, etc). When Linux kernel boots up, this superblock is read by the md kernel module and a minor device number is assigned to the array. Even if you erase your partition table or mbr, this superblock won’t be erased.
The problem arises when you try to add a disk with an existing superblock to a computer that already has another array in place (for instance when replacing a faulty RAID1 or RAID5 disk): if md driver recognises a superblock, it won’t allow your added drive to join the array and will report a generic “Invalid argument” error. Furthermore, it can happen that, if a minor number is forced onto an array, when booting a system with two parts of arrays trying to grab the same minor, none of them can get through and therefore md devices are not available.
So, instead of zeroing the whole disk with dd if=/dev/zero of=/dev/path, which can take a certain amount of time and is quite useless (if you’re rebuilding RAID1 or RAID5, your disk contents will be overwritten by raid reconstruction anyway), you can use the command explained at the beginning to erase just the bad superblock and fix the problem.

Just a final notice: another problem with replacing disks in RAID1 and RAID5 happens when people try to use a volume which is slightly smaller than the others in the array (even if advertised capacity is the same of the old drives, there can be slight differences in actual number of blocks). In this case, the error reported from md upon loading is the same as above: “Invalid argument”. So if your disk is unused, this is probably the first thing to check, otherwise try the following command on the disk device to check for existing superblocks:

# mdadm -E devicepath

The installation was straightforward and hardware support was really seamless (whew! ACPI works like a charm on new Dells), however I noticed that once the system is running, the “works with clueless user” claim (they call it “alternative to Windows“, but that’s the actual meaning) lasts only five minutes.

There are two main issues on the table:

1. Documentation: several aspects of the system have changed but documentation lags behind. For instance, upstart replaced old service management facilities and Services applet was removed from System -> Administration menu, but documentation still refers to it. Furthermore, internal documentation search feature is very primitive compared to Mac OS X or Windows ones and “search the forums” option cannot really replace a knowledge base like Microsoft’s.
2. Robustness: Ubuntu should take the release-early-release-often agility rule with a grain of salt: if replacing a subsystem completely takes a certain amount of time, you cannot really split replacement in two phases just to respect release scheduling. A lot of users are complaining about the fact that every upgrade adds more quirks than the ones it solves.

Open source applications that build up the Ubuntu’s image of Windows alternative are getting more and more mature and a lot of work has been done to make Linux look like a desktop operating system and not just a bunch of pieces put together. Now it’s time for distributions like Ubuntu to step up and fix higher quality standards for the whole development community.

For a start, they could set a common standard for documentation and knowlegde base: to offer a real post-install support, you need to have a common error reporting API, clear error messages (not like Microsoft’s “contact your System Administrator” message) and a central repository for documentation and solutions, with a common writing style (haven’t you noticed a regression since man pages days?), a decent search engine and translations in supported languages. Since no Linux distribution has enough work force to make it all alone, the only alternative is to find an agreement for an interoperability standard with large software projects (Samba, Open Office, Gnome, Cups, etc).

So now my mom is using Ubuntu: I’ve enabled remote controlling (ssh and rdesktop) to help her with the transition and I’ve installed a VirtualBox instance with Windows XP, just in case. I replaced standard theme and wallpaper with something she could find attractive (first impression counts!). If you have any good tips for making Ubuntu experience more comfortable, please share them using the comment box below.

One of the key technologies of enterprise server market is Storage Area Network: an infrastructure that abstracts storage resources. When Linux companies started to compete in server market, Linux had support for accessing SAN storages (Fibrechannel and iSCSI drivers), advanced disk partitioning support (LVM and EVMS), but no free shared-storage filesystem. So RedHat acquired Sistina’s GFS, a shared-storage filesystem, imported some work from OpenGFS developers, released it under Open Source license and evolved it to GFS2.

In the meanwhile, Novell looked around and found that Oracle had an ongoing open source project named OCFS2. It was a general purpose refactoring of original OCFS filesystem, which Oracle had developed years before to deal with clustering of its database product. So Novell decided to integrate OCFS2 into its Suse Linux Enterprise Server platform and advertise it as their top-notch mature filesystem for clustering in SAN environment.

Unluckily, what Novell marketing dept didn’t actually know is that OCFS2 has never been production-ready, yet.

In the last two years, I’ve deployed a number of OCFS2 filesystems with Novell SLES 10 SP2 and experienced the following troubles:

1. In certain situations, filesystem reports “Not enough disk space” even if df reports 50-60% usage, due to a bug in inode allocation when disk is very fragmented. This bug was reported over two years ago and is still “in the wild”!
2. If a node crashes, it has no support for intelligent fencing like RedHat’s, so if your cluster has several nodes, you may need to restore quorum manually.
3. There are several racing conditions in file locking that lead to corruption in shared bdb databases or similar faults.
4. Sharing OCFS2 folders with Samba on the nodes crashes the kernel, due to a bug in distributed locking routines. This bug was posted over one year ago and is still marked as “NEW” in Oracle’s bugzilla.
5. In the event of a system crash, OCFS2 may not recover automatically and needs a fsck. In this case, fsck takes forever, may report critical errors and finally fail, leaving the filesystem unusable and unrecoverable.
6. Restoring from backup a SAN filesystem of several Terabytes on OCFS2 takes longer. How longer? More.

Any attempt to fix these problems using Novell rpm packages, Oracle-released source packages, Linux stock kernels, Linux experimental branches and patches found on bugzilla failed miserably. Furthermore, it’s pretty clear that Oracle treats users as if they were beta-testers.

Buyers beware: OCFS2 sucks.

Is GFS2 any better? Yes (it’s really designed as an enterprise product and integrated with RedHat clustering suite), but it’s still too slow for enterprise applications.

Bottom line: Don’t believe the marketing vapor, Linux on a SAN in 2010 is still a no-go.

Shell prompt

First of all, you should check you’re using bash shell:

$ env | grep ^SHELL=
SHELL=/bin/bash

If not, you should check if bash package is already installed. If it’s already there, just change your user shell (and possibly root shell) with chsh or consult your OS manual. Be careful: if bash is not listed in /etc/shells, you might lock yourself out.

To set Gentoo-like colors on bash prompt, edit ~/.profile (or /etc/profile for system-wide defaults) and add the following:

if [[ ${EUID} == 0 ]] ; then
        PS1='\[\033[01;31m\]\h\[\033[01;34m\] \W \$\[\033[00m\] '
else
        PS1='\[\033[01;32m\]\u@\h\[\033[01;34m\] \w \$\[\033[00m\] '
fi

GNU ls

If you use GNU ls (use –version to check) and dircolors utility is available (from GNU coreutils), you can have colorful outputs by adding the following snippet to your bash profile (see above):

if type -P dircolors >/dev/null ; then
        if [[ -f ~/.dir_colors ]] ; then
                eval $(dircolors -b ~/.dir_colors)
        elif [[ -f /etc/DIR_COLORS ]] ; then
                eval $(dircolors -b /etc/DIR_COLORS)
        fi
fi
alias ls='ls --color=auto'

To enable colors, you need to save Gentoo color defs to ~/.dir_colors (or /etc/DIR_COLORS for system-wide defaults).
Alternatively, if you miss dircolors binary on your system, save output from dircolors on a Gentoo machine and copy’n'paste it into bash profile:

LS_COLORS='...weird colon-separated string...'
export LS_COLORS
alias ls='ls --color=auto'

BSD ls implementation

A special trick is required for FreeBSD and Mac OS X: add the following line to your bash profile:

export CLICOLOR=1 LSCOLORS="ExGxFxDxCxDxDxhbhdacEc"

If you want to further customize colors on your Mac OS X Terminal, you can use SIMBL TerminalColours plugin (Snow Leopard version).

GNU grep

You can enable coloring of the matched part by appending the following alias to your bash profile:

alias grep='grep --colour=auto'

As usual, if you have any coloring tips’n'tricks you want to share, please use the comment box below.