When performing zfs perambulations after install (as noted in ZFS and NAStie and i386 and not reading the specification), the chance of finger fumbling errors is high. To prevent those errors, I modified the zfsinstall script included in the mfsBSD 8.2-RELEASE-p2 ISO to configure the paths to different dataset objects as described on the FreeBSD Wiki.

A patch is available for future use/reference, and the contents of the patch are displayed below.

--- zfsinstall.original 2011-11-26 14:48:37.000000000 -0500
+++ zfsinstall  2011-11-26 14:53:19.000000000 -0500
@@ -4,7 +4,7 @@
 # mfsBSD ZFS install script
 # Copyright (c) 2011 Martin Matuska <mm at FreeBSD.org>
 #
-FS_LIST="var tmp"
+FS_LIST="home tmp usr usr/ports usr/ports/distfiles usr/ports/packages usr/src var var/crash var/db var/db/pkg var/empty var/log var/mail var/run var/tmp srv"

 usage() {
    echo "Usage: $0 [-h] -d geom_provider [-d geom_provider ...] -t archive_file [-r mirror|raidz] [-m mount_point] [-p zfs_pool_name] [-V zfs_pool_version] [-s swap_partition_size] [-z zfs_partition_size] [-c] [-l] [-4]"
@@ -275,16 +275,40 @@
 echo " ... done"
 echo -n "Creating ${POOL} partitions:"
 for FS in ${FS_LIST}; do
+   ZFSOPTS=""
    if [ "${LEGACY}" = 1 ]; then
        MNTPT="-o mountpoint=legacy"
    else
        MNTPT=
    fi
-   if ! /sbin/zfs create ${MNTPT} ${POOL}/root/${FS} > /dev/null 2> /dev/null; then
+   # Added to follow http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
+   case "${FS}" in
+   home|usr|var)
+       ZFSOPTS=""
+       ;;
+   srv|usr/src|var/crash|var/log|var/mail)
+       ZFSOPTS="-o compression=lzjb -o exec=off -o setuid=off"
+       ;;
+   tmp|var/db/pkg|var/tmp)
+       ZFSOPTS="-o compression=lzjb -o exec=on -o setuid=off"
+       ;;
+   usr/ports)
+       ZFSOPTS="-o compression=lzjb -o setuid=off"
+       ;;
+   usr/ports/distfiles|usr/ports/packages)
+       ZFSOPTS="-o compression=off -o exec=off -o setuid=off"
+       ;;
+   var/db|var/empty|var/run)
+       ZFSOPTS="-o exec=off -o setuid=off"
+       ;;
+   esac
+   # Added ZFSOPTS to follow http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
+   if ! /sbin/zfs create ${MNTPT} ${ZFSOPTS} ${POOL}/root/${FS} > /dev/null 2> /dev/null; then
        echo " error"
        exit 1
    fi
    echo -n " ${FS}"
+
 done
 echo " ... done"
 echo -n "Setting bootfs for ${POOL} to ${POOL}/root ..."
@@ -313,6 +337,10 @@
 echo " done"
 fi

+# Added to follow http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
+chmod 1777 ${MNT}/tmp
+chmod 1777 ${MNT}/var/tmp
+
 echo -n "Extracting FreeBSD distribution ..."
 if [ "${DIRECT_TAR}" = "1" ]; then
    if ! /usr/bin/tar -C ${MNT} -x -f ${ARCHIVE} > /dev/null 2> /dev/null; then
@@ -395,6 +423,9 @@
    echo "Error mounting devfs on ${MNT}/dev"
 fi

+# Added to follow http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror
+zfs set readonly=on ${POOL}/root/var/empty
+
 echo ""
 echo "Installation complete."
 echo "The system will boot from ZFS with clean install on next reboot"

Ahem.

Apparently one shouldn't skim the docs when putting together a server. Apparently one should actually read the portion which tells you what type of processor you have (for example, if you have a 64-bit processor). Occasionally that type of information would be useful... like... say... when you're trying to get your FreeBSD installation using ZFS to be stable, which it is NOT on 32-bit (ok, some small portion of the population has aligned the stars, but I did not)! One should not depend on one's memory when one thinks, "Self, that system is old so the processor must be 32-bit," and doesn't check to be sure.

Yep... I noticed a few days ago the Pentium 4 2.8 GHz processor in my server is indeed 64-bit, silly me. So I immediately attempted to load 9.0-RC2, only to have it fail to bood. RC1 (off of PC-BSD) failed as well, and then I was off and running in an attempt to find out what caused this PR (filed recently).

After digging a bit I discovered the mfsBSD images, a ZFS boot howto, and re-discovered the FreeBSD Wiki page on ZFS booting. Those gave me everything I needed to get back up and running using an amd64 image.

So, first thing was to download the ISO from the mfsBSD page (8.2p2 special edition) and boot... it works! The zfsinstall command line I used follows.

zfsinstall -d ad6 -t /cdrom/8.2-RELEASE-p2-amd64.tar.xz -s 8G -p nastie

After install I rebooted (instead of chrooting into /mnt) into the new environment, and it had that new car smell... mmmm. Anyway, tasks I did right after reboot.

I created lots of other mountpoints (the default only creates /, /var, /tmp) following the FreeBSD Wiki page.

Note about this: Remember to make sure you move any directories/files out of the way before creating the mount point... confusion reigned for a moment until my brain caught up with my fingers.

New mountpoints created (from the wiki - all commands copied from the wiki and modified for my setup) follow. BUYER BEWARE: I'm only showing the commands necessary to create mountpoints and set options, not those necessary to destroy the already created mountpoints or move files out of the way while creating a mountpoint which was already created as a directory when installing mfsBSD.

Set the checksum algorithm.

# zfs set checksum=fletcher4 zroot

Create /tmp and set the proper permissions.

# zfs create -o compression=lzjb -o exec=on -o setuid=off nastie/root/tmp
# chmod 1777 /tmp

Create the /usr and /usr/home.

# zfs create nastie/root/usr
# zfs create nastie/root/usr/home
# cd / ; ln -s /usr/home home

Create the ports hierarchy.

# zfs create -o compression=lzjb -o setuid=off nastie/root/usr/ports
# zfs create -o compression=off -o exec=off -o setuid=off nastie/root/usr/ports/distfiles
# zfs create -o compression=off -o exec=off -o setuid=off nastie/root/usr/ports/packages

Use the src, Luke!

# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/usr/src

A variety of various subdirectories of var.

# zfs create nastie/root/var
# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/var/crash
# zfs create -o exec=off -o setuid=off nastie/root/var/db
# zfs create -o compression=lzjb -o exec=on -o setuid=off nastie/root/var/db/pkg
# zfs create -o exec=off -o setuid=off nastie/root/var/empty
# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/var/log
# zfs create -o compression=gzip -o exec=off -o setuid=off nastie/root/var/mail
# zfs create -o exec=off -o setuid=off nastie/root/var/run
# zfs create -o compression=lzjb -o exec=on -o setuid=off nastie/root/var/tmp
# chmod 1777 /var/tmp
# zfs set readonly=on nastie/var/empty

Finally the service NAStie will provide.

# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/srv

That sets up the file systems. Now all that needs to be done is add a normal user and start setting up the services on the machine. It is interesting that this was much faster than installing normally. Have to look into building my own custom release sometime.

I'll add anything I notice must be done, but between this post, Getting NAStie with ZFS, and Rebuilding NAStie I should be able to reconstruct this server.

When installing CYGWIN in the future, remember to visit this page. It tells you how to use bash.exe --login -i instead of the batch file.

A tiny reminder to (when playing with Thunderbird) do the following in the config editor.

mail.check_all_imap_folders_for_new = true
mailnews.default_sort_order = 22
mailnews.default_sort_type = 1
mailnews.default_view_flags = 1
mailnews.headers.showSender = true
mailnews.headers.showUserAgent = true

Sigh... much better.

Oh, yeah, found at http://www.everything-mdaemon.com/general/threading-messages-in-thunderbird.

In Debian on Macbook Pro 5.2 I mentioned using the xterm*font resource to get rid of the nasty fonts, and then couldn't find the ttf fonts in xfontsel/xlsfonts (despite using my previous post about xorg font hell), and began to get very frustrated.

Apparently, the frustration was not necessary. Apparently, while I've spent my time in OS X land, the Linux universe was moving further and further from having X provide the fonts, instead using fontconfig and other tools. Also xterm has a method to access ttf fonts separate from the normal X fonts.

The following (in combination with a recent article on the top 10 programming fonts) fixed me up.

xterm*faceName:     DejaVu Sans Mono
xterm*faceSize:     8

I just plopped it in my .Xresources file, removed the xterm*font line, merged the resources using xrdb -merge ~/.Xresources, and relaunched my xterms.

Now, why is FireFox giving me fits?

So, Mac OS X got on my last nerve and led me back to the holy land (linux, what else?). While FreeBSD beckoned there are certain pieces of software I must have working for this Quarter. It's the first of my Master's, so I want to get in the proper habits (such as not staying up all night getting Mathematica or MATLAB working in FreeBSD, when Linux versions are available ;)).

Regardless, some notes about stumbling blocks would be helpful if I need to do this again, so here goes.

WIFI - The wireless is a Broadcom Corporation BCM4322.

$ lspci -s 04:00.0
04:00.0 Network controller: Broadcom Corporation BCM4322 802.11a/b/g/n Wireless LAN Controller (rev 01)

Checking the debian wiki page about Broadcom wireless cards indicated the fix isn't difficult... oh, unless you don't have a connection to the Internet so you can't get the software so you don't have conne... ahem.

The solution (some might have simply plugged in upstairs by the router and continued on their merry way - but certainly not I) was to download the needed debs on another computer (thanks wif!), and create a small temporary repository to hold them until I got connected. Why didn't I just use dpkg to install the debs? Good question: The dpkg method doesn't handle dependencies, the local respository method does.

The method used was from a handy obsolete page about repositories and allowed me to install the two packages ([broadcom-sta-common and broadcom-sta-source]) with all their dependencies, and move on to the configuration (remember, every time the kernel changes, you must rebuild the broadcom-sta module).

Configuring wireless is fairly painless... at least it is once you realize the wlan0 interface doesn't actually exist! That's right folks, it's a red herring to trick you into typing silly things like sudo ifup wlan0 (when what you really want is sudo ifup eth1). Heh.

Next annoying thing to attract my wrath was the function keys... whoever decided it was a good idea to make the function keys act like the cute little alternate functions... well, let's just say there's a reason I left Mac OS X. I'll probably pick up an MS Natural Keyboard just to have an MS product hooked to a Mac... well, that and (other than the plus shaped extra arrow keys useless for gaming) it's my favorite keyboard of all time.

The fix to the function key problem was on a [page about the Apple Keyboard][patak], go figure.

Huh... go figure someone created a daemon which fixed all the problems I had with function keys (without updating initramfs) and the brightness adjustment which was my next stop on the way to insanity! Check out pommed if you're installing on a Mac.

Next to fix was... erm... oh, yeah, nvidia drivers... wait! I almost forgot, the annoyingly small font in xterm. The line I used in my .Xresources file follows.

xterm*font:     -misc-fixed-medium-r-normal-*-17-*-*-*-*-*-iso10646-1

NOW for the nvidia drivers which was very anticlimactic. First, I'm running testing, so YMMV if you're not. All I did was install the nvidia-kernel-dkms package, then put the following in an /etc/X11/xorg.conf.

Section "Device"
    Identifier  "My MASSIVE GPU"
    Driver      "nvidia"
EndSection

That's it, and it worked. Weird.

Last problem: Fonts don't want to show up with the new 'modularitalized' xorg, and my xorg font hell post didn't help, so I'll have to go digging. Oh, yeah, firefox all of a sudden starts giving me fits (CPU hogging, machine lagging, but never crashing fits). So, I've installed chromium (chrome's little brother) until I can figure it out... wonder if adblock plus is available for chromium.

That's all for this set of adventures. Hopefully I'll remember to look here next time I install on a Mac.

According to various sources, stability of ZFS on i386 boxes is going to be... not a priority. Regardless, I decided to take up the challenge.

My first attempt to get everything stable was to use zero settings in the loader.conf file. That lasted until my first big ports build. That didn't work.

A similar (not the actual, I've received plenty and just chose the last one) error message follows.

panic: kmem_malloc(86016): kmem_map too small: 520044544 total allocated
cpuid = 0
KDB: enter: panic
[ thread pid 0 tid 100228 ]
Stopped at      kdb_enter+0x3a: movl    $0, kdb_why

This is consistent every time I load the disks heavily (such as trying to copy the entirety of HEAD's svn repository to another directory). Now, my goal is to attempt to get things stable under heavy load. I started using the following loader.conf settings.

# /boot/loader.conf excerpt
vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="128M"
vfs.zfs.vdev.cache.size="5M"

That didn't work. To avoid such quoting throughout, I will refer to such settings in the future as vm.kmem_size/vm.kmem_size_max/vfs.zfs.arc_max/vfs.zfs.vdev.cache.size. Using that the above values would be 512M/512M/128M/5M. Now, since that didn't work, and after some IRC goodness on #freebsd, I decided to just try 384M/384M/64M/5M. Now, there's no real empirical reason to use those, but they are close to the 330M/330M/40M/5M which worked when NAStie had 784M of memory, so perhaps this will do the trick.

We'll see, now to rsync and compile a lot!

Update: Hrm... after transfering the latest Mathematica download (about 800 MB), and then removing the Mathematica 7 ISO for Mac (can't remember how much), system dumped again. I'm going back to the 330M/330M/40M/5M which worked for me when I only had 768 MB of memory. We'll see how that works.

Update: Odd, had a middle of the night shutdown, didn't notice until this evening. I'll reboot and see what's what. No change in values.

Update: Even odder, can't get the server to stay stable for longer than (about) a day. Every time I stress the drives it dies (technical term). Going to try moving to AMD64 (more in the new post).