Setting up a NAS Box with FreeBSD and a Raspbery Pi 4

27th June, 2019

At present I've got an old Dell PC acting as my NAS box. I have two 6T HDD drives set up as a ZFS mirror pool. This is running Ubuntu server fine.

I know that the system is chewing a not insignificant amount of power running 24/7. It also has a two drive limit, but I want to be able to have up to four drives attached. This will allow me to:

The Raspberry Pi 4 with 4G of RAM has just been announced. It has a proper 1G Ethernet port and doesn't use much power. It also has a USB 3 port. So I've decided to buy a RPi4 and one of these Orico 6648US3-C 4-bay SATA to USB 3 docking stations. This will connect to one of the RPi4's USB 3 ports and make a NAS box. I can install my existing two HDDs and have two bays spare.

Trying ZFS Out on the Raspberry Pi

While I wait for the RPi4 and the Orico 6648US3-C to arrive, I've dug out my RPi3 so that I can get ZFS up and running on it.

To cut a long story short, I tried Raspian and couldn't get the ZFS modules to compile and load. Then I tried Ubuntu for the RPi3 and got the same result. Then I remembered that FreeBSD runs on the Raspberry Pi. This has native support for ZFS and I used to use FreeBSD a lot.

So, let's give FreeBSD a whirl and see if it will meet my needs. Here's how it has gone so far. I'll keep updating the blog entry as I get things working.

Putting FreeBSD on the SD Card

I went to the Download FreeBSD page and got the FreeBSD 12.0 SD card image for the RPI3.

Back on my desktop box, I did these commands to decompress the file and burn it to my SD card:

unxz FreeBSD-12.0-RELEASE-arm64-aarch64-RPI3.img.xz
sudo ddrescue -D --force FreeBSD-12.0-RELEASE-arm64-aarch64-RPI3.img /dev/sdc

I put the SD card into my RPi3 and booted it up. There's no GUI, just a console screen. I logged in as root and password root.

Basic Configuration

I used adduser to create a user called wkt. I used passwd to set root's password. I edited /etc/group and put wkt into the wheel group; this allows wkt to run the su command and become root:

wheel:*:0:root,freebsd,wkt

I set a static IP address for the Pi by adding these lines to /etc/rc.conf:

ifconfig_ue0="inet 10.10.1.90 netmask 255.255.255.0"
defaultrouter="10.10.1.1"

With this done, I can set up the ue0 interface:

/etc/rc.d/netif restart ue0

To set up the DNS server, I created the /etc/resolv.conf file with the details of the servers and the domain to use for names with no dots:

search local.net
nameserver 10.10.1.1
nameserver 8.8.8.8

At this point I can remove the USB keyboard and HDMI monitor, remotely login with ssh as user wkt and use the su command to get to a root shell.

Packages

I'm going to run several services on this device, so I installed these packages by running these commands as root:

pkg install apache24 bash minidlna proftpd rsnapshot rsync samba48 sudo vtun

At this point I haven't done any configuration for these packages.

Sudo

I'm used to running sudo on Linux, so I've installed the sudo package. In the file /usr/local/etc/sudoers, I removed the comment at the beginning of this line so that wheel group members can sudo:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL) ALL

NTP

The Pi doesn't have a battery-backed clock, so I added these lines to /etc/rc.conf to start the NTP daemon and to set the system clock at boot:

ntpd_enable="YES"
ntpdate_enable="YES"                    # Run ntpdate to sync time on boot
ntpdate_hosts="pool.ntp.org"            # List of ntpdate(8) servers.

To set the system's timezone, I linked the relevant zone file thus:

ln -s /usr/share/zoneinfo/Australia/Brisbane /etc/localtime

ZFS Setup

I can't connect my existing ZFS mirror pool to the RPi3 as they are SATA drives; I wouldn't want to anyway, not until I know that things work.

Luckily, I have an external USB hard drive which is configured as a ZFS pool with the name TAFE2T. The nice thing about FreeBSD is that ZFS is already set up and ready to go. All I had to do was plug the USB drive into my existing RPi3 and do (as root):

zfs import TAFE2T

and this is what I see:

[root@naspi /etc]# df -h
Filesystem                Size    Used   Avail Capacity  Mounted on
/dev/ufs/rootfs            14G    3.0G     10G    23%    /
devfs                     1.0K    1.0K      0B   100%    /dev
/dev/msdosfs/MSDOSBOOT     50M     13M     37M    26%    /boot/msdos
tmpfs                      50M    4.0K     50M     0%    /tmp
TAFE2T                    1.7T    171G    1.6T    10%    /TAFE2T
TAFE2T/Henry              1.6T     20G    1.6T     1%    /TAFE2T/Henry

Now, how do I ensure that this gets auto-mounted at boot time? This article says to do the zfs import so that the system knows the pool names, and add this line to /etc/rc.conf:

zfs_enable="YES"

OK, so let's reboot the system and see what happens:

[root@naspi /etc]# reboot
Connection to naspi closed by remote host.

After waiting about a minute for the RPi3 to reboot, I log back in with ssh and I see ... no ZFS pools. Hmm, I can do zfs mount -a and the pools get mounted. So there is something going on that I haven't figured out yet.

It looks like ZFS starts before the USB drives are attached. So my solution is to add this Bourne shell script to /etc/rc.local:

#!/bin/sh
# Try to mount the ZFS USB filesystems at boot time
done="no"
for i in 1 2 3 4 5 6 7 8 9 10
do if [ -c "/dev/da0" -a $done = "no" ]
   then logger "Mounting the ZFS stuff"
    zfs mount -a
        done="yes"; break
   else
     logger "Waiting for /dev/da0"; sleep 30
   fi
done

I probably don't need the done variable and can rely on the break to leave the loop.

More ZFS Settings

When I read the output of dmesg. I see these ZFS warnings:

ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
             Consider tuning vm.kmem_size and vm.kmem_size_max
             in /boot/loader.conf.

So I looked these up. The consensus is to leave prefetch disabled when there is less than 4G of RAM. Hopefully when the RPi4 arrives (with 4G) this will not be a problem. I've also tuned the kmem_size by adding these lines to /boot/loader.conf:

# Tune the VMS settings for ZFS
vm.kmem_size="512M"
vm.kmem_size_max="512M"

ProFTPD

I have a security camera which stores videos and images to a server using the FTP protocol, so I've installed proftpd on the RPi3. In /usr/local/etc/proftpd.conf I turned UseIPv6 off as I don't use it. In /etc/rc.conf I added:

proftpd_enable="YES"

to enable the service at boot time. I started it manually by doing:

service proftpd start

I can ftp in as a user and see my home directory. Now I'll set up a second user (with adduser) who has /usr/sbin/nologin as their shell (so they can't ssh in) and with a home directory where the videos and images will be stored.

Ah, I can't ftp in as the second user as proftpd doesn't know that their shell is valid. So I edited /etc/shells and added /usr/sbin/nologin as a valid shell.

Apache Web Server

Once I installed the apache24 package, there is a vanilla Apache 2.4 web service ready to go. The ServerRoot is /usr/local/www/apache24/data and there is a user www and group www to own the files. To start it, add the line

apache24_enable="YES"

to /etc/rc.conf and then run the command

service apache24 start

Samba

The NAS box is going to be used by myself and my wife, so I'll need to set up SMB file sharing with Samba. To do this, I'm copying the existing Samba config file from the existing Dell server. On FreeBSD, this is the file /usr/local/etc/smb4.conf. Here is most of my file:

[global]
wins support = yes
workgroup = WKTHOME
server string = %h server (Samba, FreeBSD)
log file = /var/log/samba4/log.%m
log level = 3
max log size = 1000
hosts allow = 10.10.1., 10.10.2., 192.168.2., 127.0.0.1
security = user
; Allow HP printer to login with XP-style authentication
ntlm auth = yes

[Music]
comment = Music
path = /usr/500/Music
public = yes
writable = no

[Scan]
comment = Public Stuff
path = /usr/Winshare/Scan
public = yes
writeable = yes
printable = no
write list = @staff

Samba keeps its own username/password database, so now I need to set up a Samba user:

]# smbpasswd -a -U wkt      
New SMB password:
Retype new SMB password:
Forcing Primary Group to 'Domain Users' for wkt
Forcing Primary Group to 'Domain Users' for wkt
Added user wkt.

As usual, we enable the service by adding a line to /etc/rc.conf:

samba_server_enable="YES"

and run a command to start the service:

service samba_server start

MiniDLNA Service

I keep a store of music and videos on my existing NAS box and serve these out to my Smart TV with minidlna. On the RPi3, I had to create the directories for my files:

mkdir -p /usr/Winshare/Photos /usr/500/Photos /usr/500/Video /usr/500/Music
chmod 755 /usr/Winshare/Photos /usr/500/Photos /usr/500/Video /usr/500/Music

Then I edited /usr/local/etc/minidlna.conf, removed the existing media_dir line and added my own lines:

< media_dir=/opt
---
> media_dir=P,/usr/Winshare/Photos
> media_dir=P,/usr/500/Photos
> media_dir=V,/usr/500/Video
> media_dir=A,/usr/500/Music

As always, edit /etc/rc.conf to enable the service at boot:

minidlna_enable="YES"

and start the service:

service minidlna start

NFS

I use a Linux desktop, so I want to mount directories from the NAS box using the NFS protocol. I had troubles with connecting the Linux client to the FreeBSD server using NFSv4. So here is my current FreeBSD configuration. Firstly, /etc/rc.conf:

mountd_enable="YES"
mountd_flags="-r -n"
nfs_server_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-verbose"
nfsv4_server_enable="YES"
rpcbind_enable="YES"

My /etc/exports file looks like this:

/usr/Winshare /usr/500 -maproot=root 10.10.1.2 10.10.1.14

so I export two mount points to two clients and allow the root uid on the client to map to the root uid on the server.

On the Linux client (as root), I can manually mount the exported directories:

mount.nfs -v -o nfsvers=3 naspi:/usr/500 /usr/500
mount.nfs -v -o nfsvers=3 naspi:/usr/Winshare /usr/Winshare

Note that I'm forcing the client to use NFSv3. Once the Dell server goes away, on my Linux box I'll change my /etc/fstab to have this:

naspi:/usr/Winshare   /usr/Winshare   nfs     soft,noauto,nfsvers=3   0    0
naspi:/usr/500        /usr/500        nfs     soft,noauto,nfsvers=3   0    0

NFS and ZFS

OK, so apparently there is a separate exports file for ZFS filesystems. And as both my /usr/Winshare and /usr/500 are ZFS filesystems, here is what I had to do.

First, empty the existing /etc/exports:

echo -n > /etc/exports

Now, create this file /etc/zfs/exports:

/usr/Winshare -maproot=root 10.10.1.2 10.10.1.14
/usr/500 -maproot=root 10.10.1.2 10.10.1.14