27th June, 2019
At present I've got an old Dell PC acting as my NAS box. I have two 6T HDD drives set up as a ZFS mirror pool. This is running Ubuntu server fine.
I know that the system is chewing a not insignificant amount of power running 24/7. It also has a two drive limit, but I want to be able to have up to four drives attached. This will allow me to:
The Raspberry Pi 4 with 4G of RAM has just been announced. It has a proper 1G Ethernet port and doesn't use much power. It also has a USB 3 port. So I've decided to buy a RPi4 and one of these Orico 6648US3-C 4-bay SATA to USB 3 docking stations. This will connect to one of the RPi4's USB 3 ports and make a NAS box. I can install my existing two HDDs and have two bays spare.
While I wait for the RPi4 and the Orico 6648US3-C to arrive, I've dug out my RPi3 so that I can get ZFS up and running on it.
To cut a long story short, I tried Raspian and couldn't get the ZFS modules to compile and load. Then I tried Ubuntu for the RPi3 and got the same result. Then I remembered that FreeBSD runs on the Raspberry Pi. This has native support for ZFS and I used to use FreeBSD a lot.
So, let's give FreeBSD a whirl and see if it will meet my needs. Here's how it has gone so far. I'll keep updating the blog entry as I get things working.
I went to the Download FreeBSD page and got the FreeBSD 12.0 SD card image for the RPI3.
Back on my desktop box, I did these commands to decompress the file and burn it to my SD card:
unxz FreeBSD-12.0-RELEASE-arm64-aarch64-RPI3.img.xz
sudo ddrescue -D --force FreeBSD-12.0-RELEASE-arm64-aarch64-RPI3.img /dev/sdc
I put the SD card into my RPi3 and booted it up. There's no GUI, just a console screen. I logged in as root and password root.
I used adduser to create a user called wkt. I used passwd to set
root's password. I edited /etc/group and put wkt into the wheel
group; this allows wkt to run the su command and become root:
wheel:*:0:root,freebsd,wkt
I set a static IP address for the Pi by adding these lines to /etc/rc.conf:
ifconfig_ue0="inet 10.10.1.90 netmask 255.255.255.0"
defaultrouter="10.10.1.1"
With this done, I can set up the ue0 interface:
/etc/rc.d/netif restart ue0
To set up the DNS server, I created the /etc/resolv.conf file with the
details of the servers and the domain to use for names with no dots:
search local.net
nameserver 10.10.1.1
nameserver 8.8.8.8
At this point I can remove the USB keyboard and HDMI monitor, remotely
login with ssh as user wkt and use the su command to get to a root shell.
I'm going to run several services on this device, so I installed these packages by running these commands as root:
pkg install apache24 bash minidlna proftpd rsnapshot rsync samba48 sudo vtun
At this point I haven't done any configuration for these packages.
I'm used to running sudo on Linux, so I've installed the sudo package.
In the file /usr/local/etc/sudoers, I removed the comment at the beginning
of this line so that wheel group members can sudo:
## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL) ALL
The Pi doesn't have a battery-backed clock, so I added these lines to
/etc/rc.conf to start the NTP daemon and to set the system clock at boot:
ntpd_enable="YES"
ntpdate_enable="YES" # Run ntpdate to sync time on boot
ntpdate_hosts="pool.ntp.org" # List of ntpdate(8) servers.
To set the system's timezone, I linked the relevant zone file thus:
ln -s /usr/share/zoneinfo/Australia/Brisbane /etc/localtime
I can't connect my existing ZFS mirror pool to the RPi3 as they are SATA drives; I wouldn't want to anyway, not until I know that things work.
Luckily, I have an external USB hard drive which is configured as a ZFS pool with the name TAFE2T. The nice thing about FreeBSD is that ZFS is already set up and ready to go. All I had to do was plug the USB drive into my existing RPi3 and do (as root):
zfs import TAFE2T
and this is what I see:
[root@naspi /etc]# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/rootfs 14G 3.0G 10G 23% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/msdosfs/MSDOSBOOT 50M 13M 37M 26% /boot/msdos
tmpfs 50M 4.0K 50M 0% /tmp
TAFE2T 1.7T 171G 1.6T 10% /TAFE2T
TAFE2T/Henry 1.6T 20G 1.6T 1% /TAFE2T/Henry
Now, how do I ensure that this gets auto-mounted at boot time?
This article says to do the zfs import so that the system knows the pool names,
and add this line to /etc/rc.conf:
zfs_enable="YES"
OK, so let's reboot the system and see what happens:
[root@naspi /etc]# reboot
Connection to naspi closed by remote host.
After waiting about a minute for the RPi3 to reboot, I log back in with ssh
and I see ... no ZFS pools. Hmm, I can do zfs mount -a and the pools
get mounted. So there is something going on that I haven't figured out yet.
It looks like ZFS starts before the USB drives are attached. So my solution
is to add this Bourne shell script to /etc/rc.local:
#!/bin/sh
# Try to mount the ZFS USB filesystems at boot time
done="no"
for i in 1 2 3 4 5 6 7 8 9 10
do if [ -c "/dev/da0" -a $done = "no" ]
then logger "Mounting the ZFS stuff"
zfs mount -a
done="yes"; break
else
logger "Waiting for /dev/da0"; sleep 30
fi
done
I probably don't need the done variable and can rely on the break to leave the loop.
When I read the output of dmesg. I see these ZFS warnings:
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
Consider tuning vm.kmem_size and vm.kmem_size_max
in /boot/loader.conf.
So I looked these up. The consensus is to leave prefetch disabled when
there is less than 4G of RAM. Hopefully when the RPi4 arrives (with 4G)
this will not be a problem. I've also tuned the kmem_size by adding these
lines to /boot/loader.conf:
# Tune the VMS settings for ZFS
vm.kmem_size="512M"
vm.kmem_size_max="512M"
I have a security camera which stores videos and images to a server using
the FTP protocol, so I've installed proftpd on the RPi3. In
/usr/local/etc/proftpd.conf I turned UseIPv6 off as I don't use it.
In /etc/rc.conf I added:
proftpd_enable="YES"
to enable the service at boot time. I started it manually by doing:
service proftpd start
I can ftp in as a user and see my home directory. Now I'll set up a
second user (with adduser) who has /usr/sbin/nologin as their shell
(so they can't ssh in) and with a home directory where the videos and
images will be stored.
Ah, I can't ftp in as the second user as proftpd doesn't know that
their shell is valid. So I edited /etc/shells and added /usr/sbin/nologin
as a valid shell.
Once I installed the apache24 package, there is a vanilla Apache 2.4
web service ready to go. The ServerRoot is /usr/local/www/apache24/data
and there is a user www and group www to own the files. To start it,
add the line
apache24_enable="YES"
to /etc/rc.conf and then run the command
service apache24 start
The NAS box is going to be used by myself and my wife, so I'll need to set
up SMB file sharing with Samba. To do this, I'm copying the existing
Samba config file from the existing Dell server. On FreeBSD, this is
the file /usr/local/etc/smb4.conf. Here is most of my file:
[global]
wins support = yes
workgroup = WKTHOME
server string = %h server (Samba, FreeBSD)
log file = /var/log/samba4/log.%m
log level = 3
max log size = 1000
hosts allow = 10.10.1., 10.10.2., 192.168.2., 127.0.0.1
security = user
; Allow HP printer to login with XP-style authentication
ntlm auth = yes
[Music]
comment = Music
path = /usr/500/Music
public = yes
writable = no
[Scan]
comment = Public Stuff
path = /usr/Winshare/Scan
public = yes
writeable = yes
printable = no
write list = @staff
Samba keeps its own username/password database, so now I need to set up a Samba user:
]# smbpasswd -a -U wkt
New SMB password:
Retype new SMB password:
Forcing Primary Group to 'Domain Users' for wkt
Forcing Primary Group to 'Domain Users' for wkt
Added user wkt.
As usual, we enable the service by adding a line to /etc/rc.conf:
samba_server_enable="YES"
and run a command to start the service:
service samba_server start
I keep a store of music and videos on my existing NAS box and serve these out to my Smart TV with minidlna. On the RPi3, I had to create the directories for my files:
mkdir -p /usr/Winshare/Photos /usr/500/Photos /usr/500/Video /usr/500/Music
chmod 755 /usr/Winshare/Photos /usr/500/Photos /usr/500/Video /usr/500/Music
Then I edited /usr/local/etc/minidlna.conf, removed the existing media_dir
line and added my own lines:
< media_dir=/opt
---
> media_dir=P,/usr/Winshare/Photos
> media_dir=P,/usr/500/Photos
> media_dir=V,/usr/500/Video
> media_dir=A,/usr/500/Music
As always, edit /etc/rc.conf to enable the service at boot:
minidlna_enable="YES"
and start the service:
service minidlna start
I use a Linux desktop, so I want to mount directories from the NAS box using
the NFS protocol. I had troubles with connecting the Linux client to the
FreeBSD server using NFSv4. So here is my current FreeBSD configuration.
Firstly, /etc/rc.conf:
mountd_enable="YES"
mountd_flags="-r -n"
nfs_server_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-verbose"
nfsv4_server_enable="YES"
rpcbind_enable="YES"
My /etc/exports file looks like this:
/usr/Winshare /usr/500 -maproot=root 10.10.1.2 10.10.1.14
so I export two mount points to two clients and allow the root uid on the client to map to the root uid on the server.
On the Linux client (as root), I can manually mount the exported directories:
mount.nfs -v -o nfsvers=3 naspi:/usr/500 /usr/500
mount.nfs -v -o nfsvers=3 naspi:/usr/Winshare /usr/Winshare
Note that I'm forcing the client to use NFSv3. Once the Dell server goes
away, on my Linux box I'll change my /etc/fstab to have this:
naspi:/usr/Winshare /usr/Winshare nfs soft,noauto,nfsvers=3 0 0
naspi:/usr/500 /usr/500 nfs soft,noauto,nfsvers=3 0 0
OK, so apparently there is a separate exports file for ZFS filesystems.
And as both my /usr/Winshare and /usr/500 are ZFS filesystems, here is
what I had to do.
First, empty the existing /etc/exports:
echo -n > /etc/exports
Now, create this file /etc/zfs/exports:
/usr/Winshare -maproot=root 10.10.1.2 10.10.1.14
/usr/500 -maproot=root 10.10.1.2 10.10.1.14