Configuring Linux OS cache servers

Starting with Resilio Active Everywhere platform v4.0.0, Linux OS (x64) servers can be used as cache servers. Linux cache server utilizes FUSE based solution to provide access to files to 3rd party applications that are not physically present on cache server. Together with Ganesha NFS server it allows to access this virtual file system over NFS v3 or v4. Linux cache server is used in in a File cache or Hybrid work jobs.

Resilio Agent installation
Third party packages installation
Configuring Ganesha NFS server
Configuring NFS Clients accessing Ganesha NFS exports
Configuring File caching/Hybrid work jobs
Applying any job changes or restarting the Gateway Agent

Resilio Agent installation

Supported for: Linux x64.
Not supported: docker installations for Gateway.

Root is mandatory

Run the Resilio Agent as root. When installing via packages - take care to update your .service files to run agent as root

Install Resilio Agent via package or tarball unpacking.

Third party packages installation

UbuntuRocky LinuxOracle Linux 9Oracle Linux 8CentosAmazon Linux 2023

You need to ensure following packages are installed: fuse3, libfuse2, lsof, nfs-ganesha-5, nfs-ganesha-vfs, software-properties-common

Follow these commands to connect necessary repositiries deploy all of the above. All commands must be started as root:

apt install -y software-properties-common
apt update && sudo apt-get upgrade -y
apt install -y fuse3 libfuse2 nfs-ganesha nfs-ganesha-vfs

ensure /usr/bin/lsof is available either as binary or as a symbolic link

Following packages are required:
libnfsidmap libntirpc libprometheus-cpp libtirpc libwbclient nfs-ganesha nfs-ganesha-vfs nfs-utils rpcbind

Follow these commands to connect necessary repositiries deploy all of the above. All commands must be started as root:

setenforce 0
dnf install centos-release-nfs-ganesha5
dnf install fuse3
dnf download libnfsidmap.x86_64 libntirpc.x86_64 libprometheus-cpp.x86_64 libtirpc.x86_64 libwbclient.x86_64 nfs-ganesha.x86_64 nfs-ganesha-vfs.x86_64 nfs-utils.x86_64 rpcbind.x86_64
rpm -i --nodeps libnfsidmap.x86_64
rpm -i --nodeps libntirpc.x86_64
rpm -i --nodeps libprometheus-cpp.x86_64
rpm -i --nodeps libtirpc.x86_64
rpm -i --nodeps libwbclient.x86_64
rpm -i --nodeps nfs-ganesha.x86_64
rpm -i --nodeps nfs-ganesha-vfs.x86_64
rpm -i --nodeps nfs-utils.x86_64
rpm -i --nodeps rpcbind.x86_64

If any of packages informs it's installed already - ignore the message.

Following packages are required:
libnfsidmap libntirpc libprometheus-cpp libtirpc libwbclient nfs-ganesha nfs-ganesha-vfs nfs-utils rpcbind

Follow these commands to connect necessary repositiries deploy all of the above. All commands must be started as root:

setenforce 0
wget https://internal.resilio.com/install/centos-release-nfs-ganesha5.rpm
rpm -i --nodeps centos-release-nfs-ganesha5.rpm
dnf download libnfsidmap.x86_64 libntirpc.x86_64 libprometheus-cpp.x86_64 libtirpc.x86_64 libwbclient.x86_64 nfs-ganesha.x86_64 nfs-ganesha-vfs.x86_64 nfs-utils.x86_64 rpcbind.x86_64
rpm -i --nodeps libnfsidmap.x86_64
rpm -i --nodeps libntirpc.x86_64
rpm -i --nodeps libprometheus-cpp.x86_64
rpm -i --nodeps libtirpc.x86_64
rpm -i --nodeps libwbclient.x86_64
rpm -i --nodeps nfs-ganesha.x86_64
rpm -i --nodeps nfs-ganesha-vfs.x86_64
rpm -i --nodeps nfs-utils.x86_64
rpm -i --nodeps rpcbind.x86_64

If any of packages informs it's installed already - ignore the message.

Old Ganesha NFS

Oracle Linux 8 only allows installation of Ganesha NFS v3.5. For a newer version of Ganesha NFS ensure you are runnning Oracle Linux 9+

Following packages are required:
fuse lsof glusterfs-server oracle-gluster-release-el8 nfs-ganesha-vfs

Follow these commands to connect necessary repositiries deploy all of the above. All commands must be started as root:

setenforce 0		
dnf install -y lsof fuse3
dnf install -y oracle-gluster-release-el8
dnf config-manager --enable ol8_addons ol8_UEKR6 ol8_appstream
dnf install -y glusterfs-server nfs-ganesha-vfs
setsebool -P ganesha_use_fusefs 1

Following packages are required:
fuse lsof glusterfs-server oracle-gluster-release-el8 nfs-ganesha-vfs

Follow these commands to connect necessary repositiries deploy all of the above. All commands must be started as root:

setenforce 0
dnf install -y centos-release-nfs-ganesha5
dnf install -y libnfsidmap libntirpc libprometheus-cpp libtirpc libwbclient nfs-ganesha nfs-ganesha-vfs nfs-utils rpcbind

You need to ensure following packages are installed: rpcbind, nfs-utils, libtirpc, libwbclient, libnfsidmap, fuse, fuse-libs, nfs-ganesha, nfs-ganesha-vfs, libntirpc, libprometheus-cpp

Follow these commands to connect necessary repositiries deploy all of the above. All commands must be started as root:

dnf install rpcbind nfs-utils libtirpc libwbclient libnfsidmap fuse3 fuse-libs
wget https://internal.resilio.com/install/centos-release-nfs-ganesha5.rpm
rpm -i --nodeps centos-release-nfs-ganesha5.rpm
mkdir packages
cd packages
dnf download nfs-ganesha nfs-ganesha-vfs libntirpc libprometheus-cpp
rpm -i --nodeps -v *.rpm

No standard NFS daemon

Exposing Resilio virtual TSS folder is not supported with standard NFS daemon. Please use Ganesha NFS v5 or newer instead.

Configuring Ganesha NFS server

Make export directory writable for all
```
chmod 0777 /mnt/ganesha-test
```
Ensure nfs-ganesha server is stopped, then edit the file /etc/ganesha/ganesha.conf:
- Use example config file for Ganesha as a template
```
NFSv4 {
  Only_Numeric_Owners = true;
      }
EXPORT
  {
  Export_Id = 12345;
  Path = /mnt/ganesha-test;
  Pseudo = /mnt/ganesha-test;
  Protocols = 3,4;
  Access_Type = RW;
  Squash = none;
  FSAL {
         Name = VFS;
       }
  }        
```
- Configuration file supports several EXPORTs. For each new unique job location a new EXPORT must be added to the config. Export_Id, Path and Pseudo must be different for every export.
- The folder that represents Resilio job does not have to be a root of exported folder, it may be a part of subtree.
Start nfs-ganesha to apply new configuration
systemctl restart nfs-ganesha

NFS server delay

By default both NFS server and NFS clients cache FS objects metadata, so NFS users will not see new files until those caches are up-to-date. Having shorter TTL for cache entries can help to see new files sooner, but it affects overall performance. Default Ganesha NFS rescan interval is 60s, it can be configured in /etc/ganesha/ganesha.conf, for example 5s :

EXPORT
  {
  ...
  Attr_Expiration_Time = 5;
  ...
  }

NFS client cache entry TTL can be set in mount options, for example 5s: actimeo=5

Configuring NFS clients accessing Ganesha NFS exports

Mount exported NFS folder from another Linux host

mount -v -o async,nconnect=16,noatime <server>:/mnt/ganesha-test /mnt/ganesha-test

Ensure read and write operations work normally and cause files to hydrate in Resilio virtual folder (number of local files in Management Console will grow every time you open a new file)
Mount exported NFS folder on several Linux hosts and make sure agent does not stuck under load

Configuring File caching/Hybrid work job

Linux cache server can be added to the job only after the Agent and 3rd party packages were installed and properly configured. Two locations must be selected for Caching Gateway agent.
Note, the selected paths cannot be changed later after the job is saved! See here for more details.

Linux path: it is the "Path" matching Ganesha config file above, or a directory inside it. Is the file access location which is made available over NFS remotely. Files are virtually represented by this location for further accessing.
This path must exist and be empty. Linux cache: the actual file download location, not available for file access.

Syncing home folders

For the synchronization of Linux users’ home folders it is recommended to add the following lines to File and folder filter in Job profile.

.Xauthority
.Xauthority*

Applying any job changes or restarting the Gateway Agent

Should you need to apply any changes in the job or restart Resilio Agent or its host, please follow the pattern below:

Stop any application / daemon using the NFS share exported by Ganesha
(optional, just speeds things up) Unmount exported shares
Stop nfs-ganesha daemon
Stop Resilio Agent / restart host

Start all components in the opposite order:

Start Resilio Agent first
Start nfs-ganesha daemon
Ensure exported servers are mounted on Linux machines that will consume the data
Start your applications that will work with the data on Ganesha exports