Each Agent needs to detect file changes / updates before it starts the procedure of file indexing and delivery. This only applies to Synchronization jobs: Distribution and Consolidation jobs only scan files during start while Script job does not operate files at all.
The Agent uses following techniques of file changes detection:
- Filesystem Notifications
The fastest way for the agent to detect changes inside the synchronized folder. When FS notifications work properly the agent starts processing file immediately after the update. Although FS notifications have a number of limitations.
- Scheduled folder scan
It starts immediately after job creation or adding the agent to the job. After that it runs every 24 hours (can be adjusted via Agent Profile). This is a very reliable way though it is slow and costs some disk performance.
- On-demand folder rescan
Can be triggered with "Rescan" button (Job runs -> [Name of run] -> Agents -> [Agent] -> Activity). Can also be triggered with Console API call. Works the same way as scheduled scan.
After the agent detects a new or updated file, it records a file event "Updated" in Management Console. This event means that the agent detected file update.
Agent will needs to recheck the updated file and discover what was changed. Once rechecked and all agents know that there's a newer file in the mesh, the file is added to download queue on other agents.
Filesystem Notifications limitations
- OS may not send FS notifications for files and folders with paths length more than 260 characters
- If a watched folder resides on an SMB/CIFS share and either client or server does not support SMBv3 or newer (introduced in Windows 8 and newer), FS notifications will not work
- OS may skip some notifications for bulk file updates
- If total amount of folders observed is greater than 8192 (all apps, not only Agent) - the agent will only monitor a part of folders. This value can be increased up to 2BLN of folders though popular distros like Debian or Redhat in practice allow to monitor up to 524288 folders.
- NFS-mounted shares do not support FS notifications
BSD-based systems (not including OS X)
- File system notifications are not supported on FreeBSD, FreeNAS, TrueNAS and other systems forked from FreeBSD
Mixed environments limitations and known issues
- Windows machines do not receive FS notifications from Linux-hosted shares mounted over NFS
- Windows machines can receive FS notifications from Linux-hosted shares mounted over SMB if:
- Windows machine supports SMBv3
- Linux machine supports Samba v4.1 (which is the first version supporting SMBv3 for Linux)
- Linux machine has enough inotify watches to actually know of file changes itself
- Windows machines can receive FS notifications from FreeNAS-hosted ZFS shares mounted over SMB though in practice FS notifications may not work sometimes for no reason.
Windows machines can receive FS notifications from Synology-hosted shares mounted over SMB though same limits as in point 2 apply.