Resilio Connect agent requires in average 2GB or RAM per million of files in sync job and even more when NTFS permissions synchronization is turned on.
If you see that you have a lack of RAM but it was not used by Connect Agent (check in Task manager), it makes sense to check filesystem buffers. You can check which RAM is used for with RAMMap tool by Sysinternals.
If most of the RAM is used by FS buffers, "Unbuffered disk IO" parameter might help. You can set it from agent profile (agent restart is required). It should be set to "Yes" (though it is going to cost performance + some extra disk usage).
If syncing on ReFS volume, check the following Microsoft instructions on fixing heavy RAM usage on ReFS drives:
As a quick fix, you can flush memory used by filesystem buffers from RAMMap "Empty -> Working set"