Articles in this section

Best practices for synchronizing large files which change very frequently

One of the use cases to illustrate the solution is managing user profiles in VDI environment. 

The problem that Resilio Connect solves is fast and reliable delivery of user profiles to a remote computer for the user to be able to login shortly after. 

Basic architecture involves centralized storage, usually a fast RAID, which keeps the user Profiles and from which the Profiles are delivered to remote computers. Depending on the chosen topology, those might be multiple computers, but usually, these are a few load-balanced servers from which users reach out for files via SMB. 

Challenges that Resilio Connect faces with user profiles are the following:

1) they are usually very large;
2) they change frequently as users stay logged in;
3) they remain locked (while in use) for Resilio Connect Agent, or the Agent lacks access permissions; 
4) Agents need to work with several such files at a moment.

While Resilio Connect successfully deals with each of these, they may produce some synchronization problems and delays when combined together.

The magic begins right after the user logs out and the updated Profile needs to be made available for further login asap. Resilio Connect Agent is required to check for file updates and discover which of the blocks of files changed in order to deliver only these ones (and cut down on network traffic).  Current implementation of the Agent is such that it will be attempting to do it as soon as it notices that the file is changed, but will surely fail as the file is still in use. 

Below are the suggested configuration of Agents settings that will help to eliminate or at least improve the narrow places mentioned above. Add these parameters to the Agent's Profile (don't confuse with users' Profile in VDI) and restart the Agents. 

Starting with version 2.12.5, the parameters from below are available through Profile preset for VDI, but editing FileDelayConfig is still explicitly required (see below).
VDI.png

 

1) Custom parameter recheck_locked_files_interval, value is in seconds. 
It is the frequency at which the Agent will come back and see if the file is still locked (in cases when it's not notified by the system). Default value is 600. Put there 60 (=1 minute) if it fits your use case. Don't put a too frequent interval as the Agent will be only busy with rechecking the file. Don't put a too rare interval either as some delays in synchronization will be observed. 

 

2) Custom parameter torrent_rolling_checksum, can be true or false
Put it as false. The Agent won't use the rolling checksum algorithm in order to find changed blocks. The Agent will still transfer only changed pieces, but only if these didn't shift! Since blocks of VDI files are pretty large, chances of block shifts are minimal.

 

3) Disable differential sync, can be Yes or No.
Set it as Yes for slow storage disks. In this case, the Agent will prefer to re-transfer the file instead of calculating the changed pieces. 
Set it as No for fast disks so that the Agent rechecks the file and transfers only changed pieces. This saves network.

 

4) Custom parameter download_priority, can be 0, 1 or 2 
Default value is 0. Value 1 means "newer first", value 2 means "older first", zero means 'no specific order".

For this parameter to work, file system notifications must be working. Note, that cloud storages and some of Linux-based systems and storages don't support file system notifications. 
This is the sorting of files in active upload\download queue for the receiving Agent. Please note, it's not prioritization of files across the whole set of data! This parameter only applies to the active queue of files. For example, it's known that AgentA needs to transfer 5 files to AgentB, file1 is the latest. We may assume that AgentA is still rechecking file1, so we may put value 2 for Agent B to start receiving the oldest file first which is most likely is already checked by source agent. 

 

Additionally to those, manually do the following adjustments (not included in the VDI preset). 

 

5) Reduce file size for versioning in Archive

By default files of size smaller than 1000 Mb keep previous versions in Archive. the size threshold is controlled by Profile parameter Max archive file size. If files change frequently, a lot of file versions will be kept in the Archive, which will take a lot of storage space.
Reduce the value to 100 (MB). With this, file versions of bigger files won't be stored in Archive.

 

6) Edit local FileDelayConfig to add VDI extensions

On every Agent that may introduce changes to the large files, edit its FileDelayConfig file located in the storage folder. Add primary VDI extensions there or edit their values if they are already present: set a 300-second delay for processing these files (for example, "*.vhdx":300). Be sure to preserve json format of the file. This helps the Agent to access frequently modified files less often.

Case sensitive

Please keep in mind that FileDelayConfig is case sensitive. Therefore, if you get some vhdx files with capitalized extension, add one more filter: "*.VHDX":300

With these, it's expected that the Agents will perform minimum disk operations and won't stuck on continuous attempt to synchronize the files which apparently cannot be synchronized at the given moment.
Seen metric is 3-3,3 minutes to deliver 2Gb of changes in a 10Gb file, from the moment user logged out (=user's Profile changed) to the moment Profile is fully available on 16 other computers. 

Containerized profile best practices with Resilio

  • Use Dynamic Profiles
  • Profile Mode 1 (Read / Write) this will help overcome locked file complications.
  • Do not use an office container unless the use case explicitly needs it
  • Cloud cache intermittently accesses the files and can cause conflicts it is not recommended to use Resilio and cloud cache at the same time without careful consideration. 
  • If using cloud cache along with Resilio exclude the lock and meta tags. Cloud cache replicates these on its own.
  • Set wait timers for VHD(X) in Agents Profile
    •   "file_delay": "{\"*.vdi\": 300, \"*.VDI\": 300, \"*.vhd*\": 300, \"*.VHD*\": 300, \"*.vbox\": 300, \"*.vmdk\": 300, \"*.VMDK\": 300}"
  • Paste the following lines into "File and folder filter" parameter in the job profile: 
    Pending
    *.vhd.lock
    *.VHD.LOCK
    *.vhdx.lock
    *.VHDX.LOCK
    *RW.VHDX
    *rw.vhdx
    *RW.VHD
    *rw.vhd
    *._ODFC.VHDX
    *._odfc.vhdx
    *._ODFC.VHD
    *._odfc.vhd
  • If synchronizing FSLogix log files, set file delay to 1 minute

Helpful Tweaks to keep profiles small

 

 

Related articles