Best practices: optimizing agent for large amount of small files delivery

By default Resilio Agent is optimized for a mixed dataset delivery: few millions of files with average size of ~5 megabytes.

This guide provides strategies to efficiently manage and optimize use cases involving millions of small files, each averaging just a few kilobytes. Here are optimization steps suggested:

1. Update Agent Profile:

• set Number of disk I/O threads to 12 (default value is 4)
  This will help Agent to access more files at the same time
• add custom parameter fs_prefer_sequential_indexing with valuefalse (default true)
  This allows Agent to process multiple small files in parallel without waiting previous files indexing to complete.
• set custom parameter peer_max_active_downloads to 2000 (default 500)
  This allows Agent to increase safety net of parallel files delivery from 500 to 2000 per connection to another Agent.
• set custom parameter max_active_downloads to 2000 (default 1000).
  This increases global Agent-scale limit of simultaneous files delivery
• [Only if cloud storage is a source or destination] set custom parameter net.http.threads to 4 ... 8 (default 0 which stands for reusing existing TCP threads).
  This allows agent to use multiple independent http threads when pushing data to the cloud.
• [Only for Windows machines] set Asynchronous disk I/O to Yes (default No).
  This allows OS to queue disk I/O requests on a very low level, resulting in faster processing and spending less I/O operations on every file.

2. Restart the Agents that have this updated profile assigned.