Deleting files from source after transfer in distribution jobs – Resilio Active Everywhere

Available in version: 4.1
Supported for: Distribution and Consolidation jobs. Local, network and cloud storages.

When files are cleared from source.
What files are cleared and not cleared from source
Where the files are deleted from source and how they are stored there
Peculiarities and considerations

Option to enable clearing the files from storage on source is available from SETTINGS tab in Distribution and Consolidation jobs.
If enabled, the source Agent(s) will clear the local files (move to Archive or skip the Archive) after the file transfer is complete and all job triggers are executed.

Avoid dataloss

If the option "Delete files to archive" is disabled, files are deleted from the system skipping system bin. Files can be restored only from the destination Agents or a third-party backup.

Below outlined are specifics of the functionality.

When files are cleared from source

Clearing the files from source is the final action done by the Agents in the job.
Source Agent will clear the files only after it verifies with all the destination Agents that they have a copy and after all job triggers are executed. An offline destination Agent blocks this operation, keeping the job in progress until aborted or timed out.
This also means that files are not cleared from source if the job is aborted before it successfully finishes.

Two new statuses can be reported by the source Agent: "resolving peers availability before cleanup" and "cleaning up source folder".

What files are cleared and not cleared from source

All files that were transferred to destination Agents, including the pre-seeded files (that were not transferred to destination because there is already a copy of the file).

These files can be seen in Job run -> Agents -> <source_agent> -> Files events, by status "Archived".

Files are not cleared from source in the following cases:
- not confirmed to be synced to destination, e.g. skipped because of an error or are in IgnoreList;
- not all destinations confirmed they received the files, e.g. a destination Agent is offline.
- files were removed from the destination Agent(s) in a trigger.
- this is a large dataset and/or slow storage, and job run was aborted during the clearing process, manually, by error or parameter "Wait for job run to complete before starting new one" in job scheduler.

Where the files are deleted from source and how they are stored there

By default, files are deleted to Archive, unless the option is unchecked manually by the Admin. Archive is located inside the job folder on the source in the hidden .sync directory. It is created even if the source folder was empty.
Agent performs a move operation.

This is not the Archive which is created in Synchronization, File Cache and Hybrid jobs, and on destination Agents in Distribution and Consolidation jobs, which is designed to store file version and deleted files. It is not managed by the "Use archive" and "Max archive file age" parameters in Agent and Job profiles.

Archive name includes the shareID (internal ID calculated by Agent and not visible to user or admin) and runID (this the ID of the job run as seen in Job Runs table, column ID on the MC).

Files are stored in the Archive per job run during the number of days configured in the job.
Files will be cleared from the archive skipping the system bin after that time.

Storing the files in the Archive is not dependent on the existence of the job itself on the Management Console. Even if the job is already deleted, the Archive will remain on the storage and files will be deleted from in accordance with the setting in the job run.

How to check when the files will be deleted from the Archive

If the job or job run is already deleted from the MC, there's no way to do that. Rely on the Agent's mechanism to delete the files rom Archive when the time comes. If this is an emergency question, please contact support.

For still existing job run:
- in the storage on source agent, spot the Archive folder of interest and its run.ID. In the image above, let it be Archive with run.99.
- in the Management Console, open job runs and in column ID find the corresponding job run. Open it and check tab settings.
- the files will be deleted in 5 days.

Peculiarities and considerations

1. Storage space requirements. Ensure sufficient storage on source Agents for multiple copies of source data or reduce Archive storage duration.

2. In scenarios when the directory on the Agent is included in Synchronization job and a source in Distribution/Consolidation job:

- do not use transfer_job_remove_prefs_folder in the Agent or Job profile. Otherwise, it will delete the whole .sync service folder, including all the underlying Archives.

- clearing files from source within Distribution/Consolidation job also affects the files in the Synchronization job and propagates this deletion to the other Agents.