Starting from v2.4.5 Resilio Connect product is high availability ready. This articles outlines how to deploy Resilio Connect MC and configure jobs to ensure high availability (HA) of your whole setup.
Setting up Management Console Availability watchdog includes the following steps:
Install 2 Management Consoles; consider one to be primary and the other to be secondary. Secondary MC should NOT be running by default.
Set up a DNS CNAME for your MC, ensure your local DNS server points to primary MC address by default.
Ensure you always put a DNS name of MC into Agents configuration file
Edit primary MC's configuration file fields
- point backup location to either shared network drive / SAN OR*
- point backup location to secondary MC over any preferred network protocol*
- set backup frequency to 1 hour using provided Cron syntax.
Setup crossover procedure, as described below.
Management Console crossover procedure:
1. Install python3 and pip3 (scripts are tested using python3.6, so python3 should point to python3.6) and requests, simplejson libraries:
sudo apt-get install python3 python3-pip
sudo python3 -m pip install requests simplejson
Download and install python3.6 and python3-pip from official site. Install requests and simplejson libraries:
python3 -m pip install requests simplejson
2. Download and unpack archive with scripts on server with secondary MC. On Windows unarchive them into same directory where secondary MC is installed (where
srvctrl.cmd are located).
config.json as necessary:
3. Place code to be executed into *.sh files if you're on Linux or replace those files with cmd scripts if you're on Windows. Be sure to adjust the corresponding parameters in config:
Pre script gets executed when failover logic is triggered but before launching the secondary management console; post script gets executed after attempt to launch the secondary management console. Notificator script is executed to notify administrator about important events. It should contain a custom code that accepts message as the first argument and sends this message somewhere.
4. Navigate to the directory where scripts are located and launch the script:
python3 main.py --config /path/to/config.json --logging debug
For example, if
config.json is in the same folder as
python3 main.py --config config.json --logging debug
This is the main executable script and monitors primary MC and if necessary launches secondary. Logging (--logging) prints events to stdout. It's not compulsory to enable debug logging, but highly advisable.
5. Add this script to system startup.
The script will be polling primary server at intervals as
REQUESTER_LOOP_INTERVAL states. If valid json is not returned, script jumps to failover part and starts secondary MC. After launching secondary server, script starts requesting secondary server status and if secondary server doesn't respond - script triggers notificator.sh to notify admin about it and exits.
- Install additional Tracker Server on machine other than primary MC
- Set next Default Agent Profile parameters:
- Custom trackers - to tracker server which runs on primary MC and your spare tracker server.
- Custom tracker mode - High Availability
Agents & Jobs HA
While deploying agents, ensure to deliver a sync.conf file which contains DNS name of your MC, not IP address.
Each Job's HA depends on a type of job you want to make highly available.
HA available only if you have a limited (1-3) number of RW agents. As a HA measure just set up one more RW agent on a separate machine.
Set up one more agent as "Destination" on separate machine. Please note, that destination agents MUST NOT save data from sources to the same physical location (i.e. data cannot be saved to same network or SAN location).
Set up one more agent as "Destination" on separate machine. Ensure, it has fastest connection possible to the source and has data pre-seeded. Alternatively, reserve destination machine may point to the same data location as source does (i.e. same network / SAN location).
Note, that in either case reserve agent will still need some time to receive metadata from source. The exact time depends on data size, CPU power and storage speed. Once reserve agent got the metadata, it'll keep seeding data to all other destinations even if source dies.