Looking at the many of out assumptions in our code, it is pretty clear that synchronization on your network is crusial.
To make it as easy as possible for administrators to manage their entire network, we want to have one specific plugin that allows for proper synchronization. Here what we need is the same time on all clocks of all the computers in the network.
This is particularly important when dealing with permissions since removing a permission needs to happen, no matter what. This could fail, though, if the clocks on various computers on your network are not properly synchronized.
Note that at this point this is important only computers running snapserver and/or snapbackend. The database itself and snap.cgi do not need to be synchronized that tightly because they are not directly dealing with the timestamps saved in the database.
We have two main solutions at our disposition:
I am thinking that snap_communicator can be used with this plugin to setup either synchronization method on all the computers relative to each others to a very small amount such as 1µs.
See the attachment as well, which shows the heavy variance Intel processors have (CISC processors have that problem) and thus a good reason for the impossibility to setup a clock at an extremely high precision.
If you decide to use NTP and want to have good accuracy, I suggest you setup several servers as usual, that is with a list of external servers.
For example, there is how one can setup a server:
restrict 0.north-america.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery server 0.north-america.pool.ntp.org
Further, the few servers that have access to external NTP servers want to allow access from local servers to get time. You want to adjust the IP and mask to your need. This one is very unrestrictive:
restrict 10.0.0.0 mask 255.0.0.0 nomodify notrap
The number of computers that offer the NTP service to your cluster depends on the total number of computers you are running with. With 5 or less, you probably want 2 masters. As you increase the total number of computer, so should the total of masters.
Now, to setup the other computers, instead of using external servers in the server statements, use your NTP masters:
server 10.0.0.1 server 10.0.0.2 ...
Then make a note somewhere to make sure that you are not going to ever shutdown all your NTP masters all at once.
I upgraded one of my computers to 16.04 and did not notice right away, but on a reboot ntpd was not running. The logs were not clear at all. After some research, I found our that the ntpdate tool was still running at the time the ntpd daemon was being started. Both cannot run simultaneously. Purging the ntpdate tool fixes the problem. Assuming you have an ntpd properly setup and your hardware clock works sufficiently well (i.e. battery is not dead) then you should have no need for ntpdate anyway.