Network Synchronization [Core]

Looking at the many of out assumptions in our code, it is pretty clear that synchronization on your network is crusial.

To make it as easy as possible for administrators to manage their entire network, we want to have one specific plugin that allows for proper synchronization. Here what we need is the same time on all clocks of all the computers in the network.

This is particularly important when dealing with permissions since removing a permission needs to happen, no matter what. This could fail, though, if the clocks on various computers on your network are not properly synchronized.

Note that at this point this is important only computers running snapserver and/or snapbackend. The database itself and snap.cgi do not need to be synchronized that tightly because they are not directly dealing with the timestamps saved in the database.

We have two main solutions at our disposition:

  • NTP, well known and used (since 1985) in many places, although that may not give us a very good synchronization in the sense that the computers may easily be synchronized to only 1ms (local networks only, over the Internet it can be as bad as 100ms); however, in Snap! we deal with timestamp with a precision of 1µs.
  • PTP, which is a more recent (2002, v2 in 2008) protocol allowing a local network to have computers synchronized to less than 1µs. (the accuracy is expected to be in the nanoseconds.)
  • Time tracking PCI card (using a feature such as GPS time)

I am thinking that snap_communicator can be used with this plugin to setup either synchronization method on all the computers relative to each others to a very small amount such as 1µs.

Sources: Time synchronization in an heterogenous environment

How accurate can we get computer clocks synchronized?

Synchronizing Clocks in a Cassandra Cluster with Solution

See the attachment as well, which shows the heavy variance Intel processors have (CISC processors have that problem) and thus a good reason for the impossibility to setup a clock at an extremely high precision.

Setting up NTP using the WAN for a few masters and the LAN for the rest

If you decide to use NTP and want to have good accuracy, I suggest you setup several servers as usual, that is with a list of external servers.

For example, there is how one can setup a server:

restrict 0.north-america.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
server 0.north-america.pool.ntp.org

Further, the few servers that have access to external NTP servers want to allow access from local servers to get time. You want to adjust the IP and mask to your need. This one is very unrestrictive:

restrict 10.0.0.0 mask 255.0.0.0 nomodify notrap

The number of computers that offer the NTP service to your cluster depends on the total number of computers you are running with. With 5 or less, you probably want 2 masters. As you increase the total number of computer, so should the total of masters.

Now, to setup the other computers, instead of using external servers in the server statements, use your NTP masters:

server 10.0.0.1
server 10.0.0.2
...

Then make a note somewhere to make sure that you are not going to ever shutdown all your NTP masters all at once.

NTP daemon and ntpdate bug in 16.04

I upgraded one of my computers to 16.04 and did not notice right away, but on a reboot ntpd was not running. The logs were not clear at all. After some research, I found our that the ntpdate tool was still running at the time the ntpd daemon was being started. Both cannot run simultaneously. Purging the ntpdate tool fixes the problem. Assuming you have an ntpd properly setup and your hardware clock works sufficiently well (i.e. battery is not dead) then you should have no need for ntpdate anyway.

Reference: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1577596

AttachmentSize
ia-32-ia-64-benchmark-code-execution-paper.pdf384.41 KB

Snap! Websites
An Open Source CMS System in C++

Contact Us Directly