Installing and Upgrading Snap! Websites on a cluster

As we are working on Snap! Websites we run in various problems, one of which is the installation process and later the upgrade of a cluster of Snap! Websites servers.

Security Consideration

Snap! Websites are built to run in a cluster. Because of that, the security of the cluster is mainly insured by the computers in the cluster and not by the servers themselves.

There are a few known points that render the Snap! Websites service weak in terms of security if people other than you can access your computers via a network connection or directly:

  • Most of the tools (snapserver, snapbackend) accept UDP commands from anyone without any secret; your firewall should block access to the all UDP ports used by Snap! Websites
  • Cassandra is used without a log in and password; that would slow down things and when you get hundreds of hits every second, that adds up quickly
  • Cassandra data is not encrypted while transferred (nor in the database)1; although we tried to make it work with SSL, we did not yet succeed in that area, although we anyway prefer to tunnel all traffic between our local network systems via a VPN; again Cassandra ports like 9160 should be protected from the outside anyway
  • [not implemented yet] Writing to logs will make use of a snaplogger service so we can better handle our logs: protected logs will be accessible from any users in writing
  • snap.cgi can be run by anyone, not just Apache2 so someone with access to a server can simulate an Apache request

Note: these considerations do not include the security of accessing the servers via an HTTP port.

Server Installation

Installing the Snap! Websites packages is very easy. It is just as easy as any other packages in Ubuntu. However, the snapserver will not properly run until you properly configure it and enable it. You also want to only install what you want to run on each machine. With experience, you will learn what is really necessary on each one of your machines. Obviously, apt-get and other similar tools will make sure that dependencies all get installed.

For example:

apt-get install snapserver

DNS

Assuming you want to use a cluster of front end systems (we recommand at least 2 front end computers for a properly working cluster, read more about this in the upgrade section,) you want to install a name server that can do a form or another of round robin for your front end systems. That way the load gets shared between many front end systems.

At this point we do not offer such a feature, however, you could also look into forcing people who are logged in to use a set of IPs and people who are not logged (anonymous) in to use another. The concept here is simple: people who are not logged in should never have to wait. People who are logged in may have to wait a little longer for an answer, but since they are logged in you would expect them to wait for such. This can be achieved by forcing users who logged in to use a different sub-domain and that address ends up being different (i.e. "www.snapwebsites.org" and "edit.snapwebsites.org".)

Mail Server

The Snap! Websites sendmail backend makes use of the system sendmail tool. It has to be installed. Which may server you use is not important, but it has to be a sendmail tool that understands the standard way of sending emails with such. We use the Postfix version for which we also have the snapbounce package.

Note

The snapbounce package includes documentation on how to setup postfix to get bounce emails to your Cassandra cluster via snapbounce.

On the Mail Server computer you want to install the snapbounce, snapserver, and snapserver-core-plugins packages. Start at least the snapbackend with the sendmail action (using the snapinit, you will list sendmail as one of the servers) and snapbounce has to be on the computer which receives mail to work. It saves bounced emails in your database allowing you to see what works and what does not work.

apt-get install postfix snapserver snapbounce

Cassandra System

In most cases you will want to Install 3 or more Cassandra Systems that are backend computers (i.e. not accessible from the outside, except maybe from SSH). 3 is the minimum you will need to make a proper Cassandra cluster with duplication that works efficiently. Many more is of course better if you get millions of hits per day. Cassandra defines various ways for you to determine the number of nodes required. It includes statistics that one can use to see what is happening with your Cassandra cluster.

If you are not in control of the local network and thus others have access to your local network, you may want to create a VPN so all transactions between computers remain encrypted. This imposes a slowdown so if you can afford your own local network, all the better.

IMPORTANT NOTE

You should have computers with at least 4Gb of memory to be sure that your nodes do not crash once in a while. We tested with 1Gb and 2Gb and it tends to crash, meaning that you need a lot of maintenance to repair nodes that crashed... so to make it easier on yourselves, 4Gb or more. Cassandra will use 50% of the RAM and leave the other 50% for the OS to use as buffers.

Specialized Backend Servers

In order to get a system run as smoothly as possible, Snap! Websites is broken up in one front end which can be paralellized and a number of backends defined by various plugins. As mentioned earlier, we have one that can be used just to send (and ultimately receive) emails.

You may run all the backend servers on the same computer, although once the computer CPU and/or I/O gets used at 50% or more, we strongly suggest that you break up the backends on multiple computers. You cannot run the same backend more than once (we do not currently have a lock, so you must be very careful about that yourselves otherwise it is not unlikely to create problems.)

As noted above, the sendmail backend must run on a computer that has sendmail. It is not required for that computer to be the one with postfix configure to send mail to the outside. You could have a local postfix install that communicates with your mail server or have a sendmail system that knows how to do so.

Right now we offer the following backends:

sendmail
pagelist
images

The backends to run on a computer need to be listed in the snapserver.conf file.

Backend Server

Snap! Websites include a CRON like feature that runs various backend processes. These can be run once every 5 minutes or so. Depending on your load, whether you use a separate server, whether you have many Cassandra nodes, and some other parameters, the timing can be adjusted to lower the load on the server as a whole.

services=backend  (WARNING: we are working on this one right now, it still uses CRON at this point)

This backend server can run on its own server or with other backends. If you separate your front ends, you should not run this on a front end. Like with other backends, only one computer can run the main backend (CRON) process at a time. We do not yet offer a lock to avoid multiple simulataneous runs which could cause problems.

Frontend Servers

The Snap! Website frontend servers (snapserver) handle the incoming HTTP traffic and generate the outgoing HTTP traffic. These servers can be on your local network and not on the same computers as Apache and snapcgi. One reason to put these on separate computers is to allow for higher security.

Although future versions may directly answer the HTTP port, at this time we use Apache2 as the main HTTP server. This adds another layer making it possible to have snapserver's on separate computers and create proxies which snapserver would not offer.

sudo apt-get install snapserver

The server requires some settings to be defined in the /etc/snapwebsites/snapserver.conf file. At least, you should specify server in the list of items that snapinit should start.

services=server

HTTP Server

Right now, the snapcgi is an Apache2 module, it may work with other HTTP servers, but we only test and use it with Apache2. Since this is a CGI tool, you do not need any special handling, only to tell Apache to use snapinit to handle requests that arrive on specific domains (or all domains.)

The server with Apache2 also needs to have snapcgi which you can install by itself.

sudo apt-get install snapcgi

We offer a snap-apache2.conf file showing you how to define an entry in your Apache2 configuration to allow Apache to call snapcgi properly. snapcgi then connects to one of your Front End Snap! Servers (snapserver).

The Crux of the script are these three lines of code:

RewriteEngine on
RewriteRule snap\.cgi - [S=1]
RewriteRule ^(.*)$ /cgi-bin/snap.cgi?q=/$1 [L,PT,QSA,E=CLEAN_SNAP_URL:1]

The second line tells the server to ignore the request if it is snap.cgi directly.

The last line transforms standard HTTP requests in such a way that snap.cgi runs with it instead of the default Apache2 reaction to load a static page from your disk.

Installing a Website

With all the servers in place, you can start the installation of a website.

We will be looking into ways to simplify the process, but right now you have quite a few steps to follow. Please note that until a first website was initialized, none of the backends can be running.

Defining Domains and Websites in Cassandra

First, you want a direct connection between your Cassandra cluster and snapmanager. You can install snapmanager on your own computer as long as you can open a connection to your Cassandra cluster from there. Preferably, use a VPN or tunneled connection so it remains secure and don't forget to close that connection once done.

When you run snapmanager, you can first test that you can connect to your Cassandra cluster. Once that works (test button on the first page) then you can click on Domains tab. There you can add domains.

Once you have one or more domains, you can add websites for these domains. The snapserver needs both, domains and websites, in order to figure out how to handle such to give you access to what will look like a website.

Detailed help for snapmanager is available in our help section.

First Website Installation

We currently run in many concurrency problems with attempting to create a website with everything turned on at the same time2. So... for the very first website installation you want to follow the following steps quite strictly:

  1. Make sure only the Cassandra cluster and one snapserver are running
  2. Make sure you can connect to that one snapserver using snapmanager
  3. In snapmanager, select Tools » Initialize a Website
  4. Enter your website URL, if there is already a URL, make sure it is from the website you want to install
  5. Enter a port, in most cases 80 or 443
  6. Click on the Send Request button

Then wait for the process to finish. You should not see any errors appearing in the Status console. The first few entries are about snapmanager connecting to the snapserver service. This is a direct connect (i.e. without Apache2 or snapcgi). If that part does not work, check your firewalls. By default the server uses port 4004, however, you may have changed the port. Also if you use a tunnel or a VPN you may be using a different IP and/or port.

First Test of Live Site

Once the First Website Installation succeeds, you are ready to test the website live using your browser. Now you want to make sure Apache2 runs and you can start all the backends.

Finally, in your browser you should be able to just access your website home page. The default theme shows a couple of sentences on the home page and a log in screen on the left side of the screen. The log in screen includes a Register link. Using that link you can create an account on your new website. Note that accounts on Snap! Websites clusters are common to all the websites, so once you created one account, you can access all the websites you own, comment on all websites that allow a user to comment, etc.

To further configure your new website, you probably want to make yourself the root or administrator user. This is done with the backend command as follow (run ONE of the following commands):

sudo snapbackend -a makeadministrator -p ROOT_USER_EMAIL=you@example.com
# or
sudo snapbackend -a makeroot -p ROOT_USER_EMAIL=you@example.com

Right now sudo is necessary to allow access to the log file. We will fix that problem later with the implementation of the snaplogger.

Installing Snap! Watchdog

In order to manage large clusters, one wants a tool that monitors all the computers without having to log in each one of them to know what the heck is happening. For this purpose we have the snapwatchdog which serves several purposes:

  1. It reads various statistics (CPU/memory usage, etc.) and save them in the Cassandra database
  2. It makes sure that its neighbors are still running/responding
  3. It agglomerates various data files (i.e. logs) on one central computer
  4. It sends emails to administrators to report important events (i.e. loss of connectivity with another server)
  5. It verifies important files information and computing their MD5 to verify them against a local server repository of such files and MD5 information, warn administrators on errors

You want to install the snapwatchdog on all your computers:

sudo apt-get install snapwatchdog

Once installed, make sure to review the configuration file. It determines what will be watched over on that computer and especially it will define a list of other computers to be checked to make sure the network does not go down.

Note: the snapwatchdog tool already exists, but it is currently limited to just point (1) above.

Upgrading Snap! Websites

Individual Server

If you don't have a full cluster, I guess you can pretty much do whatever you want... although it may be a good idea to stop the server while upgrading. The default is to stop the server only when snapserver is being upgraded which can cause problems with core plugins, snapcgi, and other parts being upgraded simultaneously. At a later time, we will offer a tool that you can use to turn the server off in a manner that allows you to show a specific message to the end user instead of just a 500 error.

Cluster of Servers

For Snap! Websites Clusters what you certainly want to do is first reconfigure your cluster so existing systems stop using the computer you want to upgrade. Once the computer you want to upgrade is not being accessed anymore (a simple tool like netstat should be enough to let you know that) then proceed with the upgrade:

sudo apt-get upgrade

If necessary, make updates to the various configuration files. Otherwise, just proceed with re-establishing the computer in the cluster.

Upgrading a Cassandra Node

The process of updating a Cassandra Node is more problematic. The fact is that if you turn off a Cassandra node, now all the other nodes try to access it. If you completely remove the node from the cluster, you run in another problem: the Cassandra cluster stops working right because the math used to find the data of a node breaks.

The easiest from what we can currently see is to simply run the upgrade command as usual. That will stop the node for just a little while and it should recover everything as expected within seconds. Even if you need to restart the computer, it should be done in that order.

Note

The current script from the Apache2 Cassandra package for Ubuntu is not shutting down nodes properly. It just kills them when it should first send them a signal to give them the time to do some usually necessary cleaning up (especially for constantly busy nodes). At some point we will add information for that and offer a fix if it does not get done otherwise. We may even add the cassandra node startup capability to our snapinit.

Setting up a secure local network on a DigitalOcean server

One of our installations makes use of DigitalOcean VPN servers. These are quite practical, but you always end up with a public network. In order to make data transfered between your various servers secure, you want to install a VPN tunnel. This is done with tinc. At this time we do not offer a way to install a node with some helper function, but we have plans to do something of the sort since in most cases you will end up having to do that sort of a thing over and over again.

For now, so, you can use the documentation offered by DigitalOcean in that regard. Their documentation is really thorough.

How To Install Tinc and Set Up a Basic VPN on Ubuntu 14.04

 

  • 1. Note that only applies to data that generally does not need encryption, various plugins may encrypt some information they use.
  • 2. The main reason is that we have to create tables in Cassandra, then create many fields in pages and backend process do not have a real good clue whether things are ready before the main process is done.

Snap! Websites
An Open Source CMS System in C++

Contact Us Directly