Snap! Websites
An Open Source CMS System in C++
As we are working on Snap! Websites we run in various problems, one of which is the installation process and later the upgrade of a cluster of Snap! Websites servers.
Snap! Websites are built to run in a cluster. Because of that, the security of the cluster is mainly insured by the computers in the cluster and not by the servers themselves.
There are a few known points that render the Snap! Websites service weak in terms of security if people other than you can access your computers via a network connection or directly:
Note: these considerations do not include the security of accessing the servers via an HTTP port.
Installing the Snap! Websites packages is very easy. It is just as easy as any other packages in Ubuntu. However, the snapserver will not properly run until you properly configure it and enable it. You also want to only install what you want to run on each machine. With experience, you will learn what is really necessary on each one of your machines. Obviously, apt-get and other similar tools will make sure that dependencies all get installed.
For example:
apt-get install snapserver
Assuming you want to use a cluster of front end systems (we recommand at least 2 front end computers for a properly working cluster, read more about this in the upgrade section,) you want to install a name server that can do a form or another of round robin for your front end systems. That way the load gets shared between many front end systems.
At this point we do not offer such a feature, however, you could also look into forcing people who are logged in to use a set of IPs and people who are not logged (anonymous) in to use another. The concept here is simple: people who are not logged in should never have to wait. People who are logged in may have to wait a little longer for an answer, but since they are logged in you would expect them to wait for such. This can be achieved by forcing users who logged in to use a different sub-domain and that address ends up being different (i.e. "www.
The Snap! Websites sendmail backend makes use of the system sendmail tool. It has to be installed. Which may server you use is not important, but it has to be a sendmail tool that understands the standard way of sending emails with such. We use the Postfix version for which we also have the snapbounce package.
Note
The snapbounce package includes documentation on how to setup postfix to get bounce emails to your Cassandra cluster via snapbounce.
On the Mail Server computer you want to install the snapbounce, snapserver, and snapserver-core-plugins packages. Start at least the snapbackend with the sendmail action (using the snapinit, you will list sendmail as one of the servers) and snapbounce has to be on the computer which receives mail to work. It saves bounced emails in your database allowing you to see what works and what does not work.
apt-get install postfix snapserver snapbounce
In most cases you will want to Install 3 or more Cassandra Systems that are backend computers (i.e. not accessible from the outside, except maybe from SSH). 3 is the minimum you will need to make a proper Cassandra cluster with duplication that works efficiently. Many more is of course better if you get millions of hits per day. Cassandra defines various ways for you to determine the number of nodes required. It includes statistics that one can use to see what is happening with your Cassandra cluster.
If you are not in control of the local network and thus others have access to your local network, you may want to create a VPN so all transactions between computers remain encrypted. This imposes a slowdown so if you can afford your own local network, all the better.
IMPORTANT NOTE
You should have computers with at least 4Gb of memory to be sure that your nodes do not crash once in a while. We tested with 1Gb and 2Gb and it tends to crash, meaning that you need a lot of maintenance to repair nodes that crashed... so to make it easier on yourselves, 4Gb or more. Cassandra will use 50% of the RAM and leave the other 50% for the OS to use as buffers.
In order to get a system run as smoothly as possible, Snap! Websites is broken up in one front end which can be paralellized and a number of backends defined by various plugins. As mentioned earlier, we have one that can be used just to send (and ultimately receive) emails.
You may run all the backend servers on the same computer, although once the computer CPU and/or I/O gets used at 50% or more, we strongly suggest that you break up the backends on multiple computers. You cannot run the same backend more than once (we do not currently have a lock, so you must be very careful about that yourselves otherwise it is not unlikely to create problems.)
As noted above, the sendmail backend must run on a computer that has sendmail. It is not required for that computer to be the one with postfix configure to send mail to the outside. You could have a local postfix install that communicates with your mail server or have a sendmail system that knows how to do so.
Right now we offer the following backends:
sendmail pagelist images
The backends to run on a computer need to be listed in the snapserver.conf file.
Snap! Websites include a CRON like feature that runs various backend processes. These can be run once every 5 minutes or so. Depending on your load, whether you use a separate server, whether you have many Cassandra nodes, and some other parameters, the timing can be adjusted to lower the load on the server as a whole.
services=backend (WARNING: we are working on this one right now, it still uses CRON at this point)
This backend server can run on its own server or with other backends. If you separate your front ends, you should not run this on a front end. Like with other backends, only one computer can run the main backend (CRON) process at a time. We do not yet offer a lock to avoid multiple simulataneous runs which could cause problems.
The Snap! Website frontend servers (snapserver) handle the incoming HTTP traffic and generate the outgoing HTTP traffic. These servers can be on your local network and not on the same computers as Apache and snapcgi. One reason to put these on separate computers is to allow for higher security.
Although future versions may directly answer the HTTP port, at this time we use Apache2 as the main HTTP server. This adds another layer making it possible to have snapserver's on separate computers and create proxies which snapserver would not offer.
sudo apt-get install snapserver
The server requires some settings to be defined in the /etc/snapwebsites/snapserver.conf file. At least, you should specify server in the list of items that snapinit should start.
services=server
Right now, the snapcgi is an Apache2 module, it may work with other HTTP servers, but we only test and use it with Apache2. Since this is a CGI tool, you do not need any special handling, only to tell Apache to use snapinit to handle requests that arrive on specific domains (or all domains.)
The server with Apache2 also needs to have snapcgi which you can install by itself.
sudo apt-get install snapcgi
We offer a snap-apache2.conf file showing you how to define an entry in your Apache2 configuration to allow Apache to call snapcgi properly. snapcgi then connects to one of your Front End Snap! Servers (snapserver).
The Crux of the script are these three lines of code:
RewriteEngine on RewriteRule snap\.cgi - [S=1] RewriteRule ^(.*)$ /cgi-bin/snap.cgi?q=/$1 [L,PT,QSA,E=CLEAN_SNAP_URL:1]
The second line tells the server to ignore the request if it is snap.cgi directly.
The last line transforms standard HTTP requests in such a way that snap.cgi runs with it instead of the default Apache2 reaction to load a static page from your disk.
With all the servers in place, you can start the installation of a website.
We will be looking into ways to simplify the process, but right now you have quite a few steps to follow. Please note that until a first website was initialized, none of the backends can be running.
First, you want a direct connection between your Cassandra cluster and snapmanager. You can install snapmanager on your own computer as long as you can open a connection to your Cassandra cluster from there. Preferably, use a VPN or tunneled connection so it remains secure and don't forget to close that connection once done.
When you run snapmanager, you can first test that you can connect to your Cassandra cluster. Once that works (test button on the first page) then you can click on Domains tab. There you can add domains.
Once you have one or more domains, you can add websites for these domains. The snapserver needs both, domains and websites, in order to figure out how to handle such to give you access to what will look like a website.
Detailed help for snapmanager is available in our help section.
We currently run in many concurrency problems with attempting to create a website with everything turned on at the same time2. So... for the very first website installation you want to follow the following steps quite strictly:
Then wait for the process to finish. You should not see any errors appearing in the Status console. The first few entries are about snapmanager connecting to the snapserver service. This is a direct connect (i.e. without Apache2 or snapcgi). If that part does not work, check your firewalls. By default the server uses port 4004, however, you may have changed the port. Also if you use a tunnel or a VPN you may be using a different IP and/or port.
Once the First Website Installation succeeds, you are ready to test the website live using your browser. Now you want to make sure Apache2 runs and you can start all the backends.
Finally, in your browser you should be able to just access your website home page. The default theme shows a couple of sentences on the home page and a log in screen on the left side of the screen. The log in screen includes a Register link. Using that link you can create an account on your new website. Note that accounts on Snap! Websites clusters are common to all the websites, so once you created one account, you can access all the websites you own, comment on all websites that allow a user to comment, etc.
To further configure your new website, you probably want to make yourself the root or administrator user. This is done with the backend command as follow (run ONE of the following commands):
sudo snapbackend -a makeadministrator -p ROOT_USER_EMAIL=you@example.com # or sudo snapbackend -a makeroot -p ROOT_USER_EMAIL=you@example.com
Right now sudo is necessary to allow access to the log file. We will fix that problem later with the implementation of the snaplogger.
In order to manage large clusters, one wants a tool that monitors all the computers without having to log in each one of them to know what the heck is happening. For this purpose we have the snapwatchdog which serves several purposes:
You want to install the snapwatchdog on all your computers:
sudo apt-get install snapwatchdog
Once installed, make sure to review the configuration file. It determines what will be watched over on that computer and especially it will define a list of other computers to be checked to make sure the network does not go down.
Note: the snapwatchdog tool already exists, but it is currently limited to just point (1) above.
If you don't have a full cluster, I guess you can pretty much do whatever you want... although it may be a good idea to stop the server while upgrading. The default is to stop the server only when snapserver is being upgraded which can cause problems with core plugins, snapcgi, and other parts being upgraded simultaneously. At a later time, we will offer a tool that you can use to turn the server off in a manner that allows you to show a specific message to the end user instead of just a 500 error.
For Snap! Websites Clusters what you certainly want to do is first reconfigure your cluster so existing systems stop using the computer you want to upgrade. Once the computer you want to upgrade is not being accessed anymore (a simple tool like netstat should be enough to let you know that) then proceed with the upgrade:
sudo apt-get upgrade
If necessary, make updates to the various configuration files. Otherwise, just proceed with re-establishing the computer in the cluster.
The process of updating a Cassandra Node is more problematic. The fact is that if you turn off a Cassandra node, now all the other nodes try to access it. If you completely remove the node from the cluster, you run in another problem: the Cassandra cluster stops working right because the math used to find the data of a node breaks.
The easiest from what we can currently see is to simply run the upgrade command as usual. That will stop the node for just a little while and it should recover everything as expected within seconds. Even if you need to restart the computer, it should be done in that order.
Note
The current script from the Apache2 Cassandra package for Ubuntu is not shutting down nodes properly. It just kills them when it should first send them a signal to give them the time to do some usually necessary cleaning up (especially for constantly busy nodes). At some point we will add information for that and offer a fix if it does not get done otherwise. We may even add the cassandra node startup capability to our snapinit.
One of our installations makes use of DigitalOcean VPN servers. These are quite practical, but you always end up with a public network. In order to make data transfered between your various servers secure, you want to install a VPN tunnel. This is done with tinc. At this time we do not offer a way to install a node with some helper function, but we have plans to do something of the sort since in most cases you will end up having to do that sort of a thing over and over again.
For now, so, you can use the documentation offered by DigitalOcean in that regard. Their documentation is really thorough.
How To Install Tinc and Set Up a Basic VPN on Ubuntu 14.04
Snap! Websites
An Open Source CMS System in C++
