libQtCassandra

A C++ library to access Cassandra servers

libQtCassandra LogoThe libQtCassandra library is an advanced C++ library used to access Cassandra servers in C++.

Contrary to the basic Cassadra server interface, this C++ library brings you separate objects that handle each level of the server data, i.e. the cluster, contexts, table, rows, cells.

Details for developers can be found on the reference pages (a 100% complete Doxygen documentation of the library including source code and working examples.)

You may also want to refer to the Installation Instructions to get Cassandra on Ubuntu page on how to install Cassandra on your Ubuntu server. Note that the libQtCassandra library works under MS-Windows as well.

Requirements

You have 2 requirements to compile the library:

  1. cmake, make, and a C++ compiler for the build process
  2. Qt 4.8+
  3. Thrift -- this library is directly included in our tarball! (download)
  4. Cassandra 2.0.x or better (download .deb); the library works with Cassandra since 0.8.4 and is very likely to work with your current version1

Obviously, I suspect you're using Linux. Other Unix systems should be capable of compiling the code. MS-Windows, I have no clue, but I would imagine so. I'll gladly accept comments and patches (questions, comments, patches, anger should be posted on SourceForge.net instead.)

Ubuntu Packages

You can now directly install the Debian packages for Ubuntu 12.10, 13.04, 13.10, 14.04. We build the packages as part of the Snap! project using launchpad (packages).

Download

You can download the source code from SourceForge.net.

We also offer ready to install packages on LaunchPad: Snap CPP (for Ubuntu users.)

Programmer Documentation

Search for the libQtCassandra entries under References.

The documentation is also found inline in the .cpp files of the project. Corrections to the documentation are very welcome!

Supported Features

The following are the main features of libQtCassandra:

  • Create contexts, tables & counters, rows, columns, and cells; individually.
  • Support array syntax to access the data (read [getter] and write [setter])
  • For counters, support the increment/decrement operators: ++, --, +=, -=.
  • Cache the data read or written for much faster data access.
  • Lamport's Bakery Algorithm offering a lock capability using Cassandra.

Cassandra Proxy Server (to be implemented)

Servers Organization using a Proxy (click to enlarge)When connecting to a Cassandra Cluster, one should always use the proxy server. This is a small front end that allows your software to connect to a Cassandra Cluster opposed to just connecting to a Cassandra Node. The concept is pretty simple: if you directly connect to a node, then the connection may fail because that specific node may be down when the cluster as a whole may still be working just fine.

The Cassandra Proxy Server resolves that problems by connecting to multiple Cassandra Nodes and maintaining statistics about the whole cluster. It is capable of connecting to additional nodes when it loses connections to existing nodes. It can also load balance requests so you avoid hitting nodes that are really busy.

The concept is simple, your server connects to the proxy server, which we expect is running on the same local network, and sends normal Cassandra requests to that proxy. The proxy passes your requests to one of the Cassandra nodes it is already connected with and returns the answer as is from Cassandra, so in effect the proxy is transparent to your process.

Adding an additional network connection can cause some slowness, but since this is a local network connection, it should still be really fast (time to copy data buffers in memory,) especially because this one connection does not require any encryption.

Quick Compilation Instructions (see also INSTALL.txt)

To recompile libQtCassandra you need cmake, make, g++, and all the dependencies (Cassandra, thrift, Qt...)

The instructions to compile the library are something like:

tar xf libQtCassandra-0.5.0.tar.gz
mkdir BUILD
cd BUILD
cmake ../libQtCassandra-0.5.0
make
make install

You can create a Debian package or a Windows Package with "make package". Create the package with wpkg is not yet fully supported.

For other targets that you can build, try "make help".

If you cd back out of the BUILD folder, you can use "make -C BUILD" to run make inside the BUILD folder.

Although you can directly run cmake in the libQtCassandra directory, if you plan to do any development work in that folder, I strongly advice for the creation of a separate build folder so you don't mix original source files and generated files.

Issues

Patches, problems, please report on the SourceForce.net page.

You may also find posts of interest in our Snap! C++ Journal under Cassandra such as these few:

The latest source published on SourceForge.net (tarball) has a really bad problem in the content::clearCache() function. It will clear the tables from memory, but it will completely lose track of all the tables, even those that still exist in Cassandra. Version 5.5 has a fix. You can find the code in the SourceForge.net project under Code (in git). We also offer a pre-compiled version for Ubuntu via launchpad.

Projects using this library

Got a project now officially using the libQtCassandra library? Let us know and we'll include a link to your project/product here.

Common Problems Compiling

Missing dependencies

Building CXX object src/CMakeFiles/QtCassandra.dir/QCassandra.cpp.o
Cassandra.h:15:24: fatal error: TProcessor.h: No such file or directory

The thrift library needs to be compiled to work with C++. When it configures (look at the output) it tells you which language extensions it creates that version for. If you don't see C++ selected (...: yes), then you will get that error saying that TProcessor.h cannot be found.

This happens because some dependencies are missing. Unfortunately, I do not know exactly which dependencies. I will update this entry as I discover such. Also we intend, at some point, to detect missing dependencies and generate a clear error instead of moving forward and having strange compilation errors.

Thrift not working?

As you start build, the configure script of the thrift library should be run by cmake. At some point the output should include something that looks like the following. This includes the thrift library version and the generators. You must have at least C++ indicated. If not, the configure script could not find your C++ compiler and thus skipped on it.

thrift 0.8.0

Building code generators ..... :

Building C++ Library ......... : yes
Building C (GLib) Library .... : no
Building Java Library ........ : no
Building C# Library .......... : no
Building Python Library ...... : no
Building Ruby Library ........ : no
Building Haskell Library ..... : no
Building Perl Library ........ : no
Building PHP Library ......... : no
Building Erlang Library ...... : no
Building Go Library .......... : no

Building TZlibTransport ...... : yes
Building TNonblockingServer .. : yes

It is good if you have the TZ included since it will compress the data being sent on the network.

Other Systems of Interest

You may find a need for other systems that are required for your environment to function as expected. For example, if you have heavy needs for locks or work that needs to be serialized, then you may want to look into getting Apache ZooKeeper (a C library implementing a barrier (lock) and a queue.)

Lock Mechanism along Cassandra

There are many solutions for locks. I think that the one most often referenced is Apache ZooKeeper, but there are other solutions depending on your needs. Search around before making a decision.

  • Apache ZooKeeper - server written in C, supports Queues and Barriers
  • ActiveMQ - server written in Java, supports Queues (message based really)

For us, we use snapdb, a daemon that comes with Snap! At first we had a lock object in the libQtCassandra library, but that system does not work when you may end up sending orders to any number of Cassandra nodes. Actually, our lock mechanism opens a single connection to communicate with a single, specific snaplock daemon. Our implementation has very little in limitations, outside of the fact that you have to process one lock with one specific snapdb. For example, we do not have a centralized lock master node. If you are running 5 instances of snapdb, all 5 participate in the locking ability. If one goes down, the lock still continues to work with zero downtime. We could not find another external lock tool that had such a feature.

Right now our snapdb daemon makes use of our snapcommunicator communication system, but we plan to have a version that is a standalone and can be used with any project.

Changes / History

  • Version 0.6+

This version uses CQL instead of thrift. However and although it works great for us, it is not that well adapted to the outside as the previous version was. We are still weighing the pros and cons of whether we want to publish an official version of 0.6+. You may get the source code though.

I found a potential problem in the lock mechanism where a crash (abort, segmentation fault) or a process being killed (kill -KILL <pid>) could leave a lock remain in the database that would last forever preventing further locks with the same object name. I now put a TTL on the entering key so if such a crash occurs the key still disappears within seconds.

Bumped copyright noticed to 2016.

Various clean ups.

Note: the jump in version is due to the fact that I did not post source packages for a while on SourceForge.net; it is also due to our nightly build which first was incrementing the wrong version number.

Added support for a regular expression to filter rows as they are read from the lowest level. This is through the row predicate class. This feature is SLOW but can be useful in special cases where you do not have an index and will not be running such requests over and over again.

Changed all shared pointers from the Qt version to the std version so that way we can properly make use of weak pointers.

Make use of the controlled_vars enum capability and avoid many casts.

Tweaked the CMakeLists.txt to define the system headers as such.

Repaired some warnings that the newer version of g++ generated.

Added a fix to clearTable() so it works as expected.

Documented the fact that QMap sorts from top to bottom even when reading data with the Reverse flag turned on.

Removed some debug code.

Compiled with version Cassandra interface version 2.0.1 and thrift version 0.9.0.

Fixed bug with QCassandraRow::exists(), I needed to test the return value of a function.

Created a Debian compatible changelog file.

Added an implementation of a Lambert's bakery lock (and a corresponding test.) See documentation.

Read consistency level can now be specified.

Added a synchronization function which is necessary if you are working on a cluster (more than 1 node) and want to create or change the schema to a context (table/column definitions.)

Updated all the tests so they also can work on a cluster of 3+ nodes.

Added a QCassandraValue test out of which I fixed the comparison operators (<, <=, >, >=).

Added support to use an index with QCassandraValue buffers (read-only right now.)

Moved the byte array data reads to the global scope.

Added Bool support to the QCassandraValue class.

Fixed the findContext() so it loads contexts first if not loaded yet.

Fixed the disconnected() so a QCassandra object can now be reused properly.

Fixed the snitch function which now returns the snitch (instead of the protocol version).

Fixed two use of column keys that would use a QString instead of a QByteArray (i.e. a null would inadvertendly end the column key.)

Fixed the dropCell() so it doesn't attempt to read the cell first.

Fixed the CMakeLists.txt so the libQtCassandra library is linked against the thrift library (so your tools do not have to know about thrift directly.) Also removed references to the boost_system library.

Reviewed the SSL connection capability. It is still not considered to be working but the password can now be specified from your application.

Updated documentation to be more accurate and define some missing entries.

  • Version 0.4.6

Added direct support for QUuid as row and column keys.

Added direct support for char * and wchar_t * so we do not have to first cast strings to QString everywhere.

Fixed bug testing row key size to limit of 64535 instead of 65535.

Added a test as row and column keys cannot be empty. It will now throw an error immediately if so.

Updated some documentation accordingly and with enhancements.

  • Version 0.4.5

Added a first_char and last_char variables (QChar) in column predicate which can be used to define "[nearly] All column names".

Fixed the names of two functions: setFinishColumnName() and setFinishColumnKey() are now setEndColumnName() and setEndColumnKey() respectively (as documented and so it matches the getters.)

Added support for indexes defined with columns. The column predicate now has a setIndex() function and that allows you to call readCells() repititively until all the columns matching the predicate were returned (very similar to reading a large set of rows.)

Fixed a few things in the documentation.

Added support for composite columns. It was functional before but with knowledge on how to build the column key which is actually quite complicated (okay, not that hard, but libQtCassandra is here to hide that sort of thing!) Use the compositeCell() function of your QCassandraRow objects.

  • Version 0.4.3

Added support for counters.

Fixed several usage of keys so 0 bytes works as expected. (in getValue() and insertValue())

Small fixes to documentation.

Fixed the QCassandraTable::readRows() so it automatically updates the row predicate with the last row as the new start key. This is very important because the rows returned to you get sorted by key in the table, whereas, in Cassandra they are not sorted that way at all. (At least not by default when you use the RandomPartitioner which is very likely.)

Fixed the QCassandraContext::descriptionOption() which would create empty options when the sought option did not exist in the context.

Upgraded the version of Thrift to 0.8.0. There are some problems with the output of the thrift command line option (some missing #include and invalid references.) I fixed the generated code as required so it compiles and the result works as expected.

Made updates to the code so it works with version 1.1 of Cassandra. This includes proper support for the replication factor which was deprecated as a direct field in the KsDef structure. The other deprecated fields are simply ignored at this point (those are in Tables, see CfDef in interface/cassandra.thrift of Cassandra 1.1)

Fixed replicateOnWrite() which now returns the expected value.

Fixed all the context and table get...() functions so if the value is marked as unset, empty or zero is returned instead of the current value saved in the object (which may not reflect what the database is defined as.)

Added the million_rows test to ensure we can create over 1 million rows and read them back. At this time, in my environment, it often crashes the Cassandra server... Java problems?

Added functions that return the partitioner and snitch information from the cluster.

Fixed QCassandraContext::prepareContextDefinition() which would force the replication factor to 1 instead of the user defined value.

The CMakeLists.txt now properly defines the folder where the compiled thrift library lies so it can link with it in the standalone version of the library.

  • Version 0.4.1

Fixed the size of the buffer used to save 64 bit integers.

Fixed the size of integers used to handle floating points.

Fixed the double being read as 8 bytes and somehow converted to a float instead of a double.

Fixed the test of the string set in a value to limit the UTF-8 version of the string to 64Mb (instead of the number of UCS-2 characters held by a QString.)

Enhanced documentation about the findRow() and findCell() which do not look for a row or cell in the Cassandra system, it only checks in memory!

Better support older versions of g++ (4.1 cannot properly cast the controlled variables for enumerations) -- thank you to John Griswold for reporting the problem.

Added some missing documentation.

  • Version 0.4.0

Enhanced the cmake scripts to make it even easier (find/use Qt, Thrift) and thus I jumped to version 0.4.0 because this is a pretty major change from 0.3.x

Removed the Qt sub-folder names from #include.

Made the getValue() function return false so we can know when it fails and react accordingly.

Fixed the use of the slice predicate and ignore the strings null terminator as they ought to be (i.e. a key can include a nul character.)

Added some try/catch to avoid a certain number of fairly legal exceptions (i.e. missing value or column.)

Removed all unwanted files from source package using a CPACK option.

Strip folder name from documentation to make it smaller.

Updated all copyrights to include 2012.

  • Version 0.3.2

Fixed the creation of a row predicate as it wasn't defining a column predicate which is necessary when we call readRows() with the default parameters on a table.

  • Version 0.3.1

Added support for installation targets and generation of binary packages.

  • Version 0.3.0

Added a dropContext() in the QCassandra object.

Added proper unparenting of the context and table classes.

Started to make use of the Controlled Variables (requires 1.3.0 or better.)

TODO

The following are things we intend to add at some point:

  • Support for multi-functions for faster retrievals (this is counter intuitive in our current environment though)
  • Support batch writes (switch your context to record all updates, inserts, and deletes and execute them all at once with one command call!)
  • See to accept all the default Cassandra types accepted as row and column keys. At this time we support UTF-8 (BytesTypes and UTF8Type) and UUIDs (LexicalUUIDType). Although all the other types are supported via the QByteArray, it is difficult (an annoyance really) to make use of a QByteArray for all the basic types. The missing types are: Ascii, Boolean, Date, Double, Float, Int32, Integer, Long.
  • Finish up the composite column support.
  • Proper support for SSL (there is some code in place, but it has not worked for us yet.) with more options.

 

  • 1. The download link is on the front page. Also, the version changes very quickly at this time as it is very actively being developed. We first developed libQtCassandra with 0.8.4, and made an update (0.4.2) to run with 1.1.0, but it should work with any version starting with 0.8.0 to 2.0.1 (libQtCassandra version 0.5.0 was used with 2.0.1 of Cassandra).

Comments

Replied

Re: libQtCassandra

Please, if you have questions, now post them on SourceForge.net as a ticket. That will make it a lot easier to follow different conversations and in case of bug reports, to actually complete fixes.

https://sourceforge.net/p/libqtcassandra/tickets/

Replied

Re: Bug fix in libQtCassandra

You are absolutely correct! The key may include null characters and thus the size must be specified which a c_str() misses completely. Thank you for sharing the fix. I put it in the git for now. (click on the Code link at the top)

Thank you!
Alexis

Replied

Bug fix in libQtCassandra

There is a bug due to which row.readCells() throws exception (when using column name of type UUID and have 00 in between) or goes to infinite loop when the pattern 00 appears at the start.

 

The fix for that is

Changed QCassandraPrivate.cpp:1035 from

  range->setLastKey(it->column.name.c_str());
to
  range->setLastKey(cell_key);
 
This fixed the case "\0" being part of key.
 
Did not know where else to report this so adding it in comment.
Replied

Re: libQtCassandra

Finally I worked around by installing thrift first and then manually changing links.txt to link -lthrift to compile libQtCassandra. 

 

Replied

Re: libQtCassandra

I do see that it it is building thrift with CPP

 

thrift 0.8.0
 
Building code generators ..... :
 
Building C++ Library ......... : yes
Building C (GLib) Library .... : no
Building Java Library ........ : no
Building C# Library .......... : no
Building Python Library ...... : no
Building Ruby Library ........ : no
Building Haskell Library ..... : no
Building Perl Library ........ : no
Building PHP Library ......... : no
Building Erlang Library ...... : no
Building Go Library .......... : no
 
Building TZlibTransport ...... : yes
Building TNonblockingServer .. : yes
 
Replied

Re: libQtCassandra

From what I've seen, these happen when you do not have the CXX module compiled in thrift. When the configure script of thrift goes, you must make sure it says "Yes" for the CXX or C++ entry. (I have that in my list of compilation problems.)

The other thing to do to see what could be missing is to turn on verbosity. You do that by defining the VERBOSE environmnt variable. Something like this:

VERBOSE=1 make -C build

Then see whether the thrift library is indeed included in the list of -l options.

Replied

Re: libQtCassandra

I have a similar if not the same problem in my Mac. I could successfuly compile in Linux.

Here is the compiler output in Mac (sorry for the long listing). Any idea what is going on?

 

[ 12%] Built target thrift-build
Linking CXX shared library libQtCassandra.dylib
Undefined symbols for architecture x86_64:
  "apache::thrift::TApplicationException::read(apache::thrift::protocol::TProtocol*)", referenced from:
      org::apache::cassandra::CassandraClient::recv_describe_snitch(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)in Cassandra.cpp.o
      org::apache::cassandra::CassandraClient::recv_describe_partitioner(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)in Cassandra.cpp.o
      org::apache::cassandra::CassandraClient::recv_describe_version(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)in Cassandra.cpp.o
      org::apache::cassandra::CassandraClient::recv_describe_cluster_name(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)in Cassandra.cpp.o
      org::apache::cassandra::CassandraClient::recv_set_cql_version()    in Cassandra.cpp.o
      org::apache::cassandra::CassandraClient::recv_set_keyspace()    in Cassandra.cpp.o
      org::apache::cassandra::CassandraClient::recv_truncate()    in Cassandra.cpp.o
      ...
  "apache::thrift::transport::TSSLSocketFactory::authenticate(bool)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::createSocket(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::createSocket(int)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::createSocket()", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::loadPrivateKey(char const*, char const*)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::loadCertificate(char const*, char const*)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::loadTrustedCertificates(char const*)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::overrideDefaultPasswordCallback()", referenced from:
      QtCassandra::QCassandraPrivate::QCassandraPrivate(QtCassandra::QCassandra*)in QCassandraPrivate.cpp.o
      QtCassandra::QCassandraPrivate::QCassandraPrivate(QtCassandra::QCassandra*)in QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::ciphers(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::randomize()", referenced from:
      vtable for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::TSSLSocketFactory()", referenced from:
      QtCassandra::(anonymous namespace)::MySocketFactory::MySocketFactory()in QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSSLSocketFactory::~TSSLSocketFactory()", referenced from:
      QtCassandra::(anonymous namespace)::MySocketFactory::~MySocketFactory()in QCassandraPrivate.cpp.o
      QtCassandra::(anonymous namespace)::MySocketFactory::~MySocketFactory()in QCassandraPrivate.cpp.o
  "apache::thrift::transport::TSocket::TSocket(std::basic_string<char, std::char_traits<char>, std::allocator<char> >, int)", referenced from:
      QtCassandra::QCassandraPrivate::connect(QString const&, int, bool)in QCassandraPrivate.cpp.o
  "apache::thrift::TApplicationException::write(apache::thrift::protocol::TProtocol*) const", referenced from:
      org::apache::cassandra::CassandraProcessor::process(boost::shared_ptr<apache::thrift::protocol::TProtocol>, boost::shared_ptr<apache::thrift::protocol::TProtocol>, void*)in Cassandra.cpp.o
      org::apache::cassandra::CassandraProcessor::process_fn(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int, void*)in Cassandra.cpp.o
      org::apache::cassandra::CassandraProcessor::process_login(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)in Cassandra.cpp.o
      org::apache::cassandra::CassandraProcessor::process_set_keyspace(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)in Cassandra.cpp.o
      org::apache::cassandra::CassandraProcessor::process_get(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)in Cassandra.cpp.o
      org::apache::cassandra::CassandraProcessor::process_get_slice(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)in Cassandra.cpp.o
      org::apache::cassandra::CassandraProcessor::process_get_count(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)in Cassandra.cpp.o
      ...
  "typeinfo for apache::thrift::transport::TSSLSocketFactory", referenced from:
      typeinfo for QtCassandra::(anonymous namespace)::MySocketFactoryin QCassandraPrivate.cpp.o
  "vtable for apache::thrift::transport::TFramedTransport", referenced from:
      apache::thrift::transport::TFramedTransport::TFramedTransport(boost::shared_ptr<apache::thrift::transport::TTransport>)in QCassandraPrivate.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make[2]: *** [src/libQtCassandra.dylib] Error 1
make[1]: *** [src/CMakeFiles/QtCassandra.dir/all] Error 2
make: *** [all] Error 2
Replied

Re: libQtCassandra

Not us so far... you need boost for Thrift, then you need to compile Thrift. Also obviously you need Qt. So that's quite a bit of work to make it all compile under Windows.

Replied

Re: libQtCassandra

Someone tried to compile it in Windows? On Linux works fine, but I would like it to run on Windows.

Replied

Re: multi-column write

Unfortunately, I have not yet worked on batches.

I have been thinking of a way to allow writes to only happen in the cache (memory) and at the end to do a dump of all the modified data in one go using the batch_mutate() function. That would include inserts, updates, and deletes. At this point I have a problem with the deletes though (when deleting it removes the memory cache...) but have been thinking to instead create some form of a journal of all the changes on a per context basis.

Note that this method has a great advantage when writing the same value over and over again while running a server and you only need to save the last value to the database!

Note that the dump could be activated whenever you'd like. You would not have to wait the very end of the lifetime of your Cassandra C++ object.

Replied

multi-column write

Hi,

is it possible to do a multi-column write in a row (instead of multiple calls/writes for each column in the row)?

Replied

Re: libQtCassandra

Ah! Okay! Understood!

The Qt library is bizarre as far as includes go. The default is to now use

#include <QObject>

So the include with a class name by itself. The Qt library has many folders and they all need to appear on your g++ command line with the -I option.

For example I have this on my computer:

ls /usr/include/qt4/
Qt          QtCore        QtDeclarative  QtGui
QtNetwork   QtScript      QtSql          QtTest
QtWebKit    QtXmlPatterns Qt3Support     QtDBus
QtDesigner  QtHelp        QtOpenGL       QtScriptTools
QtSvg       QtUiTools     QtXml

In your case, the -I should at least include:

-I/usr/include/qt4/QtCore

This is done automatically when you use cmake and whatever code we have in there (see cmake/qt.cmake in the libQtCassandra project.)

Replied

Re: libQtCassandra

all library tests get compiled properly and seem to be working fine. i can connect to a remote cassandra node successfuly. i changed some examples and rebuilt them by running make cluster/read_write_data, etc ... and all is good.

what i am trying to build now (previous question) is my own simple example outside of the libQtCassandra build system. i have libQtCassandra and thrift header files/libraries in usr/local, qt4 4.8.4 is installed properly (core, dev packages etc ...) but getting

/usr/local/include/QtCassandra/QCassandraColumnDefinition.h:39:19: fatal error: QObject: No such file or directory

basically i want to link against libQtCassandra.so and thrift.so in my app using my own make file outside of the libQtCassandra cmake.

Replied

Re: libQtCassandra

I actually made a change to the libQtCassandra CMakeLists.txt file so it would include the libraries in its link process.

In the file src/CMakeLists.txt, after the add_library, I now have this:

target_link_libraries(QtCassandra
    thrift
    ${QT_LIBRARIES}
)

That seems to help linking and running under Ubuntu 12.10.

However, from what you're saying, it somehow did not find the QT headers while compile. Looking at the files, I see a globale addition of many include directories including the QT ones... I'm not too sure why it wouldn't be visible from the tests folder in your case.

One way to see whether you've got the correct -I option is to run with something like this:

make -C BUILD VERBOSE=1

That will give you the g++ command line in full.

Replied

Re: libQtCassandra

 

thanks. now that the compilation goes through, i ran sudo make install. when i tried to compile slightly modified cluster.cpp from cli, i got this:
 
In file included from /usr/local/include/QtCassandra/QCassandraTable.h:39:0,
                 from /usr/local/include/QtCassandra/QCassandraContext.h:39,
                 from /usr/local/include/QtCassandra/QCassandra.h:40,
                 from cluster.cpp:38:
/usr/local/include/QtCassandra/QCassandraColumnDefinition.h:39:19: fatal error: QObject: No such file or directory
 
am i still missing some dependencies?
Replied

Re: libQtCassandra

Are you on Ubuntu 12.10? I just had to fix a few problems in the CMakeLists.txt to get it to work right. Also it may be that you have some missing dependencies. I just had the problem and fixed it by fiddling around.

I should get 0.4.7 out very soon (it includes a lock feature) and that should help you, if you can wait 2 or 3 more days...

Note that at that point you're not compiling thrift anymore, you're already onto compile the libQtCassandra library. It says that the thrift headers seems to be missing.

One way I went around that a while back was to install the thrift library under my /usr/local folder and then you'd get the headers under /usr/local/include/thrift/... However you have to be careful with that method, if you update to a newer version, then you'd want to remember removing the thrift files under /usr/local.

Replied

Re: libQtCassandra

please ignore my compilation issues. i didn't have all required dependencies for building thrift for c++ environment. it compiles now. thanks.

Replied

Re: libQtCassandra

Followed the build instructons but compilation fails for me during the thrift build (failed to find TProcessor.h):

 

[  4%] Building thrift
Making all in compiler/cpp
Making all in lib
Making all in test
[  8%] Installing thrift
Making install in compiler/cpp
  /bin/bash ../../libtool   --mode=install /usr/bin/install -c thrift
libtool: install: /usr/bin/install -c thrift
 
Making install in lib
Making install in test
[ 12%] Built target thrift-build
[ 16%] Building CXX object src/CMakeFiles/QtCassandra.dir/QCassandra.cpp.o
Cassandra.h:15:24: fatal error: TProcessor.h: No such file or directory
make[2]: *** [src/CMakeFiles/QtCassandra.dir/QCassandra.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/QtCassandra.dir/all] Error 2
make: *** [all] Error 2
 
 
I'm building the latest library libQtCassandra-0.4.6 on ubuntu 12.04; qt4 library is 4.8.1.
 
Thanks

 

Replied

Re: libQtCassandra

In the million_rows.cpp sample you have these few lines around line 148:

const QtCassandra::QCassandraRows& r(table->rows());
for(QtCassandra::QCassandraRows::const_iterator o(r.begin());
                                    o != r.end(); ++o, ++i) {
    const QtCassandra::QCassandraCells& c(o.value()->cells());

After the call to cells(), the variable 'c' has cell data and also cell names. The names can be retrieved with the columnName() function. (and the data with the value() function.)

I would think that the cells() call will work because you do not have to specify the column names in this case.

Replied

Re: libQtCassandra

Thanks for all your assistance in this. Can you tell me how to read the entire row from Cassandra to see how they stored these keys ?

Replied

Re: libQtCassandra

I would suggest you look in more details what there is in the database by reading the data (read the entire row from libQtCassandra and print that out.) Then replicate what you see using the different options offered. I'm finishing up a new version which will include more array handling in the QCassandraValue.h header file. If they save a 'text' field as UCS-2, then you cannot use a QString with libQtCassandra because that's saved as UTF-8. Also, if they expect the name of the column to be included in the query, you'll need to find out how that's done. I have no clue on my end because I don't use CQL at all.

Replied

Re: libQtCassandra

I tried the suggestion by adding the column name before each of the two keys with the same error as before. I even tried to add a ":" between the column name and the key with the same failure.

Replied

Re: libQtCassandra

Again, I've not been using CQL, so I cannot really tell you for sure. However, if I read the errors properly: the first one probably means that they want you to include the name of each column in the composite along with their value. I would try to add a string for each name.

    QtCassandra::QCassandraValue a1n(QString("id"));
    QtCassandra::appendUInt16Value(byte_row_key, a1n.size());
    QtCassandra::appendBinaryValue(byte_row_key, a1n.binaryValue());
    QtCassandra::appendUnsignedCharValue(byte_row_key, 0);

The u['KEY'] size of 1 sounds to me like the 'text' type would be UCS-2, or maybe UTF-16, and not UTF-8 as we use. But you used the name "John" which is a fit for UCS-2 since it has an even number of characters.

Replied

Re: libQtCassandra

Hi,

I looked at the sample program composite_type.cpp for the usage of composite keys. Here, the composite keys are not used as row keys. I would like to use them as row keys as below.

 

Using CQL3:

cqlsh> describe keyspace test_ks

 

CREATE KEYSPACE test_ks WITH strategy_class = 'SimpleStrategy'

  AND strategy_options:replication_factor = '1';

 

USE test_ks;

 

CREATE TABLE test_tbl (

  id text,

  ts timestamp,

  float_val float,

  city text,

  PRIMARY KEY (id, ts)

) WITH

  comment='' AND

  caching='KEYS_ONLY' AND

  read_repair_chance=0.100000 AND

  gc_grace_seconds=86400 AND

  replicate_on_write='true' AND

  compaction_strategy_class='SizeTieredCompactionStrategy' AND

  compression_parameters:sstable_compression='SnappyCompressor';

 

Here id and ts are composite columns that are part of the composite row key. But still they are queryable using individual conditions.

 

cqlsh> select * from test_ks.test_tbl where id = 'John' and ts > '2012-11-01';

 id        | ts                       | float_val  | city

-----------+--------------------------+------------+----------

 John      | 2012-11-01 20:00:00+0000 |   197      | Boston

 John      | 2012-11-02 20:00:00+0000 |   193      | Detroit

 

I tried to use the composite key as row key and it didn’t work. The exists() methods returns false. Any ideas on why this didn’t work ?

---------------

    QByteArray byte_row_key;

 

    QtCassandra::QCassandraValue a1(static_cast<uint64_t>(1351814400000));

    QtCassandra::QCassandraValue a2(QString("John"));

 

    QtCassandra::appendUInt16Value(byte_row_key, a1.size());

    QtCassandra::appendBinaryValue(byte_row_key, a1.binaryValue());

    QtCassandra::appendUnsignedCharValue(byte_row_key, 0);

 

    QtCassandra::appendUInt16Value(byte_row_key, a2.size());

    QtCassandra::appendBinaryValue(byte_row_key, a2.binaryValue());

    QtCassandra::appendUnsignedCharValue(byte_row_key, 0);

 

 

    if(cassandra[ks][tbl].exists(byte_row_key))

---------------

 

Further, I noticed that there was an issue with the definition of column family qt_cassandra_test_ct created in composite_type.cpp when I use CQL 3.

 

Using CQL3:

cqlsh> describe keyspace qt_cassandra_test_ct

 

CREATE KEYSPACE qt_cassandra_test_ct WITH strategy_class = 'SimpleStrategy'

  AND strategy_options:replication_factor = '1';

 

USE qt_cassandra_test_ct;

 

/apps2/apache-cassandra-1.1.6/bin/../pylib/cqlshlib/cql3handling.py:771: UnexpectedTableStructure: Unexpected table structure; may not translate correctly to CQL. expected composite key CF to have column aliases, but found none

/apps2/apache-cassandra-1.1.6/bin/../pylib/cqlshlib/cql3handling.py:794: UnexpectedTableStructure: Unexpected table structure; may not translate correctly to CQL. expected [u'KEY'] length to be 2, but it's 1. comparator='org.apache.cassandra.db.marshal.CompositeType
(org.apache.cassandra.db.marshal.UTF8Type,org.apache
.cassandra.db.marshal.IntegerType)'

CREATE TABLE qt_cassandra_test_table (

  "KEY" blob PRIMARY KEY

) WITH

  comment='' AND

  caching='KEYS_ONLY' AND

  read_repair_chance=0.100000 AND

  gc_grace_seconds=3600 AND

  replicate_on_write='true' AND

  compaction_strategy_class='SizeTieredCompactionStrategy' AND

  compression_parameters:sstable_compression='SnappyCompressor';

 

 

Replied

Re: libQtCassandra

Composite columns are one long name as defined on this page:

https://snapwebsites.org/journal/2012/09/adding-support-composite-column-names-libqtcassandra-043

Once you understand that, then you understand that the one normal column key is used to represent n column keys in a composite key. Assuming all your column keys are strings, then it's the size of the string on 2 bytes, the string itself, then a one byte terminator; repeated for each column name in the composite name.

Since the size is 2 bytes, the name of a key in a composite column key is limited to 64Kb. (opposed to the 2Mb for a full key)

Replied

Re: libQtCassandra

Hi,

I downloaded the new version 0.4.6 version of libQtCassandra and now am able to perform lookup by UUID.

I was refering to composite primary keys as referred in the section "Using Composite Primary Keys".

 

If there are two columns that are part of a primary key, how can I perform the search using libQtCassandra 0.4.6 ? Is it even supported in thrift 0.8.0 ?

I was able to execute the CQL 3 query that uses column family with composite keys from thrift 0.8.0 using the execute_cql_query() C++ interface.

But, couldn't figure out querying of column family with composite primary key using thrift/libQtCassandra standard interface.

Thanks

Replied

Re: libQtCassandra

Hi AJ,

I haven't used any UUID myself as I use simple counters instead. But you want to use the QUuid class to create an instance and then use the toByteArray()toRfc4122() function to retrieve the data.

QUuid uuid("{13818e20-1dd2-11b2-0000-49660bcef5ff}");
...exists(uuid.toRfc4122())...

There is also a toByteArray() function, but that outputs the string in a QByteArray which is not what you want. The toRfc4122() function returns the UUID in binary form and that's what Cassandra expects. There is the Java documentation for the UUID that Cassandra uses:

https://docs.oracle.com/javase/6/docs/api/java/util/UUID.html

Easier support for composite columns was added in version 0.4.4 of libQtCassandra. This being said, it is just a way to define the name of the columns. You could reuse the code and integrate it in an older version of libQtCassandra.

I started using Cassandra 1.1 in version 0.4.2 of libQtCassandra. Older versions work with Cassandra 0.8.

I think that the newer code will work with Cassandra 0.8 only some fields don't exist so you'd have to comment them out to compile the library.

I'm not looking forward to CQL myself so I'm not aware of any C++ binding in that regard. If it's not listed on the Cassandra website, it probably doesn't exist yet.

Replied

Re: libQtCassandra

Hi,

I have a column family with column key of type uuid. The column family has a row as follows.

 

cqlsh:qt_cql2_ks > select * from bar_data1 ;

id | field1 | field2 | field3 | ts

--------------------------------------+-------+------+--------+--------------------------

13818e20-1dd2-11b2-0000-49660bcef5ff | 123 | 234 | test | 2012-11-01 00:00:00+0000

 

I was able to retrieve this row using the CLI interface as below

 

 [default@qt_cql2_ks] get bar_data1['13818e20-1dd2-11b2-0000-49660bcef5ff'] ;

= > ( column=field, value=123.4, timestamp=1354720350056002 )

= > ( column=field2, value=234.5, timestamp=1354720350056000 )

= > ( column=field3, value=test, timestamp=1354720350056001 )

= > ( column=ts, value=2012-11-01 00:00:00-0400, timestamp=1354720350057000 )

Returned 4 results.

 

However, I’m unable to check or extract the row using the uuid key.

 

const char* ks = "qt_cql2_ks" ; ;

const char* tbl = "bar_data1" ; ;

if ( cassandra[ks][tbl].exists (QString("13818e20-1dd2-11b2-0000-49660bcef5ff")) )

 

The above method always returns false. Is there a different technique to pass 16 byte uuid as the key in libQtCassandra ?

 

Few other general question:

  1. Does the version 0.3.2 of libQtCassandra built with thrift 0.8.0 support the storage and retrieval of data using composite column key (multiple column key) as introduced in CQL3 ?
  2. Are you aware of any C++ driver for calling a CQL from C++ code ?

 

I appreciate your help with this.

 

Replied

Re: libQtCassandra

I used the setDescriptionOption() method to set the replication factor and that resolved the issue. Thanks for your help.

Replied

Re: libQtCassandra

Yeah... the Replication Factor has changed in the latest version and went from being a field to being an option. That's probably what your problem is.

Do you call the setDescriptionOptions() function? If so, make sure you include a "replication_factor" option set to 1.

Otherwise, check out the parseContextDefinition() function. That's the one that's used to define the factor value in the thrift structure. The ks->__isset.replication_factor flag should be true in your case (around line 572 of src/QCassandraContext.cpp).

Snap! Websites
An Open Source CMS System in C++

Contact Us Directly