TLD Library (libtld)

Introduction

The libtld is a library used to extract the TLD from a URI. This allows you to extract the exact domain name, sub-domains, and all the TLD (top level, second level, third level, etc.)

The problem with TLDs is that you cannot know where the domain starts. Some domains can use one top-level domain, others use two, etc. However, it may be useful to know where the domain is to have the exact list of sub-domains. For example, if you want to force www. at the start of the domain name if no other sub-domains are specified, then you need to know exactly how many TLD are defined in a URI.

The libtld offers one main function: tld(), which gives you a way to extract the TLD from any URI. The result is the offset where the TLD starts. This gives you enough information to extract everything else you need.

Download

You can download the TLD Library from SourceForge.net (source, binaries for Linux in .deb, documentation).

If you want to live on the edge you can get the latest source code from git.

We also offer ready to install packages for Ubuntu on LaunchPad: Snap CPP (packages)

Support

You got a problem with the library? An idea to improve it? Please post a ticket in the Support area of SourceForget.net.

You like the libtld library?
Please spread the word by sharing a post about libtld.

Features

The library offers:

  • A complete world wide list of all Top-Level Domain names.
  • One C function to find the Top-Level Domain name (TLD).
  • One C++ class to handle sub-domains, domains, and TLDs.
  • A set of C functions to check email address according to RFC 5322.
  • A C++ class to check lists of email addresses.
  • One PHP extension to access the power of the libtld directly in your PHP scripts (3 functions at this point.)
  • A complete reference of all the C and C++ declarations, functions, etc.
  • A full coverage test suite to ensure the validity of the library.

Requirements

The development environment required CMake to generate the Makefiles.

Some files include headers from the boost library.

The XML parser makes use of C++ and Qt4. Note that you may just compile the libtld library using the dev/libtld-only-CMakeListst.txt which allows you to skip on the Qt4 dependency. Later we will have a better set of CMakeLists.txt with better tests to automatically avoid dependencies when not available.

The library itself does not have any requirements (other than a C or C++ compiler, obviously.) It comes with one header (tld.h).

The PHP extension requires the php5-dev environment (also called Zend) to get compiled.

Documentation

The libtld has a small documentation since it includes only a very few functions. The documentation is available in the References section of this website.

Compiling 1.4.5 and newer

Newer versions have been modified to make use of an external set of CMake modules. This version requires you to down the snapcmakemodules_x.y.z.tar.gz. This then needs to be specified on the cmake command line to make it all work:

tar xf snapcmakemodules_x.y.z.tar.gz
tar xf libtld-x.y.z.tar.gz
mkdir BUILD
cd BUILD
cmake -DCMAKE_MODULE_PATH=../snapCMakeModules/Modules ../libtld
make
sudo make install

The files that you extract may include a directory with a version. Adjust the paths accordingly:

cmake -DCMAKE_MODULE_PATH=../snapCMakeModules-1.0.8/Modules ../libtld-1.4.5

At this point, though, the Debian packager does not put a version so we leave it out.

Compiling 1.4.2 and older

To compile the library, you need cmake. If this is your first time with cmake, do not be afraid, it is very easy to use. There are the few steps to get the library compiled:

cd to/directory/with/libtld-1.4.0.tar.gz
tar xf libtld-1.4.0.tar.gz
mkdir build
cd build
cmake ../libtld-1.4.0
make
sudo make install

The last line is not required if you do not want to install the library on your system. To change the installation path, you can use cmake this way:

cmake -DCMAKE_INSTALL_PREFIX=/absolute/path/to/somewhere/else ../libtld-1.4.0

and now the files will go under /absolute/path/tp/somwhere/else.

If you have warnings appearing, edit the main CMakeLists.txt and change the lines setting up the C and CXX flags. I already offer the ones with just -O3. Note that I will look into a way for future versions to offer a better solution (i.e. a cmake flag that can be used to switch the warnings ON instead of having them ON by default.)

Other TLD Projects

Public Suffix List

The Mozilla Foundation keeps a list of top-level domain names as a text file including comments. The project is called Public Suffix List.

I'm thinking to add a test that uses that link to check the libtld against URLs generated using this list. That way we can easily find discrepancies. From a quick look, it seems more complete, but at the same time, it looks like it may include valid URLs.

The list is very specifically used to handle cookies and prevent users from assigning a cookie at the wrong level (i.e. you may assign snapwebsites.org as the domain of a cookie, but not just .org; see Supercookie in wikipedia.)

idn library

The idn library includes support for checking TLDs in a string. Their interface is 100% in C and it includes the possibility to have additional definitions (overrides) of the TLD data.

On an Ubuntu system you can install the development library with:

sudo apt-get install libidn11-dev

At this time I did not test that library, however, I changed the name of the libtld library header from just tld.h to libtld/tld.h to avoid the header conflict (because libidn also called their TLD header tld.h).

You can find manual pages by looking up the existing functions in the /usr/include/tld.h from the libidn11-dev package and use man to find the corresponding pages. For example:

man tld_get_z

It feels like they offer a check of the characters of a domain as per each TLD defined rules.

Changes

1.4.20

  • Added even more new gTLDs and updated country gTLDs that changed.
  • Updated the tests accordingly.
  • Note: we changed our internal build system to avoid incrementing the patch version of the library, so again we will be moving one version at a time (i.e. next version will be 1.4.21)

1.4.17

  • Added new gTLDs as defined by the ICANN.
  • Updated various documents accordingly.
  • Note: intermediate versions are generated by our build system and do not include additional changes.

1.4.6

  • Added new gTLDs as defined by the ICANN.
  • Added a "brand" category.
  • Added a test to verify all the extensions offered by GoDaddy.
  • Added a script to build the source using the Debian tool dpkg-source.
  • Added some documentation in different files (INSTALL.txt, README...)
  • Updated the coverage test to work in the new environment.
  • Fixed the copyright notice with 2014.
  • Bumped the version in the CMakeLists.txt to the correct version.
  • Bumped version for publication to Sourceforge.net.

1.4.5

  • Got this version in the PPA.

1.4.4

  • Free the allocated memory as otherwise expected, avoiding memory leaks.

1.4.3

  • Got the debian directory to work with launchpad

1.4.2

  • Fixed the licenses in source files to MIT as it was supposed to be.
  • Moved the ChangeLog file to debian/changelog.
  • Added a debian/copyright file.
  • Added the validate_tld command line tool to check TLDs in shell scripts.
  • Changed the cmake scripts that get included by the main CMakeLists.txt.

1.4.1

  • Added new TLDs added in the last few months.

1.4.0 (see also)

  • Added the debian folder to build packages for the PPA (not working yet).
  • Corrected the installation process with proper components.
  • Fixed the ChangeLog dates and indentation for Debian.
  • Reorganized the package with sub-folders.
  • Fixed a few things so the library compiles under MS-Windows with cl.
  • Added support to parse email addresses.
  • Enhanced several of the unit tests.
  • Added a script to run a full coverage over the library.
  • Added a command line tool for scripting.
  • Enhanced documentation as first version was really just about tld().
  • dev/libtld-only-CMakeLists.txt to compile libtld on other platforms.
  • Known Bugs:
    • The code does not properly escape non-atom characters in the resulting email
    • The canonicalized email also does not escape non-atom characters
    • Note that these bugs prevent you from using the results as is, but they do not prevent you from validating email addresses.

1.3.0 (see also)

  • Added the ChangeLog file.
  • Added a function to check the validity of a complete URI.
  • Added a C++ class to easily handle URIs in C++.
  • Added a PHP extension so [check_]tld() can be used in PHP.
  • Added a static version of the library.
  • Updated the TLDs as of Feb 2013.
  • Updated copyright notices.
  • Updated the tests as required.
  • Enhanced the tests with better errors and added tests.
  • Added a target to run all the TLD tests at once.
  • Fixed the TLD exceptions which now return a valid answer.
  • Fixed the Doxygen generation so we get the proper documentation.
  • Fixed/enhanced the documentation as I was at it.
  • Fixed the references to Qt through the CMakeLists.txt file.
  • Fixed data test so it doesn't crash if it can find its XML data file.

1.2.0

  • Added support for exceptions so we now properly support .uk for domains such as nic.uk, but forbid it for all the sites that are not exceptions.
  • Updated the tests accordingly.
  • Added a test for the XML file to make sure it respects the DTD.
  • Fixed the offsets of the data table, since these are unsigned short, -1 is not the best value to use to represent an invalid value. Instead we use USHRT_MAX now.
  • Completed the .us entries.

1.1.1

  • Added many TLDs as defined by the Public Suffix List.
  • Wrote a new test to check our data against the Public Suffix List.
  • Updated the existing tests to work with the new library.
  • Added a new category called Entrepreneurial (i.e. official domain names used to resale sub-domains; i.e. .uk.com.)

1.1.0

  • Added the version inside the tld.h header file.
  • Added a function to retrieve the version at run time (see tld_version().)
  • Added the debian packaging capability.
  • Fixed the documentation so it directly outputs in the correct folder.
  • Fixed the output folder name and file name for the documentation.
  • Added a printf() in each test to print out the library version.
  • Added a logo for the library. Nothing fancy at this point...
  • Added an SVG file presenting the functionality in one page.

1.0.0

  • First release.
Syndicate content

Snap! Websites
An Open Source CMS System in C++

Contact Us Directly