O'Reilly Databases

MySQL backups using ZFS snapshot

2008-04-11T20:34:15Z

One of the significant features in version 2.0 of Zmanda Recovery Manager for MySQL is MySQL backups using Solaris ZFS. Doing MySQL backups using
filesystem snapshots has minimal impact on the MySQL databases. The MySQL databases are not available for updates for less than a second and the application impact is not dependent on the size of the database.

ZRM 2.0 can be downloaded from Zmanda downloads page. It supports all Linux and Solaris distributions. The documentation is available on ZRM wiki.
ZRM forums can be used to get questions answered about the project.

This article shows an example of how to install, configure, backup and restore MySQL databases using Zmanda Recovery Manager (ZRM) for MySQL on OpenSolaris. The example takes advantage of ZFS snapshots to do full backups.

The example assumes that the ZRM server and MySQL server are the same machine. We are backing up MySQL database “myisamnetflix” to the same machine running Solaris Express Community Edition snv_77 on AMD platform.

ZRM for MySQL installation

* Installation has to be done as super user.

* ZRM for MySQL works with Perl 5.8.4 from SUNWperl584core package.

* Install perl-DBD and perl-XML-parser modules. SUNWperl-xml-parser is part for Solaris Express Community Edition.

* Download ZRM for MySQL Solaris packages from Zmanda downloads page.

* Install ZRM for MySQL (ZRM server package is sufficient because MySQL server and ZRM server are the same machine).
# gunzip MySQLZrm-2.0-SunOS5.10-noarch.pkg.gz # pkgadd -d MySQLZrm-2.0-SunOS5.10-noarch.pkg

MySQL server configuration

* Check to see if MySQL server is running. If MySQL server is not installed, please install SUNWmysqlr, SUNWmysqlt, SUNWmysqlu packages from the distribution. Update the “root” MySQL server with a password using mysqladmin command (/usr/sfw/bin/mysqladmin –user root password boot12). We are using “boot12″ as the root password. This user will be used for doing MySQL backups and restores. It is better to user a specific user with minimal privileges to do MySQL backups instead of using “root” MySQL user.

* The MySQL server has to run as “mysql” user and “mysql” OS user should belong to “mysql” group. The default installation of ZRM for MySQL requires MySQL server to run as “mysql” user.

* Enable binary logging on the MySQL server. Binary logging must be enabled to do incremental backups of the MySQL server.

* Edit /etc/my.cnf configuration file. Add “log-bin” in mysqld section
[mysqld] log-bin

* Create a ZFS filesystem as shown below. We will be storing MySQL data in /testpool/testfs directory.

# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT testpool 29.8G 628M 29.1G 2% ONLINE - # zfs list NAME USED AVAIL REFER MOUNTPOINT testpool 521M 23.8G 33.2K /testpool testpool/testfs 521M 23.8G 520M /testpool/testfs

* Change /etc/my.cnf configuration file so that datadir points to the ZFS filesystem.

[mysqld] datadir = /testpool/testfs/mysql/data/

* Restart MySQL server. The MySQL server is listening to default port 3036.

mysql 616 592 0 Mar 03 console 1:2:13 /usr/sfw/sbin/mysqld --basedir=/usr/sfw --datadir=/testpool/testfs/mysql/data/
* We have mysql database “myisamnetflix” that contains two tables. We will be backing this database. This database uses MyISAM storage engine


 mysql> use myisamnetflix;

 Reading table information for completion of table and column names

 You can turn off this feature to get a quicker startup with -A
 Database changed

 mysql> show tables;

 +-------------------------+

 | Tables_in_myisamnetflix |

 +-------------------------+

 | MovieID                 |

 | MovieRatings            |

 +-------------------------+

 2 rows in set (0.00 sec)

mysql> select count(*) from MovieID; +----------+ | count(*) | +----------+ | 17770 | +----------+
* MySQL client commands are installed in /usr/bin/ directory. If they are not, accordingly configure the client command location and binary log location in mysql-zrm.conf.

ZRM configuration

* This should be done as mysql user

$ id uid=100(mysql) gid=100(mysql)

* Create the backup set directory. The backup set is called “zfstest”.
$ mkdir /etc/mysql-zrm/zfstest

* Create mysql-zrm.conf configuration file. The “myisamnetflix” database is being backed up. The location of MySQL client commands have specified in “mysql-binpath” parameter. ZFS snapshot plugin has been specified as the backup method.
$ cat /etc/mysql-zrm/zfstest/mysql-zrm.conf mysql-binpath="/usr/sfw/bin" host="localhost" databases="myisamnetflix" password="boot12" user="root" snapshot-plugin="/usr/share/mysql-zrm/plugins/zfs-snapshot.pl"

Perform ZRM backups

* This should be done as mysql user

* Perform full backup of the database immediately using mysql-zrm-scheduler.

$ mysql-zrm-scheduler --now --backup-set zfstest --backup-level 0 schedule:INFO: ZRM for MySQL Community Edition - version 2.0 Logging to /var/log/mysql-zrm/mysql-zrm-scheduler.log backup:INFO: ZRM for MySQL Community Edition - version 2.0 zfstest:backup:INFO: START OF BACKUP zfstest:backup:INFO: PHASE START: Initialization zfstest:backup:INFO: backup-set=zfstest zfstest:backup:INFO: backup-date=20080326053921 zfstest:backup:INFO: mysql-server-os=Linux/Unix zfstest:backup:INFO: host=localhost zfstest:backup:INFO: backup-date-epoch=1206535161 zfstest:backup:INFO: mysql-zrm-version=ZRM for MySQL Community Edition - version 2.0 zfstest:backup:INFO: mysql-version=4.0.24-log zfstest:backup:INFO: backup-directory=/var/lib/mysql-zrm/zfstest/20080326053921 zfstest:backup:INFO: backup-level=0 zfstest:backup:INFO: backup-mode=raw zfstest:backup:INFO: PHASE END: Initialization zfstest:backup:INFO: PHASE START: Running pre backup plugin zfstest:backup:INFO: PHASE END: Running pre backup plugin zfstest:backup:INFO: PHASE START: Flushing logs zfstest:backup:INFO: PHASE END: Flushing logs zfstest:backup:INFO: PHASE START: Creating snapshot based backup zfstest:backup:INFO: File Position Binlog_do_db Binlog_ignore_db mysql-bin.045 4 zfstest:backup:INFO: innodb-data=/testpool/testfs/mysql/data/ibdata1; zfstest:backup:INFO: innodb-logs=/testpool/testfs/mysql/data/./ib_logfile* zfstest:backup:INFO: raw-databases-snapshot=myisamnetflix zfstest:backup:INFO: PHASE END: Creating snapshot based backup zfstest:backup:INFO: PHASE START: Find table type zfstest:backup:INFO: PHASE END: Find table type zfstest:backup:INFO: PHASE START: Calculating backup size & checksums zfstest:backup:INFO: next-binlog=mysql-bin.045 zfstest:backup:INFO: backup-size=261.87 MB zfstest:backup:INFO: PHASE END: Calculating backup size & checksums zfstest:backup:INFO: read-locks-time=00:00:01 zfstest:backup:INFO: flush-logs-time=00:00:00 zfstest:backup:INFO: backup-time=00:02:19 zfstest:backup:INFO: backup-status=Backup succeeded zfstest:backup:INFO: Backup succeeded zfstest:backup:INFO: PHASE START: Running post backup plugin zfstest:backup:INFO: PHASE END: Running post backup plugin zfstest:backup:INFO: PHASE START: Mailing backup report zfstest:backup:INFO: PHASE END: Mailing backup report zfstest:backup:INFO: PHASE START: Cleanup zfstest:backup:INFO: PHASE END: Cleanup zfstest:backup:INFO: END OF BACKUP /usr/bin/mysql-zrm started successfully

* Delete some entries from the “myisamnetflix” database (so that we can do incremental backup of the database). We are deleting all movies that start with “Sherlock”
mysql> use myisamnetflix; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A


 Database changed

mysql> delete from MovieID where MovieTitle regexp 'Sherlock*'; Query OK, 31 rows affected (0.13 sec)

* Perform incremental backup of the backup set.
$ mysql-zrm-scheduler --now --backup-set zfstest --backup-level 1

ZRM backup reports

* Use mysql-zrm-reporter to look at the status of backups available
$ /usr/bin/mysql-zrm-reporter --where backup-set=zfstest --show backup-status-info


 REPORT TYPE : backup-status-info

backup_set backup_date backup_level backup_status comment ----------------------------------------------------------------------------------------------------------- zfstest Wed Mar 26 07:10:26 2008 1 Backup succeeded ---- zfstest Wed Mar 26 05:39:21 2008 0 Backup succeeded ----

* ZRM reports can also provide information on impact on MySQL application.

$ /usr/bin/mysql-zrm-reporter --where backup-set=zfstest --show backup-app-performance-info


 REPORT TYPE : backup-app-performance-info

backup_set backup_date backup_level backup_size backup_time read_locks_time flush_logs_time ------------------------------------------------------------------------------------------------------------------------------------- zfstest Wed Mar 26 07:10:26 2008 1 0.00 MB 00:00:01 00:00:00 00:00:00 zfstest Wed Mar 26 05:39:21 2008 0 261.87 MB 00:02:19 00:00:01 00:00:00

Database recovery

* Use ZRM reporting tool to identify the location of MySQL backup images.
$ /usr/bin/mysql-zrm-reporter --where backup-set=zfstest --show restore-info


 REPORT TYPE : restore-info

backup_set backup_date backup_level backup_directory backup_status comment ----------------------------------------------------------------------------------------------------------------------------------------------------- zfstest Wed Mar 26 07:10:26 2008 1 /var/lib/mysql-zrm/zfstest/20080326071026 Backup succeeded ---- zfstest Wed Mar 26 05:39:21 2008 0 /var/lib/mysql-zrm/zfstest/20080326053921 Backup succeeded ----

* You can parse incremental backups to identify database events of interest. In our example, we will look for the “DELETE” event.

$ /usr/bin/mysql-zrm-parse-binlogs --source-directory /var/lib/mysql-zrm/zfstest/20080326071026 --mysql-binpath /usr/sfw/bin parse-binlogs:INFO: ZRM for MySQL Community Edition - version 2.0 ------------------------------------------------------------ Log filename | Log Position | Timestamp | Event Type | Event ------------------------------------------------------------ /var/lib/mysql-zrm/zfstest/20080326071026/mysql-bin.045 | 4 | 08-03-26 07:08:59 | Query | use myisamnetflix; delete from MovieID where MovieTitle regexp 'Sherlock Holmes*'; /var/lib/mysql-zrm/zfstest/20080326071026/mysql-bin.045 | 110 | 08-03-26 07:09:27 | Query | delete from MovieID where MovieTitle regexp 'Sherlock*'; /var/lib/mysql-zrm/zfstest/20080326071026/mysql-bin.045 | 209 | 08-03-26 07:10:26 | Rotate to mysql-bin.046 pos: 4 | ------------------------------------------------------------
* Restore the database from the full backup done at 05:39:21
$ /usr/bin/mysql-zrm-restore --mysql-binpath /usr/sfw/bin --user=root --password=boot12 --source-directory=/var/lib/mysql-zrm/zfstest/20080326053921 restore:INFO: ZRM for MySQL Community Edition - version 2.0 BackupSet1:restore:INFO: Restored database from raw backup: myisamnetflix BackupSet1:restore:INFO: Restore done in 336 seconds. MySQL server has been shutdown. Please restart after verification.

* Restart the MySQL server. ZRM for MySQL shuts down the database after recovery so that the contents can be verified.

* Verify database recovery

mysql> use myisamnetflix; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A

Now that you have read till the end, it is time to download and try.

Improved Snapshot interface for MySQL backups

2008-04-04T18:41:36Z

We have significantly improved the snapshot interface for doing MySQL backups using ZRM. This work has been released as part of ZRM 2.0. ZRM 2.0 has couple of snapshot plugins - Linux LVM and Solaris ZFS that uses the interface.

Changes in ZRM 2.0:
* Solaris packages
* ZRM clients for Linux (RPM/Debian) and Solaris
* Tested on Gentoo distribution
* Improved Snapshot plugin interface
* Solaris ZFS snapshot plugin
* Backup of remote servers using snapshots
* Asychronous checksum computation for improved backup performance
* Backup compression on the fly for logical backups

Download it from Zmanda downloads page and give it a try. I will write more about how to use the plugin interface next week.

Reporting MySQL Internals with Information Schema plug-ins

2008-02-10T23:04:51Z

Last week, I described how to use the MySQL plug-in API to write a minimal ‘Hello world!’ information schema plug-in. The main purpose of that plug-in is to illustrate the bare essentials of the MySQL information schema plug-in interface.

In this article, I’d like to take that to the next level and demonstrate how to write an information schema plug-in that can access some of the internals of the MySQL server. For this particular purpose, we will focus on a plug-in that reports all the SAVEPOINTs available in the current session. This MYSQL_SAVEPOINTS plug-in may be of some value when debugging scripts and stored routines that rely on complex scenarios using transactions and savepoints.

In a forthcoming article, I will describe a few information schema plug-ins that are arguably more interesting, such as a plug-in to list the currently existing TEMPORARY tables, user-defined variables, and the contents of the query cache. Although the plug-in described in this article may be of some use, its main purpose is to illustrate the minimal requirements for plug-ins that can access the server’s internals.

A Quick Recapitulation

You might recall that:

The MySQL plug-in API is one of the new features in MySQL 5.1, and forms a generic extension point of the MySQL database server, allowing privileged database users to add functionality to the MySQL Server by loading a shared library from the plug-in directory
Loading and unloading a plug-in is a completely dynamic process controlled using the INSTALL PLUGIN and UNINSTALL PLUGIN syntax, and does not involve compiling the server or even restarting it
There are several types of plug-ins, the most well-known being storage engines and full-text parsers. Less well-known types include information schema and daemon plug-ins.
An information schema plug-in provides the implementation of a table (or actually, a system view) in the information_schema database
Plug-ins are usually implemented in C/C++. To implement a plug-in, the implementor must include the header file plugin.h and provide an initialized instance of the st_mysql_plugin structure. In addition, the implementor must provide code to implement the plug-in type dependent part of the interface
The plug-in type dependent part of the interface for information schema plug-ins consists of two things: the column definitions of the information schema table and a fill_table function that is called whenever the server wants to retrieve the rows of data from that table.

How Information Schema plug-ins can access MySQL Server internals

Before we discuss the MYSQL_SAVEPOINTS information schema plug-in in detail, let’s take a look at the way information schema plug-ins can obtain access to the internals of the MySQL server.

Like I just recapitulated from last week’s article, the plug-in type dependent inferface for information schema plug-ins consists of two things:

An array of ST_FIELD_INFO structures, each of which defines a column of the information schema table
A fill_table function that is called by the server when it needs to retrieve the data from table

The column definitions may be considered a static part of the interface - they simply define the structure of the table - no more, no less. The fill_table function is a different matter. Let’s take a look at the signature of the signature of that function:

int fill_table(THD *thd, TABLE_LIST *tables, COND *cond);

The TABLE_LIST *tables argument provides the handle to the information schema table that is being filled, and the COND *cond argument represents the WHERE condition of the SQL statement that is currently being handled, allowing the fill_table function to directly filter rows (instead of relying on the query execution engine to do that). As such, these arguments are occupied with the actual delivery of rows of data to the server.

The first argument to the fill_table function offers all kinds of interesting opportunities to see what is going on inside the server. We will discuss it in more detail in the next section.

Public accessors to the current `THD` instance

The first argument to the fill_table function is THD *thd. This is the so-callled thread descriptor - something that is best thought of as a handle to the current session. Note that in a MySQL context, the terms connection, thread and session are often used interchangeably. However, I find the term thread too broad, and the term connection too narrow. As there are many parts in THD that maintain state regarding the events that occurred since a connection is established, it seems most sensible to think of THD as the server-side implementation of a session.

This session handle or thread descriptor has the form of a pointer to an instance of the THD class. The plugin.h header file contains a forward declaration to this class, but the actual declaration is contained in sql/sql_class.h. The THD class is one of the key data structures in understanding the workings of the MySQL server as it is passed as an argument to many internal server functions. Consequently it provides a wealth of possibilities to create interesting new information schema plug-ins. In fact, the number of possibilities are so great, that a number of common usages has been explicitly set aside in the plugin.h header file.

The plugin.h header file contains a number of function declarations and macros that provide access to the members of a THD instance. I will not discuss all of them here, but highlight just a few just to give you an idea:

thd_test_options() - Find out which options are set. This can be used to find out whether a number of boolean options like big_tables, (general and binary) logging, and autocommit are enabled or disabled.
thd_proc_info() - Should be used by the plug-in implementor before starting a potentially time-consuming operation so the rest of the world can monitor what this session doing. The code set here corresponds to the value reported in the State column by the SHOW PROCESSLIST statement
thd_killed() - Can be used by the plug-in implementor to find out if the thread in which this session lives was killed. If the plug-in is involved in a potentially time-consuming process, the plugin-in implementor should periodically check this and gracefully abort the plugin-ins work when it detects that the thread was killed.
thd_alloc() - Allocates some memory from this session’s memory pool. If the plug-in requires some small amount of memory, plug-in implementors should use this rather than the standard malloc() function. Calling thd_alloc(); is likely to be faster because it takes memory out of a pre-allocated pool, reducing contention. In addition, it is more convenient because the memory need not be explicitly freed: it is automatically reclaimed by the pool after handling the current statement.

If you are interested in seeing all these declarations, just open plugin.h and look for comments like this:

/*************************************************************************
  Miscellaneous functions for plugin implementors
*/

`plugin.h` describes a public interface

In plugin.h, the declarations as described in the previous section together form a public interface to the current session. They are there for the convenience of plug-in implementors and represent a ’safe’ way to work with the THD pointer passed to the fill_table function.

To say that these form a public interface is to stay that these are officially supported by MySQL AB. That is: they will be supported officially once the MySQL 5.1 Server is a generally available release. From that point on you can rely on these functions when writing plug-ins in the sense that you do not have to be afraid that they will change. At least, the public interface will remain the same for all forthcoming builds of the 5.1 server. Any interface changes in future releases will involve a proper process, giving everybody the chance to update their code well in time.

Unfortunately, not every function declaration in plugin.h has source code comments. This means that for now, you sometimes need to do some digging in the server’s source code to find out what you can do with them. I admit that this situation is not exactly perfect. However, the matter has been reported as a bug, and hopefully, it will be adressed soon.

Beyond the public interface

I just described the public interface plug-in implementors can rely on. A distinct advantage of the public interface is that it takes away a lot of the complexity of the underlying internals of the MySQL Server. However, there will always be cases where the public interface does not offer the features you really need. In those cases, you simply need to be able to work directly on the server internals.

The advantage of directly referencing the server’s internals is that you can access all the interesting nuts and bolts and bits and pieces. The downside is that there is absolutely no guarantee that your code will work in another version of the server. The internals are by definition the parts that are not meant to be exposed. As such, it is possible that your code does not work or behaves unexpectedly in another version of the server.

Let’s not dwell too long on the disadvantages. Instead, let’s focus on the merits of pluggable information schema tables. Granted, it is inconvenient that we may need to make our code resilient to each different build of the server. However, for many applications, it is not very likely that we have to constantly do that.

Even if we do have to change our code, the burden will be on the developer of the plug-in. For each specific build of the server, your code may need to be different. Even if the code itself does not change, you will probably at least have to recompile your plug-in for each specific build of the server. However, your users are still not required to recompile the server itself. They can still install the plug-in without stopping or restarting the server, which in many cases seems more important than bearing the burden of changing the code.

You need to break some eggs to bake an omelet - so if you’re hungry, you better get over it and start breaking some eggs ;-)

The `MYSQL_SERVER` define

In order to access the server’s internals beyond the public interface, we need to use some C/C++ preprocessor magic and define MYSQL_SERVER. This define needs to be present before we include any MySQL header (or source) files:

#ifndef MYSQL_SERVER
#define MYSQL_SERVER
#endif

Throughout the MySQL codebase, there are many sections that are conditionally included or excluded depending on whether MYSQL_SERVER is defined. It is hard to pinpoint the exact effect of adding this definition, because there many spots that use this definition to control conditional compilation.

Normally the MYSQL_SERVER definition need be present only when compiling the server proper, but in this case we need it to let the plug-in code work with internal structures such as THD instances directly, that is, without using the accessors provided by the public interface.

To be absolutely clear: using the MySQL_SERVER define in your code does not mean you must compile your plug-in as part of the server. On the contrary - you can compile your plug-ins separately from the server, and still (un)install them at runtime. The only thing the MySQL_SERVER define does, is pull in the declarations that are normally considered to be ‘internal’. They will for example allow us to work directly with the members of the THD class instead of being required to use the public accessors defined in plugin.h.

Implementing the `MYSQL_SAVEPOINTS` Information Schema plug-in

Now that we sketched the backgrounds, we can quickly proceed and discuss the implementation of the MYSQL_SAVEPOINTS information schema plug-in. (Note that you can download the source code file mysql_is_savepoints.cc here.)

Most of the things are rather similar to what was described in the article describing the MYSQL_HELLO plug-in, for which you can still download the mysql_is_hello.cc source code.

We will do like we did last week and assume the following things are in place on your system:

g++, the GNU C++ compiler
The MySQL 5.1.22 source distribution - we need to include some of the header files
A text editor or IDE (like Eclipse with CDT)

Creating the source file

First, we need to create a C++ source file. We will assume that the working directory is ~/mysql_is_savepoints/, and that the source file is called mysql_is_savepoints.cc.

The MYSQL_SERVER define

Like we explained in the previous sections, we need to define MYSQL_SERVER so we can directly reference the members of the THD class passed to our fill_table function.

#ifndef MYSQL_SERVER
#define MYSQL_SERVER
#endif

Because this affects how the included files are processed, we do this at the very top of our source file.

Include files

We can use the same list of includes we used for the MYSQL_HELLO plug-in - the MYSQL_SERVER define is responsible for including all the additional things we require to write the MYSQL_SAVEPOINTS plug-ins.

#include <mysql_priv.h>
#include <stdlib.h>
#include <ctype.h>
#include <mysql_version.h>
#include <mysql/plugin.h>
#include <my_global.h>
#include <my_dir.h>

Defining the columns

For the MYSQL_SAVEPOINTS plug-in, we will define two columns: SAVEPOINT_ID and SAVEPOINT_NAME. At the SQL level, it will look something like this:

+----------------+-------------+------+-----+---------+-------+
| Field          | Type        | Null | Key | Default | Extra |
+----------------+-------------+------+-----+---------+-------+
| SAVEPOINT_ID   | bigint(0)   | NO   |     | 0       |       |
| SAVEPOINT_NAME | varchar(64) | NO   |     |         |       |
+----------------+-------------+------+-----+---------+-------+

…and this is what it looks like in the C/C++ source file:

#define COLUMN_SAVEPOINT_ID 0
#define COLUMN_SAVEPOINT_NAME 1

static ST_FIELD_INFO mysql_is_savepoints_field_info[]=
{
  {"SAVEPOINT_ID", 0, MYSQL_TYPE_LONGLONG, 0, 0, "Savepoint Id"},
  {"SAVEPOINT_NAME", 64, MYSQL_TYPE_STRING, 0, 0, "Savepoint Name"},
  {NULL, 0, MYSQL_TYPE_NULL, 0, 0, NULL, 0}
};

This time, in addition to creating the ST_FIELD_INFO array of column definitions, we also create #defines for the array entry indexes. The defines allow us to refer to the column definitions using the names rather than the raw, literal integer array indexes. This has the advantage that we do not have to change code should we want to change the positions of the columns. Another advantage is that our fill_table code will be easier to read: by consistently referring to COLUMN_SAVEPOINT_ID and COLUMN_SAVEPOINT_NAME rather than 0 and 1 it will be much easier to see what is going on.

Filling the table

Now we come to the heart of the matter: generating a row for each SQL SAVEPOINT available in the current session.

The savepoints for the current session are available in the transaction member of the THD class. The transaction member is an instance of the st_transactions struct, which is declared locally inside the THD class:

class THD :public Statement,
           public Open_tables_state
{

...many, many lines here...

public:

  struct st_transactions {
    SAVEPOINT *savepoints;

    ...a few more lines here...

  } transaction;

...many, many more lines here ...

};

Now you might recall that the THD class is declared in sql/sql_class.h. However, you might have some trouble locating the transaction member, because the declaration of the THD class is extremely large and long-winded: in the MySQL 5.1.22-rc source distribution, it ranges from lines 960 to 1886(!!) - and those 900 something lines make up only the declaration!

(Although the official explanation for the name of the THD class is that it is an acronym for THread Descriptor, some developers* explained that it is one of the few class names that is spelled in capitals because it is so incredibly heavy. According to this anecdote, its name should be pronounced as “…THUD!!…THUD!!…” because of the sound it makes each time it is dumped into the argument list of a function that makes up the servers source code.

* = Thanks to Eric Herman for painting this creative and tangible likeness ;-)

Anyway, you will find it easier when you look for st_transactions, or go directly to line 1149, but note that the line number is likely to be different in other versions of the server code.

Now, we can see that the st_transaction struct contains a pointer to a SAVEPOINT pointer called savepoints. As we shall see later, this is actually the list of savepoints we need. But what kind of type is this SAVEPOINT exactly?

Well, to get past this point, you really need some patience and a set of tools that allow you to search the source code. In the case of SAVEPOINT, it turns out that this is actually a typedef for the st_savepoint structure. Now, the odd thing is that this typedef appears in sql/handler.h:

typedef struct st_savepoint SAVEPOINT;

But the structure st_savepoint itself is declared in sql/sql_class.h - that is, the same file that declares THD, which seems to prefer SAVEPOINT rather than st_savepoint!

Well - it is beyond me why it was done like this. For our purpose it doesn’t really matter though, let’s examine the declaration of st_savepoint instead:

struct st_savepoint {
  struct st_savepoint *prev;
  char                *name;
  uint                 length, nht;
};

Here we can see that each st_savepoint has a char * member called name, which is presumably whatever name the user provided in the savepoint syntax:

mysql> SAVEPOINT my_savepoint;

So in this case, the name member of the st_savepoint instance corresponding to this SQL SAVEPOINT will point to the character string "my_savepoint".

Apart from the name we can also see that each st_savepoint has itself a pointer to another st_savepoint called prev. This suggests a single linked list of savepoints.

This is about all the information we need to implement the fill_table function. So, here it is:

int mysql_is_savepoints_fill_table(THD *thd, TABLE_LIST *tables, COND *cond)
{
  int status = 0;                           /* return value for this func, 0=success, 1=error*/
  CHARSET_INFO *scs= system_charset_info;   /* need this to store field into table */
  TABLE *table= (TABLE *)tables->table;     /* handle to the I_S table. class declared in table.h */
  uint savepoint_id = 0; 

  SAVEPOINT *sv= thd->transaction.savepoints;

  while(sv && !status)
  {
    /* store the savepoint sequence into the table column */
    table->field[COLUMN_SAVEPOINT_ID]->store(++savepoint_id, 0);
    /* store the savepoint name into the table column */
    table->field[COLUMN_SAVEPOINT_NAME]->store(sv->name, strlen(sv->name), scs);

    status= schema_table_store_record(thd, table);
    sv= sv->prev;
  }
  return status;
}

The biggest difference with the fill_table function we used in the MYSQL_HELLO example is that instead of just storing one single row, we have a loop, storing one row for each iteration. The loop is initialized by assigning the SAVEPOINT pointer from the transaction member from the THD instance that is passed as the first argument to the fill_table function to a local sv variable:

SAVEPOINT *sv= thd->transaction.savepoints;

Of course, it is possible that there are no savepoints in the current session, in which case thd->transaction.savepoints will be the NULL pointer. However, if there are savepoints, a pointer to the last savepoint that was created in the current session will now be stored in sv. We can now set up the actual loop:

  while(sv && !status)
  {

    ...lines here...

    status= schema_table_store_record(thd, table);
    sv= sv->prev;
  }

Note that the loop will be entered only if sv points to a savepoint. If it does, data from the savepoint is written to the columns of our information schema table.

In the bottom of the loop, we store the current record using the schema_table_store_record, which we discussed already for the MYSQL_HELLO example:

    status= schema_table_store_record(thd, table);

Interestingly, we were required to make a forward declaration to it in the MYSQL_HELLO example. Now, we don’t have to do this, presumably because we defined MYSQL_SERVER.

You might recall that schema_table_store_record function returns 0 in case of success and 1 instead of failure. Note that if a failure occurs at this point, the loop will not iterate again, as the while condition requires status to be not true (that is, zero).

After storing the row, the last step of the loop is to move back and examine the previous savepoint:

    sv= sv->prev;

If the end of the list is reached, sv will be NULL, preventing the loop to iterate again. However, if there is in fact a previous savepoint, the loop will run once again and create a new row for that savepoint too, on and on until we reach the end of the list of savepoints.

In the top of the loop, we store data into the columns of our information schema table:

    /* store the savepoint sequence into the table column */
    table->field[COLUMN_SAVEPOINT_ID]->store(++savepoint_id, 0);
    /* store the savepoint name into the table column */
    table->field[COLUMN_SAVEPOINT_NAME]->store(sv->name, strlen(sv->name), scs);

This time, we use our defines COLUMN_SAVEPOINT_ID and COLUMN_SAVEPOINT_NAME instead of the literal numerical field indexes. We already demonstrated in the MYSQL_HELLO example how to store a string, so we won’t discuss the line that stores the savepoint’s name. Instead, let’s find out how we can store an integer value by looking at the line that stores the savepoint id.

As you can see, we stipulate the value for the SAVEPOINT_ID column ourselves by simply adding one for each row:

    table->field[COLUMN_SAVEPOINT_ID]->store(++savepoint_id, 0);

Savepoints by no means have a numerical ID of their own, but it makes sense to make one up in order to unambigously indicate the order in which the savepoints were created during this session. Note the second argument to the store method, which is always 0 here:

    table->field[COLUMN_SAVEPOINT_ID]->store(++savepoint_id, 0);

This second argument is there to tell the store method whether the value represents a signed or an unsigned value. In this case, we are storing an unsigned value - it should be 1 for an unsigned value.

The rest of the implementation

The remainder of the implementation is quite similar to what was discussed for the MySQL_HELLO example. The most important difference is actually the plug-in name, but otherwise the implementation is identical. Therefore, it is not discussed here further.

Building and Installing

The build and install process is pretty much similar to that for the MYSQL_HELLO plug-in.

Compiling

We can compile the plug-in like this:

g++ -DMYSQL_DYNAMIC_PLUGIN -Wall -shared
-I/home/user/mysql-5.1.22-rc/include
-I/home/user/mysql-5.1.22-rc/regex
-I/home/user/mysql-5.1.22-rc/sql
-o mysql_is_savepoints.so mysql_is_savepoints.cc

This will create the shared library mysql_is_savepoints.so.

Installing the plug-in

You might recall that the shared library needs to be moved to the plug-in directory. After that, we can install the plug-in using the INSTALL PLUGIN syntax:

mysql> INSTALL PLUGIN MYSQL_SAVEPOINTS soname 'mysql_is_savepoints.so';
Query OK, 0 rows affected (0.00 sec)

Using the plug-in

Now we can finally see our plug-in in action. At first, there will be no savepoints present:

mysql> SELECT * FROM information_schema.MYSQL_SAVEPOINTS;
Empty set (0.02 sec)

Even if we set one, we won’t see it immediately:

mysql> SAVEPOINT A;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM information_schema.MYSQL_SAVEPOINTS;
Empty set (0.00 sec)

This is beause by default, the session has autocommit enabled. As each statement is wrapped in its own transaction, we will never be able to see any savepoints. So we disable autocommit:

mysql> SET autocommit = OFF;
Query OK, 0 rows affected (0.00 sec)

And now we can really witness the behaviour of our plug-in:

mysql> SAVEPOINT A;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM information_schema.MYSQL_SAVEPOINTS;
+--------------+----------------+
| SAVEPOINT_ID | SAVEPOINT_NAME |
+--------------+----------------+
|            1 | A              |
+--------------+----------------+
1 row in set (0.00 sec)

mysql> SAVEPOINT B;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM information_schema.MYSQL_SAVEPOINTS;
+--------------+----------------+
| SAVEPOINT_ID | SAVEPOINT_NAME |
+--------------+----------------+
|            1 | B              |
|            2 | A              |
+--------------+----------------+
2 rows in set (0.00 sec)

mysql> ROLLBACK TO SAVEPOINT A;
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT * FROM information_schema.MYSQL_SAVEPOINTS;
+--------------+----------------+
| SAVEPOINT_ID | SAVEPOINT_NAME |
+--------------+----------------+
|            1 | A              |
+--------------+----------------+
1 row in set (0.00 sec)

Learn More

This has been quite a ride! In this article it was demonstrated how we can use information schema plug-ins to report some of the things that are going on inside the current session. As such, the MYSQL_SAVEPOINTS plug-in is a big step forward compared to the MYSQL_HELLO plug-in.

However, this is just the tip of the iceberg - the current session harbours much more interesting information about the current session, and in a forthcoming article I will demonstrate a number of other usages. In particular, I will present a plug-in to report the user variables in the current session, and the temporary tables defined in the current session.

In another article, we will also see that it is sometimes possible to look beyond the current session and report on the status of server-wide structures, such as the query cache.

Meet us at the user’s conference

When I discussed the MYSQL_HELLO plug-in, I already hinted that there will be a lot there for those people that want to learn more about extending the server with (information schema) plug-ins. You can find all those links in the bottom of that article.

In addition, you can learn a lot about the MySQL Server internals. And…you can learn it from one of the founding fathers: Monty himself will be doing A tour into MySQL’s internals. So, I guess that’s going to be one of those occasions where you get the opportunity to clear up some of those details in the source code you never quite managed to wrap your head around.

If you register before the 26th of februari, you’ll get a $200 discount. There are more discounts available depending on whether you attended before, or if you register together with a number of colleagues; there’s special student and non-profit discounts too - check it out here.

See you at the conference! (Bonus points for the first one to ask Monty in the Q&A why SAVEPOINT is typedef-ed in sql/handler.h instead of sql/sql_class.h; double bonus points for the first one to ask Monty if THD is really called like that because it is so heavy ;-)

MySQL Information Schema Plugins: the best kept secret of MySQL 5.1

2008-02-01T13:01:08Z

MySQL 5.1 offers an extremely useful feature called information_schema plugins. This feature allows dynamic runtime loading of a shared library into the MySQL server to implement a table in the information_schema database. The SQL standard (ISO/IEC 9075-11:2003) allows database implementations to extend the information_schema. MySQL 5.1 transfers the possibility to do this directly to privileged database users so they can extend the information_schema themselves, in any way they see fit.

In this article, we will demonstrate how to create a minimal “Hello, World!” MySQL information schema plugin. In a forthcoming article, we’ll demonstrate how information schema plugins may be used to report some of the server’s internals such as the contents of the query cache, session level objects such as the currently defined TEMPORARY tables, user-defined variables and SAVEPOINTs.

The MySQL Plug-in API

Information Schema plug-ins are a subfeature of the MySQL plug-in API.
The plug-in API is one of the new features in the upcoming release of the MySQL database server, MySQL 5.1 (which is currently a release candidate). In essence, the MySQL plugin API provides a generic extension point to the MySQL server. It allows users to load a shared library in order to add new functionality to the server.

Plug-ins can be loaded and unloaded using the MySQL specific INSTALL PLUGIN and UNINSTALL PLUGIN syntax respectively. A key feature is that this process is completely dynamic - the server need not be re-compiled and need not be stopped in order to benefit from the functionality of a new plugin. Hence, new functionality can be added without suffering any downtime.

In some respects, the new plugin feature resembles the since long supported user-defined function (UDF) feature. Both plugins and UDFs involve dynamically loading a shared library to extend the server’s functionality. Like UDFs, plug-ins are usually written in C/C++. The difference between UDFs and plug-ins is that the UDF feature can be used only for adding new functions to use within the server’s SQL dialect. The concept of a plug-in is more broadly applicable and can be used to extend the server in more ways.

Currently, plug-ins are not supported on Microsoft Windows. MySQL is working to lift this limitation but it is at present unclear when this feature will be available for windows.

Currently supported plug-in types

Currently, the MySQL plug-in API supports the following types of plugins:

Storage engine: can be used to implement special-purpose row stores for data, which can then be accessed through SQL. Arguably, the Pluggable Storeage Engine Interface is one of the key benefits of MySQL 5.1
Full-Text parser: can be used for custom indexing of text-data as well as specialized handling of FULLTEXT query-expressions.
Daemon: a daemon plug-in can be used to execute a background process.
Information Schema table: are used to implement a ‘virtual’ table in the MySQL information_schema to report the status of for example the operating system or the server’s internals

Currently the plug-in API does not provide support for UDFs but it is expected that in due time, the current user-defined function feature will be merged into the plug-in API.

A closer look at the plug-in API

(Note: in this article I will repeatedly refer to a number of C/C++ header and source files that are part of the MySQL 5.1.22-rc source distribution. Any path that starts with mysql-5.1.22-rc/ should be taken to mean the corresponding path beneath the root of an unpacked MySQL 5.1.22-rc source distribution.)

For all plug-in types, the interface comprises a generic plug-in description structure, which is a struct called st_mysql_plugin. This struct is defined in the plugin.h header file, located in the mysql-5.1.22-rc/include/mysql directory.

The declaration of this structure is as follows:

/*
  Plugin description structure.
*/

struct st_mysql_plugin
{
  int type;             /* the plugin type (a MYSQL_XXX_PLUGIN value)   */
  void *info;           /* pointer to type-specific plugin descriptor   */
  const char *name;     /* plugin name                                  */
  const char *author;   /* plugin author (for SHOW PLUGINS)             */
  const char *descr;    /* general descriptive text (for SHOW PLUGINS ) */
  int license;          /* the plugin license (PLUGIN_LICENSE_XXX)      */
  int (*init)(void *);  /* the function to invoke when plugin is loaded */
  int (*deinit)(void *);/* the function to invoke when plugin is unloaded */
  unsigned int version; /* plugin version (for SHOW PLUGINS)            */
  struct st_mysql_show_var *status_vars;
  struct st_mysql_sys_var **system_vars;
  void * __reserved1;   /* reserved for dependency checking             */
};

Through this structure, the plug-in implementor provides the following things:

A number of data members that convey some meta data about the plug-in. These members are used to list the plug-in in the information_schema.PLUGINS system view once it is loaded. Among these is the member char *name. The value that the plug-in implementor provides for the name member is the name that is to be used for the INSTALL PLUGIN and UNINSTALL PLUGIN SQL syntax. For information schema plug-ins, the name member is also used as the table name of the information schema table implemented by the plug-in.
Optionally, an array of system variables through which the plug-in may be controlled, and an array of status variables so the plug-in may report its (dynamic) status to the server.
pointers to functions that are called upon when the plug-in is loaded (the plugin_init function) and unloaded (the plugin_deinit function)
A pointer to an object or structure that provides the actual functionality to implement a plug-in of the specific type

The type-dependent part of a plug-in implementation is usally a struct containing a number of function pointers (’hooks’) which are called in a particular sequence that is appropriate for that particular type of plug-in.

Information Schema Plugins

So what is an information_schema plug-in exactly? Well, the information_schema is a virtual database which is primarily intended as a meta data facility: it acts as a container for a collection of read-only ‘tables’ or rather system views that serve to provide data about the database server itself. As such, it is defined as part of the SQL standard (ISO/IEC 9075-11:2003).

The SQL standard describes a number of views that should appear in the information_schema, and mysql provides partial built-in support for these standard information schema views. The standard also expressly allows database implementors to extend the information_schema by adding new views, or extending the specified tables by adding columns. MySQL information schema plug-ins simply form an interface to allow privileged database users to extend the information schema themselves by writing their own information schema table implementations.

The type-specific API for Information Schema plug-ins

The type-specific part of the plug-in API for information schema plug-ins is formed by the struct called ST_SCHEMA_TABLE, which is defined in mysql-5.1.22-rc/sql/table.h.

When the plug-in is loaded, the server calls the plugin_init function associated with the int (*init)(void *) member of the st_mysql_plugin struct. When called, the server passes a pointer to an instance of a ST_SCHEMA_TABLE struct to the init function. Inside the plugin_init function, it is expected that the plug-in implementor initializes two members of that structure:

fields_info - an array of ST_FIELD_INFO structures. Each entry in this array corresponds to a column in the information schema table. The ST_FIELD_INFO structure is also declared in mysql-5.1.22-rc/sql/table.h
fill_table - a pointer to a function that is called whenever the server wants to obtain rows from the information_schema table

The ST_SCHEMA_TABLE struct has more members still, but only these two need to be initialized by the plug-in implementor. As long as the plug-in is loaded, the information_schema contains a new table that has a definition corresponding to the column definitions provided by the array assigned to the fields_info member. Whenever that table is queried, the function assigned to the fill_table member is called to actually construct the table’s rows.

These two elements really is all there is to the specific API for information schema plug-ins.

Writing an Information Schema Plugin

In this section, I will demonstrate how to write a minimal “Hello World!” MySQL information schema plugin. If you like, you can download the C++ source code.

Apart from a text editor, writing a simple, bare bones information schema plug-in requires no more than a C++ compiler and a number of MySQL’s C/C++ header files.

The following examples assume a GNU/Linux environment, a simple text editor, the g++ compiler and the MySQL header files. In order to get the necessary header files it is best to obtain a MySQL source distribution.

(For the purpose of this article, we will assume MySQL 5.1.22-rc. Until MySQL 5.1 is generally available, you are advised to always compile your plug-ins using only the header files shipped with the version of the product whereto the plug-in will be deployed.)

Creating the source file

First, we need to create a C++ source file. We will assume that the working directory is ~/mysql_is_hello/, and that the source file is called mysql_is_hello.cc

Includes

In the top of our source file, we need the include the following header files:

#include <mysql_priv.h>
#include <stdlib.h>
#include <ctype.h>
#include <mysql_version.h>
#include <mysql/plugin.h>
#include <my_global.h>
#include <my_dir.h>

The inclusion of mysql/plugin.h is most important, but currently the other includes are pulled in at some point. So far, this set of includes seems to do the trick.

Defining the columns

It was just mentioned that at some point, the plug-in implementor must provide an array of ST_FIELD_INFO structures that defines the columns of the information schema table. For this example, we’ll set up a minimal table definition consisting of one VARCHAR(64) column in order to show a “Hello, world” message. If we flash-forward for a moment - this is the structure of the table we’d like to define:

mysql> desc information_schema.MYSQL_HELLO;
+-------+-------------+------+-----+---------+-------+
| Field | Type        | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| HELLO | varchar(64) | NO   |     |         |       |
+-------+-------------+------+-----+---------+-------+
1 row in set (0.00 sec)

In order to achieve that, we need to declare our ST_FIELD_INFO array like this:

static ST_FIELD_INFO mysql_is_hello_field_info[]=
{
  {"HELLO", 64, MYSQL_TYPE_VARCHAR, 0, 0, "Hello", NULL},
  {NULL, 0, MYSQL_TYPE_NULL, NULL, NULL, NULL, NULL}
};

The last entry of this array is a dummy that serves as a marker for the end of the array. It is very important to always conclude the array with one such entry. Without such a trailing entry, the server does not know where the array ends. This would of course be a very bad thing, and is likely to result in a crash as soon as the plugin is loaded or the information schema table is accessed.

Now, the first ST_FIELD_INFO entry in the array defines the actual column. When we look at the declaration of ST_FIELD_INFO in mysql-5.1.22-rc/sql/table.h we get a better idea of what an individual column definition is made of:

typedef struct st_field_info
{
  const char* field_name;
  uint field_length;
  enum enum_field_types field_type;
  int value;
  uint field_flags;        // Field atributes(maybe_null, signed, unsigned etc.)
  const char* old_name;
  uint open_method;
} ST_FIELD_INFO;

field_name - this is used as column name
field_length - for string-type columns, this is the maximum number of characters. Otherwise, it is the ‘display-length’ for the column
field_type - a value drawn from enum_field_types. This enum is declared in mysql-5.1.22-rc/include/mysql_com.h and denotes a data type for the column. For the most part, there seems to be one entry in the enum for each SQL data type, although there seem to be a number of additional entries in the enum.
value - Unfortunately, I haven’t got the faintest idea what this is supposed to do. It does not seem to be used by any of the built-in information schema tales
field_flags - This is used to set column attributes. By default, columns are NOT NULL and SIGNED, and you can deviate from the default by setting the appopriate flags. You can use either one of the flags MY_I_S_MAYBE_NULL and MY_I_S_UNSIGNED or combine them using the bitwise or operator |. Both flags are defined in mysql-5.1.22-rec/sql/table.h.
old_name - I believe this is used by the built-in information schema tables to denote the column name for the corresponding SHOW statement. It’s not really applicable to plug-ins I guess, but I always provide a value here, usually a lower case version of the value for the name member
open_method - this should be one of SKIP_OPEN_TABLE, OPEN_FRM_ONLY or OPEN_FULL_TABLE. I admit I don’t really understand when to choose which option here. It seems likely that it defines in what manner the server must interact with the table, but I can’t boast any detailed knowledge in this matter. However, I do know that we can simply use SKIP_OPEN_TABLE for this simple example.

Filling the table

We explained that apart from defining the table columns, the plug-in implementor must also provide a function to actually deliver the rows. Before we can implement the actual fill_table function, we first need a forward declaration of a function that stores one single row in an information schema table:

bool schema_table_store_record(THD *thd, TABLE *table);

This function is defined in mysql-5.1.22-rc/sql/sql_show.cc. It’s role will become clear in a moment.

Now we can create the actual fill_table function:

int mysql_is_hello_fill_table(
  THD *thd
, TABLE_LIST *tables
, COND *cond
)
{
  int status;
  CHARSET_INFO *scs= system_charset_info;
  TABLE *table= (TABLE *)tables->table;

  const char *str = "plugin: hello, information_schema!!!";
  table->field[0]->store(
    str
  , strlen(str)
  , scs
  );

  status = schema_table_store_record(
    thd
  , table
  );

  return status;
}

The server passes a number of arguments to the fill_table function:

THD *thd - this is a pointer to an instance of the thread descriptor class. In practice, you can think of this as a direct handle to the current session. The THD is declared in mysql-5.1.22-dev/sql/sql_class.h.
TABLE_LIST *tables - This is a list of st_table objects. The first entry in that list corresponds to the runtime representation of the table we are implementing as it appears in a query. TABLE_LIST and st_table are defined in mysql-5.1.22-rc/sql/table.h. From what I could infer, it seems that in many cases TABLE (which is a typedef of st_table defined in mysql-5.1.22-dev/sql/handler.h) is used rather than st_table.
COND - This is used to pass the instance of the Item class that holds the internal representation of the WHERE-clause. This could then be used by the plugin to return only the rows that are required as per the WHERE condition - something like a pushed down condition. We will ignore this argument for now - the plug-in implementor may use it to optimize the fill_table function, but is not required to do so. So ignoring this argument will not lead to wrong results.

The fill method first obtains a handle to the runtime representation of the information schema table proper:

  TABLE *table= (TABLE *)tables->table;

Note that TABLE or st_table structure is the runtime representation of a table, i.e. a table as it appears in a SQL query. This is different from the ST_SCHEMA_TABLE structure which is used to define an information schema table.

We then write the message "plugin: hello, information_schema!!!" to the table’s column:

  char *str = "plugin: hello, information_schema!!!";
  table->field[0]->store(
    str
  , strlen(str)
  , scs
  );

So, each TABLE structure (a.k.a st_table) has an array of Field instances which is the runtime representations of table’s columns. Here, one of the store methods is called on the Field entry at the 0th index in the field array. Note that the Field at the 0th index corresponds to the 1st column in our SQL table.

Like we just saw for ST_SCHEMA_TABLE versus TABLE, the Field class is a runtime representation of a table column, not to be confused with the ST_FIELD_INFO which merely defines a column.

The Field class is declared in mysql-5.1.22-rc/sql/field.h. It declares a number of overloaded methods to store a value into the current row. In this example we use a variant that is appropriate to store a character string:

/* Store functions returns 1 on overflow and -1 on fatal error */
virtual int  store(const char *to, uint length,CHARSET_INFO *cs)=0;

Finally, we get to call the schema_table_store_record function. This finalizes the process of storing a row into the information schema table. This method returns either 0 (in case of success) or 1 (in case of error). Because our example plugin stores just one row, we can simply conclude by returning the result of schema_table_store_record as the result of our fill_table function.

In most practical applications, one would likely have a loop to repeatedly store a row in the table, and one would have to interrupt the normal completion of that loop as soon as the schema_table_store_record returns something not equal to 0.

Putting together the plug-in type-specific implementation

At this point we’ve taken care of the individial elements to implement an information schema table: we created a ST_FIELD_INFO array to define the columns of the table, and we provided an implementation for the fill_table function to actually deliver the rows. What we still need to do though is hook these elements up with the ST_SCHEMA_TABLE structure that forms the type-specific part of an information schema plug-in.

Like mentioned before, a pointer to a ST_SCHEMA_TABLE instance is passed to the plug-in’s init_plugin function. We can now implement the init_plugin function for our specfic plug-in and set it up to use our fields_info and fill_table implementations:

static int mysql_is_hello_plugin_init(void *p)
{
  ST_SCHEMA_TABLE *schema= (ST_SCHEMA_TABLE *)p;

  schema->fields_info= mysql_is_hello_field_info;
  schema->fill_table= mysql_is_hello_fill_table;

  return 0;
}

As we can see, there is very little to do here - we simply assign our implementations to the appropriate members of the ST_SCHEMA_TABLE instance, and return 0 to indicate a successful initialization.

We can immediately take care of the plugin_deinit function too:

static int mysql_is_hello_plugin_deinit(void *p)
{
  return 0;
}

In this case, we can get away with this simple dummy implementation.

In real-world examples, a plug-in might require some resource like memory or a file. In these cases, the plugin_init function would claim these resources, and the plugin_deinit function would be used to free up those resources again.

Putting together the generic plug-in implementation

With the previous step, we concluded the process of creating the plug-in type specific implementation for an information_schema plug-in. The final touch is that we have to hook this up to the generic part of plug-in interface, that is, we have to provide a st_mysql_plugin structure for our plug-in.

Rather than doing so directly, we use the predefined macros mysql_declare_plugin and mysql_declare_plugin_end for that. (Both these macros are defined in mysql-5.1.22-rc/include/mysql/plugin.h):

struct st_mysql_information_schema mysql_is_hello_plugin=
{ MYSQL_INFORMATION_SCHEMA_INTERFACE_VERSION };

mysql_declare_plugin(mysql_is_hello)
{
  MYSQL_INFORMATION_SCHEMA_PLUGIN,                 /* type constant    */
  &mysql_is_hello_plugin,                          /* type descriptor  */
  "MYSQL_HELLO",                                   /* Name             */
  "Roland Bouman (http://rpbouman.blogspot.com/)", /* Author           */
  "Says hello.",                                   /* Description      */
  PLUGIN_LICENSE_GPL,                              /* License          */
  mysql_is_hello_plugin_init,                      /* Init function    */
  mysql_is_hello_plugin_deinit,                    /* Deinit function  */
  0x0010,                                          /* Version (1.0)    */
  NULL,                                            /* status variables */
  NULL,                                            /* system variables */
  NULL                                             /* config options   */
}
mysql_declare_plugin_end;

We pass mysql_is_hello to the mysql_declare_plugin macro, and end our plug-in descriptor with the mysql_declare_plugin_end. As far as I can see, these macros are there to take care of some plumbing to allow definition of multiple plug-ins within the same source file.

An important element in putting together the plug-in descriptor is the assignment of the plugin_init and plugin_deinit functions which we discussed in the previous section. Assigning them here to the appropriate members of the plug-in descriptor ensures that the server knows what it must do in order to instantiate the plug-in.

Another important element is assigning the plug-in name, which we chose to be MYSQL_HELLO. We already explained that this name is later used in the INSTALL PLUGIN and DEINSTALL PLUGIN syntax, and that it is also used as the table name for the information_schema table.

Building and installing the plugin

Now that we have created the source file we must compile it and then install the plugin into our server.

Compiling the plugin source file

Assuming the source file mysql_is_hello.cc is located in the current working directory and /home/user/mysql-5.1.22-rc is the path to the MySQL 5.1.22 source distribution, the following line can be used to compile the source file:

g++ -DMYSQL_DYNAMIC_PLUGIN -Wall -shared
-I/home/roland/mysql-5.1.22-rc/include
-I/home/roland/mysql-5.1.22-rc/regex
-I/home/roland/mysql-5.1.22-rc/sql
-o mysql_is_hello.so mysql_is_hello.cc

Note that this is all on one line - I added line breaks to make it easier to read. If all goes well, this should result in a shared object file called mysql_is_hello.so.

Apart from the common g++ options -I (include path), -W (warnings, -Wall means “all warnings”), -shared (to compile as a shared, dynamically linkable library), and -o (output file) we see the specific -DMYSQL_DYNAMIC_PLUGIN. The -D option is there to define a constant (like a #define directive). -DMYSQL_DYNAMIC_PLUGIN triggers some conditional compilation magic that allows the relevant plug-in definitions to be exposed so they are visible from a program that dynamically links the shared object file. So, this option is required to make the plug-in pluggable.

The plugin directory

Once we obtained the mysql_is_hello.so shared object file, we must copy it to the plug-in directory of our installed MySQL 5.1.22 binary. Note that it is not necessary to build MySQL from source - you should be able to install the plug-in in a pre-built MySQL installed from a binary distribution.

The exact location of the plug-in directory is dependent upon specific MySQL distribution and configuration. You can find out its current location by querying the value of the plugin_dir system variable:

mysql> show variables like 'plugin_dir';
+---------------+-----------------------------------------+
| Variable_name | Value                                   |
+---------------+-----------------------------------------+
| plugin_dir    | /home/roland/mysql-5.1.22-dev/lib/mysql |
+---------------+-----------------------------------------+
1 row in set (0.01 sec)

So in this case, the shared library mysql_is_hello.so should be copied to /home/roland/mysql-5.1.22-dev/lib/mysql.

Installing the plugin

Once the shared library is in place we can install it using the INSTALL PLUGIN syntax:

mysql> install plugin MYSQL_HELLO soname 'mysql_is_hello.so';
Query OK, 0 rows affected (0.00 sec)

Note that we use the name MYSQL_HELLO for the plugin: this is what we defined earlier as the name member of the st_mysql_plugin plug-in descriptor. Likewise, we use mysql_is_hello.so, which is the file name of our shared object file as soname. The plug-in directory is implied - it should not be possible to install a shared library located at any place outside the plug-in directory.

In order to install a plug-in in this manner, the user has to have privileges to INSERT into the mysql.plugin table, or have the SUPER privilege.

There is a common problem that might occur at this point:

ERROR 1127 (HY000): Can't find symbol '_mysql_plugin_interface_version_' in library

If you see a message like this, it is likely that you forgot to include the -DMYSQL_DYNAMIC_PLUGIN option when compiling the plugin. Adding this option to the g++ compile line is required to create a dynamically loadable plug-in.

Verifying installation

We can now check if the plug-in is correctly installed. We do this by querying the PLUGINS table in the information_schema:

mysql> select * from information_schema.plugins
    -> where plugin_name = 'MYSQL_HELLO'G
*************************** 1. row ***************************
           PLUGIN_NAME: MYSQL_HELLO
        PLUGIN_VERSION: 0.16
         PLUGIN_STATUS: ACTIVE
           PLUGIN_TYPE: INFORMATION SCHEMA
   PLUGIN_TYPE_VERSION: 50122.0
        PLUGIN_LIBRARY: mysql_is_hello.so
PLUGIN_LIBRARY_VERSION: 1.0
         PLUGIN_AUTHOR: Roland Bouman (http://rpbouman.blogspot.com/)
    PLUGIN_DESCRIPTION: Says hello.
        PLUGIN_LICENSE: GPL
1 row in set (0.00 sec)

Using the plug-in

Finally, we get to test our plug-in ;-)

mysql> select * from information_schema.mysql_hello;
+--------------------------------------+
| HELLO                                |
+--------------------------------------+
| plugin: hello, information_schema!!! |
+--------------------------------------+
1 row in set (0.00 sec)

Of course, this is a gloriously useless application of information schema plug-ins. In a next installment I’ll demonstrate that you can do pretty cool stuff with these information schema plug-ins, such as peeking inside the query cache, listing the currently defined savepoints, temporary tables, user variables and much more.

Uninstalling the plugin

When you get tired of the plugin you can uninstall it using the UNINSTALL PLUGIN syntax:

mysql> uninstall plugin MYSQL_HELLO;
Query OK, 0 rows affected (0.00 sec)

Note that currently, due to a bug, you must be sure to use the exact same name for uninstalling the plugin as you did for installing it. I suspect this will be fixed soon, but for now it is best to simply always use the same name, for example the exact name used in the code, MYSQL_HELLO.

Learn More

I will be posting more about information schema plugins shortly. In particular, I will demonstrate how you can report status on server internals such as the query cache to discover which queries are in the cache, the number of blocks they are using and the number of bytes they actually occupying. However, the best way to learn more about extending the MySQL server, the MySQL plug-in API, and the MySQL information_schema is to visit the MySQL user’s conference, April 14-17 2008 in Santa Clara CA, USA.

There is a number of great sessions on this and related topics:

Of course, once you are at the conference, there will be many MySQL server developers giving you ample opportunity to ask them about some particular details, or maybe have them look at your code. And if you Register by February 26, 2008 you’ll save up to $200.

If you can’t wait: other great sources of information may be found in the list below:

Cheers, and happy hacking ;-)

Sun buys MySQL

2008-01-16T14:35:42Z

Earlier today, Sun announced that it will be acquiring MySQL. This is an interesting turn of events in Oracle’s silent battle over MySQL. With Falcon still years away from being production-ready, and Oracle owning the most popular and stable storage engine for MySQL (InnoDB), what are your thoughts on this acquisition and the effects (both positive and negative) it may bring to end-users?

Several of the announcements can be found below:

http://blogs.sun.com/jonathan/entry/winds_of_change_are_blowing
http://blogs.mysql.com/kaj/sun-acquires-mysql.html/
http://biz.yahoo.com/bw/080116/20080116005349.html?.v=1

Calculating the Financial Median in MySQL

2007-12-17T10:47:43Z

I believe I found a new method to calculate the median in MySQL. I would not be surprised if this method has been figured out by somebody else already. However, I can’t seem to find any resources on the internet describing this method, so for now I flatter myself by assuming the method is original.

(Please do post your comments to this blog to correct me on that should I be wrong so I have a chance to rectify.)

The method I’m describing is a one-pass, pure SQL method. It does not require subqueries, cursors or user variables. However, it does rely on the MySQL specific functions GROUP_CONCAT() and SUBSTRING_INDEX()

I’ll be maintaining a snippet for this method at MySQL Forge.
If you want to know what the median is, and how my snippet works, read on.

Some background

Like the mean and the mode, the median is an important metric to characterize the distribution of values in a collection. If we have a ordered collection of (numerical) values, the median is the value for which the number of entries that has a value that is higher than the median is exactly equal to the number of entries that has a value that is lower than the median. If there is an odd number of entries in the collection, the value of the median corresponds to the value of the entry that lies exactly in the middle of the list. If there is an even number of entries, the median is calculated as the mean of the two middle values.

MySQL offers a number of aggregate functions. Unfortunately, MySQL does not offer a function to calculate the median.

Even though MySQL does not support a MEDIAN() function natively, it is still possible to calculate it. You can:

install one of the numerous UDF’s floating around on the internet
Use pure SQL like suggested in Chapter 23 of Joe Celko’s SQL for Smarties

The snippet

Here’s a snippet that shows how to calculate the median replacement cost for a film in the sakila sample database:

select
        (
            substring_index(                          -- left median: max value in lower half:
                substring_index(
                    group_concat(                     --   list all values in ascending order
                        f.replacement_cost
                        order by f.replacement_cost
                    )
                ,   ','
                ,   ceiling(count(*)/2)               --   left half of the list
                )
            ,   ','
            ,   -1                                    --   keep only the last value in list
            )
        +   substring_index(                          -- right median: min value in upper half:
                substring_index(
                    group_concat(                     --   list all values in ascending order
                        f.replacement_cost
                        order by f.replacement_cost
                    )
                ,   ','
                ,   -ceiling(count(*)/2)              --   right half of the list
                )
            ,   ','
            ,   1                                     --   keep only the first value in list
            )
        ) / 2                                         -- average of left and right medians
        as median
from    sakila.film f;

(For the latest version, refer to MySQL Forge)

So, how does this snippet work? In the remainder of this post, I’ll explain the inner workings of this method in a top-down fashion.

The mean of the left and right median

The method I’m describing always takes the mean of the ‘left’ and ‘right’ median.

select
        (
            left-median(f.replacement_cost)
        +   right-median(f.replacement_cost)
        ) / 2                                         -- average of left and right medians
        as median
from    sakila.film f;

(Note that the usage of left-median() and right-median() is just an explanation of the structure - in reality there are not two distinct functions by that name)

The terms ‘left median’ and ‘right median’ are not common so they need an explanation.

Let’s visualize the process to determine the median. We can do this by imagining that we have an ordered list of values and that we point our left index finger to the lowest value in the list and our right index finger to the highest value in the list.

Now, we look at our fingers. If there is more than one entry between our right and left finger, we move or left finger one entry to the right and our right finger one entry to the left, and we keep doing that until there are no more entries between our left and right finger. Once we’re there, the value of the entry that is pointed to by our left finger is the ‘left median’ and the value of the entry pointed to by our right finger is the ‘right median’.

If we had an even number of entries, then the left and right median each correspond to distinct entries - if there was an odd number of entries then the left and right median correspond to one and the same entry.

At any rate, once we found the left and right median, it is clear that their mean is the true median. If we have an even number of entries, we have to calculate the mean of the two middle values anyway, and if there is an odd number of entries, taking the mean of two identical values results of course in that same value which does thus result in a correct value for the median.

`GROUP_CONCAT`: an ordered list of values

In the example, we use GROUP_CONCAT to generate a list of values in ascending order:

GROUP_CONCAT(                     --   list all values in ascending order
    f.replacement_cost
    ORDER BY f.replacement_cost
)

This gets us a string consisting of concatenated replacement_cost values in ascending order, separated by the default separator, which is a comma (',').

We use the same list in both the calculation of the left and the right median.

Note that the length of the concatenation result returned by GROUP_CONCAT() is limited. By default, it is as small as 1024 bytes. Personally I think this is way too small so I have it configured to be 64K by default. You can set the length at runtime too like this:

    SET group_concat_max_len := 65535

You can specify larger values than 65535 too, and I suspect that the maximum packet size is the practical maximum:

    SET group_concat_max_len := @@max_allowed_packet

To inspect the current value, you can do this:

    SELECT @@group_concat_max_len

Getting the ‘left’ half of the list

Once we have the list of values, we can split it in two halves with little effort. We do this using the SUBSTRING_INDEX() function.

The SUBSTRING_INDEX() function processes a string argument and gets a substring based on the position of a particular occurrence of another substring.

In this case, the comma ',' is the substring that separates the values in our ordered list. But of all the commas in the list, which occurrence of the comma do we need to find?

Suppose our list contains 4 values. Then, these values are separated by three comma’s:

values: '1,2,3,4'
commas:   ^ ^ ^
          1 2 3

If we want to divide this list in two equal halves, then the second comma is the divisor between the left and right halves of our list. With SUBSTRING_INDEX() this expression would get us the left half of the this list:

    SUBSTRING_INDEX('1,2,3,4', ',', 2) -- the substring up to the 2nd occurrence of ','

and the result will be:

    '1,2'

So now we have the first half of the list, and by definition, the last entry in that list, '2' is the left median of the original list '1,2,3,4'.

Now what if we would’ve had an odd number of entries in our list? Supose our list would’ve been like this:

values: '1,2,3'
commas:   ^ ^
          1 2

In this case too, we need the second comma to end up with a left substring that has the left median as last entry in the list (which also happens to be the proper median because this is a list with an odd number of entries).

As it turns out, we can conveniently generalize the SUBSTRING_INDEX expression like this:

    SUBSTRING_INDEX(list, separator, CEILING(#entries/2))

In other words, if we divide the number of entries in our list by two, and then round to the nearest higher integer, this gives us the particular occurrence of the separator what we are looking for to halve our list as required. Of course, calculating the number of entries is simply a matter of using the COUNT() aggregate function.

Excising the ‘left’ median

To actually obtain the left median itself, we just need to excise the last value from the left half of the list of values. We do this by applying SUBSTRING_INDEX() again.

Again, we need to make a substring in terms of the occurrence of the comma that separates the values in our list. This time, we need to get the substring found directly after the last comma in the list. With SUBSTRING_INDEX() we can conveniently express this in the following manner:

    SUBSTRING_INDEX(list, separator, -1)

This means: search the list from right to left and find the first occurrence of the separator. Return the substring that appears after the separator (that is, the substring appearing on the right hand of the separator).

Getting the ‘right’ median

The process to obtain the right median is a mirror of obtaining the left median: instead of obtaining the last value in the left half of the ordered list of values, we now need to obtain the first value of the right half of the list. This is actually as simple as reversing the sign of the occurrence argument in the SUBSTRING_INDEX() calls:

    left half:   SUBSTRING_INDEX(list, separator,  CEILING(COUNT(*)/2))
    right half:  SUBSTRING_INDEX(list, separator, -CEILING(COUNT(*)/2))

    last entry:  SUBSTRING_INDEX(list, separator, -1)
    first entry: SUBSTRING_INDEX(list, separator,  1)

A few remarks

I think that in many cases, this can be a fair method to calculate the median. The advantage of this method is that it is relatively fast because the query itself is relatively simple.

It would be interesting to see how this method behaves when handling millions of rows. Maybe I will run some benchmarks on that later on.

In the mean while, feel free to post your thoughts, suggstions or critique on this blog.

Random RDBMS and SQL Myths debunked

2007-11-09T09:44:25Z

A few times now, I’ve been wanting to write this down. I know: a lot of people will go *shrug*. Others may find me pedantic. Some of will say I’m being a smart-ass. Whatever…but I just got to write down a few of these common misconceptions that keep floating around.

None of these misconceptions are really harmful - in most cases, they do not lead to misunderstanding or miscommunication. However, when you are writing about these subjects, you’ll often find that a sloppy definition you used in some place will bite you in the tail, and make it harder to explain something later on. So, that is why I from time to time get kind of obsessed with finding just the right words.

I’m not pretending I have the right words though. But there are a few informal ways of saying things that at a glance look right but are in fact wrong. Here’s a random list of some of them:

PRIMARY KEY and UNIQUE constraints are unique indexes

Wrong - an index is just a convenient implementation to detect duplicate entries, and this is used by all RDBMS-es I am familiar with to implement PRIMARY KEY and UNIQUE constraints. However, the fact that there is a distinction is evident in for example the Oracle SQL syntax. For example, in ALTER TABLE ... DROP CONSTRAINT you can specify whether the associated index should be kept or also discarded.

Some people argue that it does not make sense to make the distinction in case the RDBMS does not maintain the constraint and index as separate objects. (This is the case in for example MySQL.)

Well, maybe…but I disagree. When I think about constraints, I’m thinking about business rules and guarding them to maintain database integrity. When talking about indexes, I’m thinking about performance and access paths. Quite different things, and in my opinion a shame to throw away the words to express the difference in my opinion.

A table consists of rows and columns

No - there is nothing wrong with an empty table. In other words, it does not consist of rows. It may or may not contain rows, but that is a different story.

A scalar subquery returns one column and one row

Wrong - first of all, a scalar subquery may return zero rows, in which case it evaluates to null, which is perfectly valid. But there is more to it.

Whether something is or is not a subquery is matter of syntaxis. The SQL grammer is defined so that if you encounter a query between parenthesis where a scalar value is appropriate, then that query (including the parentheses) will be parsed as a scalar subquery. In other words, the text satisfies the production rule for the non-terminal symbol “scalar subquery”.

The parser will usually be smart enough to verify whether the subquery yields one column, but the number of rows returned is a runtime affair.

Suppose the query that makes up the scalar subquery would in fact return more than one row…would it suddenly not be a scalar subquery anymore? Of course not. It is still a scalar subquery - it just happens to be impossible to execute it. In other words, it violates the semantics of a scalar subquery and is therefore invalid. But the mere fact that we can conlcude that must imply that it is a scalar subquery.

A subquery is a SELECT statement that appears as a part of another SELECT statement

Wrong - For the same reasons as the previous issue. A statement is a syntactical construct. It has to do with discovering a pattern in a piece of text so that it satisfies a particular rule in the SQL grammer. That grammar does not have a rule that allows statements to be nested - not in pure SQL anyway (Of course, in stored procedures, one can have statement blocks like BEGIN...END, IF...END IF etc that really can contain other statements)

Of course, if we would take the SELECT that makes up the subquery and run it in isolation, it would be a SELECT-statement. Bit that is exactly the heart of the matter: because we are regarging it as part of another statement, it cannnot be a statement itself. This is simply a matter of definition of course - most people will immediately understand what is meant.

What would be better to say though is that a subquery is a query or query expression that appears as part of another SQL statement. However, this is also not correct: CREATE VIEW for example does contain a query expression, but this would most likely not be called a subquery. For this particular case, you can argue that there is nothing sub-ish about the query expression, because it is simply an essential part of the CREATE VIEW statement.

But what to think of CREATE TABLE...AS SELECT... and INSERT INTO...SELECT? The query expression is certainly not an essential part of CREATE TABLE and INSERT INTO, and in that sense, the query does look like it is subordinate to the statement it is part of.

You could argue that a query is a subquery if it appears inside another query. That seems sound, but what to think of UPDATE ... SET = (SELECT ...)? Personally I am reluctant to call an UPDATE statement a query - I tend to think of a query as a SELECT statement or sometimes a query expression.

I can think of only one thing that really is a defining characteristic of a subquery though - that is that the query expression must appear within parentheses. So, again, a matter of syntax more than a matter of semantics. I must admit I’m still not very satisfied with this though…What do you think?

NULL is the absence of a value

Variants of this statement go like “NULL is a missing value” or “NULL is not a value”.

With slight doubt, I say: wrong. It certainly is true that many people use NULL to convey that something is not there or that something is not applicable. But this is a matter of choice, it does not change the meaning of NULL itself. If we use the same line of reasoning as we used for the subquery myth, we must conclude that NULL is certainly a valid value expression. It can legally appear anywhere where we can put a value. It is IMO also perfectly ok to say things like “…that expression evaluates to NULL“.

So what does the SQL standard say? Well, here’s a quote:

…the null value is neither equal to any other value nor not equal to any other value — it is unknown
whether or not it is equal to any given value….

So, I’m in that camp too: NULL is a value, and if we have a NULL in say, the integer domain, we just don’t know which of all possible integers it is.

Foreign keys must reference a primary key

Wrong - a Unique constraint is mostly just as acceptable.

In MySQLs InnoDB it is even more relaxed - the foreign key only needs to reference the prefix of an index in the parent table, although this is so exotic, it should probably be ignored.

This table is not normalized - it still contains redundancy

Wrong - a table is normalized when it is in the first Normal form. There are a few different opinions what that means exactly, but it usually works to say that a table is not normalized when it contains repeating groups.

A slightly stronger statement is to say that a table is not normalized when it contains data that is not atomic. This is stronger because it does not cover only repeating groups, but also columns that, for a single row, do not contain a single value. For example, a first name/last name combination in one column is not atomic, and therefore, a table that contains such values is not normalized. (There are opinions that require even more than this, but for practical purposes the sense of atomic values works pretty well.)

The source of confusion is in what happens beyond the first normal form. Although a table maybe normalized, it can still contain redundancy. By removing redundancy, you can progressively achieve a higher normal form. In many cases, one would require at least third normal form or the Boyce-Codd normal form for building database schemas. Many people say “normalized” when they actually mean “in at least the third normal form”.

So - what do you think? Pedantic? Have other myths? Maybe you have a good, satisfactory definition for subqueries? Or maybe you find an error in my debunkings?

Just drop me a comment on this post - thanks in advance.

Kettle Tip: Using java locales for a Date Dimension

2007-09-03T23:18:06Z

The Date dimension is a well known construct in general data warehousing. In many cases, the data for a date dimension is generated using a database stored procedure or shell-script.

Another approach to obtain the data for a date dimension is to generate it using an ETL tool like Pentaho Data Integration, a.k.a. Kettle. I think this approach makes sense for a number of reasons:

When you tend to use a particular ETL tool, you will be able to reuse the date dimension generator over an over, and on different database platforms.
You won’t need special database privileges beyond the ones you need already. Privileges for creating tables and to perform DML will usually be available, whereas you might need to convince a DBA that you require extra privileges to create and execute stored procedures.

In addition to these general considerations, you can pull a neat little trick with Kettle to localize the data and format of the date attributes. I wouldn’t go as far as to say that this feature is Kettle specific: rather, it relies on the localization support built into the java platform and the way you can put that to use in Kettle transformations.

Prerequisites

In this tip, the steps to create a date dimension are described using Kettle 2.5.1 (Generally available Release) and MySQL 5.1.20 (Beta). You will be able to follow through the example using earlier (and later) versions of both products though - I am not using any functionality that is specific to these particular version of the products. The recipe does not really require that you understand anything about data warehouses or date dimensions, but you will probably appreciate it better if you do ;)

Overview

The transformation to generate the data for the date dimension follows a pretty straightforward design. The graphical representation of the transformation is shown below:

First, the dimension table is created (Prepare). After that, rows are generated to fill it (Input). However, the generated rows are almost empty and barren - we still need to derive and add data to fill the attributes of the date dimension (Transformation). Finally, the data is stored in the date dimension table (Output).

Step-by-Step

The remainder of this article describes in detail how to build this transformation. The majority of steps is probably not very interesting to moderately experienced Kettle users, but may be of use to beginning users.

Note for users that are completely new to Kettle - it is advisable to review the first few chapters of the Spoon user guide (Spoon is the name of Kettle tool you use to design the ETL process). It explains how to start up the tool, create a new transformation, add and connect steps etc. You can find it in the docs/English directory beneath the Kettle home directory.

MySQL JDBC driver: setting the characterEncoding property to UTF8

You need to create a (JDBC) connection to MySQL in the usual, straightforward way:

In addition, you need to set the characterEncoding property of the JDBC driver:

This ensures MySQL will be able to understand the utf8 encoded data that we may produce to generate a date dimension in the, say, Chinese language. Note that you cannot just use a statement like SET NAMES utf8 to do this. This is not specific to Kettle, but has to do with the way the MySQL JDBC driver (Connector/J) handles character sets. Please refer to the “Using character sets and unicode” section of the Connector/J documentation for more information on this topic.

Creating the date dimension table

In this particular case, it seemed convenient to create the dimension table as part of the transformation. This is done using the “Execute SQL Script” step shown below:

The “Execute SQL Script” step executes the following script to create the date dimension table:

DROP TABLE IF EXISTS dim_date
;
CREATE TABLE IF NOT EXISTS dim_date (
  date_key                smallint unsigned NOT NULL,
  date                    date              NOT NULL,
  date_short              char(12)          NOT NULL,
  date_medium             char(16)          NOT NULL,
  date_long               char(24)          NOT NULL,
  date_full               char(32)          NOT NULL,
  day_in_year             smallint unsigned NOT NULL,
  day_in_month            tinyint  unsigned NOT NULL,
  is_first_day_in_month   char(10)          NOT NULL,
  is_last_day_in_month    char(10)          NOT NULL,
  day_abbreviation        char(3)           NOT NULL,
  day_name                char(12)          NOT NULL,
  week_in_year            tinyint  unsigned NOT NULL,
  week_in_month           tinyint  unsigned NOT NULL,
  is_first_day_in_week    char(10)          NOT NULL,
  is_last_day_in_week     char(10)          NOT NULL,
  month_number            tinyint  unsigned NOT NULL,
  month_abbreviation      char(3)           NOT NULL,
  month_name              char(12)          NOT NULL,
  year2                   char(2)           NOT NULL,
  year4                   year              NOT NULL,
  quarter_name            char(2)           NOT NULL,
  quarter_number          tinyint           NOT NULL,
  year_quarter            char(7)           NOT NULL,
  year_month_number       char(7)           NOT NULL,
  year_month_abbreviation char(8)           NOT NULL,
  PRIMARY KEY(date_key),
  UNIQUE(date)
)
ENGINE=MyISAM
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_unicode_ci

This is by no means a complete date dimension. The most important limitation is that it only contains attributes that are immediately derivable from the calendar. So, attributes to denote business specific periods like the fiscal year and holidays are not included.

Generating 10 years worth of days

The grain of the date dimension is days - a row in the date dimension represents a single day. In this case, the “Generate Rows” is configured to generate 3660 rows, which roughly corresponds with enough days to last 10 years:

In the example, this step is also used to provide parameters to generate the date dimension data. As we’ll see in a moment, the inital_date field effectively specifies the first date that goes into the date dimension. The language_code and country_code fields are used to localize the textual attributes of the date dimension, and the local_yes and local_no fields are used for boolean attributes.

There are other ways to get these parameters into our transformation. For example, we could have used an “Add Constants” step with a similar result. Another possibility would be to get this data from the environment using a “Get Variables” step, and this would allow the parameters to be specified at transformation run-time.

Counting the days

Although we certainly generate enough rows, they are all identical. In order to have each row represent a single distinct day, we need a way to ‘count’ the generated rows. We do this by adding a “Sequence” step:

In this case, we use the “Add sequence” step to generate an incrementing number within the scope of the transformation. As we’ll see later on, we can add this to our initial date to get a series of consecutive dates.

Calculating date dimension Attributes

The previous steps form a basis from which we can derive all of the attributes that currently make up our date dimension. To actually calculate the date attributes, we use a “Modified Java Script Value” step:

Kettle comes with an embedded Rhino javascript engine. The “Modified Java Script Value” step lets you use it to run javascript code to as part of the transformation.

The javascript code is executed for each row that comes out of the previous steps. In the script code, one can reference the values from the input rows, perform some processing on them, and generate new output fields.

One of the fortunate characteristics of the Rhino engine is that it lets us use java classes inside the javascript code. Let’s take a look at the script to see how we can use that to generate the localized data for our data dimension attributes.

Initialization

The first thing we do in the javascript code is to get data from the current input row. In the “Generate Rows” step, we added the language_code and country_code fields to specify a locale. Here, in the script, we use the following piece of code to turn that into a java.util.Locale object:

//Create a Locale according to the specified language code
var locale = new java.util.Locale(
    language_code.getString()    //get the ISO639 language_code from the input row
,   country_code.getString()     //get the ISO3166 country_code from the input row
);

The java.util.Locale class represents a particular cultural region. It forms a cornerstone of the internationalization support built into the java platform, and provides information to many other classes to generate appropriately localized output.

We will be using the locale on a number of occasions, but first, we use our it to initialize a java.util.Calendar object:

//Create a calendar, use the specified locale
var calendar = new java.util.GregorianCalendar(locale);

(Note that the java platform currently only provides one concrete Calender Class: the java.util.GregorianCalendar. Unfortunately, java does not seem to provide a built-in recipe for dealing with, for example, Islamic or Hebrew calendars).

We require the calendar object to obtain an instance of the java.util.Date Class that represents the date corresponding to the current row. To do that, we first set the calendar’s current date using the initial_date field that was specified in the “Generate Rows” step:

//Set the initial date
calendar.setTime(initial_date.getDate());

We need this to add the number of days generated by our “Add Sequence” step:

//set the calendar to the current date by adding DaySequence days
calendar.add(calendar.DAY_OF_MONTH,DaySequence.getInteger() - 1);

(Note that we substract 1 from the DaySequence value. This is because our sequence starts at 1, and we want the specified initial date to be included in our date dimension).

We conclude the initialization of the script by retrieving a java.util.Date object that represents the date for the current row.

//get the calendar date
var date = new java.util.Date(calendar.getTimeInMillis());

This java.util.Date instance is assigned to the date variable in the script, allowing it to be used as an output field of the javascript step. We require this in order to fill the date column of the date dimension table. We will also be using the date variable later on in this script to derive the value of other date dimension attributes.

Getting Text representations of full dates

Our date dimension has a number of attributes to denote a complete date containing day, month and year parts, in various formats: date_short, date_medium, date_long and date_full. These are all generated using the java.text.DateFormat class.

To do that, we first need to create an appropriate DateFormat instance using the static getDateInstance() method, passing our locale object as well as a constant that specifies whether we want to short, medium, long or full format. Then, we can pass the java.util.Date object for which we want to obtain the textual representation to the format method of the newly created java.text.DateFormat instance:

//en-us example: 9/3/07
var date_short  = java.text.DateFormat.getDateInstance(
                      java.text.DateFormat.SHORT
                  ,   locale
                  ).format(date);
//en-us example: Sep 3, 2007
var date_medium = java.text.DateFormat.getDateInstance(
                      java.text.DateFormat.MEDIUM
                  ,   locale
                  ).format(date);
//en-us example: September 3, 2007
var date_long   = java.text.DateFormat.getDateInstance(
                      java.text.DateFormat.LONG
                  ,   locale
                  ).format(date);
//en-us example: Monday, September 3, 2007
var date_full   = java.text.DateFormat.getDateInstance(
                      java.text.DateFormat.FULL
                  ,   locale
                  ).format(date);

Formatting date parts

Extracting and formatting different date parts is most easily done by applying the format function on a subclass of java.text.Dateformat, the java.text.SimpleDateFormat class. The java.text.SimpleDateFormat class allows formatting of dates based on date and time patterns:

//day in year: 1..366
var simpleDateFormat = java.text.SimpleDateFormat("D",locale);
var day_in_year              = simpleDateFormat.format(date);

In this example, we pass both the locale and a date pattern to the constructor to create an instance of the java.text.SimpleDateFormat class. The pattern is passed as the string "D", specifying a day-in-year format.

Once we created the java.text.SimpleDateFormat instance, we can apply a new pattern to it using the applyPattern() method. Calling the format method again, we obtain the date in the desired format:

//day in month: 1..31
simpleDateFormat.applyPattern("d");
var day_in_month       = simpleDateFormat.format(date);
//en-us example: "Monday"
simpleDateFormat.applyPattern("EEEE");
var day_name           = simpleDateFormat.format(date);
//en-us example: "Mon"
simpleDateFormat.applyPattern("E");
var day_abbreviation   = simpleDateFormat.format(date);
//week in year, 1..53
simpleDateFormat.applyPattern("ww");
var week_in_year       = simpleDateFormat.format(date);
//week in month, 1..5
simpleDateFormat.applyPattern("W");
var week_in_month      = simpleDateFormat.format(date);
//month number in year, 1..12
simpleDateFormat.applyPattern("MM");
var month_number       = simpleDateFormat.format(date);
//en-us example: "September"
simpleDateFormat.applyPattern("MMMM");
var month_name         = simpleDateFormat.format(date);
//en-us example: "Sep"
simpleDateFormat.applyPattern("MMM");
var month_abbreviation = simpleDateFormat.format(date);
//2 digit representation of the year, example: "07" for 2007
simpleDateFormat.applyPattern("y");
var year2              = simpleDateFormat.format(date);
//4 digit representation of the year, example:  2007
simpleDateFormat.applyPattern("yyyy");
var year4              = simpleDateFormat.format(date);

Dealing with Quarters

Although the java.text.SimpleDateFormat class is useful, it does not provide any functionality for working with quarters. We do want our date dimension to contain attributes to represent the quarter, so we have to reside to computing these manually:

//handling Quarters is a DIY
var quarter_name = "Q";
var quarter_number;
switch(parseInt(month_number)){
    case 1: case 2: case 3: quarter_number = "1"; break;
    case 4: case 5: case 6: quarter_number = "2"; break;
    case 7: case 8: case 9: quarter_number = "3"; break;
    case 10: case 11: case 12: quarter_number = "4"; break;
}
quarter_name += quarter_number;

Although this will do for now, this solution doesn’t really cut it because it does not produce localized output. Anyway, it is better than nothing so we’ll just have to make do with it.

Period demarcation flags

Our date dimension has a few attributes that are used to indicate the start and end of week and month periods. We use simple yes/no type flags, but we allow the actual “yes” and “no” values to be specified by the user in the “Generate Rows” step. We retrieve them with the following piece of code:

//get the local yes/no values
var yes = local_yes.getString();
var no = local_no.getString();

We can now use them these to flag the start and end of week and month periods.

The start (and of course, also the end) of the week are subject to the locale. In order to find out if we are dealing with the first day of a week, we use the getFirstDayOfWeek() method of the java.util.Calendar class. By comparing its return value with the day of week of the current row, we can see if we happen to be dealing with the first day of the week:

//initialize for week calculations
var first_day_of_week = calendar.getFirstDayOfWeek();
var day_of_week = java.util.Calendar.DAY_OF_WEEK;

//find out if this is the first day of the week
var is_first_day_in_week;
if(first_day_of_week==calendar.get(day_of_week)){
    is_first_day_in_week = yes;
} else {
    is_first_day_in_week = no;
}

Note that we obtain the current day of the week by passing the value of the DAY_OF_WEEK constant to the get method of the java.util.Calendar object that we initialized at the start of the script.

In order to set the value for the is_last_day_in_week attribute of the date dimension, we simply find out if the next day happens to be the first day of the week. If it is, then by definition, the current row represents the last day of the week:

//calculate the next day
calendar.add(calendar.DAY_OF_MONTH,1);

//get the next calendar date
var next_day = new java.util.Date(calendar.getTimeInMillis());

//find out if this is the first day of the week
var is_last_day_in_week;
if(first_day_of_week==calendar.get(day_of_week)){
    is_last_day_in_week = yes;
} else {
    is_last_day_in_week = no;
}

(Note that we have already used similar code to add a day to a date when we added the day sequence to the initial date.)

We can use similar logic to calculate the values for the is_first_day_in_month and is_last_day_in_month indicators. This is actually easier, because the first day in the month is not dependant upon the locale (at least - not within one calendar). So, we only need to find out if the day of month is equal to one:

//find out if this is the first day of the month
var is_first_day_of_month;
if(day_in_month == 1){
    is_first_day_in_month = yes;
} else {
    is_first_day_in_month = no;
}

//find out if this is the last day in the month
var is_last_day_of_month;
if(java.text.SimpleDateFormat("d",locale).format(next_day)==1){
    is_last_day_in_month = yes;
} else {
    is_last_day_in_month = no;
}

A few more date attributes

We conclude the computation of the date attributes by adding a few more useful labels:

//a few useful labels
var year_quarter            = year4 + "-" + quarter_name;
var year_month_number       = year4 + "-" + month_number;
var year_month_abbreviation = year4 + "-" + month_abbreviation;

Like when we calculated the quarters, this is actually not a very good method because the results will not be localized. That said, the result will make sense for many locales, and we don’t really have a better way to deal with it right now.

Defining the step outputs

We just calculated all the required values to fill the attributes of our date dimension. We just need to get them out of the script and into the outputs of the step.

Every variable declared in the javascript (using the var keyword) can be used as an output field of the javascript step. The easiest way to generate the outputs is by hitting the “Get Variables” button at the bottom of the dialog. This simply adds an output field for each variable declared in the script:

By default, the data type for all the outputs added in this way is set to the String type. Although it is good practice to choose a more specific data type, it is almost always unncessary in this case, as all integer type values will be correctly converted implicitly when we insert them into the database. There is one exception in this case, and that is the date output. Inside the script, it is an instance of a java.util.Date class, and we must set the type to “Date” in the output too. Otherwise, the (java) string representation of the java.util.Date object will be sent as output, and this is not automatically recognized as a date by MySQL.

Discarding Fields

We are now almost ready to insert the rows into the date dimension table. We only need to discard all fields in the stream that do not correspond with any of the columns in our date dimension table. We use a “Select Values” step to do that:

We use the “Get Fields To Select” button to pull in all available fields, and after that, simply select and delete each field that we do not need. As a final step, we rename the DaySequence field to date_key to map it to the date_key column in our date dimension table.

Inserting data into the table

In the final step, we add the generated data to the dim_date table we created in the very first step of the transformation:

We only need to specify the connection and the table name here, and the step will then automatically attempt to map the fields of the incoming rows to table columns.

We could have used the “Insert / Update” step, or even the “Execute SQL Script” step too to write the data to the dimension table, but that would require a little bit of extra work.

Running the transformation

After building the transformation, you can run it by hitting the “running man” icon on the toolbar. This will open a dialog where you can set a number of properties for the transformation. Hit “Launch” button there and after that, the transformation will be executed:

Closing Notes

I hope you enjoyed this tip. If you want to, you can download the kettle transformation here, and use it as you see fit.

If you are interested in open source data warehousing, register for the MySQL Enterprise Data Warehousing Seminar, Thursday, September 06, 2007 and hear what Robin Schumacher has to say about that subject. (Note that this is a general MySQL data warehousing seminar - this post and the seminar are unrelated)

Top 5 considerations while setting up your MySQL backup

2007-08-22T00:03:22Z

List of top 5 items that have to be considered before deciding on a MySQL backup implementation are:
* How fast and how easy do you want the MySQL Recovery process to be?
* What will be the impact of MySQL Backup process on your Application?
* What will your backup configuration look like? (the What, Where, When, and How of MySQL Backup)?
* How will you manage your backup process and backed up data?
* What kind of tracking, reporting and compliance requirements does your business have from your MySQL backup implementation?

The white paper provides detailed insights about the above considerations. Your feedback is welcome.

Seeking MySQL backup console feedback

2007-06-26T21:59:16Z

We are working on Zmanda Management Console for our MySQL backup product line: Zmanda Recovery Manager (ZRM) for MySQL. ZRM for MySQL is an enterprise backup and recovery solution for MySQL.

We are looking for MySQL administrators in San Francisco bay area who would be interested in providing functionality and usability feedback for the user interface. We are particularly interested in MySQL administrators responsible for implementing backup solution for their MySQL databases. Please send an email to me (paddy-at-zmanda-dot-com) with your availability and why you would meet our requirement. We will pay a small stipend for your time.

What MySQL can do to enter the off-line Web

2007-06-03T10:16:10Z

Disclaimer - views expressed in this blog (and this entry) are my own and do not necessarily reflect the views of MySQL AB

Ever since I wrote my blog entry about Google Gears and the query tool for the browser embedded offline Google Gears database service, I have been wondering how MySQL might fit in here.

I have heard an idea to write a MySQL storage engine for SQLite and although I do not think this is necessarily a bad idea, I don’t think it will be immediately useful for typical applications powered by Google Gears. Personally, I think the following things might be of more use:

A modification of the Google Gears browser extension that allows a local MySQL database to be used instead of the embedded SQLite database

This would be interesting in case you would need a particular MySQL feature that is not available in SQLite. I’m thinking mainly of stored procedures but more so about replication.

The local MySQL server could be setup as a master that replicates to a remote slave. The remote slave would be hosted by your company and that backend would somehow merge all the private slaves from its employees into the corporate database.

Of course, the idea for the SQLite storage engine could fit here too - a local MySQL Server with the SQLite storage engine could be set up as master to replicate to the corporate slave.

A Google Gears Worker Pool application that synchronizes the embedded SQLite database with the remote, corporate MySQL database

This idea would of course require some server component that may be accessed by the Worker application, and it would presumably take the form of a web-service. This could be implemented either as a piece of middleware that sits in a HTTP server or as a component that is part of the MySQL server itself. This product would achieve in part the same functionality as replication from the client’s master to the remote slave but I think there are a number of advantages here.

First of all, the implementation would not be intrusive to Google Gears - users can still use the ordinary Google Gears extension, and MySQL AB does not need to ensure that their modifications to the browser extension are compatible with future Google Gears developments.

Second, this application could in principle allow a true synchronization rather than just replication. In other words, it could pull data from the corporate database as needed, and push local modifications to be merged with the remote database. Obviously the possibilities here would depend on the flexibility of the server-side component accessed by the worker pool application.

Thirdly, users would not need to install an extra MySQL server on their webclient host. Arguably this could be seen as a disadvantage for MySQL AB as it would result in fewer installations of MySQL Server, but personally I don’t think this is the right angle. I think that for most people, an extra MySQL Server that sits on their machine just for the purpose of replication will not be very attractive.

To me, using the Google Gears Worker Pool to drive synchronization seems like the most sensible choice that fits good with the whole offline-web idea. But what do you think? Can MySQL play a role in the offline web like SQLite can now?

Google Gears: Webbrowser embedded Database

2007-06-02T15:10:08Z

Google Gears is an open source browser extension created by Google. It provides a framework that allows the creation of offline webbrowser applications. At the moment it provides three services:

Local Server: A data store for static resources. This allows efficient caching of images, scripts and webpages
Database: An embedded relation database management system, based on SQLite. SQLite should be familiar to most PHP developers, as it is being shipped with PHP since version 5
Worker Pool: A form of threading support inside the browser that allows webapplications to initiate long running processes without hampering the responsiveness of the user interface.

All these services can be accessed from within the browser using a javascript API.

Users only needs to install the extension in order for the browser to be able to access the services when browsing pages.

If you want to get an immediate taste of the database service, be sure to install Google Gears and take a look at my offline, browser-based database client.

Currently, the extension is available for mozilla Firefox and Internet Explorer.

What does this buy us?

Google Gears offers an enormous potential for web application developers, for a number of reasons:

The local server can be used to setup a specialized cache that can help to run web applications off-line.
The database service offers not only a structural solution for maintaining state at the client side, it even allows for persistence. That is, if you close the browser, shut down the computer, and restart, the application can pick up where it left off!
I am not really sure about the worker pool but I can imagine this can be very useful to perform tasks to perform tasks that require a connection with a server. For example, the worker could be instantiated that attempts to synchronize the local state and data gathered by the application to a webservice that acts as the terminal destination.

Of course, we all have to see real-world results, but I think this truly is a revolutionary development. It finally makes sense to start thinking of “browser applications” instead of “web applications”, because the application does not necessarily have to connect to a HTTP server anymore to be useful.

I would not be surprised if this would initiate a new wave of emancipation of relational databases. Literally everybody that can operate a webbrowser will now own a database, and all the glue to make it work together with the user interface is already there. Also, a number of mobile service applications might be implemented on top the Google Gears embedded SQLite database instead of a specific standalone database product.

The web browser offers a reasonably sufficient environment for creating data oriented end-user applications, and the combination might prove to be a killer solution. The SQLite database can still be accessed outside the browser too, and be integrated with other applications.

Prerequisites

There are few preparations that need to be taken before you can use google gears in a web page. The prerequisites are that the browser that will view the page has the Google Gears browser extension installed. The page itself needs to include the gears_init.js script in order to use the javascript API to access the services. This single line in the <head> element of the page allows access to the API:

    <script type="text/javascript" src="gears_init.js"></script>

The gears_init.js script needs to reside at the same location as the page in this case, but it may be referenced from any location.

One of the fascinating things is that neither the page nor the script need be on a webserver. You can simple store the script and the page on the local filesystem.

When opening the page in the browser, you will be prompted to allow the page to access the Google Gears services:

So the user needs to allow access for each different site that wants to use the services.

A closer look at the Database service

The Google Gears database service can be accessed using a simple API. All of the Google Gears APIs are javascript (ecmascript) APIs.

Creating a database object and opening a database

First a database object needs to be created using a line like this:

    var db = google.gears.factory.create("beta.database", "1.0");

The variable db can then be used as a handle to open the actual database. The following line will create a database named GGDB:

    db.open("GGDB");

Database names are unique per application, or rather origin. All pages that are located in a particular scheme, host and port are considered to be part of the same application. (For the web location http://www.foo.com/, http is the scheme, www.foo.com is the domain and the port is the default port for the scheme, which is 80 in this case). Pages can only access databases that were created by a page in the same domain.

This ’same domain’ restriction is an understandable security measure: if we visit https://www.mybank.com/, we do not want the data stored in any of the databases used by that site to be accessible by pages from http://www.bankrobbers.com/.

Creating a table

The database object implements the execute method, which allows one to execute SQL statements. For example, if we want to create a table to store contacts, we could try something like this:

db.execute(
    "CREATE TABLE persons("
+   "    id         INTEGER PRIMARY KEY AUTOINCREMENT"
+   ",   first_name"
+   ",   last_name"
+   ")"
);

When you are not used to SQLite’s specific philosophy of data storage, the previous statement must’ve made you frown at least once.

Yup people, SQLite does not require us to specify column data types. It is allowed to include an identifier where you’d normally put a data type name. However, this ‘data type’ does not restrict the values that maybe stored in that column. There is only one exception to that rule: when a column is defined with INTEGER PRIMARY KEY AUTOINCREMENT, it denotes a surrogate key that will store only integral values.

In spite of the fact that SQLite does not use data types for table definitions, SQLite does have a type system. SQLite has a concept called storage classes: each value entered into the database is associated with one of the following storage classes:

NULL

A special storage class for SQL NULL values

INTEGER

Storeas integral values

REAL

Stores numerical floating point values

TEXT

Variable length strings

BLOB

Variable length binary strings

However, a storage class is bound on the level of a value, not a column. So, one column can contain multiple values, and each is associated to one of the storage classes.

Inserting data

In Google Gears, all interaction with the database is performed by issuing SQL statements through the execute method of the Database object. So, if we want to insert data into our persons table, we need to do this with a SQL INSERT statement.

The first argument of the execute method is the actual SQL statement, and we can use this to pass a complete INSERT statement, including any value literals. However, we can also use parameterized statements, using placeholders instead of value literals. The remainder of the argumentlist must then be used to provide values for the placeholders in the SQL statement text:

db.execute(
    "INSERT INTO persons (first_name,last_name)"
+   "VALUES (?,?)"
,   "Roland"
,   "Bouman"
);

This manner of passing values offers the advantage that you do not need to escape quotes from the values: this is automatically taken care of by SQLite.

Processing resultsets

A call to the execute method always returns a ResultSet object. In the previous examples, execution of the statements did not result in a set of rows, but a javascript ResultSet object is returned regardless.

The following snippet illustrates how you can process the rows in the ResultSet object:

var result = db.execute(                 //execute query, get result
    "SELECT * FROM persons"
);
var fieldCount = result.fieldCount();   //get number of columns
while(result.isValidRow()){             //as long as we did not process all rows
    for(var i=0; i<fieldCount; i++){    //loop over all columns
        var value = result.field(i);    //get value of the current column
    }
    result.next();                      //process the next row
}
result.close();

The fieldCount method of the ResultSet object can be used to return the number of columns associated with the ResultSet object. If the SQL statement that yielded the ResultSet object did not return a set of rows, the fieldCount method returns 0.

The isValidRow() method of the ResultSet object returns a boolean that indicates whether there are still rows associated with the ResultSet object that may be processed. The next() method of the ResultSet object proceeds to the next row in the resultset. After calling the next() method, the isValidRow() method must be used to detect whether the resultset is already exhausted. Together, the isValidRow() and next() methods can be uses this to drive a loop to iterate over all the rows associated with the ResultSet object.

The field(columnPosition) method of the ResultSet object can be used to obtain the value for the column at the specified position. The fieldByName(columnName) method of the ResultSet object can also be used to obtain the value of a column, but as indicated by the method name, the column must be specified using the name of the column. The fieldName(columnPosition) may be used to retrieve the name of the column at the specified position.

When we are done processing the resultset, we are required to call the close() method of the ResultSet object. There are plans to implement a way to automatically close the ResultSet object when it isn’t used anymore, but even then it can’t hurt to close the ResultSet object explicitly.

Handling Errors

Although Google Gears is great, we still are not in a perfect world. Runtime errors may and will occur, and to handle them you need to use the javascript try...catch syntax. This is not so much a Google Gears thing, but as the database API will throw a runtime error for syntax errors, database constraint violations etcetera, you simply myst use try...catch blocks everywhere.

The following snippet illustrates how to do this:

try {

    //stuff that can go wrong    

} catch(e){

    //use the message property
    //from the Exception object
    alert(e.message);

}

A Quick and Dirty Command Line Client

To play around with the Google Gears Database service, I made a quick and dirty browser-based Command line client for the SQLite database. You can use the online version here. If you want to use it offline, no problem, just download save the page anywhere on your local disk. Just be sure to save a copy of the gears_init.js script in the same directory. That’s it….on to the offline web!

Debunking GROUP BY Myths

2007-05-21T18:16:07Z

There is a popular myth about the SQL GROUP BY clause. The myth holds that ’standard SQL’ requires columns referenced in the SELECT list of a query to also appear in the GROUP BY clause, unless these columns appear exclusively in an aggregated expression. MySQL is often accused of violating this standard.

In this article I will attempt to debunk this myth, and to provide a more balanced view regarding MySQL’s treatment of GROUP BY at the same time.

To do that, I will first demonstrate that MySQL can be instructed to only accept GROUP BY clauses that include all non-aggregated expressions referred to in the SELECT list, thus making MySQL’s behaviour conform more to that of other well-known rdbms-products.

Second, I will show that it is very important to clearly define which version of the SQL-standard is being referred to. The two most recent versions use a rather sophisticated way of defining the required relationships between expressions appearing in the GROUP BY clause and the SELECT list. Contrary to a popular belief, these standards do not literally require that all non-aggregated SELECT list columns appear in the GROUP BY clause.

Third, I will use a simple yet realistic example to illustrate in an informal manner what I believe is the intent expressed in the more recent versions of the SQL standard. Hopefully I will be able to convince you why it may even be better to not blindly include all non-aggregated columns from the SELECT list in the GROUP BY clause.

Before we dive into the details, I’ll start with a brief introduction with regard to GROUP BY for those that are not too familiar at all with the construct. In the introduction, I will illustrate why most database products require all non-aggregated columns that are referenced in the SELECT list to appear in the GROUP BY clause, and why users run into trouble sometimes due to MySQL’s treatment of GROUP BY clause.

The `GROUP BY`-clause

So, what is the GROUP BY-clause, and what does it do?

The GROUP BY-clause is an optional element of SQL SELECT expressions. Syntactically, the GROUP BY-clause consists of the keyword sequence GROUP BY, followed by a comma-separated list of (scalar) expressions. If the SELECT expression contains a GROUP BY-clause, it must appear after the WHERE clause. (If the WHERE clause is omitted, the GROUP BY clause will immediately follow after the FROM clause.)

When included, GROUP BY specifies that rows from the intermediate result set are to be divided in a number of groups, returning one single row for each such group. The list of expressions provided in the GROUP BY list defines how the grouping takes place. All rows that have the same combination of values for all expressions specified in the GROUP BY are in the same group.

Let’s do a few simple example queries to illustrate the effect of the GROUP BY-clause. (For these examples I’ll use the pet table from the menagerie database.) The following query retrieves all rows from the pet table in the menagerie database:

SELECT  *
FROM    menagerie.pet

The query returns a result that might look like this:

+----------+--------+---------+------+------------+------------+
| name     | owner  | species | sex  | birth      | death      |
+----------+--------+---------+------+------------+------------+
| Fluffy   | Harold | cat     | f    | 1993-02-04 | NULL       |
| Claws    | Gwen   | cat     | m    | 1994-03-17 | NULL       |
| Buffy    | Harold | dog     | f    | 1989-05-13 | NULL       |
| Fang     | Benny  | dog     | m    | 1990-08-27 | NULL       |
| Bowser   | Diane  | dog     | m    | 1979-08-31 | 1995-07-29 |
| Chirpy   | Gwen   | bird    | f    | 1998-09-11 | NULL       |
| Whistler | Gwen   | bird    | NULL | 1997-12-09 | NULL       |
| Slim     | Benny  | snake   | m    | 1996-04-29 | NULL       |
| Puffball | Diane  | hamster | f    | 1999-03-30 | NULL       |
+----------+--------+---------+------+------------+------------+

(Because we did not specify an ORDER BY-clause, the rows are returned in some order determined by the database, so your results might not look exactly like this. However, for this example, the actual rows are important - not the order.)

Now, suppose we want to make groups for each species. The following addition of the GROUP BY-clause does just that:

SELECT   species
FROM     menagerie.pet
GROUP BY species -- make one group for each species

The query returns this result:

+---------+
| species |
+---------+
| bird    |
| cat     |
| dog     |
| hamster |
| snake   |
+---------+

At a glance, it seems as if the GROUP BY clause does nothing more than scan for unique occurrences in the species column and return those. However, it is better to think of each row in the GROUP BY result as a summary row that represents a group of rows that have the same value in the species column. So, in this case, the bird row represents the group of pets that are birds, (”Chirpy” and “Whistler”); the cat row represents the group of pets that are cats (”Fluffy” and “Claws”), and so on and so forth.

Calculating Aggregates for a group of rows

A GROUP BY query allows one to apply aggregate functions on the collection of rows associated with each group defined by the GROUP BY clause. An aggregate function can process expressions for each row in a group of rows to compute a single return value. A number of well-known standard aggregate functions are COUNT, MIN, MAX, and SUM.

(Aggregate functions can also be used without a GROUP BY clause, in which case the entire intermediate resultset is treated as one big group. Try imagining the effect of the GROUP BY operation with an empty GROUP BY list: the query will return just one row that summarizes all rows from the intermediate result set).

Expanding our previous example query for GROUP BY, the following example illustrates the effect of some of these aggregate functions:

SELECT   species
,        GROUP_CONCAT(name) -- make a list of pets per species
,        COUNT(*)           -- count pets per species
,        MIN(birth)         -- birthdate of oldest pet per species
,        MAX(birth)         -- birthdate of youngest pet per species
FROM     menagerie.pet
GROUP BY species

This example also includes usage of the MySQL specific GROUP_CONCAT aggregate function which will prove to be very useful to illustrate the effect of the GROUP BY-clause.

The result looks something like this:

+---------+--------------------+----------+------------+------------+
| species | GROUP_CONCAT(name) | COUNT(*) | MIN(birth) | MAX(birth) |
+---------+--------------------+----------+------------+------------+
| bird    | Chirpy,Whistler    |        2 | 1997-12-09 | 1998-09-11 |
| cat     | Fluffy,Claws       |        2 | 1993-02-04 | 1994-03-17 |
| dog     | Buffy,Fang,Bowser  |        3 | 1979-08-31 | 1990-08-27 |
| hamster | Puffball           |        1 | 1999-03-30 | 1999-03-30 |
| snake   | Slim               |        1 | 1996-04-29 | 1996-04-29 |
+---------+--------------------+----------+------------+------------+

Again, we see one row for each group of rows that have an identical value in the species column, but this time, we also see the effect of processing the individual rows for each species using aggregate functions:

The GROUP_CONCAT function was applied to the name column. For each species in our pet table, there maybe multiple pets, and GROUP_CONCAT concatenates their names, separating the individual names by default with a comma. Thus in this example, the GROUP_CONCAT expression reveals the make-up of each group of pets of a single species.
The COUNT function is used with a wildcard *, instructing it to count the number of rows associated with each group. Verifying this with the previous column, we can immediately see that the number of rows in the group is consistent with the number of names concatenated by the GROUP_CONCAT expression
The MIN and MAX functions are applied to the birth column and for each species, respectively report the birth date of whichever pet is oldest (MIN(birth), the birth date that is smaller than any of the other birth dates) and youngest (MAX(birth), the birth date that is larger than any of the other birth dates).

So, aggregate functions are applied to expressions taken from a group of rows and have the effect of ‘condensing’ (aggregating) the group, yielding a single value. Most aggregate functions calculate or determine some kind of statistical metric which serves to characterize the group as a whole. The MySQL specific GROUP_CONCAT function is an exception: it simply enumerates all members in the group passed to the function, and as such it is not a statistical function. However, it still exposes the main property of aggregate functions, namely the ability to turn the expressions from a group of rows into a single value.

Running into trouble with `GROUP BY`

So far, we’ve seen a few examples with GROUP BY that make perfect sense. Yet is easy to run into trouble with GROUP BY. Take a look at the following query:

SELECT   species
,        MIN(birth)  -- birthdate of oldest pet per species
,        MAX(birth)  -- birthdate of youngest pet per species
,        birth       -- birthdate of ... uh oh...!
FROM     menagerie.pet

This query is similar to the previous query, where we calculated a few aggregates for each group of pets belonging to the same species. However, this time, we also include the a plain reference to the birth column in the SELECT list.

An attempt to run this query on Oracle results in an error:

SQL> SELECT  species
 2  ,        MIN(birth)
 3  ,        MAX(birth)
 4  ,        birth
 5  FROM     sakila.pet
 6  GROUP BY species;
,       birth
       *
ERROR at line 4:
ORA-00979: not a GROUP BY expression

Running this query on MySQL however does return a result, which may look something like this:

mysql> SELECT   species
    -> ,        MIN(birth)  -- birthdate of oldest pet per species
    -> ,        MAX(birth)  -- birthdate of youngest pet per species
    -> ,        birth       -- birthdate of ... uh oh...!
    -> FROM     menagerie.pet
    -> GROUP BY species;
+---------+------------+------------+------------+
| species | MIN(birth) | MAX(birth) | birth      |
+---------+------------+------------+------------+
| bird    | 1997-12-09 | 1998-09-11 | 1998-09-11 |
| cat     | 1993-02-04 | 1994-03-17 | 1993-02-04 |
| dog     | 1979-08-31 | 1990-08-27 | 1989-05-13 |
| hamster | 1999-03-30 | 1999-03-30 | 1999-03-30 |
| snake   | 1996-04-29 | 1996-04-29 | 1996-04-29 |
+---------+------------+------------+------------+
5 rows in set (0.00 sec)

What is happening here? Why do we see such different behaviours? In fact, what is MySQL’s behaviour in this case? Sometimes, the birth column reports a value that looks like the maximum value for birth within the species (first row), and sometimes we see the maximum value (second row). We even see one case where the returned value is in between the minimum and maximum values (row 3). How can we explain this seemingly random behaviour?

Understanding the Problem

It’s not too hard to deduce what is happening here. All we need to do is go back to our explanation of the effect of the GROUP BY clause, and see how it applies to our last query.

It was already explained that the GROUP BY clause returns one row for each group of rows in the intermediate result, and that the groups are defined by the expression list defined in the GROUP BY clause. So, in this case, we are creating one result row for each group of rows that belong to the same species because the GROUP BY list only contains the species column. Yet there are several pets that may belong to a specific species, so the birth column may have (and often has) a different value for each row in a particular species group.

Having realized that, we can now ask ourselves: assuming there are multiple values in the birth column for a particular value of species, which one should be returned? What did we mean when we specified the birth column in the SELECT list?

There is no good answer to this question. It is certainly possible to select just one of the possible values for the birth column: in fact, this is exactly what MySQL does. However, it is impossible to define the significance of which ever of the possible values is chosen. That is, it makes no sense to want to mix the plain values from the rows that belong to the group with the group itself. Therefore it does not make sense to even include the birth column in the SELECT list of the query.

Another way of looking at it is to say that the ‘grain’ of species values is different (and therefore, incompatible) with the grain of the birth values. It does not mean we cannot access the values in the birth column; it merely means we must use an aggregate function to compute the ‘right’ value from a whole group of them.

Avoiding the Problem

What about the behaviour such as exercised by Oracle? Doesn’t it make more sense to issue an error message rather than returning non-sense data? Put this way, most people will probably agree. Of course, the error message itself is somewhat puzzling:

,       birth
       *
ERROR at line 4:
ORA-00979: not a GROUP BY expression

This seems to suggest that the problem is that the birth column in the SELECT list is not included in the GROUP BY clause. In turn, this raises the question whether the problem would be solved if we would have included the birth column in the GROUP BY clause.

Well, including the birth column in the GROUP BY clause certainly gets rid of the error message. However, a lot of users with entry level skills in SQL fail to understand that this yields quite a different query. The original GROUP BY species yields one group for each species, whereas GROUP BY species, birth yields a group for each combination of values in species and birth - most probably not at all what is intended.

On the other hand, we cannot expect the database management system to know what we were thinking when we included the birth in the SELECT list in the first place. So, despite that the error message may seem a bit puzzling, it is still preferable over silently returning non-sense data.

But do we really have to put up with this? The answer is “No!”.

Including `ONLY_FULL_GROUP_BY` in MySQL’s `sql_mode`

Nowadays, MySQL is capable of detecting this problem too, and it is perfectly possible to make MySQL reject the previous query to avoid the problem of returning non-sense data. This is achieved by including ONLY_FULL_GROUP_BY in the sql_mode.

Like many server settings, we can specify the sql_mode using the --sql-mode command line argument to the MySQL server executable (mysqld), or we can include it in an option file. For example, including the following line in the option file will enable ONLY_FULL_GROUP_BY when the server starts up:

sql_mode=ONLY_FULL_GROUP_BY

Beginning with MySQL 4.1, it is also possible to set the sql_mode at runtime using the SET syntax. In this way, the sql_mode can be set globally or for the session level. The latter is the most useful, as it allows one to setup a sql_mode most suited for a particular application without affecting any other applications that run on the server. (Some applications don’t expect anything other than the default setting and may run into trouble with a particular sql_mode.)

The following snippet illustrates how to include ONLY_FULL_GROUP_BY in the sql_mode at runtime:

mysql> SET sql_mode := CONCAT('ONLY_FULL_GROUP_BY,',@@sql_mode);
Query OK, 0 rows affected (0.00 sec)

The @@sql_mode server variable contains a possibly empty, comma-separated string of current sql_mode settings. The CONCAT expression prepends whatever the current setting is of the sql_mode with ONLY_FULL_GROUP_BY. Note the comma immediately following ONLY_FULL_GROUP_BY. If the value of @@sql_mode is the empty string, the value of the CONCAT expression will have a trailing comma, but this is allowed in the assignment (trimming off the comma in the process).

When we now attempt to execute our query again, it fails with an error message:

mysql> SELECT   species
    -> ,        MIN(birth)  -- birthdate of oldest pet per species
    -> ,        MAX(birth)  -- birthdate of youngest pet per species
    -> ,        birth       -- birthdate of ... uh oh...!
    -> FROM     menagerie.pet
    -> GROUP BY species;
ERROR 1055 (42000): 'menagerie.pet.birth' isn't in GROUP BY

The error messages indicates that we did not include the birth column in the GROUP BY clause. Now, MySQL behaves similar to Oracle for this query.

(Alas, MySQL’s ONLY_FULL_GROUP_BY is not as clever as it should be, and there are particular cases where ONLY_FULL_GROUP_BY is too restrictive in enforcing only full GROUP BY clauses. The details are described here. The good news is that the community can help to fix this bug! Go to MySQL Forge and check out Worklog task 2489.)

What does ‘the’ SQL standard say

In the previous sections, we’ve seen how Oracle and MySQL react very differently to the same SQL GROUP BY query. But what do the standards say? How is the GROUP BY clause supposed to behave?

In the introduction of this article, I claimed I would debunk a popular myth that holds that

…standard SQL requires columns referenced in the SELECT list of a query to also appear in the GROUP BY clause, unless these columns appear exclusively in an aggregated expression.

Now, I don’t want to pretend I’m an expert as far as the SQL standard (ISO/IEC 9075) is concerned. In fact, I’ve noticed repeatedly that the sheer volume of the documentation as well as the persistent formal wording prevent me from obtaining a clear overview of it. But, let’s give it a try anyway.

The 1992 version of the standard, 7.9 - 7 states that:

If T is a grouped table, then each <column reference> in each <value expression> that references a column of T shall reference a grouping column or be specified within a <set function specification>.

As said, I’m not an expert in this area, but the way I read it it boils down to:

Queries that include a GROUP BY clause can only include column references in SELECT-ed expressions if the column appears in the GROUP BY clause, or if that column appears as part of an aggregate.

Now, in the 7.12 - 15 of the 2003 version of the standard we find this:

If T is a grouped table, then let G be the set of grouping columns of T. In each <value expression> contained in <select list>, each column reference that references a column of T shall reference some column C that is functionally dependent on G or shall be contained in an aggregated argument of a <set function specification> whose aggregation query is QS.

The 1999 version of the standard contains a similar rule. The important thing to note here is that both versions that succeeded the 1992 version stopped requiring explicitly that all non-aggregated columns in the SELECT list must be present in the GROUP BY clause. Instead they require that any non-aggregated column appearing in the SELECT list is functionally dependent upon the GROUP BY clause.

Functional dependencies

What would the 1999 and 2003 version of the SQL standard mean by the term “functionally dependent”? The answer to that question is also defined by the standard. Unfortunately, it cannot be illustrated by a simple quote, as the formal definition of what exactly constitutes a functional dependency according to the standard is fairly extensive and complicated.

Luckily, the concept of functional dependencies can be easily illustrated in a less formal way. Suppose we have two expressions, A and B. Now, B is functionally dependent upon A if B has exactly one value for a particular value of A. Consider this snippet of code:

mysql> SELECT @A:=1      AS A
    -> ,      @B:=@A + 1 AS B;
+---+------+
| A | B    |
+---+------+
| 1 |    2 |
+---+------+

Here, the column B is functionally dependent upon column A. The value of B can be derived from the value of A in a very straightforward manner, namely by adding 1 to whatever the value of A happens to be. We know how to compute the value of B for any given value of A, and for any given value of A, the corresponding value of B will always be the same.

The functional dependency concept can also be be applied to multiple columns:

SELECT CONCAT(A,B) C
FROM   someTable

Column C is defined as the result of CONCAT(A,B) expression. If we have the values of both A and B we can compute the result of CONCAT(A,B). Of course, for a given pair of expressions for A and B the result of CONCAT(A,B) will always be the same. Therefore, C is functionally dependent upon the column pair A and B.

Note that it is not enough just to only know the method to calculate B out of A. Consider this example:

mysql> SELECT @A:=1           AS A
    -> ,      @B:=@A + RAND() AS B;

Here, we know the recipe to derive the value of A from the value of A: we have to take the value of A and add the value returned by a call to the RAND() function. However, the RAND() function will return a different value each time it is called. Therefore, the value of B will be different too every time, and hence we cannot say that B is functionally dependent upon A. There is thus not a single value for B for a given value of A, and thus B is not functionally dependent upon A.

Functional dependency and Normalization

The term functional depencency is also used with regard to normalization. Part of the normalization process involves discovering functional dependencies between different groups of columns.

Normalization requires that each table has at least one key. A key is a column or group of columns that may be used to identify a single record in the table. By definition, if we have a key, all non-key columns are functionally dependent upon the key. Another way to think about this is to imagine that we look up a row using a key. If the key entry exists, this will result in exactly one row. By definition, each column in that row has exactly one value, so the value of each non-key column can be determined by the key.

Functional dependencies between a group of columns that makes up a key and any other group of columns is allowed, but functional dependencies between two groups of non-key columns are eliminated by the normalization process. (This is done by splitting off the groups of columns that expose the functional dependency to a new table, making one of the groups of columns a key in the new table.)

Functional dependency and `GROUP BY`

We have just seen that the 1999 and 2003 versions of the SQL standard require that the columns appearing in the SELECT list are functionally dependent upon the groups defined by the GROUP BY clause. In other words, if we know that a column contains only one value for any given combination of values in the columns appearing in the GROUP BY clause, we may reference the column in the SELECT list even if it does not appear in an aggregate expression.

We’ve also seen that if we have a (primary or unique) key, all columns that are not included in the key are by definition functionally dependent upon the key. This means that if we include all key columns in the GROUP BY clause, we can reference any column we like in the SELECT list, even if they appear outside an aggregate expression.

The following example from the sakila sample database might help to illustrate this:

mysql> SELECT   film_id       -- primary key
    -> ,        title         -- non-key column
    -> ,        COUNT(*)      -- one row per group
    -> FROM     sakila.film
    -> GROUP BY film_id;      -- group by on primary key
+---------+-----------------------------+----------+
| film_id | title                       | COUNT(*) |
+---------+-----------------------------+----------+
|       1 | ACADEMY DINOSAUR            |        1 |
.         .                             .          .
.         .                             .          .
[...not showing 998 rows...]
.         .                             .          .
.         .                             .          .
|    1000 | ZORRO ARK                   |        1 |
+---------+-----------------------------+----------+
1000 rows in set (0.05 sec)

Here, we query the film table. The primary key of the film table consists of only the film_id column. The GROUP BY clause contains the film_id column. As a result, the query returns a collection of groups, each of which summarizes only one row. Of course, because there is only one row per group, there can be only one value in each of the other columns of the film table. Therefore, it is safe to include whatever column we like in the SELECT list. For this reason, it is perfectly ok to include the film_title column in the SELECT list.

Of course, the GROUP BY in the previous query does not make sense logically. Because the primary key of the film table consists of only the film_id column, we already know there can be only one row for any given value of film_id. However, it becomes interesting when we include another table in the query. Consider the next example:

mysql> SELECT    f.film_id
    -> ,         f.title
    -> ,         COUNT(fa.actor_id)
    -> FROM      film        f
    -> LEFT JOIN film_actor  fa
    -> ON        f.film_id = fa.film_id
    -> GROUP BY  f.film_id;
+---------+-----------------------------+----------+
| film_id | title                       | COUNT(*) |
+---------+-----------------------------+----------+
|       1 | ACADEMY DINOSAUR            |       10 |
.         .                             .          .
.         .                             .          .
[...not showing 998 rows...]
.         .                             .          .
.         .                             .          .
|    1000 | ZORRO ARK                   |        3 |
+---------+-----------------------------+----------+
1000 rows in set (0.02 sec)

Here, we have added a LEFT JOIN to calculate the number of actors per film. This time, the GROUP BY clause on the film_id does make sense: we now get one group of actors that play a role in each film. At the same time, we know that all columns in the film table are functionally dependent upon the film_id column. Each group returned by the GROUP BY clause corresponds to exactly one row from the film table, and this means that for each group, there is only one value in any column of the film table. So, it’s perfectly safe to reference the columns from the film table in the SELECT list, even if we don’t use them in an aggregate expression.

It is important to realize that we cannot reference just any column in the SELECT list: we can only reference those columns that are functionally dependent upon the film_id column of the film table. This means that it is wrong to reference any column of the film_actor table in the SELECT list directly: we may do so only in an aggregate expression.

The previous example demonstrates a pattern. The film table acts as the so-called master, and the film_actor table acts as the detail. The master-detail pattern is very common: Order and Order Items, Vendor and Products, Country and Cities are all examples of this pattern.

So why would I do that?

Ok, hopefully, I’ve been able to explain under which circumstances it is safe for GROUP BY queries to reference columns in the SELECT list directly. One might wonder though what the advantages and disadvantages are. I mean, just because you can doesn’t mean you should, right?

Disadvantages?

Well, there certainly are reasons for always writing a full GROUP BY clause.

First of all, many rdbms products will only allow a full GROUP BY clause anyway, so in those cases there really is no choice. In MySQL however, we do have the choice as long as we are not using ONLY_FULL_GROUP_BY in the sql_mode.

Another reason for always writing the GROUP BY clause in full is that other developers might not understand under which circumstances it is ok to use a partial GROUP BY clause. In many cases, they’ve spent considerable time to learn to blindly repeat all SELECT columns in the GROUP BY clause, and they will usually point out that it is wrong to not adhere to that rule.

Of course, it is impossible to distinguish between a query that intentionally omits columns from the GROUP BY clause and one that accidentally forgot to include them. When it is the intention to always write a full GROUP BY clause, it is easy to verify whether the column references in the GROUP BY clause and the the SELECT list match.

I have heard people argue that including only the key in the GROUP BY clause will lead to problems when the definition of the key is changed. Personally, I think this is a bogus argument. When you are considering to change the definition of the key, you are most likely going to review all your queries anyway, because all your joins will need to reflect this change too. I just mean to say that changing a few GROUP BY’s here and there is probably least of your problems when you are considering to change the definition of a key.

Another argument I have heard is that it is somehow ‘more clear’, ‘cleaner’ or ‘prettier’ to repeat all columns referenced in the SELECT list in the GROUP BY clause. Personally, I think this is a bogus argument too. At the very least, these are all a matter of opinion.

Advantages

Personally, I feel it is more clear and prettier to GROUP BY only on key columns where possible. I argued that it is a matter of opinion what is ‘clear’ or ‘pretty’, so I must discard this argument likewise.

I would argue that full GROUP BY clauses are harder to maintain. Many changes will require two edits of the code instead of one. Of course, this might or might not outweigh any of the advantages of a full GROUP BY clause.

A full GROUP BY clause might be slower than a partial one. The following query finds all film titles that gathered more than 300 $ worth of payments:

mysql> SELECT    f.film_id
    -> ,         f.title
    -> ,         sum(p.amount) sum_amount
    -> FROM      film f
    -> LEFT JOIN inventory i
    -> ON        f.film_id = i.film_id
    -> LEFT JOIN rental r
    -> ON        i.inventory_id = r.inventory_id
    -> LEFT JOIN payment p
    -> ON        r.rental_id = p.rental_id
    -> GROUP BY  f.film_id
    -> HAVING    sum_amount > 300;
Empty set (0.18 sec)

Using only the film_id column in the GROUP BY clause, it takes 0.18 seconds to discover there are no film titles that satisfy this criterion. Now, let’s compare this to the equivalent query using a full GROUP BY clause:

mysql> SELECT    f.film_id
    -> ,         f.title
    -> ,         sum(p.amount) sum_amount
    -> FROM      film f
    -> LEFT JOIN inventory i
    -> ON        f.film_id = i.film_id
    -> LEFT JOIN rental r
    -> ON        i.inventory_id = r.inventory_id
    -> LEFT JOIN payment p
    -> ON        r.rental_id = p.rental_id
    -> GROUP BY  f.film_id
    -> ,         f.title
    -> HAVING    sum_amount > 300;
Empty set (0.51 sec)

This query takes almost three times as long to complete! With EXPLAIN we can retrieve the execution plans for these queries. Without a full GROUP BY clause, we see a fairly normal execution plan:

mysql> EXPLAIN
    -> SELECT    f.film_id
    -> ,         f.title
    -> ,         sum(p.amount) sum_amount
    -> FROM      film f
    -> LEFT JOIN inventory i
    -> ON        f.film_id = i.film_id
    -> LEFT JOIN rental r
    -> ON        i.inventory_id = r.inventory_id
    -> LEFT JOIN payment p
    -> ON        r.rental_id = p.rental_id
    -> GROUP BY  f.film_id
    -> HAVING    sum_amount > 300
    -> G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: f
         type: index
possible_keys: NULL
          key: PRIMARY
      key_len: 2
          ref: NULL
         rows: 953
        Extra:
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: i
         type: ref
possible_keys: idx_fk_film_id
          key: idx_fk_film_id
      key_len: 2
          ref: sakila.f.film_id
         rows: 2
        Extra: Using index
*************************** 3. row ***************************
           id: 1
  select_type: SIMPLE
        table: r
         type: ref
possible_keys: idx_fk_inventory_id
          key: idx_fk_inventory_id
      key_len: 3
          ref: sakila.i.inventory_id
         rows: 1
        Extra: Using index
*************************** 4. row ***************************
           id: 1
  select_type: SIMPLE
        table: p
         type: ref
possible_keys: fk_payment_rental
          key: fk_payment_rental
      key_len: 5
          ref: sakila.r.rental_id
         rows: 1
        Extra:
4 rows in set (0.00 sec)

With a full GROUP BY list, we notice a difference for the film table:

mysql> EXPLAIN
    -> SELECT    f.film_id
    -> ,         f.title
    -> ,         sum(p.amount) sum_amount
    -> FROM      film f
    -> LEFT JOIN inventory i
    -> ON        f.film_id = i.film_id
    -> LEFT JOIN rental r
    -> ON        i.inventory_id = r.inventory_id
    -> LEFT JOIN payment p
    -> ON        r.rental_id = p.rental_id
    -> GROUP BY  f.film_id
    -> ,         f.title
    -> HAVING    sum_amount > 300
    -> G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: f
         type: index
possible_keys: NULL
          key: idx_title
      key_len: 767
          ref: NULL
         rows: 953
        Extra: Using index; Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: i
         type: ref
possible_keys: idx_fk_film_id
          key: idx_fk_film_id
      key_len: 2
          ref: sakila.f.film_id
         rows: 2
        Extra: Using index
*************************** 3. row ***************************
           id: 1
  select_type: SIMPLE
        table: r
         type: ref
possible_keys: idx_fk_inventory_id
          key: idx_fk_inventory_id
      key_len: 3
          ref: sakila.i.inventory_id
         rows: 1
        Extra: Using index
*************************** 4. row ***************************
           id: 1
  select_type: SIMPLE
        table: p
         type: ref
possible_keys: fk_payment_rental
          key: fk_payment_rental
      key_len: 5
          ref: sakila.r.rental_id
         rows: 1
        Extra:
4 rows in set (0.01 sec)

In case you did not yet notice, the film table now has Extra: Using index; Using temporary; Using filesort.

What I think that is happening is that MySQL takes the GROUP BY clause literally and performs the GROUP BY algorithm for each of the specified expressions. MySQL implements GROUP BY by sorting the rows according to the GROUP BY expressions. In this particular case, adding the title column to the GROUP BY clause does not allow the server to sort the rows in-memory, and forces the GROUP BY to be evaluated using a temporary table and a file sort. This requires extra IO operations which is causing a decrease in performance.

Of course, it would be nice if MySQL was smart enough to deduce that the result cannot possibly be different due to including the title column in the GROUP BY list. It could attempt to detect that a key of the film table is included in the GROUP BY list and that the title column can be completely ignored when evaluating the GROUP BY clause, because there will be exactly one value in the title column for each film_id column. But, then again, MySQL does not require us to write a full group by list. So if performance is paramount, be smart and do not write a full GROUP BY clause.

Aggregating on functionally dependent columns

We just argued that it is safe to include columns in the SELECT list as long as these columns are functionally dependent upon the GROUP BY list. The reasoning is that since the columns are functionally dependent upon the GROUP BY clause, there will be one value for each result group anyway. It was also shown that it may be a bad idea to include these columns in the GROUP BY clause, as it can hurt performance.

For some people, it may still seem unacceptable to include functionally dependent columns in the SELECT list without referencing these columns in the GROUP BY clause. For example, you might be using a rdbms that requires all columns that are not referenced in the GROUP BY to be aggregated. In those cases is might be better to apply an aggregate function to the functionally dependent column rather than including it in the GROUP BY clause. Consider the following query:

mysql> SELECT    f.film_id
    -> ,         MAX(f.title) AS title
    -> ,         sum(p.amount) sum_amount
    -> FROM      film f
    -> LEFT JOIN inventory i
    -> ON        f.film_id = i.film_id
    -> LEFT JOIN rental r
    -> ON        i.inventory_id = r.inventory_id
    -> LEFT JOIN payment p
    -> ON        r.rental_id = p.rental_id
    -> GROUP BY  f.film_id
    -> HAVING    sum_amount > 300;
Empty set (0.20 sec)

Again, we GROUP BY on the film_id column, which makes up the primary key of the film table. This time however, we apply the MAX aggregate function on the title column. We know that there is only one value in the title column for each value in the film_id column, so the aggregation will not influence the result. In fact, we could’ve used MIN equally well.

These aggregate functions will return the right value for exactly the same reason why it is safe to not include the functionally dependent column in the GROUP BY clause. Technically the aggregation is therefore unnecessary: it is just a trick to fool the rdbms.

Most likely, applying the aggregate function will be somewhat slower than not applying the aggregate. However, it will in most cases be faster than including the functionally dependent column in the GROUP BY clause.

Conclusion

Contrary to popular belief, the SQL standard does not require GROUP BY queries to reference all non-aggregated columns from the SELECT list in the GROUP BY clause. As of the 1999 version of the SQL standard, it is explicitly allowed for the SELECT list to reference non-aggregated expressions as long as they are functionally dependent upon the GROUP BY list.

Each expression that has exactly one value for each group defined by the GROUP BY clause is functionally dependent upon the GROUP BY clause. Functional dependencies can be witnessed in a common query pattern: whenever we have a join between a master and a detail table to calculate aggregates over the detail rows for each row from the master, we can GROUP BY over the primary or unique key from the master. All non-key columns of the master row will be functionally dependent upon the key, and can thus appear in the SELECT-list outside an aggregate expression.

In MySQL, one can write GROUP BY queries that reference non-aggregated columns in the SELECT list that are not included in the GROUP BY clause, even if these columns are not functionally dependent upon the GROUP BY clause. This behaviour conforms to none of the SQL standard’s versions. It is possible to avoid this behaviour by including ONLY_FULL_GROUP_BY in the sql_mode server setting, but it might make more sense to take advantage of the ability to write only partial GROUP BY clauses.

In a nutshell:

It is completely safe to write partial GROUP BY clauses as long as all non-aggregated columns in the SELECT list are functionally dependent upon the GROUP BY clause.
A partial GROUP BY list can result in better performance, because it keeps the server from evaluating the entire GROUP BY list.
If one does not want to write partial GROUP BY clauses, consider using MIN or MAX to ‘aggregate’ the functionally dependent columns in the SELECT list rather than moving the functionally dependent columns to the GROUP BY clause.

Hacking MySQL table logs

2007-05-08T18:30:23Z

Shortly before MySQL Users Conference I announced that I would be cover new ground in table logs management.
I am keeping that promise, and in addition I am also showing some related hacks.

The announced facts from last year usability report were that you can't change log tables at will, as you can do with log files, and you can't change the log table engine to FEDERATED. Both claims, as it turned out, were incorrect. You can do such things, albeit not in a straightforward manner. As a bonus side effect, you can also:

add triggers to log tables;
filter log tables depending on user defined criteria, such as query type, user database, or time;
centralize logs from several servers.

DISCLAIMER. The information in this article is essentially a hack. It is not recommended nor endorsed by MySQL AB.

Overview: switching logs basics

Switching log files is easily done in just one step, assuming that you are using files as log output.
To enable file usage, you must enable the general_log variable and set the log_output variable to 'FILE'.
In this situation, changing to a dedicated sometask.log file requires just one command:


set global general_log_file='/tmp/sometask.log';

Going back to the default log file, or to any other, is just a command away:


set global general_log_file='/usr/local/mysql/data/general.log';

Switching tables is a little trickier. The default log table location (_mysql.general_log_) is not negotiable. Thus, we need to adopt a more devious approach.

First, we create a table with the same structure as the default one.


DROP TABLE IF EXISTS gl0, gl1,
CREATE TABLE mysql.gl1 like mysql.general_log;

If needed, we change the engine to a something more efficient than CSV. This means basically MyISAM, since any other engine will be refused. But the following hacks will circumvent this limitation as well. One of the reasons for changing engine is if we want to add a key that we plan to user later for searches.
```
ALTER TABLE gl1 ENGINE=MyISAM, KEY (user_host);
```

Then, we rename both tables at once:


RENAME TABLE general_log to gl0, gl1 to general_log;

That's all. The default table has always the same name, but we can switch to one table to another, if we need to log action during a given task.
The above method is also important because it is the basis for the following hacks, which make dynamic logging much more interesting.

How to: smarter logs

We are now ready to take the hack to a new dimension. The original usability report mentions that you can only use the CSV and MyISAM engines for logging. Any attempt to assign a different engine will fail. While this is technically accurate, we can use the above trick to cheat the server into using a FEDERATED table for logging.

If you have dealt with general logs before, you would know that logs introduce two administrative problems:

they grow large very quickly, eating up precious storage space;
if the server crashes for a hardware failure, your logs are lost.

A simple way of dealing with the first problem is using the Archive storage engine for logging. You can't assign it directly, but you can rename an existing table.


# Squeezing table logs
DROP TABLE IF EXISTS gl0, gl1,
CREATE TABLE gl1 like general_log;
ALTER TABLE gl1 ENGINE=ARCHIVE;
RENAME TABLE general_log to gl0, gl1 to general_log;

The ARCHIVE table will work exactly as the CSV one, with the important difference that it will only take 15~20% of its storage. Since log tables must only support INSERT statements, using an Archive engine makes sense.

How to: centralized remote logs

Another way is to send the logs to a different server. The second issue can be solved with the same trick. Current table logging implementation does not allow triggers, but if you have federated your log table to a remote one, nothing prevents the usage of triggers as well.

To create remote logs, act as follows:

Install a MySQL server on a box with sufficient storage for your purposes. The version is not important, but if you want to use triggers, version 5.0 or higher is necessary.
On the remote server, create a database for logging.
Inside that database, create a table with the same data structure of the general_log table, using your favorite storage engine.
In the local server, create a table with the same structure as the general log, federated to the remote server.
Use the technique illustrated in the previous paragraph to rename the new table to the general_log table.
Enable table logging and start seeing the effects.


# remote server
CREATE SCHEMA logs;
USE logs;
CREATE TABLE mylog LIKE mysql.general_log;
ALTER TABLE mylog ENGINE=MyISAM, KEY (user_host);

# local server
create server logserver foreign data wrapper mysql options (
host 'remote_server.net',
database 'logs',
port 3306,
user 'remote_user_name',
password 'remote_secret');

use mysql;
DROP TABLE IF EXISTS gl1, gl0;
CREATE TABLE gl1 (
event_time timestamp NOT NULL,
user_host mediumtext,
thread_id int DEFAULT NULL,
server_id int DEFAULT NULL,
command_type varchar(64) DEFAULT NULL,
argument mediumtext
) ENGINE = FEDERATED
CONNECTION = 'logserver/mylog';
# Note: MySQL 5.1.18 is required for the above syntax
# For older versions, you can use the old syntax instead:
# CONNECTION='mysql://user:password@hostname:port/schema/table'

RENAME TABLE general_log to gl0, gl1 to general_log;

When the preparation is over, your general log will end up in the remote server.

Once this organization is in place, you may start thinking about expanding it. If you can relocate log activity from one server, you can do the same for multiple ones.

What happens if you repeat the same steps for another server, pointing to the same remote server,schema and table? Exactly what you would expect. All the queries from both servers will end up in the same table. You will be centralizing your logs, but you need a way of telling the queries from different servers apart. It can be easily accomplished. The log table includes a server_id column, which records the unique ID of the server. If you assign a different server ID to each server, then you can use a centralized remote log for several servers.
On each server, issue the command


SET GLOBAL server_id = <a unique number>;

and your centralized logs will be also easily searchable.

More hacks: filtering with triggers

We mentioned before that logs don't allow triggers. That's a limitation builtin in the mysql database, where no table can have triggers. However, we are federating a table to an remote server, in a normal database, where triggers are not restricted.
Thus, although we can't add a trigger to the log table itself, nothing prevents us from adding one to the shadow table in the remote server.

Let's consider the common case where the DBA wants to log in two separate tables events related to sales and customers, and discard all the rest.
That is simply done. Filtering the wanted records and inserting them into the appropriate tables can be done inside a trigger. Discarding the rest is tricky, and it could not be done with standard SQL. You need to assign the remote log table to the BlackHole storage engine, which is an engine that simply discards everything. The data is passed to the table, and thus a BEFORE INSERT index will work just fine.
The steps to achieve our goal are:

Create the remote log tables that will store the wanted records;
Alter the remote log table, so that it uses the BlackHole engine;
create a trigger that filters the records and inserts the right ones in the auxiliary tables.

In the following example, the trigger performs a brutish check to see if the query contains 'customer' or 'sales'. In a real world situation you may want to perform a more exhaustive check, but for now this would do.


delimiter //

create trigger logs_bi
    BEFORE INSERT
    ON logs
    FOR EACH ROW
begin
    if (new.argument regexp 'customer')
    then
        insert into log_customers
            (event_time, user_host, thread_id,
            server_id, command_type, argument)
        values
            (new.event_time, new.user_host, new.thread_id,
            new.server_id, new.command_type, new.argument);
    end if;
    if (new.argument regexp 'sales')
    then
        insert into log_sales
            (event_time, user_host, thread_id,
            server_id, command_type, argument)
        values
            (new.event_time, new.user_host, new.thread_id,
            new.server_id, new.command_type, new.argument);
    end if;
end //

delimiter ;

Wrapping up

Table logs are an excellent addition in MySQL 5.1. They allow logging on demand, which is one of the most appreciated features for DBAs.
If you add some flexibility in your choice of storage engines for logging, this feature can be truly a bless for demanding administrative tasks.

It would be desirable to change log tables storage engine without much fuss. MySQL developers are working on it. In the meantime, you can use the workarounds in this article to twist your logs the way you want them.

I would like to thank Tobias Asplund and Petr Chardin for useful hints to complete these hacks.

Your Turn to Play Stump the Author...

2007-05-08T17:51:05Z

Thought I’d pass along this note I received, that may be of interest to the MySQL addicts out there:

>> Sasha Pachev, whose book Understanding MySQL Internals was released
>> last month by O’Reilly, is leading an online seminar at MySQL AB on
>> “Improving query performance through a better understanding of the
>> optimizer”:
>>
>> http://www.mysql.com/news-and-events/web-seminars/sasha.php
>>
>> You can present Sasha with your own SQL queries during this webinar
>> and learn how to interpret output of the EXPLAIN command to improve
>> your performance. This webinar is also a useful accompaniment to
>> Understanding MySQL Internals, which contains extensive information
>> on EXPLAIN and the behavior of the optimizer exposed by it

O'Reilly Databases

MySQL backups using ZFS snapshot

ZRM for MySQL installation

MySQL server configuration

ZRM configuration

Perform ZRM backups

ZRM backup reports

Database recovery

Improved Snapshot interface for MySQL backups

Reporting MySQL Internals with Information Schema plug-ins

A Quick Recapitulation

How Information Schema plug-ins can access MySQL Server internals

Public accessors to the current THD instance

plugin.h describes a public interface

Beyond the public interface

The MYSQL_SERVER define

Implementing the MYSQL_SAVEPOINTS Information Schema plug-in

Creating the source file

The MYSQL_SERVER define

Include files

Defining the columns

Filling the table

The rest of the implementation

Building and Installing

Compiling

Installing the plug-in

Using the plug-in

Learn More

Meet us at the user’s conference

MySQL Information Schema Plugins: the best kept secret of MySQL 5.1

The MySQL Plug-in API

Currently supported plug-in types

A closer look at the plug-in API

Information Schema Plugins

The type-specific API for Information Schema plug-ins

Writing an Information Schema Plugin

Creating the source file

Includes

Defining the columns

Filling the table

Putting together the plug-in type-specific implementation

Putting together the generic plug-in implementation

Building and installing the plugin

Compiling the plugin source file

The plugin directory

Installing the plugin

Verifying installation

Using the plug-in

Uninstalling the plugin

Learn More

Sun buys MySQL

Calculating the Financial Median in MySQL

Some background

The snippet

The mean of the left and right median

GROUP_CONCAT: an ordered list of values

Getting the ‘left’ half of the list

Excising the ‘left’ median

Getting the ‘right’ median

A few remarks

Random RDBMS and SQL Myths debunked

Kettle Tip: Using java locales for a Date Dimension

Prerequisites

Overview

Step-by-Step

MySQL JDBC driver: setting the characterEncoding property to UTF8

Creating the date dimension table

Generating 10 years worth of days

Counting the days

Calculating date dimension Attributes

Initialization

Getting Text representations of full dates

Formatting date parts

Dealing with Quarters

Period demarcation flags

A few more date attributes

Defining the step outputs

Discarding Fields

Inserting data into the table

Running the transformation

Public accessors to the current `THD` instance

`plugin.h` describes a public interface

The `MYSQL_SERVER` define

Implementing the `MYSQL_SAVEPOINTS` Information Schema plug-in

`GROUP_CONCAT`: an ordered list of values

The `GROUP BY`-clause

Running into trouble with `GROUP BY`

Including `ONLY_FULL_GROUP_BY` in MySQL’s `sql_mode`

Functional dependency and `GROUP BY`