Women in Technology

Hear us Roar



Article:
  Writing Apache's Logs to MySQL
Subject:   Database logging modules very bad idea
Date:   2005-02-21 22:14:31
From:   Coyote-Moon
It is a very bad idea to use an Apache module that logs directly to the database. I know there are countless implementations, but most are toys that don't handle even the most trivial real world issues such as connection pooling and temporarily inaccessible databases. Screw it up and cycling your database (e.g., to apply a security upgrade) could cause your web server to hang. Oops.


A far better design is to use "piped logs" that write logs messages to a separate process's stdin. This gives you clean separation between Apache and the database-enabled logger and there are ways to ensure the pipe is never blocked. (E.g., one thread copies stdin to an internal buffer, a second thread consumes that buffer to feed the database. This allows you to support bursts that far exceed your ability to write to the database.)


Could these tricks be done within an Apache module? Probably, but you would have to get it working reliably within a far more complex environment. Why not simplify your life and use a separate process?


P.S., it shouldn't be hard to normalize your data before writing it to the logs, it's not like much needs to be done. In my implementation the only thing I needed to handle is a separate IP table that has the canonical host name, RBL information, etc. It's trivial to support with a placeholder and separate thread every time a previously unknown IP address is seen.