Blog

I will be speaking about ModSecurity at ApacheCon Europe in Amsterdam later this year. I hear ApacheCon Europe 2007 (also in Amsterdam) was great so I am looking forward to participating this year. Interestingly, for some reason or another, this will be the first time ModSecurity will be "officially" presented to the Apache crowd, in spite of the fact we've been going at it for years. As always, the best part is meeting the people you've been communicating with for years.

"Intrusion detection is a well-known network security technique -- it introduces monitoring and correlation devices to networks, enabling administrators to monitor events and detect attacks and anomalies in real-time. Web intrusion detection does the same but it works on the HTTP level, making it suitable to deal with security issues in web applications. This session will start with an overview of web intrusion detection and web application firewalls, discussing where they belong in the overall protection strategy. The second part of the talk will discuss ModSecurity and its capabilities. ModSecurity is an open source web application firewall that can be deployed either embedded (in the Apache HTTP server) or as a network gateway (as part of a reverse proxy deployment). Now in its fifth year of development, ModSecurity is mature, robust and flexible. Due to its popularity and wide usage it is now positioned as a de-facto standard in the web intrusion detection space."

A very interesting research paper titled "Apache Prefork MPM Vulnerabilities" was released a few days ago, as you can see in the corresponding Bugtraq post. The paper describes, in detail, the dangers of allowing third-parties to run code under the same account as the Apache web server. This normally happens when dynamic content is produced using Apache modules (e.g. PHP) or when CGI scripts are configured to run without suEXEC. This topic itself is not new. You will find several articles on runtime process infection following this Google search link. I warn about this problem throughout my book and especially in Chapter 6, which is dedicated to those situations when more than one party is using the one Apache installation. However, it is one thing to know that something is possible and another to demonstrate, step by step, how it is done. Another interesting finding resulting from this paper is that it is possible to send a SIGUSR1 signal, as root, to any process on the system instead of just to Apache children processes. This is an issue that will have to be fixed in one of the future versions of Apache.

This problem with running code as the same identity as the web server is well understood (and has been for years) among the advanced Apache users. The solution is to always execute CGI scripts through suEXEC and to never allow third parties access to any of the modules. The real problem is that, as with any other product, there are few people who understand Apache inside out (and they can protect themselves) but there also those who are using the technology but do not have the luxury of learning everything there is about it (and there are many legitimate reasons for that).

The solution is obvious. Apache must be safe out of the box! We should dispense with the idea of running things in the same process. Process isolation facilities (either suEXEC or something else) should be installed and running by default on all installations. We can and should make provisions for those who know what they are doing to shoot themselves in the foot, of course. But the only reason to attempt to run things in the same process is performance and I suspect, in this day and age, virtually all users will be happy with the performance of their web server doing things in a secure manner.

Last week I spent some time stress-testing Apache 2.2.3 configured to work as a reverse proxy. I discovered (actually, re-discovered would be more accurate) two issues worth sharing.

1. Memory consumption of an Apache process will steadily increase as the number of processed requests rises. This is very easy to see if you send thousands of requests per second, with each request going to the same process. This has to be either a memory leak or a memory fragmentation issue. To deal with this you need to recycle processes before they become too large (and cause the operating system to start swapping). The MaxRequestsPerChild directive is meant to help with this. By setting its value to something other than zero (which means "unlimited") you are telling Apache to shut down every process that goes over the limit. No problems there. Except that it's where the second problem comes in.

2. The MaxRequestsPerChild directive does not work as the name suggests. Apache does not count requests - it counts *connections*. This creates a problem if you have persistent connections enabled in your configuration - you don't know how many requests will come over a connection. It is probably safe to assume the number will not be large in most cases but you won't know if someone will try to abuse this problem and force a large number of requests over a single connection (e.g. by using a specially programmed script). To be on the safe side you need to divide your ideal MaxRequestsPerChild value with the MaxKeepAliveRequests value. This will prevent the Apache processes from growing too large. But there's a side effect - Apache will now recycle its worker processes more often. As your final step you need to make sure there are enough idle processes around (using MinSpareServers) to jump in as soon as an active process goes down. Yo need to have a few of these processes because there is a performance penalty associated with the creation of a new process and because Apache creates new processes at a rate of one every second.
   

My book was translated to Japanese and published by O'Reilly Japan! This is, apparently, old news, as they did it back in 2005, but I only found out about it from the three-montly royalties statement I received in April.

While we are on the subject of writing, I am starting to get the itch again. There are two or three topics I would like to explore further. Topics such as web application firewalls and ModSecurity, web application security, and application security patterns. On the other hand, I have a few compelling reasons against writing another book:

• It takes a lot of time (time better spent building Thinking Stone into a stronger business).
  
• It's lonesome (but this can be dealth with by finding someone to co-author the book with me.
  
• My hands and arms haven't fully recovered from the first book. (This one is the most compelling reason of all - I barely managed to finish Apache Security in the first place. If you are using keyboard extensively make sure to read about RSI and always keep Workrave active.)
  

It's been exactly one year since my book, Apache Security, was published. I was very glad to learn Amazon.com are now enabling book authors to talk to their audience. It is unfortunate this feature did not exist at the time - I would have loved the opportunity to point those looking at this page to the book's web site - http://www.apachesecurity.net.

I have always believed publication is just a first step in the life of a book (a long step in my case, as I spent eight months writing), and that the best stuff comes only after a book has been in use for a year or two. Let's face it, we (the authors) don't know nearly as much as our collective readership does. Therefore I invite you, the reader, to send me your feedback and make the second edition of Apache Security much better!

I was recently involved with a project where we needed to configure an Apache server that was intended to run multiple web sites/applications. It's a pretty common assignment. To ensure the setup is secure I decided to start by creating a separate user account for each application. This allowed me to correctly configure file permissions to allow Apache to serve the static files directly. To take care of the dynamic content, I configured suEXEC to execute each application's scripts under its own account. (In case you are wondering, this particular server is fast enough to run the scripts as CGIs. But if process creation becomes a bottleneck we can always seamlessly switch to FastCGI to avoid the performance penalty. Nothing to worry about, then.)

SuEXEC is a great tool but I'd love it to be capable of jailing (via the chroot system call) the binaries it executes. However, this feature is not present in the stock version. Having been responsible for the internal chroot feature of ModSecurity, I think I have a pretty good idea of why this is the case: unless you know what you're doing it's pretty easy to break applications with chroot. And if that happens you are going to ask for help... from those that created the feature, right? Of course! As it turns out, chrooting is notoriously difficult to debug remotely and that's why the developers would much rather not deal with it.

But, if do you know your way around feel free to use my suexec chroot patch, which I have just added to the Apache Tools project. But, please, don't write to me if it's not working as you are expecting :)

I have been involved with Apache programming for several years now. During this time I've been following the main Apache development list and the module programming one. This is how I got to meet Nick Kew, probably the most helpful person on these two lists. (Perhaps on other lists too, but I only follow these two.) Rumours that Nick is writing a book (spread by the author himself) have been circulating for many months now. I am happy to say this is now official; Nick's book, Apache Programming (I am not sure if this is the official title or not) will be published by Prentice Hall in their Open Source Series. Nick has been kind to invite me to help him as a technical reviewer. This is great news for the Apache community! Apache is a fantastic web server but its growth is being slowed down by the lack of proper documentation for programmers. I only wish I had this book a couple of years back when I was starting with ModSecurity!

A new beta version of the Apache web server has been released. This release is important because it is a beta version in the 2.1.x development branch, meaning Apache is one step closer to an official 2.2 version. The most important changes are the refactoring of the authentication/authorisation backend, faster and better caching support, and the much improved mod_proxy with support for load balancing.

I have made the PHP chapter from Apache Security available for free download. When we made the decision to set the installation and configuration chaper free, several months ago, I did not realise this chapter only told one half of the story. Most people need to configure Apache *and* PHP. This is now fixed, and the two chapters together make a valuable resource. My long-term plans are to convert both chapters to DocBook, keem them up-to-date, and publish them as PDF and as HTML. But not yet--the content is still very fresh!

My book, Apache Security, is finally out, after a year and a half of hard labour. I began talking to the publisher in the summer of 2003, and began research shortly after. I began writing in March 2004 and finished in December 2004. O'Reilly had it in stock on March 1st 2005, but it only become widely available in late March.

The work itself was great fun. It is a great privilege to be able to explore the boundaries of your own knowledge in detail. I did have periods of despair, of course. But I was prepared for them from reading blog entries of other book writers. I knew how difficult it was going to be. The biggest challenge I had was deciding what to cover, and what to leave out. It was never going to be a book for absolute beginners (officially, it's an "intermediate to advanced" type of book) but I didn't want to write a book that would be understood only by a few people who are already Apache experts. On an another level, it was also not possible to look at the security of the Apache web server in isolation. A book that pretends to provide "everything you need to know to secure your Apache web server" must delve into topics such as networks security, host security, and web application security. As it turned out I had to deal with these questions every single day. It was a struggle to keep the book from growing too much. Initially, the book was supposed to be around 280 pages long. In the end, it grew to over 400 pages.

As my work progressed I began to think more and more about the process. The traditional book writing process restricts the author to his experience, the experience of his immediate peers, and the experience of the technical reviewers. While this may work in some, or even many cases, I came to believe that a piece of technical writing can achieve its best only through the collaboration process with the readers. Now that the book is out, this is exactly what I am looking forward to.

The first step, the book's web site, is already completed. In the following months I plan to put more material from the book online, start adding fresh content, and generally try to engage the public by offering them the stuff they are interested in. The way I see it, my work has just began.