2010-03-10

Apache mpm_worker and mpm_prefork with Python/Django

Although the Django documentation clearly states that mpm_prefork is the way to go we decided to give mpm_workers with mod_wsgi a try. Django 1.1.1 is thread safe so why not? Also there is some good reasons why people wants to use mpm_worker instead of mpm_prefork.

Our website get about 100 requests by second spread on 3 frontend servers. That's really not much. First it went well. Then, when traffic increased a bit, everything started to slow down. It was slowing so much that the site was almost unusable. A little bit of profiling showed that a lot of CPU time was spent into some Django's template filters. We tried to use a Django template caching technics but it didn't helped much.

At the time the mpm_worker config was something like that:

ServerLimit 16
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25

Python is known to not do well with threading because of the GIL so I try to experiment a bit with these values. I try to reduce the ThreadsPerChild value and it's what I found:

Value of ThreadsPerChildAverage response time in millisecondes
20.13
40.16
80.19
160.21

The performance improvement is clear. With less threads, the CPU usage was going down and the server started to feel a lot more snappy. These numbers don't even reflect how bad the situation was with more traffic. A single server was chocking with less than 100 requests by second.

Conclusion

If you really have to use mpm_worker with python code, make sure to test everything under high concurrent load. If the performances are bad, reduce the value of ThreadsPerChild. I would recommend 2-5 threads maximum.