Apache mpm_worker and mpm_prefork with Python/Django
Although the Django documentation clearly states that mpm_prefork is the way to go we decided to give mpm_workers with mod_wsgi a try. Django 1.1.1 is thread safe so why not? Also there is some good reasons why people wants to use mpm_worker instead of mpm_prefork.
Our website get about 100 requests by second spread on 3 frontend servers. That's really not much. First it went well. Then, when traffic increased a bit, everything started to slow down. It was slowing so much that the site was almost unusable. A little bit of profiling showed that a lot of CPU time was spent into some Django's template filters. We tried to use a Django template caching technics but it didn't helped much.
At the time the mpm_worker config was something like that:
ServerLimit 16 StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25
Python is known to not do well with threading because of the GIL so I try to experiment a bit with these values. I try to reduce the ThreadsPerChild value and it's what I found:
| Value of ThreadsPerChild | Average response time in millisecondes |
|---|---|
| 2 | 0.13 |
| 4 | 0.16 |
| 8 | 0.19 |
| 16 | 0.21 |
The performance improvement is clear. With less threads, the CPU usage was going down and the server started to feel a lot more snappy. These numbers don't even reflect how bad the situation was with more traffic. A single server was chocking with less than 100 requests by second.
Conclusion
If you really have to use mpm_worker with python code, make sure to test everything under high concurrent load. If the performances are bad, reduce the value of ThreadsPerChild. I would recommend 2-5 threads maximum.