Lighttpd does a much better job generating an index page for lots of files. 12,049 files to be precise, at time of writing. I had set up a long time ago my website to be a fallback mirror for Source Mage, so if a source file got moved, lost, deleted, or the site hosting it went down, away forever, whatever. I remember disabling Apache’s Indexes option on that file list because it would take foreverto get anything rendered, as well as consuming plenty of CPU and resources.
Well, I was using an internal lighttpd to render the same thing internally, and I didn’t disable its directory listing at all. Whilst I was working on setting up collectd to monitor my Apache scoreboards, I noticed that it could also monitor the lighttpd scoreboard. So I was going about the config to set that up, and I noticed that the Apache wasn’t proxying it’s requests through to lighttpd, and was taking a lot of time and resources to do something simple.
Welp, having ADD like I do, I got sidetracked on doing that instead and started to rewrite my configs to not serve files directly, but proxy it to the lighttpd internally.
Originally, I had two aliases set up “/sourcemage” and “/sourcemage/fallback” which both pointed to the same spot on disk, This way you could reference things using “/sourcemage/file.tar.bz2” or “/sourcemage/fallback/file.tar.bz2”. The reasoning behind this is simple: I set up my fallback mirror before Source Mage standardized on a fallback URL, and I wanted to keep both paths functional. Easy to do with aliases. Oh, and I also have an alias to “/sourcemage/codex” for all my local codex needs.
Switching to a proxied setup was a bit more complicated than I anticipated. The alias stuff doesn’t work the same way everything else does, in that what it finds first it goes with. I needed to have mod_rewrite rules wired in to properly redirect things and manipulate the URL sufficiently. But I have two special cases. I can’t simply redirect everything “/sourcemage/*” to “/sourcemage/fallback.” Also, I wanted to be able to continue to use “/sourcemage/file.tar.bz2”
My solution is relatively simple and follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
#aliases Alias /sourcemage/codex "/srv/webMirrors/sourcemage.org/codex" #rewrite /sourcemage/ to /sourcemage/fallback #some very fancy rewrite rules to keep the old alias functionality RewriteEngine On RewriteRule ^/sourcemage/?$ /sourcemage/fallback/ [R] RewriteRule ^/sourcemage/codex/?$ - [L] RewriteRule ^/sourcemage/([a-zA-Z0-9.i_\-+]+)$ /sourcemage/fallback/$1 [R] # do the actual passing through to the lighttpd (which handles huge # directory listing much much better ProxyPass /sourcemage/fallback http://fallback.shlrm.org/ ProxyPassReverse /sourcemage/fallback http://fallback.shlrm.org/ # old notes kept here for reference as to what the above rewrite # rules are doing -- dkowis 2011-09-07 # going to proxy these to the lighttpd #Alias /sourcemage/fallback "/var/spool/sorcery/" #Alias /sourcemage "/var/spool/sorcery/" # directories! #<Directory "/var/spool/sorcery/" > # Options none # AllowOverride None # Order allow,deny # Allow from all #</Directory> <Directory "/srv/webMirrors/sourcemage.org/codex/" > Options Indexes Order allow,deny Allow from all </directory> # vi: set ft=apache:
The rewriting rules are the most complicated part of this. The trick was getting it not to try to rewrite anything when the url matched “/sourcemage/codex” After that, the rest was really easy. Some matching logic to pick up on “/sourcemage/file.tar.bz2” and redirect that to “/sourcemage/fallback/$1” and everything worked as it did before, except way faster.