Then each went to his own home

And now to something completely different (No, I won’t post cat photos now)

Trackbacks are out, unfortunately. Probably due to a misuse of spammers a lot of blogs have closed their trackback ping urls.
However, as I want to know what is currently written about my articles: criticism, praise or different views or whatever, I need to somehow track new referrers to my blog. I tried different solutions that all weren’t satisfying:

• I tried to skim through my awstats statistics but it was a pain to extract the new trackbacks from the old ones. Awstats way of displaying referrers is ok to have an idea of how my visitors reach my site but it falls short in showing me new referrers.
• I created a watch list at technorati, but it seems to track just a portion of the whole cake.

I ended up writing a python script for it. As a beginner in python, programming this task was easy enough to have a good result in small time.
In the danger of presenting the 1324th solution to this problem I hereby present my script. If you already have got a solution I’d be interested since I didn’t find any service or blog post concerning this problem.

latestReferrers.py

The script reads from an apache access log file, counts how many times a referrer occurs and writes the output in RSS 2.0 format using PyRSS2Gen. Have a look at my referrer feed to see how this looks. The the script needs and no javascript, no database and no cron job (as it is directly called by the http request). All it needs is an apache log file.
Now, apache log files mostly are log rotated, that means working on the latest log file just covers the last day or the last week (this depends on how much traffic your website has). This somehow is a problem as I don’t want referrer urls showing up again after each log rotation.
Now the neat thing about rss aggregators is that they just show up new items so you won’t see referrers over and over again. I just have to subscribe to my latestReferrer.py rss feed and tell the aggregator that it shouldn’t notice me if the feed has changed and voilà, I’ve got a poor mans solution to my “new referrers” problem.

If you want to install latestReferrers.py: put the python scripts into a directory where apache grants the right to execute (i.e. cgi-bin), edit the few variables at the start (regex of urls/agents you don’t want to show up in your referrers, number of times a url has to show up in the access log so it shows up in you rss, etc.), subscribe to your new feed via a feed aggregator and then leave a comment here.. :-)
The code is published under LGPL.

Other solutions

• reeferss offers a “latest referrers RSS feed” (requires javascript on your site). No installation required. Neat!

This entry was posted on Tuesday, March 28th, 2006 at 1:42 pm and is filed under Blog. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.