So, after a few hours trying different solutions, and reading trac’s source code, I think I got where the bottleneck was.Well, it was http://www.sqlite.org/download.html which was causing the bottleneck. Trac uses an object CachedRepository to access the repositories. Whenever we want to get the chagesets, a function to synchronize the repository is called:

class CachedRepository(Repository):
  def get_changeset(self, rev):
    if not self.synced:
      self.sync()
      self.synced = 1
      return CachedChangeset(self.repos.normalize_rev(rev), self.db, self.authz)

and such method, sync(), makes a call to:

youngest_stored = self.repos.get_youngest_rev_in_cache(self.db)

which is all this:

def get_youngest_rev_in_cache(self, db):
    """Get the latest stored revision by sorting the revision strings
    numerically
    """
    cursor = db.cursor()
    cursor.execute("SELECT rev FROM revision ORDER BY -LENGTH(rev), rev DESC LIMIT 1")
    row = cursor.fetchone()
    return row and row[0] or None

And that SQL query was taking around 1-2 seconds each time it was executed. It happened that we were running an old version of sqlite and pysqlite, so a ./cofigure && make && make install using the recommended installation saved my day

Hope it is useful to anybody if it gets indexed by Google.

Eversince I have implemented various related services, basically playing around in my free time. Some of them are a RSS service and a jabber service (which you can subscribe by adding sevillatemp@jabber.org to your jabber application – even Gtalk -). There are people that use it on a daily basis and I am glad that they do

So, I had this idea in mind since some time, I just wanted to create a web service so other people could use it on their applications, be it web or desktop application. that would me let me play with webservices and, also, with Ruby.

So, I did some coding and implemented something that I would like you to know.

The web service

The temperature is read from a couple of sensors from Dallas Semiconductor using a program called digitemp. It is a unix command line program that is used by a set of shell scripts that create all the fancy graphics you can see at http://temp.roncero.org using rrdtool. These scripts are executed every five minutes to both insert the data into the database and generate the graphs. The system usually works ok, but it has some flaws, mainly because of old hardware and by the fact that I am living in London and the machine is in Seville.

In order to avoid problems with the actual system, I added a ruby script to the previous shell scripts to generate an XML file that would be later on used by the web service. The file content is

    1030AC15000800CB
    27.38
    Wed Aug 16 03:00:05 CEST 2006
    Celsius

    10952F1A00080063
    21.44
    Wed Aug 16 03:00:06 CEST 2006
    Celsius

This is what I am using to feed the web service. Notice that each of the sensors has a unique serial number and that would be used by the calls to the webservice. I created a small ruby program mapped to the url http://temp.roncero.org/temperature in order to work. There are two different methods now:

• list_sn* used to discover all the S/N available.
• read_temp(sn)* used to read the data from a sensor. *sn* is the serial number of the desired sensor.

Client programs

So here are two different client little programs that you can use to access the data. One in ruby and another one in python. Very simple, no error check.

Ruby

#!/usr/bin/env ruby

require 'soap/rpc/driver'
proxy = SOAP::RPC::Driver.new("http://temp.roncero.org/temperature",
"http://temp.roncero.org/temperature")
proxy.add_method('read_temp', 'sn')
proxy.add_method('list_sn')
puts "List S/N: #{proxy.list_sn}"
puts "reading from s/n  1030AC15000800CB"
puts "#{proxy.read_temp("1030AC15000800CB")}"
puts "reading from s/n  10952F1A00080063"
puts "#{proxy.read_temp("10952F1A00080063")}"

Python

#!/usr/bin/env python

from SOAPpy import SOAPProxy

url = 'http://temp.roncero.org/temperature'
namespace = 'http://temp.roncero.org/temperature'
server = SOAPProxy(url, namespace)
print "Printing S/N"
server.list_sn()
print "Temperature for 1030AC15000800CB"
print server.read_temp("1030AC15000800CB")
print "Temperature for 10952F1A00080063"
print server.read_temp("10952F1A00080063")

you can test both right now or come up with a new version in another language or improve these ones. Here is a typical output:

$ ./client.rb
List S/N: 1030AC15000800CB, 10952F1A00080063
reading from s/n  1030AC15000800CB
27.31 degrees at Wed Aug 16 03:20:05 CEST 2006
reading from s/n  10952F1A00080063
21.25 degrees at Wed Aug 16 03:20:06 CEST 2006

So

Yeah, this is a work-in-progress thing. I have no idea if this is the way to do it. No WSDL, not yet. Expect it to fail at some time, as I have yet to implement more error checking, etc.

So, I would like all of you, if you have an opinion on these kind of things, to comment or suggest, say if you would like it to do something different or better.

I have done some assumptions basically because my lack of knowledge on the matter.

1. I have created the webservice using ruby’s webrick own webserver on a high port and mapping it to an apache url using proxypass directives. Is this the way to do it?
2. I only created two methods. Would make sense to have more methods? Like one that would only return the temperature value?

Have something to say? Go ahead

So, I spent some time investigating PIL, the Python Imaging
Library, a set of python modules to manipulate and create images.

Basically what I do is create an small image using PIL, write the number on it and a small grid in two very close colors (grey). You can
have a look at the result below in this page.

To display the image, I just use a CGI python script that generates this image from the parameter passed to it like this way:

<img src="/snumber.png?hash=eef334ab8..." />

hash being the filename I was talking about yesterday.

The generation of the image is pretty simple, the code is like:

def generateImage(number):
    font = ImageFont.truetype(fontPath, fontSize)
    im = Image.new("RGB", imageSize, bgColor)
    draw = ImageDraw.Draw(im)

    xsize, ysize = im.size

    # Do we want the grid start at 0,0 or want some offset?
    x, y = 1,1

    draw.setink(gridInk)
    while x <= xsize:
        draw.line(((x, 0), (x, ysize)))
        x = x + xstep
    while y <= ysize:
        draw.line(((0, y), (xsize, y)))
        y = y + ystep

    draw.setink(fontInk)
    draw.text((3, 2), number, font=font)

    return im

which creates the image. The rest of the implementation consist in writing the img to stdout (web) among the HTTP headers.

Well, this is a little hack to do the trick. I believe this code can be further enhaced by, for example, creating a class for generating this
kind of images.

You can find the source code here for snumber.png. Just rename it to
a name of your choice and make it execute as an CGI. It should work

This and yesterday’s work are a fast hack to avoid spam. It surely can be enhanced. I intend to do it in the next weeks, when I have free time. Feel free to send comments or tell me if you are using it on your system (and if it works). Suggestion, patches and critics
are welcome

Investigating a bit, I learned he uses the UserAgent AIRF
string, which happens to be used by this program:
roboform, a program for automating
the filling of forms, somehow a paradise for spammers

As It says in the
FAQ, newer versions
disable the use of that UserAgent string so it makes it virtually
impossible to filter those comments from a web server point of view. My
spammer must also use many open proxies as all the request seem to come
from different IPs. Very annoying.

While thinking about the different possibilities of castration if I
could grab the spammer, I thought I could implement a human verification
process to post comments, the same way
gmail and others implement it, using a
distorted image to display a number which you have to manually type in
the form. I am not a very experienced python programmer, so I took it as
an exercise to see what I could come up with. I spent a whole evening
reading part of the pybloxsom code and specially the comments.py
plugin.

So this is what I did to
implement the logic of the verification (I have yet to think about the
generation of the images, or let others do it

I modified the comment-form.html template an added a couple of
variables, like this:

<input type="hidden" name="hash" value="$hash">
Secret Number ($snumber):
<input maxlength="5" name="snumber" size="6" type="text" value=""/>

So, the idea is this one: Each time an entry is displayed, I generate a
text file in a directory readable and writable by the web server
containing inside a random number. The name of the file would be a md5
hash ended by a “.txt”. I use this hash in the form to pass it to the
routines in charge of processing the POST action, along with the number
the poster must have typed beforehand (supposedly from the image, not
now at the moment). The POST routine should get the hash from the hash
variable, construct the file name and locate it in the hard disc from
the directory it was written which, by the way, is out of the web tree.
This way, it can retrieve the random number and compare it to what the
user typed. This way, I believe, I can know if it is a bot or a human
behind.

if form.has_key("title") and form.has_key("author") and form.has_key("body")
    and form.has_key("hash") and form.has_key("snumber"):

    hash = form["hash"].value
    diskHash = getHashNumber(hash)
    try:
        form_snumber = int(form["snumber"].value)
    except ValueError:
        # The user did not enter a number
        form_snumber = -1
    if diskHash == form_snumber:

This, and a couple of auxiliary functions make all the magic, and it
looks like it works

As I said before, I am not an expert, neither in Python nor http and web
programming and I don”t know if this approach has severe flaws or not,
if it is a bottle neck or it is just fine (looks like it works to me).
So I would really appreciate your comments on the topic, especially if
this a valid starting point to build a more secure comment system in
pybloxsom.

I believe there are other aspects to work on, like the removal of old
files from the temp directory and the posibility, although highly
unlikely, to have two identical files names. Other is the dynamic
generation of the images. I found some scripts, but in
perl [Sorry, the link
is in Spanish, but you get the idea].

The source code is here.

There is a followup of this article at

http://notreally.org/blog/devel/Python/pybloxsomnospam2