Comments on: Tagsystems: performance tests

By: Philipp Keller

Philipp Keller — Sat, 30 Jan 2010 21:59:37 +0000

@c-a: thanks for the hint, I’ve corrected the link

By: c-a

c-a — Mon, 18 Jan 2010 23:01:38 +0000

«tags doesn’t map to sql at all. so use partial indexing.»[Joshua Schachter at Carson Summit]

FYI the link to Joshua is dead.

Thanks for all the useful information.

By: Pat

Pat — Mon, 08 Jun 2009 22:59:32 +0000

only 2 years since last comment and 4 since the post.

But at about the time of this post, I was working for LinkedIn.com I ran some performance tests comparing Lucene, MySQL FULL Text, Oracle Full Text for searching people’s profile. Hands down Lucene was the winner.

Ever wonder why there is no obvious way to break a connection in LinkedIn? Its because the Lucene index is incrementally added to. Removing a connection from the search results is an expensive operation.

Of course things change — would be interesting to see the results of 4 years worth of work on all three products.

By: Peufeu

Peufeu — Mon, 15 Oct 2007 10:45:50 +0000

Efficiently handling tags is similar to attributes in dating sites (ie. +blonde +tits -fat) except there are a lot more tags than profile attributes.

In order to extract any kind of acceptable performance from a SQL database, you will have to forget about LIKE (full table scan), and foreign keys (scuttle/toxi solution).

Basically you need a SQL database which supports one of the following :

- efficient star join support (that means Oracle),
- Bitmap index support (coming up in Postgres),
- efficient fulltext search support (ie Postgres)
- vectors (arrays) as column types and specific index methods to make boolean queries (ie. Postgres) on the values contained in said vectors.

MySQL is not part of the solution ; besides MySQL FULLTEXT is lucicrous.

A better solution may be to use a full text search engine. I tried Xapian and found that, on large data sets consisting of up to a million forum posts, it massively outperformed Postgresql’s fulltext search, which itself massively outperformed MySQL’s fulltext search. This can be used for tags, and obviously to search the articles’ full text. Obviously, Lucene is also a solution, however it is less user-friendly than Xapian (uses Java, bleh, hard to interface with Python for update scripts, etc).

By: orderlord

orderlord — Wed, 26 Sep 2007 05:57:05 +0000

What about benchmarks when the queries have ORDER BY (such as ORDER BY date).

For example, when a user wants to see all items with certain tags, sorted newest item first. How is performance then?

By: Ryan

Ryan — Tue, 25 Sep 2007 03:19:23 +0000

It looks like the first commenter (“Go back to school dude!
Rework the queries for the toxi schema and use JOINs.”) never pursued his argument, and it looks like you never “got” what he was really saying.

I think what he meant was that a query of this form:
SELECT … WHERE name IN (“word1″, “word2″, “word3″)
HAVING …

should instead be rewritten as:
SELECT … WHERE name = “word1″
INNER JOIN
SELECT … WHERE name = “word2″
INNER JOIN
SELECT … WHERE name = “word3″

Same thing for the UNION query. Use actual UNIONS instead of “WHERE name IN …”.

I think this is supposed to give you a good performnance boost.
Would you consider redoing your benchmarks on the TOXI schema using these queries above?

By: Philipp Keller

Philipp Keller — Sun, 22 Jul 2007 14:39:12 +0000

Geoff: Sure, go for the indexes! Have a look at http://www.pui.ch/phred/modules/tag_database_schemas.sql. I added indexes on tag.tagname, bookmark_tag.tag, bookmark_tag.bookmark and bookmark.url. If you can improve the performence by altering that indexes let me know.

By: Geoff

Geoff — Thu, 12 Jul 2007 19:17:42 +0000

This is probably a dumb question but here goes. If I opt for the toxi approach, is there any performance benefit to indexing any of the columns?

By: Octablog » A review of the Zend Framework - Part 3

Octablog » A review of the Zend Framework - Part 3 — Tue, 19 Jun 2007 23:06:53 +0000

[...] Zend_Search_Lucene allowed me to tackle the complicated issue of tagging – Tagging is a known problem to map effectively to databases (A dude named Phillip Keller wrote a blog on different tagging schemas, and conducted a performance comparison of the schemas. Another dude named Nirin Borwankar suggested yet another schema for tagging. The tagging issue is a long and complicated one.) To quote del.icio.us creator, John Schachter – “tags don’t map to sql at all. so use partial indexing.” Using Zend_Search_Lucene to index tagged items allowed us to implement tags in the Octabox project while still enjoying high performance, which was something that I was quite worried over before. [...]

By: Tjerk

Tjerk — Wed, 08 Nov 2006 15:11:27 +0000

I just finished a course advanced database systems, and we
learnede how to increase performance for specific queries schema’s.

For example do you know wich query-plans where used?
You can check that with the sql EXPLAIN command.

I would recommend a hash index on the tag collumn because you are
not doing any reange searches .. searching for a specific tag would require constant time.

Als it is better to measure your performance test in the number of page transfers between the hard disc and the memory/cpu. Because this is the bottleneck in performance. The miliseconds say more about your system than about those queries.

Anyways, another question: Which indexes did you use? B+ indexes? ISAM indexes? Hash indexes? Which where the search-keys for these indexes?