Comments on: Tagsystems: performance tests

By: return

return — Fri, 01 Apr 2011 10:54:28 +0000

Hello,
I was looking at the Toxi method, is there a CRUD system/GUI/idea already done to update the db correctly,somewhere?
Thanks in advance

By: Mahesh

Mahesh — Fri, 04 Feb 2011 12:52:46 +0000

I think full text search is better ,for small scale web applications and maintain query cache like using doctrine like ORMs.

Because mostly people see the performance than the data redundancy.

Nice comparison

By: Philipp Keller

Philipp Keller — Sat, 30 Jan 2010 21:59:37 +0000

@c-a: thanks for the hint, I’ve corrected the link

By: c-a

c-a — Mon, 18 Jan 2010 23:01:38 +0000

«tags doesn’t map to sql at all. so use partial indexing.»[Joshua Schachter at Carson Summit]

FYI the link to Joshua is dead.

Thanks for all the useful information.

By: Pat

Pat — Mon, 08 Jun 2009 22:59:32 +0000

only 2 years since last comment and 4 since the post.

But at about the time of this post, I was working for LinkedIn.com I ran some performance tests comparing Lucene, MySQL FULL Text, Oracle Full Text for searching people’s profile. Hands down Lucene was the winner.

Ever wonder why there is no obvious way to break a connection in LinkedIn? Its because the Lucene index is incrementally added to. Removing a connection from the search results is an expensive operation.

Of course things change — would be interesting to see the results of 4 years worth of work on all three products.

By: Peufeu

Peufeu — Mon, 15 Oct 2007 10:45:50 +0000

Efficiently handling tags is similar to attributes in dating sites (ie. +blonde +tits -fat) except there are a lot more tags than profile attributes.

In order to extract any kind of acceptable performance from a SQL database, you will have to forget about LIKE (full table scan), and foreign keys (scuttle/toxi solution).

Basically you need a SQL database which supports one of the following :

- efficient star join support (that means Oracle),
- Bitmap index support (coming up in Postgres),
- efficient fulltext search support (ie Postgres)
- vectors (arrays) as column types and specific index methods to make boolean queries (ie. Postgres) on the values contained in said vectors.

MySQL is not part of the solution ; besides MySQL FULLTEXT is lucicrous.

A better solution may be to use a full text search engine. I tried Xapian and found that, on large data sets consisting of up to a million forum posts, it massively outperformed Postgresql’s fulltext search, which itself massively outperformed MySQL’s fulltext search. This can be used for tags, and obviously to search the articles’ full text. Obviously, Lucene is also a solution, however it is less user-friendly than Xapian (uses Java, bleh, hard to interface with Python for update scripts, etc).

By: orderlord

orderlord — Wed, 26 Sep 2007 05:57:05 +0000

What about benchmarks when the queries have ORDER BY (such as ORDER BY date).

For example, when a user wants to see all items with certain tags, sorted newest item first. How is performance then?

By: Ryan

Ryan — Tue, 25 Sep 2007 03:19:23 +0000

It looks like the first commenter (“Go back to school dude!
Rework the queries for the toxi schema and use JOINs.”) never pursued his argument, and it looks like you never “got” what he was really saying.

I think what he meant was that a query of this form:
SELECT … WHERE name IN (“word1″, “word2″, “word3″)
HAVING …

should instead be rewritten as:
SELECT … WHERE name = “word1″
INNER JOIN
SELECT … WHERE name = “word2″
INNER JOIN
SELECT … WHERE name = “word3″

Same thing for the UNION query. Use actual UNIONS instead of “WHERE name IN …”.

I think this is supposed to give you a good performnance boost.
Would you consider redoing your benchmarks on the TOXI schema using these queries above?

By: Philipp Keller

Philipp Keller — Sun, 22 Jul 2007 14:39:12 +0000

Geoff: Sure, go for the indexes! Have a look at http://www.pui.ch/phred/modules/tag_database_schemas.sql. I added indexes on tag.tagname, bookmark_tag.tag, bookmark_tag.bookmark and bookmark.url. If you can improve the performence by altering that indexes let me know.

By: Geoff

Geoff — Thu, 12 Jul 2007 19:17:42 +0000

This is probably a dumb question but here goes. If I opt for the toxi approach, is there any performance benefit to indexing any of the columns?