Exception when query would benefit from an index

thread: 3 messages  |  last: about 5 years ago  |  started: saturday, december 2, 2006, 1:06 pm pst


#1  |  Mazzie (Southwest Florida) United States of America
Saturday, December 2, 2006, 1:06 PM PST

This would definitely be in the “luxury” category of features, but it popped in my head today when I was going back through my rather large app doing optimizations, now that its well past the protoype stage of development.

It would be so cool if somehow you could enable a flag, that in addition to, or along with profiling, would cause QQ to throw an exception if you are trying to `WHERE`, `ORDER BY`, or `GROUP BY` on a column that is not indexed.  (Especially `ORDER BY`)

I have three major reasons why this would be useful.

1) I know that all of the indexes should have been meticulously planned out in the data model before hand, but hey, we are all guilty of oversight, laziness, mistakes at one time or another.

2) Code-genned data grids automagically let you sort on any field by clicking the column header.  Unless you index every field, users can beat up the database by sorting on poorly optimized fields.  The alternative is to go back and override those columns to disable sorting by default, or modify the codegen template.  In that case you are making assumptions about how the users will interact with the app, which in my experience, I never 100% guess correctly.  Imagine being able to cut a few users loose on the protoype and ask them to tell you what column they tried to sort on when they receive the exception for no index.  It would be easy to catch the exception and print out a nice message too, something like, “Sorting on the `foo` column results in a slower query, please pass this message onto the administrator blah blah blah”.  You can then mull over the feedback and decide which fields you want to index based on the performance requirements of the project.  At that point you could turn off the flag, or continue to catch the exception and log it, or change the message to something like “sort on this column only if you really need to”, lots of possibilities.

3) This idea plays into the Qcodo idea of getting a protoype up and running quickly, and worrying about optimization later.

#2  |  Mike Ho (Sunnyvale, CA) United States of America Qcodo Administrator
Monday, December 4, 2006, 10:50 AM PST

mazzie, it's a good suggestion, and i especially like the line of “I know that all of the indexes should have been meticulously planned out in the data model before hand, but hey, we are all guilty of oversight, laziness, mistakes at one time or another.” =)

But a few points:
** First of all, I'm not sure about the issue you bring up on order by and indexes -- please correct me if i'm wrong (and this may be DB dependent), but does order by actually use a given column index to perform ordering on all DB's?  Or is a given resultset ordered by the DB engine after the rows are retrieved regardless of indexing?
** DB Profiling was intended to help users limit the issue of N+1 roundtripping to the DB.  This feature, combined with things like Expand and ExpandAsArray (e.g. Early binding of relationships) was meant to specifically address the proverbial argument of the “ORM/ER Impedance Mismatch” and to aid in the optimization of what is usually the biggest performance sin in ORM-based frameworks/systems.  It was and still is not meant to replace any db-specific profiling tools that come with all the major db platforms, which give much better visibility and granularity to the use of table/row reads, indexes, scans, etc.  And becuase of this, I personally think that db-platform specific tools are much better suited for this than anything we could come up with at the qcodo/php/application level.
** Moreover, it might give inaccurate information -- dependning on the db platform and +how</I> that platform uses/optimizes against specific indexes, there may be times when you are running a specific query which may or may not use a specific index due to the way the db platform is implemented... and qcodo could very easily give a false positive or a false negative as to the use of that index.

#3  |  Mazzie (Southwest Florida) United States of America
Monday, December 4, 2006, 12:18 PM PST

Argh, typed out a long reply and somehow it didn't go through.  Anyways, now a shorter reply.

MySQL uses indexes for ORDER BY in the most common situations.
http://dev.mysql.com/doc/refman/4.1/en/order-by-optimization.html

I always forget that Qcodo has to support many DB platforms =)

I still think my idea would be a good development tool because QQ lets you put together queries at almost the speed of thought.  You are like a kid in a candy store, grabbing all of the data you want, how you want it.  I have caught myself several times throwing in a QQ::OrderBy clause for a field that was not indexed resulting in a huge performance hit.  When a project has 100+ tables, 60+ relationships, and 100+ indexes, sometimes I just plain forget the details of optimization until I get a call at 3am.  Then it all comes back to me =)



Copyright © 2005 - 2012, Quasidea Development, LLC
This open-source framework for PHP is released under the terms of The MIT License.