Generate QListBox and without Objects. Arrays are much faster.

thread: 13 messages  |  last: about 4 months ago  |  started: tuesday, march 30, 2010, 3:17 am pdt


#1  |  tronics (VIE, AUT) Austria
Tuesday, March 30, 2010, 3:17 AM PDT

Hello,

A while ago I posted comparisons of different ORMs finding that generating objects is much more expensive then generating arrays. We have the same situation with Qcodo.
Expecially QListBoxes with more entries can slow down a page, especially when there are more of them. After testing it out I found caching helps only a little bit, as the Objects still need a lot of memory.

I'm not sure how the new code would look like but it would be awesome to have a new separate QForm that generates Listboxes without Objects. Arrays are really much faster.

It could look like excactly like the normal QListBox but the functions eg AddItem work in a way where the data is stored as an array only.
I think also using Qquery in a way where only an array is returned would complement that. However getting the data would in most cases a custom mysql query.

$this->lstPersons = new QListBoxArray($this);
$this->lstPersons->AddItem('- Select One -', null);

$arrPersons = Person::LoadAllArray(QQ::Clause(QQ::OrderBy(QQN::Person()->LastName, QQN::Person()->FirstName)));

if ($arrPersons) foreach ($arrPersons as $arrPerson) {
$this->lstPersons->AddItem($arrPerson->LastName . ', ' . $arrPerson->FirstName, $arrPerson);
}

Cheers,
tronics

#2  |  Mike Ho (Sunnyvale, CA) United States of America Qcodo Administrator
Tuesday, April 6, 2010, 5:40 PM PDT

Yeah, it's definitely a valid point.  But again, the main issue is -- how do we know which property to use as the “display” of each listitem?  Or, better yet, with your example, what if you want the “display” of each listitem to be a concatenation of multiple fields in the object or some other calculated value?

#3  |  tronics (VIE, AUT) Austria
Thursday, April 8, 2010, 6:42 PM PDT

>But again, the main issue is -- how do we know which property to use as the “display” of each listitem?

Hmm I hoped the example would be useful :)
What about having 2 parameters like above:

addItem(parameter1,parameter2);

parameter1 = $arrPerson;
    <- this is for internal use, mainly for the ID

parameter2 = $arrPerson->field1 . " " . ($arrPerson->field2+$arrPerson->field3);
    <- this is a string that can contain whatever we want, concat 2 fields or even do calculation. Similar then the now obsolete toString() function for the models.
Ok and this string is then used for the info on the listbox item.

Regards,
tronics

#4  |  Fernando Lordán (Barcelona, CAT, Spain) Spain
Thursday, April 8, 2010, 11:26 PM PDT

Mike, if I'm not wrong, Tronics is not talking about active records here, but about a kind of QListBox control that work without QListItem objects.

Tronics, I really see a curious background conflict in choosing a fully OO framework and then tend to avoid using objects everywhere you make an intensive use of them. I find it justified when talking about query results, that tend to be massively instantiated, but there must be a compromise that doesn't break the framework philosophy.

About this case, in my opinion, a listbox that contains tenths of items might be substituted with a different control just for its bad HTML usability. Maybe an autocompletion textbox or a modal lightbox with a paginated selection datagrid would suit the need.

I don't understand when you say that the toString() method is obsolete for the model. In your example you can bypass the second parameter entirely just relying on it, if you're not displaying special calculations.

By the way, caching precompiled code (with APC) helps a lot in my case, from 30% to 90% gains in memory, depending on the nature of the request calculations. Anyway I usually don't do websites, but web applications with a very intensive use of a lot of classes, so I don't know if this will help you.

#5  |  tronics (VIE, AUT) Austria
Friday, April 9, 2010, 11:16 AM PDT

Thank you for your inputs.
I tried to come up with a better and not so longwinded answer below.

#6  |  tronics (VIE, AUT) Austria
Saturday, April 10, 2010, 11:05 AM PDT

Hello Fernando, thank you for your inputs.

Please let me clarify..(I edited this posting to make it more to the point)

With this thread I just want to suggest to create one additional QForm element  QListBoxArray that is very performant by using Arrays.

This QForm would then be a great usecase of a performant implementation using arrays and could easily be adjusted for different additional uses (eg by combining with some JQuery plugins that also just need very fast some array values in JSON Format or something..... Example are JS datagrids or trees).

So the usefulness is not only about a QListbox replacement here..
Rather having a best practice way to make performant QForm Controls and there a QListBox really is often needed.

Mike has already made a fantastic tool with the autocompleteTextbox which is very useful for larger count entries. Thanks for that it is great.

However tackling directly the QListBox would also be awesome.

Other ORMs like Doctrine and Propel offer this possibility and they are widely used. The performance difference is of course depending on your application but benchmarks in these ORMs showed amazing 'up to 60 times faster' improvements.

Here is also a relatively concrete Example:

If you have:
- 10 Listboxes
- with each 40 entries on one form
-> I generate 400 Objects from the model
-> and also additional Qquery and Qform objects etc..

...at least xxxx Objectinstances are flowing around in the memory.
But we are still only talking about 10 Listboxes, that is not like a bloated UI or something.
Yet some of these Listboxes might come from different very populated tables in terms of many columns (Person 70 Columns, Item 80 Columns, etc.) with foreignkeys and that makes the whole thing more costly.

The option to cut down a few of these by using arrays would really cut down a lot of performance cost.
That same example with datagrid is probably even worse on the clientside because of all the UI elements, additional clicks and large JS footprint in the HTML delivered.

--

Hope this makes a little clear where I'm coming from.
Regarding the QListBoxArray - It would not blow up the framework nor make anything obsolete we have achieved - just a nice performant addition that is supported right out of the box (and therefore no individual hacking required for people wanting to use arrays)

Regards,
tronics

#7  |  Mike Ho (Sunnyvale, CA) United States of America Qcodo Administrator
Monday, April 12, 2010, 2:46 PM PDT

Tronics,

Just so that I understand you clearly...

Are you stating that instead of QListBox storing a list of QListBoxItem objects, you want QListBox to simply store an array of items?

If so, then I'm curious to know where do you feel the performance would increase?  Remember that the form- and control-state data, if stored in session, never ends up going to the client.  The only stuff that goes to the client is the <option> tags for the listbox.  I do see that the memory footprint of the overall QForm would decrease with your approach... but I wonder by how much and, more importantly, what impact that has on the overall performance of the application.

I'm not doubting you... I'd just like to see some data to back it up.  After all, Fernando does bring up some good points about the consistency of the framework in adhering as closely as possible to OO principles ... what you are suggesting would violate some of those principles, which is not necessarily a bad thing, as long as it is well justified.

I look forward to your response... thanks!

#8  |  tronics (VIE, AUT) Austria
Wednesday, May 26, 2010, 5:08 PM PDT

Sorry for coming back so late.
Somehow I overlooked the topic that is so important for me.

Memory footprint is a huge problem of all ORMs.
Basically I do believe that the reason why Qcodo is in some respects not 'very fast' lies in the high quality of abstration, say too many objects.

Other frameworks have the same problem:
http://stackoverflow.com/questions/1412762/doctrine-query-memory-usage

But they provide mechanisms relying on arrays instead of objects to conquer that:
http://www.doctrine-project.org/documentation/manual/1_1/0_10/improving-performance:conservative-fetching

Basically I believe having the option to retrieve only the data needed as array to build the dropdown (or other qform) via the ORM would be great.

Like the principle of conservative fetching in doctrine.
The important thing is having a shortcut that is still matching our way of doing things (so it is not a hack) but much less generation of objects.



#9  |  Mike Ho (Sunnyvale, CA) United States of America Qcodo Administrator
Wednesday, May 26, 2010, 11:14 PM PDT

Good point...

However, I think it's important to maintain the concept of a QListBoxItem class, for backward compatibility reasons as well as for consistency.

Probably what makes the most sense is to have QListBox still take in QListBoxItem objects in AddItem() -- and return QListBoxItem objects when requesting a specific item (e.g. with GetItemAt() and GetAllItems().

However, those methods could likely be rewritten to have stuff stored in an array format instead of QListBoxItem internally.

I think that should yield the benefit of speed/memory footprint that you're looking for, while maintaining the clean, OO-centric approach.  No?

Is this something you'd like to take on? =)

#10  |  tronics (VIE, AUT) Austria
Thursday, May 27, 2010, 1:20 PM PDT

Ok, I will try to find a concrete testcase where we can see an advantage and eventually a solution. If I find something usefull I will need your expertise on the OO design.
Btw. I have integrated a different ORM (featuring this array approach) just for fun. But I could not benchmark it together with QForm so far.
That would probably be an interesting first step as there are a lot of objects generated starting with ->LoadAll() and ending at QListBoxItem. Most probably more of these for the ORM then for the QListBox.

Cheers,
tronics



Copyright © 2005 - 2012, Quasidea Development, LLC
This open-source framework for PHP is released under the terms of The MIT License.