just built a service in PHP that can process 15,000 rows of data, with a complexity of about O(1000*(10|100)n) and it runs in 4.5min. I was looking to beat 15m. I expect to get that down further.
right now i use a LIMIT of 100 to minimize sql queries, and memory. i might try going to 1000 rows or about 10k in data, but in my experience when the DB has to return that much data you are not really gaining that much.
what do you think? any other optimizations i should look to?
One optimization is to write big O notation in minimized form. It looks more impressive to write O(n).
And to try being helpful: do aggregation in the db where possible, be sure indexes are good, play around with the number of rows you process at a time, use joins if appropriate to minimize queries, if your queries are expensive at all use EXPLAIN, depending on the database, look into using a cursor (maybe?).
I was using LIMIT and OFFSET, did you mean something else?
update: I processes 200k-300k rows in a few hours, and since the DB is on the same box, as the php app, there is no need to make this more complicated.
right now i use a LIMIT of 100 to minimize sql queries, and memory. i might try going to 1000 rows or about 10k in data, but in my experience when the DB has to return that much data you are not really gaining that much.
what do you think? any other optimizations i should look to?