OutofMemory while retrieving data from table with lot of data ..

[Vineel]

would like to iterate thru it & process one record at a time, the following seems to cause OutOfMemory exception on tables with lot of data ..

sql.query("select * from mytable t ") { rs ->
  println "In the closure:";
  while( rs.next() ) {
    .. do process ..
  }
}

if I change the query to "select * from mytable limit 1,10" it does work.  what are the recommended approaches for iterating thru the complete data..  would I have to get the total count & iterate thru it in batches ..? something like ..

count = Mytable.count();
rowslimit = 1000;
offset = 0;
numbatches = (count/rowslimit)  + ((count%rowslimit)==0 ? 0 : 1)..

for( batch in 0..numbatches){
  sql.query( "select * from mytable limit ${offset},${rowslimit}") { rs ->
    while( rs.next() ) {
      .. do process ..
    }
  }
  offset += rowslimit;
}

or any other recommended approaches using the domain objects like Mytable.list( ... ) etc..

Thanks
-Vineel

[Graeme Rocher-4]

see http://stackoverflow.com/questions/3730113/need-help-improving-the-performance-of-large-datasets-in-grails

On Sat, Jul 28, 2012 at 11:47 PM, Vineel <[hidden email]> wrote:

> would like to iterate thru it & process one record at a time, the following
> seems to cause OutOfMemory exception on tables with lot of data ..
>
> sql.query("select * from mytable t ") { rs ->
>   println "In the closure:";
>   while( rs.next() ) {
>     .. do process ..
>   }
> }
>
> if I change the query to "select * from mytable limit 1,10" it does work.
> what are the recommended approaches for iterating thru the complete data..
> would I have to get the total count & iterate thru it in batches ..?
> something like ..
>
> count = Mytable.count();
> rowslimit = 1000;
> offset = 0;
> numbatches = (count/rowslimit)  + ((count%rowslimit)==0 ? 0 : 1)..
>
> for( batch in 0..numbatches){
>   sql.query( "select * from mytable limit ${offset},${rowslimit}") { rs ->
>     while( rs.next() ) {
>       .. do process ..
>     }
>   }
>   offset += rowslimit;
> }
>
> or any other recommended approaches using the domain objects like
> Mytable.list( ... ) etc..
>
> Thanks
> -Vineel
>
>
>
>
> --
> View this message in context: http://grails.1312388.n4.nabble.com/OutofMemory-while-retrieving-data-from-table-with-lot-of-data-tp4632380.html
> Sent from the Grails - user mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

--
Graeme Rocher
Grails Project Lead
SpringSource - A Division of VMware
http://www.springsource.com

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email