Quantcast

Cannot eagerly fetch collection, join and select not working???

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Cannot eagerly fetch collection, join and select not working???

johnrellis
Hey,

I may be doing something stupid, long week.

I have a domain class

class Customer {
   static hasMany = [favourites : Customer]
   String name
}

I create a customer with 50 favourites. I want to load the favourites so when i iterate over the collection and call getName() on each of them it doesn't call the database 50 times.

[fetch: [favourites: 'select']] doesn't seem to do anything.

[fetch: [favourites: 'join']] reduces the number of queries by 1.  i.e. customer.favourites is not a seperate query.

Here's what I have tried to eagerly load the collection :

52 Queries :

def customer = Customer.findByName('John', [fetch: [favourites: 'select']])//1 query
assert customer.favourites.name.size() == 50//1 query to get favourites + 50 individual queries to get name


51 Queries:

def customer = Customer.findByName('John', [fetch: [favourites: 'join']]) //1 query
assert customer.favourites.name.size() == 50 // 50 individual queries to get name

Any idea on how I can reduce the number of queries when iterating over a collection retrieved in this way???

Many thanks in advance!

Here is my entire test suit, two fails at the bottom :

class CustomerTests {

  Sql sql
  SessionFactory sessionFactory

  @Before
  void setUp() {
    BankingTransaction.withNewSession {session ->
      final customerOne = new Customer(name: "John", account: new Account(accountNumber: 123456)).save(failOnError: true)
      final customerTwo = new Customer(name: "Joe", account: new Account(accountNumber: 1234567)).save(failOnError: true)
      100.times {num ->
        def anotherCustomer = new Customer(name: "Customer ${num}", account: new Account(accountNumber: num)).save(failOnError: true)
        if (num % 2) {
          customerOne.addToFavourites(anotherCustomer)
        } else {
          customerTwo.addToFavourites(anotherCustomer)
        }
      }
      customerOne.save(failOnError: true)
      customerTwo.save(failOnError: true)
    }

    sql = new Sql(sessionFactory.currentSession.connection())
    sql.metaClass.getNumberOfSelects = {->
      return delegate.rows('SHOW STATUS LIKE "Com_select"')[0].Value as Long
    }
  }

  @Test
  void testNotRetrievingFavouritesCollection() {
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
    def customer = Customer.findByName('John')//1 query

    //assert customer.favourites

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest
    //is it what we expect??
    assert numberOfSelects == 1
  }

  @Test
  void testRetrievingFavouritesCollection() {
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
    def customer = Customer.findByName('John')//1 query

    assert customer.favourites// 1 query

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest
    //is it what we expect??
    assert numberOfSelects == 2//not sure
  }

  @Test
  void testRetrievingFavouritesLazily() {
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
    def customer = Customer.findByName('John')//1 query

    assert customer.favourites.name.size() == 50//1 query to get favourites + 50 individual queries to get name

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest
    //is it what we expect??
    assert numberOfSelects == 52
  }

  @Test
  void testRetrievingFavouritesEagerly() {
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
//    def customer = Customer.findByName('John', [lazy:[favourites:false]])//1 query
    def customer = Customer.findByName('John', [fetch: [favourites: 'join']])//1 query

    assert customer.favourites.name.size() == 50

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest
    //is it what we expect??
    assert numberOfSelects < 50 //FAIL!!!!!!!!!!!!!!!!!!!!!
  }

  @Test
  void testRetrievingFavouritesEagerlyAgain() {
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
//    def customer = Customer.findByName('John', [lazy:[favourites:false]])//1 query
    def customer = Customer.findByName('John', [fetch: [favourites: 'select']])//1 query

    assert customer.favourites.name.size() == 50

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest
    //is it what we expect??
    assert numberOfSelects < 50 //FAIL!!!!!!!!!!!!!!!!!!!!!
  }

}

--
John Rellis

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

pledbrook
> I create a customer with 50 favourites. I want to load the favourites so
> when i iterate over the collection and call getName() on each of them it
> doesn't call the database 50 times.
>
> [fetch: [favourites: 'select']] doesn't seem to do anything.
>
> [fetch: [favourites: 'join']] reduces the number of queries by 1.  i.e.
> customer.favourites is not a seperate query.

I can't see why this wouldn't work. Do you have a reproducible project
that we can work with?

Thanks,

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

johnrellis
Hi Peter,

Thanks for the reply.  Sure, the the project is hosted here :


Really hope I haven't done something stupid in my tests.  Many thanks!

John

On Fri, Aug 17, 2012 at 12:34 PM, Peter Ledbrook <[hidden email]> wrote:
> I create a customer with 50 favourites. I want to load the favourites so
> when i iterate over the collection and call getName() on each of them it
> doesn't call the database 50 times.
>
> [fetch: [favourites: 'select']] doesn't seem to do anything.
>
> [fetch: [favourites: 'join']] reduces the number of queries by 1.  i.e.
> customer.favourites is not a seperate query.

I can't see why this wouldn't work. Do you have a reproducible project
that we can work with?

Thanks,

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
John Rellis

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

pledbrook
> Thanks for the reply.  Sure, the the project is hosted here :
>
> https://github.com/johnrellis/OrmSimplicity
>
> Really hope I haven't done something stupid in my tests.  Many thanks!

I think I know what the problem is. Because you have a unidirectional
one-to-many relationship and haven't customised the mapping, you have
a link table. The link table is being retrieved eagerly, but the
relation on the other side of the link table is not. If you can do
without the link table, you could specify a direct foreign key in the
custom mapping.

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

johnrellis
OK, thanks for your time and effort Peter.  

From some quick searching, all answers suggest using a belongsTo.

Is there a way to do this without a belongsTo in  classes that don't make sense to have a belongsTo??  Example :

class ContactList {
 
  static hasMany = [contacts: Person]

  static mapping {
      //something goes here???
  }

}

class Person {

}

Many thanks!
John


On Fri, Aug 17, 2012 at 3:15 PM, Peter Ledbrook <[hidden email]> wrote:
> Thanks for the reply.  Sure, the the project is hosted here :
>
> https://github.com/johnrellis/OrmSimplicity
>
> Really hope I haven't done something stupid in my tests.  Many thanks!

I think I know what the problem is. Because you have a unidirectional
one-to-many relationship and haven't customised the mapping, you have
a link table. The link table is being retrieved eagerly, but the
relation on the other side of the link table is not. If you can do
without the link table, you could specify a direct foreign key in the
custom mapping.

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
John Rellis

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

pledbrook
> Is there a way to do this without a belongsTo in  classes that don't make
> sense to have a belongsTo??  Example :

Have you seen this?

    http://assarconsulting.blogspot.co.uk/2010/10/grails-one-to-many-mapping-with-foreign.html

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

johnrellis
Thanks.  Taking a look at this now.

On Fri, Aug 17, 2012 at 4:22 PM, Peter Ledbrook <[hidden email]> wrote:
> Is there a way to do this without a belongsTo in  classes that don't make
> sense to have a belongsTo??  Example :

Have you seen this?

    http://assarconsulting.blogspot.co.uk/2010/10/grails-one-to-many-mapping-with-foreign.html

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
John Rellis

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

johnrellis
Hmmm, ok, this isn't going to work for me because many people can have the same person in their contact list so I think join table is required for this type of behaviour, although the monday fog could be clouding my mind.  Otherwise, I would be fine with using a belongsTo.

To offer a possible solution to achieve the DB performance required without a bi-directional relationship, I suppose I could do something like

     //carry out the query
    def customer = Customer.findByName('John', [fetch: [favourites: 'join']])
    def favourites = Customer.findAllByIdInList(customer.favourites.id)
    assert favourites.name.size() == 50

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest
    //is it what we expect??
    assert numberOfSelects == 2

// non nice code done....

Is this something that needs to be improved on a hibernate level?  Curious on opinion, it should be possible to have a unidirectional relationship with the ability to eagerly take the collection back. 

Anywho, thanks again for your help!

On Mon, Aug 20, 2012 at 10:28 AM, John Rellis <[hidden email]> wrote:
Thanks.  Taking a look at this now.


On Fri, Aug 17, 2012 at 4:22 PM, Peter Ledbrook <[hidden email]> wrote:
> Is there a way to do this without a belongsTo in  classes that don't make
> sense to have a belongsTo??  Example :

Have you seen this?

    http://assarconsulting.blogspot.co.uk/2010/10/grails-one-to-many-mapping-with-foreign.html

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
John Rellis




--
John Rellis

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

pledbrook
> Hmmm, ok, this isn't going to work for me because many people can have the
> same person in their contact list so I think join table is required for this
> type of behaviour, although the monday fog could be clouding my mind.
> Otherwise, I would be fine with using a belongsTo.

I suggest you look into managing your own link domain class. It's
uglier, but at least you'll have more control over the way that
relations are fetched. And raise an issue. Alternatively, configure
the mapping yourself in Hibernate XML:

    http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/associations.html#assoc-unidirectional-join

This should also allow you to specify the default fetch strategy:

    http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/performance.html#performance-fetching-custom

Hope that helps,

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Cannot eagerly fetch collection, join and select not working???

johnrellis
That's great peter, thanks very much.

I have never used hibernate outside of grails so I always find this stuff fairly interesting.  Can lead me down some interesting rabbit holes!

On Mon, Aug 20, 2012 at 1:55 PM, Peter Ledbrook <[hidden email]> wrote:
> Hmmm, ok, this isn't going to work for me because many people can have the
> same person in their contact list so I think join table is required for this
> type of behaviour, although the monday fog could be clouding my mind.
> Otherwise, I would be fine with using a belongsTo.

I suggest you look into managing your own link domain class. It's
uglier, but at least you'll have more control over the way that
relations are fetched. And raise an issue. Alternatively, configure
the mapping yourself in Hibernate XML:

    http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/associations.html#assoc-unidirectional-join

This should also allow you to specify the default fetch strategy:

    http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/performance.html#performance-fetching-custom

Hope that helps,

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
John Rellis

Loading...