Don't use range validator...

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Don't use range validator...

Volnei Munhoz
... especially for large ranges!

Hi all, 

I'm writing just to share with you my terrible experience with the range validator.
I and my team are working on a large project myself with hundreds of domain classes, all with their validators and we are very careful in assigning them.

Well, part of this project was to produce in the last week and to our surprise it was a terrible experience. Especially when having a transaction insert into one of our domain classes . The server CPU reached 100 % and stayed there while the performance falls every minute. I confess that I felt frustrated.
We started investigating the causes of slowdowns and after exhausting any possibility of the database, triggers, constraints, retentions, locks and more... imagine that it could be something related to the behavior of hibernate.
Again we did not have any developments, we read several times the GORM Gotchas and some very good performances from Burt Beckwith, but we were still with the problem .
The light at the end of the tunnel appeared when disabling validation with a save (validate: false) and the application started to respond with a satisfactory time. But did not make sense that some constraints could generate so much delay.
Then investigated constraint by constraint and disabling one by one to identify what was being more costly and came to the conclusion that it was a range constraint we had in a field of a domain class. Something like this:

field range : 0 .. 9999999999

No biggie right? Wrong!

We understand what the operation of the range and to our surprise is not something simple like a:

it> from && it < to

The Range of the Groovy class actually creates a list and runs the same to see if the value is valid . Understand that I am not questioning the operation of the same , only saying that her performance can be very degrading . The following script takes only 1 milesegundo to run:

def r = 1 .. 9999999999
println r.contains ( 1 )

The same script to a larger value takes more than 1 second.

def r = 1..9999999999
println r.contains(20000000)

And with the maximum value takes up to 5 minutes.

Not terrible? After removing all constraints of range back to being satisfied with this framework that fills my eyes. And that does not pass through the same situation, I decided to share with you.

Enjoy!
Volnei Munhoz
Reply | Threaded
Open this post in threaded view
|

Re: Don't use range validator...

longwa
I think this is a problem with your code. The value "9999999999" is > Integer.MAX_VALUE and therefore causing the Groovy compiler to have to do some weird stuff when used as the end of the range.

For instance:

def r = 1..Integer.MAX_VALUE
println r.contains(20000000)

returns "true" in milliseconds as does:

def r = 1..2147483647 // MAX Integer value
println r.contains(20000000)

However, 

def r = 1..2147483648 // MAX + 1
println r.contains(20000000)

returns "false" because it ends up being 0 (or maybe -1, my two's complement is bad).

Anyway, if you use 1..Integer.MAX_VALUE as your upper range, you shouldn't have performance issues.

-Aaron

On Thu, May 22, 2014 at 1:37 PM, Volnei <[hidden email]> wrote:
... especially for large ranges!

Hi all, 

I'm writing just to share with you my terrible experience with the range validator.
I and my team are working on a large project myself with hundreds of domain classes, all with their validators and we are very careful in assigning them.

Well, part of this project was to produce in the last week and to our surprise it was a terrible experience. Especially when having a transaction insert into one of our domain classes . The server CPU reached 100 % and stayed there while the performance falls every minute. I confess that I felt frustrated.
We started investigating the causes of slowdowns and after exhausting any possibility of the database, triggers, constraints, retentions, locks and more... imagine that it could be something related to the behavior of hibernate.
Again we did not have any developments, we read several times the GORM Gotchas and some very good performances from Burt Beckwith, but we were still with the problem .
The light at the end of the tunnel appeared when disabling validation with a save (validate: false) and the application started to respond with a satisfactory time. But did not make sense that some constraints could generate so much delay.
Then investigated constraint by constraint and disabling one by one to identify what was being more costly and came to the conclusion that it was a range constraint we had in a field of a domain class. Something like this:

field range : 0 .. 9999999999

No biggie right? Wrong!

We understand what the operation of the range and to our surprise is not something simple like a:

it> from && it < to

The Range of the Groovy class actually creates a list and runs the same to see if the value is valid . Understand that I am not questioning the operation of the same , only saying that her performance can be very degrading . The following script takes only 1 milesegundo to run:

def r = 1 .. 9999999999
println r.contains ( 1 )

The same script to a larger value takes more than 1 second.

def r = 1..9999999999
println r.contains(20000000)

And with the maximum value takes up to 5 minutes.

Not terrible? After removing all constraints of range back to being satisfied with this framework that fills my eyes. And that does not pass through the same situation, I decided to share with you.

Enjoy!

Reply | Threaded
Open this post in threaded view
|

Re: Don't use range validator...

Volnei Munhoz
So if the follow code returns "false"

def r = 1..<a href="tel:2147483648" value="+552147483648" target="_blank" style="font-family:&#39;courier new&#39;,monospace;font-size:13px">2147483648 // MAX + 1
println r.contains(20000000)

Should follow code also returns "false", right?

def r = 1..99999999999999
println r.contains(2)

But it returns "true"




To me, there something wrong in this code:

def r = 1..9999999999
println r.contains(2000000)

def r1 = 1..Integer.MAX_VALUE
println r1.contains(2000000)

def r2 = 1..Integer.MAX_VALUE + 1
println r2.contains(2000000)

def r3 = 1..2147483647 + 1
println r3.contains(2000000)

def r4 = 1..2147483648
println r4.contains(2000000)


So as I told the "large ranges" can cause problems. 

I'm just alerting people who can have the same problem with no compiler or runtime exceptions.

Thanks


________________________________________
Volnei Granado Munhoz


2014-05-22 17:12 GMT-03:00 Aaron Long <[hidden email]>:
I think this is a problem with your code. The value "<a href="tel:9999999999" value="+559999999999" target="_blank">9999999999" is > Integer.MAX_VALUE and therefore causing the Groovy compiler to have to do some weird stuff when used as the end of the range.

For instance:

def r = 1..Integer.MAX_VALUE
println r.contains(20000000)

returns "true" in milliseconds as does:

def r = 1..<a href="tel:2147483647" value="+552147483647" target="_blank">2147483647 // MAX Integer value
println r.contains(20000000)

However, 

def r = 1..<a href="tel:2147483648" value="+552147483648" target="_blank">2147483648 // MAX + 1
println r.contains(20000000)

returns "false" because it ends up being 0 (or maybe -1, my two's complement is bad).

Anyway, if you use 1..Integer.MAX_VALUE as your upper range, you shouldn't have performance issues.

-Aaron

On Thu, May 22, 2014 at 1:37 PM, Volnei <[hidden email]> wrote:
... especially for large ranges!

Hi all, 

I'm writing just to share with you my terrible experience with the range validator.
I and my team are working on a large project myself with hundreds of domain classes, all with their validators and we are very careful in assigning them.

Well, part of this project was to produce in the last week and to our surprise it was a terrible experience. Especially when having a transaction insert into one of our domain classes . The server CPU reached 100 % and stayed there while the performance falls every minute. I confess that I felt frustrated.
We started investigating the causes of slowdowns and after exhausting any possibility of the database, triggers, constraints, retentions, locks and more... imagine that it could be something related to the behavior of hibernate.
Again we did not have any developments, we read several times the GORM Gotchas and some very good performances from Burt Beckwith, but we were still with the problem .
The light at the end of the tunnel appeared when disabling validation with a save (validate: false) and the application started to respond with a satisfactory time. But did not make sense that some constraints could generate so much delay.
Then investigated constraint by constraint and disabling one by one to identify what was being more costly and came to the conclusion that it was a range constraint we had in a field of a domain class. Something like this:

field range : <a href="tel:0%20..%209999999999" value="+559999999999" target="_blank">0 .. 9999999999

No biggie right? Wrong!

We understand what the operation of the range and to our surprise is not something simple like a:

it> from && it < to

The Range of the Groovy class actually creates a list and runs the same to see if the value is valid . Understand that I am not questioning the operation of the same , only saying that her performance can be very degrading . The following script takes only 1 milesegundo to run:

def r = <a href="tel:1%20..%209999999999" value="+5519999999999" target="_blank">1 .. 9999999999
println r.contains ( 1 )

The same script to a larger value takes more than 1 second.

def r = <a href="tel:1..9999999999" value="+5519999999999" target="_blank">1..9999999999
println r.contains(20000000)

And with the maximum value takes up to 5 minutes.

Not terrible? After removing all constraints of range back to being satisfied with this framework that fills my eyes. And that does not pass through the same situation, I decided to share with you.

Enjoy!


Volnei Munhoz
Reply | Threaded
Open this post in threaded view
|

Re: Don't use range validator...

longwa
This is generally true of anything if you use a value larger the Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is actually an IntRange implementation, I believe).

Type this into your groovyConsole:

int foo = 99999999999999

and you'll see that the result is actually 276447231. This is essentially what you get when you try to stuff a huge number into a signed 32-bit field.

I understand what you are saying, I just wanted to point out that it's really a misuse of the range that is causing the performance problems.

-Aaron

Reply | Threaded
Open this post in threaded view
|

Re: Don't use range validator...

Jeff Scott Brown-2


On May 22, 2014 at 3:59:54 PM, Aaron Long ([hidden email]) wrote:

> This is generally true of anything if you use a value larger the
> Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is
> actually an IntRange implementation, I believe).
>  
> Type this into your groovyConsole:
>  
> int foo = 99999999999999
>  
> and you'll see that the result is actually 276447231. This is essentially
> what you get when you try to stuff a huge number into a signed 32-bit field.
>  
> I understand what you are saying, I just wanted to point out that it's
> really a misuse of the range that is causing the performance problems.
>  
> -Aaron
>  


Can you guys take this to the new Google group or the Groovy mailing list?  We have deprecated the Grails mailing lists.  See http://grails.1312388.n4.nabble.com/Deprecating-The-Grails-Mailing-Lists-td4656735.html.

Thanks.



JSB

Jeff Scott Brown
[hidden email]

Find The Cause ~ Find The Cure  
http://www.autismspeaks.org/



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Don't use range validator...

Volnei Munhoz
George, yes! I actually do this, the porpouse for this post is just to alert others.

Jeff, sute!! 

Thanks..

________________________________________
Volnei Granado Munhoz


2014-05-22 18:32 GMT-03:00 Jeff Scott Brown <[hidden email]>:


On May 22, 2014 at 3:59:54 PM, Aaron Long ([hidden email]) wrote:
> This is generally true of anything if you use a value larger the
> Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is
> actually an IntRange implementation, I believe).
>
> Type this into your groovyConsole:
>
> int foo = 99999999999999
>
> and you'll see that the result is actually 276447231. This is essentially
> what you get when you try to stuff a huge number into a signed 32-bit field.
>
> I understand what you are saying, I just wanted to point out that it's
> really a misuse of the range that is causing the performance problems.
>
> -Aaron
>


Can you guys take this to the new Google group or the Groovy mailing list?  We have deprecated the Grails mailing lists.  See http://grails.1312388.n4.nabble.com/Deprecating-The-Grails-Mailing-Lists-td4656735.html.

Thanks.



JSB

Jeff Scott Brown
[hidden email]

Find The Cause ~ Find The Cure
http://www.autismspeaks.org/



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Volnei Munhoz
Reply | Threaded
Open this post in threaded view
|

Re: Don't use range validator...

Volnei Munhoz
Aaron,

I agree with you, is a misuse! Yes it's a IntRange but it also handle Long values... try this

def range = 1..9999999999999
println range.to.class

Now the to is a Long with 9999999999999 value.

See you in the Google Group.

Thanks!



________________________________________
Volnei Granado Munhoz


2014-05-22 19:15 GMT-03:00 Volnei <[hidden email]>:
George, yes! I actually do this, the porpouse for this post is just to alert others.

Jeff, sute!! 

Thanks..

________________________________________
Volnei Granado Munhoz


2014-05-22 18:32 GMT-03:00 Jeff Scott Brown <[hidden email]>:



On May 22, 2014 at 3:59:54 PM, Aaron Long ([hidden email]) wrote:
> This is generally true of anything if you use a value larger the
> Integer.MAX_VALUE for something that holds an Integer (Range in Groovy is
> actually an IntRange implementation, I believe).
>
> Type this into your groovyConsole:
>
> int foo = 99999999999999
>
> and you'll see that the result is actually 276447231. This is essentially
> what you get when you try to stuff a huge number into a signed 32-bit field.
>
> I understand what you are saying, I just wanted to point out that it's
> really a misuse of the range that is causing the performance problems.
>
> -Aaron
>


Can you guys take this to the new Google group or the Groovy mailing list?  We have deprecated the Grails mailing lists.  See http://grails.1312388.n4.nabble.com/Deprecating-The-Grails-Mailing-Lists-td4656735.html.

Thanks.



JSB

Jeff Scott Brown
[hidden email]

Find The Cause ~ Find The Cure
http://www.autismspeaks.org/



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email




Volnei Munhoz