Pretty urgent: Grails 2.2.3 and sudden CPU hunger

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Pretty urgent: Grails 2.2.3 and sudden CPU hunger

theirish81
Hello everyone,
so we have this big grails application with lots and lots of pages with grails.gsp.enable.reload=true. Pages aren't changed frequently and our permgen monitors say everything is ok with it.
The app doesn't use hibernate, resources plugin... it's bare minimal.
Also, the application lives with other 2 applications we would consider "ok" for now, but it's the only one using grails.

What happens is very subtle. This is the most relevant test we did:
- we start the server with all applications active. All apps generate traffic but everything works fine with  little cpu load
- we set the grails app in draining (not necessarily, but this gives you a hint)
- hours later, the tomcat server starts demanding CPU, reaching the 100%
- setting all the apps in draining brings peace to the CPU
- if we reactivate ANY of the apps, the deadly shows starts back

Now what's interesting is:
- if we start the server and leave the grails app in draining, this peak will never happen
- one request to the grails app, and sooner or later we'll see this show
- nothing shows up in the logs
- the memory usage looks ok

Other notes:
- Xerces and Xalan are installed in the tomcat/lib directory
- All three applications use EhCache 2.6.3

So my feeling is somehow the grails app activates something that will turn disruptive sooner or later. Now, since the app itself works on live requests and doesn't really starts threads or processes, I'm starting to think it might be related to the platform, but I'm far from being sure.

Anyone experienced anything like this? any hint for a close-to-desperate dev?
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Miles

Have you looked at visual vm and tc server too get an idea of what calls are taking so long and if any particular objects are churning or doing something odd?

On 17 Dec 2013 09:23, "theirish81" <[hidden email]> wrote:
Hello everyone,
so we have this big grails application with lots and lots of pages with
grails.gsp.enable.reload=true. Pages aren't changed frequently and our
permgen monitors say everything is ok with it.
The app doesn't use hibernate, resources plugin... it's bare minimal.
Also, the application lives with other 2 applications we would consider "ok"
for now, but it's the only one using grails.

What happens is very subtle. This is the most relevant test we did:
- we start the server with all applications active. All apps generate
traffic but everything works fine with  little cpu load
- we set the grails app in draining (not necessarily, but this gives you a
hint)
- hours later, the tomcat server starts demanding CPU, reaching the 100%
- setting all the apps in draining brings peace to the CPU
- if we reactivate ANY of the apps, the deadly shows starts back

Now what's interesting is:
- if we start the server and leave the grails app in draining, this peak
will never happen
- one request to the grails app, and sooner or later we'll see this show
- nothing shows up in the logs
- the memory usage looks ok

Other notes:
- Xerces and Xalan are installed in the tomcat/lib directory
- All three applications use EhCache 2.6.3

So my feeling is somehow the grails app activates something that will turn
disruptive sooner or later. Now, since the app itself works on live requests
and doesn't really starts threads or processes, I'm starting to think it
might be related to the platform, but I'm far from being sure.

Anyone experienced anything like this? any hint for a close-to-desperate
dev?



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

theirish81
Thanks for the reply, Miles,
we looked at visual VM and what we saw is pretty bizarre. Every thread, even the dumbest, simply consume lots more CPU than usual.
So the usual threads all of a sudden start consuming N times what they normally do. There seems to be no "blocking" thread, but everything looks overdriven.
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Miles

Hmm, and you don't see this effect when reloading is disabled?

Quick one, what os are you running?

On 17 Dec 2013 09:50, "theirish81" <[hidden email]> wrote:
Thanks for the reply, Miles,
we looked at visual VM and what we saw is pretty bizarre. Every thread, even
the dumbest, simply consume lots more CPU than usual.
So the usual threads all of a sudden start consuming N times what they
normally do. There seems to be no "blocking" thread, but everything looks
overdriven.



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531p4652534.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Graeme Rocher-2
In reply to this post by theirish81
This sounds like when you hit a low memory situation and get into "metaclass churn"

Cheers


On Tue, Dec 17, 2013 at 10:22 AM, theirish81 <[hidden email]> wrote:
Hello everyone,
so we have this big grails application with lots and lots of pages with
grails.gsp.enable.reload=true. Pages aren't changed frequently and our
permgen monitors say everything is ok with it.
The app doesn't use hibernate, resources plugin... it's bare minimal.
Also, the application lives with other 2 applications we would consider "ok"
for now, but it's the only one using grails.

What happens is very subtle. This is the most relevant test we did:
- we start the server with all applications active. All apps generate
traffic but everything works fine with  little cpu load
- we set the grails app in draining (not necessarily, but this gives you a
hint)
- hours later, the tomcat server starts demanding CPU, reaching the 100%
- setting all the apps in draining brings peace to the CPU
- if we reactivate ANY of the apps, the deadly shows starts back

Now what's interesting is:
- if we start the server and leave the grails app in draining, this peak
will never happen
- one request to the grails app, and sooner or later we'll see this show
- nothing shows up in the logs
- the memory usage looks ok

Other notes:
- Xerces and Xalan are installed in the tomcat/lib directory
- All three applications use EhCache 2.6.3

So my feeling is somehow the grails app activates something that will turn
disruptive sooner or later. Now, since the app itself works on live requests
and doesn't really starts threads or processes, I'm starting to think it
might be related to the platform, but I'm far from being sure.

Anyone experienced anything like this? any hint for a close-to-desperate
dev?



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
Graeme Rocher
Grails Project Lead
SpringSource
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Miles

Graeme, how would this surface in visual vm? Tons of eden churn?

On 17 Dec 2013 10:53, "Graeme Rocher" <[hidden email]> wrote:
This sounds like when you hit a low memory situation and get into "metaclass churn"

Cheers


On Tue, Dec 17, 2013 at 10:22 AM, theirish81 <[hidden email]> wrote:
Hello everyone,
so we have this big grails application with lots and lots of pages with
grails.gsp.enable.reload=true. Pages aren't changed frequently and our
permgen monitors say everything is ok with it.
The app doesn't use hibernate, resources plugin... it's bare minimal.
Also, the application lives with other 2 applications we would consider "ok"
for now, but it's the only one using grails.

What happens is very subtle. This is the most relevant test we did:
- we start the server with all applications active. All apps generate
traffic but everything works fine with  little cpu load
- we set the grails app in draining (not necessarily, but this gives you a
hint)
- hours later, the tomcat server starts demanding CPU, reaching the 100%
- setting all the apps in draining brings peace to the CPU
- if we reactivate ANY of the apps, the deadly shows starts back

Now what's interesting is:
- if we start the server and leave the grails app in draining, this peak
will never happen
- one request to the grails app, and sooner or later we'll see this show
- nothing shows up in the logs
- the memory usage looks ok

Other notes:
- Xerces and Xalan are installed in the tomcat/lib directory
- All three applications use EhCache 2.6.3

So my feeling is somehow the grails app activates something that will turn
disruptive sooner or later. Now, since the app itself works on live requests
and doesn't really starts threads or processes, I'm starting to think it
might be related to the platform, but I'm far from being sure.

Anyone experienced anything like this? any hint for a close-to-desperate
dev?



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
Graeme Rocher
Grails Project Lead
SpringSource
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Graeme Rocher-2
Not sure, but basically the root cause is common Java classes like String, Integer, List etc. do not have a 'metaClass' property, so Groovy has to get a MetaClass for these classes from the registry. When memory is low, the MetaClass is retrieved from the registry, then immediately garbage collected. If you are doing a lot of String, or Integer operations (as most web apps do) this results the MetaClass being collected and recreated for each method call, which turns your application into a zombie and spikes the CPU.

So, never run a Grails app without the required amount of memory :)


On Tue, Dec 17, 2013 at 12:04 PM, Miles Burton <[hidden email]> wrote:

Graeme, how would this surface in visual vm? Tons of eden churn?

On 17 Dec 2013 10:53, "Graeme Rocher" <[hidden email]> wrote:
This sounds like when you hit a low memory situation and get into "metaclass churn"

Cheers


On Tue, Dec 17, 2013 at 10:22 AM, theirish81 <[hidden email]> wrote:
Hello everyone,
so we have this big grails application with lots and lots of pages with
grails.gsp.enable.reload=true. Pages aren't changed frequently and our
permgen monitors say everything is ok with it.
The app doesn't use hibernate, resources plugin... it's bare minimal.
Also, the application lives with other 2 applications we would consider "ok"
for now, but it's the only one using grails.

What happens is very subtle. This is the most relevant test we did:
- we start the server with all applications active. All apps generate
traffic but everything works fine with  little cpu load
- we set the grails app in draining (not necessarily, but this gives you a
hint)
- hours later, the tomcat server starts demanding CPU, reaching the 100%
- setting all the apps in draining brings peace to the CPU
- if we reactivate ANY of the apps, the deadly shows starts back

Now what's interesting is:
- if we start the server and leave the grails app in draining, this peak
will never happen
- one request to the grails app, and sooner or later we'll see this show
- nothing shows up in the logs
- the memory usage looks ok

Other notes:
- Xerces and Xalan are installed in the tomcat/lib directory
- All three applications use EhCache 2.6.3

So my feeling is somehow the grails app activates something that will turn
disruptive sooner or later. Now, since the app itself works on live requests
and doesn't really starts threads or processes, I'm starting to think it
might be related to the platform, but I'm far from being sure.

Anyone experienced anything like this? any hint for a close-to-desperate
dev?



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email





--
Graeme Rocher
Grails Project Lead
SpringSource



--
Graeme Rocher
Grails Project Lead
SpringSource
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

theirish81
Server: RedHat 5.7
Memory: 64GB ram -> 48GB of heap, 500MB of perm (128 used)

The system occupies little memory even when it goes nuts.

It would be very tough for us to disable reloading, but we'll find a way to give it a try

Thanks everyone
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Miles

48gig!? Have you enabled gc logging?  Which jvm are you using?

Server: RedHat 5.7
Memory: 64GB ram -> 48GB of heap, 500MB of perm (128 used)

The system occupies little memory even when it goes nuts.

It would be very tough for us to disable reloading, but we'll find a way to
give it a try

Thanks everyone



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531p4652546.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

theirish81
Not bad uh? :)
As Garbage Collector we're using G1. The logging states a regular GC every few seconds (around 10 seconds), and we mostly see young memory collections. This frequency is definitely compatible with the traffic we normally have.
The JVM is a Oracle JVM 1.7.40

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

theirish81
I forgot to mention we use Tomcat 6
Reply | Threaded
Open this post in threaded view
|

Re: Pretty urgent: Grails 2.2.3 and sudden CPU hunger

Lari Hotari -
In reply to this post by theirish81

See http://jira.grails.org/browse/GRAILS-10852 and http://jira.grails.org/browse/GRAILS-10882 .

Lari

17.12.2013 11.23 kirjoitti "theirish81" <[hidden email]>:
Hello everyone,
so we have this big grails application with lots and lots of pages with
grails.gsp.enable.reload=true. Pages aren't changed frequently and our
permgen monitors say everything is ok with it.
The app doesn't use hibernate, resources plugin... it's bare minimal.
Also, the application lives with other 2 applications we would consider "ok"
for now, but it's the only one using grails.

What happens is very subtle. This is the most relevant test we did:
- we start the server with all applications active. All apps generate
traffic but everything works fine with  little cpu load
- we set the grails app in draining (not necessarily, but this gives you a
hint)
- hours later, the tomcat server starts demanding CPU, reaching the 100%
- setting all the apps in draining brings peace to the CPU
- if we reactivate ANY of the apps, the deadly shows starts back

Now what's interesting is:
- if we start the server and leave the grails app in draining, this peak
will never happen
- one request to the grails app, and sooner or later we'll see this show
- nothing shows up in the logs
- the memory usage looks ok

Other notes:
- Xerces and Xalan are installed in the tomcat/lib directory
- All three applications use EhCache 2.6.3

So my feeling is somehow the grails app activates something that will turn
disruptive sooner or later. Now, since the app itself works on live requests
and doesn't really starts threads or processes, I'm starting to think it
might be related to the platform, but I'm far from being sure.

Anyone experienced anything like this? any hint for a close-to-desperate
dev?



--
View this message in context: http://grails.1312388.n4.nabble.com/Pretty-urgent-Grails-2-2-3-and-sudden-CPU-hunger-tp4652531.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email