events-push lease threads

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

events-push lease threads

longwa
This is our second attempt at using events-push plugin and we keep seeing the same issue each time. Last time we thought it was because we weren't using NIO, but now we are setup with NIO connector and using SSE protocol.

We eventually end up with a dead server and hundreds of threads like this:

"http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800 nid=0x2cc1 waiting on condition [0x00007f079b4f0000]      java.lang.Thread.State: WAITING (parking)      at sun.misc.Unsafe.park(Native Method)      - parking to wait for <0x00000006372e0460> (a java.util.concurrent.CountDownLatch$Sync)      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)      at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)      at org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)      at org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)      at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)      at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)      at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)      at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)      at org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)      at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)      at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)      at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)      at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)      at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)      at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)

-Aaron
Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

longwa
Should be leaks threads, not lease threads...autocorrect fail.

Here's a more readable stack:

"http-nio-8080-exec-217" daemon prio=10 tid=0x00007f0830147000 nid=0x2e1c waiting on condition [0x00007f079359e000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x000000078c4b69a0> (a java.util.concurrent.CountDownLatch$Sync)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)

        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)

        at org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)

        at org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)

        at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)

        at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)

        at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)

        at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)

        at org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)

        at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)

        at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)



On Mon, Nov 11, 2013 at 10:22 AM, Aaron Long <[hidden email]> wrote:
This is our second attempt at using events-push plugin and we keep seeing the same issue each time. Last time we thought it was because we weren't using NIO, but now we are setup with NIO connector and using SSE protocol.

We eventually end up with a dead server and hundreds of threads like this:

"http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800 nid=0x2cc1 waiting on condition [0x00007f079b4f0000]      java.lang.Thread.State: WAITING (parking)      at sun.misc.Unsafe.park(Native Method)      - parking to wait for <0x00000006372e0460> (a java.util.concurrent.CountDownLatch$Sync)      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)      at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)      at org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)      at org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)      at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)      at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)      at org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)      at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)      at org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)      at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)      at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)      at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)      at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)      at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)      at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)

-Aaron

Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

zyro
that stacktrace does look like tomcat is using nio
("http-nio-8080-exec") but atmosphere does not recognize it or cannot
use it:

...org.atmosphere.container.Tomcat7BIOSupportWithWebSocket...
...BlockingIOCometSupport...

- did you set servlet version to 3.0 everywhere (BuildConfig.groovy and
prior to 2.3.0 in application.properties as well)?
- did you make sure all filters that the requests goes through and the
meteorServlet have <async-supported>true</async-supported>?
- did you try including/excluding the atmosphere-native dependencies?

if those do not help consider using a more recent version of atmosphere
(remember then to update atmosphere.js as well (if required)).

zyro

-------- Original Message  --------
Subject: [grails-user] Re: events-push lease threads
From: Aaron Long <[hidden email]>
To: [hidden email] <[hidden email]>
Date: Mon, 11 Nov 2013 10:25:32 -0500

> Should be leaks threads, not lease threads...autocorrect fail.
>
> Here's a more readable stack:
>
> "http-nio-8080-exec-217" daemon prio=10 tid=0x00007f0830147000
> nid=0x2e1c waiting on condition [0x00007f079359e000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>         at sun.misc.Unsafe.park(Native Method)
>
>         - parking to wait for  <0x000000078c4b69a0> (a
> java.util.concurrent.CountDownLatch$Sync)
>
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>
>         at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
>
>         at
> org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
>
>         at
> org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
>
>         at
> org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
>
>         at
> org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
>
>         at
> org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
>
>         at
> org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
>
>         at
> org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
>
>         at
> org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>         at
> org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
>
>         at
> org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
>
>         at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>         at
> org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>
>
> On Mon, Nov 11, 2013 at 10:22 AM, Aaron Long <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     This is our second attempt at using events-push plugin and we keep
>     seeing the same issue each time. Last time we thought it was because
>     we weren't using NIO, but now we are setup with NIO connector and
>     using SSE protocol.
>
>     We eventually end up with a dead server and hundreds of threads like
>     this:
>
>     "http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800
>     nid=0x2cc1 waiting on condition [0x00007f079b4f0000]    
>      java.lang.Thread.State: WAITING (parking)      at
>     sun.misc.Unsafe.park(Native Method)      - parking to wait for
>     <0x00000006372e0460> (a java.util.concurrent.CountDownLatch$Sync)  
>        at
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)  
>        at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>          at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>          at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>          at
>     java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)  
>        at
>     org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
>          at
>     org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
>          at
>     org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
>          at
>     org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
>          at
>     org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
>          at
>     org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
>          at
>     org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
>          at
>     org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
>          at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>          at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
>          at
>     org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
>          at
>     org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
>
>     -Aaron
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

longwa
Yeah, those are all good suggestions. I noticed that Tomcat7 seems to refuse to use the NIO adapter even though I've done everything that you suggested.

Looks like the latest builds of events-push are against atmosphere-2.0.2 which some people claim fixes issues such as this one. I might try switching to the snapshot build and see what happens.


On Mon, Nov 11, 2013 at 10:51 AM, zyro <[hidden email]> wrote:
that stacktrace does look like tomcat is using nio
("http-nio-8080-exec") but atmosphere does not recognize it or cannot
use it:

...org.atmosphere.container.Tomcat7BIOSupportWithWebSocket...
...BlockingIOCometSupport...

- did you set servlet version to 3.0 everywhere (BuildConfig.groovy and
prior to 2.3.0 in application.properties as well)?
- did you make sure all filters that the requests goes through and the
meteorServlet have <async-supported>true</async-supported>?
- did you try including/excluding the atmosphere-native dependencies?

if those do not help consider using a more recent version of atmosphere
(remember then to update atmosphere.js as well (if required)).

zyro

-------- Original Message  --------
Subject: [grails-user] Re: events-push lease threads
From: Aaron Long <[hidden email]>
To: [hidden email] <[hidden email]>
Date: Mon, 11 Nov 2013 10:25:32 -0500

> Should be leaks threads, not lease threads...autocorrect fail.
>
> Here's a more readable stack:
>
> "http-nio-8080-exec-217" daemon prio=10 tid=0x00007f0830147000
> nid=0x2e1c waiting on condition [0x00007f079359e000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>         at sun.misc.Unsafe.park(Native Method)
>
>         - parking to wait for  <0x000000078c4b69a0> (a
> java.util.concurrent.CountDownLatch$Sync)
>
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>
>         at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
>
>         at
> org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
>
>         at
> org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
>
>         at
> org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
>
>         at
> org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
>
>         at
> org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
>
>         at
> org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
>
>         at
> org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
>
>         at
> org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>         at
> org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
>
>         at
> org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
>
>         at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>         at
> org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>
>
> On Mon, Nov 11, 2013 at 10:22 AM, Aaron Long <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     This is our second attempt at using events-push plugin and we keep
>     seeing the same issue each time. Last time we thought it was because
>     we weren't using NIO, but now we are setup with NIO connector and
>     using SSE protocol.
>
>     We eventually end up with a dead server and hundreds of threads like
>     this:
>
>     "http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800
>     nid=0x2cc1 waiting on condition [0x00007f079b4f0000]
>      java.lang.Thread.State: WAITING (parking)      at
>     sun.misc.Unsafe.park(Native Method)      - parking to wait for
>     <0x00000006372e0460> (a java.util.concurrent.CountDownLatch$Sync)
>        at
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>        at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>          at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>          at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>          at
>     java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
>        at
>     org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
>          at
>     org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
>          at
>     org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
>          at
>     org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
>          at
>     org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
>          at
>     org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
>          at
>     org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
>          at
>     org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
>          at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>          at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
>          at
>     org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
>          at
>     org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
>
>     -Aaron
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

smaldini
Any updates ?


On Mon, Nov 11, 2013 at 6:30 PM, Aaron Long <[hidden email]> wrote:
Yeah, those are all good suggestions. I noticed that Tomcat7 seems to refuse to use the NIO adapter even though I've done everything that you suggested.

Looks like the latest builds of events-push are against atmosphere-2.0.2 which some people claim fixes issues such as this one. I might try switching to the snapshot build and see what happens.


On Mon, Nov 11, 2013 at 10:51 AM, zyro <[hidden email]> wrote:
that stacktrace does look like tomcat is using nio
("http-nio-8080-exec") but atmosphere does not recognize it or cannot
use it:

...org.atmosphere.container.Tomcat7BIOSupportWithWebSocket...
...BlockingIOCometSupport...

- did you set servlet version to 3.0 everywhere (BuildConfig.groovy and
prior to 2.3.0 in application.properties as well)?
- did you make sure all filters that the requests goes through and the
meteorServlet have <async-supported>true</async-supported>?
- did you try including/excluding the atmosphere-native dependencies?

if those do not help consider using a more recent version of atmosphere
(remember then to update atmosphere.js as well (if required)).

zyro

-------- Original Message  --------
Subject: [grails-user] Re: events-push lease threads
From: Aaron Long <[hidden email]>
To: [hidden email] <[hidden email]>
Date: Mon, 11 Nov 2013 10:25:32 -0500

> Should be leaks threads, not lease threads...autocorrect fail.
>
> Here's a more readable stack:
>
> "http-nio-8080-exec-217" daemon prio=10 tid=0x00007f0830147000
> nid=0x2e1c waiting on condition [0x00007f079359e000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>         at sun.misc.Unsafe.park(Native Method)
>
>         - parking to wait for  <0x000000078c4b69a0> (a
> java.util.concurrent.CountDownLatch$Sync)
>
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>
>         at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
>
>         at
> org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
>
>         at
> org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
>
>         at
> org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
>
>         at
> org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
>
>         at
> org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
>
>         at
> org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
>
>         at
> org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
>
>         at
> org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>         at
> org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
>
>         at
> org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
>
>         at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>         at
> org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>
>
>
> On Mon, Nov 11, 2013 at 10:22 AM, Aaron Long <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     This is our second attempt at using events-push plugin and we keep
>     seeing the same issue each time. Last time we thought it was because
>     we weren't using NIO, but now we are setup with NIO connector and
>     using SSE protocol.
>
>     We eventually end up with a dead server and hundreds of threads like
>     this:
>
>     "http-nio-8080-exec-80" daemon prio=10 tid=0x00007f0830051800
>     nid=0x2cc1 waiting on condition [0x00007f079b4f0000]
>      java.lang.Thread.State: WAITING (parking)      at
>     sun.misc.Unsafe.park(Native Method)      - parking to wait for
>     <0x00000006372e0460> (a java.util.concurrent.CountDownLatch$Sync)
>        at
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>        at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>          at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>          at
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>          at
>     java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
>        at
>     org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:153)
>          at
>     org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
>          at
>     org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.doService(Tomcat7BIOSupportWithWebSocket.java:65)
>          at
>     org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:84)
>          at
>     org.atmosphere.container.Tomcat7BIOSupportWithWebSocket.service(Tomcat7BIOSupportWithWebSocket.java:61)
>          at
>     org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1370)
>          at
>     org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:286)
>          at
>     org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:272)
>          at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>          at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.processFilterChain(UrlMappingsFilter.java:377)
>          at
>     org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:244)
>          at
>     org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:139)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>          at
>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>          at
>     org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
>
>     -Aaron
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email






--
Stéphane
--

Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

Richard Polderman
This post was updated on .
I am experiencing the same issues. Is there any news on how to solve this?


Regards,
Richard
Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

Richard Polderman
I am experiencing the same issues. Is there any news on how to solve this?

Regards,
Richard
Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

longwa
There's a discussion just started on the dev Google group about the future of events and events-push in Grails.


-Aaron


On Thu, May 15, 2014 at 9:35 AM, Richard Polderman <[hidden email]> wrote:
I am experiencing the same issues. Is there any news on how to solve this?

Regards,
Richard



--
View this message in context: http://grails.1312388.n4.nabble.com/events-push-lease-threads-tp4651349p4656905.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: events-push lease threads

Richard Polderman
Hi Aaron,

I wil follow that discussion!

Thanks,
Richard