zero down time with tc server and parallel deployment

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view

zero down time with tc server and parallel deployment

Julien ROTT

I'm trying to use the parallel deployment feature of tc server in order to have zero down time.

In my tests, this works fine, but after a few deployments the server crashes with memory errors.

I ran the server with the -XX:+HeapDumpOnOutOfMemoryError option to create a dump and used Eclipse Memory Analyzer Tool to analyze the dump.

It appears that there are as many instances of org.apache.catalina.loader.WebappClassLoader as the number of versions of my app I deployed.

What can I do to be sure that my app is completely undeployed when a new version is deployed ?



Julien ROTT.
Reply | Threaded
Open this post in threaded view

Re: zero down time with tc server and parallel deployment

Are the errors PermGen related?

A year or so ago I was working on a similar thing where we wanted to be able to hot redeploy the app, but after a few redeploys it would do what you describe, crash with PermGen memory errors.  We were using plain Tomcat.  A lot of research and we eventually started finding stuff about the class loaders not releasing all the resources.  I'd have to dig back in my notes and emails to get more specific.

Eventually through trial and error I tracked it down to the MySQL JDBC driver.  I think I remember creating a blank app with no database and redeploying it without problems, then once I added the JDBC driver it would exhibit the problem.

The fix was to move the JDBC driver out of the project and into Tomcat itself.  So in the BuildConfig we changed the dependency to "provided 'mysql:mysql-connector-java:5.1.23'", and put the JDBC driver into the Tomcat's lib directory.  We also changed the DataSource to use JNDI connections rather than putting the URLs there.  So the DataSource.groovy would have something like this:

 production {
                dataSource {

And then define the JNDI data source in Tomcat.  After that we didn't have memory problems with redeployment.

At least until we started to add other libraries, a few times the problem would pop back up, and eventually we had to test every single time we added a dependency into the project, sometimes it would pop back up and we'd have to either try moving it outside Tomcat (which became a deployment process nightmare) or sometimes look for an alternative library.

I'm not sure if your problem is the same, but it sounds similar so hopefully that's helpful.