Grails app fails to automatically start and aborts if datasource is unavailable

[gatherer]

Ocasionally, when one of my servers has to reboot for instance after a long blackout, one of my datasources (datasources plugin) is not available for a while. When my grails app tries to start, it cannot reach that datasource (Network is unreachable: connect Exception) and aborts the whole application but tomcat is still running. That means the app is not available until someone manually restarts it, which is not ideal.

Is there any way to tell grails to keep trying until it gets a connection with all the datasources instead of just giving up?

[gatherer]

Any ideas?

[pledbrook]

> Ocasionally, when one of my servers has to reboot for instance after a long
> blackout, one of my datasources (datasources plugin) is not available for a
> while. When my grails app tries to start, it cannot reach that datasource
> (Network is unreachable: connect Exception) and aborts the whole application
> but tomcat is still running. That means the app is not available until
> someone manually restarts it, which is not ideal.
>
> Is there any way to tell grails to keep trying until it gets a connection
> with all the datasources instead of just giving up?

Will your application actually work if that data source isn't available?

Also, do you have a stacktrace? That will at least help determine what
is throwing the exception and hence where you should focus on coding
your own solution.

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[bksaville]

We've noticed this as well and always happens in the internals of grails.  It would GREAT to have the application continue to start and keep trying, or just to start and report the datasource as down, as it does after the application is started and the datasource goes down.  Do you think we could file a JIRA on this to get it addressed in grails-core?

Thanks,

-Brian

On Tue, Sep 25, 2012 at 7:03 AM, Peter Ledbrook <[hidden email]> wrote:

> Ocasionally, when one of my servers has to reboot for instance after a long
> blackout, one of my datasources (datasources plugin) is not available for a
> while. When my grails app tries to start, it cannot reach that datasource
> (Network is unreachable: connect Exception) and aborts the whole application
> but tomcat is still running. That means the app is not available until
> someone manually restarts it, which is not ideal.
>
> Is there any way to tell grails to keep trying until it gets a connection
> with all the datasources instead of just giving up?

Will your application actually work if that data source isn't available?

Also, do you have a stacktrace? That will at least help determine what
is throwing the exception and hence where you should focus on coding
your own solution.

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Christian Ramseyer]

> On Tue, Sep 25, 2012 at 7:03 AM, Peter Ledbrook <[hidden email]
> <mailto:[hidden email]>> wrote:
>     Also, do you have a stacktrace? That will at least help determine what
>     is throwing the exception and hence where you should focus on coding
>     your own solution.


On 9/25/12 5:22 PM, Brian Saville wrote:> We've noticed this as well and
always happens in the internals of
> grails.  It would GREAT to have the application continue to start and
> keep trying, or just to start and report the datasource as down, as it
> does after the application is started and the datasource goes down.  Do
> you think we could file a JIRA on this to get it addressed in grails-core?



Yes that would be a great feature, especially for applications with
multiple datasources.

I'm currently working on an application with many datasources, and not
being able to start it just because 5% of the functionality isn't there
somehow sucks.

Here's a stack trace of what happens in our application if e.g. a
Oracle-based datasource is not available during startup (sorry, long and
badly line-wrapped):

| Loading Grails 2.1.0
| Configuring classpath.
| Environment set to development.....
| Packaging Grails application.....
| Running Grails application
ERROR web.context.GrailsContextLoader - Error executing bootstraps:
Error creating bean with name 'voice.BusinessunitValidator_voice':
Cannot resolve reference to bean 'sessionFactory_voice' while setting
bean property 'sessionFactory'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'sessionFactory_voice': Cannot resolve reference to bean
'lobHandlerDetector_voice' while setting bean property 'lobHandler';
nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'lobHandlerDetector_voice': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'voice.BusinessunitValidator_voice': Cannot resolve
reference to bean 'sessionFactory_voice' while setting bean property
'sessionFactory'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'sessionFactory_voice': Cannot resolve reference to bean
'lobHandlerDetector_voice' while setting bean property 'lobHandler';
nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'lobHandlerDetector_voice': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'sessionFactory_voice': Cannot resolve
reference to bean 'lobHandlerDetector_voice' while setting bean property
'lobHandler'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'lobHandlerDetector_voice': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        ... 5 more
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'lobHandlerDetector_voice': Invocation of
init method failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        ... 5 more
Caused by: org.springframework.jdbc.support.MetaDataAccessException:
Error while extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        ... 5 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
        at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
        at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        ... 5 more
Caused by: java.sql.SQLException: Io exception: The Network Adapter
could not establish the connection
        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:179)
        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:334)
        at oracle.jdbc.ttc7.TTC7Protocol.handleIOException(TTC7Protocol.java:3668)
        at oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:353)
        at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:371)
        at
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:551)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:351)
        at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
        at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
        at
org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
        at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[bksaville]

http://jira.grails.org/browse/GRAILS-9442 submitted as a bug, even though this could be considered an improvement I guess.

-Brian

On Tue, Sep 25, 2012 at 10:29 AM, Christian Ramseyer <[hidden email]> wrote:

> On Tue, Sep 25, 2012 at 7:03 AM, Peter Ledbrook <[hidden email]

> <mailto:[hidden email]>> wrote:
>     Also, do you have a stacktrace? That will at least help determine what
>     is throwing the exception and hence where you should focus on coding
>     your own solution.

On 9/25/12 5:22 PM, Brian Saville wrote:> We've noticed this as well and

always happens in the internals of
> grails.  It would GREAT to have the application continue to start and
> keep trying, or just to start and report the datasource as down, as it
> does after the application is started and the datasource goes down.  Do
> you think we could file a JIRA on this to get it addressed in grails-core?

Yes that would be a great feature, especially for applications with
multiple datasources.

I'm currently working on an application with many datasources, and not
being able to start it just because 5% of the functionality isn't there
somehow sucks.

Here's a stack trace of what happens in our application if e.g. a
Oracle-based datasource is not available during startup (sorry, long and
badly line-wrapped):

| Loading Grails 2.1.0
| Configuring classpath.
| Environment set to development.....
| Packaging Grails application.....
| Running Grails application
ERROR web.context.GrailsContextLoader - Error executing bootstraps:
Error creating bean with name 'voice.BusinessunitValidator_voice':
Cannot resolve reference to bean 'sessionFactory_voice' while setting
bean property 'sessionFactory'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'sessionFactory_voice': Cannot resolve reference to bean
'lobHandlerDetector_voice' while setting bean property 'lobHandler';
nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'lobHandlerDetector_voice': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'voice.BusinessunitValidator_voice': Cannot resolve
reference to bean 'sessionFactory_voice' while setting bean property
'sessionFactory'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'sessionFactory_voice': Cannot resolve reference to bean
'lobHandlerDetector_voice' while setting bean property 'lobHandler';
nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'lobHandlerDetector_voice': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'sessionFactory_voice': Cannot resolve
reference to bean 'lobHandlerDetector_voice' while setting bean property
'lobHandler'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'lobHandlerDetector_voice': Invocation of init method
failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        ... 5 more
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'lobHandlerDetector_voice': Invocation of
init method failed; nested exception is
org.springframework.jdbc.support.MetaDataAccessException: Error while
extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        ... 5 more
Caused by: org.springframework.jdbc.support.MetaDataAccessException:
Error while extracting DatabaseMetaData; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        ... 5 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Io exception: The Network Adapter could not
establish the connection)
        at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
        at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
        at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        ... 5 more
Caused by: java.sql.SQLException: Io exception: The Network Adapter
could not establish the connection
        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:179)
        at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:334)
        at oracle.jdbc.ttc7.TTC7Protocol.handleIOException(TTC7Protocol.java:3668)
        at oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:353)
        at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:371)
        at
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:551)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:351)
        at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
        at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
        at
org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
        at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Lari Hotari -]

25.09.2012 20:13, Brian Saville wrote:
> http://jira.grails.org/browse/GRAILS-9442 submitted as a bug, even
> though this could be considered an improvement I guess.
> -Brian

In plain Spring applications, I've been using
http://static.springsource.org/spring/docs/3.1.x/javadoc-api/org/springframework/jdbc/datasource/LazyConnectionDataSourceProxy.html
to get around this type of problems.

my comment on the issue:
http://jira.grails.org/browse/GRAILS-9442?focusedCommentId=72403&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-72403

Lari

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[sergiomichels]

Definitely a good feature! Upvoted.

I've tried to deploy my app in a OC4J (Oracle) container and had problems because I was using JNDI to define the datasource and it isn't available during the deploy. I lost a lot of time to discover some way to define the datasource globally in the container as "workaround" for this.

--

Sérgio Michels

On Wed, Sep 26, 2012 at 2:39 PM, Lari Hotari <[hidden email]> wrote:

<a href="tel:25.09.2012%2020" value="[phone]">25.09.2012 20:13, Brian Saville wrote:
> http://jira.grails.org/browse/GRAILS-9442 submitted as a bug, even
> though this could be considered an improvement I guess.
> -Brian

In plain Spring applications, I've been using
http://static.springsource.org/spring/docs/3.1.x/javadoc-api/org/springframework/jdbc/datasource/LazyConnectionDataSourceProxy.html
to get around this type of problems.

my comment on the issue:
http://jira.grails.org/browse/GRAILS-9442?focusedCommentId=72403&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-72403

Lari

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[burtbeckwith]

I did a blog post on this a while back and did an updated post here: http://burtbeckwith.com/blog/?p=1565

Burt

bksaville wrote

http://jira.grails.org/browse/GRAILS-9442 submitted as a bug, even though
this could be considered an improvement I guess.
-Brian

[bksaville]

Awesome, thanks Burt.  I actually remember that blog post now, but I ignored it since it didn't apply to me at the time :)

One question though, how would this work with the MongoDB plugin/Grails data mapping project?  Is lobHandlerDetector still used, etc?

Thanks!

-Brian

On Wed, Sep 26, 2012 at 12:51 PM, burtbeckwith <[hidden email]> wrote:

I did a blog post on this a while back and did an updated post here:
http://burtbeckwith.com/blog/?p=1565

Burt


bksaville wrote

> http://jira.grails.org/browse/GRAILS-9442 submitted as a bug, even though
> this could be considered an improvement I guess.
> -Brian

--
View this message in context: http://grails.1312388.n4.nabble.com/Grails-app-fails-to-automatically-start-and-aborts-if-datasource-is-unavailable-tp4634985p4635512.html

Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[gatherer]

I think I have at least partially solved my issue thanks to your great posts Burt, now I can define "optional" datasources that do not have to be online when the application starts.

For this particular project I am using grails 1.3.x with the datasources plugin, and the problematic database connection is one of the extra datasources.

I have followed the steps you described in your post http://burtbeckwith.com/blog/?p=1565.

1- Define the dialect in each datasource so that grails doesn't have to query it at startup.

2- Redefine the lobHandlerDetector for each problematic datasource that is not needed at startup.

3- Add temp.use_jdbc_metadata_defaults = false in the datasource hibernate clause. I couldn't and did not do it!!! See below...

As I am using grails 1.3.X and the datasources plugin, the hibernate clause syntax in each of the additional datasources is different than on grails 2.X, and no matter what I tried I couldn't get it to recognize the temp member (I suspect it is not mapped or soemthing). So in the end I didn't add it and it still worked! It does throw some exceptions when starting if the datasources are not avalable, but it succeeds and as soon as they're back online I can use the functionality that depends on then.

I don't think I'm breaking anything by not including the third step as I'm only "touching" the optional datasources which are read-only and leaving the main DB/datasource alone, but it would be nice if someone could confirm it.