ANN: Database Migration Plugin v0.1 released

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

ANN: Database Migration Plugin v0.1 released

burtbeckwith

I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

The plugin will only work with Grails 1.3 and above.

This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

Burt

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

Jean Barmash 1
Congratulations, Burt!  Looks amazing.  My understanding is that this is an initial version for migrations code for future versions of grails, am I correct?  What would you say the current status of this plugin - alpha, beta, or are there production systems out there that already use this?   I guess trying to gauge how careful I should be trying it out, especially knowing that it's built on a pretty solid foundation.  

Also, what is its relationship with autobase plugin - was the DSL designed to be somewhat compatible? 

Thanks,

Jean

On Sun, Jan 9, 2011 at 11:04 PM, Burt Beckwith <[hidden email]> wrote:

I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

The plugin will only work with Grails 1.3 and above.

This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

Burt


Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

scryan
In reply to this post by burtbeckwith
<base href="x-msg://48/">Burt,
Great plugin.  Are you using the Liquibase Groovy DSL on Github or did you develop you own?  Is there a standard way to extend this plugin so I can access multiple database using a strategy other than the grails configuration?  We have this capability in the Multi tenant plugin to be able to migrate our databases via that method using the plugin we developed.  I want to avoid duplicate plugins that do the same thing so since our duplicates the one you just released and yours is more official I will migrate ours to leverage anything that yours supports.   The capability we need is to run the upgrade/rollback scripts agains hundreds of databases via a JNDI list managed via the existing Multi Tenant console.

Thanks again for some great work

Scott Ryan


On Jan 9, 2011, at 9:04 PM, Burt Beckwith wrote:

I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

The plugin will only work with Grails 1.3 and above.

This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

Burt

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

burtbeckwith
In reply to this post by Jean Barmash 1

It's not being used yet, but it has solid test coverage and like the Liquibase plugin is primarily a wrapper around Liquibase which is obviously very robust. There certainly may be integration issues though.

Backups and test runs are always a good idea when doing migrations and the same goes for this plugin. I'm planning on keeping the version pre-1.0 until it's been tested by the community and any issues are fixed.

The DSL is very similar to the Autobase DSL, the one significant difference being that scripts are wrapped in a databaseChangeLog = { ... } block in this plugin. I wasn't able to find many examples of Autobase changelogs so there may be others, but both are fairly straightforward Groovy-izations of the Liquibase XML, so conversion should be simple.

Burt

> Congratulations, Burt! Looks amazing. My understanding is that this is an

> initial version for migrations code for future versions of grails, am I

> correct? What would you say the current status of this plugin - alpha,

> beta, or are there production systems out there that already use this? I

> guess trying to gauge how careful I should be trying it out, especially

> knowing that it's built on a pretty solid foundation.

>

> Also, what is its relationship with autobase plugin - was the DSL designed

> to be somewhat compatible?

>

> Thanks,

>

> Jean

>

> On Sun, Jan 9, 2011 at 11:04 PM, Burt Beckwith <[hidden email]>wrote:

>

> > I've released version 0.1 of the Database Migration plugin. This uses

> > Liquibase 2.0 and has all of the features of the current Liquibase plugin

> > and most of the features of the Autobase plugin, in particular support for

> > writing changes in a Groovy DSL. You can also use Liquibase XML.

> >

> > In addition the plugin has GORM integration, so you can make changes in

> > your domain classes and run a script that will generate a changelog based on

> > the diff between your code and any arbitrary database (without having to

> > export the schema to a temp database first).

> >

> > The plugin will only work with Grails 1.3 and above.

> >

> > This is an official Grails plugin, so the code is hosted at

> > https://github.com/grails-plugins/grails-database-migration

> >

> > The standard wiki page is at http://grails.org/plugin/database-migration but

> > the documentation is at

> > http://grails-plugins.github.com/grails-database-migration/

> >

> > Feedback, suggestions for improvement and contributions are always welcome.

> > Please report any bugs or feature requests on the user list or in JIRA at

> > http://jira.codehaus.org/browse/GRAILSPLUGINS under the

> > 'Grails-Database-Migration' component.

> >

> > Burt

> >

>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

burtbeckwith
In reply to this post by scryan

I looked at the the DSL at Github (you're talking about the one that Tim Berglund created, right?) but by that point I had already created my own and it had 100% of the features of the XML file format (since I based the DSL parser on the XML parser) whereas the other DSL was incomplete.

I didn't design the plugin or the DSL with extensions in mind, but I think it'd be great to support the multitenant approach.

Currently you choose which database to run against by running the scripts in the environment that corresponds to that database in DataSource.groovy, e.g. 'grails prod dbm-update', 'grails -Dgrails.env=staging2 dbm-update', etc. But to make things more flexible I'm planning on adding support for specifying arbitrary connection info from the commandline in the next release - would this be sufficient or is the process more involved than that?

Burt

> Burt,

> Great plugin. Are you using the Liquibase Groovy DSL on Github or did you develop you own? Is there a standard way to extend this plugin so I can access multiple database using a strategy other than the grails configuration? We have this capability in the Multi tenant plugin to be able to migrate our databases via that method using the plugin we developed. I want to avoid duplicate plugins that do the same thing so since our duplicates the one you just released and yours is more official I will migrate ours to leverage anything that yours supports. The capability we need is to run the upgrade/rollback scripts agains hundreds of databases via a JNDI list managed via the existing Multi Tenant console.

>

> Thanks again for some great work

>

> Scott Ryan

>

>

> On Jan 9, 2011, at 9:04 PM, Burt Beckwith wrote:

>

> > I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

> > In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

> > The plugin will only work with Grails 1.3 and above.

> >

> > This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

> > The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

> >

> > Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

> >

> > Burt

>

>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

scryan
<base href="x-msg://90/">Great I just wanted to understand which DSL you were using as I have been testing with Tim's version but it was a little incomplete so wanted to make sure I converted over my definitions before I tried your plugin.  Being able to call this plugin with artibtrary connections via JNDI or URL would be all that I would need.  I have build a grails application that allows you to store all the JNDI or JDBC connection strings to access all the tenant databases.  I can then run the liquibase commands against them one by one or can group them by a type like QA, UAT, Production , etc and it returns a PDF or HTML of the success of each run against each database.  I normally run against one database to make sure the conversion works and then hit all my databases in an single environment which is typically 10 -100.  

I love the groovy interaction available with this plugin because many of our interactions are fairly complex and having the ability to use groovy to assist is awesome

Thanks again for great plugin  I can't wait to test it later this week.  Let me know if I can help with the arbitrary data source feature as that I could use day one.  I guess for now I can create a datasource in my grails config for each database and write a script to cycle through them


Scott Ryan


On Jan 10, 2011, at 10:34 AM, Burt Beckwith wrote:

I looked at the the DSL at Github (you're talking about the one that Tim Berglund created, right?) but by that point I had already created my own and it had 100% of the features of the XML file format (since I based the DSL parser on the XML parser) whereas the other DSL was incomplete.

I didn't design the plugin or the DSL with extensions in mind, but I think it'd be great to support the multitenant approach.

Currently you choose which database to run against by running the scripts in the environment that corresponds to that database in DataSource.groovy, e.g. 'grails prod dbm-update', 'grails -Dgrails.env=staging2 dbm-update', etc. But to make things more flexible I'm planning on adding support for specifying arbitrary connection info from the commandline in the next release - would this be sufficient or is the process more involved than that?

Burt

> Burt,
> Great plugin. Are you using the Liquibase Groovy DSL on Github or did you develop you own? Is there a standard way to extend this plugin so I can access multiple database using a strategy other than the grails configuration? We have this capability in the Multi tenant plugin to be able to migrate our databases via that method using the plugin we developed. I want to avoid duplicate plugins that do the same thing so since our duplicates the one you just released and yours is more official I will migrate ours to leverage anything that yours supports. The capability we need is to run the upgrade/rollback scripts agains hundreds of databases via a JNDI list managed via the existing Multi Tenant console.
>
> Thanks again for some great work
>
> Scott Ryan
>
>
> On Jan 9, 2011, at 9:04 PM, Burt Beckwith wrote:
>
> > I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.
> > In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).
> > The plugin will only work with Grails 1.3 and above.
> >
> > This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration
> >
> > Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.
> >
> > Burt
>
>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

Antoine Roux
Hi,
I continued the maintenance of Autobase plug-in as I needed database migrations. Now that there is an official plug-in, I may try to check the compatibility and end the support of Autobase if it has all the features I need (or propose some additions if needed).

Jean: I just read the documentation for now, but it looks very similar. If you already use Autobase, you should probably be able to migrate with only some copy/paste or sed.


Antoine



On Mon, Jan 10, 2011 at 6:48 PM, Scott Ryan <[hidden email]> wrote:
Great I just wanted to understand which DSL you were using as I have been testing with Tim's version but it was a little incomplete so wanted to make sure I converted over my definitions before I tried your plugin.  Being able to call this plugin with artibtrary connections via JNDI or URL would be all that I would need.  I have build a grails application that allows you to store all the JNDI or JDBC connection strings to access all the tenant databases.  I can then run the liquibase commands against them one by one or can group them by a type like QA, UAT, Production , etc and it returns a PDF or HTML of the success of each run against each database.  I normally run against one database to make sure the conversion works and then hit all my databases in an single environment which is typically 10 -100.  

I love the groovy interaction available with this plugin because many of our interactions are fairly complex and having the ability to use groovy to assist is awesome

Thanks again for great plugin  I can't wait to test it later this week.  Let me know if I can help with the arbitrary data source feature as that I could use day one.  I guess for now I can create a datasource in my grails config for each database and write a script to cycle through them


Scott Ryan


On Jan 10, 2011, at 10:34 AM, Burt Beckwith wrote:

I looked at the the DSL at Github (you're talking about the one that Tim Berglund created, right?) but by that point I had already created my own and it had 100% of the features of the XML file format (since I based the DSL parser on the XML parser) whereas the other DSL was incomplete.

I didn't design the plugin or the DSL with extensions in mind, but I think it'd be great to support the multitenant approach.

Currently you choose which database to run against by running the scripts in the environment that corresponds to that database in DataSource.groovy, e.g. 'grails prod dbm-update', 'grails -Dgrails.env=staging2 dbm-update', etc. But to make things more flexible I'm planning on adding support for specifying arbitrary connection info from the commandline in the next release - would this be sufficient or is the process more involved than that?

Burt

> Burt,
> Great plugin. Are you using the Liquibase Groovy DSL on Github or did you develop you own? Is there a standard way to extend this plugin so I can access multiple database using a strategy other than the grails configuration? We have this capability in the Multi tenant plugin to be able to migrate our databases via that method using the plugin we developed. I want to avoid duplicate plugins that do the same thing so since our duplicates the one you just released and yours is more official I will migrate ours to leverage anything that yours supports. The capability we need is to run the upgrade/rollback scripts agains hundreds of databases via a JNDI list managed via the existing Multi Tenant console.
>
> Thanks again for some great work
>
> Scott Ryan
>
>
> On Jan 9, 2011, at 9:04 PM, Burt Beckwith wrote:
>
> > I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.
> > In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).
> > The plugin will only work with Grails 1.3 and above.
> >
> > This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration
> >
> > Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.
> >
> > Burt
>
>


Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

jerrydboonstra
In reply to this post by burtbeckwith
<base href="x-msg://77/">
Great plugin, I was previously using liquibase 2.0.1-SNAPSHOT on the command line, using an extra db in my development environment in order to perform diffs -- that is what I could get working.

One problem, under Grails 1.3.6:
I'm using a root changelog with includes and I'm getting these errors at "run-app" time, one error per file in grails-app/conf/migrations:

ERROR org.codehaus.groovy.grails.commons.DefaultGrailsApplication  - The class         [migrations.baseChangeLog] was not found when attempting to load Grails application. Skipping.

("baseChangeLog.groovy" is the file for the example above).

Both inner and root changelog files have an outer "databaseChangeLog = {" closure (not sure if that is valid or not as I can't find a DSL reference).

A workaround for me is to relocate the migrations directory from 'grails-app/conf/migrations' to 'grails-app/migrations', and set this in Config.groovy:

grails.plugin.databasemigration.changelogLocation='grails-app/migrations'

Is this a bug, or something specific to 1.3.6, or? It appears that Grails is attempting to parse those files as class files.  I'm happy to file a ticket.

j e r r y

On Jan 9, 2011, at 8:04 PM PST, Burt Beckwith wrote:

I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

The plugin will only work with Grails 1.3 and above.

This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

Burt

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

burtbeckwith

Sorry about that, I forgot to add a note about this in the docs. This error should be ignored and doesn't appear in 1.4. The issue is that the .groovy files are discovered as resources since they're under grails-app but they're not compiled, so Grails complains when it can't find the .class file. The plugin parses the Groovy files directly so this isn't a problem.

You can relocate the file location but then the files won't be automatically added to the classpath, and autorun from a war file will fail. If that's not something you need (i.e. if you explicitly run migrations while deploying) then that won't affect you.

Burt

> Great plugin, I was previously using liquibase 2.0.1-SNAPSHOT on the command line, using an extra db in my development environment in order to perform diffs -- that is what I could get working.

>

> One problem, under Grails 1.3.6:

> I'm using a root changelog with includes and I'm getting these errors at "run-app" time, one error per file in grails-app/conf/migrations:

>

> ERROR org.codehaus.groovy.grails.commons.DefaultGrailsApplication - The class [migrations.baseChangeLog] was not found when attempting to load Grails application. Skipping.

>

> ("baseChangeLog.groovy" is the file for the example above).

>

> Both inner and root changelog files have an outer "databaseChangeLog = {" closure (not sure if that is valid or not as I can't find a DSL reference).

>

> A workaround for me is to relocate the migrations directory from 'grails-app/conf/migrations' to 'grails-app/migrations', and set this in Config.groovy:

>

> grails.plugin.databasemigration.changelogLocation='grails-app/migrations'

>

> Is this a bug, or something specific to 1.3.6, or? It appears that Grails is attempting to parse those files as class files. I'm happy to file a ticket.

>

> j e r r y

>

> On Jan 9, 2011, at 8:04 PM PST, Burt Beckwith wrote:

>

> > I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

> > In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

> > The plugin will only work with Grails 1.3 and above.

> >

> > This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

> > The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

> >

> > Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

> >

> > Burt

>

>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

jerrydboonstra
<base href="x-msg://86/">
This error should be ignored and doesn't appear in 1.4

Ok, cool, glad to know that its a spurious message that will go away.

One more problem.  I'm attempting to perform a "custom sql" change set, but the change fails silently and the change gets recorded in 'databasechangelog' table as successfully applied (I'm using Postgres 8.4).

Here is the included changeset file:

databaseChangeLog = {

    // This change does not get applied for some reason.
    // This change will fail silently if these fail (say, for permissions reasons).
    // They still fail on a database that we have proper permissions for, and they can be
    // run via PSQL
    changeSet(author: "jerry", id: "1294692516740-1") {
        sql(splitStatements: "false") {'''
            SELECT AddGeometryColumn('itagstuffdev', 'map_point', 'coords', 4326, 'POINT', 2);
            ALTER TABLE itagstuffdev.map_point ALTER COLUMN coords SET NOT NULL;
            CREATE INDEX map_point_coords_idx ON itagstuffdev.map_point USING GIST (coords);
            CLUSTER map_point using map_point_coords_idx;
            VACUUM ANALYZE itagstuffdev.map_point (coords);
            '''
        }
    }
}

At first I had permissions issues which were causing the silent failure, which I troubleshot by applying the SQL by hand via psql.  Those errors should have terminate the changeset application.

When I resolved that and ran again, I would still find that the changes were not successfully applied but were recorded as such in the database.  Liquibase or databasemigration DEBUG logs show those changesets run successfully, but the actual custom SQL statements appear as blank in the log message.

I'd expect that any underlying exceptions thrown would result in a failed changeset application, but not in this case.

Any pointers here?  I'm thinking next of trying to view the wire protocol to see if the actual statements are being transmitted.

j e r r y

On Jan 10, 2011, at 7:25 PM PST, Burt Beckwith wrote:

Sorry about that, I forgot to add a note about this in the docs. This error should be ignored and doesn't appear in 1.4. The issue is that the .groovy files are discovered as resources since they're under grails-app but they're not compiled, so Grails complains when it can't find the .class file. The plugin parses the Groovy files directly so this isn't a problem.

You can relocate the file location but then the files won't be automatically added to the classpath, and autorun from a war file will fail. If that's not something you need (i.e. if you explicitly run migrations while deploying) then that won't affect you.

Burt

> Great plugin, I was previously using liquibase 2.0.1-SNAPSHOT on the command line, using an extra db in my development environment in order to perform diffs -- that is what I could get working.
>
> One problem, under Grails 1.3.6:
> I'm using a root changelog with includes and I'm getting these errors at "run-app" time, one error per file in grails-app/conf/migrations:
>
> ERROR org.codehaus.groovy.grails.commons.DefaultGrailsApplication - The class [migrations.baseChangeLog] was not found when attempting to load Grails application. Skipping.
>
> ("baseChangeLog.groovy" is the file for the example above).
>
> Both inner and root changelog files have an outer "databaseChangeLog = {" closure (not sure if that is valid or not as I can't find a DSL reference).
>
> A workaround for me is to relocate the migrations directory from 'grails-app/conf/migrations' to 'grails-app/migrations', and set this in Config.groovy:
>
> grails.plugin.databasemigration.changelogLocation='grails-app/migrations'
>
> Is this a bug, or something specific to 1.3.6, or? It appears that Grails is attempting to parse those files as class files. I'm happy to file a ticket.
>
> j e r r y
>
> On Jan 9, 2011, at 8:04 PM PST, Burt Beckwith wrote:
>
> > I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.
> > In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).
> > The plugin will only work with Grails 1.3 and above.
> >
> > This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration
> >
> > Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.
> >
> > Burt
>
>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

burtbeckwith
In reply to this post by Antoine Roux

I've included the features of Autobase that I thought made sense. I borrowed code from the liquibase-runner plugin to support running migrations on start, but it's not the default behavior - you have to configure it. I also looked at making migrations artifacts like you did for the most recent release but didn't include it since I didn't see the benefit, and I don't see how it would work if you mix XML and Groovy files (for ordering anyway).

Competition is good but I'd prefer to collaborate so there aren't three plugins all based on Liquibase.

I'm planning on adding a section to the plugin docs on converting from the current Liquibase and Autobase plugins to this one - there shouldn't be much work involved since they're all quite similar.

Burt

> Hi,

> I continued the maintenance of Autobase plug-in as I needed database

> migrations. Now that there is an official plug-in, I may try to check the

> compatibility and end the support of Autobase if it has all the features I

> need (or propose some additions if needed).

>

> Jean: I just read the documentation for now, but it looks very similar. If

> you already use Autobase, you should probably be able to migrate with only

> some copy/paste or sed.

>

>

> Antoine

>

>

>

> On Mon, Jan 10, 2011 at 6:48 PM, Scott Ryan <[hidden email]> wrote:

>

> > Great I just wanted to understand which DSL you were using as I have been

> > testing with Tim's version but it was a little incomplete so wanted to make

> > sure I converted over my definitions before I tried your plugin. Being able

> > to call this plugin with artibtrary connections via JNDI or URL would be all

> > that I would need. I have build a grails application that allows you to

> > store all the JNDI or JDBC connection strings to access all the tenant

> > databases. I can then run the liquibase commands against them one by one or

> > can group them by a type like QA, UAT, Production , etc and it returns a PDF

> > or HTML of the success of each run against each database. I normally run

> > against one database to make sure the conversion works and then hit all my

> > databases in an single environment which is typically 10 -100.

> >

> > I love the groovy interaction available with this plugin because many of

> > our interactions are fairly complex and having the ability to use groovy to

> > assist is awesome

> >

> > Thanks again for great plugin I can't wait to test it later this week.

> > Let me know if I can help with the arbitrary data source feature as that I

> > could use day one. I guess for now I can create a datasource in my grails

> > config for each database and write a script to cycle through them

> >

> >

> > Scott Ryan

> >

> >

> > On Jan 10, 2011, at 10:34 AM, Burt Beckwith wrote:

> >

> > I looked at the the DSL at Github (you're talking about the one that Tim

> > Berglund created, right?) but by that point I had already created my own and

> > it had 100% of the features of the XML file format (since I based the DSL

> > parser on the XML parser) whereas the other DSL was incomplete.

> >

> > I didn't design the plugin or the DSL with extensions in mind, but I think

> > it'd be great to support the multitenant approach.

> >

> > Currently you choose which database to run against by running the scripts

> > in the environment that corresponds to that database in DataSource.groovy,

> > e.g. 'grails prod dbm-update', 'grails -Dgrails.env=staging2 dbm-update',

> > etc. But to make things more flexible I'm planning on adding support for

> > specifying arbitrary connection info from the commandline in the next

> > release - would this be sufficient or is the process more involved than

> > that?

> >

> > Burt

> >

> > > Burt,

> > > Great plugin. Are you using the Liquibase Groovy DSL on Github or did you

> > develop you own? Is there a standard way to extend this plugin so I can

> > access multiple database using a strategy other than the grails

> > configuration? We have this capability in the Multi tenant plugin to be able

> > to migrate our databases via that method using the plugin we developed. I

> > want to avoid duplicate plugins that do the same thing so since our

> > duplicates the one you just released and yours is more official I will

> > migrate ours to leverage anything that yours supports. The capability we

> > need is to run the upgrade/rollback scripts agains hundreds of databases via

> > a JNDI list managed via the existing Multi Tenant console.

> > >

> > > Thanks again for some great work

> > >

> > > Scott Ryan

> > >

> > >

> > > On Jan 9, 2011, at 9:04 PM, Burt Beckwith wrote:

> > >

> > > > I've released version 0.1 of the Database Migration plugin. This uses

> > Liquibase 2.0 and has all of the features of the current Liquibase plugin

> > and most of the features of the Autobase plugin, in particular support for

> > writing changes in a Groovy DSL. You can also use Liquibase XML.

> > > > In addition the plugin has GORM integration, so you can make changes in

> > your domain classes and run a script that will generate a changelog based on

> > the diff between your code and any arbitrary database (without having to

> > export the schema to a temp database first).

> > > > The plugin will only work with Grails 1.3 and above.

> > > >

> > > > This is an official Grails plugin, so the code is hosted at

> > https://github.com/grails-plugins/grails-database-migration

> > > > The standard wiki page is at

> > http://grails.org/plugin/database-migration but the documentation is at

> > http://grails-plugins.github.com/grails-database-migration/

> > > >

> > > > Feedback, suggestions for improvement and contributions are always

> > welcome. Please report any bugs or feature requests on the user list or in

> > JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the

> > 'Grails-Database-Migration' component.

> > > >

> > > > Burt

> > >

> > >

> >

> >

> >

>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

burtbeckwith
In reply to this post by jerrydboonstra

The SQL string shouldn't be in a closure, it should be the last parameter, i.e.

sql(splitStatements: "false", '''

SELECT AddGeometryColumn('itagstuffdev', 'map_point', 'coords', 4326, 'POINT', 2);

ALTER TABLE itagstuffdev.map_point ALTER COLUMN coords SET NOT NULL;

CREATE INDEX map_point_coords_idx ON itagstuffdev.map_point USING GIST (coords);

CLUSTER map_point using map_point_coords_idx;

VACUUM ANALYZE itagstuffdev.map_point (coords);

'''

)

Burt

> > This error should be ignored and doesn't appear in 1.4

>

> Ok, cool, glad to know that its a spurious message that will go away.

>

> One more problem. I'm attempting to perform a "custom sql" change set, but the change fails silently and the change gets recorded in 'databasechangelog' table as successfully applied (I'm using Postgres 8.4).

>

> Here is the included changeset file:

>

> databaseChangeLog = {

>

> // This change does not get applied for some reason.

> // This change will fail silently if these fail (say, for permissions reasons).

> // They still fail on a database that we have proper permissions for, and they can be

> // run via PSQL

> changeSet(author: "jerry", id: "1294692516740-1") {

> sql(splitStatements: "false") {'''

> SELECT AddGeometryColumn('itagstuffdev', 'map_point', 'coords', 4326, 'POINT', 2);

> ALTER TABLE itagstuffdev.map_point ALTER COLUMN coords SET NOT NULL;

> CREATE INDEX map_point_coords_idx ON itagstuffdev.map_point USING GIST (coords);

> CLUSTER map_point using map_point_coords_idx;

> VACUUM ANALYZE itagstuffdev.map_point (coords);

> '''

> }

> }

> }

>

> At first I had permissions issues which were causing the silent failure, which I troubleshot by applying the SQL by hand via psql. Those errors should have terminate the changeset application.

>

> When I resolved that and ran again, I would still find that the changes were not successfully applied but were recorded as such in the database. Liquibase or databasemigration DEBUG logs show those changesets run successfully, but the actual custom SQL statements appear as blank in the log message.

>

> I'd expect that any underlying exceptions thrown would result in a failed changeset application, but not in this case.

>

> Any pointers here? I'm thinking next of trying to view the wire protocol to see if the actual statements are being transmitted.

>

> j e r r y

>

> On Jan 10, 2011, at 7:25 PM PST, Burt Beckwith wrote:

>

> > Sorry about that, I forgot to add a note about this in the docs. This error should be ignored and doesn't appear in 1.4. The issue is that the .groovy files are discovered as resources since they're under grails-app but they're not compiled, so Grails complains when it can't find the .class file. The plugin parses the Groovy files directly so this isn't a problem.

> > You can relocate the file location but then the files won't be automatically added to the classpath, and autorun from a war file will fail. If that's not something you need (i.e. if you explicitly run migrations while deploying) then that won't affect you.

> > Burt

> > > Great plugin, I was previously using liquibase 2.0.1-SNAPSHOT on the command line, using an extra db in my development environment in order to perform diffs -- that is what I could get working.

> > >

> > > One problem, under Grails 1.3.6:

> > > I'm using a root changelog with includes and I'm getting these errors at "run-app" time, one error per file in grails-app/conf/migrations:

> > >

> > > ERROR org.codehaus.groovy.grails.commons.DefaultGrailsApplication - The class [migrations.baseChangeLog] was not found when attempting to load Grails application. Skipping.

> > >

> > > ("baseChangeLog.groovy" is the file for the example above).

> > >

> > > Both inner and root changelog files have an outer "databaseChangeLog = {" closure (not sure if that is valid or not as I can't find a DSL reference).

> > >

> > > A workaround for me is to relocate the migrations directory from 'grails-app/conf/migrations' to 'grails-app/migrations', and set this in Config.groovy:

> > >

> > > grails.plugin.databasemigration.changelogLocation='grails-app/migrations'

> > >

> > > Is this a bug, or something specific to 1.3.6, or? It appears that Grails is attempting to parse those files as class files. I'm happy to file a ticket.

> > >

> > > j e r r y

> > >

> > > On Jan 9, 2011, at 8:04 PM PST, Burt Beckwith wrote:

> > >

> > > > I've released version 0.1 of the Database Migration plugin. This uses Liquibase 2.0 and has all of the features of the current Liquibase plugin and most of the features of the Autobase plugin, in particular support for writing changes in a Groovy DSL. You can also use Liquibase XML.

> > > > In addition the plugin has GORM integration, so you can make changes in your domain classes and run a script that will generate a changelog based on the diff between your code and any arbitrary database (without having to export the schema to a temp database first).

> > > > The plugin will only work with Grails 1.3 and above.

> > > >

> > > > This is an official Grails plugin, so the code is hosted at https://github.com/grails-plugins/grails-database-migration

> > > > The standard wiki page is at http://grails.org/plugin/database-migration but the documentation is at http://grails-plugins.github.com/grails-database-migration/

> > > >

> > > > Feedback, suggestions for improvement and contributions are always welcome. Please report any bugs or feature requests on the user list or in JIRA at http://jira.codehaus.org/browse/GRAILSPLUGINS under the 'Grails-Database-Migration' component.

> > > >

> > > > Burt

> > >

> > >

>

>

Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

spidie
In reply to this post by burtbeckwith
Hi Burt

Great plugin - it's looking like just what we need. Works fine for me
(thanks for the help you gave Lee with it) - but it doesn't seem to
run for me in the test environment (ie. grails test dbm-update or
similar).

Any ideas? There's nothing in the logs (apart from the one to do with
class files mentioned in this thread), I'm on grails 1.3.6 (ubuntu
10.10)

Regards
Steve

On Mon, Jan 10, 2011 at 2:04 PM, Burt Beckwith <[hidden email]> wrote:

> I've released version 0.1 of the Database Migration plugin. This uses
> Liquibase 2.0 and has all of the features of the current Liquibase plugin
> and most of the features of the Autobase plugin, in particular support for
> writing changes in a Groovy DSL. You can also use Liquibase XML.
>
> In addition the plugin has GORM integration, so you can make changes in your
> domain classes and run a script that will generate a changelog based on the
> diff between your code and any arbitrary database (without having to export
> the schema to a temp database first).
>
> The plugin will only work with Grails 1.3 and above.
>
> This is an official Grails plugin, so the code is hosted at
> https://github.com/grails-plugins/grails-database-migration
>
> The standard wiki page is at http://grails.org/plugin/database-migration but
> the documentation is at
> http://grails-plugins.github.com/grails-database-migration/
>
> Feedback, suggestions for improvement and contributions are always welcome.
> Please report any bugs or feature requests on the user list or in JIRA at
> http://jira.codehaus.org/browse/GRAILSPLUGINS under the
> 'Grails-Database-Migration' component.
>
> Burt
>



--
Refactor
Engage. Succeed. Repeat.
PO Box 802, Labrador, Q 4215, Australia
tel: +61 (0)7 5668 3424 web: refactor.com.au

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

Sebastien ARBOGAST
It's going to be a very useful plugin but right now I'm still
struggling with my Oracle database: when I generate the initial
changelog using dbm-generate-gorm-changelog, it does not create the
HIBERNATE_SEQUENCE that is created by hbm2ddl in create or create-drop
mode.

---
Sébastien Arbogast
http://about.me/sebastienarbogast/bio




2011/1/11 Steve Dalton <[hidden email]>:

> Hi Burt
>
> Great plugin - it's looking like just what we need. Works fine for me
> (thanks for the help you gave Lee with it) - but it doesn't seem to
> run for me in the test environment (ie. grails test dbm-update or
> similar).
>
> Any ideas? There's nothing in the logs (apart from the one to do with
> class files mentioned in this thread), I'm on grails 1.3.6 (ubuntu
> 10.10)
>
> Regards
> Steve
>
> On Mon, Jan 10, 2011 at 2:04 PM, Burt Beckwith <[hidden email]> wrote:
>> I've released version 0.1 of the Database Migration plugin. This uses
>> Liquibase 2.0 and has all of the features of the current Liquibase plugin
>> and most of the features of the Autobase plugin, in particular support for
>> writing changes in a Groovy DSL. You can also use Liquibase XML.
>>
>> In addition the plugin has GORM integration, so you can make changes in your
>> domain classes and run a script that will generate a changelog based on the
>> diff between your code and any arbitrary database (without having to export
>> the schema to a temp database first).
>>
>> The plugin will only work with Grails 1.3 and above.
>>
>> This is an official Grails plugin, so the code is hosted at
>> https://github.com/grails-plugins/grails-database-migration
>>
>> The standard wiki page is at http://grails.org/plugin/database-migration but
>> the documentation is at
>> http://grails-plugins.github.com/grails-database-migration/
>>
>> Feedback, suggestions for improvement and contributions are always welcome.
>> Please report any bugs or feature requests on the user list or in JIRA at
>> http://jira.codehaus.org/browse/GRAILSPLUGINS under the
>> 'Grails-Database-Migration' component.
>>
>> Burt
>>
>
>
>
> --
> Refactor
> Engage. Succeed. Repeat.
> PO Box 802, Labrador, Q 4215, Australia
> tel: +61 (0)7 5668 3424 web: refactor.com.au
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: ANN: Database Migration Plugin v0.1 released

Brigette
In reply to this post by burtbeckwith
Folks can also take a look at Tomas Lin's blog post on this here: http://fbflex.wordpress.com/2011/01/19/working-with-the-grails-database-migration-plugin/ 
I found this very helpful.

I've also found issues with the plugin not using the hibernate dialect wrt creation of proper sequences when using dbm-gorm-diff. I thikn for now we are going to use Tomas' approach, using dbm-diff.

An issue I have found with dbm-diff generated changsets is that it records the schema. For constraint and sequence creation, when the dbm-update will be applied across multiple databases (eg dev, test, uat, and one day production) having the incorrect (for the database/environment) schema in the changeset breaks the  dbm-update (table does not exist etc). Current workaround is to remove the schema references for these types of changeset. I added the --default-schema, but this didn't help, and running dbm-update without --default-schema, and without the schema settings in the changesets works fine.

I've played with the dbm-tag and dbm-rollback feature- this functionality seems to work very smoothly.

Do we have a planned release date for 0.2?

Brigette