Liquibase usage and alternatives
Locked
 
|
This question is directed at Nathan or anybody who uses Liquibase.
I've been evaluating different strategies for schema migration (side note: this is an area that is a bit thin in the Grails docs and feature-set since GORM/Hibernate update is too naive to work in anything other than home projects - but I digress). Seems the balance leans towards Liquibase as opposed to dbmigrate, so having looked at Liquibase, I see how it caters for structural changes, but in my previous experience there are times where one needs to move data to, say, a temporary table in order to effectively alter a table and then move the data back in. How does Liquibase deal with this? Does it deal with this? I can't seem to figure out how. Certainly you can't do it 'scripting' via XML. An example is: A [1:M] B [1:M] C ... A has 1 to many rel with B, which has 1 to many with C. Now in my current project, I actually want to remove B out of the equation since the business rules have changed, so now I need to have: A [1:M] C ... but to do this I'll need to move the data out of C into a temporary table, alter C to have its foreign key reference A, and then re-insert the data into C with appropriate values for that foreign key. Any idea how that'll be done with Liquibase? I'm assuming the answer is "you can't" but just confirming. If so, surely this is a potential scenario in any meaningful business application. How do you guys deal with this in a way that doesn't require a lot of manual intervention from version to version upgrade? Thanks, Darryl |
Re: Liquibase usage and alternatives
|
If you can do everything you need to do for your complex migration in
straight SQL, then you can use the "custom SQL" or "custom SQL file" refactorings that liquibase offers. See the "Custom Refactorings" section here: http://www.liquibase.org/manual/home Liquibase (and the Grails plugin for it) has worked well for the other migrations I've needed to do (which admittedly have been fairly straight forward). On May 21, 2008, at 10:47 AM, Darryl Pentz wrote: > > This question is directed at Nathan or anybody who uses Liquibase. > > I've been evaluating different strategies for schema migration (side > note: > this is an area that is a bit thin in the Grails docs and feature- > set since > GORM/Hibernate update is too naive to work in anything other than home > projects - but I digress). > > Seems the balance leans towards Liquibase as opposed to dbmigrate, > so having > looked at Liquibase, I see how it caters for structural changes, but > in my > previous experience there are times where one needs to move data to, > say, a > temporary table in order to effectively alter a table and then move > the data > back in. How does Liquibase deal with this? Does it deal with this? > I can't > seem to figure out how. Certainly you can't do it 'scripting' via XML. > > An example is: > > A [1:M] B [1:M] C ... A has 1 to many rel with B, which has 1 to > many with > C. Now in my current project, I actually want to remove B out of the > equation since the business rules have changed, so now I need to have: > > A [1:M] C ... but to do this I'll need to move the data out of C > into a > temporary table, alter C to have its foreign key reference A, and then > re-insert the data into C with appropriate values for that foreign > key. Any > idea how that'll be done with Liquibase? I'm assuming the answer is > "you > can't" but just confirming. > > If so, surely this is a potential scenario in any meaningful business > application. How do you guys deal with this in a way that doesn't > require a > lot of manual intervention from version to version upgrade? > > Thanks, > Darryl > -- > View this message in context: http://www.nabble.com/Liquibase-usage-and-alternatives-tp17364331p17364331.html > Sent from the grails - user mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
|
In reply to this post by jondo_w
Hi Mike,
So how do you work - what's your process? Do you allow GORM to update the database in development mode and then diff those changes into your change log? Also, how exactly do you use Liquibase as a plugin in a production app? It doesn't seem to be something that can be invoked as part of your apps startup? I'm not having much joy with Liquibase to be honest. I created an initial change log, then reran Grails with some changes on a fresh db and diffed that with the previous version which was appended to my changelog.xml. But when I ran liquibase command line to update the db, it tried to rerun from the first changeset which obviously breaks because those tables already exist. It was meant to pick up the trail from ~ change set # -87 or so. Dunno. I'm not feeling that bullish about Liquibase in a meaningful business app at this time. :-( - DP ----- Original Message ---- From: Mike Hugo <[hidden email]> To: [hidden email] Sent: Wednesday, May 21, 2008 7:59:29 PM Subject: Re: [grails-user] Liquibase usage and alternatives If you can do everything you need to do for your complex migration in straight SQL, then you can use the "custom SQL" or "custom SQL file" refactorings that liquibase offers. See the "Custom Refactorings" section here: http://www.liquibase.org/manual/home Liquibase (and the Grails plugin for it) has worked well for the other migrations I've needed to do (which admittedly have been fairly straight forward). On May 21, 2008, at 10:47 AM, Darryl Pentz wrote: > > This question is directed at Nathan or anybody who uses Liquibase. > > I've been evaluating different strategies for schema migration (side > note: > this is an area that is a bit thin in the Grails docs and feature- > set since > GORM/Hibernate update is too naive to work in anything other than home > projects - but I digress). > > Seems the balance leans towards Liquibase as opposed to dbmigrate, > so having > looked at Liquibase, I see how it caters for structural changes, but > in my > previous experience there are times where one needs to move data to, > say, a > temporary table in order to effectively alter a table and then move > the data > back in. How does Liquibase deal with this? Does it deal with this? > I can't > seem to figure out how. Certainly you can't do it 'scripting' via XML. > > An example is: > > A [1:M] B [1:M] C ... A has 1 to many rel with B, which has 1 to > many with > C. Now in my current project, I actually want to remove B out of the > equation since the business rules have changed, so now I need to have: > > A [1:M] C ... but to do this I'll need to move the data out of C > into a > temporary table, alter C to have its foreign key reference A, and then > re-insert the data into C with appropriate values for that foreign > key. Any > idea how that'll be done with Liquibase? I'm assuming the answer is > "you > can't" but just confirming. > > If so, surely this is a potential scenario in any meaningful business > application. How do you guys deal with this in a way that doesn't > require a > lot of manual intervention from version to version upgrade? > > Thanks, > Darryl > -- > View this message in context: http://www.nabble.com/Liquibase-usage-and-alternatives-tp17364331p17364331.html > Sent from the grails - user mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
Re: Liquibase usage and alternatives
|
|
Hi Darryl,
I have dbCreate commented out in my datasource.groovy and have been manually keeping the change log up to date with changes I make to the domain model. True, it's a two step process, but at least I know exactly what's going on under the covers. Personally I prefer that liquibase NOT automatically get invoked as part of application startup - instead I run migrate as part of the deployment process. Most places I've worked have a DBA do the database changes separately from the engineer who does the code deployment, so this actually works out OK. Mike On May 21, 2008, at 3:13 PM, Darryl Pentz wrote: > Hi Mike, > > So how do you work - what's your process? Do you allow GORM to > update the database in development mode and then diff those changes > into your change log? Also, how exactly do you use Liquibase as a > plugin in a production app? It doesn't seem to be something that can > be invoked as part of your apps startup? > > I'm not having much joy with Liquibase to be honest. I created an > initial change log, then reran Grails with some changes on a fresh > db and diffed that with the previous version which was appended to > my changelog.xml. But when I ran liquibase command line to update > the db, it tried to rerun from the first changeset which obviously > breaks because those tables already exist. It was meant to pick up > the trail from ~ change set # -87 or so. Dunno. I'm not feeling that > bullish about Liquibase in a meaningful business app at this time. :-( > > - DP > > ----- Original Message ---- > From: Mike Hugo <[hidden email]> > To: [hidden email] > Sent: Wednesday, May 21, 2008 7:59:29 PM > Subject: Re: [grails-user] Liquibase usage and alternatives > > If you can do everything you need to do for your complex migration in > straight SQL, then you can use the "custom SQL" or "custom SQL file" > refactorings that liquibase offers. See the "Custom Refactorings" > section here: http://www.liquibase.org/manual/home > > Liquibase (and the Grails plugin for it) has worked well for the other > migrations I've needed to do (which admittedly have been fairly > straight forward). > > On May 21, 2008, at 10:47 AM, Darryl Pentz wrote: > >> >> This question is directed at Nathan or anybody who uses Liquibase. >> >> I've been evaluating different strategies for schema migration (side >> note: >> this is an area that is a bit thin in the Grails docs and feature- >> set since >> GORM/Hibernate update is too naive to work in anything other than >> home >> projects - but I digress). >> >> Seems the balance leans towards Liquibase as opposed to dbmigrate, >> so having >> looked at Liquibase, I see how it caters for structural changes, but >> in my >> previous experience there are times where one needs to move data to, >> say, a >> temporary table in order to effectively alter a table and then move >> the data >> back in. How does Liquibase deal with this? Does it deal with this? >> I can't >> seem to figure out how. Certainly you can't do it 'scripting' via >> XML. >> >> An example is: >> >> A [1:M] B [1:M] C ... A has 1 to many rel with B, which has 1 to >> many with >> C. Now in my current project, I actually want to remove B out of the >> equation since the business rules have changed, so now I need to >> have: >> >> A [1:M] C ... but to do this I'll need to move the data out of C >> into a >> temporary table, alter C to have its foreign key reference A, and >> then >> re-insert the data into C with appropriate values for that foreign >> key. Any >> idea how that'll be done with Liquibase? I'm assuming the answer is >> "you >> can't" but just confirming. >> >> If so, surely this is a potential scenario in any meaningful business >> application. How do you guys deal with this in a way that doesn't >> require a >> lot of manual intervention from version to version upgrade? >> >> Thanks, >> Darryl >> -- >> View this message in context: http://www.nabble.com/Liquibase-usage-and-alternatives-tp17364331p17364331.html >> Sent from the grails - user mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
Hey Guys,
I'm not sure how closely Nathan follows this list, so if you have specific questions about functionality of Liquibase, your best bet is probably the Liquibase users list. Also, Mike is right about needing to comment out/remove the dbCreate property from DataSource.groovy. This will stop Hibernate from automatically trying to create/update the schema. You then need to define the schema in the Liquibase changelog.xml file (located in grails-app/migrations/changelog.xml). You can initially create this file using the generate-changelog script. If you do this against a database that's already populated, you'll need to run the changelog-sync script to tell Liquibase that all changes have been applied. After you have your changelog and a development/test database, your development cycle will look something like this: - Update your domain model in GORM and/or Java mapped classes (annotated or with hbm.xml files). - Add to your changelog.xml file the equivalent DB migrations - (optional) Take a peak at what Liquibase will generate with the migrate-sql script. - (optional) If all looks good, tag the state of your database prior to the migration with the tag script - Do the migration with the migrate script - If there's a problem with migration use one of the rollback scripts (i.e. rollback, rollback-count, rollback-to-date) - If all is well, then check in your changelog.xml file tell the rest of you dev team (if any) that a migrate will be required when they update (i.e. via svn, etc). Finally, you can add a little bit of code into Bootstrap.groovy to run the migration if the environment is production. This uses the Liquibase API directly and looks something like this: def init = { servletContext -> if (GrailsUtil.environment == 'production') { Liquibase liquibase = null try { def c = dataSource.getConnection() if (c == null) { throw new RuntimeException("Connection could not be created."); } def fileOpener = new ClassLoaderFileOpener() def database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(c) database.setDefaultSchemaName(c.catalog) liquibase = new Liquibase("changelog.xml", fileOpener, database); liquibase.update(null) } finally { if (liquibase && liquibase.database) { liquibase.database.close() } } } } Anyway, that's how I've setup Liquibase in our project and it has worked pretty well so far. It's much better than the alternative of using hbm2ddl, setting the dbCreate property to update and crossing your fingers ;^) Dave -- On Wed, May 21, 2008 at 5:02 PM, Mike Hugo <[hidden email]> wrote: Hi Darryl, |
|
|
In reply to this post by jondo_w
David, I have to say that your post is *the* most useful thing I've read about Liquibase to date. This really should be added to the Liquibase plugin docs/wiki page. Thank you very much because this is really what I've been looking for - it answered a few aspects of Liquibase use I just wasn't getting from the docs. The Liquibase guys would do well to include this in their manual, or at least in the Grails plugin portion. So, thank you *very* much. - Darryl ----- Original Message ---- From: David White <[hidden email]> To: [hidden email] Sent: Thursday, May 22, 2008 3:19:45 AM Subject: Re: [grails-user] Liquibase usage and alternatives Hey Guys, I'm not sure how closely Nathan follows this list, so if you have specific questions about functionality of Liquibase, your best bet is probably the Liquibase users list. Also, Mike is right about needing to comment out/remove the dbCreate property from DataSource.groovy. This will stop Hibernate from automatically trying to create/update the schema. You then need to define the schema in the Liquibase changelog.xml file (located in grails-app/migrations/changelog.xml). You can initially create this file using the generate-changelog script. If you do this against a database that's already populated, you'll need to run the changelog-sync script to tell Liquibase that all changes have been applied. After you have your changelog and a development/test database, your development cycle will look something like this: - Update your domain model in GORM and/or Java mapped classes (annotated or with hbm.xml files). - Add to your changelog.xml file the equivalent DB migrations - (optional) Take a peak at what Liquibase will generate with the migrate-sql script. - (optional) If all looks good, tag the state of your database prior to the migration with the tag script - Do the migration with the migrate script - If there's a problem with migration use one of the rollback scripts (i.e. rollback, rollback-count, rollback-to-date) - If all is well, then check in your changelog.xml file tell the rest of you dev team (if any) that a migrate will be required when they update (i.e. via svn, etc). Finally, you can add a little bit of code into Bootstrap.groovy to run the migration if the environment is production. This uses the Liquibase API directly and looks something like this: def init = { servletContext -> if (GrailsUtil.environment == 'production') { Liquibase liquibase = null try { def c = dataSource.getConnection() if (c == null) { throw new RuntimeException("Connection could not be created."); } def fileOpener = new ClassLoaderFileOpener() def database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(c) database.setDefaultSchemaName(c.catalog) liquibase = new Liquibase("changelog.xml", fileOpener, database); liquibase.update(null) } finally { if (liquibase && liquibase.database) { liquibase.database.close() } } } } Anyway, that's how I've setup Liquibase in our project and it has worked pretty well so far. It's much better than the alternative of using hbm2ddl, setting the dbCreate property to update and crossing your fingers ;^) Dave -- On Wed, May 21, 2008 at 5:02 PM, Mike Hugo <[hidden email]> wrote: Hi Darryl, |
|
|
No problem. Yeah, something like this should be on the Liquibase Grails plugin page. Basically a "best practice" for using Liquibase with Grails. The lack of this was what stumped me for a while as well when I was trying to get it to work in our environment.
Dave -- On Thu, May 22, 2008 at 2:40 AM, Darryl Pentz <[hidden email]> wrote:
|
Re: Liquibase usage and alternatives
|
In reply to this post by jondo_w
I agree that that was a very useful message. I've been trying to follow the grails-user list for LiquiBase related items to reply to, but since I don't do a ton of grails-related dev I'm not as helpful as others can be.
If you or anyone else ever doesn't get the info you are looking for on the grails-user list, feel free to post to the liquibase-user list as well. I am better at keeping up with that one. Also, please feel free to update the LiquiBase documentation wherever you find it lacking or in need of improvement. Nathan On Thu, May 22, 2008 at 1:40 AM, Darryl Pentz <[hidden email]> wrote:
|
|
In reply to this post by dtabwhite
Hi Dave,
do you use one of the event hooks to copy changelog.xml onto your classpath so that new Liquibase("changelog.xml", fileOpener, database) in Bootstrapp.groovy can resolve the file? I'm planning on have the migrations run in test and prod mode so need it to work via run-app and as a deployed WAR cheers Lee 2008/5/22 David White <[hidden email]>: Hey Guys, |
|
|
An event hook seems to work quite well:
eventPackagingEnd = { Ant.copy(file:'grails-app/migrations/changelog.xml',todir:'web-app/WEB-INF') } The only downside is that you get two copies of it in your project and you have to remember (and educate your team) not to modify the one in web-app/WEB-INF cheers Lee 2008/6/11 Lee Butts <[hidden email]>: Hi Dave, |
RE: Liquibase usage and alternatives
|
In reply to this post by Lee Butts
Hi Lee, I found a way to find your changelog.xml
file without resorting to having to copy it onto the classpath: liquibase = new
Liquibase("changelog.xml", new GrailsFileOpener(), database); I leave changelog.xml in its default place
(grails-app/migrations/changelog.xml) Works slick! Jay Guidos From: Lee Butts [mailto:[hidden email]]
Hi Dave, 2008/5/22 David White <[hidden email]>: Hey Guys, On Wed, May 21, 2008 at 5:02 PM, Mike Hugo <[hidden email]> wrote: Hi Darryl,
Hi Mike,
|
|
|
Hi Jay,
good find, however does that still work when running as a deployed WAR inside tomcat for example? I don't think it will, the GrailsFileOpener trys to resolve getResourceAsStream("grails-app/migrations/"+file) which will not work as the changelog.xml is not copied into WEB-INF/grails-app cheers Lee 2008/6/12 Jay Guidos <[hidden email]>:
|
RE: Liquibase usage and alternatives
|
|
Hmm, ya that could be. I have yet to
deploy a war. Looking at the code, I think you are right, your only
option on WAR build is to hook into the “WarStart” event to get the
changelog into your war. IMHO that’s still better than having to
keep it in two places all the time. Jay From: Lee Butts
[mailto:[hidden email]] Hi Jay, 2008/6/12 Jay Guidos <[hidden email]>: Hi Lee, I found a way to find your changelog.xml file without
resorting to having to copy it onto the classpath: liquibase = new
Liquibase("changelog.xml", new GrailsFileOpener(), database); I leave changelog.xml in
its default place (grails-app/migrations/changelog.xml) Works slick! Jay Guidos From: Lee Butts [mailto:[hidden email]]
Subject: Re:
[grails-user] Liquibase usage and alternatives Hi Dave, 2008/5/22
David White <[hidden email]>: Hey Guys, On Wed,
May 21, 2008 at 5:02 PM, Mike Hugo <[hidden email]> wrote: Hi
Darryl,
Hi Mike,
|
|
I am following your recommendations to run the changelog.xml when deploying the war in the web server, it works perfectly, I am using several files and I am including them in the changelog.xml, I have something like this:
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd"> <include file="./releases/20091015-changeLogSchema.xml" relativeToChangelogFile="true"/> <include file="./bugs/20091115-45896.xml" relativeToChangelogFile="true"/> </databaseChangeLog> I am copying the migrations folder to WEB-INF/grails-app/migrations, this works great, but when in DEV, if I do grails migrate, I get the following exception java.io.FileNotFoundException: /Projects/Test/grails-app/migrations/changelog.xml./releases/20091015-changeLogSchema.xml (No such file or directory) I think this is because I am setting the relativeToChangelogFile attribute to true. Any idea what might be wrong with these settings when in DEV env? Thanks!!
- Maricel
|
| Powered by Nabble | See how NAML generates this page |
