LiquiBase

Martin Fowler makes good points in his Incremental Migration Bliki post. Data migration is easy to push off or forget, and can often kill an otherwise great release–especially when valuable historical data gets lost due to a bad script.

I think reason many people put off data migration scripts is because they mistakenly think that they are for release day only. In reality, however, data migration is needed throughout the development process including updating developer databases and migrating integration test data.

Doing your data migrations incrementally and appending to your database change log on every refactoring will help you catch migration logic errors early and keep everyone’s databases in sync.

To keep up with your database migration scripts, you need to use the update your test/code/test cycle to:

1. Write Test
2. Run Test
3. Code Change
4. Append database changes to change log
5. Run LiquiBase to update database
6. Run Test, See Green

Making no database changes outside of your change log script and dev cycle will ensure that no changes get forgotten and that there are no differences between what happened on the developer’s database and what happens on other developer’s and production databases.

To integrate LiquiBase into your build process, there are Ant, and Maven front-ends available. This will allow you to migrate your integration tests data sets before executing them and allow you to find errors in your database update logic with each run.

[Note: This post was re-published due to Blog hosting change]

We have moved our blog to a new hosting provider and to WordPress from Blogger.

Please let us know if you have any issues or if any RSS feeds don’t transfer correctly.

It is LiquiBase 1.0’s first birthday, but you get the gift: LiquiBase 1.7.0 is now available.

Major changes include:

• Added loadData change which allows importing of data from CSV files
• Added support for changelog parameters for more dynamic and flexible change logs
• Allow preconditions in changeSet tags to act as sanity checks during a database update
• Can export data with generateChangeLog
• Added markNextChangeSetRan command
• Added validCheckSum tag to changeSet
• Can use any refactoring tags or reference previous changeSets inside a rollback tag
• Allow custom preconditions
• RowMapper interface is now public
• Misc bug fixes

The core library can be downloaded from http://www.liquibase.org/download while the Maven, Grails, and IntelliJ plug-ins can be installed from their respective plug-in managers.

LiquiBase 1.6.1 has been released.

This is primarily a bug fix release, but also includes a release of the 1.6 version to the Maven (still propagating), Grails, and IntelliJ repositories.

Download LiquiBase 1.6.1 from: http://www.liquibase.org/download

The LiquiBase team is proud to announce the 1.6.0 LiquiBase Core release.

This release focuses primarily on Hibernate integration which allows you to use the diffChangeLog functionality available with the command line and Ant to automatically generate LiquiBase change sets based on new Hibernate mapping information.

This functionality is similar to what is offered with hbm2ddl, but with the added ability to control and modify the changes before they are applied. For more information, see http://www.liquibase.org/manual/hibernate

Additional functionality includes a new “tagDatabase” tag, a “primaryKeyName” attribute on the column tag, as well as many bug fixes.

NOTE: Due to a bug in the change set md5 checksum generation in pre-1.6 releases, existing md5sum information needs to be cleared out of the databasechagelog table before running 1.6 for the first time. This can be done by running “liquibase clearCheckSums” or by running “UPDATE DATABASECHANGELOG SET MD5SUM=NULL”

Upgrading is simply a matter of replacing the liquibase.jar file. To take advantage of newer change log features, change your XSD declaration to:

<databasechangelog xmlns=”http://www.liquibase.org/xml/ns/dbchangelog/1.6″
xsi=”http://www.w3.org/2001/XMLSchema-instance”
schemalocation=”http://www.liquibase.org/xml/ns/dbchangelog/1.6
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.6.xsd”>

LiquiBase 1.6 can be downloaded from http://www.liquibase.org/download and, as always, let us know if you have any questions or issues.

The Maven and Grails plug-ins will be updated over the next week.

LiquiBase 1.5.2 has been released.

Changes include:

• Fixed bug with runOnChange and MD5sum check
• Handles generating changelog for tables with composite primary keys
• Other minor bug fixes

The LiquiBase core library can be downloaded via http://www.liquibase.org/download.

The Maven, Grails, and IntelliJ plug-ins have been updated and can be downloaded via their respective plug-in manager.

There was a great post the other day on Coding Horror titled “Get Your Database Under Version Control“. It references a good series of posts by K. Scott Allen on database version control, but also makes the point that “When it comes to version control, the database is often a second or even third-class citizen.”

I have noticed this in the past as well, and have wondered why?

Lack of tools?
Subversion, CVS, RCS and others have been around for years, but when I came the the realization that the database must be version controlled I could find no tools that fit my need and so created LiquiBase. I think lack of tools is a symptom of the problem rather than a cause, however. Developers have never been a group to sit around waiting for a tool to solve a pressing need they see.

DBA Overlords?
In some organization changes to the database must go through a database change process managed by DBAs. The fact that the database changes are managed by an external group could create a “not our problem” situation where the developers depend on the DBAs to track changes. Again, I don’t see this as a reason because the majority of projects do not have such a process in place and so wouldn’t be depending on it.

Only Now A Big Problem?
The answer that makes most sense to me is that database versioning is a relatively new problem. Code changes need to be propagated to every developer on a team quickly and reliably and therefore automated tools like Subversion and CVS have been around for a long time and no one would ever consider a group project without them. Most developers would not even consider a solo project without them.

Databases, on the other hand, do not change as often and so manual and error-ridden processes have worked well enough in the past. As databases have become more and more central to projects of all types the old manual database update scripts are showing their limitations.

I think the final straw in the ad-hoc database management schemes has come from the growth of agile processes. Pre-agile, even smaller projects would often design the database up-front and changes to it could be managed in a piecemeal fashion throughout the project. As agile does away with as much of the up-front design as possible, the number of database changes introduced throughout a project increases dramatically and a way to quickly, reliably, and automatically apply changes becomes a necessity.

I see the existing database versioning tools like LiquiBase, DBDeploy, and ActiveRecord:Migration as the RCS of database versioning: they are a great start, but there is a lot of ground left to cover. Recent version of LiquiBase have added features such as database change rollback, database comparisons, DBDoc, and change contexts, but there is still a lot to do. That doesn’t mean, however, that you shouldn’t Get your databse under version control!

LiquiBase Core 1.5.1 has been released along with the initial 1.5.x series Grails and IntelliJ plugins.

Changes to the core library are minor bug fixes including:

• Fixed failOnError logic
• Improved serial column detection in Postgres
• Rollback connections before close for DB2
• Other minor bug fixes

The LiquiBase core library can be downloaded, as usual, from http://www.liquibase.org/download. The Grails and IntelliJ plugins are installed via their respective plugin managers

The 1.5 Maven plugin should be released next week after further testing.

LiquiBase Core 1.5.0 is now available for download from http://www.liquibase.org/download

1.5.0 includes a major refactoring which should not affect most users except for the following items:

BREAKING CHANGES

---

• Servlet Migrator: The web.xml parameter names have changed. See http://www.liquibase.org/manual/servlet_listener for more information.
• If you used the “database.migrator.should.run”, it is must now be changed to “liquibase.should.run”
• If you have extended or embedded LiquiBase classes or calls directly in your code, changes will be necessary.

BACKWARDS-COMPATIBLE CHANGES

---

• Servlet Migrator: The classes to reference in web.xml have changed to liquibase.servlet.LiquibaseStatusServlet and liquibase.servlet.LiquibaseServletListener. The old classes are now simply subclasses of the new and are deprecated so they should work.
• Spring Migrator: The class to reference in your spring config has changed to liquibase.spring.SpringLiquibase. The old class still exists as a subclass of the new so existing configurations should continue to work.
• Command Line: The “migrate” command has been changed to “update”.
  “migrate” is now an alias for “update” so existing calls should continue to work

ENHANCEMENTS

---

IMPROVED SCHEMA SUPPORT
There is now a “defaultSchemaName” parameter available for setting default schema. This schema will be used for all ambiguous database objects as well as for storing the databasechangelog and databasechangeloglock tables.

IMPROVED ANT TASKS
Ant support has been greatly expanded and now covers most of the functionality available in the command line application. See http://www.liquibase.org/manual/ant for more information.

NEW COMMANDS

• changeLogSync
• updateCount
• updateCountSQL

NEW REFACTORINGS

• update change
• delete change

OTHER CHANGES

---

• Custom Database implementations can be specified with the databaseClassName parameter
• “replaceIfExists” attribute added to createView
• createTable can specify uniqueConstraintName
• Setting value/valueNumeric/valueBoolean/valueDate on addColumn will update all existing rows with the given value
• Database table comments saved to generated change log
• Changelog file comparisons are case-insensitive on windows
• Output warning of old schema version
• Added comments tag to generated SQL output
• Rollback commands can specify contexts
• XSDs are not pulled from network
• Handles Postgres datatypes better
• Bug fixes

Upgrading

---

Upgrading is simply a matter of replacing the liquibase.jar file. To take advantage of newer change log features, change your XSD declaration to:

<databasechangelog xmlns=”http://www.liquibase.org/xml/ns/dbchangelog/1.5″
xsi=”http://www.w3.org/2001/XMLSchema-instance”
schemalocation=”http://www.liquibase.org/xml/ns/dbchangelog/1.5
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.5.xsd”>

Depending on feedback received from this release, the 1.5.0.0 releases of the various plug-ins (Maven, Grails, IntelliJ, Eclipse) should be released over the next few days.

As usual, be sure to let us know if you have any questions or issues.

I would like to thank Yasuo Honda for all his hard work on the Japanese translation of the LiquiBase documentation. He was also a great help in determining the best wiki tool to use to support translations.

If you are interested in assisting with translating the LiquiBase documentation, feel free to edit the site wiki directly, or contact the documentation mailing list for more information.

You can view the Japanese translation by starting at http://www.liquibase.org/ja/home or by clicking the “ja” box on any page.