Liquibase

I have been working hard on LiquiBase 2.0 lately, and it feels very, very good to be updating and cleaning up the codebase. I am putting the SOLID principals to work, increasing test coverage, and building in integration hooks.

So you do not worry, The changelog XML format will stay the same and be completely backwards compatible with the 1.x series so the vast majority of users will see little to no changes. It is only the underlying java classes that will change significantly. If you are using any liquibase classes directly, this will be a large change for you, but the number of people using the API is small enough that it is worth simplifying and standardizing now, before really opening up the library for extension like the plan is. We will provide a migration guide for anyone needing it.

Curently I feel I am about 1/2 done with the main code update, which means that in about 1 month we should have a beta 1 release. The plan is to have a series of beta releases before the final release.

With beta 1, I am hoping to open up a plug-in contest. There will be many new integration points around modifying what SQL gets ran for a change, new database support, new changes, new changelog parsers, and more. There are many tools and enhancements that would be very helpful to some, but not general purpose enough to make it into the core application that would make great plugins, so get thinking. Despite having no budget, I would like to kick off the extensibility support with a bang, hopefully I can find a company willing to throw in a prize for some PR. Wost case, the grand prize will be a copy of mysql community server and sqlserver desktop edition

Despite not being finished with the refactoring, feel free to review the code and give me any feedback (or patches) you have. There are many use cases i have heard over the years that I plan on supporting, but there the best ones are often the ones I never thought of.

I am starting the extensibility refactoring at the *Statement class level and moving up from there.  I committed an initial version of what I am thinking to trunk, and am looking for feedback (either in comments, or directly to me at nathan [at] voxland.net).

Renamed liquibase.database.sql package to liquibase.database.statement.
We may want to support non-”sql” statements in the future, so the name change seemeded best

Separated out the creation of SQL from the *Statement class.
Before each Statement class had a “String getSqlStatement(Database)” method that would create a string containing the database specific sql required to run the statement.  Now, there is a liquibase.database.statement.generator package that contains classes that take a Statement instance containing the information about a statement (table name, column names, etc) and converts them into SQL strings.

There is now a  SqlGeneratorFactory singleton with a  “SqlGenerator getBestGenerator(SqlStatement statement, Database database)”  method.  When we get to the point where we have Statement classes that need to be executed, liquibase will call this method to get the correct SqlGenerator instance for a given statement and database and use that instance to create the sql to execute.  (this change has not been made yet)  .

The SqlGeneratorFactory has a list of all SqlGenerators available.  That list is built up via a reflection package search in the liquibase.database.statement.generator package and sub-packages, or via a SqlGeneratorFactory.register() method.  There will be a standard package of liquibase.database.statement.generator.ext that can be used by 3rd parties to have their statements registered automatically without using the register ethod.

I have started breaking up the Statement classes (although I haven’t removed the old getSqlStatement methods yet).  In the liquibase.database.statement.generator package there is an SqlGenerator interface which provides a getSpecializationLevel method, an isValid method, and a generateSql method.   In the getBestGenerator() method, the SqlGeneratorFactory will loop through all known generators and call the isValid method to determine if a given SqlGenerator would work for that combination of database and statement.  Once it has a list of good generators, it will call the getSpecialiazationLevel() method which returns an int corresponding to how “good” that generator is for that statement/database combo.  There should usually be a default generator for each statement that will have a level of 1.  There will often be database-specific versions of the generator that return a level of 5.  3rd party Statement generators can return any level they want to override the built in ones or other 3rd party generators.  If no generators are found for a database/statement combo, tyere is a NotImplementedGenerator that is returned that simply throws an exception.

New generateSql  method returns an array.
Before, there was a one-to-one correspondence between a Statement class, and the SQL generated.  I think there are situations, however, where a Statement may generate multiple SQL commands and so we should allow the return of an array.

generateSql returns instances of “Sql”, not Strings
This is the part I go back and forth on.  Since the goal of this refactoring is to give 3rd party developers an API to code against, I want to do my best to make sure that API is not going to change.  There has been talk in the past about building more flexibility into the system by giving back an syntax tree that can be modified before being executed.  The idea with the Sql interface, is that we would only have an “UnparsedSql” implementation at first, but may make smarter implementations in the future.

The question is: is this too much?  We now have a mechanism where we can override what sql string is generated for a particular database and/or statement,  is that going to cover all the low-level-sql-tweaking that we would need?  I feel like the Sql interface may allow some sort of cross-cutting sql modification, or similar, but it is just a half-thought out idea.

Improving tests
I am going to repurpose the test framework around the Statement classes that actually run them against the database.  This has not been done yet.

My plan for 1.10 is to focus on extensibility. I wanted to make sure I have good goals for that in mind, and that what we are planning to do will match up with those goals.

1.    More Extensibile API:
a.    Ability to override what SQL is sent to the database for standard databases and refactorings (Add engine innodb to all create table statements, LoadData on MSSQL should call “set identity insert on/off”, etc)
b.    Ability to add custom Change classes and use them from the changeset
c.    Ability to create a custom changelog parser as an alternative to the standard XML
d.    Better documentation of how the liquibase API can be used within an application
e.    Ensure that all liquibase APIs will remain stable for the foreseeable future

2.    Create 3rd party community. There are many refactorings that we do not want to support in the core because they are database specific or limited in scope (vacuum database, set identity_insert on, etc).
a.    Create a “contrib” area on the site that people can upload useful extensions that are not in the core
b.    Developer documentation on extending liquibase
c.    Better issue/feature tracker that supports extensions?
d.    Better mailing list/newsgroup setup
e.    Ensure that all documented extension points will remain stable for the foreseeable future

3.    Make core codebase more approachable to submissions:
a.    Better developer documentation
b.    Document migration path from 3rd party extension to core functionality
c.    Refactoring of codebase to make it easier to add new Database, Change, and Statement classes without touching as much other code
d.    Better unit test coverage to stop regression errors

Is there any particular use cases related to extensibility you want to make sure are addressed?  What am I missing?

I plan to follow up with requests for in-depth comments on each of these steps as we get to them and can say exactly what we are implementing.