Maven, JIRA, Bamboo and release management – ARRHH!

(this is a long one!  Hope your sitting comfortably)

I just don’t get it, I really don’t – life just shouldn’t be this hard!

First, some (simplified and made-up) background:

• there is a single product
• we host the products and each client will always be running the latest production
• we care about the “mainline” code, the version currently being tested and the version currently in production

What I want is the following process:

•  developer does stuff in git – enough is done to justify a roll-out.  Maybe “enough” is “a single chunk of work” (i.e. continual deployment)
•  some trigger starts the “test” process which means I now care about a set of release candidates that will eventually go into production as an atomic unit
• developers do some more stuff which should go into this release, so more stuff is added to the *existing* test process
• eventually testers say “yeah!” and all the current release candidates go into a single production build

Now I know you are all (at least the 2 people who read this blog) shouting “what’s the problem – just number your versions!”.  Well, I don’t want to  – I think numbered releases *in this environment* is an abstraction too far.  I *only* care about “untested work”, “work in test” and “work in production”.

Essentially, when stuff is on production the “in tests” is no longer interesting.  In essence, at that point we only care about the fact that production is actually (git’s) SHA1 XYZ.  We wouldn’t even care about that if it weren’t for the need for emergency bug fixes (although Martin Fowler’s http://martinfowler.com/bliki/FeatureToggle.html is helping mitigate even that).

To be clear, I view anything between that isn’t deployed as inventory, which according to the Theory of Constraints is a *bad thing*.

The way we contort the tools to support this workflow is:

[git and bamboo]

Git has a “master”, “test” and “prod” branch.  At any point master is good to go.  Always.  At some point (maybe as the result of a successful Bamboo build from master? :)) the master branch will be merged into test.

Bamboo will then pick up this change and start building and testing it. If this works it will be deployed into the test environment (http://www.puppetlabs.com/ rocks by the way!).

At this point real people start sanity checking it (our automated tests just aren’t good enough at the moment…) .  Maybe another chunk of code makes it into test as well, so now test has two discrete chunks of work.  At some point the real people say “yeah”, so the test branch is merged into the prod branch which causes Bamboo to build it, test it and then deploy it onto the production servers.  At this point, the prod branch is really a tag against a git SHA.

Nothing gets into prod except through test and nothing gets into test except through master.  If a “quick fix” needs to be applied to test, then tough – test is never edited directly!  Accepting that master isn’t good enough to go into test is too high a cost.

[jira]

So, one question that needs to be answered is “what coming up in the next version?” so that testers can be ready and the client can be ready.  Maybe, on a good day, the client was even involved in the discussion of which issues to release into development 🙂

After many years experience, I have given up trying to plan beyond the next few weeks of work, therefore we do not have the next 15 releases in JIRA of the form “1.1, 1.2, 1.3… 2.0”.  Instead we have a version called “current” which contains JIRAs currently being worked on.  A version called “next” which contains agreed, and prioritised work (i.e. this is the upstream queue) and “unscheduled” for work which has been agreed, but isn’t scheduled.  Developers pick from the ordered list in “current” and managers feed “next” from “unscheduled”, and then current from next.

When a batch is ready to be released for testing, nothing happens in JIRA.  When a version is ready to go to production, the “current” version is renamed to the current date – all unfinished JIRAs are migrated to “next” and a new “current” version is created.

The answer to “what’s in production” is now easy – “go look at the last (date-stamped) version”.  Great.

The answer to “what’s in test” is “no more than what is in current”.  Blurgh.  The obvious improvement is to continually deploy master to test, which in effect removes the need for an explicit test; test is the delta between the last production and now.

[Bamboo]

Bamboo has separate projects for listening to master, test and prod branchs in the git repository.  After a successful test build or prod build, the WARs are deployed to our puppet file store which means within 30 minutes (the default checking time for a puppet client) the WARs will be deployed to the testing or production infrastructure.

I really want Bamboo though, to tell me exactly *what* is in each testing “phase”.  It *will* tell me what is in each *build*, but there might (for example) be multiple builds in the testing phase.  There is nothing in Bamboo that will tell me that production contains test builds 1-10.  Sure, *JIRA* will tell me that version XYZ is comprised of test build 1-10 and production build 3, but, er, it is all a bit loose.  Even worse, the “test” plan in Bamboo will list *all* the issues that are in the test branch, not just the ones in this current round of testing.

The problem I think, is that none of the tools (git, JIRA and Bamboo) talk the same language.  None of them have a notion of a “unit of work” – JIRA issues are too fine grained, JIRA versions are too coarse.  GIT branches are in fact created for each release cycle, but because they are from master, they still contain the entire history.

If you are still reading – well done 🙂

Some “solutions”:

[don’t merge but rebase into test and prod]

Well this is rebasing in the wrong direction – there is no new work on test so there is nothing to rebase.

[create a new branch for each release cycle]

This actually doesn’t help at all because the branch still inherits its ancestors commit log…

[create a new *repo* for every release cycle]

Sure – that gives me a nice clean commit per group of work (i.e. one commit to the test repo is essentially a release candidate)

[go back to architecting and coding and let some-one else deal with all this!]

Sounds good to me 🙂

[use maven’s release management plugin]

That sort of falls down as it requires a new branch for each release – I don’t want to do that and I certainly don’t want to have to update Bamboo’s source repository settings every time, even if I could automate that!

[use bamboos’s release management plugin]

That seems to work best when there is a nice future release plan (i.e. now it is v1.4, we plan for v1.5, 1.6…v1.12 and v2.0).  It won’t automatically rename the “current” JIRA release into a date stamp and create a new “current” JIRA release.

Any solution I can think of using those tools just works against my notion of only caring about “current, test and prod” and is very verbose.  What with merging into different branches and keeping JIRA up to date there isn’t enough time to do anything that needs releasing :).  Moving to a pure continual deployment model is just going to make this even worse!

So, first – have I completely missed the point here?  If so, what process do you follow that supports such a “light touch” workflow (“untested work”, “currently being tested work” and “currently in production” are the only things I care about)?  If not, then how on earth do you manage all this stuff?

Tags: agile, bamboo, git, jira, maven, release management, tech