My First Groovy DSL

I decided to have a go at automating the management of our Jenkins jobs. Basically most of our projects have similar builds, and I want to keep the job configs standard – they tend to diverge when we manage them via the UI. I’ve previously played with the Jenkins API using HttpBuilder to post XML config files. My DSL doesn’t do anything clever to generate the XML, but it makes the Jenkins config very readable:

jenkins {   
    url = "http://jenkins.url/"
    username = "jenkins"
    apiKey = "asdfadsgdsfshgdsfg"

    buildGroup {
        name = "Maven Builds"
        xml {
            feature = "release/xml/maven-test-config.xml"
            develop = "release/xml/maven-install-config.xml"
        }
        repos {
            project1 = "ssh://git.repo/project1.git"
            project2 = "ssh://git.repo/project2.git"
        }
    }

    buildGroup {
        name = "Grails Builds"
        xml {
            feature = "release/xml/grails-test-config.xml"
            develop = "release/xml/grails-install-config.xml"
        }
        repos {
            project4 = "ssh://git.repo/project4.git"
            project5 = "ssh://git.repo/project5.git"
        }
    }

}

I think this is so easy to read.

I’m not really sure if I’ve gone the right way about writing this, but here goes:

First, I have a main method, which parses the file. It binds the “jenkins” method to an instance of my JenkinsDsl class.

    Binding binding = new Binding();
    JenkinsDsl jenkinsDsl = new JenkinsDsl();
    binding.setVariable("jenkins",jenkinsDsl);
    GroovyShell shell = new GroovyShell(binding);
    Script script = shell.parse(file);
    script.run();

The JenkinsDsl class overrides the `call` method, and uses `methodMissing` to define how we handle the buildGroups. The really cool bit is I didn’t have to write anything special to get the url, username and password from the jenkins config file – groovy is automatically calling a ‘setProperty’ method on the delegate.

class JenkinsDsl {

    def call(Closure cl) {
        log.info "Processing Main Configuration"

        cl.setDelegate(this);
        cl.setResolveStrategy(Closure.DELEGATE_ONLY)
        cl.call();
    }

    def url
    def apiKey
    def username

    def jenkinsHttp

    def methodMissing(String methodName, args) {

        if (methodName.equals("buildGroup")) {

            Closure closure = args[0]
            closure.setDelegate(new JenkinsBuildGroup())
            closure.setResolveStrategy(Closure.DELEGATE_ONLY)
            closure()
            closure.delegate.updateJenkins(getJenkinsHttp())
        } else {
            log.error "Unsupported option: " + methodName
        }
    }

    def getJenkinsHttp() {
        if (!jenkinsHttp) {
            jenkinsHttp = new JenkinsHttp(url, username, apiKey)
        }
        jenkinsHttp
    }   
}

The JenkinsBuildGroup class is very similar – to save time I’ve used the ‘Expando’ class to collect the xml and repository details.

class JenkinsBuildGroup {

    def name
    def xml
    def repoList

    /**
     * Use methodMissing to load the xml and repos closures
     * @param methodName
     * @param args
     */
    def methodMissing(String methodName, args) {

        if (methodName.equals("xml")) {
            xml = new Expando()
            Closure closure = args[0]
            closure.setDelegate(xml)
            closure.setResolveStrategy(Closure.DELEGATE_ONLY)
            closure()
        } else if (methodName.equals("repos")) {
            repoList = new Expando()
            Closure closure = args[0]
            closure.setDelegate(repoList)
            closure.setResolveStrategy(Closure.DELEGATE_ONLY)
            closure()
        } else {
            log.error "Unsupported option: " + methodName
        }
    }

    def updateJenkins(def jenkinsHttp) {
        xml.getProperties().each { String jobType, String filePath ->

            def configXml = new File(filePath).text

            repoList.getProperties().each { String jobName, String repo ->
                String jobXml = configXml.replaceAll("REPOSITORY_URL", repo)
                jenkinsHttp.createOrUpdate(jobName + "-" + jobType, jobXml)
            }
        }
    }
}

To actually update jenkins, I substitute the correct repository in the template XML file and post it to the Jenkins API. The JenkinsHttp class is just a wrapper for HTTP Builder. It checks if a job exists so that it can correctly create or update.

I’m not sure if this is the right way to go about writing a groovy DSL, but it works! And once I’ve finalised the template XML files, Jenkins is going to be a vision of consistency.

  

Groovy Grails Exchange 2013

This week I attended the 2013 Groovy Grails Exchange, organised by Skills Matter (with whom I’ve also been lucky enough to attend a couple of courses over the years).

Finally got time to jot down a few thoughts – I came back with an almost full notebook, but these are the highlights:

1. What a great conference! Skills Matter did an amazing job of hosting. The speakers were varied and interesting. It was fantastically motivating to be amongst all these people who are actually pushing the technology forwards. As I work in a very small dev team (out in the sticks) I found it really interesting to see what goes on in the rest of the world.

2. Must. Learn. Spock.

3. Must also get back on board with Geb. I dabbled a little a while ago, and now that we’re about to start a new Grails project, functional testing with Spock & Geb has to be central to it. Lots of food for thought in the DevQA talk by Alvaro Sanchez-Mariscal, applicable even though we don’t have a dedicated QA team for our internal dev.

4. Forces on code – what makes code “good” depends on the context. Common sense really but well illustrated in the talk by by David the Coder

5. Do I need to learn a JavaScript framework? I enjoyed the very persuasive talk on Developing SPI Applications by Alvaro Sanchez-Mariscal. Separating the front end and back end into independent apps makes a lot of sense especially if you have dedicated UI developers. I’m not sure we have the resources to move away from GSPs just yet. But I’m certainly going to have a browse around ToDoMVC to get an idea of the options.

6. Jeff Brown’s live coding demo of Metaprogramming With The Groovy Runtime was a great refresher – nothing particularly that hadn’t been covered on the Groovy course I attended last year, but a reminder that I am pretty guilty of just writing Java like code inside .groovy files and not taking full advantage of groovy’s awesomeness.

7. The “Open Source and You” session by Peter Ledbrook made me think a bit more deeply about Open Source software – what to expect from it, the costs involved, how to manage a successful open source project. I’m definitely motivated to get more involved.

8. Gradle for deployment? @danveloper’s talk on Groovy for Sysadmins gave me lots to think about. I doubt I’ll ever end up hacking the kernel, but I like the software centric approach to deploying and maintaining servers.

9. Must pay more attention to release notes and road maps. New and upcoming versions of Groovy and Grails have some great new features that I’m looking forward to using. Changes notes for Groovy 2.2 and Grails 2.3 definitely worth a look. Also looking forward to plugin and build system changes in Grails 3.0 some time in 2014.

10. And finally, how come all these Open Source aficionados use Macs?! I’ve not got an iAnything yet and don’t plan to. Linux rocks :)

Anyway, in summary, well worth going for anyone working with Groovy or Grails. Podcasts of all the talks are available on the Skills Matter website, but I think attending in real is a fantastic opportunity to absorb knowledge from an enthusiastic and knowledgeable crowd, and worth every penny. Better get my early bird ticket for next year…

  

Issue Security in Jira sub-tasks and the Script Runner plugin

I’ve been trying to do some advanced JIRA configuration recently, with mixed success.

The requirement was to have issues in a project which could be viewed only by a specific user (not necessarily the assignee).

I discovered that “Issue Security Schemes” can have levels which allow access based on a “User Custom Field Value”. Fantastic. I added a custom field, let’s call it “Employee”, and added it to an Issue Security Level. Applied to the project and bingo, users can only see issues where they have been entered into the “Employee” field.

Too easy. I added a subtask to an issue where I was the employee. No error message, but no subtask appeared either.

My understanding was that you could not set issue security on subtasks, because they inherit the issue security of the parent. I assumed the intention of this was that if you could see an issue, you should be able to see its children. But no – they specifically inherit the issue security setting from the parent, rather than its visibility – as confirmed in Atlassian Answers. So because the sub-task does not have an “Employee” field, I can’t see it. Bother.

Of course, I could use a workflow post-function so that when subtasks are created, the employee field is set to that of the parent. But what if the value is changed in the parent? It’s a nuisance to have to change all the sub-tasks too.

Time to investigate the Script Runner plugin.

Here’s the code for my Scripted field, which gets the value of the Employee field in the parent issue.

import com.atlassian.jira.ComponentManager
import com.atlassian.jira.issue.CustomFieldManager
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.issue.fields.CustomField

def componentManager = ComponentManager.getInstance()
def parentIssue = issue.getParentObject();
CustomFieldManager cfManager = ComponentManager.getInstance().getCustomFieldManager()


CustomField customField = cfManager.getCustomFieldObject(99999)
parentIssue.getCustomFieldValue(customField)

It is almost too easy. I think it may be the answer to some of the other requests I’ve had for custom fields in JIRA, but unfortunately it doesn’t solve this problem. The calculated value appears perfectly on the page when viewing a sub-task. But, when I go to the “Issue Security Level” and click in the “User Custom Field Value” drop down, it doesn’t appear! It’s configured with the “User Picker” Searcher and template, so I don’t think there’s much more I can do.

Back to the drawing board…

  

Unexpected side effects using Groovy MockFor in a Grails test

Recently, I was trying to write a Unit Test for a Grails method which instantiates an HTTPBuilder object within a method.

I couldn’t use the normal Grails ‘mockFor’ syntax, because the HTTBuilder was not injected or accessible from outside the class. No problem, because standard Groovy MockFor is also available.

With a bit of help from google (http://stackoverflow.com/questions/9101084/groovy-httpbuilder-mocking-the-response) I came up with the test below.

def testDoSomething() {
    def requestDelegate = [response: [:], uri: [:]]

    def mockHttpBuilder = new MockFor(groovyx.net.http.HTTPBuilder)
			
    mockHttpBuilder.demand.request { Method method, Closure body ->
        body.delegate = requestDelegate
        body.call()
        requestDelegate.response.success() // make it call success
    }

    mockHttpBuilder.use {
        // All calls to HTTPBuilder will be intercepted within this block of code.      
        MyClient myClient = new MyClient();
        assert myClient.doSomething("Some Param")
    }
}

This works really well in isolating the code under test from the HTTP request.

However after spending a bit of time working on this test, and getting it to pass, I ran test-app on the entire project, and was very confused to see that some totally unrelated tests had begin to fail. Grails was reporting errors like “No more calls to ‘get’ expected at this point.” in tests where I had used no mocking at all. I was bewildered.

Eventually I came across GRAILS-8535. Although unit tests are supposed to be just that, it seems that grails continues to use the proxy in subsequent tests. This appears to be fixed in Grails 2.2.3, but as of the time of writing, this wasn’t released.

Luckily, it is possible to manually reset the MetaClass in the Grails MetaClassRegistry at the end of the test.

At the start of the test which uses MockFor, I record what the class was originally:

MetaClassRegistry registry = GrailsMetaClassUtils.getRegistry()
def originalMetaClass = registry.getMetaClass(HTTPBuilder)

And at the end of the test, set it back:

MetaClassRegistryCleaner.addAlteredMetaClass(HTTPBuilder, originalMetaClass) 
  

Writing a maven 3 plugin in groovy

I thought I’d document this, as we found it a bit confusing to get going, and the documentation is fairly sparse.

We wanted to knock up a quick maven plugin which would parse some XML files, and output some HTML. Great, a chance to do some groovy foo! Parsing and writing XML with groovy is so easy, we thought we’d try writing the plugin in groovy. Once compiled, groovy is just java, so Maven should not care what the plugin is written in.

First, DON’T follow these instructions: http://www.sonatype.com/books/mcookbook/reference/writing-plugins-alternative-sect-writing-groovy.html. They only work in Maven 2.

For reference, the error message when compiling the plugin:

Failed to execute goal org.codehaus.mojo.groovy:groovy-maven-plugin:1.0-beta-3:generateStubs (default) on project firstgroovy-maven-plugin: Execution default of goal org.codehaus.mojo.groovy:groovy-maven-plugin:1.0-beta-3:generateStubs failed: An API incompatibility was encountered while executing org.codehaus.mojo.groovy:groovy-maven-plugin:1.0-beta-3:generateStubs: java.lang.NoSuchMethodError: org.codehaus.plexus.PlexusContainer.hasChildContainer(Ljava/lang/String;)Z

The correct approach for maven 3 is documented here:
http://docs.codehaus.org/display/GMAVEN/Implementing+Maven+Plugins.

It still doesn’t work straight away. If you follow the guidelines and create the pom and mojo as in this page (specifiying 1.5 for the gmaven versions), the mojo appears to install but then when you execute it from another project, you get the following error:

 org.sonatype.guice.bean.reflect.Logs$JULSink warn
WARNING: Error injecting: sample.plugin.GreetingMojo
java.lang.NoClassDefFoundError: Lorg/codehaus/groovy/reflection/ClassInfo;

The answer is on this page: http://jira.codehaus.org/browse/GMAVEN-110.

We need to add the following to the gmaven-plugin configuration:

<configuration>
    <providerSelection>1.5</providerSelection>
</configuration>

I’ve attached a zip containing a working mojo and a test project: groovy-maven-plugin.tar.gz. Run mvn install on the first to install it in your local repository, then mvn compile onthe second to see it execute.

  

Groovy use of the Jira REST API to update a custom field

We use Jira for managing a specific type of project at work. For some time, we’ve used the description field of certain issues to hold a particular piece of information. We recently decided to make a custom field for this instead, so that we could use the description field for its intended purpose, and include our new custom field in issue lists.

The challenge was to update the new custom field with the data from the description field for all historical issues. I couldn’t see a way to do it through Jira directly, so I thought I’d take the opportunity to exercise my groovy skills, and explore the Jira REST API.

I decided to use the HTTPBuilder library. Here is the grab and the imports:

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.5.2' )

import groovy.json.*

import groovyx.net.http.RESTClient
import groovy.util.slurpersupport.GPathResult
import groovyx.net.http.ContentType;

Good practice next, I don’t want to hard code my username and password in a script, so lets prompt for them:

String username = System.console().readLine("%s ", ["Jira Username:"] as String) 
String password = String.valueOf(System.console().readPassword("%s ", ["Jira Password:"] as String[]))

Next we make a client for all our rest requests:

def jiraClient = new RESTClient("https://my.jira.url");

Now I’m going to set up the authorisation header, and some settings that I’ll reuse each time I make a REST request. I spent a while working out how to do the authorisation. You can post to the /auth/1/session resource and then store a cookie, but sending an Authorisation Header seems nice and simple

def authString = "${username}:${password}".getBytes().encodeBase64().toString()
def requestSettings = [contentType: ContentType.JSON, requestContentType: ContentType.JSON, headers:['Authorization':"Basic ${authString}"]]

Here’s the first REST request, to the search resource. The clever groovy bit is adding two maps together, using the plus operator. I spent a while looking for a special function to do that. Sometimes groovy makes things too easy! The plus operator adds two maps and returns a new one. In contrast using the left shift operator would alter the map on the left.

def jqlQuery = "type = \"My Issue Type\" and cf[11720] is EMPTY and project = MYPROJ"
def issueRequest = jiraClient.post(settings + [path: "/rest/api/latest/search", body:  [jql: jqlQuery, maxResults: 10000]])

Next we run a REST PUT request for each issue returned. The groovy JsonBuilder makes is really easy to construct a correctly formatted JSON String to send. The syntax is a bit complicated – as well as the REST API documentation I found this example helpful: https://developer.atlassian.com/display/JIRADEV/JIRA+REST+API+Example+-+Edit+issues.

issueRequest.data.issues.each() { issue ->
  def json = new groovy.json.JsonBuilder()
  def root =  json.update {  
		customfield_11720 ( [{
			set "${issue.fields.description}"
		}] )
  }
  
  try {
      def result = jiraClient.put(settings + [path: "/rest/api/latest/issue/${issue.key}", body:  json.toString()] )
      println "${issue.key} | ${issue.fields.summary} | ${issue.fields.description} |  ${result.status}"
  } catch (Exception ex) {
      println "${issue.key} | ${issue.fields.summary} | ${issue.fields.description} |  FAIL"
  }
  
}

Unfortunately some of the put requests failed, because we’ve made more fields mandatory since we started and some of the older issues were missing data for mandatory fields, but it still saved a lot of time!

  

Updating from a list in Grails

This week I had the pleasure of attending both Groovy and Grails courses at Skills Matter in London. The courses were taught by Dierk König, a committer to both Groovy and Grails, and the author of Groovy in Action. He’s very knowledgeable and the courses were really interesting – I would have liked more time for advanced grails topics, but I guess you can’t fit it all in to two days!

During the course I asked for help with one of my pet grails problems – the best way to update multiple records at once from a list view. For example, you might have a set of records with a check box field, and want to tick/un-tick for several rows and then save all with a single click. We didn’t quite get it working in class, but I got some helpful advice which meant I was able to finish it off on the train home.

The domain class is nothing special:

class Person {
 String firstName
 String lastName
 boolean available
}

My PersonController looks like this (I’ve left the scaffolding in, so you get all the standard pages too):

class PersonController {
   static scaffold = true
   def multiEdit = {
      List<Person> list = Person.list()
      [personInstanceList: list]
   }

    def multiSave = { MultiplePersonCommand command ->
     command.people.each { p ->
     Person person = Person.get(p.id)
     person.properties = p.properties
     person.save()
   }
   redirect action: list
 }
}
class SinglePersonCommand {
 Integer id
 boolean available
}
class MultiplePersonCommand {
 List<SinglePersonCommand> people = [].withDefault({ new SinglePersonCommand() } )
}

The important thing here is the use of Command objects. I’ve defined these in the same class as the Controller but they could be in a separate file. The really important tip is the use of `withDefault` on the list in the MultiplePersonCommand. When the binding takes place, we can’t guarantee what order it will be in. For example it might try to bind the second list item before the first. This would cause an error without the `withDefault` method.

And finally, multiEdit.gsp looks like this:

<g:form action="multiSave">
 <g:each var="person" in="${personInstanceList}" status="i">
     <div id="person${i}">
     <g:hiddenField name='people[${i}].id' value='$person.id'/>
     <g:fieldValue bean="${person}" field="firstName"/>
     <g:checkBox name='people[${i}].available' value='1' checked="${person.available}"/>
   </div>
 </g:each>
 </div>
<g:submitButton name="action" />
</g:form>

The important thing here is the use of the $i variable in square brackets on the fields in question. This means that the params that come back to the Controller will effectively contain people[0].id, people[0].available, people[1].id, people[1].available and so on. Grails is clever enough to bind all the people[0] values to the first SinglePersonCommand in the people list inside the MultiplePersonCommand, and so on. Then I can access this list and copy across the values to People objects and save them.

I hope this is useful to someone. I’m looking forward to spending some time on Groovy and Grails development, so hopefully more here soon!