What the F is f:table

Today I needed to knock up a quick interface to some database tables for inputting and editing data to be used in the demonstration of some data extraction software. Aha I thought; I’ll try using Grails Scaffolding. I haven’t used grails scaffolding in earnest since taking a grails training course few years back. However, today I really did just need some simple CRUD functionality.

In Grails 3, you have two options for scaffolding – annotating the controller and having the views and controller methods auto-generated at runtime, or running a command to generate them statically so that you can modify them. I chose the latter, assuming that I’d want to do some customisation.

grails generate-all com.domain.Thing

You can then inspect the generated controller and views, and make any changes necessary. And this is where it all started to go wrong. The table containing the list of existing records didn’t look very nice. I’d removed the default ‘application.css’ which comes with grails, and used bootstrap to style the app. Without the default styles, the table has no spacing, and looks pretty awful.

No problem, I just need to add class=”table” to the table and I’ll get a standard bootstrap styled table. However, the generated index.gsp doesn’t contain a table tag. All I found was this:

<f:table collection="${thingList}"/>

The <f:table/> tag was a new one to me. Google suggests this comes from the grails fields plugin, but the documentation is very sparse: Grails 3 fields plugin.
The documentation doesn’t even mention the <f:table/>http://grails3-plugins.github.io/fields/snapshot/ref/Tags/table.html which helped a bit, in that it showed how to configure which fields to show in the table but didn’t help in changing styles or other formatting.

The main grails scaffolding documentation suggests running

grails install-templates

to get local copies of the templates used in scaffolding, but this doesn’t include anything to do with the fields plugin.

More detective work led to this Stackoverflow post, and onward to the fields plugin source code.

Finally… how to customise the f:table tag:

Place a file called _table.gsp in /grails-app/views/templates/_fields/

The default file contents are here: _table.gsp

After adding this file to the project and amending to use the required styles, the <f:table/> tag can be used throughout the project with reckless abandon.

My table looks nice now, but I think this sums up why I struggle with the grails plugin ecosystem; it feels a bit half-finished to be using an undocumented tag as part of what should be a quick start process for new users.

Using JDBI with Postgres JSON data

I’ve been migrating some raw JDBC code over to JDBI, and joyfully stripping out lines of boilerplate code for preparing statements, opening record sets, sometimes remembering to close them, handling SQL exceptions which won’t ever occur anyway, and so on. Using the SQL Object API means the only code you have to write is the SQL and a ResultSetMapper to determine how to create your domain objects from the resultset. It really promotes adherence to the single responsibility principle and discourages you from mixing logic in with your database access code.

The database in question has a number of fields containing JSON data. More specifically, they use the PostgreSQL jsonb data type. This has required a little more tinkering to get working.

Inserting jsonb data

Out of the box, JDBI provides two annotations for binding parameters. The @Bind annotation binds a single named argument, and @BindBean binds bean properties with matching names. However, to insert jsonb data, you need to first create an PGobject instance and bind that. To do this, I created a new Binding annotation, following the guidance here: http://jdbi.org/sql_object_api_argument_binding/

The annotation code looks like this:

BindingAnnotation(BindJson.JsonBinderFactory.class)
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.PARAMETER})
public @interface BindJson {
    String value();

    public static class JsonBinderFactory implements BinderFactory {
        @Override
        public Binder build(Annotation annotation) {
            return new Binder<BindJson, String>() {                
                @Override
                public void bind(SQLStatement q, BindJson bind, String jsonString) {
                    try {
                        PGobject data = new PGobject();
                        data.setType("jsonb");
                        data.setValue(jsonString);
                        q.bind(bind.value(), data);                        
                    } catch (SQLException ex) {
                        throw new IllegalStateException("Error Binding JSON",ex);
                    }
                }
            };
        }
    }
}

To use it, annotate the json parameter with the new annotation:

@SqlUpdate("insert into my_table (id,data) VALUES (:id,:data)")
void insertJson(@Bind("id") int id, @BindJson("data") String jsonString);

And that’s it; it just works.

Querying json dynamically

I had a requirement where the parameter supplied to the query was the name of the json element to return. For example, consider the json below. I wanted to be able to paramterise a query to return either one of the key values.

{
   "element": {
      "key1": "value1",
      "key2": "value2",
      "key3": "value3"
   }
}

Using raw JDBC it was possible (although not very pretty) to concatenate a suitable sql statement and then execute it:

String sql = "select data->'element1'->'" + subKeyName + "' as value from mytable"
...

This isn’t possible when the SQL string is specified as a JDBI annotation. I found some useful Postgres json processing functions, including jsonb_extract_path_text which allows you to bind parameters normally:

@SqlQuery("select jsonb_extract_path_text(data,'element1',:subKeyName) as value from mytable")
List<String> getSubKey(@Bind("subKeyName") String subKeyName)

So far I haven’t come across any other issues using JDBI with a PostgreSQL JSON data store. I’m looking forward to trying out the new jsonb functionality in PostgreSQL 9.5 which supports writing partial updates to json fields, yippee!

My First Groovy DSL

I decided to have a go at automating the management of our Jenkins jobs. Basically most of our projects have similar builds, and I want to keep the job configs standard – they tend to diverge when we manage them via the UI. I’ve previously played with the Jenkins API using HttpBuilder to post XML config files. My DSL doesn’t do anything clever to generate the XML, but it makes the Jenkins config very readable:

jenkins {   
    url = "http://jenkins.url/"
    username = "jenkins"
    apiKey = "asdfadsgdsfshgdsfg"

    buildGroup {
        name = "Maven Builds"
        xml {
            feature = "release/xml/maven-test-config.xml"
            develop = "release/xml/maven-install-config.xml"
        }
        repos {
            project1 = "ssh://git.repo/project1.git"
            project2 = "ssh://git.repo/project2.git"
        }
    }

    buildGroup {
        name = "Grails Builds"
        xml {
            feature = "release/xml/grails-test-config.xml"
            develop = "release/xml/grails-install-config.xml"
        }
        repos {
            project4 = "ssh://git.repo/project4.git"
            project5 = "ssh://git.repo/project5.git"
        }
    }

}

I think this is so easy to read.

I’m not really sure if I’ve gone the right way about writing this, but here goes:

First, I have a main method, which parses the file. It binds the “jenkins” method to an instance of my JenkinsDsl class.

    Binding binding = new Binding();
    JenkinsDsl jenkinsDsl = new JenkinsDsl();
    binding.setVariable("jenkins",jenkinsDsl);
    GroovyShell shell = new GroovyShell(binding);
    Script script = shell.parse(file);
    script.run();

The JenkinsDsl class overrides the `call` method, and uses `methodMissing` to define how we handle the buildGroups. The really cool bit is I didn’t have to write anything special to get the url, username and password from the jenkins config file – groovy is automatically calling a ‘setProperty’ method on the delegate.

class JenkinsDsl {

    def call(Closure cl) {
        log.info "Processing Main Configuration"

        cl.setDelegate(this);
        cl.setResolveStrategy(Closure.DELEGATE_ONLY)
        cl.call();
    }

    def url
    def apiKey
    def username

    def jenkinsHttp

    def methodMissing(String methodName, args) {

        if (methodName.equals("buildGroup")) {

            Closure closure = args[0]
            closure.setDelegate(new JenkinsBuildGroup())
            closure.setResolveStrategy(Closure.DELEGATE_ONLY)
            closure()
            closure.delegate.updateJenkins(getJenkinsHttp())
        } else {
            log.error "Unsupported option: " + methodName
        }
    }

    def getJenkinsHttp() {
        if (!jenkinsHttp) {
            jenkinsHttp = new JenkinsHttp(url, username, apiKey)
        }
        jenkinsHttp
    }   
}

The JenkinsBuildGroup class is very similar – to save time I’ve used the ‘Expando’ class to collect the xml and repository details.

class JenkinsBuildGroup {

    def name
    def xml
    def repoList

    /**
     * Use methodMissing to load the xml and repos closures
     * @param methodName
     * @param args
     */
    def methodMissing(String methodName, args) {

        if (methodName.equals("xml")) {
            xml = new Expando()
            Closure closure = args[0]
            closure.setDelegate(xml)
            closure.setResolveStrategy(Closure.DELEGATE_ONLY)
            closure()
        } else if (methodName.equals("repos")) {
            repoList = new Expando()
            Closure closure = args[0]
            closure.setDelegate(repoList)
            closure.setResolveStrategy(Closure.DELEGATE_ONLY)
            closure()
        } else {
            log.error "Unsupported option: " + methodName
        }
    }

    def updateJenkins(def jenkinsHttp) {
        xml.getProperties().each { String jobType, String filePath ->

            def configXml = new File(filePath).text

            repoList.getProperties().each { String jobName, String repo ->
                String jobXml = configXml.replaceAll("REPOSITORY_URL", repo)
                jenkinsHttp.createOrUpdate(jobName + "-" + jobType, jobXml)
            }
        }
    }
}

To actually update jenkins, I substitute the correct repository in the template XML file and post it to the Jenkins API. The JenkinsHttp class is just a wrapper for HTTP Builder. It checks if a job exists so that it can correctly create or update.

I’m not sure if this is the right way to go about writing a groovy DSL, but it works! And once I’ve finalised the template XML files, Jenkins is going to be a vision of consistency.

Groovy Grails Exchange 2013

This week I attended the 2013 Groovy Grails Exchange, organised by Skills Matter (with whom I’ve also been lucky enough to attend a couple of courses over the years).

Finally got time to jot down a few thoughts – I came back with an almost full notebook, but these are the highlights:

1. What a great conference! Skills Matter did an amazing job of hosting. The speakers were varied and interesting. It was fantastically motivating to be amongst all these people who are actually pushing the technology forwards. As I work in a very small dev team (out in the sticks) I found it really interesting to see what goes on in the rest of the world.

2. Must. Learn. Spock.

3. Must also get back on board with Geb. I dabbled a little a while ago, and now that we’re about to start a new Grails project, functional testing with Spock & Geb has to be central to it. Lots of food for thought in the DevQA talk by Alvaro Sanchez-Mariscal, applicable even though we don’t have a dedicated QA team for our internal dev.

4. Forces on code – what makes code “good” depends on the context. Common sense really but well illustrated in the talk by by David the Coder

5. Do I need to learn a JavaScript framework? I enjoyed the very persuasive talk on Developing SPI Applications by Alvaro Sanchez-Mariscal. Separating the front end and back end into independent apps makes a lot of sense especially if you have dedicated UI developers. I’m not sure we have the resources to move away from GSPs just yet. But I’m certainly going to have a browse around ToDoMVC to get an idea of the options.

6. Jeff Brown’s live coding demo of Metaprogramming With The Groovy Runtime was a great refresher – nothing particularly that hadn’t been covered on the Groovy course I attended last year, but a reminder that I am pretty guilty of just writing Java like code inside .groovy files and not taking full advantage of groovy’s awesomeness.

7. The “Open Source and You” session by Peter Ledbrook made me think a bit more deeply about Open Source software – what to expect from it, the costs involved, how to manage a successful open source project. I’m definitely motivated to get more involved.

8. Gradle for deployment? @danveloper’s talk on Groovy for Sysadmins gave me lots to think about. I doubt I’ll ever end up hacking the kernel, but I like the software centric approach to deploying and maintaining servers.

9. Must pay more attention to release notes and road maps. New and upcoming versions of Groovy and Grails have some great new features that I’m looking forward to using. Changes notes for Groovy 2.2 and Grails 2.3 definitely worth a look. Also looking forward to plugin and build system changes in Grails 3.0 some time in 2014.

10. And finally, how come all these Open Source aficionados use Macs?! I’ve not got an iAnything yet and don’t plan to. Linux rocks :)

Anyway, in summary, well worth going for anyone working with Groovy or Grails. Podcasts of all the talks are available on the Skills Matter website, but I think attending in real is a fantastic opportunity to absorb knowledge from an enthusiastic and knowledgeable crowd, and worth every penny. Better get my early bird ticket for next year…

Updated JIRA sub-task sorting plugin

Following some user feedback, I’ve made some changes to the JIRA plugin I wrote for dragging sub-tasks, first mentioned in this post Jira web-resource plugin to drag and drop subtasks.

If you’ve used it, you may have found it annoying that it was actually too easy to drag subtasks, causing page refreshes when they weren’t wanted! Copying and pasting from the subtask list was impossible. You could also drag issues in the Issue Navigator, although it didn’t achieve anything.

Just a few teething problems then.

First I put a delay in the jQuery sortable options. I also stopped the page from submitting if the list order had not been changed. This was a definite improvement, but users still wanted to copy text from the list, which wasn’t possible. So now I’ve limited the “drag handle” to the cell in the table where the up and down arrows appear for sorting. You can only drag if you click in that area, anywhere else won’t work. It’s still not ideal, as it’s not immediately obvious how to drag. Better ideas welcomed!

You can get the updated source from github: https://github.com/anorakgirl/subtask-dragger/releases/tag/v0.6.
Or if you don’t feel like packaging it yourself, there’s a jar for download here: subtask-dragger-0.6.jar. USE AT YOUR OWN RISK!

Issue Security in Jira sub-tasks and the Script Runner plugin

I’ve been trying to do some advanced JIRA configuration recently, with mixed success.

The requirement was to have issues in a project which could be viewed only by a specific user (not necessarily the assignee).

I discovered that “Issue Security Schemes” can have levels which allow access based on a “User Custom Field Value”. Fantastic. I added a custom field, let’s call it “Employee”, and added it to an Issue Security Level. Applied to the project and bingo, users can only see issues where they have been entered into the “Employee” field.

Too easy. I added a subtask to an issue where I was the employee. No error message, but no subtask appeared either.

My understanding was that you could not set issue security on subtasks, because they inherit the issue security of the parent. I assumed the intention of this was that if you could see an issue, you should be able to see its children. But no – they specifically inherit the issue security setting from the parent, rather than its visibility – as confirmed in Atlassian Answers. So because the sub-task does not have an “Employee” field, I can’t see it. Bother.

Of course, I could use a workflow post-function so that when subtasks are created, the employee field is set to that of the parent. But what if the value is changed in the parent? It’s a nuisance to have to change all the sub-tasks too.

Time to investigate the Script Runner plugin.

Here’s the code for my Scripted field, which gets the value of the Employee field in the parent issue.

import com.atlassian.jira.ComponentManager
import com.atlassian.jira.issue.CustomFieldManager
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.issue.fields.CustomField

def componentManager = ComponentManager.getInstance()
def parentIssue = issue.getParentObject();
CustomFieldManager cfManager = ComponentManager.getInstance().getCustomFieldManager()


CustomField customField = cfManager.getCustomFieldObject(99999)
parentIssue.getCustomFieldValue(customField)

It is almost too easy. I think it may be the answer to some of the other requests I’ve had for custom fields in JIRA, but unfortunately it doesn’t solve this problem. The calculated value appears perfectly on the page when viewing a sub-task. But, when I go to the “Issue Security Level” and click in the “User Custom Field Value” drop down, it doesn’t appear! It’s configured with the “User Picker” Searcher and template, so I don’t think there’s much more I can do.

Back to the drawing board…

Unexpected side effects using Groovy MockFor in a Grails test

Recently, I was trying to write a Unit Test for a Grails method which instantiates an HTTPBuilder object within a method.

I couldn’t use the normal Grails ‘mockFor’ syntax, because the HTTBuilder was not injected or accessible from outside the class. No problem, because standard Groovy MockFor is also available.

With a bit of help from google (http://stackoverflow.com/questions/9101084/groovy-httpbuilder-mocking-the-response) I came up with the test below.

def testDoSomething() {
    def requestDelegate = [response: [:], uri: [:]]

    def mockHttpBuilder = new MockFor(groovyx.net.http.HTTPBuilder)
			
    mockHttpBuilder.demand.request { Method method, Closure body -&amp;gt;
        body.delegate = requestDelegate
        body.call()
        requestDelegate.response.success() // make it call success
    }

    mockHttpBuilder.use {
        // All calls to HTTPBuilder will be intercepted within this block of code.      
        MyClient myClient = new MyClient();
        assert myClient.doSomething(&amp;quot;Some Param&amp;quot;)
    }
}

This works really well in isolating the code under test from the HTTP request.

However after spending a bit of time working on this test, and getting it to pass, I ran test-app on the entire project, and was very confused to see that some totally unrelated tests had begin to fail. Grails was reporting errors like “No more calls to ‘get’ expected at this point.” in tests where I had used no mocking at all. I was bewildered.

Eventually I came across GRAILS-8535. Although unit tests are supposed to be just that, it seems that grails continues to use the proxy in subsequent tests. This appears to be fixed in Grails 2.2.3, but as of the time of writing, this wasn’t released.

Luckily, it is possible to manually reset the MetaClass in the Grails MetaClassRegistry at the end of the test.

At the start of the test which uses MockFor, I record what the class was originally:

MetaClassRegistry registry = GrailsMetaClassUtils.getRegistry()
def originalMetaClass = registry.getMetaClass(HTTPBuilder)

And at the end of the test, set it back:

MetaClassRegistryCleaner.addAlteredMetaClass(HTTPBuilder, originalMetaClass) 

Best method for functional testing in grails

or ‘why isn’t there more quality control on the internet…?’

This week I tried to add some automated functional tests to a grails project. (See my post on agile best practices; we agreed that our definition of ‘Done’ should definitely include functional tests)

There are a few plugins for grails which look like they could do the job.

My final selection criteria turned out to be: the only one which I could get working within less than half a day.

Canoo Webtest

http://grails.org/plugin/webtest

I came away from a recent grails training course thinking that this was the de-facto standard for functional testing. In fact, I have since then used the standalone Webtest tool to test a legacy PHP application, with good success.

However, trying to get going in grails was a different matter.

UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: net.sourceforge.htmlunit#htmlunit;2.8-SNAPSHOT: not found
::::::::::::::::::::::::::::::::::::::::::::::

I found the following bug report, and following the advice to specify the htmlunit version did allow me to get started.

http://jira.grails.org/browse/GPWEBTEST-72

But I was already disappointed.  If I install a plugin using the grails install-plugin command, I think there should be some level of assurance that the plugin version to be installed will be a tested version, dependent on NON-snapshot versions of any required libraries.

Grails functional-test plugin

I then realised that other plugins were available, as mentioned in the grails functional testing user guide.
The documentation for this one defines the dependency as

compile ":functional-test:2.0.RC1"

Again, I do not want the install-plugin command to install a Release Candidate. I want the tested and released version. I tried using an older version as well, but gave up on this pretty quickly too.

selenium-rc

See: http://grails.org/plugin/selenium-rc
The grails plugin page states:

This plugin is no longer maintained. Consider looking at Geb.

So this is what I did.

Geb

After a bad day, I was pleasantly surprised to get going quickly with Geb. And I really like the separation between modelling the available functions on the page and calling them in tests. Although the plugin version is only 0.9.0 it feels like it is current and will be maintained.

I’m surprised at how much trouble I had getting going with functional testing. It feels like there is a lack of quality control on the publicly available Grails plugins, and little attempt to ensure versions are compatible with other standard plugins such as jQuery.

I’ll be spending some more time learning to use Geb in the near future, so I’ll try and report my progress…

Grails unit tests failing on jenkins

I spent a while confused today, when some fairly straightforward unit tests were running fine locally, but failing on our continuous integration server, jenkins.

The error was occurring in the setUp method, annotated @Before, where we were setting a flash variable prior to running tests.

java.lang.NullPointerException: Cannot invoke method getFlashScope() on null object
at grails.test.mixin.web.ControllerUnitTestMixin.getFlash(ControllerUnitTestMixin.groovy:159)

After much head scratching I came across this link
http://www.mopri.de/2013/grails-unit-testing-and-a-little-fun-with-before/
which explains a similar problem.

It appears that because the Controller uses a Mixin, it effectively extends ControllerUnitTestMixin which is not abstract and also has a method annotated with @Before. Junit does not guarantee the order in which the two methods annotated with @Before will be run (See http://junit.sourceforge.net/doc/faq/faq.htm#tests_2)

If the Mixin method does not run before my setUp method, the flash scope has not been mocked, hence the failure.

The solution suggested in the post is to call super.bindGrailsWebRequest() in the setUp method. In our case it was simple enough to set the flash variable within the individual tests, instead of the setUp method.

Immutability and Collections.unmodifiableList

Or: why is my list still mutable?

Today I learnt a new thing about Collections.unmodifiableList.

First, I noted that Sonar wasn’t raising a violation where I thought it should.

Take the sample class below

public final class MyImmutableClass {
    
    private final List<String> myList;

    public MyImmutableClass(List<String>myList) {
        this.myList = myList;
    }

    public List<String> getMyList() {
        return myList;
    }
    
}

I would expect sonar to raise the following:

Malicious code vulnerability - May expose internal representation by incorporating reference to mutable object (Findbugs: EI_EXPOSE_REP2)
Malicious code vulnerability - May expose internal representation by returning reference to mutable object (Findbugs: EI_EXPOSE_REP)

But it didn’t. This is a mystery in itself, as I am sure it has picked me up on this before.

Anyway, no problem, I wrote a test for the class myself:

/**
     * Test of getMyList method, of class MyImmutableClass.
     */
    @Test
    public void testGetMyList() {
        List<String> list = new ArrayList<String>();
        list.add("Potato");
        list.add("Apple");
        
        MyImmutableClass instance = new MyImmutableClass(list);
        
        assertEquals(2,instance.getMyList().size());
        
        //check for immutability
        list.add("item 3");
        assertEquals(2,instance.getMyList().size());
        
        try {
            instance.getMyList().add("Message3");
            fail("Should not be possible");
        } catch (Exception ex) {
            assertTrue( ex instanceof UnsupportedOperationException);
        }
    }

Of course, the test fails with the code as it is.

So I modified it to this:

public final class MyImmutableClass {
    
    private final List<String> myList;

    public MyImmutableClass(List<String>myList) {
        this.myList = Collections.unmodifiableList(myList);
    }

    public List<String> getMyList() {
        return Collections.unmodifiableList(myList);
    }
    
}

And it STILL failed. I was very confused. How come Collections.unmodifiableList in the constructer wasn’t stopping the list inside the class from changing?

It took some googling to find the answer. For example: this stackoverflow post.
If you pay proper attention to the javadoc for Collections.unmodifiableList, it makes sense.


     * Returns an unmodifiable view of the specified list.  This method allows
     * modules to provide users with "read-only" access to internal
     * lists.  Query operations on the returned list "read through" to the
     * specified list, and attempts to modify the returned list, whether
     * direct or via its iterator, result in an
     * UnsupportedOperationException.

So, this just wraps the original list inside an unmodifiable view. I can’t modify the one inside my class, but if I modify the one I used to create the class, the view reflects the update.

The correct way to make my class immutable is:

public final class MyImmutableClass {
    
    private final List<String> myList;

    public MyImmutableClass(List<String>myList) {
        this.myList = Collections.unmodifiableList(new ArrayList<String>(myList));
    }

    public List<String> getMyList() {
        return Collections.unmodifiableList(myList);
    }
    
}

Phew. As an afterthought, because sonar and findbugs did not pick this up, I’m thinking of taking a look at this: http://mutability-detector.blogspot.co.uk/. We like to make all classes immutable, unless there is a good reason not to. It would be interesting to see what else has slipped through.