jenkins plugin and grails commandline exit code

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

jenkins plugin and grails commandline exit code

hanswesterbeek
Hi all,

I have come across an issue that I think can only be solved by making
modifications to Grails itself, therefore this post. I made a post
about this on the user list previously but that's the wrong place, i
have found out.

I wanted to do some work on https://issues.jenkins-ci.org/browse/JENKINS-8535

Basically, the plugin marks a build as 'failed' instead of 'unstable'
when 1 or more tests fail, which is incorrect. After some local
debugging (well, ok, System.out.println :) ) of the plugin I have
found out that the grails test-app command returns the following:

Case A: sucessfull build, all tests pass: 0
Case B: one or more tests fail: 1
Case C: compiler error: 1

The commandline is: grails -test-app -xml --non-interactive -unit
Grails is 1.3.7

The problem is that from the jenkins plugin's perspective it can't
tell the difference between case B and C. For the purposes of using
Jenkins in your project those are very different though, only a case
like C warrants a build to be marked as 'FAILED'. Case of B means it
should be marked 'UNSTABLE'.

I think it's pretty important for all of us that we have a good plugin
for Jenkins and this issue is a royal pain for project that take
continuous integration and code quality metrics seriously.

My questions:
* Who else thinks this is a valid issue?
* Is there any other way I the plugin would be able to distinguish the
results? I would like to implement that and make a pull-request.

Any pointers into the right direction are appreciated...

Cheers,
Hans

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: jenkins plugin and grails commandline exit code

hanswesterbeek
Hmm... seems that if I am somehow able to call the method 'getFailedTests' which is in scripts/_GrailsTest.groovy, i could figure out, post-build wether to tell jenkins if the build actually failed, eg:

IF build returns 0, just report success to jenkins
if build result = 1, check for test failures, if those exist, return UNSTABLE to jenkins,

That would leave one more problem: situation where test failures exist but the build has also FAILED in say, the 'war' command that the user may have invoked after 'test-app'

Any thoughts?
F'k
Reply | Threaded
Open this post in threaded view
|

Re: jenkins plugin and grails commandline exit code

F'k
I think this is a valid suggestions, because its blocking proper integration with jenkins.
UNSTABLE should be fine. As long as that doesn't override the failure of other commands. Is there a possibility that you could check for compilation errors or other objects indicating the failure of commands?
If that wouldn't be possible then we could at least return some other code when test failure exists and on jenkins we would have a separate steps for test/war commands as a workaround (then we would separate failures from unstables)
Reply | Threaded
Open this post in threaded view
|

Re: jenkins plugin and grails commandline exit code

alxndrsn
In reply to this post by hanswesterbeek
hanswesterbeek wrote
Hi all,

I have come across an issue that I think can only be solved by making
modifications to Grails itself, therefore this post. I made a post
about this on the user list previously but that's the wrong place, i
have found out.

I wanted to do some work on https://issues.jenkins-ci.org/browse/JENKINS-8535

Basically, the plugin marks a build as 'failed' instead of 'unstable'
when 1 or more tests fail, which is incorrect. After some local
debugging (well, ok, System.out.println :) ) of the plugin I have
found out that the grails test-app command returns the following:

Case A: sucessfull build, all tests pass: 0
Case B: one or more tests fail: 1
Case C: compiler error: 1

The commandline is: grails -test-app -xml --non-interactive -unit
Grails is 1.3.7

The problem is that from the jenkins plugin's perspective it can't
tell the difference between case B and C. For the purposes of using
Jenkins in your project those are very different though, only a case
like C warrants a build to be marked as 'FAILED'. Case of B means it
should be marked 'UNSTABLE'.

I think it's pretty important for all of us that we have a good plugin
for Jenkins and this issue is a royal pain for project that take
continuous integration and code quality metrics seriously.

My questions:
* Who else thinks this is a valid issue?
* Is there any other way I the plugin would be able to distinguish the
results? I would like to implement that and make a pull-request.
Hi Hans,

I also think this is a valid issue and have added some thought on implementation to the Jenkins JIRA ticket.

It seems like it would be very simple to make this change in the Grails core though - to return a different error code when there is a test failure, it seems like you just need to modify the number `1` at https://github.com/grails/grails-core/blob/master/scripts/_GrailsTest.groovy#L226

Is this kind of change a possibility for Grails?  I can see it might be viewed as "breaking", but as currently all non-zero exit codes are `1`, I'd expect most code relying on this will be checking for non-zero rather than explicitly for `1`.

Other possible fixes that come to mind:
* processing the text output of the grails test-app command (as mentioned on the JIRA ticket)
* override the `allTests` target in `_GrailsTest`, call the parent and change the return value if it's `1`??
* override the test-app script???

All thoughts welcome!

Alex

P.S. it strikes me that at least the basis of this discussion (are `unstable` builds a useful distinction in Jenkins and should jenkins-grails-plugin support that) would be suited to the user mailing list, but not sure how this could be moved there and if it should.
Reply | Threaded
Open this post in threaded view
|

Re: jenkins plugin and grails commandline exit code

pledbrook
> It seems like it would be very simple to make this change in the Grails core
> though - to return a different error code when there is a test failure, it
> seems like you just need to modify the number `1` at
> https://github.com/grails/grails-core/blob/master/scripts/_GrailsTest.groovy#L226
>
> Is this kind of change a possibility for Grails?  I can see it might be
> viewed as "breaking", but as currently all non-zero exit codes are `1`, I'd
> expect most code relying on this will be checking for non-zero rather than
> explicitly for `1`.

I don't see a problem with  this change, and this is what exit codes
are for (although we should be careful overloading 'standard' shell
exit codes).

Peter

--
Peter Ledbrook
Developer Advocate
VMware

t: @pledbrook

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: jenkins plugin and grails commandline exit code

Jeff Scott Brown

On Oct 11, 2012, at 5:36 AM, Peter Ledbrook wrote:

>> It seems like it would be very simple to make this change in the Grails core
>> though - to return a different error code when there is a test failure, it
>> seems like you just need to modify the number `1` at
>> https://github.com/grails/grails-core/blob/master/scripts/_GrailsTest.groovy#L226
>>
>> Is this kind of change a possibility for Grails?  I can see it might be
>> viewed as "breaking", but as currently all non-zero exit codes are `1`, I'd
>> expect most code relying on this will be checking for non-zero rather than
>> explicitly for `1`.
>
> I don't see a problem with  this change, and this is what exit codes
> are for (although we should be careful overloading 'standard' shell
> exit codes).
>

+1



JSB

--
Jeff Brown
SpringSource
http://www.springsource.com/

Autism Strikes 1 in 166
Find The Cause ~ Find The Cure
http://www.autismspeaks.org/


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: jenkins plugin and grails commandline exit code

alxndrsn
>>> Is this kind of change a possibility for Grails?  I can see it might be
>>> viewed as "breaking", but as currently all non-zero exit codes are `1`, I'd
>>> expect most code relying on this will be checking for non-zero rather than
>>> explicitly for `1`.
>>
>> I don't see a problem with  this change, and this is what exit codes
>> are for (although we should be careful overloading 'standard' shell
>> exit codes).
>>
>
> +1

Great, have created on JIRA http://jira.grails.org/browse/GRAILS-9481;
vote there!

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email