What's up with /srv directory & Grails?? - failed builds

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

What's up with /srv directory & Grails?? - failed builds

Krystian
Hi,

I'm having a strangest thing today!
We are using Bamboo for CI and after I pushed my changes to git the build failed. So I went to investigate, curious, as tests went fine on my machine.
The error was very strange, far far away from functionality I have changed.
The error comes from the Converters:
org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: java.lang.NullPointerException: Cannot get property 'id' on null object

Please note: since last build there were no changes to Grails version/plugins/libs/server. None at all.

The very same source builds and goes through tests on other machines just fine.

So I decided to login onto the box where bamboo runs, I switched to bamboo system user, cloned the repository and tried to run the build manually.
Same error.

Now I went back to my user on the box and did the same thing and... build went fine!

Went back to bamboo user, removed .grails and .ivy2 directories, grails test-app, test failed. Same as before.

So i've said ok... maybe there are other things I forgot to remove.
I switched to a user which runs Jira, cloned the repo and ran grails test-app.
Test failed. Same thing as with bamboo user.

With nothing better coming to my mind I switched to a newly created normal user, cloned repo, ran test-app. It went fine!

So I had 2:2 failed:successful.

Now a little details about my setup.
All the apps like Jira/bamboo run from /srv/<app>-home directories and these are home directories for users running those apps.
Normal users reside in /home directory.

Because I don't like draws I decided to go for another atlassian app running user - crowd this time. Build failed.
I decided to move the cloned repo to /tmp directory and run the build from there. Well... it was successful.
I've tried moving it to /srv/test directory and ... it failed.
Tried renaming it o /srv/test124 and... it failed again.
Tried moving it to /test124 and... it succeeded.

There seems to be something wrong with my /srv directory O_o

I have no clue about what to do now with it. I'm going to get some sleep and get back to it tomorrow.

I would appreciate some suggestions as to what could this be.

Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

ld@ldaley.com
I don't quite understand what is going on for you, but I'd seriously consider setting the following in BuildConfig.groovy

grails.project.work.dir = "target"

This will prevent Grails from using your user home folder for storing plugins, .class files etc. which is really problematic on a busy CI server.

On 13/01/2011, at 11:30 AM, Krystian wrote:

>
> Hi,
>
> I'm having a strangest thing today!
> We are using Bamboo for CI and after I pushed my changes to git the build
> failed. So I went to investigate, curious, as tests went fine on my machine.
> The error was very strange, far far away from functionality I have changed.
> The error comes from the Converters:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> java.lang.NullPointerException: Cannot get property 'id' on null object
>
> Please note: since last build there were no changes to Grails
> version/plugins/libs/server. None at all.
>
> The very same source builds and goes through tests on other machines just
> fine.
>
> So I decided to login onto the box where bamboo runs, I switched to bamboo
> system user, cloned the repository and tried to run the build manually.
> Same error.
>
> Now I went back to my user on the box and did the same thing and... build
> went fine!
>
> Went back to bamboo user, removed .grails and .ivy2 directories, grails
> test-app, test failed. Same as before.
>
> So i've said ok... maybe there are other things I forgot to remove.
> I switched to a user which runs Jira, cloned the repo and ran grails
> test-app.
> Test failed. Same thing as with bamboo user.
>
> With nothing better coming to my mind I switched to a newly created normal
> user, cloned repo, ran test-app. It went fine!
>
> So I had 2:2 failed:successful.
>
> Now a little details about my setup.
> All the apps like Jira/bamboo run from /srv/<app>-home directories and these
> are home directories for users running those apps.
> Normal users reside in /home directory.
>
> Because I don't like draws I decided to go for another atlassian app running
> user - crowd this time. Build failed.
> I decided to move the cloned repo to /tmp directory and run the build from
> there. Well... it was successful.
> I've tried moving it to /srv/test directory and ... it failed.
> Tried renaming it o /srv/test124 and... it failed again.
> Tried moving it to /test124 and... it succeeded.
>
> There seems to be something wrong with my /srv directory O_o
>
> I have no clue about what to do now with it. I'm going to get some sleep and
> get back to it tomorrow.
>
> I would appreciate some suggestions as to what could this be.
>
>
> --
> View this message in context: http://grails.1312388.n4.nabble.com/What-s-up-with-srv-directory-Grails-failed-builds-tp3215087p3215087.html
> Sent from the Grails - user mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

mickdavis

Out of curiosity Luke,  why is that problematic?   I've never used a CI server but I know I will someday soon.   Is there by any chance some docs that these sorts of recommendations that you're aware of?   Or perhaps a book?

Thanks.

Mick

On Jan 12, 2011 9:04 PM, "Luke Daley" <[hidden email]> wrote:
> I don't quite understand what is going on for you, but I'd seriously consider setting the following in BuildConfig.groovy
>
> grails.project.work.dir = "target"
>
> This will prevent Grails from using your user home folder for storing plugins, .class files etc. which is really problematic on a busy CI server.
>
> On 13/01/2011, at 11:30 AM, Krystian wrote:
>
>>
>> Hi,
>>
>> I'm having a strangest thing today!
>> We are using Bamboo for CI and after I pushed my changes to git the build
>> failed. So I went to investigate, curious, as tests went fine on my machine.
>> The error was very strange, far far away from functionality I have changed.
>> The error comes from the Converters:
>> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
>> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
>> java.lang.NullPointerException: Cannot get property 'id' on null object
>>
>> Please note: since last build there were no changes to Grails
>> version/plugins/libs/server. None at all.
>>
>> The very same source builds and goes through tests on other machines just
>> fine.
>>
>> So I decided to login onto the box where bamboo runs, I switched to bamboo
>> system user, cloned the repository and tried to run the build manually.
>> Same error.
>>
>> Now I went back to my user on the box and did the same thing and... build
>> went fine!
>>
>> Went back to bamboo user, removed .grails and .ivy2 directories, grails
>> test-app, test failed. Same as before.
>>
>> So i've said ok... maybe there are other things I forgot to remove.
>> I switched to a user which runs Jira, cloned the repo and ran grails
>> test-app.
>> Test failed. Same thing as with bamboo user.
>>
>> With nothing better coming to my mind I switched to a newly created normal
>> user, cloned repo, ran test-app. It went fine!
>>
>> So I had 2:2 failed:successful.
>>
>> Now a little details about my setup.
>> All the apps like Jira/bamboo run from /srv/<app>-home directories and these
>> are home directories for users running those apps.
>> Normal users reside in /home directory.
>>
>> Because I don't like draws I decided to go for another atlassian app running
>> user - crowd this time. Build failed.
>> I decided to move the cloned repo to /tmp directory and run the build from
>> there. Well... it was successful.
>> I've tried moving it to /srv/test directory and ... it failed.
>> Tried renaming it o /srv/test124 and... it failed again.
>> Tried moving it to /test124 and... it succeeded.
>>
>> There seems to be something wrong with my /srv directory O_o
>>
>> I have no clue about what to do now with it. I'm going to get some sleep and
>> get back to it tomorrow.
>>
>> I would appreciate some suggestions as to what could this be.
>>
>>
>> --
>> View this message in context: http://grails.1312388.n4.nabble.com/What-s-up-with-srv-directory-Grails-failed-builds-tp3215087p3215087.html
>> Sent from the Grails - user mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>> http://xircles.codehaus.org/manage_email
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
> http://xircles.codehaus.org/manage_email
>
>
Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

ld@ldaley.com

On 13/01/2011, at 12:58 PM, Mick Davis wrote:

> Out of curiosity Luke,  why is that problematic?

By default, Grails stores working files in ~/.grails/«version»/«projectname»

If you have either two projects with the same names or different build variants of the same project and you have concurrent builds then the builds can stomp on each other. If you store all working files within the checkout that your CI server is using there is no chance of this happening.

>   I've never used a CI server but I know I will someday soon.   Is there by any chance some docs that these sorts of recommendations that you're aware of?   Or perhaps a book?

Not that I know of, but I am sure there are many non Grails specific resources out there that explain these common problems.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Krystian
In reply to this post by ld@ldaley.com
Hi Luke,

thanks for the reply.
It's not a busy server. Just this one application, no concurrent builds at the moment.
Like I've mentioned in my first mail/post, I have removed the .grails and .ivy2 directory to test if they are the cause of issue.

I went and added:
grails.project.work.dir = "/srv/bamboo-home/test"
to the BuildConfig.groovy but it didn't help :/

It was ~70th build on the server when it happened and settings didn't change before that.

Any other thoughts?

Thanks,
Krystian

Luke Daley wrote
I don't quite understand what is going on for you, but I'd seriously consider setting the following in BuildConfig.groovy

grails.project.work.dir = "target"

This will prevent Grails from using your user home folder for storing plugins, .class files etc. which is really problematic on a busy CI server.

On 13/01/2011, at 11:30 AM, Krystian wrote:

>
> Hi,
>
> I'm having a strangest thing today!
> We are using Bamboo for CI and after I pushed my changes to git the build
> failed. So I went to investigate, curious, as tests went fine on my machine.
> The error was very strange, far far away from functionality I have changed.
> The error comes from the Converters:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> java.lang.NullPointerException: Cannot get property 'id' on null object
>
> Please note: since last build there were no changes to Grails
> version/plugins/libs/server. None at all.
>
> The very same source builds and goes through tests on other machines just
> fine.
>
> So I decided to login onto the box where bamboo runs, I switched to bamboo
> system user, cloned the repository and tried to run the build manually.
> Same error.
>
> Now I went back to my user on the box and did the same thing and... build
> went fine!
>
> Went back to bamboo user, removed .grails and .ivy2 directories, grails
> test-app, test failed. Same as before.
>
> So i've said ok... maybe there are other things I forgot to remove.
> I switched to a user which runs Jira, cloned the repo and ran grails
> test-app.
> Test failed. Same thing as with bamboo user.
>
> With nothing better coming to my mind I switched to a newly created normal
> user, cloned repo, ran test-app. It went fine!
>
> So I had 2:2 failed:successful.
>
> Now a little details about my setup.
> All the apps like Jira/bamboo run from /srv/<app>-home directories and these
> are home directories for users running those apps.
> Normal users reside in /home directory.
>
> Because I don't like draws I decided to go for another atlassian app running
> user - crowd this time. Build failed.
> I decided to move the cloned repo to /tmp directory and run the build from
> there. Well... it was successful.
> I've tried moving it to /srv/test directory and ... it failed.
> Tried renaming it o /srv/test124 and... it failed again.
> Tried moving it to /test124 and... it succeeded.
>
> There seems to be something wrong with my /srv directory O_o
>
> I have no clue about what to do now with it. I'm going to get some sleep and
> get back to it tomorrow.
>
> I would appreciate some suggestions as to what could this be.
>
>
> --
> View this message in context: http://grails.1312388.n4.nabble.com/What-s-up-with-srv-directory-Grails-failed-builds-tp3215087p3215087.html
> Sent from the Grails - user mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Sven Arnold-3
In reply to this post by Krystian
Is /srv on a separate (logical) volume?
You could have a corrupted filesystem or ran out of disk space...

Sven

Am Mittwoch, den 12.01.2011, 17:30 -0800 schrieb Krystian:

> Hi,
>
> I'm having a strangest thing today!
> We are using Bamboo for CI and after I pushed my changes to git the build
> failed. So I went to investigate, curious, as tests went fine on my machine.
> The error was very strange, far far away from functionality I have changed.
> The error comes from the Converters:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> java.lang.NullPointerException: Cannot get property 'id' on null object
>
> Please note: since last build there were no changes to Grails
> version/plugins/libs/server. None at all.
>
> The very same source builds and goes through tests on other machines just
> fine.
>
> So I decided to login onto the box where bamboo runs, I switched to bamboo
> system user, cloned the repository and tried to run the build manually.
> Same error.
>
> Now I went back to my user on the box and did the same thing and... build
> went fine!
>
> Went back to bamboo user, removed .grails and .ivy2 directories, grails
> test-app, test failed. Same as before.
>
> So i've said ok... maybe there are other things I forgot to remove.
> I switched to a user which runs Jira, cloned the repo and ran grails
> test-app.
> Test failed. Same thing as with bamboo user.
>
> With nothing better coming to my mind I switched to a newly created normal
> user, cloned repo, ran test-app. It went fine!
>
> So I had 2:2 failed:successful.
>
> Now a little details about my setup.
> All the apps like Jira/bamboo run from /srv/<app>-home directories and these
> are home directories for users running those apps.
> Normal users reside in /home directory.
>
> Because I don't like draws I decided to go for another atlassian app running
> user - crowd this time. Build failed.
> I decided to move the cloned repo to /tmp directory and run the build from
> there. Well... it was successful.
> I've tried moving it to /srv/test directory and ... it failed.
> Tried renaming it o /srv/test124 and... it failed again.
> Tried moving it to /test124 and... it succeeded.
>
> There seems to be something wrong with my /srv directory O_o
>
> I have no clue about what to do now with it. I'm going to get some sleep and
> get back to it tomorrow.
>
> I would appreciate some suggestions as to what could this be.
>
>



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Krystian
Hi Sven,

thanks for interest.

Yes, it is on a separate logical volume, however there's a lot of space in there:
/dev/cciss/c0d0p3     248G   60G  176G  26% /srv

Also I've unmounted the volume and tried running fsck [with force as filesystem was clean] and it went without reporting any issues.

After mounting the volume again I've run grails test-app and unfortunately it fails again.

Damn...

Have you got any other ideas? :/ I'm kind of running out of them :/

Thanks,
Krystian


Sven Arnold-3 wrote
Is /srv on a separate (logical) volume?
You could have a corrupted filesystem or ran out of disk space...

Sven

Am Mittwoch, den 12.01.2011, 17:30 -0800 schrieb Krystian:
> Hi,
>
> I'm having a strangest thing today!
> We are using Bamboo for CI and after I pushed my changes to git the build
> failed. So I went to investigate, curious, as tests went fine on my machine.
> The error was very strange, far far away from functionality I have changed.
> The error comes from the Converters:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> org.codehaus.groovy.grails.web.converters.exceptions.ConverterException:
> java.lang.NullPointerException: Cannot get property 'id' on null object
>
> Please note: since last build there were no changes to Grails
> version/plugins/libs/server. None at all.
>
> The very same source builds and goes through tests on other machines just
> fine.
>
> So I decided to login onto the box where bamboo runs, I switched to bamboo
> system user, cloned the repository and tried to run the build manually.
> Same error.
>
> Now I went back to my user on the box and did the same thing and... build
> went fine!
>
> Went back to bamboo user, removed .grails and .ivy2 directories, grails
> test-app, test failed. Same as before.
>
> So i've said ok... maybe there are other things I forgot to remove.
> I switched to a user which runs Jira, cloned the repo and ran grails
> test-app.
> Test failed. Same thing as with bamboo user.
>
> With nothing better coming to my mind I switched to a newly created normal
> user, cloned repo, ran test-app. It went fine!
>
> So I had 2:2 failed:successful.
>
> Now a little details about my setup.
> All the apps like Jira/bamboo run from /srv/<app>-home directories and these
> are home directories for users running those apps.
> Normal users reside in /home directory.
>
> Because I don't like draws I decided to go for another atlassian app running
> user - crowd this time. Build failed.
> I decided to move the cloned repo to /tmp directory and run the build from
> there. Well... it was successful.
> I've tried moving it to /srv/test directory and ... it failed.
> Tried renaming it o /srv/test124 and... it failed again.
> Tried moving it to /test124 and... it succeeded.
>
> There seems to be something wrong with my /srv directory O_o
>
> I have no clue about what to do now with it. I'm going to get some sleep and
> get back to it tomorrow.
>
> I would appreciate some suggestions as to what could this be.
>
>



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Ian Roberts
On 13/01/2011 10:35, Krystian wrote:
>
> Hi Sven,
>
> thanks for interest.
>
> Yes, it is on a separate logical volume, however there's a lot of space in
> there:
> /dev/cciss/c0d0p3     248G   60G  176G  26% /srv

Does the server enforce per-user quotas?

Also (long shot), have a look at df -i as well as df -h.  If a volume
has a lot of small files on it it is possible to run out of inodes
before you run out of space (at least on ext2/3/4, xfs doesn't have this
problem as it doesn't fix the maximum number of inodes up front).

Ian

--
Ian Roberts               | Department of Computer Science
[hidden email]  | University of Sheffield, UK

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Krystian
Ian, thanks for the interest.

I ran df -i and here's the output:
/dev/cciss/c0d0p3    16482304  124222 16358082    1% /srv

looks ok to me.

There are no quotas on a per user basis, besides I've copied my repository 10x already and didn't hit any limitations.

I guess it's going to be something small & stupid & somewhere else than we look... but still I can't figure out what could it be :/

Ian Roberts wrote
On 13/01/2011 10:35, Krystian wrote:
>
> Hi Sven,
>
> thanks for interest.
>
> Yes, it is on a separate logical volume, however there's a lot of space in
> there:
> /dev/cciss/c0d0p3     248G   60G  176G  26% /srv

Does the server enforce per-user quotas?

Also (long shot), have a look at df -i as well as df -h.  If a volume
has a lot of small files on it it is possible to run out of inodes
before you run out of space (at least on ext2/3/4, xfs doesn't have this
problem as it doesn't fix the maximum number of inodes up front).

Ian

--
Ian Roberts               | Department of Computer Science
i.roberts@dcs.shef.ac.uk  | University of Sheffield, UK

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Krystian
For the moment I have changed bamboo configuration and moved repository cache outside /srv and builds are successful. This one gives me headaches and I have to have this figured out soon.

Anyone got some suggestions? Something I could try?

Thanks,
Krystian

Krystian wrote
Ian, thanks for the interest.

I ran df -i and here's the output:
/dev/cciss/c0d0p3    16482304  124222 16358082    1% /srv

looks ok to me.

There are no quotas on a per user basis, besides I've copied my repository 10x already and didn't hit any limitations.

I guess it's going to be something small & stupid & somewhere else than we look... but still I can't figure out what could it be :/

Ian Roberts wrote
On 13/01/2011 10:35, Krystian wrote:
>
> Hi Sven,
>
> thanks for interest.
>
> Yes, it is on a separate logical volume, however there's a lot of space in
> there:
> /dev/cciss/c0d0p3     248G   60G  176G  26% /srv

Does the server enforce per-user quotas?

Also (long shot), have a look at df -i as well as df -h.  If a volume
has a lot of small files on it it is possible to run out of inodes
before you run out of space (at least on ext2/3/4, xfs doesn't have this
problem as it doesn't fix the maximum number of inodes up front).

Ian

--
Ian Roberts               | Department of Computer Science
i.roberts@dcs.shef.ac.uk  | University of Sheffield, UK

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

Reply | Threaded
Open this post in threaded view
|

Re: What's up with /srv directory & Grails?? - failed builds

Krystian
I have found the problem with the code.
There was a bug when associating two domain objects and one of them was not persisted to the database, thus Custom Marshaller was getting an error.
Funny thing is, this code was there for ~ 1,5 months and it only now started to act like this.
Better yet -  the strange behaviour depending on the directory app was run.
I have no clue why it acted this way, all the time I used the same codebase.

Anyway... got that sorted out.

Thanks for all the help!