How best to store uploaded files in a Grails app?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

How best to store uploaded files in a Grails app?

Ben Klein
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

sameetn
Are you managing these files as a part of your application? Or are these just additional files that are used for display etc. just curious

Sent from my iPhone

On Mar 25, 2013, at 9:28 PM, Benjamin Klein <[hidden email]> wrote:

> I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?
>
> Thanks!
>
> Ben
>
> --
> b
>
> Sent from my iPhone
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

ebiester
In reply to this post by Ben Klein

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

RE: How best to store uploaded files in a Grails app?

netwiser
For me, I stored all files in the MySql database, it has over 22G now, which causes a big headache for me. I am running the server on Amazon AWS,  I can't push the static images to the CDN or S3, this does not harm too much.  The most worst part is, every time I try to optimize or backup the database, I have to take the server offline for over 30 minutes, let my users watching a funny sorry server maintaining page..... Also I had to resize the EBS volume couple of times. 

I wish I put the static big files on hard drive from the beginning. 

James



Date: Mon, 25 Mar 2013 19:17:34 -0700
From: [hidden email]
To: [hidden email]
Subject: Re: [grails-user] How best to store uploaded files in a Grails app?

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Ben Klein
In reply to this post by ebiester
These are static always-publicly-available files. They’re displayed (and linked to) on public pages, although only uploaded by authenticated users. Basically, a file can be uploaded as part of the creation of a Publication; it can be replaced as part of an update to a Publication; and it can be deleted as part of the deletion of a Publication. All of this is done without direct involvement on the part of the user, except insofar as is required for the user to select a file for upload.

I avoid the database mainly for simplicity, and because I like to have the files in all of their file-y goodness freely available in a Web-accessible directory rather than as bytes specially gathered from the database and hand-assembled when a user requests them. It is not a Big Issue, but I do prefer to use the filesystem for files whenever possible.

To reply more directly to Sameet: These files are uploaded through the app, but are not managed by hand. Once uploaded they are indeed mostly used for, as you say, “display” purposes, and are treated as static files.

--
b

Sent from my iPhone

On Mar 25, 2013, at 9:17 PM, Eric Biesterfeld <[hidden email]> wrote:

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Alexandre Jacques
If they can be accessed by anyone, here's what I do: save these files (or move them after uploaded) to a location where you can "map" your web server (a different <Location> in Apache, for example) and let it serve these files.

Regards,

Alexandre Jacques


On Tuesday, March 26, 2013, Benjamin Klein wrote:
These are static always-publicly-available files. They’re displayed (and linked to) on public pages, although only uploaded by authenticated users. Basically, a file can be uploaded as part of the creation of a Publication; it can be replaced as part of an update to a Publication; and it can be deleted as part of the deletion of a Publication. All of this is done without direct involvement on the part of the user, except insofar as is required for the user to select a file for upload.

I avoid the database mainly for simplicity, and because I like to have the files in all of their file-y goodness freely available in a Web-accessible directory rather than as bytes specially gathered from the database and hand-assembled when a user requests them. It is not a Big Issue, but I do prefer to use the filesystem for files whenever possible.

To reply more directly to Sameet: These files are uploaded through the app, but are not managed by hand. Once uploaded they are indeed mostly used for, as you say, “display” purposes, and are treated as static files.

--
b

Sent from my iPhone

On Mar 25, 2013, at 9:17 PM, Eric Biesterfeld <<a href="javascript:_e({}, &#39;cvml&#39;, &#39;eric@ebiester.com&#39;);" target="_blank">eric@...> wrote:

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <<a href="javascript:_e({}, &#39;cvml&#39;, &#39;ben@silver-chalice.com&#39;);" target="_blank">ben@...> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email




--
Alexandre Jacques
Reply | Threaded
Open this post in threaded view
|

RE: How best to store uploaded files in a Grails app?

gem fox
In reply to this post by netwiser
MongoDB GridFS is another option, :)

jian

From: [hidden email]
To: [hidden email]
Date: Mon, 25 Mar 2013 23:12:38 -0400
Subject: RE: [grails-user] How best to store uploaded files in a Grails app?

For me, I stored all files in the MySql database, it has over 22G now, which causes a big headache for me. I am running the server on Amazon AWS,  I can't push the static images to the CDN or S3, this does not harm too much.  The most worst part is, every time I try to optimize or backup the database, I have to take the server offline for over 30 minutes, let my users watching a funny sorry server maintaining page..... Also I had to resize the EBS volume couple of times. 

I wish I put the static big files on hard drive from the beginning. 

James



Date: Mon, 25 Mar 2013 19:17:34 -0700
From: [hidden email]
To: [hidden email]
Subject: Re: [grails-user] How best to store uploaded files in a Grails app?

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Marc Palmer Local
In reply to this post by Ben Klein

On 26 Mar 2013, at 03:21, Benjamin Klein <[hidden email]> wrote:

> These are static always-publicly-available files. They’re displayed (and linked to) on public pages, although only uploaded by authenticated users. Basically, a file can be uploaded as part of the creation of a Publication; it can be replaced as part of an update to a Publication; and it can be deleted as part of the deletion of a Publication. All of this is done without direct involvement on the part of the user, except insofar as is required for the user to select a file for upload.
>
> I avoid the database mainly for simplicity, and because I like to have the files in all of their file-y goodness freely available in a Web-accessible directory rather than as bytes specially gathered from the database and hand-assembled when a user requests them. It is not a Big Issue, but I do prefer to use the filesystem for files whenever possible.
>
> To reply more directly to Sameet: These files are uploaded through the app, but are not managed by hand. Once uploaded they are indeed mostly used for, as you say, “display” purposes, and are treated as static files.

Just use S3, unless it is a totally toy project.

Use the AWS plugin for grails, uploading to S3 is trivial. Even better use a direct S3 uploader in the browser, and there's nothing to do except store the URL of the file in your DB.

It's easy.

Marc

~ ~ ~
Marc Palmer
Freelancer (Grails/Groovy/Java/UX)

Currently available for hire, see http://marcpalmer.net

Blog: http://www.anyware.co.uk
Twitter: http://twitter.com/wangjammer5
App.Net: https://alpha.app.net/marcpalmer 






---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

sameetn
In reply to this post by Ben Klein
The reason I asked that question is we use a CMS to manage files like these. And we actually store only the reference to these in the database. Is that a possible option?

Sent from my iPhone

On Mar 25, 2013, at 11:21 PM, Benjamin Klein <[hidden email]> wrote:

These are static always-publicly-available files. They’re displayed (and linked to) on public pages, although only uploaded by authenticated users. Basically, a file can be uploaded as part of the creation of a Publication; it can be replaced as part of an update to a Publication; and it can be deleted as part of the deletion of a Publication. All of this is done without direct involvement on the part of the user, except insofar as is required for the user to select a file for upload.

I avoid the database mainly for simplicity, and because I like to have the files in all of their file-y goodness freely available in a Web-accessible directory rather than as bytes specially gathered from the database and hand-assembled when a user requests them. It is not a Big Issue, but I do prefer to use the filesystem for files whenever possible.

To reply more directly to Sameet: These files are uploaded through the app, but are not managed by hand. Once uploaded they are indeed mostly used for, as you say, “display” purposes, and are treated as static files.

--
b

Sent from my iPhone

On Mar 25, 2013, at 9:17 PM, Eric Biesterfeld <[hidden email]> wrote:

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Geoff Hopson
In reply to this post by Marc Palmer Local
+1 for Marc's suggestion. There is also the jets3t library and Amazon's own S3 java libs if you want to use those.

Also, if S3 is not palatable, writing some file shuffling code to move the uploaded file to a known folder/filename structure under a local webserver/CDN serving static content is not hard. We did exactly this at Sky for the video.sky.com web site, creating folders that mapped to object instances, transcoding videos, creating thumbnails etc all under a simple Apache installation. The workflow had a "publish" stage which is when images/videos were moved from a "work" area to the published area.

HTH
Geoff
Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

lucastex
You can see how to upload files to S3 using grails aws plugin here: http://blanq.github.com/grails-aws/1.2.12.1/index.html

Easy as this:

def file = request.getFile('photo')
def uploadedFile = file.inputStream.s3upload(file.originalFilename) {
    bucket "file-upload-from-inputstream"
}

[]s,

Lucas Teixeira



On Tue, Mar 26, 2013 at 8:29 AM, ɢεσғғ нσρƨσп <[hidden email]> wrote:
+1 for Marc's suggestion. There is also the jets3t library and Amazon's own S3 java libs if you want to use those.

Also, if S3 is not palatable, writing some file shuffling code to move the uploaded file to a known folder/filename structure under a local webserver/CDN serving static content is not hard. We did exactly this at Sky for the video.sky.com web site, creating folders that mapped to object instances, transcoding videos, creating thumbnails etc all under a simple Apache installation. The workflow had a "publish" stage which is when images/videos were moved from a "work" area to the published area.

HTH
Geoff

Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Dean Del Ponte-2
We currently store 5000+ images and files in our MySQL database.

The downside is the  database can grow quite large (ours is 9G) which causes backups/restores to take longer.

The plus side is it's easy to manage because all of your data is in one place.

Videos are the one file you probably don't want to store in a database because of the overhead.

Good luck!

- Dean Del Ponte


On Tue, Mar 26, 2013 at 8:18 AM, Lucas F. A. Teixeira <[hidden email]> wrote:
You can see how to upload files to S3 using grails aws plugin here: http://blanq.github.com/grails-aws/1.2.12.1/index.html

Easy as this:

def file = request.getFile('photo')
def uploadedFile = file.inputStream.s3upload(file.originalFilename) {
    bucket "file-upload-from-inputstream"
}

[]s,

Lucas Teixeira



On Tue, Mar 26, 2013 at 8:29 AM, ɢεσғғ нσρƨσп <[hidden email]> wrote:
+1 for Marc's suggestion. There is also the jets3t library and Amazon's own S3 java libs if you want to use those.

Also, if S3 is not palatable, writing some file shuffling code to move the uploaded file to a known folder/filename structure under a local webserver/CDN serving static content is not hard. We did exactly this at Sky for the video.sky.com web site, creating folders that mapped to object instances, transcoding videos, creating thumbnails etc all under a simple Apache installation. The workflow had a "publish" stage which is when images/videos were moved from a "work" area to the published area.

HTH
Geoff


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

bdrhoa
In reply to this post by Ben Klein
I've seen arguments for against putting files in db vs keeping them on the file system. I haven't found any argument either way to be definitive. But I decided on keeping them on the file system. However, I keep them outside of web app. That way I can upload a new war file without having to make sure I save the documents and then moving them back. To do that I use the very old static-resources plugin.

http://maflt.org/ibidem


---------------------------
www.maf.org/rhoads
www.ontherhoads.org


On Mon, Mar 25, 2013 at 7:28 PM, Benjamin Klein <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Ben Klein
In reply to this post by sameetn
Well, technically the app itself is the CMS. :) My approach all along has been to store the filename in the database, and then to use that to refer to the actual files, which were where they had been placed in a specific directory by the app.

--
b

On Mar 26, 2013, at 6:18 AM, Sameet Nasnodkar <[hidden email]> wrote:

The reason I asked that question is we use a CMS to manage files like these. And we actually store only the reference to these in the database. Is that a possible option?

Sent from my iPhone

On Mar 25, 2013, at 11:21 PM, Benjamin Klein <[hidden email]> wrote:

These are static always-publicly-available files. They’re displayed (and linked to) on public pages, although only uploaded by authenticated users. Basically, a file can be uploaded as part of the creation of a Publication; it can be replaced as part of an update to a Publication; and it can be deleted as part of the deletion of a Publication. All of this is done without direct involvement on the part of the user, except insofar as is required for the user to select a file for upload.

I avoid the database mainly for simplicity, and because I like to have the files in all of their file-y goodness freely available in a Web-accessible directory rather than as bytes specially gathered from the database and hand-assembled when a user requests them. It is not a Big Issue, but I do prefer to use the filesystem for files whenever possible.

To reply more directly to Sameet: These files are uploaded through the app, but are not managed by hand. Once uploaded they are indeed mostly used for, as you say, “display” purposes, and are treated as static files.

--
b

Sent from my iPhone

On Mar 25, 2013, at 9:17 PM, Eric Biesterfeld <[hidden email]> wrote:

1. Why are you avoiding the database? I'm not saying it's ideal, but it does affect the rest of the advice.

2. How much control do you have over your environment? I mean, you should have a common directory at the least.

3. Are all of these files public? Do they depend on authentication?

On Mar 25, 2013 6:36 PM, "Benjamin Klein" <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Ben Klein
In reply to this post by bdrhoa
Thanks for the responses everybody. :) It sounds like I will probably be using S3.

Ben

--
b

On Mar 26, 2013, at 2:41 PM, Brad Rhoads <[hidden email]> wrote:

I've seen arguments for against putting files in db vs keeping them on the file system. I haven't found any argument either way to be definitive. But I decided on keeping them on the file system. However, I keep them outside of web app. That way I can upload a new war file without having to make sure I save the documents and then moving them back. To do that I use the very old static-resources plugin.

http://maflt.org/ibidem


---------------------------
www.maf.org/rhoads
www.ontherhoads.org


On Mon, Mar 25, 2013 at 7:28 PM, Benjamin Klein <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email




Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Marc Palmer Local
In reply to this post by bdrhoa

On 26 Mar 2013, at 19:41, Brad Rhoads <[hidden email]> wrote:

> I've seen arguments for against putting files in db vs keeping them on the file system. I haven't found any argument either way to be definitive. But I decided on keeping them on the file system. However, I keep them outside of web app. That way I can upload a new war file without having to make sure I save the documents and then moving them back. To do that I use the very old static-resources plugin.

Transferring large blobs from the DB is really slow, and then you have to serve that to the client. You'll also want to be sure that the BLOB is not eagerly loaded when you load domain objects just to access other properties. This can kill you.

In my experience a minimum of 2x the size of the blob is required in heap memory to transfer it to/from the DB. This massively affects your ability to serve highly concurrent loads. This is because of the way the JDBC layer / driver and hibernate work.

If you do not need to serve the files directly from your server for security reasons, DO NOT SERVE THEM AND DO NOT USE THE DB :)

Take it from me, we do this for my client's SPOKEnPHOTO service, but this was purely as a compromise because at the time of launch we did not have time to tackle the issues required to make mobile applications work with S3 securely using the Amazon S3 token generation mechanism, to avoid the "super user" credentials being in the client application.

That is being resolved soon because of the above issues - it costs a small fortune to get enough external bandwidth to serve any kind of decent concurrent load, as well as tons of RAM/VMs.

If its a toy project, go to town. Toy means less than a few dozen concurrent users and/or small file sizes.

Marc
~ ~ ~
Marc Palmer
Freelancer (Grails/Groovy/Java/UX)

Currently available for hire, see http://marcpalmer.net

Blog: http://www.anyware.co.uk
Twitter: http://twitter.com/wangjammer5
App.Net: https://alpha.app.net/marcpalmer 






---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Marc Palmer Local
In reply to this post by Ben Klein

On 26 Mar 2013, at 20:05, Benjamin Klein <[hidden email]> wrote:

> Well, technically the app itself is the CMS. :) My approach all along has been to store the filename in the database, and then to use that to refer to the actual files, which were where they had been placed in a specific directory by the app.
>

For small single-server apps that is a good solution Ben. This is what Weceem does. The fun starts if you have to move to multiple servers accessing the shared FS.

Marc

~ ~ ~
Marc Palmer
Freelancer (Grails/Groovy/Java/UX)

Currently available for hire, see http://marcpalmer.net

Blog: http://www.anyware.co.uk
Twitter: http://twitter.com/wangjammer5
App.Net: https://alpha.app.net/marcpalmer 






---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

burtbeckwith
In reply to this post by Marc Palmer Local
If you do use BLOBs (and/or CLOBs) you can use http://grails.org/plugin/lazylob to avoid eager loading.

Burt

Marc Palmer Local wrote
On 26 Mar 2013, at 19:41, Brad Rhoads <[hidden email]> wrote:
...

Transferring large blobs from the DB is really slow, and then you have to serve that to the client. You'll also want to be sure that the BLOB is not eagerly loaded when you load domain objects just to access other properties. This can kill you.

...
Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

themarchoffolly
In reply to this post by Ben Klein
One further option you may want to consider is MongoDB and specifically GridFS. A good place to start is the FAQ article When should I use GridFS. I'm using this myself for storing and streaming large, binary files in a production application.

David

On Tue, Mar 26, 2013 at 4:10 PM, Benjamin Klein <[hidden email]> wrote:
Thanks for the responses everybody. :) It sounds like I will probably be using S3.

Ben

--
b

On Mar 26, 2013, at 2:41 PM, Brad Rhoads <[hidden email]> wrote:

I've seen arguments for against putting files in db vs keeping them on the file system. I haven't found any argument either way to be definitive. But I decided on keeping them on the file system. However, I keep them outside of web app. That way I can upload a new war file without having to make sure I save the documents and then moving them back. To do that I use the very old static-resources plugin.

http://maflt.org/ibidem


---------------------------
www.maf.org/rhoads
www.ontherhoads.org


On Mon, Mar 25, 2013 at 7:28 PM, Benjamin Klein <[hidden email]> wrote:
I have a Grails app (2.0.4, for what that's worth) in which I need to create Publication instances with associated static files (PDF and image). My approach up to this point has been to store the files in a directory under web-app in the expanded WAR, and to save their filenames along with the associated Publication for purposes of linking to them later on. This approach is of course made of problems and I am not at all recommending it -- indeed I have been told multiple times that this is not the way to do it -- but I have not yet come across any satisfactory alternate means by which to store these files without using the database. How might I best accomplish this, with minimal damage to life and property?

Thanks!

Ben

--
b

Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email







--
"It's possible that my whole purpose in life is simply to serve as a warning to others."
Reply | Threaded
Open this post in threaded view
|

Re: How best to store uploaded files in a Grails app?

Marc Palmer Local

On 26 Mar 2013, at 20:26, David Russell <[hidden email]> wrote:

> One further option you may want to consider is MongoDB and specifically GridFS. A good place to start is the FAQ article When should I use GridFS. I'm using this myself for storing and streaming large, binary files in a production application.

Yes, that would work too, but you still have the bandwidth requirements and don't want to serve those files through Servlet API (Grails) unless you can rig up some NIO type stuff to make sure there aren't multiple copies of the entire object in RAM.

The issue with bandwidth is not just cost. For example, if your ISP's network cannot even sell you more than say 20MBit/s external connectivity... some simple math will give you the maximum number of customers you can serve concurrently at a speed of say 200KBytes/second. It isn't much.

Again, it depends on the app, size of files, usage patterns etc.

Marc

~ ~ ~
Marc Palmer
Freelancer (Grails/Groovy/Java/UX)

Currently available for hire, see http://marcpalmer.net

Blog: http://www.anyware.co.uk
Twitter: http://twitter.com/wangjammer5
App.Net: https://alpha.app.net/marcpalmer 






---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email