Collecting ideas for AWS Beanstalk integration

[lucastex]

Hello everybody,

I'm starting to work on the Grails-AWS plugin Beanstalk integration.
I'd like to collect suggestions of people who is already using or who just tried this Amazon service on how would be the best way to provide this integration.

I myself get concerned about adding infrastructure related scripts directly in the application. This adds, IMO, potentially dangers, since a script that deploy and make an application available could be triggered by mistake in place of other one.

So, I keep asking myself what level of integration would be great to provide. That's why I'm asking the list.

I've mapped the following possible integration points:

1) One script that would upload an application version (and create the beanstalk application if not previously created).

2) One script that checks DNS CNAME availability on Beanstalk domain

3) One simple web interface to create configuration templates for beanstalk environments

*4) One script to build environments using configuration templates described in item 3

*5) One script to restart app application servers of a selected environment

*6) One script to rebuild one entire environment

*7) One script to get and watch latests events that are happening in one environment

*8) One script to deploy a generated war to some beanstalk environment previously created

The '*' marked items (4,5,6,7,8) reflects what I've said on infrastructure vs. development issue.
What you guys think about it? Should the plugin provide all these scripts?

Or should it provide just helpers (upload, create templates, etc). This way, all the infrastructure work will have to be done using AWS web console.

Any opinions regarding this? Or any other ideas of integration to implement?

Thanks a lot,

Lucas

[lucastex]

no one ?

On Tue, Feb 22, 2011 at 7:08 PM, Lucas F. A. Teixeira <[hidden email]> wrote:

Hello everybody,

I'm starting to work on the Grails-AWS plugin Beanstalk integration.
I'd like to collect suggestions of people who is already using or who just tried this Amazon service on how would be the best way to provide this integration.

I myself get concerned about adding infrastructure related scripts directly in the application. This adds, IMO, potentially dangers, since a script that deploy and make an application available could be triggered by mistake in place of other one.

So, I keep asking myself what level of integration would be great to provide. That's why I'm asking the list.

I've mapped the following possible integration points:

1) One script that would upload an application version (and create the beanstalk application if not previously created).

2) One script that checks DNS CNAME availability on Beanstalk domain

3) One simple web interface to create configuration templates for beanstalk environments

*4) One script to build environments using configuration templates described in item 3

*5) One script to restart app application servers of a selected environment

*6) One script to rebuild one entire environment

*7) One script to get and watch latests events that are happening in one environment

*8) One script to deploy a generated war to some beanstalk environment previously created

The '*' marked items (4,5,6,7,8) reflects what I've said on infrastructure vs. development issue.
What you guys think about it? Should the plugin provide all these scripts?

Or should it provide just helpers (upload, create templates, etc). This way, all the infrastructure work will have to be done using AWS web console.

Any opinions regarding this? Or any other ideas of integration to implement?

Thanks a lot,

Lucas

[tomas lin]

What about checkout and deployment of github projects in a way similar
to heroku's git support?
http://devcenter.heroku.com/articles/git

I think this would be cool as a wrapper around Amazon Web Service
BeanStalk Command Line (
http://aws.typepad.com/aws/2011/01/aws-beanstalk-apis-and-command-line-tools.html
), but would be scared of using it if there was no guarantee that the
scripts are up-to-date with AWS.

( The GAE plugin deploy was broken for a little bit because of version
updates, and the fear would be that if this plugin gets abandoned or
out of date, entire builds would fail. )

On Tue, Feb 22, 2011 at 10:08 PM, Lucas F. A. Teixeira
<[hidden email]> wrote:

> Hello everybody,
>
> I'm starting to work on the Grails-AWS plugin Beanstalk integration.
> I'd like to collect suggestions of people who is already using or who just
> tried this Amazon service on how would be the best way to provide this
> integration.
>
> I myself get concerned about adding infrastructure related scripts directly
> in the application. This adds, IMO, potentially dangers, since a script that
> deploy and make an application available could be triggered by mistake in
> place of other one.
>
> So, I keep asking myself what level of integration would be great to
> provide. That's why I'm asking the list.
>
> I've mapped the following possible integration points:
>
> 1) One script that would upload an application version (and create the
> beanstalk application if not previously created).
>
> 2) One script that checks DNS CNAME availability on Beanstalk domain
>
> 3) One simple web interface to create configuration templates for beanstalk
> environments
>
> *4) One script to build environments using configuration templates described
> in item 3
>
> *5) One script to restart app application servers of a selected environment
>
> *6) One script to rebuild one entire environment
>
> *7) One script to get and watch latests events that are happening in one
> environment
>
> *8) One script to deploy a generated war to some beanstalk environment
> previously created
>
> The '*' marked items (4,5,6,7,8) reflects what I've said on infrastructure
> vs. development issue.
> What you guys think about it? Should the plugin provide all these scripts?
>
> Or should it provide just helpers (upload, create templates, etc). This way,
> all the infrastructure work will have to be done using AWS web console.
>
> Any opinions regarding this? Or any other ideas of integration to implement?
>
> Thanks a lot,
>
> Lucas
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[lucastex]

Thanks Tomas,

Actually, Heroku works a little bit different Beanstalk, and that's why stuff like these work for them.

1) Rails uses exploded file structure, and we're using Java WARs.

This is much easier when we have a directory structure, using rails+git+heroku you can just push latest changes on selected files and heroku will pull it for you in their local repository where servers are looking into.
I won't dig too much deeper on this, since there is another thread on updating applications partially :)

2) Heroku uses filesystem to deploy the app.

Since heroku pulls the users changes to the filesystem, the "application server" is up to run and look into it. Beanstalk needs the application to be stored in a S3 Bucket. So, again I point we'll have to build the entire war and upload it to S3. If S3 let us manipulate it as a filesystem (not pseudo-mounting it) could work, but we would have to install git clients (or svn or whatever) directly in S3. See?

Unfortunately, we're stuck on:

- large and single war files
- having application servers looking into S3 and not filesystems

Any thoughts?

Thanks!


Lucas Frare Teixeira .·.
- [hidden email]
- lucastex.com.br
- blog.lucastex.com
- twitter.com/lucastex

On Wed, Feb 23, 2011 at 10:44 AM, Tomas Lin <[hidden email]> wrote:

What about checkout and deployment of github projects in a way similar
to heroku's git support?
http://devcenter.heroku.com/articles/git

I think this would be cool as a wrapper around Amazon Web Service
BeanStalk Command Line (
http://aws.typepad.com/aws/2011/01/aws-beanstalk-apis-and-command-line-tools.html
), but would be scared of using it if there was no guarantee that the
scripts are up-to-date with AWS.

( The GAE plugin deploy was broken for a little bit because of version
updates, and the fear would be that if this plugin gets abandoned or
out of date, entire builds would fail. )

On Tue, Feb 22, 2011 at 10:08 PM, Lucas F. A. Teixeira
<[hidden email]> wrote:
> Hello everybody,
>
> I'm starting to work on the Grails-AWS plugin Beanstalk integration.
> I'd like to collect suggestions of people who is already using or who just
> tried this Amazon service on how would be the best way to provide this
> integration.
>
> I myself get concerned about adding infrastructure related scripts directly
> in the application. This adds, IMO, potentially dangers, since a script that
> deploy and make an application available could be triggered by mistake in
> place of other one.
>
> So, I keep asking myself what level of integration would be great to
> provide. That's why I'm asking the list.
>
> I've mapped the following possible integration points:
>
> 1) One script that would upload an application version (and create the
> beanstalk application if not previously created).
>
> 2) One script that checks DNS CNAME availability on Beanstalk domain
>
> 3) One simple web interface to create configuration templates for beanstalk
> environments
>
> *4) One script to build environments using configuration templates described
> in item 3
>
> *5) One script to restart app application servers of a selected environment
>
> *6) One script to rebuild one entire environment
>
> *7) One script to get and watch latests events that are happening in one
> environment
>
> *8) One script to deploy a generated war to some beanstalk environment
> previously created
>
> The '*' marked items (4,5,6,7,8) reflects what I've said on infrastructure
> vs. development issue.
> What you guys think about it? Should the plugin provide all these scripts?
>
> Or should it provide just helpers (upload, create templates, etc). This way,
> all the infrastructure work will have to be done using AWS web console.
>
> Any opinions regarding this? Or any other ideas of integration to implement?
>
> Thanks a lot,
>
> Lucas
>
>

[Wolfgang Schell]

Hi Lucas,

tanks for your work on the AWS plugin!

lucastex wrote

I myself get concerned about adding infrastructure related scripts directly
in the application. This adds, IMO, potentially dangers, since a script that
deploy and make an application available could be triggered by mistake in
place of other one.

You're talking about scripts in <mygrailsapp>/scripts (or rather <awsplugin>/scripts), right? The scripts don't end up in the WAR file, so I think there's no problem with this.

lucastex wrote

1) One script that would upload an application version (and create the
beanstalk application if not previously created).

This is the bare minimum. It could create the WAR file and accept all required parameters (grails env, beanstalk app name, beanstalk env and everything to create the app, if necessary. See script (4) below).

All parameters (except grails env) should be configurable in BuildConfig.groovy.

lucastex wrote

2) One script that checks DNS CNAME availability on Beanstalk domain

As this is a simple call to the corresponding Beanstalk cli "elastic-beanstalk-check-dns-availability"), I would leave this aside.

lucastex wrote

3) One simple web interface to create configuration templates for beanstalk environments

Where would this interface be running? Inside your app as part of the Beanstalk plugin? A separate web app just for configuration? Could you please elaborate some more on this idea?

Basically I would prefer simple scripts.

lucastex wrote

*4) One script to build environments using configuration templates described in item 3

Yes, I think this script would create an app and/or an environment with all required parameters like instance type, count, key name, solution stack, CNAME, etc. similar to my bash script [1]. Also it would form the base for script (1), if the app and/or environment does not yet exist.

lucastex wrote

*5) One script to restart app application servers of a selected environment
*6) One script to rebuild one entire environment

Yes, these should be rather simple scripts.

lucastex wrote

*7) One script to get and watch latests events that are happening in one
environment

Again, as this is a simple call to the corresponding Beanstalk cli "elastic-beanstalk-describe-events"), I would leave this aside.

lucastex wrote

*8) One script to deploy a generated war to some beanstalk environment previously created

Isn't this the same as (1)?


[1] http://blog.jetztgrad.net/2011/02/script-to-start-an-amazon-elastic-beanstalk-application/


Regards,

Wolfgang

[lucastex]

Hi Wolfgang!

Thanks for replying! :)

lucastex wrote:
>
> I myself get concerned about adding infrastructure related scripts
> directly
> in the application. This adds, IMO, potentially dangers, since a script
> that
> deploy and make an application available could be triggered by mistake in
> place of other one.
>

You're talking about scripts in <mygrailsapp>/scripts (or rather
<awsplugin>/scripts), right? The scripts don't end up in the WAR file, so I
think there's no problem with this.

Not exactly this, my concern is actually making devs capable of using simple scripts to deploy applications and interfere in prod envs.
This is fine when you have a small team, or a test application. But you can imagine a 15 people team, and a new guy just gets involved and accidentally (or by curiosity) calls the deploy script :)

I grew with the idea on separating concepts, development and infrastructure/deployment. No problem if you're the guy that do both, but I'm talking about making things isolated, got it?
 

lucastex wrote:
>
> 1) One script that would upload an application version (and create the
> beanstalk application if not previously created).
>

This is the bare minimum. It could create the WAR file and accept all
required parameters (grails env, beanstalk app name, beanstalk env and
everything to create the app, if necessary. See script (4) below).

All parameters (except grails env) should be configurable in
BuildConfig.groovy.

Yeah, this one is the minimum, as you said :)
But in my mind, this one would create just the application and the application version, without environments or other things, this is just to upload the war file to Amazon.
Similar to this script: https://github.com/blanq/beanstalk-upload

lucastex wrote:
>
> 2) One script that checks DNS CNAME availability on Beanstalk domain
>

As this is a simple call to the corresponding Beanstalk cli
"elastic-beanstalk-check-dns-availability"), I would leave this aside.

Yes, the same! This can still be useful for those who doesn't have beanstalk cli tools installed.
Not that much people will have it... Will achieve the same, but using webservices directly instead of the cli lib.
 

lucastex wrote:
>
> 3) One simple web interface to create configuration templates for
> beanstalk environments
>

Where would this interface be running? Inside your app as part of the
Beanstalk plugin? A separate web app just for configuration? Could you
please elaborate some more on this idea?

Basically I would prefer simple scripts.

I thought initially to make a simple view (controller+view) to make things easier, like selecting stack solution on combo boxes, adding JVM parameters, setting JDBC_CONNECTION_STRING and acceskey/secretkey params, etc... This would have to be protected in prod envs of course, but would be there just to provide an easy way to create configuration templates, to let people configure environments before starting them.

This can be easily implemented in gant scripts as well, something like:

-------
Choose your solution stack:

[1] Tomcat 6 32 bits
[2] Tomcat 6 64 bits

Select (1 or 2): __
----------

I myself prefer scripts just like you, a web interface would be easier to use, but of course, would have to be protected in prod. Not sure what I must do.
If I was implementing just for me, would use gant scripts, for sure.

lucastex wrote:
>
> *4) One script to build environments using configuration templates
> described in item 3
>

Yes, I think this script would create an app and/or an environment with all
required parameters like instance type, count, key name, solution stack,
CNAME, etc. similar to my bash script [1]. Also it would form the base for
script (1), if the app and/or environment does not yet exist.

I myself doesn't like long/lot of parameters. That's why I've suggested item (3) to build the configurations and use it in this other script.
But this is it :)

I have checked your script before when reading your blog, and it is awesome :) great work on it!
 

lucastex wrote:
>
> *5) One script to restart app application servers of a selected
> environment
> *6) One script to rebuild one entire environment
>

Yes, these should be rather simple scripts.

yeah! :)
 

lucastex wrote:
>
> *7) One script to get and watch latests events that are happening in one
> environment
>

Again, as this is a simple call to the corresponding Beanstalk cli
"elastic-beanstalk-describe-events"), I would leave this aside.

 

Same case before, to let quick access on it, it would use the credentials previously configured in the application to do it.
And most people won't have beanstalk cli tools installed :)

lucastex wrote:
>
> *8) One script to deploy a generated war to some beanstalk environment
> previously created
>

Isn't this the same as (1)?

Actually not, the (1) would upload the generated war and create one version poiting to it.
This script would deploy that version in the pointed environment. No uploads on this one...
 

[1]
http://blog.jetztgrad.net/2011/02/script-to-start-an-amazon-elastic-beanstalk-application/

I've checked before, this got great! :)


Lucas

 

Regards,

Wolfgang
--
View this message in context: http://grails.1312388.n4.nabble.com/Collecting-ideas-for-AWS-Beanstalk-integration-tp3320152p3321225.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[Jos-4]

On Wed, Feb 23, 2011 at 3:26 PM, Lucas F. A. Teixeira
<[hidden email]> wrote:
>
> Unfortunately, we're stuck on:
>
> - large and single war files
> - having application servers looking into S3 and not filesystems
>
> Any thoughts?

Just a thought.

1. create an instance on AWS
2. instance loads war from S3 to its file system
3. instance unzips war
4. rsync changes from local to this instance (like you would do for a
single server)
5. update exixting beanstalk instances ( I am not sure if this step is
necessary, or if beanstalk does this automatically)
6. instance zips up war and puts it back on S3
7. stop or terminate the instance

This shouldn't take too long and will cost you about 10 cents.

--Jos

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[lucastex]

Hi Jos,
 

This shouldn't take too long and will cost you about 10 cents.

Tell me, do you think that creating a new instance, starting it up, download the war, unpack it, and then after rsync, upload it again would be faster than just uploading it from local to s3?

Do you have any metrics on this?

I've thought about it once, but really don't know if would be faster or slower...

And what do you guys think about adding "deploy cost" to the "plugin". IMO, don't really think it would make people use it tho...

[]s,

 

--Jos

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[Jos-4]

On Thu, Feb 24, 2011 at 5:26 PM, Lucas F. A. Teixeira
<[hidden email]> wrote:
> Hi Jos,
>
>>
>> This shouldn't take too long and will cost you about 10 cents.
>>
>
> Tell me, do you think that creating a new instance, starting it up, download
> the war, unpack it, and then after rsync, upload it again would be faster
> than just uploading it from local to s3?

Of course this only makes sense if your upstream bandwidth is low. I
think most of the time will be spent in building the war. The instance
should be up by that time.

--Jos

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Ken Liu]

Hopefully in this situation there would be some other means to prevent
accidental deployment such as a password or other configuration that
is known only to the "deployer" person.

I think it would be best to provide tools to allow smooth and easy
deployment during development, and let the people responsible for
production ops/deployment make sure that they don't shoot themselves
in the foot. Frankly, if deployment processes are set up in a way that
someone can accidentally deploy something, then "you're doing it
wrong."

Look at Google AppEngine for example; they provide a very nice Eclipse
plugin that allows you to either run the app in a local Jetty or
deploy directly to GAE using a single click. The app console can
control whether the deployed version is the actual "live" version.
Being able to deploy to the cloud with a single click is a huge
benefit during the development process. You don't always want to
separate development and deployment processes because usually
developers need to deploy to a development server, whether it's in the
office or in the cloud.

Ken

> Not exactly this, my concern is actually making devs capable of using simple
> scripts to deploy applications and interfere in prod envs.
> This is fine when you have a small team, or a test application. But you can
> imagine a 15 people team, and a new guy just gets involved and accidentally
> (or by curiosity) calls the deploy script :)
>
> I grew with the idea on separating concepts, development and
> infrastructure/deployment. No problem if you're the guy that do both, but
> I'm talking about making things isolated, got it?
\

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[lucastex]

For sure Ken,

This is what was described.
You could "deploy" (would actually upload the version and let there standing) in item (1) of my first e-mail.
And setting the live version would be in item (8) of the same email.

Having more than one version live at the same time like AppEngine is not possible with Beanstalk in the same environment.
We would have to launch one environment for each version, but as we can't have "idle" environments, and each one have its own EC2 instances, it would cost a lot of money...

Thanks!

[]s,


Lucas Frare Teixeira .·.
- [hidden email]
- lucastex.com.br
- blog.lucastex.com
- twitter.com/lucastex

On Thu, Feb 24, 2011 at 3:10 PM, Ken Liu <[hidden email]> wrote:

Hopefully in this situation there would be some other means to prevent
accidental deployment such as a password or other configuration that
is known only to the "deployer" person.

I think it would be best to provide tools to allow smooth and easy
deployment during development, and let the people responsible for
production ops/deployment make sure that they don't shoot themselves
in the foot. Frankly, if deployment processes are set up in a way that
someone can accidentally deploy something, then "you're doing it
wrong."

Look at Google AppEngine for example; they provide a very nice Eclipse
plugin that allows you to either run the app in a local Jetty or
deploy directly to GAE using a single click. The app console can
control whether the deployed version is the actual "live" version.
Being able to deploy to the cloud with a single click is a huge
benefit during the development process. You don't always want to
separate development and deployment processes because usually
developers need to deploy to a development server, whether it's in the
office or in the cloud.

Ken

> Not exactly this, my concern is actually making devs capable of using simple
> scripts to deploy applications and interfere in prod envs.
> This is fine when you have a small team, or a test application. But you can
> imagine a 15 people team, and a new guy just gets involved and accidentally
> (or by curiosity) calls the deploy script :)
>
> I grew with the idea on separating concepts, development and
> infrastructure/deployment. No problem if you're the guy that do both, but
> I'm talking about making things isolated, got it?

\

[tomas lin]

Hey Lucas,

Now that we also have Amazon CloudFormation, would it make sense to
build default grails application templates and have the AWS service
support those?

http://aws.amazon.com/cloudformation/

On Thu, Feb 24, 2011 at 6:16 PM, Lucas F. A. Teixeira
<[hidden email]> wrote:

> For sure Ken,
>
> This is what was described.
> You could "deploy" (would actually upload the version and let there
> standing) in item (1) of my first e-mail.
> And setting the live version would be in item (8) of the same email.
>
> Having more than one version live at the same time like AppEngine is not
> possible with Beanstalk in the same environment.
> We would have to launch one environment for each version, but as we can't
> have "idle" environments, and each one have its own EC2 instances, it would
> cost a lot of money...
>
> Thanks!
>
> []s,
>
>
> Lucas Frare Teixeira .·.
> - [hidden email]
> - lucastex.com.br
> - blog.lucastex.com
> - twitter.com/lucastex
>
>
> On Thu, Feb 24, 2011 at 3:10 PM, Ken Liu <[hidden email]> wrote:
>>
>> Hopefully in this situation there would be some other means to prevent
>> accidental deployment such as a password or other configuration that
>> is known only to the "deployer" person.
>>
>> I think it would be best to provide tools to allow smooth and easy
>> deployment during development, and let the people responsible for
>> production ops/deployment make sure that they don't shoot themselves
>> in the foot. Frankly, if deployment processes are set up in a way that
>> someone can accidentally deploy something, then "you're doing it
>> wrong."
>>
>> Look at Google AppEngine for example; they provide a very nice Eclipse
>> plugin that allows you to either run the app in a local Jetty or
>> deploy directly to GAE using a single click. The app console can
>> control whether the deployed version is the actual "live" version.
>> Being able to deploy to the cloud with a single click is a huge
>> benefit during the development process. You don't always want to
>> separate development and deployment processes because usually
>> developers need to deploy to a development server, whether it's in the
>> office or in the cloud.
>>
>> Ken
>>
>> > Not exactly this, my concern is actually making devs capable of using
>> > simple
>> > scripts to deploy applications and interfere in prod envs.
>> > This is fine when you have a small team, or a test application. But you
>> > can
>> > imagine a 15 people team, and a new guy just gets involved and
>> > accidentally
>> > (or by curiosity) calls the deploy script :)
>> >
>> > I grew with the idea on separating concepts, development and
>> > infrastructure/deployment. No problem if you're the guy that do both,
>> > but
>> > I'm talking about making things isolated, got it?
>> \
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Wolfgang Schell]

A Grails template for Amazon CloudFormation using Beanstalk for the App and RDS for MySQL would be great. There is an example template, which might already provide everything we need: https://s3.amazonaws.com/cloudformation-samples-us-east-1/ElasticBeanstalk-1.0.0.template

Wolfgang