Advice on Queuing technologies

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Advice on Queuing technologies

jonspalmer

I’m hoping to get some advice on choosing a queuing technology for our Grails app. Our primary use case is enqueuing various long running background tasks. At the moment we’re using the jesque plugin but we’re increasingly finding it is not stable enough for our needs. I’m curious to understand what alternatives people would recommend? I see plugins for ActiveMQ, RabbitMQ, JMS and others. Which are stable and ready for prime time? Any thoughts would be much appreciated.

 

Thanks

Jon

 

Jon Palmer | Software Engineering Manager

Care.com, Inc.
201 Jones Road | Suite 500 | Waltham, MA | 02451
(p) 781.693.1784 | (f) 781.899.1294 | (e) [hidden email]


cid:3392194686_747741


Care.com | There for you
For Children, Adults & Seniors, Pets, Home & Lifestyle


Become a Care.com Fan on Facebook:
http://www.facebook.com/caredotcom

This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.

Care.com and "There for you" are service marks or registered service marks of Care.com, Inc. © 2007-2012 Care.com, Inc. All rights reserved.

 

Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

a.shneyderman

On Tue, Oct 9, 2012 at 5:35 AM, Jon Palmer <[hidden email]> wrote:

I’m hoping to get some advice on choosing a queuing technology for our Grails app. Our primary use case is enqueuing various long running background tasks. At the moment we’re using the jesque plugin but we’re increasingly finding it is not stable enough for our needs. I’m curious to understand what alternatives people would recommend? I see plugins for ActiveMQ, RabbitMQ, JMS and others. Which are stable and ready for prime time? Any thoughts would be much appreciated.


You can always go it alone. JMS stuff, you can setup without amq plugin, so is rabbit. As the matter of fact you should do it even if you decide to use plugin in the end. Plugins wrap around some spring integration. You can always setup spring integration with said broker without any plugin. Some of the things that are important for you to effectively use rabbit, for example, are very well hidden from you by the plugin. I doubt it is all that useful, especially once your code hits production and you have no clue where those exceptions that are logged coming from. Plugins do provide convenience. For example, rabbit provides sendRabbit method to your artifacts, but you can also wire rabbitTemplate and do the same thing. At the end I think you should read the source of the plugin if you decide to use it. It will let you understand how plugin sets up your context. 

As to whether to go with rabbit or JMS. I think the choice is clear if you have that choice. Rabbit wins in that contest, no matter how you look at it. ActiveMQ just does not provide you with enough juice. Its web console is pathetic as compared to Rabbit's. Rabbit's ability to route messages is not something that is provided by basic AMQ installation. The speed and stability of Rabbit is superb. It has out of the box clustering, which will allow you to scale and provides you with HA features if/when you need them (good reference is RabbitMQ in Action book, so is their web site). The only thing that I know of that beats Rabbit is ZeroMQ. Unfortunately, it requires a lot of fiddling, there are no plugins that I know of, and you have to completely change your view about MQ broker. But it is an option, not an easy one but still it is out there :-) 

Also, there is this http://grailsrocks.github.com/grails-platform-core/guide/events.html. You might find that to be useful if event buses is something you like. IMHO, they are bit harder to understand and manage then your regular centralized message brokers (something that is also true about ZeroMQ, BTW).

Cheers,
Alex.

Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

ideasculptor


On Mon, Oct 8, 2012 at 11:00 PM, Alex Shneyderman <[hidden email]> wrote:

On Tue, Oct 9, 2012 at 5:35 AM, Jon Palmer <[hidden email]> wrote:

I’m hoping to get some advice on choosing a queuing technology for our Grails app. Our primary use case is enqueuing various long running background tasks. At the moment we’re using the jesque plugin but we’re increasingly finding it is not stable enough for our needs. I’m curious to understand what alternatives people would recommend? I see plugins for ActiveMQ, RabbitMQ, JMS and others. Which are stable and ready for prime time? Any thoughts would be much appreciated.


You can always go it alone. JMS stuff, you can setup without amq plugin, so is rabbit. As the matter of fact you should do it even if you decide to use plugin in the end. Plugins wrap around some spring integration. You can always setup spring integration with said broker without any plugin. Some of the things that are important for you to effectively use rabbit, for example, are very well hidden from you by the plugin. I doubt it is all that useful, especially once your code hits production and you have no clue where those exceptions that are logged coming from. Plugins do provide convenience. For example, rabbit provides sendRabbit method to your artifacts, but you can also wire rabbitTemplate and do the same thing. At the end I think you should read the source of the plugin if you decide to use it. It will let you understand how plugin sets up your context. 

As to whether to go with rabbit or JMS. I think the choice is clear if you have that choice. Rabbit wins in that contest, no matter how you look at it. ActiveMQ just does not provide you with enough juice. Its web console is pathetic as compared to Rabbit's. Rabbit's ability to route messages is not something that is provided by basic AMQ installation. The speed and stability of Rabbit is superb. It has out of the box clustering, which will allow you to scale and provides you with HA features if/when you need them (good reference is RabbitMQ in Action book, so is their web site). The only thing that I know of that beats Rabbit is ZeroMQ. Unfortunately, it requires a lot of fiddling, there are no plugins that I know of, and you have to completely change your view about MQ broker. But it is an option, not an easy one but still it is out there :-) 

I'm not sure why openMQ is ignored here, since it has out of the box clustering and high availability that is very easy to set up and use, as well.  I like and use rabbitmq, but mostly because I have clients that aren't java interacting with my messaging system, not because I have found openMQ to be lacking in features compared to rabbitmq.  I'm not sure what oracle have done with openmq since the acquisition, but I found it really stable, easy to use, and feature rich last time I had need of a JMS solution.  It was every bit as easy to use as rabbitmq.  That said, rabbit most definitely gets the job done as well. I've got no complaints.

And I don't use any plugins for my spring-integration functionality.  Getting things working requires nothing more than wiring some beans together via spring and annotating some methods in service classes, so I didn't see any reason to bury that inside a plugin. Most of my developers hardly even know spring integration is installed, since I have services wired together directly in the environments they use for development.  I've got test/CI systems, production, and my own development system wired together with spring-integation and rabbit and that's about it.  Unless someone else writes code that happens to break when it runs via spring-integration in continuous integration builds, other devs don't need to know anything about setting up and running messaging, and most stuff just works. Obviously, they know it is there, but it doesn't interject itself into their day-to-day coding very often.


Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

bond_
Guys as the discussion came up I have a question .

Apollo seems to be the successor of ActiveMQ any usage based comments on that?

Links:
  1. http://activemq.apache.org/new-features-in-60.html
  2. http://activemq.apache.org/apollo/

On Tue, Oct 9, 2012 at 5:41 PM, Samuel Gendler <[hidden email]> wrote:


On Mon, Oct 8, 2012 at 11:00 PM, Alex Shneyderman <[hidden email]> wrote:

On Tue, Oct 9, 2012 at 5:35 AM, Jon Palmer <[hidden email]> wrote:

I’m hoping to get some advice on choosing a queuing technology for our Grails app. Our primary use case is enqueuing various long running background tasks. At the moment we’re using the jesque plugin but we’re increasingly finding it is not stable enough for our needs. I’m curious to understand what alternatives people would recommend? I see plugins for ActiveMQ, RabbitMQ, JMS and others. Which are stable and ready for prime time? Any thoughts would be much appreciated.


You can always go it alone. JMS stuff, you can setup without amq plugin, so is rabbit. As the matter of fact you should do it even if you decide to use plugin in the end. Plugins wrap around some spring integration. You can always setup spring integration with said broker without any plugin. Some of the things that are important for you to effectively use rabbit, for example, are very well hidden from you by the plugin. I doubt it is all that useful, especially once your code hits production and you have no clue where those exceptions that are logged coming from. Plugins do provide convenience. For example, rabbit provides sendRabbit method to your artifacts, but you can also wire rabbitTemplate and do the same thing. At the end I think you should read the source of the plugin if you decide to use it. It will let you understand how plugin sets up your context. 

As to whether to go with rabbit or JMS. I think the choice is clear if you have that choice. Rabbit wins in that contest, no matter how you look at it. ActiveMQ just does not provide you with enough juice. Its web console is pathetic as compared to Rabbit's. Rabbit's ability to route messages is not something that is provided by basic AMQ installation. The speed and stability of Rabbit is superb. It has out of the box clustering, which will allow you to scale and provides you with HA features if/when you need them (good reference is RabbitMQ in Action book, so is their web site). The only thing that I know of that beats Rabbit is ZeroMQ. Unfortunately, it requires a lot of fiddling, there are no plugins that I know of, and you have to completely change your view about MQ broker. But it is an option, not an easy one but still it is out there :-) 

I'm not sure why openMQ is ignored here, since it has out of the box clustering and high availability that is very easy to set up and use, as well.  I like and use rabbitmq, but mostly because I have clients that aren't java interacting with my messaging system, not because I have found openMQ to be lacking in features compared to rabbitmq.  I'm not sure what oracle have done with openmq since the acquisition, but I found it really stable, easy to use, and feature rich last time I had need of a JMS solution.  It was every bit as easy to use as rabbitmq.  That said, rabbit most definitely gets the job done as well. I've got no complaints.

And I don't use any plugins for my spring-integration functionality.  Getting things working requires nothing more than wiring some beans together via spring and annotating some methods in service classes, so I didn't see any reason to bury that inside a plugin. Most of my developers hardly even know spring integration is installed, since I have services wired together directly in the environments they use for development.  I've got test/CI systems, production, and my own development system wired together with spring-integation and rabbit and that's about it.  Unless someone else writes code that happens to break when it runs via spring-integration in continuous integration builds, other devs don't need to know anything about setting up and running messaging, and most stuff just works. Obviously, they know it is there, but it doesn't interject itself into their day-to-day coding very often.





--
Ravi Teja Lokineni | Software Engineer
SemanticBits India Pvt. Ltd.



Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

John Fletcher-3
In reply to this post by a.shneyderman
One thing where I like ActiveMQ over what I can see from Rabbit is networking brokers. I can run an ActiveMQ broker on my localhost and with a line or two in the config tell it to connect to a broker on a remote server and send it messages whenever the connection is available. The disconnect/reconnect handling logic is well done.
 
The upshot of this is that when I need to connect my application to the main message broker via an unreliable network connection, I can offload the connection handling issues to ActiveMQ  by running a broker on the same machine as my application and queueing messages there, as this broker is always available to my application. The broker will store messages and pass them on to the main broker whenever the link is up.
 
If I don't do this, I effectively have to write a message queue in my own code to handle connection errors, which defeats part of the purpose of running a message queue at all.
 
I understand that what I'm doing might be possible in Rabbit but I would have to install some plugin or another and it didn't sound like it was a primary use-case to me. So it may well work, I haven't tried it. But it works nicely in ActiveMQ.
 
John

2012/10/9 Alex Shneyderman <[hidden email]>

On Tue, Oct 9, 2012 at 5:35 AM, Jon Palmer <[hidden email]> wrote:

I’m hoping to get some advice on choosing a queuing technology for our Grails app. Our primary use case is enqueuing various long running background tasks. At the moment we’re using the jesque plugin but we’re increasingly finding it is not stable enough for our needs. I’m curious to understand what alternatives people would recommend? I see plugins for ActiveMQ, RabbitMQ, JMS and others. Which are stable and ready for prime time? Any thoughts would be much appreciated.


You can always go it alone. JMS stuff, you can setup without amq plugin, so is rabbit. As the matter of fact you should do it even if you decide to use plugin in the end. Plugins wrap around some spring integration. You can always setup spring integration with said broker without any plugin. Some of the things that are important for you to effectively use rabbit, for example, are very well hidden from you by the plugin. I doubt it is all that useful, especially once your code hits production and you have no clue where those exceptions that are logged coming from. Plugins do provide convenience. For example, rabbit provides sendRabbit method to your artifacts, but you can also wire rabbitTemplate and do the same thing. At the end I think you should read the source of the plugin if you decide to use it. It will let you understand how plugin sets up your context. 

As to whether to go with rabbit or JMS. I think the choice is clear if you have that choice. Rabbit wins in that contest, no matter how you look at it. ActiveMQ just does not provide you with enough juice. Its web console is pathetic as compared to Rabbit's. Rabbit's ability to route messages is not something that is provided by basic AMQ installation. The speed and stability of Rabbit is superb. It has out of the box clustering, which will allow you to scale and provides you with HA features if/when you need them (good reference is RabbitMQ in Action book, so is their web site). The only thing that I know of that beats Rabbit is ZeroMQ. Unfortunately, it requires a lot of fiddling, there are no plugins that I know of, and you have to completely change your view about MQ broker. But it is an option, not an easy one but still it is out there :-) 

Also, there is this http://grailsrocks.github.com/grails-platform-core/guide/events.html. You might find that to be useful if event buses is something you like. IMHO, they are bit harder to understand and manage then your regular centralized message brokers (something that is also true about ZeroMQ, BTW).

Cheers,
Alex.


Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

a.shneyderman
On Wed, Oct 10, 2012 at 11:06 AM, John Fletcher <[hidden email]> wrote:

> One thing where I like ActiveMQ over what I can see from Rabbit is
> networking brokers. I can run an ActiveMQ broker on my localhost and with a
> line or two in the config tell it to connect to a broker on a remote server
> and send it messages whenever the connection is available. The
> disconnect/reconnect handling logic is well done.
>
> The upshot of this is that when I need to connect my application to the main
> message broker via an unreliable network connection, I can offload the
> connection handling issues to ActiveMQ  by running a broker on the same
> machine as my application and queueing messages there, as this broker is
> always available to my application. The broker will store messages and pass
> them on to the main broker whenever the link is up.
>
> If I don't do this, I effectively have to write a message queue in my own
> code to handle connection errors, which defeats part of the purpose of
> running a message queue at all.
>
> I understand that what I'm doing might be possible in Rabbit but I would
> have to install some plugin or another and it didn't sound like it was a
> primary use-case to me. So it may well work, I haven't tried it. But it
> works nicely in ActiveMQ.

http://www.rabbitmq.com/federation.html

not really sure what you mean by primary case. Federation will hardly
ever be a primary case. It just needs to be possible at some point of the
product evolution.

plugin or no plugin at the end of the day do you really care how it is
done, as long as it works ...

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

John Fletcher-3


2012/10/10 Alex Shneyderman <[hidden email]>
http://www.rabbitmq.com/federation.html

not really sure what you mean by primary case. Federation will hardly
ever be a primary case. It just needs to be possible at some point of the
product evolution.

plugin or no plugin at the end of the day do you really care how it is
done, as long as it works ...
 
True, the plugin thing only matters if it's in the category "alpha/test/no-one uses". This federation link looks good, it's actually different to the plugin I identified when a looked into this a year or two ago. Very interesting.
 
John
 
Reply | Threaded
Open this post in threaded view
|

Re: Advice on Queuing technologies

Drake Emko
In reply to this post by jonspalmer
I've been using Amazon's SQS message queueing service, which is part of their AWS infrastructure. This may not be useful to you unless you are already hosting in their cloud, but I've found it to be very simple, reliable, and low-cost, and it does away with the headaches of hosting and administering your own MQ service.

On Oct 8, 2012, at 9:35 PM, Jon Palmer <[hidden email]> wrote:

I’m hoping to get some advice on choosing a queuing technology for our Grails app. Our primary use case is enqueuing various long running background tasks. At the moment we’re using the jesque plugin but we’re increasingly finding it is not stable enough for our needs. I’m curious to understand what alternatives people would recommend? I see plugins for ActiveMQ, RabbitMQ, JMS and others. Which are stable and ready for prime time? Any thoughts would be much appreciated.

 

Thanks

Jon

 

Jon Palmer | Software Engineering Manager

Care.com, Inc.
201 Jones Road | Suite 500 | Waltham, MA | 02451
(p) 781.693.1784 | (f) 781.899.1294 | (e) [hidden email]


<image001.jpg>


Care.com | There for you
For Children, Adults & Seniors, Pets, Home & Lifestyle


Become a Care.com Fan on Facebook:
http://www.facebook.com/caredotcom

This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.

Care.com and "There for you" are service marks or registered service marks of Care.com, Inc. © 2007-2012 Care.com, Inc. All rights reserved.