ElasticSearch vs Searchable plugin

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ElasticSearch vs Searchable plugin

jasenj1
We're at the point with an app we're developing to start adding search features. I see ElasticSearch is the new Compass. Is the ElasticSearch plugin ready to replace Searchable? I need something fairly stable and can't be chasing frequent releases and dealing with beta-type bugs. But I also don't want to adopt something that is being abandoned.

- Jasen.
mcf
Reply | Threaded
Open this post in threaded view
|

Re: ElasticSearch vs Searchable plugin

mcf
So... why not using Solr?

----
Marcelo Carvalho Fernandes
+55 21 8272-7970
+55 21 2205-2786


On Mon, Mar 12, 2012 at 11:04 AM, jasenj1 <[hidden email]> wrote:
We're at the point with an app we're developing to start adding search
features. I see ElasticSearch is the new Compass. Is the ElasticSearch
plugin ready to replace Searchable? I need something fairly stable and can't
be chasing frequent releases and dealing with beta-type bugs. But I also
don't want to adopt something that is being abandoned.

- Jasen.

--
View this message in context: http://grails.1312388.n4.nabble.com/ElasticSearch-vs-Searchable-plugin-tp4466097p4466097.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: ElasticSearch vs Searchable plugin

jasenj1
My understanding is that Solr is a stand-alone Lucene server. I don't need a separate server at this point.
The searchable plugin uses Compass which is built on top of Lucene. So I'm going to guess there is a way to point the searchable plugin at a Solr server if we want to.

The Solr Grails plugin looks very, very, out of date. Maybe it works fine and doesn't need any updating. But I don't like to use plugins that haven't been updated in over a year - especially since Grails has jumped to 2.0.

The searchable and elasticsearch plugins provide a handy way to annotate domain classes with search info. I'd like to stick with that model.
Reply | Threaded
Open this post in threaded view
|

Re: ElasticSearch vs Searchable plugin

Enrique Medina Montenegro
In reply to this post by jasenj1
Jasen,

I'm afraid you're misunderstanding what Elasticsearch is. Let me explain.

Elasticsearch is a distributed, fault tolerant, scalable index server on top of Lucene, create by the same guy who implemented Compass in the past. So it's not a new Compass, but a whole index server in the cloud (which probably takes lots of ideas from Compass, sharing the same author). Elasticsearch can be compared to Solr from the "index server" point of view, if you will.

Grails has a plugin for Elasticsearch which automatically and transparently starts up the Elasticsearch server, and provides transparent mapping from Grails domain classes into their JSON representations (serializer/deserializer). That's all.

Next, there's the Searchable plugin which is just a layer on top of Lucene (client API, not server like Elasticsearch or Solr) which again transparently maps domain classes into Lucene artefacts.

So Elasticsearch plugin will never replace Searchable plugin because they live at different abstraction levels, if you see what I mean.

Regards.

On Mon, Mar 12, 2012 at 3:04 PM, jasenj1 <[hidden email]> wrote:
We're at the point with an app we're developing to start adding search
features. I see ElasticSearch is the new Compass. Is the ElasticSearch
plugin ready to replace Searchable? I need something fairly stable and can't
be chasing frequent releases and dealing with beta-type bugs. But I also
don't want to adopt something that is being abandoned.

- Jasen.

--
View this message in context: http://grails.1312388.n4.nabble.com/ElasticSearch-vs-Searchable-plugin-tp4466097p4466097.html
Sent from the Grails - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: ElasticSearch vs Searchable plugin

Mike Powers
Hello,
  I just thought I would share my experiences with both ElasticSearch
& Searchable. I should start off by sayings that we are learning as we
go, I am not an expert in any of this....and we like both plugins for
different reasons.

Over the past few years, I have used the searchable plugin on many
different Grails applications. Most apps where fairly low traffic and
small datasets. In these cases, searchable worked really well for us.
It's pretty easy to setup, and a reasonable amount of documentation is
available (although we did have to figure out some advanced queries
through trial and error.)

On our latest project, we had a larger dataset to work with. In this
case, we currently have around 300k domain objects that required
indexing, and approximately 50k domain objects need to be re-indexed
daily. The re-indexing would occur in a quartz background job that is
running 24/7. We originally used searchable for this project as well,
but as our dataset grew, performance dropped off significantly.
(Searching would take roughly 8 seconds to return results).

Because of the slow performance, I decided to look into elastic search
as an alternative path forward. It did not take very long to switch
our project over to use elasticsearch, but we did run into a few
problems along the way. Ultimately it took a few weekends of patching
the elasticsearch plugin, and tweaking our production environment to
make ElasticSearch stable.

Admittedly, we are not experts with Lucene or Compass, maybe there was
something we could of done to make searchable run faster in this
environment. Just yesterday I learned about the 'optimize' command for
Lucene indexes (
http://stackoverflow.com/questions/119994/should-an-index-be-optimised-after-incremental-indexes-in-lucene
).... I have no idea if searchable will automatically do this for you,
or it's up to you to optimize your indexes on occasion.

To summarize our experiences with both plugins:
searchable:
* works well out of the box
* relatively stable
* reasonable documentation
* doesn't seem to perform well if you have a background job constantly
re-indexing large amounts of data.
* I doubt very much effort will go into developing Compass /
searchable further (maybe I'm wrong?).
* its difficult to pull searchable off your web server, and onto a
separate server. It is also difficult to cluster / sync indexes for
projects that use searchable

elasticsearch
* plugin still needs some work
* elasticsearch service requires lots of file handles. I did not
realize this early on (guess I should read instructions :-)
http://www.elasticsearch.org/guide/reference/setup/installation.html
). We would hit a issue where elasticsearch would run out of file
handles, and it would take down our entire site. (even though our
elasticsearch service is run on a separate server from our tomcat
server...)
* we ran into a couple other problems with the elasticsearch plugin,
but wrapping the problematic code with try/catch blocks got us up and
running. Admittedly I have not had time to track down the true
problems, but the patched code seems to be working fine. I suspect
some of the elasticsearch issues we run into, are being triggered by
the fact that we do indexing in a background quartz job.
* elasticsearch can be run on a separate machine from your web server,
a big plus if you need to scale out and run on multiple servers.

Sorry this email seems a little disjointed, but I hope it helps give
you insight into some of the pros/cons of both plugins. At the end of
the day, both plugins saved us a lot of time, and got us pretty close
to workable full-index searching. Both required some work, but none of
the problems we hit where extremely difficult to solve or work around.

If you don't mind getting your hands dirty, and would like to help
improve the quality of elasticsearch, give it a shot. I'm sure the es
devs wouldn't mind feedback / bug reports / patches. I think
elasticsearch has a promising future, and I'm happy we made the
switch. If you don't need to re-index lots of data constantly, and you
only need to run your project on 1 server, searchable will probably
work fine, (and be pretty easy to setup).

 I suppose it depends on the scope of your project, but I didn't find
the actual migration from searchable to elasticsearch very difficult
for our project.

-Mike



On Tue, Mar 13, 2012 at 1:15 PM, Enrique Medina Montenegro
<[hidden email]> wrote:

> Jasen,
>
> I'm afraid you're misunderstanding what Elasticsearch is. Let me explain.
>
> Elasticsearch is a distributed, fault tolerant, scalable index server on top
> of Lucene, create by the same guy who implemented Compass in the past. So
> it's not a new Compass, but a whole index server in the cloud (which
> probably takes lots of ideas from Compass, sharing the same author).
> Elasticsearch can be compared to Solr from the "index server" point of view,
> if you will.
>
> Grails has a plugin for Elasticsearch which automatically and transparently
> starts up the Elasticsearch server, and provides transparent mapping from
> Grails domain classes into their JSON representations
> (serializer/deserializer). That's all.
>
> Next, there's the Searchable plugin which is just a layer on top of Lucene
> (client API, not server like Elasticsearch or Solr) which again
> transparently maps domain classes into Lucene artefacts.
>
> So Elasticsearch plugin will never replace Searchable plugin because they
> live at different abstraction levels, if you see what I mean.
>
> Regards.
>
> On Mon, Mar 12, 2012 at 3:04 PM, jasenj1 <[hidden email]> wrote:
>>
>> We're at the point with an app we're developing to start adding search
>> features. I see ElasticSearch is the new Compass. Is the ElasticSearch
>> plugin ready to replace Searchable? I need something fairly stable and
>> can't
>> be chasing frequent releases and dealing with beta-type bugs. But I also
>> don't want to adopt something that is being abandoned.
>>
>> - Jasen.
>>
>> --
>> View this message in context:
>> http://grails.1312388.n4.nabble.com/ElasticSearch-vs-Searchable-plugin-tp4466097p4466097.html
>> Sent from the Grails - user mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: ElasticSearch vs Searchable plugin

jasenj1
Thank you for the detailed responses. Very helpful.

Enrique Medina Montenegro wrote
Grails has a plugin for Elasticsearch which automatically and transparently starts up the Elasticsearch server, and provides transparent mapping from Grails domain classes into their JSON representations (serializer/deserializer). That's all.

Next, there's the Searchable plugin which is just a layer on top of Lucene (client API, not server like Elasticsearch or Solr) which again transparently maps domain classes into Lucene artefacts.
The mapping of domain objects to some sort of search engine/index is what I care about at my current level of understanding. And as long as there is a query language (Lucene) that can be used to build queries and get results back, that is good for now. Performance and scalability concerns will have to wait for serious evaluation until we see how our data acts.

It is encouraging to hear that migrating to ElasticSearch was not very hard.

My current understanding is that Compass shreds domain objects into Lucene indexes. The ElectricSearch plugin shreds domain objects into JSON which ElasticSearch puts into Lucene indexes. ElasticSearch also does a bunch of fancy stuff under the covers which may be useful if our dataset proves too big for Compass and however it stores the Lucene indexes.

I've started working with the searchable plugin and so far I'm happy with the way it maps domain objects to search fields.  It seems I would need to do this process regardless of the search engine/plugin chosen. I like that I'm able to do everything within the Grails app - vs. a completely separate Solr server.

The Solr plugin for Grails looks like it's been abandoned. So I'm not going down that road. If searchable/Compass proves inadequate I'll either give ElasticSearch a try or go to raw Solr.

Thanks again for helping this search newbie learn more.

- Jasen.
Reply | Threaded
Open this post in threaded view
|

Re: ElasticSearch vs Searchable plugin

Hanan
In reply to this post by Enrique Medina Montenegro
Thanks for your information; It really helped me