Katta and Solr

Replies to This Discussion

Permalink Reply by Jason Venner on August 27, 2009 at 2:51pm

I have my first big index build in hadoop complete today. there are a number of log messages that I need to go through, related to odd ball failures.
Most of them related to path issues, for finding the solr config data, or missing solr index files.
I don't at this point know if this is an issue with some of my task tracker configurations, or and issue with the solr-1301 patch and concurrent tasks on a tasktracker

• ▶ Reply

Permalink Reply by Jason Venner on August 30, 2009 at 11:50am

Jason Rutherglen and I have been having a hackathon this weekend, on this, including working on putting realtime support in.

Hopefully in the not to distant future some patches will appear.

• ▶ Reply

Permalink Reply by Jason Rutherglen on August 30, 2009 at 8:21pm

The jira issue is at http://issues.apache.org/jira/browse/SOLR-1395

• ▶ Reply

Permalink Reply by Jason Venner on August 30, 2009 at 8:23pm

We have made some good progress laying down the basics.
Getting the unit tests going is interesting as they require the solr and katta configuration to run.

• ▶ Reply

Permalink Reply by Jason Venner on August 31, 2009 at 7:08pm

Jason Rutherglen is coming over again and we are going to beat down on this some.
He has the first test cases beginning to run.

I have been deploying solr indexes into katta, and searching with lucene just as a test case, and it has been working for me.

• ▶ Reply

Permalink Reply by Jason Venner on August 31, 2009 at 10:36pm

Working on the unit test for the core classes that are used to plug into katta.
Clearly I have missed something as all the tests, when run from eclipse give me this:

localteSolrHome INFO: JNDI not configured for solr (NoInitialContextEx)

• ▶ Reply

Permalink Reply by Jason Venner on September 1, 2009 at 7:50am

Turns out that to run from eclipse I need to set up the test case execution directory as the configuration files are loaded by relative file system path, not by class path resources.
In this case, I needed to set the test case to run from test/org
${workspace_loc:solr-1395/test/org}
Now to work through the actual code that handles running the requests :)

• ▶ Reply

Permalink Reply by Jason Venner on September 1, 2009 at 10:40pm

Jason Rutherglen and I made good progress today.
I have the core piece that will run queries to the embedded solr servers running with basic unit tests
and the front end piece that will talk to katta to distribute the queries to the katta nodes that serve the indexes.

Jason Rutherglen has the index deployment working with tests and will hang my piece for the embedded servers in and tie up the communication piece tomorrow.

• ▶ Reply

Permalink Reply by Jason Venner on September 15, 2009 at 7:44am

Well, I have managed to deploy one of my hadoop built, solr shards into my katta cluster.

I spent quite a few hours trying to debug issues with my Katta master node exiting right away, which turned out to be a configuration error on my part.

I have been using an external zookeeper cluster, and I had set the hosts for the cluster in my katta.zk.properties file, but had left the zookeeper.embedded property to true. This resulted in the masterNode, starting, running 1 loop and exiting.

• ▶ Reply

Permalink Reply by Jason Venner on September 24, 2009 at 9:24am

Jason Rutherglen and I have been working on this quite a bit.
I have my first 24 node katta cluster with 65million documents over 160 shards running, with solr serving the queries.

I am working on getting the ruby blacklight visualizer up and running sensibly with my dataset.

• ▶ Reply

Permalink Reply by Jason Venner on October 2, 2009 at 8:56am

I now have a test case working with 9 shards and 65 million documents.
It took a bit to get the facet counting to aggregate correctly, updated patch later this weeked

• ▶ Reply

Permalink Reply by Saju K K on May 2, 2010 at 8:02am

Hi,

I would like to what could be the best practise in SOLR/KATTA/HADOOP integration .

I do have a small experimental cluster where i do have a katta cluster (with zoo keeper ) and the indexs in hadoop . Was also able to successfully deployed the hadoop cluster to katta cluster.

Also would like to know how to get SOLR integrated with the KATTA .

--Saju

--saju
Jason Venner said:

Well, I have managed to deploy one of my hadoop built, solr shards into my katta cluster.

I spent quite a few hours trying to debug issues with my Katta master node exiting right away, which turned out to be a configuration error on my part.

I have been using an external zookeeper cluster, and I had set the hosts for the cluster in my katta.zk.properties file, but had left the zookeeper.embedded property to true. This resulted in the masterNode, starting, running 1 loop and exiting.

• ▶ Reply

▶ Reply to Discussion