SolrCloud Cluster (Single Collection) Deployment

Update: revised an article up to Solr v4.9.x

Objective of this article is to provide insight & an idea on how to design SolrCloud Cluster (Single Collection) deployment.  I recommend you to check out an post SolrCloud (aka Apache Solr) to know Solr Distributed Terminology, Bird Eye: SolrCloud vs Classic Solr, Articles and upcoming Articles.


Designing SolrCloud Collection

Design phase is considered to be a vital & crucial phase of the project.  We are going to execute the following design plan for designing SolrCloud cluster.  Collection & Shard are logical elements in the SolrCloud cluster, multiple Solr core(s) collective brings SolrCloud formation. Underneath it just a physical Solr Core.

  • Creation of single SolrCloud collection named europe-collection in Europe Data Center
  • ZooKeeper ensemble with 5 replicated ZooKeeper server(s)
  • 3 Node(s) of SolrCloud Instance(s) with replication factor 3
  • 3 Shard(s) distributed among 3 SolrCloud Instance(s)
  • Manual distribution of Shard replica(s) among 3 Shard(s) on 3 Node(s)

Technical Elements:

  • Apache Tomcat 7.0.x (same steps applies other version too)
  • Apache ZooKeeper 3.3.x
  • Apache Solr 4.x

Kindly Note: The above described design plan is going to be executed in a single box as for as this article is concerned. The server instances will be simulated within the same box. However the design plan can be executed / simulated cross multiple boxes.

Directory structures are -

The described design has been represented in below diagram:


ZooKeeper Ensemble Deployment

Check out an article ZooKeeper Cluster (Multi-Server) Setup to get acquaintance with the steps to deploy a ZooKeeper ensemble.  Go ahead and deploy ZooKeeper server(s).


Handy SolrCloud ZkCLI Commands

SolrCloud comes with really handy ZooKeeper CLI commands for upload, download, linking configuration set to Collection.  Let’s prepare, it’s very simple.  All you have to do is

  • Extract the apache-solr-4.x.x.war
  • Get all the jar(s) from lib directory and place it in local directory

Let’s perform these steps:

Download Apache Solr 4 artifact to tmp directory

Extract zip file and war file in to tmp directory

Create a directory and copy it over there

Copying Logger libraries to solr-cli-lib [this step is applicable only to Solr 4.3 & above]

Now, we are ready to take advantage SolrCloud ZooKeeper CLI handy commands in sub sequent article sections.


Creating Solr Configuration schema.xml, solrconfig.xml, etc

Every organization is unique and has its own requirement and hence create Solr configuration files adhering to the  unique business requirements of the the organization.  Of-course every Solr deployment is unique in nature around the world with its schema (fields and data structure), solrconfig (analyzers, tokenizers, etc), stopwords, Boosting & Blocking, synonyms, etc.

Create your own solr configurations and place it in a directory.  For this article I’m planning to use example configuration shipped with Solr 4 artifacts.

Listing out copied Solr configuration


Uploading Solr Configuration into ZooKeeper ensemble

We have already prepared the required Solr CLI libs so let us use it now.  Listing down the points that we should be aware of -

  • ZooKeeper host addresses and client port numbers for zkhost param
  • Solr Configuration directory for confdir param
  • Configuration name in the ZooKeeper binding, let’s say myconf

Uploading a Solr Configuration

Linking Uploaded Solr configuration with collection

We have uploaded and linked solr configuration, lets verify it.

Connecting to ZooKeeper (we have five servers, let’s connect to one among the 5 servers)

Querying uploaded configuration and collection path in ZooKeeper

Looks good move on!


Deploying SolrCloud in Tomcat

In this section we will be setting up tomcat instance(s) and SolrCloud Cluster.  It would take more number of steps to understand the SolrCloud cluster deployment.  I’m going to describe those steps in simplified way.  Stay with me!

Step 1

Directory Structure creation for our deployment

Step 2

Downloading Tomcat 7 & Solr 4 artifacts

Step 3

Extracting downloaded artifacts and place it in respective directories

Step 4

Creating a setenv.sh for respective tomcat(s) serves for our customization.  I’m describing for one tomcat instance and similarly follow the same steps for other two tomcat servers with appropriate values (Solr home, app server port no.)

And place the following configuration snippet in it and FORGET NOT to save  setenv.sh

Incorporated notes from Flavio Pompermaier (via comment):

  • It is important to specify in the solr.xml the host parameter and than in SOLR_OPTS too otherwise every node will advertise itself with its IP & hostname (127.0.0.1 & localhost). Since its defaults to first localhost address found
  • If you keep solr configs in specific directory on ZooKeeper, kindly specify/append that directory name in the last node of the -DzkHost param.  For e.g.: SOLR_OPTS will look like-

Step 5

Creating solr.xml for Solr Home

And place following lines into it and save  solr.xml

Configuration for Solr v4.4.0 and above

Configuration up to Solr v4.3.1

Now copy above solr.xml to home2 and  home3

Step 6

Applying necessary permissions

Step 7

Updating tomcat server.xml for port number, shutdown port number. and comment out AJP connector (we don’t need it here).  After modification don’t forget to save server.xml

Tomcat 1:  /Users/jeeva/dc-1/tomcat1/conf/server.xml

  • Port No. => 7070
  • Shutdown port no. => 7005
  • comment out Java AJP connector

Tomcat 2:  /Users/jeeva/dc-1/tomcat2/conf/server.xml

  • Port No. => 8080
  • Shutdown port no. => 8005
  • comment out Java AJP connector

Tomcat 3:  /Users/jeeva/dc-1/tomcat3/conf/server.xml

  • Port No. => 9090
  • Shutdown port no. => 9005
  • comment out Java AJP connector

Step 8

Start the Tomcat(s) servers one by one and then access any of the tomcat server for e.g. http://localhost:8080/solr; and be surprised to see the page similar to one shown below

Starting Tomcat(s)

Screenshot – Solr Admin UI without Cores:

Solr Admin UI without Cores

Solr Admin UI without Cores

Now we are ready use Collections and CoreAdmin API’s to create our SolrCloud Collection.


Creating Collection, Shard(s), Replica(s) in SolrCloud

Let’s make use of Solr Collections and CoreAdmin API’s to create collection, shard(s), Replica(s) and replication factor.  These handy API lets you control solr core for replica’s on specific Solr node.

Creation a Collection ‘europe-collection’ and passing following parameters

  • action => CREATE
  • collection name => europe-collection
  • Number of Shards => 3
  • Replicator Factor => 3 (no. of document copies in the collection)
  • maxShardsPerNode => 3 (Since Solr v4.2, thanks to Ariel Lieberman)

Keenly notice to observe that we are not providing a collection.configName param because we have already linked the Solr configuration with Collection.

Above command creates collection, shard in each SolrCloud node and Solr core in each Shard.

Creating Shard replica’s and distributing into 3 Solr Node(s) as per above design

We are specifically mentioning Shard Name and choosing particular Solr node for creating a replica

Shard 1 Replica’s -

Shard 2 Replica’s

Shard 3 Replica’s

We have achieved SolrCloud replica distribution in a fashion as shown in following diagram

Now take a look at following solr.xml, and the newly created solr core configuration persisted in that.


Let’s Perform few documents indexing

Let us use the exampledocs shipped in Solr 4 artifcats for indexing and we will use each solr node we have created above.

Querying an indexed document by hitting the following URL in the browser and the result would display 5 documents.


Exploring Newly Created SolrCloud Cluster Availability

As per above design we have evenly distributed Shard and it’s replica among 3 Solr Node(s).  Hence we can expect good availability; because we have deployed the shard replica’s across Solr Node(s). Design used in this article is kind of startup; and this can be scaled further to deploy high availability clusters.  Let’s test it.

Before checking the availability verify the number of documents in the europe-collection =>  http://localhost:9090/solr/europe-collection/select?q=*:*

And this results in total 5 documents in number.

Stop/Kill Tomcat 1 (running on port 7070) – Solr Node 1

Query the Solr: http://localhost:9090/solr/europe-collection/select?q=*:*

This will provide 5 documents as search results.

Stop/Kill Tomcat 2 (running on port 8080) – Solr Node 2

Query the Solr: http://localhost:9090/solr/europe-collection/select?q=*:*

This will still provide 5 documents as search results.

Take a look at Graph in the Solr Node 3 (http://localhost:9090/solr/#/~cloud), and find the following representation of cluster.  Replica becomes leader and takes care of user queries.

solr-cluster-availability-check

Test it again with some indexing some more documents in Solr which is up and running, start the Solr Node 1 & Node 2 it will get sync with Node 3 in Recovery process of Solr cluster.


Your Journey Starts Here

I hope this article gives an idea & insight of deploying your SolrCloud Cluster (Single Collection) customized for your needs.  All the best!

Please leave a comment if you have any queries :)

  • I would like to add that we can also create multiple tomcat instances by using the java based portable utility which I developed for automating the steps required to edit the server config files.

    • ATP

      hi Java ,
      Nice technical setup and explanation, could you please guide how to query and search habase table data with help of solr?
      is there any reference please share, and how to direct solr to hbase.

      Thanks

  • alex

    Hi,

    What is the point of running zookeeper instances in localhost? I have read all available articles about solr cloud setup and all of them are setting shards, zookeeper and solr instances in one linux box. Why not consider a case with two linux boxes. In short what will be equivalent of pre solr4 architecture with two shards each one in a separate linux box?

    Thanks.

    Alex.

    • Hello Alex – Thanks for sharing your thought. Yes I have used the one box from article standpoint, we can use multiple box(s) based on capacity & availability planning of ZooKeeper and Solr 4. No blocker on that. It’s recommended to have multiple physical box(s)/VM slice(s) for Production environment.

      Equivalent in pre solr4 architecture; having master-solr (configured as repeater mode) and minimum two slave-solr(s), each linux box will have one slave-solr instance. Indexing on master-solr, search query(s) against slave-solr(s).

      Cheers,
      Jeeva

  • Somen

    Hi Jeeva,

    I’m having trouble uploading the Solr configuration to the Zookeeper ensemble.

    I first noticed the error because when I brought up Solr in a browser, it said:

    Unable to load environment info from null/admin/system?wt=json.

    This interface requires that you activate the admin request handlers in all SolrCores by adding the following configuration to your solrconfig.xml:

    Then I double checked the configuration upload. I used the command:

    java -classpath .:/Users/somen/dc-1/solr-cli-lib/* org.apache.solr.cloud.ZkCLI -cmd upconfig -zkhost localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2185 -confdir /Users/somen/dc-1/config-files -confname myconf

    However, after executing the command, at the very end of the output, it says:

    -confdir and -confname are required for upconfig

    I also noticed that when I query the uploaded configuration with “ls /”, configs does not show in the list.

    Please help me to understand why the configs may not be getting uploaded? Please let me know anymore info or logs you may need to see.

    Thanks,

    Somen

    • Hello Somen – Can you please share your logs to my email id. I will have a look.

      PS: Apologies for delay response, I was bit occupied.

      Cheers,
      Jeeva

  • Prateek

    Hello Sir,

    I am student experimenting with solr, i was trying to index an 11.3gb data of the type xml but couldn’t. i was using the commands given bove ie using post.jar on my linux machines.

    it gives errors like :

    “Exception in thread “main” java.lang.IllegalArgumentException: invalid content length

    at java.net.HttpURLConnection.setFixedLengthStreamingMode(Unknown Source)

    at org.apache.solr.util.SimplePostTool.postData(SimplePostTool.java:832)

    at org.apache.solr.util.SimplePostTool.postFile(SimplePostTool.java:746)

    at org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:478)

    at org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:398)

    at org.apache.solr.util.SimplePostTool.doFilesMode(SimplePostTool.java:276)

    at org.apache.solr.util.SimplePostTool.execute(SimplePostTool.java:169)

    at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:159)

    i am not sure why this has occurred, please help me out with this if you could.

    Why is it giving invalid content length is it because of some “r” or some other return characters?

    or is that post doesnt work with big files.

    Eagerly waiting for a reply.

  • Ariel Lieberman

    Jeeva,

    Thanks for very helpful post.

    In Solr 4.2 collections CREATE has been modified

    It creates the shard replicas too.

    curl ‘http://localhost:7070/solr/admin/collections?action=CREATE&name=europe-collection&numShards=3&replicationFactor=3&maxShardsPerNode=3′

  • Suppose we have 3 nodes of Solr. Is there a way to query by using a single url? Can zookeeper act like a load balancer and provide a single url to query the solr? Basically based on the load, zookeeper should route to any one of the free solr node.

    • Hello Aawardhan – SolrCloud by default handles requests in distributed way. For example: If you hit any solr node belongs to same collection, solr will internal query with other solr node of same collection returns a total results (http://localhost:8080/solr/europe-collection/select?q=*:*).

      Also you can control requests to shard, refer this link

      ZooKeeper is used to maintain Cluster State not a load balancer. If you like to have load balancer, please try out software/hardware load balancer.

      Cheers,
      Jeeva

  • Chris

    Hi, The curl create command is throwing a NPE with solr 4.2.1:

    Any ideas?

    curl ‘http://mysolr1.com:8080/solr/admin/collections?action=CREATE&name=collection&numShards=4&replicationFactor=1′

    50011java.lang.NullPointerException

    at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:149)

    at org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:277)

    at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:112)

    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)

    at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:591)

    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)

    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)

    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)

    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)

    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)

    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)

    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)

    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:722)

    500

    • Hello Chris –

      It seems Solr unable to communicate or Solr is not getting a expected response from ZooKeeper.

      As per Solr 4.2.1 Source Code:
      Line 149 # QueueEvent event = coreContainer.getZkController()
      .getOverseerCollectionQueue()
      .offer(ZkStateReader.toJSON(m), DEFAULT_ZK_TIMEOUT);

      So can you please verify:
      – Connect to ZooKeeper and verify Solr configuration presence (may be this reference will help)

      Let me know the outcome!

      Cheers,
      Jeeva

  • wanghua_zwh

    Hi,Jeeva,

    Thanks for very helpful post.

    I’m having trouble when I creation a Collection ‘europe-collection’ in solr4.2.1

    can you help me! thank you!

    [[email protected] dc-1]$ curl ‘http://localhost:7070/solr/admin/collections?action=CREATE&name=europe-collection&numShards=3&replicationFactor=3&maxShardsPerNode=3′

    400247org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Cannot create collection europe-collection. Value of maxShardsPerNode is 3, and the number of live nodes is 1. This allows a maximum of 3 to be created. Value of numShards is 3 and value of replicationFactor is 3. This requires 9 shards to be created (higher than the allowed number)Cannot create collection europe-collection. Value of maxShardsPerNode is 3, and the number of live nodes is 1. This allows a maximum of 3 to be created. Value of numShards is 3 and value of replicationFactor is 3. This requires 9 shards to be created (higher than the allowed number)400Cannot create collection europe-collection. Value of maxShardsPerNode is 3, and the number of live nodes is 1. This allows a maximum of 3 to be created. Value of numShards is 3 and value of replicationFactor is 3. This requires 9 shards to be created (higher than the allowed number)400

    [[email protected] dc-1]$

    • @wanghua_zwh – The reason you get this error is, value of maxShardsPerNode=3 requires minimum 3 solr node/instance running. That’s why solr complaining “the number of live nodes is 1″ in the exception.

      Cheers,
      Jeeva

  • cloud beginer

    Hello,

    Thank you for the helpful post!

    I would like to update that I had some problems in ver 4.3 since the logging jars were missing , I found them in …example/lib/ext/

    I have a question regarding shards deployment.

    I might want to have several shards belonging to the same collection to be deployed on the same server/tomcat. Because I have large indexes Its’ more efficient to split the index to several shards. But I don’t want them to be several cores because then they are still managed by the same solr instance. Before cloud this was very easy I just deployed new Solr instances on the same tomcat and then added them to the shards param (shards=s1,s2…)

    Now in solr cloud it looks like I can either deploy a new core or deploy an entire new tomcat. It looks like the SOLR OPTS (solr home param) are restricting me :

    SOLR_OPTS=”-Dsolr.solr.home=/Users/jeeva/dc-1/solr/home1 -Dport=7070 -DhostContext=solr..

    what is you suggestion ?

    There is a way to deploy several shards for the same collection on same tomcat ?

    Use Cores instead

    Deploy several tomcats on same machine?

    • Apologies for delay, I was on holidays.

      Yes, changes has happened in 4.3 version. Okay let’s move on to your queries.
      We can control solr.home value in ways-

      1. JVM system property: -Dsolr.solr.home=/Users/jeeva/dc-1/solr/home1
      2. JNDI lookup: in web.xml

      I would suggest you to go for multiple tomcat and join them in SolrCloud Cluster (ZooKeeper), it will give you best balance around in ongoing maintenance.

      Cheers,
      Jeeva

  • Gopal Bhat

    Hi Jeeva, Thanks 4 posting such a valuable experience.

    I have 2 important question

    1. how the application server(asume php service) which is calling solr shard collection select API will know which shard ip/host name to call in case of one of shard is down. Because in the select call the shard ip/host name is mentioned in the API url. So there is no automation redirect to available shard URL by zookeeper ensemble.

    2. I tried 4th shard tomcat4 and solrhome 4 but after all my setup of 3 shards. Is that mean once the solr cloud cluster is implemented we cannot add one more additional shard to cluster.

    Thanks once again, gr8 work buddy :)

    • Apologies for delay, I was on holidays. Let’s come to your quires:

      For question 1: SolrCloud automatically maintains cluster information with help of ZooKeeper ensemble. You don’t have to worry/or pay attention to.

      For Example:
      – Let’s say you have 4 instance in SolrCloud, look at cluster state in Solr admin. You will have 4 live node (check from any solr node)
      – Now Turn off one node, check again solr cluster state from available solr node. You will get 3 live node

      So, take is ‘from your application query any solr node in the cluster, your request is distributed automatically among live solr node and caller gets response’

      For Question 2:
      Actually, you can add servers without any issues to host shards until you reach num_servers = numShards(collection creation param) and you can add new
      replicas for existing shards whenever you need them. Underneath problem is only
      exists when you try to grow over your numShards initial configuration.

      Cheers,
      Jeeva

  • Malathi Subramaniam

    Hi,
    I have installed solr 4.2 with 8 cores and 8 shards (where each shard corresponds to one core (e..g shard1->core1, shard2->core2…shard8->core8 in solr.xml file). Currently, I am using tomcat with solr. I read few articles about routing i.e. compositeId. When I viewed the solr admin interface, the clusterstate has the router=>compositeId attribute. When I added documents (documentid being the unique key), the documents were evenly distributed among the cores which was expected. Since the index was getting too big, at that stage I thought I will add documents to a particular core. Now I am wondering if I am able to add documents to a particular core (shard). Because, when I added a shardkey (e.g shard1) to the documentid i.e. shard1!12345 where 12345 is the documentid, the documents were added to the core8 instead of core1. When I changed the documentid to shard4!12345, the document is still added to core8. In some posts I read that _shard_ parameter has to be sent but I was unable to find any examples. Could you please clarify the below questions and correct me if I was missing something or misunderstood the concept?
    1. For the above routing method, can I explicitly mention the shard id (using _shards_ parameter)when the documents are added?
    2. If I am able to send the _shard_ parameter, can you provide an example. Currently I am adding documents from an xml file using the URL http://localhost:8080/solr/update?commit=true

    Thanks,
    Malathi

    • Hello Malathi – Your understanding is correct. Before we get into your questions, I would like to share some input for you.

      * If you specify numShards when you create a collection, you will use the “compositeId” router by default, which will then allow you to send documents with a prefix in the document ID
      * If you did not create the collection with the numShards parameter, you will be using the “implicit” router by default. In this case, you could use the “_shard_” parameter or a field to name a specific shard

      You can see which router is assigned to each collection in the clusterstate.json by checking the “router” attribute.

      Please find my response below-
      For Question #1: Yes, you can send _shard_ parameter, while adding document
      For Question #2: I believe now you know, how to send _shard_ value. i.e.
      [composite_id/_shard_]![document_id]

      Also, Just have a look on http://searchhub.org/2013/06/13/solr-cloud-document-routing/

      I hope this helps!

      Cheers,
      Jeeva

      • Malathi Subramaniam

        Hi Jeeva,

        Thanks for you reply. Can you given an example for question1?
        Even when I pass the shardid (without compositeid) as a prefix to document id irrespective of whether it is shard1 or shard2,.. or shard8, the documents were added to core1 instead of corresponding core. When I added a completely different prefix like apple!documentid, the documents were sent to a different core i.e core3 in my example. It is evident that the hash generated for the prefixes shard1, shard2,…. shard8 were similar and the documents were sent to the same core. So, what I did was, I have modified the url to add documents to a particular core as http://localhost:8080/solr/core8. So that documents will be sent to correspoding core herre it is core8. If I want to send the documents to core4, I have changed the url to http://localhost:8080/solr/core4 and there is no need to specify the prefix to the documentid.

        Thanks,
        Malathi

        • Hello Malathi – Typically while indexing, whichever the core you use; always all the document goes to leader then leader decides the place for the document.

          As you described (using specific URL) that is strange. It seems to me some misconfiguration.

          I don’t have a example program handy with me.

          Cheers,
          Jeeva

  • Alessandro Borriello

    Hi,

    I configured solrCloud with 2 shards. Solr is deployed within Tomcat. If I access the solritas (/browse) I get java.lang.ClassNotFoundException: solr.VelocityResponseWriter.

    Shall I upload some libraries into zookeeper?

    Thanks in advance.

    Alessandro

    • Hello Alessandro – If you get any Class not found exception in Solr. Collect required jars and put in [solr-home]/lib directory, where the solr directory is same that contains solr.xml used to configure solr cores you will also need to add following line to solrconfig.xml

      For this issue, ensure following library from following directories are present in solr/lib –
      apache-solr-*/dist/apache-solr-velocity-*.jar
      apache-solr-*/contrib/velocity-tools-*.jar
      apache-solr-*/contrib/velocity-*.jar
      apache-solr-*/contrib/commons-collections-*.jar
      apache-solr-*/contrib/commons-beanutils-*.jar

      Cheers,
      Jeeva

  • sathish

    Hi,

    I have a question on loading core in solrcloud, i have uploaded the db_data_config, solrconfig files into solr cloud. I have solrcore.properties files specific to core can this be upload into zookeeper?

    Below is solr.xml for Core ABC, core is not created using below solr.xml. Is this correct way of referring?

    • @sathish – Yes, it can be uploaded into zookeeper and also you can customize the solr.xml then upload it to zookeeper, there is no harm in that. Everything depends on your requirement.

      Cheers,
      Jeeva

  • Aditya

    Hi, I have created 3 server solr cloud and 1 zookeeeper and I am facing the following problems with my set up.

    1) When I create a new core using the collections API , the core are created, but all are in down state.

    I edited the clusterstate.json to get them active and then they become active.

    2 ) I have configured a collection to have 2 shards and 2 replicas and added documents to the collection. But when I query the servers, I am getting inconsistent results. I have in one shard 241 documents and in another 230 documents. When I query any server in the cloud I get 471, 230 and 241. Could you suggest as to where the problem might be.

    Thanks,
    Aditya

  • Guest

    Error with “Uploading a Solr Configuration”

    Exception in thread “main” java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

    at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:66)

    at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:164)

    Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory

    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

    … 2 more

    • Ashok Allu

      I am also getting the same issue . I was not able to move forward anymore after this step

  • Guest

    Hi,

    I have an error with “Uploading a Solr Configuration”

    Exception in thread “main” java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

    at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:66)

    at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:164)

    Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory

    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

    … 2 more

  • Jacques Gariépy

    when I go to the “http:myipserver/8080/solr” is gives me the following error “HTTP Status 404 – /solr”

  • @jackboy70 – It seems solr war is not deployed properly along with solr.xml.

    Please share your error trace!

    Cheers,
    Jeeva

  • Try adding slf4j-log4j12.jar and slf4j-api-1.6.6.jar to lib directory

    Cheers,
    Jeeva

  • Pingback: SolrCloud Cluster (Single Collection) Deploymen...()

  • Marius M

    Hi Jeeva! I have some unclarities. In the overview of the system, let’s take node 1. Does that node represent a server? If yes, there are two replicas there, who do they replicate there ? shard2 and 3? thanks!

    • Hello Marius – Yes, node represents single tomcat instance. As per article approach each shard replicas reside in other two nodes respectively to ensure availability & failover.
      For example: Shard1 in node1 and replica of shard1 reside in node2 & node3 respectively.

      Cheers,
      Jeeva

  • Jaya

    Hi Jeeva,

    Please refer to below link for my Solr Cluster design. I am setting up for 2 zookeeper and 2 solr instance.

    http://wiki.apache.org/solr/SolrCloudTomcat

    Below are my issues :

    1. When I access using admin UI, I can always see only Leader node is active while 2nd node is always in recovering mode. How can I bring the recovering node to active ?

    URL : http://localhost:8080/solr-4.4.0/#/~cloud
    Navigation : Cloud–>Graph (left side of Admin UI)

    2. I added one document from leader node and try to access the same from 2nd node which is in recovering status. I am getting below error.

    ERROR
    RecoveryStrategy
    Error while trying to recover.
    core=collection1:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
    Server at http://localhost:8080/solr returned non ok status:404,​
    message:Not Found

    Can you please provide solution for these two issue.

    Thanks in advance !!

    Cheers !!
    Jaya

    • @disqus_BxJUKxTtEx:disqus – Mentioned article look fine. My inputs for you!

      For #1: You have mentioned 2 ZooKeeper servers. Typically ZooKeeper ensemble servers count should to be in odd number to form a successful QuorumPeers. So try 1, 3, 5, 7 so on..

      For #2: During a recovery state, you will not be able to access Solr node.

      Sort out the first one! it may solve your issues.

      Cheers,
      Jeeva

      • Jaya

        Hi Jeeva,

        I really appreciate your prompt response on my query. As suggested by you I tried with 1 and 3 zookeeper with 2 Solr instance. In both case(with 1 zookeper and 3 zookeeper) only my leader Solr node shows active while 2nd Solr node is in recovering status.

        Do we need solr_cluster.properties here to maintain active status of each node. If yes can you please provide some details on this one.

        Cheers !!
        Jaya

        • @disqus_BxJUKxTtEx:disqus – Hmm, node recovering status still bothering you. Can you please drop me an email (jeeva at myjeeva dot com) with your configuration files, if possible will try to simulate mentioned design structure, then share my observation.

          Cheers,
          Jeeva

  • sathish

    Hi, while starting a node following approach, getting below error .

    SEVERE: Unable to create core: collection1
    java.lang.IllegalArgumentException: Path must not end with / character
    at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1020)
    at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:202)
    at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:199)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
    at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:199)
    at org.apache.solr.cloud.ZkController.createCollectionZkNode(ZkController.java:970)
    at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:937)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:624)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
    Oct 7, 2013 8:47:37 AM org.apache.solr.common.SolrException log
    SEVERE: null:org.apache.solr.common.SolrException: Unable to create core: collection1
    at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1654)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1039)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
    at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:624)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException: Path must not end with / character
    at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1020)
    at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:202)
    at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:199)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
    at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:199)
    at org.apache.solr.cloud.ZkController.createCollectionZkNode(ZkController.java:970)
    at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:937)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1031)
    … 10 more

    • @disqus_h1JDfRBD5K:disqus – As per exception, it seems ZooKeeper config path issue. can you please share directory structure of ZooKeeper?

      Cheers,
      Jeeva

  • Grigory Kazachenkov

    Hello, thank you for such a great article. I’m using Solr 4.5 and i saw few differencies with collection creating guidelines defined in your article. For example all replicas are created automatically on all nodes after first CREATE command. And many jar’s must be copied to lib directory in solr/home dir.

    But that’s not a problem. The problem problem that I encountered is about node shutdown (for example one with 7070 port. Other nodes are active and healthy). I shutdown tomcat with port 7070. And then start in up. After node is started, it is shown in down state on graph. And on localhost:7070/solr page it’s shown that No cores are available.

    And i have no idea how to make it working. Could you please help me?

    P.S. Collection is empty. It’s created, but i didn’t post any content there

    • @grigorykazachenkov:disqus – Will look have on Solr 4.5 differences, share for information. However, it should be fine :)

      It seems leader election or node recovery state is causing a issue, could you please share configs and details to jeeva at myjeeva dot com

      Cheers,
      Jeeva

  • Balamurugan Ramasamy

    Hi Jeeva,

    I followed all these steps, did set up zoo keeper (3 instances), and tomcat (4 instances on ports 6060, 7070, 8080 and 9090). However i get “Solr instance is not running in SolrCloud mode” message when trying to create the shards and replicas by this url “http://localhost:6060/solr/admin/collections?action=CREATE&name=liferay-collection&numShards=2&replicationFactor=2&maxShardsPerNode=1″

    Can you please help?

    • Share your configs by email ‘jeeva at myjeeva dot com’? Article flow will sure get your SolrCloud :)

      As per error, it seems misconfiguration.

      Cheers,
      Jeeva

      • Balamurugan Ramasamy

        Thanks Jeeva for the reply. I later fugure out that i had done misconfiguration in setenv.sh, and i have resolved it now. Thanks anyway.

  • Flavio Pompermaier

    Amazing guide! Just to share my experience that can save a couple of hours when deploying in production: it is important to specify in the solr.xml the host parameter and than in SOLR_OPTS too otherwise every node will advertise itself with its hostname IP (im my cas every node was advertising itself with 127.0.0.1 IP…). Moreover, if I put solr stuff under a folder in zookeeper (and not in the root as in this guide) append the folder only to the last node )in my case I had the solr stuff in the solr folder in zookeeper). Example of my SOLR_OPTS:

    SOLR_OPTS=”-Dsolr.solr.home=/opt/solrhome -Dhost=myhost1.mydomain.org -Dport=7070 -DhostContext=solr -DzkClientTimeout=20000 -DzkHost=myhost1.mydomain.org:2181,myhost2.mydomain.org:2181/solr”

    • @flaviopompermaier:disqus Thanks, I have incorporated your notes in the article.

      Cheers,
      Jeeva

  • KRiddolls

    Hi Jeeva,
    I’ve been following this exercise to gain more knowledge of SolrCloud (same configs). Thank you for posting, it’s been very helpful so far.
    The only difference is I’m doing this on windows 7 professional 64 bit.

    I’ve made it through most of the exercise with expected results but things blew up on Step 8.
    When I try to reference the tomcat instance through the browser I get the following error, which seems to suggest it can’t find solrconfig.xml.
    Note: I did see solrconfig.xml in the zookeeper directories under myconf.
    I also find this path odd
    Error loading solr config from solr/collection1solrconfig.xml

    Chrome browser error:
    {msg=SolrCore ‘collection1′ is not available due to init failure: Could not load core configuration for core collection1,trace=org.apache.solr.common.SolrException: SolrCore ‘collection1′ is not available due to init failure: Could not load core configuration for core collection1 at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:753) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:307) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2442) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2431) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.solr.common.SolrException: Could not load core configuration for core collection1 at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:66) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:554) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:261) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:253) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) … 1 more Caused by: org.apache.solr.common.SolrException: Error loading solr config from solr/collection1solrconfig.xml at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:148) at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:79) at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61) … 9 more Caused by: java.io.IOException: Can’t find resource ‘solrconfig.xml’ in classpath or ‘c:Solrdc-1tomcat2binsolrcollection1conf’ at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:362) at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:308) at org.apache.solr.core.Config.(Config.java:116) at org.apache.solr.core.Config.(Config.java:86) at org.apache.solr.core.SolrConfig.(SolrConfig.java:161) at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:144) … 11 more ,code=500}

    • @kriddolls:disqus based on exception it seems “-Dsolr.solr.home” property value is missing. It is trying to load a solrconfig.xml from incorrect location.

      Can you please verify it and let me know?

      Cheers,
      Jeeva

      • KRiddolls

        Thanks for the quick reply Jeeva,
        This is my SOLR_OPTS setting in setenv.sh. I was mimicking your setting with the slight directory change. the full path is C:Solrdc-1solrhome1 Once I get to dc-1 the directory structure is the same as what you’ve provided.

        SOLR_OPTS=”-Dsolr.solr.home=/Solr/dc-1/solr/home1 -Dhost=solr1.example.org -Dport=7070 -DhostContext=solr -DzkClientTimeout=20000 -DzkHost=localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2185″

      • KRiddolls

        Hi Jeeva, I’ve replied to this once and it seems to have disappeared. hopefully I’m not replying twice.

        Here is my SOLR_OPTS setting in setenv.sh

        SOLR_OPTS=”-Dsolr.solr.home=/Solr/dc-1/solr/home1 -Dhost=solr1.example.org -Dport=7070 -DhostContext=solr -DzkClientTimeout=20000 -DzkHost=localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2185″

      • KRiddolls

        I think it’s not finding the path I’ve set so it’s ignoring it and the error is caused…
        I need to determine what the right path should be for windows.

        • @kriddolls:disqus for windows you can use path in following formats C:/Solr/dc-1/solr/home1 or
          C:Solrdc-1solrhome1

          Cheers,
          Jeeva

          • KRiddolls

            Thanks for your help Jeeva.
            I had a few more errors…. several actually.
            When running in windows setenv.sh… needs to be a batch file.
            setenv.bat I had left this as .sh I should have recognized that.

            I had to change the variable settings to match .bat windows commands.

            This is what I ended up with and it’s working now:

            file: setenv.bat

            rem !/bin/sh
            rem author: Jeevanandam M.

            set “JAVA_HOME=%JAVA_HOME%”

            rem run JVM in server mode
            rem memory
            rem Timezone and JVM file encoding
            rem also addedin SOLR_OPTS
            set “JAVA_OPTS= -server -Xms128m -Xmx2048m -XX:PermSize=64m -XX:MaxPermSize=128m -XX:+UseG1GC -Duser.timezone=UTC -Dfile.encoding=UTF8 -Dsolr.solr.home=C:Solrdc-1solrhome3 -Dhost=solr1.example.org -Dport=9090 -DhostContext=solr -DzkClientTimeout=20000 -DzkHost=localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2185

            rem solr.solr.home => solr home for this app instance
            rem host => hostname, defaults to first localhost address for.eg.: solr1.exmaple.org
            rem port => app server port no.
            rem hostContext => wepapp context name
            rem zkHost => ZooKeeper ensemble host names and its client port no.
            rem zkClientTimeout => ZooKeeper client timeout
            set “SOLR_OPTS= -Dsolr.solr.home=C:Solrdc-1solrhome1 -Dhost=solr1.example.org -Dport=7070 -DhostContext=solr -DzkClientTimeout=20000 -DzkHost=localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2185″

            I can now move on! Thanks!

          • Okay, good to hear!

            Of course you need a .bat file for windows, forgot to mention that; sorry!

            Cheers,
            Jeeva

  • Bala Kumar

    Hi Jeeva ,

    I have a requirement to implement search and retrieve information from Rich documents(PDF,XLSX,DOC and TXT) which are in DFS . Can u please help me out how solrcloud ll fit in this ? I am completely new to Solr and SolrCloud .

  • @disqus_VeyKfDOOor:disqus Solr fits here “Indexing rich documents from DFS and searching those rich documents from Solr”

    As you described rich documents in DFS. so you have configure Solr to index those rich document from DFS using http://wiki.apache.org/solr/ExtractingRequestHandler then you can search a those documents from Solr.

    I hope it gives direction for you.

    Cheers,
    Jeeva

    • Bala Kumar

      Hey Thanks for your reply… Sorry i made a mistake . The Rich Documents i have mentioned needs to be in DFS , then i should be able to search(Contents) it . So is there any way in solr where i can put the documents in DFS and then i can follow the above step u have mentioned .

    • Bala Kumar

      http://wiki.apache.org/solr/ExtractingRequestHandler .
      From the link i did the following steps :
      1. Copied the files i need to post to exampledocs (Where post.jar is present) .
      2. Used java -Dauto -jar post.jar *.doc *.xls *.pdf command to post the documents to solr .
      3. Then using this http://localhost:8983/solr/select?q=Gradient i was able to query the document .

      Questions :
      1. Are these steps i had followed is right ?
      2. I done these in my system so does this work in one Master and one Slave basis ?
      3. If its right , How can i make a cluster(DFS) ?

      Thanks in Advance .
      (Please bare with me , I am completely new to Solr )

  • 1. Followed steps are fine
    2. There is no master/slave concept in SolrCloud (from Solr 4 onwards)
    3. What do you mean DFS? in general DFS is distributed file system. Which cluster are you referring Solr Cluster or DFS cluster?

    Cheers,
    Jeeva

    • Bala Kumar

      1. Actually i didn’t defined any schema for the data . I just followed the steps and posted data . Will the above step works well with large amount of data ?
      2. I need to store the data in the form of Distributed file system and use solr above it for querying the data.Is it possible ?

      Awaiting for your response.
      Bala

  • @disqus_VeyKfDOOor:disqus Sorry for late response. I was bit occupied with my work. I hope you have sorted out the issue.

    Cheers,
    Jeeva

  • Praveena

    Hi Jeeva,

    I followed the steps as mentioned.. and im trying to create a 2 shard, 2 replica(on 4 nodes).

    When i added the collection the create failed and i got error saying
    org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://solr3.example.org:7072/solr

    I had to remove the -DHost=”” option in the SOLR_OPTS

    After that deleted the collection and recreated it.

    But though i can see the success response, i dont see the info persisted in solr.xml. Hence i am not able to see the cloud graph in the UI. I am using SOLR 4.6.1.

    What can I do to see the graph in the SolrCloud ui?

    Thanks and Regards,
    Praveena

  • Roy

    Hi Jeeva,

    I am setting up for 3 zookeeper and 2 solr instance.

    Here is my issue :

    1. When I access using admin UI, I can always see only Leader node is active while 2nd node is always in recovering mode. How can I bring the recovering node to active ?

    Thank you in advance.

    Roy.

  • Dhaivat

    Hi Jeeva,

    Thank you so much for writing this article on solr cloud. It’s very helpful. It’s not just setting up but it also explains the terminology of solr cloud. I tried to follow the instructions you have written here. However I am getting one exception while creating the collection.

    Exception
    ====

    org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/europe-collection

    I have setup zookeeper and solr on single machine.

    Any idea why this is happening?

    Many Thanks
    Dhaivat

  • Shuddhashil Ray

    Hi Jeeva,

    First of all thanks for the detailed description! I stepped on a situation where I need some clarification, if you could kindly elaborate

    Shard 1

    ————————-
    Leader A – Running
    Replica B – Running
    Replica B – goes down for some reason
    Leader A – is running and receives lets say x number of updates
    Leader A – then goes down for some reason
    Replica B – then starts up and keeps on trying to recover without success.

    Can Replica B ever become the leader in case A never comes up? Else Shard 1 is literally down without any Leader.

  • perezbjc

    HI,

    I am getting this error in catalina.out

    ERROR org.apache.solr.core.CoreContainer – Error creating core [collection1]: Could not load conf for core collection1: Error loading solr config from solr/collection1/solrconfig.xml

    org.apache.solr.common.SolrException: Could not load conf for core collection1: Error loading solr config from solr/collection1/solrconfig.xml

  • perezbjc

    beside this file /Users/jeeva/dc-1/solr/home1/solr.xml,

    which other files are in this directory? Am I supposed to copy the example folder in here as well?

  • sumit

    when i am trying to create collection using below url:-

    http://localhost:7070/solr/admin/collections?action=CREATE&name=europe-collection&numShards=3&replicationFactor=3&maxShardsPerNode=3

    getting below exception.Can you please help why i am getting this issue?

    400342org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Cannot create collection europe-collection. Value of maxShardsPerNode is 3, and the number of live nodes is 1. This allows a maximum of 3 to be created. Value of numShards is 3 and value of replicationFactor is 3. This requires 9 shards to be created (higher than the allowed number)Cannot create collection europe-collection. Value of maxShardsPerNode is 3, and the number of live nodes is 1. This allows a maximum of 3 to be created. Value of numShards is 3 and value of replicationFactor is 3. This requires 9 shards to be created (higher than the allowed number)400Cannot create collection europe-collection. Value of maxShardsPerNode is 3, and the number of live nodes is 1. This allows a maximum of 3 to be created. Value of numShards is 3 and value of replicationFactor is 3. This requires 9 shards to be created (higher than the allowed number)400

    • @disqus_zwyr8hXwMA:disqus – This is happening because of your no of. live nodes. As per exception, you have only 1 live node but requesting to create 3 shardspernode. it won’t happen. Please follow the complete article, which will help you to create 3 live node then trigged this command.

      I hope it helps!

      Cheers,
      Jeeva

  • senthil_jr

    Hi Jeeva,

    Is there a way to deploy Zookeeper on weblogic cluster.

    Regards,

    Senthil

    • Cesar Fuentes

      i have the same question, @Jeeva is this possible?? Thanks for your, reply

  • Arun

    Hi Jeeva, I have solrCloud with solr 4.8.0 on hadoop cluster and I need to upgrade it to 4.10.2. I was thinking of copying all the collection information from solr-4.8.0/example/solr to solr-4.10.2/example/solr. Also, modify the startup & shutdown commands to use the same set of zookeeper ensemble. Is this going to be sufficient ? I didnt find anywhere on internet on how to upgrade from one working solr version to a later version. Appreciate your inputs

    • @disqus_KrOQP3BYlp:disqus – It is sufficient to achieve upgrade. If you come across any issues, post it here!

      Cheers,
      Jeeva

  • jyotiska biswas

    Hi Jeeva,

    Great tutorial. However, I’m running into 503 errors. I’m using Zookeeper 3.4.6, and Solr 4.10.3 on Windows. However, I have set it up the way you said. I am using 3 zookeepers ensemble (on 2181, 2182, and 2183), and two tomcats (tomcat1, and tomcat2) of solr instances. However, on the step “Screenshot – Solr Admin UI without Cores:” I see 503 Server is Shutting Down or Failed to Initialize. However, on the Tomcat Web Application Manager, I see that “solr/” as running. Do you have any idea what would cause this 503 error. In the tomcat/lib directories, I have copied the files from “tmpsolr-4.10.3examplelibext”. I have been stuck on this issue for several days. Any assistance would be immensely appreciated!

    • @jyotiskabiswas:disqus – Can you share exception details?

      Cheers,
      Jeeva

      • jyotiska biswas

        Hey Jeeva, thanks for your quick reply. I am not really getting any exceptions, however I am getting the following log4j warnings when I run the tomcat server with “tomcat1binstartup.bat”:

        log4j:WARN No appenders could be found for logger (org.apache.solr.client.solrj.impl.HttpClientUtil).
        log4j:WARN Please initialize the log4j system properly.
        log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

        I also get the following warnings when I upload my Solr configuration and linking them to zookeeper
        > java -classpath .;C:solr_cloudsolr-cli-lib* org.apache.solr.cloud.ZkCLI -cmd linkconfig -collection collection1 -confname myconf -zkhost localhost:2181,localhost:2182,localhost:2183

        log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
        log4j:WARN Please initialize the log4j system properly.
        log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

        I’m not sure if that helps, but please let me know what information I can give you to further help me.

      • jyotiska biswas

        As far as i know, no exceptions are being thrown. I copied and pasted the solr.war into my tomcat/webapps. When I run startup.bat, the solr.war gets deployed; however, when I try to access it I get 503. Is there any log file you want to see in particular?

  • Bharat

    Hi Jeeva,

    I am looking for multiple DIH configurations into solrCloud – 5.0
    Should I need to create 5 collections or there is any other way to configure it?

    Thanks!

  • Oded Sofer

    Do you know of any limitation with the number of nodes of single collection and single zooKeeper? I have 50 nodes in my solrCloud and planning to enhance it to 250 nodes.

    • @odedsofer:disqus – As for as I know no arbitrary limits in number collections or number of nodes in single collections. However you might end up with single ZK limitations on maintaining cluster state if you go over 1000 nodes or near by. See https://issues.apache.org/jira/browse/SOLR-5381

      As per your numbers you should be good to go!

      Cheers,
      Jeeva

  • Landon Campbell

    Hi Jeeva, I have another question (gave up on basic authentication, using iptables instead): I just set-up a SolrCloud with 3 separate ZooKeeper instances on three separate servers, and everything is fine, except I can’t upload my configuration! I’m not getting any errors, but the config never shows up in ZooKeeper. This worked perfectly when I was running with all zk nodes on a single machine, but now, no luck. All the ports are open, and the communication between all the parts of the cloud are fine, just no config. Do you have any thoughts what might be the problem? Thanks.

  • SR40

    Where would I add bootstrap in this configuration??

  • JLamb

    Where would I add bootstrap in this configuration?

  • @sr40:disqus – It’s typically at solr.home/conf in the solr instance where you have configured DIH handler.

  • TwylaNix1

    Nice suggestions ! With regards to witch , if anybody is interested in merging of PDF files , my business stumbled across post here http://goo.gl/iNhqTc