A Community for Hadoop Users

datanode can not connect to the namenode in a small hadoop cluster

Hey there I have a hadoop cluster build on 2 servers (2 laptops). One node (A)
contains the namenode, a datanode, the jobtraker and a tasktraker.

The other node(B) just has a datanode and a tasktraker.
I set up correctly hdfs with ./start-hdfs.sh

When I try to set up MapReduce with ./start-mapred.sh the TaskTraker of node (B) can not connect to the namenode. The tasktracker log will
keep throwing:

INFO org.apache.hadoop.ipc.Client: Retrying connect to server: mynamenode/192.168.0.13:8021. Already tried 8 time(s)

("mynamenode" and "192.168.0.13" are coorect)
I have properly set the properties:
fs.default.name
dfs.name.dir
dfs.data.dir
fs.checkpoint.dir
mapred.job.tracker
mapred.local.dir
mapred.system.dir,

I think maybe I am missing something in /etc/hosts file or this hadoop property is not set correctly:
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:50010</value>
<description>
The address where the datanode server will listen to.
If the port is 0 then the server will start on a free port.
</description>
</property>

I have noticed I can not telnet to the 8021 port even from the server containing the namenode (as localhost):
telnet localhost 8021
telnet 192.168.0.13 8021
Both cases I get:
telnet: Unable to connect to remote host: Connection refused

Has someone experienced this problem before?
*both servers can connect to each other via passwordless ssh without problems.
I am running hadoop 0.19.2
Thanks in advance

Comment

You need to be a member of Hadoop Professionals to add comments!

Join Hadoop Professionals

Comment by Marc Sturlese on February 15, 2010 at 3:07pm: Solved. The problem was the /etc/hosts. As far as I understand now all nodes must have the ip and hostname of all members of the cluster (and themselves, of course). So, my /etc/hosts for both nodes now looks like (I am running under ubuntu boxes):
127.0.0.1 localhost
192.168.0.13 mymasternode
192.168.0.10 myslavenode

Running jps command and web interfaces I can see everything works perfect now:
http://localhost:50030/
http://localhost:50060/
http://localhost:50070/

Comment by Marc Sturlese on February 15, 2010 at 8:01am: Yes, looking at the logs seems they are. Actually if I stop the daemonds with ./stop-hdfs.sh and ./stop.mapred.sh I can see the messages that namenode, jobtracker and the 2 datanodes and tasktrackers are being shut down.
When they are up I can use hadoop comands like fs -mkdir with no problems.
Looking at the mapred web interface just the tasktraker that is in the same machine as the namenode and jobtraker will appear (the one in the other machine is the one that fails trying to connect to the namenode).

Comment by Jason Venner on February 15, 2010 at 7:36am: Can you verify that the Namenode, and Datanode server processes are running.

datanode can not connect to the namenode in a small hadoop cluster

You need to be a member of Hadoop Professionals to add comments!

Groups

NoSql

Karma Sphere Studio Users

HBase Users

Hadoop Cascading

Hadoop Administrators