Hadoop Professionals - A Community for Hadoop Users

A Community for Hadoop Users

This network is a place to discuss and learn Hadoop, Solr, Katta, Map Reduce and a place to discuss hadoop resources such as hadoop books.

Members

View All

Latest Activity

Jason Venner replied to yigang chen's discussion 'Legacy code running in mapper'

You can build your own symbolic link by running a command from java, you just need to verify where the data is unpacked, and then build a link to it. A quick search turned up the following page for sample java code for you: http://www.giannistsakir…

yesterday

Jason Venner replied to sanjo cole's discussion 'hadoop or mysql?'

If your data is highly relational, your users will have a simpler time accessing it if it is stored in a more traditional data warehouse. The sizes you are talking about are very small, I have some of the higher end solid state devices for storage,…

yesterday

Rodrigo Carvalho Rezende and sanjo cole joined Hadoop Professionals

yesterday

yigang chen replied to yigang chen's discussion 'Legacy code running in mapper'

So that means I'd need to modify the legacy code, i.e., change the hard coded: "a/relative/path/to/my/file.xml" to: "./mymeta.zip/a/relative/path/to/my/file.xml" Is there a way at all to NOT change the legacy code?

on Monday

sanjo cole added a discussion

hadoop or mysql?

hi,i'm working on a data warehouse and am deciding whether to use hadoop or mysql.the dataset is currently likely to be no bigger than 40gb for the first year, then perhaps 80gb for the next year, and possibly 120gb the year after.we want to be able…

on Monday

Anand updated their profile

on Monday

Anand and yigang chen joined Hadoop Professionals

on Monday

dachuan huang replied to dachuan huang's discussion 'how could i setup a hadoop source project?'

where can i find this code bundle? at present, i just want to run some simple examples which just some internal classes are involed, it's not necessary to re-build the whole hadoop source project. thank you.

on Monday

Jason Venner replied to yigang chen's discussion 'Legacy code running in mapper'

If you pass -archives mymeta.zip there will be a symbolic link in the current working directory for the map or reduce task mymeta.zip, which points to the directory that the archive was unpacked in. so if you use ./mymeta.zip/path_in_archive/file.xm…

on Sunday

Jason Venner replied to dachuan huang's discussion 'how could i setup a hadoop source project?'

In the code bundle for the book ProHadoop is a full eclipse environment for running with either hadoop 18.3, 19.0 or 19.1. At present I typically use that and maven for building my production code

on Sunday

yigang chen added a discussion

Legacy code running in mapper

Hi, I'm trying to move my legacy data procfessing code to hadoop. My issue is the legacy code relies on local file system - it both reads and writes meta data. When the code access local data it typically uses relative path, like this: "meta-dir/g…

on Sunday

dachuan huang replied to dachuan huang's discussion 'how could i setup a hadoop source project?'

I've already used an awkward way to solve this problem, just copy the source folder one by one manually into my eclipse project, and then copy the jars into my eclipse project, this is not a perfect solution because anytime i have made some modifica…

on Sunday

dachuan huang commented on nijil's blog post 'hello...is this site a bit inactive or am i the only one feeling it...please make sm noise'

shout.

on Sunday

dachuan huang replied to Adeel's discussion 'do all mappers finish before reducer starts'

based on my understanding and experiences, the reducers starts before mappers finish. because there are four stages in "reduce": copy phase, append phase, sort phase, reduce phase. exactly speaking, append phase starts after all mappers finishes, bu…

on Sunday

dachuan huang added a discussion

how could i setup a hadoop source project?

hi, i am new to this forum.when i read the book "OReilly.Hadoop.The.Definitive.Guide.pdf", i found many interesting examples, and i want to run them. this might be done by importing "hadoop-0.18.3-core.jar" and some other "libraries" into my classpa…

on Sunday

dachuan huang is now a member of Hadoop Professionals

Welcome Them!

on Sunday

Photos

Loading…

Help With Hadoop

A great place to learn Hadoop, and to tune your map reduce jobs.

Ask specific Hadoop questions here to get help from an expert :)

Forum

Legacy code running in mapper 3 Replies

Started by yigang chen. Last reply by Jason Venner 1 day ago.

hadoop or mysql? 1 Reply

Started by sanjo cole. Last reply by Jason Venner 1 day ago.

how could i setup a hadoop source project? 3 Replies

Started by dachuan huang. Last reply by dachuan huang Mar 8.

Events

Add an Event

Blog Posts

datanode can not connect to the namenode in a small hadoop cluster

Hey there I have a hadoop cluster build on 2 servers (2 laptops). One node (A)

contains the namenode, a datanode, the jobtraker and a tasktraker.

The other node(B) just has a datanode and a tasktraker.

I set up correctly hdfs with ./start-hdfs.sh

When I try to set up MapReduce with ./start-mapred.sh the
TaskTraker of node (B) can not connect to the namenode. The tasktracker log will

keep throwing:

INFO org.apache.hadoop.ipc.Client: Retrying conne… Continue

Posted by Marc Sturlese on February 15, 2010 at 7:00am — 3 Comments

hello...is this site a bit inactive or am i the only one feeling it...please make sm noise

....

Posted by nijil on February 14, 2010 at 6:40am — 1 Comment

seeking advice on word vectors

Hello all,

Hope all is well in the community. I am inquiring on how to apply hadoop to retrieve information from various blogs, news feeds, etc.. in a particular fashion.

I have identified three groups of word pairs that are valuable to me. I would like to explore the clustering patterns among particular URL's of these particular word pairs in their respective blog spaces, news feeds, etc.

So, given that I have an expec

… Continue

Posted by Mark Cejas on February 13, 2010 at 10:41am — 2 Comments

.bashrc file error

Hello all,

I hope that the holidays are going well,
I finally have my graduate school work behind me and have more time to learn about this wonderful Hadoop tool. I work on a Fedora 11 distribution and upon getting my JAVA_HOME and HADOOP_HOME paths set, I started to encouter the following error. The error is is observed upon establishing root user as follows:

[rasaan@rasaan ~]$ su
Password:
bash: /root/.bashrc: line 9: unexpected EOF while looking for matching `)'
bash: /root/.bashrc: line 14… Continue

Posted by Mark Cejas on December 31, 2009 at 12:23pm — 1 Comment