"Applications can specify a comma separated list of paths which would be present in the current working directory of the task using the option -files. The -libjars option allows applications to add jars to the classpaths of the maps and reduces. The…"
A Community for Hadoop Users
Jason Venner's Groups
-
NoSql
6 members
-
Karma Sphere Studio Users
4 members
-
HBase Users
11 members
Jason Venner's Discussions
Notes and Presentation from the HUG talk on Nov 18th - integrating solr & katta for scaled reliable search
3 Replies
Started this discussion. Last reply by Jason Venner Dec 3, 2009.
Please post job openings here.
Started Nov 17, 2009
Katta and Solr
12 Replies
Started this discussion. Last reply by Saju K K May 2, 2010.
Gifts Received
Jason Venner has not received any gifts yet
Jason Venner's Page
Latest Activity
Jason Venner commented on Yigang Chen's blog post How to set classpath to 'hadoop jar' command line?
NoSql
A group for discussion various distributed random access datastores that work well with the hadoop ecosystem tools
Jason Venner commented on Bhavesh Shah's blog post Query related Hadoop's Map-reduce"If I understand you, you have to data sets, A & B, and for each record of A, you have to operate on every record of B.
The simplest way would be to use A as the input data set for your map reduce job, and to open and scan through B be in side…"
Jason Venner replied to Oleksiy's discussion Run hadoop app on the cluster from another machine"the simple way is to copy the config files from your master to the hadoop conf dir on the remote machine."
Jason Venner replied to mariaprabudass's discussion Help needed to view the php file which is located in hadoop path?Jason Venner replied to mariaprabudass's discussion Help needed to view the php file which is located in hadoop path?"apache can not see hdfs paths.
either fuse mount your hdfs or copy the file from hdfs to the local file system"
Jason Venner replied to mariaprabudass's discussion hadoop files not viewed in the group HBase Users"it is very likely that the path /user/hadoop/input is an hdfs path, apache can not see hdfs paths, only local file system paths.
you need something to copy the file into the local file system, or to mount hdfs as a local file system"
Jason Venner replied to Abhishek Sagar's discussion Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out."In the 19 line, the each line of the slaves file had the host name of a slave in it
Each slave will use that names from that file to contact it's peers.
If the slaves do not have identical name -> ip mapping for their peer's names, the…"
Jason Venner replied to Garry Boyce's discussion java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text"You have to specify the that the input is in TextInputFormat."
Deepak Kumar joined Jason Venner's groupNoSql
A group for discussion various distributed random access datastores that work well with the hadoop ecosystem tools
Hardick Satiya joined Jason Venner's groupNoSql
A group for discussion various distributed random access datastores that work well with the hadoop ecosystem tools
Profile Information
- Hadoop Experience Level
- Expert
- Interests
- Science Fiction, Spirituality, Aviation, Physics, Biology
- Expertise
- OpenStack, Hadoop, Java, Linux, Performance Tuning, Scaling, Architecture
- Past Projects
- Distributed Cloud Architectures for Commerce, Large Solr Search Indexes, Web scale media crawling,fingerprinting and matching.
- Current Project
- Advising CIO's on the use of Cloud and Big Data techniques
- Available for Consulting
- Yes
- Your Website
- http://www.brokerage.com
- Search Expertise
- Intermediate
- HBase Expertise
- Novice
- Machine Learning Expertise
- Novice
Jason Venner's Blog
I am giving a talk at the HUG on Wed, scaling search with hadoop, katta and solr
Jason Rutherglen will be providing the in depth lucene/solr pieces.
Hope to see you there.
Hope to see you there.
Posted on November 17, 2009 at 12:57pm
Thanks to Stephane for a fun Katta Meetup last night.
There were good discussions on Katta, Solr machine learning and general machine performance
Posted on September 30, 2009 at 7:29am
Cloudera folds Hbase into their 0.20 hadoop distribution
Per Michael Stack,
Our Andrew Purtell working with Chad Metcalf over at Cloudera have added HBase to the CDH2 Cloudera distribution. Andrew has a guest blog over on Cloudera here: http://su.pr/27zIMw St.Ack
Enjoy!
Our Andrew Purtell working with Chad Metcalf over at Cloudera have added HBase to the CDH2 Cloudera distribution. Andrew has a guest blog over on Cloudera here: http://su.pr/27zIMw St.Ack
Enjoy!
Posted on September 29, 2009 at 8:22am
Scripts that are missing from the source code bundle
I somehow missed including the Perl scripts for the aggregate streaming in chapter 8 and various shell scripts from earlier chapters.
I have attached them in scripts.zip
I have attached them in scripts.zip
Posted on July 24, 2009 at 6:48am
Slow responding this week
I am overly booked and not getting back to people
Posted on July 21, 2009 at 10:58pm
Comment Wall (20 comments)
- At 10:16am on January 26, 2010, G Sondeep said…
- Thankyou So Much Jason
I really appreciate your response
- At 10:09am on January 26, 2010, Jason Venner said…
- Not looking Sondeep
- At 10:08am on January 26, 2010, G Sondeep said…
- Please accept my apologies in case you feel that you have received this message in error
Thankyou
- At 10:06am on January 26, 2010, G Sondeep said…
- Hi Jason
GoodDay!
I am Sandeep from a staffing company would like to speak to you regarding a Job opportunity with my direct client
Sandeep
510-493-2104X625
- At 12:29pm on December 13, 2009,
sheeraz mughal said… - Hi,
Thank you very much for your reply and its was really informative. I have been given a responsiblity to head a research group in one of the leading Universities in Pakistan to create and head a research group on behalf of universities funding. I am very much interested in Hadoop and related products so i was thinking to do research work in Hadoop and related technologies like Hive map reduce etc.
so in the regard i request if you could give me some solid ideas or directions towards any problem areas in hadoop or releated tech so that i can jump into it and start the research. I thank you very much and it would be an honour for me if you leave something for me as early as possible.
- At 10:50pm on December 7, 2009,
Wade Xiao said… - thank you~ I'm a student and just doing some research on Hadoop. Currently I'm interested in the storage of MapReduce applications, including HDFS and HBase.
- At 10:03am on November 13, 2009, sheeraz mughal said…
- Hi,
Can we encrypt the data file in HDFS with any triple DES compatible algorithm or any else and then decrypt the input while map methods before passing it onwards for any business logic and then further to reducer??? I am like beginner in Hadoop technology so if my question sounds stupid ;) then please do comment and correct me and guide me about any Security model hadoop is having???
Thanks
sheeraz
- At 9:54pm on November 8, 2009, sheeraz mughal said…
- hi,
To all Hadoop professionals and others i am working in a organization having world's largest Biometric Database and by keeping this fact in mind kindly let me know what can be built on top of it using hadoop where hadoop's core significance could shine as compare to other technologies. I request you all to please suggest any idea as i have to submit my research proposal in few days. Thanks a lot
sheeraz
- At 11:30pm on October 5, 2009,
Stefan Groschupf said… - Thanks! ... sure I will. :)
- At 8:33pm on September 15, 2009,
wang zhengkui said… - After reading your book, I generate some questions which I want to know. Firstly, if I want to let the reducers to fetch more partitions files from map out put, is that ok? For instance, now reducer one can fetch all the partition 1 from mappers, how I implement that reducer one can fetch all the partition 1 and also 2 to go to reducer 1? If can , How could I implement that?
Secondly, in the map phase, one recorder can only be written into one partition file according to the partitioner function. If I want to write one recorder to multi-partition files, how can I do that? For example, there are M reducers and there should be M partition files in map phase. Now one recorder can only be output to one of M partition files. If I want to output one recorder to multi-partition files, is there any way to do this?
Groups
-
NoSql
6 members
-
Karma Sphere Studio Users
4 members
-
HBase Users
11 members
-
Hadoop Cascading
1 member
-
Hadoop Administrators
9 members
© 2012 Created by Jason Venner.
