yigang chen's Discussions

Legacy code running in mapper
3 Replies

Tags: cache, distributed

Started this discussion. Last reply by Jason Venner Mar 10, 2010.

View All

yigang chen's Page

Gifts Received

yigang chen has not received any gifts yet

Give yigang chen a Gift

Latest Activity

Jason Venner replied to yigang chen's discussion 'Legacy code running in mapper'

You can build your own symbolic link by running a command from java, you just need to verify where the data is unpacked, and then build a link to it. A quick search turned up the following page for sample java code for you:…

Mar 10, 2010

yigang chen replied to yigang chen's discussion 'Legacy code running in mapper'

So that means I'd need to modify the legacy code, i.e., change the hard coded: "a/relative/path/to/my/file.xml" to: "./mymeta.zip/a/relative/path/to/my/file.xml" Is there a way at all to NOT change the legacy code?

Mar 8, 2010

Jason Venner replied to yigang chen's discussion 'Legacy code running in mapper'

If you pass -archives mymeta.zip there will be a symbolic link in the current working directory for the map or reduce task mymeta.zip, which points to the directory that the archive was unpacked in. so if you use…

Mar 7, 2010

Legacy code running in mapper

Hi, I'm trying to move my legacy data procfessing code to hadoop. My issue is the legacy code relies on local file system - it both reads and writes meta data. When the code access local data it typically uses relative path, like this: "meta-dir/group/my-meta.xml". From the O'Reilly book (Tom White), I'm thinking of using distributed cache to copy the local files to the task nodes. For example I could zip the entire meta data dirrectory tree and use -archives mymeta.zip My question is: How do I…See More

Discussion posted by yigang chen Mar 7, 2010

3 Comments

yigang chen is now a member of Hadoop Professionals Mar 7, 2010

Welcome Them!