Data Migration: Oracle to Oracle

Replies to This Discussion

Permalink Reply by Jason Venner on June 9, 2009 at 8:01am

My small sample of experience using hadoop map tasks to read from a database or to write to a database resulted in the database falling over almost immediately due to excessive loading. On the flip side the db was mysql running on a single 8way linux box.

There have been a series of patches for people using the database input format, to support various databases.

This mail thread has oracle DBInputFormat references: http://markmail.org/message/eomox6uvleyhf62e
This patch has some fixes for oracle in 19: https://issues.apache.org/jira/browse/HADOOP-5616
The cloudera blog has an article on it also: http://www.cloudera.com/blog/tag/dbinputformat/

From my perspective, once the mechanics of oracle are sorted out as listed above, the key is ensuring that the db doesn't get totally hammered and that contention is managed.

The default nature of input splits in hadoop imply that multiple tasks will be beating on sections of one table or join at the same time, and it they are transforming and writing this opens the door for hidous lock contention

• ▶ Reply

Permalink Reply by Jason Venner on June 11, 2009 at 9:48am

Cloudera just announced an opensource package for bulk loading of data from from databases for use by Map Reduce applications. The application is called sqoop, written primarily by aaron kimbal.
http://www.cloudera.com/hadoop-sqoop

• ▶ Reply

Permalink Reply by Ram on June 11, 2009 at 3:46pm

Yes Jason. I did attend the sumit. I came to know about it. Thanks for the information.

Regards

Ram

Jason Venner said:

Cloudera just announced an opensource package for bulk loading of data from from databases for use by Map Reduce applications. The application is called sqoop, written primarily by aaron kimbal.
http://www.cloudera.com/hadoop-sqoop

• ▶ Reply

▶ Reply to Discussion