Posts tagged: Application Engine

Session: Data Processing Model Besides Map/Reduce

Big Data Workshop, April 23, 2010
Session 5A

Title: Data Processing Model Besides M/R
Convener: Stanley Poon
Notes-taker: Stanley

  • M/R is relatively new
  • Limited in way to partition problem
  • Latency high
  • Not enough parallelism: Map ha to finish before reduce
  • IBM – tool to reduce many existing algorithms to map reduce. Project name unknown at time.
  • CloudComp 2009 has some papers on comparing map reduce with MPI
  • Graph processing model as a more general, Pregel from Google.
  • An example using M/R to process video streams: Streams are processed by mappers and fed to reducers. Each reducers will put the frames into sequence. Frame boundary is a natural demarcation to break down stream.

WordPress Themes