Big Data Workshop, April 23, 2010
Title: Data Processing Model Besides M/R
Convener: Stanley Poon
- M/R is relatively new
- Limited in way to partition problem
- Latency high
- Not enough parallelism: Map ha to finish before reduce
- IBM – tool to reduce many existing algorithms to map reduce. Project name unknown at time.
- CloudComp 2009 has some papers on comparing map reduce with MPI
- Graph processing model as a more general, Pregel from Google.
- An example using M/R to process video streams: Streams are processed by mappers and fed to reducers. Each reducers will put the frames into sequence. Frame boundary is a natural demarcation to break down stream.