Distributed File System (DFS)

  • MapReduce
  • GFS
  • Collossus
  • Google BigTable (LSM over DFS)

API

  • Create(p)
  • Delete(p)
  • Copy(s, d)
  • PRead(p, offset, size)
  • Write(p, offset, data)
  • Append(p, data)

Hardware

Block Device
[   |   |   |B_0|   |B_1]
     4kb

Design

File [   |   |   ]       | Data      ->  ChunkStore
      B_0 B_1 B_2        |
                         |
Tree:   { INode }        |
path -> <B_0, B_1, ...>  | MetaData  ->  MetaStore

Read

  1. MS.ListChunks(p, o, s) -> [c_i]
  2. CS.Get([c_i])

Append

  1. CS.Put(d) -> id
  2. MS.Append(p, id)

ChunkStore

  • Get(..)
  • Put(..)
  • Unordered - G-Set
  • x3 replication -> x1.5 replication (correction codes) -> x1.33 (LRC)

MetaStore

  • Ordered - RSM

See Also