Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
a library to simplify and unify the lower layers
of big data systems on modern resource managers.
• One cluster used by all workloads
(interactive, batch, streaming, …)
• Resources are handed out as
containers
Container ...
Resource
Managers
Enable true multi-tenancy…
Many workloads:
Streaming, Batch, Interactive…
Many users:
Production, Ad-Hoc...
Resource
Managers
Enable true multi-tenancy…
Many workloads:
Streaming, Batch, Interactive…
Many users:
Production, Ad-Hoc...
Resource
Managers
Enable true multi-tenancy…
Many workloads:
Streaming, Batch, Interactive…
Many users:
Production, Ad-Hoc...
Resource
Managers
Enable true multi-tenancy…
Many workloads:
Streaming, Batch, Interactive…
Many users:
Production, Ad-Hoc...
Resource
Managers
Enable true multi-tenancy…
Many workloads:
Streaming, Batch, Interactive…
Many users:
Production, Ad-Hoc...
Example 1:
SQL / MapReduce
Elasticity
Fault tolerance
σ
π
⋈
Example 1:
SQL / MapReduce
Elasticity
Fault tolerance
σ
π
⋈ ⋈⋈ ⋈⋈
Example 1:
SQL / MapReduce
Elasticity
Fault tolerance
π
⋈ ⋈⋈ ⋈⋈
Example 2:
Machine learning
Fault tolerance
Elasticity
Iterative computations
Example 3:
Graph processing
Fault tolerance
Elasticity
Iterative computations
Low latency communication
Silos
Silos are hard to build
Each duplicates the same
mechanisms under the hood
In practice, silos form pipelines
• In ea...
Fun new problems,
boring old ones
Control flow setup
Master / Slave
Membership
Via heartbeats
Task Submission
Inter-Task M...
REEF
Breadth
Mechanism over Policy
Avoid silos
Recognize the need
for different models
But: allow them to be composed
RM p...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
REEF Control Flow
Yarn ( ) handles resource
management (security, quotas,
priorities)
Per-job Drivers ( ) request
resource...
Hello World
1. Client submits the Driver
allocation request to the
Resource Manager ( )
$…
Hello World
1. Client submits the Driver
allocation request to the
Resource Manager ( )
2. Once started, Driver ( )
reques...
Hello World
1. Client submits the Driver
allocation request to the
Resource Manager ( )
2. Once started, Driver ( )
reques...
Hello World
1. Client submits the Driver
allocation request to the
Resource Manager ( )
2. Once started, Driver ( )
reques...
Hello World
1. Client submits the Driver
allocation request to the
Resource Manager ( )
2. Once started, Driver ( )
reques...
public final class HelloTask implements Task {
@Inject
private HelloTask() {}
@Override
public byte[] call(final byte[] me...
public final class StartHandler implements EventHandler<StartTime> {
@Override
public void onNext(final StartTime startTim...
final Configuration runtimeConfig = YarnRuntimeConfiguration.CONF.build();
final Configuration driverConfig = DriverConfig...
Tang
Command = ‘ls’
Error:
container-487236457-02.stderr:
NullPointerException at:
java…eval():1234
ShellTask.helper():546...
Tang
Command = ‘ls’
Error:
Unknown parameter "Command"
Missing required parameter "cmd"
Configuration is hard
- Errors oft...
Tang
cmd = ‘ls’
ShellTask
Evaluator
Error:
Required instanceof Evaluator
Got ShellTask
Configuration is hard
- Errors ofte...
Tang
cmd = ‘ls’
ShellTask
Task
YarnEvaluator
Evaluator
Configuration is hard
- Errors often show up at runtime only
- Stat...
Wake:
Events + I/O
Event-based
programming and remoting
API: A static subset of Rx
- Static checking of event flows
- Aggr...
Distributed Shell
1. Client submits Driver configuration
to YARN runtime. Configuration
has cmd shell command, and n
numbe...
Distributed Shell
1. Client submits Driver configuration
to YARN runtime. Configuration
has cmd shell command, and n
numbe...
Distributed Shell
1. Client submits Driver configuration
to YARN runtime. Configuration
has cmd shell command, and n
numbe...
Distributed Shell
1. Client submits Driver configuration
to YARN runtime. Configuration
has cmd shell command, and n
numbe...
Distributed Shell
1. Client submits Driver configuration
to YARN runtime. Configuration
has cmd shell command, and n
numbe...
@Inject
private ShellTask(@Parameter(Command.class) final String command) {
this.command = command;
}
@Override
public byt...
@NamedParameter(doc="Number of evaluators", short_name="n", default_value="1")
public final class NumEvaluators implements...
public final class EvaluatorAllocatedHandler
implements EventHandler<AllocatedEvaluator> {
@Override
public void onNext(fi...
final JavaConfigurationBuilder driverConfig =
tang.newConfigurationBuilder(DriverConfiguration.CONF
.set(DriverConfigurati...
YARN Example: 1333 lines of code
YARN Example: 1333 lines of code
Simple REEF Application: 83 lines
94%
YARN Example: 1333 lines of code
Simple REEF Application: 83 lines
Interactive Fault-Tolerant
Web Application on REEF: 361...
•
•
•
•
•
Group
Communication
Also: Collective Communication
Communications with many
participants
Contrast: Peer-to-peer communicat...
Broadcast
Mechanism
The sender (S) sends a value
All receivers (R) receive the identical value
Use case
Distribute a model...
Reduce
Mechanism
The senders (S) each send a value
Those values are aggregated
The receiver (R) receives the aggregate tot...
// We can send and receive any Java serializable data, e.g. jBLAS matrices
private final Broadcast.Sender<DoubleMatrix> mo...
http://reef.apache.org
http://github.com/cmssnu/dolphin
http://github.com/Microsoft-CISL/TensorFactorization-LDA
Contact dev@reef.apache.org
Website http://reef.apache.org
Start with a random 𝑤0
Until convergence:
Compute the gradient
𝜕 𝑤 = ෍
𝑥,𝑦 𝜖 𝑋
𝑤, 𝑥 − 𝑦
Apply gradient and regularizer to ...
On REEF
Driver requests Evaluators
On REEF
Driver requests Evaluators
Driver sends Tasks to load & parse
data
On REEF
Driver requests Evaluators
Driver sends Tasks to load & parse
data
Driver sends ComputeGradient and
master Tasks
On REEF
Driver requests Evaluators
Driver sends Tasks to load & parse
data
Driver sends ComputeGradient and
master Tasks
C...

Start with a random 𝑤0
Until convergence:
Compute the gradient
𝜕 𝑤 = ෍
𝑥,𝑦 𝜖 𝑋
2 𝑤, 𝑥 − 𝑦
Apply gradient to the model
𝑤𝑡+1...
On REEF
First iteration
On REEF
Second Iteration
On REEF
End state
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Apache REEF - stdlib for big data
Próxima SlideShare
Cargando en…5
×

Apache REEF - stdlib for big data

19.397 visualizaciones

Publicado el

Slides from my talk at 2016 Apache: Big Data conference.

Resource managers like Apache YARN and Mesos have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very low-level. We present Apache REEF, a powerful yet simple framework that helps developers of big data systems to retain fine-grained control over the cloud resources and address common problems of fault-tolerance, task scheduling and coordination, caching, interprocess communication, and bulk-data transfers. We will guide the developers through a simple REEF application and discuss current state of Apache REEF project and its place in the Hadoop ecosystem.

Publicado en: Software
  • Sé el primero en comentar

Apache REEF - stdlib for big data

  1. 1. a library to simplify and unify the lower layers of big data systems on modern resource managers.
  2. 2. • One cluster used by all workloads (interactive, batch, streaming, …) • Resources are handed out as containers Container is slice of a machine Fixed RAM, CPU, I/O, … • Examples: Apache Hadoop YARN Apache Mesos Google Borg Resource Managers
  3. 3. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps
  4. 4. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance
  5. 5. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance
  6. 6. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance Preemption
  7. 7. Resource Managers Enable true multi-tenancy… Many workloads: Streaming, Batch, Interactive… Many users: Production, Ad-Hoc, Experiments… …but, only for sophisticated apps Fault tolerance Preemption Elasticity
  8. 8. Example 1: SQL / MapReduce Elasticity Fault tolerance σ π ⋈
  9. 9. Example 1: SQL / MapReduce Elasticity Fault tolerance σ π ⋈ ⋈⋈ ⋈⋈
  10. 10. Example 1: SQL / MapReduce Elasticity Fault tolerance π ⋈ ⋈⋈ ⋈⋈
  11. 11. Example 2: Machine learning Fault tolerance Elasticity Iterative computations
  12. 12. Example 3: Graph processing Fault tolerance Elasticity Iterative computations Low latency communication
  13. 13. Silos Silos are hard to build Each duplicates the same mechanisms under the hood In practice, silos form pipelines • In each step: Read from and write to HDFS • Synchronize on complete data between steps  Slow! σ π ⋈
  14. 14. Fun new problems, boring old ones Control flow setup Master / Slave Membership Via heartbeats Task Submission Inter-Task Messaging … σ π ⋈
  15. 15. REEF Breadth Mechanism over Policy Avoid silos Recognize the need for different models But: allow them to be composed RM portability Make app independent from low- level Resource Manager APIs. Bridge the JVM/CLR/… divide Different parts of the computation can be in either of them. Resource Manager and DFS := Cluster OS REEF := stdlib
  16. 16. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… + 3
  17. 17. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc. + 3
  18. 18. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ + 3
  19. 19. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  20. 20. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  21. 21. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  22. 22. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ σ σ σ
  23. 23. REEF Control Flow Yarn ( ) handles resource management (security, quotas, priorities) Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc… REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks ( , , , , , , etc…) to use the same cached state. πσ
  24. 24. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) $…
  25. 25. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN $…
  26. 26. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN 3. Driver submits HelloWorld Task to the newly allocated Evaluator $…
  27. 27. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN 3. Driver submits a HelloWorld Task to the newly allocated Evaluator 4. HelloWorld Task prints a greeting to the log and quits $… LOG.log(Level.INFO, "Hello REEF!");
  28. 28. Hello World 1. Client submits the Driver allocation request to the Resource Manager ( ) 2. Once started, Driver ( ) requests one Evaluator container from YARN 3. Driver submits a HelloWorld Task to the newly allocated Evaluator 4. HelloWorld Task prints a greeting to the log and quits 5. Driver receives CompletedTask notification, releases the Evaluator and quits $…
  29. 29. public final class HelloTask implements Task { @Inject private HelloTask() {} @Override public byte[] call(final byte[] memento) { LOG.log(Level.INFO, "Hello REEF!"); return null; } }
  30. 30. public final class StartHandler implements EventHandler<StartTime> { @Override public void onNext(final StartTime startTime) { requestor.submit(EvaluatorRequest.newBuilder() .setNumber(1).setMemory(64).setNumberOfCores(1).build()); } } public final class EvaluatorAllocatedHandler implements EventHandler<AllocatedEvaluator> { @Override public void onNext(final AllocatedEvaluator allocatedEvaluator) { allocatedEvaluator.submitTask(TaskConfiguration.CONF .set(TaskConfiguration.IDENTIFIER, "HelloREEFTask") .set(TaskConfiguration.TASK, HelloTask.class) .build(); } }
  31. 31. final Configuration runtimeConfig = YarnRuntimeConfiguration.CONF.build(); final Configuration driverConfig = DriverConfiguration.CONF .set(DriverConfiguration.DRIVER_IDENTIFIER, "HelloREEF") .set(DriverConfiguration.GLOBAL_LIBRARIES, EnvironmentUtils.getClassLocation(HelloDriver.class)) .set(DriverConfiguration.ON_DRIVER_STARTED, HelloDriver.StartHandler.class) .set(DriverConfiguration.ON_EVALUATOR_ALLOCATED, HelloDriver.EvaluatorAllocatedHandler.class) .build(); DriverLauncher .getLauncher(runtimeConfig) .run(driverConfig, JOB_TIMEOUT);
  32. 32. Tang Command = ‘ls’ Error: container-487236457-02.stderr: NullPointerException at: java…eval():1234 ShellTask.helper():546 ShellTask.onNext():789 YarnEvaluator.onNext():12 Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process
  33. 33. Tang Command = ‘ls’ Error: Unknown parameter "Command" Missing required parameter "cmd" Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process Our approach: - Use Dependency Injection - Configuration is pure data  Early static and dynamic checks
  34. 34. Tang cmd = ‘ls’ ShellTask Evaluator Error: Required instanceof Evaluator Got ShellTask Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process Our approach: - Use Dependency Injection - Configuration is pure data  Early static and dynamic checks
  35. 35. Tang cmd = ‘ls’ ShellTask Task YarnEvaluator Evaluator Configuration is hard - Errors often show up at runtime only - State of receiving process is unknown to the configuring process Our approach: - Use Dependency Injection - Configuration is pure data  Early static and dynamic checks
  36. 36. Wake: Events + I/O Event-based programming and remoting API: A static subset of Rx - Static checking of event flows - Aggressive JVM event inlining Implementation: “SEDA++” - Global thread pool - Thread sharing where possible
  37. 37. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on $…
  38. 38. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN $…
  39. 39. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN 3. Driver submits ShellTask with cmd to each Evaluator #! $… #! #!
  40. 40. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN 3. Driver submits ShellTask with cmd to each Evaluator 4. Each ShellTask runs the command, logs its stdout, and quits #! $… #! #!
  41. 41. Distributed Shell 1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and n number of Evaluators to run it on 2. Once started, Driver requests n Evaluators from YARN 3. Driver submits ShellTask with cmd to each Evaluator 4. Each ShellTask runs the command, logs its stdout, and quits 5. Driver receives CompletedTask notifications, releases Evaluators and quits when all Evaluators are gone $…
  42. 42. @Inject private ShellTask(@Parameter(Command.class) final String command) { this.command = command; } @Override public byte[] call(final byte[] memento) { final String result = CommandUtils.runCommand(this.command); LOG.log(Level.INFO, result); return CODEC.encode(result); }
  43. 43. @NamedParameter(doc="Number of evaluators", short_name="n", default_value="1") public final class NumEvaluators implements Name<Integer> {} @NamedParameter(doc="The shell command", short_name="cmd") public final class Command implements Name<String> {}
  44. 44. public final class EvaluatorAllocatedHandler implements EventHandler<AllocatedEvaluator> { @Override public void onNext(final AllocatedEvaluator allocatedEvaluator) { final JavaConfigurationBuilder taskConfigBuilder = tang.newConfigurationBuilder(TaskConfiguration.CONF .set(TaskConfiguration.IDENTIFIER, "ShellTask") .set(TaskConfiguration.TASK, ShellTask.class) .build()); taskConfigBuilder.bindNamedParameter(Command.class, command); allocatedEvaluator.submitTask(taskConfigBuilder.build()); } }
  45. 45. final JavaConfigurationBuilder driverConfig = tang.newConfigurationBuilder(DriverConfiguration.CONF .set(DriverConfiguration.DRIVER_IDENTIFIER, "DistributedShell") .set(DriverConfiguration.GLOBAL_LIBRARIES, EnvironmentUtils.getClassLocation(ShellDriver.class)) .set(DriverConfiguration.ON_DRIVER_STARTED, ShellDriver.StartHandler.class) .set(DriverConfiguration.ON_EVALUATOR_ALLOCATED, ShellDriver.EvaluatorAllocatedHandler.class) .build()); new CommandLine(driverConfigBuilder) .registerShortNameOfClass(Command.class) .registerShortNameOfClass(NumEvaluators.class) .processCommandLine(args); DriverLauncher.getLauncher(runtimeConfig).run(driverConfig.build(), JOB_TIMEOUT);
  46. 46. YARN Example: 1333 lines of code
  47. 47. YARN Example: 1333 lines of code Simple REEF Application: 83 lines 94%
  48. 48. YARN Example: 1333 lines of code Simple REEF Application: 83 lines Interactive Fault-Tolerant Web Application on REEF: 361 lines 27%
  49. 49. • • • • •
  50. 50. Group Communication Also: Collective Communication Communications with many participants Contrast: Peer-to-peer communication Most commonly used interface: MPI
  51. 51. Broadcast Mechanism The sender (S) sends a value All receivers (R) receive the identical value Use case Distribute a model Distribute a descent direction for line search Optimizations Trees to distribute the data Be mindful of the topology of machines Do peer-to-peer sends … S
  52. 52. Reduce Mechanism The senders (S) each send a value Those values are aggregated The receiver (R) receives the aggregate total Use case Aggregate gradients, losses Optimizations Aggregation trees Pipelining
  53. 53. // We can send and receive any Java serializable data, e.g. jBLAS matrices private final Broadcast.Sender<DoubleMatrix> modelSender; private final Broadcast.Receiver<DoubleMatrix[]> resultReceiver; // Broadcast the model, collect the results, repeat. do { this.modelSender.send(modelSlice); // ... final DoubleMatrix[] result = this.resultReceiver.reduce(); } while (notConverged(modelSlice, prevModelSlice));
  54. 54. http://reef.apache.org
  55. 55. http://github.com/cmssnu/dolphin http://github.com/Microsoft-CISL/TensorFactorization-LDA
  56. 56. Contact dev@reef.apache.org Website http://reef.apache.org
  57. 57. Start with a random 𝑤0 Until convergence: Compute the gradient 𝜕 𝑤 = ෍ 𝑥,𝑦 𝜖 𝑋 𝑤, 𝑥 − 𝑦 Apply gradient and regularizer to the model 𝑤𝑡+1 = 𝑤𝑡 − 𝜂 𝜕 𝑤 + 𝜆 Data parallel in X  Reduce Needed by Partitions  Broadcast
  58. 58. On REEF Driver requests Evaluators
  59. 59. On REEF Driver requests Evaluators Driver sends Tasks to load & parse data
  60. 60. On REEF Driver requests Evaluators Driver sends Tasks to load & parse data Driver sends ComputeGradient and master Tasks
  61. 61. On REEF Driver requests Evaluators Driver sends Tasks to load & parse data Driver sends ComputeGradient and master Tasks Computation commences in sequence of Broadcast and Reduce
  62. 62.
  63. 63. Start with a random 𝑤0 Until convergence: Compute the gradient 𝜕 𝑤 = ෍ 𝑥,𝑦 𝜖 𝑋 2 𝑤, 𝑥 − 𝑦 Apply gradient to the model 𝑤𝑡+1 = 𝑤𝑡 − 𝜕 𝑤 Not having some machines means training on a (random) subset of X
  64. 64. On REEF First iteration
  65. 65. On REEF Second Iteration
  66. 66. On REEF End state

×