In this session we’ll first discuss our experience extending Hadoop development to new platforms & languages and then discuss our experiments and experiences building supporting developer tools and plugins for those platforms. First, we’ll take a hands on approach to showing our experiments and successes extending Hadoop to languages such as JavaScript and .NET with LINQ. Second, we’ll walk through some of the developer & developer ops tools and plugins we’ve experimented with in an effort to simplify life for the Hadoop developer across both on premises and cloud-based projects.

1. BUILDING TOOLS FOR THE
HADOOP DEVELOPER
matt winkler
@mwinkle
Operated
By: mike
flasko
@mflasko

3. C#, F# Map/Reduce, LINQ to Hive, .NET management clients
Node.js management SDK
Hive, Pig, Mahout, Cascading, Scalding, Scoobi, Pegasus…
PowerShell, Cross Platform CLI tools

4. Existing Ecosystem
 Actively contributing to:
 Core
 Pig
 Hive
 HCatalog
 Branching to other projects
 Streamlined, Simple Deploy
 Simple one-box developer install on Windows
 Simple scale up/out to the cloud

5. .NET
 Map/Reduce
 LINQ to Hive
 Client API’s
 WebHCat
 Ambari
 WebHDFS
 Azure / Cloud
 Visual Studio Tooling
 Local debugging support

6. JavaScript
 Node.js client API’s
 WebHCat
 WebHDFS
 Ambari
 Azure / Cloud

7. Management
 UI Tooling
 Cluster usage
 Job authoring
 Result consumption in common tools
 PowerShell & Cross platform scripting
 API Surface
 RDFE – Azure provisioning
 Ambari – Cluster monitoring
 WebHCatalog – Metadata and job submission
 WebHDFS, Blob Storage – Storage
>_

8.  Sources
 http://hadoopsdk.codeplex.com
 http://www.github.com/windowsazure
 NuGet packages
 Microsoft.Hadoop.MapReduce
 Microsoft.Hadoop.Hive
 Microsoft.Hadoop.WebClient
 NPM packages
 Azure
 Azure-cli
open