As big data technology matures, you’d think there would be more talent available to hire. Although the number of people interested and engaged in the big data world has dramatically increased, job demand is far ahead. Companies like Google or Facebook have access to the best talent — thousands of engineers with PhDs from the best schools, which is why they are able to innovate. How can a company close the skills gap while innovating and creating product advantage? This talk highlights how the right technology can allow you to compete without having an army of PhDs at your disposal. At iPass, we’ve created an environment where our engineers can be empowered to create value without getting bogged down by big data and Ops challenges. As a result, we have been able to more easily recruit internal engineers to our big data team, leveraging their current expertise, while bringing them up to speed on big data projects much faster. Join this talk to learn how you can do the same for your organization.

1. Tomasz Magdanski, iPass
HOW APACHE SPARK CHANGED
THE WAY WE HIRE PEOPLE
#EntSAIS17

2. What if the war for talent ended and
your company lost?
2#EntSAIS17
• War for talent
– Late ’90s warning from McKinsey about talent
shortage
– Urged companies to prioritize strategies around
recruiting, retaining and developing key employees
• One percent problem

3. Hiring is tough
3#EntSAIS17
Source: edureka

4. And its going to get worse
4#EntSAIS17
Source: Hour of Code

5. Since war for talent started we
have made a full circle
5#EntSAIS17
• Apart from hiring skilled engineers companies look inside
to fill in the gap
• Create path to grow within your organization
• But wait a minute ?
Didn't we just say there is a big skills gap ?

6. What are we building ?
6#EntSAIS17

7. Goals
• Scalable platform
• Cost effective
• No data loss
• Code portability
• Easy R&D
• Extendable
• Support many languages
• Support batch, stream
• ML enabled
• Collaborative
7#EntSAIS17

8. Who we were looking for ?
• MapReduce
• Hadoop / HDFS
• Hive / Pig
• Storm
• Caching
• Avro / Parquet
• Distributed Computing
• Manage Clusters and Infrastructure
• Integrate tools
• Data Warehousing and Modeling
8#EntSAIS17

9. Who we were looking for ?
• CAP Theorem
• Data Transformation
• Data Collection
• SQL
• Cassandra / Hbase / MongoDB / mysql
• Kafka
• AWS
• Scala / Java / Python
• Understanding data structures and algorithms
• Visualization and Data Analysis
• Team player
• SPECIFIC INDUSTRY KNOWLEDGE
9#EntSAIS17

10. Spark changed what we are
looking for
10#EntSAIS17

11. Spark
• Simple
• Easy to learn
• High abstraction API
• Build in connectors to major data sources
• Supports Batch, Stream
• Highly optimized and extendible
• ML library to run at scale
• Spark provides transactional writes and exact once semantic
11#EntSAIS17

12. Spark and Databricks
• A single platform that unifies data engineering and data science
• Automated cluster management / zero-management infrastructure
• Intuitive notebooks supporting multiple programming languages
• Makes collaboration easy
• Blends Data Engineering and Data Science workloads
• APIs to integrate with other tools
12#EntSAIS17

13. Spark and Databricks
• Integrated meta store
• Integrated Managed and unmanaged tables
• Workspace API
• Engineering and Customer Support including Solution Architects
• Easy dashboards
• DbUtils
• SBT tools for easy deployment
13#EntSAIS17

14. Who we are looking for now?
• Experience in programming using APIs
• Understanding data structures and algorithms
• Scala / Java / Python / R
• Visualization and Data Analysis
• Team player
• SPECIFIC INDUSTRY KNOWLEDGE
14#EntSAIS17

15. Business needs
• Wi-Fi connectivity patterns
• Our business and customers
• Existing system architecture
• Skip lengthy onboarding process
• One stack to learn
15#EntSAIS17

16. How did that change the way we hire ?
• We have internally hired:
– QA engineer
– App developers
– Ex developer / product manager
– Backend engineer
• Externally hired:
– One very experienced senior Data Engineer
– Few collage grads – Junior data engineers
16#EntSAIS17

17. Summary
• Thanks to Databricks we didn’t have to build a
platform
• Hired mixed of internal and external candidates
• Focus on business needs
• Created 6 data products
• New seven digit revenue stream for our company
• Continue to innovate
17#EntSAIS17