Connected Vehicle Data Platform

8. Ford Copyright 2016 © Connected Vehicle Data Platform How much data can a Vehicle Generate? A single Vehicle can generate 25GB of Controller Area Network (CAN) Traffic in an hour. What does the data look like ? TkEiASwYVQxwFSRRM0ZBNl AwU1UyRFIzMzIzNTkBaCILA AF+ItYvADkl0AoYVQxvAggiF wAQAAQEAAAAAAjgKyQAB AAAKIAB/4H/gF4ABiXcAAY mFQA/AjgCcBACCwgPBb0BI IFRQb/AACAgpQQAgsINjB+I tYvADkl0 ……

11. Connected Vehicle Data Platform Ford Copyright 2016 © Receive Azure Event Hub SFTP RESTful WS Internal DataBase SqoopStorm Spout & Bolts Oozie Workflow Java application Yes Java Web Service ORC Hive TablesData Access Data Loading Transform Scheduling Landing Ingestion HDFS No HBase Java MapReduce PiG De-dup Logic Data Classification Hive Partitioning Validation Security Ranger

12. Ford Copyright 2016 © Connected Vehicle Data Platform Opportunities Data Privacy Integration with Partners and Suppliers Security Standards & Frameworks Country Specific Regulations Storage & Updates

13. Ford Copyright 2016 © Connected Vehicle Data Platform Production: • 261 Data Nodes • 5220 CPU Cores • 65 TB RAM • 5 PB Usable Storage • 10 Racks / 130 kW Power Hadoop Capacities Note: Usable storage assumes 3x replication in cluster.

Notas del editor

Hello everybody, I’m Tom Bryans, Deliver Manager for Ford Motor Company’s Connected Vehicle Data Platform. Joining me is Dan Totten from our Enterprise Architecture Group. We also have Joe Niemiec from who is our resident architect from HortonWorks. What I would like to provide you is, an overview of our Ford Connected Vehicle Journey. After I get through my presentation, we should have time for a Q&A at the end. who owns a Ford Mustang ? What is your name ? <Simon.. Hey <Simon>, thanks for driving a Ford. I have a mustang as well, and it is awesome!!!!!
Fordhas been undergoing a transformation We are going from a Car Company where Revenue is generated from Vehicles, Parts and Accessories. To a Mobility Company What is a mobility company ? A mobility company helps to change the way the world moves. Mobility is about human progress. getting Food to storesAmbulances to emergencies It is about Getting to work and getting home. There are 4 megatrends that Ford is looking at 1 -Urbanization – growing population in urban environments Here, the Infrastructure simply can’t keep up with number of vehicles 2 - Global middle class growth – it is expected to double from 2B-4B by 2030 – and ione of the middle class dreams is to … own a car 3 - Air Quality – we are very concerned about urban air pollution; 4 - Changing consumer attitudes – things like car sharing and car request services. Ford has a Blueprint for Mobility – that blueprint includes: Vehicles that talk to one another as well as the infrastructure People that want to share vehicles Mobility vs Gridlock And every one is aware of Autonomous Vehicles This blueprint may mean less vehicles but more usage At the center of that transformation is the connected vehicle.
So, What is a Connected Vehicle ? <Click> A connected Vehicle has the means to Transmit Data <Click> As well as the means to Receive Data By receiving Data, Think of things like Over the Air Software Updates or A feature such as a remote start request. <Click>
So, How do we collect data from our connected vehicles. There are three basic ways <Click> The first is Data Over Voice – This is where we use the consumers cell phone to transmit a very small payload over a cellular voice connection – think a 2400 baud modem. <Click> Next, we have Modems Embedded in the vehicle that use 4G cellular data to transmit a more robust payload. <Click> And formally, we have Plug In Devices that plug into the vehicles OBD II port and provide a very rich and frequent payload. <Click>
Where does data go from the Vehicle ? As the Data leaves the Vehicle it goes to various Cloud providers. <Click> The Data over Voice packages goes to one of our partners - Airbiquity <Click> The Embedded Modem data goes to partners Accenture, Telogis or the Ford Cloud <Click> The Plug In data goes to the Delphi, Azure or Ford Clouds <Click>
Where does the data go from the cloud? <Click> From the Cloud, the data securely travels via various protocols Including: HTTP via rest based web services. Secure FTP And streaming using AMQP A big challenge is that this data is all in different formats? <Click> The data is then stored in our Ford Enterprise Hadoop Data Platform <Click> Where we Land and Transform it <Click> We do that utilizing various Hadoop components including Storm, MapReduce, Oozie, PIG and Java. <Click> The Data is then stored <Click> in HIVE and HBASE<Click>
What data is collected from our vehicles ? There are more lines of code in our vehicles than there are in airplanes. Data that we Collect from the Vehicle includes: <Click> CAN Signals and Messages These are things like Odometer, Engine RPM, Location and Temperatures <Click> Diagnostic Trouble Codes or DTC’s which include thing like: Cylinder Misfires and Emissions Faults. <Click> And Warning Indicator Lamp Status which include things like: Service Engine Soon, Low Fuel, Low Tire Pressure. <Click>
How much Data can a Vehicle Generate ? <Click> A single vehicle can generate 25GB of CAN traffic in an hour. Of course, not all this data is transferred back, most of the traffic is computer to computer communications. <Click> What Does the Data that we receive look like ? <Click> What we receive is an encrypted; base64 encoded blob; which then has to be decrypted; decoded; transformed; and stored. <Click>
This is a Use Case of the Remote Start Feature that is available in our Sync Connect equipped vehicles – You have your Car, the Ford Cloud and our Hadoop Data Platform <Click> Using your Ford Pass Mobile App, <Click> you send a remote start request which travels to the Ford Cloud. <Click> The Ford Cloud will send that request to the vehicle as wall as to the data platform where we record that a Remote Start Request was Received from the Mobile device and that a Remote Start Request was sent to the Vehicle. <Click> Upon success, the vehicle will send a response back to the cloud which will route the confirmation to the data platform <Click> as well as to the FordPass mobile App. <Click>
The next use Case, is our Ford Credit Variable Lease <Click> Let’s say that Simon leases a mustang for a term of $350/month for 24 months and gets 24,000 miles. Simon ride shares to work every day, and just uses the Mustang on the weekends <Click> As you know, Data like odometer is transmitted from the vehicle, to the Ford Cloud and back to the Enterprise Data Platform <Click> Ford Credit pulls the Odometer from our data store <Click> and with that information is able to offer Simon a variable lease of $250/month Base Plus a variable usage cost based on the actual miles driven that month. <Click>
A little bit about our Architecture patterns As discussed earlier, We can receive data from Azure Event hubs, secure FTP, and restful Webservices We also collect data from our Internal Ford databases. We use Oozie to schedule our workflows; and Ingest the data using Storm Spouts, Java Applications, Java web Services; or we Sqoop it from internal databases. The raw Data is brought directly into HDFS Where we transform it using Java, MapReduce and PIG And utilize De-dup logic, classify the data, partition the data, validate the data and; We than Load it into Hive and HBASE tables. <Long Pause> For security, We control Access to the Data with the help of Ranger. One other thing to note; We are currently conducting POC’s utilizing NiFi as an ingestion framework
Some of the Opportunities and challenges that we have faced include: Country Specific Regulations GPS not allowed to leave certain countries Right to be Forgotten Data Privacy Ford strictly follows all Automotive Privacy Alliance Principles Customers must explicitly Opt In Integration with Partners and Suppliers Each has a different Interface and data specification Difficult performing Integration Testing & Creating Test Data Storage & Updates of Data In Place updates in HIVE is challenging We use, HIVE for vehicle data HBASE for reference data Standards and Frameworks Created a CVDP common framework STORM for real time data Knox to land data in HDFS through WAS Security Data must be Encrypted At Rest and In Transit Country level partitioning
A little bit about Ford’s Hadoop Cluster We have 261 Data Nodes With over 5200 CPU cores 65 TB of RAM 5 PB usable storage In 10 Racks
Thank You Very much for your time, and I would like to open the floor for Q&A. Any business oriented questions I would be happy to answer; Any technical questions, Dan can handle; And any really technical questions can go to Simon or Joe.

Connected Vehicle Data Platform

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Connected Vehicle Data Platform

Similar a Connected Vehicle Data Platform (20)

Más de DataWorks Summit/Hadoop Summit

Más de DataWorks Summit/Hadoop Summit (20)

Último

Último (20)

Connected Vehicle Data Platform

Notas del editor