Tracking Pandemic Recovery Using Graphs
•Descargar como PPTX, PDF•
0 recomendaciones•13 vistas
Tracking Pandemic Recovery Using Graphs
Recomendados
Recomendados
Más contenido relacionado
Similar a Tracking Pandemic Recovery Using Graphs
Similar a Tracking Pandemic Recovery Using Graphs (20)
Más de Neo4j
Más de Neo4j (20)
Último
Último (20)
Tracking Pandemic Recovery Using Graphs
- 1. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. Erik Erickson Chief Data Officer Alexander Long Data Engineer Tracking Pandemic Recovery using Graphs
- 2. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. Hennepin County • MN’s most populous county • 34th most populous county in the United States • ~1.26 million residents • ~9,000 employees • State & local partnership for public health ◦ State, county, and city entities ◦ Local delivery
- 3. © 2022 Neo4j, Inc. All rights reserved. Pandemic response • March 12th, 2020 – First confirmed COVID-19 case in Hennepin County • March 13th, 2020 – Hennepin County activated its emergency response structure ◦ April 2020 – data work to support the response had begun ◦ May 2020 – first iteration of operational dashboards established
- 4. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. Critical Gap • Limited data insights into how the pandemic was impacting our community ◦ Official statistics have a significant lag • Launched effort to try and better understand the community impact of covid-19 using timelier, yet non-traditional indicators The depth of the downturns induced by covid-19 has put a premium on swift intelligence.” -The Economist (July 2020) “
- 5. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. Making sense of non- traditional data sources • Indicators abound related to key aspects of interest ◦ Google search trends ◦ Open table reservations ◦ Anonymized cell phone data ◦ Etc. • Will the whole be greater than the sum of its parts?
- 6. © 2022 Neo4j, Inc. All rights reserved. 6 ZHVI Zillow Home Value Index Monthly (~1 mo. lag) Neighborhood level Challenges beyond data quality Affinity Credit/debit spending data Weekly (~2 wk lag) County level BTS Bureau of transportation experimental data Daily (~1 day) County level
- 7. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. In addition to challenges inherent in each indicator, how can we meaningfully integrate these data?
- 8. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. Graphs as a solution • Looked towards graph databases to meaningfully integrate data. • Pilots in early 2020 ◦ CosmosDB ◦ TigerGraph ◦ Neo4J • Looked towards graph databases to meaningfully integrate data. • Pilots in early 2020 ◦ CosmosDB ◦ TigerGraph ◦ Neo4J Exploring graphs as a solution From Carlson Analytics Lab – "Hennepin County Use Case Presentation", May 2020
- 9. © 2022 Neo4j, Inc. All rights reserved. Iterative process w/ iterative design • Implementation ◦ New data or improved front-end functionality • Feedback ◦ Interviewed or requested written feedback from domain experts • Planning ◦ Use feedback to prioritize next steps
- 10. © 2022 Neo4j, Inc. All rights reserved. Final Product: Power BI Dashboard
- 11. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. As the pandemic wore on, the graph helped us fulfill two key duties: - Interpret complex data - React to a changing situation
- 12. © 2022 Neo4j, Inc. All rights reserved. Interpret complex data • Continuity between logical and storage models ◦ Data relationships are built into storage structure ◦ Intuitive to understand = easier to use
- 13. © 2022 Neo4j, Inc. All rights reserved. React to changing situation • Schemaless provides flexibility ◦ COVID-19 data changing constantly ◦ Imperative that we adapt quickly Implement Feedback Planning
- 14. © 2022 Neo4j, Inc. All rights reserved. A brief peak at the tech stack • Starting point: Cypher / APOC load CSVs from "import" folder • Now: Using Neo4j Python library to automate scripts • Soon: Migrating code to the cloud, integrating with existing data pipelines and cloud storage
- 15. © 2022 Neo4j, Inc. All rights reserved. Lessons learned and next steps • Making better use of the graph ◦ Graph Data Science library • Integrate person records – explore person/place relationships • Integrate property records – graph enables "spatial" queries without requiring special geospatial tools
- 16. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. What questions do you have?
- 17. © 2022 Neo4j, Inc. All rights reserved. © 2022 Neo4j, Inc. All rights reserved. Erik Erickson Chief Data Officer Alexander Long Data Engineer Thank you Tracking Pandemic Recovery using Graphs
Notas del editor
- Data helping guide decisions in COVID-19 response | Hennepin County Established an analytics infrastructure to monitor Number of cases, hospitalizations and deaths Key county response activities (vaccine administration, PPE delivery, small business support, etc.)
- "Official statistics" - referencing ACS / census data here?
- Had an inkling that graphs could help us with this. In early 2020 we piloted a few graph technologies CosmosDB (Microsoft shop) TigerGraph Internuntius consulting Partnership with Carlson Analytics Lab (University of Minnesota) Successfully modelled interaction between SNAP (food stamp) recipients and community demographics to inform the placement of food shelves. With these promising results we began to work on a database to collect metrics on the impact of COVID in our community.
- It was obvious to everyone that the situation with COVID was changing quickly. Day to day changes in data availability are hard to handle for any organization, and we don't have the most mature tech stack in the world. We knew we needed to follow an iterative process with fast cycles. Step 1) implement a new idea, like adding new data, a new summary measure, or improved functionality to the user-facing dashboard Step 2) Gather feedback on our implementation, and identify gaps alongside subject matter experts Step 3) Use that feedback to fix issues or plan future improvements As the project came together, these iterations get closer and closer together. The schemaless nature of the graph storage was a key factor to increasing our development speed.
- Here's a look at our final product, an interactive dashboard built in Power BI, using a Neo4j database as its main back end data store. This was a win in multiple ways. First, it confirms our ability to write efficient cypher queries to support these types of reports (both in terms of execution time and storage space, both of which an interactive dashboard has little). Second, it showcases our ability to aggregate geography and date hierarchies at any desired summary level. On the right side, we show a monthly indicator at city-level geographies, but that same rollup can be done for quarters or years, for the full county or comissioner districts, only by attaching relationships to those "dimensional" nodes.
- One of the first benefits we realized with our COVID graph DB is that having a storage model that closely follows the logical structure allows for greater ease of use. This includes faster development, and easier for us to explain technical details to business partners. For example, "can we summarize this at the county level?" or "can I see all the housing measures for this city?". A simple modification to the query rather than a complicated join or worse, having to completely rewrite tables to fit unexpected schemas. 1) Graph relationships - Easier to communicate capabilities and limitions of each data point
- https://commons.wikimedia.org/wiki/File:Jenga_distorted.jpg We said earlier that the volatile nature of the problem, especially early in the pandemic, impressed upon us the need to iterate rapidly. The schemaless data storage of the graph db was instrumental in this. We have some measures that have been around for years, some that became available just as the first COVID cases appeared in Minnesota, and others that weren't available for months after. As the pandemic wound down, some indicators have stopped reporting. Others have changed summary levels or collection methodology. Since we weren't pinned down by a particular table schema, we could quickly add new indicators, remove old ones, and modify relationships to keep ourselves on track.
- Cypher / APOC library to load CSVs from "import" folder Shifted to python scripts running on the database's server, executing Cypher via the transactional API Today we run python scripts remotely, loading data using Neo4j python library Soon: moving those scripts to the cloud (Databricks on Azure), integrating more closely with existing data pipelines and cloud storage
- Q&A
- Q&A