Microsoft SQL Server - StreamInsight Overview Presentation
•Descargar como PPTX, PDF•
2 recomendaciones•1,588 vistas
![Presentation Outline (hidden slide) ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (13)
Similar a Microsoft SQL Server - StreamInsight Overview Presentation
Similar a Microsoft SQL Server - StreamInsight Overview Presentation (20)
Más de Microsoft Private Cloud
Más de Microsoft Private Cloud (20)
Último
Último (20)
Microsoft SQL Server - StreamInsight Overview Presentation
- 1. SQL Server 2008 R2 StreamInsight Overview Name Title Microsoft Corporation
- 3. Intended Audience: ITDMs,DBAs, Database Developers, Database Architects, BI specialists
- 4. Objectives (what do you want the audience to take away):
- 5. Introduction to Complex Event Processing (CEP) and Microsoft’s StreamInsight solution
- 6. Technical overview of CEP architecture and development environment
- 8. Overview - Information Platform Vision, SQL Server 2008 Foundation, SQL Server 2008 R2 Value Proposition
- 9. Introduction to Complex Event Processing (CEP)
- 10. Business Challenges and key CEP benefits
- 12. CEP and LINQ Queries
- 14. Query Binding & Configuration
- 18. Scalable relational database platform Consistent, familiar model & tools Self-managed, highly available cloud services MPP support for 10s to 100s TB DW Highly scalable appliances Seamless integration with Microsoft BI A Comprehensive Platform Managed Self-Service BI Multi-server management Virtualization & Live Migration
- 19. TRUSTED, SCALABLE PLATFORM IT & DEVELOPER EFFICIENCY MANAGED SELF-SERVICE BI Enterprise-level security, scalability High-scale, complex event processing Data consistency across heterogeneous systems Multi-server management Virtualization & Live Migration Accelerated development& deployment Self-service analytics Self-service reporting Streamlined collaboration& management COMPREHENSIVE PLATFORM FOR IT VALUE MPP support for 10s-100s TB DW Highly scalable appliances Seamless BI Integration Scalable relational database platform Consistent, familiar model & toolsSelf-managed, highly available cloud service
- 20. What is CEP? Event Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. request output stream input stream response 6
- 21. Latency Scenarios for Event Processing Relational Database Applications CEP Target Scenarios Operational Analytics Applications, e.g., Logistics, etc. Data Warehousing Applications Web Analytics Applications Manufacturing Applications Financial trading Applications Monitoring Applications Aggregate Data Rate (Events/sec.) 7
- 23. Outages
- 24. Smart grids
- 26. React through device controllers
- 27. Aggregated data
- 30. Page layout
- 36. Event correlation from multiple sources
- 38. Virtuous Cycle: Monitor, Manage, Mine 9
- 39. Microsoft’s CEP Solution Data Sources, Operations, Assets, Feeds, Sensors, Devices Input Data Streams OutputData Streams Input Data Streams CEP Engine Monitor & Record Manage & Benefit Mine & Design f(x) f'(x) g(y) h(x,y) Results Deploy History Operational Data Store & Archive CEP Engine f(x) g(y) f'(x) h(x,y) 10
- 41. In the mid-tier – consolidate related data sources,
- 42. In the data center – historical archive, mining, large scale correlationWeb servers Sensors CEP CEP Feeds Devices CEP CEP CEP CEP CEP CEP CEP Complex Analytics & Mining 11
- 44. Low latency processing (< milliseconds)
- 45. High volume (>100,000 events /second)
- 46. Correlated events from multiple sources
- 47. Time interval oriented standing queries
- 48. Manufacturing: Process monitoring i.e. scheduling, control, inventory
- 49. Financial: algorithmic trading, compliance monitoring, fraud detection
- 50. Web: Click stream analysis ,ad-delivery model
- 53. Networking: traffic monitoring ,security intrusion detection
- 56. Event Types Events in Microsoft’s CEP platform use the .NET type system Events are structured and can have multiple fields Fields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristics Event sources populate time stamp fields
- 60. methods to handle event and stream behavior, and
- 61. properties to indicate adapter features to the engineAdapter features Feature Problem 16
- 62. Core CEP Query Engine Event Event Event Event Event Event Event Event Event CEP Engine Hosts “standing queries” Operators consume and produce streams Queries are composable Query results are computed incrementally Standing Queries Output Adapters Input Adapters Query instance management: Submit, start, stop Runtime statistics Takeaway: CEP engine does the heavy lifting for you when processing temporal event data 17
- 63. Typical CEP Queries Typical CEP queries require combination of functionality Complex type describes event properties Calculations introduce additional event properties Grouping by one or more event properties Aggregation for each event group over a pre-defined period of time, typically a window Multiple event groups monitored by the same query Correlate event streams Check for absence of activity with a data source Enrich events with reference data Collection of assets may change over time We want to make writing and maintaining those queries easy or even effortless
- 64. CEP Query Features Operators over streams Calculations (PROJECT) Correlation of streams from different data sources (JOIN) Check for absence of activity with a data source (EXISTS) Selection of events from streams (FILTER) Stream partitioning (GROUP & APPLY) Aggregation (SUM, COUNT, …) Ranking and heavy hitters (TOP-K) Temporal operations: hopping window, sliding window Extensibility – to add new domain-specific operators Queries are written over specific event types They can be evaluated on all data sources with the same event type Support for streaming data, reference data (lookup), and historical data (replay)
- 65. LINQ Query Examples LINQ Example – JOIN, PROJECT, FILTER: frome1inMyStream1 joine2inMyStream2e1.IDequalse2.ID wheree1.f2 = “foo” select new { e1.f1, e2.f4 }; Filter Project Window Grouping Join LINQ Example – GROUP&APPLY, WINDOW: frome3inMyStream3 groupe3bye3.iinto SubStreams froms4inSubStreams frome4ins4.SlidingWindow(FiveMinutes,ThreeSeconds) select new { pl =newMyNewPayload(e4.i, e4.f)};
- 66. Extensibility Built-in operators do not cover all functionality Need for domain-specific extensions Integrate with functionality available in existing libraries Support for extensions in the CEP platform: User-defined operators, functions, aggregates Code written in .NET, deployed as .NET assembly Query operators can refer to functionality of the assembly LINQ queries can easily refer to user-defined functionality Temporal snap-shot operator framework Interface to implement user-defined operators Manages operator state and snapshot changes Framework does the heavy lifting to deal with intricate temporal behavior such as out-of-order events
- 68. Server applicationsCompelling story for domain-specific extensions CEP development environment Builds on top of .NET Microsoft Visual Studio as IDE Applications written in C# Queries written in LINQ LINQ: varqueryFilter = fromcinTestEventStream wherec.Field1 > 1 selectc; 22
- 71. Re-use existing queries for new data source types
- 73. CEP Query Configuration Query Query binding: Coupling adapters with a query Event types required by the query need to be matched by types delivered by adapters Query can be re-used for all data sources of the same type No changes to query necessary for re-use Output Adapter Type Type Query Template Proj. Join Type Type Type Type Input Adapter Input Adapter
- 74. Recap: CEP Platform from Microsoft Event Event Event Event Event Event Event Event Event Development experience with .NET, C#, LINQ and Visual Studio 2008 CEP Application Development CEP platform from Microsoft to build event-driven applications Event targets Event sources Event-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementally CEP Engine Standing Queries Output Adapters Input Adapters Flexible adapter SDK with high performance to connect to different event sources and sinks The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data Static reference data 26
- 75. CEP Deployment Scenarios Custom CEP Application Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application ISV Application with CEP Engine .NET, C#, LINQ CEP Engine CEP Engine KPIs KPI mining ETL Pipeline with CEP engine CEP Engine CEP CEP CEP CEP Scenario 4: Operational Intelligence w/ CEP Scenario 3: CEP Enabled Device Reference data Device with Embedded CEP Engine Reference data Madison CEP Engine
- 76. CEP Platform Roadmap Focus on Custom development platform for CEP applications CEP platform for Microsoft partners TAP (Technology Adopter Program) will launch in July 2009 Product availability targeted for 2010 CTP2 TAP program CTP3 TAP continues General Availability time 2010 Late 2009 July 2009
- 78. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Notas del editor
- Now more than ever, organizations have the need to use information management to compete and grow in a difficult market by reducing costs and identifying the highest value opportunities for their organization. As we engage with customers, you explain the pressures you have to deliver more real-time information through rich applications while also reducing costs in this new economy. The converging IT trends of virtually free storage space, the rapid adoption of virtualization and the increasing capabilities of industry standard hardware, the emergence of the Cloud as a deployment option, and the need for real-time business information for all employees through easy to use tools are driving the data explosion we see today. You need a complete approach to managing, accessing and delivering information across your organization to accelerate and improve business decisions.. We also understand that at the center of delivering on these business needs there are people - IT Professionals, who support the expanding information needs of the business through IT services, Developers who build solutions in time to capture business opportunities in an increasingly competitive market and Business Users, who quickly mine volumes of data for business insights to help increase customer satisfaction and drive business results. Microsoft, along with our worldwide partners, is committed to deliver an Information Platform which enables your people and provides you with a complete set of technologies and tools to help you realize more value from your information at the lowest total cost of ownership. As we invest in the Information Platform we think about what we need to deliver to you in each of these 4 areas: Mission-Critical Platform Mission-Critical Platform for all your application requirements, delivered with the industry’s best TCOEnsuring your applications and systems are reliable, highly available, secure and deliver superior, predictable performance is top priority for IT and for Microsoft. At our core, we are focused on delivering an information platform which meets the needs for performance, scalability, availability, and security for your most mission critical business applications at a lower cost of ownership, including acquisition costs and the ongoing costs of support, management and maintenance. From the server to the datacenter we offer our customers a reliable information platform that grows with their needs at a third of the cost of our enterprise DB competitor.Secure and available infrastructureWe have seen tremendous growth in our Enterprise business – investing significantly in our datacenter capability on the Windows Server platform, building out the industry’s most widely deployed database platform. The largest enterprises on the planet are now running their top-tier applications on the Microsoft platform and seeing sustained high-performance and uptime that directly benefits their businesses’ bottom line. Scales to all business requirementsWith storage costs declining, and increasing data volumes, IT departments need the ability to scale up seamlessly to process an increasing number of transactions, as well as storing and processing larger volumes of information. Microsoft’s Information Platform provides high scalability and performance for OLTP and BI as well as new Data Warehousing capabilities to scale to hundreds of terabytes to deliver on the most critical business needs. A complete and interoperable platform that empowers IT to be more productive and agile The boundaries of the IT environment continue to be stretched and IT Professionals have increased responsibility to manage application requirements for the datacenter, across mobile devices and the desktop, and now out in the cloud. To successfully manage these resources across the enterprise IT professionals require a consistent, and productive platform, including easy to use and robust tools. Increased DBA Productivity and ControlSQL Server has long had a focus on DBA productivity – by providing rich management tools that empower administrators to take control of their environment. We continue to invest in tools that enable administrators to manage multiple servers at once via policies and to proactively manage system performance. We’re also investing to provide complementary tools for areas such as data integration and master data management. Consolidation and VirtualizationConsolidation and virtualization can deliver significant resource and operational benefits in the datacenter, particularly at a time when IT professionals are facing budget pressure along with the need to continue the same level of service. Together, Windows Server, System Center and SQL Server can deliver a virtualized Data Center, providing increased utilization of server resources to reduce costs, greater standardization to reduce administration overhead and greater agility to dynamically scale to support changing and growing business needs. The Power of Choice - Harnessing the CloudThe cloud offers new ways for IT to reduce the friction of connecting with customers and partners, the ability to quickly provision resources, and a way to reduce operational costs. Microsoft is delivering SQL Azure as part of the Windows Azure Platform to enable IT to manage these resources with a platform and tools that are consistent with their existing SQL Server environment.Dynamic DevelopmentIntegrated tools to help Developers more quickly build rich, intuitive and connected applications Developers are continually asked to deliver richer applications and services while lowering the time-to-solution to adapt to business needs and opportunities. With Visual Studio, the .NET Framework, and SQL Server developers have a highly productive platform to deliver data through their applications and to collaborate more efficiently with IT on deployment needs and requirements. The consistent development experience also gives developers the power to build applications for deployment across devices, desktops, the datacenter, and the cloud. Tight Integration across the stackSQL Server has deep integration with Visual Studio and .NET to provide developers with a rich, seamless development experience. Developers can use familiar tools and the power of a model-driven approach to vastly improve their productivity, by abstracting away from database specifics and instead focusing on the business logic and outcomes required. Platform for all DataBeing able to bring unstructured data alongside structured data in a single application gives developers the power to bring the relational model to new types of data. They can extend existing applications and develop new types of applications that incorporate location aware services, real time streaming data – all in a secure, synchronized way. Rich and embeddable user experiencesDevelopers can embed powerful user experiences into their applications through components such as spatial data for rich mapping scenarios, embeddable BI controls and data synchronization. Looking ahead, developers can integrate value-added cloud services into applications for a differentiated value prop and additional services opportunities.Pervasive InsightA complete Business Intelligence platform that enables users across the organization to derive greater insights through familiar toolsBusiness Intelligence continues to be the #1 priority for CIOs. Even as budgets are cut IT is being asked to do more to deliver information to the business. Getting the right information to the right person at the right time is critical for success now more than ever. This requires providing access through familiar tools to end users while also providing IT the ability to manage and secure access to the information. By enabling access to all types of users through familiar tools, in a secure and well managed way – businesses can get more value from their information than ever before. Microsoft helps companies consolidate information, deliver access and empower users at all levels of the business: strategic, tactical and operational decision makers. Integrate all your data and scale your reporting systemsWith SQL Server, you can integrate the many sources of data in your organization into large scale data warehouses and deliver rich, high-performance reporting to your whole organization. Enable users to derive insights through familiar toolsIncreased processing power and memory for desktops and laptops are enabling information workers to do more advanced data mining and analysis to improve decision making. We are extending our capabilities to deliver the power of self-service business intelligence to individual users through the familiar tools they use everyday such as Microsoft Excel. Enable collaborative decision making Today organizations have relevant data that lives in the datacenter, on user devices and desktops, and through external services. It is critical for users to be able to draw upon the wealth of information that is distributed throughout the organization and to do analysis and reporting with the data sources that are most relevant. The real value is that individual insight can be easily shared across work groups and across the organization. With Microsoft Office SharePoint, individual information workers can share and publish insights and analysis tools to scale reach and deliver greater value to the organization.ClosingLooking across the Information Platform, we think about how these capabilities can better support our customers’ information needs. For example a national retailer could use technology to track the demographics, volume and location of customers throughout stores across the country. By processing large volume of data in real-time, they could then evaluate the needs of their customer base at any given moment and make changes to promotions or the store environment like the background music, to better appeal to the audience and drive real-time purchase. Looking across the Information Platform, the ability to dynamically respond to customer needs could be built by working with your development organization to develop a new application or extending an existing application, partnering with IT to support the application through greater utilization of existing assets, and delivering information to business users through familiar tools. By providing business information pervasively throughout the organization in real-time this organization would be able to much more quickly identify and capture business opportunities.The Information Platform delivers the rich capabilities of SQL Server and SQL Azure to enable IT, developers and end users to deliver on your most critical business needs and maximize the value of your information throughout the organization at a low cost of ownership.
- The purpose of this slide is to communicate the next wave of SQL innovation and how it should be messaged. There are three key areas of innovation: SQL Server 2008 R2 – the next generation release of the SQL Server database platform, Parallel Data Warehouse, appliances for high scale data warehousing and SQL Azure – scalable relational database platform service in the cloud. When you talk to each of these innovations, you want to emphasize that each is built on the solid foundation of SQL Server 2008.SQL Server 2008 R2 enables businesses to make more timely, better informed decisions by enabling users to access, integrate, analyze and share information using tools they are already familiar with – Microsoft Office. DBAs can more efficiently manage databases and applications at scale with multi-server management. Enhanced support for Virtualization will provide opportunities to further optimize resources through consolidation.SQL Azure – delivers on the Microsoft Data Platform vision of extending the SQL Server capabilities to the cloud as web-based services. SQL Azure provides an internet-facing database and advanced query processing services and is ideal solution for customers building new applications or integrating with existing investments in the cloud. Built on SQL Server technologies, SQL Azure provides the same enterprise-class availability, scalability , and security with the benefits of built-in data protection, self healing and disaster recovery as well as a consistent SQL Server programming model & tools. SQL Azure is scheduled for release later this year.Parallel Data Warehouse is a highly scalable data warehouse appliance. It uses Massively Parallel Processing (MPP) to deliver high performance and scalability on SQL Server 2008, Windows Server 2008 and industry-standard hardware. Parallel Data Warehouse is a separate release/edition but will launch jointly with R2 in the first half of 2010.
- The “hero” offering that we’ll really sell broadly is SQL Server 2008 R2. You’ll notice that the messaging pillars for R2 are aligned with the Trusted, Productive, Intelligent messaging pillars for SQL Server 2008.To provide a comprehensive view of the R2 release (especially for SA discussions) this slide focuses on the key technology investments for each of the pillars. Trusted, Scalable Platform, it’s important to re-emphasize the enterprise-level security, high availability and scale delivered in SQL 2008 for mission critical applications. What’s new to the R2 release is StreamInsight for Complex Event Processing, Master Data Services for data consistency across heterogeneous systems, SQL Server support for up to 256 Logical Processors – providing greater flexibility in hardware choice, and enhanced data compression including Unicode compression for greater storage efficiency.IT & Developer Efficiency, the release-defining scenarios for R2 are Virtualization and Multi-server Management to increase IT productivity and reduce data management costs. This is enabled by the new manageability enhancements delivered in Management Studio plus R2 support for Windows Server 2008 R2 Hyper-V Virtualization and Live Migration. With R2, we introduce a new concept to the market, a single unit of deployment which will significantly increase the efficiencies around data-tier application deployments and upgrades resulting in dramatically streamlined consolidation and management efforts. Managed Self Service BI – Self-service analytics is delivered with PowerPivot – which is about enabling users to get the data they need without IT having to do a 6-month project to create a new application. PowerPivot gives users the power and flexibility to take internal data, integrate it with other information – and within Excel do all the slicing and dicing you need to get to the view of information you really need for decision making. This is all done “in memory” through really high compression in RAM – which is very powerful. None of the other major BI vendors can do this. User can easily share their application with others by publishing it to SharePoint.Once published, there’s a PowerPivot Operations Dashboard in SharePoint that provides IT Pros with the ability to manage business intelligence through SharePoint like it was just another service they provide. This new dashboard allows IT Pros to look at which reports are being run, who’s running them, how often are they running them, is the server being over utilized, how can I move the app dynamically to another server…BI is basically another managed object for IT. The reports the users publish is the object and we enable IT to understand how it is being used and apply rules and policies to that.
- Your free PASS membership enables access to 12 Virtual Chapters, 24 Hours of PASS broadcasts, local PASS Chapters, professional development resources, newsletters, events calendar, and much moreMeet up with like-minded professionals throughout the year at face-to-face PASS Chapter meetings, discuss topical SQL Server issues, share tips and tricks, network, enjoy special guest speakers and get access to all things PASSAt PASS Summit,meet top industry experts , learn about best practices, effective troubleshooting, how to prevent issues, save money, and build a better SQL Server environment