L’obiettivo è quello di fare una panoramica dello stato dell’arte sulle tecnologie a supporto dei database. Alcuni esempi sono la tecnologia in-memory integrata con le funzionalità di analisi operative in tempo reale e della tecnologia Always Encrypted per la protezione dei dati utilizzati in locale o durante gli spostamenti. La tecnologia in-memory consente di migliorare di 30 volte le performance delle transazioni utilizzando hardware standard di settore. Inoltre i Big Data e l'analisi sono diventati un importante fattore di differenziazione competitivo, ma la gestione delle enormi quantità di dati correlate a un tempo di attività 24 ore su 24 continua a essere una sfida per l'IT. Oggi è più importante che mai soddisfare a livello aziendale l'esigenza di prestazioni, disponibilità e sicurezza efficace per gestire carichi di lavoro mission-critical a un costo contenuto. Le soluzioni Microsoft fissano un nuovo standard nelle performance mission-critical.
2. SQL Server
2008
SQL Server
2008 R2
SQL Server
2000XML ● KPIs
Compression ● Policy-Based Mgmt ● Programmability
PowerPivot ● SharePoint Integration ● Master Data Services
SQL Server
2012
AlwaysOn ● ColumnStore Index ● Data Quality Services ● Power View ● Cloud Connectivity
SQL Server
2014
In-Memory Across Workloads ● Performance & Scale ● Hybrid Cloud Optimized ● HDInsight ● Cloud BI
Management Studio ● Mirroring
SQL Server
2005
SQL Server
2016
Enhanced Always On ● In-Memory OLTP ● Stretch DB ● Temporal Tables ● Enhanced Backup to Cloud ● Polybase ● Real-
Time Operational Analytics ● Row-Level Security ● Query Store ● R Services● Always Encrypted ● Mobile BI
The evolution of Microsoft SQL Server
3. How we develop SQL
Cloud-first but not cloud-only
Use SQL Database to improve core SQL Server features and cadence
Many interesting and compelling on-premises cloud scenarios
SQL Server
and APS
Azure
SQL Virtual
Machines
Azure
SQL Database
Azure
SQL Data
Warehouse
6. ALTER TABLE Sales.SalesOrderDetail
ALTER INDEX PK_SalesOrderID
REBUILD
WITH (BUCKET_COUNT=100000000)
ALTER support
Full schema change support: add/alter/drop
column/constraint
Add/drop index supported
Performance
The ALTER TABLE syntax is used for making changes to the table schema, as well as for
adding, deleting, and rebuilding indexes
Indexes are considered part of the table definition
Key advantage is the ability to change the BUCKET_COUNT with an ALTER INDEX
statement
7. CREATE PROCEDURE [dbo].[usp_1]
WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER
AS BEGIN ATOMIC WITH
(
TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE =
N'us_english'
)
SELECT c1, c2 from dbo.T1
END
GO
ALTER PROCEDURE [dbo].[usp_1]
WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER
AS BEGIN ATOMIC WITH
(
TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE =
N'us_english'
)
SELECT c1 from dbo.T1
END
GO
You can now perform ALTER
operations on natively compiled
stored procedures using the
ALTER PROCEDURE statement
Use sp_recompile to
recompile stored procedures on
the next execution
Altering natively compiled stored procedures
Performance
8. Transact-SQL
Full support for all Collation and Unicode Support
(var)char columns can use any code page supported by SQL Server
Character columns in index keys can use any SQL Server collation
Expressions in natively compiled modules as well as constraints on memory-
optimized tables can use any SQL Server collation
Scalar User-Defined Functions for In-Memory OLTP
Create, drop, and alter natively compiled, scalar user-defined functions
Native compilation improves performance of the evaluation of UDFs in T-SQL
Performance
10. In SQL Server 2016, the
storage for memory-
optimized tables will be
encrypted as part of enabling
TDE on the database
Simply follow the same steps
as you would for a disk-based
database
Support for Transparent Data Encryption (TDE)
Windows Operating System
Level Data Protection
SQL Server
Instance Level
User Database
Level Database Encryption Key
Service Master Key
DPAPI encrypts the Service Master Key
Master
Database Level Database Encryption Key
Service Master Key Encrypts the Database
master Key for the master Database
Database Master Key of the master Database
creates a certificate in the master database
The certificate encrypts the database
Encryption Key in the user database
The entire user database is secured by the
Datbase Encryption Key (DEK) of the user
database by using transparent database
encryption
Created at a time of SQL Server setup
Statement:
CREAT MASTER KEY…
Statement:
CREATE CERTIFICATE…
Statement:
CREATE DATABASE ENCRYPTION KEY…
Statement:
ALTER DATABSE… SET ENCRYPTION
Performance
11. New Transaction Performance Analysis Overview report
New report replaces the
need to use the
Management Data
Warehouse to analyze
which tables and stored
procedures are candidates
for in-memory
optimization
Performance
13. Data changes over time
Tracking and analyzing changes is often important
Temporal in DB
Automatically tracks history of data changes
Enables easy querying of historical data states
Advantages over workarounds
Simplifies app development and maintenance
Efficiently handles complex logic in DB engine
Why temporal
Time travel Data audit
Slowly changing
dimensions
Repair record-level
corruptions
Performance
14. No change in programming model New Insights
INSERT / BULK INSERT
UPDATE
DELETE
MERGE
DML SELECT * FROM temporal
Querying
How to start with temporal
CREATE temporal
TABLE PERIOD FOR
SYSTEM_TIME…
ALTER regular_table
TABLE ADD
PERIOD…
DDL
FOR SYSTEM_TIME
AS OF
FROM..TO
BETWEEN..AND
CONTAINED IN
Temporal
Querying
ANSI 2011
compliant
Performance
15. Temporal table (actual data)
Insert / Bulk Insert
* Old versions
Update */ Delete *
How does system-time work?
History table
Performance
16. Temporal table (actual data)
Temporal queries *
(Time travel, etc.)
How does system-time work?
History table
Regular queries
(current data)
* Include historical
version
Performance
18. Data disclosure
prevention
Client-side encryption of
sensitive data using keys that
are never given to the
database system
Queries on
encrypted data
Support for equality
comparison, including join,
group by, and distinct
operators
Application
transparency
Minimal application changes
via server and client library
enhancements
Allows customers to securely store sensitive data outside of their trust boundary.
Data remains protected from high-privileged, yet unauthorized, users.
The need for Always Encrypted
Security
19. trust boundary
"SELECT Name FROM Customers
WHERE SSN = @SSN","111-22-3333"
How it works
Help protect data at rest and in motion, on-premises and in the cloud
Name
Wayne Jefferson
ADO .NET
Name
0x19ca706fbd9a
Result SetResult Set
Client
Name SSN Country
0x19ca706fbd9a 0x7ff654ae6d USA
SQL Server or SQL Database
"SELECT Name FROM Customers
WHERE SSN = @SSN",0x7ff654ae6d
ciphertext
Encrypted sensitive data and corresponding keys
are never seen in plaintext in SQL Server
dbo.Customers
ciphertext
Security
20. Randomized encryption
Encrypt('123-45-6789') = 0x17cfd50a
Repeat: Encrypt('123-45-6789') = 0x9b1fcf32
Allows for transparent retrieval of encrypted
data but NO operations
More secure
Deterministic encryption
Encrypt('123-45-6789') = 0x85a55d3f
Repeat: Encrypt('123-45-6789') = 0x85a55d3f
Allows for transparent retrieval of encrypted
data AND equality comparison
E.g. in WHERE clauses and joins, distinct,
group by
Two types of encryption
available
Randomized encryption uses a method
that encrypts data in a less predictable
manner
Deterministic encryption uses a method
that always generates the same encrypted
value for any given plaintext value
Types of encryption for Always Encrypted
Security
22. Configuration made easy in new Azure portal
Policy-driven at table and column level, for
defined set of users
Data masking applied in real time to query
results based on policy
Multiple masking functions available, such as full
or partial, for various sensitive data categories
(credit card numbers, SSN)
SQL Database
SQL Server 2016
Table.CreditCardNo
4465-6571-7868-5796
4468-7746-3848-1978
4484-5434-6858-6550
Real-time data masking,
partial masking
Prevent abuse of sensitive data
by hiding it from users
Security
Dynamic Data Masking
23. Regulatory
compliance
Sensitive data
protection
Agility and
transparency
Data is masked on the fly, with
underlying data in database
remaining intact (transparent to
application and applied
according to user privilege)
Limit access to sensitive data by defining policies to obfuscate specific database fields, without
affecting database integrity
Security
Benefits of Dynamic Data Masking
24. Dynamic data masking walkthrough
ALTER TABLE [Employee] ALTER COLUMN [SocialSecurityNumber]
ADD MASKED WITH (FUNCTION = ‘SSN()’)
ALTER TABLE [Employee] ALTER COLUMN [Email]
ADD MASKED WITH (FUNCTION = ‘EMAIL()’)
ALTER TABLE [Employee] ALTER COLUMN [Salary]
ADD MASKED WITH (FUNCTION = ‘RANDOM(1,20000)’)
GRANT UNMASK to admin1
1) Security officer defines dynamic data masking policy in T-SQL over sensitive data in Employee table
2) Application user selects from Employee table
3) Dynamic data masking policy obfuscates the sensitive data in the query results
SELECT [Name],
[SocialSecurityNumber],
[Email],
[Salary]
FROM [Employee]
Security
26. Nurse
Database
One
Policy manager creates a filter predicate and security policy
Two
App user (e.g., nurse) selects from Patients table
Three
Security Policy transparently rewrites query to apply filter predicate
CREATE FUNCTION dbo.fn_securitypredicate(@wing int)
RETURNS TABLE WITH SCHEMABINDING AS
return SELECT 1 as [fn_securitypredicate_result] FROM
StaffDuties d INNER JOIN Employees e
ON (d.EmpId = e.EmpId)
WHERE e.UserSID = SUSER_SID() AND @wing = d.Wing;
CREATE SECURITY POLICY dbo.SecPol
ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing) ON Patients
WITH (STATE = ON)
Filter
Predicate:
INNER
JOIN…
Security
Policy
Application
Patients
SELECT * FROM Patients
SELECT * FROM Patients
SEMIJOIN APPLY dbo.fn_securitypredicate(patients.Wing);
SELECT Patients.* FROM Patients,
StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId)
WHERE e.UserSID = SUSER_SID() AND Patients.wing = d.Wing;
Security
RLS in three steps
Policy Manager
27. -- The following syntax creates a security policy with a filter
predicate for the Customer table, and leaves the security policy
disabled
CREATE SECURITY POLICY [FederatedSecurityPolicy]
ADD FILTER PREDICATE
[rls].[fn_securitypredicate]([CustomerId])
ON [dbo].[Customer];
-- Create a new schema and predicate function, which will use the
application user ID stored in CONTEXT_INFO to filter rows.
CREATE FUNCTION rls.fn_securitypredicate (@AppUserId int)
RETURNS TABLE
WITH SCHEMABINDING
AS
RETURN (
SELECT 1 AS fn_securitypredicate_result
WHERE
DATABASE_PRINCIPAL_ID() = DATABASE_PRINCIPAL_ID('dbo') --
application context
AND CONTEXT_INFO() = CONVERT(VARBINARY(128), @AppUserId);
GO
Creates security policy for
row-level security
The following examples
demonstrate use of CREATE
SECURITY POLICY syntax
For an example of a complete
security policy scenario, see
Row-Level Security
Create security policy
Security
28. Capability
Row-Level Security provides fine-grained access control
over rows in a table based on conditions you set up
Benefits
Store data for many users in same databases and tables
while limiting access by other users who share same tables
Security
Summary: Row-Level Security
29. Security enhancements
Built-in tools for enabling compliance: SQL Server audit tools
Create server audits, with server audit specifications (audited events
can be written to event logs or to audit files)
User-defined audit: Allows middle-tier application to write custom events into audit log, which
enables more flexibility to store audit information
Audit filtering: Provides greater flexibility to filter wanted events in audit log
Audit resilience: Audit logging is now tolerant to loss of connectivity to target directory and will
recover automatically once network connection is re-established
Security
30. Azure Key Vault support
Azure Key Vault:
Central key management that leverages hardware security modules (HSMs), separation of key
management from data management
Support for AKV available through SQL Server Connector for AKV
Extensible Key Management (EKM) provider for SQL Server
Leverage Azure Key Vault for managing encryption keys
Both on-premises and SQL Server-in-a-VM users can assume control of encryption keys for
Transparent Data Encryption (TDE), Column Level Encryption (CLE), and Backup Encryption while
leveraging additional security benefits of Azure Key Vault
Security
31. Transparent Data Encryption (TDE)
Encrypt data when it is stored on disk, and decrypt
it when read into memory
Developers can encrypt database files, log files, and backup files without
changing existing applications
Intel AES-NI hardware encryption acceleration
Support for storage of memory-optimized OLTP tables (new)
Exclusive to SQL Server Enterprise edition
Security
32. Transparent Data Encryption (TDE)
Encrypt backup by specifying encryption algorithm and encryptor
Supports on-premises and Azure storage locations
Configurable for Managed Backup to Windows Azure
Backup encryption now supported with compression, using AES-
NI hardware acceleration
Security
33. Summary: Security
Always encrypted: Sensitive data always encrypted (and queryable)
Dynamic Data Masking: Real-time obfuscation of data
Row-Level Security: Fine-grained access control of table rows
Audit success/failure of database operations
TDE support for storage of In-Memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record
changes
Security
34. AlwaysOn
Failover on SQL Server instance level
Shared storage (SAN/SMB)
Failover can take minutes based on load
Multi-node clustering
Passive secondary nodes
Failover on database level
Direct attached storage
Failover takes seconds
Multiple secondaries
Active secondaries
Availability
Failover Cluster Instances
for servers
Availability Groups
for groups of databases
35. WSFC node WSFC node
Failover Cluster Instances
Server failover
Shared storage
Multi-node clustering
Passive secondary nodes
Failover in minutes
SQL Server 2016
Shared storage
SQL Server 2016SQL
Server
Failover
Cluster
Instance
Availability
37. Availability Groups + Failover Clustering
AlwaysOn:
Failover Cluster Instances
and Availability Groups
work together to ensure
data is accessible despite
failures
Availability
Windows Server Failover Clustering (WSFC) Cluster
Network Subnet Network Subnet
Node NodeNodeNodeNode
SQL Server
Instance
SQL Server
Instance
SQL Server
Instance
AlwaysOn SQL Server
Failover Cluster Instance
Primary Replica Secondary Replica Secondary Replica Secondary Replica
AlwaysOn Availability Group
Instance
Network Name
WSFC
Configuration
WSFC
Configuration
WSFC
Configuration
WSFC
Configuration
WSFC
Configuration
Instance
Network Name
Instance
Network Name
Instance
Network Name
Availability Group Listener Virtual Network Name
Storage Storage Storage Share Storage
39. Interest in big data spurs customer demand
Adoption of big data technologies like Hadoop
Increase in number and
variety of data sources
that generate large
quantities of data
Realization that data is
“too valuable” to delete
Dramatic decline in
hardware cost,
especially storage
$
40. PolyBase and queries
Provides a scalable, T-SQL-compatible query processing
framework for combining data from both universes
Access any data
41. PolyBase View
Execute T-SQL queries against
relational data in SQL Server and
semi-structured data in Hadoop or
Azure Blob Storage
Leverage existing T-SQL skills and BI
tools to gain insights from different
data stores
Access any data
PolyBase View in SQL Server 2016
43. Components introduced in SQL Server 2016
PolyBase Engine Service
PolyBase Data Movement Service
(with HDFS Bridge)
External table constructs
MR pushdown computation support
Access any data
Head Node
SQL 2016
PolyBase Engine
PolyBase DMS
44. Step 1: Set up a Hadoop cluster…
Hortonworks or Cloudera distributions
Hadoop 2.0 or above
Linux or Windows
On-premises or in Azure
Access any data
Hadoop Cluster
Namenode Datanode Datanode Datanode Datanode
File
System
AB 01 01 01 01
File
System
File
System
File
System
45. Step 1: …or set up an Azure Storage Blob
Azure Storage Blob (ASB) exposes an HDFS layer
PolyBase reads and writes from ASB using Hadoop
RecordReader/RecordWrite
No compute pushdown support for ASB
Access any data
Azure
Azure
Storage
Volume
Azure
Storage
Volume
Azure
Storage
Volume
46. Access any data
Query relational
and non-relational
data, on-premises
and in Azure
T-SQL query
Apps
SQL Server Hadoop
Summary: PolyBase
Query relational and non-relational data with T-SQL
49. Analysis Services Overview
Analysis Services (SSAS) is an online analytical data engine used in decision support and
business analytics
It provides analytical data for business reports and client applications such as Reporting
Services reports, Power BI, Excel, and other third-party data visualization tools
SSAS supports two modelling modes—multidimensional and tabular
The multidimensional mode also includes a data mining engine
50. Analysis Services Overview
One Semantic Model - Two Ways to Develop
Dimensions and measure groups
Highly scalable and mature
Feature rich and complex
Multidimensional models
Tables and relationships
Fast by design with in-memory
Easy to get started and simple
Tabular models
51. Analysis Services Architectural Overview
BI SEMANTIC MODEL
Data access
Use SQL Server
Data Tools for
Visual Studio to
create BI semantic
models
DATA SOURCES
CLIENT TOOLS
Data model
Business logic
Queries
52. Reporting Services Overview
Reporting Services (SSRS) delivers enterprise, web-enabled reporting functionality
Implemented as a Report Server, in one of two modes—Native and SharePoint—it can
scale to support thousands of users
SSRS 2016 is extended with a new capability to deliver mobile reports, and is enhanced
with a new modern web portal
To contrast classic RDL reports with the new mobile reports, SSRS reports are now referred to as either
paginated or mobile
53. Reporting Services architecture
Web APIs
Report server
database
Web portal
Web browsers
Power BI Mobile
Report Builder
Mobile Report Publisher
Visual Studio
SQL Server
SQL Server
Analysis Services
Oracle Teradata
ODBC, OLE DB More/custom
Processing and
rendering
Scheduling and
delivery
Email File share
SharePoint Power BI
Custom
REPORT VIEWING (PCs, mobile)
REPORT DESIGN (PCs)
REPORT SERVER DATA SOURCES
REPORT DELIVERY DESTINATIONS
Custom apps/LOB
54. SQL Server 2016
Enhanced analysis
Modern reports
Rich tools
Mobile reports
Hybrid BI
SharePoint 2016
support
55. New SSRS Features
Supports the current versions of Microsoft.NET Framework 4
This includes 4.0 and 4.5.1
If no version of .NET Framework 4.x is installed, SQL Server setup installs .NET 4.0 during the feature
installation
Render paginated reports as PowerPoint files
Text boxes and images render as PowerPoint text boxes and images
Data visualizations render as images
The PowerPoint slide layout can be adjusted to suit, by moving and resizing individual objects
56. New SSRS Features
Ability to deliver subscriptions by using a single file share account, reusable across
multiple subscriptions
Pin report items to Power BI dashboards
Requires registering the Report Server with Power BI
Can pin charts, gauge panels, maps and images, and configure how often the data is refreshed
Dashboard tile refresh is achieved with subscriptions
Supported by all SQL Server editions
Support of SharePoint mode for SharePoint 2016
57. New SSRS Features
Paginated report authoring:
Custom parameter pane, allowing
design-time control over report
parameter layout
Two new chart types: Tree Map and
Sunburst charts, which effectively
visualize hierarchical data
Report Builder
Now has a new modern look and feel,
with streamlined UI elements
Supports High DPI (Dots Per Inch)
scaling and devices
58. New SSRS Features
Updated data source support:
Oracle Database 12c (uses ODP.NET)
Oracle Essbase 11
SAP BW 7.5
Teradata 15
Personalized connection strings:
Enables data sources to pass current user name as custom data
Data Source=localhost;Initial Catalog=Sales Analysis;CustomData={{ UserID }}
59. New SSRS Features
Key Performance Indicators (KPIs):
Deliver visual cues that communicate the amount of progress made toward a goal
Typically valuable for teams, managers, and businesses by enabling them to quickly evaluate the
progress made toward measurable goals
Developed in, and displayed by, the new SSRS web portal
Supported by Enterprise edition, and Native mode only
60. New SSRS Features
Key Performance Indicators (KPIs):
Enable defining value, goal, status and trend metrics
The value, goal and status metrics are single values—typically numeric, though text is also possible
The trend metric is designed to use a time series (e.g. monthly values)
Values can be sourced from shared datasets, or by manually entered values
Datasets can be parameterized
Datasets must use caching
Cache plans should be used to schedule the refresh
Can be configured to launch related content—either a mobile report, or a custom URL
The status colors—by default—are green (1), amber (0), or red (-1), and these can be overridden by custom
branding
61. New SSRS Features
The web portal is a new HTML5 application that targets modern web
standards and modern browsers
Report Manager has been removed and the web portal is now the default used to
browse/manage/consume the Report Server catalog
Stores, manages and secures reports and related assets:
Paginated reports
Mobile reports
Key Performance Indicators (KPIs)
Power BI Desktop files
Excel workbooks
62. New SSRS Features
Supports custom branding, allowing the customization of logo, web portal colors and a mobile report
theme
63. New SSRS Features
PDF replaces ActiveX for remote printing (plug-in free printing)
New enhanced subscription management:
Easily enable/disable
Provide a description
User can change the owner
65. Consume mobile reports
Native apps for all major mobile platforms
One mobile app for Power BI and
SSRS mobile reports
Native mobile experience
optimized for touch
Keep data up-to-date with real-time
query or scheduled data refresh
Interact with mobile reports even
when disconnected
67. Revolution R Enterprise and SQL
Big data analytics platform
Based on open source R
High-performance, scalable, full-featured
Statistical and machine-learning algorithms are performant,
scalable, and distributable
Write once, deploy anywhere
Scripts and models can be executed on a variety of
platforms, including non-Microsoft (Hadoop,
Teradata in-DB)
Integration with the R Ecosystem
Analytic algorithms accessed via R function with similar
syntax for R users (with arbitrary R functions/packages)
Advanced analytics
Data source
integration
Parallel external-memory
algorithm library
Data
Compute context
integration
Resources
Requests
R scripts +
CRAN
algorithms
68. SQL Server 2016 R integration scenario
Exploration
Use Revolution R Enterprise (RRE) from R
integrated development environment
(IDE) to analyze large data sets and build
predictive and embedded models with
compute on SQL Server machine (SQL
Server compute context)
Operationalization
Developer can operationalize R
script/model over SQL Server data by
using T-SQL constructs
DBA can manage resources, plus secure
and govern R runtime execution in SQL
Server
Advanced analytics
69. R script usage from SQL Server
Original R script:
IrisPredict <- function(data, model){
library(e1071)
predicted_species <- predict(model, data)
return(predicted_species)
}
library(RODBC)
conn <- odbcConnect("MySqlAzure", uid = myUser, pwd =
myPassword);
Iris_data <-sqlFetch(conn, "Iris_Data");
Iris_model <-sqlQuery(conn, "select model from my_iris_model");
IrisPredict (Iris_data, model);
Calling R script from SQL Server:
/* Input table schema */
create table Iris_Data (name varchar(100), length int, width int);
/* Model table schema */
create table my_iris_model (model varbinary(max));
declare @iris_model varbinary(max) = (select model from
my_iris_model);
exec sp_execute_external_script
@language = 'R'
, @script = '
IrisPredict <- function(data, model){
library(e1071)
predicted_species <- predict(model, data)
return(predicted_species)
}
IrisPredict(input_data_1, model);
'
, @parallel = default
, @input_data_1 = N'select * from Iris_Data'
, @params = N'@model varbinary(max)'
, @model = @iris_model
with result sets ((name varchar(100), length int, width int
, species varchar(30)));
Values highlighted in yellow are SQL queries embedded in the original R script
Values highlighted in aqua are R variables that bind to SQL variables by name
Advanced analytics
70. Capability
Extensible in-database analytics, integrated with R,
exposed through T-SQL
Centralized enterprise library for analytic models
Benefits
SQL Server
Analytical engines
Integrate with R
Become fully extensible
Data management layer
Relational data
Use T-SQL interface
Stream data in-memory
Analytics library
Share and collaborate
Manage and deploy
R +
Data Scientists
Business
Analysts
Publish algorithms, interact
directly with data
Analyze through T-SQL,
tools, and vetted algorithms
DBAs
Manage storage and
analytics together
Summary: R integration and advanced analytics
Advanced analytics
72. Ever-growing data, ever-shrinking IT
What to do?
Expand server and storage
Move data elsewhere
Delete
Massive tables (hundreds of
millions/billions of rows, TBs size)
Users want/need to retain data
indefinitely
Cold data infrequently accessed
but must be online
Datacenter consolidation
Maintenance challenges
Business SLAs at risk
Hybrid solutions
73. Capability
Stretch large operational tables
from on-premises to Azure with
the ability to query
Benefits
Stretch SQL Server into Azure
Securely stretch cold tables to Azure with remote query processing
SQL
SERVER
2016
Azure
Hybrid solutions
74. Stretch Database architecture
How it works
Creates a secure linked server definition
in the on-premises SQL Server
Targets remote endpoint with linked
server definition
Provisions remote resources and begins
to migrate eligible data, if migration
is enabled
Queries against tables run against both
local database and remote endpoint
Remote
endpoint
Remote
data
Azure
Internetboundary
Local
database
Local
data
Eligible
data
Linked servers
Hybrid solutions
75. Queries continue working
Business applications continue
working without disruption
DBA scripts and tools work as
before (all controls still held in local
SQL Server)
Developers continue building or
enhancing applications with existing
tools and methods
Hybrid solutions
76. Advanced security features supported
Data in motion always via secure
channels (TLS 1.1/1.2)
Always Encrypted supported if
enabled by user (encryption key
remains on-premises)
Row-Level Security and Auditing
supported
Hybrid solutions
77. Backup and restore benefits
Hybrid solutions
DBAs only back up/restore local
SQL Server hot data
StretchDB ensures remote data is
transactionally consistent with local
Upon completion of local restore,
SQL Server reconciles with remote
using metadata operation, not
data copy
Time of restore for remote not
dependent on size of data
78. Order history
Name SSN Date
Jane Doe cm61ba906fd 2/28/2005
Jim Gray ox7ff654ae6d 3/18/2005
John Smith i2y36cg776rg 4/10/2005
Bill Brown nx290pldo90l 4/27/2005
Sue Daniels ypo85ba616rj 5/12/2005
Sarah Jones bns51ra806fd 5/22/2005
Jake Marks mci12hh906fj 6/07/2005
Eric Mears utb76b916gi 6/18/2014
Rachel Hogan px61hi9306fj 7/1/2014
Sam Johnson ol43bi506gd 7/12/2014
David Simon tx83hal916fi 7/29/2014
Order history
Name SSN Date
Jane Doe cm61ba906fd 2/28/2005
Jim Gray ox7ff654ae6d 3/18/2005
John Smith i2y36cg776rg 4/10/2005
Bill Brown nx290pldo90l 4/27/2005
Customer data
Product data
Order History
Stretch to cloud
App
Query
Microsoft
Azure
Jim Gray ox7ff654ae6d 3/18/2005
Hybrid solutions
Notas del editor
Speaker notes:
This slide shows the evolution of SQL server overtime. Today SQL Server is leader in Database Management System according to Gartner.
Microsoft has made tremendous investments in the platform, in order to make SQL Server 2016 the fastest release ever produced. SQL Server 2016 has been in production in Microsoft Cloud through Azure SQL Database for more than 1 year and now is available to customers and partners to run on-premises or in Service Providers’ datacenter.
Source: https://msdn.microsoft.com/en-us/library/dn269114(v=sql.130).aspx
In SQL Server 2016 you can perform ALTER operations on memory-optimized tables by using the ALTER TABLE statement. The database application can continue to run, and any operation that is accessing the table is blocked until the alteration process is completed.
Source: https://msdn.microsoft.com/en-us/library/dn957475(v=sql.130).aspx
In SQL Server 2016 , you can perform ALTER operations on natively compiled stored procedures using the ALTER PROCEDURE statement.
In the previous release of SQL Server, you had to complete manual steps to modify natively compiled stored procedures.
When executing ALTER PROCEDURE on a natively compiled stored procedure, the procedure is recompiled using a new definition. While recompilation is in progress, the old version of the procedure continues to be available for execution. Once compilation completes, procedure executions are drained, and the new version of the procedure is installed.When you alter a natively compiled stored procedure, you can modify the following options.
Parameter
EXECUTE AS
TRANSACTION ISOLATION LEVEL
LANGUAGE
DATEFIRST
DATEFORMAT
DELAYED_DURABILITY
For more information on ALTER PROCEDURE functionality and syntax, see ALTER PROCEDURE (Transact-SQL)
You can execute sp_recompile on a natively compiled stored procedure, which causes the stored procedure to recompile on the next execution.
https://msdn.microsoft.com/en-us/library/dn935012.aspx
The sp_recompile (Transact-SQL)stored procedure can be used with the natively compiled, scalar user-defined function. It will result in the function being recompiled using the definition that exists in metadata.
Support for using temporal system-versioning with In-Memory OLTP. For more information, see https://msdn.microsoft.com/en-us/library/mt590207.aspx
Query store support for natively compiled code from In-Memory OLTP workloads. For more information, see Using the Query Store with In-Memory OLTP https://msdn.microsoft.com/en-us/library/mt590480.aspx
Row-Level Security in Memory-Optimized Tables https://msdn.microsoft.com/en-us/library/dn511014.aspx#rls
Source: https://msdn.microsoft.com/en-us/library/dn688968(v=sql.130).aspx
Encryption
In SQL Server 2016, the storage for memory-optimized tables will be encrypted as part of enabling TDE on the database. For more information, see Transparent Data Encryption (TDE).
Note that in SQL Server 2014 you needed to set up a Management Data Warehouse (MDW) in order to run the In-Memory OLTP reports. SQL Server 2016 now includes lightweight reports available in SSMS without setting up MDW.
Open SSMS and right click on the AdventureWorks2016 database and select Reports | Standard Reports | Transaction Performance Analysis Overview.
Select the Tables Analysis report link.
This presents you with a prioritized plot chart that graphs the amounts of potential gain against the potential amount of work to migrate the top tables to In-Memory OLTP. This analysis is based upon the workload that we just ran with the batch file. This helps you see the highest potential gain with the lowest potential amount of migration work required.
Select the SalesOrderDetail_ondisk to explore further.
Note that this report contains information on latch statistics, lock statistics, recommended In-Memory index type, and more.
Click the Navigate Backwards button twice in the upper left of the report bar (see illustration above) to go back to the main Report and click on Stored Procedure Analysis.
Source: https://msdn.microsoft.com/en-us/library/dn935015(v=sql.130).aspx
A temporal table is a new type of table that provides correct information about stored facts at any point in time. Each temporal table consists of two tables actually, one for the current data and one for the historical data. The system automatically ensures that when the data changes in the table with the current data, the previous values are stored in the historical table. Querying constructs are provided to hide this complexity from users. For more information, see Temporal Tables.
Introduction to Key Components and Concepts
What is a Temporal Table?
A temporal table is a table for which a PERIOD definition exists and which contains system columns with a datatype of datetime2 into which the period of validity is recorded by the system, and which has an associated history table into which the system records all prior versions of each record with their period of validity. With a temporal table, the value of each record at any point in time can be determined, rather than just the current value of each record. A temporal table is also referred to as a system-versioned table.
Why Temporal?
Real data sources are dynamic and more often than not business decisions rely on insights that analysts can get from data evolution. Use cases for temporal tables include:
Understanding business trends over time
Tracking data changes over time
Auditing all changes to data
Maintaining a slowly changing dimension for decision support applications
Recovering from accidental data changes and application errors
Source: https://msdn.microsoft.com/en-us/library/dn935015(v=sql.130).aspx
How Does Temporal Work?
System-versioning for a table is implemented as a pair of tables, a current table and a history table.
Within each of these tables, two additional datetime (datetime2 datatype) columns are used to define the period of validity for each record – a system start time (SysStartTime) column and a system end time (SysEndTime) column.
The current table contains the current value for each record. The history table contains the each previous value for each record, if any, and the start time and end time for the period for which it was valid.
INSERTS: On an INSERT, the system sets the value for the SysStartTime column to the UTC time of the current transaction based on the system clock and assigns the value for the SysEndTime column to the maximum value of 9999-12-31 – this marks the record as open.
UPDATES: On an UPDATE, the system stores the previous value of the record in the history table and sets the value for the SysEndTime column to the UTC time of the current transaction based on the system clock. This marks the record as closed, with a period recorded for which the record was valid. In the current table, the record is updated with its new value and the system sets the value for the SysStartTime column to the UTC time for the transaction based on the system clock. The value for the updated record in the current table for the SysEndTime column remains the maximum value of 9999-12-31.
DELETES: On a DELETE, the system stores the previous value of the record in the history table and sets the value for the SysEndTime column to the UTC time of the current transaction based on the system clock. This marks this record as closed, with a period recorded for which the previous record was valid. In the current table, the record is removed. Queries of the current table will not return this value. Only queries that deal with history data return data for which a record is closed.
MERGE: On a MERGE, MERGE behaves as an INSERT, an UPDATE, or a DELETE based on the condition for each record.
Source: https://msdn.microsoft.com/en-us/library/dn935015(v=sql.130).aspx
The SYSTEM_TIME period columns used to record the SysStartTime and SysEndTime values must be defined with a datatype of datetime2.
Source: https://msdn.microsoft.com/en-us/library/dn935015(v=sql.130).aspx
The SYSTEM_TIME period columns used to record the SysStartTime and SysEndTime values must be defined with a datatype of datetime2.
Source: https://msdn.microsoft.com/en-us/library/mt163865(v=sql.130).aspx
Selecting Deterministic or Randomized Encryption
Always Encrypted supports two types of encryption: randomized encryption and deterministic encryption:
Deterministic encryption uses a method that always generates the same encrypted value for any given plaintext value. Using deterministic encryption allows grouping, filtering by equality, and joining tables based on encrypted values, but can also allow unauthorized users to guess information about encrypted values by examining patterns in the encrypted column. This weakness is increased when there is a small set of possible encrypted values, such as True/False, or North/South/East/West region. Deterministic encryption must use a column collation with a binary2 sort order for character columns.
Randomized encryption uses a method that encrypts data in a less predictable manner. Randomized encryption is more secure, but prevents equality searches, grouping, indexing, and joining on encrypted columns.
Use deterministic encryption for columns that will be used as search or grouping parameters, such as a government ID number. Use randomized encryption for data such as confidential investigation comments, which are not grouped with other records, or for data used to join tables (in the row that contains the encrypted column of interest).
Source: https://msdn.microsoft.com/en-us/library/mt130841(v=sql.130).aspx
Dynamic Data Masking limits sensitive data exposure by masking it to non-privileged users. It helps prevent unauthorized access to sensitive data by enabling customers to designate how much sensitive data to reveal with minimal impact on the application layer. A policy-based security feature, it hides sensitive data in the result set of a query over designated database fields, while the data in the database is not changed. Dynamic Data Masking is easy to use with existing applications, since masking rules are applied in the query results, and there is no need to modify existing queries.
For example, a call center support person may identify callers by several digits of their Social Security Number or credit card number, but those data items should not be fully exposed to the support person. A developer can define a masking rule to be applied to each query result that masks all but the last four digits of any Social Security Number or credit card number in the result set. For another example, by using the appropriate data mask to protect personally identifiable information (PII) data, a developer can query production environments for troubleshooting purposes without violating compliance regulations.
Dynamic Data Masking limits the exposure of sensitive data and prevents accidental viewing by engineers that access directly databases for troubleshooting purposes or non-privileged application users. It doesn’t aim to prevent privileged database users from connecting directly to the database and running exhaustive queries that expose pieces of the sensitive data. Dynamic Data Masking complements other SQL Server security features (auditing, encryption, Row-Level Security…), and it is highly recommended to enable them in addition in order to protect better sensitive data in the database.
Since data is masked just before being returned to the user, changing the data type to an unmasked type will return unmasked data.
Dynamic Data Masking is available in SQL Server 2016. However, to enable it, you must use trace flags 209 and 219. For SQL Database, see Get started with SQL Database Dynamic Data Masking (Azure Preview portal).
Source: https://msdn.microsoft.com/en-us/library/bb510411(v=sql.130).aspx#RLS
Row-Level Security introduces predicate-based access control. It features a flexible, centralized, predicate-based evaluation that can take into consideration metadata (such as labels) or any other criteria the administrator determines as appropriate. The predicate is used to determine whether the user has the appropriate access to the data based on user attributes. Label-based access control can be implemented by using predicate-based access control. For more information, see Row-Level Security.
Policy manager creates a filter predicate and security policy
App user (e.g., a nurse) selects from the Patients table
The query is transparently rewritten to apply the filter predicate
Notice: No app changes!
Source: https://msdn.microsoft.com/en-us/library/dn765131(v=sql.130).aspx
Row-level filtering of data selected from a table is enacted through a security predicate filter defined as an inline table valued function. The function is then invoked and enforced by a security policy. The policy can restrict the rows that may be viewed (a filter predicate), but does not restrict the rows that can be inserted or updated from a table (a blocking predicate). There is no indication to the application that rows have been filtered from the result set; if all rows are filtered, then a null set will be returned.
Filter predicates are applied while reading data from the base table, and it affects all get operations: SELECT, DELETE (user cannot delete rows that are filtered), and UPDATE (user cannot update rows that are filtered, although it is possible to update rows in such way that they will be subsequently filtered). Blocking predicates are not available in this version of RLS, but equivalent functionality (user cannot INSERT or UPDATE rows such that they will subsequently be filtered) can be implemented using check constraints or triggers.
Filter predicates and security policies have the following behavior:
Define a security policy that filters the rows of a table. The application is unaware that any rows have been filtered for SELECT, UPDATE, and DELETE operations, including situations where all the rows have been filtered out. The application can INSERT any rows, regardless of whether they will be filtered during any other operation.Define a predicate function that joins with another table and/or invokes a function. The join/function is accessible from the query and works as expected without any additional permission checks.
Issue a query against a table that has a security predicate defined but disabled. Any rows that would have been filtered or restricted are not affected.
The dbo user, a member of the db_owner role, or the table owner queries against a table that has a security policy defined and enabled. Rows are filtered/restricted as defined by the security policy.
Attempts to alter the schema of a table bound by a security policy will result in an error. However, columns not referenced by the filter predicate can be altered.
Attempts to add a predicate on a table that already has one defined (regardless of whether it is enabled or disabled) results in an error.
Attempts to modify a function used as a predicate on a table within a security policy results in an error.
Defining multiple active security policies that contain non-overlapping predicates, succeeds.
Source: https://msdn.microsoft.com/en-us/library/azure/dn765131.aspx
Auditing an instance of the SQL Server Database Engine or an individual database involves tracking and logging events that occur on the Database Engine. SQL Server audit lets you create server audits, which can contain server audit specifications for server-level events, and database audit specifications for database-level events. Audited events can be written to the event logs or to audit files. All editions of SQL Server support server-level audits. Database-level auditing is limited to the Enterprise, Developer, and Evaluation editions.
Azure Key Vault offers central key management, leverages hardware security modules (HSMs), and promotes the separation of key management from the management of data to help meet regulatory compliances.
SQL Server’s support for AKV is available through the SQL Server Connector for AKV for all enterprise versions of SQL Server starting with 2012 through the 2016 release. The Connector is a downloadable DLL file that serves as an Extensible Key Management (EKM) provider for SQL Server.
The Connector is especially important to customers using SQL Server-in-a-VM who want to leverage Azure Key Vault for managing their encryption keys. SQL Server-in-a-VM allows for quick deployment of SQL Server and is ideally suited for re-hosting existing SQL Server applications in the cloud or for extending portions of their database management operations into the cloud.
With Azure Key Vault integration, both on-premises and SQL Server-in-a-VM users can assume control of encryption keys for Transparent Data Encryption (TDE), Column Level Encryption (CLE), and backup encryption while leveraging the additional security benefits of Azure Key Vault.
Transparent Data Encryption (TDE) in SQL Server, exclusive to SQL Server Enterprise edition
SQL Server encrypts data while creating a backup. By specifying the encryption algorithm and encryptor (a certificate or asymmetric key) when backing up, you can create an encrypted backup file. On-premises and Windows Azure storage locations are supported for this process. In addition, encryption options can be configured for SQL Server Managed Backup to Windows Azure operations, a new feature introduced in SQL Server 2014. To encrypt during backup, you must specify an encryption algorithm, and an encryptor to secure the encryption key.
The following encryption options are supported:
Encryption algorithm – The supported encryption algorithms are AES 128, AES 192, and AES 256
Encryptor – A certificate or asymmetric key
Backup encryption – Now supported with compression and automatically will use Intel AES-NI hardware acceleration
When it comes to key BI investments, we are making it much easier to manage relational and non-relational data. PolyBase technology allows you to query Hadoop data and SQL Server relational data through a single T-SQL query.
One of the challenges we see with Hadoop is there are not enough people knowledgeable in Hadoop and MapReduce, and this technology simplifies the skill set needed to manage Hadoop data. This can also work across your on-premises environment or SQL Server running in Azure.
Key Points:
SQL Server Analysis Services Themes
Talk Track:
Make it easier to create models
Improve tabular modeling and performance
Use Analysis Services as a semantic model only
High-performance DirectQuery for tabular
Strengthen multidimensional
Performance and data source updates
Technical talking points:
DirectQuery and tabular models
Loading and analyzing data
New analytic functions
Data modeling relationships
Updated developer tools
MOLAP performance improvements
Modeling
DirectQuery
Performance
Dev Ops improvements
Key Points:
SQL Server Reporting Services offers a single, on-premises solution for deploying and managing reports – both traditional, paginated reports and responsive mobile reports.
Talk track:
Report Design
Report Builder enables report authors to design paginated reports and publish them to a report server.
Mobile Report Publisher enables report authors to design mobile reports and publish them to a report server.
SQL Server Data Tools (SSDT) integrates into Visual Studio 2015 and enables developers to design paginated reports within the Visual Studio integrated development environment and take advantage of projects, source control, build, and deployment.
Report Viewing
Modern browsers (Microsoft Edge, Microsoft Internet Explorer, Google Chrome, Mozilla Firefox, Apple Safari) provide users with access to their reports across platforms and devices.
Power BI Mobile apps are available for all major mobile platforms – Windows, iOS, and Android – and enable users to view KPIs and mobile reports on their phones and tablets.
Custom Apps
Third-party and custom apps can embed reports (both paginated and mobile) using HTML5 technology.
ASP.NET web apps can embed the Report Viewer control for paginated report viewing.
Apps can further integrate with Reporting Services using APIs and web services.
Report Server
A brand-new web portal built on HTML5 enables users to browse, search, view, and manage reports (both paginated and mobile) using modern browsers.
The same web APIs that serve the web portal, Power BI Mobile apps, and report design apps also enable third-party and custom apps to integrate with Reporting Services and report server content.
An enterprise-scale processing and rendering engine connects to and queries various types of data sources; processes data (filtering, sorting, grouping, aggregation); evaluates expressions and custom code; and renders paginated reports to various formats.
A scheduling and delivery agent refreshes datasets and executes reports on a schedule, and delivers paginated reports to users via email and other means.
The report server database, built on the SQL Server Database Engine, stores and manages the catalog of content, including data sources, datasets, paginated reports, mobile reports, and KPIs. The database can be on the report server or on a different server running SQL Server.
Multiple report servers can operate in a scale-out configuration for additional scale and availability.
Data Sources
Reporting Services provides native data source connectors for SQL Server, SQL Server Analysis Services, Oracle Database, Oracle Essbase, SAP BW, Teradata, SharePoint lists, and XML.
It also provides generic connectors based on standard ODBC and OLE DB protocols for access to many more data source types.
Developers can create custom data processing extensions for connectivity to other, proprietary data sources.
Report Delivery Destinations
Reporting Services can deliver paginated reports on a schedule to email, file shares, and SharePoint libraries.
Power BI users can pin items from Reporting Services paginated reports (charts, gauges, maps, images) to their Power BI dashboards. The report server updates their dashboard tiles on a schedule so the data is up-to-date.
Developers can create custom delivery extensions to deliver reports to other destinations.
Key points: Key investments areas in SQL Server 2016 BI capabilities.
Talk track:
Both SQL Server Reporting Services and Analysis Services receive major enhancements and deliver modern and mobile reporting and powerful modeling capabilities.
Let’s take a look at what’s new in Reporting Services.
Key points:
The Microsoft SQL Server Mobile Report Publisher is the single point for designing and publishing of mobile reports.
Talk track:
With the Mobile Report Publisher you can connect to the SQL Server Reporting Services to access enterprise SQL server data sources.
When providing an optimal experience on mobile devices, you need the ability to design different type of layouts: a responsive layout that adapts to very different screen sizes and orientations. With Report Publisher you can easily create beautiful reports for master, tablet and phone view.
Connect visualizations to data sources and then publish them to all devices for access by other people in the organization.
The Mobile Report Publisher Preview for Windows 7+ is available from the Microsoft Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=50400)
Key points:
Mobile reports dynamically adjust the content to fit your screen (or browser window). You never see content scrolled out of view or flowing onto a different page. While you can view these reports on your PC as well, they're really optimized for mobile devices.:
Talk track:
You only need one mobile app to access Power BI and SSRS mobile reports. Power BI apps are available for all major mobile platforms.
Connect to your on-premises SSRS server and view your KPIs and mobile reports
In addition to access via mobile devices you can also access your data via HTML rendering from your browser.
The Power BI app to view Reporting Services mobile reports and KPIs is available in preview here: https://itunes.apple.com/us/app/microsoft-power-bi/id929738808?mt=8
Take out in initial version of NDA roadmap, until fully committed.
Source: https://msdn.microsoft.com/en-us/library/dn935011(v=sql.130).aspx
Stretch Database lets you archive your historical data transparently and securely. In SQL Server 2016, Stretch Database stores your historical data in the Microsoft Azure cloud. After you enable Stretch Database, it silently migrates your historical data to an SQL Database.
You don't have to change existing queries and client apps. You continue to have seamless access to both local and remote data.
Your local queries and database operations against current data typically run faster.
You typically enjoy reduced cost and complexity.
Source: https://msdn.microsoft.com/en-us/library/mt169378(v=sql.130).aspx
Concepts and architecture for Stretch Database
Terms
Local database. The on-premises SQL Server 2016 database.
Remote endpoint. The location in Microsoft Azure that contains the database’s remote data. In SQL Server 2016, this is an Azure SQL Database server. This is subject to change in the future.
Local data. Data in a database with Stretch Database enabled that will not be moved to Azure based on the Stretch Database configuration of the tables in the database.
Eligible data. Data in a database with Stretch Database enabled that has not yet been moved, but will be moved to Azure based on the Stretch Database configuration of the tables in the database.
Remote data. Data in a database with Stretch Database enabled that has already been moved to Azure.
Architecture
Stretch Database leverages the resources in Microsoft Azure to offload archival data storage and query processing.
When you enable Stretch Database on a database, it creates a secure linked server definition in the on-premises SQL Server. This linked server definition has the remote endpoint as the target. When you enable Stretch Database on a table in the database, it provisions remote resources and begins to migrate eligible data, if migration is enabled.
Queries against tables with Stretch Database enabled automatically to run against both the local database and the remote endpoint. Stretch Database leverages processing power in Azure to run queries against remote data by rewriting the query. You can see this rewriting as a "remote query" operator in the new query plan.