2. Bienvenida
• Ing. Eduardo Castro, PhD
• MVP de Microsoft SQL Server
• PASS Regional de Mentor
• PASS Board of Directors
• http://www.youtube.com/eduardocastrom
7. SQL Server en Linux: ¿Cómo?
• Mismo código de alta calidad y normas de seguridad
• El mismo valor en términos de funcionalidad y escalabilidad
• Compatibilidad de aplicaciones entre SQL Server en Windows y Linux
• Las características y correcciones deben aparecer inmediatamente a través de las
plataformas
• Establecer una base para otros servicios de SQL Server
8. SQL Server en Linux: ¿Cómo?
• Migración a Linux puro
• Reescribir 70 + millones de líneas de código C++
• Una base de código que cambia constantemente
• Tarea de plomería tediosa de varios años.
• No hay garantía de éxito o rendimiento
9. SQL Server en Linux: ¿Cómo?
• Encontrar una forma de reutilizar la mayoría del código
SQLOS Drawbridge
10. Drawbridge
10
• Las máquinas virtuales tienen recursos generales de recursos
grandes en términos de espacio de disco, memoria, CPU y costos
administrativos.
• Drawbridge combina dos ideas de la literatura, el picoproceso y
la biblioteca OS , para proporcionar una nueva forma de
computación, que conserva los beneficios del aislamiento seguro,
la compatibilidad persistente y la continuidad de la ejecución,
pero con recursos de recursos drásticamente más bajos.
11. Drawbridge
11
• El picoproceso es un contenedor de aislamiento
ligero y seguro.
• Se construye desde un espacio de direcciones de
proceso de SO, pero con todos los servicios de SO
tradicionales eliminados.
• La interfaz binaria de aplicación (ABI) entre el código
que se ejecuta en el picoproceso y el sistema
operativo sigue los patrones de diseño de las
máquinas virtuales de hardware;
• Consiste en un conjunto cerrado de 45 downcalls
con semántica fija que proporcionan una interfaz sin
estado.
• Todas las llamadas ABI son atendidas por el monitor
de seguridad, que desempeña un papel similar al
hipervisor o monitor VM en diseños de hardware VM
tradicionales.
12. Biblioteca de sistema operativo
12
• Una biblioteca de sistema operativo es un sistema operativo
refactorado para ejecutarse como un conjunto de bibliotecas
dentro del contexto de una aplicación.
• Mientras Drawbridge puede ejecutar muchos sistemas operativos
posibles, una contribución clave de Drawbridge es una versión de
Windows que ha sido adaptada para ejecutarse dentro de un solo
picoproceso de Drawbridge.
14. LibOS
14
• Como se mencionó anteriormente, Drawbridge contiene una
implementación de las llamadas de sistema de Windows y las
APIs necesarias en Windows.
• Por lo tanto Drawbridge implementada un tipo de versión de
espacio de usuario del kernel de NT (conocido como NTUM -
"modo de usuario NT").
• Además, LibOS cuenta con una biblioteca que representa a las
API de Windows en el espacio de usuario.
• En comparación con las implementaciones reales en un entorno
normal de Windows, ambos elementos se mantienen mucho más
delgados, pero resultan en un entorno de tiempo de ejecución
que no se comporta muy diferente a la implementación
convencional de Windows.
15. SQLPAL
15
• En combinación con Drawbridge y SQLOS permiten que
servidores MSSQL se ejecuten en Linux.
• El equipo de Microsoft decidió extraer las partes pertinentes de
Drawbridge, LibOS y la capa de abstracción de SQLOS para
construir una nueva capa de abstracción de plataforma (PAL).
• Éste sólo contiene las ABI relevantes para SQL Server todo lo que
se requiere para proporcionar un entorno de tiempo de ejecución
para MSSQL.
• La capa inferior consiste en un binario de Linux (en el espacio de
usuario).
19. ¿Qué es SQL Server en Linux?
• Es sólo SQL Server. Simplemente funciona
• Inicialmente dirigido al motor de base de datos
• Actualmente alrededor del 95% de las características son
soportadas.
20. Plataformas y características soportadas
• Plataformas soportadas
• Ubuntu 16.10, Ubuntu 16.04 LTS
• Red Hat Enterprise Linux 7.3 Workstation, Server, and Desktop and earlier
• SUSE Enterprise Linux Server v12 SP2
• Docker Engine 1.8 and later
• Características
• Core Database Engine
• Including CLR, In-Memory OLTP, Columnstore Index, Query Store, DMV, Security and
Encryption features
• Herramientas
• Adminsitración: SQLCMD, SSMS, mssql-conf, Powershell
• Desarrollo: SSMS, Visual Studio Code, SSDT, Connectivity libraries (C#, Java, PHP, Node.js,
Python, Ruby, C++)
21. Características no compatibles (CTP 2.0)
Database Engine High Availability Security Services
Replication Database Mirroring Active Directory
Authentication
SQL Server Browser
FileTable Windows Authentication SQL Server R Services
Stretch DB Extensible Key
Management
SQL Server Python Services
Polybase Use of user-provided
certificate for SSL or TLS
StreamInsight
Distributed Query Analysis Services
CLR assemblies with the
EXTERNAL_ACCESS or
UNSAFE permission set
Reporting Services
System extended stored
procedures (XP_CMDSHELL,
etc.)
Integration Services
Data Quality Services
Master Data Services
22. Instalar SQL Server en Linux
• Ningún medio de instalación o programa por separado
• Instalación de software utiliza el sistema de gestión de paquetes
de Linux
23. Active Directory Authentication con SQL Server on Linux
23
# RHEL
sudo yum install realmd krb5-workstation
# SUSE
sudo zypper install realmd krb5-client
# Ubuntu
sudo apt-get install realmd krb5-user software-properties-common python-software-
properties packagekit
sudo realm discover contoso.com –v
sudo realm join contoso.com -U 'user@CONTOSO.COM' -v
<...>
* Successfully enrolled machine in realm
27. Alta disponibilidad en Linux
• Clustering
• Linux a Linux
• Grupos de alta disponibilidad
• Puede ser Linux a Linux
• Puede ser Windows a Linux
• Backup y restore
• Las bases de datos se pueden respaldar en Windows y restaurar en Linux
• Las bases de datos pueden ser respaldadas en Linux y restauradas en Windows
• Log Shipping
• Las bases de datos se pueden hacer respaldo en Windows y restaurar en Linux
28. SQL Server en Linux HA Cluster
• Failover Cluster Instance on RHEL 7.3
• Basado en Red Hat Enterprise Linux (RHEL) HA add-on
• Utiliza Corosync y Pacemaker
29. SQL Server en Linux HA
• Mission-critical HADR con Always On Availability Groups
• Ejecución de aplicaciones de misión criítca en SQL Server Linux
30. SQL Server en Linux HA
• HADR de misión crítica con siempre en grupos de disponibilidad
• Balanceo de carga de trabajo para aumentar la escalabilidad y el rendimiento
31. SQL Server en Linux HA
• HADR de misión crítica con Always On Availability Groups
• Migración sin problemas entre plataformas
Version 1.0 of SQL Server Released 1989 on OS/2
Version 4.21a in 1993 for Windows NT
Current Version v14
Output of Query run against various Linux and Windows Instances
2 Virtual Linux Boxes
2 Azure Linux Instances
2 Docker Images
1 Windows Instance on Windows 2016
SQL Server RDBMS and other services that ship with it in the SQL Server product suite account for more than 40 million lines of C++ code.
If you include all of products (SSIS, SSRS etc) then have mort than 70 million lines of C++ code.
Pure port could take years just to get compiling and booting let alone figuring out things like performance and feature parity with SQL Server on Windows
SQL Server innovation is happening in the same codebase and would have been a daunting task and keep the team in a close to endless catch-up game.
Getting to run only fraction of work need to deliver Product Also need to account for Lab Automation, Testing, Packaging, Servicing, Security, High Availability, Support Tools & Third Party Tools etc
SQLOS – Created For SQL Server 2005
- Purpose to Abstract the operating system from the Rest of SQL Server Engine
- Provides user-level operating system services to rest of Server
- Built to Take advantage of Win32 Subsystem. Substituted some services but Win32 calls blead through entire engine
If just used SQLOS need to either bring Win32 awareness deep inside or re-platform all existing Win32 calls across full code base
Drawbridge – Microsoft Research project from 2011
- Initially designed as a Container Technology
- Enables ntoskrnl to run in User Mode – Called LibOS (Library OS)
- Provides the Win32 Subsystem right inside your Process
Slave Oks - Responsible for Initial Design of SQLOS
- Spent 2017 to 2014 working on MIDORI operating system
- While developing Midori utilised Drawbridge to provide Windows Programs inside Midori
- Developed Concept of combining SQLOS with Drawbridge to create SQLOSV2 or SQLPAL
- Joined Project Helsinki in early 2005
- Produced a “running version” of SQL Server on Linux on 28 February 2005
LibOS
As mentioned earlier, Drawbridge contains a complete user space implementation of the Windows system calls and the APIs needed in Windows. So Drawbridge implemented a type of user space version of the NT kernel (known as NTUM - "user mode NT"). Plus, LibOS has a library which represents the Windows APIs in the user space. Compared to the actual implementations in a normal Windows environment, both elements are kept much leaner, but they result in a run-time environment which does not behave any differently than a conventional Windows one.
SQL PAL
In combination, Drawbridge and SQLOS allow for MSSQL servers to be run in Linux. But in their implementation, they go way too far in some respects and partly solve the same problems. For this reason, the team at Microsoft decided to extract the relevant parts and use Drawbridge, LibOS and the SQLOS abstraction layer to construct a new platform abstraction layer (PAL). This one only contains the ABI and APIs relevant for SQL servers and features everything that's required in order to provide a run-time environment for MSSQL. The bottom layer consists of a Linux binary (in the user space) which initiates the rest of the environment.
To-do list for SOS and LibOS
The current status is that MSSQL is already running smoothly in Linux, but the Microsoft team is not yet finished consolidating SOS and LibOS. So for the time being, there are two versions of SOS / LibOS within the SQL server for Linux. The "upper" version forwards calls to SQL PAL, which then uses the newer version of SOS to send queries to the Linux kernel via a host bridge. Microsoft is currently working intensively on eliminating the need for these two versions and harmonising these two instances so that, in the end, the entire abstraction can be performed in SQL PAL.
The final architecture at a glance
As already mentioned, Drawbridge and SQLOS have a few overlaps which Microsoft is still working on eliminating. Overall, SQL PAL comprises the run-time environment for MSSQL and the RPM/DEB packet brings all executables, as well the necessary additional Windows libraries along with it in order to be able to run MSSQL, based on SQL PAL. According to Microsoft, around 81MB uncompressed Windows libraries will be included in the supply (that's just about 1% of a full Windows installation). The SQL PAL binary is about 8MB.
The software is installed in /opt and is managed with systemd. Microsoft supplies everything ready-to-use and with man pages included.
Generally, the process for starting MSSQL on Linux can be described as follows: the host component (standard Linux binary) first starts SQL PAL, which then initiates the MSSQL server in the "emulated" Windows environment.
The final architecture looks about like this:
SOSv2
SOS and LibOS merged
81 MB of uncompressed Windows libraries, SQL PAL itself ~8MB
The sqlserver.exe, sqlmin.dll, … are the same, exact, PE binaries we ship for Windows (our traditional box product), the first question is always: “How can the Linux kernel understand a PE image?”
ELF image format is the image format know to the Linux kernel.
PE image format is the image format known the Windows kernel.
Microsoft SQL Server runs on Linux through a containerised approach called Drawbridge
Drawbridge implements a Linux loader and a minimal set of ABI calls to allow an in-process NT user mode kernel to run
SQL Server runs on top of a SQL platform layer (called SQL OS) that could be ported to run on Drawbridge
SQL Server had supportability commands added to allow the state of the system to be measured with SQL calls
A number of efficiency gains were applied to both the Drawbridge components and the SQL Server code to bring performance to within 20% of the equivalent process running on Windows
Overhead for transition between PE and ELF < 100 instructions
This is just Engineering needed to get running on SQL Server
All of supporting tools, documentation needs to be updated
600,000 + Unit Tests needed to be modified
Support Engineers need training
Debuging Tools needed to be upgraded – DBGBridge (1 hour presentation and Blog Posts from Bob Dorr)
March 7th 2016 announced plans to bring SQL Server to Linux.
Private Preview with > 20,000
We are bringing the core relational database capabilities to preview today, and are targeting availability in mid-2017.
And despite the fact that it was only announced in spring 2016, you shouldn’t think of SQL Server on Linux as being a subset of the full Windows offering, says Rohan Kumar, general manager of the Microsoft database systems group.
“It’s the same SQL Server product,” he tells CIO.com. “We have one SQL Server product, and Windows and Linux are your deployment choices.”
Some features fall into Category of Partially Supported
SQL Agent – Partial Support (CMD shell support)
Some features planned but not yet available
FullText Search
Some features will never be supported
FileTable – depends on Windows
It is just SQL Server
Includes 2016 SP1 Improvements
TRIM
CONCAT_WS
STRING_AGG
TRANSLATE
sys.dm_exec_query_statistics_xml (with TF 7412)
Dedicated Administrator connection is not enabled
Cannot start in Single-User mode (Can’t restore
SQL Server Database Engine (CTP 1.3)
Indirect checkpoint performance improvements.
Cluster-less Availability Groups support added.
Minimum Replica Commit Availability Groups setting added.
Availability Groups can now work across Windows-Linux to enable cross-OS migrations and testing.
Temporal Tables Retention Policy support added,
New DMV SYS.DM_DB_STATS_HISTOGRAM
Online non-clustered columnstore index buill and rebuild support added
5 new dynamic management views to return information about Linux process. For more information, see Linux Process Dynamic
In Linux distributions, a package refers to a compressed file archive containing all of the files that come with a particular application. The files are usually stored in the package according to their relative installation paths on your system. Most packages also contain installation instructions for the OS, as well as a list of any other packages that are dependencies, prerequisites required for installation.
Common types of Linux packages include .deb, .rpm, and .tgz. Since Linux packages do not usually contain the dependencies necessary to install them, many Linux distributions use package managers that automatically read dependencies files and download the packages needed before proceeding with the installation. Some examples of package managers are APT, YUM, and zipper
Packages are downloaded into the local repository and then installed using the Package Manager.
Full instructions at :https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup
No Internet then manually download from : http://packages.microsoft.com/
Packages for Ubuntu, RHEL and SLES
Can also install Linux version of:
SQLCMD
BCP
Similar process to installing Server
https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-tools
You can use this utility to set the following parameters:+
TCP port: Change the port where SQL Server will listen for connections.
Default data directory: Change the directory where the new SQL Server database data files (.mdf).
Default log directory: Changes the directory where the new SQL Server database log (.ldf) files are created.
Default dump directory: Change the directory where SQL Server will deposit the memory dumps and other troubleshooting files by default.
Default backup directory: Change the directory where SQL Server will send the backup files by default.
Set traceflags: Set the traceflags that the service is going to use.
Set collation: Set a new collation for SQL Server on Linux.
+
Released with CTP 1.3
https://blog.dbi-services.com/introducing-alwayson-availability-groups-on-linux/
Log shipping can be set-up using CronTab and CronJobs
https://dbafromthecold.wordpress.com/2017/01/25/transaction-log-shipping-in-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-with-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-and-multi-subnet-scenarios-with-sql-server-on-linux/?platform=hootsuite
Log shipping can be set-up using CronTab and CronJobs
https://dbafromthecold.wordpress.com/2017/01/25/transaction-log-shipping-in-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-with-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-and-multi-subnet-scenarios-with-sql-server-on-linux/?platform=hootsuite
Log shipping can be set-up using CronTab and CronJobs
https://dbafromthecold.wordpress.com/2017/01/25/transaction-log-shipping-in-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-with-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-and-multi-subnet-scenarios-with-sql-server-on-linux/?platform=hootsuite
Log shipping can be set-up using CronTab and CronJobs
https://dbafromthecold.wordpress.com/2017/01/25/transaction-log-shipping-in-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-with-sql-server-on-linux/
http://blog.dbi-services.com/introducing-high-availability-and-multi-subnet-scenarios-with-sql-server-on-linux/?platform=hootsuite