Red Hat Software Collections, OpenShift and the Red Hat Container Development Kit open up many new possibilities for Python developers targeting Red Hat Enterprise Linux. At the same time, the wider Python ecosystem is undergoing two significant transitions - one being the ongoing migration from Python 2 to Python 3, and the other the shift to correctly validating HTTPS connections by default. In this session we will cover the currently available options for developing with Python on Red Hat platforms, as well as provide some insight into where things are headed in the context of the wider Python ecosystem.

1. Developing in Python
on Red Hat platforms
Nick Coghlan
Senior Software Engineer
Graham Dumpleton
Developer Advocate for OpenShift
June 28th
2016

2. Using Python on Red Hat Platforms
● Python for Network Services
● Python for Applications
● Python for System Administration

3. Current Transitions in the Python Ecosystem
● Migration to Python 3
● Modernizing the Python 2.7 Network Security Stack
● Defaulting to HTTPS Certificate Verification

4. Python for Network Services

12. $ oc new-app https://gitlab.com/osevg/python-django-modwsgi.git
--> Found image 772dc19 (4 weeks old) in image stream "python" in project "openshift" under tag "3.4" for "python"
Python 3.4
----------
Platform for building and running Python 3.4 applications
Tags: builder, python, python34, rh-python34
* The source repository appears to match: python
* A source build using source code from https://gitlab.com/osevg/python-django-modwsgi.git will be created
* The resulting image will be pushed to image stream "python-django-modwsgi:latest"
* This image will be deployed in deployment config "python-django-modwsgi"
* Port 8080/tcp will be load balanced by service "python-django-modwsgi"
* Other containers can access this service through the hostname "python-django-modwsgi"
--> Creating resources with label app=python-django-modwsgi ...
imagestream "python-django-modwsgi" created
buildconfig "python-django-modwsgi" created
deploymentconfig "python-django-modwsgi" created
service "python-django-modwsgi" created
--> Success
Build scheduled, use 'oc logs -f bc/python-django-modwsgi' to track its progress.
Run 'oc status' to view your app.
$ oc expose svc/python-django-modwsgi
route "python-django-modwsgi" exposed

13. Source to Image (S2I)
https://github.com/openshift/source-to-image

14. $ s2i build https://gitlab.com/osevg/python-django-modwsgi.git centos/python-34-centos7 python-django-modwsgi
I0610 10:02:07.422470 84805 docker.go:352] Image "centos/python-34-centos7:latest" not available locally, pulling ...
I0610 10:02:36.501637 84805 clone.go:32] Downloading "https://gitlab.com/osevg/python-django-modwsgi.git" ...
I0610 10:02:38.831859 84805 install.go:251] Using "assemble" installed from "image:///usr/libexec/s2i/assemble"
I0610 10:02:38.831913 84805 install.go:251] Using "run" installed from "image:///usr/libexec/s2i/run"
I0610 10:02:38.831943 84805 install.go:251] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts"
I0610 10:02:38.832495 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources
---> Copying application source ...
---> Installing dependencies ...
...
---> Collecting Django static files ...
I0610 10:03:00.876021 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources
$ docker run --rm -p 8080:8080 python-django-modwsgi
---> Running application from Python script (app.py) ...
[Fri Jun 10 00:07:33.580013 2016] [mpm_event:notice] [pid 1:tid 139758878566464] AH00489: Apache/2.4.6 (CentOS)
mod_wsgi/4.5.2 Python/3.4.2 configured -- resuming normal operations
[Fri Jun 10 00:07:33.580149 2016] [core:notice] [pid 1:tid 139758878566464] AH00094: Command line: 'httpd (mod_wsgi-
express) -f /tmp/mod_wsgi-localhost:8080:1001/httpd.conf -D MOD_WSGI_MULTIPROCESS -D
MOD_WSGI_WITH_PROXY_HEADERS -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D
MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D
MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

15. Docker Base Images (S2I Enabled)
● RHEL 7
– Python 2.7 → registry.access.redhat.com/rhscl/python-27-rhel7
– Python 3.3 → registry.access.redhat.com/openshift3/python-33-rhel7
– Python 3.4 → registry.access.redhat.com/rhscl/python-34-rhel7
● CentOS 7
– Python 2.7 → docker.io/centos/python-27-centos7
– Python 3.3 → docker.io/openshift/python-33-centos7
– Python 3.4 → docker.io/centos/python-34-centos7

16. Migrating from OpenShift 2 to 3
● Converting of existing applications.
● Backward compatible S2I builder.
● Guidelines for porting applications.
● Templates to aid in porting applications.

18. Python for Applications

19. Why *Not* Containers?
● Containers are the recommended option for network services
● However:
– Container support for rich desktop applications is currently limited
– Container runtime may impose unwanted overhead on dedicated systems
– Containers may give more isolation than is wanted
– Applications may require non-trivial modification to run as a privileged container
● Software Collections aim to offer “minimum viable runtime isolation”
– Add new executable directories to front of PATH
– Add new shared library directories to front of LD_LIBRARY_PATH

20. Why Software Collections for Python?
● Use newer Python runtimes without impacting system components
● Use a common Python runtime across multiple operating system versions
● Python 3 for Red Hat Enterprise Linux 6 & 7!

21. Red Hat Software Collections
● Platforms
– Red Hat Enterprise Linux 6 & Red Hat Enterprise Linux 7
– CentOS 6 and 7 (via softwarecollections.org)
– Basis for OpenShift language runtimes
● Available versions (as of RHSCL 2.2)
– Python 2.7.8 (+ selected backports)
– Python 3.5.1 (+ selected backports)
– Python 3.4.2 (+ selected backports)
– Python 3.3.2 (+ selected backports)

22. Python Virtual Environments
● Software Collections allow multiple runtimes to share a system without conflicting
● Virtual environments allow multiple Python applications to share a runtime
● Low or no runtime overhead: just add/replace directories in Python’s import path
● Cleanly isolate application dependencies from platform components
● Dependencies within the environment managed with pip
● Created via:
– python3 -m venv (Python 3.4+)
– virtualenv (Python 2.7, Python 3.x)
● Not included in the Red Hat Enterprise Linux System Python

24. Constructing a Layered Application
● Base platform (via system package manager):
– Operating system (e.g. kernel, C runtime)
– Language runtime (from Software Collections)
– Other external dependencies (e.g. OpenSSL)
● Modify shared library loading (via enabled Software Collection)
● Modify Python import configuration (via virtualenv or standard library’s venv)
● Inside the virtual environment:
– Application dependencies (managed via pip)
– Application source code (managed via pip, or direct from source control)

25. Deploying a Layered Application (1)
● Application RPM with generated artifacts in SRPM:
– Create full installation in representative environment
– Bundle entire virtualenv and other desired components into SRPM
– Also include scripts to appropriately activate SCL and virtual environment
● Some generated files will end up in the SRPM
– Pre-compiled Python files
– Executable wrappers for pip managed Python scripts

26. Deploying a Layered Application (2)
● Application RPM with source-only SRPM:
– SRPM contains source for application and any application level dependencies
– virtual environment created and configured during RPM build process
● Not yet fully supported in pip
– some pip generated metadata will incorrectly include RPM buildroot paths
– shouldn’t matter for most RPM based deployment use cases

27. Managing Application Dependencies
● The Python Package Index is not an App Store!
● Designed to minimize barriers to publication:
– No pre-publication review
– Publisher Terms of Service ensure right to redistribute, not to run or modify
– Publishers may delete (but not replace) previously published versions
● Recommendation: use caching proxies and a component review process
– Commercially supported options: JFrog Artifactory, Sonatype Nexus
– Community/self-supported options: devpi, Python plugin for Pulp
– Check licensing, export restrictions, project governance, ...
● Security monitoring & response also becomes a dev team responsibility!

28. Python for System Administration

29. Why *Not* Software Collections?
● Containers are the recommended option for network services
● Software Collections are recommended for applications
● However:
– some platform bindings are only installed in the main System Python
– the System Python is available on all systems by default
● Some system administration tasks are best handled with the System Python

30. System Python
● Red Hat Enterprise Linux 6
– Python 2.6.6 (+ selected backports)
● Red Hat Enterprise Linux 7
– Python 2.7.5 (+ selected backports)
● Fedora 23 and later
– Recent Python 3.x (rebases rather than backports)
– Recent Python 2.7 available as system package, may not have all system bindings

31. Caveats and Challenges
● Restricted to features of System Python on oldest supported platform
● Community maintained libraries and frameworks often require newer runtimes
● Conflict between supporting:
– Red Hat Enterprise Linux 5 (Python 2.4 lacks Py3 forward compatibility features)
– Fedora 23+ (Python 3.x as System Python)
● May want to consider higher level system abstractions like Ansible

32. Migration to Python 3

33. Python 2.7 Support Timeline
● Anticipated community end-of-life for Python 2.7 in January 2020
– https://docs.python.org/devguide/#status-of-python-branches
● Supported in Red Hat Enterprise Linux 7 until June 2024
– https://access.redhat.com/support/policy/updates/errata
● Anticipate community project support for Python 2 declining sharply post-2020
– Already seeing new community projects starting as Python 3 only

34. Python 3 Migration Techniques
● General “refactoring enablement” techniques:
– automated regression testing frameworks
– static structural analysis tools
● Recommended approach for applications and network services:
– Migrate to the Python 2.7 SCL or OpenShift image (if using the system Python)
– Follow https://docs.python.org/3/howto/pyporting.html
– Migrate to the latest Python 3.x SCL or OpenShift image
● Recommended approach for system administration tools:
– Consider using Fedora 23+ to look for potential pain points

35. Python 3 Migration Notes
● Common subset of Python 2.6+ and 3.3+ is quite large
● Many deprecated idioms can be updated automatically
● Key data & workload driven pain points
– Explicit bytes/unicode separation
– Removal of implicit cross-type comparisons
● Automated refactoring and compatibility testing tools continue to improve

36. Modernizing the Python 2.7
Network Security Stack

37. New Security Features in Python 2.7
● https://docs.python.org/2/whatsnew/2.7.html#pep-466-network-security-enhancements-
for-python-2-7
● Constant-time comparison (hmac.compare_digest())
● Password storage hashing (hashlib.pbkdf2_hmac())
● ssl module rebase on Python 3.4 implementation
– Server Name Indication support
– SSLContext for SSL configuration
– Configuration support for TLS 1.x
– Access to system certificate stores
– ...

38. Availability in Red Hat Products
● Red Hat Enterprise Linux 7.2
– backported to System Python
● Red Hat Software Collections 2.0+
– default in Python 3.4
● Red Hat Software Collections 2.2+
– backported to Python 2.7
– default in Python 3.5

39. Third Party Module Compatibility
● ssl module rebase changed several private implementation details
● Some third party libraries had used internal APIs instead of requesting public ones
● Backport offers greater compatibility than upstream rebase
● Testing before upgrading is still recommended
● Report problems through the usual channels

40. Defaulting to HTTPS
Certificate Verification

41. What does “HTTPS” Mean?
● Historical Python standard library answer:
– “HTTP connection with SSL/TLS enabled”
– didn’t check certificate validity or remote host identification
● Modern Python standard library answer:
– “What web browsers say it means”
– still a HTTP connection with SSL/TLS enabled
– also checks for certificate validity
– also checks remote host identification against system certificate store

42. Verifying HTTPS Certificates
● https://docs.python.org/2/whatsnew/2.7.html#pep-476-enabling-certificate-verification-by-
default-for-stdlib-http-clients
● Default behavior of standard library HTTPS clients in:
– Python 2.7.9+
– Python 3.4.3+
– Python 3.5.0+
● Turns a silent security failure into a noisy connection failure
● Potential problems:
– Self-signed internal certificates
– Expired certificates
– Internal CAs not configured on client system

43. On Red Hat Platforms
● File-based opt-in:
– Config setting in /etc/python/cert-verification.cfg
– Red Hat Enterprise Linux 7.2+ System Python
– Red Hat Software Collections 2.2+ Python 2.7 collection
● Default behavior:
– Red Hat Software Collections Python 3.5 collection
● Details in Knowledge Base
– https://access.redhat.com/articles/2039753

44. Future Configuration Options
● https://docs.python.org/2/whatsnew/2.7.html#pep-493-https-verification-migration-tools
-for-python-2-7
● Adds new Python 2.7 specific configuration options
– ssl._https_verify_certificates() API
– PYTHONHTTPSVERIFY environment variable
● Can be used to revert Python 2.7.12+ to Python 2.7.8 behavior
● Python 2.7 only, not supported by any version of Python 3

45. Resources

46. OpenShift
● OpenShift
– https://www.openshift.com
● OpenShift Online Preview
– https://www.openshift.com/devpreview/register.html
● OpenShift Container Development Kit
– https://developers.redhat.com/products/cdk/overview/
● OpenShift Origin
– https://www.openshift.org
● OpenShift Origin All-In-One VM
– https://www.openshift.org/vm

47. Python
● Red Hat Software Collections
– http://developers.redhat.com/products/softwarecollections/
– Access: https://access.redhat.com/solutions/472793
● Software Collections Upstream
– https://wiki.centos.org/SpecialInterestGroup/SCLo
– https://www.softwarecollections.org

48. Related DevNation 2016 Sessions
● OpenShift
– OpenShift Enterprise 3 walk-through with Docker and Kubernetes
● Container Development Kit
– CDK 2.0: Docker, Kubernetes, and OSE on your desk
– Container development for command line developers
● Software Collections
– Software Collections: Easy access to the cutting edge