SlideShare una empresa de Scribd logo
1 de 33
Descargar para leer sin conexión
Making Django and NoSQL
Play Nice
Alex Gaynor
Berlin
NoSQL


Any database that doesn’t speak SQL
Usually non-relational databases
e.g. Cassandra, Redis, MongoDB
2 Part Talk


         50%                50%




               Current Internals
               Coming Changes
What does playing nice
mean?
  from mongoengine import connection


  def my_view(request):
      objects = connection.do_something()




              BAD
# settings.py

DATABASES = {
    "default": {
        "ENGINE": "django_mongo",
    }
}

# models.py

from django.db import models

class MyPerfectlyNormalModel(models.Model):
    name = models.CharField(max_length=12)




                GOOD
Why do we care?
Admin
Forms
Serializers
Model validation
API Generators
Metadata
Makes my brain hurt less
Into the rabbit hole we go!
Lay of the land
 Models
 Managers
 QuerySets
 Queries
 Compilers
 Backends
Models
from django.db import models


class Category(models.Model):
    name = models.CharField(max_length=100)
    slug = models.SlugField()
    parent = models.ForeignKey("self", null=True)
Managers


     Category.objects
QuerySets


Category.objects.get_query_set()
Queries

Category.objects.get_query_set().query
Compilers


qs = Category.objects.get_query_set()

qs.query.get_compiler(qs.db)
QuerySets
The whole damned thing
QuerySet

This is the top layer of query state. From here on out
it’s like an onion.
Not backend specific.
_db
_result_cache
_iter
query
Query

Right now this holds all state for a query.
It’s semi-backend specific. Right now there’s one, and
it’s specific to SQL backends.
Computes most JOINS, aggregates, etc.
Translates Q objects into Where objects.
The Query Problem

It’s something of a lossy translation.
Translating filter(), values(), and other calls into internal
datastructures is lossless with respect to the database
being used, but not with respect to other databases.
If you’ve got all SQL databases you’re fine, but if you
mix in a non-relational DB you’ve got problems.
More on this later.
SQLCompiler

Takes a Query and a connection and turns it into SQL
(and executes it).
This also does some computation of joins (for
select_related).
This is the totally backend specific part (only part that
knows about the actual connection and database).
The rest of the chain just *assumes* a SQL db.
django.db.backends.*

This is where backends live.
Not super exciting.
A bunch of flags and methods to control very small
parts of SQL creation.
Also introspection, creation, and shell.
You call methods on a QuerySet
  Which calls methods on a Query
    Which mutates some datastructures
You evaluate a QuerySet
  Which asks it’s Query for a Compiler
    Which generates some SQL
      Which calls some methods on the backend
    Which gets a cursor and evaluates it
The Problem
Query thinks in terms of
SQL

 It chooses between join types
 it generates table aliases
 it splits filters between HAVING and WHERE
 and probably some other stuff
Why is this an issue

 How do I ask a MongoDBCompiler to compile a LEFT
 OUTER JOIN vs. an INNER JOIN?
 Or a HAVING vs a WHERE?
 These concepts don’t map cleanly, so the translation is
 lossy across backends
Design Decisions
Not everything is a technical problem
Do we emulate JOINs?


Category.objects.filter(parent__parent__name="Tech")
Do we maintain secondary
indices?


   Category.objects.filter(name="Tech")
Different databases have
different features

 True of SQL databases, but more so for non-relational
 databases.
 No lingua franca like SQL is.
A solution
Or something close enough...
Make Query do less


Instead of generating two trees of WHERE and
HAVING, generate a single tree of filters.
Don’t generate JOINs at all.
Push that all down to the compiler.
Make SQLCompiler do more

Generate all JOINs
Split filter tree into HAVING vs WHERE
Can generate more efficient JOINs with global
knowledge.
Probably makes it easier to do fix some other ORM
bugs.
Plan of action

 Change the ORM up
 Build MongoDB prototype backend
 ???
 Profit
http://alexgaynor.net/

Slides will be up there
Making Django and NoSQL Play Nice

Más contenido relacionado

La actualidad más candente

Django tech-talk
Django tech-talkDjango tech-talk
Django tech-talkdtdannen
 
Django Architecture Introduction
Django Architecture IntroductionDjango Architecture Introduction
Django Architecture IntroductionHaiqi Chen
 
Hybrid Web Applications
Hybrid Web ApplicationsHybrid Web Applications
Hybrid Web ApplicationsJames Da Costa
 
Django best practices for logging and signals
Django best practices for logging and signals Django best practices for logging and signals
Django best practices for logging and signals flywindy
 
Django Overview
Django OverviewDjango Overview
Django OverviewBrian Tol
 
Introduction to Django
Introduction to DjangoIntroduction to Django
Introduction to Djangocolinkingswood
 
Django Framework Overview forNon-Python Developers
Django Framework Overview forNon-Python DevelopersDjango Framework Overview forNon-Python Developers
Django Framework Overview forNon-Python DevelopersRosario Renga
 
Create responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJSCreate responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJSHannes Hapke
 
Djangocon 2014 angular + django
Djangocon 2014 angular + djangoDjangocon 2014 angular + django
Djangocon 2014 angular + djangoNina Zakharenko
 
Django and Mongoengine
Django and MongoengineDjango and Mongoengine
Django and Mongoengineaustinpublic
 
The Best (and Worst) of Django
The Best (and Worst) of DjangoThe Best (and Worst) of Django
The Best (and Worst) of DjangoJacob Kaplan-Moss
 
[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules
[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules
[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rulesSrijan Technologies
 
Owl: The New Odoo UI Framework
Owl: The New Odoo UI FrameworkOwl: The New Odoo UI Framework
Owl: The New Odoo UI FrameworkOdoo
 
Django for mobile applications
Django for mobile applicationsDjango for mobile applications
Django for mobile applicationsHassan Abid
 
Web application development with Django framework
Web application development with Django frameworkWeb application development with Django framework
Web application development with Django frameworkflapiello
 
The Django Web Application Framework 2
The Django Web Application Framework 2The Django Web Application Framework 2
The Django Web Application Framework 2fishwarter
 

La actualidad más candente (20)

Django tech-talk
Django tech-talkDjango tech-talk
Django tech-talk
 
Django Architecture Introduction
Django Architecture IntroductionDjango Architecture Introduction
Django Architecture Introduction
 
Hybrid Web Applications
Hybrid Web ApplicationsHybrid Web Applications
Hybrid Web Applications
 
Django a whirlwind tour
Django   a whirlwind tourDjango   a whirlwind tour
Django a whirlwind tour
 
Backbone.js
Backbone.jsBackbone.js
Backbone.js
 
Django best practices for logging and signals
Django best practices for logging and signals Django best practices for logging and signals
Django best practices for logging and signals
 
Django Overview
Django OverviewDjango Overview
Django Overview
 
Introduction to Django
Introduction to DjangoIntroduction to Django
Introduction to Django
 
Django Framework Overview forNon-Python Developers
Django Framework Overview forNon-Python DevelopersDjango Framework Overview forNon-Python Developers
Django Framework Overview forNon-Python Developers
 
Create responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJSCreate responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJS
 
Django Mongodb Engine
Django Mongodb EngineDjango Mongodb Engine
Django Mongodb Engine
 
Djangocon 2014 angular + django
Djangocon 2014 angular + djangoDjangocon 2014 angular + django
Djangocon 2014 angular + django
 
Django and Mongoengine
Django and MongoengineDjango and Mongoengine
Django and Mongoengine
 
Django
DjangoDjango
Django
 
The Best (and Worst) of Django
The Best (and Worst) of DjangoThe Best (and Worst) of Django
The Best (and Worst) of Django
 
[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules
[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules
[Srijan Wednesday Webinars] Ruling Drupal 8 with #d8rules
 
Owl: The New Odoo UI Framework
Owl: The New Odoo UI FrameworkOwl: The New Odoo UI Framework
Owl: The New Odoo UI Framework
 
Django for mobile applications
Django for mobile applicationsDjango for mobile applications
Django for mobile applications
 
Web application development with Django framework
Web application development with Django frameworkWeb application development with Django framework
Web application development with Django framework
 
The Django Web Application Framework 2
The Django Web Application Framework 2The Django Web Application Framework 2
The Django Web Application Framework 2
 

Destacado

PyCon India 2012: Rapid development of website search in python
PyCon India 2012: Rapid development of website search in pythonPyCon India 2012: Rapid development of website search in python
PyCon India 2012: Rapid development of website search in pythonChetan Giridhar
 
Django with Mongo using Mongoengine
Django with Mongo using MongoengineDjango with Mongo using Mongoengine
Django with Mongo using Mongoenginedudarev
 
Desenvolvimento web simples com Python e DJango
Desenvolvimento web simples com Python e DJangoDesenvolvimento web simples com Python e DJango
Desenvolvimento web simples com Python e DJangoRafael Nunes
 
Graph Databases in Python (PyCon Canada 2012)
Graph Databases in Python (PyCon Canada 2012)Graph Databases in Python (PyCon Canada 2012)
Graph Databases in Python (PyCon Canada 2012)Javier de la Rosa
 
Python e Django na Globo.com
Python e Django na Globo.comPython e Django na Globo.com
Python e Django na Globo.comricobl
 
MongoDB for Beginners
MongoDB for BeginnersMongoDB for Beginners
MongoDB for BeginnersEnoch Joshua
 
Sql vs NoSQL
Sql vs NoSQLSql vs NoSQL
Sql vs NoSQLRTigger
 
Introduction to NoSQL Databases
Introduction to NoSQL DatabasesIntroduction to NoSQL Databases
Introduction to NoSQL DatabasesDerek Stainer
 

Destacado (11)

PyCon India 2012: Rapid development of website search in python
PyCon India 2012: Rapid development of website search in pythonPyCon India 2012: Rapid development of website search in python
PyCon India 2012: Rapid development of website search in python
 
Django with Mongo using Mongoengine
Django with Mongo using MongoengineDjango with Mongo using Mongoengine
Django with Mongo using Mongoengine
 
Why Django for Web Development
Why Django for Web DevelopmentWhy Django for Web Development
Why Django for Web Development
 
Desenvolvimento web simples com Python e DJango
Desenvolvimento web simples com Python e DJangoDesenvolvimento web simples com Python e DJango
Desenvolvimento web simples com Python e DJango
 
Relational vs. Non-Relational
Relational vs. Non-RelationalRelational vs. Non-Relational
Relational vs. Non-Relational
 
Graph Databases in Python (PyCon Canada 2012)
Graph Databases in Python (PyCon Canada 2012)Graph Databases in Python (PyCon Canada 2012)
Graph Databases in Python (PyCon Canada 2012)
 
Python e Django na Globo.com
Python e Django na Globo.comPython e Django na Globo.com
Python e Django na Globo.com
 
MongoDB for Beginners
MongoDB for BeginnersMongoDB for Beginners
MongoDB for Beginners
 
Sql vs NoSQL
Sql vs NoSQLSql vs NoSQL
Sql vs NoSQL
 
Mongo db
Mongo dbMongo db
Mongo db
 
Introduction to NoSQL Databases
Introduction to NoSQL DatabasesIntroduction to NoSQL Databases
Introduction to NoSQL Databases
 

Similar a Making Django and NoSQL Play Nice

Testing database content with DBUnit. My experience.
Testing database content with DBUnit. My experience.Testing database content with DBUnit. My experience.
Testing database content with DBUnit. My experience.Serhii Kartashov
 
Entity Framework Database and Code First
Entity Framework Database and Code FirstEntity Framework Database and Code First
Entity Framework Database and Code FirstJames Johnson
 
Linq 1224887336792847 9
Linq 1224887336792847 9Linq 1224887336792847 9
Linq 1224887336792847 9google
 
High Performance Jdbc
High Performance JdbcHigh Performance Jdbc
High Performance JdbcSam Pattsin
 
Linq To The Enterprise
Linq To The EnterpriseLinq To The Enterprise
Linq To The EnterpriseDaniel Egan
 
Obevo Javasig.pptx
Obevo Javasig.pptxObevo Javasig.pptx
Obevo Javasig.pptxLadduAnanu
 
Ejb3 Struts Tutorial En
Ejb3 Struts Tutorial EnEjb3 Struts Tutorial En
Ejb3 Struts Tutorial EnAnkur Dongre
 
Ejb3 Struts Tutorial En
Ejb3 Struts Tutorial EnEjb3 Struts Tutorial En
Ejb3 Struts Tutorial EnAnkur Dongre
 
SF Elixir Meetup - RethinkDB
SF Elixir Meetup - RethinkDBSF Elixir Meetup - RethinkDB
SF Elixir Meetup - RethinkDBPeter Hamilton
 
SQL for NoSQL and how Apache Calcite can help
SQL for NoSQL and how  Apache Calcite can helpSQL for NoSQL and how  Apache Calcite can help
SQL for NoSQL and how Apache Calcite can helpChristian Tzolov
 
NoSQL Endgame DevoxxUA Conference 2020
NoSQL Endgame DevoxxUA Conference 2020NoSQL Endgame DevoxxUA Conference 2020
NoSQL Endgame DevoxxUA Conference 2020Thodoris Bais
 
Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)David McCarter
 
MVC and Entity Framework 4
MVC and Entity Framework 4MVC and Entity Framework 4
MVC and Entity Framework 4James Johnson
 
Myth busters - performance tuning 102 2008
Myth busters - performance tuning 102 2008Myth busters - performance tuning 102 2008
Myth busters - performance tuning 102 2008paulguerin
 
Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)David McCarter
 

Similar a Making Django and NoSQL Play Nice (20)

Testing database content with DBUnit. My experience.
Testing database content with DBUnit. My experience.Testing database content with DBUnit. My experience.
Testing database content with DBUnit. My experience.
 
La sql
La sqlLa sql
La sql
 
Entity Framework Database and Code First
Entity Framework Database and Code FirstEntity Framework Database and Code First
Entity Framework Database and Code First
 
Linq 1224887336792847 9
Linq 1224887336792847 9Linq 1224887336792847 9
Linq 1224887336792847 9
 
High Performance Jdbc
High Performance JdbcHigh Performance Jdbc
High Performance Jdbc
 
Linq To The Enterprise
Linq To The EnterpriseLinq To The Enterprise
Linq To The Enterprise
 
4 jdbc step1
4 jdbc step14 jdbc step1
4 jdbc step1
 
Obevo Javasig.pptx
Obevo Javasig.pptxObevo Javasig.pptx
Obevo Javasig.pptx
 
Ejb3 Struts Tutorial En
Ejb3 Struts Tutorial EnEjb3 Struts Tutorial En
Ejb3 Struts Tutorial En
 
Ejb3 Struts Tutorial En
Ejb3 Struts Tutorial EnEjb3 Struts Tutorial En
Ejb3 Struts Tutorial En
 
Green dao
Green daoGreen dao
Green dao
 
Django Pro ORM
Django Pro ORMDjango Pro ORM
Django Pro ORM
 
Sql
SqlSql
Sql
 
SF Elixir Meetup - RethinkDB
SF Elixir Meetup - RethinkDBSF Elixir Meetup - RethinkDB
SF Elixir Meetup - RethinkDB
 
SQL for NoSQL and how Apache Calcite can help
SQL for NoSQL and how  Apache Calcite can helpSQL for NoSQL and how  Apache Calcite can help
SQL for NoSQL and how Apache Calcite can help
 
NoSQL Endgame DevoxxUA Conference 2020
NoSQL Endgame DevoxxUA Conference 2020NoSQL Endgame DevoxxUA Conference 2020
NoSQL Endgame DevoxxUA Conference 2020
 
Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)
 
MVC and Entity Framework 4
MVC and Entity Framework 4MVC and Entity Framework 4
MVC and Entity Framework 4
 
Myth busters - performance tuning 102 2008
Myth busters - performance tuning 102 2008Myth busters - performance tuning 102 2008
Myth busters - performance tuning 102 2008
 
Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)
 

Último

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"DianaGray10
 

Último (20)

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
 

Making Django and NoSQL Play Nice

  • 1. Making Django and NoSQL Play Nice Alex Gaynor Berlin
  • 2. NoSQL Any database that doesn’t speak SQL Usually non-relational databases e.g. Cassandra, Redis, MongoDB
  • 3. 2 Part Talk 50% 50% Current Internals Coming Changes
  • 4. What does playing nice mean? from mongoengine import connection def my_view(request): objects = connection.do_something() BAD
  • 5. # settings.py DATABASES = { "default": { "ENGINE": "django_mongo", } } # models.py from django.db import models class MyPerfectlyNormalModel(models.Model): name = models.CharField(max_length=12) GOOD
  • 6. Why do we care? Admin Forms Serializers Model validation API Generators Metadata Makes my brain hurt less
  • 7. Into the rabbit hole we go!
  • 8. Lay of the land Models Managers QuerySets Queries Compilers Backends
  • 9. Models from django.db import models class Category(models.Model): name = models.CharField(max_length=100) slug = models.SlugField() parent = models.ForeignKey("self", null=True)
  • 10. Managers Category.objects
  • 15. QuerySet This is the top layer of query state. From here on out it’s like an onion. Not backend specific. _db _result_cache _iter query
  • 16. Query Right now this holds all state for a query. It’s semi-backend specific. Right now there’s one, and it’s specific to SQL backends. Computes most JOINS, aggregates, etc. Translates Q objects into Where objects.
  • 17. The Query Problem It’s something of a lossy translation. Translating filter(), values(), and other calls into internal datastructures is lossless with respect to the database being used, but not with respect to other databases. If you’ve got all SQL databases you’re fine, but if you mix in a non-relational DB you’ve got problems. More on this later.
  • 18. SQLCompiler Takes a Query and a connection and turns it into SQL (and executes it). This also does some computation of joins (for select_related). This is the totally backend specific part (only part that knows about the actual connection and database). The rest of the chain just *assumes* a SQL db.
  • 19. django.db.backends.* This is where backends live. Not super exciting. A bunch of flags and methods to control very small parts of SQL creation. Also introspection, creation, and shell.
  • 20. You call methods on a QuerySet Which calls methods on a Query Which mutates some datastructures You evaluate a QuerySet Which asks it’s Query for a Compiler Which generates some SQL Which calls some methods on the backend Which gets a cursor and evaluates it
  • 22. Query thinks in terms of SQL It chooses between join types it generates table aliases it splits filters between HAVING and WHERE and probably some other stuff
  • 23. Why is this an issue How do I ask a MongoDBCompiler to compile a LEFT OUTER JOIN vs. an INNER JOIN? Or a HAVING vs a WHERE? These concepts don’t map cleanly, so the translation is lossy across backends
  • 24. Design Decisions Not everything is a technical problem
  • 25. Do we emulate JOINs? Category.objects.filter(parent__parent__name="Tech")
  • 26. Do we maintain secondary indices? Category.objects.filter(name="Tech")
  • 27. Different databases have different features True of SQL databases, but more so for non-relational databases. No lingua franca like SQL is.
  • 28. A solution Or something close enough...
  • 29. Make Query do less Instead of generating two trees of WHERE and HAVING, generate a single tree of filters. Don’t generate JOINs at all. Push that all down to the compiler.
  • 30. Make SQLCompiler do more Generate all JOINs Split filter tree into HAVING vs WHERE Can generate more efficient JOINs with global knowledge. Probably makes it easier to do fix some other ORM bugs.
  • 31. Plan of action Change the ORM up Build MongoDB prototype backend ??? Profit

Notas del editor