Aurynn Shaw
Stored Procedures Made Easier
In this talk, we will be covering the Simpycity and Exceptable libraries, a Python-based database abstraction which provides an ORM-equivalent stack around stored procedure development.
We will be covering why stored procedures are an important tool in database application development, how to design business logic abstractions for your database, and how Exceptable aids in providing for cleaner client code.
Finally, we will cover example model code implementing the Simpycity abstractions, and cover how these abstractions isolate the underlying data model without damaging functionality or data access.
1. Simpycity and
Exceptable
A Q u e ry - F o c u s s e d
D a t a b a s e A b s t r a c t i o n
A u ry n n S h a w
P o s t g r e S Q L E a s t , 2 0 1 0
Saturday, March 27, 2010 1
3. Whys and
Wherefores
Lowest level of our data
design
The humble table
Saturday, March 27, 2010 3
4. Whys and
Wherefores
This is our ideal -
normalized, good natural
keys, good constraints,
minimal data duplication.
The humble table
Best possible relational representation
Saturday, March 27, 2010 4
5. Whys and
Wherefores
Are tables the best
representation to model?
The humble table
Best possible relational representation
Generally, ORMs model tables
Saturday, March 27, 2010 5
7. But wait...
Objects aren’t like relations
Saturday, March 27, 2010 7
8. But wait...
Objects encapsulate
attributes, properties,
methods, all the stuff
that constitutes a single
entity
Objects aren’t like relations
A single, coherent concept
Saturday, March 27, 2010 8
9. But wait...
Proper normalization,
users are a good example.
Objects aren’t like relations
A single, coherent concept
Spanning multiple tables
Saturday, March 27, 2010 9
10. So...
Why are we using ORMs
to represent individual
rows in our tables? Our
objects aren’t relations.
Saturday, March 27, 2010 10
11. Data Modesty
They are our physical
data representation -
the lowest abstraction
we work with.
Physical data
Saturday, March 27, 2010 11
12. Data Modesty
Exposing your underlying
tables means that you
end up being locked to
that representation -
even if you don’t want to
be.
Physical data We argue this is an
important reason for
object abstractions, why
Exposure is Commitment is it different for
relational abstractions?
Saturday, March 27, 2010 12
13. Data Modesty
What we should be
doing is aiming for a
logical
representation of
our data, as opposed
to matching the
Physical data physical
representation. We
use methods and
Exposure is Commitment attributes on our
objects to
manipulate data, we
Tables aren’t the logical interface don’t directly fiddle
the private
attributes.
Saturday, March 27, 2010 13
14. If we’re not
modelling tables...
What are we modelling?
What should we be
modelling?
Saturday, March 27, 2010 14
15. What is Simpycity?
Why not queries?
Queries are the
mechanism by which
we interact with the
physical data layer,
and queries bind
multiple tables
Query Mapping library together via JOIN, so,
are they not a good
representation for our
business layer?
Saturday, March 27, 2010 15
16. What is Simpycity?
The core is arbitrary
methods invoking
arbitrary queries,
seamlessly and deliciously.
Query Mapping library
Methods invoke Queries
Saturday, March 27, 2010 16
17. How to work with
Simpycity systems
firstly, setting up the
data connection
And now, some Code
Saturday, March 27, 2010 17
18. And now, some Code
>>> from simpycity.context import Context
>>> ctx = Context(dsn=“dbname=‘foo’
username=‘bar’”)
a Context is a single
point from which all
data constructs are
derived - the DB
connection can be
committed and closed
from this point.
Saturday, March 27, 2010 18
19. Basic Queries
>>> i = ctx.Raw(“““SELECT *
... FROM users.user
... WHERE id = 1”””)
...
>>> user = i()
>>> user[‘id’]
1 Most absolutely basic
usage of Simpycity’s Raw
>>> datatype.
Saturday, March 27, 2010 19
20. Parameterized
Queries
>>> i = ctx.Raw(“““SELECT *
... FROM users.user
... WHERE id = %s”””,
... [‘id’])
...
>>> user = i(1)
>>> user[‘id’] Parameterization -
Methods suddenly look
1 like actual Python
functions!
>>>
Saturday, March 27, 2010 20
21. Parameterized
Queries
>>> i = ctx.Raw(“““SELECT *
... FROM users.user
... WHERE id = %s”””,
... [‘id’])
...
>>> user = i(1)
>>> user[‘id’] Simpycity callables even
1 work with standard
keyword
>>> user = i(id=1) parameterization.
>>> user[‘id’]
1
>>>
Saturday, March 27, 2010 21
22. Procedural Logic
>>> inst = ctx.Function(“““get_user”””,
[‘id’])
>>> user_positional = inst(1)
>>> user_positional[‘id’]
1
>>> user_keyword = inst(id=1)
>>> user_keyword[‘id’]
1 Functions work exactly
the same - Same
>>> arguments, syntax, and
parameterization
semantics.
Saturday, March 27, 2010 22
23. Positional and
Keyword
>>> i = ctx.Function(“““complex_get”””,
[‘id’,‘table’,‘schemaname’])
>>> rs = i(1, ‘some_table’, ‘public’)
- OR -
>>> rs = i(id=1, table=‘some_table’,
schemaname=‘public’);
- OR -
>>> rs = i(schemaname=‘public’, id=1,
table=‘some_table’);
A huge advantage of Simpycity is treating the argument chain as positional or keyword arguments -
Allowing your APIs to be indistinguishable from normal Python.
Saturday, March 27, 2010 23
25. Upshots You’re dealing with
constructs that act and
respond like a Python
method should act and
respond - It allows for a
very consistent
interface
Running queries is calling a method.
Saturday, March 27, 2010 25
26. Upshots Instead of writing our
queries using a half-
assed badly-
implemented subset of
SQL, we write queries
directly in SQL, and
never have to worry
about the query
generator writing
ridiculous queries.
Instead, we write our
Running queries is calling a method. own ridiculous
queries. ;)
Underlying DB is abstracted away
Saturday, March 27, 2010 26
27. Downsides Due to Simpycity’s
current architecture,
insert/update/delete
statements aren’t
really supported - it
definitely expects to
get something *Back*
from the DB.
Insert/Update/Delete requires a procedure..
Saturday, March 27, 2010 27
28. Downsides Simpycity’s queries also
don’t have a concept of
defaults, as yet - all
arguments declared by
the definition *must* be
present in the call.
Insert/Update/Delete requires a procedure..
..All arguments must be accounted for..
Saturday, March 27, 2010 28
29. This won’t work.
>>> inst = ctx.Function(“““complex_get”””,
[‘id’,‘table’,‘schemaname’])
>>> item = inst(1, ‘some_table’)
Traceback (most recent call last):
.. <SNIP> ..
Exception: Insufficient arguments: Expected
3, got 2
So this isn’t going to work.
Saturday, March 27, 2010 29
31. Downsides
Insert/Update/Delete requires a procedure..
..All arguments must be accounted for..
..Big resultsets will be entirely pulled into
memory..
Another disadvantage is running a query that has a lot of results will pull them *all* into
memory, by default. This is a limitation of the underlying result set representation, for
reasons I’ll get into in a moment.
Saturday, March 27, 2010 31
32. Downsides Simpycity doesn’t do
*any* actual query
generation - the most it
will do is the select *
from function. None of
the more advanced query
generation exists here.
Insert/Update/Delete requires a procedure..
All arguments must be accounted for
Big resultsets will be entirely pulled into
memory
.. And, you’ll be writing a lot of SQL...
Saturday, March 27, 2010 32
33. This is all it does
>>> inst = ctx.Function(“““complex_get”””,
[‘id’,‘table’,‘schemaname’])
Becomes
SELECT * FROM complex_get(%s, %s, %s)
Saturday, March 27, 2010 33
34. SQL is programming -
Upshots code. It’s as important
as the rest of your
application, and you’re
better at writing it
than a computer is.
Even more, computers
cannot extract
semantic meaning
from your relational
design, and cannot
Running queries is calling a method. build appropriate
representations - only
the programmer can.
Underlying DB is abstracted away
...but you should be anyway.
Saturday, March 27, 2010 34
35. While the queries are
useful on their own,
they don’t really provide
an easy way to manage
data at an application
level.
For that,
Applications!
Saturday, March 27, 2010 35
36. Applications!
The reasoning behind that
Need Logical Representations is that applications require
logical abstractions that
make sense from the
application perspective -
Saturday, March 27, 2010 36
37. Applications!
Need Logical Representations
Business models!
Divorcing us from the
underlying table
representations. Instead
of modelling tables, we
should be modelling
*objects*, concepts that
are complete unto
themselves.
Saturday, March 27, 2010 37
38. Models in Simpycity
Don’t model tables
Models in Simpycity
follow this logical chain -
we don’t model the tables.
Instead, we work to find
what the best
representation of a given
object is,
Saturday, March 27, 2010 38
39. Models in Simpycity
Don’t model tables
Aligned towards application requirements
the one that most clearly fits what the
application itself requires, in terms of internal
architecture and logical consistency.
Instead of fighting to make relational concepts
fit into objects, we should be making our objects
accurately represent the concepts we need.
Saturday, March 27, 2010 39
40. Models in Simpycity
Don’t model tables
Aligned towards application requirements
Still allow for Active Record-style manipulation
At the same time, the Active Record pattern has a lot of useful concepts, like direct
instancing and .save() on dirty objects.
For these reasons, the model pattern that Simpycity uses is less Active Record, as
we’re not modelling result sets, but more
Saturday, March 27, 2010 40
41. Active Object.
Let’s have a look at how
Simpycity handles
Active Object.
Active Object
Saturday, March 27, 2010 41
42. Basic Models
>>> base = ctx.Model()
- OR -
>>> class base(ctx.Model()):
... pass
First, we create a base class that all our models
derive from - This allows us to add additional
functionality on a global level to our application
models.
For instance, Vertically Challenged creates a base
model with authentication tokens baked in.
Saturday, March 27, 2010 42
43. Basic Models
>>> base = ctx.Model()
>>> class ourUser(base):
... table = [“id”, “username”]
Declaring the basic model - our instance, and the
table declares what our internal attributes are.
Note how we don’t really enforce data types -
this just declares what the business object looks
like.
Saturday, March 27, 2010 43
44. Basic Models
base = ctx.Model()
class ourUser(base):
table = [“id”, “username”]
__load__ = ctx.Function(“get_user”,[‘id’])
__load__ is the basic instancing mechanism in a
Simpycity - under the covers, any arguments
passed to the instancing of a new object will be
passed to this function, and then mapped to the
object’s attributes.
Saturday, March 27, 2010 44
45. Basic Models
>>> base = ctx.Model()
>>> class ourUser(base):
... table = [“id”, “username”]
... __load__ = ctx.Function(“get_user”,
[‘id’])
From load, we instance our
>>> u = ourUser(1) models just as if we were
running the query directly -
>>> u.id only now, we have the model
attributes available, that we
1 didn’t have before.
>>> u.username
“Test User”
>>>
Saturday, March 27, 2010 45
46. Basic Models For instance, we can update the model
with new values.
But, they’re just a part of that
particular model instance. They’re not
persisted out to the database.
>>> u = ourUser(1) Fortunately, Simpycity can deal with
this, too:
>>> u.id
1
>>> u.username
“Test User”
>>> u.username = “PGEast Demo”
>>> u.username
“PGEast Demo”
>>>
Saturday, March 27, 2010 46
47. Basic Models
class ourUser(base):
table = [“id”, “username”]
__load__ = ctx.Function(“get_user”,[‘id’])
__save__ = ctx.Function(“save_user”,
[‘id’,‘username’])
>>> u.username = “PG Demo”
Coming back to our model, we’ve added
>>> u.username another property - __save__.
By declaring __save__ on our model, we
“PG Demo” get access to the dynamic .save() method.
>>> u.save() This method will take the current model’s
state, test for any dirty values, and use
>>> ctx.commit() the provided callable to save those values.
The callable is allowed to be anything you
like - it doesn’t have to be a Simpycity
function.
Saturday, March 27, 2010 47
48. Basic Models
class ourUser(base):
table = [“id”, “username”, “password”]
__load__ = ctx.Function(“get_user”,[‘id’])
__save__ = ctx.Function(“save_user”,
[‘id’,‘username’])
delete = ctx.Function(“delete_user”,
[‘id’])
Additionally, models need custom methods, and Simpycity
covers that too - assigning a Simpycity function to an
attribute will be automatically converted into a bound
callable, and will map any arguments named in the table as
that value.
In this case, we’ve provided a delete method on the model,
which Simpycity does not currently support, and anything else
you can conceive works in this way.
Saturday, March 27, 2010 48
49. Other Instance
Patterns
So, at this point, we have a model that can be instanced, manipulated, and
deleted, similar to a normal ORM. All we know about the underlying
database is there’s a couple of functions, and what their arguments are.
Unfortunately, we only have a single instancing mechanism. This works for
limited cases, but is really lacking in flexibility.
What we need is multiple instancing patterns, a variety of ways to pull data
from the database.
Simpycity can handle that, too - with a nifty little pattern.
Saturday, March 27, 2010 49
50. Other Instance
Patterns
class ourUser(base):
... # Remember the Code.
by_id = ctx.Function(“get_user”,[‘id’],
returns_type=base)
by_username = ctx.Function(“get_user”,
[‘username’], return_type=ourUser)
Here, we’re stepping back to the core Raw and Function mechanisms, and
passing a new argument, return_type.
Return type takes the provided class and will map the results of the query to
that object.
If the query returned multiple rows, then you’ll get a normal list of objects, all
correctly mapped.
This functionality even allows for the easy creation of new generators, such as
Saturday, March 27, 2010 50
51. Newly Minted
Objects
class ourUser(base):
... # Code was here.
new = ctx.Function(“new_user”,[‘username’,
‘password’], return_type=ourUser)
with this new method.
By declaring the database function to return the new user object, we both
insert our new record and get our working model in a single call.
And, since it’s already persisted to our database, any methods hanging on the
model that require database backing will still work.
Any single aspect goes wrong, and it all rolls back, just as it should.
Saturday, March 27, 2010 51
52. How about Search?
>>> search = ctx.Function(“user_search”,
[‘content’], return_type=ourUser)
>>> u = search(“content in their profile”)
>>>
A search method would even work great in this format - any
function that can generate the target object type is a great fit.
Saturday, March 27, 2010 52
53. It’s all about good
abstractions
So far, everything that we’ve talked about has been discussing building good
abstractions at the business logic level of our app - abstracting the
underlying able design into views and stored procedures, as necessary.
But this does overlook a single critical aspect of building good APIs, and that
is Exceptions.
Saturday, March 27, 2010 53
54. Exceptions
By default, exceptions in plpgsql are very generic - a single
exception type, with a text argument.
While functional, this does not provide an abundance of
manipulatable error types.
Stored Procedures get RAISE EXCEPTION
Saturday, March 27, 2010 54
55. Exceptions
When this single exception type reaches the Python
layer, psycopg2 will convert it into an InternalError -
the string is preserved.
This is somewhat useful, but,
In stored procedures, RAISE EXCEPTION
In Python, this becomes an InternalError
Saturday, March 27, 2010 55
56. Current Exceptions
CREATE OR REPLACE FUNCTION except_test()
RETURNS VOID AS $$
but, this is what we
BEGIN currently have to work
with from the DB.
RAISE EXCEPTION 'Test!';
Not all that useful,
END; right?
$$ LANGUAGE PLPGSQL;
>>> c.Function("except_test")()
Traceback (most recent call last):
... <SNIP> ...
psycopg2.InternalError: Test!
Saturday, March 27, 2010 56
57. Exceptions
In stored procedures, RAISE EXCEPTION
This becomes an InternalError
But what was it *really*?
Unless you’re deeply adhering to a consistent style in your exception text - not always
easily done - you’re going to end up with inconsistencies.
Because of this, you’ll end up in a position where you’re parsing error strings, looking for
specific errors. Not the best design practise.
Saturday, March 27, 2010 57
58. Exceptable
Consistency of exception text
The first advantage that
Exceptable in Simpycity
brings is significantly more
consistent exception text -
allowing for Exceptable to
consistently parse and re-
raise exceptions.
Saturday, March 27, 2010 58
59. Exceptable
Consistency of exception text
Simple DB API
Working with Exceptable at
the DB level is also incredibly
easy: there’s really only 2
queries that need to be
remembered.
Saturday, March 27, 2010 59
60. Easy!
CREATE OR REPLACE FUNCTION except_test()
RETURNS VOID AS $$
SELECT exceptable.raise(
Firstly, raising exceptions
‘YourException’, - easily handled, though
slightly more verbose
than a standard
‘This is the Error Text’); exception.
$$ LANGUAGE SQL;
Saturday, March 27, 2010 60
61. and in Python
>>> c.Function("except_test")()
Traceback (most recent call last):
... <SNIP> ...
psycopg2.InternalError: YourException::
This is the Error Text
This is what the
error string looks
like, now!
Saturday, March 27, 2010 61
62. Adding new
Exceptions
your_database=> SELECT exceptable.register
(‘YourException’,‘This is our custom
exception!’, NULL);
register
----------
t
The second aspect is adding new exceptions to the Exceptable
tables - without this, attempting to use an exception will
throw an error.
This is done solely so that, even though it’s not yet
implemented, Exceptable can introspect the exceptions table
and automatically generate exceptions.
It also allows for enforced consistency - Typos happen to even
the best of us.
Saturday, March 27, 2010 62
63. Exceptable
Consistency of exception text All of this is trying to work
to bringing *good*
exceptions to the
application fabric - An
Simple DB API easily parsed exception
from the database, and the
Exceptable integration in
Good Application Exceptions Simpycity means, we can
have first-class Python
exceptions from our stored
procedures.
Here’s how:
Saturday, March 27, 2010 63
64. Register the
Exception..
>>> from simpycity.exceptions import base
>>> class YourException(Exception):
... pass
...
>>> base.register(“YourException”,
YourException)
The syntax for registering a new exception
at the Python layer is similar to the DB
layer, with the first argument providing the
same text value as the exception type in the
DB.
The second argument is the class definition,
and instances will be used any time your
exception is raised from the database.
Saturday, March 27, 2010 64
65. And then use it.
>>> c.Function(“except_test”)()
Traceback (most recent call last):
... SNIP ...
YourException: This is the Error text
And using it is identical to any other
Python exception. Simple and easy.
Saturday, March 27, 2010 65
66. PG Error Codes
But wait! 8.4 supports custom error codes!
A cool feature of 8.4 and
higher is the ability to raise an
exception inside a given error
code. There’s a large list of
error codes, and a sub-feature
of that is the ability to raise
custom error codes.
Saturday, March 27, 2010 66
67. PG Error Codes
But wait! 8.4 supports custom error codes
But Exceptable doesn’t! (yet) This is one of those features
that Exceptable *will* be
supporting, eventually.
Instead of using a regex to
map exceptions, exceptable
will just compare a list of pg
error codes - much simpler
to implement.
Saturday, March 27, 2010 67
68. See! Custom!
CREATE OR REPLACE FUNCTION ex_test()
RETURNS void
AS $body$
BEGIN
RAISE EXCEPTION 'testing'
USING ERRCODE= 'EX111';
END;
$body$ LANGUAGE PLPGSQL;
Saturday, March 27, 2010 68
69. And the Python
>>> try:
... c.Function(“ex_test”)()
... except Exception, e:
... print e.pgcode
...
EX111 And our ex111 exception gets properly
propagated upwards, and Exceptable would
be able to catch it and map it correctly.
Saturday, March 27, 2010 69
70. And, thus
A n y Q u e s t i o n s ?
Saturday, March 27, 2010 70
71. Get it!
h t t p s : / /
p r o j e c t s . c o m m a n d p r o m p t .
c o m / p u b l i c / s i m p y c i t y
and
h t t p s : / /
p r o j e c t s . c o m m a n d p r o m p t .
c o m / p u b l i c / e x c e p t a b l e
Saturday, March 27, 2010 71
72. And finally,
Thank you.
Saturday, March 27, 2010 72