slides for YAPC::Asia 2012

1. DBD::SQLite
Recipes, Issues,
and Plans
Kenichi Ishigaki
(@charsbar)
YAPC::Asia 2012
Sep 28, 2012

2. Thank you
for coming.

3. Today I‘ll show
you 10 recipes
and 10 issues.

4. Recipe 1: Bulk Insert
Recipe 2: In-memory Database And Backup
Recipe 3: Alter Table
Recipe 4: Attach Database
Recipe 5: SQLite Hooks
Recipe 6: Unicode
Recipe 7: Get More Information
Recipe 8: Troubleshooting
Recipe 9: Full Text Search
Recipe 10: Custom Extensions

5. Issue 1: Transaction Mode
Issue 2: Refactoring of "execute"
Issue 3: Optimization
Issue 4: Compiler Issues
Issue 5: iOS
Issue 6: Better Errors
Issue 7: Type Information
Issue 8: Async IO
Issue 9: Cookbook
Issue 10: SQLite 4

6. I don’t think
everything should
be covered today.

7. Questions and
suggestions are
all welcome.

8. Recipe 1
Bulk Insert

9. This is fairly SLOW.
my $dbh = DBI->connect(...);
my $stmt = "INSERT INTO foo VALUES(?)";
for (@data) {
$dbh->do($stmt, undef, $_);
}

10. Much faster.
my $sth = $dbh->prepare($stmt);
for (@data) {
$sth->execute($_);
}

11. A bit more faster.
$dbh->begin_work;
for (@data) {
$sth->execute($_);
}
$dbh->commit;

12. A bit more faster (maybe).
my $ct = 0;
$dbh->begin_work;
for (@data) {
$sth->execute($_);
# this number is arbitrary
unless (++$ct % 1000) {
$dbh->commit;
$dbh->begin_work;
}
}
$dbh->commit;

13. A convenient way.
$dbh->{AutoCommit} = 0;
for (@data) {
$sth->execute($_);
unless (++$ct % 1000) {
$dbh->commit;
}
}
$dbh->commit;
$dbh->{AutoCommit} = 1;

14. Much faster
(but dangerous; SQLite only)
$dbh->do("PRAGMA synchronous = OFF");
for (@data) {
...
}
$dbh->do("PRAGMA synchronous = FULL");

15. Much faster (since 1.37)
# INSERT INTO foo VALUES (?),(?),...
my $placeholders = join ",", ('(?)') x 500;
my $stmt = "INSERT INTO foo VALUES $placeholders";
my $sth = $dbh->prepare($stmt);
while (@data) {
if (@data > 500) {
$sth->execute(splice @data, 0, 500);
} else {
...
}
}

16. Summary
 Use "prepare".
 Don't insert one by one.
 Don't insert everything at once.
 Do The Right Thing when ACIDity matters.
 Multi-row insert may help.
 Beware SQLITE_MAX_VARIABLE_NUMBER
(default: 999)

17. Recipe 2
In-memory Database
And Backup

18. An in-memory database
is quite fast.
# volatile, in-memory database
my $dbh = DBI->connect('dbi:SQLite::memory:'...);
# cf. volatile database based on a temporary file
# my $dbh = DBI->connect('dbi:SQLite:'...);

19. It's also handy
for tests.
{
my $dbh = DBI->connect('dbi:SQLite::memory:'...);
... # do whatever you need
}
# everything is gone
(Cleaning-up files may be a bit tricky)

20. What to do
if the tests
should fail?

21. Back it up as
necessary.
if (!Test::More->builder->is_passing) {
$dbh->sqlite_backup_to_file("failed.db");
}

22. Load into an in-
memory database.
my $dbh = DBI->connect('dbi:SQLite::memory:‘...);
$dbh->sqlite_backup_from_file("backup.db");

23. CAVEATS
 Loading time may matter.
 Everything is REPLACED (naturally).
 Just copying a file may suffice.
 Shared (read) lock helps.
 http://www.sqlite.org/backup.html

24. Recipe 3
Alter Table

25. SQLite only supports
"ADD COLUMN".
my $dbh = DBI->connect(...);
$dbh->do("ALTER TABLE foo ADD COLUMN new");
# and renaming a table

26. Use a temporary
table if you want
more.

27. Remove a column (col2)
$dbh->begin_work;
eval {
$dbh->do("CREATE TEMP TABLE t (col1, col3)");
$dbh->do("INSERT INTO t SELECT col1, col3 FROM foo");
$dbh->do("DROP TABLE foo");
$dbh->do("CREATE TABLE foo (col1, col3)");
$dbh->do("INSERT INTO foo SELECT col1, col3 FROM t");
$dbh->do("DROP TABLE t");
};
!$@ ? $dbh->commit : $dbh->rollback;

28. Rename a column
(col3 -> col4)
$dbh->begin_work;
eval {
$dbh->do("CREATE TEMP TABLE t (col1, col4)");
$dbh->do("INSERT INTO t SELECT col1, col3 FROM foo");
$dbh->do("DROP TABLE foo");
$dbh->do("CREATE TABLE foo (col1, col4)");
$dbh->do("INSERT INTO foo SELECT col1, col4 FROM t");
$dbh->do("DROP TABLE t");
};
!$@ ? $dbh->commit : $dbh->rollback;

29. NOTE
 ACIDity matters here.
 A temporary table is reasonably fast.
 Want some wrapper?
 http://sqlite.org/faq.html#q11

30. Recipe 4
Attach Database

31. One big file with
all tables
=
Easier to copy/move.

32. One big file with
all tables
=
Easier to be locked.

33. Splitting a
database into
smaller databases
may help.

34. ATTACH databases
if you need to join.
$dbh->do("ATTACH DATABASE 'sub.db' AS sub");

35. You can ATTACH
in-memory databases.
$dbh->do("ATTACH DATABASE ':memory:' AS mem");

36. CAVEATS
 ATTACH before you begin a transaction.
 Beware SQLITE_MAX_ATTACHED
(default: 10)
 Split a database if concurrency really
matters.
 Or use better server/client databases.

37. Recipe 5
SQLite Hooks

38. May be a problem.
my $select =
"SELECT id FROM foo WHERE status = 0 LIMIT 1";
my $update = "UPDATE foo SET status = 1 WHERE id = ?";
my ($id) = $dbh->selectrow_array($select);
$dbh->do($update, undef, $id);

39. Update first to
get a write lock.
my $update = q{
UPDATE foo SET status = 1
WHERE id = (
SELECT id FROM foo
WHERE status = 0 LIMIT 1
)
};

40. What should we do
to find an updated
row?
# $dbh->last_insert_id doesn't work

41. "update_hook" may help.
my ($action, $database, $table, $rowid);
$dbh->sqlite_update_hook(sub {
($action, $database, $table, $rowid) = @_;
...
# you can't do anything that modifies
# the database connection (incl. "prepare")
});
$dbh->do($update);

42. Retreive the row
with the id.
my $stmt = "SELECT * FROM foo WHERE ROWID = ?";
my $row = $dbh->selectrow_arrayref(
$stmt, undef, $rowid
);

43. NOTE
 Also sqlite_(commit|rollback)_hook
 One hook per connection
 Optional "UPDATE ... LIMIT" clause?
 SQLITE_ENABLE_UPDATE_DELETE_LIMIT
 Amalgamated source doesn't support this
 (because part of the source needs to be
regenerated).

44. Recipe 6
Unicode

45. You can use unicode
column/table names.
$dbh->do(‘
create table テーブル
(カラム,カラム,カラム)
');

46. Basic rules
 decode what you get from a
database
 encode what you put into a
database

47. sqlite_unicode => 1
 what you get will be
decoded
 what you put will be ...

48. Summary
 Don't need to care usually.
 Be careful when you use
_info() methods.
 More work may be needed.

49. Recipe 7
Get More Information

50. Supported *_info() methods
 my $sth = $dbh->table_info(...);
 my $sth = $dbh->column_info(...);
 my $sth = $dbh->primary_key_info(...);
 my $sth = $dbh->foreign_key_info(...);
(since 1.38_01)

51. table_info()
 to get table names
(accepts wild cards)

52. column_info()
 to get column names
(accepts wild cards)
 to see if a column is nullable
 to get type definition

53. primary_key_info()
 to get primary key names

54. foreign_key_info()
 to get foreign key names
 or referred tables

55. SQLite Pragmata to
get information
 PRAGMA database_list
 PRAGMA table_info('table')
 PRAGMA foreign_key_list('table')

56. To get info of
attached databases.
 PRAGMA db.table_info('table')
 PRAGMA db.foreign_key_list('table')

57. System tables
SELECT * FROM sqlite_master;
SELECT * FROM sqlite_temp_master;
# SQLs stored in system tables are not
always what you used to create tables

58. Recipe 8
Troubleshooting

59. sqlite_trace()
$dbh->sqlite_trace(sub {
my ($stmt) = @_;
say $stmt;
});

60. bound params are
embedded.
$dbh->do(q{
INSERT INTO foo VALUES (?)
}, undef, 1);
# INSERT INTO foo VALUES ('1')

61. sqlite_profile()
$dbh->sqlite_profile(sub {
my ($stmt, $time) = @_;
say "$stmt (elapsed: $time ms)";
});

62. bound params are
not embedded.
$dbh->do(q{
INSERT INTO foo VALUES (?)
}, undef, 1);
# INSERT INTO foo VALUES (?)
(elapsed: 0 ms)

63. EXPLAIN
EXPLAIN QUERY PLAN
For interactive analysis
and troubleshooting only.

64. Check "SCAN TABLE"
without using indices.
my $stmt = "SELECT * FROM foo WHERE id = ?";
my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt");
$sth->execute(1);
while(my $plan = $sth->fetchrow_hashref) {
my $detail = $plan->{detail};
if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/)
{
...
}
}

65. In case you want to know
memory usage etc...
 my $status = DBD::SQLite::sqlite_status();
 my $status = $dbh->sqlite_db_status();
 my $status = $sth->sqlite_st_status();

66. Recipe 9
Full Text Search

67. SQLite supports
full text search.
 http://sqlite.org/fts3.html

68. "perl" tokenizer is
supported
$dbh->do('
CREATE VIRTUAL TABLE foo
USING fts3 (
content,
tokenize=perl "main::tokenizer"
)
');

69. my $mecab = Text::MeCab->new;
sub tokenizer { return sub {
my $node = $mecab->parse($_[0]);
my ($index, $pos) = (0, 0);
return sub {
my $token = $node->surface or return;
my $length = $node->length;
my $start = $pos;
my $end = $pos += $length;
$node = $node->next;
return ($token, $length, $start, $end, $index++);
};
}}
http://d.hatena.ne.jp/charsbar/20100828/1282937592

70. Search::Tokenizer
 word
 word_locale
 word_unicode
 unaccent

71. Want more?

72. Recipe 10
Custom Extensions

73. Before you use
extensions...
$dbh->sqlite_enable_load_extension(1);

74. Use load_extension()
function to load
$dbh->do(q{
SELECT load_extension('./ex.dll')
});

75. Limitation of
load_extension()
 Can't register other functions
from the extension.

76. Use sqlite_load_extension()
$dbh->sqlite_load_extension('./ex.dll');

77. How to write an extension
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1;
int sqlite3_extension_init(
sqlite3 *db,
char **error,
const sqlite3_api_routines *api
){
SQLITE_EXTENSION_INIT2(api);
/* do whatever you like */
return SQLITE_OK;
}

78. Use XS for more portability.
#include <EXTERN.h>
#include <perl.h>
#include <XSUB.h>
#include "sqlite3ext.h“
SQLITE_EXTENSION_INIT1;
int sqlite3_extension_init(...) {
SQLITE_EXTENSION_INIT2(api);
...
}
MODULE = DBD::SQLite::Extension PACKAGE
= DBD::SQLite::Extension
PROTOTYPES: DISABLE

79. Prepare Makefile.PL
use strict;
use warnings;
use ExtUtils::MakeMaker;
# Get installed header files
get_sqlite_header("sqlite3.h") or exit;
get_sqlite_header("sqlite3ext.h") or exit;
WriteMakefile(
NAME => 'DBD::SQLite::Extension',
CONFIGURE_REQUIRES => {
'File::ShareDir' => '1.0', # to get headers
'DBD::SQLite' => '1.38_01',
},
FUNCLIST => ['sqlite3_extension_init'],
DL_FUNCS => {'DBD::SQLite::Extension' => []},
);

80. It Works.
my $dll =
"./blib/arch/auto/DBD/SQLite/Extension/Extension.dll";
my $dbh = DBI->connect('dbi:SQLite::memory:');
$dbh->sqlite_enable_load_extension(1);
$dbh->sqlite_load_extension($dll);

81. NOTE
 Needs more tricks to make complex
extensions (malloc/free conflicts,
DBI macros etc)
 Use always the same headers as used
in DBD::SQLite
 "Using SQLite" (O’Reilly) helps.

82. Questions so far?

83. Issue 1
Transaction Mode

84. A controversial change
in DBD::SQLite 1.38_01
The default
transaction mode
becomes
"IMMEDIATE".

85. Deferred Transaction
 SQLite default
 Maximum concurrency
 May cause a deadlock (multiple
clients in transactions wait for
others releasing their locks)

86. Immediate Transaction
 DBD::SQLite’s current default
 Immediately reserve a write lock
 No deadlock (if you can begin a
transaction, you'll most probably be
able to commit either).

87. Why changed?
 by request (#56444, in April 2010)
(other reports on locking issues:
#42205, #46289 (both in 2009))
 to avoid unexpected errors/deadlocks
while testing.
 ditto for smaller web applications.
 After all, who uses DBD::SQLite most?

88. Use "Deferred" when
you know a database
is almost read-only.

89. To defer a transaction
$dbh->{sqlite_use_immediate_transaction} = 0;
$dbh->do("BEGIN");

90. Issue 2
Refactoring of
"execute"

91.  Was suggested to move
parameter binding to
sqlite_bind_ph
 Broke tests / Would change
behaviors
 Not sure if it's worth trying
 begin_work

92. Issue 3
Optimization

93.  Asked to use the same
optimization as perl.
 Reverted as it broke an
R-Tree test (floating
point issue).

94. Issue 4
Compiler Issues

95. Anyone using... ?
 AIX 6.1
 Sun C compiler

96. Issue 5
iOS

97.  A reporter says
some databases
are said to be
"encrypted"...

98. Issue 6
Better Errors

99.  sqlite3_extended_errcode
 sqlite3_extended_result_codes

100. Issue 7
Type Information

101.  $dbh->type_info(_all) is not
implemented yet.
 $sth->{TYPE} returns an array
ref of strings. (Not conformed
to the DBI spec.)
 Dynamic typing / Type affinity

102. Issue 8
Async IO

103.  Nice to have.
 Not amalgamated.
 Would need significant
refactoring.
 Separate distribution?

104. Issue 9
Cookbook

105.  Almost forgotten.
 Maybe these
slides help.
 Suggestions?

106. Issue 10
SQLite 4

107. SQLite4 is an alternative,
not a replacement, for
SQLite3.
- http://sqlite.org/src4/doc/trunk/www/design.wiki

108. Still in its
earliest stage.
 No downloadable packages.
 Not enough features yet.

109. Wanna try?
 fossil clone http://www.sqlite.org/src4
sqlite4.fossil
 mkdir sqlite4 && cd sqlite4
 fossil open ../sqlite4.fossil
 you probably need to tweak Makefile (or
apply proper compile options)

110. DBD::SQLite4?
 Will probably be needed eventually
 Should share the same sqlite_
prefix?
 Able to share the common part?

111. Questions?

112. Thank you