ESUG 2011, Edinburgh

1. Mariano Martinez Peck
marianopeck@gmail.com
http://marianopeck.wordpress.com/
1
Monday, August 22, 2011

2. THANKS a lot!!!
SummerTalk 2011
Student: Martin Dias
Mentor: Mariano Martínez Peck
2
Monday, August 22, 2011

3. Object references
3
Monday, August 22, 2011

4. Object graph
4
Monday, August 22, 2011

5. An objet graph serializer.
5
Monday, August 22, 2011

6. Serialize
Input: an object graph
Output: stream of bytes
6
Monday, August 22, 2011

7. Materialize
(deserialize)
Input: stream of bytes
Output: an object graph
7
Monday, August 22, 2011

8. Once serialized...
Stream of bytes
Database File Memory Socket
8
Monday, August 22, 2011

9. Fuel’s main goals
Provide fast object serialization and materialization.
Be flexible and easy to customize.
Have a good OO design, well tested and
benchmarked.
No need of special support from the VM.
Be a general purpose serializer.
Allow tools to be built on top of Fuel.
9
Monday, August 22, 2011

10. Key features
Fast serialization and materialization.
Class reshape support.
Serialization of any kind of object.
Cycles support.
Global objects references.
Buffered writing.
Support for some “hook methods”.
10
Monday, August 22, 2011

11. Key Characteristics
Pickle format.
Objects grouped in clusters.
Analysis phase before writing.
Stack over recursion.
Two phases for writing instances and
references.
Iterative graph recreation.
11
Monday, August 22, 2011

12. Pickle format
Invest more time in serialization so that objects can then be
materialized much faster.
12
Monday, August 22, 2011

13. Grouping objects in
clusters
“Similar” objects (they share writing/loading information) are
grouped together in clusters. The most common case, yet not the
only one, takes place when a class is a cluster for its instances.
13
Monday, August 22, 2011

14. 14
Monday, August 22, 2011

15. Each jar has
a specific
type of
element
14
Monday, August 22, 2011

16. Each jar has
a specific
type of
element
Jars are in
order
14
Monday, August 22, 2011

17. Label:
- What’s inside?
Each jar has - How much?
a specific
type of
element
Jars are in
order
14
Monday, August 22, 2011

18. Label:
- What’s inside?
Each jar has - How much?
a specific
type of
element
Different sizes
and different
Jars are in amounts of
order elements
14
Monday, August 22, 2011

19. s/jar/cluster
Label:
- What’s inside?
Each jar has - How much?
a specific
type of
element
Different sizes
and different
Jars are in amounts of
order elements
14
Monday, August 22, 2011

20. s/jar/cluster
Label:
- What’s inside?
Each jar has - How much?
a specific
type of
element
Different sizes
and different
Jars are in amounts of
order elements
14
Monday, August 22, 2011

21. s/jar/cluster
Label:
- What’s inside?
- How much?
Each cluster
has a specific
type of object
Different sizes
and different
Jars are in amounts of
order elements
14
Monday, August 22, 2011

22. s/jar/cluster
Label:
- What’s inside?
- How much?
Each cluster
has a specific
type of object
Different sizes
and different
Jars are in amounts of
order elements
14
Monday, August 22, 2011

23. s/jar/cluster
Label:
- What’s inside?
- How much?
Each cluster
has a specific
type of object
Different sizes
and different
Clusters are amounts of
in order elements
14
Monday, August 22, 2011

24. s/jar/cluster
Label:
- What’s inside?
Each cluster - How much?
has a specific
type of object
Different sizes
and different
Clusters are amounts of
in order elements
14
Monday, August 22, 2011

25. s/jar/cluster
Label:
- Cluster ID
Each cluster - Amount of
has a specific objects
type of object
Different sizes
and different
Clusters are amounts of
in order elements
14
Monday, August 22, 2011

26. s/jar/cluster
Label:
- Cluster ID
Each cluster - Amount of
has a specific objects
type of object
Different sizes
and different
Clusters are amounts of
in order elements
14
Monday, August 22, 2011

27. s/jar/cluster
Label:
- Cluster ID
Each cluster - Amount of
has a specific objects
type of object
Different sizes
and different
Clusters are amounts of
in order objects
14
Monday, August 22, 2011

28. Pickle format basic
Stream
15
Monday, August 22, 2011

29. Pickle format basic
Stream
15
Monday, August 22, 2011

30. Pickle format basic
Stream
15
Monday, August 22, 2011

31. Pickle format basic
Stream
15
Monday, August 22, 2011

32. Why the pickle format is so fast in materialization?
Standard serializers Fuel pickle format
16
Monday, August 22, 2011

33. Why the pickle format is so fast in materialization?
Standard serializers Fuel pickle format
ive ive n
rat tio
urs tion
ec za Ite iza
R ial
eri ali at er
m at m
16
Monday, August 22, 2011

34. Pickle advantages
Batch/Bulk/Iterative materialization.
Efficient since types are stored and fetch only
once.
Fast because at materialization we know the
size of everything.
The generated stream is smaller.
More next....
17
Monday, August 22, 2011

35. There is no silver
bullet...
ation
rializ e)
Fa st se ickl
(with out p
Slow
seri
(wit aliza
h pi tion
ckle
)
18
Monday, August 22, 2011

36. Fuel requires
Traversing the object graph.
Mapping each object to a specific cluster.
This is done in a phase before
serialization called “Analysis”.
19
Monday, August 22, 2011

37. Analysis phase
1) Traverse
(#fuelAccept: aVisitor)
2) Fill dictionary
(#fuelSerializer)
cluster1 aPoint ...
cluster2 x y ...
... ...
key (a cluster) value (a set)
20
Monday, August 22, 2011

38. Serialization
cluster1 aPoint ...
cluster2 x y ...
... ...
key (a cluster) value (a set)
Cluster
ID
serialize: anObject on: aStream
materializeFrom: aStream
A cluster defines how its objects
are serialized and materialized.
21
Monday, August 22, 2011

39. Stack over recursion
To traverse the object graph, Fuel uses a custom
stack implementation rather than a recursion.
22
Monday, August 22, 2011

40. Basic steps
Serialization
1. Analyze.
2. Serialize header.
3. Serialize instances.
4. Serialize references.
5. Serialize root.
Materialization
1. Materialize header.
2. Materialize instances.
3. Materialize references.
4. Materialize root.
23
Monday, August 22, 2011

41. Fuel for far)
(so
software
Moose export utility.
SandstoneDB persistence.
Pier kernel persistence.
Newspeak language.
Marea (my own research project!).
24
Monday, August 22, 2011

42. Future Work
Continue efforts on performance optimization.
Create a tool for loading class and trait
packages.
Support user-defined Singletons.
Fast statistics/brief info extraction of a stored
graph.
Partial loading of a stored graph.
25
Monday, August 22, 2011

43. Future work 2
Enable to deploy serialization and
materialization behavior independently.
Support object replacement for serialization
and materialization.
Allow cycle detections to be disabled.
Partial loading.
26
Monday, August 22, 2011

44. Serialization of primitive objects
Memory based stream
27
Monday, August 22, 2011

45. Serialization of primitive objects
Memory based stream
StOMP
SRP
Fuel 27
Monday, August 22, 2011

46. Serialization of primitive objects
Memory based stream File based stream
StOMP
SRP
Fuel 27
Monday, August 22, 2011

47. Serialization of primitive objects
Memory based stream File based stream
StOMP Fuel
SRP SRP
Fuel 27 StOMP
Monday, August 22, 2011

48. Materialization
of primitive objects
28
Monday, August 22, 2011

49. Materialization
of primitive objects
Fuel
StOMP
ImageSegment
28
Monday, August 22, 2011

50. Non primitive objects
29
Monday, August 22, 2011

51. Non primitive objects
StOMP
SRP
Fuel 29
Monday, August 22, 2011

52. Non primitive objects
StOMP
SRP
Fuel 29
Monday, August 22, 2011

53. Non primitive objects
StOMP Fuel
SRP ImageSegment
Fuel 29 StOMP
Monday, August 22, 2011

54. Links
Website: http://rmod.lille.inria.fr/web/pier/
software/Fuel
Issue tracker: http://code.google.com/p/fuel
Source repository: http://www.squeaksource.com/
Fuel
Continuous integration server: https://pharo-
ic.lille.inria.fr/hudson/job/Fuel/
30
Monday, August 22, 2011

55. Conclusion
for us
Excellent performance without special support
from VM and good OO design.
31
Monday, August 22, 2011

56. Conclusion
for YOU
Fuel is a vehicle. It is infrastructure. You can
build cool stuff on top of it.
32
Monday, August 22, 2011

57. Thanks!
Mariano Martinez Peck
marianopeck@gmail.com
http://marianopeck.wordpress.com/
Monday, August 22, 2011

58. Concrete example
34
Monday, August 22, 2011

59. 35
Monday, August 22, 2011

60. Analysis phase
Analysis cluster1 aRectangle
phase cluster2 anOrigin aCorner
cluster3 10 20 30 40
key (a cluster) value (a IdentitySet)
20
36
Monday, August 22, 2011

61. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner instances
cluster3 10 20 30 40 phase
Monday, August 22, 2011

62. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner instances
cluster3 10 20 30 40 phase
Monday, August 22, 2011

63. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner instances
cluster3 10 20 30 40 phase
instance variable count instance variable names
instance
cluster ID 1 Rectangle 2 origi corne 1 count
class name
Monday, August 22, 2011

64. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner instances
cluster3 10 20 30 40 phase
instance variable count instance variable names
instance
cluster ID 1 Rectangle 2 origi corne 1 count
class name
Monday, August 22, 2011

65. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner instances
cluster3 10 20 30 40 phase
instance variable count instance variable names
instance
cluster ID 1 Rectangle 2 origi corne 1 count
class name 1 Point 2 x y 2
Monday, August 22, 2011

66. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner instances
cluster3 10 20 30 40 phase
instance variable count instance variable names
instance
cluster ID 1 Rectangle 2 origi corne 1 count
class name 1 Point 2 x y 2
54 10 20 30 40
Monday, August 22, 2011

67. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner references
cluster3 10 20 30 40 phase
instancesIndex
aRectangle 20 40 aCorner anOrigi 10 30
(IdentityDictionary)
1 5 7 3 2 4 6
54 10 20 30 40
instances
references
Monday, August 22, 2011

68. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner references
cluster3 10 20 30 40 phase
instancesIndex
aRectangle 20 40 aCorner anOrigi 10 30
(IdentityDictionary)
1 5 7 3 2 4 6
54 10 20 30 40
instances
2 3
references
Monday, August 22, 2011

69. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner references
cluster3 10 20 30 40 phase
instancesIndex
aRectangle 20 40 aCorner anOrigi 10 30
(IdentityDictionary)
1 5 7 3 2 4 6
54 10 20 30 40
instances
2 3
references
4 5 6 7
Monday, August 22, 2011

70. cluster1 aRectangle Serialization
cluster2 anOrigin aCorner references
cluster3 10 20 30 40 phase
instancesIndex
aRectangle 20 40 aCorner anOrigi 10 30
(IdentityDictionary)
1 5 7 3 2 4 6
54 10 20 30 40
instances
2 3
references
4 5 6 7
Monday, August 22, 2011

71. final stream
{
1 Rectangle 2 origi corne 1
instances 1 Point 2 x y 2
54 10 20 30 40
references
{ 2 3
4 5 6 7
Monday, August 22, 2011

72. final stream
objectsCounts
header { 7 3 clustersCount
{
1 Rectangle 2 origi corne 1
instances 1 Point 2 x y 2
54 10 20 30 40
references
{ 2 3
4 5 6 7
trailer { 1 root
Monday, August 22, 2011

73. Materialization
objectsCounts
header { 7 3 clustersCount
1 Rectangle 2 origin corner 1
instances
{ 1 Point 2 x y 2
54 1 2 3 4
2 3
references {
4 5 6 7
trailer {
1 root
Monday, August 22, 2011

74. 41
Monday, August 22, 2011