The document describes the mechanism of msgpack::unpacker, which is a library for unpacking MessagePack serialized data in C++. It discusses two scenarios: 1) Receiving msgpack data in fixed length packets with a length header. The unpacker allocates a buffer based on the length, unpacks the data, and checks if more data is needed on each call. 2) Receiving a continuous msgpack stream without length headers. The unpacker allocates an initial buffer and resizes it as needed, unpacking data in chunks to avoid copying. It handles partially unpacked objects across buffer resizes. The internal structures of the unpacker are also outlined, including how it manages memory buffers

1. The mechanism of
msgpack::unpacker
msgpack-c committer
Takatoshi Kondo
17/16/2014 Copyright OGIS-RI Co., Ltd.

2. Overview
Scenario 1
2
length
The length of the
“length” field is fixed size
length
msgpack msgpack msgpack msgpack
Scenario 2
Receiving a fixed length msgpack packets
msgpack packets with no length information (e.g. streaming)
msgpack msgpack msgpack msgpack
7/16/2014 Copyright OGIS-RI Co., Ltd.

3. Overview
Client
msgpack::unpacker
• Client receives msgpack data asynchronously.
• Data is separated as packets.
• The packet format is as follows:
length payload(msgpack format )
2. receive “length”
3.1. allocate length bytes memory
3.reserve_buffer(length)
unpacker may allocate more
than “length” bytes for
optimization.
1.create
4.buf = buffer()
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
8.1. set unpacked contents
9.obj = get()
10.z = zone()
3
Scenario 1
7/16/2014 Copyright OGIS-RI Co., Ltd.
6.buffer_consumed(received)
5. reveived = receive payload into buf

4. Overview
Client
msgpack::unpacker
3.reserve_buffer(length)
1.create
4.buf = buffer()
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
9.obj = get()
10.z = zone()
do application
processes using obj
and z
47/16/2014 Copyright OGIS-RI Co., Ltd.
• Client receives msgpack data asynchronously.
• Data is separated as packets.
• The packet format is as follows:
Scenario 1
length payload(msgpack format )
2. receive “length”
3.1. allocate length bytes memory
8.1. set unpacked contents
6.buffer_consumed(received)
5. reveived = receive payload into buf

5. Overview
Client
msgpack::unpacker
3.reserve_buffer(length)
1.create
4.buf = buffer()
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
9.obj = get()
10.z = zone()
loop[bRet==true]
next() returns true while an
object is unpacked.
57/16/2014 Copyright OGIS-RI Co., Ltd.
• Client receives msgpack data asynchronously.
• Data is separated as packets.
• The packet format is as follows:
Scenario 1
length payload(msgpack format )
2. receive “length”
3.1. allocate length bytes memory
8.1. set unpacked contents
6.buffer_consumed(received)
5. reveived = receive payload into buf

6. loop[bRet==true]
Overview
Client
msgpack::unpacker
3.reserve_buffer(length)
1.create
4.buf = buffer()
6.buffer_consumed(received)
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
9.obj = get()
10.z = zone()
loop
infinite loop for data receiving
67/16/2014 Copyright OGIS-RI Co., Ltd.
• Client receives msgpack data asynchronously.
• Data is separated as packets.
• The packet format is as follows:
Scenario 1
length payload(msgpack format )
2. receive “length”
3.1. allocate length bytes memory
5. reveived = receive payload into buf
8.1. set unpacked contents

7. loop
[total received
< length ]
loop[bRet==true]
Overview
Client
msgpack::unpacker
3.reserve_buffer(length)
1.create
4.buf = buffer()
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
9.obj = get()
10.z = zone()
When payoads are received
intermittently, this loop is effective.
If msgpack object is incomplete and additional receive is
required, the return false.
The position of buf is advanced to
received size.
77/16/2014 Copyright OGIS-RI Co., Ltd.
• Client receives msgpack data asynchronously.
• Data is separated as packets.
• The packet format is as follows:
Scenario 1
length payload(msgpack format )
6.buffer_consumed(received)
2. receive “length”
3.1. allocate length bytes memory
5. reveived = receive payload into buf
8.1. set unpacked contents

8. loop
loop[bRet==true]
Overview
Client
msgpack::unpacker
3.reserve_buffer(length)
1.create
4.buf = buffer()
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
9.obj = get()
10.z = zone()
8
The combination of each loop.
7/16/2014 Copyright OGIS-RI Co., Ltd.
• Client receives msgpack data asynchronously.
• Data is separated as packets.
• The packet format is as follows:
Scenario 1
length payload(msgpack format )
6.buffer_consumed(received)
2. receive “length”
3.1. allocate length bytes memory
5. reveived = receive payload into buf
8.1. set unpacked contents
loop
[total received
< length ]

9. payload (msgpack format variable length data)
loop
loop
[cap > threashold]
loop[bRet==true]
Overview
Client
msgpack::unpacker
msgpack packets with no length information (e.g. streaming)
2.1. allocate at least N bytes memory
2.reserve_buffer(specific value N)
1.create
3.buf = buffer()
5.R=copy at most “cap” size data into buf
6.buffer_consumed(R)
msgpack::unpacked
7. unp = create
8.bRet = next(unp)
9.obj = get()
10.z = zone()
R is an actual received size
e.g. N=1024
4.cap = buffer_capacity() cap >= N
9
Scenario 2
while cap is enough,
continue receiving
7/16/2014 Copyright OGIS-RI Co., Ltd.
8.1. set unpacked contents

10. Internal Structure
memory management of unpacker
unpacker
char* m_buffer
size_t m_used
size_t m_free
size_t m_off
size_t m_parsed
size_t m_initial_buffer_size
zone m_z
context m_ctx
char* buffer()
size_t buffer_capacity()
void buffer_consumed(size)
bool next()
...
m_buffer+ m_used
m_free
m_used += size
m_free += size
10
context unpack_stack
unsigned int m_top
unpack_stack m_stack[]
...
undsigned int m_container_type
object m_obj
size_t m_count
...
7/16/2014 Copyright OGIS-RI Co., Ltd.
https://github.com/msgpack/msgpack-c/tree/poc/0.6
The names of data members are based on the
following branch:

11. Internal Structure
m_parsed = 0
m_buffer
COUNTER_SIZE
The start bytes of m_buffer is used as a reference
counter
constructor(init_size=default_init_size)
If init_size is less than COUNTER_SIZE, then init_size=COUNTER_SIZE.
1
m_used = COUTNER_SIZE
m_free = init_size - m_used
m_off = COUNTER_SIZE
construct
m_buffer m_buffer+m_used
m_m_initial_buffer_size = init_size
117/16/2014 Copyright OGIS-RI Co., Ltd.
unpacker
char* m_buffer
size_t m_used
size_t m_free
size_t m_off
size_t m_parsed
size_t m_initial_buffer_size
zone m_z
context m_ctx
char* buffer()
size_t buffer_capacity()
void buffer_consumed(size)
bool next()
...
m_buffer+ m_used
m_free
m_used += size
m_free += size
memory management of unpacker

12. received data1
Internal Structure
m_parsed = 0
Write received data into the memory that is returned
by buffer()
1
m_used = COUTNER_SIZE
m_free = init_size - m_used
m_off = COUNTER_SIZE
Writing received data
m_buffer m_buffer+m_used
m_initial_buffer_size = init_size
received data1
m_parsed = 0
m_used += size; m_free -= size;
1
m_free
m_off = COUNTER_SIZE
buffer_consumed(size)
m_buffer
m_initial_buffer_size = init_size
m_buffer+m_used
msgpack(array, size=3)
bool uint32 string
received data1
num
number of array
elements
127/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker

13. uint32
boolnum
num
Internal Structure
received data1
m_parsed = 0
Parse from m_buffer+m_off
1
m_free
next()
m_buffer m_buffer + m_usedm_buffer+m_off
received data1
Parse “num”, then store object into the top level of m_ctx.
Advance m_off, increase m_parsed.
1
m_buffer m_buffer + m_off
In order to parse the elements of array, decrement an
internal level of the m_ctx. Parse “bool” and set object
on m_ctx. Advance m_off, increase m_parsed.
msgpack(array, size=3)
bool uint32 string
received data1
num
boolnum
received data11
m_buffer m_buffer + m_off
There are not enough data to parse“uint32”, then return
false. m_off and m_parsed are not updated.
initialize the top level count of
the m_ctx with num
decrement the top level count of
the m_ctx
num
m_parsed
num
m_parsed
bool
13
“num” field means that this is an
array object that has three elements.
array size=numm_stack
m_top
m_stack
m_top
Array structure processes recursively. In order
to implement these processes as loop,
unpacker uses a stack structure.
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker

14. uint32boolnum
uint32boolnum
received data2
Internal Structure
received data1
Do nothing because m_free >= size.
1
m_free
reserve_buffer(size)
m_buffer m_buffer + m_usedm_buffer + m_off
m_used += size; m_free -= size;
buffer_consumed(size)
received data2received data11
m_free
m_buffer m_buffer + m_used
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
m_buffer+ m_off
num
m_parsed
bool
14
m_stack
m_top
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker
Writing received data
Write received data into the memory that is returned by buffer()

15. uint32uint32boolnum
m_parsed
uint32boolnum
m_buffer + m_off
m_parsed
string
Internal Structure
received data2received data11
m_free
m_buffer m_buffer + m_usedm_buffer + m_off
next()
received data2received data11
m_free
m_buffer m_buffer + m_used
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
num
m_parsed
bool
15
m_stack
m_top
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker
Parse from m_buffer+m_off
Parse “uint32”, then store object into m_ctx.
Advance m_off, increase m_parsed.
There are not enough data to parse“string”, then return
false. m_off and m_parsed are not updated.

16. 1
string
not parsed data
stringuint32boolnum
Internal Structure
m_free < size then allocate a new buffer.
reserve_buffer(size)
received data2received data1
m_free
m_buffer m_buffer+m_used
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
m_buffer+m_off
received data3
next_size = m_initial_buffer_size
m_initial_buffer_size
not_parsed
not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer.
1
COUNTER_SIZE
While next_size is less than size + not_parsed + COUNTER_SIZE,
next_size *= 2
m_buffer+m_usedm_buffer+m_off
m_free
Update m_off, m_used, m_free as follows:
m_off = COUNTER_SIZE
m_used = not_parsed + COUNTER_SIZE
m_free = next_size - m_used
Decrement the previous buffer’s reference count and if it is zero,
then free it.
uint32bool
num
m_parsed
m_parsed is not updated
16
Copy the not parsed data to newly allocated buffer.
m_stack
m_top
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker

17. 10
string
not parsed data
stringuint32boolnum
Internal Structure
m_free < size then allocate a new buffer.
reserve_buffer(size)
received data2received data1
m_free
m_buffer m_buffer+m_used
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
m_buffer+m_off
received data3
next_size = m_initial_buffer_size
m_initial_buffer_size
not_parsed
not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer.
1
COUNTER_SIZE
While next_size is less than size + not_parsed + COUNTER_SIZE,
next_size *= 2
m_buffer+m_usedm_buffer+m_off
m_free
Update m_off, m_used, m_free as follows:
m_off = COUNTER_SIZE
m_used = not_parsed + COUNTER_SIZE
m_free = next_size - m_used
Decrement the previous buffer’s reference count and if it is zero,
then free it.
uint32bool
num
m_parsed
m_parsed is not updated
17
Copy the not parsed data to newly allocated buffer.
m_stack
m_top
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker

18. string
not parsed data
Internal Structure
m_free < size then allocate a new buffer.
reserve_buffer(size)
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
received data3
next_size = m_initial_buffer_size
not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer.
1
COUNTER_SIZE
While next_size is less than size + not_parsed + COUNTER_SIZE,
next_size *= 2
m_buffer+m_usedm_buffer+m_off
m_free
Update m_off, m_used, m_free as follows:
m_off = COUNTER_SIZE
m_used = not_parsed + COUNTER_SIZE
m_free = next_size - m_used
Decrement the previous buffer’s reference count and if it is zero,
then free it.
uint32bool
num
m_parsed
m_parsed is not updated
18
Copy the not parsed data to newly allocated buffer.
m_stack
m_top
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker

19. 1
received data3
string
not parsed data
Internal Structure
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
received data3
1
COUNTER_SIZE
m_buffer+m_usedm_buffer+m_off
m_free
m_used += size; m_free -= size;
buffer_consumed(size)
received data3
string
not parsed data1
COUNTER_SIZE
m_buffer+m_usedm_buffer+m_off
m_free
next()
uint32bool
num
m_parsed
received data3
string
not parsed data
COUNTER_SIZE
m_buffer+m_used
m_buffer+m_off
m_free
Decrement the top level count
of m_ctx
string
Because of the top level count of m_ctx is zero, msgpack object has
fully constructed. As the result of that next() returns true.
19
m_stack
m_top
array size=num
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker
Writing received data
Parse from m_buffer+m_off
Write received data into the memory that is
returned by buffer()
Parse “string”, then store object into m_ctx.
Advance m_off, increase m_parsed.

20. 12
received data3
string
not parsed data
Internal Structure
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
received data3
1
COUNTER_SIZE
m_buffer+m_usedm_buffer+m_off
m_free
m_used += size; m_free -= size;
buffer_consumed(size)
received data3
string
not parsed data1
COUNTER_SIZE
m_buffer+m_usedm_buffer+m_off
m_free
next()
uint32bool
num
m_parsed
received data3
string
not parsed data
COUNTER_SIZE
m_buffer+m_used
m_buffer+m_off
m_free
Decrement the top level count
of m_ctx
string
Because of the top level count of m_ctx is zero, msgpack object has
fully constructed. As the result of that next() returns true.
When the msgpack object type is string or bin,
increment the reference counter
20
m_stack
m_top
array size=num
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker
Writing received data
Parse from m_buffer+m_off
Write received data into the memory that is
returned by buffer()
Parse “string”, then store object into m_ctx.
Advance m_off, increase m_parsed.

21. 12
received data3
string
not parsed data
Internal Structure
msgpack(array, size=3)
bool uint32 string
received data1
num
received data2
size
received data3
1
COUNTER_SIZE
m_buffer+m_usedm_buffer+m_off
m_free
m_used += size; m_free -= size;
buffer_consumed(size)
received data3
string
not parsed data1
COUNTER_SIZE
m_buffer+m_usedm_buffer+m_off
m_free
next()
uint32bool
num
m_parsed
received data3
string
not parsed data
COUNTER_SIZE
m_buffer+m_used
m_buffer+m_off
m_free
Decrement the top level count
of m_ctx
string
Because of the top level count of m_ctx is zero, msgpack object has
fully constructed. As the result of that next() returns true.
When the msgpack object type is string or bin,
increment the reference counter
Instead of copy, msgpack object has
a reference of string or bin in the
buffer.
21
m_stack
m_top
array size=num
7/16/2014 Copyright OGIS-RI Co., Ltd.
memory management of unpacker
Writing received data
Parse from m_buffer+m_off
Write received data into the memory that is
returned by buffer()
Parse “string”, then store object into m_ctx.
Advance m_off, increase m_parsed.

22. zone, unpacker, context, object
22
client
unpacker
create and request unpack
7/16/2014 Copyright OGIS-RI Co., Ltd.

23. zone, unpacker, context, object
23
client
unpacker
zone
context
buffer
create
createcreate
7/16/2014 Copyright OGIS-RI Co., Ltd.
create and request unpack

24. zone, unpacker, context, object
24
client
unpacker
zone
context object
buffer
create
createcreate
reference
object(bin,string)
create
place the object on the zone
place the object on the zonemanaged by a reference count
7/16/2014 Copyright OGIS-RI Co., Ltd.
create and request unpack

25. zone, unpacker, context, object
25
client
unpacker
zone
context object
buffer
create
createcreate
reference
object(bin,string)
create
acquire the ownership from the unpacker
7/16/2014 Copyright OGIS-RI Co., Ltd.
create and request unpack

26. zone, unpacker, context, object
26
client
zone
object
buffer
reference
object(raw,string)
7/16/2014 Copyright OGIS-RI Co., Ltd.
acquire the ownership from the unpacker