Unpack mechanism of the msgpack-c

1. The mechanism of msgpack::unpacker msgpack-c committer Takatoshi Kondo 17/16/2014 Copyright OGIS-RI Co., Ltd.

2. Overview Scenario 1 2 length The length of the “length” field is fixed size length msgpack msgpack msgpack msgpack Scenario 2 Receiving a fixed length msgpack packets msgpack packets with no length information (e.g. streaming) msgpack msgpack msgpack msgpack 7/16/2014 Copyright OGIS-RI Co., Ltd.

3. Overview Client msgpack::unpacker • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 3.reserve_buffer(length) unpacker may allocate more than “length” bytes for optimization. 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 8.1. set unpacked contents 9.obj = get() 10.z = zone() 3 Scenario 1 7/16/2014 Copyright OGIS-RI Co., Ltd. 6.buffer_consumed(received) 5. reveived = receive payload into buf

4. Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() do application processes using obj and z 47/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 8.1. set unpacked contents 6.buffer_consumed(received) 5. reveived = receive payload into buf

5. Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() loop[bRet==true] next() returns true while an object is unpacked. 57/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 8.1. set unpacked contents 6.buffer_consumed(received) 5. reveived = receive payload into buf

6. loop[bRet==true] Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() 6.buffer_consumed(received) msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() loop infinite loop for data receiving 67/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 2. receive “length” 3.1. allocate length bytes memory 5. reveived = receive payload into buf 8.1. set unpacked contents

7. loop [total received < length ] loop[bRet==true] Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() When payoads are received intermittently, this loop is effective. If msgpack object is incomplete and additional receive is required, the return false. The position of buf is advanced to received size. 77/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 6.buffer_consumed(received) 2. receive “length” 3.1. allocate length bytes memory 5. reveived = receive payload into buf 8.1. set unpacked contents

8. loop loop[bRet==true] Overview Client msgpack::unpacker 3.reserve_buffer(length) 1.create 4.buf = buffer() msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() 8 The combination of each loop. 7/16/2014 Copyright OGIS-RI Co., Ltd. • Client receives msgpack data asynchronously. • Data is separated as packets. • The packet format is as follows: Scenario 1 length payload(msgpack format ) 6.buffer_consumed(received) 2. receive “length” 3.1. allocate length bytes memory 5. reveived = receive payload into buf 8.1. set unpacked contents loop [total received < length ]

9. payload (msgpack format variable length data) loop loop [cap > threashold] loop[bRet==true] Overview Client msgpack::unpacker msgpack packets with no length information (e.g. streaming) 2.1. allocate at least N bytes memory 2.reserve_buffer(specific value N) 1.create 3.buf = buffer() 5.R=copy at most “cap” size data into buf 6.buffer_consumed(R) msgpack::unpacked 7. unp = create 8.bRet = next(unp) 9.obj = get() 10.z = zone() R is an actual received size e.g. N=1024 4.cap = buffer_capacity() cap >= N 9 Scenario 2 while cap is enough, continue receiving 7/16/2014 Copyright OGIS-RI Co., Ltd. 8.1. set unpacked contents

10. Internal Structure memory management of unpacker unpacker char* m_buffer size_t m_used size_t m_free size_t m_off size_t m_parsed size_t m_initial_buffer_size zone m_z context m_ctx char* buffer() size_t buffer_capacity() void buffer_consumed(size) bool next() ... m_buffer+ m_used m_free m_used += size m_free += size 10 context unpack_stack unsigned int m_top unpack_stack m_stack[] ... undsigned int m_container_type object m_obj size_t m_count ... 7/16/2014 Copyright OGIS-RI Co., Ltd. https://github.com/msgpack/msgpack-c/tree/poc/0.6 The names of data members are based on the following branch:

11. Internal Structure m_parsed = 0 m_buffer COUNTER_SIZE The start bytes of m_buffer is used as a reference counter constructor(init_size=default_init_size) If init_size is less than COUNTER_SIZE, then init_size=COUNTER_SIZE. 1 m_used = COUTNER_SIZE m_free = init_size - m_used m_off = COUNTER_SIZE construct m_buffer m_buffer+m_used m_m_initial_buffer_size = init_size 117/16/2014 Copyright OGIS-RI Co., Ltd. unpacker char* m_buffer size_t m_used size_t m_free size_t m_off size_t m_parsed size_t m_initial_buffer_size zone m_z context m_ctx char* buffer() size_t buffer_capacity() void buffer_consumed(size) bool next() ... m_buffer+ m_used m_free m_used += size m_free += size memory management of unpacker

12. received data1 Internal Structure m_parsed = 0 Write received data into the memory that is returned by buffer() 1 m_used = COUTNER_SIZE m_free = init_size - m_used m_off = COUNTER_SIZE Writing received data m_buffer m_buffer+m_used m_initial_buffer_size = init_size received data1 m_parsed = 0 m_used += size; m_free -= size; 1 m_free m_off = COUNTER_SIZE buffer_consumed(size) m_buffer m_initial_buffer_size = init_size m_buffer+m_used msgpack(array, size=3) bool uint32 string received data1 num number of array elements 127/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker

13. uint32 boolnum num Internal Structure received data1 m_parsed = 0 Parse from m_buffer+m_off 1 m_free next() m_buffer m_buffer + m_usedm_buffer+m_off received data1 Parse “num”, then store object into the top level of m_ctx. Advance m_off, increase m_parsed. 1 m_buffer m_buffer + m_off In order to parse the elements of array, decrement an internal level of the m_ctx. Parse “bool” and set object on m_ctx. Advance m_off, increase m_parsed. msgpack(array, size=3) bool uint32 string received data1 num boolnum received data11 m_buffer m_buffer + m_off There are not enough data to parse“uint32”, then return false. m_off and m_parsed are not updated. initialize the top level count of the m_ctx with num decrement the top level count of the m_ctx num m_parsed num m_parsed bool 13 “num” field means that this is an array object that has three elements. array size=numm_stack m_top m_stack m_top Array structure processes recursively. In order to implement these processes as loop, unpacker uses a stack structure. 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker

14. uint32boolnum uint32boolnum received data2 Internal Structure received data1 Do nothing because m_free >= size. 1 m_free reserve_buffer(size) m_buffer m_buffer + m_usedm_buffer + m_off m_used += size; m_free -= size; buffer_consumed(size) received data2received data11 m_free m_buffer m_buffer + m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size m_buffer+ m_off num m_parsed bool 14 m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Write received data into the memory that is returned by buffer()

15. uint32uint32boolnum m_parsed uint32boolnum m_buffer + m_off m_parsed string Internal Structure received data2received data11 m_free m_buffer m_buffer + m_usedm_buffer + m_off next() received data2received data11 m_free m_buffer m_buffer + m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size num m_parsed bool 15 m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Parse from m_buffer+m_off Parse “uint32”, then store object into m_ctx. Advance m_off, increase m_parsed. There are not enough data to parse“string”, then return false. m_off and m_parsed are not updated.

16. 1 string not parsed data stringuint32boolnum Internal Structure m_free < size then allocate a new buffer. reserve_buffer(size) received data2received data1 m_free m_buffer m_buffer+m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size m_buffer+m_off received data3 next_size = m_initial_buffer_size m_initial_buffer_size not_parsed not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer. 1 COUNTER_SIZE While next_size is less than size + not_parsed + COUNTER_SIZE, next_size *= 2 m_buffer+m_usedm_buffer+m_off m_free Update m_off, m_used, m_free as follows: m_off = COUNTER_SIZE m_used = not_parsed + COUNTER_SIZE m_free = next_size - m_used Decrement the previous buffer’s reference count and if it is zero, then free it. uint32bool num m_parsed m_parsed is not updated 16 Copy the not parsed data to newly allocated buffer. m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker

17. 10 string not parsed data stringuint32boolnum Internal Structure m_free < size then allocate a new buffer. reserve_buffer(size) received data2received data1 m_free m_buffer m_buffer+m_used msgpack(array, size=3) bool uint32 string received data1 num received data2 size m_buffer+m_off received data3 next_size = m_initial_buffer_size m_initial_buffer_size not_parsed not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer. 1 COUNTER_SIZE While next_size is less than size + not_parsed + COUNTER_SIZE, next_size *= 2 m_buffer+m_usedm_buffer+m_off m_free Update m_off, m_used, m_free as follows: m_off = COUNTER_SIZE m_used = not_parsed + COUNTER_SIZE m_free = next_size - m_used Decrement the previous buffer’s reference count and if it is zero, then free it. uint32bool num m_parsed m_parsed is not updated 17 Copy the not parsed data to newly allocated buffer. m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker

18. string not parsed data Internal Structure m_free < size then allocate a new buffer. reserve_buffer(size) msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 next_size = m_initial_buffer_size not_parsed = m_used - m_off The data that is not parsed yet in the previous buffer. 1 COUNTER_SIZE While next_size is less than size + not_parsed + COUNTER_SIZE, next_size *= 2 m_buffer+m_usedm_buffer+m_off m_free Update m_off, m_used, m_free as follows: m_off = COUNTER_SIZE m_used = not_parsed + COUNTER_SIZE m_free = next_size - m_used Decrement the previous buffer’s reference count and if it is zero, then free it. uint32bool num m_parsed m_parsed is not updated 18 Copy the not parsed data to newly allocated buffer. m_stack m_top 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker

19. 1 received data3 string not parsed data Internal Structure msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free m_used += size; m_free -= size; buffer_consumed(size) received data3 string not parsed data1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free next() uint32bool num m_parsed received data3 string not parsed data COUNTER_SIZE m_buffer+m_used m_buffer+m_off m_free Decrement the top level count of m_ctx string Because of the top level count of m_ctx is zero, msgpack object has fully constructed. As the result of that next() returns true. 19 m_stack m_top array size=num 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Parse from m_buffer+m_off Write received data into the memory that is returned by buffer() Parse “string”, then store object into m_ctx. Advance m_off, increase m_parsed.

20. 12 received data3 string not parsed data Internal Structure msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free m_used += size; m_free -= size; buffer_consumed(size) received data3 string not parsed data1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free next() uint32bool num m_parsed received data3 string not parsed data COUNTER_SIZE m_buffer+m_used m_buffer+m_off m_free Decrement the top level count of m_ctx string Because of the top level count of m_ctx is zero, msgpack object has fully constructed. As the result of that next() returns true. When the msgpack object type is string or bin, increment the reference counter 20 m_stack m_top array size=num 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Parse from m_buffer+m_off Write received data into the memory that is returned by buffer() Parse “string”, then store object into m_ctx. Advance m_off, increase m_parsed.

21. 12 received data3 string not parsed data Internal Structure msgpack(array, size=3) bool uint32 string received data1 num received data2 size received data3 1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free m_used += size; m_free -= size; buffer_consumed(size) received data3 string not parsed data1 COUNTER_SIZE m_buffer+m_usedm_buffer+m_off m_free next() uint32bool num m_parsed received data3 string not parsed data COUNTER_SIZE m_buffer+m_used m_buffer+m_off m_free Decrement the top level count of m_ctx string Because of the top level count of m_ctx is zero, msgpack object has fully constructed. As the result of that next() returns true. When the msgpack object type is string or bin, increment the reference counter Instead of copy, msgpack object has a reference of string or bin in the buffer. 21 m_stack m_top array size=num 7/16/2014 Copyright OGIS-RI Co., Ltd. memory management of unpacker Writing received data Parse from m_buffer+m_off Write received data into the memory that is returned by buffer() Parse “string”, then store object into m_ctx. Advance m_off, increase m_parsed.

22. zone, unpacker, context, object 22 client unpacker create and request unpack 7/16/2014 Copyright OGIS-RI Co., Ltd.

23. zone, unpacker, context, object 23 client unpacker zone context buffer create createcreate 7/16/2014 Copyright OGIS-RI Co., Ltd. create and request unpack

24. zone, unpacker, context, object 24 client unpacker zone context object buffer create createcreate reference object(bin,string) create place the object on the zone place the object on the zonemanaged by a reference count 7/16/2014 Copyright OGIS-RI Co., Ltd. create and request unpack

25. zone, unpacker, context, object 25 client unpacker zone context object buffer create createcreate reference object(bin,string) create acquire the ownership from the unpacker 7/16/2014 Copyright OGIS-RI Co., Ltd. create and request unpack

26. zone, unpacker, context, object 26 client zone object buffer reference object(raw,string) 7/16/2014 Copyright OGIS-RI Co., Ltd. acquire the ownership from the unpacker

Unpack mechanism of the msgpack-c

Unpack mechanism of the msgpack-c

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente(20)

Similar a Unpack mechanism of the msgpack-c

Similar a Unpack mechanism of the msgpack-c(20)

Más de Takatoshi Kondo

Más de Takatoshi Kondo(10)

Último

Último(20)

Unpack mechanism of the msgpack-c