Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains

198 visualizaciones

Publicado el

Выступление на PYCON RUSSIA 2017

Publicado en: Internet
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains

  1. 1. Debugging in Python 3.6: Better, Faster, Stronger Elizaveta Shashkova JetBrains

  2. 2. Bio • Software developer of PyCharm IDE at JetBrains • Debugger • Saint Petersburg 2
  3. 3. Debugging • Adding print statements • Logging 3
  4. 4. Debugging • Adding print statements • Logging • Special tools: debuggers 4
  5. 5. Debugger’s Performance 5 Run Debug Debuggers are 30 times slower
  6. 6. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 6
  7. 7. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 7
  8. 8. 8 def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc) 1 2 3 4 5 6 sys.settrace(tracefunc) - set the system tracing function Tracing Function
  9. 9. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 9
  10. 10. 1 2 3 4 5 6 7 8 9 1 calldef foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 10
  11. 11. 1 2 3 4 5 6 7 8 9 1 call 2 line def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 11
  12. 12. 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 12
  13. 13. 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 13
  14. 14. 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 14
  15. 15. Build Python Debugger • Breakpoints • Stepping 15
  16. 16. Tracing Debugger • Suspend program if breakpoint’s line equals frame.f_lineno • Handle events for stepping 16
  17. 17. Performance def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return 17
  18. 18. Example 1 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum 1 2 3 4 5 6 7 8 9 18
  19. 19. Example 1 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum def tracefunc(frame, event, arg): return tracefunc 1 2 3 4 5 6 7 8 9 19
  20. 20. Performance 20 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
  21. 21. Performance 21 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
  22. 22. Performance 22 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
  23. 23. Performance 23 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec ~ 25 times slower!
  24. 24. Problem • Tracing call on every line 24
  25. 25. Optimizations 25
  26. 26. Optimization 1 • Skip scopes without breakpoints • frame.f_code.co_name - name of the function 26
  27. 27. Optimization 2 • Rewrite tracing function in C 27
  28. 28. Optimization 2 • Rewrite tracing function in C • Works only for CPython • ~ 4 times faster 28
  29. 29. Problem • Tracing call on every line 29
  30. 30. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 30
  31. 31. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 31
  32. 32. Python 3.6 32
  33. 33. Python 3.6 • New frame evaluation API • PEP 523 33
  34. 34. PEP 523 • Handle evaluation of frames • Add a new field to code objects • C API 34
  35. 35. Frame Evaluation cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 35
  36. 36. cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name f line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 36
  37. 37. cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 37
  38. 38. cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 38
  39. 39. cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 39
  40. 40. cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) def set_frame_eval(): cdef PyThreadState *state = PyThreadState_Get() state.interp.eval_frame = frame_eval 1 2 3 4 5 6 7 8 9 Frame Evaluation 40
  41. 41. Example 1 2 3 4 5 6 7 8 9 10 11 def first(): second() def second(): third() def third(): pass set_frame_eval() first() 41
  42. 42. Example def first(): second() def second(): third() def third(): pass set_frame_eval() first() 1 2 3 4 5 6 7 8 9 10 11 1 first 4 second 7 third 42
  43. 43. Custom Frame Evaluation • It works • Executed while entering a frame • Access to frame and code object 43
  44. 44. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 44
  45. 45. Problem • Tracing call on every line 45
  46. 46. Problem • Tracing call on every line • Remove the tracing function 46
  47. 47. Replace tracing function with frame evaluation function 47
  48. 48. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 48
  49. 49. Build Python Debugger • Breakpoints • Stepping 49
  50. 50. Breakpoints • Access to the whole code object 50
  51. 51. Breakpoints • Access to the whole code object • Insert breakpoint’s code into frame’s code 51
  52. 52. Breakpoints def maximum(a, b): if a > b: return a else: return b 1 2 3 4 5 6 7 8 52
  53. 53. Breakpoints def maximum(a, b): if a > b: return a # breakpoint else: return b 1 2 3 4 5 6 7 8 53
  54. 54. Breakpoints def maximum(a, b): if a > b: return a # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 54
  55. 55. Breakpoints def maximum(a, b): if a > b: breakpoint() return a # breakpoint else: return b 1 2 3 4 5 6 7 8 55
  56. 56. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 56
  57. 57. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 57
  58. 58. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 58
  59. 59. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 59
  60. 60. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 60
  61. 61. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 61
  62. 62. Python Bytecode 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 62
  63. 63. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 63
  64. 64. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 64
  65. 65. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 65
  66. 66. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() 66
  67. 67. Bytecode Modification • Insert breakpoint’s code • Update arguments and offsets 67
  68. 68. Bytecode Modification • Insert breakpoint’s code • Update arguments and offsets • 200 lines in Python 68
  69. 69. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() ?! 69
  70. 70. Breakpoint Bytecode def _stop_at_break(): # a lot of code here def breakpoint(): _stop_at_break() 1 2 3 4 5 6 7 8 0 LOAD_GLOBAL 0 2 CALL_FUNCTION 0 4 POP_TOP 6 LOAD_CONST 0 8 RETURN_VALUE 70
  71. 71. Build Python Debugger • Breakpoints • Stepping 71
  72. 72. Stepping • Inserting temporary breakpoint on every line • Use old tracing function 72
  73. 73. Frame evaluation debugger is ready 73
  74. 74. Example 1 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum 1 2 3 4 5 74
  75. 75. Example 1 75 Run Tracing Frame 
 evaluation 0,80 sec 19,81 sec 0,81 sec
  76. 76. Example 1 76 Run Tracing Frame 
 evaluation 0,80 sec 19,81 sec 0,81 sec
  77. 77. Example 2 def foo(): pass def calculate(): sum = 0 for i in range(10 ** 7): foo() sum += i return sum 1 2 3 4 5 6 7 8 9 77
  78. 78. Example 2 78 Run Tracing Frame 
 evaluation 1,73 sec 43,58 sec 37,41 sec
  79. 79. PEP 523 • Handle evaluation of frames • Add a new field to code objects 79
  80. 80. PEP 523 • Expand PyCodeObject struct • co_extra - “scratch space” for the code object • Mark frames without breakpoints 80
  81. 81. Mark Frames Cython 1 2 3 4 5 6 7 8 9 81 cdef frame_eval(PyFrameObject *frame, int exc): flag = _PyCode_GetExtra(frame.f_code, index) if flag == NO_BREAKS_IN_FRAME: return _PyEval_EvalFrameDefault(frame, exc) # check for breakpoints ...
  82. 82. Example 2 82 Run Tracing Frame 
 evaluation 1,73 sec 43,58 sec 1,91 sec
  83. 83. PEP 523 • Handle evaluation of frames • Add a new field to code objects 83
  84. 84. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 84
  85. 85. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 85
  86. 86. Real Life Example 86
  87. 87. Real Life Example • Included into PyCharm 2017.1 • Works in production 87
  88. 88. PyCharm 88 Tracing
 w/o Cython Tracing
 with Cython Frame 
 evaluation 11,59 sec 5,66 sec 0,28 sec
  89. 89. Disadvantages • More complicated • Only with CPython • Only with Python 3.6 89
  90. 90. Frame Evaluation • Let’s move to Python 3.6! 90
  91. 91. Frame Evaluation • Let’s move to Python 3.6! • Let’s find another use cases 91
  92. 92. Use cases def maximum(a, b): if a > b: return a # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 92
  93. 93. PEP 523 • Pyjion project • JIT for Python 93
  94. 94. Frame Evaluation • Let’s move to Python 3.6! • Let’s find another use cases 94
  95. 95. Links • Prototype: https://github.com/ Elizaveta239/frame-eval • PyCharm Community Edition source code • bytesinsert on PyPi 95
  96. 96. • Prototype: https://github.com/ Elizaveta239/frame-eval • PyCharm Community Edition source code • bytesinsert on PyPi Questions? 96 @lisa_shashkova

×