Git and Testing

42 visualizaciones

Publicado el

Describe Git related tools and practices to help testing.

Publicado en: Software
0 comentarios
0 recomendaciones
Estadísticas
Notas
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Sin descargas
Visualizaciones
Visualizaciones totales
42
En SlideShare
0
De insertados
0
Número de insertados
0
Acciones
Compartido
0
Descargas
0
Comentarios
0
Recomendaciones
0
Insertados 0
No insertados

No hay notas en la diapositiva.

Git and Testing

  1. 1. Git and Testing December, 2015 Christian Couder chriscool@tuxfamily.org
  2. 2. A Distributed Version Control System (DVCS): ● created by Linus Torvalds ● maintained by Junio Hamano ● since 2005 ● preferred VCS About Git
  3. 3. ● started developing Git in 2006 ● worked especially on git bisect ● independent consultant ● started working on IPFS in 2014 About myself
  4. 4. ● “The Permanent Web”, see http://ipfs.io/ ● Content addressed, versioned, peer to peer filesystem ● Like: Web + Git + BitTorrent ● Alpha software written using the Go language ● Has a Command Line Interface About IPFS (1)
  5. 5. ● In “http://example.com/foo/bar.png“, “example.com” is translated into an IP address like 10.20.30.40 which is a location; the Web is location addressed ● In “/ipns/example.com/foo/bar.png”, “example.com” is translated into a hash of some content like QmW98pJrc6FZ6; IPFS is content addressed About IPFS (2)
  6. 6. ● Pre-alpha software ● No black box tests, only some unit tests ● Heavy development ● A lot of regressions => Really needed black box tests IPFS in October 2014 (last year)
  7. 7. ● 10 years ago Junio Hamano created the Git test framework ● Developed in shell (POSIX /bin/sh compatible) ● Extracted 4 years ago by Mathias Lafeldt into a separate project called Sharness ● It's the reason why Git has always been very stable Sharness
  8. 8. #!/bin/sh test_description=”Sharness example” . sharness/sharness.sh test_expect_success “Testing echo” ' echo Hello world >actual && grep Hello actual ' test_done Sharness example (1)
  9. 9. $ ./example.t ok 1 - Testing echo # passed all 1 test(s) 1..1 Sharness example (2)
  10. 10. Sharness feature: prerequisites Some tests will be launched only if some condition is satisfied, for example: test "$TEST_EXPENSIVE" = 1 && test_set_prereq EXPENSIVE and then: test_expect_success EXPENSIVE,FUSE “Name” ' ...expensive code using fuse...'
  11. 11. Sharness feature: result aggregation
  12. 12. Other Sharness features ● Easily extensible with your own test functions ● Many options: verbose output, stop immediately if a test fails, ... ● Many builtin functions: test_expect_failure, test_pause, test_must_fail,... ● Output in the “test anything” protocol, so compatibility with other tools
  13. 13. ● Now IPFS has around 24 Sharness test scripts ● That means around 300 tests ● Run automatically on each commit of each pull request ● On MacOS and Linux ● Using Travis CI and CircleCI IPFS Sharness Tests now
  14. 14. ● Add both unit tests and black box tests ● Avoid breaking backward compatibility even for pre-alpha software Avoid Regressions (1)
  15. 15. ● Regressions still happen sometimes as tests don't cover everything => git bisect ● Each commit must compile and pass tests => GIT_EDITOR=true git rebase -i --exec “make test” master Avoid Regressions (2)
  16. 16. Commits in Git form a DAG (Directed Acyclic Graph) ● history direction is from left to right ● new commits point to their parents
  17. 17. First Bad Commit B ● B introduces a bad behavior called "bug" or "regression" ● red commits are called "bad" ● blue commits are called "good"
  18. 18. Idea: ● help find a first bad commit ● use a binary search algorithm for efficiency if possible Benefits: ● manually verifying the source code changes from only one commit is relatively easy ● the commit gives extra information: commit message, author, ... Git bisect
  19. 19. 2 ways to do it: $ git bisect start $ git bisect bad [<BAD>] $ git bisect good [<GOOD>...] or $ git bisect start <BAD> <GOOD> [<GOOD>...] where <BAD> and <GOOD> can be resolved to commits Starting a bisection and bounding it
  20. 20. (toy example with the linux kernel) $ git bisect start v2.6.27 v2.6.25 Bisecting: 10928 revisions left to test after this (roughly 14 steps) [2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit $ => the commit you should test has been checked out Starting example
  21. 21. 1.test the current commit 2.tell "git bisect" whether it is good or bad, for example: $ git bisect bad Bisecting: 5480 revisions left to test after this (roughly 13 steps) [66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm repeat step 1. and 2. until the first bad commit is found... Driving a bisection manually
  22. 22. $ git bisect bad 2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux- foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 :100644 100644 5cf8258195331a4dbdddff08b8d68642638eea57 4492984efc09ab72ff6219a7bc21fb6a957c4cd5 M Makefile First bad commit found
  23. 23. When the first bad commit is found: ● you can check it out and tinker with it, or ● you can use "git bisect reset", like that: $ git bisect reset Checking out files: 100% (21549/21549), done. Previous HEAD position was 2ddcca3... Linux 2.6.26-rc1 Switched to branch 'master'   to go back to the branch you were in before you started bisecting End of bisection
  24. 24. At each bisection step a script or command will be launched to tell if the current commit is good or bad. Syntax: $ git bisect run COMMAND [ARG...] Example to bisect a broken build: $ git bisect run make Driving a bisection automatically
  25. 25. $ git bisect start v2.6.27 v2.6.25 Bisecting: 10928 revisions left to test after this (roughly 14 steps) [2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit $ $ git bisect run grep '^SUBLEVEL = 25' Makefile running grep ^SUBLEVEL = 25 Makefile Bisecting: 5480 revisions left to test after this (roughly 13 steps) [66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm running grep ^SUBLEVEL = 25 Makefile Automatic bisect example (1)
  26. 26. SUBLEVEL = 25 Bisecting: 2740 revisions left to test after this (roughly 12 steps) [671294719628f1671faefd4882764886f8ad08cb] V4L/DVB(7879): Adding cx18 Support for mxl5005s ... ... running grep ^SUBLEVEL = 25 Makefile Bisecting: 0 revisions left to test after this (roughly 0 steps) [2ddcca36c8bcfa251724fe342c8327451988be0d] Linux 2.6.26-rc1 running grep ^SUBLEVEL = 25 Makefile Automatic bisect example (2)
  27. 27. 2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux- foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 :100644 100644 5cf8258195331a4dbdddff08b8d68642638eea57 4492984efc09ab72ff6219a7bc21fb6a957c4cd5 M Makefile bisect run success Automatic bisect example (3)
  28. 28. 0 => good 1-124 and 126-127 => bad 125 => skip 128-255 => stop "skip": mark commit as "untestable", "git bisect" will choose another commit to be tested "stop": bisection is stopped immediately, useful to abort bisection in abnormal situations Run script and exit code
  29. 29. Problem when bisecting Sometimes the commit that introduced a bug will be in an untestable area of the graph. For example: X X1 X2 X3W Y Z Commit X introduced a breakage, later fixed by commit Y.
  30. 30. There are only 'skip'ped commits left to test. The first bad commit could be any of: 15722f2fa328eaba97022898a305ffc8172db6b1 78e86cf3e850bd755bb71831f42e200626fbd1e0 e15b73ad3db9b48d7d1ade32f8cd23a751fe0ace 070eab2303024706f2924822bfec8b9847e4ac1b We cannot bisect more! Possible end of bisection
  31. 31. Possible solutions Possible solutions to bisect anyway: ● apply a patch before testing and remove it afterwards (can be done using "git cherry-pick"), or ● create a fixed up branch (can be done with "git rebase -i"), for example: X X1 X2 X3W Y Z X + Y X1' X2' X3' Z' Z1
  32. 32. A good solution X X1 X2 X3W Y Z X + Y X1' X2' X3' Z' Z1 The idea is that we will replace Z with Z' so that we bisect from the beginning using the fixed up branch. $ git replace Z Z'
  33. 33. ● ./example.t ● vi t0040-add-and-cat.sh ● TEST_EXPENSIVE=1 ./t0040-add-and-cat.sh -v ● cd test/sharness && make ● git rebase -i --exec “make test” master ● git bisect run ./test_script.sh Demos
  34. 34. ● https://github.com/mlafeldt/sharness ● https://github.com/chriscool/sharnessify ● http://git-scm.com ● http://ipfs.io Links
  35. 35. Conclusion ● Black box tests with Sharness are quite easy and very useful ● Check every commit, not just every pull request ● Use git bisect, automate it if possible
  36. 36. Many thanks to: ● Junio Hamano, Linus Torvalds, ● Juan Benet, Jeromy Johnson, Mathias Lafeldt ● many other great people in the IPFS, Git and Linux communities ● LinuxCon organizers and attendants, ● SoftAtHome, the company I am working for.
  37. 37. Questions?

×