The document discusses different types of bugs at various stages of development and production. It provides strategies for preventing bugs including writing unit tests, automating processes, monitoring systems, and working smarter by refactoring code and documenting assumptions. When bugs occur in production, it recommends gathering detailed bug reports, profiling code with Xdebug, tracing code execution, and potentially remote debugging to identify issues. However, remote debugging should only be used temporarily due to performance impacts and confidentiality concerns. The document concludes with a plug for the author's company which provides application development and monitoring services.

1. It Works On Dev ByMarcel Esser Code Samurai, CROSCON Contact +1 202 730 9728 +49 176 785 69729 marcel.esser@croscon.com

2. The Many Faces of Bugs What are bugs, really? On dev, a bug is an oversight or human error. We’ve used x or y incorrectly, or our flow control is incorrect. On test, a bug is a failed assertion; x does not equal y, although it’s supposed to. On stage, a bug is handbrake. You can recover gracefully and roll back. On live, any problem is a bug. It prevents the normal course of business operations.

3. Prevention An apple a day…

4. Prevention: Overview How do I prevent live bugs? Practice pre-emptive warfare.Encourage behavior and practices that detect bugs early. Automate processes to prevent human error.If an action can be automated with reasonable effort, do so. Monitor the eco-system.Your application is not just PHP; it’s the entire platform. Work smarter.Identify where investing more time yields greater rewards.

5. Prevention: Pre-emptive Warfare Things you should do: Write unit tests (and preferably, follow TDD). K.I.S.S. Complicated code is hard to maintain. Develop functional tests (for example, Selenium IDE). Re-factor often. Maintain automatic documentation (for example, phpdoc). Ask someone stupid. Know your tools.

6. Prevention: Automation Automation Examples Write scripts to push your configuration and application files to your staging and production machines. rsync and ssh work great. Test upgrades and reconfigurations on dev->test and script them for dev->stage. This is your test for pushing to production. Run customized functional test suites against live after pushes to ensure that your code works as expected.

7. Prevention: Automation Production Push from SVN #!/bin/sh sshroot@prod "/etc/init.d/apache2 stop" rm -rf /tmp/x; mkdir -p /tmp/x svn export https://devs/svn/repo/tags/STABLE /tmp/x/ rsync -avz /tmp/x/ root@prod:/var/www/app/ sshroot@prod "/etc/init.d/apache2 start"

8. Prevention: Monitoring Monitoring: Best effort to lazy time ratio Sometimes bugs aren’t bugs; monitor your application servers to keep track of resource usage and predict load situations. For example, Munin or Zabbix. Use a service-monitoring tool such as Nagios or Zabbix to script actions that ensure your application is available. If you are doing background processing, develop scripts to monitor the status of your custom daemons.

9. Prevention: Be Smarter Working a little smarter Don’t fix problems twice. Keep a log. When you think of a potential problem, make a note. Don’t assume. Create incentives for bug killers. Schedule time for preventative work.

10. Debugging on live? Surely, you jest.

11. Debugging on live? Where do live bugs come from? Very common causes: Differences between dev/test/stage/live platforms. Differences in application state (for example, the database). Differences in configuration. Differences in hardware. Poor quality assurance. Incorrect assumptions.

12. Debugging on live? How to Gather Intelligence on Live Your objectives: Bug reports Profiling Tracing Remote debugging Identifying bugs in your application server

13. Debugging on live: Bug reports What’s a good bug report? Ideally, you have: The account of the user that experienced the error. The page that generated the error. What the user was doing before the error occurred. The user’s client software. Competent support personnel.

14. Debugging on live: Profiling What is Xdebug? PHP debugging extension written by DerickRethans. It profiles. It traces. It breakpoints. It has good client support (Eclipse/NetBeans). Zend Debugger is also good – but not everyone has it.

15. Debugging on live: Profiling with Xdebug # load as a ZE extensionzend_extension=/path/xdebug.so# enable debugging trigger # i.e. XDEBUG_PROFILE=1 in GET, POST,# or cookie xdebug.profiler_enable_trigger = 1 # where to save profile output xdebug.profiler_output_dir = /tmp # filename specifier; <path>.<time>.out xdebug.profiler_output_name = %s.%t.out

16. Debugging on live: Profiling with Xdebug Visit your URL:http://127.0.0.1/cake_1.2.5/?XDEBUG_PROFILE=1 2) Grab the profile dump from the server. 3) Fire up KCacheGrind. 4) Profit.

17. Debugging on live: Tracing with Xdebug # load as a ZE extensionzend_extension=/path/xdebug.so # turn on automatic tracing xdebug.auto_trace=On # where to store the tracedumps xdebug.trace_output_dir=/tmp/ # more memory reporting, please xdebug.show_mem_delta=On

18. Debugging on live: Tracing with Xdebug Visit your URL:http://127.0.0.1/worksondev/index.php 2) Grab the trace dump from the server. 3) Fire up a text editor. 4) Profit.

19. Debugging on live: Debugging with Xdebug # load as a ZE extensionzend_extension=/path/xdebug.so # turn on automatic tracing xdebug.remote_enable = on # debugger protocolxdebug.remote_handler=dbgp # debugger client hostxdebug.remote_host=localhost # debugger client portxdebug.remote_port=9000

20. Debugging on live: A Word of Caution Before you get ahead of yourself… A couple of notes about remote debugging: There is a performance penalty. Enable it only when you need to, and turn it off after you get your data. Remember that debugging information is extremely sensitive and should be kept confidential. Use automatic scripts to turn debug extensions/settings on/off automatically so you don’t accidentally leave “leavings”.

21. Debugging on live: Identifying Server Bugs When All Else Fails Remember that you are computer programmers and: Take the problem machine out of rotation. Setup Apache/Yourwebserver to spawn 1 worker process. Attach a tracer to that process, like strace or dtrace. Figure out where it dies. Map that to the appropriate PHP code or extension. Don’t do whatever you did.

22. Oh, and btw. In case you forgot…

23. Shameless Plug Hi, my name is Marcel Esser. I work for CROSCON. We do: Bespoke application development Customized service monitoring MyCourt productivity software Info? marcel.esser@croscon.com 202.730.9728