The document discusses elements of monitored processes like PID files, their long-running nature, and daemonization. It provides examples of using PID files to track processes and ensure only one instance runs. It demonstrates managing long-running processes and using God to monitor and restart them if they crash or exceed resource thresholds. While God provides an easy way to monitor processes, its Ruby DSL is unnatural and it can be a moderately expensive process itself.
13. LONG RUNNING PROCESSES
require "pid_file"
module LongRunningProcess
module_function
def manage(name, args = ARGV, &job)
pid_path = "/Users/james/Desktop/#{name}.pid"
if args.include? "stop"
if pid = File.read(pid_path).to_i rescue nil
Process.kill("TERM", pid)
else
puts "Not running."
end
else
if PIDFile.create(pid_path)
job.call
else
puts "Already running."
end
end
end
end
14. LONG RUNNING PROCESSES
IN ACTION
require "long_running_process"
LongRunningProcess.manage("show_long_running_process") do
puts "Running as #{Process.pid}..."
at_exit do
puts "Shutting down."
end
loop do
sleep
end
end
15. LONG RUNNING PROCESSES
IN ACTION
require "long_running_process"
LongRunningProcess.manage("show_long_running_process") do
puts "Running as #{Process.pid}..."
at_exit do
puts "Shutting down."
end
loop do
sleep
end $ ruby -I . show_long_running_process.rb start
end Running as 6970...
Shutting down.
Terminated
16. LONG RUNNING PROCESSES
IN ACTION
require "long_running_process"
LongRunningProcess.manage("show_long_running_process") do
puts "Running as #{Process.pid}..."
at_exit do
puts "Shutting down."
end
loop do
sleep
end $ ruby -I . show_long_running_process.rb start
end Running as 6970...
Shutting down.
Terminated
$ cat show_long_running_process.pid
cat: show_long_running_process.pid: No such file or directory
$ cat show_long_running_process.pid
6970
$ ruby -I . show_long_running_process.rb start
Already running.
$ ruby -I . show_long_running_process.rb stop
$ cat show_long_running_process.pid
cat: show_long_running_process.pid: No such file or directory
18. SUPPORTING
DAEMONIZATION
def create(path, &daemonize)
open(path, File::CREAT | File::EXCL | File::WRONLY) do |pid|
pid.flock(File::LOCK_EX)
begin
if daemonize.nil? or (daemonize.call rescue false)
pid.puts Process.pid
else
return false
end
ensure
pid.flock(File::LOCK_UN)
end
end
at_exit do
remove(path)
end
true
rescue Errno::EEXIST # file already exists
false
end
21. THE SMALLEST POSSIBLE GOD
DESKTOP = "/Users/james/Desktop"
RUBY = "ruby -I #{DESKTOP}"
God.watch do |w|
w.name = "show_long_running_process"
w.interval = 30.seconds
w.start = "#{RUBY} #{DESKTOP}/show_long_running_process.rb start"
w.stop = "#{RUBY} #{DESKTOP}/show_long_running_process.rb stop"
w.start_grace = 10.seconds
w.restart_grace = 10.seconds
w.pid_file = "#{DESKTOP}/show_long_running_process.pid"
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
end
22. IN ACTION
$ rvm 1.9.2
$ rvmsudo god -c minor_god.god -D
I [2011-01-13 15:42:43] INFO: Loading minor_god.god
I [2011-01-13 15:42:43] INFO: Syslog enabled.
I [2011-01-13 15:42:43] INFO: Using pid file directory: /var/run/god
I [2011-01-13 15:42:43] INFO: Started on drbunix:///tmp/god.17165.sock
I [2011-01-13 15:42:43] INFO: show_long_running_process move 'unmonitored' to 'up'
I [2011-01-13 15:42:43] INFO: show_long_running_process moved 'unmonitored' to 'up'
I [2011-01-13 15:42:43] INFO: show_long_running_process [trigger] process is not running
(ProcessRunning)
I [2011-01-13 15:42:43] INFO: show_long_running_process move 'up' to 'start'
I [2011-01-13 15:42:43] INFO: show_long_running_process start: ruby -I /Users/james/Desktop /
Users/james/Desktop/show_long_running_process.rb start
I [2011-01-13 15:42:53] INFO: show_long_running_process moved 'up' to 'up'
I [2011-01-13 15:42:53] INFO: show_long_running_process [ok] process is running
(ProcessRunning)
I [2011-01-13 15:42:58] INFO: show_long_running_process [ok] process is running
(ProcessRunning)
I [2011-01-13 15:43:03] INFO: show_long_running_process [ok] process is running
(ProcessRunning)
I [2011-01-13 15:43:09] INFO: show_long_running_process [ok] process is running
(ProcessRunning)
^C/Users/james/.rvm/gems/ruby-1.9.2-p136/gems/god-0.11.0/lib/god.rb:656:in `join': Interrupt
from /Users/james/.rvm/gems/ruby-1.9.2-p136/gems/god-0.11.0/lib/god.rb:656:in `start'
from /Users/james/.rvm/gems/ruby-1.9.2-p136/gems/god-0.11.0/lib/god.rb:667:in `at_exit'
from /Users/james/.rvm/gems/ruby-1.9.2-p136/gems/god-0.11.0/lib/god.rb:700:in `block in <top
(required)>'
23. DOING EVIL
UNDER GOD’S GAZE
$ rvmsudo god -c minor_god.god
$ ps auxww | grep show_long_running_process
root 1986 0.2 0.0 2448168 952 ?? S 3:56PM 0:00.02 ruby -
I /Users/james/Desktop /Users/james/Desktop/show_long_running_process.rb start
james 1988 0.0 0.0 2435116 532 s000 S+ 3:57PM 0:00.00 grep
show_long_running_process
$ sudo kill 1986
$ ps auxww | grep show_long_running_process
root 1996 0.2 0.0 2448168 936 ?? S 3:57PM 0:00.00 ruby -
I /Users/james/Desktop /Users/james/Desktop/show_long_running_process.rb start
james 1998 0.0 0.0 2435116 532 s000 S+ 3:57PM 0:00.00 grep
show_long_running_process
$ rvmsudo god terminate
..
Stopped all watches
Stopped god
$ ps auxww | grep show_long_running_process
james 2005 0.0 0.0 2435116 532 s000 S+ 3:57PM 0:00.00 grep
show_long_running_process
26. CONDITIONAL RESTARTS
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 150.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
restart.condition(:cpu_usage) do |c|
c.above = 50.percent
c.times = 5
end
end
27. CONTROLLING DEATH
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
28. THE POWER OF GOD
# determine the state on startup
w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) do |c|
c.running = true
end
end
# determine when process has finished starting
w.transition([:start, :restart], :up) do |on|
on.condition(:process_running) do |c| # start if process is not running
c.running = true w.transition(:up, :start) do |on|
end on.condition(:process_exits)
end
# failsafe
on.condition(:tries) do |c| # restart if memory or cpu is too high
c.times = 5 w.transition(:up, :restart) do |on|
c.transition = :start on.condition(:memory_usage) do |c|
end c.interval = 20
end c.above = 50.megabytes
c.times = [3, 5]
end
on.condition(:cpu_usage) do |c|
c.interval = 10
c.above = 10.percent
c.times = [3, 5]
end
end
37. THE PRICE OF GOD
• Ruby DSL isn’t at all natural
38. THE PRICE OF GOD
• Ruby DSL isn’t at all natural
• God process in moderately expensive
39. THE PRICE OF GOD
• Ruby DSL isn’t at all natural
• God process in moderately expensive
• God leaked memory in the past (I think this is resolved now)