1. SURVEY OF DYNAMIC PROGRAMMING SOFTWARES
IE-5306- Dynamic Optimization
Fall 2015
Project Report
Date: 12/1/15
Instructor:
Dr. Herbert Corley
Submitted by:
Ashkan Aliabadi
Ukesh Chawal
Nilabh Ohol

2. Abstract
This report surveys some of the commercially available softwares to solve dynamic programming
problems. Specific software for solving dynamic programming problem is not available and due to its
vast application and versatility of problems a number of softwares can be used to solve different type of
problems. Keeping in mind the scope of the course and in an attempt for users to be able to solve some
simple problems, an overview of such solvers has been put forth with the hope that using this knowledge
more complex problems can be tackled by the users. Different types of solvers require different skill set
which is also being highlighted in the report.
Introduction
Background:
Dynamic Programming solves optimization problems by breaking a complex problem into a sequences of
sub-problems, each having decisions such that they can be solved in like fashion to obtain an optimal
solution for each sub problem such that the optimal solution for a the original problem can be found out
from the optimal solutions of the sub problems. This method is based on Bellman’s Principal of
optimality which can be phrased as follows:
“An optimal policy has the property that whatever the initial state and initial decision are, the remaining
decisions must constitute an optimal policy with regard to the state resulting from the first decision.”(Lew
and Mauch, 2007)
A large variety of problems can be solved using dynamic programming, both discrete and continuous.
Some of the popular problems include allocation problems, optimal stopping rule which are discrete
problems while continuous problems include linear programming, non-linear programming, minimax
problems. Dynamic programming is popular since it is a generic approach to solving optimization
problems. This report proposes the work to be done over coming weeks to survey various dynamic
optimization softwares. The motivation behind the survey, the methodology to complete the study and the
expected results are discussed in the following sections.
Motivation:
Curse of dimensionality refers to phenomena that arises when analyzing or organizing data with high
dimensional spaces. This is a common problem while solving dynamic optimization problems since there
are a number of problems where the dimension of state variables is large. Thus in order to improve
computational simplicity, a computer program, that can handle such a problem, is necessary. Also, apart

3. from handling high dimension spaces, it is always advantageous to have a tool which can solve a dynamic
optimization problem. Software tool to solving dynamic programming is much more difficult to design in
part because the problem specification task in itself presents difficulties. Unlike linear programming
where there are specific solvers that handle the problem, dynamic programming problems do not have a
dedicated solver largely due to its generality. However there are certain solvers available that can handle
dynamic programming problems. Hence it is necessary to investigate such softwares and highlight their
advantages and disadvantages and their applications in dynamic programming world. Following sections
cover the problem definition, enlists different softwares and solves some popular dynamic programming
problems. The algorithm embedded in different solvers is also highlighted to show different approaches to
solving dynamic programming problems. Finally the learning objectives are summarized in the
conclusion section. The flow of the report closely follows the course structure.
Problem Definition:
The aim of the study is to survey various dynamic programming softwares, compare them to highlight
their advantages and disadvantages along with their usefulness. As a part of the study numerous examples
will be solved to illustrate the working of these softwares.
Survey of Softwares
LINDO
An Overview:
The name LINDO is an abbreviation of Linear INteractive Discrete Optimization. With the LINDO, users
can easily create their own optimization applications. It allows them to plug the power of the LINDO
solvers right into customized applications and mathematical programs that they have written.
Like almost all LP packages, LINDO solves real world linear programs using the simplex method. Along
with the solution to the problem, the program will also provide ordinary sensitivity analysis of the
Objective Function Coefficients (called Cost Coefficients) and the Right-hand-side (RHS) of the
constraints.

4. Key Benefits of the LINDO:
Fast, Easy Application Development
The LINDO makes it easy for you to seamlessly integrate optimization into your own application. The
developer interface has been designed for maximum ease-of-use and flexibility.
Powerful Solvers
LINDO provides you with an arsenal of powerful solvers for Stochastic, Linear, Nonlinear (convex &
nonconvex/Global), Quadratic, Quadratically Constrained, Second Order Cone and Integer optimization.
All solvers incorporate numerous enhancements for maximum speed and robustness.
Comprehensive Set of Routines
Whether your application is big or small, simple or complex, the LINDO API provides the flexibility and
functionality that you'll need. It includes dozens of routines to formulate, solve, query, and modify your
problems.
Extensive Documentation and Help
LINDO API provides all of the tools you will need to get up and running quickly. You get the LINDO
API User Manual that includes detailed function definitions for all routines. Also included in the manual
is a discussion to assist you in writing your own applications.
Analyze Infeasible and Unbounded Models
LINDO API includes tools that allow you to track down what has caused a model to be infeasible or
unbounded. The tools isolate a portion of the original model as the source of the problem. This allows you
to focus your attention on a relatively small subsection of the model to look for formulation or data entry
errors. On infeasible linear, nonlinear, quadratic and integer models, the tools can find an irreducibly
inconsistent set of constraints (IIS), and on unbounded models, the tools can find an irreducibly
unbounded set of columns (IUS).
Create Web and Intranet Applications
The LINDO API is thread safe to allow you to create web and network applications that handle multiple
user sessions concurrently. Web and network applications require special licensing.
Assignment Problem
Q) Consider a school with 5 teachers and 5 courses. The performance of a teacher i in charge of course j
is xij. There will be fixed unit of performance done by each teacher in each course. For example, 7 units of

5. performance is done by teacher 1 in course 1. Similarly, 8 units of performance is done by teacher 5 in
course 4 and so on.
course 1 course 2 course 3 course 4 course 5
Teacher 1 7 5 4 4 5
Teacher 2 7 9 7 9 4
Teacher 3 4 6 5 8 5
Teacher 4 5 4 5 7 4
Teacher 5 4 5 5 8 9
Chair has assign teachers to courses to maximize performance.
Traditional Dynamic Programming Solution Method
Stages: teachers
Decision variables: the course chosen
States: courses left
Returns: performance
𝑚𝑎𝑥 ∑ ∑ 𝑥 𝑖𝑗
𝑛
𝑗=1
𝑛
𝑖=1
s.t
∑ 𝑥 𝑖 𝑗 = 1
𝑛
𝑖=1
∀𝑖 = 1, …… , 𝑛
∑ 𝑥 𝑖 𝑗 = 1
𝑛
𝑗=1
∀𝑗 = 1,… …, 𝑛
𝑥 𝑖 𝑗 ≥ 0 ∀𝑖, 𝑗

6. Since it is pretty much impossible to solve it by hand (Traditional Dynamic Programming Solution
Method), we took help from Dynamic Programming Software, LINDO to solve this problem.
Formulating the Problems:
𝑀𝑎𝑥 7𝑥11 + 5𝑥12 + 4𝑥13 + 4𝑥14 + 5𝑥15 + 7𝑥21 + 9𝑥22 + 7𝑥23 + 9𝑥24 + 4𝑥25
+ 4𝑥31 + 6𝑥32 + 5𝑥33 + 8𝑥34 + 5𝑥35 + 5𝑥41 + 4𝑥42 + 5𝑥43 + 7𝑥44
+ 4𝑥45 + 4𝑥51 + 5𝑥52 + 5𝑥53 + 8𝑥54 + 9𝑥55
𝑠𝑡
𝑥11 + 𝑥12 + 𝑥13 + 𝑥14 + 𝑥15 = 1
𝑥21 + 𝑥22 + 𝑥23 + 𝑥24 + 𝑥25 = 1
𝑥31 + 𝑥32 + 𝑥33 + 𝑥34 + 𝑥35 = 1
𝑥41 + 𝑥42 + 𝑥43 + 𝑥44 + 𝑥45 = 1
𝑥51 + 𝑥52 + 𝑥53 + 𝑥54 + 𝑥55 = 1
𝑥11 + 𝑥21 + 𝑥31 + 𝑥41 + 𝑥51 = 1
𝑥12 + 𝑥22 + 𝑥32 + 𝑥42 + 𝑥52 = 1
𝑥13 + 𝑥23 + 𝑥33 + 𝑥43 + 𝑥53 = 1
𝑥14 + 𝑥24 + 𝑥34 + 𝑥44 + 𝑥54 = 1
𝑥15 + 𝑥25 + 𝑥35 + 𝑥45 + 𝑥55 = 1
𝑥 𝑖 𝑗 ≥ 0 ∀𝑖, 𝑗
T1T4 T3 T2
d1
s4 s3
s2
r4 r3
r2 r1
d4 d3
d2
s1
T5
s5
d5
r5

7. Input in LINDO

8. Solution from LINDO
From the solution window, we can see the number of iterations needed to solve it is 19.
We can interpret that maximum performance is 38units.
We can also see x11 =1 meaning 7 units of performance done by teacher 1 in course 1. Similarly, x22 = 1
meaning 9 units of performance done by teacher 2 in course 2. x34 = 1 meaning 8 units of performance
done by teacher 3 in course 4. x43 =1 meaning 5 units of performance done by teacher 4 in course 3. x55 =
1 meaning 9 units of performance done by teacher 5 in course 5 which sums up to be 38 units.

9. Limitations
1) Non – Negativity Conditions – Imposed on all variables by default
2) Shadow Prices – Non-consistent
3) Optimal Solution to non-standard form problem – May need conversion
4) Sensitivity Ranges – May need conversion which is not an easy task.
The maximum sized problem LINGO can handle depends on the version you have. The current limits for
the various versions are:
Version
Total
Variables
Integer
Variables
Nonlinear
Variables
Global
Variables Constraints
Demo/Web 300 30 30 5 150
Solver Suite 500 50 50 5 250
Super 2,000 200 200 10 1,000
Hyper 8,000 800 800 20 4,000
Industrial 32,000 3,200 3,200 50 16,000
Extended Unlimited Unlimited Unlimited Unlimited Unlimited
TORA
AN OVERVIEW OF TORA
The Temporary-Ordered Routing Algorithm (TORA), An Operations Research Software is an algorithm
i.e. a mathematical set of instructions or programs. It is an optimization system in the area of operations
research which is very easy to use. Further, TORA is menu-driven and Windows-based which makes it
very user friendly.

10. TORA is invented by Vincent Park and M.Scott Corson from University of Maryland. The main objective
of TORA is to limit control message propagation in the highly dynamic mobile computing environment.
Each node has to explicitly initiate a query when it needs to send data to a particular destination.
TORA software deals with the following algorithms:
•Solution of simultaneous linear equations
•Linear programming
•Transportation model
•Integer programming
•Network models
•Project analysis by CPM/PERT
•Poisson queuing models
•Zero-sum games
Like LINDO, TORA also solves linear programming programs using the simplex method.
Linear Programming - Transportation Problem
Q) There are four factories (sources) and four markets (sinks). Each of the factories have certain
maximum capacity to supply. Each of the markets have certain maximum demand. For example, Factory
1 has a capacity of producing 250 units, Factory 2 has a capacity of 300 and so on. Market 1 has a
demand of 250 units, Market 2 has 200 units and so on. There will be fixed cost of transporting 1 unit of
material from each factory to each market. For example, there is a 5 dollar cost for transporting a material
from Factory 1 to Market 1. Similarly there is a 6 dollar cost for transporting from Factory 4 to Market 3
and so on.
Market 1 Market 2 Market 3 Market 4 Capacity
Factory 1 $5 $4 $6 $3 250
Factory 2 $7 $8 $8 $1 300
Factory 3 $4 $4 $3 $6 100
Factory 4 $8 $7 $6 $8 200
Demand 250 200 200 200

11. Minimize the total transportation cost.
Traditional Dynamic Programming Solution Method
Stages: factory
Decision variables: quantity of material shipped from a factory i to market j.
States: quantity left
Returns: transportation cost
Since it is pretty much impossible to solve it by hand (Traditional Dynamic Programming Solution
Method), we took help from Dynamic Programming Software, TORA to solve this problem.
Input in Tora
F1F4 F3 F2
d1
s4 s3
s2
r4 r3
r2 r1
d4 d3
d2
s1

12. Solutions from TORA
From the solution window, we can interpret that total minimum cost to transport the required quantities is
$ 3550.
We would sell 50 units of materials from factory 1 to market 1, 200 units of material from factory 1 to
market 2, 100 units of materials from factory 2 to market 1, 200 units of materials from factory 2 to
market 4, 100 units of materials from factory 3 to market 1 and 200 units of materials from factory 4 to
market 3.
Limitations: Not able to solve following DP Problems
1) Non-linear Programming Problem
2) Probabilistic Problem
3) Mini-max Problem
4) Optimal Stopping Rule Problem
Cross Checking the TORA answer with LINDO.
Formulating the Problems in LINDO:
𝑀𝑖𝑛 5𝑥11 + 4𝑥12 + 6𝑥13 + 3𝑥14 + 7𝑥21 + 8𝑥22 + 8𝑥23 + 𝑥24 + 4𝑥31 + 4𝑥32 + 3𝑥33
+ 6𝑥34 + 8𝑥41 + 7𝑥42 + 6𝑥43 + 8𝑥44

13. 𝑠𝑡
𝑥11 + 𝑥12 + 𝑥13 + 𝑥14 <= 250
𝑥21 + 𝑥22 + 𝑥23 + 𝑥24 <= 300
𝑥31 + 𝑥32 + 𝑥33 + 𝑥34 <= 100
𝑥41 + 𝑥42 + 𝑥43 + 𝑥44 <= 200
𝑥11 + 𝑥21 + 𝑥31 + 𝑥41 = 250
𝑥12 + 𝑥22 + 𝑥32 + 𝑥42 = 200
𝑥13 + 𝑥23 + 𝑥33 + 𝑥43 = 200
𝑥14 + 𝑥24 + 𝑥34 + 𝑥44 = 200
Input in LINDO

14. Solution from LINDO
From the solution window, we can see total minimum transportation cost is $ 3550. We would sell 50
units of materials from factory 1 to market 1, 200 units of material from factory 1 to market 2, 100 units
of materials from factory 2 to market 1, 200 units of materials from factory 2 to market 4, 100 units of
materials from factory 3 to market 1 and 200 units of materials from factory 4 to market 3.
The Answer from LINDO exactly match with that of TORA.
MATLAB
MATLAB is a multi-paradigm numerical environment and a fourth generation programming language
intended primarily for numerical computing. A proprietary programming language developed by
MathWorks, MATLAB allows matrix manipulation, plotting of functions and data, implementation of
algorithms. It shares a robust interface with programs written in other languages, including C, C++, Java,
FORTRAN and Python. The application is built around the MATLAB scripting language and involves
using a command window as an interactive mathematical shell or executing text files containing
MATLAB codes. The ability to create a separate function file which can be called from a main program
makes it a very strong tool for solving custom made dynamic optimization problems.

15. The optimization toolbox in MATLAB contains an exhaustive list of functions that handles a variety of
optimization problems. Problems such as linear programming problems, non-linear programming, convex
and non-convex, problems, stochastic programming problems, minmax and multi-objective problems can
be solved in MATLAB using with the help of the optimization tool box. A typical dynamic programming
(DP) problem uses a backward recursive algorithm which in not necessarily embedded in all the solvers.
A custom DP problem can be coded in a MATLAB script file without the use of the existing toolbox in
the software. An example of this is a travelling salesman problem. A function was developed to solve the
travelling sales man problem using dynamic programming. The function code is attached in appendix and
is developed based on the paper by Held and Karp from 1962. The DP is guaranteed to provide accurate
result to travelling salesman problem, but due to the time complexity is limited to at most 15 cities. The
output is also attached in the appendix. The objective of this function is to highlight the fact that a custom
program can be developed to solve a DP problem in the traditional way.
Minimax Problem
Another problem that MATLAB effectively handles is a minimax problem. A minimax problem can be
converted into a linear programming problem and solved in a traditional way. However MATLAB
function fminimax finds the minimum of a problem specified by:
min
𝑥
max
𝑖
𝐹𝑖 ( 𝑥) 𝑠𝑢𝑐ℎ 𝑡ℎ𝑎𝑡
{
𝑐( 𝑥) ≤ 0
𝑐𝑒𝑞( 𝑥) = 0
𝐴 ∙ 𝑥 ≤ 𝑏
𝐴𝑒𝑞 ∙ 𝑥 = 𝑏𝑒𝑞
𝑙𝑏 ≤ 𝑥 ≤ 𝑢𝑏
where b and beq are vectors, A and Aeq are matrices, and c(x), ceq(x), and F(x) are functions that return
vectors. F(x), c(x), and ceq(x) can be nonlinear functions. x, lb, and ub can be passed as vectors or
matrices. A maximin problem can be solved using the identity,
max
𝑥
min
𝑖
𝐹𝑖 ( 𝑥) = −min
𝑥
max
𝑖
(−𝐹𝑖 ( 𝑥))
This is illustrated by a problem,
𝐹𝑖𝑛𝑑 𝑡ℎ𝑒 𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝑥 𝑡ℎ𝑎𝑡 𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒𝑠 𝑡ℎ𝑒 𝑚𝑎𝑥𝑖𝑚𝑢𝑚 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓
[ 𝑓1( 𝑥), 𝑓2( 𝑥), 𝑓3( 𝑥), 𝑓4( 𝑥), 𝑓5( 𝑥)]
𝑤ℎ𝑒𝑟𝑒,
𝑓1( 𝑥) = 2𝑥1
2
+ 𝑥2
2
− 48𝑥1 − 40𝑥2 + 304
𝑓2( 𝑥) = −𝑥1
2
− 3𝑥2
2
𝑓3( 𝑥) = 𝑥1 + 3𝑥2 − 18

16. 𝑓4( 𝑥) = −𝑥1 − 𝑥2
𝑓5( 𝑥) = 𝑥1 + 𝑥2 − 8
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜, 𝑥1
2
+ 𝑥2
2
≤ 12
First a function file is written that computes all the functions at x. The code for the function file is given
below:
function f = minimax(x)%objective functions are defined here
f(1) = 2*x(1)^2+x(2)^2-48*x(1)-40*x(2)+304; %notice some of the functions are
nonlinear
f(2) = -x(1)^2-3*x(2)^2;
f(3) = x(1)+3*x(2)-18;
f(4) = -x(1)-x(2);
f(5) = x(1)+x(2)-18;
Next step is to define the non-linear constraint in a different function. This is written as follows:
function [c,ceq] = constr(x)%non-linear constraints are defined here
c(1) = x(1)^2 + x(2)^2 - 12;%non-linear inequality constraint
ceq = [];%no non-linear equality constraint present
Notice that the ceq, which is the non-linear inequality constraint, is defined as a null set since it is not
present in the problem. It is imperative to define all the constraints which are missing in the problem as
null sets for the function fminimax to work. The syntax is discussed later on.
After the two functions are defined a main program is written which calls out these two functions and
defines other parameters. This is given below:
%%Minimax main Program which calls the two previously defined functions%%
clc;
clear all;
x0 = [0;0];%Starting point of the algorithm
A=[];%No linear constrains
b=[];
ub=[];%Upper bound is not defined
lb=[0;0];%x1 and x1 >=0
Aeq=[];%no linear equality constraints for the problem
beq=[];
[x,fval] = fminimax(@minimax,x0,A,b,Aeq,beq,lb,ub,@constr);%fminmax
calculates the minimax for the objective functions
The syntax fminimax(@minimax,x0,A,b,Aeq,beq,lb,ub,@constr) minimizes the maximum value of te
functions. In this syntax, @minimax calls the function that computes all the functions at x. x0 is te
starting point from which the algorithm begins. In this case it is defined as [0;0] for x1 and x2. It makes
sense because the lower bound lb is defined as 0, A and b is the left and right hand side of the linear

17. inequality constraint respectively. Since these constraints are not present in the problem, they are defined
as null sets. Similarly Aeq and beq which are the left and right hand side of the linear equality constraints
are defined as null sets. lb and ub is the lower and upper bound respectively. @constr calls the function
that defines the nonlinear constraints. The output parameter x returns the value of x1 and x2 which
minimizes the maximum value of the functions and fval gives the values of the five functions. The output
of the program is shown below:
Figure: MATLAB Output of the Minimax Problem
fminimax internally reformulates the minimax problem into an equivalent Nonlinear Linear Programming
problem by appending additional (reformulation) constraints of the form Fi(x) ≤ γ to the constraints given
in equation, and then minimizing γ over x. fminimax uses a sequential quadratic programming (SQP)
method to solve the problem discussed. Thus at x1 = 2.5345 and x2 = 2.3614 the maximum values of the
functions are minimized. These values are shown in the output. Although the algorithm used by fminimax

18. is different from the traditional DP algorithm, it gives the same solution in much less time. Also with the
increasing number of functions and variables, the number of stages in recursive DP method increases, use
of software is warranted.
Problem with Multi-Objective Functions (Pareto Optima)
There are many cases where solution of a multi-objective function is required. For example knapsack
problems where the objective can be to minimize weight volume along with a new objective say
dissatisfaction level. This then becomes a multi-objective problem. It is known that multi-objective
maximization and minimization problems can be solved using dynamic programming methods. However
with increasing number of variables it becomes necessary to use software techniques to solve such
problems. Solver such as Cplex and Couenne cannot easily handle multi objective problems. However
there is an inbuilt function in MATLAB which enables users to solve multi objective functions easily. An
example is illustrated to show how to perform multi-objective optimization using multi-objective genetic
algorithm function gamultiobj in the global optimization toolbox.
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑣𝑚𝑖𝑛(2𝑥1 − 3𝑥2,4𝑥1 + 𝑥2
2)
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ,
𝑥1 + 𝑥2 ≤ 12
𝑥1, 𝑥2 ≥ 0
The first step is to define the functions as was done for the minimax problem. The code is given below:
function f = vmin(x)%objective functions are defined here
f(:,1) = 2*x(1)-3*x(2);
f(:,2) = 4*x(1)+x(2).^2;%notice some of the functions are nonlinear
It can be seen that function f2 is non-linear. The next step is to define the constraints. It is not necessary to
define a new function for the constraints since all the constraints are linear in nature. These are defined in
the main program. The main program is given below:
%Program for Pareto Optima%
clc;
clear all;
close all;
numberofvariables = 2;
lb = [0;0];%define lower bounds
ub=[];%undefined upper bounds
A = [1 1];%Defining inequality linear constraints
b = [12];
Aeq = [];%no equality linear constraints are present
beq = [];

19. options = gaoptimset('PlotFcns',{@gaplotpareto,@gaplotscorediversity});%plot
the objective functions on the axis
[x,fval] =
gamultiobj(@vmin,numberofvariables,A,b,Aeq,beq,lb,ub,options);%Finding Pareto
optima
fprintf('The number of points on the Pareto front was: %dn',size(x,1));
Defining constraints is similar in this case as was discussed earlier. The solver gamultiobj attempts to
create a set of Pareto optima for a multi-objective minimization. The definition of the solver is the same
as it calls the function where the function values are defined, along with the number of variables in the
problem, the constraints and the options option which creates a plot of the two objective functions. This
plot is shown below:
Figure: MATLAB plot of the Pareto front
The first figure shows the pareto front which shows the pareto optimal points with the value of their
respective objective functions. The second plot shows the score diversity for each objective. The output is
shown in below: All the 27 pareto optimal points and their respective function values are shown.
-40 -35 -30 -25 -20 -15 -10 -5 0
0
50
100
150
Objective 1
Objective2
Pareto front
-40 -20 0 20 40 60 80 100 120 140 160
0
5
10
15
20
Score Histogram
Score (range)
Numberofindividuals
fun1 [-36.0019 -0.00132518]
fun2 [3.94134e-05 144.019]

20. Figure: MATLAB output
Note on the Algorithm: gamultiobj uses a controlled elitist genetic algorithm. An elitist GA always favors
individuals with better fitness value (rank). A controlled elitist GA also favors individuals that can help
increase the diversity of the population even if they have a lower fitness value. It is important to maintain
the diversity of population for convergence to an optimal Pareto front. Diversity is maintained by
controlling the elite members of the population as the algorithm progresses. Two options, ParetoFraction
and DistanceFcn, control the elitism. ParetoFraction limits the number of individuals on the Pareto front
(elite members). The distance function, selected by DistanceFcn, helps to maintain diversity on a front by
favoring individuals that are relatively far away on the front. The algorithm stops if the spread, a measure
of the movement of the Pareto front, is small.

21. AMPL
AMPL (A Mathematical Programming Language) is an algebraic modeling language for describing and
solving high complexity problems for large scale mathematical computation. AMPL supports lots of
solvers, both open source and commercial. It also supports the entire optimization modeling lifecycle i.e.
formulation, testing, development and maintenance. Problems in AMPL are comparatively easy to
formulate and since it supports a large number of solvers it becomes an important part of this study.
AMPL creates models and data for optimization and the solvers compute them to obtain optimal
solutions. Solvers such as CPLEX, Gurobi, Xpress handle linear and convex quadratic optimization
problems where the variables can be both continuous and integer. On the other hand solvers such as
CONOPT, KNITRO, MINOS, SNOPT provide most effective way of finding locally optimal solution to
problems involving smooth non-linear functions. Alternative solver such as BARON and Couenne can
also be interfaced with AMPL to solve optimization problems. CPLEX is a popular solver for linear
programming problems and uses a branch and bound algorithm to solve LP problems whereas Couenne is
extensively used for solving smooth non-linear programming problems. However in this report, MINOS
is used to solve smooth non-linear programming problem with continuous constraints. Since linear
programming problems were discussed earlier, an example problem using CPLEX is solved for a smooth
quadratic objective function. A traditional dynamic programming approach is solves both linear and non-
linear problems however with increasing number of variables, the computational complexity also
increases and thus use of software technique is warranted.
CPLEX
CPLEX follows a branch and bound algorithm and is very effective in solving high dimensional linear
programming problems. A simple linear programming problem is solved with seconds however with the
increasing number of variables (say in thousands) the computational time increases. This is usually
tackled by setting a time bound on the solver. In doing so, under worst case, the solver runs as much
iteration as possible within the specified time frame to return the best possible feasible solution. With new
addition to the solver, CPLEX can also handle quadratic objective functions. An example is shown below
to illustrate this:
:Portfolio Selection: An investor has $5000 and two potential investments. Let xj for j = 1 and j = 2 denote his
allocation to investment j in thousands of dollars. From historical data, investments 1 and 2 have an expected
annual return of 20 and 16 percent, respectively.Also, the total risk involved with investments 1 and 2,as measured
by the variance of total return, is given by, 2𝑥1
2
+ 𝑥2
2
+ (𝑥1 + 𝑥2)2, so that risk increases with total investment and
with the amount of each individual investment.The investor would like to maximize his expected return and at the
same time minimize his risk. Clearly,both of these objectivescannot, in general, be satisfied simultaneously. There
are several possible approaches.For example, he can minimize risk subject to a constraint imposing a lower bound
on expected return. Alternatively, expected return and risk can be combined in an objective function, to give the
model:

22. 𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝑓( 𝑥) = 20𝑥1 + 16𝑥2 − [2𝑥1
2
+ 𝑥2
2
+ ( 𝑥1 + 𝑥2)3]
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜,
𝑥1 + 𝑥2 ≤ 5
𝑥1, 𝑥2 ≥ 0
Such a problem obtains a quadratic form and can be modeled in AMPL and solved using CPLEX.
The code for modeling the above problem is shown below:
#Define Variables
var x1;
var x2;
#Define Objective Function
maximize z: 20*x1+16*x2-2*x1^2-x2^2-(x1+x2)^2;
#Define Constraints
subject to c1: x1+x2<=5;
subject to c2: x1>=0;
subject to c3: x2>=0;
The output is given below:
ampl: model nlp.mod;
ampl: option solver cplex;
ampl: solve;
CPLEX 12.6.1.0: optimal solution; objective 46.33333321
17 QP barrier iterations
No basis.
ampl: display x1;
x1 = 2.33333
ampl: display x2;
x2 = 2.66667
ampl:
The model file is stored as a .mod file which is called in the AMPL workspace. The solver is then selected
to run the model file. The solution obtained is the maximum value of the objective function for the given
constraints and can be seen from the output above.
Now if a cubic term is added in the objective function and the code is run using CPLEX solver, it can be
seen that CPLEX returns an error warning.
ampl: model nlp.mod;
ampl: option solver cplex;

23. ampl: solve;
CPLEX 12.6.1.0: C:UsersIE-FAC~1AppDataLocalTempat3936.nl contains a nonlinear
objective.
ampl:
This shows that CPLEX does not handle non-linear problems. MINOS is used to solve this problem. The
problem definition remains unchanged only difference is that while running the code in AMPL
workspace, MINOS is selected chosen instead of CPLEX.
ampl: model nlp.mod;
ampl: option solver minos;
ampl: solve;
MINOS 5.51: optimal solution found.
5 iterations, objective 25.59249598
Nonlin evals: obj = 11, grad = 10.
ampl: display x1;
x1 = 1.39724
ampl: display x2;
x2 = 0.794486
ampl:
The results can be seen from the output and are pretty straightforward to interpret. These solvers provide a
considerable amount of advantage when the number of variables is very large. Since a nonlinear
programming problem with a high number of variables increases computational complexity, if solved
using traditional dynamic programming method.
DP Solver
Developed by Paul A. Jensen, a professors at Mechanical engineering department at UT Austin. The link
to his all OR software is https://www.me.utexas.edu/~jensen/ORMM/
This DP add-in provides a general structure with which most problems appropriate for DP can be
modeled and solved. The add-ins in this package are programmed using the Microsoft macro language,
Visual Basic for Applications (VBA). The source code is available for viewing and changing when the
add-in is installed in Excel. Some of the add-ins are not password protected. These include the Teach DP
and Stochastic Models add-ins. They include specific model classes that are provided through VBA
subroutines. They are left open so the user can add new problem classes.
Limitation: 2/20/2011: The MP Model Builder add-in, Jensen LP/IP solver add-in, and Math
Programming add-in were modified to allow more than 32000 nonzero coefficients in the LP constraint
matrix. The Jensen Network Solver add-in was modified to have more than 32000 arcs.

24. Figure 1: DP collection Add-in
The Jensen Software Add-in covers a variety of optimization problem:
 LinearProgramming
 NonlinearProgramming
 IntegerProgramming
 MixedIntegerProgramming
 NetworkFlowProgramming
 TransportationModel
 Stochasticprogramming
 Dynamicprogramming
 Combinatorics
 Queueing
 Morkov Decisionprocess
 Simulation
DP Collection Add-in:

25. The DP Data add-in
The DP Data add-in constructs tables that describe data for instances of specific application areas of
dynamic programming or Markov analysis. The DP Data add-in calls the DP Models add-in to construct
the model form to hold the decisions and stochastic features of the model. The DP Data add-in then fills
the model form with parameters linked to the data form. The resulting model will ultimately be used to
provide data for the DP Solver or Markov Analysis add-ins.
Although the model forms available in the DP Models add-in are very general, it is somewhat difficult for
the unskilled user of Excel to build a model. The DP Data add-in removes this difficulty by
automatically building the model for specific problem classes. At this time only five classe s are
available: birth-death, finite queue, random walk, inventory and resource. For each class there are
several variations.
Addition of new problem classes requires VBA programming. The programs of this add-in are not
password protected so you can examine the programs for existing classes. As a user of this add-in, you
may find it interesting to add new classes.
The Start and Finish commands are used to respectively add and remove buttons from the worksheets.
This is important when moving models from one computer to another. Always delete the buttons with the
Finish command before attempting to open a model on another computer. Otherwise you will experience
an error message regarding links. Add buttons back with the Start command. Although the Finish
command deletes all buttons from the workbook, the Start command only adds buttons for the add-in
from which the command was called.
The picture shows the ORMM menu when the DP add-in collection is
installed. Each add-in has an individual role. The DP Data add-in accepts
data concerning general situations and calls the DP Models add-in to
construct models. The DP Models add-in calls the DP Solver add-in to
obtain numerical solutions.
The DP Models add-in also models Markov Chain problems. These are
described by states and events alone. No actions are involved. Models
constructed can either be analyzed with the DP Solver add-in or the
Markov Analysis add-in.
Markov Decision Processes are subsumed under this discussion as they
are similar to stochastic dynamic programming models.
Here, we provides a brief introductions to the three add-ins:

26. Although the add-ins do not restrict the size of the problem entered, some computational methods are
impractical for large problems. The size ofthe worksheetin terms ofrows and columns limits some
models. Some computations in the DP Solver add-in use the Excel INVERSE function that is
limited to about 50 stages, although the limit seems to be greater for Excel 2007. The size of a model, as
measured by the number of stages, decisions, and transitions, is an important feature of a DP model
because for many problems these numbers may be very large. In the parlance of dynamic programming
this is the curse of dimensionality.
Dynamic Programming Models
The DP Models add-in provides a structure to abstractly define several kinds of models. The first step is
to choose the kind of model. Four model types are allowed.
add-ins consume part of the memory allotted to Excel. Add-ins also increase the time required for Excel
to launch. If you experience trouble, a possible remedy is to decrease the number of add-ins installed.
This section describes a general model form constructed on an Excel workbook that seems to be useful
for a variety of problems that are related to DP. It is too early to tell if the structure is sufficient for all
problems that involve stages, States and events, but it has been used for several problems that have
appeared in the literature.
Stages Stages Events Stage
Subsets
Decisions Transitions
Markov Chain X X Optional X
Deterministic DP X X Optional Optional X
Stochastic DP
or Markov Decision
Process
X X X Optional Optional X
States and transitions are required for every model type. Stages and decisions are added for optimization
problems. Events are added for problems involving probabilities. The optional elements are useful for
restricting combinations that are infeasible. For example the State Subsets element is used to identify
subsets of stages that have different characteristics. Most deterministic DP models start at some Initial
state and finish at one or more Final stages. The State Subset element is used to define the stages where
the process terminates.

27. The example of allocation problem With DP Solver
The DDP model of the resource problem has only states and actions. No events are required because there
is no uncertainty in this model. We use the four resource model as an illustration.
This is the linear version of the problem that for most instances is easily solved by math programming
solution techniques. The add-in creates a DP version of the problem. It can handle separable nonlinear as
well as linear models. Here we consider the Resource problem with a deterministic DP model. When this
page is written, only the DDP model is available.
•
•
• The facility has four machines of type 1, five of type 2, three of type 3 and seven of type 4. Each
machine operates 40 hours per week. The problem is to determine the optimum weekly
production quantities for the products. The goal is to maximize total profit.
• Maximize Z = 18X1+ 25X2 + 10X3 + 12X4 + 15X5
• Solution: X1 = 59, X2 = 62, X3 = 0, X4 = 10
Machine Quantity Product 1 Product 2 Product 3 Product
4
Product 5
M1 160 h 1.2 1.3 0.7 0.0 0.5
M2 200 h 0.7 2.2 1.6 0.5 1.0
M3 120 h 0.9 0.7 1.3 1.0 0.8
M4 280 h 1.4 2.8 0.5 1.2 0.6
Unit profit, $ —— 18 25 10 12 15

28. Fill the name field with a short name with no spaces or punctuation. The name, number of items, and
number of resources are fixed once the worksheet is created. The maximum of each item can be changed
on the data worksheet.
By checking the Make Random Problem box, the model parameters are assigned randomly. The
Maximize Objective button determines the direction of optimization.
• Clicking the Build Data Table button constructs tables for the revenue and resource use data. This
allows the representation of nonlinear forms that calls the add-in to make tables for Revenue and
resource use as a function of the number of items selected.
Clicking the Build Model button on the top of the data worksheet, calls the DP Models add-in to construct
the model worksheet. The Build/Modify Model button also constructs the model worksheet, but first
presents the Model dialog.
Z = 18X1+ 25X2 + 10X3 + 12X4 + 15X5
Suppose Z= 18 𝑥1
2
then the row for item one should be
changed to 18, 72, 162, and so on.

29. The solver cannot solve a problem and return error message. Since The enumeration process finds all
combinations of five state variables. 6*161*201*121*281 = 6,601,838,166, and it is beyond the capacity
of excel to handle it, then the software cannot solve a problem.
DP2PN2Solver
The solver software DP2PN2Solver presented in this section is a general, flexible, and expandable
software tool that solves DP problems. It consists of modules on two levels. A level one module takes the
specification of a discrete DP problem instance as input and produces an intermediate Petri net (PN)
representation called Bellman net (Lew, 2002; Lew, Mauch, 2003, 2004) as output — a middle layer,
which concisely captures all the essential elements of a DP problem in a standardized and mathematically
precise fashion. The optimal solution for the problem instance is computed by an “executable” code (e.g.
Java, Spreadsheet, etc.) derived by a level two module from the Bellman net representation.
DP2PN2Solver’s unique potential lies in its Bellman net representation. In theory, a PN’s intrinsic
concurrency allows to distribute the computational load encountered when solving a single DP problem
instance to several computational units. It will tell you how and where to deploy the files to your
computer and how to invoke the two major modules DP2PN and PN2Solver.
A Petri net is a directed bipartite graph, in which the nodes represent transitions (i.e. events that may
occur, signified by bars) and places (i.e. conditions, signified by circles). The directed arcs describe which
places are pre- and/or postconditions for which transitions (signified by arrows).

30. Disadvantage of current software system reviewed by Mauch
Attempts to develop general-purpose DP computer codes have been made in the early 70’s by Hastings
(1974) and others. While Hasting’s “DYNACODE” is frequently referenced (e.g. in Hastings, 1973),
detailed information about this system is unavailable as of today. Sniedovich characterizes these codes as
problem specific and concludes that “no general-purpose dynamic programming computer codes seem to
be available commercially” (Sniedovich, 1992, p. 193). Other, non-general DP codes have been
developed in Hubert, Arabie, Meulman (2001), Lin and Bricker (1990), Sniedovich (1993, 1994). DP
software accompanying textbooks such as Hillier and Lieberman (1990) does not generalize well either.
The shortcoming of these attempts is that they are usually restricted to solve instances of one special
problem. Mailund’s DPROG system at www.daimi.au.dk/~mailund assumes that the user has a
substantial amount of expertise in software engineering practice. As of today it seems that major parts of
the system are still under construction.
The optimization package system LINGO / LINDO solves optimization problems across a wide problem
domain with a variety of solution methods. While its strength lies in the areas of linear programming, and
to some extent in integer programming and nonlinear programming, it is only of very limited use to tackle
DP problems. In particular, it is not obvious at all how to retrieve the series of optimal decisions that lead
to the optimal solution. A disadvantage of this and many other optimization software systems is that they
incorporate search mechanisms. This can in the best case be very efficient, but in the worst case not lead
to the optimal solution at all. The most common approach taken today for solving real-world DP
problems
is to start a specialized software development project for each and every problem. Even taking into
account that a module-oriented development philosophy allows considerable reuse of components, this is
still a rather expensive approach. A general solver solution like the DP2PN2Solver introduced here
should be able to save software development costs for many DP problems arising in practice.

31. gDPS Language
DP problems can can take on various forms and shapes. This is why it is hard to specify a strict and fixed
specification format like for linear programming (LP). But there are common themes across all DP
problems and those are captured in the general DP specification (gDPS) language. A special purpose
language should allow for both a convenient description of a DP problem instance and efficient parsing of
the specification. The gDPS language has been designed as a general source language that can describe a
variety of DP problems. It offers the flexibility needed for the various types of DP problems arise in
reality.
Advantage of using DP2PN2SOLVER
• Automatically solve many optimization problems of practical relevance with DP
• The DP2PN2Solver user no longer needs deep skills in software development to solve a problem
• It is no longer necessary to worry about the DP solution process itself
• DP2PN2Solver guarantees that the optimal solution to a DP instance will be found, provided of
course that the DP functional equation is correct and its solution does exist.
• The gDPS language is easy to learn.
Solving Traveling salesman with DP2PN2Solver
For instance let,
The procedure in getting the solution is the following:
1. Invocation of DP2PN
2. Consequences of Invocation: Desired Bellman net
3. Invocation of PN2Solver
4. Consequences of Invocation:
1. Spreadsheet solver: update the cell and get new solution
2. Java Solver
3. Solution Tree in text
The solution tree of TSP:
• The optimal value is: 39.0
• The solution tree is:

32. State (0,{0}) has optimal value: 39.0 Decision d=1 State (1,{0,1}) has optimal value: 38.0 Decision d=3
State (3,{0,1,3}) has optimal value: 35.0 Decision d=4 State (4,{0,1,3,4}) has optimal value: 20.0
Decision d=2 Base state (2,{0,1,2,3,4}) has initial value: 7.0
Conclusion
This report surveyed softwares that could be used to solve dynamic programming problems. It can be
seen from the report that even though there does not exist a specific software to handle dynamic
programming problems, a variety of open source and commercially available softwares can solve a range
of DP problems. Some of the softwares highlighted in this report are,LINDO,TORA,MATLAB, APML,
Excel and DP2PN2 solver. Although not all of the solvers use a backward recursion approach, the
algorithms embedded in the solvers are a good fit to the problem definitions. A lot of research is going on
in this area where scripting languages are used to create toolboxes for such softwares that would handle
more complex problems.

33. References
http://home.ubalt.edu/ntsbarsh/Business-stat/opre/LindoCitrix.pdf
https://www.youtube.com/watch?v=qjM_5RP-eJw
http://taha.ineg.uark.edu/tora.html
http://www.lindo.com/index.php?option=com_content&view=article&id=1&Itemid=9
http://www.ampl.com/BOOKLETS/ampl-osl.pdf
http://www.ampl.com/BOOKLETS/ampl-cplex3.pdf
http://ampl.com/products/solvers/solvers-we-sell/minos/
http://www.math.clemson.edu/~mjs/courses/mthsc.440/integer
http://ampl.com/BOOK/CHAPTERS/21-nonlinear.pdf
http://ampl.com/products/solvers/open-source/
https://www.tu-
chemnitz.de/mathematik/part_dgl/teaching/WS2009_Grundlagen_der_Optimierung/amplguide.pdf
http://www.mathworks.com/help/gads/examples/performing-a-multiobjective-optimization-using-the-
genetic-algorithm.html
http://www.mathworks.com/help/gads/gamultiobj.html
http://www.mathworks.com/help/gads/using-gamultiobj.html
http://www.mathworks.com/help/optim/ug/constrained-nonlinear-optimization-algorithms.html#f26684

34. Appendix
Travelling Salesman Problem
%=======Traveling Salesman Problem Dynamic Programming Function ===========
%= This function solves the Traveling Salesman Problem (TSP) using Dynamic=
%= programming (DP). =
%= The function is based on the paper by Held and Karp from 1962. The DP =
%= is guaranteed to provide the accurate (optimal) result to the TSP, but =
%= the time complexity of this algorithm is O(2^n n^2), which limits the =
%= the use of this algorithm to 15 cities or less. =
%= NOTE: For reasonable runtime, please do not try to calculate a tour of =
% more than 13 cities. DP is not for large sets of cities. =
%= Inputs: =
% Cities: an n-by-2 (or n-by-3) matrix of city locations. If this =
% input is not given, a set of 10 cities will be randomly =
% generated. =
% Dmatrix: This is an n-by-n matrix of the distances between the =
% cities. If not given, this matrix will be calculated at =
% initialization. =
%= Outputs: =
% OptimalTour: a vector with indices of the cities, according to the=
% the optimal order of cities to visit. It is a closed tour =
% that will return to city 1, which is assumed to be the =
% salesman's origin (depot). =
%= mincost: the total length of the optimal route. =
%= Developed by: Elad Kivelevitch =
%= Version: 1.0 =
%= Date: 15 May 2011 =
%==========================================================================
function [OptimalTour,mincost]=tsp_dp1(cities, Dmatrix)
% Initialization Process
if nargin==0
cities=random('unif',0,100,10,2);
end
[NumOfCities,dummy]=size(cities);
Primes=primes(NumOfCities*10);
if nargin<2 %No Dmatrix used
D=diag(inf*ones(1,NumOfCities)); %Assign an inifinite cost to traveling
from a city to itself
for i=1:NumOfCities %Assign the distances between pairs of cities
for j=i+1:NumOfCities
D(i,j)=norm(cities(i,:)-cities(j,:));
D(j,i)=D(i,j);
end
end
else
D=Dmatrix;
end
NumOfDataSets=1;
for i=2:NumOfCities
NumOfDataSets=NumOfDataSets+nchoosek(NumOfCities,i);
end
Data(NumOfDataSets).S=[];

35. Data(NumOfDataSets).l=0;
Data(NumOfDataSets).cost=inf;
Data(NumOfDataSets).pre=[];
Data(NumOfDataSets).m=[];
LookUpTable(NumOfDataSets)=0;
%Define a data structure that holds the following pieces of data we need
%for later. This data structure uses the same notation used in the paper
%by Held and Karp (1962):
% S - the set of cities in the tour.
% l - the last city visited in the set S.
% cost - the cost of a tour, which includes all city in S and ends at l.
%In addition, the following data items are used in the dataset for reducing
%runtime:
% Pre - the index of predecessor dataset, i.e. the one with Spre=S-{l}
% m - the city in S-{l} that yielded the lowest cost C(Spre,m)+D(m,l).
% This index will facilitate the generation of the optimal tour without
% further calculations.
Data(1).S=[1];
Data(1).l=1;
Data(1).cost=0;
Data(1).Pre=[];
Data(1).m=[];
for s=2:NumOfCities
Data(s).S=[Data(1).S,s];
Data(s).l=s;
Data(s).cost=D(s,1);
Data(s).Pre=1;
Data(s).m=1;
LUT=calcLUT(Data(s).S,s,Primes);
LookUpTable(s)=LUT;
end
IndexStartPrevStep=2; %index into Data that marks the beginning of the
previous step
IndexLastStep=NumOfCities; %index into Data that marks the end of the
previous step
CurrentData=IndexLastStep; %index into Data that marks the current dataset
%This is the main Dynamic Programming loop
for s=3:NumOfCities
%generate possible sets with s-1 cities out of the possible N-1 cities
%(not including city 1)
TempSets=nchoosek(2:NumOfCities,s-1);
NumOfSets=size(TempSets);
for j=1:NumOfSets(1) %iterate through all the sets
for k=1:NumOfSets(2) %iterate through all the elements in each set
SminuskSet=[1,TempSets(j,1:k-1),TempSets(j,k+1:NumOfSets(2))];
%this is the set S-{k}
candidatecost(2:length(SminuskSet))=inf;
indices=[];
for mm=2:length(SminuskSet) %choose a city in S-{k} that will be
last
LUV=calcLUT(SminuskSet,SminuskSet(mm),Primes);
index=find(LUV==LookUpTable(IndexStartPrevStep:IndexLastStep));
index=index+IndexStartPrevStep-1;
if index==0
candidatecost(mm)=inf;

36. else
candidatecost(mm)=Data(index).cost+D(SminuskSet(mm),TempSets(j,k));
indices(mm)=index;
end
end
[mincost,indexcost]=min(candidatecost(2:end));
CurrentData=CurrentData+1;
Data(CurrentData).S=[1,TempSets(j,:)];
Data(CurrentData).l=TempSets(j,k);
Data(CurrentData).cost=mincost;
Data(CurrentData).Pre=indices(indexcost+1);
Data(CurrentData).m=SminuskSet(indexcost+1);
LookUpTable(CurrentData)=calcLUT(Data(CurrentData).S,TempSets(j,k),Primes);
end
end
IndexStartPrevStep=IndexLastStep+1;
IndexLastStep=CurrentData;
end
mm=0;
%Now add the distance back from the last city to city 1
for i=IndexStartPrevStep:IndexLastStep
mm=mm+1;
candidatecost(mm)=Data(i).cost+D(Data(i).l,1);
end
%Find the one that minimizes the total distance
[mincost,indexcost]=min(candidatecost);
Temp=Data(IndexStartPrevStep+indexcost-1);
%Generate the optimal tour by traversing back from the last city to its
%predecessors
OptimalTour=1;
while ~isempty(Temp.Pre)
OptimalTour=[OptimalTour,Temp.l];
Temp=Data(Temp.Pre);
end
OptimalTour=[OptimalTour,1];
function LUT=calcLUT(vec,last,Primes)
j=length(vec);
LUT=Primes(last);
for i=2:j
LUT=LUT*Primes(vec(i));
end
Output