Scilab optimization workshop

The free and open source software for numerical computation
Optimization with Scilab
June 29th
2011
Michaël BAUDIN & Vincent COUVERT
Scilab Consortium

The free and open source software for numerical computationThe free software for numerical computation
Part 1 - What's new in Scilab 5 ?
Focus on the Nelder-Mead component
Part 2 - Optimization in Scilab: Matlab® compatibility
Part 3 - OMD2 project: Scilab Platform Development
Part 4 - Conclusion
What is missing in Scilab ?
Outline

1. Introduction
1.1 What's in Scilab ?
1.2 What's new in Scilab v5 ?
1.3 What's new on Atoms ?
Part 1 – What's new in Scilab 5 ?
2. The Nelder-Mead Component
2.1 Introduction
2.2 The algorithm
2.3 Test cases
2.4 Conclusions

Objective Bounds Equality Inequalities Size Gradient
Needed
Solver
Linear yes linear linear medium - linpro
Quadradic yes linear linear medium - quapro
Quadratic yes linear linear large - qpsolve
Quadratic yes linear linear medium - qld
Non-Linear yes large yes optim
Non-Linear small no fminsearch
Non-Linear yes small no neldermead
Non-Linear yes small no optim_ga
Non-Linear small no optim_sa
N.Li.Lea.Sq. large optional lsqrsolve
N.Li.Lea.Sq. large optional leastsq
Min-Max yes medium yes optim/nd
Multi-Obj yes small no optim_moga
Semi-Def. lin. (spectral) large no semidef
L.M.I. lin. (spectral) lin. (spectral) large no lmisolve

● Genetic Algorithms: nonlinear objective, bounds, global optimization
● Simulated Annealing: nonlinear objective, global optimization
● The Nelder-Mead component: nonlinear objective, unconstrained,
derivative-free, local optimization
● fminsearch: Matlab® compatible
1.2 What's new in Scilab 5 ?

● Optimization Solvers:
– Quapro: linear or quadratic objective, linear constraints (full
matrices),
– SciIpopt: an interface to Ipopt. Nonlinear objective, nonlinear
constraints (beta version),
– Fmincon: nonlinear objective, nonlinear constraints (alpha
version) – Matlab® compatible,
– Other modules: Cobyla, Particle Swarm Optimization,
Optkelley, …
● Test Problems:
– Uncprb: 35 unconstrained optimization problems,
– AMPL: load AMPL problems into Scilab,
– And also: CUTEr.
1.3 What's new on ATOMS ?

Outline
1. Introduction
1.2 What's new in Scilab v5 ?
1.3 What's new on Atoms ?
2.1 Introduction
2.2 The algorithm
2.3 Test cases
2.4 Conclusions

John Ashworth Nelder (8 October 1924 – 7 August 2010)
Source: http://www.guardian.co.uk/technology/2010/sep/23/john-nelder-obituary
2.1 Introduction

● We are interested in solving the unconstrained continuous
optimization problem:
Minimize f(x)
with unbounded, real, multidimensional, variable x.
● A direct search algorithm:
– Uses only function values (no gradient needed),
– Does not approximate the gradient.
● « A simplex method for function minimization », John Nelder, Roger
Mead, Computer Journal, vol. 7, no 4, 1965, p. 308-313
2. The Nelder-Mead Algorithm
2.1 Introduction

2.1 Introduction
Virginia Torczon (1989) writes:
"Margaret Wright has stated that over fifty percent of the calls
received by the support group for the NAG software library
concerned the version of the Nelder-Mead simplex algorithm to be
found in that library."

A simplex: a set of n+1 vertices, in n dimensions.
In 2 dimensions.
In 3 dimensions.
2.2 The algorithm

● Steps in the Nelder-Mead algorithm
● Inputs:
– the n+1 vertices v(1), v(2), ..., v(n+1) of a nondegenerate
simplex in n dimensions,
– the associated function values f(1),...,f(n+1),
– the coefficients ρ (reflection), χ (expansion), γ (contraction),
and σ (shrinkage).
● Standard Nelder-Mead: ρ=1, χ=2, γ=1/2, and σ=1/2.
2.2 The algorithm

2.2 The algorithm

f x1, x2=x1
2
x2
2
−x1 x2
function [ y , index ] = quadratic ( x , index )
y = x(1)^2 + x(2)^2 - x(1) * x(2);
endfunction
nm = neldermead_new ();
nm = neldermead_configure(nm,"-numberofvariables",2);
nm = neldermead_configure(nm,"-function",quadratic);
nm = neldermead_configure(nm,"-x0",[2 2]');
nm = neldermead_search(nm);
xopt = neldermead_get(nm,"-xopt");
nm = neldermead_destroy(nm);
2.3 Test cases

2.3 Test cases

● Mc Kinnon, « Convergence of the neldermead simplex method to a nonstationary
point ». SIAM J. on Optimization, 1998
● Failure by repeated inside contraction
2.3 Test cases

● C. T. Kelley. « Detection and remediation of stagnation in the neldermead algorithm
using a sufficient decrease condition » SIAM J. on Optimization, 1999
● Restart the algorithm...
2.3 Test cases

● Some general facts:
– Memory requirement is O(n²)
– Shrink steps are rare
– Generally 1 or 2 function evaluations by iteration
– Convergence is slow. Typical number of iterations is 100n,
where n is the number of dimensions
– Hundreds of iterations are not rare
– Convergence can be even slower when n > 10 (Han &
Neumann, 2006)
– Restart the algorithm when in doubt for convergence (Kelley,
1999)
– Convergence is guaranteed in 1 dimension (Lagarias et al.,
1999)
2.4 Conclusions

● We should not use this algorithm just because the gradient is not
required:
– For example, if f is smooth, Quasi-Newton methods (optim)
with numerical derivatives converge much faster.
● We may use this algorithm when:
– No other property of the problem can be used (e.g. non linear
least squares can be solved by lsqrsolve),
– The objective function is nonsmooth or "noisy" (Kelley, 1999),
– We do not need too much accuracy (Torzcon, 1989),
– The number of parameters is moderate (Han & Neumann,
2006).
2.4 Conclusions

1. Introduction
2. Scilab Coverage
3. Overview
Part 3 –
Optimization in Scilab: Matlab® compatibility

● Matlab® has many functions for optimization:
– Minimization,
– Equation solving,
– Datafitting and nonlinear least squares,
– Global optimization.
● Scilab has often similar functions: let's see which ones.
Matlab is a registered trademark of The Mathworks, Inc.
1. Introduction

● For each Matlab® function, we search:
– Scilab function, if available,
– Differences of features, differences of algorithms.
● (*) : Function will be reviewed at the end of the talk,
● For most functions, the match is not 100% identical,
– But some other functions can do it : which ones ?
● We consider only Scilab Industrial Grade solvers:
– Scilab internal modules,
– ATOMS modules,
– Portables on all OS,
– Well documented,
– Tested.
1. Introduction

1. Introduction
Main differences
● Design:
– Matlab®: problem oriented (may be with several solvers),
– Scilab: solver oriented (may be several solvers).
● Function arguments:
– Matlab® nearly always provides common options,
– Scilab is less homogeneous.
● Management of the callbacks/extra-arguments:
– Matlab®: M-file or @
– Scilab: list

● Minimization:
– fminbnd Not 100% identical,
But optim can do it.
– fmincon ATOMS/fmincon (alpha version)
– fminimax Not 100% identical,
but optim/''nd'' is designed for it.
– fminsearch fminsearch 90% identical in Scilab 5.3.2.
fminsearch 99% identical in 5.4.0
2. Scilab Coverage

– fminunc Not 100% identical,
but optim/''qn'' or optim/''gc'' are designed for it.
No sparsity pattern of Hessian in Scilab.
No PCG in optim: L-BFGS instead.
– linprog 100% for full matrices: karmarkar
ATOMS/quapro: linpro
No known solver for sparse matrices (*).
– quadprog 100% for full matrices: qpsolve, qp_solve
ATOMS/quapro: quapro
No known solver for sparse matrices.
2. Scilab Coverage

● Equation Solving:
– fsolve fsolve 100% for full matrices.
No known solver with sparse Jacobian (*).
– fzero No identical function.
But fsolve can do it.
● Least Squares (Curve Fitting):
– lsqcurvefit datafit
– lsqnonlin lsqrsolve (leastsq)
2. Scilab Coverage

● Global Optimization Toolbox:
● Genetic Algorithm Not 100% identical,
But optim_ga is built-in Scilab.
No linear equality and inequality in Scilab,
but bounds are managed.
● Simulated Annealing Not 100% identical,
But optim_sa is built-in Scilab
No bounds in Scilab SA, but user can
customize the neighbour function.
2. Scilab Coverage

Matlab Problem Scilab
bintprog Binary Integer Programming -
fgoalattain Multiobjective goal attainment -
fminbd Single-variable, on interval optim
fmincon Constrained, nonlinear, multivariable ATOMS/fmincon
fminimax Minimax, constrained optim/''nd''
fminsearch Unconstrained, multivariable, derivative-free fminsearch (100%)
fminunc Unconstrained, multivariable optim/''qn'',''gc''
fseminf Semi-infinitely constrained, multivariable,
nonlinear
-
ktrlink Constrained or unconstrained, nonlinear,
multivariable using Knitro
-
linprog Linear programming karmarkar,
ATOMS/quapro
quadprog Quadratic programming qpsolve,
ATOMS/quapro
3. Overview

Matlab Problem Scilab
fsolve Solve systems of nonlinear equations fsolve
fzero Root of continuous function of one variable -
lsqcurvefi t Nonlinear least squares curve fitting datafit
lsqlin Constrained linear least squares -
lsqnonlin Nonlinear least-squares (nonlinear data-fitting) lsqrsolve, leastsq
lsqnonneg Nonnegative least squares -
optimtool GUI to select solvers, options and run problems -
Global Search Solve GlobalSearch problems -
Multi Start Solve MultiStart problems -
Genetic Algorithm Genetic Algorithms optim_ga
Direct Search Pattern Search -
Simulated Annealing Simulated Annealing optim_sa
3. Overview

Part 4 -
OMD2 project: Scilab Platform Development
1. Overview
2. Modules
2.1 Data Management
2.2 Modeling
2.3 Optimization

● OMD2 / CSDL projects collaboration
● Will be available on Scilab forge:
– http://forge.scilab.org/index.php/p/omd2/
– Private project up to first release.
● Scilab Optimization Platform:
– Batch mode (script edition, large scale execution),
– GUI mode (interactive edition, prototyping).
● Future Scilab external module available through ATOMS.
Overview

● Project management (Save & Load working data as HDF5 files)
● Wrappers:
– Scilab algorithms,
– External tools,
– Proactive.
● Mask complexity for users
● Modules:
– Data Management,
– Modeling,
– Optimization,
– Visualization.
Main functionalities

● Factors / Parameters:
– Load existing Design Of Experiments (Isight .db files, …)
– Generate Design Of Experiments:
● DoE generator wrappers (LHS, …),
● DoE generator settings.
● Responses simulation using:
– External tool (openFOAM, Catia, CCM+, …),
– Scilab function.
● 2-D visualization:
– Factor / Factor,
– Response / Factor.
Data Management Module (1/2)

Data Management Module (2/2)

● Point selection:
– Learning points used for modeling,
– Validation points used to validate model,
– Bad points (simulation issue, …).
● Modeler:
– Selected among modeler wrappers (DACE, Lolimot, …),
– Parameters configuration,
– Multiple model management with best model user selection.
● Visualization:
– 2-D models,
– Cross correlation,
– Sensibility analysis.
Modeling module (1/2)

Modeling module (2/2)

● Responses coefficients values setting
● Optimizer:
– Selection among generic wrappers (optim, fmincon, genetic
algorithms, …),
– Optimizer configuration,
– Enable two chained optimizers.
● Visualization:
– Optimal point,
– Paretos,
– Robustness.
Optimization Module (1/2)

Optimization Module (2/2)

1. What is missing ?
2. Bibliography
Part 4 - Conclusion

● High Performance Optimization:
– Use BLAS/LAPACK within optim ?
● Sparse Linear Programming:
– Update LIPSOL ?
● Non Linear Programming:
– Improve fmincon ?
● Non Linear Programming Test Cases:
– CUTEr requires a compiler on the test machine,
– Connect the Hock-Schittkowski collection ?
Conclusion
1. What is missing ?

● « Nelder-Mead User's Manual », Michaël Baudin, Consortium Scilab
– DIGITEO, 2010
● « Optimization in Scilab », Baudin, Couvert, Steer, Consortium Scilab
- DIGITEO – INRIA, 2010
● « Optimization with scilab, present and future », Michaël Baudin and
Serge Steer, 2009 IEEE International Workshop on Open Source
Software for Scientific Computation, pp.99-106, 18-20 Sept. 2009
● « Introduction to Optimization with Scilab », Michaël Baudin,
Consortium Scilab – DIGITEO, 2010
● « Unconstrained Optimality Conditions with Scilab », Michaël Baudin,
Consortium Scilab – DIGITEO, 2010
Conclusion
2. Bibliography

Thanks for your attention
www.scilab.org

Some slides you won't see, unless you ask...
Extra-Slides

Some Historical References:
● Spendley, Hext, Himsworth (1962): fixed shape simplex algorithm
● Nelder, Mead (1965): variable shape algorithm
● Box (1965): simplex algo., with constraints
● O'Neill (1971): Fortran 77 implementation.
● Torczon (1989): Multi Directional Search.
● Mc Kinnon (1998): Counter examples of N-M.
● Lagarias, Reeds, Wright, Wright (1998): Proof of convergence in
dimensions 1 and 2 for strictly convex functions.
● Han, Neumann (2006): More counter examples of N-M.
The Nelder-Mead Algorithm

In what softwares N-M can be found ?
● Matlab (fminsearch)
● NAG (E04CBF)
● Numerical Recipes (amoeba)
● IMSL (UMPOL)
● … and Scilab since v5.2.0 in 2009
● … and R after Sébastien Bihorel's port of Scilab's source code.

1. Sort by function value. Order the vertices: f(1) ≤ · · · ≤ f(n) ≤ f(n+1)
2. Calculate centroid. B = (v(1)+...+v(n))/n
3. Reflection. Compute R = (1+ρ)B − ρv(n+1) and evaluate f(R).
4. Expansion. If f(R)<f(1), compute E=(1+ρχ)B − ρχv(n+1) and evaluate f(E).
If f(E)<f(R), accept E, else accept R and goto 1.
5. Accept R. If f(1) ≤ f(R) < f(n), accept R and goto 1.
6. Outside Contraction. If f(n)≤f(R)<f(n+1), compute Co=(1+ργ)B − ργv(n+1)
and evaluate f(Co). If f(Co)<f(R), then accept Co and goto 1 else, goto 8.
7. Inside Contraction. If f(n+1)≤f(R), compute Ci=(1-γ)B +γv(n+1) and
evaluate f(Ci). If f(Ci)<f(n+1), then accept Ci and goto 1 else, goto 8.
8. Shrink. Compute the points v(i)=v(1)+σ(v(i)-v(1)) and evaluate
f(i)=f(x(i)), for i=2,3,...,n+1. Goto 1.

● Lagarias, Reeds, Wright, Wright (1998)
1. In dimension 1, the Nelder-Mead method converges to a
minimizer, and convergence is eventually M-step linear, when
the reflection parameter ρ = 1.
2. In dimension 2, the function values at all simplex vertices in the
standard Nelder-Mead algorithm converge to the same value.
3. In dimension 2, the simplices in the standard Nelder-Mead
algorithm have diameters converging to zero.
● Note that Result 3 does not implies that the simplices converge to a
single point x*.

● Minimization:
– bintprog Solve binary integer programming problems
– fgoalattain Solve multiobjective goal attainment problems
– fminbnd Find minimum of single-variable function on
fixed interval
– fmincon Find minimum of constrained nonlinear
multivariable function
– fminimax Solve minimax constraint problem
– fminsearch Find minimum of unconstrained multivariable
function using derivative-free method
What's in Matlab® ?

– fminunc Find minimum of unconstrained multivariable
function
– fseminf Find minimum of semi-infinitely constrained
multivariable nonlinear function
– ktrlink Find minimum of constrained or unconstrained
nonlinear multivariable function using KNITRO
third-party libraries
– linprog Solve linear programming problems
– quadprog Quadratic programming

● Equation Solving:
– fsolve Solve system of nonlinear equations
– fzero Find root of continuous function of one variable
● Least Squares (Curve Fitting):
– lsqcurvefit Solve nonlinear curve-fitting (data-fitting)
problems in least-squares sense
– lsqlin Solve constrained linear least-squares problems
– lsqnonlin Solve nonlinear least-squares problems
– lsqnonneg Solve nonnegative least-squares constraint
problem

● Utilities:
– optimtool GUI to select solver, optimization options, and
run problems
– optimget Optimization options values
– optimset Create or edit optimization options structure

● Global Optimization Toolbox:
– GlobalSearch Create and solve GlobalSearch
problems
– MultiStart Create and solve MultiStart problems
– Genetic Algorithm Use genetic algorithm and Optimization
Tool, and modify genetic algorithm
options
– Direct Search Use direct search and Optimization Tool,
and modify pattern search options
– Simulated Annealing Use simulated annealing and
Optimization Tool, and modify simulated
annealing options
What's in Matlab® toolboxes ?

Scilab optimization workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scilab optimization workshop

Similar to Scilab optimization workshop (20)

More from Scilab

More from Scilab (20)

Recently uploaded

Recently uploaded (20)

Scilab optimization workshop