A Review of Minimization Techniques for Nonlinear Functions

Previous article Next article A Review of Minimization Techniques for Nonlinear FunctionsH. A. Spang, IIIH. A. Spang, IIIhttps://doi.org/10.1137/1004089PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout1 E. R. Lancaster, A Method for Finding a Minimum of a Multivariate Function with Applications to the Reduction of Missile and Satellite Data, 1959, Paper No. 70, Preprints of Papers Presented at 14th National Meeting of Association of Comput-ing Machinery Richard Bellman, Dynamic programming, Princeton Univeristy Press, Princeton, N. J., 1957, 34– MR0090477 Google Scholar2 R. R. Brown, Gradient Methods for the Computer Solution of System Optimization Problems, Technical Note, 57-159, Wright Air Development Center, 1957 G. E. P. Box and , P. V. Youle, The Exploration and Exploitation of Response Surfaces: An Example of the Link Between the Fitted Surface and the Basic Mechanism of the System, Biometrics, 11 (1955), 287–323 CrossrefISIGoogle Scholar3 A. D. Booth, An application of the method of steepest descents to the solution of systems of non-linear simultaneous equations, Quart. J. Mech. Appl. Math., 2 (1949), 460–468 MR0035119 0035.07502 G. E. P. Box, The Exploration and Exploitation of Response Surfaces: Some General Considerations and Examples, Biometrics, 10 (1954), 16–60 CrossrefISIGoogle Scholar4 S. H. Brooks, A Comparison of Maximum Seeking Methods, J. Operations Res. Soc., 7 (1959), 430–457 R. R. Brown, A Generalized Computer Procedure for the Design of Optimum Systems I and II, AIEE Comm. and Electr., (1959), , (July) CrossrefISIGoogle Scholar5 A. L. Cauchy, Méthode Générale pour la Résolution des Systémes d'équations Simultanées, C. R. Read Sci., Paris, 25 (1847), 536–538 J. R. Dickinson, A Computer Program for System Optimization, Rept., R58, CAP 18, Can. GE Civilian Atomic Power Dept., 1958 Google Scholar6 Haskell B. Curry, The method of steepest descent for non-linear minimization problems, Quart. Appl. Math., 2 (1944), 258–261 MR0010667 0061.26801 J. R. Dickinson, The Use of Gradient Methods in Optimizing, Rept., R59, CAP26, Can. GE Civilian Atomic Power Dept., 1959 CrossrefGoogle Scholar7 Jean Bronfenbrenner Crockett and , Herman Chernoff, Gradient methods of maximization, Pacific J. Math., 5 (1955), 33–50 MR0075676 0066.10103 George E. Forsythe, Solving linear algebraic equations can be interesting, Bull. Amer. Math. Soc., 59 (1953), 299–329 MR0056372 0050.34603 CrossrefISIGoogle Scholar8 W. C. Davidon, Variable Metric Method for Minimization, Argonne National Lab. Rept. ANL-5990 Rev. Phys. Math. TID-4500, Office of Tech. Services., Fritz John, Extremum problems with inequalities as subsidiary conditions, Studies and Essays Presented to R. Courant on his 60th Birthday, January 8, 1948, Interscience Publishers, Inc., New York, N. Y., 1948, 187–204 MR0030135 0034.10503 Google Scholar9 R. R. Brown, , J. B. Dennis and , C. Kingsley, Design of Systems Using Digital Computers, Technical Note, 56-383, Wright Air Development Center, 1956 G. Zoutendijk, Methods of feasible directions: A study in linear and non-linear programming, Elsevier Publishing Co., Amsterdam-London-New York-Princeton, N.J., 1960ii+127 MR0129119 0097.35408 Google Scholar10 Marvin L. Stein, Gradient methods in the solution of systems of linear equations, J. Research Nat. Bur. Standards, 48 (1952), 407–413 MR0050378 CrossrefISIGoogle Scholar11 G. E. Forsythe and , T. S. Motzkin, Asymptotic Properties of the Optimum Gradient Method, Bull. Am. Math. Soc., 183 (1951), , (abstract) Google Scholar12 G. E. Forsythe, Acceleration of the Optimum Gradient Method, Preliminary Report, Bull. Am. Math. Soc., 57 (1951), 304–305, (abstract) ISIGoogle Scholar13 R. W. Finkel, The Method of Resultant Descents for the Minimization of an Arbitrary Function, 1959, Paper 71, Preprints of Papers Presented at 14th National Meeting of Association of Computing Machinery Google Scholar14 R. Courant, Variational methods for the solution of problems of equilibrium and vibrations, Bull. Amer. Math. Soc., 49 (1943), 1–23 MR0007838 0063.00985 CrossrefGoogle Scholar15 L. V. Kantorovič, On the method of steepest descent, Doklady Akad. Nauk SSSR (N. S.), 56 (1947), 233–236, Reviewed by H. B. Curry in Math. Revs., 9, 308 (1948) MR0023126 Google Scholar16 R. Hooke and , T. A. Jeeves, “Direct Search” Solution of Numerical and Statistical Problems, Scientific Paper, 10-1210-1-P1, Westinghouse Res. Lab., 1959, also Assoc. for Computing Machinery, 8, 212 (1961) Google Scholar17 C. F. Wood, Application of “Direct Search” to the Solution of Engineering Problems, Scientific Paper, 6-41210-1-P1, Westinghouse Res. Lab., 1960 Google Scholar18 George B. Dantzig, Maximization of a linear function of variables subject to linear inequalitiesActivity Analysis of Production and Allocation, Cowles Commission Monograph No. 13, John Wiley & Sons Inc., New York, N. Y., 1951, 339–347 MR0056260 0045.09802 Google Scholar19 J. E. Kelley, Jr., The cutting-plane method for solving convex programs, J. Soc. Indust. Appl. Math., 8 (1960), 703–712 10.1137/0108053 MR0118538 0098.12104 LinkISIGoogle Scholar20 J. B. Rosen, The gradient projection method for nonlinear programming. I. Linear constraints, J. Soc. Indust. Appl. Math., 8 (1960), 181–217 10.1137/0108011 MR0112750 0099.36405 LinkISIGoogle Scholar21 J. B. Rosen, The gradient projection method for nonlinear programming. II. Nonlinear constraints, J. Soc. Indust. Appl. Math., 9 (1961), 514–532 10.1137/0109044 MR0135991 0231.90048 LinkISIGoogle Scholar22 Vera Riley and , Saul I. Gass, Linear programming and associated techniques, Bibliographic Reference Series No. 5, Published for Operations Research Office, The Johns Hopkins University, by the The Johns Hopkins Press, Baltimore, Md., 1958x+613 MR0093014 0088.36601 Google Scholar23 Jack B. Dennis, Mathematical programming and electrical networks, The Technology Press of The Massachusetts Institute of Technology, Cambridge, Mass., 1959vi+186, and John Wiley and Sons MR0108400 Google Scholar24 Kenneth J. Arrow and , Leonid Hurwicz, Reduction of constrained maxima to saddle-point problems, Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1954–1955, vol. V, University of California Press, Berkeley and Los Angeles, 1956, 1–20 MR0084938 0070.05804 Google Scholar25 Kenneth Arrow and , Leonid Hurwicz, Gradient methods for constrained maxima, Operations Res., 5 (1957), 258–265 MR0089097 CrossrefISIGoogle Scholar26 W. S. Dorn, On Lagrange multipliers and inequalities, Operations Res.., 9 (1961), 95–104 MR0129488 0119.16001 CrossrefISIGoogle Scholar27 Bertram Klein, Direct use of extremal principles in solving certain optimizing problems involving inequalities, J. Operations Res. Soc. Amer., 3 (1955), 168–175 MR0068758 CrossrefISIGoogle Scholar28 H. W. Kuhn and , A. W. Tucker, Nonlinear programming, Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950, University of California Press, Berkeley and Los Angeles, 1951, 481–492 MR0047303 0044.05903 Google Scholar29 Kenneth J. Arrow, , Leonid Hurwicz and , Hirofumi Uzawa, Studies in linear and non-linear programming, With contributions by H. B. Chenery, S. M. Johnson, S. Karlin, T. Marschak, R. M. Solow. Stanford Mathematical Studies in the Social Sciences, vol. II, Stanford University Press, Stanford, Calif., 1958vii+229 MR0108399 0091.16002 Google Scholar30 J. Kiefer, Sequential minimax search for a maximum, Proc. Amer. Math. Soc., 4 (1953), 502–506 MR0055639 0050.35702 CrossrefISIGoogle Scholar31 J. Kiefer, Optimum sequential search and approximation methods under minimum regularity assumptions, J. Soc. Indust. Appl. Math., 5 (1957), 105–136 10.1137/0105009 MR0092326 0081.35801 LinkISIGoogle Scholar32 D. J. Newman, Locating the Maximum on a Unimodal Surface, 1960, Presented at 18th National Meeting of Operation Res. Society, October (unpublished) Google Scholar33 S. Johnson, Best Exploration for Maximum is Fibonaccian, Rand Corp. Rept., RM-1590 Google Scholar34 S. H. Brooks, A Discussion of Random Methods of Seeking Maxima, J. Operations Res. Soc., 6 (1958), 244–251 CrossrefISIGoogle Scholar35 D. S. McArthur, Strategy in Research—Alternative Methods for Design of Experiments, I. R. E. Trans. on Engineering Management, EM-8 (1961), , (March) Google Scholar36 Philip Wolfe, Recent developments in nonlinear programmingAdvances in Computers, Vol. 3, Academic Press, New York, 1962, 155–187 MR0146002 0111.17205 CrossrefGoogle Scholar Previous article Next article FiguresRelatedReferencesCited byDetails Novel Application of Fast Simulated Annealing Method in Brushless Motor Drive (BLMD) Dynamical Parameter Identification for Electric Vehicle Propulsion22 September 2021 Cross Ref Duct Acoustics11 May 2021 | , Vol. 361 Cross Ref System Search and Optimization Cross Ref Equilibrium chemistry down to 100 K6 June 2018 | Astronomy & Astrophysics, Vol. 614 Cross Ref Bibliography6 October 2017 Cross Ref Global Optimization on an Interval20 September 2016 | Journal of Optimization Theory and Applications, Vol. 172, No. 2 Cross Ref Global Optimization on an IntervalSSRN Electronic Journal, Vol. 4 Cross Ref Application of the Characteristic Function to Evaluate and Compare Analytical Variability in an External Quality Assessment Scheme for Serum Ethanol1 July 2015 | Clinical Chemistry, Vol. 61, No. 7 Cross Ref A Least Square Method for Parameter Estimation of RSC Sub-Codes of Turbo CodesIEEE Communications Letters, Vol. 18, No. 4 Cross Ref Optimization by Direct Search: New Perspectives on Some Classical and Modern MethodsTamara G. Kolda, Robert Michael Lewis, and Virginia Torczon4 August 2006 | SIAM Review, Vol. 45, No. 3AbstractPDF (1718 KB)A Direction Set Based Algorithm for Adaptive Least Squares Problems in Signal Processing Cross Ref Performance improvement of a hybrid optimization algorithm for electromagnetic devices designIEEE Transactions on Magnetics, Vol. 35, No. 3 Cross Ref Exponential analysis in physical phenomenaReview of Scientific Instruments, Vol. 70, No. 2 Cross Ref A direction set based algorithm for adaptive filteringIEEE Transactions on Signal Processing, Vol. 47, No. 2 Cross Ref SUMT (Revisited)Operations Research, Vol. 46, No. 6 Cross Ref A direction set based algorithm for least squares problems in adaptive signal processingLinear Algebra and its Applications, Vol. 284, No. 1-3 Cross Ref Bibliography28 August 2014 Cross Ref Partial surface and volume matching in three dimensionsIEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 9 Cross Ref Core-based portal image registration for automatic radiotherapy treatment verificationInternational Journal of Radiation Oncology*Biology*Physics, Vol. 33, No. 5 Cross Ref Automatic registration of CT and MR brain images using correlation of geometrical featuresIEEE Transactions on Medical Imaging, Vol. 14, No. 2 Cross Ref Global optimization in biology and medicineMathematical and Computer Modelling, Vol. 20, No. 6 Cross Ref Bibliography Cross Ref Medical image matching-a review with classificationIEEE Engineering in Medicine and Biology Magazine, Vol. 12, No. 1 Cross Ref NONLINEAR OPTIMIZATION OF AIR POLLUTION MONITORING NETWORKS: ALGORITHMIC CONSIDERATIONS AND COMPUTATIONAL RESULTSEngineering Optimization, Vol. 19, No. 4 Cross Ref Curve fitting XPS spectraJournal of Electron Spectroscopy and Related Phenomena, Vol. 46, No. 1 Cross Ref Distribution of the filtered output of a quadratic rectifier computed by numerical contour integrationIEEE Transactions on Information Theory, Vol. 32, No. 4 Cross Ref Computation of photoelectron counting distributions by numerical contour integration1 May 1985 | Journal of the Optical Society of America A, Vol. 2, No. 5 Cross Ref Numerical solutions of nonlinear problems of continua—IComputers & Structures, Vol. 19, No. 5-6 Cross Ref Chapter 12 Computational problems and methods Cross Ref Valuation of American call options on dividend-paying stocksJournal of Financial Economics, Vol. 10, No. 1 Cross Ref References Cross Ref Linearized ridge-path method for function minimizationJournal of Optimization Theory and Applications, Vol. 30, No. 2 Cross Ref A phaseshift analysis of experimental angular distributions of electrons elastically scattered from He, Ne and Ar over the range 0.5 to 20 eV17 January 2001 | Journal of Physics B: Atomic and Molecular Physics, Vol. 12, No. 2 Cross Ref An introduction to designh optimality with an overview of the literatureCommunications in Statistics - Theory and Methods, Vol. 7, No. 14 Cross Ref Literatur Cross Ref Minimization methods with constraintsJournal of Soviet Mathematics, Vol. 5, No. 1 Cross Ref Approximation methods for the unconstrained optimization of functions of several variablesJournal of Soviet Mathematics, Vol. 4, No. 6 Cross Ref A Predictive Model of Urban Stock and Activity: 1. Theoretical Considerations24 July 2016 | Environment and Planning A: Economy and Space, Vol. 7, No. 8 Cross Ref Nonlinear RegressionThe American Statistician, Vol. 29, No. 2 Cross Ref Structure electronique de derives sulfures—xTetrahedron, Vol. 31, No. 5 Cross Ref Optimization of variational trial functionsThe Journal of Chemical Physics, Vol. 58, No. 9 Cross Ref Experiment versus analysis: Computational techniques for the description of static material responseInternational Journal for Numerical Methods in Engineering, Vol. 5, No. 4 Cross Ref On sequential search for the maximum of an unknown functionJournal of Mathematical Analysis and Applications, Vol. 41, No. 1 Cross Ref A monte carlo evaluation of three nonmetric multidimensional scaling algorithmsPsychometrika, Vol. 37, No. 4 Cross Ref Statistics and stability of small assemblies of atomsJournal of Crystal Growth, Vol. 17 Cross Ref An adaptive randomized pattern search Cross Ref A Sequential Method Seeking the Global Maximum of a FunctionBruno O. Shubert14 July 2006 | SIAM Journal on Numerical Analysis, Vol. 9, No. 3AbstractPDF (856 KB)“Repairing” Response Surface DesignsTechnometrics, Vol. 14, No. 3 Cross Ref Optimal Real and Reactive Power Operation in a Hydrothermal SystemIEEE Transactions on Power Apparatus and Systems, Vol. PAS-91, No. 4 Cross Ref Extension of classical experimental techniques for characterizing composite-material behaviorExperimental Mechanics, Vol. 12, No. 1 Cross Ref Etude theorique par la methode CNDO II de l'addition du chlore sur les systemes ethyleniquesTetrahedron, Vol. 28, No. 22 Cross Ref Parameter Optimization-An Aid to Thermal Protection DesignJournal of Spacecraft and Rockets, Vol. 9, No. 1 Cross Ref Exploratory Calibration of a Retail Location Model Using Search by Golden Section1 December 1971 | Environment and Planning A: Economy and Space, Vol. 3, No. 4 Cross Ref A computational algorithm for the identification of nonlinear systemsJournal of the Franklin Institute, Vol. 291, No. 6 Cross Ref Location of facilities with rectangular distances among point and area destinationsNaval Research Logistics Quarterly, Vol. 18, No. 1 Cross Ref Bibliography Cross Ref Tour d'horizon : programmation non linéaire4 May 2009 | Revue française d'informatique et de recherche opérationnelle. Série rouge, Vol. 5, No. R1 Cross Ref Mathematical Optimization Techniques in Drug Product Design and Process AnalysisJournal of Pharmaceutical Sciences, Vol. 59, No. 11 Cross Ref A Computer Method for Pumping-Test AnalysisGround Water, Vol. 8, No. 5 Cross Ref Derivative synthesis of plane mechanisms to generate functions of two variablesJournal of Mechanisms, Vol. 5, No. 2 Cross Ref A Survey of Numerical Methods for Unconstrained OptimizationM. J. D. Powell18 July 2006 | SIAM Review, Vol. 12, No. 1AbstractPDF (2399 KB)BIBLIOGRAPHY Cross Ref A direction set based algorithm for adaptive filtering Cross Ref Comparison of some conjugate direction procedures for function minimizationJournal of the Franklin Institute, Vol. 288, No. 6 Cross Ref Linear Circuit Models Derived via Computer OptimizationIEEE Transactions on Education, Vol. 12, No. 3 Cross Ref Microwave Circuit Design by Digital ComputerIEEE Transactions on Microwave Theory and Techniques, Vol. 17, No. 8 Cross Ref Optimization Methods for Computer-Aided DesignIEEE Transactions on Microwave Theory and Techniques, Vol. 17, No. 8 Cross Ref Determination of orbits of planetary artificial satellites and planetary gravitational fields.AIAA Journal, Vol. 7, No. 8 Cross Ref Extension of Davidon’s Variable Metric Method to Maximization Under Linear Inequality and Equality ConstraintsDonald Goldfarb12 July 2006 | SIAM Journal on Applied Mathematics, Vol. 17, No. 4AbstractPDF (2500 KB)Some statistical considerations in multidimensional scalingPsychometrika, Vol. 34, No. 2 Cross Ref A method of solution for non-linear simultaneous algebraic equationsInternational Journal for Numerical Methods in Engineering, Vol. 1, No. 2 Cross Ref Lattice Approximations to the Minima of Functions of Several VariablesJournal of the ACM, Vol. 16, No. 2 Cross Ref A method of descent for chebyshev approximationBIT, Vol. 8, No. 4 Cross Ref Optimization of the representation of sampled data signals on orthonormal bases Cross Ref Problems of grouping individuals and the method of modalityBehavioral Science, Vol. 13, No. 6 Cross Ref Optimal Power Flow SolutionsIEEE Transactions on Power Apparatus and Systems, Vol. PAS-87, No. 10 Cross Ref Time Domain Approximations and an Active Network Realization of Transfer Functions Derived from Ideal FiltersIEEE Transactions on Circuit Theory, Vol. 15, No. 3 Cross Ref Solution of reaction and heat flow problems by nonlinear estimationThe Canadian Journal of Chemical Engineering, Vol. 46, No. 4 Cross Ref Forecasting with an Input-Output ModelIEEE Transactions on Power Apparatus and Systems, Vol. PAS-87, No. 6 Cross Ref Electron Correlation and Separated‐Pair Approximation. An Application to Berylliumlike Atomic SystemsThe Journal of Chemical Physics, Vol. 48, No. 8 Cross Ref The automatic design of fractional factorial experiments for adaptive process optimizationInformation and Control, Vol. 11, No. 5-6 Cross Ref An optimization technique for real and reactive power allocationProceedings of the IEEE, Vol. 55, No. 11 Cross Ref A selected bibliography on parameter optimization methods suitable for hybrid computation19 August 2016 | SIMULATION, Vol. 8, No. 6 Cross Ref On the Convergence of Some Feasible Direction Algorithms for Nonlinear ProgrammingDonald M. Topkis and Arthur F. Veinott, Jr.18 July 2006 | SIAM Journal on Control, Vol. 5, No. 2AbstractPDF (1369 KB)An Initial Value Method for Trajectory Optimization Problems1 1A major portion of this report was supported by Air Force Contracts AF 33(657)-7383 and AF 33(615)-1858. Cross Ref 9 Synthesis of Optimal Controls Cross Ref On the relative efficiencies of gradient methods1 January 1967 | Mathematics of Computation, Vol. 21, No. 99 Cross Ref A COMPARISON OF OPTIMIZATION METHODS FOR FITTING CURVES TO INFRARED BAND ENVELOPESCanadian Journal of Chemistry, Vol. 44, No. 24 Cross Ref The computer revolution in psychometricsPsychometrika, Vol. 31, No. 4 Cross Ref On parameter optimization of linear systemsIEEE Transactions on Automatic Control, Vol. 11, No. 4 Cross Ref Factor analysis by minimizing residuals (minres)Psychometrika, Vol. 31, No. 3 Cross Ref AN EXPERIMENTAL INVESTIGATION OF A MATHEMATICAL LEARNING MODEL1, 28 August 2014 | ETS Research Bulletin Series, Vol. 1966, No. 1 Cross Ref SOME STATISTICAL CONSIDERATIONS IN MULTIDIMENSIONAL SCALING8 August 2014 | ETS Research Bulletin Series, Vol. 1966, No. 1 Cross Ref Testing a simple structure hypothesis in factor analysisPsychometrika, Vol. 31, No. 2 Cross Ref Minimization by Successive ApproximationGerald Berman14 July 2006 | SIAM Journal on Numerical Analysis, Vol. 3, No. 1AbstractPDF (1065 KB)OPTIMIZING I — HILL-CLIMBING METHODS Cross Ref Minimal-time Control of Linear Systems using Functional Analysis†International Journal of Control, Vol. 2, No. 1 Cross Ref TESTING A SIMPLE STRUCTURE HYPOTHESIS IN FACTOR ANALYSIS1,28 August 2014 | ETS Research Bulletin Series, Vol. 1965, No. 1 Cross Ref A computational method for feedback control optimizationInformation and Control, Vol. 8, No. 2 Cross Ref Sequential Search: A Method for Solving Constrained Optimization ProblemsJournal of the ACM, Vol. 12, No. 1 Cross Ref Nonlinear Programming with the Aid of a Multiple-Gradient Summation TechniqueJournal of the ACM, Vol. 11, No. 4 Cross Ref Nonmetric multidimensional scaling: A numerical method1 June 1964 | Psychometrika, Vol. 29, No. 2 Cross Ref Some Algorithms for Minimizing a Function of Several VariablesB. V. Shah, R. J. Buehler, and O. Kempthorne13 July 2006 | Journal of the Society for Industrial and Applied Mathematics, Vol. 12, No. 1AbstractPDF (3393 KB) Volume 4, Issue 4| 1962SIAM Review History Submitted:17 October 1961Published online:18 July 2006 InformationCopyright © 1962 Society for Industrial and Applied MathematicsPDF Download Article & Publication DataArticle DOI:10.1137/1004089Article page range:pp. 343-365ISSN (print):0036-1445ISSN (online):1095-7200Publisher:Society for Industrial and Applied Mathematics