{"id":519,"date":"2025-04-09T22:42:53","date_gmt":"2025-04-09T14:42:53","guid":{"rendered":"https:\/\/www.toothlessos.xyz\/?p=519"},"modified":"2025-04-23T20:54:31","modified_gmt":"2025-04-23T12:54:31","slug":"the-optimization-set-gauss-newton-method","status":"publish","type":"post","link":"https:\/\/www.toothlessos.xyz\/index.php\/2025\/04\/09\/the-optimization-set-gauss-newton-method\/","title":{"rendered":"The optimization set: Gauss-Newton method"},"content":{"rendered":"\n\n\n<p>This log is an extension for the least square problems (introduced in a previous log: <a href=\"http:\/\/38.246.252.17:8080\/index.php\/2025\/03\/30\/the-optimization-set-qr-factorization\/\">The optimization set: QR factorization &#8211; ToothlessOS Log<\/a>). Specifically, we will look at how to apply the results to <strong>non-linear regression<\/strong> problems.<\/p>\n\n\n<div class=\"wp-block-aioseo-table-of-contents\"><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-theory\">Theory<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-least-square-problems\">Least-square problems<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-adjusted-for-non-linear-regression\">The Gauss-Newton method<\/a><\/li><\/ul><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-practice\">Practice<\/a><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-step-1-data-preparation\">Step 1: Data preparation<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-step-2-choose-a-model\">Step 2: Choose a model<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-step-3-implementation\">Step 3: Implementation<\/a><\/li><\/ul><\/li><\/ul><\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"aioseo-theory\">Theory<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"aioseo-least-square-problems\">Least-square problems<\/h3>\n\n\n\n<p>We recall that the <strong>least square solution<\/strong> is an approximate solution for <strong>overdetermined<\/strong> linear system <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-054f87a07025338170db3a4b50c1f7a9_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#120;&#32;&#61;&#32;&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"55\" style=\"vertical-align: 0px;\"\/>, with the <strong>criteria<\/strong> of minimizing the residue <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-c409433a9e2dfcdb83360a974d243f18_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#114;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"8\" style=\"vertical-align: 0px;\"\/>:<\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-bb1fb6c5ccd9cef1ee008b20bdd56998_l3.png\" height=\"22\" width=\"172\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#109;&#105;&#110;&#32;&#124;&#124;&#32;&#114;&#32;&#124;&#124;&#94;&#50;&#32;&#61;&#32;&#124;&#124;&#32;&#65;&#120;&#32;&#45;&#32;&#98;&#32;&#124;&#124;&#94;&#50;&#32;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>For these problems, we can compute the approximate solution easily: <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-79f07cadb37f981de3e7ad451672213d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#61;&#32;&#65;&#94;&#123;&#92;&#100;&#97;&#103;&#103;&#101;&#114;&#125;&#98;&#32;&#61;&#32;&#123;&#40;&#65;&#94;&#123;&#84;&#125;&#65;&#41;&#125;&#94;&#123;&#45;&#49;&#125;&#65;&#94;&#123;&#84;&#125;&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"24\" width=\"187\" style=\"vertical-align: -5px;\"\/>. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"aioseo-adjusted-for-non-linear-regression\">The Gauss-Newton method<\/h3>\n\n\n\n<p>Unfortunately, sometimes we cannot formulate problems in this nice linear form. Therefore, we have to make some adjustments: introducing <strong>the gauss-newton<\/strong> method!<\/p>\n\n\n\n<p>That sounds really familiar to the <strong>Newton-Raphson<\/strong> solver mentioned in the previous log (<a href=\"http:\/\/38.246.252.17:8080\/index.php\/2025\/03\/31\/the-optimization-set-non-linear-solvers\/\">The optimization set: Non-linear solvers &#8211; ToothlessOS Log<\/a>). Actually, the ideas behind them are quite similar, in terms of the <strong>linear approximation<\/strong> used.<\/p>\n\n\n\n<p>The following content is taken from Prof. Peng Sun&#8217;s slides of MATH304 at DKU.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"504\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-1024x504.png\" alt=\"\" class=\"wp-image-543\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1024x504.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-300x148.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-768x378.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1536x756.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image.png 1758w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"370\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-1-1024x370.png\" alt=\"\" class=\"wp-image-544\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1-1024x370.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1-300x108.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1-768x278.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1-1536x555.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-1.png 1745w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"409\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-2-1024x409.png\" alt=\"\" class=\"wp-image-545\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-2-1024x409.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-2-300x120.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-2-768x306.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-2-1536x613.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-2.png 1749w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"372\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-3-1024x372.png\" alt=\"\" class=\"wp-image-546\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-3-1024x372.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-3-300x109.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-3-768x279.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-3-1536x558.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-3.png 1755w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"410\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-4-1024x410.png\" alt=\"\" class=\"wp-image-547\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-4-1024x410.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-4-300x120.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-4-768x307.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-4-1536x615.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-4.png 1751w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"451\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-5-1024x451.png\" alt=\"\" class=\"wp-image-548\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-5-1024x451.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-5-300x132.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-5-768x338.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-5-1536x677.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-5.png 1763w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"479\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/image-6-1024x479.png\" alt=\"\" class=\"wp-image-549\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-6-1024x479.png 1024w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-6-300x140.png 300w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-6-768x359.png 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-6-1536x719.png 1536w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/image-6.png 1722w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"aioseo-practice\">Practice<\/h2>\n\n\n\n<p>In the practice section, we will implement the gauss-newton method on GDP data of China from scratch with <strong>MATLAB<\/strong>. This section is designed with reference to: <a href=\"https:\/\/daddyawesome.github.io\/data_science\/non%20linear%20regression%20analysis\/python\/machine%20learning\/2020\/04\/27\/MachineLearningRegNonLinearRegression.html\">Non Linear Regression Analysis | My Data Science Projects<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"aioseo-step-1-data-preparation\">Step 1: Data preparation<\/h3>\n\n\n\n<p>For the dataset, we will use the China GDP data from 1960 to 2014. The dataset can downloaded <a href=\"https:\/\/s3-api.us-geo.objectstorage.softlayer.net\/cf-courses-data\/CognitiveClass\/ML0101ENv3\/labs\/china_gdp.csv\" title=\"\">here<\/a>.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#24292e\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"% Data preparation\ndata = readtable(&quot;china_gdp.csv&quot;);\n\n% Preview\ndata\nplot(data, &quot;Year&quot;, &quot;Value&quot;);\n\n% Normalization\ndata.Year = data.Year \/ max(data.Year);\ndata.Value = data.Value \/ max(data.Value);\nplot(data, &quot;Year&quot;, &quot;Value&quot;);\" style=\"color:#e1e4e8;display:none\" aria-label=\"\u590d\u5236\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki github-dark\" style=\"background-color: #24292e\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #6A737D\">% Data preparation<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data = readtable(<\/span><span style=\"color: #9ECBFF\">&quot;china_gdp.csv&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Preview<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data, <\/span><span style=\"color: #9ECBFF\">&quot;Year&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;Value&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Normalization<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data.Year = data.Year <\/span><span style=\"color: #F97583\">\/<\/span><span style=\"color: #E1E4E8\"> max(data.Year);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data.Value = data.Value <\/span><span style=\"color: #F97583\">\/<\/span><span style=\"color: #E1E4E8\"> max(data.Value);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data, <\/span><span style=\"color: #9ECBFF\">&quot;Year&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;Value&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p>(Data preview)<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"444\" height=\"348\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/untitled1.png\" alt=\"\" class=\"wp-image-537\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/untitled1.png 444w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/untitled1-300x235.png 300w\" sizes=\"auto, (max-width: 444px) 100vw, 444px\" \/><\/figure>\n\n\n\n<p>\uff08Normalized\uff09<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"427\" height=\"332\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/norm_data.png\" alt=\"\" class=\"wp-image-539\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/norm_data.png 427w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/norm_data-300x233.png 300w\" sizes=\"auto, (max-width: 427px) 100vw, 427px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"aioseo-step-2-choose-a-model\">Step 2: Choose a model<\/h3>\n\n\n\n<p>Based on the growth trend of the data, we choose 2 non-linear models to fix it: the <strong>exponential function<\/strong> and the <strong>sigmoid\/logistic function<\/strong>.<\/p>\n\n\n\n<p><em>The exponential function<\/em>:<\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 19px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-64c616501f6aef7d22d09de21d57a475_l3.png\" height=\"19\" width=\"158\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#102;&#40;&#120;&#124;&#97;&#44;&#98;&#44;&#99;&#41;&#32;&#61;&#32;&#97;&#32;&#43;&#32;&#98;&#99;&#94;&#120;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>, where <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-1c8fa84096b6a255b6d9bd6e90ac83b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;&#32;&#92;&#110;&#101;&#32;&#48;&#44;&#32;&#99;&#32;&#62;&#32;&#48;&#44;&#32;&#99;&#32;&#92;&#110;&#101;&#32;&#49;\" title=\"Rendered by QuickLaTeX.com\" height=\"17\" width=\"135\" style=\"vertical-align: -4px;\"\/> and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-ede05c264bba0eda080918aaa09c4658_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"10\" style=\"vertical-align: 0px;\"\/> is any real number<\/p>\n\n\n\n<p>Now, in order to construct the matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-1a516b672dd1ad729114aed2410398fc_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#91;&#90;&#95;&#123;&#106;&#125;&#93;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"25\" style=\"vertical-align: -6px;\"\/>, we want to compute the partial derivatives of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-a7ee323bc5a3f73ad5e066b13bed5504_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#102;&#40;&#120;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"34\" style=\"vertical-align: -5px;\"\/> with respect to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-5c53d6ebabdbcfa4e107550ea60b1b19_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#97;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"9\" style=\"vertical-align: 0px;\"\/>, <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-f56d50c26583f9a035ff6b4e3c0ca5c0_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-41a04eeea923a1a0c28094a8a4680525_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#99;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"8\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-70989434ddd2178e0269ec9687c807c8_l3.png\" height=\"38\" width=\"79\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#97;&#125;&#32;&#61;&#32;&#49;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-bba2e40bf542c229330f9389486b4a91_l3.png\" height=\"38\" width=\"87\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#98;&#125;&#32;&#61;&#32;&#99;&#94;&#120;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-71b3f3a397b2406f211b3b53bbef4201_l3.png\" height=\"38\" width=\"121\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#99;&#125;&#32;&#61;&#32;&#98;&#120;&#99;&#94;&#123;&#120;&#45;&#49;&#125;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p><em>The sigmoid function<\/em>:<\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-9df563c3b77d2a5db95b2ef1f3240228_l3.png\" height=\"38\" width=\"219\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#102;&#40;&#120;&#124;&#97;&#44;&#98;&#44;&#99;&#44;&#100;&#41;&#32;&#61;&#32;&#97;&#32;&#43;&#32;&#92;&#102;&#114;&#97;&#99;&#123;&#98;&#125;&#123;&#49;&#43;&#99;&#94;&#123;&#120;&#45;&#100;&#125;&#125;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>Similarly, we need to find the partial derivatives:<\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-70989434ddd2178e0269ec9687c807c8_l3.png\" height=\"38\" width=\"79\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#97;&#125;&#32;&#61;&#32;&#49;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 40px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-b4b9d19f5cf1959cd8e4e838c0b83385_l3.png\" height=\"40\" width=\"138\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#98;&#125;&#32;&#61;&#32;&#92;&#102;&#114;&#97;&#99;&#123;&#49;&#125;&#123;&#49;&#43;&#99;&#94;&#123;&#120;&#45;&#100;&#125;&#125;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-abacc3b4b8e23c1f6a4f2cbc4d74dc0e_l3.png\" height=\"38\" width=\"295\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#99;&#125;&#32;&#61;&#32;&#45;&#98;&#40;&#49;&#43;&#99;&#94;&#123;&#120;&#45;&#100;&#125;&#41;&#94;&#123;&#45;&#50;&#125;&#40;&#120;&#45;&#100;&#41;&#99;&#94;&#123;&#120;&#45;&#100;&#45;&#49;&#125;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 38px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-8b1e81b2bc6cc95518042491e0c82c82_l3.png\" height=\"38\" width=\"246\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#91;&#92;&#102;&#114;&#97;&#99;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#102;&#40;&#120;&#41;&#125;&#123;&#92;&#112;&#97;&#114;&#116;&#105;&#97;&#108;&#32;&#100;&#125;&#32;&#61;&#32;&#98;&#40;&#49;&#43;&#99;&#94;&#123;&#120;&#45;&#100;&#125;&#41;&#94;&#123;&#45;&#50;&#125;&#99;&#94;&#123;&#120;&#45;&#100;&#125;&#108;&#110;&#40;&#99;&#41;&#92;&#93;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"aioseo-step-3-implementation\">Step 3: Implementation<\/h3>\n\n\n\n<p>When playing around with the implementation, I find that <strong>preprocessing of the data<\/strong> is very important. If the data is not aligned with the target function , we could face horrible <strong>divergence<\/strong> and numerical <strong>instability<\/strong> issues. This is especially important for <strong>exponential functions<\/strong> which can explodes to infinity! <\/p>\n\n\n\n<p>Therefore, the implementation is a bit different for here. We fix the parameters <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-5c53d6ebabdbcfa4e107550ea60b1b19_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#97;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"9\" style=\"vertical-align: 0px;\"\/> = 0 and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-f56d50c26583f9a035ff6b4e3c0ca5c0_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> = 1 and only update <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/ql-cache\/quicklatex.com-41a04eeea923a1a0c28094a8a4680525_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#99;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"8\" style=\"vertical-align: 0px;\"\/>, making it the simple form of the exponential function. At the same time, we shift and scale the original dataset so that it passed through (1, 1), aligning it with the exponential function. After this, we are able to get better fitting results.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#24292e\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"% Fitting with exponential functions\n% Data preparation\ndata = readtable(&quot;china_gdp.csv&quot;);\n\n% Preview\ndata\nplot(data, &quot;Year&quot;, &quot;Value&quot;);\n\n% Normalization (scale and shift)\ndata_norm = data;\ndata_norm.Year = (data.Year \/ max(data.Year) - 1) * 200;\ndata_norm.Value = data.Value \/ max(data.Value);\nplot(data_norm, &quot;Year&quot;, &quot;Value&quot;);\n\n% Case 1: Using exponential function\n% Initialize the parameters\na = 0;\nb = 1;\nc = 1;\n\n% Parameter delta matrix\ndelta_A = zeros(1,1);\n\n% Convergence threshold\neps = 1e-5;\n\n% Set the max iterations here\nfor i=1:100\n    % Jacobian(Derivatives) matrix\n    % Vectorized implementation\n    Z = zeros(length(data_norm.Year),3);\n    % Partial derivative: a\n    Z(:,1) = 1;\n    % Partial derivative: b\n    Z(:,2) = c.^data_norm.Year;\n    % Partial derivative: c\n    Z(:,3) = b.*data_norm.Year.*c.^(data_norm.Year-1);\n    \n    % The difference matrix\n    D = zeros(length(data_norm.Year),1);\n    D(:,1) = data_norm.Value - (a + b*c.^data_norm.Year);\n\n    % Now, we can solve for least square solution delta_A\n    % This tells us how to update the parameters\n    % We use QR factorization here\n    [Q, R] = qr(Z);\n    delta_A = Q'*D\\R;\n\n    update = abs(delta_A(3)\/c);\n\n    % a = a + delta_A(1);\n    % b = b + delta_A(2);\n    c = c + delta_A(3);\n\n    % Check convergence condition\n    if update &lt; eps\n        disp(&quot;Convergence cond reach at &quot; + i + &quot; iteration&quot;)\n        break\n    end\nend\n\ndisp(&quot;Update: &quot; + update);\ndisp(&quot;Loss: &quot; + norm(data_norm.Value - (a + b*c.^data_norm.Year)))\nx_axis = linspace(-6,0,100);\ny_axis = a + b*c.^x_axis;\nplot(data_norm.Year, data_norm.Value, x_axis, y_axis);\" style=\"color:#e1e4e8;display:none\" aria-label=\"\u590d\u5236\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki github-dark\" style=\"background-color: #24292e\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #6A737D\">% Fitting with exponential functions<\/span><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Data preparation<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data = readtable(<\/span><span style=\"color: #9ECBFF\">&quot;china_gdp.csv&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Preview<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data, <\/span><span style=\"color: #9ECBFF\">&quot;Year&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;Value&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Normalization (scale and shift)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data_norm = data;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data_norm.Year = (data.Year <\/span><span style=\"color: #F97583\">\/<\/span><span style=\"color: #E1E4E8\"> max(data.Year) <\/span><span style=\"color: #F97583\">-<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">) <\/span><span style=\"color: #F97583\">*<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #79B8FF\">200<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data_norm.Value = data.Value <\/span><span style=\"color: #F97583\">\/<\/span><span style=\"color: #E1E4E8\"> max(data.Value);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data_norm, <\/span><span style=\"color: #9ECBFF\">&quot;Year&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;Value&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Case 1: Using exponential function<\/span><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Initialize the parameters<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">a = <\/span><span style=\"color: #79B8FF\">0<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">b = <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">c = <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Parameter delta matrix<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">delta_A = zeros(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Convergence threshold<\/span><\/span>\n<span class=\"line\"><span style=\"color: #79B8FF\">eps<\/span><span style=\"color: #E1E4E8\"> = <\/span><span style=\"color: #79B8FF\">1e-5<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Set the max iterations here<\/span><\/span>\n<span class=\"line\"><span style=\"color: #F97583\">for<\/span><span style=\"color: #E1E4E8\"> i=<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">:<\/span><span style=\"color: #79B8FF\">100<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Jacobian(Derivatives) matrix<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Vectorized implementation<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z = zeros(length(data_norm.Year),<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: a<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">) = <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: b<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">2<\/span><span style=\"color: #E1E4E8\">) = c.^data_norm.Year;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: c<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">) = b.*data_norm.Year.*c.^(data_norm.Year-<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% The difference matrix<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    D = zeros(length(data_norm.Year),<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    D(:,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">) = data_norm.Value <\/span><span style=\"color: #F97583\">-<\/span><span style=\"color: #E1E4E8\"> (a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> b*c.^data_norm.Year);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Now, we can solve for least square solution delta_A<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% This tells us how to update the parameters<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% We use QR factorization here<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    [Q, R] = qr(Z);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    delta_A = Q<\/span><span style=\"color: #F97583\">&#39;<\/span><span style=\"color: #E1E4E8\">*D\\R;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    update = abs(delta_A(<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">)\/c);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% a = a + delta_A(1);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% b = b + delta_A(2);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    c = c <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> delta_A(<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Check convergence condition<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #F97583\">if<\/span><span style=\"color: #E1E4E8\"> update <\/span><span style=\"color: #F97583\">&lt;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #79B8FF\">eps<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">        disp(<\/span><span style=\"color: #9ECBFF\">&quot;Convergence cond reach at &quot;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> i <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #9ECBFF\">&quot; iteration&quot;<\/span><span style=\"color: #E1E4E8\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">        <\/span><span style=\"color: #F97583\">break<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #F97583\">end<\/span><\/span>\n<span class=\"line\"><span style=\"color: #F97583\">end<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">disp(<\/span><span style=\"color: #9ECBFF\">&quot;Update: &quot;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> update);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">disp(<\/span><span style=\"color: #9ECBFF\">&quot;Loss: &quot;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> norm(data_norm.Value <\/span><span style=\"color: #F97583\">-<\/span><span style=\"color: #E1E4E8\"> (a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> b*c.^data_norm.Year)))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">x_axis = linspace(-<\/span><span style=\"color: #79B8FF\">6<\/span><span style=\"color: #E1E4E8\">,<\/span><span style=\"color: #79B8FF\">0<\/span><span style=\"color: #E1E4E8\">,<\/span><span style=\"color: #79B8FF\">100<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">y_axis = a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> b*c.^x_axis;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data_norm.Year, data_norm.Value, x_axis, y_axis);<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p>Results: <strong>L2-Loss after normalization<\/strong>: 0.16822<\/p>\n\n\n\n<p>(Visualization)<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"407\" height=\"325\" src=\"http:\/\/38.246.252.17:8080\/wp-content\/uploads\/2025\/04\/exp_fit-1.png\" alt=\"\" class=\"wp-image-572\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/exp_fit-1.png 407w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2025\/04\/exp_fit-1-300x240.png 300w\" sizes=\"auto, (max-width: 407px) 100vw, 407px\" \/><\/figure>\n\n\n\n<p>Unfortunately, we ran into divergence issues again with the <strong>sigmoid function<\/strong>. <\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#24292e\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"% Data preparation\ndata = readtable(&quot;china_gdp.csv&quot;);\n\n% Preview\ndata\nplot(data, &quot;Year&quot;, &quot;Value&quot;);\n\n% Normalization (scale and shift)\ndata_norm = data;\ndata_norm.Year = data.Year \/ max(data.Year);\ndata_norm.Value = data.Value \/ max(data.Value);\nplot(data_norm, &quot;Year&quot;, &quot;Value&quot;);\n\n% Case 1: Using exponential function\n% Initialize the parameters\na = 0;\nb = 1;\nc = exp(1);\nd = 0;\n\n% Parameter delta matrix\ndelta_A = zeros(4,1);\n\n% Convergence threshold\neps = 1e-5;\n\n% Set the max iterations here\nfor i=1:1000\n    % Jacobian(Derivatives) matrix\n    % Vectorized implementation\n    Z = zeros(length(data_norm.Year),4);\n    % Partial derivative: a\n    Z(:,1) = 1;\n    % Partial derivative: b\n    Z(:,2) = 1\/(1+c.^(data_norm.Year-d));\n    % Partial derivative: c\n    Z(:,3) = -(b.*(data_norm.Year-d).*c.^(data_norm.Year-d-1)).\/(1+c.^(data_norm.Year-d)).^2;\n    % Partial derivative: d\n    Z(:,4) = (b.*c.^(data_norm.Year-d).*log(c)).\/(1+c.^(data_norm.Year-d)).^2;\n    \n    % The difference matrix\n    D = zeros(length(data_norm.Year),1);\n    D(:,1) = data_norm.Value - (a + b.\/(1+c.^(data_norm.Year-d)));\n\n    % Now, we can solve for least square solution delta_A\n    % This tells us how to update the parameters\n    % We use QR factorization here\n    [Q, R] = qr(Z);\n    delta_A = Q'*D\\R;\n\n    update = max([abs(delta_A(1)\/a) abs(delta_A(2)\/b) abs(delta_A(3)\/c) abs(delta_A(4)\/d)]);\n\n    a = a + delta_A(1);\n    b = b + delta_A(2);\n    c = c + delta_A(3);\n    d = d + delta_A(4);\n\n    % Check convergence condition\n    if update &lt; eps\n        disp(&quot;Convergence cond reach at &quot; + i + &quot; iteration&quot;)\n        break\n    end\nend\n\ndisp(&quot;Update: &quot; + update);\ndisp(&quot;Loss: &quot; + norm(data_norm.Value - (a + b\/(1+c.^(data_norm.Year-d)))));\nx_axis = linspace(0.97,1,100);\ny_axis = a + b.\/(1+c.^(x_axis-d));\nplot(data_norm.Year, data_norm.Value, x_axis, y_axis);\n\nlegend([&quot;original&quot;, &quot;fit&quot;])\" style=\"color:#e1e4e8;display:none\" aria-label=\"\u590d\u5236\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki github-dark\" style=\"background-color: #24292e\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #6A737D\">% Data preparation<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data = readtable(<\/span><span style=\"color: #9ECBFF\">&quot;china_gdp.csv&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Preview<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data, <\/span><span style=\"color: #9ECBFF\">&quot;Year&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;Value&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Normalization (scale and shift)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data_norm = data;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data_norm.Year = data.Year <\/span><span style=\"color: #F97583\">\/<\/span><span style=\"color: #E1E4E8\"> max(data.Year);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">data_norm.Value = data.Value <\/span><span style=\"color: #F97583\">\/<\/span><span style=\"color: #E1E4E8\"> max(data.Value);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data_norm, <\/span><span style=\"color: #9ECBFF\">&quot;Year&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;Value&quot;<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Case 1: Using exponential function<\/span><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Initialize the parameters<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">a = <\/span><span style=\"color: #79B8FF\">0<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">b = <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">c = exp(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">d = <\/span><span style=\"color: #79B8FF\">0<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Parameter delta matrix<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">delta_A = zeros(<\/span><span style=\"color: #79B8FF\">4<\/span><span style=\"color: #E1E4E8\">,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Convergence threshold<\/span><\/span>\n<span class=\"line\"><span style=\"color: #79B8FF\">eps<\/span><span style=\"color: #E1E4E8\"> = <\/span><span style=\"color: #79B8FF\">1e-5<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #6A737D\">% Set the max iterations here<\/span><\/span>\n<span class=\"line\"><span style=\"color: #F97583\">for<\/span><span style=\"color: #E1E4E8\"> i=<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">:<\/span><span style=\"color: #79B8FF\">1000<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Jacobian(Derivatives) matrix<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Vectorized implementation<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z = zeros(length(data_norm.Year),<\/span><span style=\"color: #79B8FF\">4<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: a<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">) = <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: b<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">2<\/span><span style=\"color: #E1E4E8\">) = <\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">\/(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">+c.^(data_norm.Year-d));<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: c<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">) = -(b.*(data_norm.Year-d).*c.^(data_norm.Year-d-<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">)).\/(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">+c.^(data_norm.Year-d)).^2;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Partial derivative: d<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    Z(:,<\/span><span style=\"color: #79B8FF\">4<\/span><span style=\"color: #E1E4E8\">) = (b.*c.^(data_norm.Year-d).*log(c)).\/(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">+c.^(data_norm.Year-d)).^2;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% The difference matrix<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    D = zeros(length(data_norm.Year),<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    D(:,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">) = data_norm.Value <\/span><span style=\"color: #F97583\">-<\/span><span style=\"color: #E1E4E8\"> (a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> b.\/(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">+c.^(data_norm.Year-d)));<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Now, we can solve for least square solution delta_A<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% This tells us how to update the parameters<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% We use QR factorization here<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    [Q, R] = qr(Z);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    delta_A = Q<\/span><span style=\"color: #F97583\">&#39;<\/span><span style=\"color: #E1E4E8\">*D\\R;<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    update = max([abs(delta_A(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">)\/a) abs(delta_A(<\/span><span style=\"color: #79B8FF\">2<\/span><span style=\"color: #E1E4E8\">)\/b) abs(delta_A(<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">)\/c) abs(delta_A(<\/span><span style=\"color: #79B8FF\">4<\/span><span style=\"color: #E1E4E8\">)\/d)]);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    a = a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> delta_A(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    b = b <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> delta_A(<\/span><span style=\"color: #79B8FF\">2<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    c = c <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> delta_A(<\/span><span style=\"color: #79B8FF\">3<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    d = d <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> delta_A(<\/span><span style=\"color: #79B8FF\">4<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #6A737D\">% Check convergence condition<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #F97583\">if<\/span><span style=\"color: #E1E4E8\"> update <\/span><span style=\"color: #F97583\">&lt;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #79B8FF\">eps<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">        disp(<\/span><span style=\"color: #9ECBFF\">&quot;Convergence cond reach at &quot;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> i <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #9ECBFF\">&quot; iteration&quot;<\/span><span style=\"color: #E1E4E8\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">        <\/span><span style=\"color: #F97583\">break<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">    <\/span><span style=\"color: #F97583\">end<\/span><\/span>\n<span class=\"line\"><span style=\"color: #F97583\">end<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">disp(<\/span><span style=\"color: #9ECBFF\">&quot;Update: &quot;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> update);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">disp(<\/span><span style=\"color: #9ECBFF\">&quot;Loss: &quot;<\/span><span style=\"color: #E1E4E8\"> <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> norm(data_norm.Value <\/span><span style=\"color: #F97583\">-<\/span><span style=\"color: #E1E4E8\"> (a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> b\/(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">+c.^(data_norm.Year-d)))));<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">x_axis = linspace(<\/span><span style=\"color: #79B8FF\">0.97<\/span><span style=\"color: #E1E4E8\">,<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">,<\/span><span style=\"color: #79B8FF\">100<\/span><span style=\"color: #E1E4E8\">);<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">y_axis = a <\/span><span style=\"color: #F97583\">+<\/span><span style=\"color: #E1E4E8\"> b.\/(<\/span><span style=\"color: #79B8FF\">1<\/span><span style=\"color: #E1E4E8\">+c.^(x_axis-d));<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">plot(data_norm.Year, data_norm.Value, x_axis, y_axis);<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #E1E4E8\">legend([<\/span><span style=\"color: #9ECBFF\">&quot;original&quot;<\/span><span style=\"color: #E1E4E8\">, <\/span><span style=\"color: #9ECBFF\">&quot;fit&quot;<\/span><span style=\"color: #E1E4E8\">])<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p>The divergence issue probability lies in the hidden assumptions of the Gauss-Newton method. For this method to work, the first-order Taylor approximation must be effective. Otherwise, it is possible that we get completely lost!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This log is an extension for the least square problems  [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[37],"class_list":["post-519","post","type-post","status-publish","format-standard","hentry","category-math","tag-optimization"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts\/519","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/comments?post=519"}],"version-history":[{"count":50,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts\/519\/revisions"}],"predecessor-version":[{"id":675,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts\/519\/revisions\/675"}],"wp:attachment":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/media?parent=519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/categories?post=519"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/tags?post=519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}