{"id":783,"date":"2026-02-28T17:07:51","date_gmt":"2026-02-28T09:07:51","guid":{"rendered":"https:\/\/www.toothlessos.xyz\/?p=783"},"modified":"2026-02-28T17:07:52","modified_gmt":"2026-02-28T09:07:52","slug":"singular-value-decomposition-svd-from-scratch-with-linear-operators","status":"publish","type":"post","link":"https:\/\/www.toothlessos.xyz\/index.php\/2026\/02\/28\/singular-value-decomposition-svd-from-scratch-with-linear-operators\/","title":{"rendered":"Singular Value Decomposition (SVD) from scratch with linear operators"},"content":{"rendered":"\n<p><strong>Singular Value Decomposition (SVD)<\/strong> is definitely one of the most important tools in Linear Algebra &#8211; and it can be tricky to understand. In this log, I will be deriving SVD from scratch with <strong>linear operators<\/strong> and reviewing its <strong>computation<\/strong> and <strong>applications<\/strong>.<\/p>\n\n\n<div class=\"wp-block-aioseo-table-of-contents\"><ul><li><a class=\"aioseo-toc-item\" href=\"#aioseo-prerequisites-2\">Prerequisites<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-derivation-3\">Derivation<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-compuation-4\">Computation<\/a><\/li><li><a class=\"aioseo-toc-item\" href=\"#aioseo-applications-6\">Applications<\/a><\/li><\/ul><\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"aioseo-prerequisites-2\">Prerequisites<\/h2>\n\n\n\n<p><strong>Key concepts and tools<\/strong>:<\/p>\n\n\n\n<p>Spectral theorem, Positive Semi-definite Operator, Self-adjoint, Normal &amp; Isometry<\/p>\n\n\n\n<p>If you are not familiar with these concepts, the book <a href=\"https:\/\/linear.axler.net\/\">Linear Algebra Done Right<\/a> is a great reference.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"713\" height=\"1024\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-713x1024.jpg\" alt=\"\" class=\"wp-image-789\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-713x1024.jpg 713w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-209x300.jpg 209w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-768x1103.jpg 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-1069x1536.jpg 1069w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-1426x2048.jpg 1426w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_1-scaled.jpg 1782w\" sizes=\"auto, (max-width: 713px) 100vw, 713px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"aioseo-derivation-3\">Derivation<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"734\" height=\"1024\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-734x1024.jpg\" alt=\"\" class=\"wp-image-791\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-734x1024.jpg 734w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-215x300.jpg 215w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-768x1071.jpg 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-1101x1536.jpg 1101w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-1468x2048.jpg 1468w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_2-scaled.jpg 1835w\" sizes=\"auto, (max-width: 734px) 100vw, 734px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"729\" height=\"1024\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-729x1024.jpg\" alt=\"\" class=\"wp-image-792\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-729x1024.jpg 729w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-214x300.jpg 214w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-768x1079.jpg 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-1094x1536.jpg 1094w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-1458x2048.jpg 1458w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_3-scaled.jpg 1823w\" sizes=\"auto, (max-width: 729px) 100vw, 729px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"aioseo-compuation-4\">Computation<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"717\" height=\"1024\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-717x1024.jpg\" alt=\"\" class=\"wp-image-793\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-717x1024.jpg 717w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-210x300.jpg 210w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-768x1098.jpg 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-1075x1536.jpg 1075w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-1433x2048.jpg 1433w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_4-scaled.jpg 1791w\" sizes=\"auto, (max-width: 717px) 100vw, 717px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"aioseo-applications-6\">Applications<\/h2>\n\n\n\n<p><strong>Key points<\/strong>: Projection, Least Squares, Image Compression &amp; PCA<\/p>\n\n\n\n<p>I&#8217;d like to elaborate a bit more on the image compression section here, which gives some important insights on <strong>matrix multiplication<\/strong>. Recall when you were learning linear algebra for the first time &#8211; matrix multiplication is just &#8220;row-column&#8221; inner product and &#8220;filling in the blanks&#8221;. However, SVD gives us a different way to think about it &#8211; taking the <strong>outer product<\/strong> instead of the inner product and factoring the matrix into the weighted sum of rank-1 approximations. That&#8217;s exactly how to compress the image &#8211; truncate at the top-k singular values!<\/p>\n\n\n\n<p>You will probably see more of these kind of decompositions if you go on to learn courses such as graph-matrix analysis. Believe me, this can be very intriguing!<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"719\" height=\"1024\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-719x1024.jpg\" alt=\"\" class=\"wp-image-794\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-719x1024.jpg 719w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-211x300.jpg 211w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-768x1094.jpg 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-1079x1536.jpg 1079w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-1438x2048.jpg 1438w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_5-scaled.jpg 1798w\" sizes=\"auto, (max-width: 719px) 100vw, 719px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"722\" height=\"1024\" src=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-722x1024.jpg\" alt=\"\" class=\"wp-image-795\" srcset=\"https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-722x1024.jpg 722w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-211x300.jpg 211w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-768x1090.jpg 768w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-1082x1536.jpg 1082w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-1443x2048.jpg 1443w, https:\/\/www.toothlessos.xyz\/wp-content\/uploads\/2026\/02\/ToothlessOS-Log-SVD_6-scaled.jpg 1804w\" sizes=\"auto, (max-width: 722px) 100vw, 722px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Singular Value Decomposition (SVD) is definitely one of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28,5],"tags":[38,9],"class_list":["post-783","post","type-post","status-publish","format-standard","hentry","category-cs","category-math","tag-linear-algebra","tag-review"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts\/783","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/comments?post=783"}],"version-history":[{"count":9,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts\/783\/revisions"}],"predecessor-version":[{"id":798,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/posts\/783\/revisions\/798"}],"wp:attachment":[{"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/media?parent=783"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/categories?post=783"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.toothlessos.xyz\/index.php\/wp-json\/wp\/v2\/tags?post=783"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}