The Mathematics Behind Gradient Descent: Geometric Insights (Part 2)

2020-01-12

2. Some Mathematical Concepts Involved


Previous Review: The Mathematical Principles Behind Gradient Descent Geometry? (Part 1)

5. Mean Absolute Error


The mean absolute error measures the average error in a set of predicted samples without evaluating their direction or vector. It can be modeled using the following formula.


6. Mean Squared Error


The mean squared error finds the average squared difference between predicted values and actual values. Apart from the fact that the final value is squared instead of absolute, the principle of mean squared error (MSE) is similar to that of mean absolute error (MAE). Instead of measuring the distance between points in a coordinate system, MSE measures the shape (usually a square) formed by the distances between measurement points.


7. Gradient Descent (Introduction Again)


Let's consider an analogy to further understand the intuitive principle of gradient descent!



Imagine that you are standing at the top of Mount Everest and now have the task of reaching the base of the mountain. This sounds quite simple and intuitive, right?
However, there is a small detail you need to consider - you are completely blind. This introduces a small piece of information that you need to reconsider - you are blind.
This undoubtedly makes the task more challenging, but the goal is not entirely impossible to achieve. Before you start moving towards steeper slopes, you need to take small steps. This method requires countless iterations before reaching the destination at the bottom of the mountain.
This essentially mimics the concept of gradient descent, where the model reaches the lowest point of the mountain through backpropagation.
The mountain is similar to the data plot drawn in space, the length of the steps represents the learning rate, and the steepness of the terrain represents the gradient of the algorithm calculating the parameters of the data set.
If the assumption is correct, the chosen direction will reduce the cost function. The bottom of the mountain represents the optimal value of the machine's weights (the cost function has been minimized).




8. Linear Regression


Regression analysis is often used in all statistical modeling disciplines to study the relationship between multivariate functions for predictive analysis.
The line representing the error between the expected value and the experimental value is called the regression line, and each residual value can be depicted by a vertical line segment connected to its variance and the best fit line.
The following formula represents x as the input training data (parameters for a univariate or single input variable), assuming supervised learning, and y represents the label of the data.



Let's further understand this through the following example.
Elon works as a part-time marketing director at salesx and has collected data on the payment amount and sales amount of promotional activities over the past year, providing some guidance for future sales and promotions.
Elon believes that the data should be linear, so he presents this information in the form of a scatter plot, with the number of new customers and the cost of expenditure as the x and y coordinates, respectively. Elon constructs a regression line to better understand and predict how many customers salesx will gain through new marketing concepts.

9. Polynomial Regression


Linear regression can effectively show the structure and trend of two related variables in a dataset. However, considering the behavior of linear functions, they cannot accurately reflect the regression relationship in nonlinear relationships, where some correlation can still be clearly demonstrated.
Polynomial regression models the relationship between n-th degree functions and can fit certain datasets with error function values lower than linear regression.
Although polynomial regression can better fit the curvature of a function and accurately represent the relationship between two variables, they are highly sensitive to outliers, which can easily cause data deviation.





Original article link: https://medium.com/datadriveninvestor/the-math-and-intuition-behind-gradient-descent-13c45f367a11

RECENT AI NEWS

RECENT AI TOOLS