Regularization_BY_MOHAMED_ESSAM.pptx

1. Regularization
BY:MOHAMED ESSAM

2. Agenda
 Overfitting VS Underfitting
 Regularization
 Lasso, Ridge Regularization
 Early Stopping
 Dropout
 Data Augmentation
 Implementation

3. Before We Go!
 The main goal of each machine learning model is to generalize well.
 Here generalization defines the ability of an ML model to provide a suitable
output by adapting the given set of unknown input.
 It means after providing training on the dataset, it can produce reliable and
accurate output.
 Hence, the underfitting and overfitting are the two terms that need to be
checked for the performance of the model and whether the model is
generalizing well or not.

4. Before We Go!
let's understand some basic term that will help to understand this topic well:
 Noise: Noise is unnecessary and irrelevant data that reduces the
performance of the model.
 Bias: Bias is a prediction error that is introduced in the model due to
oversimplifying the machine learning algorithms. Or it is the difference
between the predicted values and the actual values.
 Variance: If the machine learning model performs well with the training
dataset, but does not perform well with the test dataset, then variance
occurs.

5. Example to Understand Overfitting
We can understand overfitting with a general example. Suppose there are three
students, X, Y, and Z, and all three are preparing for an exam. X has studied only
three sections of the book and left all other sections. Y has a good memory, hence
memorized the whole book. And the third student, Z, has studied and practiced all
the questions. So, in the exam, X will only be able to solve the questions if the exam
has questions related to section 3. Student Y will only be able to solve questions if
they appear the same as given in the book. Student Z will be able to solve all the
exam questions in a proper way.

6. Example to Understand Overfitting
 The same happens with machine learning; if the algorithm learns from a small part of the
data, it is unable to capture the required data points and hence underfitted.
 Suppose the model learns the training dataset, like the Y student. They perform very well
on the seen dataset but perform badly on unseen data or unknown instances. In such
cases, the model is said to be Overfitting.
 And if the model performs well with the training dataset and with the test/unseen
dataset, like student Z, it is said to be a good fit.

7. Overfitting
 Overfitting occurs when the model fits more data than required, and it tries
to capture each datapoint fed to it. Hence it starts capturing noise and
inaccurate data from the dataset, which degrades the performance of the
model.
 An overfitted model doesn't perform accurately with the test/unseen
dataset and can’t generalize well.
 An overfitted model is said to have low bias and high variance.

8. How to Detect Overfitting?
 Overfitting in the model can only be detected once you test the data. To detect the
issue, we can perform Train/test split.
 In the train-test split of the dataset, we can divide our dataset into random test and
training datasets. We train the model with a training dataset which is about 80% of the
total dataset. After training the model, we test it with the test dataset, which is 20 % of
the total dataset.

9. How to Detect Overfitting?
 Now, if the model performs well with
the training dataset but not with the
test dataset, then it is likely to have
an overfitting issue.
 For example, if the model shows 85%
accuracy with training data and 50%
accuracy with the test dataset, it
means the model is not performing
well.

10. Ways to prevent the Overfitting
Although overfitting is an error in Machine learning which reduces the performance of
the model, however, we can prevent it in several ways.
Below are several ways that can be used to prevent overfitting:
 Regularization
 Cross-Validation
 Early Stopping
 Data Augmentation

11. Underfitting
 In the case of underfitting, the model is not able to learn enough from the
training data, and hence it reduces the accuracy and produces unreliable
predictions.
 An underfitted model has high bias and low variance.

12. How to Avoid Underfitting?
 By increasing the training time of the model.
 By increasing the number of features.

13. Overfitting And Underfitting

14. Why Regularization?
 Regularization is one of the most important concepts of machine learning. It is
a technique to prevent the model from overfitting by adding extra information
to it.
 This technique can be used in such a way that it will allow to maintain all
variables or features in the model by reducing the magnitude of the variables.
Hence, it maintains accuracy as well as a generalization of the model.
 It mainly regularizes or reduces the coefficient of features toward zero. In
simple words, "In regularization technique, we reduce the magnitude of the
features by keeping the same number of features."

15. How Does Regularization Work?
 Regularization works by adding a penalty or complexity term to the
complex model. Let's consider the simple linear regression equation:
 In the above equation, Y represents the value to be predicted
 X1, X2, …Xn are the features for Y.
 β0,β1,…..βn are the weights or magnitude attached to the features,
respectively. Here represents the bias of the model, and b represents the
intercept.

16. How Does Regularization Work?
 Linear regression models try to optimize the β0 and b to minimize the
cost function. The equation for the cost function for the linear model is
given below:
 Now, we will add a loss function and optimize parameter to make the
model that can predict the accurate value of Y. The loss function for the
linear regression is called as RSS or Residual sum of squares.

17. Techniques of Regularization
There are mainly two types of regularization techniques, which are given
below:
 Ridge Regression
 Lasso Regression

18. Ridge Regression
 Ridge regression is one of the types of linear regression in which a small
amount of bias is introduced so that we can get better long-term predictions.
 Ridge regression is a regularization technique, which is used to reduce the
complexity of the model. It is also called as L2 regularization.
 In this technique, the cost function is altered(changed) by adding the penalty
term to it. The amount of bias added to the model is called Ridge Regression
penalty. We can calculate it by multiplying with the lambda to the squared
weight of each individual feature

19. Ridge Regression
 The equation for the cost function in ridge regression will be:
 In the above equation, the penalty term regularizes the coefficients of the model, and
hence ridge regression reduces the amplitudes of the coefficients that decreases the
complexity of the model.
 As we can see from the above equation, if the values of λ tend to zero, the equation
becomes the cost function of the linear regression model. Hence, for the minimum
value of λ, the model will resemble the linear regression model.
 A general linear or polynomial regression will fail if there is high collinearity between
the independent variables, so to solve such problems, Ridge regression can be used.
 It helps to solve the problems if we have more parameters than samples.

20. Lasso Regression
 Lasso regression is another regularization technique to reduce the complexity
of the model. It stands for Least Absolute and Selection Operator.
 It is like the Ridge Regression except that the penalty term contains only the
absolute weights instead of a square of weights.
 Since it takes absolute values, hence, it can shrink the slope to 0, whereas
Ridge Regression can only shrink it near to 0.

21. Lasso Regression
 It is also called as L1 regularization. The equation for the cost function of Lasso
regression will be:
 Some of the features in this technique are completely neglected for model
evaluation.
 Hence, the Lasso regression can help us to reduce the overfitting in the model
as well as the feature selection.

22. Key Difference between
Ridge Regression and Lasso Regression
 Ridge regression is mostly used to reduce the overfitting in the model, and it
includes all the features present in the model. It reduces the complexity of the
model by shrinking the coefficients.
 Lasso regression helps to reduce the overfitting in the model as well as feature
selection.

23. Dropout

24. Problem of overfitting

25. Complex network
● If you’ve built a neural network before , you know how
complex they are . This makes them more prone to overfitting

26. Dropout
● It is a fairly simple algorithm: at every training step, every neuron
(including the input neurons but excluding the output neurons) has a
probability p of being temporarily "dropped out," meaning it will be entirely
ignored during this training step, but it may be active during the next step

27. Data Augmentation
● Consists of generating new training (rotating, resizing, flipping, and cropping)
instances from existing ones, artificially boosting the size of the training set.
● This will reduce overfitting, making this a regularization technique. The trick is
to generate realistic training instances.

30. CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon, and infographics & images by Freepik
THANK
YOU