[ML 2021 (English version)] Lecture 2: Predicting the views of this channel - ML Introduction (2/2)

Linear Models and Model Bias

In this section, the speaker discusses the limitations of linear models and introduces the concept of model bias.

Linear Models are Too Simple

• Linear models have a simple relationship between x1 and y, which is always a straight line.

• As x1 increases, y also increases in a linear model.

• However, this may not be true in reality as there could be more complicated relationships between x1 and y.

Model Bias

• The limitations of linear models are called model bias.

• Due to these limitations, linear models cannot simulate real-life situations accurately.

• To overcome these limitations, we need to write more complicated functions with unknown parameters.

Piecewise Linear Curves

• A piecewise linear curve can be seen as a constant function plus a bunch of blue functions.

• The blue function has a special property where its output is constant when the input value is less than a certain threshold and then changes slope after that threshold.

• By adding multiple blue functions together with different slopes and thresholds, we can create more complex curves that better fit real-life data.

Piecewise Linear Curves and Sigmoid Functions

In this section, the speaker explains how to use piecewise linear curves and sigmoid functions to approximate any continuous curve. They also discuss how adjusting variables can create different shapes of sigmoid functions.

Using Piecewise Linear Curves

• To approximate a non-piecewise linear curve, choose some points on the curve and connect them to transform it into a piecewise linear curve.

• More complicated piecewise linear curves with more turning points require more blue functions.

Sigmoid Functions

• The formula for a sigmoid function is y = c / (1 + e^(-b-wx1)).

• Adjusting variables b, w, and c can create different shapes of sigmoid functions.

• The hard sigmoid is a commonly used blue function that can be approximated using the normal sigmoid function.

• Stacking up different sigmoid functions with different values of b, w, and c can approximate all kinds of curves.

Introduction to Sigmoid Functions

In this section, the speaker introduces sigmoid functions and how they can be used to create flexible functions with unknown parameters.

Sigmoid Functions

• Sigmoid functions can be written in the form of c1 times the sigmoid of w times x1 plus b.

• The variables that differ between sigmoid functions are w, b, and c.

• By stacking different sigmoid functions together, we can create a new function that is more flexible.

Creating More Flexible Functions

In this section, the speaker discusses how to reduce model bias by creating more flexible functions with unknown parameters.

More Flexible Function Formula

• A more flexible function formula can be written as y equals b plus summation ci times the sigmoid of wi times x1 plus bi.

• By choosing different values for ci, bi, and wi, we can create all kinds of different functions.

Using Multiple Features

In this section, the speaker explains how multiple features can be used in creating flexible functions.

Expanding Function Formula

• We use j to represent the number of features.

• The function formula expands to include summation over j wijxj instead of just wx1.

• Different values for ci, bi, and wij can create different functions.

Intuitive Understanding

In this section, the speaker provides an intuitive understanding of how these formulas work.

Drawing Out Formulas

• When drawn out with three features (j = 1 to 3), each i represents a blue function or a sigmoid function.

• Each sigmoid has parentheses containing weights multiplied by previous viewership numbers (x1-x3).

• Different values for ci, bi and wij result in different curves.

Understanding the Sigmoid Function

In this section, we learn about the sigmoid function and its role in calculating r1, r2, and r3.

The Sigmoid Function

• There are three sigmoid functions used in this calculation.

• The second sigmoid function multiplies x1 by w21, x2 by w22, x3 by w23, adds them up and adds b2.

• The third sigmoid function puts 123x1x2x3 multiplied by w31,w32,and w33 respectively and adds b3.

• R1, R2, and R3 are represented as vectors. X1,X2,X3,R1,R2,R3 are related through matrix multiplication.

Simplifying the Calculation with Linear Algebra

In this section, we learn how to simplify the calculation using linear algebra.

Simplifying the Calculation

• We can simplify the calculation using matrix multiplication.

• X is multiplied by a matrix called W (containing nine values), then B is added to get vector R.

• This simplification allows us to use a more commonly used representation in linear algebra.

Passing Through Sigmoid Function

In this section, we learn about passing through the sigmoid function.

Passing Through Sigmoid Function

• R1,R2,and R3 pass through the sigmoid function separately.

• The output of the sigmoid function is multiplied by C and added by B to get Y.

• A1,A2, and A3 are obtained from R1,R2, and R3 respectively.

Flexible Formula

In this section, we learn about a more flexible formula for the calculation.

Flexible Formula

• The calculation can be expressed in terms of multiplications between vectors and matrices defined in linear algebra.

• X is multiplied by W, added by B, passed through sigmoid, then multiplied by C transpose and added B to get Y.

Redefining Symbols

In this section, the speaker redefines the symbols used in machine learning and introduces the concept of unknown parameters.

Redefining Symbols

• The x represents input features.

• There are four symbols for unknown parameters: w, b, c, and another b.

• The two b's are not the same; one is a vector while the other is a scalar.

• These unknown parameters are represented by a long vector called theta.

Brute Force Search vs Gradient Descent

This section discusses optimization methods for finding unknown parameters.

Optimization Methods

• Brute force search can be used when there are only a few parameters to find.

• Gradient descent is needed when there are many parameters to find.

• Theta refers to all unknown parameters collectively and can be found using gradient descent.

Number of Sigmoid Functions

This section discusses how many sigmoid functions should be used in an example.

Number of Sigmoid Functions

• The number of sigmoid functions used is up to the user.

• More sigmoid functions generate more complicated piecewise linear functions that can approximate more complex functions.

• Three sigmoid functions were used in the early example as an example but users can decide on their own number of sigmoid functions.

Hard Sigmoid Function

This section discusses using hard sigmoid instead of regular sigmoid function.

Using Hard Sigmoid Function

• Hard sigmoid function may be more complicated than regular sigmoid function but it can still be used if desired.

• There are many other ways to approximate hard sigmoid function besides using regular sigmoid function.

Introduction to Optimization

In this section, the speaker introduces the concept of optimization and explains how it is used in machine learning.

The Optimization Problem

• The optimization problem involves finding a set of parameters that can make the loss function as small as possible.

• To find the optimal set of parameters, we use an algorithm called gradient descent.

Gradient Descent Algorithm

• We randomly choose an initial value for our parameter vector, which we call theta zero.

• We calculate the differential for each unknown parameter using the loss function and represent them as a vector called gradient (g).

• We update our parameters by subtracting the product of learning rate and gradient from our current parameter values.

• We repeat this process until we reach a point where we cannot update our parameters anymore or until we decide to stop.

Dividing Data into Batches

In this section, the speaker explains how to divide data into batches and why it is necessary.

Dividing Data into Batches

• Divide n pieces of data into individual batches randomly.

• Capital b represents the number of pieces of data in each batch.

• Group them into b pieces each group is called a batch.

• Calculate loss by taking only one batch instead of the entire data. This loss is called l1.

• Gradient is calculated using l1, l2, and l3 from batches instead of capital L for the entire dataset.

• Processing all the batches once is called an epoch. Each time we update parameters, it's called an update.

Number of Updates in an Epoch

• If there are 10,000 examples with a batch size set to 10, then there will be 1000 batches formed. Therefore, in an epoch, you have updated the parameters 1000 times.

• If there are 1000 examples with a batch size set to 100, then there will be ten updates in an epoch.

Variations on Models

In this section, the speaker discusses variations on models and different ways to write functions.

Hard Sigmoid vs Soft Sigmoid

• There are other ways besides switching from hard sigmoid to soft sigmoid.

• The formula for hard sigmoid can be viewed as the sum of two rectified linear units (ReLU).

Rectified Linear Unit (ReLU)

• ReLU has a horizontal line with a turning point somewhere afterward it becomes a slope.

• The formula of this function is written as c multiplied by max 0 b plus w x 1.

• If b plus wx1 is less than 0, then the output is 0. If b plus wx1 is greater than 0, the output is b plus wx1.

• This line can be written as c max 0 b plus w x 1.

• By using different w, different b, and different c, you can move its position and change the slope of this line.

• Stacking two ReLUs becomes hard sigmoid.

Activation Functions in Machine Learning

In this section, the speaker discusses activation functions in machine learning and compares two common ones - sigmoid and ReLU.

Sigmoid vs. ReLU

• Sigmoid is a common activation function in machine learning.

• ReLU is another popular activation function that can be used instead of sigmoid.

• The speaker will discuss which one is better in the next session but chose to use ReLU for the next experiment.

• Using 100 ReLUs can create more complicated curves than using just one or ten, resulting in better performance on testing data.

Modifying Models with Multiple Layers

In this section, the speaker discusses modifying models by adding multiple layers and performing a series of operations on input features.

Adding Layers

• A model can be modified by passing through a series of operations on input features and multiplying them by different parameters.

• The number of times to repeat these operations is a hyperparameter that must be decided upon.

• By repeating these operations twice or three times, there was significant improvement in loss reduction from 0.28k to 0.18k and then to 0.14k respectively.

Results of Experiment

In this section, the speaker presents the results of their experiment with multiple layers.

Predictions vs Real Data

• The speaker presents a graph comparing the predicted data to the real data.

• The machine's prediction is fairly accurate at low points in the data, but it overestimates the number of viewers on some days.

• There is an interesting discrepancy where the machine predicts a low point one day late.

Conclusion

In this section, the speaker concludes their presentation and asks if anyone has any ideas about why there was a discrepancy in predictions.

Final Thoughts

• The speaker concludes their presentation and asks if anyone has any ideas about why there was a discrepancy in predictions.

Introduction to Deep Learning

In this section, the speaker introduces deep learning and explains how it got its name.

What is Deep Learning?

• The speaker discusses the need for a good name for deep learning models.

• Neurons are introduced as the building blocks of neural networks, which simulate the human brain.

• A large network with many hidden layers is called deep learning.

Benefits of Deep Learning

• The speaker discusses various deep learning models and their error rates in image recognition.

• Training deep networks requires tips and tricks that will be discussed later.

Why "Deep" Learning?

• The speaker explains that any continuous function can be approximated with enough sigmoids and ray lews, so why add more layers?

• The term "deep" was chosen because it sounds better than "fat" or other alternatives.

• Networks with hundreds of layers are now common in deep learning.

Challenges in Deep Learning

• There can be issues with overfitting on training data versus generalizing to new data.

Introduction to Deep Learning

In this section, the instructor introduces deep learning and explains how it can be used to predict unknown data.

Choosing a Model

• The instructor asks the audience whether they would choose a 3-layer or 4-layer network to predict the number of viewers on February 26th.

• Most people choose a three-layer network because it performs well on unread data.

• The result on training data from four-layer network is better, but we care about the result of the unread data.

Predicting Viewership

• The instructor uses a three-layer neural network to predict the number of viewers on February 25th and gets an estimate of 5250 viewers.

• Using this estimate, he predicts that there will be only 3960 viewers on February 26th due to fewer people watching videos on Fridays.

• He asks the audience which prediction is more accurate and most people believe that his prediction has greater error than the true value.

Conclusion

• The instructor concludes by providing links to lecture videos for those who want a general introduction to deep learning and backpropagation.

• He also encourages viewers to click on his videos in order to reduce errors in predicting future viewership.