Applied Statistics for Machine Learning Engineers

Applied Statistics for Machine Learning Engineers

Introduction to Applied Statistics in Python for Machine Learning Engineers

In this section, the instructor introduces himself and provides an overview of what the course will cover.

Course Overview

  • The field of statistics is essential to working through predictive modeling problems with machine learning.
  • Statistical methods refer to a range of techniques from simple summary statistics to hypothesis testing and estimation statistics.
  • This course teaches step-by-step basis for statistical methods with concrete and executable examples in python.
  • The course covers six core aspects of statistics for machine learning.

Six Core Aspects of Statistics for Machine Learning

In this section, the instructor discusses the six core aspects of statistics that will be covered in the course.

Core Aspects

  • Introduction to the field of statistics, its relationship to machine learning, and its importance when working through a predictive modeling problem.
  • Building a foundation of statistics specific to machine learning including descriptive statistics, data visualization, random numbers, and important findings such as the law of large numbers and central limit theorem.
  • Hypothesis testing including statistical hypothesis tests for comparing populations or samples and interpretation using p-values and critical values.
  • Resampling methods used in economics to evaluate predictive models such as k-fold cross-validation and bootstrap.
  • Estimation statistics which are an alternative to hypothesis testing called estimation statistics including tolerance intervals conference intervals prediction intervals
  • Non-parametric statistical hypothesis testing methods used when data does not meet expectations of parametric tests i.e. not in normal distribution.

By the end of this course, students will have a solid foundation on how to apply statistics to their machine learning models.

Introduction to Machine Learning and Statistics

In this section, the instructor introduces the course and explains what students can expect to learn. He emphasizes that machine learning and statistics are closely related fields, and that a solid understanding of statistics is essential for success in machine learning.

Course Overview

  • The course will cover topics such as importing data, replacing missing values, descriptive statistics, inferential statistics, hypothesis testing, analysis of variance, and more.
  • The target audience for the course includes programmers who want to transition into machine learning as well as individuals new to programming.
  • A solid understanding of applied statistics is necessary for success in machine learning.

Missing Values in Machine Learning

In this section, the instructor discusses missing values in machine learning and how they can be replaced using statistical techniques called imputation.

Missing Values

  • A missing value in Python is represented by NaN (not a number).
  • Imputation is a statistical technique used to replace missing values.
  • Replacing missing values is important because it can improve the accuracy of your model.

Is This Course Right For You?

In this section, the instructor explains who would benefit from taking this course and what skills are necessary for success.

Target Audience

  • Programmers who want to transition into machine learning
  • Individuals new to programming
  • Anyone moving from another field into machine learning
  • Solid understanding of applied statistics is necessary for success in machine learning

Applied Statistics in Machine Learning

In this section, the instructor explains how applied statistics plays a crucial role in machine learning projects.

Applied Statistics

  • Statistics refers to a collection of methods for working with data and using that data to answer questions.
  • Descriptive statistics provides information that describes the data in some manner.
  • Inferential statistics is about using data from a sample and then making inferences about that larger population from which the sample was drawn.
  • Statistics and machine learning are closely related fields, and there are methods that clearly belong to the field of statistics that are invaluable when working on machine learning projects.

Descriptive Statistics

In this section, the instructor explains what descriptive statistics is and how it can be used to analyze data.

Descriptive Statistics

  • Descriptive statistics provides information that describes the data in some manner.
  • Examples of descriptive statistics include measures of central tendency (mean, median, mode), measures of variability (range, variance, standard deviation), and graphical representations of data (histograms, bar charts, pie graphs).

Inferential Statistics

In this section, the instructor explains what inferential statistics is and how it can be used to make predictions about a larger population based on a smaller sample.

Inferential Statistics

  • Inferential statistics is about using data from a sample and then making inferences about that larger population from which the sample was drawn.
  • The main goal of inferential statistics is to draw conclusions from a sample and generalize them to a population.
  • Hypothesis testing and analysis of variances are common methodologies used in inferential statistics.

Understanding Data Science

This section provides an overview of the different stages involved in data science, including exploratory data analysis, data mining, and data understanding.

Framing the Problem

  • Exploratory Data Analysis (EDA) allows you to visualize your data to gain insights.
  • Data Mining is the discovery of relationships and patterns in your data.
  • Data Understanding means having an intimate graph of both the distribution of variables and the relationships between those variables.

Statistical Methods for Data Understanding

  • Summary statistics are used to summarize the distribution and relationships between variables using statistical quantities.
  • Data visualization methods are used to summarize the distribution and relationships between variables using visualization of charge plots and graphs.

Data Cleansing

  • The process of identifying and repairing issues within the data is called data cleansing.
  • Outlier detection methods are used for identifying observations that are far from the expected value in a distribution.
  • Imputation approaches are used for repairing and showing incorrupt or missing values and observations.

Data Selection

  • The process of reducing the scope of the data to those elements that are most important for making predictions is often called data selection.
  • One approach to data selection is data sampling, which systematically creates smaller representative samples from larger datasets.
  • Feature selection methods automatically identify those variables that are most relevant to the outcome variable.

Data Preparation

  • Transformations may be required in order to change the shape or structure of the data to make it more suitable for modeling.
  • Statistical methods such as standardization and normalization can be used for this purpose.

-Encoding methods such as integer encoding or one-hot encoding can also be applied.

Experimental Design

  • Planning a predictive model requires experimental design which involves resampling techniques like cross-validation.
  • Hyperparameters allow machine learning models to be tailored specifically for a problem.
  • Model selection involves selecting one method as a solution based on criteria from stakeholders and interpretation of the estimated skill of methods evaluated for the problem.

Prediction

  • The final model is used to make predictions on new data where the real outcome is unknown.
  • Estimation statistics are used to quantify the uncertainty for a prediction via prediction intervals.

Data Distribution and Measurement Scales

In this section, the speaker discusses data distribution and measurement scales used in statistics.

Types of Data

  • Discrete data consists of whole numbers that cannot be divided. For example, the number of people in a classroom.
  • Continuous data is data that continues on forever and can be broken down into smaller units.

Measurement Scales

  • Nominal scales are used for labeling variables without any quantitative value. They are simply called labels. For example, preferred colors of five people.
  • Ordinal scales indicate the order of values but not the differences between them. For example, happiness levels such as okay, unhappy, happy or very happy.
  • Interval scales indicate both order and exact differences between values. Celsius temperature is an example where the difference between each value is measurable.
  • Ratio scales tell us about the order and exact value between units with absolute zero allowing for a wide range of descriptive statistics to be applied. Examples include height, weight, and duration.

Samples vs Populations

  • A sample is a subset of all observations from a group while population refers to all possible observations from that group.
  • Statistical methods are used on samples versus populations when making claims about a population using only observations in the sample.
  • There is always some uncertainty when working with samples so it's important to understand and report that uncertainty.

Gaussian Distribution

This section introduces the concept of a Gaussian distribution and how it is used to describe data. It also covers generating sample data from a Gaussian distribution.

Introduction to Gaussian Distribution

  • A normal distribution is formally named for Carl Friedrich Gauss.
  • Data that looks Gaussian is referred to as normally distributed or Gaussian.
  • Examples of observations that fit a Gaussian distribution are people's heights, IQ scores, and temperatures.

Generating Sample Data from a Gaussian Distribution

  • Two key parameters define the Gaussian distribution: mean and standard deviation.
  • The rand function in NumPy can be used to generate sample distributions of data drawn from a Gaussian distribution with a mean of 0 and standard deviation of 1.
  • The generated numbers can be scaled by adding the desired mean and multiplying by the desired standard deviation.
  • Plotting the generated data as a histogram highlights an important facet of distributions: most of the time, the data will not look like a perfect Gaussian distribution but will have similar qualities.
  • Treating such data as if it were normally distributed allows us to use all statistical tools designed for normally distributed data.

Measures of Central Tendency

  • Mean and median are two commonly used measures of central tendency in statistics.
  • The arithmetic mean is an estimate of the parameter underlying the population's true mean from which our sample was drawn. It can be calculated using NumPy's mean function.
  • Outlier values can influence the mean value, making it less reliable than other measures in cases where outliers exist or when dealing with non-Gaussian distributions.
  • Median is another measure of central tendency that calculates the middle value in sorted data. It is less influenced by outlier values than mean.

Variance and Randomness in Machine Learning

In this section, the speaker discusses variance as a measure of the spread of distribution and how it is calculated. They also explain standard deviation and its importance in specifying any Gaussian distribution. The speaker then moves on to discuss randomness in machine learning, its sources, and how it is used as a tool to help learning algorithms be more robust.

Variance and Standard Deviation

  • Variance measures the spread of distribution.
  • The variance can be calculated using the var function in numpy.
  • Standard deviation is described using the square root of variance.
  • Standard deviation along with mean are two key parameters required to specify any Gaussian distribution.

Randomness in Machine Learning

  • Pseudorandom number generator is a mathematical trick that generates a sequence of nearly random numbers used as a source of randomness in machine learning.
  • Sources of randomness include noise in data, limited access to observations from a domain, algorithm evaluation, and stochastic algorithms.
  • Examples of randomness used in machine learning algorithms include shuffling training data prior to each training epoch, stochastic gradient descent, random subset of input features chosen for split points in random forest algorithms, and random initial weights in artificial neural networks.

Controlling Randomness with Seed

This section discusses how to control randomness by setting the seed to ensure that code produces the same results every time.

Generating Random Floating Point Variables

  • Random floating point values can be generated using the random function.
  • Values are generated in the range between 0 and 1, specifically the interval 0-1.
  • Values are drawn from a uniform distribution, meaning each value has an equal chance of being drawn.

Generating Random Integers

  • Random integer values can be generated using the randint function.
  • This function takes two arguments: start and end of the range for generated integer values.
  • Random integers are generated within and including the start and end of the range values, specifically in the interval start-end.
  • Values are drawn from a uniform distribution.

Numpy Pseudo-Random Number Generator

  • Libraries like Scikit-Learn and Keras use Numpy under their covers, which makes working with vectors and matrices of numbers very efficient.
  • Numpy has its own implementation of a pseudo-random number generator (PRNG).
  • The PRNG is configured using functions such as seed() or rand().

Seeding Numpy Pseudo-Random Number Generator

  • The numpy pseudorandom number generator is different from Python's standard library pseudorandom number generator. It must be seeded separately.
  • The seed function can be used to seed the numpy pseudorandom number generator taking an integer as a seed value.
  • Seeding ensures that receding will result in generating exactly same sequence of random numbers.

When to Seed Pseudorandom Number Generators?

This section discusses when to seed a random number generator.

Data Preparation

  • During data preparation, randomness may be used such as shuffling the data or selecting values.
  • Data preparation must be consistent so that the data is always prepared in the same way during fitting evaluation and when making predictions with the final model.

Data Splits

  • The splits used in the data such as train test split or k-fold cross-validation must be made consistently to ensure that each algorithm is trained and evaluated in the same way on the same sub-samples of data.

Seeding Pseudorandom Number Generators

  • You may wish to seed pseudorandom number generators once before each task or once before performing a batch of tasks.
  • Sometimes you may want an algorithm to behave consistently perhaps because it's trained on exactly the same data each time.
  • This may happen if an algorithm is used in a production environment or if you're demonstrating an algorithm in a tutorial environment.

Evaluating Machine Learning Models

In this section, the speaker discusses two core aspects of data uncertainty when evaluating machine learning models. They also explain how evaluating an algorithm on multiple splits of data and multiple evaluations in the same split can provide insight into the model's performance.

Sources of Data Uncertainty

  • Two core aspects of data uncertainty to consider are changes to the training and testing data.
  • Evaluating an algorithm on multiple splits of the data will give insight into how its performance varies with changes to the training and testing data.
  • Evaluating an algorithm multiple times in the same splits of data will provide insight into how its performance varies alone.
  • Reporting on both sources of uncertainty combined will give a fair description of model performance.

The Law of Large Numbers

  • The law of large numbers is a theorem from probability and statistics that suggests that repeating an experiment multiple times will better approximate the true or expected underlying results.
  • As sample size increases, the mean value of the sample will better approximate the mean or expected value in the population.
  • Larger samples have less error, which is what we expect from this theorem.

The Central Limit Theorem

  • The central limit theorem is often confused with law of large numbers but has important implications for making inferences about machine learning models' skill.
  • As sample size increases, the distribution of means across samples more approximates a Gaussian distribution.
  • Each time you add a die, as in rolling dice, to an experiment, it gets closer to normal because of this theorem.

Demonstration

  • A demonstration using code shows how random numbers can be generated using numpy functions.

Central Limit Theorem and Statistical Hypothesis Testing

This section covers the central limit theorem, which states that the distribution of sample means will be Gaussian. It also discusses statistical hypothesis testing, which is used to interpret data by assuming a specific structured outcome and using statistical methods to confirm or reject the assumption.

Central Limit Theorem

  • By repeating a process multiple times, such as taking a thousand samples, the central limit theorem states that the distribution of these samples will be Gaussian.
  • As the size of the sample increases, in this case to 50, then the better the sample means will approximate a Gaussian distribution.
  • Data must be interpreted in order to discover meaning. In statistics, when you wish to start asking questions about data and interpret results, you are using statistical methods that provide confidence or likelihood about answers.

Statistical Hypothesis Testing

  • Statistical hypothesis tests are used whenever you want to make a claim about the distribution of data or whether one set of results is different from another set of results in applied machine learning.
  • A hypothesis test calculates some quantity under a given assumption. The result of the test allows you to interpret whether the assumption holds or whether it has been violated.
  • The assumption of a statistical test is called a null hypothesis (H0). It is often called the default assumption or an assumption that nothing has changed. A violation of this assumption is always called an alternative hypothesis (H1).
  • There are two common forms that a result from a statistical hypothesis can take and they must be interpreted in different ways. They are the p-value and critical values.
  • The p-value is a quantity that we can use to interpret or quantify the result of a test and either reject or fail to reject the null hypothesis. This is done by comparing the p-value to a threshold value chosen beforehand called a significance level.
  • To reject the null hypothesis is one of two possible outcomes of a hypothesis test. The other is failing to reject the null hypothesis.

Null Hypothesis and Gaussian Distribution

This section covers the concept of null hypothesis and how it relates to a proposal. It also introduces the Gaussian distribution, which is a well-known probability density function used to describe the grouping or density of observations.

Null Hypothesis

  • The null hypothesis states that there is no difference, change, or effect.
  • Before the proposal, they were not engaged to be married.
  • If there is no difference in the relationship as a result of his proposal, then the null hypothesis holds true.
  • If she rejects the null hypothesis, this indicates that she does want to be married because there is a significant change in the outcome.

Gaussian Distribution

  • A sample of data will always form a distribution.
  • The most well-known distribution is the Gaussian distribution (also known as normal distribution).
  • This distribution describes the grouping or density of observations called probability density function.
  • You can calculate the likelihood of an observation having a value equal or lesser than given value using cumulative density function.

Describing Distributions with Functions

  • A distribution is simply a collection of data or scores on a variable usually arranged from smallest to largest and presented graphically.
  • Many data conform to well-known mathematical functions such as Gaussian distributions.
  • Once a distribution function is known, it can be used as shorthand for describing and calculating related quantities such as likelihoods of observations and plotting relationships between observations in domain.

Density Functions

This section covers density functions - probability density functions (PDF) and cumulative density functions (CDF). PDF calculates probability while CDF calculates cumulative likelihood for an observation equal to or less than given value.

Probability Density Function (PDF)

  • PDF calculates the probability of observing a given value.
  • It can also be used to summarize the likelihood of observations across the distribution sample space.
  • Plots of PDF show a familiar shape of the distribution such as a bell curve or Gaussian distribution.

Cumulative Density Function (CDF)

  • CDF calculates the cumulative likelihood for an observation and all prior observations in that space.
  • It allows you to quickly understand and comment on how much the distribution lies before and after a given value.
  • A CDF is often plotted as a curve from 0 to 1 for the distribution.

Checking Randomness

This section covers checking randomness by plotting data and using statistical tests. If observations are normally distributed, they can be summarized by just mean and variance calculated directly on samples.

Checking Randomness

  • A sample of data can be checked to see if it's random by plotting it and checking for the familiar normal shape or by using statistical tests.
  • If observations are normally distributed, they are summarized by just mean and variance calculated directly on samples.

Normal Distribution and Student's t-Distribution

This section covers the normal distribution, the 68-95-99.7 rule, and Student's t-distribution.

Normal Distribution and 68-95-99.7 Rule

  • The normal distribution has a heuristic that defines the percentage of data covered by a range by the number of standard deviations from the mean.
  • The 68-95-99.7 rule is an approximate percentage of data covered by ranges defined by one, two, or three standard deviations from the mean.
  • For example, in a distribution with a mean of 50 and a standard deviation of 5, you would expect that 95 percent of data to be covered by values that are two standard deviations from the mean or between 40 and 60.
  • The middle 95 percent can be defined using the percentage point function with values of 2.5 at the low end and 97.5 at the high end.

Student's t-Distribution

  • Student's t-distribution is used instead of normal distribution when dealing with small samples.
  • The larger the sample size, the more it looks like a normal distribution; for samples larger than 20 degrees of freedom, it is almost exactly like a normal distribution.
  • You may estimate values from student's t-distribution required as parameters in other statistical methods such as statistical significance tests but not use it directly.
  • Degrees of freedom describe how many pieces of information are used to describe population quantity; for example, if all n observations in a sample are used to calculate an estimate for population mean then degrees of freedom is n.
  • The number of degrees of freedom is a key factor in using the t-distribution.

Chi-Squared Distribution

This section covers the chi-squared distribution and its use in statistical methods.

Chi-Squared Distribution

  • The chi-squared distribution is used for goodness-of-fit tests and to quantify uncertainty in statistical methods drawn from a Gaussian distribution.
  • It has one parameter, degrees of freedom, where chi is an observation that has a chi-square distribution similar to student's t-distribution.
  • Data does not fit a chi-square distribution; instead, observations are drawn from this distribution when calculating statistical methods for a sample of Gaussian data.

Scipy Stats Module

This section covers the stats module in Scipy.

Scipy Stats Module

  • Scipy provides the stats.chi2 module for calculating statistics for a chi-squared distribution.
  • The stats.chi2.pdf function can be used to calculate the chi-square distribution for a sample between 0 and 50 with 20 degrees of freedom.

Understanding Chi-Square Distribution

In this section, we will learn about the chi-square distribution and how it can be used to calculate the cumulative density function over the sample space.

Chi-Square Distribution

  • The sum of squared random observations from the standard Gaussian distribution would sum to just under the number of degrees of freedom.
  • The chi-squared CDF function can be used to calculate the cumulative density function over the sample space.
  • A plot of the cumulative density function for the chi-square distribution helps you see the likelihood of a specific chi-square value.

Critical Values in Hypothesis Testing

In this section, we will learn about critical values and their importance in hypothesis testing.

What are Critical Values?

  • A critical value is a value that helps you decide whether or not to accept or reject a null hypothesis.
  • It changes as you change the level of significance.
  • It depends on which type of hypothesis testing you are doing (one-tail or two-tail).

Anatomy of a Critical Value

  • It's used in hypothesis testing and assumes a normal or Gaussian distribution.
  • There is an acceptance region and a rejection region, with the line separating those regions being the critical value.

Steps Involved in Using Critical Values

  • Find the critical value based on your level of significance and degrees of freedom.
  • Run your test and plot your observation value on a chart.
  • If your observation falls beyond the critical value, reject your null hypothesis.

One-Tailed vs Two-Tailed Tests

In this section, we will learn about one-tailed tests versus two-tailed tests.

One-Tailed Tests

  • Allows you to determine if mean is greater or less than another mean but not both.
  • A direction must be chosen prior to testing.

Two-Tailed Tests

  • Allows you to determine if mean is different from another mean in either direction.
  • No direction needs to be chosen prior to testing.

Two-Tailed Test and Percent Point Function

This section covers the concept of two-tailed tests, when to use them, and how to calculate critical values using the percent point function.

Two-Tailed Test

  • A two-tailed test determines if two means are different from each other.
  • It is used when you are willing to accept any of the following:
  • One mean being greater than another
  • One mean being lower than another
  • Both means being different from each other.

Percent Point Function

  • The percent point function (PPF) returns an observation value for a provided probability that is less or equal to the provided probability from a distribution.
  • You can calculate PPF using the inverse survival function (ISF) and psi pi.
  • Three distributions commonly used to calculate critical values are:
  • Gaussian distribution
  • Student's t-distribution
  • Chi-squared distribution.

Correlation and Sir Charles Galton

This section covers correlation as a measure of how closely two variables are related, its predictive relationship for future behavior, and its attribution to Sir Charles Galton.

  • Correlation measures how closely two variables are related. The higher the correlation, the more closely they are related.
  • Testing correlation between factors allows you to establish a predictive relationship for future behavior.
  • Sir Charles Galton first attributed correlation in the 1880s while studying patterns related to physical traits in humans and what behaviors could be predicted by them.
  • He later developed regression and applied statistical scales to his observations that could prove or disprove.

Correlation and Covariance

In this section, the speaker explains what correlation and covariance are, how they can be used to measure the relationship between variables, and their limitations.

Correlation

  • Correlation is a statistical measure that describes the relationship between two variables.
  • Correlation can be positive, negative or neutral (zero). Positive correlation means both variables move in the same direction; negative correlation means when one variable value increases, the other variable's value decreases; neutral correlation means that the variables are unrelated.
  • Multicollinearity occurs when two or more variables are tightly related. This can cause some algorithms' performance to deteriorate.

Demo of Strong Positive Correlation

  • A demo is shown where a strong positive correlation between two variables is created using random numbers drawn from a Gaussian distribution with a mean of 100 and standard deviation of 20 for one variable and values from the first variable with Gaussian noise added with a mean of 50 and standard deviation of 10 for another variable.
  • The scatter plot shows how the two variables are created because this is a contrived data set you know that there is a strong positive correlation between the two variables.

Covariance

  • Covariance measures how much two random variables vary together. It tells us whether they change in similar ways or not at all.
  • If variance is zero, it means all data points are identical. Variance is nothing but the average of squared differences from mean. Standard deviation is square root of variance.
  • Covariance is a measure of how much two variables change together. It can be used to summarize the relationship between two variables called covariance.
  • Pearson's r correlation coefficient can be used to summarize the strength of the linear relationship between two data samples. It is calculated as the covariance of the two variables divided by the product of the standard deviation of each sample.

Pearson's r Correlation Coefficient

In this section, we learn about the Pearson's r correlation coefficient and how it can be used to calculate the correlation between two data samples with the same length. We also learn how to evaluate the relationship between more than two variables using a correlation matrix.

Pearson's r Correlation Coefficient

  • The Pearson's r sci pi function can be used to calculate the Pearson's r correlation coefficient between two data samples with the same length.
  • The correlation between two variables in a test problem can be calculated by running an example that calculates and prints the Pearson's correlation coefficient and interprets the p-value.
  • The result of calculating a matrix of relationships between each pair of variables in a dataset is called a correlation matrix, which is a symmetric matrix with a value of 1 along the diagonal as each column is always perfectly correlated with itself.

Parametric vs Non-parametric Statistical Tests

In this section, we learn about parametric and non-parametric statistical tests and their differences based on data distribution.

Parametric vs Non-parametric Statistical Tests

  • Statistical tests are divided into two separate sections: parametric and non-parametric statistical tests.
  • Parametric statistical tests assume that the data sample will be drawn from a specific distribution, such as normal or Gaussian distribution, while non-parametric statistical tests are drawn from skewed data distributions.
  • A typical question we may have about two or more samples of data is whether they have the same distribution. Parametric significance tests assume that the data was drawn from a Gaussian distribution with identical mean and standard deviation parameters for both populations being compared.

Student's t-test

In this section, we learn about the Student's t-test and how it can be used to compare the means between two data samples.

Student's t-test

  • The Student's t-test is a parametric statistical test that compares the means of two independent populations. It assumes that both populations are normally distributed with equal variances.
  • The test is available via Python using the test_ind SciPy function, which takes two samples as arguments and returns the calculated statistic and p-value.
  • If you wish to compare the means between two data samples that are related in some way, such as repeated or dependent measures, you cannot use the Student's t-test. Instead, you can use a modified version of the test called paired Student's t-test.

Paired and Unpaired t-tests, ANOVA, and Effect Size

In this section, the speaker discusses statistical hypothesis tests such as paired and unpaired t-tests, ANOVA, and effect size.

Paired and Unpaired t-tests

  • The paired student's t-test can be used when two data samples are dependent.
  • The unpaired student's t-test can be used when two data samples are independent.
  • You can perform the student's t-test pairwise on each combination of multiple independent data samples to determine which samples have different means.

ANOVA

  • ANOVA is a statistical test that assumes the mean across two or more groups are equal.
  • The test requires that the data samples are Gaussian distributions, independent, and have the same standard deviation.
  • The f1-way scipy function in Python can be used to perform an ANOVA test.

Effect Size

  • Effect size refers to a suite of statistical tools for quantifying the size of an effect in experimental results.
  • There are two main groups of methods for calculating effect size: association and difference.
  • The result of an effect size calculation must be interpreted based on the specific statistical methods used.
  • Three types of calculated results include standardized result, original unit result, and unit-free result.

Effect Size and Power Analysis

This section covers the concept of effect size, statistical power, and power analysis.

Effect Size

  • Effect size is a unit-free measurement that quantifies the magnitude of a result present in the population.
  • It can be calculated using specific statistical measures such as Pearson's correlation coefficient.

Statistical Power

  • Statistical power is one of four related parts that make up the puzzle of power analysis.
  • The other three parts are effect size, sample size, and significance level.
  • All four variables are related, and increasing one can affect the others.
  • Power analysis involves estimating one parameter given values for the other three parameters.

Power Analysis

  • A power analysis can estimate the minimum sample size required for an experiment.
  • Multiple power analyses can provide a curve of one parameter against another to help with experimental design.

Data Sampling and Resampling

This section covers data sampling and resampling techniques used in predictive modeling problems.

Data Sampling

  • Data sampling refers to statistical methods for selecting observations from a domain to estimate population parameters.
  • A sample is a subset of the population selected through sampling techniques such as probability or non-probability sampling.

Data Resampling

  • Data resampling refers to methods for using collected data sets to improve estimates of population parameters and quantify uncertainty.

Sampling Techniques

Probability Sampling

  • Probability sampling uses randomization to ensure every element has an equal chance of being selected.
  • Simple random sampling, systemic sampling, and stratified sampling are examples of probability sampling techniques.

Non-Probability Sampling

  • Non-probability sampling does not rely on randomization and may be biased.

Sampling and Resampling Methods

In this section, the speaker discusses the process of estimation and generalization using a data sample. They explain how errors can be introduced into the sample, including selection bias and sampling errors. The speaker also covers resampling methods for estimating population parameters.

Types of Errors in Sampling

  • Selection bias occurs when the method of drawing observations skews the sample in some way.
  • Sampling error is caused due to the random nature of drawing observations skewing the samples in some way.
  • Systemic errors may be present such as measurement corruption and over or under-representation of observation classes.

Simple Methods for Error Detection

  • Reviewing raw observations, summary statistics, and visualizations can help expose simple errors such as measurement corruption and over or under-representation of observation classes.

Resampling Methods for Estimating Population Parameters

  • Resampling methods involve estimating population parameters multiple times from a data sample to address variability or uncertainty in estimates.
  • Statistical resampling methods are procedures that describe how to economically use available data to estimate a population parameter with more accurate estimates and quantification of uncertainty.
  • Bootstrap samples are drawn from a dataset with replacement allowing an observation to be included in each small sample more than once. This approach is called sampling with replacement.

K-Fold Cross Validation

  • K-fold cross-validation involves partitioning a dataset into k groups where each group is given an opportunity to be used as held-out test sets leaving the remaining group in the training set. This method lends itself well to evaluating predictive models repeatedly trained on subsets of data evaluated on held-out subsets (testing/training sets).

The Bootstrap Method

In this section, the speaker discusses the bootstrap method for estimating quantities about a population by averaging estimates from multiple small data samples. They explain how samples are constructed by drawing observations from large data samples one at a time and returning them to the data sample after they've been chosen.

Steps in the Bootstrap Process

  • Randomly choose a sample from the population.
  • Write down the sample.
  • Put it back in the sample.

Bootstrap Method and Cross-Validation

In this section, we learn about the bootstrap method and cross-validation. We also learn how to implement these techniques using Python libraries.

Bootstrap Method

  • The bootstrap method involves repeatedly resampling a dataset with replacement to estimate a population parameter.
  • To perform the bootstrap method, you take a row from the population and put it back into the population. This process is repeated any number of times.
  • One observation is pulled out and then put back in, and this process is repeated one more time. Note that you've pulled out the same sample twice; this can happen because on each iteration, you're putting the sample back into the population.
  • The final result is one bootstrap sample with three observations.

Implementing Bootstrap Method Using Python Libraries

  • You don't have to implement the bootstrap method manually; instead, you can use an implementation provided by Python libraries such as Scikit-Learn. The resample function can be used to create a single bootstrap sample of a dataset. It takes as arguments the data array, whether or not to sample with replacement, the size of the sample, and seed of the pseudorandom number generator used prior to sampling for reproducibility purposes.
  • Unfortunately, Scikit-Learn's API doesn't include any mechanism to easily gather out-of-bag observations that could be used as a test set to evaluate model fit in univariate cases. However, you can gather out-of-bag observations using a simple Python list comprehension in some cases.

Cross-Validation

  • Cross-validation is a resampling procedure used to evaluate machine learning models on limited data by partitioning the dataset into k bins. This kind of cross-validation is known as k-fold cross-validation, where k is simply the number of segments called bins.
  • The choice of k is usually 5 or 10, but there's no formal rule. As k gets smaller, the difference in size between the training set and resampling subsets gets smaller, and as the difference decreases, the bias of the technique becomes smaller with each sample.
  • In machine learning, rows are often referred to as samples or observations. Each time you run a model on four bins or folds and test it on one bender fold (left out for testing), it's running an iteration. Each iteration will have different subsets of data set aside for testing. Once all iterations have been completed, the model will have been tested on the entire dataset each time leaving out a subset of data for testing. An average score from each iteration will be taken to use for modeling purposes.

Implementing Cross-Validation Using Python Libraries

  • Scikit-Learn's KFold class can be used to perform k-fold cross-validation by taking as arguments the number splits (k), whether or not to shuffle the sample, and seed of pseudorandom number generator used prior to shuffling for reproducibility purposes.
  • Other variations of cross-validation include train-test split, leave-one-out cross-validation (LOOCV), stratified splitting governed by criteria such as ensuring that each fold has an equal proportion of observations with a given categorical value such as a class outcome value (stratified cross-validation), and repeated k-fold cross-validation procedure where importantly data samples are shuffled prior to each repetition which results in a different split for each repetition.

Introduction to Train Test Split and Estimation Statistics

In this section, the instructor introduces train test split as an alternative to k-fold cross-validation. They also discuss estimation statistics as a group of methods that attempt to quantify findings.

Train Test Split

  • The train test split lives within the model selection in the scikit-learn library.
  • The data is separated into target variable and training/test data using train_test_split().
  • A seed is created for reproducibility, and 20% of the data is held back for testing while 80% is used for training.
  • An XGBoost model is created and fit to the training data. The model's performance on the testing data is outputted.

Estimation Statistics

  • Statistical hypothesis tests can indicate whether differences between two samples are due to random chance but cannot comment on the size of the difference.
  • Estimation statistics refer to methods that attempt to quantify findings, including effect size, interval estimation, and meta-analysis.
  • There has been a push towards using estimation statistics instead of p-values in research literature because they are easier to analyze and interpret in context.

Effect Size

  • Effect size describes how different one method or treatment is from another by quantifying the size of an effect or outcome.

Association and Difference

This section discusses the two main techniques to quantify the magnitude of an effect in research: association and difference. It also explains how to calculate effect size using correlation coefficients and Cohen's statistic.

Quantifying Effect Size

  • Association is the degree to which samples change together, while difference is the degree to which samples are different.
  • Correlation coefficients such as Pearson's coefficient and R-squared can be used to calculate association effect size.
  • Cohen's statistic provides a standardized measure for how the means of two populations differ, making it useful for calculating difference effect size.

Interval Estimation

This section covers statistical methods for quantifying uncertainty in observations, including tolerance intervals, confidence intervals, and prediction intervals.

Types of Intervals

  • Tolerance intervals define the bounds or coverage of a proportion of the distribution with a specific level of confidence.
  • Confidence intervals define the bounds of estimates for a population parameter.
  • Prediction intervals define the bounds of a single observation.

Applications

  • Tolerance intervals can help set expectations on observations in a population or identify outliers.
  • Confidence intervals can be used to interpret the range for a mean data sample that could become more precise as sample size increases.
  • Prediction intervals can provide a range for a forecast from a model.

Statistical Tolerance Intervals

This section explains what statistical tolerance intervals are and how they differ from tolerance intervals in engineering. It also covers the difference between confidence intervals and tolerance intervals.

Calculating Tolerance Intervals

  • Statistical tolerance intervals define the upper and lower bounds within which a certain percent of the process output falls with a stated confidence.
  • There are two main methods for calculating tolerance intervals: parametric and non-parametric methods.

Parametric vs. Non-Parametric Tolerance Intervals

  • Parametric tolerance intervals use knowledge of the population distribution to specify both coverage and confidence, often referring to a Gaussian distribution.
  • Non-parametric tolerance intervals use rank statistics to estimate coverage and confidence, resulting in less precision and wider intervals due to lack of information about the distribution.

Overall, this transcript covers various statistical techniques for quantifying effect size and uncertainty in observations. It also explains how to calculate different types of intervals and provides examples of their applications.

Gaussian Tolerance Interval

This section explains how to calculate a Gaussian tolerance interval.

Calculation of the Interval

  • To calculate the tolerance interval, you must estimate the mean and standard deviation from the sample and take this uncertainty into account.
  • Degrees of freedom are used in the calculation of critical values and in calculating the interval. In this example, there are 100 observations, so we have 99 degrees of freedom.
  • The proportional coverage of data is specified. In this example, we're interested in the middle 95% of data.
  • The critical value for 95% is 1.96. You can calculate it directly in Python given the percentage (2.5%) of the inverse survival function.
  • Retrieve the critical value from chi-squared distribution for a given number of degrees of freedom and desired probability using chi2.isf function from Scipy.
  • Calculate Gaussian tolerance interval using all pieces calculated above.

Confidence Interval

  • A confidence interval provides bounds on a population parameter such as mean or standard deviation. It quantifies uncertainty on an estimate.
  • The confidence interval is different from a tolerance interval that describes balance data sampled from distribution and prediction intervals that describe bounds on single observation.
  • Confidence intervals may be used to graphically depict uncertainty on estimates through error bars.
  • Larger samples result in more precise estimates with smaller confidence intervals while larger confidence intervals indicate less precise estimates.

Classification Problems

  • Classification problems involve predicting label or class outcome variable given input data.
  • Classification accuracy or error is used to describe the skill of a predictive model.

Confidence Intervals with Gaussian Distribution

This section discusses how to calculate confidence intervals using the assumption of a Gaussian distribution of the proportion, such as classification accuracy or error. It also covers how to use the bootstrap resampling method as a non-parametric method for calculating confidence intervals.

Calculating Confidence Intervals with Gaussian Distribution

  • Use the assumption of a Gaussian distribution of the proportion (classification accuracy or error) to easily calculate the confidence interval.
  • The radius of interval can be calculated using code on screen where interval is the radius of the confidence interval error and accuracy or classification error and accuracy respectively. n is the size of the sample, sqrt is square root function, and z is number of standard deviations from a Gaussian distribution.
  • Commonly used numbers for standard deviations from a Gaussian distribution and their corresponding significance levels are presented on screen.
  • Example: A model with an error rate of 20% on a validation dataset with 50 samples (n = 50). You can calculate the 95% confidence interval where z = 1.96. Running this example calculates results for confidence interval calculation. You can then make claims that:
  • The classification error of model is 20% +/- 11%
  • The true classification error likely lies between 9% and 31%
  • Sample size has an impact on precision of estimate in terms of radius of confidence interval. Running example shows that increasing sample size drops confidence interval to about 7%, increasing precision.

Bootstrap Resampling Method

  • When you don't know distribution for chosen performance measure or analytical way to calculate such confidence and rules for a score, bootstrap resampling method can be used as non-parametric method for calculating confidence intervals.
  • Bootstrap is simulated Monte Carlo method where samples are drawn from fixed finite dataset with replacement and parameters estimated on each sample. This procedure leads to robust estimate of true population parameters via sampling.
  • Procedure can be used to estimate skill of predictive model by fitting model on each sample and evaluating skill of model on those samples not included in the sample. Mean or median skill of model can be presented as an estimate of the model skill when a value is weighted on unseen data.
  • Confidence intervals can be added to this estimate by selecting observations from the sample of skill scores at specific percentiles. Percentile is observation value drawn from sorted sample where percentage of observations in the sample fall below that value.
  • Example: Choose significance level for confidence level such as 95% represented as 5%. For example, 100 - 95 = 5%. Because confidence interval is symmetric around mean, choose observations in the 2.5 percentile and the 97.5 percentile to give full range.
  • Example: Assume you have a dataset with a thousand observations of values between 0.5 and 1 drawn from a uniform distribution.

Bootstrap Statistics

This section explains how to calculate central tendency and confidence intervals using bootstrap statistics.

Calculating Central Tendency and Confidence Intervals

  • To calculate central tendency, use the median or 50th percentile.
  • Calculate the confidence interval as the middle 95% of observed statistical values centered around the median.
  • Retrieve observations at the desired lower percentile based on the chosen confidence interval.
  • Do the same thing for the upper boundary of the confidence interval.
  • The distribution of bootstrap sample statistics is summarized, including the 2.5, 50th (median), and 97.5 percentile.
  • Use these observations to make a claim about the sample distribution.

Prediction Intervals

This section explains what prediction intervals are and how they differ from confidence intervals.

Understanding Prediction Intervals

  • A prediction interval estimates an interval in which future observation will fall with a certain level of confidence given already observed data.
  • It provides probabilistic upper and lower bounds on an estimate of an outcome variable.
  • Prediction intervals are most commonly used when making predictions or forecasts with regression models where uncertainty is being predicted.
  • A prediction interval surrounds a model's prediction and hopefully covers the range of two outcomes.

Differences Between Prediction Intervals and Confidence Intervals

  • A prediction interval quantifies uncertainty associated with error in a single forecast, while a confidence interval quantifies uncertainty on an estimated population variable such as mean or standard deviation.
  • In predictive modeling, a confidence interval can be used to quantify uncertainty in an estimated skill of a model, whereas a prediction interval can be used to quantify uncertainty in a single forecast.
  • Prediction intervals will almost always be wider than confidence intervals because they account for more uncertainty associated with error.

Linear Regression Prediction Intervals

This section explains how to make a case for linear regression prediction intervals with an example.

Example of Simple Two-Variable Data Set

  • The output variable y depends on the input variable x with some Gaussian noise.
  • The mean and standard deviations of the two variables are printed, and a plot is created to show the clear linear relationship between the variables.
  • The spread of points highlights the noise or random error in the relationship.

Developing a Simple Linear Regression Model

  • A simple linear regression model is developed using scikit-learn library.
  • The model's coefficients and intercept are printed, along with its R-squared value.
  • A scatter plot is created to visualize the model's predictions against actual values.

Calculating Prediction Intervals

  • Prediction intervals are calculated using bootstrapping techniques.
  • The lower and upper bounds of prediction intervals are plotted on top of the scatter plot.

Simple Linear Regression Model

In this section, the speaker explains how to use a simple linear regression model to make predictions and plot them on top of a scatter plot.

Fitting the Model

  • The resulting points form a line that represents the learned relationship.
  • Running an example fits and prints the coefficients.
  • The coefficients are then used as inputs from the data set to make predictions.
  • The resulting inputs and predicted y values are plotted as a line on top of the scatter plot for the data set.

Prediction Interval with Simple Linear Regression Model

In this section, the speaker demonstrates how to add a prediction interval to a simple linear regression model.

Making Predictions

  • A sample is taken out from the data set to demonstrate the prediction interval.
  • The input is used to make the prediction.
  • Calculate the prediction interval for the prediction and compare it with expected values.

Calculating Standard Deviation in Prediction Direction

  • Estimate standard deviation in prediction direction using numpy arrays.

Calculating Prediction Interval

  • Calculate 95% significance level which is 1.96 standard deviations.
  • Once calculated, summarize bounds of prediction to user.

Presenting Results

  • Estimates y-hat standard deviations and calculates confidence interval.
  • Once calculated, present prediction interval to user for given input variable.
  • Plot raw data set as scatter plot, predictions for data set as red line, and prediction interval as black dot and line respectively.

Turn any video into a summary like this

YouTube links, meetings, lectures. With transcripts, search, and chat.

Video description

Complete Course on Applied Statistics for Machine Learning Engineers. It's all the statistics you'll need to know for a career in machine learning.