Video 48 - Determining Statistical Confidence - ESTIEM LSS Course

Understanding Statistical Confidence in Hypothesis Testing

Theoretical Components of Hypothesis Testing

• The discussion begins with a review of two theoretical components in hypothesis testing: sampling and P-values. The focus now shifts to the third component, which is statistical confidence.

• Statistical confidence relates to how much trust we can place in our sample size and decision rules. It is expressed through confidence intervals, which are measured in physical dimensions like standard deviation or mean.

Types of Probability

• There are three types of probability discussed:

• Personal Probability: Based on individual belief (e.g., predicting rain at 30% without data).

• Rational Probability: Derived from frequency distribution analysis (e.g., calculating the odds of drawing an ace from a deck).

• Causative Probability: Focused on understanding causation effects, which will be elaborated upon later.

Confidence Intervals and Their Calculation

• A formula for calculating a 95% confidence interval is introduced, where Alpha equals 0.005. This involves using statistics such as the mean or proportion plus or minus a calculated value based on Z-scores.

• For a 95% confidence interval, the Z-score is fixed at 1.96. This score is multiplied by the standard deviation divided by the square root of sample size (S/√n).

Impact of Sample Size on Confidence Intervals

• As sample size increases, S/√n decreases, leading to smaller confidence intervals. This highlights that large populations may show statistically significant results even if they are not practically meaningful.

• Three different sampling error estimates are mentioned:

• Sample error of the mean

• Standard error of the mean

• Variance over sample size

Changes in Confidence Levels

• A comparison between different confidence levels shows that increasing from a 5% to a 1% significance level narrows the width of the confidence interval significantly.

Understanding Confidence Intervals and Data Quality Concerns

Confidence Intervals and Sample Size

• At a 99% confidence level, the confidence interval is widest, indicating higher risk of error as the confidence level decreases to 66%, where the interval narrows.

• Holding sample size constant while varying it shows that a sample size of five is often used in control charts, such as X-bar charts.

• As sample size increases from five to 30, the confidence interval decreases significantly; however, at a sample size of 30, the underlying probability distribution shifts.

• Above a sample size of 30, the Z distribution applies adequately; below this threshold, there’s distortion in data representation.

• For samples under 30, it's recommended to use Student's T distribution instead of normal distribution due to significant differences in confidence intervals.

Short-term vs Long-term Data Concerns

Short-term Concerns

• Short-term concerns focus on single samples and include validity—ensuring measures accurately reflect what they intend to measure.

• Timeliness is crucial; delays between data recording and decision-making can compromise analysis quality. Real-time data collection versus delayed analysis presents different challenges.

• Accuracy assesses whether results represent true process performance while precision checks if results are repeatable under similar conditions.

• Calibration ensures measurement equipment accurately reflects standard measures (e.g., verifying if one centimeter truly equals one centimeter).

• Foolproofing involves establishing procedures for data collection and analysis that minimize human errors during observations.

Long-term Concerns

• Long-term concerns encompass multiple short-term issues and focus on stability over time—how consistent measurements remain across various periods.

• Reliability indicates whether short-term performance can be sustained long term; consistent reliability allows for future performance projections based on past data.

• The gauge run chart serves as a tool for measuring long-term reliability or stability within processes discussed earlier in the measure phase.