R Programming Tutorial - Learn the Basics of Statistical Computing

Name: R Programming Tutorial - Learn the Basics of Statistical Computing
Uploaded: 2019-06-06T14:12:38.000Z
Duration: 4 h 20 min 39 s

Creating Informative Graphs in R

Introduction to Plotting in R

The speaker demonstrates how to create a plot in R, including adding titles and axis labels. This process is initiated by executing commands using Command or Control Enter.

The base command for plotting allows users to generate visually appealing and informative graphs, showcasing the flexibility of the plot function beyond mere data representation.

Advanced Plotting Techniques

The speaker illustrates how to use the plot command with mathematical functions like cosine, exponential distributions, and normal density (D norm), providing examples of each graph type.

A refined bell curve is created using D norm with customizations such as color changes and line width adjustments, emphasizing the power of R's default plotting capabilities.

Bar Charts: A Fundamental Graphic Tool

Transitioning to bar charts, the speaker highlights their simplicity and effectiveness for basic data analysis. Bar charts serve as an excellent starting point for visualizing categorical data.

Working with Datasets

The dataset "empty cars" from Motor Trend magazine (1974) is introduced. It includes various car specifications such as MPG, number of cylinders, horsepower, etc., setting up for further analysis.

An overview of the dataset reveals key variables that will be used in creating visual representations. The speaker notes that these are older models with specific characteristics relevant to their time.

Creating Bar Charts in R

Initial attempts at creating a bar chart using RS bar plot command fail due to raw data format issues; it requires summarization first.

To resolve this issue, a summary table is created using the table command on the cylinder variable from the dataset before proceeding with generating a proper bar chart.

After successfully summarizing data into a 'cylinders' object, a functional bar chart displays counts of cars based on cylinder numbers. This reinforces that simple graphics can effectively convey categorical information.

Understanding Histograms in R

Introduction to Histograms

The most basic type of chart is a histogram, which is used for quantitative data that is measured or scaled. It helps visualize the distribution of data.

Analyzing Histogram Characteristics

Key aspects to observe in a histogram include:

Shape of the distribution (symmetrical, skewed, unimodal, bimodal).

Presence of gaps or empty spaces in the distribution.

Identification of outliers that may distort analyses.

Practical Application in R

To create histograms in R, start by loading datasets using the library command and accessing specific datasets like Iris.

The Iris dataset contains sepal and petal measurements for three species of Iris flowers. Basic commands can be used to generate histograms for these variables.

Generating Basic Histograms

Using the hist command allows users to create histograms for specific variables such as sepal length. The output includes frequency counts and automatically adjusts axes.

Observations from generated histograms reveal patterns such as bell curves and gaps indicating interesting data distributions.

Grouping Data for Comparative Analysis

To analyze multiple groups within a dataset (e.g., different species), histograms can be arranged into small multiples using parameters like par(mfrow=c(3,1)).

By specifying conditions (e.g., species type), separate histograms can be created for each group while maintaining consistent x-axis scales across all charts.

Visualizing Species-Specific Distributions

Creating distinct color-coded histograms for each species (Setosa, Versicolor, Virginica) facilitates easy comparison between their distributions.

Understanding Scatter Plots and Data Visualization

Importance of Graphical Parameters

When changing graphical parameters in data visualization, it's crucial to revert them back to their original settings after use.

Histograms are effective for examining quantitative variables and understanding complications arising from different categories with varying scores.

Exploring Univariate Distributions

Previous discussions covered basic graphics for single variables, including bar charts for categorical data and histograms for quantitative data.

The next step involves scatter plots, which visualize the association between two quantitative variables.

Analyzing Scatter Plots

Scatter plots are primarily used to assess linear associations between two quantitative variables.

Key aspects to observe in a scatter plot include:

Linearity of the association.

Consistent spread across scores (checking for heteroscedasticity).

Identification of outliers that may skew interpretations.

Correlation presence between the two variables.

Practical Application with Datasets

To analyze associations, start by loading relevant datasets (e.g., 'mtcars') and reviewing initial cases.

It's beneficial to examine univariate distributions before assessing relationships; histograms can be created for each variable involved (e.g., weight and miles per gallon).

Creating Enhanced Visualizations

Using R's generic plot command allows automatic selection of scatter plots when inputting two quantitative variables like weight and MPG.

Enhancements such as color coding, point size adjustments, and labeling improve clarity in visual patterns observed in the data.

Overlaying Plots for Insightful Analysis

Overlaying plots can increase information density, providing more insights within a limited space.

Understanding Data Visualization Techniques

Importance of Restraint in Data Graphics

The speaker emphasizes the need for cognitive clarity when creating data graphics, suggesting that just because a technique is possible doesn't mean it should be used.

A key principle discussed is to use complementary views in graphics that enhance information without competing against each other.

Introduction to the Lynx Dataset

The session introduces a new dataset concerning Canadian Lynx trappings from 1821 to 1934, highlighting its historical significance.

The dataset consists of a time series represented by a single line of numbers, starting from 1821 and extending through the years.

Creating Initial Visualizations

A default histogram is created to visualize lynx trapping data, revealing a skewed distribution with most observations at lower values.

Options are introduced for enhancing the histogram's complexity, including specifying bin sizes and changing color schemes for better visual appeal.

Analyzing Distribution Patterns

To assess normality in data distribution, a normal distribution curve is overlaid on the histogram using specific commands in R.

The speaker notes significant deviations from normality due to pronounced spikes at low values in the dataset.

Advanced Visualization Techniques

Kernel density estimators are introduced as non-parametric alternatives to normal distributions, allowing for more flexible representation of data shapes.

A rug plot is added beneath the histogram to indicate individual data points visually, providing deeper insight into how each observation contributes to overall trends.

Final Thoughts on Data Representation

The final visualization combines multiple perspectives on lynx trapping data, illustrating how different graphical representations can yield richer insights.

Data Analysis with R: Summary Functions

Introduction to Data Analysis

The session begins with a focus on analyzing quantitative variables using R, emphasizing the simplicity of the process.

The Iris dataset is introduced, which includes measurements for sepal and petal lengths and widths across three species of Iris flowers.

Summary Functions in R

A summary function is demonstrated for categorical variables, specifically focusing on the 'species' variable within the Iris dataset. It reveals counts for each category: setosa (50), versicolor (50), and virginica (50).

For quantitative analysis, sepal length is summarized, providing key statistics such as minimum (4.3), first quartile (5.1), median, mean, third quartile, and maximum (7.9). This helps assess data distribution.

Comprehensive Data Summaries

A full summary of the entire dataset is executed using summary(Iris), displaying vertical arrangements of various statistics including counts for species.

The basic summary function provides essential descriptive statistics that prepare users for further analyses in R.

Advanced Descriptive Statistics

Users may seek more detailed information than what base functions offer; thus, the describe function from the psych package is introduced as a solution.

The describe function outputs comprehensive metrics including sample size (n), mean, standard deviation, median, trimmed mean, range skewness, kurtosis, and standard errors.

Utilizing Additional Packages

Instructions are provided on installing necessary packages using Pacman to facilitate easier management of dependencies when working with datasets like Iris.

Guidance on accessing help documentation through Pacman is shared; it can be viewed online or within R's viewer by setting specific arguments.

Practical Application of Describe Function

The describe function focuses solely on quantitative variables; an example using sepal length illustrates its output format.

Introduction to Basic Statistics in R

Understanding Categorical Variables and Data Description

The discussion begins with the importance of understanding categorical variables, emphasizing that certain lines can be ignored for clarity. An asterisk is used to denote this.

The describe function is highlighted as a valuable tool for obtaining detailed statistics such as standard deviation and skewness, which are essential for a comprehensive data analysis.

It is suggested that using describe complements visualizations like histograms and box plots, providing a more precise image of the dataset.

Selecting Cases in R

The section transitions into selecting cases within datasets, allowing analysts to focus on specific subsets of data based on categories or values.

A practical example using the iris dataset illustrates how to select cases by category (species) or value (petal length), enhancing analytical focus.

Visualizing Data with Histograms

A histogram of petal length from the iris dataset is created, revealing distribution patterns including gaps and normal distributions among species.

Summary statistics are generated for petal length, showcasing minimum values, quartiles, and means to provide context for the histogram.

Case Selection Techniques

The process of selecting cases by category is demonstrated through histograms for different species: versicolor, virginica, and setosa. Each selection results in distinct visual outputs.

Emphasis is placed on ensuring correct spelling and capitalization when specifying categories in selections.

Advanced Selection Methods

Another method discussed involves selecting cases based on quantitative variables; an example shows how to filter petal lengths less than 2 while maintaining consistent titles across diagrams.

Combining multiple selectors allows users to refine their analyses further; an example filters virginica species with shorter petals using logical operators.

Creating Subsets for Repeated Use

For efficiency in repeated analyses, creating new datasets from selected cases is recommended. This involves specifying rows and columns while utilizing assignment operators.

A new object named i.setosa is created from the iris dataset containing only setosa species. This subset retains all columns but limits rows to enhance focused analysis.

Conclusion on Data Accessing Techniques

Understanding Data Formats in R

Overview of Data Types

The discussion begins with the importance of understanding data formats, particularly when dealing with fundamentally different types of data.

Data can vary by type (e.g., numeric, character, logical) and structure. Numeric variables can be integers or floating-point numbers (single/double precision).

Key data types include:

Numeric

Character (text)

Logical (Boolean)

Complex numbers

Raw data types

Common Data Structures

Vector: A one-dimensional array where all elements are of the same type; a single number is still considered a vector of length one.

Matrix: A two-dimensional structure with rows and columns; all columns must be the same length and class.

Data Frame: A two-dimensional collection that allows for multiple data types within its vectors, akin to a spreadsheet. All vectors must have the same length.

List: The most flexible format in R, allowing any class or structure. Lists can contain other lists, creating nested structures.

Coercion in Data Types

Coercion refers to changing data objects from one type to another (e.g., character to logical). This is beneficial in data science as it allows for flexibility in handling different variable types.

Examples include converting matrices to data frames or changing double precision values to integers.

Practical Demonstration in R

The demonstration starts without needing additional packages; basic numeric operations will be shown using an example variable n1.

By assigning n1 a value of 15, it illustrates how R defaults to double precision for numeric variables.

Understanding Data Types and Structures in R

Logical Variables

In R, logical values are represented as TRUE or FALSE, which must be written in all caps.

When a logical value is printed without quotes, it confirms its type as logical rather than character.

Abbreviations can also be used for logical variables; for example, 'T' for true and 'F' for false.

Vectors

A vector is a one-dimensional collection created using the c() function to concatenate values.

Vectors can contain different data types, including numeric, character, and logical values.

The output of a vector displays the elements without commas but requires commas during input.

Matrices

A matrix is a two-dimensional structure created using the matrix() function with specified rows and columns.

You can organize matrices by specifying whether to fill them by rows or columns using the argument byrow = TRUE.

Arrays

An array allows for multi-dimensional data storage; dimensions are defined when creating an array using the colon operator.

The structure of an array includes tables, rows, and columns displayed separately.

Data Frames

A data frame combines vectors of different types into a single structure while maintaining their individual data types.

Using cbind() may lead to unintended coercion where all variables convert to the most general format (character).

Lists

Lists can hold multiple objects of varying types; they are created using the list() function.

Nested lists can be formed by including existing lists within new lists, leading to complex structures.

Coercing Data Types

Understanding Data Types and Structures in R

Coercion of Data Types

By using parentheses around a variable, it automatically saves the response and converts all elements to character type, which is the most general format.

Specific coercion can be performed; for example, converting a variable to an integer using as.integer(), demonstrating how data types can be manipulated in R.

A character variable containing numerical values can be coerced into numeric format with as.numeric(), losing quotes and defaulting to double precision.

Working with Matrices and Data Frames

Creating a matrix of numbers allows for coercion into a data frame, which enables access to functions exclusive to data frames.

The structure of matrices versus data frames is highlighted; while they may look similar, their internal organization differs significantly (e.g., row indices vs. variable names).

Understanding different data types and structures is crucial for effective analysis in R.

Introduction to Factors

Factors are essential for categorizing data; they define possible values and their order within vectors.

Demonstrating factors involves creating artificial data sets in R, showcasing how variables can represent categorical information.

Creating and Manipulating Data Frames

Combining variables into a new data frame (df) illustrates how R handles differing lengths by repeating shorter vectors.

The type of variables within the data frame can be checked; initially integers but can be converted into factors using as.factor().

Reclassifying Variables as Factors

Converting existing integer variables into factors changes their classification without altering displayed values.

New factors created from existing variables show levels distinctly when checked, emphasizing the importance of factor levels in categorical analysis.

Understanding Factor Variables in R

Creating and Labeling Factor Variables

The speaker introduces the creation of a factor variable df four with labels for three operating systems: Mac OS, Windows, and Linux. The order of these labels is crucial as it aligns with their corresponding numerical values (1 for Mac OS, 2 for Windows, 3 for Linux).

After running the code, the output shows that the factor has been converted to text variables despite being entered numerically. The type remains an integer even though it displays words.

The structure of the factor reveals three levels with numeric representations underneath each label. This concept parallels SPSS's value labeling system where numeric values can have descriptive labels.

Reordering Factor Levels

A new variable x five is created and bound to another data frame (df five). The speaker changes the order of levels when listing them, demonstrating how this affects the display of labels in R.

Upon execution, the output reflects a new arrangement of labels which indicates their order through less than signs. This flexibility allows users to customize how factors are represented in analyses.

Importance of Factors in Data Analysis

The speaker emphasizes that factors allow researchers to assign meaningful labels to numerical variables, enhancing clarity during analysis—especially important in experimental research contexts.

Entering Data Manually in R

Ad Hoc Data Entry Methods

Manual data entry is introduced as a method for quickly inputting small datasets directly into R. This approach is termed "ad hoc" data entry since most data would typically be imported from external sources.

Various methods for entering data are discussed including:

Colon operator

seq() function (for sequences)

c() function (for concatenation)

scan()

rep()

Assignment Operator in R

The assignment operator <- is explained as a means to assign values to variables. Unlike other programming languages that use an equal sign (=), R uses this arrow-like symbol which reads as "gets."

A shortcut within R Studio allows users to easily insert this assignment operator using Option + Dash on Mac or Alt + Minus on Windows.

Generating Sequences

The colon operator is demonstrated by creating a sequence from zero to ten (0:10). For descending sequences, reversing the order (e.g., 10:0) achieves similar results.

The seq() function provides more control over sequence generation by allowing specification of starting points and increments (e.g., counting down by threes).

Data Entry and Importing in R

Entering Data into R

To enter a collection of numbers in various orders, the c() function (concatenate) is used to combine them into a data object.

The scan() function allows for live data entry, creating an object where users can input numbers directly via the console.

After entering numbers with scan(), hitting Enter twice finalizes the input, allowing users to view their entered data by calling the object's name.

The rep() function is introduced for repetition; it can replicate values multiple times, such as repeating "TRUE" five times.

Users can also create sets of repeated values (e.g., true/false), demonstrating flexibility in how data can be structured for analysis.

Importing Data into R

The next focus is on importing data, which is often the most common method of getting datasets into R efficiently.

Various file types are discussed for import: CSV (comma-separated values), TXT (text files), XLSX (Excel spreadsheets), and JSON (JavaScript Object Notation).

R has built-in functions for importing these formats; however, using the rio package simplifies this process by consolidating import functions with consistent syntax.

A practical example involves accessing course files containing Google Trends data related to composers Mozart, Beethoven, and Bach in different formats.

Important advice regarding Excel files emphasizes exporting data from Excel as tab-delimited or CSV format before importing it into R to avoid complications.

Best Practices for Data Import

Importing Data with Rio in R

Using Rio for File Imports

The speaker introduces the use of the Rio package to simplify data importation, emphasizing its ease of use.

Demonstrates importing a CSV file using the command import, without needing to specify file type or headers; shows successful data frame creation.

Explains that the same import command can be used for text files by adding .txt to the filename, resulting in identical data structure as before.

Highlights that Rio can also handle Excel files (.xlsx) seamlessly, provided they are formatted similarly to other file types.

Introduces a Data Viewer feature within R that allows users to visually inspect and sort imported data like a spreadsheet.

Built-in R Commands for Reading Files

Discusses built-in R commands such as read.table, which require specifying headers and separators; notes potential errors due to missing values.

Emphasizes the need for specificity when reading tab-delimited files, demonstrating how to correctly set parameters for successful imports.

Mentions that CSV files do not require delimiter specification since they are inherently comma-separated, simplifying their import process.

Introduction to Data Modeling

Overview of Statistical Modeling Techniques

The speaker transitions into discussing statistical modeling, indicating this is an introductory overview rather than an exhaustive course on algorithms.

Introduces hierarchical clustering as a method for identifying similar cases or observations within datasets based on defined criteria of similarity and distance measures.

Hierarchical Clustering Explained

Describes hierarchical clustering's approach of grouping similar cases together while allowing flexibility in defining group numbers (K).

Differentiates between divisive clustering (starting with one group and splitting apart) and agglomerative clustering (starting with individual groups and combining them).

Understanding Car Data Analysis

Initial Data Exploration

The dataset contains 32 cars, and the speaker will analyze similarities among them based on various variables such as MPG, cylinders, and displacement.

A new data frame called "cars" is created to include only selected columns from the original dataset, omitting less useful variables.

The head of the new dataset shows a reduced set of variables: mpg, cylinders, displacement, weight, horsepower, and quarter mile time.

Hierarchical Clustering Process

The speaker introduces hierarchical clustering using the Dplyr package in R for efficient data processing through pipes.

A dissimilarity matrix is computed to measure distances between observations in Euclidean space before applying hierarchical clustering with hclust.

The results are visualized in a dendrogram that illustrates how different car models cluster together based on their features.

Insights from Clustering Results

Notable clusters include Honda Civic and Toyota Corolla being closely related; other groupings make sense based on historical context (e.g., Lincoln Continental with Cadillac Fleetwood).

An outlier identified is the Mangiarotti Bora, which stands apart due to its unique characteristics compared to other cars in the dataset.

Validity of Clusters

It's emphasized that clustering validity depends on the chosen variables; different selections could yield varied clustering outcomes.

Enhancing Cluster Visualization

To improve readability of clusters in the dendrogram, colored boxes are drawn around groups to highlight relationships among car models more clearly.

Practical Applications of Clustering

This method can be beneficial for market analysis by identifying audience segments or consumer behavior patterns based on grouped data insights.

Dimensionality Reduction via PCA

Introduction to Principal Component Analysis (PCA)

PCA is introduced as a technique for reducing noise and unhelpful variables while retaining meaningful information from datasets.

Conceptual Analogy for PCA

Understanding Principal Component Analysis (PCA)

Introduction to PCA and Data Transformation

The speaker introduces a scatterplot illustrating a strong linear association between two variables, indicating the initial step in performing PCA.

A regression line is drawn through the dataset, with perpendicular distances from each data point to this line highlighted by red lines, emphasizing the importance of these distances in PCA.

The process of collapsing data points down to the regression line is described, leading to a one-dimensional representation while retaining essential information from the original two-dimensional dataset.

Benefits of Dimensionality Reduction

By reducing dimensions, analysis becomes simpler and more reliable; fewer dimensions facilitate easier interpretation and understanding of complex datasets.

The speaker prepares to demonstrate PCA using RStudio with the 'mtcars' dataset, focusing on creating a subset of variables for analysis.

Performing PCA in R

The speaker computes PCA using prcomp, specifying centering and scaling options to ensure all variables are on the same scale for accurate analysis.

Two methods for specifying variables in PCA are discussed: including them directly or using a formula notation with tilde (~), both yielding similar results.

Analyzing PCA Results

A summary of the principal components reveals that there are nine components corresponding to the original variables; however, their significance varies greatly.

Standard deviations for each component indicate varying levels of importance; PC1 has a standard deviation significantly higher than 1, suggesting it captures substantial variance from original data.

Visualizing Component Importance

A scree plot is introduced as a tool for visualizing how much variance each principal component explains; PC1 stands out as particularly significant compared to others.

Criteria for determining important components are briefly mentioned; currently, visual inspection suggests that PC1 is crucial while PC2 holds minor relevance.

Exploring Individual Variable Contributions

The rotation matrix shows associations between individual variables and principal components, allowing insights into which variables contribute most significantly to each component's variance.

Biplots and Regression Analysis

Understanding Biplots

The speaker introduces the concept of a biplot, which is a two-dimensional plot that charts the first two principal components of data analysis.

The biplot displays individual variables' contributions through red lines, with cases labeled to show their positions. Notably, the Maasai Bora stands out as unusual in its placement.

Clustering is discussed, highlighting how big cars (heavy displacement, weight, cylinders, horsepower) cluster together while smaller cars (like Honda Civic and Porsche 911) are grouped separately based on speed and efficiency.

The dimensions of "big vs. small" and "slow vs. fast" emerge as significant insights for further analysis.

Introduction to Regression

Regression is presented as a powerful analytical method used to predict one outcome variable from multiple predictor variables.

Various adaptations of regression make it flexible for different tasks; the speaker plans to demonstrate these adaptations using R programming.

Working with Data in R

The dataset used is from US judge ratings, containing scores on various attributes like diligence and demeanor. The goal is to predict judges' retention based on these scores.

A matrix object called 'x' will be created from all predictor variables except for the retention score (the outcome variable).

Performing Regression Analysis

Simultaneous entry regression uses all predictor variables at once to predict a single outcome variable using linear models in R (lm function).

The model's formula indicates that retention (y) is predicted by all other variables (x), allowing for comprehensive analysis without needing explicit variable names each time.

Interpreting Results

Coefficients from the regression output are examined; they indicate relationships between predictors and the outcome variable. Each coefficient reflects how changes in predictors affect retention predictions.

Regression Analysis and Predictive Modeling

Understanding Coefficients and Statistical Significance

The analysis begins with a summary of individual coefficients, highlighting their values, standard errors, t-tests, and probability values. Asterisks indicate significance below the 0.05 threshold.

The intercept is expected to be below the significance level. Notably, integrity significantly influences judgments on whether a judge should be retained.

Model Performance Metrics

The multiple R-squared value indicates that the selected variables predict judges' retention decisions very effectively.

A 95% confidence interval for coefficients is presented, showing lower (2.5%) and upper (97.5%) bounds for predictions.

Residual Analysis

Residual data can be complex; thus, visualizing them through histograms provides clearer insights into prediction accuracy.

A histogram reveals that residuals are mostly centered around zero with some skewness, indicating generally good predictive performance.

Exploring Different Regression Techniques

Introduction of additional libraries: Lars (Least Angle Regression) and caret (Classification and Regression Training).

Stepwise regression is performed quickly despite its criticisms; followed by stage-wise regression which offers better generalizability.

Comparing Predictive Abilities of Models

Various regression methods including Lasso (Least Absolute Shrinkage and Selection Operator) are compared based on their R-squared values.

All models demonstrate high predictive ability; however, variations may occur depending on specific datasets or contexts.

Conclusion and Next Steps in Learning R

Encouragement to explore further resources available at data lab.cc for advanced learning in R.

Suggestion to compare R with Python as both languages offer similar functionalities in data science applications.

Emphasizing Data Visualization Skills

Importance of mastering data visualization concepts alongside technical skills in R to enhance understanding and design quality visuals.

Machine Learning Applications