Random Forest is a common tree model that uses the bagging technique. Many trees are built up in parallel and used to build a single tree model. In this article, we will learn how to use random forest in r.
For this tutorial, we will use the Boston data set which includes
housing data with features of the houses and their prices. We would like
to predict the medv
column or the medium value.
library(MASS)
data(Boston)
str(Boston)
## 'data.frame': 506 obs. of 14 variables:
## $ crim : num 0.00632 0.02731 0.02729 0.03237 0.06905 ...
## $ zn : num 18 0 0 0 0 0 12.5 12.5 12.5 12.5 ...
## $ indus : num 2.31 7.07 7.07 2.18 2.18 2.18 7.87 7.87 7.87 7.87 ...
## $ chas : int 0 0 0 0 0 0 0 0 0 0 ...
## $ nox : num 0.538 0.469 0.469 0.458 0.458 0.458 0.524 0.524 0.524 0.524 ...
## $ rm : num 6.58 6.42 7.18 7 7.15 ...
## $ age : num 65.2 78.9 61.1 45.8 54.2 58.7 66.6 96.1 100 85.9 ...
## $ dis : num 4.09 4.97 4.97 6.06 6.06 ...
## $ rad : int 1 2 2 3 3 3 5 5 5 5 ...
## $ tax : num 296 242 242 222 222 222 311 311 311 311 ...
## $ ptratio: num 15.3 17.8 17.8 18.7 18.7 18.7 15.2 15.2 15.2 15.2 ...
## $ black : num 397 397 393 395 397 ...
## $ lstat : num 4.98 9.14 4.03 2.94 5.33 ...
## $ medv : num 24 21.6 34.7 33.4 36.2 28.7 22.9 27.1 16.5 18.9 ...
To create a basic Random Forest model in R, we can use the
randomForest
function from the randomForest
function. We pass the
formula of the model medv ~.
which means to model medium value by all
other predictors. We also pass our data Boston
.
library(randomForest)
## Warning: package 'randomForest' was built under R version 4.0.5
## randomForest 4.6-14
## Type rfNews() to see new features/changes/bug fixes.
model = randomForest(medv ~ ., data = Boston)
model
##
## Call:
## randomForest(formula = medv ~ ., data = Boston)
## Type of random forest: regression
## Number of trees: 500
## No. of variables tried at each split: 4
##
## Mean of squared residuals: 9.968035
## % Var explained: 88.19
We will now see how to model a ridge regression using the Caret
package. We will use this library as it provides us with many features
for real life modeling. To do this, we use the train
method. We pass
the same parameters as above, but in addition we pass the
method = 'rf'
model to tell Caret to use a lasso model.
library(caret)
## Loading required package: lattice
## Loading required package: ggplot2
##
## Attaching package: 'ggplot2'
## The following object is masked from 'package:randomForest':
##
## margin
set.seed(1)
model <- train(
medv ~ .,
data = Boston,
method = 'rf'
)
model
## Random Forest
##
## 506 samples
## 13 predictor
##
## No pre-processing
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 506, 506, 506, 506, 506, 506, ...
## Resampling results across tuning parameters:
##
## mtry RMSE Rsquared MAE
## 2 3.717373 0.8494336 2.455635
## 7 3.357573 0.8664933 2.254325
## 13 3.502874 0.8521564 2.347024
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was mtry = 7.
Here we can see that caret automatically trained over multiple hyper parameters. We can easily plot those to visualize.
plot(model)
One feature that we use from Caret is preprocessing. Often in real life
data science we want to run some pre processing before modeling. We will
center and scale our data by passing the following to the train method:
preProcess = c("center", "scale")
.
set.seed(1)
model2 <- train(
medv ~ .,
data = Boston,
method = 'rf',
preProcess = c("center", "scale")
)
model2
## Random Forest
##
## 506 samples
## 13 predictor
##
## Pre-processing: centered (13), scaled (13)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 506, 506, 506, 506, 506, 506, ...
## Resampling results across tuning parameters:
##
## mtry RMSE Rsquared MAE
## 2 3.710690 0.8498796 2.452022
## 7 3.354881 0.8667967 2.252195
## 13 3.506762 0.8519703 2.348413
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was mtry = 7.
Often when we are modeling, we want to split our data into a train and
test set. This way, we can check for overfitting. We can use the
createDataPartition
method to do this. In this example, we use the
target medv
to split into an 80/20 split, p = .80
.
This function will return indexes that contains 80% of the data that we should use for training. We then use the indexes to get our training data from the data set.
set.seed(1)
inTraining <- createDataPartition(Boston$medv, p = .80, list = FALSE)
training <- Boston[inTraining,]
testing <- Boston[-inTraining,]
We can then fit our model again using only the training data.
set.seed(1)
model3 <- train(
medv ~ .,
data = training,
method = 'rf',
preProcess = c("center", "scale")
)
model3
## Random Forest
##
## 407 samples
## 13 predictor
##
## Pre-processing: centered (13), scaled (13)
## Resampling: Bootstrapped (25 reps)
## Summary of sample sizes: 407, 407, 407, 407, 407, 407, ...
## Resampling results across tuning parameters:
##
## mtry RMSE Rsquared MAE
## 2 3.953246 0.8241597 2.592503
## 7 3.626163 0.8409663 2.414241
## 13 3.814123 0.8218665 2.533308
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was mtry = 7.
Now, we want to check our data on the test set. We can use the subset
method to get the features and test target. We then use the predict
method passing in our model from above and the test features.
Finally, we calculate the RMSE and r2 to compare to the model above.
test.features = subset(testing, select=-c(medv))
test.target = subset(testing, select=medv)[,1]
predictions = predict(model3, newdata = test.features)
# RMSE
sqrt(mean((test.target - predictions)^2))
## [1] 2.492674
# R2
cor(test.target, predictions) ^ 2
## [1] 0.936557
In practice, we don’t normal build our data in on training set. It is
common to use a data partitioning strategy like k-fold cross-validation
that resamples and splits our data many times. We then train the model
on these samples and pick the best model. Caret makes this easy with the
trainControl
method.
We will use 10-fold cross-validation in this tutorial. To do this we
need to pass three parameters method = "cv"
, number = 10
(for
10-fold). We store this result in a variable.
set.seed(1)
ctrl <- trainControl(
method = "cv",
number = 10,
)
Now, we can retrain our model and pass the ctrl
response to the
trControl
parameter. Notice the our call has added
trControl = set.seed
.
# set.seed(1)
model4 <- train(
medv ~ .,
data = training,
method = 'rf',
preProcess = c("center", "scale"),
trControl = ctrl
)
model4
## Random Forest
##
## 407 samples
## 13 predictor
##
## Pre-processing: centered (13), scaled (13)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 367, 366, 367, 366, 365, 367, ...
## Resampling results across tuning parameters:
##
## mtry RMSE Rsquared MAE
## 2 3.716290 0.8521267 2.527913
## 7 3.342535 0.8708394 2.309503
## 13 3.418426 0.8595248 2.351986
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was mtry = 7.
plot(model4)
This results seemed to have improved our accuracy for our training data. Let’s check this on the test data to see the results.
test.features = subset(testing, select=-c(medv))
test.target = subset(testing, select=medv)[,1]
predictions = predict(model4, newdata = test.features)
# RMSE
sqrt(mean((test.target - predictions)^2))
## [1] 2.376367
# R2
cor(test.target, predictions) ^ 2
## [1] 0.9428691
To tune a random forest model, we can give the model different values of
mtry
. Caret will retrain the model using different mtry and select the
best version.
set.seed(1)
tuneGrid <- expand.grid(
mtry = c(2:4)
)
model5 <- train(
medv ~ .,
data = training,
method = 'rf',
preProcess = c("center", "scale"),
trControl = ctrl,
tuneGrid = tuneGrid
)
model5
## Random Forest
##
## 407 samples
## 13 predictor
##
## Pre-processing: centered (13), scaled (13)
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 367, 366, 367, 366, 365, 367, ...
## Resampling results across tuning parameters:
##
## mtry RMSE Rsquared MAE
## 2 3.716290 0.8521267 2.527913
## 3 3.457209 0.8695897 2.375834
## 4 3.382420 0.8719238 2.325126
##
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was mtry = 4.
Finally, we can again plot the model to see how it performs over different tuning parameters.
plot(model5)