Trees and random forest

We need the following packages:

library(tidyverse)
library(leaps)
library(glmnet)
library(mlbench)
library(rpart)
library(rpart.plot)
library(visNetwork)
library(ISLR)
library(kernlab)
library(randomForest)
library(plotROC)
library(pROC)
library(ranger)
library(caret)

Trees

Exercise 1 (tree regression)

We consider the following dataset

n <-50
set.seed(1234)
X <- runif(n)
set.seed(5678)
Y <- 1*X*(X<=0.6)+(-1*X+3.2)*(X>0.6)+rnorm(n,sd=0.1)
data1 <- data.frame(X,Y)
ggplot(data1)+aes(x=X,y=Y)+geom_point()+theme_classic()

1. Fit a tree to explain \(Y\) by \(X\) (use rpart).

2. Draw the tree with prp and rpart.plot.

3. Write the tree regression function.

4. Add on the graph of question 1 the partition defined by the tree and the prediction.

Exercise 2 (classification tree)

We consider the following dataset

n <- 50
set.seed(12345)
X1 <- runif(n)
set.seed(5678)
X2 <- runif(n)
Y <- rep(0,n)
set.seed(54321)
Y[X1<=0.45] <- rbinom(sum(X1<=0.45),1,0.85)
set.seed(52432)
Y[X1>0.45] <- rbinom(sum(X1>0.45),1,0.15)
data2 <- data.frame(X1,X2,Y)
library(ggplot2)
ggplot(data2)+aes(x=X1,y=X2,color=Y)+geom_point(size=2)+scale_x_continuous(name="")+scale_y_continuous(name="")+theme_classic()

1. Fit a tree to explain \(Y\) by \(X_1\) and \(X_2\). Draw the tree. What happens?

2. Write the classification rule and the score function induced by the tree.

The classification rule is \[\widehat g(x)=\mathbf{1}_{X_1<0.44}.\]

The score function is \[\widehat S(x)=\widehat P(Y=1|X=x)=0.83\mathbf{1}_{X_1<0.44}+0.07\mathbf{1}_{X_1\geq 0.44}.\]

4. Add on the first graph the partition defined by the tree.

Exercise 3 (categorical input)

We consider the following dataset

n <- 100
X <- factor(rep(c("A","B","C","D"),n))
set.seed(1234)
Y[X=="A"] <- rbinom(sum(X=="A"),1,0.9)
Y[X=="B"] <- rbinom(sum(X=="B"),1,0.25)
Y[X=="C"] <- rbinom(sum(X=="C"),1,0.8)
Y[X=="D"] <- rbinom(sum(X=="D"),1,0.2)
Y <- as.factor(Y)
data3 <- data.frame(X,Y)

1. Fit a tree to explain \(Y\) by \(X\).

2. Explain how the tree is fitted in this context.

Exercise 4 (pruning)

We consider the dataset Carseats from the ISLR package.

data(Carseats)

The problem is to explain the Sales variable by the other variables.

1. Fit a tree with rpart to explain Sales by the other variables.

2. Explain the output of the printcp command

printcp(tree)

3. Draw the tree with 8 split (use prune).

4. Remark : visTree function from visNetwork package allows to draw interactive graphs

visTree(tree)

A shiny web application is also proposed to visualise the sequence of subtrees

visTreeEditor(Carseats)

5. Split the dataset into a training set of size 250 and a test set of size 150.

n.train <- 250
set.seed(12345)
perm <- sample(nrow(Carseats))
train <- Carseats[perm[1:n.train],]
test <- Carseats[-perm[1:n.train],]

6. We fit a sequence of trees on the train sample with

set.seed(12345)
tree <- rpart(Sales~.,data=train,cp=0.00000001,minsplit=2)

In this sequence of tree, select

• a very simple tree (with 2 or 3 splits)
• a very large tree
• an optimal tree (with the classical pruning strategy)

7. Estimate the quadratic error of these 3 trees with the test sample.

Random Forest

Exercise 5 (random Forest)

We again consider the spam dataset from package kernlab.

data(spam)

1. Explain the following graph

rf1 <- randomForest(type~.,data=spam)
plot(rf1)

2. Fit anoter random forest with mtry=1. Make a comparison between the two forests.

3. Split the data into a training set of size 3000 and a test set of size 1601.

4. Fit 2 random forest on the training set: one with the default value of mtry and one with mtry=1.

5. Estimate the misclassification error of the 2 RF with the test set.

6. Use caret package to select mtry parameter. You can select this parameter in the grid seq(1,30,by=5).

7. Fit a tree on the train sample.

8. Draw ROC curves and compute AUC for the 2 random forests and the tree.

9. Represent the 10 most important variables of the best random forest with a bar chart.

10. Fit a forest on the train dataset with the ranger function of the ranger package. What do you notice?

Exercice 6

Make a comparison between random Forest, ridge and lasso for the spam dataset. To do that, you will compute the classical risks (error probability, ROC…) by 10-folds cross validation. Be carefull wih the selection of the parameters.