neural network.Rmd

---
title: "Classifying movie reviews: a binary classification example"
output: 
  prettydoc::html_pretty:
    theme: Tactile
    highlight: github
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

# Two-class classification, or binary classification, 

In this example, we will learn to classify movie reviews into "positive" reviews and "negative" reviews, just based on the text content of the reviews.

# execute "Data_management.rmd" first

```{r}
library(tidymodels)
library(tidyverse)
```

## Executer "data_magagement.Rmd"



```{r, results='hide', eval=F}
library(keras)

```


# Définition de trainning set et test set
```{r}

set.seed(1234)


ames_split <- initial_split(gravity, prob = 0.80)
ames_train <- training(ames_split)
ames_test <- testing(ames_split)
# ames_test_small <- ames_test %>% head(5)

dim(ames_train)
dim(ames_test)
colnames(ames_train)
glimpse(ames_train)

```


# Fabriquer matrice_design pour neural_network training set
```{r}
ames_rec <-
  recipe(grav_or_not ~ ., data = ames_train) %>%
#   step_other(UniqueCarrier, threshold = 0.01) %>%
#   # step_mutate(DepDelay = log(1-min(DepDelay)+DepDelay))%>%
#   step_mutate(DepDelay=log(1-min(DepDelay)+DepDelay))%>%
  step_dummy(all_nominal(), -grav_or_not)
  # %>%
#   step_interact( ~ DepDelay:starts_with("UniqueCarrier_") ) %>%
#   step_ns(DepDelay, deg_free = 20)

# ces lignes de code dessous pour visualiser la design matrice
ames_rec_prepped <- prep(ames_rec) # si je met pas de parametre data, il va prendre le meme que la recette


matrice_design <- bake(ames_rec_prepped, new_data = NULL)
# matrice_design
dim(matrice_design)

```

# Fabriquer matrice_design_test pour neural_network test set
```{r}
ames_rec_test <-
  recipe(grav_or_not ~ ., data = ames_test) %>%
  step_dummy(all_nominal(), -grav_or_not)


# ces lignes de code dessous pour visualiser la design matrice
ames_rec_prepped_test <- prep(ames_rec_test) # si je met pas de parametre data, il va prendre le meme que la recette


matrice_design_test <- bake(ames_rec_prepped_test, new_data = NULL)
# matrice_design_test
dim(matrice_design_test)
```





# gestion de format des données en entrée : utiliser directement la design matrice

```{r}

# Traitement de X : trainig_set

matrice_design_DL_X<- matrice_design %>% select(-lat,-long, -age)
matrice_design_DL_quanti<- matrice_design %>% select(lat,long,age)

matrice_design_DL_X <- as.matrix(matrice_design_DL_X)
matrice_design_DL_quanti <- as.matrix(matrice_design_DL_quanti)

# head(matrice_design_DL_X)
# head(matrice_design_DL_quanti)
x_train_test <- to_categorical(matrice_design_DL_X)
dim(x_train_test)
dim(matrice_design_DL_X)

x_train_test <- (x_train_test[,,2])
x_train_test <- cbind(x_train_test, matrice_design_DL_quanti)

# head(x_train_test)
# test <- head(x_train)
# view(test)
# class(x_train) # "matrix" "array" 
class(x_train_test)
# dim(x_train) # 25000 10000
dim(x_train_test) # 2475 260



# Traitement de Y : trainig_set
matrice_design_DL_Y<- matrice_design %>% select(grav_or_not)
matrice_design_DL_Y
y_train_test <- as.numeric(as.vector(t(matrice_design_DL_Y)))
dim(y_train_test) # NULL
#dim(y_train) # NULL
length(y_train_test) # 2475
#length(y_train) # 25000
# head(y_train_test, 20)
#head(y_train, 20)
# head(matrice_design_DL_Y, 20)

# Traitement de X : test_set

matrice_design_test_DL_X<- matrice_design_test %>% select(-lat,-long, -age)
matrice_design_tes_DL_quanti<- matrice_design_test %>% select(lat,long,age)


matrice_design_test_DL_X <- as.matrix(matrice_design_test_DL_X)
matrice_design_tes_DL_quanti <- as.matrix(matrice_design_tes_DL_quanti)

# head(matrice_design_test_DL_X)
x_test_test <- to_categorical(matrice_design_test_DL_X)

x_test_test <- (x_test_test[,,2])
x_test_test <- cbind(x_test_test, matrice_design_tes_DL_quanti)
dim(x_test_test) # 826 338
# head(x_test_test)
# test <- head(x_test)
# view(test)
# class(x_test) # "matrix" "array" 
class(x_test_test) #"matrix" "array" 
# dim(x_test) # 25000 10000



# Traitement de Y : test_set
matrice_design_test_DL_Y<- matrice_design_test %>% select(grav_or_not)
matrice_design_test_DL_Y
y_test_test <- as.numeric(as.vector(t(matrice_design_test_DL_Y)))
dim(y_test_test) # NULL
# dim(y_test) # NULL
length(y_test_test) # 826
# length(y_test) # 25000
# head(y_test, 20)
# head(y_test_test, 20)
# head(matrice_design_test_DL_Y, 20)


```

Now our data is ready to be fed into a neural network.

## Building our network

The input data is vectors, and the labels are scalars (1s and 0s): this is the easiest setup you'll ever encounter. A type of network that performs well on such a problem is a simple stack of fully connected ("dense") layers with `relu` activations: `layer_dense(units = 16, activation = "relu")`.

The argument being passed to each dense layer (16) is the number of hidden units of the layer. A _hidden unit_ is a dimension in the representation space of the layer. 

Having 16 hidden units means that the weight matrix `W` will have shape `(input_dimension, 16)`, i.e. the dot product with `W` will project the input data onto a 16-dimensional representation space (and then we would add the bias vector `b` and apply the `relu` operation). You can intuitively understand the dimensionality of your representation space as "how much freedom you are allowing the network to have when learning internal representations". Having more hidden units (a higher-dimensional representation space) allows your network to learn more complex representations, but it makes your network more computationally expensive and may lead to learning unwanted patterns (patterns that will improve performance on the training data but not on the test data).

There are two key architecture decisions to be made about such stack of dense layers:

* How many layers to use.
* How many "hidden units" to chose for each layer.

Here's what our network looks like:

![3-layer network](https://s3.amazonaws.com/book.keras.io/img/ch3/3_layer_network.png)

And here's the Keras implementation, very similar to the MNIST example you saw previously:



###  383 variables donc c(383)
```{r}

model <- keras_model_sequential() %>% 
  layer_dense(units = 16, activation = "relu", input_shape = c(394)) %>% 
  layer_dense(units = 16, activation = "relu") %>% 
  layer_dense(units = 1, activation = "sigmoid")
```


Lastly, we need to pick a loss function and an optimizer. 

* Since we are facing a binary classification problem and the output of our network is a probability (we end our network with a single-unit layer with a sigmoid activation), is it best to use the `binary_crossentropy` loss. 


You may try : `mean_squared_error`
It isn't the only viable choice: you could use, for instance, `mean_squared_error`.

Crossentropy is a quantity from the field of Information Theory, that measures the "distance" between probability distributions, or in our case, between the ground-truth distribution and our predictions.

Here's the step where we configure our model with the `rmsprop` optimizer and the `binary_crossentropy` loss function. Note that we will also monitor accuracy during training.

```{r eval = F}
model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)
```

You're passing your optimizer, loss function, and metrics as strings, which is possible because `rmsprop`, `binary_crossentropy`, and `accuracy` are packaged as part of Keras. Sometimes you may want to configure the parameters of your optimizer or pass a custom loss function or metric function. The former can be done by passing an optimizer instance as the `optimizer` argument: 


```{r eval = F}
model %>% compile(
  optimizer = optimizer_rmsprop(lr=0.001),
  loss = "binary_crossentropy",
  metrics = c("accuracy")
) 
```

The latter can be done by passing function objects as the `loss` or `metrics` arguments:

```{r eval = F}
model %>% compile(
  optimizer = optimizer_rmsprop(lr = 0.001),
  loss = loss_binary_crossentropy,
  metrics = metric_binary_accuracy
) 
```

## Validating our approach (sans cv, l'approche cv est en bas)

In order to monitor during training the accuracy of the model on data that it has never seen before, we will create a "validation set" by setting apart 10,000 samples from the original training data:

```{r eval = F}
val_indices <- 1:25000
x_val_test <- x_train_test[val_indices,]
partial_x_train_test <- x_train_test[-val_indices,]
y_val_test <- y_train_test[val_indices]
partial_y_train_test <- y_train_test[-val_indices]

dim(x_val_test)
length(y_val_test)


dim(partial_x_train_test)
length(partial_y_train_test)
```





We will now train our model for 20 epochs (20 iterations over all samples in the `x_train` and `y_train` tensors), in mini-batches of 512 samples. At this same time we will monitor loss and accuracy on the 10,000 samples that we set apart. This is done by passing the validation data as the `validation_data` argument:

```{r, echo=TRUE, results='hide', eval=F}
model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)

```


```{r}
history <- model %>% fit(
  partial_x_train_test,
  partial_y_train_test,
  epochs = 5,
  batch_size = 512,
  validation_data = list(x_val_test, y_val_test)
)

dim(partial_x_train_test)
length(partial_y_train_test)
dim(x_val_test)
length(y_val_test)

```

On CPU, this will take less than two seconds per epoch -- training is over in 20 seconds. At the end of every epoch, there is a slight pause as the model computes its loss and accuracy on the 10,000 samples of the validation data.

Note that the call to `fit()` returns a `history` object. Let's take a look at it:

```{r eval = F}
str(history)

mean(history$metrics$accuracy)
```

The `history` object includes various parameters used to fit the model (`history$params`) as well as data for each of the metrics being monitored (`history$metrics`).

The `history` object has a `plot()` method that enables us to visualize the training and validation metrics by epoch:

```{r eval = F}
plot(history)
```

The accuracy is plotted on the top panel and the loss on the bottom panel. Note that your own results may vary 
slightly due to a different random initialization of your network.



The dots are the training loss and accuracy, while the solid lines are the validation loss and accuracy. Note that your own results may vary slightly due to a different random initialization of your network.

As you can see, the training loss decreases with every epoch, and the training accuracy increases with every epoch. 

That's what you would expect when running a gradient-descent optimization -- the quantity you're trying to minimize should be less with every iteration. 


But that isn't the case for the validation loss and accuracy: they seem to peak at the fourth epoch. This is an example of what we warned against earlier: a model that performs better on the training data isn't necessarily a model that will do better on data it has never seen before. In precise terms, what you're seeing is _overfitting_: 

In this case, to prevent overfitting, you could stop training after three epochs. In general, you can use a range of techniques to mitigate overfitting.

Let's train a new network from scratch for four epochs and then evaluate it on the test data.

```{r, echo=TRUE, results='hide', eval = F}
# model <- keras_model_sequential() %>% 
#   layer_dense(units = 16, activation = "relu", input_shape = c(10000)) %>% 
#   layer_dense(units = 16, activation = "relu") %>% 
#   layer_dense(units = 1, activation = "sigmoid")
# model %>% compile(
#   optimizer = "rmsprop",
#   loss = "binary_crossentropy",
#   metrics = c("accuracy")
# )
# model %>% fit(x_train, y_train, epochs = 4, batch_size = 512)
```

###  Cross validation for keras à la main (Eric)
```{r}

nbbloc <-10
set.seed(1234)
bloc <- sample(rep(1:nbbloc,length=nrow(x_train_test)))
RES <- data.frame(Y=y_train_test, pred=0)
list_of_history <- list()
vector_of_accuracy <- c()
for(ii in 1:nbbloc){
  print(ii)
  partial_x_train_test <- x_train_test[bloc!=ii,] # dim(partial_x_train_test) : 1183   15
  partial_y_train_test <- y_train_test[bloc!=ii] # length(partial_y_train_test) = 1183
  x_val_test <- x_train_test[bloc==ii,] # dim(x_val_test) = 132  15
  y_val_test <- y_train_test[bloc==ii] # length(y_val_test) = 132
  
  model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)
  
  history <- model %>% fit(
  partial_x_train_test,
  partial_y_train_test,
  epochs = 5,
  batch_size = 512,
  validation_data = list(x_val_test, y_val_test)
)
  list_of_history <- c(list_of_history, list(history))
  
  vector_of_accuracy <-c(vector_of_accuracy, mean(history$metrics$accuracy)) 
  RES[bloc==ii,"pred"] <- model %>% predict(x_val_test[,])
  
}


RES

```

# creer une liste de history (1 history pour un boucle de CV)
```{r}
str(history)
str(list_of_history[[2]]) # extraire un history et montrer la visu
plot(list_of_history[[8]])

# calculer la moyen de accuracy de CV
accuracy <- vector_of_accuracy

#list_of_history[c(1,2)]
```

# calculer à la main les metrics (roc, accuracy, AUC)
```{r}
library(pROC)
plot(roc(RES[,1],RES[,2]))


for(jj in 2:ncol(RES)){
  print(paste("pour la methode ",colnames(RES)[jj],
              " l'AUC vaut :",
              round(auc(RES[,1],RES[,jj]),2),sep=""))
}

res <- NULL
for(jj in 2:(ncol(RES))){
    tmp <- roc(RES[,1],RES[,jj])
    res <- rbind(res,coords(tmp,"best"))
}
rownames(res) <- colnames(RES)[-1]
res$tot = res$sensitivity+res$specificity
res

```


# Evaluation avec test_set
```{r}
results <- model %>% evaluate(x_test_test, y_test_test)
```


```{r eval = F}
results
```

Our fairly naive approach achieves an accuracy of 88%. With state-of-the-art approaches, one should be able to get close to 95%.

## Using a trained network to generate predictions on new data

After having trained a network, you'll want to use it in a practical setting. You can generate the likelihood of reviews being positive by using the `predict` method:

```{r eval = F}


model %>% predict(x_test_test[,])
```

As you can see, the network is very confident for some samples (0.99 or more, or 0.02 or less) but less confident for others. 

## Further experiments


* We were using 2 hidden layers. Try to use 1 or 3 hidden layers and see how it affects validation and test accuracy.
* Try to use layers with more hidden units or less hidden units: 32 units, 64 units...
* Try to use the `mse` loss function instead of `binary_crossentropy`.
* Try to use the `tanh` activation (an activation that was popular in the early days of neural networks) instead of `relu`.

These experiments will help convince you that the architecture choices we have made are all fairly reasonable, although they can still be improved!

## Conclusions


Here's what you should take away from this example:

* You usually need to do quite a bit of preprocessing on your raw data in order to be able to feed it -- as tensors -- into a neural network. Sequences of words can be encoded as binary vectors, but there are other encoding options, too.
* Stacks of dense layers with `relu` activations can solve a wide range of problems (including sentiment classification), and you'll likely use them frequently.
* In a binary classification problem (two output classes), your network should end with a dense layer with one unit and a `sigmoid` activation. That is, the output of your network should be a scalar between 0 and 1, encoding a probability.
* With such a scalar sigmoid output on a binary classification problem, the loss function you should use is `binary_crossentropy`.
* The `rmsprop` optimizer is generally a good enough choice, whatever your problem. That's one less thing for you to worry about.
* As they get better on their training data, neural networks eventually start _overfitting_ and end up obtaining increasingly worse results on data they've never seen before. Be sure to always monitor performance on data that is outside of the training set.