---
title: "Solution / First Steps in Data Analytics with R"
author: "Martin Schedlbauer, PhD"
output:
  pdf_document: default
  html_document:
    df_print: paged
---

## Data Analysis

```{r echo = F}
url <- "http://artificium.us/assignments/55.data-analytics/a-55-103/customertxndata.csv"

df <- read.csv(url,
               header = T,
               stringsAsFactors = F)
```

The customer data set contains `r nrow(df)` transactions and has the following structure:

```{r echo = F, comment=""}
str(df)
```

```{r echo=F}
tot.rev <- sum(df$rev, na.rm = T)
mean.vis <- mean(df$numvisits, na.rm = T)
med.rev <- median(df$rev, na.rm = T)
sd.rev <- sd(df$rev, na.rm = T)

# calculate most commonly occurring gender
uniqv <- unique(df$gender)
mode.gender <- uniqv[which.max(tabulate(match(df$gender, uniqv)))]
```

The total revenue of all transactions is `r paste("$",format(round(tot.rev,0), big.mark = ",", scientific = FALSE))`, while the median transaction amount is `r paste("$", round(med.rev,2))` with a standard deviation of `r round(sd.rev, 2)`. The average number of visits was `r round(mean.vis, 0)`. The majority of customers were `r paste0('"',mode.gender,'"')`.

```{r echo = F}
# calculate revenue by gender
rev.male <- sum(df$rev[which(df$gender == "Male")])
rev.female <- sum(df$rev[which(df$gender == "Female")])

# combine values into a vector
revByGender <- c(rev.female, Male = rev.male)

# Note: to do the above grouping more generally requires either
# the use of the group_by function from "dplyr" or a SQL statement
# applied using "sqldf"

barplot(revByGender,
        main = "Revenue by Gender",
        xlab = "Gender",
        ylab = "Revenue",
        names.arg = c("Female", "Male"),
        col = "darkred",
        horiz = FALSE)
```

The barplot indicates that the revenue from "male" customers was almost twice as much as those of "female" customers.

### Correlation Analysis

```{r echo = F}
cor.vr <- cor(df$numvisits, df$rev)
```

There `r ifelse(cor.vr < 0.6, "does not appear", "appears")` to be a correlation between the number of visits a customer made and the revenue spent on a transaction (*r = `r round(cor.vr, 2)`*).

### Missing Data Analysis

```{r echo = F}
missing.gender <- length(which(is.na(df$gender)))
missing.os <- length(which(is.na(df$OS)))
missing.txns <- length(which(is.na(df$numtxns)))
```

The data set has a few columns that have missing data, specifically, there are `r missing.gender` values for "gender" and `r missing.txns` values for "numtxns" missing from the data set. When analyzing the data we could either ignore transactions where the gender is missing, or we could either assume a default gender, assign a neutral gender values, or impute the gender with the most commonly occurring gender. For the number of transactions, we could impute missing values with the mean or using a machine learning algorithm to estimate them from the other variables for the row.

### Imputing Missing Values

We will use mode to impute missing gender values and mean to impute missing values for the number of transactions.

```{r}
df$gender[which(is.na(df$gender))] <- mode.gender
df$numtxns[which(is.na(df$numtxns))] <- mean(df$numtxns, na.rm = T)
```

Of course, imputing changes the values for the various statistics, although it might be more meaningful than to simply ignore them. Often the context determines whether ignoring missing values or using imputed values is best.

```{r echo=F}
tot.rev <- sum(df$rev, na.rm = T)
mean.vis <- mean(df$numvisits, na.rm = T)
med.rev <- median(df$rev, na.rm = T)
sd.rev <- sd(df$rev, na.rm = T)

# calculate most commonly occurring gender
uniqv <- unique(df$gender)
mode.gender <- uniqv[which.max(tabulate(match(df$gender, uniqv)))]
```

Using the imputed values, we have a total revenue of all transactions of `r paste0("$ ",format(round(tot.rev,0), big.mark = ",", scientific = FALSE))`, while the median transaction amount is `r paste("$", round(med.rev,2))` with a standard deviation of `r round(sd.rev, 2)`. The average number of visits was `r round(mean.vis, 0)`. The majority of customers were `r paste0('"',mode.gender,'"')`.

### Training and Validation Samples

Splitting the data into two subsets is common in machine learning. The larger subset is used to train the model, while the smaller subset is held back for validation of the model. For now, we will split the data between training and validation evenly and use the odd numbered cases for training and the even numbered cases for validation, *i.e.*, rows 1, 3, 5, 7, *etc.* are training data while rows 2, 4, 6, *etc.* are validation data.

```{r}
even.rows <- seq(from = 2, to = nrow(df), by = 2)
odd.rows <- seq(from = 1, to = nrow(df), by = 2)

df.train <- df[even.rows,]
df.val <- df[odd.rows,]
```

```{r echo = F}
train.mean.rev <- mean(df.train$rev)
val.mean.rev <- mean(df.val$rev)

t <- t.test(df.train$rev,df.val$rev)
```

Assuming that the rows are somewhat random, the mean revenue between the validation and training data sets should be about the same. The actual values are `r round(train.mean.rev,1)` for the training sample and `r round(val.mean.rev,1)` for the validation sample. The difference is minor and a *t*-test shows, as expected, that the difference is not statistically significant (t = `r round(t$statistic,3)`, *p* \> 0.05).

Rather than evenly splitting the data, it is better to take a random sample, particularly when we are not sure that the data is not somehow ordered. The code below takes a random sample of 60% for training, 20% for testing, and another 20% for validation.

```{r}
# Set the seed so that same sample can be reproduced in the future
set.seed(101) 

# Draw 60% of the data from total 'n' rows of the data  
sample <- sample.int(n = nrow(df), size = floor(.60*nrow(df)), replace = F)
train <- df[sample, ]

# Take a 50% sample (half) from the remaining 40%
remaining  <- df[-sample, ]
sample <- sample.int(n = nrow(remaining), size = floor(.50*nrow(remaining)), replace = F)
testing <- remaining[sample,]
validation <- remaining[-sample,]
```
