{"id":2155,"date":"2018-07-19T17:25:51","date_gmt":"2018-07-19T15:25:51","guid":{"rendered":"https:\/\/blogs.fu-berlin.de\/reseda\/?page_id=2155"},"modified":"2018-09-27T15:17:20","modified_gmt":"2018-09-27T13:17:20","slug":"learning-curve","status":"publish","type":"page","link":"https:\/\/blogs.fu-berlin.de\/reseda\/learning-curve\/","title":{"rendered":"Learning Curve"},"content":{"rendered":"

One question that is asked again and again is the amount of training data needed for a supervised classification. This question is not easy to answer, because every classification and regression problem is different.
\nHowever, there is an ideal analysis method to assess this problem for certain: creating a learning curve<\/a>:
\nSimply repeat your training procedure multiple times with different train sample sizes and compare the performance differences!
\nPlotting those results will (hopefully) show a decreasing error rate with increasing amounts of train data.<\/p>\n

In the following, a complete script has been developed, which performs the RF classification from the previous section<\/a> with a maximum of \\(\\{2, 4, 8, 16, …, 2048\\}\\) samples per class (steps<\/span>). In order to minimize any random effects and specify a confidence interval, all classifications were repeated 20 times each (nrepeat<\/span>).<\/p>\n

In order not to unnecessarily inflate the code, we use for loops to iterate over the individual sample sizes and iterations and write the respective results into a matrix r<\/span>. The following code is comprehensively documented. <\/p>\n

\r\n# import libraries\r\nlibrary(raster)\r\nlibrary(randomForest)\r\n\r\n# import image (img) and shapefile (shp)\r\nsetwd(\"\/media\/sf_exchange\/landsatdata\/\")\r\nimg <- brick("LC081930232017060201T1-SC20180613160412_subset.tif")\r\nshp <- shapefile("training_data.shp")\r\n\r\n# extract samples with class labels and put them all together in a dataframe\r\nnames(img) <- c("b1", "b2", "b3", "b4", "b5", "b6", "b7")\r\nsmp <- extract(img, shp, df = TRUE)\r\nsmp$cl <- as.factor(shp$classes[match(smp$ID, seq(nrow(shp)))])\r\nsmp <- smp[-1]\r\n\r\n# number of samples per class chosen for each run\r\nsteps <- 2^(1:11)\r\n# number of repetitions of each run\r\nnrepeat = 20\r\n\r\n# create empty matrix for final results\r\nr <- matrix(0, 5, length(steps))\r\n\r\nfor (i in 1:length(steps)) {\r\n  # create empty vector for OOB error from each repetition\r\n  rtmp <- rep(0, nrepeat)\r\n  \r\n  for (j in 1:nrepeat) {\r\n    # shuffle all samples and subset according to size defined in "steps"\r\n    sub <- smp[sample(nrow(smp)),]\r\n    sub <- sub[ave(1:(nrow(sub)), sub$cl, FUN = seq) <= steps[i], ]\r\n    \r\n    # RF classify as usual\r\n    sub.size <- rep(min(summary(sub$cl)), nlevels(sub$cl))\r\n    rfmodel <- tuneRF(x = sub[-ncol(sub)],\r\n                      y = sub$cl,\r\n                      sampsize = sub.size,\r\n                      strata = sub$cl,\r\n                      ntree = 250,\r\n                      doBest = TRUE,plot = FALSE\r\n                      )\r\n    # extract OOB error rate of last tree (the longest trained) and save to rtmp\r\n    ooberrors <- rfmodel$err.rate[ , 1]\r\n    rtmp[j] <- ooberrors[length(ooberrors)]\r\n  }\r\n  \r\n  # use repetitions to calc statistics (mean, min, max, CI) & save it to final results matrix\r\n  ttest <- t.test(rtmp, conf.level = 0.95)\r\n  r[ , i] <- c(mean(rtmp), ttest$conf.int[2], ttest$conf.int[1], max(rtmp), min(rtmp))\r\n}\r\n<\/pre>\n
When the individual experiments have been processed, we can create a nice plot that summarizes the findings:<\/p>\n
\r\n# conversion in percent\r\nr <- r * 100\r\n# plot empty plot without x-axis\r\nplot(x = 1:length(steps),\r\n     y = r[1,],\r\n     ylim = c(min(r), max(r)),\r\n     type = "n",\r\n     xaxt = "n",\r\n     xlab = "number of samples per class",\r\n     ylab = "OOB error [%]"\r\n     )\r\n# complete the x-axis \r\naxis(1, at=1:length(steps), labels=steps)\r\n# add a grid\r\ngrid()\r\n# draw min-max range of OOB errors\r\npolygon(c(1:length(steps), rev(1:length(steps))), \r\n        c(r[5, ], rev(r[4, ])),\r\n        col = "grey80", \r\n        border = FALSE\r\n        )\r\n# draw confidence interval 95%\r\npolygon(c(1:length(steps), rev(1:length(steps))), \r\n        c(r[3, ], rev(r[2, ])),\r\n        col = "grey60", \r\n        border = FALSE\r\n        )\r\n# draw line of mean OOB \r\nlines(1:length(steps), r[1, ], lwd = 3)\r\n# add a legend\r\nlegend("topright",\r\n       c("mean", "t-test CI 95%", "min-max range"),\r\n       col = c("black", "grey80", "grey60"),\r\n       lwd = 3,\r\n       bty = "n"\r\n       )\r\n<\/pre>\n
\"\"<\/a><\/p>\n
Conclusion for this example: From 512 samples per class the achieved improvement is negligible. So 512 samples per class should be enough to get a robust classification.<\/p>\n
<\/br><\/br><\/p>\n
\n
\n