{"id":1578,"date":"2018-06-25T13:25:35","date_gmt":"2018-06-25T11:25:35","guid":{"rendered":"https:\/\/blogs.fu-berlin.de\/reseda\/?page_id=1578"},"modified":"2018-10-15T11:30:29","modified_gmt":"2018-10-15T09:30:29","slug":"landsat-8-preprocessing","status":"publish","type":"page","link":"https:\/\/blogs.fu-berlin.de\/reseda\/landsat-8-preprocessing\/","title":{"rendered":"Landsat 8 Preprocessing"},"content":{"rendered":"

Landsat 8 ships as a tar-archived file<\/a> with the spectral bands as individual georeferenced tif images. We want to stack these images into a single geotiff-file, i.e., into a so-called raster stack. Afterwards, it is much easier to work with such a raster stack. While this could also be done in QGIS, we will use R for this preprocessing, because it is easier to automate things this way. This results in the following \tintermediate steps we have to check off:<\/p>\n

    \n
  1. unzip your downloaded L8 files<\/li>\n
  2. put together the spectral band files of your choice into a rasterstack<\/li>\n
  3. save this rasterstack to your hard drive<\/li>\n
  4. (optional) delete all redundant data <\/li>\n
  5. (optional) create pyramid layers for a better visualization in QGIS <\/li>\n<\/ol>\n

    We will practice everything exemplarily on the basis of a single L8 Level-2 data set. There will be an exhaustive explanation for each line of code. Based on that we will develop a script that will automatically do everything for you in the future. <\/p>\n

    <\/br><\/p>\n

    Prerequisite<\/h1>\n

    The following content requires that you have either successfully downloaded some Landsat 8 scenes as part of the Download Section Exercise<\/a>, or that you have acquired some datasets from the USGS EarthExplorer by your own. If this is not the case, look into the chapter Landsat \/ Earthexplorer<\/a>!<\/p>\n

    Done? – Then start R-Studio<\/a> now!<\/p>\n

    <\/br><\/a><\/p>\n

    Preprocess a single dataset<\/h1>\n

    We recommend writing the following code in the script window of R-Studio and executing it from there (see chapter R-Studio<\/a>).
    \nThe example below assumes that you have one or more Landsat 8 scenes in a “landsatdata”-folder located in the exchange folder of your VM. Otherwise, adjust the folder according to your data!<\/p>\n

    We will use the raster library<\/a> to write a raster file later. Additional libraries should always be loaded first, using the function library()<\/span>:<\/p>\n

    \r\nlibrary(raster)\r\n## Loading required package: sp\r\n<\/pre>\n
    A few libraries make use of other libraries. So does the raster library with the sp-package<\/a>, which will be loaded automatically.<\/p>\n
    Then define the Landsat 8 product of your choice with its entire file path and save it as the string variable product<\/span>:<\/p>\n
    \r\nproduct <- "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412.tar.gz"\r\n\r\nfile.exists(product)\r\n## [1] TRUE\r\n<\/pre>\n
    NOTE:<\/strong> this is just an example – you have to change the file and its path according to your own settings!
    \nYou can use the function file.exists() <\/span> to check whether the file can be found on your system or not. If it returns FALSE<\/span>, make sure you did not mess up the file name.<\/p>\n
    We want to unpack the file into a folder with the same name as the file. The character variable product<\/span> already contains the full name of the Landsat scene. We just have to get rid of the suffix “.tar.gz”. Therefore we can use the function substr()<\/span> to delete the last seven characters (=”.tar.gz”) of the string:<\/p>\n
    \r\nproductname <- substr(product, 1, nchar(product) - 7) \r\n\r\nproductname\r\n## [1] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412"\r\n<\/pre>\n
    The substring function substr()<\/span> accepts three arguments here: a string (the entire file path with the suffix), and two integer values – one for a start (“1” = start with the first character) and one for a stop position within the given string. In order to define the stop position, we need to count the number of all characters of the file path via nchar(product)<\/span> (which is 77 in our case) and substract 7.<\/p>\n
    Unzip the Landsat product using the untar()<\/span> method and define the directory to which all data will be extracted by setting the argument exdir = productname<\/span>:<\/p>\n
    \r\nuntar(product, exdir = productname)\r\n<\/pre>\n
    You should notice a new folder being created next to your initial data product. This could take a short moment. During processing, a red exclamation point can be seen in the upper right corner of the console window. NOTE:<\/strong> If this step fails, your data is corrupt. In this case, you will need to download the file again because something went wrong during data transfer.
    \nAfter unpacking is complete, we want to have a look in the newly extracted folder and save the file names of all files in this folder into a vector productfiles<\/span> using the function list.files<\/span>. By providing the argument full.names = TRUE<\/span>, the full paths are returned for each file:<\/p>\n
    \r\nproductfiles <- list.files(productname, full.names = TRUE)\r\n\r\nproductfiles\r\n##  [1] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_ANG.txt"       \r\n##  [2] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_MTL.txt"       \r\n##  [3] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_pixel_qa.tif"  \r\n##  [4] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_radsat_qa.tif" \r\n##  [5] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_aerosol.tif"\r\n##  [6] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band1.tif"  \r\n##  [7] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band2.tif"  \r\n##  [8] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band3.tif"  \r\n##  [9] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band4.tif"  \r\n## [10] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band5.tif"  \r\n## [11] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band6.tif"  \r\n## [12] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band7.tif"  \r\n## [13] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1.xml"\r\n<\/pre>\n
    These are all 13 files that come with a Level-2 product (see Landsat 8<\/a> for more information). As a double check, you can have a look at the identical files in your file explorer:<\/p>\n
    <\/a><\/p>\n
    Now we want to select all the spectral bands that we want to put into our new data stack. Have a deeper look at the files in productfiles<\/span>. The files are named after the corresponding spectral channels at the end of the file name, e.g., “band1”, “band2”, and so on. We use these terms to extract the bands by utilizing the grep()<\/span> function. The following code looks a little bloated. Maybe it is. But he gives you the freedom to exclude individual bands that you may not need. NOTE:<\/strong> This is an example of level 2 data. Landsat Level-1 scenes potentially have 11 channels. In the case of Level 1 data, you could easily enter the remaining lines here.<\/p>\n
    \r\nbands <- c(grep('band1', productfiles, value=TRUE),\r\n           grep('band2', productfiles, value=TRUE),\r\n           grep('band3', productfiles, value=TRUE),\r\n           grep('band4', productfiles, value=TRUE),\r\n           grep('band5', productfiles, value=TRUE),\r\n           grep('band6', productfiles, value=TRUE),\r\n           grep('band7', productfiles, value=TRUE)\r\n           ) \r\n\r\nbands\r\n## [1] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band1.tif"\r\n## [2] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band2.tif"\r\n## [3] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band3.tif"\r\n## [4] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band4.tif"\r\n## [5] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band5.tif"\r\n## [6] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band6.tif"\r\n## [7] "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412\/LC08_L1TP_193023_20170602_20170615_01_T1_sr_band7.tif"\r\n<\/pre>\n
    The grep()<\/span> function searches for the string, which is given as the first argument, e.g., ‘band1’, in the vector of strings (2nd argument). Normally, when the function finds a match, it only returns the position of this find in the vector productfiles<\/span>. By setting the argument value=TRUE<\/span> we can write out the content of the vector at the respective position instead. By doing so, we create an vector bands<\/span> containing all file paths via the standard combine-function c()<\/span>. <\/p>\n
    Now we can create a rasterstack based on the vector of bands in the variable bands<\/span>. In order to do so, we use the function stack()<\/span>, which is part of the raster-library we loaded in the beginning! This rasterstack function<\/a> creates a rasterstack object based on a list of filenames. You will learn more about the beauty of rasterstacks in the chapter Visualization<\/a>. <\/p>\n
    \r\nrasterstack <- stack( bands )\r\n<\/pre>\n
    We can save this rasterstack and store it on your hard drive via writeRaster<\/span>:<\/p>\n
    \r\nwriteRaster(rasterstack, \r\n            paste0(productname, ".tif"), \r\n            format = "GTiff",\r\n            overwrite = TRUE\r\n            )\r\n<\/pre>\n
    The writerRaster fuction<\/a>  is a powerful tool, which is also provided by the raster package<\/a>. In line 1 we set our rasterstack object as the first argument. In line 2 we define the output name of the new file we will create. To do this, just add the suffix “.tif” to our filename using the handy function paste0<\/span>, which just put all the strings together to one string. Line 3 explicitly defines the output format “GTiff”. How should I know this string “GTiff”, you ask? These strings are fixed by the function and are also listed for other data formats in the raster documentation<\/a> on page 220. The fourth argument in line 4 only gives the authority to overwrite existing files.<\/p>\n
    DONE!<\/strong> A new file containing all spectral bands is now written directly next to the initial packed file you downloaded! <\/p>\n
    There are still two useful additional things left to do:
    \nFirst, we can now delete the folder we created by uncompressing your Landsat data because it is no longer needed. So, if you want to save disk space, use the command unlink()<\/span> to simply delete the folder. By setting the argument recursive=TRUE<\/span>, all files within the folder will be deleted:<\/p>\n
    \r\nunlink(productname, recursive=TRUE)\r\n<\/pre>\n
    Second, it is advisable to create so-called pyramid layers<\/a> for each Landsat dataset. Pyramid layers are used to improve performance. They are a downsampled version of the original raster and speed up the display of raster data by retrieving only the data at a specified resolution that is required for the display. We can automatically create them by using the Geospatial Data Abstraction Library<\/a> (GDAL). Initially, GDAL has nothing to do with R, but it can be operated via R using rgdal<\/a>. However, we use the standalone installation of GDAL at this point, or more precisely, the function gdaladdo<\/a>. If you do not use our VM, you have to install GDAL on your machine first. We build the appropriate command to run gdaladdo<\/span> using paste0()<\/span> and pass that command to the system()<\/span> function. The system()<\/span> function executes a command exactly as if you entered it yourself in the command window of Linux. Try it!<\/p>\n
    \r\nmakeOVR <- paste0("gdaladdo -ro ", productname, ".tif 2 4 8 16")\r\n\r\nmakeOVR\r\n## [1] "gdaladdo -ro \/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412.tif 2 4 8 16"\r\n\r\nsystem( makeOVR )\r\n<\/pre>\n
    Run the code and you will create a .ovr-file next to your .tif-file. This ovr-file holds the pyramid information, which will prove useful later in QGIS.<\/p>\n
    Phew! This was A LOT of (explanatory) text by now. Fortunately, the code is actually much shorter! Here is the complete code for preprocessing one exemplary L8 Level-2 file:<\/p>\n
    \r\nlibrary(raster)\r\n\r\nproduct <- "\/media\/sf_exchange\/landsatdata\/LC081930232017060201T1-SC20180613160412.tar.gz"\r\n\r\nproductname <- substr(product, 1, nchar(product) - 7) \r\n\r\nuntar(product, exdir = productname)\r\n\r\nproductfiles <- list.files(productname, full.names = TRUE)\r\n\r\nbands <- c(grep('band1', productfiles, value=TRUE),\r\n           grep('band2', productfiles, value=TRUE),\r\n           grep('band3', productfiles, value=TRUE),\r\n           grep('band4', productfiles, value=TRUE),\r\n           grep('band5', productfiles, value=TRUE),\r\n           grep('band6', productfiles, value=TRUE),\r\n           grep('band7', productfiles, value=TRUE)\r\n           ) \r\n\r\nrasterstack <- stack( bands )\r\n\r\nwriteRaster(rasterstack, \r\n            paste0(productname, ".tif"), \r\n            format = "GTiff",\r\n            overwrite = TRUE\r\n            )\r\n\r\nunlink(productname, recursive=TRUE)\r\n\r\nmakeOVR <- paste0("gdaladdo -ro ", productname, ".tif 2 4 8 16")\r\nsystem( makeOVR )\r\n<\/pre>\n
    Learn how to automate things for many datasets in the next section!<\/p>\n
    \n
    \n