{"id":1776,"date":"2018-07-02T12:54:16","date_gmt":"2018-07-02T10:54:16","guid":{"rendered":"https:\/\/blogs.fu-berlin.de\/reseda\/?page_id=1776"},"modified":"2019-07-29T17:27:33","modified_gmt":"2019-07-29T15:27:33","slug":"s2-bulk-preprocessing","status":"publish","type":"page","link":"https:\/\/blogs.fu-berlin.de\/reseda\/s2-bulk-preprocessing\/","title":{"rendered":"Sentinel 2 Bulk Preprocessing"},"content":{"rendered":"<p>This section contains advanced techniques. Understanding these techniques is particularly relevant if you want to pre-process dozens of Sentinel 2 data in an identical way.<br \/>\nWe will create a graph in SNAP and then modify it to process all data automatically via the Graph Processing Tool (GPT)!<\/p>\n<p>Open SNAP and click on the graph builder icon <img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-899\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/05\/SNAP_buttons_05.png\" alt=\"\" width=\"28\" height=\"32\" \/> in the Toolbar. You will see a new window pop up, which shows a standard graph (containing one <strong>Read <\/strong>and one <strong>Write <\/strong>node) in the upper area and some options in the lower area. Right click somewhere in the white area at upper window to add new nodes, i.e., processing steps, to the graph. Navigate to <em>Add <\/em>&gt; <em>Raster <\/em>&gt; <em>Geometric <\/em>&gt; <em>Resample<\/em> to add an <strong>Resample <\/strong>and a <strong>Subset <\/strong>node (those are the steps we used in the above section):<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_13.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1739\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_13.png\" alt=\"\" width=\"650\" height=\"749\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Add new nodes to the graph via a right click<\/span><\/div>\n<p>Each processing step is visualized by a rectangle in the graph. You can drag those rectangles, or nodes, to whichever position you want with pressed left mouse button. Your graph should now look something like this:<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_14.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1742\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_14.png\" alt=\"\" width=\"468\" height=\"101\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Unconnected graph<\/span><\/div>\n<p>The individual nodes are not yet linked. If you rest the mouse pointer over the right edge of a rectangle, you will see a red arrow. Drag this arrow with pressed left mouse button to the next node in the following order:<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_15.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1743\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_15.png\" alt=\"\" width=\"468\" height=\"101\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Connected graph<\/span><\/div>\n<p>Next, click on the <strong>Read <\/strong>Node and load one of your ZIP-compressed Sentinel 2 scenes into the Source Product setting. This could take a short moment. <strong>Attention: <\/strong> From now on, it will always take a little while to switch back and forth between the processing nodes, as SNAP will process and read your data as virtual products in the background!<\/p>\n<p>Now either click through the tabs at the bottom of the Graph Builder window or through the rectangles at the top to set the following options for the <strong>Resampling<\/strong>, the <strong>Subset <\/strong>and the <strong>Write <\/strong>node (according to the settings made in the previous section):<\/p>\n<div class=\"h5p-iframe-wrapper\"><iframe id=\"h5p-iframe-11\" class=\"h5p-iframe\" data-content-id=\"11\" style=\"height:1px\" src=\"about:blank\" frameBorder=\"0\" scrolling=\"no\" title=\"preproc Sentinel 2\"><\/iframe><\/div>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Graph settings<\/span><\/div>\n<p>The spatial section in the Subset node should be made using the geographic coordinates. However, the linked map is quite blurred and therefore not much help. However, under the map you can define any spatial subset using the Well-Known Text (WKT) format. There are a number of <a href=\"http:\/\/arthur-e.github.io\/Wicket\/sandbox-gmaps3.html\" target=\"_blank\" rel=\"noopener\">simple online tools<\/a> for creating such WKTs. Copy the entire &#8220;Polygon(())&#8221; expression into the line below the map within the subset Node:<\/p>\n<p>POLYGON((13.11 52.65, 13.70 52.65, 13.70 52.35, 13.11 52.35, 13.11 52.65))<\/p>\n<p>Of course, you can add more processing steps later for your own analysis. When you are done click Save on the bottom of the Graph Builder window and save your graph as &#8220;myGraph.xml&#8221; to your hard drive (exchange folder recommended):<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_20.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1750\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_20.png\" alt=\"\" width=\"532\" height=\"47\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_20.png 532w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_20-300x27.png 300w\" sizes=\"auto, (max-width: 532px) 85vw, 532px\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Save graph as .xml-file<\/span><\/div>\n<p><a name=\"1\"><\/a><\/p>\n<h1>Automate Things Using GPT<\/h1>\n<p>Close SNAP, as we do not need it anymore. Now, open a terminal, which is linked in the taskbar of our VM:<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/05\/USGS_036.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-1280\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/05\/USGS_036.png\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Linux terminal shortcut in taskbar<\/span><\/div>\n<p>It is possible to run the graph using the GPT now. The GPT can be found in our VM in the file path &#8220;\/home\/student\/snap\/bin\/gpt&#8221;. Consequently, we can execute the graph with the following command (change the file path corresponding to the location of your graph file!):<\/p>\n<pre class=\"theme:amityresedaterminal nums:false\">\/home\/student\/snap\/bin\/gpt \/media\/sf_exchange\/sentineldata\/myGraph.xml\r\n<\/pre>\n<p>Replace the path with the path of your file, copy it to the terminal and press Enter!<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_21.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1755\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_21.png\" alt=\"\" width=\"737\" height=\"372\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_21.png 737w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_21-300x151.png 300w\" sizes=\"auto, (max-width: 737px) 85vw, 737px\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Execute graph via GPT in command line<\/span><\/div>\n<p>A pre-processed file should be created directly in the same folder as your zip-compressed source file.<br \/>\nHowever, you can only process one file with the graph so far&#8230; That is not exactly what we wanted. That is why we will make some changes to the xml-file now. Right click the xml-file and choose <em>Open With<\/em> &gt; <em>Open With &#8220;Text Editor&#8221;<\/em>. Take a moment to understand how the file is structured. The individual inputs and outputs of all processes, or nodes, are saved here as a text. Each of the successive processes is opened by a <span class=\"crayon-inline theme:amityresedaterminal\">&lt;node = id &#8220;&#8221;&gt;<\/span> tag and closed by <span class=\"crayon-inline theme:amityresedaterminal\">\/node&gt; <\/span>.<br \/>\nSearch for the <strong>Read <\/strong>node. Within that node, a line should list the Sentinel 2 file you used when creating the graph (the line number may differ in your xml):<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_22.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1763\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_22.png\" alt=\"\" width=\"860\" height=\"121\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_22.png 860w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_22-300x42.png 300w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_22-768x108.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Read node in xml<\/span><\/div>\n<p>Replace the filename (marked green in the figure above) with the placeholder <span class=\"crayon-inline theme:amityresedaterminal\">$input<\/span>:<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_24.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1764\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_24.png\" alt=\"\" width=\"860\" height=\"121\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_24.png 860w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_24-300x42.png 300w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_24-768x108.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Modified Read node in xml<\/span><\/div>\n<p>Now search for the <strong>Write <\/strong>node:<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_23.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1765\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_23.png\" alt=\"\" width=\"860\" height=\"165\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_23.png 860w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_23-300x58.png 300w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_23-768x147.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Write node in xml<\/span><\/div>\n<p>And replace the file path with the placeholder <span class=\"crayon-inline theme:amityresedaterminal\">$output<\/span>:<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_25.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1766\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_25.png\" alt=\"\" width=\"860\" height=\"165\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_25.png 860w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_25-300x58.png 300w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/06\/preproc_25-768x147.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<div style=\"margin: -30px 0 20px 0;text-align: center\"><span style=\"color: #686868;font-size: small\">Modified Write node in xml<\/span><\/div>\n<p>The input and output paths have now been replaced by placeholders, which could dynamically accept any file. When calling the graph via GPT, the two variables <span class=\"crayon-inline theme:amityresedaterminal\">$input<\/span> and <span class=\"crayon-inline theme:amityresedaterminal\">$output<\/span> must always be defined with from now on. A possible call for a file might look like this:<\/p>\n<pre class=\"theme:amityresedaterminal nums:false\">\/home\/student\/snap\/bin\/gpt \/media\/sf_exchange\/sentineldata\/myGraph.xml -Pinput=\/media\/sf_exchange\/sentineldata\/S2B_MSIL1C_20170830T102019_N0205_R065_T33UUU_20170830T102531.zip -Poutput=\/media\/sf_exchange\/sentineldata\/S2B_MSIL1C_20170830T102019_N0205_R065_T33UUU_20170830T102531_sub_res.tif\r\n<\/pre>\n<p>Of course, we can take advantage of this and use a for-loop to go through and process all the Sentinel 2 datasets located in a folder one after another using R or a bash script. We have prepared a bash script for you, which does just that, located in <span class=\"crayon-inline theme:amityresedaterminal\">\/home\/student\/Documents\/S2_proc.bash<\/span>. This script tells the bash shell of Linux what it has to do, so you will need our VM or any Linux operating system. To run the script, open a terminal, reference the script and give it two arguments: first the full path to your graph .xml file and second the path where the downloaded Sentinel 2 zip-files are located. A possible command might look like this:<\/p>\n<pre class=\"theme:amityresedaterminal nums:false\">\/home\/student\/Documents\/S2_proc.bash \/media\/sf_exchange\/sentineldata\/myGraph.xml \/media\/sf_exchange\/sentineldata\/\r\n<\/pre>\n<p>The script starts as soon as you press enter and writes the output to a new folder called &#8220;Processed&#8221;, which will be created next to your S2 zip files.<\/p>\n<p><a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/07\/preproc_26.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1804\" src=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/07\/preproc_26.png\" alt=\"\" width=\"712\" height=\"414\" srcset=\"https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/07\/preproc_26.png 712w, https:\/\/blogs.fu-berlin.de\/reseda\/files\/2018\/07\/preproc_26-300x174.png 300w\" sizes=\"auto, (max-width: 712px) 85vw, 712px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<hr style=\"height: 4px;background-color: #6b9e1f\" \/>\n<div style=\"font-family: 'Noto Sans', sans-serif;line-height: 1.2;text-align: right\"><span style=\"font-size: 12px;color: #bfbfbf\"><strong><em>NEXT<\/em><\/strong><\/span><br \/>\n<a style=\"text-decoration: none\" href=\"https:\/\/blogs.fu-berlin.de\/reseda\/quality-band\/\"><span style=\"font-size: 30px;color: #6b9e1f\"><strong><em>CLOUD MASKING<\/em><\/strong><\/span><\/a><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This section contains advanced techniques. Understanding these techniques is particularly relevant if you want to pre-process dozens of Sentinel 2 data in an identical way. We will create a graph in SNAP and then modify it to process all data automatically via the Graph Processing Tool (GPT)! Open SNAP and click on the graph builder &hellip; <a href=\"https:\/\/blogs.fu-berlin.de\/reseda\/s2-bulk-preprocessing\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Sentinel 2 Bulk Preprocessing&#8221;<\/span><\/a><\/p>\n","protected":false},"author":3237,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1776","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/pages\/1776","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/users\/3237"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/comments?post=1776"}],"version-history":[{"count":12,"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/pages\/1776\/revisions"}],"predecessor-version":[{"id":3040,"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/pages\/1776\/revisions\/3040"}],"wp:attachment":[{"href":"https:\/\/blogs.fu-berlin.de\/reseda\/wp-json\/wp\/v2\/media?parent=1776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}