Dataset Structure

You should now be more comfortable with finding datasets in the Streaming Data Library (SDL). Let us take a closer look at the information and utilities available to you when you select a dataset. You will not use these utilities in this section, but it is important to know what they do and where they are for later parts of this tutorial. The Copernicus Land NDVI dataset is a commonly used dataset and is the example for this section.

Select the Copernicus Land NDVI* dataset.


There is a wealth of information about the dataset on this default 'Description' page and the dataset can be manipulated through user interactions via the tabs shown above the graphical frame. Let's look at the content and use of all of these tabs from left to right.


Select the "Description" tab (if not yet selected).

Description

The description tab shows the default page that will be returned when the user is given a choice to proceed in the analysis. At the top of the page it shows the Source selection. Below that the main menu is shown as a row of several tabs. The graphical frame underneath this shows the flowchain in going from datasource(s) to the final analytics result and shows the datasource selections in green, any applied functions/filters to the data in blue and the end result in yellow. Next to each 'bubble' a short description is shown of what it represents. You can zoom in and out in this frame and rearrange the chain's shape. Any filtering applied to the dataset will be shown here as a new bubble in the analytics chain. When multiple datasets are used or when functions/filters use multiple paths to get to an intermediate result, you can see that in a visual manner in this chain. The section below the frame shows a description of the datasource; its name, links to documentation on the dataset, links to the datasets and variables to choose from, its independent variables (or grid), some relevant information about the data and references to these.


Select the "Views" tab.

Views

A series of icons are displayed which when clicked, take you to a rendered image of the selected data and an image viewer/editor. Note that you should have selected already at least one dataset for the viewer to show it. The datasets to be selected can be chosen from the list under the Description tab or can be set via the Expert Mode tab. Let's quickly go over the different types of images that can be selected. The colored and contoured images generally display a spatial (e.g., [X,Z], [lon,lat]) image as either colors or contours. The line graph typically illustrates a time series (e.g., [data value,T]) image. Other types of views icons will appear depending on your data selection. It is important to remember that the image in the icon is representative of the image to which the icon is linked. We will talk much more about these links and their visualization capabilities in Part III - Visualization.


Select the "Data Filters" tab.

Data Filters

This option allows you to filter the dataset using a limited set of pre-defined filters shown as links. Only a small subset of available filters is shown here. Filters not shown here have to be applied manually using the Expert Mode tab. In Part II of this introduction , you will learn how to use these filters to obtain a refined dataset.


Select the "Data Selection" tab.

Data Selection

This option allows you to pick out a subset of the full dataset. This can be done by restricting the range of one or more of the grids being used manually or by selecting the 'Data Viewer' via the link in this page. The latter is an interactive viewer which lets you select your one, two or even three dimensional data.

When manually editing the ranges, just enter the limits in the respective fields under 'Setting Ranges', click on button 'Restrict Ranges', check the new settings for the grids under 'Data Selection' and if you are satisfied with the selection click button 'Stop Selecting'. If you want to edit the grid ranges interactively, click the link 'Data Viewer' shown near the top of the 'Data Selection' section. This will start the data viewer.

Select the "Data Viewer" link.

As you can see the data viewer offers many options to manipulate the grid ranges as well as the graphical layout of the rendered image of the data. You can even select the format the resulting imagery has to be written to like pdf, a link or plain html. The data viewer is explained in more detail in the 'Data Viewer' section below.


Select the "Data Files" tab.

Data Files

This option allows you to download the dataset in a variety of formats. You will learn much more about this option in Part III - Download.


Select the "Data Tables" tab.

Data Tables

This option allows you to view the data as a table in a variety of formats. You will learn much more about this option in Part III.


Select the "Expert Mode" tab.

Expert Mode

This option switches the interface into a mode that allows you to enter data-manipulating commands directly rather than clicking on a series of pages to perform an identical task. You don't have to be already familiar with the command grammar at this point. The basic commands will be introduced throughout this tutorial.

When in Expert Mode, the dataset in use is shown in the top 'Source' bar and beneath it a data selection bar and the command editor window currently containing the command equivalent of the current data selection. Commands can be added and/or edited in this window. When done, click the OK button and your commands will start being interpreted. Your view will return at the Description tab. The RESET button will take your command set back to the point when you started editing. The commands that can be entered comprise the SDL Scripting language, explained in short at the bottom of this page. The data selection bar shows the possible sub datasets or variables available to you. Right now, you should see only one main dataset 'Copernicus Land NDVI' with three sub-datasets being 'CRS', 'NDVI' and 'TIME_GRID'.

Select the "NDVI" dataset by clicking the link.

This action will take you back to the Description window. Just click the 'Expert Mode' tab again to go back to our previous window.
Because no variable has yet been selected, you can see all of the possible variables listed to the right of the data selection bar (here: longitude, latitude and time). Much more detail about expert mode, including specific examples, will be given in Parts II and III of the Tutorial.

Go back to the dataset main page by selecting the "NDVI" link in the source bar.


We wil now explain a bit more about the Source Bar and the Data Viewer and the use of grids.

Source Bar

The source bar as depicted below, indicates to the user the location of the current dataset in the Streaming Data Library (SDL) and offers the opportunity to go to any of the higher levels in the hierarchy by selecting one of the links.

SOURCES /   Copernicus /   Land /   NDVI  

For example, you can view other Copernicus datasets by selecting the "Copernicus" link and available Land datasets by selecting the "Land" link. The source bar is an extremely valuable navigation tool in the SDL. Try using its options...


Data Viewer

The data viewer is an interactive tool to manipulate the grids you use for your dataset and to change the way the data is visualized. It also enables you to select the format your imagery will be saved to. You could for example just want a link to a rendered image instead of downloading it to local storage. We will go over the options available here.

The data viewer can be accessed from the Data Selection tab with a data source having been selected.

Near the top in the data viewer window the data source is shown together with the grid ranges in use and the grid type. Next, the main window is shown. At the center there is the rendered image obtained with the last applied settings. To the left and bottom of the image the grid ranges are shown for the vertical and horizontal grids in editable text boxes. Just above the image the time grid can be manipulated by editing the date range. To the left of the image three buttons are shown together with a zoom pulldown menu. Somewhat lower under the image the selected coloring range is shown which can also be changed to a user defined range. Under this some menus are shown with which you can alter the axes of the image, edit the drawing of land, sea or country contours and change the type of image being created. At the bottom of the page several options are available to obtain information on the selected data, to edit optional plotting settings for landscape marks as rivers, borders etc. and to export your image to a number of different formats.

Many of these options will be treated in more detail in Parts II and III. For a quick help on all these options use the 'Help' button to the left of the image. The page will refresh with help information shown in the lower half of the page.

After you have changed any of the variables in f.i. the text boxes you can refresh the image by clicking the 'Redraw' button to the left of the image. Using the middle 'Zoom' button one can zoom in or out of the image afer having selected one of the zoom levels from the pulldown menu. By just clicking in the image with the mouse, the image will be re-centered along the grid axes (if possuble).

Another option worth noting here is the ability to animate over a range of dates. If the dataset is given a range of dates (in the textbox just above the image), a click on the Redraw button will start an animation, in fact looping over the results for the range of dates given. One can manually loop over the date range by clikcing the button(s) to the right of this textbox.

Just to get a feelng for it, a few examples to try:

Select a zoom magnitude from the Zoom pull-down. Click "Redraw". See how the image is zoomed in or out.

You can also limit the map by selecting latitude and longitude values. Select new lat/lon limits and click "Redraw".


Grids

You can think of grids as the ways in which the data are dependent.

Click the 'Back' button in the page and select the 'Data Selection' tab. The grid settigs are shown.

In this example, the data vary with latitude and longitude as well as with time. The information after each grid indicates the units and range of the grid. For example, the time grid shows the data taking interval and has data from Mar 26, 1998 through Oct 26, 2018 which creates 741 different time grid points. (Note: month is another common temporal grid unit.)

Grids vary from dataset to dataset, but there is a fundamental difference between those for gridded data and those for station data (i.e. data obtained at a single geographical point and pssibly multiple moments in time).

Select NOAA NCDC DAILY FSOD .PRES here.

This is a dataset having a station grid. You can see this in the grid description showing the WBAN (Weather Bureau Army Navy) numbers as an ordered grid. If you click on the 'Data Selection' tab, a particular station or range of stations can be selected.

For additional information on grids, see this page.


Select the "Documentation" tab.

Documentation

The documentaiton tab shows documentation on the dataset. This tab is not visible with every data source selected. In the case of station data for example like here, it will usually be visible.