Chapter V: Data acquisition, data assimilation and processing
Raw satellite data needs to be processed before it can be used by the forecaster. The information displayed on the screen in forecasting offices is often a combination of different data sources, at least it is modified in a way to best show the relevant data content.
The webcasts and training modules below show the process of data acquisition and data processing for various types of satellite data. This section also contains a tutorial on how to implement the NowCasting SAF software on your local machine, RGB fine-tuning, parallax correction and an introduction to the AAPP tool from ECMWF. Forecast verifications and the use of vertical cross section are also treated here.
Soil Moisture data assimilation for flood prediction (Webcast, 37+27 minutes), 2019
The reliable estimation of hydrological variables in space and time is of fundamental importance in hydrology to improve the flood predictions and hydrological cycle description. Nowadays remotely sensed data can offer a chance to improve hydrological models especially in environments with scarce ground-based data. The presentation describes how satellite soil moisture can be used in hydrological modelling trough data assimilation.
Analysis of Soil Moisture time series and spatial patterns (Webcasat, 38+39 minutes), 2019
Soil Moisture is a crucial variable in hydrological applications. It can be measured and estimates in different way and along different spatial and temporal scale. The presentation describes how soil moisture estimated trough satellite can be compared and validated with other sources of information; theoretical basis and practical example will be showed.
How to work with RZSM products: From download to visualization (Webcast, 19 minutes)
H-SAF root-zone soil moisture products are freely available online. They are currently stored in daily files on a reduced Gaussian grid in GRIB binary format, which increases storage efficiency. Examples of downloading, converting the format to regular netCDF files and visualizing the data in Python are presented. The new metview-python package is also introduced, allowing direct applications in GRIB format.
ECMWF provides the core root-zone soil moisture (SM) products for H SAF through an Extended Kalman filter assimilation system, running independently of the NWP system. Space borne scatterometer-derived surface SM observations are assimilated into the root-zone (0-1 m) SM of the H-TESSEL land surface model. In this presentation, the theory behind the near-real-time and data record products is discussed.
Towards the Standardization of European Weatherand Impact Observations (Webcast, 27 minutes), 2019
Since the release of the WMO recommendations for impact based forecasts and warnings, reliable data about weather related effects on society and public life became increasingly important for operational forecasting. Particularly densely populated urban areas are vulnerable to the forces of convective weather hazards. In this regard, operational automatic station networks or remote sensing data cannot provide complete information about the ground truth like hail size, downburst related wind damage, flash floods or damage from lightning strikes. In order to fill this gap, targeted human assessment and observations are still needed. With the help of weather- and impact reports a real-time feedback loop between forecasters and voluntary observers can be established, to improve impact based warnings and thus to increase weather related disaster resilience and mitigation. In our presentation, we introduce the Austrian weather- and impact observation system "wettermelden.at" with its applications and furthermore a concept for the swift exchange of these reports on national and European levels.
ECMWF EFI and SOT for forecasting severe weather (Webcast, 50 minutes), 2018
ECMWF provides the Extreme Forecast Index (EFI) and Shift Of Tails (SOT) as tools to help forecasting anomalous and extreme weather by measuring the difference between the Cumulative Distribution Function (CDF) of the real-time ensemble forecast and the model climate (M-climate) CDF. The definition of the EFI and SOT will be presented alongside the M-climate configuration. Verification results and EFI/SOT products will be shown with examples to demonstrate their use and interpretation. Some advantages and limitations of the EFI and SOT will be discussed. Examples focused on the available EFI/SOT for marine forecasting will be given as well. Two recent EFI/SOT products for forecasting severe thunderstorms and their interpretation and practical use will be demonstrated.
EFI and SOT Products (Webcast, 39 minutes), 2017
The Extreme Forecasting Index (EFI) and the Shift Of Tails (SOT) index are two operations products developed by ECMWF for usage in forecasting severe weather. The EFI is based on ECMWF ensemble forecasts and it compares these forecasts with the model climate (M-climate) that is generated by the model re-runs. While the high EFI tells us that the confidence level of a forecast is higher for a certain event, the positive high SOT value tells us that the event would be more extreme than the one with low SOT value. At the end Ivan shows us a few cases and demonstrate how EFI and SOT work together.
Forecast verification (Training Module, 120 minutes), 2017
This module is designed both for users of verification results, who wish to understand what the results really mean, and those who wish to dabble in verification methodology themselves.
Online engineering workshop for the NWCSAF GEO v2016 (Webcast, 180 minutes), 2016
The NWC SAF has released a new SW package GEO v2016 in November 2016. An online workshop was organized on 11 January 2017 between 9:00 and 14:00 UTC in order to introduce to the users the NWCSAF GEO v2016 from a technical point of view. The NWC SAF team explains how to install this new version, the organization of directories, auxiliary datasets, new output format...
Tuning of RGB products and new RGB products from FCI (Webcast, 30 minutes), 2016
The amount of data from the world's weather satellites is overwhelming. While each type of data is valuable, it's almost impossible to use them all operationally. It's like trying to drink from a fire hose; there's simply too much data to absorb, and much of it ends up not being used.
"Red, Blue, Green" or RGB processing is a simple but powerful technique that consolidates different channels of satellite imagery into single products that are easy for forecasters to use. RGB processing used to be a visualization technique used mainly in research. But due to its popularity, it is increasingly available to operational forecasters. A pre-requisite for this, however, is the standardization of RGB products, i.e. the selection of the most useful RGB products for operational forecasting, generated at each Meteorological Service with the same identical standard method/recipe.
The combination of individual images into RGB colour composites is modernizing the interpretation of satellite imagery. While black and white imagery still has its uses, it often cannot match the effectiveness of RGB products. In fact, RGB images are often more useful than traditional colour image enhancements.
The Flexible Combined Imager (FCI) of MTG will open new possibilities of RGB products, with higher temporal and spatial resolution and better accuracy (less noise). Of particular interest will be the new NIR2.25 band that will improve cloud phase detection and detection of hot/large fires. Two new RGBs related to this channel will be presented. Furthermore, some standard RGBs will need to be tuned to account for the slight changes in central wavelength and band width. Some Himawari examples for this tuning will be given. Finally, the issue of local versions of RGB products will be addressed using the examples of "tropical" Night Microphysics and Airmass RGBs.
The EUMETSAT SAF to support Nowcasting (NWCSAF) develops two software packages, one for geostationary imagery and one for polar satellite imagery. Both packages retrieve cloud and other parameters relevant for nowcasting and other applications relying on cloud detection. For more information see www.nwcsaf.org
The Polar Platform System (PPS) software package retrieves information on clouds and precipitation from NOAA satellites, MetOp and S-NPP. The recent release of PPS v2014 features also a number of technical updates affecting installation of PPS and interfacing to your environment and applications.
The workshop is addressed to users of PPS wanting to update their application, but also to prospective new users.
The NWCSAF kindly invites you to participate to a two-hour online training workshop on the installation, use and operation of the new PPS v2014 software. We plan to have approximately four half-hour slots around the following subjects:
- Installation
- New output format
- Operating PPS via the main script "RunAllParallel.py"
- Setup PPS in a real-time environment (no Task Manager in v2014)
Satellite products (Webcast, 30 minutes), 2014
The course gives a short introduction to some established methods to derive meteorological products from satellite data, including the benefits and downsides of products. Product examples will mainly focus on the MSG products, derived centrally at EUMETSAT and within the NWC SAF project.
AAPP and other processing tools (Webcast, 30 minutes), 2012
The ATOVS and AVHRR Pre-processing Package (AAPP) is a tool to process sounder and imager data from the NOAA, Metop and FengYun polar orbiting satellites. It was originally developed in the late 1990s to support processing of direct broadcast data from the NOAA-POES satellites and has since been extended to accommodate Metop (including IASI) and more recently Suomi-NPP. The package is maintained by the EUMETSAT Satellite Application Facility for Numerical Weather Prediction (NWP SAF). An overview will be given of the capabilities of AAPP, including how it complements other processing packages such as IMAPP and CSPP. AAPP products include calibrated, re-mapped radiances (for NWP) and derived products such as cloud mask. Examples of the products and their uses will be shown.
Satellite data assimilation (Webcast, 35 minutes), 2012
This lecture introduces the central role played by data assimilation in Numerical Weather Prediction, Climate Reanalysis and Environmental monitoring. It will be shown that radiance observations from polar orbiting satellites are the single most influential component of the global observing network and the impact of these data on forecast quality will be demonstrated. Finally, the major scientific challenges facing the successful exploitation of satellite radiance observations will be discussed - in particular issues related to vertical resolution, cloud and precipitation contamination and systematic errors.
CM SAF: Introduction to Software Tools (Webcast, 30 minutes), 2012
This presentation is an introduction on the software tools cdo and R that are well suited to handle CM SAF data. Both tools are freeware and can be installed under Windows and Unix environments. Scripts for cdo and R for basic operations on CM SAF products are provided and will be explained in this presentation. Time to install cdo and R and to order CM SAF data via the web user interface (WUI).
Meteosat IR Enhancements (Webcast, 30 minutes), 2012
This short, but interesting and to all satellite community very useful lecture is brought by Martin Setvak.
Image enhancement is a process of image modification or improvement of its quality, the aim of which is to achieve a more pleasing appearance of the final image. However, in science, the main goal for image enhancement is to increase the interpretability of the image to a human eye and brain, typically focusing on a certain feature carried by the image.
Example of this kind of enhancement you can see in everyday use of digital cameras, which have built-in image enhancement (software based). So the 'raw' images captured by the EM sensor can be further improved by simple computer post-processing (in other words - enhancement).
Processes of enhancement can be done with various methods, such as; histogram-based methods, curves adjustment, gamma function adjustment, noise reduction, utilization of advanced filters, etc. Also a few nice examples you can see here and taste the abilities of this kind of image improvement.
LSA-SAF: Messing around with data (Webcast, 60 minutes), 2011
This one-hour presentation takes focus on the characteristics of the product files that are produced and distributed by LSA SAF. The structure of the files is explained and freeware tools are presented. All the relevant information is given for the obtaining LSA SAF products. More general product information is given for documentation and file structure of data. Also, this session is explaining how to use Quick visualization by means of HDFView, GIS tools for visualization and analysis (Quantum GIS, GRASS GIS and ILWIS) and tool for georeferencing LSA SAF products. At the end of presentation there are words about how to use Python scripting for building custom scripts or programs for accessing and manipulating Land SAF data.
Assimilation of Scatterometer Winds (Webcast, 30 minutes), 2011
Over the last years the processing of ERS scatterometer winds has been refined. Subsequently, High Resolution Limited Area Model, HIRLAM, and ECMWF model data assimilation experiments have been carried out to assess the impact of one scatterometer, ERS-1 and of two scatterometers, ERS-1 and ERS-2, on the analyses and forecasts. We found that scatterometer winds have a clear and beneficial impact in the data assimilation cycle and on the forecasts. Furthermore, ECMWF has shown that ERS scatterometer data improve the prediction of tropical cyclones in 4Dvar, where unprecedented skillful medium-range forecasts result of potential large social-economic value. Nevertheless, scatterometer winds contain much sub-synoptic scale information where the smallest scales resolved are difficult to assimilate into a Numerical Weather Prediction, NWP, model. This is mainly due to the otherwise general sparsity of the observing system over the ocean. In line with this it is found that scatterometer data coverage is very important for obtaining a large impact. In that respect future scatterometer systems such as SeaWinds on QuikSCAT and ADEOSII, and ASCAT on EPS are promising.
Data acquisition, processing and applications (Webcast, 50 minutes), 2011
For the correct assessment of satellite images, the processes involved in data processing should be known very well. Data processing and applied algorithms have essential impact on the satellite images, such as for example gamma correction. This lecture leads from single counts to radiances and brightness temperatures. Finally, a well-chosen selection of applications will be shown.
Parallax Shift (Webcast, 16 minutes), 2011
Although may considered as the answer to all problems, satellite also come with some limitations. In this lecture one such limitation, Parallax Shift is explained, and how you must take this into account. Fortunately, there are some solutions available to overcome this problem, which are also explained by Martin Setvak (CHMI).
Nefodina (Webcast, 30 minutes), 2011
Davide Melfi (CNMCA) presents the development on the NEFODINA product. This is an effective model designed for the detection and the forecasting of the convective systems evolution in the Mediterranean area. It is composed by a system of neural networks and a varying temperature threshold method. It is able to detect not only the convective clusters but also all the convective cells inside them. He explained the product and gave some examples of how the products are used.
Vorticity patterns control the circulation of air masses in their vicinity. By doing this they control the location of important meteorological quantities that are essential for an accurate diagnosis and forecast of the atmosphere.
The scale of vorticity patterns in the atmosphere ranges from large-scale synoptic system circulations (low and high pressure centres) to smaller meso-scale circulations (water vapour vortices (sometimes referred as WV eddies or WV eyes). The (anti-)cyclonic rotation in the atmosphere caused by a vorticity maximum is easily seen in satellite imagery. And quite naturally, satellite imagery is the key tool to correctly locate the maximum of cyclonic and anticyclonic vorticity in the atmosphere. Moreover, satellite images are able to show the small-scale vorticity patterns that are easily overlooked and smoothed out by a NWP model.
This training module has been developed to teach you to identify these vorticity centres in Meteosat Second Generation (MSG) satellite imagery. In addition, the module will provide you with a firm physical background to help you understand why it is important to do a good diagnosis of satellite images and also provide you with a range of examples and exercises to demonstrate the impact a vorticity center may have on your weather forecast.
NWP Model Monitoring (Training Module, 60 minutes), 2008
Numerical Weather Prediction (NWP) is a fundamental part of the modern forecasting. The importance of NWP in the forecasting process is so obvious that we often forget the restrictions and limitations of NWP. Models do have errors, even a lot of them. Even with ever improving performance the NWP models still have many problems in predicting many weather elements precisely.
It is of extreme importance that the forecaster can be able to evaluate the performance of NWP in a daily operational environment, and not be misled by the NWP output.
This module will concentrate on one aspect in model error correction: the subjective monitoring the NWP output with the help of satellite images. The purpose of the module is to help forecaster to use selected satellite products for a quick analysis of how the NWP model is able to catch the features in the weather chart.
The target users for this module are operational forecasters and developers. The pre-requisites for completing this module are general knowledge of meteorology, some experience in forecasting and a basic knowledge of the satellite images and image techniques.
Nowcasting CAL (Training Module, 180 minutes), 2007
In this module the basics of Nowcasting in the forecast room are explained. Its practical use in the weather room is explained with a series of examples.
The second part deals with explanation and testing of Nowcasting for various situations such as convective events, frontal situations or Fog. By adding more source-material a student is taught how to improve his Nowcasting skills.
Vertical Cross Sections (Training Module, 60 minutes), 2007
Not immediately related to satellite meteorology but often presented in the EUMeTrain material are cross sections. In meteorology vertical cross-sections are used because they give additional information compared to the information gained from the analysis on horizontal levels. The basics on cross sections and what can be derived are discussed and tested in the CAL module.