TL;DR

SRTM is good, but Copernicus DEM is usually better. For terrain applications, FABDEM might be even better.

Intro

We use Digital Elevation (DEM) and Land Cover Models to identify obstructions to wireless communications over long distances. DEM quality is important to our business and the success of our customers. SRTM has been the go-to DEM for most industry applications. SRTM has served us well. Recently, competitors have appeared. Should we replace SRTM with a competitor?

But, how do we know if a competitor is better than SRTM? These are actual DEM quality claims from marketing and scientific material:

• < 4m (90% linear error),
• Mean Absolute Error is 0.49m less (urban) and 2.27m (forests),
• RMSE is 2.44m for NED and 3.53m for SRTM, and
• RMSE of SRTM is 6.61m in floodplains.

These statistics are as relevant to DEM quality as player height is to the quality of a basketball team. Height is important, but not as important as a player's many quirks & features. DEM quality involves many quirks & features that elude the statistics above. This article introduces DEM Explorer, a visualization tool that lets you explore DEM quality and discover its quirks & features.

DEM Explorer graphs the behavior of the 14 DEMs below, across 3 land covers and 4 slopes. Each curve plots the distribution of error [ elev_DEM - elev_LiDAR ] between a DEM and highly accurate LiDAR ground truths. Pan, zoom, pinch & swipe the graph to discover more quirks & features.

DEM Contenders

Try for Yourself

Click any graph on the right to launch DEM Explorer and see

• why DEM quality cannot be captured by a few statistics,
• where SRTM is superior to Copernicus DEM 30 (COP30),
• why NASADEM is a good replacement for SRTM,
• where COP30 is an asset and where it is a liability,
• where FABDEM amazingly reduces COP30 surface bias, but at the cost of exaggerating flooding risks,
• how rounding COP30 pixel values to the nearest integer (useful on resource constrained devices) affects quality (or not),
• where ICESat-2 could make an excellent global DEM, if more ATL08 segments can be captured,
• how much MERIT reduces SRTM forest bias,
• that SRTM has less forest bias than COP30,
• when COP90 is a good substitute for COP30, and
• why ASTER should be put out to pasture.

Methodology

We compare each DEM listed above to billions of LiDAR ground truths with a point density > 6 / m². Each colored curve on the graph is a distribution of error, created by comparing one DEM to all LiDAR ground truths with the same land cover and slope; eg. forest with moderate slope. Each curve stresses the DEM in a different way, teasing out biases. A smooth curve requires at least one million LiDAR ground truths; most curves use many more (billions in some cases) producing the smooth curves you see on the right.

These LiDAR ground truths have a vertical accuracy better than the height of a chipmunk (5 to 10 cm). We quote RMSE, MAE and other statistics to 0.1m precision; a higher precision captures only terrain noise, like chipmunks, acorns and other ephemeral clutter.

ESA WorldCover 10m 2021 V200 identifies a land cover for each LiDAR elevation: Grass / Crop, Forest or Developed. (We combine Grassland and Cropland, as they present remote sensing with a similar, short and easily permeable surface.)

Slope is calculated from a high-resolution 0.5m elevation grid created from the LiDAR ground truths, providing the most accurate ground slope possible:

Elevation Normalization (Geoid to Ellipsoid)

LiDAR and DEM surveys capture ellipsoidal elevation which are later converted to geoidal (eg. EGM96, EGM2008, NAVD88, CGG2013) for public use. Our analysis require all LiDAR and DEM elevations normalized to the WGS84 ellipsoid. Normalization uses a geoid grid and an interpolation method; each grid size & interpolation combination produces different results. Normalization error occurs if the combination we use differs from what was used when the elevation data was packaged for public use. This error can vary from centimeters to meters.

A geoid's continuous surface is defined by spherical harmonic coefficients. These coefficients are too computationally expensive to work with directly, so they are digitized once into a grid of pixels, which approximate the geoid's surface, and interpolated, on demand, to obtain geoid offsets.

Our normalization (from geoid to WGS84) should use the same grid size and interpolation used when the DEM was created. However, only NASADEM publishes these details (ie. 15 arcsecond grid with linear interpolation). We applied various geoid grid sizes and interpolation methods to COP90 to discover how it was derived from TanDEM-X 90m (ie. 60 arcsecond grid with linear interpolation); we assume the same for COP30 but cannot confirm because TanDEM-X does not publish a 1 arcsecond spacing DEM. We used a 60 arcsecond grid and spline interpolation for other DEMs, as that is the interpolation method used by the US National Geospatial Intelligence Agency (masters of the geoid) in their calculations.

These details are important wherever the geoid undulates strongly, such as Hawaii, a place we are currently studying.

Conclusion

DEM quality is not a constant and depends on use-case, nature-of-bias, budget, license terms, file size and coverage area.

DEM Explorer can help you understand a DEM's nature-of-bias from comparisions with 627 billion LiDAR ground truths in Southern Ontario and 12.3 billion more in Newfoundland, Canada.

Marketing and scientific literature often use RMSE (root mean square error) as a proxy for DEM quality. RMSE must be used with caution, because a few bad apples can spoil the results. To that end, DEM Explorer provides >10m and >20m threshold statistics, measuring the percentage of bad apples (ie. percentage of error above 10m and 20m) which sends RMSE soaring. DEM Explorer also provides MAE (mean absolute error) which is less sensitive to extreme outliers. But, RMSE or MAE — alone — are as much a sign of DEM quality as player height is to basketball team quality.

What's the answer to replacing SRTM? Switching DEMs is not a simple exercise. A new DEM brings its own quirks & features that will improve some things and worsen others. Will the mix of quirks & features net a positive outcome?

Copernicus DEM provides much more accurate surface elevations, useful for our work in wireless propagation analysis. FABDEM is a derivative of Copernicus DEM that reduces this surface bias, which you need for flood analysis. FABDEM performs this task well, at a cost of some negative bias. As well, FABDEM has restrictive terms of license which a commercial application must consider.

We use DEMs for wireless propagation analysis, which favors a DEM that captures all surface clutter (forests, shrubs, but not chipmunks or acorns). Other use-cases, like floodplain analysis, need a no-clutter DEM. These and other quirks & features are what DEM Explore can help you discover, on your search for a better DEM.

These graphs compare DEMs to LiDAR ground elevations.
Curves depict distribution of error
[ elev_DEM - elev_LiDAR ]
> Click any graph to explore further <

Figure 1: One RMSE value cannot capture DEM quality

Color	DEM	Land cover	Slope	RMSE
Yellow	SRTM	Grass/crop	Level	1.7
Red	SRTM	Developed	Moderate	3.0
Green	SRTM	Forest	Moderate	5.4

This range of RMSE values — for one DEM — shows the folly of representing DEM quality with a single RMSE value. Yet scientific and marketing literature does this.

Figure 2: Copernicus DEM quality eclipses NASADEM in low-clutter terrain

Color	DEM	RMSE	MAE	Mean	Stdev
Yellow	COP30	0.9	0.4	0.2	0.9
Green	NASADEM	1.7	1.3	-0.5	1.6

NASADEM's shallow curve isn't a sign of how bad it is (NASADEM is a good DEM). Instead, it's a sign of how good Copernicus DEM is in certain situations.

Figure 3: Copernicus DEM can't see the forest floor

Color	DEM	RMSE	MAE	Mean	Stdev
Yellow	COP30	8.8	7.3	7.2	5.0
Green	FABDEM	3.8	2.8	0.9	3.7

COP30's yellow tail to the right of the y-axis is forest clutter; a benefit to radio propagation analysis but a detriment to floodplain analysis. FABDEM's green curve reduces clutter bias, at a cost of bias left of the y-axis.

Figure 4: Copernicus DEM does better in Newfoundland, Canada.

	COP30		FABDEM
Location	RMSE	MAE	RMSE	MAE
Ontario	8.8	7.3	3.8	2.8
Newfoundland	2.7	2.2	2.4	1.7

Figures 3 & 4 show different RMSE & MAE values for the same DEM in forests with moderate slope. The only difference is place, expanding on observations in Figure 1.

Figure 5: Effects of rounding DEM pixels (no effect)

Pixel Type	RMSE	MAE
Float32	7.3	5.7
1m	7.3	5.7

Rounding Copernicus DEM pixels from float to integer significantly improves compression ratios, benefiting resource constrained devices. Quality is not compromised when rounding pixels in forests with level terrain.

Figure 6: Effects of rounding DEM pixels (some effect)

Pixel Type	RMSE	MAE
Float	0.9	0.4
Integer	1.0	0.5

The effects of rounding are slightly worse in Grass / Crop surfaces with level terrain.

Figure 7: ICESat-2, as a sparse-DEM, in forested, moderate slope areas

Color	ICESat-2 h_te_uncertainty	RMSE	MAE	Mean	Stdev
Red	< 2	1.17	0.65	-0.02	1.16
Green	< 10	1.73	0.76	-0.14	1.73

ICESat-2 ATL08 segments are used in other studies as ground truths to assess DEM quality. We turned the tables, creating two sparse-DEMs from 68,600 km² of ATL08 segments and compared them to our higher accuracy LiDAR. DEM Explorer shows this sparse-DEM to be more far accurate than any global DEM.

Figure 8: COP90 (green) and TanDEM-X (yellow) share DNA

DEM	RMSE	MAE
TDX90	1.6	0.9
COP90	1.2	0.9

COP30 and COP90 are derived from TanDEM-X. COP90 and TDX90 are often identical. This graph shows divergence for negative bias, possibly due to manual editing of COP90.

The graphs below plot the most recent 18 months of ISED SMS snapshots, by channel count (top), occupied spectrum (middle) and site count (bottom), for the three national (left) and four regional (right) carriers.

Rogers' graphs (red) naturally trend upwards, capturing the growth of its network. Telus' (green) and Bell's (blue) graphs see-saw up and down, hilighting the inconsistencies of these snapshots.

We fix these snapshots to ensure Canada Cellular Services provides the most accurate account of Canada's wireless networks available.