Loxcel Geomatics Logo
Sign In
Sign Up
News
Help

Click here for updated DEM quality analysis (includes helpful interactive tools)

Monday, 2020-Dec-07 — DEM Quality — Can One Size Fit All?

Many research studies analyzed SRTM quality and published a wide range of RMSE values: 9.73m world-wide, 8.28m in Philippines, 3.53m at 13,305 reference points in US, 3.82m near Hispaniola shoreline, 15.27m at six US sites, and 10.3m at 335 IGS stations.

And, our study around Toronto Pearson Airport published an RMSE of 5.06m. We also analyzed NASADEM, the next generation of SRTM, which presented the airport's main runway as hilly, jagged terrain!

The variance of published RMSE results raise a few questions:

Tuesday, 2020-Dec-01 — DEM Visualization

The three scenes below visualize the Digital Elevation Models (DEMs) we studied last week. Each represents 57 km² (9.0 x 6.3 km) of area. Dark areas are low surface elevations; bright areas are high surface elevations. Tiny, bright rectangles are the tops of tall structures, usually buildings. Each scene includes a LIDAR image which depicts the perfect surface DEM. Use the toggle in the top-left corner of each scene to switch between DEMs and discover where each is less than perfect, due to remote sensing errors or surface changes over time.

These 3-D scenes (the 3rd dimension is surface elevation represented as brightness) gives us another perspective into DEM quality. All DEMs have identical 30 m grid spacings but present stark differences in spatial resolution, with some preserving building footprints and others appearing foggy.

Kingston (Canada)

Dark areas at the bottom and right are Lake Ontario at 72 m elevation (EGM96). Specular highlights in AW3D30 are building rooftops confirmed by LIDAR, but absent from Copernicus DEM 30.

Image of 2m LIDAR DSM from 2009 of Kingston, Ontario, Canada






_________________________ 2 km


Halifax (Canada)

The wide, dark diagonal down the middle is an ocean harbor. LIDAR shows many building rooftops in downtown (left of harbor) and two dozen oil tanks (right of harbor). AW3D30 preserves some of this detail; Copernicus preserves none. This could be because AW3D30 downsampled from a 5 m DEM while Copernicus downsampled from a 12 m DEM.

Image of 1m LIDAR DSM from 2019 of Halifax, Canada






_________________________ 2 km


Toronto Pearson Airport (Canada)

The airport's runways have mild slopes, which should appear free of noise. ASTER has an artifact down the middle, almost vertical; perhaps it's where two strips were merged. As before, AW3D30 preserves building rooftops better than Copernicus, but Copernicus shows much less noise around the runways. SRTM / NASADEM has severe noise, far beyond what we've seen in other areas; both are unusable.

Image of 1m LIDAR DSM from 2018 over Toronto Pearson Airport






_________________________ 2 km

This profile compares DEM elevations to LIDAR (in black) along runway 05/23:

Elevation profile of Toronto Pearson runway 05-23, red=NASADEM, green=Cop30, blue=AW3D30, black=0.5m LIDAR

The graphs below plot [ DEM elevation - LIDAR elevation, in meters ] in the 81% of the 57 km² area classified as Urban / built-up by NALCMS 2015 v2. (Our other study compares DEM elevation to ICESat-2 ATL08 terrain elevation.) The right graph pulls SRTM / NASADEM from the left graph to highlight how NASADEM varies from SRTM.

Urban / built-up areas
Graph comparing 0.5m LIDAR terrain heights in urban areas with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER7,358,74112.26-10.676.034.2%68.3%
AW3D307,358,7412.930.842.8064.1%1.7%
COP307,358,7413.21-0.153.2172.1%1.9%
NASADEM7,358,7415.03-1.474.8144.2%8.5%
SRTM7,358,7415.06-0.804.9941.7%8.3%
SRTM / NASADEM shift
Graph comparing 0.5m LIDAR terrain heights in urban areas with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM

The area around Toronto Pearson Airport is relatively flat and vegetation-free, and should present few challenges to remote sensing equipment. Let us know if you have any ideas about what happened with SRTM / NASADEM.

Citations:

Thursday, 2020-Nov-26 — The Earth isn't Flat

Bicubic fits the earth's curved surfaces better than bilinear interpolation with DEM grids of 30 m or greater. To demonstrate this, we revisit our previous study, this time with bilinear interpolation. This study looks at short vegetation and barren surfaces, which reveal the earth's surface to remote sensing equipment better than other surface types. The results are below:

LabelDEMGrid SpacingInterpolation
CUBIC30COPernicus DEM30 mBicubic
LINEAR30COPernicus DEM30 mBilinear
CUBIC90COPernicus DEM90 mBicubic
LINEAR90COPernicus DEM90 mBilinear

Short Vegetation (Flat terrain)
Graph comparing ICESat-2 terrain heights in flat terrain in short vegetation with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM
LabelComparisonsRMSEMeanStDev< 2 m> 8 m
CUBIC3045,115,6401.200.121.2097.0%0.4%
LINEAR3045,122,0561.600.141.6097.0%0.4%
CUBIC9045,115,6401.260.151.2595.8%0.3%
LINEAR9045,121,4521.970.191.9695.1%0.4%
Terrain Slope



Barren (Flat terrain)
Graph comparing ICESat-2 terrain heights in flat barren terrain with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM
LabelComparisonsRMSEMeanStDev< 2 m> 8 m
CUBIC306,111,1491.22-0.121.2198.5%0.3%
LINEAR306,113,6832.37-0.072.3798.4%0.3%
CUBIC906,111,1491.30-0.121.2997.6%0.3%
LINEAR906,113,6482.64-0.082.6396.9%0.4%
Terrain Slope




Bilinear has only one advantage over bicubic: speed of computation. Our benchmarks found bilinear to be as fast to four times faster than bicubic interpolation, depending on the percentage of CPU cache misses incurred.

But you might not see these benefits. DEM tiles are very large rasters each covering 1 x 1 degree, with the most north-west elevation at the lowest memory address. If these tiles are accessed randomly, the CPU will spend more time waiting for memory than performing computations. If query throughput is your priority, before replacing bicubic with bilinear interpolation, look at how your program accesses DEM tiles. It should query all locations within one tile before querying the next tile. And within a tile, it should query north-west locations first, then move east and finally south.

Copernicus DEM 30 and 90 are accurate global DEMs. Bilinear interpolation will degrade their accuracy, especially Copernicus DEM 90. If query throughput is a priority, try to reduce CPU cache misses before replacing bicubic with bilinear interpolation.

Wednesday, 2020-Nov-25 — Digital Elevation Model Comparison

Digital Elevation Models (DEM) have many uses, from land and forest management, wireless propagation studies and satellite image orthorectification. Flight simulators use one-meter DEMs to deliver state-of-the-art visuals — for only $59.99!

Fortunately we have many DEMs to choose from, including these six which are the subjects of our comparison:


Each DEM has its own strengths, and choosing the the right one depends on evaluating metrics such as

MetricDetails
Vertical accuracy (absolute)DEM elevation compared to geodetic vertical control points, located across a range of terrain slopes and landcover types. A DEM might perform well in flat, uncluttered spaces but not on forested hillsides
Vertical accuracy (relative)Elevation difference between adjacent DEM cells. Used to derive slope (rise over run) and aspect (slope azimuth)
Horizontal uncertaintyIncreases error of slope, aspect and other attributes
Horizontal skewA shift in some or all of grid. Can prevent co-registration with landcover or other spatial databases
Grid spacingA 90 m DEM captures fewer terrain details than a 30 m DEM and can compromise terrain modelling, eg. decrease slope gradient
Storage / bandwidthA 90 m DEM uses 1/9th the storage or bandwidth of a 30 m DEM
Terrain and / or SurfaceUse a Digital Terrain Model (DTM) to analyze water flow and a Digital Surface Model (DSM) to analyze wireless signal propagation. Combine both to measure tree canopy height or forest volume
Coverage areaGlobal or local? 1, 2 or 5 m DEMs might be available only for regional areas of interest
AgeSome DEMs (not listed above) are 30+ years old
VoidsPeaks can obscure mountain side creating data voids. How many? How were they handled?
SecurityWas DEM corrupted during download, or otherwise tampered with? Are checksums available?
AccessibilityHow easy is it to download DEM? Ftp, batch wget or cumbersome website?
LicenseWhat use cases are allowed or prohibited? Can you make derivative works?
CostPrice can vary from free to thousands of dollars for a regional area of interest
MetadataIs ancillary data provided, eg. waterbody mask?
WaterbodyAre lakes flattened to one elevation, river elevations adjusted by a constant interval and shorelines elevated above adjacent waterbody?
UpdatesAll DEMs have errors. How do you report them? How often are updates published?
FormatA binary file with 1 arcsecond grid spacing (eg. SRTM) is easy to use via your own bespoke software, but a TIF with a more efficient projection might be easier to use with packaged software
Coordinates / datumDoes DEM use an obscure, local projection or vertical datum?

Our study addresses the first metric above, Vertical accuracy (absolute). We loosely followed the approach of this study which used ICESat-2 ATL08 segments as vertical control points, but with a few changes:

Our study compares 280 million ATL08 segments captured from 2018-Oct-14 to 2020-Sep-30 across North America to the six DEMs above. NALCMS 2015 v2 provided landcover type:

# ComparisonsPercentageNALCMS Landcover Type
174,238,42762.2%Short Vegetation
48,308,78017.3%Needleleaf Forest
28,604,99110.2%Barren
12,895,1844.6%Broadleaf Forest
9,683,6313.5%Mixed Forest
6,208,4902.2%Urban
279,939,503100.0%ALL
# ComparisonsPercentageATL08 terrain_slope
87,319,33731.2%Mild (0.005 - 0.02)
71,503,95725.6%Flat (< 0.005)
61,059,04421.8%Moderate (0.02 - 0.05)
39,805,80814.2%Rugged (0.05 - 0.125)
20,251,3577.2%Very Rugged (> 0.125)
279,939,503100.0%ALL

An ICESat-2 ATL08 segment reports one latitude, one longitude, one terrain and one canopy elevation from where 100s of photons struck a track 14 m wide and 100 m long, at ground level, up in a pervious vegetative canopy — or infrequently way up in the clouds! This process is at the forefront of remote sensing technology and offers an unbelievable volume of data with an unbelievable level of precision. But, what about errors and their distribution? Are they random or have systemic properties? If the 100 m track is flat terrain, we have confidence in the values reported. For rough terrain, our confidence drops. For this reason, we caution you when interpreting any results below where terrain_slope > 0.02 (ie. > 2 m / 100 m).

RMSE, Mean, StDev and x-axis are based on [ DEM elevation - ICESat-2 terrain elevation, in meters ]. Graphs omit ASTER as its curves ride near the x-axis (it's not that ASTER is bad; it's that Copernicus DEM is very good!) Toggle the Terrain Slope buttons to see results at various slopes (for uncluttered landcover types only).

Short Vegetation (Flat terrain)
Graph comparing ICESat-2 terrain heights in flat terrain in short vegetation with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER45,115,6239.44-3.918.5920.6%32.1%
AW3D3045,115,5962.681.092.4569.6%0.9%
COP3045,115,6401.200.121.2097.0%0.4%
COP9045,115,6401.260.151.2595.8%0.3%
NASADEM29,583,5482.38-0.232.3775.9%1.2%
SRTMV328,703,7992.820.152.8157.3%1.2%
Terrain Slope





Barren (Flat terrain)
Graph comparing ICESat-2 terrain heights in flat barren terrain with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER6,111,14712.13-6.929.9613.3%49.5%
AW3D306,111,1433.341.073.1672.1%1.3%
COP306,111,1491.22-0.121.2198.5%0.3%
COP906,111,1491.30-0.121.2997.6%0.3%
NASADEM1,183,2424.72-0.084.7265.5%3.3%
SRTMV31,144,2674.300.554.2752.9%3.1%
Terrain Slope






Needleleaf Forest (trees > 5 m tall)
Graph comparing ICESat-2 terrain heights in needleleaf forest with ASTER, AW3D30, Copernicus DEM, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER13,035,2909.05-3.448.3817.7%35.4%
AW3D3013,035,2903.611.843.1059.2%3.8%
COP3013,035,2903.261.502.9074.3%3.9%
COP9013,035,2903.231.562.8272.2%3.6%
NASADEM9,574,4303.160.133.1673.3%3.1%
SRTMV38,724,6113.140.063.1459.1%2.5%
Terrain Slope
< 0.5 m / 100 m  
Broadleaf Forest (trees > 5 m tall)
Graph comparing ICESat-2 terrain heights in broadleaf forest with ASTER, AW3D30, Copernicus, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER3,181,0849.040.809.0018.4%34.6%
AW3D303,181,0846.183.874.8235.4%15.9%
COP303,181,0846.123.994.6442.9%16.2%
COP903,181,0845.994.084.3838.3%15.5%
NASADEM2,871,9744.252.003.7549.1%6.7%
SRTMV32,826,9305.062.644.3240.6%11.0%
Terrain Slope
< 0.5 m / 100 m  

Mixed Forest (trees > 5 m tall)
Graph comparing ICESat-2 terrain heights in mixed forest with ASTER, AW3D30, Copernicus, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER2,170,6359.24-0.019.2418.3%35.2%
AW3D302,170,6355.953.834.5637.1%15.2%
COP302,170,6356.164.024.6742.9%17.1%
COP902,170,6356.054.124.4438.9%16.7%
NASADEM1,726,0604.001.713.6253.2%5.3%
SRTMV31,682,9474.371.993.8947.9%7.3%
Terrain Slope
< 0.5 m / 100 m  
Urban / built-up areas
Graph comparing ICESat-2 heights in urban areas with ASTER, AW3D30, Copernicus, NASADEM and SRTM
DEMComparisonsRMSEMeanStDev< 2 m> 8 m
ASTER1,890,1598.24-0.798.2023.9%24.2%
AW3D301,890,1593.611.913.0750.2%2.6%
COP301,890,1592.861.262.5776.4%2.5%
COP901,890,1592.861.382.5073.3%2.3%
NASADEM1,877,6953.140.683.0762.6%2.1%
SRTMV31,875,5764.071.663.7245.1%4.7%
Terrain Slope
< 0.5 m / 100 m  

All DEMs show significant positive elevation bias in broadleaf and mixed forests, measuring well above the forest floor. However, we use DEMs for wireless propagation studies, where hills and trees block wireless signals. So we welcome this positive bias, as long as it does not extend above the forest canopy. For you, this bias might be a concern — it all depends on your use case.

Methodology:

Geoid DEM elevations were converted to WGS84 ellipsoid elevations before being compared to ICESat-2 elevations. Gridded data (ie. DEM and vertical datum conversion) used 16-value bicubic interpolation. SRTM / NASADEM had fewer comparisons because their coverage does not extend north of 60N and 61N latitude, respectively.

Please contact us if you you have any questions about our study.

Citations: