Among many improvements, version 3.32 introduces the long-awaited feature of native point cloud processing in QGIS, enabling QGIS Desktop to become a powerful LiDAR data processing utility.
Before publishing a tutorial on the new features, I’ve gathered together the somewhat scattered information on the architecture and choices that have been made by the developers. It’s always more interesting to wait in front of a progress bar when you know what’s coming…
Classic vector points and point clouds
Recent advances in LiDAR technology and photogrammetry have made it increasingly necessary to obtain and store point cloud data. Although point cloud data is vector-based, its size is generally much larger than that of standard vector layers. Whereas typical vector datasets are made up of thousands or millions of entities, point clouds can contain millions, billions or even trillions of points. Consequently, a distinct approach is required in a GIS platform to effectively visualize, analyze and store point cloud data, given the sheer volume of points involved.
In the world of open source GIS, there are some fantastic tools for working with point clouds. PDAL (“Point Data Abstraction Library”) has become an important tool, supporting a wide range of input data formats and also offering numerous algorithms for point cloud processing. For web visualization, the Potree JavaScript library is the most common choice. To visualize point clouds in a desktop application, users usually open CloudCompare, which comes with a variety of built-in tools.
The integration of a point cloud viewer into a GIS brings considerable added value to users, compared with a specialized, dedicated point cloud viewer:
- Point cloud data can be viewed, compared and analyzed with other types of spatial data (including vector, raster and grid layers).
- Familiar user interface and workflow
- Integration with analytical tools to rapidly create derived datasets
The main difficulty in using point cloud data stems from the sheer number of points contained in the datasets: we have to process millions or even billions of points for relatively small geographical areas, while country-wide studies can involve trillions of points. With this amount of data, it is not possible to simply convert the dataset into “ordinary” vector layers – typical GIS tools and data formats such as GeoPackage or Shapefile are not optimized for such quantities of data, and any operation would be very time-consuming.
The new point cloud layers
Since version 3.26, a new map layer type has been introduced in QGIS: the point cloud layer, which is used for all point cloud data. As mentioned above, it was not possible to reuse the existing vector layer type (with point geometries) as it is not designed to support such large quantities of points.
For file-based data access, it was decided to use PDAL (“point data abstraction library”). This library follows the approach adopted by the GDAL library: it provides a common interface for reading, writing or processing point cloud data, and various drivers (“readers” and “writers” in PDAL terminology) support the specifics of different formats. What’s more, PDAL offers a pipeline concept enabling various processing algorithms to be combined.
The second thing to consider was how to display the data efficiently. Going through all the points in a dataset and drawing them would simply not work well and would take too much time. It would also be wasteful – when zoomed out, only a small amount of points needed to be displayed. A second case to consider was that when a user zooms in a lot, we should be able to render only points within the area of interest, rather than reading points outside the current view. What was needed was a way of quickly querying points in these different situations. This need is usually solved by the use of space-partitioning data structures (such as octrees) that hierarchically divide space into smaller volumes. An important feature is that not only the leaves, but also the internal nodes contain data: thanks to this, it is possible not only to quickly query points in a small area of interest, but also to obtain smaller portions of points at different zoom levels.
Point cloud indexing
Point cloud data in commonly used formats such as LAS are fairly simple sets of points that are not associated with any indexing data structure, and the order of the points can be arbitrary. Consequently, it is necessary to build such an indexing data structure when opening these files in order to be able to render them – this is also done by other viewers (such as CloudCompare). As part of the indexing process, points need to be reorganized so that they can be accessed quickly when rendering. Once indexing is complete, we have a tree-like data structure where each node contains a subset of the point cloud – this indexed point cloud can be stored on disk or retained in memory.
2D/3D map view
Most importantly, users can visualize point cloud data on a 2D or 3D map. The added benefit of full integration with QGIS is that other data (vector or raster) can be superimposed on the point cloud data.
To enable rapid 2D and 3D rendering even of large datasets, QGIS is able to render subsets of the data using the hierarchical index (generated when the data is first loaded into QGIS). When the user zooms out, QGIS draws only a small number of points, and as soon as the user zooms in, QGIS draws larger and larger numbers of points (while reducing the extent of the map).
2D rendering
The whole rendering process is driven by the QgsPointCloudLayerRenderer class (where most of the work is done in a thread, as is the case with other layer renderers). For the time being, the acceptable geometric error (i.e. the acceptable spacing between points) is determined according to the map scale – for example, at a scale of 1:1,000,000, a spacing of 1 km between points may be suitable, but at a scale of 1:1,000, a much smaller spacing, say 1 meter, is required. This parameter is exposed to users so that they can control it as a “dynamic filtering” parameter, should the user wish to override the default setting (for example, for better quality rendering or faster rendering time).
Next, we go through the index hierarchy and retrieve the nodes that intersect the map view extent and have an acceptable geometric error. Finally, we retrieve the point cloud data from these nodes – either from the layer cache (if already in memory), or from the index nodes, and draw the points one by one.
In the work carried out so far, a single implementation of the renderer has been planned. It draws points of fixed size, the color of the points is determined by one of the attributes chosen by the user (e.g. elevation, classification, return number) and the color ramp chosen. In addition, some basic filtering options are supported (e.g., only the last return, only one or more particular classification values, only a particular range of elevations).
In the future, other renderings may be added, such as the display of interpolated surface (calculated on the fly from point data), shading or contours.
3D rendering
Basically, rendering point clouds in 3D follows the same principles as 2D rendering: using the hierarchical index, we determine which nodes are to be displayed, on the basis of which point data is loaded and displayed in the 3D scene. Part of the necessary infrastructure was already in place: QGIS 3D views support “piecewise” rendering of terrain from the outset (as the user gets closer to the terrain, more detailed elevations and map textures are rendered). As the mechanism is fairly generic and has been extended to vector layer data in the past, it has also been extended to point clouds. In the past, it was limited by the fact that it always expected a quadripartite hierarchy, but this has been updated to allow octree tree hierarchies.
Styling options for 3D views are similar to those for 2D rendering: configuration of point size, coloring based on a single attribute and some simple data filtering options.
It has also been possible to add the eye-dome lighting effect, which enhances depth perception.
Point identification
The 2D and 3D map identification tool has been updated to return complete information on identified points. In addition to their XYZ coordinates, it also returns the value of each point’s other attributes (such as classification and other attributes).
New features in version 3.32
Native processing provider for point clouds¶
A new native processing provider has been integrated for point cloud algorithms: the provider uses the pdal_wrench command line tool and currently supports the following algorithms:
- Information: provides basic point cloud metadata (number of points, extent, crs, etc.).
- Convert format: converts the point cloud to a different format, e.g. from las to laz.
- Reproject: reproject the point cloud to a different CRS.
- Fix projection: fix (assign) the CRS of a point cloud file.
- Clip: clip a point cloud by cutting out one or more polygons
- Merge: merge several point clouds into a single file
- Tile: create tiles from input data
- Thin: create a thinner version of the point cloud
- Boundary: export a vector layer containing the point cloud boundaries
- Density: exports a raster file in which each cell contains the number of points in that cell’s area.
- Export to raster: exports point cloud data to a 2D raster grid
- Export to vector: export point cloud data to a vector layer with 3D points
- Export to raster (TIN): export data from a point cloud to a 2D raster grid using point triangulation.
- Filter: extract a subset of the point cloud using PDAL expressions
As pdal_wrench requires PDAL >= 2.5.0, the new processing provider will only be available if the required PDAL version is met.
There are several reasons for using pdal_wrench rather than using the PDAL API and building pipelines:
- ease of use: pdal_wrench provides a set of ready-to-use algorithms and hides the complexity of pipeline creation from the user, making LiDAR data much easier to use for the uninitiated.
- parallel execution: PDAL executes pipelines in a single thread, while pdal_wrench supports multithreaded processing on either a spatial or file basis.
- virtual point cloud support: this is similar to GDAL’s virtual raster (VRT) – a single file references several other files and represents them as a single datum.
Point cloud expression builder for Processing framework¶
A new point cloud expression builder widget is available in the GUI to create point cloud filters in a user-friendly way that is consistent with existing QGIS expression builder interfaces.
Rectangle and expression filtering added to PDAL¶ algorithms
Added the option to filter the input point cloud by rectangle (extent) and expression in the following PDAL algorithms:
- limit
- clip
- density
- export to raster (normal and TIN variants)
- export to vector
- merge
- thin
This allows you to process only a subset of the points in the input file(s), without having to perform an intermediate filtering step or generate temporary files.
For those who only require filtering, there is also a separate filtering algorithm capable of filtering either by range, expression or their combination.
Virtual point clouds (VPC)¶
A new data provider is available for virtual point cloud (VPC) files created by the pdal_wrench utility and the Build virtual point cloud (VPC) tool in the new Native point cloud Processing provider.
The VPC file is treated as a single layer in QGIS and can contain a large number of point cloud files whose indexes are loaded when the canvas is sufficiently enlarged. The individual files contained in the virtual file are displayed as rights-of-way as long as their extent is less than the width of the map window. As soon as the zoom is sufficiently large and the extent of an individual file exceeds the width of the map window, the points in the file are displayed.
A virtual point cloud is a JSON container file with a .vpc extension, referencing other files/URLs that contain the actual point cloud data. The concept is similar to that of virtual rasters (VRT) in GDAL.
Displaying virtual point clouds (VPC) in 3D¶ views
Virtual point clouds can be rendered in 3D views. For performance management, a sub-index threshold is specified so that the view renders points in the point cloud only when appropriate, and instead renders the extent of a particular sub-index when the threshold criterion is not met.
