Product Development at wetransform: the 5-year Review
Product development driven by customer need
Product development at wetransform continues to have real, needs-based discussions with a large variety of long-standing customers at its heart. We work closely with both regional and international customers to evolve our platform to meet ever-changing needs-including legislative requirements, new data and service standards, and emerging technologies that improve our infrastructure. Our growth is defined by challenges, as each new hale»connect public cloud and on-premise user brings new ideas, workflows, and technical complexities not unknown in the geospatial domain. We seek to solve those individual customer problems in a way that provides maximum benefit to the platform and users as a whole, and that has the potential to expand our existing user base.
The hale»connect platform was originally designed to reduce the amount of effort required to comply with EU and regional legislative requirements and offer implementers a fast and easy solution for publishing INSPIRE and ISO compliant datasets, metadata and associated OGC network services. With a strong focus on INSPIRE SLAs, the platform has expanded over the past 5 years to include specialized support for a broader range of specifications, including the German planning standard, XPlanung, and the Environmental Noise Directive (END). Looking ahead, product development will continue to closely follow emerging legislation and technological developments at the EU level, including the High Value Datasets (HVD) specification and the European Green Data Space. Building upon our expertise in the field, expanding in these directions is a natural extension of the original concept. We will work together with an amazing team of developers, and a network of customers and trusted partners to further define our products and long-term strategic vision.
Adapting to evolving standards
What we’ve done
Created in 2007, the INSPIRE directive continues to change and evolve based on real-world experience and feedback from a growing number of implementers. As a platform focused on providing INSPIRE implementers a compliant solution , we adapt hale»connect to support these changes.
Five years ago, we realized that most INSPIRE data specifications were still missing detailed portrayal rules. We undertook the challenge to create a complete repository of INSPIRE styled layer descriptor files (SLDs) to offer our customers compliant WMS styling for all 34 INSPIRE themes. INSPIRE compliant styles are now applied automatically to harmonized datasets containing INSPIRE feature types that are published on the platform.
During the same time in 2018, wetransform introduced the Profile Management Tools (PMT) at the INSPIRE conference in Antwerp. The PMT tool-suite enables customers to edit existing schemas to create and manage their own profiles. A profile is an adaptation of a schema, which contains additional constraints. Developed with support from LGL, the geodetic agency of Baden- Württemberg, the tools replaced manual workflows where large metadata profiles containing hundreds of conditions were managed in Excel. The Profile Management Tools are available to all hale»connect subscribers.
One of the most significant changes to INSPIRE over the past five years, was the migration from version 1.3 to version 2.0 metadata and the associated changes to the INSPIRE validator and INSPIRE monitoring. The migration from metadata version 1.3 to 2.0 involved removed requirements, new requirements, changed requirements and combined or altered requirements. The INSPIRE validator also added a new conformance class (Conformance Class 2b: INSPIRE data sets and data set series metadata for Monitoring) to test INSPIRE monitoring and reporting requirements, including spatial scope and priority dataset keywords, that were not a part of the technical guidance. Prior to the migration, the hale»connect platform could automatically generate fully compliant ISO and INSPIRE version 1.3 metadata, however offering full support for version 2.0 required a substantial effort. hale»connect generates metadata for datasets, dataset series, download services (direct and pre-defined) and view services and during the migration all types experienced changes. Metadata version 2.0 of the Technical Guidance for the implementation of INSPIRE dataset and service metadata based on ISO/TS 19139:2007 is now fully implemented on the hale»connect platform, and we continue to make updates based on on-going changes to the profile.
On a platform that supports tens of thousands of datasets, the need to view compliance on the platform level was becoming increasingly evident. From its inception, hale»connect used the GDI-DE test suite to provide validation reports. We decided to replace the GDI-DE test suite with the ETF validator to provide more flexibility and to enable users to view INSPIRE validation reports, for all published resources, directly within the application. The ETF integration has been one of the largest internal development projects undertaken. Efforts were required to adapt the UI to support the selection of conformance classes, real-time status updates across the platform and provision of reports. On the back-end, major efforts included managing the number of concurrently running tests and handling errors from the ETF. hale»connect customers can expect the release of the ETF validator integration in the coming weeks.
Building upon these developments, wetransform has recently released a new metadata-only workflow to support users with metadata creation. hale»connect users can now opt to create INSPIRE and ISO-compliant dataset metadata on the platform, without uploading any additional resources, such as the data itself. The workflow supports users who want to create valid metadata easily, without having to resort to manual, error-prone, copy-paste workflows in text editors. The metadata-only workflow is configurable on the theme level and is not available for network services.
Fifteen years after its creation, the INSPIRE directive is still a work-in-progress for many implementers. Low overall compliance rates and the lack of available supporting software tools have spurred interest in official INSPIRE alternate encodings to provide implementers additional means of serving compliant resources. wetransform has stood on the forefront of this initiative from the beginning, actively involved in developing the GeoJSON encoding rules for select INSPIRE themes . Following these efforts, CEO Thorsten Reitz developed the geopackage alternative encoding templates for the Environmental Noise Directive (END). hale»connect support for validating and publishing END-conformant GeoPackages and related transformation projects soon followed. Customers now use a streamlined, end-to-end workflow to generate compliant END deliverables using both hale»studio and hale»connect.
Things to come
The hale»connect platform will expand this year to offer support for OGC API Feature services. Customers will have the option to serve OGC API Feature services alongside their existing, legacy WFS services. We are currently testing serving the INSPIRE GeoJSON alternate encoding rules using OGC API feature services, and plan to support customers interested in this workflow.
Support for customers delivering High-Value Datasets (HVD) will continue to develop. With the recent release of a list of specific datasets to be delivered to the EU within the next 16 months, we look forward to supporting both new and existing customers in meeting their obligations.
Big Data
What we've done
Geospatial data has always been big, well before the term “big data” was coined. The past several years at wetransform has seen increased support for the “big data” our platform processes. We have increased the size threshold for individual file publishing, including support for file splitting and publishing multiple files in a dataset. At the same time, we have also added support for improved WMS rendering for large datasets, and support for a range of new standards.
One of the most significant workflows introduced to the platform over the past 5 years has been the introduction of dataset series. Dataset series offer our customers the ability to organize large numbers of related datasets which share the same schema, in a series dataset.
The hale»connect dataset series workflow is used heavily by our customers serving XPlanung GML data. Using this workflow, small to medium-sized regional governments also publish their local planning data, which amounts to hundreds, or even thousands of BP_Pläne, at a time. In the case of Xplanung, it is not the size of the individual datasets, but rather the combined size of thousands of these datasets in a single series, which makes the dataset series a challenging prospect.
Dataset series proved to be one of the most difficult developments to test in the history of wetransform.. We were faced with the challenge of scaling to manage many, many more datasets, and the resulting services. Support for XPlanung workflows also required adding support for PNG raster data in the form of georeferenced spatial planning documents, the first raster format supported on hale»connect. Fast-forward to 2023, and it is clear that wetransform will continue to expand its investment in XPlanung and scale to new heights with plans to offer an XPlanung Cloud product.
wetransform took its first steps to support native environmental data formats together with a Dutch organization and their need to harmonize NetCDF data. The hale»connect platform supports the access of API services returning raw NetCDF files. We even encountered a NetCDF file containing a complete time series reaching back to the 1950’s. A single file containing daily, or even hourly, weather sensor readings recorded over decades- that’s big! Beyond simple access, wetransform also developed the INSPIRE harmonization projects and cloud automation workflows to automatically convert, transform and publish the INSPIRE-compliant meteorological datasets on the platform.
Things to come
One of the most requested features we hear from our customers, is support for “real” raster data, or orthoimagery and multi-spectral rasters, in the form of coverage services. Although not currently supported, we will continue to investigate a variety of potential frameworks that will allow us to add such services to our public-cloud offering. Aligned with our goal to improve support for environmental data standards, we are also watching developments in the EDR (Environmental Data Retrieval) API, STAC (Spatio Temporal Asset Catalogs) and OGC API Coverages standards, with plans to build upon our support for OGC API Features.
Modernizing infrastructure
What we've done
In an effort to support ever-increasing volumes of data, we face the challenge of both storing data, and rendering data in a timely manner. The introduction of a hybrid rendering mode for WMS view services significantly improved rendering times for large datasets. The mode automatically generates a relational database structure based on the SLD, and includes automatic generalization and limits the number of features shown per tile.
Large numbers of parallel WMS requests remained a problem, with customers facing varying challenges. In response we strived to optimise caching of WMS requests with Mapproxy and made a wide range of configuration options available to sytem administrators.
As the hale»connect platform and the customers have grown, so too have the variety of implementation offerings. Clients can choose from a range of deployment options to best suit their needs, including public-cloud, private-cloud, and on-premise. Over the past five years, configuration management for on-premise customers has expanded- hale»connect can be installed and operated with Docker Compose on a single host, Docker Swarm on a single host or on multiple hosts, or in a Kubernetes cluster.
Regardless of their setup, on-premise customers have the possibility to customize and configure their own hale»connect instance with a wide range of options, from adapting the UI theme to enabling or disabling certain platform features and adapting the platform to their specific needs and infrastructure. Our configuration management system allows them to easily manage multiple instances, handle secrets securly and deploy with a single command.
In 2021, hale»connect was migrated to Kubernetes. Kubernetes is an open-source framework for automating deployment and container management. The Kubernetes migration was the largest internal infrastructure project undertaken in the past 5 years and represents a significant investment in improving the stability and resilience of the platform. Running hale»connect on Kubernetes means that managing the life cycles of containers, load-balancing, and automated monitoring and alerting have all improved. These improvements have reduced unexpected down-time and major incidents.
Things to come
Looking ahead, we will continue to improve the scalability of the platform to ensure our public-cloud customers a seamless experience. With the groundwork of a successful Kubernetes migration laid, we will work on further optimizing the framework to deliver additional benefits like better automated scaling and resilience.
Conclusion
Product development at wetransform will continue to respond to customer needs in a growing range of areas. The core, subscription-based hale»connect product will offer support for the next generation of OGC services, and a more sophisticated geoportal, offering customers the ability to visualize and analyze data on the platform. The hale»connect-based XPlanung cloud offering will continue to develop extended support for the standard, along with AI-based, value-added services such as semantic text search across all XPlanung documents.
Having been awarded two substantial dataspaces research projects, Future Forests AI (FF.ai) and Soilsense, wetransform has been working actively in the formative discussions shaping the direction of these new technologies for years. The next challenge is to consolidate these experiences with stakeholders and other early adopters and bring a wetransform interoperability solution to market. The solution should offer data providers a critical piece of the data spaces puzzle by providing the transformation service layer required to guarantee data interoperability within a data space. It is an exciting time for wetransform, as the data spaces “space” in the EU becomes more and more concrete, and as the EU begins to provide data spaces for public use. Stay tuned!