Summary
At a large Dutch company running SAP S/4HANA, SAP announced a security patch that blocked the protocol used to pull part of the data into the data warehouse, including the two heaviest financial datasets (general ledger lines and open items). Most sources already ran over a compliant protocol; the problem sat in a small number of views that leaned on the blocked protocol for their incremental refresh. After weighing up three routes, the most independent one was chosen: reading the underlying tables directly through the existing, compliant connector. Because that route brings no change detection of its own, a custom approach was built on a watermark, supplemented with logic that traces reversal and clearing postings through their reference document, and a periodic check on the remaining, dateless changes.
The brief
Make sure the SAP extraction keeps working once SAP applies the announced security patch, without losing the reliability of the incremental refresh, and preferably without becoming dependent on a new platform or a slow stopgap.
What I built
- An impact analysis that established that most of the sources already ran compliant and the risk was limited to a small number of views, verified against the live extraction configuration instead of assumed
- An assessment of three architectural routes (platform migration, HTTP variant of the blocked protocol, direct table extraction) with explicit pros and cons and a recommendation
- The simple master-data sources moved over to direct extraction of their SAP tables through the existing, compliant connector
- For the two heavy financial datasets: change detection without the blocked protocol, from a creation-date watermark, a follow-the-reference-document rule for reversals and clearings, and a periodic additional check
- A chunked initial full load (per fiscal year and period) after an unchunked attempt ran into a memory limit
- A phased, verified cutover on the development and the production environment, followed by clean-up of the old extraction layer
In detail
- Three-layer change detection without the blocked functionality: a date watermark for new postings, document-tracing logic for reversals and clearings on existing postings, and a periodic check on system change logs for the remaining, dateless amendments
- A fix for a row-width limit: one of the financial tables hit the maximum row size because of too many rarely used fields, solved by narrowing the column selection at extraction level
- A chunking strategy for the initial load: extraction in portions per fiscal year and period, each one well below the memory limit of the extraction function
- Orchestration speed-up: the refresh chain rebuilt from many separate steps into a single server-driven procedure, with a measurable gain in run time
- Phased cutover with verification: proven on development first, then on production, with control reports before and after to rule out regression
Outcome
In production on both the development and the production environment; the old extraction layer that SAP blocked has been cleaned up, and the refresh has run inside the intended time windows since.