Building a NEW “best-of-breed” sGTM logger: real-time alerts, cumulative metrics, data quality checks, and a framework that grows with you
tl;dr What does this solve for in summary?
- For analysts, marketers, and SREs – Real-time visibility: data collection problems are flagged within minutes instead of being discovered days later when someone happens to check a dashboard, closing the gap between something breaking and someone finding out.
- For marketers – Soft failures caught before they cost money: surfacing issues that never throw an error, like an e-commerce feed quietly going out of stock or a media campaign burning through its budget, the kind of problem that only shows up as a change in pattern, not a crash.
- For marketers, and SREs – Tag failure alerting: pinpoints exactly which tag has broken (a tracking pixel, a conversion API, an ad platform integration) so reporting and ad spend decisions aren’t quietly built on broken data.
- For marketers – Event spike detection: flags sudden surges, such as a jump in errors or unexpected traffic, by comparing each tag against its own recent normal rather than a fixed guessed number, so genuine anomalies get caught without drowning the team in false alarms.
- For marketers – Plateau detection: catches a tag going silent (a paused destination, a broken trigger) and tells you whether it’s an isolated issue or a site-wide outage, cutting the time spent diagnosing what’s actually wrong.
- For SREs – Monitoring that scales with the business, not against it: all of the above for roughly 0.01p a day per container, tunable up or down, so the depth of monitoring can match the value of what’s being tracked rather than being an expensive afterthought.
- For analysts, marketers, and DPOs – automatic data quality checks. Confirm your data collection is getting the right types, right formats, and PII is only where it should be, hashed (or not), and not where it shouldn’t be.
Server-side Google Tag Manager (sGTM) solves a lot of data collection challenges, but it also creates a new blind spot: once a tag runs inside a Cloud Run container instead of a browser, you lose the easy visibility a browser console and the network tab used to give you. Tags fail quietly. Volumes drift without anyone noticing. A container that looked healthy yesterday can be silently broken today, and the first sign is often a client asking why last week’s numbers look wrong – more on this scenario anon from real world experiences…
Sure, you can manage alert emails in GA that deliver insights into spikes, drops, and a range of anomalies, but email? You don’t want to manage governance lists in the reporting UI – no one does this. You need the message to be specific, relevant and actionable for the recipient. You may need multiple different types of message – specifics for management, broadcast for engineering teams.
GA doesn’t do this. GTM doesn’t do this out of the box. We need some form of automatic webhook alert to deliver the right heads-up notices on sensible data anomalies, not false flags, to the right people in real time.
For sure, there are already good sGTM loggers out there. Addingwell and Stape both deploy well-regarded templates, and we used both as a starting point. But neither one, on its own, gave us everything we wanted, so we built our own: the Duga Logger.
Why we built another logger
The motivation was not “not invented here.” It was a specific gap. We wanted near real-time alerting on failures and spikes, across every tag and every event, not just a dashboard we had to remember to check. And we wanted cumulative metrics that could catch soft failures: the kind of problem that never throws an error, because nothing is technically broken. A product feed tag that keeps succeeding but stops firing for a specific event can mean a bestseller has quietly gone out of stock. A media tag that goes quiet, or a companion tag that spikes, can mean a campaign has hit its daily budget cap. Neither of those shows up as a failure in a normal error log. They only show up as a change in a pattern over time, which is exactly what cumulative, comparison-based monitoring is good at catching.
What we borrowed, and what we changed
Addingwell and Stape each had ideas worth keeping. The table below is drawn directly from the project README, and shows where each idea for the Duga Logger came from, and what we added on top. (Side note, if I missed or incorrectly categorised any features here, LMK, happy to address and ‘fess up)
| Feature | Inspired by | Duga logger adds |
|---|---|---|
| Full incoming request (path, query, body, headers) | Stape | Configurable headers, optional body |
| Full event data as real values | Stape | event_data JSON column plus flattened key fields |
| Per-tag execution results (the outgoing side) | Addingwell | tags JSON plus flat tags_total, tags_failed, tags_max_execution_ms columns |
| exclude: true tag-metadata filter | Addingwell | Supported |
| Debug-only vs always-on logging | Stape | A Log condition field |
| ignoreUnknownValues on the BigQuery insert | Stape | Supported, plus error logging, which Stape omits |
| Custom key/value rows | Stape | A custom_data JSON column |
| Consent state (gcs/gcd), richsstsse, container version | Addingwell | Supported |
| Date partitioning | Neither | PARTITION BY DATE(event_timestamp), no date maths needed in the sandbox |
| Slack alert on tag failure, spike or plateau | Neither | Optional Incoming Webhook, the same pattern already used by our release note checker and Version Update Automation |
One difference is worth calling out on its own, because it is a genuine trade-off rather than a straightforward improvement. Addingwell’s logger hashes everything for privacy. Ours logs raw values instead: IP address, user agent, full URLs, request body. That is a deliberate choice, because raw values make debugging much easier, but it means the resulting BigQuery table needs the access controls and retention policy that any dataset containing personal data deserves. If a client’s compliance posture calls for Addingwell’s approach instead, that is a configuration decision worth making explicitly, not an oversight.
Per-event and per-tag monitoring
The core of the logger is simple: every event gets one flat row in a date-partitioned BigQuery table, and every named tag gets its own execution result recorded inside that row.
Because the schema is flat, adding a new event parameter or a new tag never means a migration. Variable-shape data (event_data, request_headers, tags, custom_data, request_body) lives in JSON columns, queried with JSON_VALUE() or JSON_QUERY_ARRAY(), or through a convenience view we ship alongside the table.
Naming a tag is a single tick box (Advanced Settings, Additional Tag Metadata, Include Tag Name), and it does two things at once: it turns a numeric tag ID into a readable name in BigQuery, and it opts that tag into Slack alerting.
An unnamed tag still logs to BigQuery, it just does not generate its own alert, which keeps noisy or unimportant tags out of the alerting picture without extra configuration.
Comparison-based normality, not guessed thresholds
Setting a single fixed threshold for “too many” or “too few” almost never fits every tag equally. A high-volume pageview tag and a low-volume checkout tag do not share a sensible spike threshold, and a threshold tuned for one will either miss real problems on the other or drown you in false positives.
Rather than ask anyone to guess a number, the logger compares each tag against its own recent behaviour.
For spikes, this hour’s successful executions for a tag are checked against last hour’s count for the same tag. A spike needs both a relative multiplier (three times last hour’s count, for example) and a minimum absolute count, so a quiet tag going from one execution to three never reads as a three times spike. This is genuinely useful for catching something like a sudden surge in 404 tracking, where the interesting signal is the shape of the change, not an absolute number nobody could have picked in advance.
For plateaus, the comparison works differently: instead of comparing a tag against its own history, you can optionally name a reference tag you trust is normally always firing, typically a pageview tag. When another tag goes quiet, the alert checks whether that reference tag has also gone quiet. If it has, the problem is likely site-wide, perhaps the whole container has broken. If the reference tag is still firing normally, the problem is likely isolated to the one tag that plateaued. Either way, you get that context in the alert itself, rather than having to go and check.
Cumulative metrics catch what a single failed request cannot
A single failed request rarely tells the full story on its own. Cumulative counts, tracked hour by hour, do a better job.
The same hourly cumulative BigQuery view that powers our Data Studio trend line is the same signal the real-time Slack alert already watches: a flat line on the dashboard chart is the plateau the alert caught hours earlier. That consistency matters, because it means the dashboard and the alert are never telling two different stories about the same underlying data.
Real-time alerting, three ways
All of the alerting reuses a pattern we already trust: an optional Slack Incoming Webhook, the same approach used by our release note checker and by our GTM Container Version Update Automation.
There are three kinds of alert, and each is independently optional:
Failure alerting fires once a named tag’s failure count, since the last container publish, crosses a configurable threshold. It latches after firing once, so you are not paged repeatedly for the same ongoing issue, and it resets automatically the next time the container is published. This type of alert is the bare minimum but it’s saved blushes for clients and agencies alike multiple times. We have a sub-ten minute turn around case study fixing a Meta tag that had an out of date API key – saved € thousand, paid for itself there and then.
Spike alerting fires when a tag’s hourly successful executions jump against the previous hour’s count by at least a configured multiple, with a minimum absolute floor so a tiny baseline cannot trivially read as a huge spike.
Plateau alerting fires when a named tag has gone quiet for longer than a configured number of minutes, checked on a roughly fifteen-minute cycle rather than waiting for a daily reset. It clears itself automatically the moment the tag next succeeds.
State for all of this lives in Firestore rather than in the tag’s own runtime storage, because that storage is scoped to a single server process instance and gets wiped whenever Cloud Run recycles or autoscales, which happens more often than you would expect under real traffic. Firestore transactions give an atomic, cross-instance counter instead. Alerting can also never break event logging: the BigQuery insert always runs first and unconditionally, before the alerting function is even called, so a Slack or Firestore problem never costs you the underlying data.
Checking the data itself, not just the pipe it flows through
Everything above catches problems with the pipe: is a tag firing, is it failing, has volume moved. None of it looks at what is actually inside each event once it lands in BigQuery, and that turns out to be its own source of quiet failures.
A parameter can carry the wrong type, a field can go missing without anyone noticing, or personal data can end up somewhere nobody meant it to be, and no failure alert, spike alert or plateau alert would ever catch any of that.
So the logger ships a data quality layer alongside the alerting: an audit query profiles every parameter actually seen in event_data, its type, its value distribution, its cardinality, and checks it against a maintained dictionary of what each parameter is supposed to look like, flagging anything undocumented, wrongly typed, or newly appeared.
A set of PII heuristics runs alongside it, catching email, phone, card number, postcode and IP-shaped values landing in a parameter nobody declared as PII. The dictionary itself is seeded from a real GA4 export rather than typed by hand, so it reflects what your events actually contain rather than what someone assumed they would.
There is scope to fold this into an existing GA4 DataForm workload so it runs automatically rather than on demand, with the results surfaced on a Data Studio dashboard alongside the same cumulative view the Slack alerts already watch, turning it into a standing record of data quality and governance rather than a one-off check.
One framework, built to grow
Every threshold in the logger is a template field, not a hard-coded number. Alert thresholds, plateau minutes, the spike multiplier, and the minimum spike count are all tuned per client, or per container, without touching code. Any tag can be named and opted in, or excluded, at any time, with no redeploy needed. The same JSON columns that make the schema flat also mean new event parameters or new tags never require a schema migration. And because the Firestore project and webhook pattern are already used elsewhere in our stack, a new container plugs into infrastructure that already exists, rather than needing its own.
What it costs
Running cost for this starts at roughly 0.01p a day per container, based on what we are seeing in production today. That figure moves with how much detail you choose to log: more request headers, more named tags, and tighter alert thresholds cost a little more; fewer headers, coarser partitioning, and longer alert windows cost less. The same template scales in either direction, so the cost follows the level of detail you actually need rather than a fixed overhead you pay regardless.
Where this goes next
A few directions look worth exploring from here.
- Adaptive baselines that account for day-of-week seasonality would improve on the current fixed spike multiplier.
- A follow-up “resolved” Slack message when a failing or spiking tag recovers, not just when it breaks, would close the loop on an incident properly.
- Alert severity tiers would let critical failures reach an urgent channel while routine warnings land in a quieter digest, and a daily or weekly rollup summary would add trend awareness alongside the real-time pings without adding to alert fatigue.
- Per-client channel routing would let one container’s alerts land in one Slack channel and another container’s in a different one, from the same template.
- And an optional per-field hashing toggle would let clients who want Addingwell’s privacy posture keep it, without giving up everything else the logger does.
If you are running sGTM and any of this sounds familiar, whether it is a dashboard you have to remember to check, a threshold you have never quite trusted, or a soft failure that only showed up once someone noticed the numbers looked off, we would be glad to talk through how this could apply to your setup.