(and now I have a dragon)
Dan and Doug have been pestering motivating me for a few months now to talk about the work we’ve been doing with Data Manager API. We have created an automated architecture in Google Cloud to send conversion data to sGTM whenever it is uploaded by our client, GTM Client Template, a Tag and two Variable Templates for write and retrieve data from Firestore, a Tag template to send data event data to Google platforms, and one to manage Audiences in Google. It’s been a busy few months… Last week a few pieces fell in place and I decided to go for it, and I presented at MeasureCamp London about our offline conversion tracking solution. I didn’t expect to be awarded the Best Newcomer presentation, nor to be handed Sebastian, my new favourite dragon and best ice breaker in a pub or riding the bus home.
A massive thank you to everyone who came along and asked questions, to Addingwell (and Lucas Rolin that helped with his superb server-side skills) and to the organisers and volunteers who made the whole day happen. MeasureCamp is awesome and I really want to participate in more of them moving forward!
Below I’ll talk a bit about DM API, share what I’ve gone through in the presentation, and something new about it that I learned because of someone’s question.
Data Manager API
The Data Manager API is a single API for sending audience and conversion data to Google platforms: Google Ads, Google Analytics, DV 360, CM360, Search Ads 360, and Google Ad Manager (literally found out last week).
One request can contain all of the above destinations, any number of events (I haven’t tried to break it though), require no API secrets to manage, and is actively being developed. This is how Google wants you to start sending server-to-server events now. From the beginning of the year they have been developing this HARD, and we’ve been testing it a lot. This month, it has come to a point where I’m really happy with it. The documentation is looking very complete, covering several platforms and use cases, all bugs we identified have been solved, and every new possibility they ask us to test is working.
Use Case: Offline Purchase Attribution
One of our clients was sending offline purchase events to GA via MP. This works to some extent, but this means that GA has two issues: 1 – GA contains all online and offline purchases now, even if the offline client never visited the website, and 2 – The offline purchases were always attributed to Direct. DM API allows to solve both of these problems. Here are the steps to do it.
1 – Collect Session Attribution
When the user makes a consent selection, their session attributes are sent to the server-side container and stored in a Firestore record. We collect parameters like client ID, session start time, landing page and referrer, user agent, campaign identifiers (gclid, gbraid, wbraid, UTMs) and, very importantly, the Google consent state at that moment.
1.5 – Storage
That record lives in Firestore, keyed to the client ID, with a time-to-live. We don’t need to keep it forever, only long enough to account for an usual online interaction to offline purchase window.
2 – Collect UPD during Lead Generation
When the user triggers a lead gen action and provides their User Provided Data (UPD), we look up the same client ID in Firestore. If there’s a match, a new record is created combining the session data with the UPD: a normalised and hashed email address and phone number.
2.5 – Storage, again
This second record is keyed to the hashed email address, and it holds everything we’ll need later:
- User identifiers: client ID, session ID, hashed email, hashed phone number
- Session attributes: landing page, referrer, user agent, campaign, click identifiers
- Consent state: the gcs and gcd strings
- Lead gen action: which interaction the user provided their data in
- Time-to-live: how long the record will be kept
3 – Reconcile
Now the offline purchase happens. The sale is sent from the back-end (point of sale, CRM, CDP, whatever the source is) to the server-side container. It contains the usual ecommerce data, and the UPD the user had previously shared. Our bespoke variable looks up that hashed email in Firestore, finds the record from step 2.5, and creates a DM API payload with the data from the event and the data stored in Firestore.
This is where the GTM tag does the heavy lifting. The Data Manager API tag has a few components worth calling out:
- Destinations: as many as you want (it probably has a limit, I haven’t found it yet)
- Validation: validateOnly lets you test the request without actually sending data
- Event Source
- Consent Settings
- Data Override / Additional Data
- Firestore Campaign Reader (which does part of the magic)
4 – Deliver
One complete, consented event, sent to every destination.
GA event with First user attribution:
Google Ads conversions:
The Surprise Slides
Consent
When can we do all of this? The only answer is: When the user consents. Since there are a lot of moving parts involved (Firestore storage, UPD collection, sending of data to Analytics and Marketing platforms), we need to know when it’s ok to do some of these things, but not the other. For this, I created a matrix that shows when we’ll perform any of these tasks based on the consent given. This is not for anyone to copy and this is not our legal recommendation as well. This is the baseline I created to talk to, and get legal sign of, from any client we’re working with. Ultimately, this is their decision and responsibility, but I’ve seen this falling in the hands of analysts way too many times for not to be proactively thinking about and creating in-built solutions for it nowadays.
Other uses
The same architecture isn’t only for offline conversions. Two other quick uses of Data Manager API are: 1 – Sending website purchase data from your back-end to Google Ads as an additional data source, deduplicated by transaction ID (Multi-Source Conversions, formerly known by the catchy PEWMDS), and 2 – add or remove members to Customer Lists for audience matching.
Google Ads Website events:
Google Ads Audience Matching:
New Knowledge
A person’s question after the presentation (part of the reason why I loved doing this was talking to people afterwards) regarding event attribution got me thinking about what we could do to make attribution even better. After researching a bit, I actually found out that it already is better than I thought! Not only First user attribution is proper for the events we’re sending, and Session attribution is as well if the event is sent within 24 hours of the session, but event data driven attribution is also present! The same transaction, when looking at the Source/Medium and other Event scoped parameters, we get DDA.
User scope (no DDA, as usual):
Event Scope (with DDA):
What Now?
Google will continue to develop DM API, and we will (hopefully) continue to test it. We’d love to see this become available for all users soon. As of now, it only works for properties that have allowed it. But we’ll definitely continue to implement it, and shout out when we have more cool things to share.
Thank you again to everyone at MeasureCamp London. If you were in the room, I’d love to hear what you thought. If you weren’t, you can find the full slide deck here.