{"id":63490,"date":"2025-02-02T03:25:24","date_gmt":"2025-02-02T03:25:24","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/enrich-server-side-data-with-cloud-firestore\/"},"modified":"2025-02-02T03:25:24","modified_gmt":"2025-02-02T03:25:24","slug":"enrich-server-side-data-with-cloud-firestore","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=63490","title":{"rendered":"Enrich Server-side Data With Cloud Firestore"},"content":{"rendered":"


\n<\/p>\n

\n

With asynchronous variables<\/a> recently released in server-side Google Tag Manager<\/a>, it\u2019s time to dig into data enrichment flows<\/strong> using another<\/em> release from the Google team.<\/p>\n

* drum roll *<\/strong><\/p>\n

We have a new Google Cloud Platform API!<\/p>\n

It\u2019s fast. It\u2019s sleek. It\u2019s beautiful. It\u2019s Firestore<\/strong>!<\/p>\n

\n

<\/p>\n

\"Firestore\"\/<\/p>\n

<\/a><\/p>\n<\/div>\n

Firestore is a NoSQL<\/strong>, transactional<\/strong>, and scalable<\/strong> database that offers near-real-time write\/read and sync operations for data.<\/p>\n

In practice, it\u2019s a great<\/em> way to enrich and widen the data that you pass through your Server container.<\/p>\n


\n X<\/span><\/p>\n

\n
\n The Simmer Newsletter<\/strong>\n <\/p>\n

\n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n

<\/span><\/p>\n

What is Firestore<\/h2>\n

Firestore stores data in documents<\/strong>, each contained within a collection<\/strong>.<\/p>\n

\n

<\/p>\n

\"Firestore<\/p>\n

<\/a><\/p>\n<\/div>\n

Documents store data using familiar data types, such as strings<\/strong>, numbers<\/strong>, complex objects<\/strong>, and even subcollections<\/strong> for more hierarchical setups.<\/p>\n

When querying the data, you can either directly reference a document by its path<\/strong> (e.g. order_history\/416cda47-cb46-435c-9c9a-15527d76343f<\/code>), or you can query<\/strong> the collection to fetch all documents that match the query (e.g. iso_time == \"2022-02-10T09:21:27.718Z\"<\/code>).<\/p>\n

As such, Firestore has great utility for widening<\/em> the data streams in the Server container.<\/p>\n

\n

<\/p>\n

\"Firestore<\/p>\n

<\/a><\/p>\n<\/div>\n

The incoming request can include a key, such as the transaction ID<\/strong>, and this key can then be used to query the Firestore for additional data.<\/p>\n

\n

<\/p>\n

\"Firestore<\/p>\n

<\/a><\/p>\n<\/div>\n

Thanks to the new asynchronous variables<\/a>, this can all be done quite easily (see below for examples).<\/p>\n

The new Firestore Lookup variable<\/h2>\n

To make it easy to read<\/strong> Firestore data, there\u2019s a new Firestore Lookup<\/strong> variable in server-side Google Tag Manager.<\/p>\n

\n

<\/p>\n

\"Blank<\/p>\n

<\/a><\/p>\n<\/div>\n

This variable lets you pull in the value from a specific key<\/strong>\/field<\/strong> in a Firestore document<\/em>.<\/p>\n

Lookup Type<\/h3>\n

The Lookup Type<\/strong> specifies how the correct document<\/strong> is retrieved. If you recall, Firestore nests all the relevant information within a document<\/em>, which is hosted in a collection<\/em>.<\/p>\n

You can fetch the document directly using the full Document Path<\/strong>, or you can search for the document with Collection Path & Query<\/strong>.<\/p>\n

Specifying the Document Path is the fastest method, because it\u2019s a direct lookup to fetch a single document.<\/p>\n

The document path typically looks like this:<\/p>\n

collection_name\/document_name<\/code><\/p>\n

\n

<\/p>\n

\"collection\/document\"\/<\/p>\n

<\/a><\/p>\n<\/div>\n

For example, to fetch the first document in the screenshot above, I would specify this as the Document Path:<\/p>\n

order_history\/416cda47-cb46-435c-9c9a-15527d76343f<\/code><\/p>\n

Alternatively, if you can\u2019t access the document directly with the path, you can also query<\/strong> it. You first need to provide a path to the collection that has the document. Then, you can add one or more query conditions<\/strong> using keys you know to exist in the document.<\/p>\n

\n

<\/p>\n

\"User<\/p>\n

<\/a><\/p>\n<\/div>\n

The query above would look for documents embedded within the user_database<\/code> collection. The variable would fetch the first<\/strong> document that has the user_id<\/code> field set to whatever is contained in the GTM {{User ID}}<\/code> variable (could be something from the incoming request, for example) and has the user_status<\/code> field set to something other than inactive<\/code>.<\/p>\n

\n

Because Firestore indexes documents by default, query performance is proportional to the size of the result set<\/strong> and not the data set being queried. So it doesn\u2019t matter how many documents there are in the collection for query performance, but it does matter how broad your query is (i.e. how many results it retrieves).<\/p>\n<\/blockquote>\n

The Firestore variable uses the first document that matches the query<\/strong> for returning the key (see below). This is something to consider in case your query is too general and returns multiple (unrelated) documents.<\/p>\n

Key Path<\/h3>\n

Assuming the lookup returns a document, the Key Path<\/strong> then specifies which value to fetch.<\/p>\n

Here\u2019s a dummy example of a Firestore document:<\/p>\n

\n

<\/p>\n

\"Dummy<\/p>\n

<\/a><\/p>\n<\/div>\n

To fetch the transaction_id<\/code>, I would set that as the Key Path<\/strong>. To fetch the value<\/code>, I would set that. To fetch the ID of the first item, I could use items.0.item_id<\/code>, and so on.<\/p>\n

If no value is found, then the Default Value<\/strong> is returned if set, or undefined<\/code> otherwise.<\/p>\n

Override Project ID<\/h3>\n

By default, the variable looks for collections and documents in the same Google Cloud Project<\/strong> that runs the GTM server.<\/p>\n

\n

Note!<\/strong> This requires that the Project ID is set in the GOOGLE_CLOUD_PROJECT<\/code> environment variable of the cloud service. This is done automatically in the App Engine deployment. If you\u2019re running Cloud Run<\/a> or some other stack, you need to add the environment variable to the service or<\/em> you can simply type the project ID into the Project ID field.<\/p>\n<\/blockquote>\n

If you want, you can override the Project ID<\/strong> and provide some other Google Cloud Project ID where the collection is stored.<\/p>\n

To do this, expand More Settings<\/strong> in the tag and check Override Project ID<\/strong>.<\/p>\n

\n

<\/p>\n

\"Override<\/p>\n

<\/a><\/p>\n<\/div>\n

\n

If you provide another Project ID, you must add your GTM Server project service account<\/strong> to the Firestore project via IAM.<\/p>\n<\/blockquote>\n

\n

<\/p>\n

\"IAM<\/p>\n

<\/a><\/p>\n<\/div>\n

If you\u2019re running your container on App Engine, locate the App Engine default service account<\/strong> in your GTM Server project IAM<\/a>, and add it as a new role into your Firestore project<\/a>.<\/p>\n

If your container is running on Cloud Run, use the Compute Engine default service account<\/strong> instead.<\/p>\n

Grant the service account a role of Cloud Datastore User<\/strong> to give the GTM Server access to the other project\u2019s Firestore.<\/p>\n

Example walkthrough<\/h2>\n

In this fictitious example, I\u2019ll show you how you could use Firestore to widen<\/strong> an incoming Google Analytics 4<\/strong> data stream.<\/p>\n

\n

<\/p>\n

\"Example<\/p>\n

<\/a><\/p>\n<\/div>\n

As you can see from the immaculately designed process flow above, there are several moving parts in this setup.<\/p>\n

    \n
  1. First, the user makes a purchase<\/strong> on the website.<\/li>\n
  2. As soon as the purchase happens, an entry is made into the local database<\/strong> (or service database) of the store.<\/li>\n
  3. Immediately, the order details are written into a Firebase collection<\/strong>, too.<\/li>\n
  4. The web server responds to the site that the purchase was successful, at which point the site sends a Google Analytics 4<\/strong> event to the server-side Google Tag Manager endpoint<\/strong>, with the Transaction ID<\/strong> as one of the parameters.<\/li>\n
  5. The server-side endpoint then takes this Transaction ID from the request and uses Firestore variables<\/strong> to pull the rest of the purchase information.<\/li>\n
  6. These variables are then added to the server-side Google Analytics tag (and other tags that need this information), in order to complete the required parameters that a well-formed Purchase event<\/strong> requires.<\/li>\n<\/ol>\n

    Because this is a made-up example, I\u2019m going to skip steps 1\u20133<\/strong>. It\u2019s a bit of a cop-out because engineering the data flow from the sales engine to Firestore isn\u2019t trivial to do. It has to happen immediately<\/strong> to avoid latency and potential race conditions, where the server container tries to query for data that doesn\u2019t (yet) exist.<\/p>\n

    Firestore collection<\/h3>\n

    In Firestore, the collection ends up looking like this:<\/p>\n

    \n

    <\/p>\n

    \"Transaction<\/p>\n

    <\/a><\/p>\n<\/div>\n

    There\u2019s a collection called order_history<\/code>, under which each transaction is its own, unique document. I\u2019m using the Transaction ID as the name of each document to make the lookup easier.<\/p>\n

    Each transaction document has the following fields. I\u2019ve chosen these fields and these types to require as little transformation in the Server container as possible. Thus, the fields and the values follow closely what GA4\u2019s ecommerce schema<\/a> expects.<\/p>\n