A recent guide<\/a> of mine introduced the Google Analytics adapter<\/a> in Snowplow<\/a>. The idea was that you can duplicate the Google Analytics<\/a> requests sent via Google Tag Manager<\/a> and dispatch them to your Snowplow analytics pipeline, too. The pipeline then takes care of these duplicated requests, using the new adapter to automatically align the hits with their corresponding data tables, ready for data modeling and analysis.<\/p>\n While testing the new adapter, I implemented a Snowplow pipeline from scratch for parsing data from my own website. This was the first time I\u2019d done the whole process from end-to-end myself, so I thought it might be prudent to document the process for the benefit of others who might want to take a jab at Snowplow but are intimidated by the moving parts.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n And make no mistake. There are plenty<\/strong> of moving parts. Snowplow leverages a number of Amazon Web Services<\/a> components, in addition to a whole host of utilities of its own. It\u2019s not like setting up Google Analytics, which is, at its most basic, a fairly rudimentary plug-and-play affair.<\/p>\n <\/p>\n <\/a><\/p>\n Image source: https:\/\/goo.gl\/D2f3xi<\/span><\/p>\n<\/div>\n Take this article with a grain of salt. It definitely does not<\/strong> describe the most efficient or cost-effective way to do things, but it should help you get started with Snowplow, ending up with a data store full of usable data for modeling and analysis. As such, it\u2019s not necessarily a guide<\/strong> rather than a description of the steps I took, in good and bad.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n WARNING:<\/strong> If you DO follow this guide step-by-step (which makes me very happy), please do note that there will be costs involved. For example, my current, very light-weight setup, is costing me a couple of dollars a day to maintain, with most costs incurred by running the collector on a virtual machine in Amazon\u2019s cloud. Just keep this in mind when working with AWS. It\u2019s unlikely to be totally free, even if you have the free tier for your account.<\/p>\n \n \n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n <\/span><\/p>\n I ended up needing the following things to make the pipeline work:<\/p>\n The Linux\/Unix command line (handily accessible via the Terminal application of Mac OS X).<\/p>\n<\/li>\n Git<\/a> client – not strictly necessary but it makes life easier to clone the Snowplow repo and work with it locally.<\/p>\n<\/li>\n A new Amazon Web Services<\/a> account with the introductory free tier (first 12 months).<\/p>\n<\/li>\n A credit card – even with the free tier the pipeline is not free.<\/p>\n<\/li>\n A domain name<\/strong> of my own (I used gtmtools.com) whose DNS records I can modify.<\/p>\n<\/li>\n A Google Analytics tag running through Google Tag Manager.<\/p>\n<\/li>\n A lot of time.<\/p>\n<\/li>\n<\/ul>\n <\/p>\n <\/a><\/p>\n<\/div>\n The bullets concerning money and custom domain name might be a turn-off to some.<\/p>\n You might be able to set up the pipeline without a domain name by using some combination of Amazon CloudFront and Route 53 with Amazon\u2019s own SSL certificates, but I didn\u2019t explore this option.<\/p>\n And yes, this whole thing is going to cost money. As I wrote in the beginning, I didn\u2019t follow the most cost-effective path. But even if I did, it would still cost a dollar or something per day to keep this up and running. If this is enough of a red flag for you, then take a look at what managed solutions<\/a> Snowplow is offering. This article is for the engineers out there who want to try building the whole thing from scratch.<\/p>\n Why do this exercise at all? Why even look towards Snowplow? The transition from the pre-built, top-down world of Google Analytics to the anarchy represented by Snowplow\u2019s agnostic approach to data processing can be daunting.<\/p>\n Let me be frank: Snowplow is not for everyone. Even though the company itself offers managed solutions, making it as turnkey as it can get, it\u2019s still you<\/strong> building an analytics pipeline<\/strong> to suit your<\/strong> organization\u2019s needs. This involves asking very difficult questions, such as:<\/p>\n What is an \u201cevent\u201d?<\/p>\n<\/li>\n What constitutes a \u201csession\u201d?<\/p>\n<\/li>\n Who owns the data?<\/p>\n<\/li>\n What\u2019s the ROI of data analytics?<\/p>\n<\/li>\n How should conversions be attributed?<\/p>\n<\/li>\n How should I measure users across domains and devices?<\/p>\n<\/li>\n<\/ul>\n If you\u2019ve never asked one of these (or similar) questions before, you might not want to look at Snowplow or any other custom-built setup yet. These are questions that inevitably surface at some point when using tools that give you very few configuration options.<\/p>\n At this point I think I should add a disclaimer. This article is not Google Analytics versus Snowplow<\/strong>. There\u2019s no reason to bring one down to highlight the benefits of the other. Both GA and Snowplow have their place in the world of analytics, and having one is not predicated on the absence of the other.<\/p>\n The whole idea behind the Google Analytics plugin, for example, is that you can duplicate tracking to both GA and to Snowplow. You might want to reserve GA tracking for marketing and advertising attribution, as Google\u2019s backend integrations are still unmatched by other platforms. You can then run Snowplow to collect this same data so that you\u2019ll have access to an unsampled, raw, relational database you can use to enrich and join with your other data sets.<\/p>\n Snowplow is NOT a Google Analytics killer<\/strong>. They\u2019re more like cousins fighting together for the honor of the same family line, but occasionally quarreling over the inheritance of a common, recently deceased relative.<\/p>\n Here\u2019s a diagram of what we\u2019re hopefully going to build in this article:<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n I wonder why no one\u2019s hired me as a designer yet\u2026<\/p>\n The process will cover the following steps.<\/p>\n The website will duplicate the payloads sent to Google Analytics, and send them to a collector written with Clojure<\/a>.<\/p>\n<\/li>\n The collector runs as a web service on AWS Elastic Beanstalk<\/a>, to which traffic is routed and secured with SSL from my custom domain name using AWS Route 53<\/a>.<\/p>\n<\/li>\n The log data from the collector is stored in AWS S3<\/a>.<\/p>\n<\/li>\n A utility is periodically executed on my local machine, which runs an ETL (e<\/strong>xtract, t<\/strong>ransform, l<\/strong>oad) process using AWS EMR<\/a> to enrich and \u201cshred\u201d the data in S3.<\/p>\n<\/li>\n The same utility finally stores the processed data files into relational tables in AWS Redshift<\/a>.<\/p>\n<\/li>\n<\/ol>\n So the process ends with a relational database that has all your collected data populated periodically.<\/p>\n The very first thing to do is register on Amazon Web Services and setup an IAM (Identity and Access Management) User that will run the whole setup.<\/p>\n So browse to https:\/\/aws.amazon.com\/,<\/a> and select the option to create a free account.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n The free account gives you access to the free tier<\/a> of services, some of which will help keep costs down for this pipeline, too.<\/p>\n Once you\u2019ve created the account, you can do the first important thing in setting up the pipeline: create an IAM User. We\u2019ll be following Snowplow\u2019s own excellent IAM setup guide<\/a> for these steps.<\/p>\n In the Services<\/strong> menu, select IAM from the long list of items.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n In the left-hand menu, select Groups<\/strong>.<\/p>\n<\/li>\n Click the Create New Group<\/strong> in the view that opens.<\/p>\n<\/li>\n Name the group Skip the Attach Policy step for now by clicking the Next Step<\/strong> button.<\/p>\n<\/li>\n Click Create Group<\/strong>.<\/p>\n<\/li>\n<\/ul>\n Now in the left-hand menu, select Policies<\/strong>.<\/p>\n Next, click Review Policy<\/strong>.<\/p>\n<\/li>\n Name the policy Finally, click Create Policy<\/strong>.<\/p>\n<\/li>\n<\/ul>\n Now go back to Groups<\/strong> from the left-hand menu, and click the snowplow-setup<\/strong> group name to open its settings.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Switch to the Permissions<\/strong> tab and click Attach Policy<\/strong>.<\/p>\n<\/li>\n From the list that opens, select snowplow-setup-policy-infrastructure<\/strong> and click Attach Policy<\/strong>.<\/p>\n<\/li>\n<\/ul>\n <\/p>\n <\/a><\/p>\n<\/div>\n Now select Users<\/strong> from the left-hand menu, and click the Add user<\/strong> button.<\/p>\n Name the user Check the box next to Programmatic access<\/strong>.<\/p>\n<\/li>\n Click Next: Permissions<\/strong>.<\/p>\n<\/li>\n With Add user to group<\/strong> selected, check the box next to snowplow-setup<\/strong>, and click the Next: Review<\/strong> button at the bottom of the page.<\/p>\n<\/li>\n Finally, click Create user<\/strong>.<\/p>\n<\/li>\n<\/ul>\n The following screen will show you a success message, and your user with an Access Key ID<\/strong> and Secret Access Key<\/strong> (click Show to see it) available. At this point, copy both of these somewhere safe<\/strong>. You will need them soon, and once you leave this screen you will not be able to see the secret key anymore. You can also download them as a CSV file by clicking the Download .csv<\/strong> button.<\/p>\n You have now created a user with which you will set up your entire pipeline. Once everything is set up, you will create a new user with fewer permissions, who will take care of running and managing the pipeline.<\/p>\n Congratulations, step 0 complete!<\/p>\n You should have successfully registered a new account on AWS.<\/p>\n<\/li>\n You should have a new IAM user named This is going to be one of the more difficult steps to do, since there\u2019s no generic guide for some of the things that need to be done with the collector endpoint. If you want, you can look into the CloudFront Collector<\/a> instead, because that runs directly on top of S3 without needing a web service to collect the hits. However, it doesn\u2019t support the Google Analytics hit duplicator, which is why in this article we\u2019ll use the Clojure collector<\/a>.<\/p>\n The Clojure collector is basically a web endpoint to which you will log the requests from your site. The endpoint is a scalable web service running on Apache Tomcat, which is hosted on AWS\u2019 Elastic Beanstalk. The collector has been configured to automatically log the Tomcat access logs directly into AWS S3 storage, meaning whenever your site sends a request to the collector, it logs this request as an access log entry, ready for the ETL process that comes soon after.<\/p>\n But let\u2019s not get ahead of ourselves.<\/p>\n The first thing you\u2019ll need to do is download the Clojure collector WAR file and store it locally in a temporary location (such as your Downloads folder).<\/p>\n![\"Snowplow](\"https:\/\/www.simoahava.com\/images\/2018\/02\/snowplow-etl-emr.jpg#ZgotmplZ\")
<\/p>\n![\"Image](\"https:\/\/www.simoahava.com\/images\/2018\/02\/ga-schema.jpg#ZgotmplZ\")
<\/p>\n![\"SQL](\"https:\/\/www.simoahava.com\/images\/2018\/02\/sql-workbench-query.jpg#ZgotmplZ\")
<\/p>\n
\n X<\/span><\/p>\n
\n The Simmer Newsletter<\/strong>\n <\/p>\nTo start with<\/h2>\n
\n
![\"The](\"https:\/\/www.simoahava.com\/images\/2018\/02\/command-line.jpg#ZgotmplZ\")
<\/p>\nWhy Snowplow?<\/h2>\n
\n
What we are going to build<\/h2>\n
![\"Snowplow](\"https:\/\/www.simoahava.com\/images\/2018\/02\/snowplow-process.jpg#ZgotmplZ\")
<\/p>\n\n
Step 0: Register on AWS and setup IAM roles<\/h2>\n
What you need for this step<\/h3>\n
\n
Register on AMR<\/h3>\n
![\"Create](\"https:\/\/www.simoahava.com\/images\/2018\/02\/create-free-account-aws.jpg#ZgotmplZ\")
<\/p>\nCreate an Identity and Account Management (IAM) user<\/h3>\n
![\"Services](\"https:\/\/www.simoahava.com\/images\/2018\/02\/services-menu.jpg#ZgotmplZ\")
<\/p>\n\n
snowplow-setup<\/code>.<\/p>\n<\/li>\n{\n \"Version\"<\/span>: \"2012-10-17\"<\/span>,\n \"Statement\"<\/span>: [\n {\n \"Effect\"<\/span>: \"Allow\"<\/span>,\n \"Action\"<\/span>: [\n \"acm:*\"<\/span>,\n \"autoscaling:*\"<\/span>,\n \"aws-marketplace:ViewSubscriptions\"<\/span>,\n \"aws-marketplace:Subscribe\"<\/span>,\n \"aws-marketplace:Unsubscribe\"<\/span>,\n \"cloudformation:*\"<\/span>,\n \"cloudfront:*\"<\/span>,\n \"cloudwatch:*\"<\/span>,\n \"ec2:*\"<\/span>,\n \"elasticbeanstalk:*\"<\/span>,\n \"elasticloadbalancing:*\"<\/span>,\n \"elasticmapreduce:*\"<\/span>,\n \"es:*\"<\/span>,\n \"iam:*\"<\/span>,\n \"rds:*\"<\/span>,\n \"redshift:*\"<\/span>,\n \"s3:*\"<\/span>,\n \"sns:*\"<\/span>\n ],\n \"Resource\"<\/span>: \"*\"<\/span>\n }\n ]\n}<\/code><\/pre>\n<\/div>\n\n
snowplow-setup-policy-infrastructure<\/code>.<\/p>\n<\/li>\n![\"IAM](\"https:\/\/www.simoahava.com\/images\/2018\/02\/iam-group.jpg#ZgotmplZ\")
<\/p>\n\n
![\"Attach](\"https:\/\/www.simoahava.com\/images\/2018\/02\/attach-policy.jpg#ZgotmplZ\")
<\/p>\n\n
snowplow-setup<\/code>.<\/p>\n<\/li>\nWhat you should have after this step<\/h3>\n
\n
snowplow-setup<\/code> with all the access privileges distributed by the custom policy you created.<\/p>\n<\/li>\n<\/ol>\nStep 1: The Clojure collector<\/h2>\n
What you need for this step<\/h3>\n
\n
Getting started<\/h3>\n
Setting up the Clojure collector<\/h3>\n
![\"Select](\"https:\/\/www.simoahava.com\/images\/2018\/02\/region-selection.jpg#ZgotmplZ\")
<\/p>\n![\"New](\"https:\/\/www.simoahava.com\/images\/2018\/02\/beanstalk-new-web-server.jpg#ZgotmplZ\")
<\/p>\n