Last updated 18 Jan 2019: Added details about the free tier<\/strong> limitations, and showed how to avoid the Dataflow jobs auto-scaling<\/strong> out of control.<\/em><\/p>\n I\u2019m (still) a huge fan of Snowplow Analytics<\/a>. Their open-source, modular approach to DIY analytics pipelines has inspired me two write articles<\/a> about them, and to host a meetup<\/a> in Helsinki. In my previous Snowplow with Amazon Web Services guide<\/a>, I walked you through setting up a Snowplow pipeline using Amazon Web Services<\/a>. This time around, I\u2019m looking at the wondrous Google Cloud Platform<\/a>, for which Snowplow introduced support in an early 2018 release<\/a>.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n I won\u2019t be offering a comprehensive comparison between GCP and AWS, nor will I walk you through every single possible customization you could do when firing up the instances and building the pipeline. Those are left for you to discover by yourself, or to consult a data engineer who can help you with scale.<\/p>\n In fact, the setup I\u2019ll walk you through will be suboptimal in many places. It won\u2019t be the most robust setup for a large flow of data through the pipeline, but what it will provide you is the comprehensive list of steps you need to make to get things running.<\/p>\n Note that Snowplow also offers their own Snowplow Insights<\/a> service for setting up and managing the pipeline so that you can jump straight into data collection and analysis. I really<\/strong> recommend this service especially if you are unsure about how to set up the pipeline in an economical and scalable way. You might accrue a lot of extra costs if you don\u2019t know how to scale the required resources to suit what you actually need for efficient analysis.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n You can also follow the official Snowplow wiki<\/a>, if you wish. The steps are similar to what I go through in this guide, though I do think this guide will be better suited for you in case you are unfamiliar with how GCP works.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n My Powerpoint visualization skills just get better and better each passing year!<\/p>\n \n \n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n <\/span><\/p>\n This probably won\u2019t come as a surprise to you, but working with a cloud platform is not free of charge<\/strong>. If you\u2019re a first-time user, you should be eligible for the free trial<\/a> of $300 to be spent during the first 12 months. This is more than enough to build a pipeline and test it out over a number of days, but it won\u2019t get you far.<\/p>\n UPDATE 18 Jan 2019<\/strong>: The free tier only gives you 8 CPUs<\/strong> total<\/em> across your entire project. That means that you must keep count of all the instances you\u2019ll be using for the collector and for the ETL machine. Start with a low number, e.g. 1 or 2 CPUs per instance to see how Snowplow works. Once you have the budget, you can shoot for more ambitious setups.<\/p>\n<\/blockquote>\n <\/p>\n <\/a><\/p>\n<\/div>\n You\u2019ll need to keep a close eye on the billing dashboard<\/a> of your project. It helps you get an idea of how much cost you have factually accumulated to this day, and what the projected cost for the whole month will be.<\/p>\n To follow this guide, you will need:<\/p>\n A Google account<\/strong> – something to log into Google services with, such as your Google Mail ID.<\/p>\n<\/li>\n A credit card<\/strong> – you\u2019ll need to enable billing in your Google Cloud account to fire up some of the services we need.<\/p>\n<\/li>\n A custom domain name<\/strong> – this is optional, but it\u2019s necessary if you want to set up the tracker as a secure HTTPS endpoint. Without a custom domain name, you\u2019re forced to use HTTP only. You can get a cheap domain name from Google Domains<\/a>. You might<\/em> be able to set something up for free using Cloud Endpoints<\/a>, but this guide will not cover this.<\/p>\n<\/li>\n<\/ul>\n Good luck! Let me know in the comments if some part of this guide was particularly unclear.<\/p>\n Here we\u2019ll setup the GCP project, make sure you have the necessary resources enabled, and we\u2019ll also install the Google Cloud SDK so that you can interact with your project via your local terminal, too.<\/p>\n The first thing you\u2019ll need to do is set up a new Google Cloud project.<\/p>\n (1)<\/strong> Browse to https:\/\/console.cloud.google.com\/cloud-resource-manager\/<\/a>. Remember to login with the Google ID which you\u2019ll use to manage this project. Click CREATE PROJECT<\/strong>.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n If this is the first time you use GCP with this account, you might see an offer for free credit<\/strong>. Take it!<\/p>\n<\/blockquote>\n (2)<\/strong> Give the project a name.<\/p>\n For convenience, it might be good to change the Project ID<\/strong> to something more legible and easy to understand.<\/p>\n If you have Organizations<\/strong> and Locations<\/strong> (such as folders) set up for your Google Cloud account, choose the appropriate ones from the menu.<\/p>\n Also, if you already have a Billing Account<\/strong> set up, choose that as well from the respective drop-down menu.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Choose CREATE<\/strong> when you\u2019re done.<\/p>\n After clicking the button, GCP will do some loading and spinning for a while, after which you should see your new project dashboard. If you don\u2019t, make sure to select it from the project selector menu.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Congratulations! You\u2019ve created the project.<\/p>\n You\u2019ll need to create a billing account<\/strong> so that GCP can invoice you if necessary (sucks, I know).<\/p>\n (1)<\/strong> Browse to https:\/\/console.cloud.google.com\/billing<\/a>.<\/p>\n (2)<\/strong> Click Add billing account<\/strong>.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Here, follow the steps to add your credit card, or, if you\u2019re eligible for the free trial<\/strong>, to activate your free trial.<\/p>\n Make sure you follow the prompts to link your Snowpow project to the Billing account you just created.<\/p>\n Next, you need to enable the services and APIs we\u2019ll need to get started. These steps let you set up the collector. When you move to the enrich and BigQuery load steps<\/a>, you\u2019ll need to enable additional services.<\/p>\n (1)<\/strong> Browse to https:\/\/console.cloud.google.com\/apis\/library<\/a>.<\/p>\n You should see a search bar, so start by searching for Compute Engine API<\/strong> and click the relevant result. We\u2019ll need the Compute Engine to fire up our virtual machines on which the Snowplow collector will reside.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Upon entering the Compute Engine API page, simply click the blue ENABLE<\/strong> button at the top.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Once it\u2019s done, you should see something like this where the Enable<\/strong> button used to be:<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Now, follow these exact same steps for the Cloud Pub\/Sub API<\/strong>. Pub\/Sub (for Pub<\/strong>lisher\/Sub<\/strong>scriber) is a real-time message queue we\u2019ll use to process the data fed into the Snowplow pipeline.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Once you\u2019re done enabling these services, you\u2019re ready to move onto the next step!<\/p>\n The next thing you\u2019ll want to do is to install the Google Cloud SDK locally in your machine. It\u2019s very convenient because it lets you access your GCP project from the command line, and it lets you test some of the services from your local machine rather than having to find the appropriate paths through the often confusing UI.<\/p>\n To install the SDK, follow the steps for your platform, starting from here<\/a>.<\/p>\n Once you\u2019ve installed the SDK, you should be able to run these commands in your terminal \/ shell:<\/p>\n The first command logs you in with Google Cloud using the Google Account you choose in the web prompt. The second command points the current <\/p>\n <\/a><\/p>\n<\/div>\n The final initialization step is to setup a service account<\/strong>. A service account is basically a Google Cloud account which has full access to your GCP services and resources. It\u2019s a necessary step if you want to run services on GCP programmatically rather than with your own, personal Google account.<\/p>\n Note that when you enable Compute Engine API<\/strong> for your project, a Compute Engine default service account<\/strong> is created for you automatically. the steps below really only apply if you wanted to create a different service account for your pipeline.<\/p>\n<\/blockquote>\n (1)<\/strong> Browse to https:\/\/console.cloud.google.com\/apis\/credentials<\/a>.<\/p>\n (2)<\/strong> Make sure you have the correct project selected in the project selector menu.<\/p>\n (3)<\/strong> Click the blue Create credentials<\/strong> selector, and choose Service account key<\/strong> from the menu.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Since you\u2019ve enabled the Compute Engine API<\/strong>, you should be able to choose the default GCE service account from the drop-down.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Keep the JSON<\/strong> option selected, and choose Create<\/strong>.<\/p>\n The browser should automatically download the JSON file, so make sure you find it in your local files and store it securely<\/strong>.<\/p>\n At this point, it would be a good idea to create a local folder for this entire Snowplow experiment<\/strong>. In that local folder, you can store this service account key and any temporary configuration files you\u2019ll work on, and other stuff such as executables and binaries you\u2019ll eventually upload to the GCP, if necessary.<\/p>\n<\/blockquote>\n Once you\u2019ve created the service account you are ready to move on. All the initial steps have now been completed, congratulations!<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Snowplow uses a collector<\/strong> as the endpoint of your tracker requests. This collector resides in a Compute Engine instance (a virtual machine, basically), and it processes the requests you send to the endpoint.<\/p>\n These requests are then published into Pub\/Sub topics<\/strong>, from where they are then passed on further down the pipeline for enrichment and parsing. Pub\/Sub is essentially a real-time messaging pipeline, which collects messages in topics<\/strong>, which are then available for subscriptions<\/strong> to access.<\/p>\n Make sure you check out Snowplow\u2019s official wiki<\/a> for more information on this part of the pipeline.<\/p>\n The first thing you\u2019ll need to do is create topics<\/strong> in the Pub\/Sub you\u2019ve enabled for your project.<\/p>\n (1)<\/strong> Browse to https:\/\/console.cloud.google.com\/cloudpubsub\/topicList<\/a>.<\/p>\n (2)<\/strong> Make sure you have the correct project selected in the project selector.<\/p>\n (3)<\/strong> Click the blue Create a topic<\/strong> button.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n (4)<\/strong> Type <\/p>\n <\/a><\/p>\n<\/div>\n (5)<\/strong> Click CREATE TOPIC<\/strong> and type <\/p>\n <\/a><\/p>\n<\/div>\n Snowplow uses these two topics to filter out hits and requests that validate ( Next, you can create a subscription<\/strong> with which you can test the stream. This is optional, but might be a good idea if you want to see how the pipeline is working.<\/p>\n (6)<\/strong> Click the (7)<\/strong> Choose CREATE SUBSCRIPTION<\/strong> from the top navigation.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n (8)<\/strong> Give the subscription a name ( (9)<\/strong> Leave the other settings as they are, and click CREATE<\/strong>.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Now that you\u2019ve created the Pub\/Sub topics, it\u2019s time to step into one of the more complicated steps of this guide: setting up the collector itself<\/strong>.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Snowplow uses a collector written in Scala<\/strong> for processing the requests sent from your website. These records are parsed by the Scala Stream Collector<\/strong> and then distributed into the Pub\/Sub topics you created in the previous chapter.<\/p>\n We\u2019ll start with a simple HTTP endpoint<\/strong> using the default IP address assigned to the Compute Engine instance you\u2019ll spin up. This is only to test that the whole pipeline works. You\u2019ll want to configure your own custom domain name with an HTTPS endpoint<\/strong><\/a> for the actual tracker!<\/p>\n In the local directory where you\u2019re storing all your project files (such as the service account credentials you created earlier), create a new file named At this time, the only line you need to edit is the one with Keep this file at hand, because you\u2019ll need to make some changes to it soon.<\/p>\n You are now ready to start a Compute Engine instance.<\/p>\n NOTE! Because this is just a test – remember to shut down the instance as soon as you\u2019re done so that you don\u2019t accumulate extra costs.<\/p>\n<\/blockquote>\n (1)<\/strong> Browse to https:\/\/console.cloud.google.com\/compute\/instances<\/a> and make sure your project is selected in the project selector.<\/p>\n (2)<\/strong> Click Create<\/strong>.<\/p>\n![\"Snowplow](\"https:\/\/www.simoahava.com\/images\/2018\/11\/snowplow-gcp.jpg#ZgotmplZ\")
<\/p>\n![\"Snowplow](\"https:\/\/www.simoahava.com\/images\/2018\/12\/snowplow-insights.jpg#ZgotmplZ\")
<\/p>\n![\"Full](\"https:\/\/www.simoahava.com\/images\/2018\/12\/full-process.jpg#ZgotmplZ\")
<\/p>\n
\n X<\/span><\/p>\n
\n The Simmer Newsletter<\/strong>\n <\/p>\nWord of warning<\/h2>\n
\n
![\"Billing](\"https:\/\/www.simoahava.com\/images\/2018\/12\/billing-dashboard.jpg#ZgotmplZ\")
<\/p>\nWhat you\u2019ll need<\/h2>\n
\n
First steps<\/h2>\n
Step 1: Set up a new project<\/h3>\n
![\"Create](\"https:\/\/www.simoahava.com\/images\/2018\/11\/gcp-create-project.jpg#ZgotmplZ\")
<\/p>\n\n
![\"Create](\"https:\/\/www.simoahava.com\/images\/2018\/11\/create-new-project.jpg#ZgotmplZ\")
<\/p>\n![\"GCP](\"https:\/\/www.simoahava.com\/images\/2018\/11\/gcp-dashboard.jpg#ZgotmplZ\")
<\/p>\nStep 2: Enable Billing<\/h3>\n
![\"Add](\"https:\/\/www.simoahava.com\/images\/2018\/11\/add-billing-account.jpg#ZgotmplZ\")
<\/p>\nStep 3: Enable the required services<\/h3>\n
![\"Compute](\"https:\/\/www.simoahava.com\/images\/2018\/11\/compute-engine-api.jpg#ZgotmplZ\")
<\/p>\n![\"Enable](\"https:\/\/www.simoahava.com\/images\/2018\/11\/compute-engine-api-enable.jpg#ZgotmplZ\")
<\/p>\n![\"Compute](\"https:\/\/www.simoahava.com\/images\/2018\/11\/compute-engine-api-enabled.jpg#ZgotmplZ\")
<\/p>\n![\"Enable](\"https:\/\/www.simoahava.com\/images\/2018\/11\/enable-pub-sub-api.jpg#ZgotmplZ\")
<\/p>\nStep 4: Install the Google Cloud SDK<\/h3>\n
$ gcloud auth login\n$ gcloud config set<\/span> project PROJECT_ID<\/code><\/pre>\n<\/div>\ngcloud<\/code> setup to the project ID you\u2019ll give (replace PROJECT_ID<\/code> with the ID you configured when you created the project). The end result should be something like this:<\/p>\n![\"Gcloud](\"https:\/\/www.simoahava.com\/images\/2018\/11\/gcloud-set-project.jpg#ZgotmplZ\")
<\/p>\nStep 5: Setup a service account<\/h3>\n
\n
![\"Create](\"https:\/\/www.simoahava.com\/images\/2018\/11\/create-credentials.jpg#ZgotmplZ\")
<\/p>\n![\"Default](\"https:\/\/www.simoahava.com\/images\/2018\/11\/default-service-account.jpg#ZgotmplZ\")
<\/p>\n\n
Setting up the Pub\/Sub topics<\/h2>\n
![\"Pub\/sub](\"https:\/\/www.simoahava.com\/images\/2018\/12\/pubsub-process.jpg#ZgotmplZ\")
<\/p>\nStep 1: Create the Pub\/Sub topics<\/h3>\n
![\"create](\"https:\/\/www.simoahava.com\/images\/2018\/11\/create-a-pubsub-topic.jpg#ZgotmplZ\")
<\/p>\ngood<\/code> after the project path as in the screenshot below, and click CREATE<\/strong>.<\/p>\n![\"Good](\"https:\/\/www.simoahava.com\/images\/2018\/11\/good-topic.jpg#ZgotmplZ\")
<\/p>\nbad<\/code> as the name of the new topic, and then click CREATE<\/strong>.<\/p>\n![\"Bad](\"https:\/\/www.simoahava.com\/images\/2018\/11\/create-bad-topic.jpg#ZgotmplZ\")
<\/p>\ngood<\/code> stream) and those that have issues and errors (bad<\/code> stream).<\/p>\nStep 2: Create a subscription (optional)<\/h3>\n
good<\/code> topic to enter its configuration page.<\/p>\n![\"Create](\"https:\/\/www.simoahava.com\/images\/2018\/11\/create-subscription.jpg#ZgotmplZ\")
<\/p>\ntest-good<\/code>).<\/p>\n![\"Test](\"https:\/\/www.simoahava.com\/images\/2018\/11\/test-good-subscription.jpg#ZgotmplZ\")
<\/p>\nCreate an HTTP endpoint with the Scala Stream Collector<\/h2>\n
![\"Pub\/sub](\"https:\/\/www.simoahava.com\/images\/2018\/12\/collector-process.jpg#ZgotmplZ\")
<\/p>\nStep 1: Create the config file<\/h3>\n
application.config<\/code>, and copy-paste the contents of this sample config<\/a> within.<\/p>\ngoogleProjectId = your-project-id<\/code>. Change your-project-id<\/code> to your actual Google Cloud project ID.<\/p>\nStep 2: Fire up a GCE instance<\/h3>\n
\n