On New Year\u2019s Eve 2018, I published an article which instructed how to scrape pages of a site and write the results into Google BigQuery<\/a>. I considered it to be a cool way to build your own web scraper, as it utilized the power and scale of the Google Cloud platform combined with the flexibility of a headless crawler<\/a> built on top of Puppeteer<\/a>.<\/p>\n In today\u2019s article, I\u2019m revisiting this solution in order to share with you its latest version, which includes a feature that you might find extremely useful when auditing the cookies<\/a> that are dropped on your site.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n The purpose of this exercise is to list all<\/em> the cookies, both 1st party and 3rd party<\/a>, the crawler encounters when automatically following links within your site. This way you have an idea of what cookie storage is actually being utilized on your site.<\/p>\n You can use this information to proactively audit and annotate the cookie use on your site, which is helpful in case you are striving for compliance with EU\u2019s cookie regulation<\/a>, for example.<\/p>\n \n \n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n <\/span><\/p>\n This is the easy part: you follow the exact<\/strong> steps as outlined in the original article<\/a>. The only change you might want to do is set the config flag <\/p>\n <\/a><\/p>\n<\/div>\n Other than that small change, just follow the steps in the original article. Just to recap, you should have the following:<\/p>\n Once the virtual machine fires up, assuming you\u2019ve followed the instructions meticulously, you\u2019ll end up with a BigQuery table that collects a stream of scraped pages, together with the newly added cookie metadata as well.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n The crawler scans cookies in both 1st party and 3rd party requests. Cookie information is parsed for name<\/strong>, value<\/strong>, size<\/strong>, domain<\/strong>, path<\/strong>, expiration<\/strong>, HttpOnly<\/strong>, secure<\/strong>, and SameSite<\/strong>.<\/p>\n Once you have the data in the table, here are some BigQuery SQL queries you can run to make the most of the new information.<\/p>\n The first query is simple: it gets you the crawled URL together with all the cookies dropped on the site.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n This second query returns just the cookies, grouping similar cookies together. It\u2019s a handy way to get a list of all the distinct cookies dropped during the crawl. Each cookie with a domain<\/strong> namespace different from your own is a 3rd party cookie<\/strong> (unless you neglected to set the You can find a screenshot of the result at the very beginning of this article.<\/p>\n There are some caveats to this solution.<\/p>\n Dynamic sites that reveal navigation links only upon a click, or that load content with lazy-load, will need to be manually configured into a custom crawler<\/a> utilizing Puppeteer\u2019s Another problem is that there might be cookies that are set only upon the user interacting with the site. Prime example is a login event, or a conversion event that doesn\u2019t rely on a retargeting cookie (which would have fired with the page load). In these cases, a comprehensive cookie audit would need the crawler to be configured with these custom navigation paths, so that all cookies would be audited accordingly.<\/p>\n It\u2019s also possible that some vendors can detect crawlers and prevent their SDKs from dropping any cookies.<\/p>\n In any case, the solution described here should be a starting point for a more comprehensive storage audit. It doesn\u2019t tackle other forms of stateful storage (e.g. Hopefully this article inspires you to take a look at the web-scraper-gcp<\/a> project again, especially with the updated cookie crawling capabilities. I believe every single organization in the world should be exercising this type of oversight and governance in the name of accountability. You owe it to your site visitors. And, if your practices fall under legal regulation from e.g. GDPR or California\u2019s CPRA<\/a>, you should be very<\/em> interested in knowing what browser storage is utilized on your site.<\/p>\n The Let me know in the comments if you have questions or feedback about the solution – I\u2019m happy to help if I can!<\/p>\n<\/p><\/div>\n\n","protected":false},"excerpt":{"rendered":" On New Year\u2019s Eve 2018, I published an article which instructed how to scrape pages of a site and write the results into Google BigQuery. I considered it to be a cool way to build your own web scraper, as it utilized the power and scale of the Google Cloud platform combined with the flexibility […]<\/p>\n","protected":false},"author":1,"featured_media":87529,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12033],"tags":[16404,28608,14315,1531],"dealstore":[],"offerexpiration":[],"class_list":["post-87528","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","tag-audit","tag-bigquery","tag-cookie","tag-google"],"yoast_head":"\n![\"Scrape](\"https:\/\/www.simoahava.com\/images\/2020\/06\/scrape-cookies-bigquery.jpg#ZgotmplZ\")
<\/p>\n
\n X<\/span><\/p>\n
\n The Simmer Newsletter<\/strong>\n <\/p>\nHow to set it up<\/h2>\n
skipExternal<\/code> to true<\/code> in the config.json<\/code> file, which means that the crawler will no longer crawl external pages that are linked to from your site. This was originally done to get the HTTP status codes of external links, but as you\u2019re focusing on a cookie audit, external pages would just add to the confusion.<\/p>\n![\"Config](\"https:\/\/www.simoahava.com\/images\/2020\/06\/config-example.jpg#ZgotmplZ\")
<\/p>\n\n
gce-install.sh<\/code> script modified<\/a> with the URL to the config file in the storage bucket.<\/li>\n![\"BigQuery](\"https:\/\/www.simoahava.com\/images\/2020\/06\/bigquery-metadata.jpg#ZgotmplZ\")
<\/p>\nSample queries<\/h2>\n
SELECT<\/span>\n final_url,\n cookies\nFROM<\/span>\n `project.dataset.table<\/span>`<\/code><\/pre>\n<\/div>\n![\"All](\"https:\/\/www.simoahava.com\/images\/2020\/06\/all-cookies.jpg#ZgotmplZ\")
<\/p>\nskipExternal<\/code> flag to true<\/code> in the configuration step).<\/p>\nSELECT<\/span>\n c<\/span>.name,\n c<\/span>.domain<\/span>,\n c<\/span>.httpOnly,\n c<\/span>.secure,\n c<\/span>.session<\/span>,\n c<\/span>.sameSite\nFROM<\/span>\n `project.dataset.table<\/span>`,\n UNNEST<\/span>(cookies) AS<\/span> c<\/span>\nGROUP<\/span> BY<\/span>\n 1<\/span>, 2<\/span>, 3<\/span>, 4<\/span>, 5<\/span>, 6<\/span>\nORDER<\/span> BY<\/span>\n 1<\/span> ASC<\/span><\/code><\/pre>\n<\/div>\nCaveats<\/h2>\n
page<\/code> APIs<\/a>. It\u2019s not trivial to set up, as you\u2019ll basically need to add only<\/em> the links scraped from a dynamic navigation click (or lazy-load event) into the headless-chrome-crawler<\/code> queue to avoid duplication.<\/p>\nlocalStorage<\/code> or IndexedDB<\/code>), but it does give you an idea of what cookie storage the scripts and tags running on your site utilize.<\/p>\nSummary<\/h2>\n
headless-chrome-crawler<\/code> project hasn\u2019t been updated for a couple of years, but it still works. I\u2019m hoping to fork it one day and fix the dependency issues, which you might have noticed if running the solution locally.<\/p>\n