There are thousands upon thousands of bots, crawlers, spiders, and other creepy-crawlies out there doing nothing but crawling through websites and harvesting the content within for whatever purposes they have been fine-tuned to. While Google Analytics<\/a> provides a bot filtering feature<\/a> to filter out \u201cspam\u201d and \u201cbot traffic\u201d from views, this is far from comprehensive enough to tackle all instances of bot traffic that might enter the site.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n You might have noticed bot traffic in your data even if you have bot filtering toggled on. For instance, such traffic could be characterized as follows:<\/p>\n Luckily, there are better ways to identify this traffic than looking at noisy Google Analytics data. Google has introduced reCAPTCHA v3<\/a> which lets you verify traffic without any user interaction required. This is done with a simple JavaScript API combined with a server-side verification endpoint.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n In this article, Sebastian Pospischil<\/strong> and Philipp Schneider<\/strong> from Resolution Media<\/a> will share the solution with you. They\u2019ll show you how to build a simple PHP API on your Apache web server to handle the requests from reCAPTCHA.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n I\u2019ll contribute by building the same thing with Node.js<\/strong>. Remember that you can have both (Apache and Node.js) running in Google Cloud<\/a> using e.g. the App Engine<\/a> standard environment. This is helpful if you do not have the resources or capabilities to run something like this on your own web servers.<\/p>\n Huge thanks to Philipp and Sebastian for sharing this nugget of wisdom with this blog!<\/p>\n \n \n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n <\/span><\/p>\n To start with, you\u2019ll need to generate a (free) site key<\/strong> and a (free) secret key<\/strong> for your reCAPTCHA API use. Head on over to https:\/\/www.google.com\/recaptcha\/admin\/create<\/a>, and register a new site to use with reCAPTCHA v3. Make sure you enter all the domains from which these verification requests will be sent.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Once you\u2019ve added everything and checked all the necessary boxes, click SUBMIT<\/strong> to get your keys.<\/p>\n <\/p>\n <\/a><\/p>\n<\/div>\n Copy the keys, as you\u2019ll need them soon when configuring the Custom HTML tag in Google Tag Manager<\/a> as well as the server-side endpoint.<\/p>\n In this chapter, Philipp and Sebastian will walk you through the steps of creating a PHP endpoint for verifying the reCAPTCHA requests. They\u2019ll also show you how to create the Custom HTML tag in Google Tag Manager for loading the reCAPTCHA JavaScript API and ascertaining the \u201cbot-ness\u201d of the user.<\/p>\n First things first – let\u2019s setup Google Tag Manager. In GTM, create a new Custom HTML Tag<\/strong>.<\/p>\n You can set this tag to fire on whatever trigger you wish, though a trigger that fires when the page loads (e.g. All Pages, DOM Ready, or Window Loaded) is recommended.<\/p>\n Make sure you edit the following:<\/p>\n This block of code runs the reCAPTCHA JavaScript API in the user\u2019s browser, and sends the tokenized result to the PHP endpoint you\u2019ll configure in the next chapter. When it gets a response, the score and success state are pushed into <\/p>\n <\/a><\/p>\n<\/div>\n The very first code block in the tag hides the reCAPTCHA badge from showing up on the site. Remember to read this part of the FAQ<\/a> to learn how to modify your site to allow the hiding of the badge.<\/p>\n Create a file NOTE!<\/strong> You can use whatever path and filename you want for the PHP file – just remember to update the Custom HTML tag\u2019s HTTP request endpoint URL accordingly.<\/p>\n<\/blockquote>\n Again, make sure you replace the This PHP code takes the response sent by Google Tag Manager, validates it against the reCAPTCHA verification API, and then sends the result object in the response, which will, in turn, be pushed into The code also writes a cookie in the HTTP response. This cookie is named In this chapter, Simo will show you how to do the same as above by using a Node.js<\/a> application instead.<\/p>\n The main difference (apart from the tech stack) to the PHP solution is that we\u2019ll log errors as well. They are sent as the value of the This is what the Custom HTML tag looks like:<\/p>\n Remember to change the The This setup is different from the PHP request. Here, we use a GET request, and we parse the response, pushing both the success and error into Also, since we load the JavaScript API asynchronously, we create a global callback method The end result is the same as with the PHP endpoint – a The Node.js<\/strong> application example here is the simplest possible thing you can build. For it to work, you need to have Node.js installed<\/a> on in the environment where you want to run the endpoint (for example, a web server running on your site or in the cloud).<\/p>\n Create an empty directory, and initialize a new This installs the required dependencies to run a simple express.js<\/strong> web server as well as the files required to run the reCAPTCHA middleware and some cookie parsing logic.<\/p>\n Then, create a file named server.js<\/strong>, open it for editing, and add the following code within:<\/p>\n![\"Bot](\"https:\/\/www.simoahava.com\/images\/2019\/09\/bot-traffic-in-ga.jpg#ZgotmplZ\")
<\/p>\n\n
![\"reCAPTCHA](\"https:\/\/www.simoahava.com\/images\/2019\/10\/recaptcha-bot-traffic.jpg#ZgotmplZ\")
<\/p>\n![\"Philipp](\"https:\/\/www.simoahava.com\/images\/2019\/10\/phil-sebastian.jpg#ZgotmplZ\")
<\/p>\n
\n X<\/span><\/p>\n
\n The Simmer Newsletter<\/strong>\n <\/p>\nGetting started<\/h2>\n
![\"Enable](\"https:\/\/www.simoahava.com\/images\/2019\/09\/enable-recaptcha.jpg#ZgotmplZ\")
<\/p>\n![\"register](\"https:\/\/www.simoahava.com\/images\/2019\/09\/register-keys.jpg#ZgotmplZ\")
<\/p>\nPHP endpoint<\/h2>\n
Custom HTML tag<\/h3>\n
style>\n \/* hides the recaptcha badge *\/<\/span>\n .grecaptcha-badge<\/span> {\n visibility<\/span>: hidden<\/span> !important<\/span>;\n }\nstyle<\/span>>\nscript src<\/span>=\"https:\/\/www.google.com\/recaptcha\/api.js?render=_reCAPTCHA_site_key_\"<\/span>>script<\/span>>\nscript>\n grecaptcha.ready(function<\/span>() {\n grecaptcha.execute('_reCAPTCHA_site_key_'<\/span>, {\n action: 'homepage'<\/span>\n }).then(function<\/span>(token) {\n var<\/span> xhr = new<\/span> XMLHttpRequest();\n xhr.onload = function<\/span>() {\n if<\/span> (xhr.response !== 'noChange'<\/span>) {\n var<\/span> greResult = JSON.parse(xhr.response);\n window<\/span>.dataLayer.push({\n event: 'recaptcha'<\/span>, \n recaptchaAnswer: greResult.success,\n recaptchaScore: greResult.score\n });\n }\n };\n xhr.open('POST'<\/span>, '\/gtm\/recaptcha.php'<\/span>, true<\/span>); \/\/replace this with URL to your PHP fil\n<\/span> xhr.setRequestHeader('Content-type'<\/span>, 'application\/x-www-form-urlencoded'<\/span>);\n xhr.send('token='<\/span> + token + '&action=homepage'<\/span>);\n });\n });\nscript<\/span>><\/code><\/pre>\n<\/div>\n\n
_reCAPTCHA_site_key<\/code> string on lines 7 and 10 with the actual site key<\/strong> you copied from the reCAPTCHA admin console in the previous chapter.<\/li>\ndataLayer<\/code>.<\/p>\n![\"reCAPTCHA](\"https:\/\/www.simoahava.com\/images\/2019\/09\/recaptcha-badge.jpg#ZgotmplZ\"){kind=icon}
<\/p>\nPHP file for your server<\/h3>\n
recaptcha.php<\/code> with the following code, and upload it to all the domains you wish to validate reCAPTCHA requests on. Place it in a subdirectory named \/gtm\/<\/code>.<\/p>\n\n
\n\t\/\/ reCaptcha info\n<\/span> $url<\/span> = 'https:\/\/www.google.com\/recaptcha\/api\/siteverify'<\/span>; \n $secret<\/span> = \"_reCAPTCHA_secret_key_\"<\/span>;\n\t$remoteip<\/span> = $_SERVER<\/span>['REMOTE_ADDR'<\/span>];\n\t\/\/ Form info\n<\/span>\t$action<\/span> = $_POST<\/span>['action'<\/span>];\n\t$response<\/span> = $_POST<\/span>['token'<\/span>];\n\t\/\/ Botscore\n<\/span>\t$botscore<\/span> = $_COOKIE<\/span>['_rbs'<\/span>];\n\n\t\/\/ Curl Request\n<\/span>\t$curl<\/span> = curl_init();\n\tcurl_setopt($curl<\/span>, CURLOPT_URL, $url<\/span>);\n\tcurl_setopt($curl<\/span>, CURLOPT_POST, true<\/span>);\n\tcurl_setopt($curl<\/span>, CURLOPT_RETURNTRANSFER, 1<\/span>);\n\tcurl_setopt($curl<\/span>, CURLOPT_POSTFIELDS, array<\/span>(\n\t\t'secret'<\/span> => $secret<\/span>,\n\t\t'remoteip'<\/span> => $remoteip<\/span>,\n\t\t'action'<\/span> => $action<\/span>,\t\t\n\t\t'response'<\/span> => $response<\/span>\n\t));\n\t$curlData<\/span> = curl_exec($curl<\/span>);\n\tcurl_close($curl<\/span>);\n\t$curlJson<\/span> = json_decode($curlData<\/span>, true<\/span>);\n\t\n\t\/\/defining the answer\n<\/span>\t$answer<\/span> = $curlData<\/span>;\n\t\n\t\/\/refresh the cookie\n<\/span>\tsetcookie('_rbs'<\/span>, $curlJson<\/span>['score'<\/span>], time()+1800<\/span>, '\/'<\/span>,''<\/span>, 0<\/span>); \n\t\n\t\/\/ only fire $answer if botscore cookie is not set or different from acutal score\n<\/span>\tif<\/span> ($botscore<\/span> != $curlJson<\/span>['score'<\/span>]) {\n\t\techo<\/span> $answer<\/span>;\n\t} else<\/span> {\n\t\techo<\/span> \"noChange\"<\/span>;\n\t}\n?><\/span>\n<\/span><\/code><\/pre>\n<\/div>\n_reCAPTCHA_secret_key<\/code> on line 4 with the secret key<\/strong> you got from the admin console.<\/p>\ndataLayer<\/code> for your tags to use.<\/p>\n_rbs<\/code> and contains the score of the reCAPTCHA action. The results are only returned if the reCAPTCHA score has changed in the last 30 minutes. If the results have not changed, the response will be \"noChange\"<\/code>, which blocks the dataLayer.push()<\/code> from taking place.<\/p>\nNode.js endpoint<\/h2>\n
Custom HTML tag<\/h3>\n
recaptchaScore<\/code> key in dataLayer<\/code>, so in your report you\u2019ll end up with a mix of valid scores and error texts, but this should be easy to filter in your tables.<\/p>\nstyle>\n \/* hides the recaptcha badge *\/<\/span>\n .grecaptcha-badge<\/span> {\n visibility<\/span>: hidden<\/span> !important<\/span>;\n }\nstyle<\/span>>\nscript src<\/span>=\"https:\/\/www.google.com\/recaptcha\/api.js?render=_reCAPTCHA_site_key&onload=__recaptchaCallback\"<\/span> async<\/span> defer<\/span>>script<\/span>>\nscript>\n window<\/span>.__recaptchaCallback = function<\/span>() {\n if<\/span> (window<\/span>.grecaptcha) {\n window<\/span>.grecaptcha.execute('_reCAPTCHA_site_key'<\/span>, {action: 'homepage'<\/span>}).then(function<\/span>(token) {\n var<\/span> url = 'https:\/\/mydomain.com:3000\/recaptcha\/'<\/span>;\n var<\/span> xhr = new<\/span> XMLHttpRequest();\n xhr.open('GET'<\/span>, url + '?g-recaptcha-response='<\/span> + token, true<\/span>);\n xhr.onreadystatechange = function<\/span>(data) {\n if<\/span> (this<\/span>.readyState === XMLHttpRequest.DONE && this<\/span>.status === 200<\/span>) {\n var<\/span> responseJson = JSON.parse(xhr.response);\n window<\/span>.dataLayer.push({\n event: 'recaptcha'<\/span>,\n recaptchaAnswer: responseJson.status,\n recaptchaScore: responseJson.recaptcha.score\n });\n }\n }\n xhr.send();\n });\n }\n };\nscript<\/span>><\/code><\/pre>\n<\/div>\n_reCAPTCHA_site_key<\/code> on lines 7 and 11.<\/p>\nurl<\/code> variable on line 12 needs to point to your Node.js application.<\/p>\ndataLayer<\/code> using a success callback for the HTTP request.<\/p>\n__recaptchaCallback<\/code>, whose name we pass in the script loader as a parameter (&onload<\/code>).<\/p>\ndataLayer.push()<\/code> event with the reCAPTCHA status and score.<\/p>\nNode.js application<\/h3>\n
npm<\/code> project with npm init<\/code>. After following through the setup steps, run:<\/p>\n$ npm install express express-recaptcha cookie-parser --save<\/code><\/pre>\n<\/div>\nconst<\/span> express = require(