Measuring Hidden Traffic not captured by Web Analytics tool

If your website consists of downloadable resources like whitepapers, case studies, article etc, then there are chances of people downloading them directly from search engines, and bypassing your web analytics tool tracking code (in case your tool uses javascript implementation)

So, the actual search engine traffic could be much higher as portrayed by analytics tool. This is a common problem faced by all analytics tool where implementation is based on either javascript getting triggered or the pixel beacon getting hit through the website.

The remedy to this problem is the server logs. You need to simply look into those entries where the referral (please request your hosting provider to enable this feature if referrals are not captured in the logs) is any of the search engines (google, yahoo, bing). Below are the simple steps you need to follow to make this happen.

1. Take all server logs (for specified time period) in a single folder.

2. Run 'Find Command' from the DOS (cmd.exe) to include only search engine and save the output file in csv format

eg. C:\>find "google" *.txt>yahoo.csv

This command will find all entries with yahoo and save in yahoo.csv

3. Similarly, take entries for yahoo and bing

4. Using Pivot tables, filter only entries with pdf or excel or mp4 (or any downloadable content your site has)

5. This will provide all clicks on search engines that your web analytics tool does not provide.

6. Further more, you can also filter out the search terms (included in 'q' parameter in referral string) as well site position in google (included in 'cd' parameter)

Do let me know your thoughts on this.