The brands are listening to the bloggers

After scanning my web server’s logs I found some unusual activity and decided to investigate further. I discovered that it seems that brands pay some internet companies to scan blog sites to see what things people are saying about them and their products.

I decided to take a look to see what kind of traffic I was getting to this website by doing a quick analysis of the web server’s log files. There are a few different kinds of ‘visitor’ that you may find visiting your website, and they each behave in different ways, so you get to recognise them after a while. They are:

• Search engine robots: You normally see the usual ‘robots’ from the big search engine companies like Google, Yahoo, Microsoft Live etc. The have a pattern of scanning a lot of pages in a fairly quick burst, then they return some days later and repeat the same scan process, in order to update their catalogue with any updates.
• Spammer robots: Spammers prey on blog sites so that they can post references to their dodgy websites for selling pharmaceuticals, porn, financial services, scams etc. They have various patterns, some sweep your whole site completely in one visit, but these are easy to spot due to the high traffic activity, so the subtler ones only try to post comment or trackback spam to about 4 or 5 pages a day. However, these leeches come back every day. The good news is that they often use a single IP address to spam from, and so you can simply ban that IP address from being able to access your web server by adding their IP address to your firewall’s blacklist. Then all attempts by them will fail, as all traffic from them is simply ignored by your firewall, so none of their requests even reach your http server. Some websites list the IP addresses these spammers use, and you can cut and paste these into your firewall configuration if you like. Very satisfying to see them not come back 🙂 Even more satisfying is to log all the network packets from them that your server drops (ignores). Then you feel like your effort has paid off. These spammer vermin sap energy from your server by wasting time and resources, so they have to be stopped. The main culprits that I discovered were coming from places like the Dominican Republic, China and Saudi Arabia. Not any more. If you find yourself adding too many IP addresses from a particular country, you can always add the whole country to your firewall blacklist, but I have resisted doing this so far.
• Then there are the genuine visitors, the humans 🙂

Well, today I found a new type of visitor. They use the IP addresses in the 142.166.x.x range (more than 65000 IP addresses!), mostly from 142.166.3.122 & 142.166.3.123, and their pattern of activity was quite different. I ran a ‘traceroute’ and ‘whois’ on them, and found that they appear to be an organisation called ‘Stentor National Integrated Communications Network’.

Here is some of the info on them from ‘whois’:

OrgName:    Stentor National Integrated Communications Network 
OrgID:      SNI1
Address:    One Brunswick Square
City:       Saint John
StateProv:  NB
PostalCode: E2L-4K2
Country:    CA

NetRange:   142.166.0.0 - 142.166.255.255 
CIDR:       142.166.0.0/16 
NetName:    ALIANT-TEL-142-166
NetHandle:  NET-142-166-0-0-1
Parent:     NET-142-0-0-0-0
NetType:    Direct Allocation
NameServer: OPAL.NBNET.NB.CA
NameServer: ONYX.NBNET.NB.CA
Comment:    
RegDate:    1992-08-26
Updated:    2002-12-04

Google found a couple of interesting pages like this one Who the heck is 142.166.3.122 and 142.166.3.123?

Stored within the User Agent field in the logs was a reference to “www.radian6.com/crawler”, so I decided to pay it a visit and found this information was provided by them:

Who is Radian6?

You have likely found you (sic) way to this page because you were curious about our crawlers, having seen them in your logs, and noted this page was present in the UserAgent string. Radian6 is a social media monitoring company that has developed a product offering specifically tailored to PR and marketing companies, that helps companies find and listen to conversations about their brands. You can find out more about our company at www.radian6.com.

If you have seen our crawlers visit your content, then that is because you have written about topics that are of interest to companies and brands that are using our system. This is good for you, as it means your voice is being heard, and the number of subscribers for your content has increased.

What is R6_FeedFetcher?

R6_FeedFetcher is our main RSS update crawler. It is responsible for checking RSS feeds periodically, looking for updates to your feed. The behavior of this agent should look much like any other RSS update agent. If you have concerns about how it is interacting with your content, or would like to have your content removed from our subscription list, please provide details and email them to support@radian6.com.

What is R6_CommentReader?

You may see R6_CommentReader visit one or more of the posts found in your RSS feed periodically for a number of days. This agent is responsible for reading comments left by your visitors to posts on your feed. In this regard, it should look similar to comment tracking services such as Commentful, CoComments, or Co.mments. It only checks content relevant to our clients, and the ones it does check follow heuristics as to how often and for how long it is checked.

Why follow comments? Social media is different than traditional media in that the ‘social’ aspect of the medium is what makes it such a powerful force. Comments left by an involved and engaged community form a conversation, and allow everyone to have a voice in that conversation. The Radian6 comment reading agent makes sure that these comments are seen by the brands and companies using our service.

If you are concerned with the behavior of this agent, or wish to have your site excluded from these checks, please provide details and email them to support@radian6.com.

So it would appear that this visitor is not something bad like a spammer. But I did find it interesting to think that brands pay these people to look for references to their products on bloggers’ websites. It seems that once they lock onto a web page of interest they keep returning to see what comments people post regarding your article. So it seems that the brands are keen to see what other people are saying to see if there is some kind of consensus, perhaps. Well, I decided to see which stories they were interested in and when they first started tracking these stories.

After checking archived logs, I can see that they first started scanning the site on 02 JAN 2008, coincidentally or not, when I first mentioned Panasonic, Sony & NEC products. They use multiple IP addresses, and they have taken an interest in tracking the following articles:

• How to enable your DVD player to play DivX format video from DVD+R discs (brands/products mentioned: Apple Mac Pro, Panasonic DMR-EH57, Sony DW-150A and NEC 4570)
• Apple RAM prices (brands/products mentioned: Apple Mac Pro memory and OWC memory)
• Why are my photos so bad? (brands/products mentioned: Nikon, Canon)
• Complex nature (brands/products mentioned: Nikon D200)
• Life on a hard drive (brands/products mentioned: Apple Mac OS X, generic RAID)
• Leopard: first impressions (brands/products mentioned: Apple Mac OS X)

After an initial sweep of the articles listed above, they then only check back for updates on the following articles:

1. How to enable your DVD player to play DivX format video from DVD+R discs (brands/products mentioned: Apple Mac Pro, Panasonic DMR-EH57, Sony DW-150A and NEC 4570)
2. Apple RAM prices (brands/products mentioned: Apple Mac Pro memory and OWC memory)

And I’m very happy they loved those two articles so much, as both products, although overall are fantastic products, have aspects that have intensely annoyed me. For example, Panasonic has annoyed me by not allowing their DMR-EH57 to play DivX format video from standard DVD+R discs (and in fact not DVD-RW & DVD+RW discs too!), and Apple has annoyed me by having such ridiculous high prices for Apple-branded memory for the Mac Pro computer, when compataible much cheaper alternatives exist.

While I’m on a roll, I also hate the Mac Pro’s noisy aluminium case as it is horribly, horribly noisy when you add more hard drives, due to hard drive vibrations causing the case to vibrate noisily. Apple need to add rubber or silicone grommets to dampen the HDD’s vibrations.

There, that’s told them, I feel better now 😉 So Radian6, you ‘social media monitoring’ company, pass that advice on to your paying customers please!

After reading more on the website of www.radian6.com, I found the following info:

“By using the Radian6 solution we are able to more efficiently target the ‘key influentials’ and social-media tastemakers in any given brand category,” expressed Chas Salmore, CEO of MWKS. “We are also able to track the success of our engagement strategies over time for complete analysis and comprehensive reporting to our numerous clients.”

Studies are showing that customers are increasingly discussing their experiences with brands online. Simultaneously, an increasing number of potential customers are researching online prior to purchase, taking into consideration the opinions of those who post material in various forms of social media.

“Brands are increasingly owned less by the institutions who make them and more by consumers who use them,” said Marcel LeBrun, CEO of Radian6. “MWKS understands this power shift and by using the advanced social media monitoring and analysis capabilities of the Radian6 solution they can quickly discover and participate in the very online conversations that are redefining and shaping these brands.”

In theory, this article itself might prove irresistible to their tracking robots too, as it repeats all the brands/products mentioned in the pages they are already tracking — I’ll post an update in a few days to see it they start to track this page too 🙂

However, if they start getting too funky with their probing, I might just add them to my firewall’s ignore list — after all, they aren’t paying me for all my traffic allowance that they are using, and there’s only a limited amount included in the package price 😉 Unless they feel like sharing some of that cash their clients are paying them… no, I didn’t think so 😉