The brands are listening to the bloggers

After scanning my web server’s logs I found some unusual activity and decided to investigate further. I discovered that it seems that brands pay some internet companies to scan blog sites to see what things people are saying about them and their products.

I decided to take a look to see what kind of traffic I was getting to this website by doing a quick analysis of the web server’s log files. There are a few different kinds of ‘visitor’ that you may find visiting your website, and they each behave in different ways, so you get to recognise them after a while. They are:

  • Search engine robots: You normally see the usual ‘robots’ from the big search engine companies like Google, Yahoo, Microsoft Live etc. The have a pattern of scanning a lot of pages in a fairly quick burst, then they return some days later and repeat the same scan process, in order to update their catalogue with any updates.
  • Spammer robots: Spammers prey on blog sites so that they can post references to their dodgy websites for selling pharmaceuticals, porn, financial services, scams etc. They have various patterns, some sweep your whole site completely in one visit, but these are easy to spot due to the high traffic activity, so the subtler ones only try to post comment or trackback spam to about 4 or 5 pages a day. However, these leeches come back every day. The good news is that they often use a single IP address to spam from, and so you can simply ban that IP address from being able to access your web server by adding their IP address to your firewall’s blacklist. Then all attempts by them will fail, as all traffic from them is simply ignored by your firewall, so none of their requests even reach your http server. Some websites list the IP addresses these spammers use, and you can cut and paste these into your firewall configuration if you like. Very satisfying to see them not come back 🙂 Even more satisfying is to log all the network packets from them that your server drops (ignores). Then you feel like your effort has paid off. These spammer vermin sap energy from your server by wasting time and resources, so they have to be stopped. The main culprits that I discovered were coming from places like the Dominican Republic, China and Saudi Arabia. Not any more. If you find yourself adding too many IP addresses from a particular country, you can always add the whole country to your firewall blacklist, but I have resisted doing this so far.
  • Then there are the genuine visitors, the humans 🙂

Well, today I found a new type of visitor. They use the IP addresses in the 142.166.x.x range (more than 65000 IP addresses!), mostly from 142.166.3.122 & 142.166.3.123, and their pattern of activity was quite different. I ran a ‘traceroute’ and ‘whois’ on them, and found that they appear to be an organisation called ‘Stentor National Integrated Communications Network’.

Here is some of the info on them from ‘whois’:

OrgName:    Stentor National Integrated Communications Network 
OrgID:      SNI1
Address:    One Brunswick Square
City:       Saint John
StateProv:  NB
PostalCode: E2L-4K2
Country:    CA

NetRange:   142.166.0.0 - 142.166.255.255 
CIDR:       142.166.0.0/16 
NetName:    ALIANT-TEL-142-166
NetHandle:  NET-142-166-0-0-1
Parent:     NET-142-0-0-0-0
NetType:    Direct Allocation
NameServer: OPAL.NBNET.NB.CA
NameServer: ONYX.NBNET.NB.CA
Comment:    
RegDate:    1992-08-26
Updated:    2002-12-04

Google found a couple of interesting pages like this one Who the heck is 142.166.3.122 and 142.166.3.123?

Stored within the User Agent field in the logs was a reference to “www.radian6.com/crawler”, so I decided to pay it a visit and found this information was provided by them:

Who is Radian6?

You have likely found you (sic) way to this page because you were curious about our crawlers, having seen them in your logs, and noted this page was present in the UserAgent string. Radian6 is a social media monitoring company that has developed a product offering specifically tailored to PR and marketing companies, that helps companies find and listen to conversations about their brands. You can find out more about our company at www.radian6.com.

If you have seen our crawlers visit your content, then that is because you have written about topics that are of interest to companies and brands that are using our system. This is good for you, as it means your voice is being heard, and the number of subscribers for your content has increased.

What is R6_FeedFetcher?

R6_FeedFetcher is our main RSS update crawler. It is responsible for checking RSS feeds periodically, looking for updates to your feed. The behavior of this agent should look much like any other RSS update agent. If you have concerns about how it is interacting with your content, or would like to have your content removed from our subscription list, please provide details and email them to support@radian6.com.

What is R6_CommentReader?

You may see R6_CommentReader visit one or more of the posts found in your RSS feed periodically for a number of days. This agent is responsible for reading comments left by your visitors to posts on your feed. In this regard, it should look similar to comment tracking services such as Commentful, CoComments, or Co.mments. It only checks content relevant to our clients, and the ones it does check follow heuristics as to how often and for how long it is checked.

Why follow comments? Social media is different than traditional media in that the ‘social’ aspect of the medium is what makes it such a powerful force. Comments left by an involved and engaged community form a conversation, and allow everyone to have a voice in that conversation. The Radian6 comment reading agent makes sure that these comments are seen by the brands and companies using our service.

If you are concerned with the behavior of this agent, or wish to have your site excluded from these checks, please provide details and email them to support@radian6.com.

So it would appear that this visitor is not something bad like a spammer. But I did find it interesting to think that brands pay these people to look for references to their products on bloggers’ websites. It seems that once they lock onto a web page of interest they keep returning to see what comments people post regarding your article. So it seems that the brands are keen to see what other people are saying to see if there is some kind of consensus, perhaps. Well, I decided to see which stories they were interested in and when they first started tracking these stories.

After checking archived logs, I can see that they first started scanning the site on 02 JAN 2008, coincidentally or not, when I first mentioned Panasonic, Sony & NEC products. They use multiple IP addresses, and they have taken an interest in tracking the following articles:

After an initial sweep of the articles listed above, they then only check back for updates on the following articles:

  1. How to enable your DVD player to play DivX format video from DVD+R discs (brands/products mentioned: Apple Mac Pro, Panasonic DMR-EH57, Sony DW-150A and NEC 4570)
  2. Apple RAM prices (brands/products mentioned: Apple Mac Pro memory and OWC memory)

And I’m very happy they loved those two articles so much, as both products, although overall are fantastic products, have aspects that have intensely annoyed me. For example, Panasonic has annoyed me by not allowing their DMR-EH57 to play DivX format video from standard DVD+R discs (and in fact not DVD-RW & DVD+RW discs too!), and Apple has annoyed me by having such ridiculous high prices for Apple-branded memory for the Mac Pro computer, when compataible much cheaper alternatives exist.

While I’m on a roll, I also hate the Mac Pro’s noisy aluminium case as it is horribly, horribly noisy when you add more hard drives, due to hard drive vibrations causing the case to vibrate noisily. Apple need to add rubber or silicone grommets to dampen the HDD’s vibrations.

There, that’s told them, I feel better now 😉 So Radian6, you ‘social media monitoring’ company, pass that advice on to your paying customers please!

After reading more on the website of www.radian6.com, I found the following info:

“By using the Radian6 solution we are able to more efficiently target the ‘key influentials’ and social-media tastemakers in any given brand category,” expressed Chas Salmore, CEO of MWKS. “We are also able to track the success of our engagement strategies over time for complete analysis and comprehensive reporting to our numerous clients.”

Studies are showing that customers are increasingly discussing their experiences with brands online. Simultaneously, an increasing number of potential customers are researching online prior to purchase, taking into consideration the opinions of those who post material in various forms of social media.

“Brands are increasingly owned less by the institutions who make them and more by consumers who use them,” said Marcel LeBrun, CEO of Radian6. “MWKS understands this power shift and by using the advanced social media monitoring and analysis capabilities of the Radian6 solution they can quickly discover and participate in the very online conversations that are redefining and shaping these brands.”

In theory, this article itself might prove irresistible to their tracking robots too, as it repeats all the brands/products mentioned in the pages they are already tracking — I’ll post an update in a few days to see it they start to track this page too 🙂

However, if they start getting too funky with their probing, I might just add them to my firewall’s ignore list — after all, they aren’t paying me for all my traffic allowance that they are using, and there’s only a limited amount included in the package price 😉 Unless they feel like sharing some of that cash their clients are paying them… no, I didn’t think so 😉

Join the conversation

25 Comments

  1. After publishing my own blog this morning and not a blog type person to begin with, I had noticed in a few hours I had a subscriber. It turned out to be this radian6 crawler. putting that in Google I ran across your article. Thanks, it explains allot! Now I know how to get a companies attention.

    Larry B. Overton
    Cold Sweat-Chicago LLC
    http://www.coldsweat-chicago.com

  2. What a Big Brother world. At least the birds are still playing outside in the sun.. (on Malta they’re often not so lucky).

  3. Sorry … but I’m going to block any bot that doesn’t adhere to the robots exclusion standard.

    For example, why can/should I have to contact R6_ via email when I’ve already got a universal disallow on one of my websites they’re hammering?

  4. Thank you for that great reading

    I thought i was being hacked and was key scrambling and password changing and ip blocking like crazy

    well i see they like my stuff so i’ll give them something to help push my custom silk neckties

    using bird feather images

    damb who would have thought a big brand would be looking at me a small time company

    well maybe i’ll be getting bigger soon or they just want to steal my ideas

    most likely steal my ideas

    tie-guy

  5. Just found your page after checking an ip from Stentor National Integrated Communications Network, To be honest at first look at that name its looks very dodgy like oh -_- someone importants reading my blog and possably might not like it? lol or FBI or something irritable, Not that im in that business of causeing that much trouble!.

    I am supprised they found me I have only had wordpress up a few days lol

    But thank you for clearing up by posting this and cleared up my worrys

    Regards
    FurPortal Editor
    Steam

  6. Stentor National Integrated Communications Network is Bell Aliant (www.aliant.net), DSL/dialup provider in Atlantic Canada… well, at least dial-up resolves to Stentor National Integrated Communications Network. I think their DSL actually says Aliant now… my IP now is 142.163.xxx.xxx and i’m on a home residential Aliant connection in NL.

  7. Thanks for this article. Stentor National Integrated Communications Network is in the process of really looking around my blog after Googling a name of a movie character I had written about! I thought it might be a fan bored at work, but perhaps there’s something more to this. At least it’s not malicious…I hope!

  8. The address:
    1 Brunswick Square
    Saint John, NB

    This is one of 3 main communication units of Bell-Aliant, the main telecommunications provider for Atlantic Canada. The other two are in Halifax, NS and St. John’s, Nfld. I have toured this facility and was most impressed at the size and breadth of operations there. They have literally hundreds of server cabinets. Some are their own, some are of their clients. When I was working for one of their clients, the name ‘Stentor’ referred to (800) phone numbers which we leased from Bell-Aliant.

  9. Noticed this too today after posting a review about the Sony Ericsson w595. Thought it was harmless, but decided to check exactly where it had come from. Glad I’m using the WordPress plug-in to see who’s online now :p

  10. I was recently monitoring traffic on my site, and as I watched Facebook completely spam me with crawls, I noticed Stentnor crawling. Having never really seen them before, I Googled, and well, here I am.

    The interesting thing is that the page they were on wasn’t mentioning a product – it didn’t even have any words, really. It was an infographic I did about Back to the Future (http://scarform.com/blog/2010/04/28/back-to-the-future/). I guess the title itself could have been taken as a “product” title…

    Anyway, thanks for the info.

  11. I built my dad a small website that gave information about a software product he designed. I had put one backlink to it and they crawled my site. No one else has ever been to the site b/c it has been up for two days. Great Post! Thank you!

  12. Interesting! I have read this page and all the comments. My site at http://ProblemSolverXYZ.com was just visited by this bot. It’s not the first time. What makes this interesting is when you start adding things up collectively.

    The only items that they scanned on our site is political or governmental posts/comments. So, does this mean, the US Government or political groups are using this company? It would be local to say that not only companies are using this technology.

    I just thought it was interesting to point out. The bot is specially interested in our post about the death of Osama Bin Laden.

    Have a great day!

    Webmaster
    ProblemSolverXYZ.com

  13. I had recent similar activity, and after having analyzed the landing pages they seemed to be interested in Ron Paul, and or the Federal Reserve.

  14. Like one of the other posters indicated, Stentor National is an Internet Service Provider. Bell Aliant FibreOp also shows under this provider in IP lookups. A few of them may be zombies running crawlers, but in all actuality, there’s a pretty good chance it just indicates that people are reading the blogs you’re writing. From the above post:

    NetRange: 142.166.0.0 – 142.166.255.255
    CIDR: 142.166.0.0/16
    NetName: ALIANT-TEL-142-166

    The NetName Even has Aliant right in it 😛

  15. You may be surprised to learn that this has been going on for over 150 years 🙂 Not with blogs for that long, obviously, but magazines, newspapers, TV, radio – any kind of media that might mention a client’s name or products. I used to work for Romeike (called Cision these days), which was the first ever “clippings service” – so called because they literally used to cut out articles of interest from the newspapers to send on to their client. In the beginning, it was a service for vain actors who wanted to know who was talking about them and their performances, but grew to service all kinds of companies that took an interest in monitoring the effectiveness of their marketing.

    See http://en.wikipedia.org/wiki/Media_monitoring_service

Leave a comment

Leave a Reply to Trever Cancel reply

Your email address will not be published. Required fields are marked *