Payout Magazine Digital Edition

New Smarter Spiders - by Kath Blackwell

With every new technology comes a set of pros and cons. Recently, the folks at SPI Dynamics released a research analysis report that focused on covert crawlers, a new breed of spiders that are coming that would be able to surf and view the Web, just as humans do and can even complete all of their tasks undetected. The technology behind these secret spiders is so advanced that the crawlers will be able to click on links, run Flash files and view JavaScript and other elements just as we do. In fact, these super-smart bugs could act just like a browser and even keep a cache, downloading only necessary files as it navigates your website from page to page.


Sound like pure Sci-fi, something that couldn’t happen in the near future? Researchers say these spiders might be crawling your site very soon. The technology to make these theories into a possibility is available now and only a few “bugs” remain to be figured out in order to make it a reality. In fact, the research report that was released by SPI Dynamics proves just how possible it would be for these covert crawlers to be crawling the Web today.


The Pros… and the Cons

The technology for these covert crawlers was intensively researched and presented by one of the engineers at SPI Dynamics (www.spidynamics.com) as a way to ferret out problems and online theft such as click-fraud, check out site-load dynamics – even get a thorough peek at competitors’ sites. According to the report, covert crawling “has all the pros of regular automated crawling” such as exhaustive search and the ability to automatically save resources to be used later for analysis. Certain tricks are employed, however, to make it look as if it multiple users from various IPs were browsing the website instead of a spider.

That’s a fairly awesome concept when you think about all of the positive applications of a technology that can surf unnoticed and scan every single page on a site. Imagine the possibilities! The technology developed by SPI Dynamics is not intended to attack a site – it was created to help developers get the most out of their Web projects. However, because of its techniques and its virtual invisibility on hosting logs and statistics, it could be used to locate the best possible group of pages to attack, store the information and later be used to launch attacks. SPI Dynamics warns the development community that this technology can and will be used for ulterior motives in the near future.

Other negative affects of this technology could affect those who work heavily in the search engine optimization fields. Because today’s Web crawlers view sites differently than human beings, the tricks and methods used by SEOs are designed to work in a way that those crawlers will appreciate most in order to earn the highest ranked listings. However, a site that is crawled with one of these super spiders in the future would have to have a different game plan, and would likely have to tweak their entire approach.

Other developers that would be affected are those who create crawler-only content in order to boost search engine rankings. While this is a con to those folks, it would actually become a benefit to honest developers who would do well being ranked upon their site’s true content and data that may have previously had poor rankings in competition with these SE-cheaters.

What to Do Today

Web developers need to take notice of the impending days of the covert crawler. Action needs to be taken to avoid potential attacks due to the recognizance work that can and will be done by competitors and hackers alike. Action needs to be taken to avoid loss of search engine ranking due to tricks and cheats currently enacted within your web project’s structure.

  • Evaluate your site’s security. If you haven’t had your site professionally checked by an expert to locate potential vulnerabilities and ways that hackers could attack your site, this should be priority number one. Your server set-up, site structure, firewalls – everything needs to be analyzed and evaluated in order to ensure that even with a covert crawler, your competition or potential hackers would be unable to excise any damage. Remember – because these spiders can crawl your entire site undetected, you will not be able to rely upon your logs for a warning against potential doom.

  • Consider adding more security – or even changing hosts. After you’ve analyzed your set-up, you should definitely consider adding additional security measures to your site structure or should consider changing hosts if the analyst determines that you are at-risk due to set-up or security on the host level. Always choose your hosting company based upon reputation, reliability and ability by the site’s support staff to meet all of your hosting needs. Choosing a host based solely upon price is a disaster in the making.

  • Re-evaluate your current SEO set-up. Make sure that your site’s ranking does not depend solely upon crawler-tricks and cheats. View your site via a browser and take a look at how your site measures up from a human’s point of view to ensure that your site’s value and ranking would not be diminished by the change from a drone crawler to a super smart spider in the future. If you are using crawler-only content, you will want to re-think your strategy and make sure that you have the true site content to support your target keywords.


Preparing for the spiders of tomorrow today will help eliminate potential problems, loss of income and downtime of your site in the future. These covert crawlers are definitely coming – it’s only a question of when.

Back to Article Page

Stay tuned and check in for coverage of all facets of industry ecommerce – whatever online verticals you are involved in we will be sharing information that will help you maximize both your efforts and revenues. We’ll be reviewing personalities, websites, software and providing plenty of how-to articles, along with global industry trends, data and analysis to stimulate ideas and get you thinking about creative ways to innovate as we all move forward.

For all our entrepreneurs, program owners, affiliate webmasters – readers working in every aspect of ecommerce – watch for insight, deals, events and opportunities you won’t see covered anywhere else. Market your products and services, promote your latest offers to our extraordinary readership – get current and forward thinking information you can use every day – really useful information combined with a little harmless diversion – get involved and move ahead.

Follow MikeB and PAYOUT Magazine around the world, hitting every significant industry event and, recording it for posterity as only MikeB can! Global distribution in print and digital issues – sponsor event coverage and engage in print and digital marketing with life well beyond the tradeshow – sponsor ad pages, photo galleries, site banners – see and be seen – PAYOUT Magazine readers are working in the industry – make sure they know who you are…