Amazon ahead in arms race to win the 'bots' battle
Earlier this year, engineers at Wal-Mart Stores who track rivals' prices online got a rude surprise: the technology they were using to check Amazon.com several million times a day suddenly stopped working.
Losing access to Amazon's data was no small matter. Like most big retailers, Wal-Mart relies on computer programs that scan prices on competitors' websites so it can adjust its listings accordingly. A difference of even 50 cents can mean losing a sale. But a new tactic by Amazon to block these programmes - known commonly as robots or bots - thwarted the US retailer.
Its technology unit, @WalmartLabs, was unable to work around the blockade for weeks, forcing it to retrieve Amazon's data through a secondary source, according to a person familiar.
The previously unreported incident offers a case study in how Amazon's technological prowess is helping it dominate the retail competition.
Now the largest online retailer in the world, Amazon is best known by consumers for its fast delivery, huge product catalogue and ambitious moves into areas like original TV programming.
But its mastery of the complex, behind-the-scenes technologies that power modern e-commerce is just as important to its success.
Dexterity with bots allows Amazon not only to see what its rivals are doing, but increasingly to keep them in the dark when it undercuts them on price or is quietly charging more. "Benchmarking against Amazon is going to become hard," said Guru Hariharan, a former Amazon manager who now sells pricing software to retailers as CEO of California-based Boomerang Commerce. A Wal-Mart spokesman declined to discuss the January episode but said the company improves its technology regularly and has multiple tools for tracking items. He said the company offers value not only through pricing but from discounts for in-store pickup and other benefits.
A spokeswoman for Amazon said the company is aware of competitors using bots to check its listings and denied any "campaign" to stop them. "Nothing has changed recently in how we manage bots on our site," she said but added, "we prioritise humans over bots as needed".
Bots can slow down a website, a big motivator for retailers to block them. Reuters interviewed 21 people familiar with bots and how they are deployed, including current and former Wal-Mart employees, former Amazon employees and outside specialists. Many spoke only on condition of anonymity because they were not authorised to discuss the issues publicly. Most pointed to Amazon's leadership in the burgeoning bot wars.
The company's technological edge has been good for its profit margin, and it's proving a winning formula for investors. Shares of the internet powerhouse have risen about 15-fold since the market's bottom in March 2009, while the S&P 500 has more than tripled in value. Amazon hit $100bn in annual sales in 2015 - faster than any company in history, it said.
Bot-driven pricing has represented a massive change for the retail industry since Amazon helped pioneer the practice more than a decade ago. Traditionally, brick-and-mortar stores changed prices no more than weekly because of the time and expense needed to swap labels by hand. In the world of e-commerce, though, retailers update prices with ease, sometimes multiple times a day, helped by algorithms that consider inventory levels, sales forecasts and rivals' pricing data. To stay in the game, companies such as online wholesaler Boxed, based in New York, depend on a variety of methods including bots to ensure they do not lag others' price moves for even 20 minutes.
"That's like a lifetime during Christmas," said CEO Chieh Huang, whose company sells bulk staples like toilet paper and pet food. "If we're not decently priced, we'll see it almost immediately" in sales declines. Using bots to view massive amounts of data on public websites - a process known as crawling or scraping - has many purposes. Google, for example, constantly crawls the web to gather information for its search engine results and to sell ads. In ecommerce, though, the use of bots has developed into a cat-and-mouse game. Companies try to thwart the practice on their own websites while aiming to penetrate their competitors' defences. Third-party services abound to help less-savvy retailers.
To protect data from rivals, some retail websites use what's known as a 'CAPTCHA' - typically a distorted string of letters and numbers that humans can read but most bots can't. Amazon shies away from the practice because it annoys some customers.
For merchants seeking to evade such defences, disguising their computer programs as real shoppers is key. Some pricing technology experts have programmed computer cursors to meander through a web page in the way a person might, instead of going directly to the prized data.
Another technique is to use multiple computer addresses so that retailers cannot track a barrage of clicks to a single source. "It is an arms race," said Keith Anderson, a senior vice president at e-commerce analytics firm Profitero, based in Ireland. "Every week or every month, there's some new approach from both sides."
The Amazon manoeuvre that halted Wal-Mart in January took aim at a specialised web browser called PhantomJS - designed specifically for programmers. It was unclear how the move, which was not aimed at Wal-Mart in particular, affected other companies. (Reuters)