I read an interesting article in The New York Times (2/12/11) this past weekend regarding a “Black Hat” trick that caused a certain retailer to continually pop-up at  the very top of a popular search engine.  Regardless of what a searcher might enter in, including brand name items, the retailer continually appeared in the top slot (of the unpaid portion of the page).  As it turned out, the reason it was deemed a “Black Hat” as opposed to a “White Hat” trick was that the results were due to invalid data that tricked an algorithm.

So, my brain began to wander and a question came to mind.  How would the fast-paced, millisecond mind space of today’s discerning researcher (whether it be a consumer shopping or an analyst determining a lift in sales) be able to validate that what was before them was actually accurate?  In the article, the newspaper was able to hire an investigator of sorts, but in the real world, such resources aren’t always available.

In business, results are often tallied, averaged, summarized and aggregated and, while there is nothing wrong with this, one should always be able to validate from whence the results came.  Should a business always trust implicitly the results in front of them?  What if something odd or unexpected should appear?  Do you have a way to validate your analytical findings? And maybe the most important question I would love to hear back on is:  do you have a transparent view into how the results presented to you were derived, and can you get to them in the instant you need them?

If you could ask any question about your business, what would that question be? While it may be a fun problem to contemplate, most  businesses find that they simply don’t have the tools to do this.  When that happens, decisions can be made based on guesses or in a vacuum.

How much does it cost your business to not be able to ask any question of your data? Unfortunately, too many businesses have tolerated suboptimal tools for measuring Key Performance Indicators (KPIs).  As a result, while there are a lot of things that they should (and could) know, they only discover what their tools can report.

The good news is that things are changing.  More and more, the press is buzzing about the need to tackle “Big Data,” and the acceptance of the private cloud as a viable alternative for getting there.  As a case in point, AutoZone recently switched their analytical toolkit to tackle not only big data but get their arms around the complex questions that they were never able to ask before.  Click here and read the Integrated Solutions story about what AutoZone discovered just by being able to ask any question.  After you read this article, we’d love it if you could share with this blog and our readers the question: “If you could ask any question of your data, what would it be?”

After attending the 100th annual National Retail Federation (NRF) convention and expo, we came away incrementally more positive on the retail environment and the software companies that serve the space. Based on our conversations with numerous private and public companies, attendance was up approximately 15-20% from last year and the prospective customers in attendance appeared to be much more serious and “real” buyers than last year.

Additionally, 1010data chose this year’s NRF to announce the Vendor Analytics Portal, a solution that makes it easier for retailers and CPG manufacturers to work together, analyze store, SKU and shopper-level data and make better decisions when it comes to optimizing promotions and other aspects of marketing and operations.

1010data attended the NGR event in Miami last week.  It was a pretty interesting format – kind of like speed dating, to connect retailers with vendors of the latest technologies.  We met with eight retailers for 40 minutes each.  The room for the “dating” sessions was a large hall with 20 tables and only limited signage was allowed.

The process prior to attending was to clearly articulate solutions so that the retailers could select those vendors that they were most interested in seeing.   Both sides needed to do their due diligence to prepare, and we found it to be a fun and engaging forum.  We did live demonstrations for all of the meetings, as it is becoming very apparent with our offering that “seeing is believing.”  I have been in this business for many years, and it is so exciting to see someone’s face light up when they see 1010data.  The ability to move mountains of POS data, with clear visibility as to what you are doing is a fundamental need of every retailer.  The proof was in the fact that I did the demos by myself, which speaks to the absolute ease-of-use-genius of 1010data.

Out of 20 vendors, we were the only real data warehousing offering and the only one that included this plus the analytics side as well.  SaaS was a dominant theme at the show – this ties in with our value proposition, as 1010data offers a turnkey hosted service.  There was also a lot of interest around mobile computing and “Omni Channel” marketing, which is the next generation of multi channel, including mobile as well as Twitter and Facebook as a medium for target marketing and risk avoidance.  IDC led some of the sessions, and the key notes were interesting.

Some of the big ideas we presented, and that resonated with the attendees included:

• Transparency (All Data & All Users)
• Speed (Performance and Scalability can be accomplished, “it’s all about speed”)
• Simplicity (No short cuts but complexity can be simple)
• Big , Raw & Disparate Data (The size and type of data should not be an impediment )
• Infinite and Linear Scalability
• Service (We handle data; You the results)
• Cost Advantage (Open systems, speed to go live, no cubes, ELTAR, no bottlenecks)
• Deep Complex Analytics (Inherent in DB)

Each one of these topics can be whole sessions in and of themselves, and require time with 1010data to understand.  We have been called by our customers the “Google for data,”  but we are so much more.  It was fun having the opportunity to show retailers something they have never seen before.

Point-of-sale transaction logs (POS tlogs) have long been proclaimed the “Holy Grail” of retail information. Tlogs present the most granular view of purchasing data, as they are a record of raw transaction-by-transaction data from every POS terminal.

However, the potential of tlogs to help retailers better understand what consumers purchase, and in what combinations (i.e. the market basket) has not been systematically fulfilled. Traditional approaches that systems take in working with tlogs involves summarizing data and then loading it into a data warehouse (or loading first and then summarizing) for use in reports.

For most retailers, the status-quo of data warehousing and reporting has been adequate for SKU-by-store level analyses. However, retailers are now seeing that they’ve outgrown the traditional summarized approach. These systems do not go far enough, deep or fast enough to satisfy the evolving needs of the strategic retailer.

The good news is that systems are now here that can work with raw tlog data and give retailers the analytical tools needed to better understand what the data means and gain actionable insight in doing so.

The Market Basked Mandate

Now more than ever, retailing requires a comprehensive understanding of the customer. The changing economy, the onslaught of new technology, and environmental challenges all contribute to changing how customers shop. Accordingly, a retailer’s repository of data must evolve from simply reporting what has been sold in aggregate to a system that can examine and answer any question as customers continue to grow and change.

Improved efficiencies due to these same sociological factors are necessary in Operations, Merchandising, Marketing and Customer Service. The increasing pressure on margins forces retailers to work smarter and get the most out of systems. Not surprisingly, Market Basket Analysis happens to also be the best analytical resource for fixing, maintaining and managing operational efficiencies.

Market Basket Analysis Answers…

• What items are in customers’ individual baskets and over time, across trips?
• What are all the metrics associated with the basket?
• What are the relationships of the items in the transaction?

These questions require working with transactional data at its most “atomic” level. Such datasets include the most granular level of basket detail. Market basket analysis is not just limited to examining the interactions of items, but also includes operational details that are impossible to glean from summarized data.

A Market Basket Analysis system will store by transaction:

• Store Number
• Date
• Loyalty Card
• UPCs/SKUs, quantities, prices
• Transaction Time Stamp
• Terminal/Lane
• Cashier
• Ring time, by item in Basket
• Net discount item detail

And can be supplemented with data like:
• Demographic characteristics by customer
• Geo-demographics surrounding stores
• Inventory data
• Promotional details
• Coupon billing

Requirements for Effective Market Basket Analysis

Speed — High-performance data warehousing technologies are available for large data volume needs.

Scalability – Market Basket Data storage requires a highly scalable database system.

Analytical Extensions – Necessary to go beyond basic metrics and really maximize data value.

Data Integration — The database should facilitate easy integration of additional data without re-design.

Retail is a highly competitive environment. Market Basket Analysis offers retailers the opportunity to transform their businesses. The time is now, the data is here, and true, scalable Market Basket Analysis has arrived.

The NY Times featured an article earlier this week about ways that retailers are leveraging data to personalize offers for their customers.  Have they Got a Deal for You describes how powerful computers are leveling the playing field and expanding sophisticated analytics beyond the most data intensive businesses like banking, airlines and credit card companies. The article reports:

… retailers have long tried to decode consumer behavior, through surveys or test panels, or by using crude forms of data analysis. During the last decade or so, many retailers have amassed huge amounts of data through loyalty programs or membership cards, like those provided by Sam’s Club and its rival, Costco… But retailers have generally done little with the information, other than using the cards as a branding opportunity or to offer broad discounts… Some retailers, however, have used the data to figure out the right product mix and layout for their stores, or to offer discounts to categories of customers.

The article goes on to cite examples of big box retailers’ use of data (combined with third party consumer information from FICO) to develop promotions tailored to the individual shopper.

It is a great topic, and one that is near and dear to our Retail Solutions Group.   Where it gets really interesting is the ability to go beyond vast stores of possibly dated loyalty customer info to examine T-Log data (i.e. transactional data from POS systems) tick by tick, almost in real time.

Retailers really do have the ability to glean the most powerful insights about their customers from their own raw data.  The ability to now discover the individual uniqueness of each customer is now possible and opens up endless opportunities for the more advanced retailers.

We are glad to see the NY times covering this type of story.

The May issue of RIS News features a cover story (What’s in your Market Basket?) on retailer Dollar General’s use of 1010data for improved market basket analysis.

This blog is about education, and we intentionally steer clear of self-promotion.  Just the same, we feel it is important to point out the third party validation for our approach, and are pretty excited about the story.  There is a lot of misinformation out there about the limitations of cloud-based analytics (see last Tuesday’s post on this blog, for example, in which we point out erroneous claims) and believe that highlighting the experiences of customers is one way to counter misinformation and portray the results that can be achieved, especially for data-intensive fields like CPG and retail.

Writer Joe Skorupa very eloquently summarizes the effort and results achieved in the opening paragraphs:

A massive cloud over Europe obscured views and disrupted air travel this spring thanks to airborne dust from an active volcano, while another large cloud within Dollar General was having the opposite effect. It was producing new levels of clarity by doing large-scale, high-speed analysis on billions of rows of stored data and delivering actionable insight to marketing and merchandising executives who were effectively putting it to use.

Insight from this cloud-based, data warehouse and reporting solution was uncovering previously hidden opportunities for key executives to make more informed decisions to increase same-store sales, which is essential to help drive one of the fastest growing retailers in the country.

Results from this system and other strategic steps taken by Dollar General in the past two years have produced impressive financial gains. At its most recent quarterly filing, on March 31, Dollar General reported a net income increase of 121 percent to $172.9 million along with 9.5 percent growth in same-store sales in 2009 on top of nine percent growth in 2008.

Regarding concerns over cloud scalability, the article states:

The Dollar General deployment goes a long way toward allaying these fears, in part because it is a $12 billion enterprise with 8,800 stores. Its cloud-based data warehouse system currently handles 45 terabytes with 70 billion rows of data. If a retailer the size of Dollar General can use a cloud-based tool to handle core business operations, then most retailers also can use it and the technology may be more bullet proof than many retail CIOs think.

Elsewhere, the article quotes Dollar General’s chairman and CEO:

“We accomplished these objectives while investing for future growth, a balance that positions us well for the long-term,” said Rick Dreiling, chairman and CEO.  “We are confident that we have the right strategy in place to continue building on our track record of profitable growth as we enter 2010.”

Please visit the following link to read the full story.

We saw a nice post on the Impact 21 blog about BI trends for this year and implications for retail.  Lesley Saitta (Impact 21 CEO) lamented the lack of innovation in BI and attributed this to industry consolidation and a shrinking pool of smaller, disruptive challengers.  She wrote:

Fortunately, there is still a lot of innovation occurring[sic] in Business Intelligence, as retailers and other industries are demanding greater access and integration of their mission critical data/information to support decision making by their users.

Lesley goes on to cite a list of trends in analytics from Driven to Perform author Nenshad Bardoliwalla.

We were pleased to see some synergies we have with the list on her blog, specifically the first trend (discusses the integration of transactional and analytics capabilities), second (highlights the need to be fast and predictive and look beyond what happened yesterday) third (talks about SaaS/cloud), fourth (importance of scalability).  Each and every one of these areas is positively impacted by the 1010data service and we see an ever increasing trend toward simplification of delivery of data to users within our retailer base. The sophistication of the end user is rapidly increasing but the delivery of their answers is being simplified by our approach. A nice trend.

One trend that was not explicitly stated that we believe is happening today is the need to massively simplify how data is being delivered to the end user. We believe this to be a major trend and the next generation of BI as users require more immediate access to detailed data and are less apt to be patient waiting for the answers. This is coupled with IT’s need to dramatically reduce the cost associated with the delivery of this user information.  Data warehousing as we traditionally know it is being turned upside down. Can you imagine letting a user have access to a billion row table with no supervision? We can and we do.  Welcome to the next generation of Business Intelligence.

I recently participated in a Q&A with TDWI where we explored how private clouds can speed up data access. It’s an important idea for retailers, check out the full article here.

1010data just finished our third NRF show and the results were very good.  This year the retail community seemed more upbeat than last year, and the attendance was up. More importantly, the attendees were interested in analytics and finding out what it can do for their businesses.

I noticed that more and more people are becoming comfortable with hosted solutions and the thought that a private cloud can be a part of their application architecture.  We showed live demos of 1010data to several retailers using over 12 billion rows of raw POS data from a chain of 1000 stores and all were very impressed.   Everyone who saw the performance and our ability to offer visibility into these lower level transactions grasped the benefits right away.

Seeing is believing, as the saying goes, and our experience has been that the best way for a retailer to understand what it means to move to the next generation of analytics is by proving it with their own data.  The fact that we do not  need to pre aggregate, or build indexes or cubes to generate the performance they need is still a mental hurdle that we are overcoming one retailer at a time.  We are convinced that the industry will realize the benefits of this approach, especially as the need for ad hoc analytics increases.  Our philosophy of getting the user as close as possible to the raw source data will allow our customers to be more consistent and flexible as their needs continue to change.

All in all, the show was a success for us and proved to us that retailers are adopting analytics as a way to differentiate themselves and become more in tune with what their customers really want and need.  We look forward to the next NRF show to further prove that analytics will separate the winners from the losers in retail.

As always your comments and suggestions are welcome.