There was a great article in the Economist about the challenges and opportunities of big data entitled Building with Big Data; The Data Revolution is Changing the Landscape of Business. The article describes the growth in various types of data, and cites a McKinsey study on the potential enterprise value to be gained by better understanding that data:

Last year people stored enough data to fill 60,000 Libraries of Congress. The world’s 4 billion mobile-phone users (12% of whom own smartphones) have turned themselves into data-streams. YouTube claims to receive 24 hours of video every minute. Manufacturers have embedded 30m sensors into their products, converting mute bits of metal into data-generating nodes in the internet of things. The number of smartphones is increasing by 20% a year and the number of sensors by 30%.

Clearly the amount of data businesses deal with every day is exploding. The article continues…

In a suitably fact-packed new report, “Big data: the next frontier for innovation, competition and productivity”, [McKinsey] argues that data are becoming a factor of production, like physical or human capital. Companies that can harness big data will trample data-incompetents. Data equity, to coin a phrase, will become as important as brand equity. MGI insists that this is not just idle futurology: businesses are already adapting to big data.

We agree with this assessment, and believe that companies need to use their rapidly expanding data to create data equity. It’s not enough just to have the data stored in legacy systems or remote silos- you must have easy access to the data and the analytic tools necessary to leverage that data to create real value. We have been speaking and blogging about related topics, like data monetization (see this recent post).

Our own VP of Marketing Tim Negris will be speaking about creating data equity through big data analysis at the Cloud Computing Expo in New York this week: Collaborative Big Data Analytics: It takes a Cloud.  If you are attending the show, we invite you to come listen to Tim or stop by our booth to learn more.

Greg Munves

Some people confuse the term “Web analytics” with the more general fields of BI and analytics. While BI and analytics can help make sense out of a wide range of transactional data, Web analytics, as its name implies, deals specifically with clicks, links and other types of Web activity.

These interactions between the user and website can be modeled as transactional data. They are both time-series based, and contain information on the interaction between the user and the business. Because of the need to leverage historical data to evaluate advertising effectiveness and social impact, to optimize e-commerce offerings, and to A/B test websites, we have seen a growing interest in cloud-based, big data analytics solutions like 1010data’s for Web analytics.  The explosion of Web data caused by the growth of social media, streaming / rich media and the real time Web is driving the need for technology that can handle massive amounts of Web-driven data and bridge the divide separating more general business analytics and Web analytics.

Solutions that can help e-commerce sites and Web publishers better understand the data is a step towards getting value from, or monetizing the data. I was reminded of this by an article in DIGIDAY: The Big Bet on Social Data.  Brian Morrissey writes:

At a time when content businesses grope for business models, it’s clear where the real money remains: data. The best part about data: no need to publish content or even attract users. The latest evidence: Clearspring Technologies, a company that started as a widget maker back in the heady days of MySpace, closed a big $20 million funding round, led by former Twitter and Zynga investor Institutional Venture Partners, to build out its social data business. Clearspring boasts it sees a billion Internet users — all without operating a single Website… “What social has done is unlock data,” he said. “The biggest outcome of social media is big data.”

Just how much data is big data? What does cloud computing really mean? How is this different than SQL? Oracle? Can you do analytics? A whirlwind of questions can be asked of many “big data” companies. Companies that help others manage large volumes of data have long had trouble describing what they truly are. This is partly because the range of solutions offered is immense. These solutions fall into two main categories.

First, data warehousing companies help companies deal with data overload. The companies may have recognized they wanted to keep their data, but couldn’t manage it themselves. They choose to buy hardware or occasionally a cloud offering to collect and maintain historical data. This offering often allows for simple analytical queries like indexing and basic aggregations. However, they lack analytic power, and either need to do pre-aggregating and indexing before each complex query or split the data set into smaller segments for analysis.

Second, there are analytic offerings. These include Excel, and for the more technical, SQL. While Excel is very flexible, it is hampered by its inability to work on large data sets and relative lack of power. SQL has relatively strong analytic capabilities, but forces data to be stored relationally. This makes the data take up much more space than it needs to, and makes many types of queries, especially time series, very sluggish on large data sets.

Companies are struggling to use these solutions to solve their problems. It’s important to consider that the biggest concern of most customers is how to leverage data to drive business decisions. The proper solution tackles this problem by providing robust analytic flexibility and power enabling both technical and non-technical decisions makers to drive strategic vision and change. This requires an incredibly fast big data platform that combines the flexibility and simplicity of Excel with the ability to do even more powerful analytics than SQL.

To truly meet our customers’ goals we have created a Big Data Analytics Platform (BigDap). We can take all the data any customer could throw at us (big data), and empower users to do lightning-fast and powerful analyses that drive business results (analytics). Perhaps most importantly, we offer non-technical end users a flexible spreadsheet like, web browser interface as intuitive as Excel, and technical users an incredibly powerful XML based query language accessible via several programmatic interfaces such as APIs, ODBC, MS Excel Add-in, and an SDK that supports most modern programming languages, all delivered securely via a managed cloud. (platform).

Using the managed cloud means that users can access the data and analytic power securely from any web browser. This ensures that there is exactly one version of the truth, and everyone working together will see the same data at the same time. It also reduces hardware costs and lets the company focus on their business needs, not how to store data.

BigDap is more than just a way to manage your data. It is more than an on demand tool to analyze some of it. This is a Big Data Analytics Platform: let your data lead the way!

E-Commerce Times had a nice story about 9 Ways to Sharpen Business Intelligence.  David Carr writes:

In a time of economic turmoil, business intelligence (BI) initiatives stand out for their potential to improve corporate performance… Compared with many enterprise IT projects, BI requires relatively modest investments… [which can] pay off the most for organizations that are serious about doing it right…

He goes on to list 9 ways to maximize BI investments (please visit the link to read the details of each):

1. Customize user interfaces/dashboards for specific roles
2. Integrate data across departments and applications
3. Foster a culture of data-driven decision making
4. Implement processes for continuous data quality improvement
5. Demonstrate improved planning, operations and other outcomes
6. Implement a formal KPI methodology (e.g., Balanced Scorecard, Six Sigma, etc)
7. Deploy alerts/notifications
8. Implement employee training
9. Improve analytics capabilities.

In addition to data warehousing, 1010data lets IT professionals build customizable UIs that provide simple and advanced reporting and querying capabilities; this directly addresses the need in the first point for roles-based UIs and dashboards.

Regarding the second point, Carr explains the problems of having data silos and disparate tools.  We have found that providing analytics and data warehousing capabilities as a service makes it easier to offer clients an enterprise-wide system with a common interface and a unified store of data (which helps them to arrive at “a single version of the truth,” as we like to say).

He also talks about challenges inherent in extracting and rolling up data, and about data quality issues; our clients appreciate that the 1010data platform works directly on raw data, so there is no need for extensive data pre-processing, grooming, aggregation, etc.  This helps accelerate “time-to-insight.”

Finally, in reference to the last point, please see our post A Modest Proposal Regarding Advanced Analytics; also take a look at the news we announced: 1010data Advances Hosted Analytics and Reporting.

Dr. Dobbs had a nice story (Big Data Inside Stormy Clouds) about how the cloud is driving improvements in BI, e.g. offering more flexibility for developers.  A source quoted in the article said that: …cloud BI represents a way for software engineers to build reporting and analysis solutions more easily. We have found that building BI/analytics solutions on top of a platform like 1010data’s allows developers to focus on the business needs they are trying to address, and not on performance or hardware issues

The article also cited an IDC study that was bullish on the use of the cloud for BI and analytics: A recent IDC survey and presentation, The Maturing Cloud: What the Grateful Dead Can Teach Us About Cloud Economics.. showed that 50 percent of respondents said it was highly likely they would pursue the public cloud for BI and analytics — and nearly 70 percent said it was likely they would pursue a private cloud deployment.

Another quote in the article said: Many large enterprises are interested in cloud BI as a horizontal tool to provide a simple, distinct, affordable ‘IT sandbox’ where software developers can work on project experimentation and evaluation can occur far from the production environment. The organization may also want to use cloud BI to develop and deliver a departmental BI project more quickly and inexpensively than other options. In both cases, avoiding the time and expense of buying and configuring server hardware, operating software, and database software holds a strong appeal and greatly accelerates the evaluation-to-deployment cycle

Dr. Dobbs is written for developers, and the article is written to appeal to that type of audience.  We agree with the benefits cited above, and would just also add that cloud-based BI can offer direct benefits for a wider range of users too.  E.g., business users can also use cloud-based BI to simplify reporting and analytics.  Further, large enterprise are increasingly using the cloud not just as a sandbox; a number of our clients have used it to deploy production enterprise data warehouses.

Andy Hayler of research firm Information difference wrote a nice article for CIO: Lots in Store for Data Warehousing (I am not sure if the pun in the title is intended or not, but it does seem to work well as one).  He covers some of the history of the technology, e.g. cites relational databases and OLTP, discusses what is hot today and how this relates to where the technology is going.  Andy writes:

Some markets which appear to be mature can suddenly become exciting once more. One of the earliest mainstream enterprise applications was the database… But once the relational database became widely accepted there was only a brief period of competition before the market was carved up between Oracle, IBM and Microsoft… Yet in the last five years or so there has been a flood of new entrants to the market, some using quite different database designs from traditional ones. What happened?

He then focuses on some of the forces that have led to disruption and new competition: chiefly, the growth of data and strides in computer design and database architectures.  Hayler continues:

This combination of specialist software and hardware aimed at data warehousing has become known as an appliance, though the definition is a little blurry, as some appliance offerings… can operate in the cloud, so do not require hardware on site.

The article focuses more on hardware and software and mentions the cloud only in passing.  We believe that the cloud can be not just a delivery mechanism and another way to deploy an appliance for data warehousing – cloud-based analytics can open up new opportunities, and lead to wider adoption of self-service, turnkey analytical tools across the enterprise.  They can help achieve another long-sought-after goal that Hayler writes about:

Sheer size of data has not been the only issue — the need to analyse large volumes of data in something close to real time has allowed further specialization…

He concludes:

…the increasing desire for near real-time analysis and the inexorable rise in the volumes of data that organisations need to handle, promise to keep things lively in the data warehouse market for some time.

1010data and others have proved that cloud-based analytics are one way to achieve these goals.

Forbes had an interesting article about how storage and data center technology giant EMC is gearing up for the industry disruption caused by the growth of cloud computing.   The article quotes EMC CEO Joe Tucci:

A … shift … is happening now, Tucci says…. the disruption is coming from the “cloud,” those massive data centers providing remote storage and computation to replace the individual servers and hard drives companies install themselves. Companies who switch to the cloud can cut their IT budgets by at least a third and manage upgrades easily. Market research firm IDC figures companies worldwide will free up perhaps $1 trillion in cost over the next four years by no longer having to maintain old systems.  The shift to the cloud will leave “most, at least half, of today’s companies totally devastated, with new competition coming up,” says Tucci.

At 1010data we have also observed these trends.  We adopted an SaaS model from our very start about 10 years ago.  These days, we use a private cloud model to deliver turnkey data warehousing and analytics for some of the biggest companies and data sets.

There has been some industry debate about the best approach for analyzing large data sets.  Some think that cloud-based solutions are immature, and that specially-tuned appliances and databases are the ways to go.  Our experiences prove that it is possible to support the most demanding analytics applications via the cloud.  In many ways, these are the days we have been waiting for – and that is why the article caught our attention.

ZDnet had an article this week that reported on IT spending plans for mid-sized businesses over the next 12-18 months.  The information was based on a recent survey of 2000 mid-sized companies from around the world.  The article reports that these businesses plan to increase spending across a number of key areas, including analytics and cloud computing.

We thought that this was noteworthy because it flies in the face of conventional wisdom, which says that sophisticated analytics and cloud computing tend to be used more often by larger enterprises rather than SMBs.  A range of solutions, including ours, are making these types of technologies more affordable and easier- to-implement for a wider range of organizations.  Of course, as has been covered extensively on this blog, part of the reason for the uptake is the growing trends to combine cloud computing with analytics; this is also in-line with the growth of self-service BI and analytics solutions that are delivered via the cloud.

According to the article:

Data analytics is a particularly important initiative. The study found 70% of midsize companies are actively pursuing analytics technology to better understand their customers, make better decisions and become more efficient. The study also shows growing adoption of cloud computing among midsize firms, with two-thirds either planning or currently deploying cloud-based technologies to improve IT systems management while lowering costs. This change is reflected in the increased adoption of analytics and predictive technologies that have become more affordable and widely available for midsize companies. The study finds that 53% of respondents expect their IT budgets to increase over the next 12 to 18 months, 31% expect they will remain unchanged and 16% think they will decrease or are unsure.

January is a month when you often see stories that predict the hot topics, trends and technologies for the coming year.  We were very pleased to see 1010data featured in just such a story today, a blog post that industry analyst Robin Bloor wrote for the Bloor Group’s blog (see 10 IT Companies to watch in 2011).

We were happy that we were first on the list in his post, and also that 1010data was in good company, and mentioned along with other innovators.

This post follows another recent story that Robin wrote about interactive analytics.  We were featured prominently there as well, on the topic of analytics for big data.

The following excerpt from today’s Top 10 story includes a link to his earlier piece.  Thanks Robin, and Bloor Group!

[1010data]… is one of a new breed of analytic databases that distinguishes itself by delivering extremely high performance on very large collections of data. It also has some unique analytical capabilities. We discussed its interactive capabilities in the posting Is Interactive Data Analytics Possible? We expect the 1010data become much more prominent this year as it becomes better known and acquires a wider customer base….

“Interactive analytics” is a buzzword that is often thrown around, and you will find no shortage of opinions about what it is and how real it is.  This week, analyst Robin Bloor cleared up some of the confusion in his great post on the Virtual Circle blog: Is Interactive Data Analytics Possible?

The topic is an important one for 1010data, and a key differentiator, especially when large data sets are involved.  We were pleased to be included in the article.

We include a few excerpts from Robin’s article below, and encourage you to visit the blog and read the entire post.  Thanks for covering this important topic Robin.

I didn’t think [highly scalable interactive data analytics was possible]… until I saw 1010data in action… I believe the capability to be unique, at the present time.

There are two parts to providing this capability:

1. Providing an interactive interface
2. Including an underlying database that is fast enough to respond to interactive requests.

Interactive Interface: 1010data provides an interface that comes close to meeting this definition [I don’t mean firing carefully crafted SQL at a database and waiting till it deigns to respond. … In essence, I’m talking about responses of a few seconds or less than a second]. There are two parts to it; the part that helps you formulate a query and the part that allows you to display the results in convenient graphical formats. … None of this is remarkable, except that 1010 data can do this on tables with billions of rows of data. There are limits to the response time, of course. But even with very large tables responses can be as low as a few seconds. Really.

The Underlying Database: I’m tempted to suggest that the underlying database is the fastest I’ve ever seen, but I don’t know that for sure. … It uses all the data base tricks I’m aware of to get speed; this includes compression, column store, parallelism, keeping intermediate results and so on…Possibly the strongest indicator that the 1010 database is a force to be reckoned with is the simple fact that it has sold extremely well amongst financial sector companies – especially to trading desks…

In Summary: This is a product to take note of…