Big Data: Technical or Organizational Problem?

You see the term “big data” everywhere. What exactly does it mean? The trouble is that there are many definitions. For clarity’s sake, I’ll pick one from Gartner Research: “Big data is the term adopted by the market to describe extreme information management and processing issues which exceed the capability of traditional information technology along one or multiple dimensions to support the use of the information assets.” Gartner's definition points out that there is an array of software and hardware solutions available to solve these big data problems.

Rachel Alt-Simmons, SAS

But is this “big data” thing really new? The reality is that companies have had big data challenges for years and years. Tackling big data opportunities now is definitely enabled by the new technologies and architectural designs being provided by the IT vendor community, but the business community needs to take a more active role in visioning a future state of business strategies and processes enabled by big data technologies. An example: During a big data proof-of-concept project, a team was able to reduce the runtime of a marketing optimization routine from ten hours to under two minutes. One of the business sponsors rightly said, “That’s great! Now we made a bad process really fast!” While the technology did exactly what it was designed to do, they realized the need to rethink their business analytic processes and workflow.

Also, analytic data and technology environments are often not well architected for their intended purpose. Many analytic groups find themselves managing operational processes, while the demand for analytics, as well as the data volume and need for speed-to-insight, grow within the organization. Over time, the processes can grow so complex that it becomes difficult to identify the root causes when the process breaks down. Is it a big data problem or simply a process, data or technology design problem?

[For more of Alt-Simmons' industry insights, see Navigating the "New Normal".]

The good news is that there are process improvement methodologies out there to help you. For analytic teams faced with this dilemma, I recommend a Design for Lean (DFL) approach. The goal of Lean is to eliminate waste and improve efficiency (and is complementary to its sister methodology Six Sigma, where the goal is to increase quality). The “design for” approach allows the analytic team to vision an ideal future state and to systematically address any people, process, data, and technology opportunities uncovered during the visioning process.

Here’s a simple way that you can put the methodology into action:

Get your process improvement team together: This group should include business and internal/external technology stakeholders. Think through what you need to solve for and create an actionable opportunity statement. For example, “We need to create more relevant customer offers that increase customer satisfaction, drive profitable relationships, and increase revenue.” Don’t forget to quantify these metrics where possible. Don’t get mired in day-to-day details here: Fixing current problems are important, but crafting a roadmap for the future is critical.

Identify high priority business parameters for design: In our marketing example, it could be an increase in the number of offers or campaigns, running analytics processes more quickly to ensure relevancy, or running predictive models on entire populations instead of samples to increase model lift. Make sure that your team brainstorms around all of the four pillars of analytics: people, process, data and technology. Don’t forget to include aspects like “I could achieve this business goal if only I had access to the data in one place” or “I can’t run this critical analytic process because the system keeps crashing,” but don’t forget to stay focused on your original opportunity statement. If multiple teams are using the analytic environment for different purposes, each team should hold their own visioning sessions. Bring the teams’ output together to look for commonalities.

Architect the vision: Once you have all of the future state business requirements, it’s time to architect the vision. Engage the right partners to help with the right tasks: Human resources groups can assist with people aspects, technology teams can define analytic data and platform architecture. If your internal partners don’t have the domain expertise needed in the design phase, engage external partners.

Create your solution document and roadmap for closing gaps: Chances are your ideal state will require considerable time, resource and financial investments. If you’ve quantified your opportunity statement, you should be able to tie your future state investments to business outcomes. Remember, you don’t have to tackle everything at once: Select projects that will drive more immediate business value and help you incrementally achieve your goal.

The great part of this methodology is that it aligns business goals with technology infrastructure. Taking a disciplined approach will help you separate the hype of market terminology from identifying the root causes inhibiting your ability to move your analytic initiatives forward.

About the Author: Rachel Alt-Simmons is Senior Industry Consultant for Insurance at Cary, N.C.-based SAS. Simmons has driven business intelligence initiatives at Travelers and Hartford Life and has been Research Director for Life & Annuity at research firm TowerGroup.