By Stewart Bond, IDC Research Director, Data Integration and Data Integrity Software
It’s a new world we live and work in. In recent years we have seen dramatic changes in distributed computing, combined with a massive increase of data volume, variety, and velocity as big data fuels digital transformation. In parallel, exponential increases in computing power and advances in machine learning are enabling artificial intelligence (AI) to deliver new solutions that turn the impossible into the possible. One such example is the advent of business-friendly, AI-enabled data preparation software solutions that enable the business to take back its ownership of the data.
According to a 2017 survey conducted by IDC, the 80/20 rule is still in effect: data professionals spend 80% of their time searching for and preparing data compared to 20% of their time performing analytics. At the heart of digital transformation is data; not the data itself, but the ability to leverage the value of analytic and operational insights derived from the data, resulting in actions that improve engagement, experience, and the bottom line. Business-oriented data preparation software solutions can now reduce the time people spend preparing data, thus increasing the time available to uncover new insights.
This new class of data preparation software is also very disruptive in the data integration and integrity (DII) software market, reducing demand for IT-oriented solutions, where data integration vendors have been incumbents. Data preparation software continues to gain momentum, demonstrating growth rates three times that of the overall DII software market, and a forecasted compound annual growth rate of 19% for the five-year period from 2018 to 2022.
Many IT departments are interested in data preparation software as it removes the IT bottleneck in data analytics projects while still allowing a level of control that will satisfy internal policies and external regulations. Governance coupled with collaboration is a key capability required in modern data preparation solutions.
As with any new software implementation, organizations need to build a business case and measure return on investment (ROI). Metrics are critical for success because what isn’t measured cannot be improved. Data preparation software lets users parse, cleanse, standardize, transform, federate, and integrate data. These capabilities are the basis on which metrics can be defined:
- Time spent accessing and preparing data
- Overall health and quality of data
- Relief from support and maintenance costs associated with legacy data preparation software
Time spent accessing and preparing data: Reducing the time spent locating, accessing, and preparing data will change the 80/20 rule. A recent IDC study on the business value of data preparation software measured the ROI associated with this metric. The study identified an 88% reduction in data preparation time spent by analysts and a 20% increase in analyst productivity. The resulting ROI was achieved in a few months. Data quality levels, trust, and compliance also improved as by-products of better data access, profiling, standardization, control, and increased collaboration among analysts. Putting data preparation into the hands of the business user will also relieve IT from the pressure of preparing data, allowing a focus on more innovative initiatives.
Overall health and quality of data: Improving the overall data quality in the organization will increase the level of data trust, ultimately resulting in better data-driven business decisions and outcomes. Data quality is defined and measured by timeliness, completeness, accuracy, and consistency, which can be impacted by data duplication and distribution, as well as errors made during creation, modification, and blending. These are metrics that can be defined and tracked across the organization, and data profiling within and across columns according to standard regular expressions and defined business rules can provide an overall data quality score. As business data owners are given more access and the ability to cleanse their data using data preparation software, the scores should improve over time, demonstrating the software’s value and ROI.
Relief from support and maintenance costs related to legacy data preparation software: Cost savings are also associated with reducing reliance on legacy data preparation technology and its attendant complexities. As data preparation shifts to the end user’s desktop browser, the capacity required in legacy ETL software may be reduced, thereby reducing infrastructure and licensing costs. Data preparation software also provides new opportunities to handle larger data sets and greater varieties of data sets, and to improve the quality of data to uncover deeper insights that can enable better business outcomes through new products, services, or new ways to go to market, which is digital transformation.
If you are looking for opportunities to digitally transform with data, you may need data preparation software. You may also need it if:
- Business users cannot access the data they need
- IT is burdened with data preparation tasks that hold up business projects
- The business no longer trusts the data to make decisions
- IT’s budget is consumed supporting legacy data technologies that require specialized resources
- Data is delivered too slowly for analysis to be valuable
Data preparation software promises to be a key capability in the new world of work.