More data facilitates better business decisions, improves operational efficiency and drives growth. But too much of a good thing creates problems.
Cloud implementation allows businesses to rapidly onboard new and better technologies and intake a lot more data — so much data that analytics teams can be overwhelmed. When companies can’t process, map and mine incoming data while also monitoring for security anomalies, value is lost.
Most companies discard a majority of data collected. Up to 90% of data mined never makes it to the analytics stage, and most companies’ tech stacks offer end-to-end observability of just 9% of data, according to the survey.
“The systems that exist today are just too slow and expensive,” said Mike Maciag, CMO at Dynatrace. “So, you have to make a hypothesis ahead of time about what kind of data might even be usable because you’re going to throw away 90% of it, which is insane.”
With IT staffing and retention woes impeding modernization efforts, companies are turning to automation and AI deployments to take up the slack. A majority of executives said they were bullish on automation technologies in an August report from Gartner, and seven in 10 reported relative ease sourcing AI talent.
In contrast, a scarcity of data science talent is hindering analytics projects. To exacerbate the gap, demand for data professionals showed significant year-over-year growth in the first half of 2022, according to a mid-year analysis of three million job postings by Dice. Data analysts, data engineers and data scientists all placed in the top ten of most in-demand tech positions.
Even without a talent shortage, data operations would be stressed.
Cloud is an adrenaline boost for enterprise IT — a blessing and a curse for data teams. As the volume, velocity and variety of data has increased, so have the number of tools needed to monitor and track that data.
With companies pushing for real-time insights across all business functions, simply managing data, preventing dreaded siloing and providing adequate access is a challenge.
“Observability logging and analytics at scale goes from something you might be able to do on your own to something you wouldn’t be able to do manually even if you could hire the people you need,” Maciag said.