Why Data Projects Fail – And What To Do About It

Data Army Director Michael Ogilvie says the root cause of project issues often runs much deeper than technology selection.

Data projects today come with high expectations of success. The progression from business intelligence to big data, to data science and now the fast-tracking and mainstreaming of AI into a broad range of operational contexts, confirms that the future of business is data driven.

But, like a lot of technology-enabled domains, success isn’t guaranteed and does not always come as anticipated. This is confirmed by the decades of collective experiences of Data Army consultants operating in this space.

Sometimes, data programs expose foundational shortcomings around data definitions, structure, formatting and cleanliness that require significant work to remediate. Additionally, a broad range of skills are required to achieve success. These include clear top-down direction and strategic leadership, data engineering and science talent to build out technical capabilities and data literacy among intended users.

Importantly, there’s often no single reason why a data project is not where it needs to be. Projects may suffer from one or more challenges that individually would not derail the project, but that collectively can become a strain on achieving key deliverables.

These challenges can affect the unlikeliest of candidates: organisations that are broadly on-board with data; that have an actively engaged staff who want to be more data-driven and who are constantly bringing new ideas or approaches to the table; and have successfully secured internal funding off the back of a strong data-driven business case.

Even with all these things, success can still be elusive.

When data projects start to slip – such as by failing to achieve key milestones, objectives or expectations – it can lead to doubt over whether the business case is as sound as it once seemed. It can also lead to negative perceptions of – and negative sentiment being directed towards – the data platform itself and particularly to the vendor technology that underpins that platform. But technology is rarely the root-cause of the problems. In a lot of cases, the chosen technology can adequately perform the task it’s been purchased for.

Often, the path to improvement is about taking the energy and enthusiasm that exists internally for data and for the program of work and more effectively channelling it to produce results.

Minor tweaks in communication, vision, ownership and prioritization can drive major improvements in ability to execute and value.

These improvements can materially steer a data project back onto the right path towards utility insights and value creation.

Communication and leadership

One key reason why data projects fail is because a gap exists between the executives that signed off on the project and the teams executing it.

A solid strategy and cohesive direction may exist, but if it isn’t effectively communicated and cascaded to the implementation teams, then the strategy won’t be as successful as it could be.

Gaps may also be present in organisations where data teams have been empowered to come up with their own overarching data strategy for the organisation.

Empowerment can lead to improved accountability, ownership and results. However, business leaders still need to provide context about the business environment, desired benefits, financial constraints, timeline constraints, risk appetite and so on, to empowered teams. Without executive input, strategies developed closer to the coalface may prioritise different goals, which can influence the data platform that is designed and provisioned.

This can lead to misaligned expectations, misunderstood requirements and underutilised data insights. It may also lead to wasted efforts, as teams may work on incorrect assumptions or incomplete information, resulting in outcomes that do not meet the project’s – and executive sponsor’s – intended objectives.

Placing project goals ahead of personal goals

Data projects provide tremendous opportunities for individuals to grow and upskill in emerging technologies. This enthusiasm should be encouraged, but also harnessed in a way that benefits the project delivery. Where things can go wrong is if the desire to develop certain skills leads individuals or teams to go in a certain direction and make technology decisions aligned to skills development, instead of what the organisation really needs to progress its data capabilities.

Similarly, individuals who have previous experience with certain technology stacks, ecosystems or methodologies may seek to prioritise their ideas or methods over what’s best for the project. While direct exposure or first-hand experience with a data technology can be invaluable, there are other factors that need to be considered as part of the decision-making. These individuals also need to remain open to other ideas and input and demonstrate that openness. Otherwise, it can cause other team members to disengage from the project, leading them to withhold their ideas and suggestions.

When team members prioritise their personal goals over the project’s objectives, it can lead to conflicts of interest and a lack of focus on what’s truly important for the project. This misalignment can derail the project from its intended outcomes, particularly as decisions are made based on individual aspirations rather than project success.

Data projects are still projects

If a data project is going off the rails, it could also be because key tenets of project management are not receiving due focus, rather than due to anything specifically related to data, vendor tools or platforms.

Project management is its own specific discipline and skillset. It may be that there are too many people with a hand in managing the project, but that no single person exists to coordinate resources and ensure everyone is moving in the right direction.

Alternatively, there may be a single overarching project manager, but they are swamped or overwhelmed by the complexity of the program of work and could benefit from some additional support to bring the program back under control.

Rather than immediately pointing the finger at the technology, organisations with data projects that are veering off track should look internally first, focusing on recognizing and resolving issues with the human aspects of the project.

The integrations and data sharing practices across various organisations fosters the exchange of valuable information and generates more and more data.

This article was originally published on 17 September 2024 in Intelligent CIO