The 7 Most Common Integration Issues… AKA Why system integration projects fail!

Sometimes integrations just stop working. Sometimes they cause datasets to become corrupted. Sometimes this results in unnecessary costs being incurred or potential revenue lost. Sometimes they cause a customer service disaster that destroys reputations and relationships. Whatever the failure mode, integration problems have a massive impact on the business.

Integration projects rarely fail due to a single root cause. The majority of integration issues are usually due to a variety of factors that combine to cause a problem. But it doesn’t have to be this way. By understanding the things that cause integration failures, you can implement the appropriate controls and contingency measures to make them far less likely.

Integration issues are typically caused by:

• Poorly defined requirements
• Integration design weaknesses
• Integration infrastructure issues
• Personnel related challenges
• Data issues
• Lack of contingency planning
• Integration development process issues

Common integration failure mode 1 – Poorly defined requirements

There are many types of integration and many ways in which any particular integration project may be approached. Each option will have different pros and cons and there are always compromises to be made. Unless the business’s requirements of the integration are clearly defined it is likely that the delivered solution may not satisfy every stakeholder and deal with every scenario or risk vector.

Commonly missed (or insufficiently defined) requirements include:

• Insufficient knowledge of target systems and upstream / downstream process flows
• Lack of awareness of data quality, volume or transactional flow rate
• Failure to consider end-to-end security requirements
• Operating parameters (e.g. anticipated throughput, expected transaction timings etc) not sufficiently defined
• Error and exception -handling processes are undefined
• Data contention and record-locking / change roll-back policies are not defined

Performing thorough up-front due diligence on the integration scope is essential if problems are to be avoided. Time invested in scoping, requirements analysis and assumption testing is never wasted.

Common integration failure mode 2 – Integration design weaknesses

People are often blinkered by their personal experiences and skills or by the limitations of their toolset of choice. This can lead to sub-optimal integration designs that have inherent systemic problems that only manifest when the deployed solution is put under real-word loading.

Commonly observed integration design issues include:

• Using the wrong approach / methodology for the use case or scenario
• Limitations of the technical platforms / tooling being used
• Insufficient documentation leads to inaccurate assumptions or poor decisions
• Selected architecture unable to handle the anticipated working load
• Selected architecture unable to flex sufficiently to accommodate peak load spikes
• Published APIs do not cover all integration scenarios / use cases
• Inefficiencies within the deployed approach prevent scaling

Whilst not every problem can be designed out of a solution the vast majority can be. Methodology selection and tooling choices should be based upon a broad array of inputs and take advantage of the collective experience of the team across many integration scenarios.

Common integration failure mode 3 – Integration infrastructure issues

The technology stack used to implement the integration matters. Every component selected must be up to the job if the integrity and efficiency of the integration is to be maintained. Not all platforms are the same. Each have strengths and weaknesses. Understanding when and how to deploy the various tooling options available is essential if the integration is to be successful.

Integration infrastructure issues that typically cause problems include:

• Middleware platform instability / poor availability
• Architectural issues such as bottlenecks or single points of failure that cause performance degradation or integration outages
• Incomplete support for transfer mechanisms / protocols or open standards
• Integration infrastructure not being monitored appropriately
• Poor configuration choices
• Excessive resource consumption / Unnecessary processing overhead

Even best-in-class technology will fail if it isn’t deployed correctly or is being used in a way it wasn’t designed for. Monitoring regimes and periodic reconfiguration efforts are needed to ensure integrations remain stable, secure and, efficient.

Common integration failure mode 4 – Personnel related challenges

People are at the heart of all integration projects. The success or failure of any project is dependent on the resources allocated to it. It’s important to recognise the impact that poor staffing decisions can have on a programme of works. The difference in contribution from good and average developers is huge. Great developers are rare. Getting the best mix of resources you can within the commercial constraints you’re operating under is key.

Common staffing challenges include:

• Availability of suitably skilled personnel
• Experts are unaffordable (and often only necessary for a relatively small proportion of the project effort) within the project budget
• Lack of resource continuity throughout the project
• Technical competence of assigned personnel insufficient
• Self-imposed constraints i.e. Blinkered by their personal experiences and skills
• Demotivating exceptional talent by requiring it to perform mundane tasks

To be successful you need to leverage a variety of skillsets of varying levels of experience throughout the project lifecycle. Accessing the collective knowledge pool and skills of a team of integration specialists helps ensure your project delivers.

Common integration failure mode 5 – Data issues

Data is at the heart of every integration. And yet many people don’t take the time to assess and analyse the datasets they are working with. It is incredibly dangerous to assume that the data being passed is 100% accurate. Integrations should proactively evaluate the data they are accessing or transferring and work within master data management policies of the business if integration errors are to be avoided.

Data related issues that may impact integration performance include:

• Data that is incomplete or erroneous
• Data volume, or update frequency, is higher than anticipated
• Data not provided in accordance with the defined input parameters
• Volume of exceptions exceeds design assumptions
• Duplicated data related reconciliation issues
• Lack of awareness of behind-the-scenes referential integrity requirements

Data integrity matters. Being able to show the provenance and chain of custody for data may be incredibly important. Safeguarding privacy, protecting sensitive information and, assuring data sovereignty is maintained, as well as being able to demonstrate regulatory compliance is essential.

Common integration failure mode 6 – Lack of contingency planning

Every integration stumbles at some point. Whether that stumble causes the business to come to a crashing halt or not depends on whether the integration was designed with service restoration and data integrity strategies in mind. Pragmatism pays. Irrespective of the resilience of the integration infrastructure incidents happen. Planning for the inevitable helps assure data quality and minimises system restoration timelines.

Factors to consider that minimise the impact of an integration outage include:

• Integration restoration process not defined
  • System (re-)synchronisation
  • Transaction queuing / batch processing
• Data integrity checkpoints / snapshots not defined
• Lack of roll-back capability
• Periodic testing of recovery processes i.e. integration outage drills

If an integration is to consistently deliver value it must be maintained and monitored. Procedures related to service reinstatement must be defined and regularly tested to assure the integrity of the end-to-end solution.

Common integration failure mode 7 – Integration development process issues

Systemic integration issues that don’t originate during the requirements capture or design phases are often inadvertently introduced during the development of the integration itself. Standard project governance processes and rigorously assuring the effectiveness of controls helps to mitigate the risk.

Common reasons why the delivered integration fails may include:

• Insufficient testing
• Insufficient integration hardening make it susceptible to exception -based issues
• Lack of governance / project assurance to ensure what needs to get done is done
• Assumptions not validated / tested and proven
• Last minute design changes, scope creep and arbitrary delivery dates
• Lack of telemetry incorporated into the delivered solution to enable integration health to be monitored

Building an integration is similar to building anything else. It requires the consistent and focused application of project management controls to ensure that what is delivered is fit for purpose.

To conclude…

Integrations are important. Many business processes (and business’s) are dependent on integrations working flawlessly every time. And yet many Systems Integrators fail to treat integration projects with the seriousness they deserve or allocate the resources needed to deliver them. This is always a false economy. Failed integrations are always a problem. The connections between systems facilitate the process. If those connections fail, then the process fails. Sometimes with catastrophic results.
Being aware of the most commonly observed issues with integrations is a good start. When planning your integrations, we would strongly recommend that you use the above list when evaluating proposals and project plans to ensure that the most significant risks are managed.

The Multishoring approach and how we stop integration projects failing

Integration is what we do. Our team has witnessed the common integration failure modes outlined above and understands how to mitigate the risks, avoid common pitfalls and, ensure your project is successful. Whether we’re in a turn-around situation helping to get an off-the-rails project back on track, or working on a net new integration, we focus on the things that truly matter and make it work within the constraints you define.

Depending on your appetite for risk, available budget, or timeline, we will work with you to define the optimal schedule of works so that you feel confident in the delivered solution and its ability to run as designed issue free for as long as you need it. We don’t over engineer projects to artificially inflate the scope of works and we don’t skimp on activities that will materially impact the integrity of the delivered integration.

Integrations may be complex, but they do not need to be complicated. We use our experience and expertise to take challenging integration scenarios and make them happen. We do what is needed to make your integrations stable, secure and scalable so that you can focus on other things.

Share article on: