Almost every day we hear of a new integration challenge, and that is only in our niche Sitecore-Salesforce space. This article looks at why it is not always easy to expose the data in one system to another.
Integrations are usually traditional "syncs" or real-time "gets" from a single source of data, or some combination of both.
A sync copies the data in one system to another at regular intervals however this can introduce undesirable complexities:
- Lumpy data transfers create unknown data states in both systems
- Syncing cannot handle deletions in the receiving system. Syncing often resurrects deleted data
- Good data changed in either system can get overwritten
- Cannot implement complex or granular transactions. Syncing cannot pre-test before updating objects. Related records cannot be easily handled
- Syncing typically uses an object's Last Modified Date so updating an irrelevant record field causes the entire record to sync
- Cannot easily sync a varying number of objects in one system to the other
A safer and far more flexible approach is to transact in real-time. This facilitates a "single source of truth" that eliminates the issues of data integrity, quality, missing information, and allows complex transactions to be processed. Despite this, real-time syncing does come with fewer challenges:
- Remote systems can introduce propagation delays
- More coding can be required to deal with complex types (that syncing cannot handle)
Integrations need an event to initiate the transaction of data between systems. These events are often fired in the wrong system at the wrong time. This leaves the data in an unknown state which degrades the usability of the integration. Integration events come in several flavors including website visitor actions (e.g. mouse clicks), an interval scheduler firing, a web page loading, a user starting an application, or an incoming request from an external system.
To understand the challenge, consider a situation where we need record data in System A to be exposed in System B. Initially the "same" data in both systems may be different. The true value of data in System B is unknown - the last update may have been many hours ago, or a user in System B may have manually changed the data after the last automated update.
Let's examine the impact of where the event occurs. The first scenario is when the event is in System A. Quite simply, code in System A updates the data in System B through an API call ("67890" becomes "12345").
It gets more complicated if the event is in System B because B needs to send a GET request to System A (dashed line) that tells A to transmit the data. Doing this requires both systems to have an API and System B to have some calling code.
All this raises several questions:
- What if I can't or don't want to write code in one system or the other?
- What if one of the systems imposes volume constraints on its API (Salesforce does this)?
- What happens with inserts and deletes e.g. if I create a new record in B, how will A know?
We've only looked at moving data in one direction. Making it work in both directions compounds the challenge manyfold. There may not be a simple answer to your integration challenge - the real world just gets in the way with constraints such as:
- A common record identifier may not be present in both systems
- A legacy system cannot be modified
- A system may not have a suitable API (cannot scale, has limits, lacks the right query methods, etc)
- Development expertise is not available to modify a system
- System owners will not permit their system to be changed
- The bandwidth between systems is insufficient
In some cases, the workarounds can get ugly e.g. screen-scraping to access data in systems that don't have an API or transferring identifiers between systems using email click-throughs.
If you are lucky to have a problem in the Sitecore to Salesforce integration space, we have probably seen it before and are happy to help. For other system integrations, we recommend the real-time single source of truth approach if this is possible. Good luck.