Salesforce Data - To Push, Pull, Batch, Sync, Real-time ...

There are many ways to transact data between Salesforce and an external system. This post considers the questions you need to ask, the leading options, and the concessions you will need to make.

The Questions

Before choosing a transaction approach first understand your data and capabilities:

• What is the data direction and which system initiates the transaction?
  • An event in Salesforce pushes data out
  • An event in Salesforce pulls data in
  • An event in the external system pushes data out
  • An event in the external system pulls data in
• What are the transaction characteristics?
  • Are the transactions constant or intermittent?
  • How many transactions will occur in 24 hours?
  • What is the size of the packet payload? What is the maximum size of each transaction?
  • Do messages need to arrive in the same order they were transmitted?
  • What is the maximum number of simultaneous transactions?
• What is the required data quality?
  • Does the data need to be 100% accurate, every time you view it?
  • If yes, favor using Salesforce as a single source of data
• How available must the data be?
  • Is the data required instantly (i.e. present in both systems)?
  • Is a short propagation delay permissible (while retrieving it from the external system)? 
• What is the required level of security?
  • Is security a key concern?
• What are the capabilities of the external system?
  • What are the technology and customization constraints?
  • What are the API constraints?
• Do I understand the constraints imposed by Salesforce?
  • How will my choices be impacted by Salesforce constraints?
  • Can I buy more capacity to mitigate the constraints? 
• Can I access skilled resources?
  • Are the skills available to create the required functionality in Salesforce and the external system?
  • How will my choices impact project cost and timeframe? 

System constraints are an important part of your decision-making process so always consider worst-case scenarios e.g. your system should not fail when social media campaign goes viral. 

The following is a summary of the leading technologies used to transact Salesforce data.

Outbound Message

An event in Salesforce pushes data to an external system.

An outbound message uses a POST call to send a SOAP message (XML) from Salesforce to an external endpoint. Messages can contain up to 100 notifications each containing an object ID and a reference to the associated sObject data. The endpoint processes the message and returns an acknowledgment saying that the entire message was processed correctly.

• Messages are queued until they are sent
• Messages are retried if the delivery fails. The order of execution and delivery time cannot be guaranteed
• Workflow is used to initiate the message so Salesforce development is not required. Only formula fields on the immediate object can initiate the send event

Callout

An event in Salesforce pushes data to (or pulls data from) the external system.

A Salesforce callout uses Apex code to execute an HTTP request method (GET, POST, etc.) to send and retrieve data. The data packet is typically in the XML or JSON format. Unlike outbound messages which are can only be secured by SSL, callouts support more options like Basic Authentication and OAuth.

• Callouts can be synchronous or asynchronous
• Callouts have no inbuilt retry process
• Callouts have constraints that limit the number of parallel calls, time for a single call, size of each transaction and number elements per transaction

The callout constraints can make it difficult to process data in volume.

Data Exports

A feature in Salesforce that pushes data to an external file system.

Salesforce has two inbuilt features that support the manual transaction of CSV files manually or by a schedule. The Data Export Wizard is accessible in Salesforce and supports for weekly and monthly exports. 

Data Loader

An external application that pushes data to Salesforce.

The Data Loader is an application that runs via a UI or from the command line. The Data Loader can also be used to import data from CSV files.

Salesforce REST API

An event in an external system pushes data to (or pulls data from) Salesforce.

Salesforce APIs expose data to an external system via a web service. An event in the external system (client) initiates the data transaction with Salesforce. The REST API is a simple and powerful web service ideal for use in web and mobile applications.

• Stateless synchronous REST requests using HTTP verbs using JSON/XML/URL
• Granular queries can be initiated in the external system in real-time or via a scheduled task
• The number of API calls in a 24-hour period is limited by the type of Salesforce org and the number of org licenses
• In a single API call, a request batch size of 2000 records is applied. It is easy to loop through and request more records but each fetch costs an API call. When pushing to Salesforce only a single record can be updated at a time

Using APIs is often the easiest approach as the enforced constraints are well understood, and can be mitigated with relative ease. In general, APIs require little or no development in less Salesforce skills but it is likely custom code will be required in the external client.

Salesforce Composite API

An event in an external system pushes data to (or pulls data from) Salesforce.

The Composite API overcomes the constraints of the REST API by executing multiple REST API requests in a single call. The output of one request can become the input of another. This orchestration enables the collective requests to be returned in a single response and counted as a single API call.

• Synchronous REST requests using HTTP verbs using JSON/XML/URL
• A single API call can have up to 25 subrequests
• A transaction can be rolled back if an error occurs
• In a single API call, up to 1020 records can be created, read, updated, or deleted. 

Salesforce SOAP API

An event in an external system pushes data to (or pulls data from) Salesforce.

This API is similar to the REST API but uses a different transaction protocol.

• Strongly-typed synchronous requests using HTTP and XML standards
• Granular queries can be initiated in the external system in real-time or via a scheduled task
• The number of API calls in a 24-hour period is limited by the type of Salesforce org and the number of org licenses
• In a single API call, a request batch size of 2000 records is applied. Each request for more records costs an API call. When pushing to Salesforce the batch size is generally 200 records 

 Salesforce Bulk API 2.0

An event in an external system pushes data to (or pulls data from) Salesforce.

The Bulk API makes it simple to process data from a few thousand to millions of records. Bulk API queries like inserts, updates, and deletes, all run asynchronously in the background. 

• REST-based bulk extraction in CSV, JSON, XML format
• No guarantee of completion
• A batch can contain a maximum of 10,000 records and 15,000 batches can be submitted in a rolling 24-hour period

The Bulk API has understandable limits and is widely used for transacting data where timeliness is not a factor.

Other APIs

It seems Salesforce has an API for every occasion but you will need to understand how their constraints will impact what you wish to achieve. The challenge is to find any gotchas before going live.

Conclusion

The options above are native to Salesforce. Many third parties offer integrations like those on the AppExchange, others like DBAmp, and if you use Sitecore, our own S4S integration.

If you have questions about accessing Salesforce data please contact us. It is, after all, what we do!