Data Integration in Salesforce: A Quick and Simple Overview

Definition:

Copying information from one or more sources to one or more targets.

Examples:

• a visitor registers on a Salesforce community, then a process calls a web service to validate his/her email address
• a Salesforce user clicks a button on an Account page, then a LWC calls Google Maps to calculate the directions from the user’s address to the account’s address
• a batch process in Salesforce collects closed won opportunities in the past 24 hours to send to a fulfillment service
• a third-party chat bot service calls a Salesforce web service to notify that a chat has been closed
• a process in another platform connects to Salesforce API to send new customer records
• an integration process in Mulesoft collects accounts created or updated since the last integration run and copies them to the marketing system
• a script to run DataLoader and upload CSV files from a folder to create records in Salesforce

Methods:

• API – callouts to services using HTTP protocol
• Middleware/ETL tools (Boomi, Informatica, Jitterbit, Mulesoft, …) that will deal with the APIs

Types of Integration:

UI integration

UI integration is not part of this document. The common scenario is of a user having to toggle between Salesforce and another application. An UI integration would make that toggle transparent. Example:  CTI apps, hyperlinks, embedded iframes/canvas apps, web browser extensions, etc.

Point to point

One to one integration: your Salesforce org calls another service to pass or to fetch data (or another process calls a Salesforce web service)

It has a big disadvantage:  each point will need a connection to the other point, more expensive to maintain since each pair is a separate integration

Hub and spoke

A central hub receives and sends data to the systems and is responsible for routing data between them.

Each system will only need one connection to the hub instead of a separate connection for each of the other systems.

The hub has to implement a generic business model, transform each application data to/from the generic model and deal with conflicts, ambiguities and compatibility issues.

Enterprise Service Bus

Evolution of a hub and spoke:  can do routing between one or more systems and orchestrate the order in which the integrations occur.

Types of Data Synchronization:

• Unidirectional – data flows from source/master to target
• Bidirectional – data flows from one direction then the reverse direction with special handling to not overwrite each other’s updates or adopting a policy of “the latest update wins”
• Two-way-unidirectional – object is synced in both directions but in different fields and there is no overlap of data

Integration patterns in Salesforce:

• request and reply – synchronous, the remote service responds quickly
• fire and forget – asynchronous, the remote service takes long to respond and we can’t wait
• batch data sync – asynchronous, involves larger amounts of data to be copied during off hours
• remote call-in – remote system calls Salesfoce APIs or Apex services to enter data
• UI update – event-driven, user is notified on-screen when relevant data changes
• data virtualization – external objects using Salesforce Connect

Questions to ask when designing an integration

Think of it as both the future ongoing integration and the initial data load as well:

• How and when the data will be ready? What is the format of the data?
• Who is the subject matter expert?
• Are there any security concerns or guidelines?
• What middleware/ETL tool to use?
• How often should the data be copied and in what directions (unidirectional, bidirectional, two-way-unidirectional, to Salesforce, from Salesforce)?
• Request integration user credentials to Salesforce (dev/UAT/Prod) and their other systems.
• What Salesforce objects are to be mapped to what external tables/services?
• When the mapping document will be ready for review and who is responsible for it?
• When will the target SF org be ready? (UAT or Prod)
• Who will do the UAT/QA and what is the org?
• How long should it take to go live?

Tools/items that help with integration:

• jwt.io
• Postman, Insomnia
• SOAP.ui
• Salesforce Workbench

Salesforce implementation items that help with integration:

• Ability to disable triggers/flows using custom setting for the integration user
• Store external system Ids as external ids in Salesforce (or vice-versa: store Salesforce Ids in the external systems)
• If a record is inserted/updated in Salesforce during integration, have triggers take care of wiring parent/children records to it instead of the process in the middleware
• Sort children records by parent to avoid record locking errors
• Have a “Source” field to store the name of the system where the record was originated
• Have a “Job Id” field to store the integration’s job id:  this identifies records updated in the last run of the integration
• Have the integration process use upsert (or check before inserting/updating) to avoid duplicates
• Log integration errors in a Salesforce object named Error_Log__c or Integration_Error__c

Other things in Salesforce that pertain to integration:

• Named Credentials
• Remote Site Settings
• Workflow Outbound Messages
• Salesforce/Lightning Connect, Heroku Connect, Apex Callouts
• SOAP API, REST API, Bulk API, Streaming API

Sources/Further study:

• https://www.researchgate.net/publication/47394009_An_integration_strategy_for_large_enterprises
• https://www.centeractive.com/content/enterprise-service-bus
• https://www.salesforceben.com/salesforce-integration/
• https://developer.salesforce.com/docs/atlas.en-us.integration_patterns_and_practices.meta/integration_patterns_and_practices/integ_pat_remote_process_invocation_state.htm
• https://resources.docs.salesforce.com/sfdc/pdf/integration_patterns_and_practices.pdf
• https://trailhead.salesforce.com/en/content/learn/superbadges/superbadge_integration
• https://resources.docs.salesforce.com/sfdc/pdf/salesforce_large_data_volumes_bp.pdf
• Developing Data Migrations and Integrations with Salesforce – David Masri

Personal experiences:

• Most common objects to integrate are (in decreasing order):  Account, Contact, Opportunity, then Product, Pricebook, PricebookEntry, OpportunityLineItem, Quote, QuoteLineItem
• Most common issue in integrations:  client doesn’t have an expert on their source/target system (or lacks availability)
• Second most common issues:  bad API (doesn’t have pagination, filtering, batching), cumbersome API (requires a lot of separate calls or returns cryptic error messages or stack dump instead of error message), lack of documentation
• Demand for integration consulting is ever increasing, up to 3x times from one year to the next