May 22, 2013

Syncing Salesforce Changes to an External System with Future/Schedule Architecture

Many integration projects involve syncing Salesforce data to external systems. For example, you may need to sync contacts and leads along with their related events and tasks to a home-grown portal, or sync financial information to an external accounting system, or sync sales data to a system that allows better mobile experience for certain verticals.

The business logic for some of these projects is very complex, but the general design of the solution usually follow known patterns. However, certain requirements present several considerations that make writing the more common integration code a bit trickier. Here are a few such considerations:
  • Sync events get fired by triggers, but triggers can’t make callouts, so syncing code must run asynchronously (@future).
  • Trigger execution order is not guaranteed, so it is possible that your sync trigger will be fired by a process that is already running asynchronously (which will create a runtime error).
  • Depending on the activity and the number of objects that get synced, you can reach callout limits.
  • Packaged sync applications often clash with other packages that also work asynchronously. Packaged apps also have wide variability of activity, so there is a good chance to hit callouts limits.
  • Any runtime error results in a sync event that is not transmitted to the external system.
  • Any delayed process needs to run soon after the triggering event occurred. Also, a record that is triggered again before its delayed processing occurs, should not be transmitted twice (unless needed by the external system).

To avoid some of these issues, or at least make it less likely for a transaction to be lost, I’ve come up with a design pattern that combines both almost-immediate (future) and delayed (scheduled) methods to communicate with external systems that need to be synced. The basic idea for this design pattern is to always try the immediate processing as the first option, but also allow for a scheduled job to be created if immediate processing is not possible at the time.

Here’s how it’s done… Please note, that I omitted (added as a comment) any business logic that is not important for the design pattern.

First of all, create two new objects to handle scheduled syncing:
  • An object called Synchronize__c to hold Ids of records that need to be synced, as well as any additional information that is needed when processing jobs. For example you may want to save the type of trigger that fired the sync event (insert or update). This object will be queried by a scheduled process, and allow the scheduled processing to reuse the normally triggered sync code.
  • An object called  CronJob__c to hold Cron Job Ids so we can keep track of what jobs were created and be able to remove them after they execute. This is necessary because querying the cron job object does not give you all the info you need to clean up old jobs.

Next, create a schedulable class called SynchronizeSched.cls that will be used to execute scheduled code. Keep it empty until the rest of the code is written:

Next, create a class (Synchronize.cls) to handle all the business logic and special processing. The next four methods will be added to this class, starting with a static class called ProcessFutureCallout that receives a set of Ids. Since the execution is starting from a trigger, specify @future(callout = true) before the method declaration to allow the class to make a callout. This method is what you would normally write to sync records to an external system, and will include any business logic that needs to run before the callout, the actual callout, and the response handling.

Now we can create a method called ScheduleCallout that can handle the delayed sync. This method will be called instead of ProcessFutureCallout when an asynchronous call cannot be made immediately. Within the method, we need to save the Ids of all the records that need to be synced, as well as any other important info, to the newly created object Synchronize__c. Then, create a cron job and save its Id to the new object CronJob__c. Note that in this example, the job is scheduled 1 to 2 minutes into the future. By adding two minutes to now, the scheduled job will run on the second to next whole minute. I think that this is the shortest time period that should be scheduled because if you add just one minute to now(), you run the risk that the code would execute around a whole minute and create a job that cannot be scheduled (You'll get the error "Based on configured schedule, the given trigger will never fire"). Of course, the delayed job can be scheduled for an hourly or daily sync. Not included in the sample, but a good addition, is a quick check that a scheduled job is not already created before creating a new one.

Now create a method ProcessTrigger to be used as the entry point from the trigger. At the beginning of the method, add the normal logic that figures out if you need to process the changed record(s). Once you know which records need to be synced, you can figure out if it is possible to continue processing asynchronously. If it is not possible, you can schedule the changed records for later execution. This method can check for all kind of governor limits and react as needed.
We have to create one more method called ProcessScheduledCallout to handle the scheduled logic. The schedulable class will call this method to run the delayed sync. This method actually does not do much other than get info from the database, call ProcessFutureCallout, and then clean up the database. Since ProcessFutureCallout already has all the logic we need, there is no point rewriting that here.
Lastly, update the schedulable class from the first step to call the ProcessScheduledCallout() method: