Help

Page tree

Building a Custom ESA

This topic provides some direction for creating a custom External System Adapter (ESA) implementation, based on the Sample ESA provided for free with every  Agiloft KB. Follow the steps provided in Using the Sample ESA, replacing the default implementation with your own ESA details. 

Read and familiarize yourself with the Javadoc comments provided with the ESA package. The ExternalSystemAdapter.java document in com/supportwizard/sync/interfaces/esa lists all of the ESA methods and explains their semantics and parameters. In addition, you should read and understand the Sample ESA code. 

Create the ExternalSystemAdapter Interface

You could start by using ExternalSystemAdapterBase as the base class. The Java file is located at com/supportwizard/sync/interfaces/esa. ExternalSystemAdapterBase implements auxiliary methods in a default manner suitable for most ESAs. If you need some advanced functionality later, you can always override these default implementations. 

  1. Based on your setup, decide on a communication style - Command Line or HTTPS

  2. Open JavaExecutableESA.java, located at com/supportwizard/remoteesa, and look for the following lines:

    JavaExecutableEsa
    // Replace the following line with "new MyESA()"
ExternalSystemAdapter esa = new SampleEsa();
// CL ESA variant
EsaRunner.runCommandLineEsa(esa);
// For HTTPs ESA, comment/uncomment following line (need to supply ewServerURL and externalSystemID)
// EsaRunner.runHttpsEsa(esa, ewServerURL, externalSystemID);
  3. Replace the following line: ExternalSystemAdapter esa = new SampleEsa(); with your new class.
    1. For a CL ESA, just reassemble the example using make.sh or make.bat scripts. You now have your own ESA ready to be launched. 
    2. For an HTTPS ESA, comment the line: EsaRunner.runCommandLineEsa(esa); and uncomment the line: // EsaRunner.runHttpsEsa(esa, ewServerURL, externalSystemID);.
    3. You will need to make further changes, both because you will need to manage the ESA life cycle yourself, and because you will need to supply the  Agiloft server URL and External System ID values. 

Implementation Tips

  1. Always use logging instead of printing on the System.out. Note that in  Agiloft's configuration JBoss log4j is pre-configured to log into cl-esa.log.
  2. Using the ESA provided parameters instead of your own private configuration mechanisms will make it easier to use the ESA in practice.
  3.   Most of the ESA methods should never return null, unless explicitly stated in the Javadoc.
  4. If a field is absent in ExternalRecord, this means 'do not change the field value if possible'. If the field is present but null, this means 'make the field blank'. 
  5. Most important ESA methods such as metadata retrieval and CRUD operations are always framed by startSync() and endSync() calls. You should normally establish connections in startSync() and close them in endSync(). 
  6. If you do cascade updates, return true from the needSyncAgain() method. Otherwise,  Agiloft will never know you did the update and will not pick up the changed data. 
  7. When mapping relations, check that:
    1. Both ends of a relation are mapped. For example, if you have a relation between Contacts and Cases, you should first map both Contacts and Cases to the  Agiloft tables. 
    2. There exists a Linked Field in  Agiloft between tables. You can check this at Setup > Tables > [select table] > Edit. 
    3. If a relation is required, the linked field should also be required.
    4. The multiplicity of a linked field and its relation should also match. 
    5. To map external relations to linked field sets in  Agiloft you may specify either 'direct' mapping - equivalent to manually entering values into fields - or map the whole set of fields to some relation in an External System. Note that you cannot combine both direct and whole set mappings - if a field is already directly mapped, that entire set of fields becomes unavailable for relation mapping. 
  8. If you need to run a clean sync - that is, before the systems are already synchronized - do not just delete records in both systems, as this will result in delete propagations on the next sync. Instead, you can optionally delete the records in both systems, then you should click the Reset Records Peering button at the bottom of the Sync Configuration wizard.
  9. The Remote Proxy has a timeout to reconnect after the sync finishes. By default, this is 30 seconds and it may cause the Waiting for ESA to connect screen to hang for a short time. The polling period timeout can be changed in the Remoting section of the Sync Configuration wizard.
  10. If your ESA reads a large amount of data, consider implementing the ExternalSystemAdapter.getModifiedPaged() and ExternalSystemAdapter.getDeletedPaged() methods. 

Delayed Updates

In some cases, an ESA cannot, or should not, immediately perform CRUD operations. A common case is 'bursting', when multiple operations can be served within a single request to the external system. Doing every operation in a single request might be too expensive - it can be ten or even 100 times slower. Another example is updating an External System in an asynchronous manner.

If the ESA decides to delay an update, it should throw ESaRecordDelayedException from the CRUD method, supplying a unique structure-scope string token, within a startSync()/endSync() period. This token is later passed into checkDelayedXXX() methods to get the actual operation result. If an operation failed and an exception should be thrown, the exception should be thrown from the checkDealyedXXX() method. It is possible to throw the EsaRecordDelayedException again. The number of update attempts is limited by sync to prevent infinite loops. 

Localization

Some methods have a Locale argument. The ESA follows the Java convention in naming locales: <language-code>_<country-code>_<variant>. Examples are en_US, ru_RU and Pt_BR. See Localization for more details. 

If the ESA doesn't support the required locale or doesn't support localization at all, it should return an American English label and hint. 

Troubleshooting Tips

  • As a rule of thumb, whenever you experience unexpected behavior check the esa.log and cl-esa.log. These reside in the current directory of the Remote Proxy, that is, the directory where you run the java -jar esa.jar command. The esa.log file will hold the remote proxy logging, including the XML message dumps, whereas cl-esa.log will only contain logging from your ESA. Most of the time these logs will contain relevant information about the events. ESA logs are very detailed and contain full XML message dumps, as well as debug information. 

  • Save your configuration before connecting an ESA to the server. Note that it is auto-saved when you click Next on the General tab. If not, you may see the following exception message:

    Error connecting to an unsaved configuration
    anatoly@anatoly
syncHome> java -jar esa.jar
Agiloft ESA Remote Proxy running.
Connecting to http://anatoly:8080 (External System ID is 123) to get sync
parameters...
IO Error (see esa.log for more details):Can't connect to
http://anatoly:8080/gui2/sync/connect?extsysid=123&acceptssync=false(http://anatoly:8080);
nested exception is:
com.supportwizard.sync.interfaces.transport.TransportException: Server error:
500

    This error commonly appears when running the ESA before leaving the Sync Configuration wizard.

  • Check that the Remote Proxy connects to the correct server and port, and has the right External System ID. If not, please use the full syntax for the connection: java -jar esa.jar -ewhost anatoly:8080 -id
    ewhost denotes the server name:port, and ID standards for the externalID in the Sync Configuration wizard, at the bottom of the General tab.  
  • Check that the command line cmd/c xxxxx\run.bat is correct.
  • Never use System.out.println for debugging. You will never see the message because it will be hidden bet ween RemoteProxy java -jar esa.jar and your ESA run.bat processes. Instead, use log traces. 
  • Unit test your code. it is difficult to debug CL ESA applications, because the process is automatically started by the RemoteProxy app. It is much easier to debug unit tests. 
  • Check that your ESA follows the Javadoc standards from the ExternalSystemAdapter interface. In particular, check the return values for create() and update() - these should never return null. 

If you still have problems, please contact Agiloft support and submit a bug report. Please provide the exact steps to reproduce the problem and all the necessary credentials and code. 

Additional Areas for Discussion

A few message or data types need special discussion.

Deletions

For many systems, tracking deleted records is problematic. The  Agiloft HelperAPI can simplify this task. If you can't easily track deletions in the external system you can either:

  • Enumerate all record IDs known by  Agiloft and check whether those records actually exist at the moment, or
  • Report deletion events to  Agiloft when deletion happens in the external system. The latter is usable if you can somehow obtain deletion notifications from the external system.

A more sophisticated approach to ID enumeration with better performance is possible if your system uses never repeating, always increasing auto-incremented IDs. If this is the case, you may process the countRange message and send a detectDeleted message to HelperApi. It will then apply a dichotomy method - binary division - to find deleted IDs. This will be considerably faster than comparing known and currently existing IDs one by one, especially if you can have a fast implement count A < id < B operation and the number of deleted records is not large.

Transactions

Sync is transaction-less, because not all external systems support transactions or expose them over integration APIs. If an external system requires transaction usage, the ESA should behave as it would with auto-commit on in SQL, meaning it should execute every operation in a separate transaction.

Collections

Collection fields may hold complex objects, which can be mapped to  Agiloft tables. For this reason, getFields, getRelations and getCollections calls should accept both structure names and collection IDs.

Note: An ESA must ensure that structure names and any collection IDs use different name spaces.

Locking - Clashing Modifications

To deal with concurrent data modifications, Sync employs an optimistic locking strategy. When the ESA is asked to modify or delete an external record, it is given a last seen timestamp. If the ESA detects that the record was modified after that time, the ESA should respond with an OptimisticLockingFailure exception and not modify or delete the record. If an external system uses the pessimistic locking approach internally, the ESA should wait until the lock is granted first, or else report a record update failure if waiting for the lock is not reasonable.

Currently, delayed updates due to locks are not supported, but this feature is likely to be added in the upcoming releases because it may speed up syncn appreciably. It is a good idea to make your ESA ready to throw some exception when a pessimistic lock is detected - this might be employed in the next Sync release.

When a record update/create/delete error occurs, the ESA should respond with EsaRecordException. Such exceptions indicate a single record related error, probably recoverable in the future and do not stop the syncn cycle. On the next cycle, sync will try to re-synchronize the erroneous records.

Cursors

In some setups, getModified/getDeleted may return a huge amount of data. It may be inefficient or even unfeasible to transfer and process such traffic in one call.

To overcome this, Data Cursors can be employed. A Data Cursor is a paginated stream of data. When requested, cursor "core" information such as ID and page number is transferred from the ESA to  Agiloft.  Agiloft requests actual pages data from the cursor passing readDataPage messages supplying cursor ID and page index.

Cursors are a more advanced technique than the simple return of all data. Their support is optional in the ESA. For the getModified function,  Agiloft first sends getModified, which has a chance to return all data at once. If and only if the getModified response is a USE_PAGES string and not an empty result,  Agiloft sends getModifiedPaged, which returns a cursor. The ESA may choose the transferring approach in runtime, depending on the amount of the data.

When a data cursor is fully read, the cursor is closed. However, due to possible communication failures, this may not happen. The ESA should retire cursors in 15 minutes since the last readDataPage or leaseCursor message.

ESA Parameters

The ESA is likely to have configuration parameters, such as external system host, login, password, etc. While the ESA may maintain them by its own configuration mechanisms, it is often desirable to have them in one place, to simplify administration and maintenance.

For this purpose, Sync may store some ESA parameters within the  Agiloft database, providing a simple GUI for changing them. To do this, Sync asks ESA for the names and types of parameters to manage. The ESA provides parameter meta-data such as type, default value, required / not required, and Sync shows them in the configuration GUI, ESA Settings page.

ESA may later request its parameter values during the syncn cycle, using parameter names. See getParameter message. Though the number of parameter types is limited they are likely to cover most of the ESA configuration needs and simplify administration greatly. If, for example, your ESA will require some configuration file, it is a good idea to put the local path to it in an ESA parameter.

Parameter names starting with config are reserved for getting Sync Configuration settings. Currently, the following are available:

  • config.pollperiod - polling period
  • config.commandline - command line to invoke a third-party command line in the ESA.
  • config.esatype - ESA type name.

It is likely that only config.pollperiod is of interest for ESA developers, and only for HTTPs ESAs.

Timestamps

The time resolution for sync is one second, and all timestamps are truncated to the closest second. All timestamps passed to or received from the ESA are in UTC - GMT, Greenwich time. It is the responsibility of the ESA to translate external system timestamps into UTC, taking time zones and daylight savings into effect.

Sync may compute the system clock difference between the external system (ESA) machine and the  Agiloft host machine. To do so, sync passes a getCurrentTime message to the ESA. If the ESA wants the clock difference to be taken into account, it should respond with the current external system UTC time. Alternatively, the ESA may respond with an empty result, setting clock difference to zero.

If you are sure that the  Agiloft host and external system host have their system clocks synchronized, for example via Network Time Protocol, NTP, it is a good idea to turn clock difference computation off by responding with an empty message. Since the Sync time resolution is less precise than NTP, it will never get better results than it could if the system clocks are really synchronized.

Due to limited time difference precision, it is possible that some records may stay unsynchronized. Though the chance of this is very low, it is always better to have machine clocks synchronized and time difference detection off.

If you are unsure that clocks are synchronized, for example, your ESA can be deployed on a number of machines, you should run getCurrentTime. The time difference calculation works best if the ESA actually measures time in the middle of a message delivery + message processing + message delivery back loop, but in practice, it is fine to just measure and respond as fast as you can.

The Time difference calculation is performed on each Sync cycle, so if system clocks are corrected between synchronizations, Sync will work correctly. It is however, presumed that system clocks are not changed during synchronization.

Last seen timestamps are always passed just as they were received from the ESA last time and thus are not affected by system clock changes.

  • No labels