Logging for audit, error, and reconcilation of interfaces

The major concern during the support and maintainance I face:

• Data not sent to destination
• Connection to FTP, database, MQ, DB2 or other platform or application is down
• Exceptions due to data validation errors
• Component of application server is not working

I majorly describe them into three categories

• System: Due to some component or configuration functinality on server/platform we see this kind of error. Yes, bad coding, we all are good coders, however we do mistakes . No proper handling of exception lead it to surprise for us.
• Network: since interfaces interact with so many systems/ application internal or external. Sometimes something is not workting over the network and it is completly out of our scope since this may the owned by some other part of enterprise or may be failure in partner systems. We only can inform them and wait for them to take action.
• Data: What if the application or system using the interface sending data which is not in proper fomat. Few simple expamle are date format mismatch, length of field is wrong for fix length data, mandatory field is missing

Ok, so I am done with the classification, what next?
This solves our simple purpose for error logging and notification

• System error should be logged to internal to error table with description of error and notification sent to development and admin team
• Network error should be logged in error and the framework should provide functionality for re-invocation of interface, however notification should be sent to the different concerned parties
• Data error should be logged in error and notification should be sent to user of the interface. Providing the detail of data error.

This clearly helps to designate the error to concerned parties and they can decide their actions to recover the failure.

Being the part of team responsible for development and maintainance I would be intrested to drill down more in to system errors. Usually their are two seperate roles one administration and other for development, mainainance and support.

• Errors from server/platform for administrator: Since administration team use their own tools for tracking of errors on server and other infrastructure. We can notify them by logging to server log or may be sending notification mails
• Errors from application/interface/service for developers: This kind of exceptions/errors should be handled properly during development, developers and architect should be notified with details in form of hourly/daily reports or email per error. And no errors should be left for surprise and to be logged on server. Additionally we can have a framework for re-invocation of interfaces.

With the above discsussion we can clearly understand keeping a re-invocation framework can keep things align and without any business data loss.

Re-invocation framework: When first time I thought of this idea it sound appealing, but complicated since there are different type of interfaces. However this is acheivable, specially when we compare the effort of re-concilation due to some failures.

Since what all resources need to re-invoke interface are within scope, we just need wrap the request and send to to re-concilation framework. And framework have to be robust enough to re-send/invoke all of interfaces.

Naming standard in Enterprise Application Integration(EAI)

In my 5+ years of experience with EAI, I have few projects and integration tools to understand with proper knowledge and few more which I know from my friends. So what importantly i have seen as naming pattern is :

• a. "interface prefix"+"sourcesystemname"+"targetsystemname"
• b. "interface prefix"+"somenumber"+"targetsystemname"
• c. "interface prefix"+"somenumber"+"systemname"+"subscribing/publishing"

number will be combination of some digit between 0-9 however the series is incremental based on the some logic.

Here let’s take an example of an interface is developed on naming system "a" as above ser_ABC_XYZ. And most of us dealing with integration get a mails or issues saying:

• 1. Order/ invoice/ customer/ payment etc is not transferred.
• 2. The database is down during next 30 minutes, consider stopping the interfaces also.
• 3. MQ Broker has some problem.
• 4. ABC application is failing to send data to application XYZ.

• a. Message type
• b. Adapter type
• c. Source system
• d. Target system

Looking into the various aspects of interface/service I think most important is message type, source system and target system. So we have to have two/three document explaining the code and description

Message Type description (document), this document will explain the mapping of message type to specific number between. Number of digits should be based on the number of message type we have to use for now and in future, mostly three digits XXX (from 000-999) should be fine.

Message Type - Message code
General ledger - 432
Customer data - 234

Source Target description(document), this document will explain the mapping of source/target system to specific number, code should be of two digits XX. Number of digits should be based on the number of message type we have to use for now and in future, mostly two digits XX (from 00-99) should be fine.

Source/Target system name - Source/Target system code
System1 (Data ware house) - 10
System2 (CRM) - 11

Using our newly defined naming standard we can name the interface from CRM to DW for customer data, with any interface prefix we can have the interface name. For example with interface prefix ser_1123410

With this naming convention and the message type, source and target system description document a developer, technical/functional designer or business user can easily co-relate the interface/service and easy life during development, support and maintenance.

PS: comments, feedbacks, suggestions will be appreciated.

xPath

In first place i thought it will be simple, in a way boring. Well not really.
Offcourse www.w3schools.com/XPath/default.asp is begining. Lot more found on http://www.xfront.com/xpath/ . I have used 2.0, though 1.0 and 2.0 is available on the website.
I have used http://try.zorba-xquery.com/ to run and get output of xPath expressions. You can have tools like Stylus, Altova, Oxygen etc or may be some online tool.

Couple of things i have noticed to remmember and get good hold of:
Sequence
Item( can be Node of atomic value)
Functions(over 100 built-in functions) includes String, dateTime, numeric and other functions
Operators
General comparision operators(=, !=, <, <=, >, >=)
Value Comparison Operators(eq, ne, lt, le, gt, ge)
Arithmetic Operators(+, -, *, div, mod, idiv)
Ueages of Regular Expressions in xPath
Predicates
for expressions
If-then-else expressions(you cannot miss else in this kind of expressions, for no action you can use else ())

xQuery functions and Operators are strongly typed. "instance of", "castable as", "cast as" operators are available(equivalent operators are available with JAVA).

Enjoy Learning!!!

xQuery

I had worked with BEA Weblogic Intergation for around 3+ years used xQuery for transformations, however it was sometimes difficult for few complex transformation , Java XML APIs as a part of tool were last resort for such complicacies.

Now after a while with JCAPS, I have started with Oracle SOA 11g and I again same xQuery. Interesting!

So first question for me was, what should be the editor for me? I have used jDeveloper(there are many just google it), where I can run the xQuery samples which I will create during learning.

There are many websites for documents and tutorials
I started with http://www.w3schools.com/xquery/default.asp for initial understanding and being familiar with the technology. For xQuery functions I have tried by hand with http://www.xqueryfunctions.com/. During this I realized xPath is important. Well switched to xPath.

Cannot forget to mention http://try.zorba-xquery.com/ for initial learning; trying lot of sample queries.

Not tired or started fresh day, you can try your hand with use cases:

http://www.w3.org/TR/xquery-use-cases/