A Brief History of Integrating

I was recently asked how I would integrate systems with similar functionality but different code bases and infrastructure. It got me thinking about how I feel I have been solving the integration problem throughout my career and how it (not surprisingly) keeps coming up. My answer was: It depends on what you have and where you want to land. I have many questions: how much of the data is the same? How reliable is the data? Is there an appetite to reduce any duplication? What is the desired recency (age) of the data? Is there a goal to become more homogenous? Regardless of the answers, I have employed several strategies/technologies over my career.

One of the first implementations I worked on was a practical solution for a PC-based application to leverage COBOL code on the mainframe. The company had invested years in the code and was not ready (or able) to rewrite it. We built an LU6.2 gateway to call the mainframe and pass data to the COBOL file's copybook. This was such a common need that later, Microsoft wrote an LU6.2 connector to create a code proxy from the copybook for you. Here we are 24 years after Y2K, and I am willing to bet that the banking and finance sectors still have a bunch of COBOL being used.

During the Object-Oriented boom, there was a big push to build an Enterprise Service Bus (ESB). We talked about technologies like MQSeries, CORBA, and RPC as ways to implement an ESB. I never saw an ESB I liked, probably because so many were never finished. Instead, I watched enterprise architects trying to resolve the enterprises' requirements into something for everyone. It's hard enough to nail this down for a single business line in an enterprise, much less many/all of them.

To some extent, we used databases as the point of integration. One system would be declared the data owner and copies of that data would be made for use by others. At the time, I saw this strategy fail more often than it succeeded because the copy would become stale and not meet our customer's expectations. Nowadays, we make copies, but it's less about integration and more about scale. To succeed, we had to figure out how to work in the world of eventual consistency.

As I shared these examples, I couldn't help but think of a few other topics. But these are the ones I wanted to share. If you've been in this industry for any time, I'm sure you have your own integration history. Integration is a persistent problem, a challenge that we all face, and it's safe to say that it's not going away anytime soon.