A Brief History of Integrating

I was recently asked how I would integrate systems with similar functionality but different code bases and infrastructure. It got me thinking about how I feel I have been solving the integration problem throughout my career and how it (not surprisingly) keeps coming up. My answer was: It depends on what you have and where you want to land. I have many questions: how much of the data is the same? How reliable is the data? Is there an appetite to reduce any duplication? What is the desired recency (age) of the data? Is there a goal to become more homogenous? Regardless of the answers, I have employed several strategies/technologies over my career.

One of the first implementations I worked on was a practical solution for a PC-based application to leverage COBOL code on the mainframe. The company had invested years in the code and was not ready (or able) to rewrite it. We built an LU6.2 gateway to call the mainframe and pass data to the COBOL file's copybook. This was such a common need that later, Microsoft wrote an LU6.2 connector to create a code proxy from the copybook for you. Here we are 24 years after Y2K, and I am willing to bet that the banking and finance sectors still have a bunch of COBOL being used.

During the Object-Oriented boom, there was a big push to build an Enterprise Service Bus (ESB). We talked about technologies like MQSeries, CORBA, and RPC as ways to implement an ESB. I never saw an ESB I liked, probably because so many were never finished. Instead, I watched enterprise architects trying to resolve the enterprises' requirements into something for everyone. It's hard enough to nail this down for a single business line in an enterprise, much less many/all of them.

To some extent, we used databases as the point of integration. One system would be declared the data owner and copies of that data would be made for use by others. At the time, I saw this strategy fail more often than it succeeded because the copy would become stale and not meet our customer's expectations. Nowadays, we make copies, but it's less about integration and more about scale. To succeed, we had to figure out how to work in the world of eventual consistency.

As I shared these examples, I couldn't help but think of a few other topics. But these are the ones I wanted to share. If you've been in this industry for any time, I'm sure you have your own integration history. Integration is a persistent problem, a challenge that we all face, and it's safe to say that it's not going away anytime soon.

My Experience of APIs

I watched Nate Jones's TicTok last week. He talked about how APIs are becoming the interface to data. As I noodled on his words, I agreed with him, and it got me thinking about my history with APIs - more generally than his point.

I have been developing or calling APIs for a long time, and this got me thinking: weren't the MSDOS interrupts (related post) a form of API? We all know the right answer, so we will not say it out loud and move on.

But seriously. I remember creating an API for the DOS terminate and stay resident (TSR) module I wrote as the proxy for the server-based DBMS we wrote. This API used INT2F as the interface to the TSR.  We didn't expect CBASIC programmers to code to INT2F, so I had a callable C library that abstracted the internals. What we built was closer to something like ODBC, a generic API for retrieving data requiring you to write the query.

There was a period when our systems had a business layer and a data layer; the business layer represented business semantics, and the data layer represented the sproc/view/query/table structure. Weren't these all APIs? During the object-oriented period (which sounds like the Jurassic period), I didn't refer to them as APIs then; they were methods on an object.

We went from RPC to SOAP to REST, each an evolution to build distributed systems. All these are forms of APIs as well. Why? APIs are about creating a dependable contract that callers can depend on to create an outcome—not about the specific implementation technology. The "interesting" part of the API design process was how to design for reusability. And by interesting, I mean the part we spent a lot of time discussing. In the Enterprise IT space, we had enterprise architects who created taxonomies, libraries, and APIs that modeled the data and/or the business processes. Building models/services for the entire company didn't work, and a "single responsibility model" and "separation of concerns" were needed. Even now, we have engineers who have API design as a skill we depend on and use at some level of governance.

From these evolutions (and more) of APIs, I have learned they are hard. Why? Because API design is hard. Why? Because reuse is hard. This is why when we talk about APIs today, we don't think of them as static. Instead, we build them to be extensible as the needs evolve. Even designing for evolution, I have had more than my fair share of painful API deprecations.

I still love a good API design session.

Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

The Easy Fallacy

I was talking to one of the younger engineers at work recently and he said that he appreciated my experience. After I realized that it wasn't one of those - your so old moments. I laughed and told him that is probably more true than he realized. For instance, we were talking about an a fix that he labeled as "easy". I told him that in my experience easy changes can be a fallacy. I could see in his expression that he knew what I saying but he didn't know what I was saying.

I can't count the number of times when me, my colleagues, and/or team fell for this one. It looks something like - the change is a small one, often innocuous - like a configuration change. If you're nodding you're head right now, then you too have fallen or been witness to this one.

This happened recently a couple of times. One looked like a change that when merged to to the main branch, inadvertently picked up an unrelated change. This change was not something that was caught until it was promoted to production and resulted in a performance impact. As a result we had a week delay two weeks before the larger launch because we had to do even more work to earn trust that it would not happen a second time.

A contributor to falling for this fallacy is that making simple changes in a complex system isn't about how small the change is - it's about everything else after the change.

Which begs the question, how do you not fall for this one? I feel like this one hurts the most when your under a time crunch and your belief in the fallacy convinces you that you should make "easy" last minute changes. Ouch. Or that you let your guard down on the "easy" change and miss something. The answer for these two is - don't do this. I am sure you have your version of this. 

Keep It Simple (kiss) Revisited

I have a calendar from a vendor we use that has some of the classic coding and design principles - one for each month.  I was rubbing my chin staring at it this morning and I wanted to share what popped into my head...

While I am sure that the KISS principle has been written about (perhaps to death) I had another instance of this today as it applies to operations and infrastructure.

Quick background - I recently inherited an Operational group.  Operations is the clean up crew of development here.  While I understand the rationale of separating them, I think I like the idea of developers supporting their own code so that they better understand the impact of what they do.  What a great teaching tool - you want to not get up in the middle of the night - fix the code, do a better job in the first place, write a utility to help you out.  We have a bunch of applications that have been around for years and over time the developers who maintain many of these have moved on.  So today I asked the question of someone about two AD groups and what they are used for.  In both cases the answer was initially I don't know - and later the answer became these are not used anymore.

Part of keeping systems simple is getting rid of the things that are not used anymore.  We have all these extraneous moving parts that we don't need.  This just creates system bloat that should be easy to remove.

Granted you cannot get to everything - right now.  But this stuff has to get cleaned up over time.  Putting it into some sort of maintenance, wish list or Kaizen log seems like an easy thing to do.

All it takes is discipline.