My Experience of APIs

I watched Nate Jones's TicTok last week. He talked about how APIs are becoming the interface to data. As I noodled on his words, I agreed with him, and it got me thinking about my history with APIs - more generally than his point.

I have been developing or calling APIs for a long time, and this got me thinking: weren't the MSDOS interrupts (related post) a form of API? We all know the right answer, so we will not say it out loud and move on.

But seriously. I remember creating an API for the DOS terminate and stay resident (TSR) module I wrote as the proxy for the server-based DBMS we wrote. This API used INT2F as the interface to the TSR.  We didn't expect CBASIC programmers to code to INT2F, so I had a callable C library that abstracted the internals. What we built was closer to something like ODBC, a generic API for retrieving data requiring you to write the query.

There was a period when our systems had a business layer and a data layer; the business layer represented business semantics, and the data layer represented the sproc/view/query/table structure. Weren't these all APIs? During the object-oriented period (which sounds like the Jurassic period), I didn't refer to them as APIs then; they were methods on an object.

We went from RPC to SOAP to REST, each an evolution to build distributed systems. All these are forms of APIs as well. Why? APIs are about creating a dependable contract that callers can depend on to create an outcome—not about the specific implementation technology. The "interesting" part of the API design process was how to design for reusability. And by interesting, I mean the part we spent a lot of time discussing. In the Enterprise IT space, we had enterprise architects who created taxonomies, libraries, and APIs that modeled the data and/or the business processes. Building models/services for the entire company didn't work, and a "single responsibility model" and "separation of concerns" were needed. Even now, we have engineers who have API design as a skill we depend on and use at some level of governance.

From these evolutions (and more) of APIs, I have learned they are hard. Why? Because API design is hard. Why? Because reuse is hard. This is why when we talk about APIs today, we don't think of them as static. Instead, we build them to be extensible as the needs evolve. Even designing for evolution, I have had more than my fair share of painful API deprecations.

I still love a good API design session.

Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

Powershell Quickee

As this title rolled off my fingers it made me laugh a little.

But hey, isn't everything in PowerShell a little quicker?  At least that is the intent.  Over the past few years I keep dabbling in PowerShell from time to time.  I have just enough experience to know what can reasonably be done in the tool - without over extending it.  <soapbox> It looks like there are some people who take this tool and use it to bang every nail they have.  But that is another blog entry. </soapbox>

Here is the script I crufted up today for showing me all the services that are on a server set to run at startup (aka Automatic) and yet are NOT running now.

I was using the get-service cmdlet but it did not seem to be return the StartMode and State properties from a remote server (works locally fine).  So I Googled up an alternative that uses WMI.

Services (part 1 of many)

I not been blogging much lately because I have felt like what I have been working on is just not blog-worthy. We "outsource" most of our infrastructure services to an internal division; which was recently turned over to a large outsourcing firm. So I have been spending much of my time "training" the folks who I interact with on a near daily basis to do what I need them to do. As it turned out this was not a trivial task. And that's all I have to say about that. :-)

I am beginning to look at building out some data services. In the Investment Management business data is king; especially market data. What I am learning (or should I say re-learning or feeling pain around) is that without the data in place the service is nearly useless. So my project is really turning into data project where I am bringing in and reconciling market data vendor feeds. Oh yeah and by the way I am exposing them as a service.

Which has me questioning myself; why a service in the first place? I just want to make sure I am actually providing real value and not just overhead. I think the key reason for a service here is provide a single well define abstraction of the backend. We have so much market data and so many sources of market data; that providing a flexible abstraction makes sense. My expectation is that we will be consuming this data right from Excel (the universal financial application platform) as well as many of our custom applications.

I remember presentations back in the mid-90’s when I was a Microsoft and we were listening to the early internal presentations on Microsoft Transaction Server (MTS) by Pat Helland and James Utzschneider. They were talking about MTS interception and how you can accomplish cool things when you can get in the middle. That is exactly what we are trying to do. Once we get the abstraction of the data, we can begin to do some cool things.

My biggest concern about building a service is performance. In past lives when I have built services I would always warn people that if you need fast performance then web services you should think long and hard about not using web services. Of course there are ways that you can “cache” data outside of the service, but now you start to run into the dirty cache problem. I want to go back and read Pat Helland’s fiefdoms articles from a bunch of years ago to get my head back into this problem space. I recall Roger Sessions also doing some similar work based on Pat’s original article.

Next decision is what Framework bits to use; WebMethod (.NET 2.0) or WCF (.NET 3.0)? I know .NET 2.0 pretty well; so I feel confident I can get something done fairly quickly on those bits. But why do something on the “legacy” asmx platform? Why not just bite the learning curve now? I assume the interop between the two just works; so if I wanted to the service to be in WCF and the client to be either that would be OK.

I was just noticed the bottom the page (I write this in Word first) coming up. And I didn’t think I had anything to write about...