Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

Classic Tech Books I Still Have My Shelf

Relating some of the books I still have on my bookshelf. They are totally obsolete but found myself reminiscing around what they represented from my past and why they still took up space on my shelf.

One thing to know about me is that by default, I am a book collector. I once had a collection of over 200 technical books on my shelf. That was before the great purge where I donated most of the books and hung onto 30 or so. The great book purge is a story for another time. Suffice it to say that the books that survived were either timeless or ones that I had a fondness for. Here are a few of the fond ones.

The Pink Shirt Book. What would I have done without this book? If you were programming to the "IBM PC" at the time, this was a must have. I pulled it off the shelf and it flopped open the chapter on INT 21. This was the system services interrupt the keyboard, files, etc. I also found a dog-eared page for the INT 27 terminate and stay resident (TSR) details as well. TSRs were the services of the time. This was mostly assembler, then bumping up to C as soon as I could.

The Genie Book. I used this book in conjunction with the Norton Pink Shirt book. It came out after the Norton book, but had more details on writing TSRs, multitasking, and expanded memory. I pulled this book off the shelf and there is a page that was used so much that it was sticking out - having come off the binding. Now that is one well used page. When grabbing the image for this post, I found that this book was available on the Internet Archive (archive.org). E-versions of technical books is one reason I was comfortable getting rid of so many of my technical books (see future purge post).

The Microsoft C 5.0 Quick Reference Guide. This is an actual picture of my copy because I could not find a stock photo. Note the cup stains on the cover. It was a C reference, the page I used the most for C were the escape sequences for printf (e.g. \n, \r) and the language type ranges (e.g. difference between long and float). I still use this reference occasionally for the last table on ASCII codes - the decimal, hex, and character representation of 0 to 255. If you have seen The Martian, then you know.

The Intel 386 Microprocessor Hardware Reference Manual. Yikes, I am not sure why I am hanging onto this one, I have not used it since the time when I needed it. I have flipped through it a couple of time since and seen all the timing and logic diagrams and thought - we actually did this stuff. I primarily used this around interrupts (maskable vs. non-maskable) and for figuring out peripheral timings for external serial devices. We were doing some pretty hard-core stuff at the time. I recall a device we borrowed from one of the hardware manufacturers that sat in between the CPU and the socket. It allowed you to debug at the CPU level and single step through code. I remember we called it the skateboard because it resembled one but have not seen or heard of one since. Hardcore.

It's worth noting that many of the folks I worked with during this time are still friends. We were young and thought anything was possible. On top of that we had a leader who inspired us to live on the edge and accomplish amazing things.

Builders gotta build

When I talk to people about what I like about what I do - there are usually two things that I tell them. First that I like to build stuff and second to solve problems. I am a builder - both in my professional and personal life. Building software is what I get paid to do because I have demonstrated that I am good at it. Building in the physical world is just fun.

Take for instance "building" at home. Fixed things in the house. Upgrades. Or creating something that I need. The one that I wished I had yesterday was a yard tool holder. I recently moved and left the yard tool holder behind. It was something that I built out of extra lumber I had around the house. It looked something like the picture I attached, but custom made for the tools and the materials I had available. I knew what I wanted / needed but was constrained by what I had available.

Sounds like a parallel to building software. There are the requirements and then there is what you can do. The components available and time are both constraints to what you can complete. This is one place where the problem solving comes into play - putting it all together to complete (deliver) something.

There is something rewarding about completing something that I built. Yesterday it was waxing poetic about my yard tool holder. Sounds like an opportunity to consider version 2.

New Coding Problems = Old Coding Problems?

Late last year and into this year (2024) I needed to take time off to recover from surgery. As my mind started to clear but my body was still healing, I started poking around some code. At the time I was playing Pokémon Go (PG) and wanted to build my own mapping application. Well not actually from scratch but build it using a bunch of Python that others had assembled. I was not successful, and it the reason why got me thinking about libraries vs. binaries. 

Library vs. Binary reuse is one of those problems that has been around as long as I can remember. The first time I ran into it was when coding in C (pre-C++) and considering an #inlcude file (.h), a library (.LIB), or a dynamic link library (.DLL). I don't plan on rehashing that thinking now because we all know the right answer and so we can just move on.

The Python code / repositories I was trying to assemble for this purpose suffered greatly from the conflicting version problem. Where code in the same namespace was dependent on different versions of the same library. This creates a Gordian Knot of mixed versions that is very difficult to work out. I am no Python expert, but my understanding from those that are is that this can be a common thing in the world of Python.

Not to give Python a bad name, I know it would happen in Java too. Yes, there are ways to work around this, but I was more focused on getting the solution running that having to unravel dependency tree conflicts.

It was a walk down memory lane that got me thinking that a lot of the old problems I had as a developer are still around. I know the tools have gotten better at helping us manage these and lessen the pain of the problem, but the problem still exists.

The Easy Fallacy

I was talking to one of the younger engineers at work recently and he said that he appreciated my experience. After I realized that it wasn't one of those - your so old moments. I laughed and told him that is probably more true than he realized. For instance, we were talking about an a fix that he labeled as "easy". I told him that in my experience easy changes can be a fallacy. I could see in his expression that he knew what I saying but he didn't know what I was saying.

I can't count the number of times when me, my colleagues, and/or team fell for this one. It looks something like - the change is a small one, often innocuous - like a configuration change. If you're nodding you're head right now, then you too have fallen or been witness to this one.

This happened recently a couple of times. One looked like a change that when merged to to the main branch, inadvertently picked up an unrelated change. This change was not something that was caught until it was promoted to production and resulted in a performance impact. As a result we had a week delay two weeks before the larger launch because we had to do even more work to earn trust that it would not happen a second time.

A contributor to falling for this fallacy is that making simple changes in a complex system isn't about how small the change is - it's about everything else after the change.

Which begs the question, how do you not fall for this one? I feel like this one hurts the most when your under a time crunch and your belief in the fallacy convinces you that you should make "easy" last minute changes. Ouch. Or that you let your guard down on the "easy" change and miss something. The answer for these two is - don't do this. I am sure you have your version of this. 

It works on my machine

One truth I had to come to terms with earlier in my career was the "it works on my machine" anti-pattern. It's a close cousin to the name of my blog - sometimes words DO mean something.

This was back in the days before Windows (yes there was such a thing) and I was writing a custom TSR (terminate and stay resident) module that we used to communicate with the custom database engine we wrote on NetWare (starting to get a sense of the decade yet?). Long story short there was a problem with the code that prevented it from working on one machine - just one. And it wasn't like it always didn't work, it only didn't work sporadically.

In my arrogance at the time, I was slow to take a look at this issue. My arrogant thinking went like (1) this is the only machine to have the issue (2) I spent a lot of time on this code and especially testing it (3) this code has been running on hundreds of other machines without an issue (4) so it must be something particular to the machine or user. Fast forward a couple of weeks and surprise a couple more people report having the issue. It was then that a colleague and I were talking, and I woke up from the spell - "this is something I need to look into" I finally told myself.

Turns out the issue was an interrupt conflict with a badly behaved driver that was slowly be rolled out in a pilot and very soon would release across the entire org and would impact my, now thousands, of customers. Once I figured this out (which was actually pretty hard to do) - the fix was simple; I just switched to a different interrupt. There were other times when I fell under the spell of my own perfume, each time learning something and making me humbler.

Computers and software have grown much more complex since then. developers have much less control over all the code they are dependent upon. In the example above, I was dependent on the CPU microcode, the BIOS and MSDOS - all of which I had the source code for. I could usually discount the microcode (I only recycled the manual last year) which meant that any issue was either with my code or the few dependencies I had. The lack of dependencies and illusion of control was part of where my arrogance came from. But today there are dependencies on dependencies; it's such a complex ecosystem. And yet developers still have quite a bit of arrogance. 

Why am I writing about this now? This time I was the one user for which the application did not work - last year (mid-2020) it started, and I would constantly send logs and traces to the development team, and nothing would happen. I told them that my fear is that my experience was the "canary in the coal mine" and that either (1) other people are also having the issue and not reporting it OR (2) that at some time in the not-too-distant future it would become more urgent. 

The later was this week.

It now works on my machine.

Yiddish for IT Leaders

I have come to the conclusion that I need to know more yiddish.  Can I use it as a code to hide what I am really thinking?  Help bypass any email filters?  Just make me feel better.  Here is my arsenal.

bupkis - As in - you don't know bupkis.

chutzpah - There is always one team member with too much of this.

glitch - Things are late again?  It must be another glitch.

kibitz - What we should call reviews.

klutz - You don't want this and a programming to go together.

kvetch - What I do when I get home from work.

nudnik - In management speak these are the team members you manage out.

schmuck - What you call someone who changes something directly in production - first.

schtik - A little off topic, but I think of Benji Bronk on the radio on my way into work.

shpiel - My weekly team briefings have at least one of these.

yutz - Yiddish has lots of fun words for describing people that bum you out.