Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

Classic Tech Books I Still Have My Shelf

Relating some of the books I still have on my bookshelf. They are totally obsolete but found myself reminiscing around what they represented from my past and why they still took up space on my shelf.

One thing to know about me is that by default, I am a book collector. I once had a collection of over 200 technical books on my shelf. That was before the great purge where I donated most of the books and hung onto 30 or so. The great book purge is a story for another time. Suffice it to say that the books that survived were either timeless or ones that I had a fondness for. Here are a few of the fond ones.

The Pink Shirt Book. What would I have done without this book? If you were programming to the "IBM PC" at the time, this was a must have. I pulled it off the shelf and it flopped open the chapter on INT 21. This was the system services interrupt the keyboard, files, etc. I also found a dog-eared page for the INT 27 terminate and stay resident (TSR) details as well. TSRs were the services of the time. This was mostly assembler, then bumping up to C as soon as I could.

The Genie Book. I used this book in conjunction with the Norton Pink Shirt book. It came out after the Norton book, but had more details on writing TSRs, multitasking, and expanded memory. I pulled this book off the shelf and there is a page that was used so much that it was sticking out - having come off the binding. Now that is one well used page. When grabbing the image for this post, I found that this book was available on the Internet Archive (archive.org). E-versions of technical books is one reason I was comfortable getting rid of so many of my technical books (see future purge post).

The Microsoft C 5.0 Quick Reference Guide. This is an actual picture of my copy because I could not find a stock photo. Note the cup stains on the cover. It was a C reference, the page I used the most for C were the escape sequences for printf (e.g. \n, \r) and the language type ranges (e.g. difference between long and float). I still use this reference occasionally for the last table on ASCII codes - the decimal, hex, and character representation of 0 to 255. If you have seen The Martian, then you know.

The Intel 386 Microprocessor Hardware Reference Manual. Yikes, I am not sure why I am hanging onto this one, I have not used it since the time when I needed it. I have flipped through it a couple of time since and seen all the timing and logic diagrams and thought - we actually did this stuff. I primarily used this around interrupts (maskable vs. non-maskable) and for figuring out peripheral timings for external serial devices. We were doing some pretty hard-core stuff at the time. I recall a device we borrowed from one of the hardware manufacturers that sat in between the CPU and the socket. It allowed you to debug at the CPU level and single step through code. I remember we called it the skateboard because it resembled one but have not seen or heard of one since. Hardcore.

It's worth noting that many of the folks I worked with during this time are still friends. We were young and thought anything was possible. On top of that we had a leader who inspired us to live on the edge and accomplish amazing things.

Builders gotta build

When I talk to people about what I like about what I do - there are usually two things that I tell them. First that I like to build stuff and second to solve problems. I am a builder - both in my professional and personal life. Building software is what I get paid to do because I have demonstrated that I am good at it. Building in the physical world is just fun.

Take for instance "building" at home. Fixed things in the house. Upgrades. Or creating something that I need. The one that I wished I had yesterday was a yard tool holder. I recently moved and left the yard tool holder behind. It was something that I built out of extra lumber I had around the house. It looked something like the picture I attached, but custom made for the tools and the materials I had available. I knew what I wanted / needed but was constrained by what I had available.

Sounds like a parallel to building software. There are the requirements and then there is what you can do. The components available and time are both constraints to what you can complete. This is one place where the problem solving comes into play - putting it all together to complete (deliver) something.

There is something rewarding about completing something that I built. Yesterday it was waxing poetic about my yard tool holder. Sounds like an opportunity to consider version 2.

It works on my machine

One truth I had to come to terms with earlier in my career was the "it works on my machine" anti-pattern. It's a close cousin to the name of my blog - sometimes words DO mean something.

This was back in the days before Windows (yes there was such a thing) and I was writing a custom TSR (terminate and stay resident) module that we used to communicate with the custom database engine we wrote on NetWare (starting to get a sense of the decade yet?). Long story short there was a problem with the code that prevented it from working on one machine - just one. And it wasn't like it always didn't work, it only didn't work sporadically.

In my arrogance at the time, I was slow to take a look at this issue. My arrogant thinking went like (1) this is the only machine to have the issue (2) I spent a lot of time on this code and especially testing it (3) this code has been running on hundreds of other machines without an issue (4) so it must be something particular to the machine or user. Fast forward a couple of weeks and surprise a couple more people report having the issue. It was then that a colleague and I were talking, and I woke up from the spell - "this is something I need to look into" I finally told myself.

Turns out the issue was an interrupt conflict with a badly behaved driver that was slowly be rolled out in a pilot and very soon would release across the entire org and would impact my, now thousands, of customers. Once I figured this out (which was actually pretty hard to do) - the fix was simple; I just switched to a different interrupt. There were other times when I fell under the spell of my own perfume, each time learning something and making me humbler.

Computers and software have grown much more complex since then. developers have much less control over all the code they are dependent upon. In the example above, I was dependent on the CPU microcode, the BIOS and MSDOS - all of which I had the source code for. I could usually discount the microcode (I only recycled the manual last year) which meant that any issue was either with my code or the few dependencies I had. The lack of dependencies and illusion of control was part of where my arrogance came from. But today there are dependencies on dependencies; it's such a complex ecosystem. And yet developers still have quite a bit of arrogance. 

Why am I writing about this now? This time I was the one user for which the application did not work - last year (mid-2020) it started, and I would constantly send logs and traces to the development team, and nothing would happen. I told them that my fear is that my experience was the "canary in the coal mine" and that either (1) other people are also having the issue and not reporting it OR (2) that at some time in the not-too-distant future it would become more urgent. 

The later was this week.

It now works on my machine.

If you smelt...you dealt it

No this entry is NOT about flatulence.  Ironically, there is a Wikipedia page that is - here.  Instead my intent is to opine on the practice at my employer about developers supporting their own code - which means that if you broke something then you are going to be the one to fix it.

When I was managing the production support team at my previous employer; the team's responsibility was to focus on the operation of key systems and ensure that they were available and meeting SLAs.  This team did a good job at keeping things moving - if something broke they got to be pretty adept at fixing issues.  If you look at this as an outcome with a fixed cost (which is how we structured it) then it can look pretty attractive.  But in the end the people on the team had many concerns/issues/complaints/etc about the challenges they faced in sustaining this model.  In the end - I feel that this model takes too short of a view of life cycle of a system and therefore is flawed.

First it is worth noting that Amazon has a list of core leadership principles that they live and breath by.  I have done a lot of chin rubbing around these as of late and the more that I play with them the more I like them.  When considering this post there were a few in particular that I felt applied..."Ownership", "Invent and Simplify" and "Insist on the Highest Standards".  If you read the short descriptions for each of these they are all pointing to how separating support from development is flawed.

If developers are not responsible for keeping their own code running or seeing how it behaves in production then how they doing any of these things?  Sure it is possible that you get outcomes in this direction; but it is not likely - especially without the ownership.

A related topic to this is how we work this into our Agile methodology to ensure that we still get things done on time.