Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

Classic Tech Books I Still Have My Shelf

Relating some of the books I still have on my bookshelf. They are totally obsolete but found myself reminiscing around what they represented from my past and why they still took up space on my shelf.

One thing to know about me is that by default, I am a book collector. I once had a collection of over 200 technical books on my shelf. That was before the great purge where I donated most of the books and hung onto 30 or so. The great book purge is a story for another time. Suffice it to say that the books that survived were either timeless or ones that I had a fondness for. Here are a few of the fond ones.

The Pink Shirt Book. What would I have done without this book? If you were programming to the "IBM PC" at the time, this was a must have. I pulled it off the shelf and it flopped open the chapter on INT 21. This was the system services interrupt the keyboard, files, etc. I also found a dog-eared page for the INT 27 terminate and stay resident (TSR) details as well. TSRs were the services of the time. This was mostly assembler, then bumping up to C as soon as I could.

The Genie Book. I used this book in conjunction with the Norton Pink Shirt book. It came out after the Norton book, but had more details on writing TSRs, multitasking, and expanded memory. I pulled this book off the shelf and there is a page that was used so much that it was sticking out - having come off the binding. Now that is one well used page. When grabbing the image for this post, I found that this book was available on the Internet Archive (archive.org). E-versions of technical books is one reason I was comfortable getting rid of so many of my technical books (see future purge post).

The Microsoft C 5.0 Quick Reference Guide. This is an actual picture of my copy because I could not find a stock photo. Note the cup stains on the cover. It was a C reference, the page I used the most for C were the escape sequences for printf (e.g. \n, \r) and the language type ranges (e.g. difference between long and float). I still use this reference occasionally for the last table on ASCII codes - the decimal, hex, and character representation of 0 to 255. If you have seen The Martian, then you know.

The Intel 386 Microprocessor Hardware Reference Manual. Yikes, I am not sure why I am hanging onto this one, I have not used it since the time when I needed it. I have flipped through it a couple of time since and seen all the timing and logic diagrams and thought - we actually did this stuff. I primarily used this around interrupts (maskable vs. non-maskable) and for figuring out peripheral timings for external serial devices. We were doing some pretty hard-core stuff at the time. I recall a device we borrowed from one of the hardware manufacturers that sat in between the CPU and the socket. It allowed you to debug at the CPU level and single step through code. I remember we called it the skateboard because it resembled one but have not seen or heard of one since. Hardcore.

It's worth noting that many of the folks I worked with during this time are still friends. We were young and thought anything was possible. On top of that we had a leader who inspired us to live on the edge and accomplish amazing things.

My Brain Doing Background Processing

Maybe you're like me where when you run into an issue - your brain is working on the problem in the background and sometime later a solution emerges. But when it's happening, I am [mostly] not aware of it. Call it my subconscious? I have had this happen over and over, but it's not something I feel entirely able to control. For instance, my brain does not support an explicit action like...

In fact, it's sort of a mystery to me which things end up getting queued up this way and which one don't. It does seem to be limited to things that I already know but have not built an association between the problem and the answer yet.

Rather than figuring out the brain mechanics I just accept that it happens. So much so that it's become part of my management style. I often suggest to folks that they "noodle" on something and see if it comes to them later (say in the shower  / positive ion generator). More frequently than not, it comes to them.

This happened to me over the weekend around my recent post on standing up a Pokemon Go mapping application. What came to me is that that problem really needed a deployment container (e.g. Docker) where a working set of all the Python code was already assembled and could be deployed as a working system. Seems obvious now, but it wasn't something I was aware of trying to work out.

Builders gotta build

When I talk to people about what I like about what I do - there are usually two things that I tell them. First that I like to build stuff and second to solve problems. I am a builder - both in my professional and personal life. Building software is what I get paid to do because I have demonstrated that I am good at it. Building in the physical world is just fun.

Take for instance "building" at home. Fixed things in the house. Upgrades. Or creating something that I need. The one that I wished I had yesterday was a yard tool holder. I recently moved and left the yard tool holder behind. It was something that I built out of extra lumber I had around the house. It looked something like the picture I attached, but custom made for the tools and the materials I had available. I knew what I wanted / needed but was constrained by what I had available.

Sounds like a parallel to building software. There are the requirements and then there is what you can do. The components available and time are both constraints to what you can complete. This is one place where the problem solving comes into play - putting it all together to complete (deliver) something.

There is something rewarding about completing something that I built. Yesterday it was waxing poetic about my yard tool holder. Sounds like an opportunity to consider version 2.

New Coding Problems = Old Coding Problems?

Late last year and into this year (2024) I needed to take time off to recover from surgery. As my mind started to clear but my body was still healing, I started poking around some code. At the time I was playing Pokémon Go (PG) and wanted to build my own mapping application. Well not actually from scratch but build it using a bunch of Python that others had assembled. I was not successful, and it the reason why got me thinking about libraries vs. binaries. 

Library vs. Binary reuse is one of those problems that has been around as long as I can remember. The first time I ran into it was when coding in C (pre-C++) and considering an #inlcude file (.h), a library (.LIB), or a dynamic link library (.DLL). I don't plan on rehashing that thinking now because we all know the right answer and so we can just move on.

The Python code / repositories I was trying to assemble for this purpose suffered greatly from the conflicting version problem. Where code in the same namespace was dependent on different versions of the same library. This creates a Gordian Knot of mixed versions that is very difficult to work out. I am no Python expert, but my understanding from those that are is that this can be a common thing in the world of Python.

Not to give Python a bad name, I know it would happen in Java too. Yes, there are ways to work around this, but I was more focused on getting the solution running that having to unravel dependency tree conflicts.

It was a walk down memory lane that got me thinking that a lot of the old problems I had as a developer are still around. I know the tools have gotten better at helping us manage these and lessen the pain of the problem, but the problem still exists.

Be a Student of Management

My experience is being a manager is hard. More specifically it's hard to do well. In my career I have tried being a manager and decided it was not for me and gone back to being an individual contributor. I liked the idea of being a manager but could tell I was not ready to be a good manager.

It wasn't until I became a manager where I was also considered an officer of the company and the company enrolled me in a year-long training. Oh wait, there is training for this? This is when I became a student of management. The assumption that the skills that made me a great individual contributor were the same as the ones I needed to be an equally great manager were quickly dispelled. I needed to grow some new muscles.

I am a naturally curious person, so this was something I wanted to learn more about. That is after I swallowed a little humility. I started off reading a handful of books recommended as part of the development program. These were helpful and gave me a foundation for the type of manager I wanted to be. But it was only through trying (and at times failing) that I learned where my weaknesses were. The biggest one was to not try and control the outcomes of using my technical expertise. This is where I came up with the pattern I called controlled failure - a topic for another post.

The key for me has been to acknowledge that as good of a manager I am, that I need to remain humble. There is always something new to learn. I have learned things at the best and worst of times. My goal is never stop learning how to be the best manager I can be - to always be a student.

Adam Grant is someone I have learned a bunch from. 

Kindness Factor

Kindness is something that has been on my mind lately. What does kindness look like at work? I was thinking on this and believe kindness starts with myself then extends to others. 

Am I the only one that has an easier time extending kindness to others than myself? Self analysis aside, I can't help but smile when being kind to others. Either way, it's me expressing the kindness - I am "in control".

The challenge is - I am not in control when looking to receive kindness. It's not something I can simply request. But it is something that when it's missing, I know it. I learned a long time ago that it's foolish to think that by modeling kindness - something will change. If it does, great, but it's not likely.

Kindness at work is happiness. Satisfaction. Feeling appreciated. All sound like good things to have in the work place - right? So why do I get the sense that there are others that see this idea as foreign / incompatible with the workplace? How about you - do you see kindness where you work?