Try Noodling on That

The name of this blog is a reference to words uttered by a friend of mine. He also added the word "noodle" to my vocabulary, which is synonymous with thinking in this context. Not to be confused with a Noodle the pug who prognosticated on what type of day we could expect to have.

Whether to myself or someone on my team, I often use the phrase "noodle on that" when the answer is non-trivial and analysis and logic are not yielding a solution. It can be relatively straightforward—I need an answer to this problem—or something more abstract—like ideation. Either way, there are times when looking for an answer is more like chasing something elusive. Finding the answer is often further compounded by some time pressure - I need to solve this by XX.

Enter Noodling.

Several times, the answer presented itself to me - but only when I took the pressure off myself. For instance, I was cramming for a math final, and the answer to a particularly gnarly calculus problem came to me in my sleep. It woke me up, presumably so I could write it down (ha). Another time, the answer came to me in the shower. Someone once attributed this to all the positive ions stimulating my brain. Whatever the reason, it's a thing - Shower Epiphanies.

As a manager, I have often suggested that they take a break from trying to solve the problem directly and noodle on it. Give yourself a break. Let the pressure off. Give your subconscious a chance to work on it. My anecdotal data is that it works the majority of the time. This has the additional benefit of letting people work it out for themselves. My belief is that things feel better when people figure something out on their own, as opposed to me giving them the answer. 

See you on comfy mountain. IYKYK.

Is Doing the Right Thing - Right?

For most of this year (the last nine months), I have been pondering whether doing the right thing is right. Let me clarify. For me, doing the right thing in the context of honesty or integrity - is ALWAYS the right thing. Is doing what I feel is the right thing to do - the right thing? I have been noodling on specific circumstances that have me questioning whether I did the right thing based on how they turned out. 

There were two circumstances where I put others' needs ahead of my own, and it didn't work out. It is not nearly as noble as when Spock sacrificed himself in Wrath of Khan, but it is my little drama. The binary part of me looks at these situations and sees the only alternative as never putting others first and looking out only for myself. Even writing that down makes me grimace, as it goes against my core beliefs. So, the answer must be in the vast gray area between what I did and doing nothing. In the classic answer for all questions, it depends.

Regarding the two circumstances I have been pondering, the others were not in their integrity, making that the easier thing to focus on. I put others (people, projects, etc.) first and myself second, and it didn't work out as I had hoped. As I healed from that, I was able to begin considering where my learning was. Even if it's a one-off, I won't trust that person in the future.

I have been thinking about reciprocity in this context; doing something for others (entities or people) without expecting what I may get in return. Being a manager/leader is being in service. The thing I struggle with is whether that also requires selflessness. My recent experience is that selflessness in the workplace is a minefield. First, it's not very likely to happen - it's just the nature of companies, especially large ones. Second, I must acknowledge that I sometimes have expectations hiding behind my actions. They may be as simple as looking for a smile, but expectations nonetheless. That doesn't make my decision/activity wrong; it just exposes that I had some expectations. It's the missed expectations that led to my disappointment.

Ultimately, I feel good about what I did and my reasons for doing it. I would make the same choice again, looking out for any expectations I have. It also means that being selfless does not mean sacrificing myself. So, maintain my integrity, do what I believe right, and watch out for personal risks.

Do or do not. There is no try.

I recently dreamed I was in a large audience where an unidentified leader asked for a volunteer to lead a big project. They looked at me, and I said something like I will try, but I couldn't do it alone. Someone else stood up and said they would take on the project, and I could feel the audience staring at me with disdain. So much so that it woke me up.

I am not the type of person who looks for a lot of meaning in my dreams, mostly because I don't remember them. But this one woke me up, and I spent the rest of the early morning lying in bed, considering why it bothered me.

This was similar to talking to others and asking them how we get something done, and their answer is something like, "We don't have the resources." That wasn't the question; the question was, what would it take? My dream was similar because the question was not, "Would you deliver this project by yourself." It's not a question of ensuring a successful outcome; it's a question of pulling together enough for a reasonable discussion on feasibility.

The response I would guide my dream self to give is: Sure, I will lead that big project for you. Then, I would document what it's going to take to get it done and present that to the stakeholders. 

This dream bothered me because I was settling for trying when I knew better.

Surprisingly good project estimation

At Amazon, we went through several cycles/iterations of planning. Regardless of how Agile an organization operated, we needed to map capacity to the roadmap to understand how much we could deliver in a given year. This does not mean we actually delivered exactly what we initially planned since requirements and priorities changed over the course of the year. But it did allow us to have a starting place that we used to have a shared vision across engineering and stakeholders.

For each iteration, I would strive for better accuracy by asking technical leaders to revise their estimates. Effort was the input we refined the most, asking teams to go from swags to more educated estimates. How much were they relying on existing services? How much had never been done before? I had a list of prompts to help them assess the complexity/unknowns and, therefore, the risk. I would take all that and put it into my model (i.e., Excel), which would auto-magically create a calendar plan.

Invariably our leaders always requested when we would be "done," typically in the context of something like Prime Day or re:Invent. To answer this question, I would use a scaled-down version of the organization planning spreadsheet. Typically, the project was underway, and varying levels of design were complete. Most of the time this meant that the effort estimates were better, and the number of resources available was known. For each estimate, I would ask managers to provide me with the factor they used to compensate for PTO, meetings, etc. - I called it non-keyboard time.

Anecdotally, the static model usually yielded a surprisingly accurate date. I would use it to drive discussions around resource and/or feature tradeoffs depending on which knob we were using to tune the outputs (typically the launch date).

Such a simple model gave many people heartburn—"How could something so simple model something so complicated?" I tried "enhancing" the model to account for other factors, for instance, parallelization. However, it was too complex to model and required too much from engineering managers or technical leads. Parallelization is needed when building a plan but doesn't work when doing estimation—it simply didn't improve the model. 

The problem was that many leaders would simply not believe the model. I found this very frustrating since they would usually choose a somewhat arbitrary date as to when they wanted to be done and tell me to use that. I would challenge that approach and eventually agree since it was obvious that they were going to get their way (an anti-pattern of Amazon's Have Backbone, Disagree, and Commit). Side note, one time, I escalated a case of this, and while the VP I escalated to was appreciative, the Director threw a tantrum in the meeting. So much so that the VP later contacted me to apologize for the Director's behavior.

In project retrospectives, we discussed how we course-corrected throughout the project. It turns out that the early model outputs were right—scary right?

I have been around long enough to know that when bringing in the date, leaders are often pressuring the teams to keep their foot on the gas. Something to watch out for is that this practice can lead to the outcomes I describe in my "Secure Apps Take A Village" post. Some of this behavior comes down to a couple of challenges leaders (including me) have to grapple with - trust and giving up control. Trust (and verify) what my teams are telling me, and accept that I don't have the same level of control I used to. Both are probably topics in and of themselves.

Things that I ran into...

1. Inflated efforts. Managers and leads pad their effort estimates to ensure they meet commitments. First, how do you know if it's inflated? Who better to know the complexity of a system than the people closest to it? If managers are really doing this, then do they understand what is being asked of them? How will it be used? If managers are padding, they are responding to a fear typically rooted in the organization's culture. This is something to go deeper into to figure out how to make it work for you.
2. Low efficacy factors. Given that keyboard time is an input into the model, this will directly impact the date/resource calculation. Is a low number accounting for a less experienced team? Or is the manager using the same number every time? Why—to pad dates? I used a generously low number, which was painful to write down but proved to be more accurate. For instance, this was especially true for teams that had high operational overhead.
3. What is DONE? Of course, done is something that not everyone defines the same way. As the owner of a launch, done to me meant the system was launched and open to customers. Invariably some folks would define done as some level of completeness for their code. Ensure that everyone is aligned on what done means. I would typically use code complete, including all automated tests and the binaries, in a shared test environment. I used this as done since the cost of integration testing and other launch activities was typically shared across teams, and I could estimate those differently.

Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

Classic Tech Books I Still Have My Shelf

Relating some of the books I still have on my bookshelf. They are totally obsolete but found myself reminiscing around what they represented from my past and why they still took up space on my shelf.

One thing to know about me is that by default, I am a book collector. I once had a collection of over 200 technical books on my shelf. That was before the great purge where I donated most of the books and hung onto 30 or so. The great book purge is a story for another time. Suffice it to say that the books that survived were either timeless or ones that I had a fondness for. Here are a few of the fond ones.

The Pink Shirt Book. What would I have done without this book? If you were programming to the "IBM PC" at the time, this was a must have. I pulled it off the shelf and it flopped open the chapter on INT 21. This was the system services interrupt the keyboard, files, etc. I also found a dog-eared page for the INT 27 terminate and stay resident (TSR) details as well. TSRs were the services of the time. This was mostly assembler, then bumping up to C as soon as I could.

The Genie Book. I used this book in conjunction with the Norton Pink Shirt book. It came out after the Norton book, but had more details on writing TSRs, multitasking, and expanded memory. I pulled this book off the shelf and there is a page that was used so much that it was sticking out - having come off the binding. Now that is one well used page. When grabbing the image for this post, I found that this book was available on the Internet Archive (archive.org). E-versions of technical books is one reason I was comfortable getting rid of so many of my technical books (see future purge post).

The Microsoft C 5.0 Quick Reference Guide. This is an actual picture of my copy because I could not find a stock photo. Note the cup stains on the cover. It was a C reference, the page I used the most for C were the escape sequences for printf (e.g. \n, \r) and the language type ranges (e.g. difference between long and float). I still use this reference occasionally for the last table on ASCII codes - the decimal, hex, and character representation of 0 to 255. If you have seen The Martian, then you know.

The Intel 386 Microprocessor Hardware Reference Manual. Yikes, I am not sure why I am hanging onto this one, I have not used it since the time when I needed it. I have flipped through it a couple of time since and seen all the timing and logic diagrams and thought - we actually did this stuff. I primarily used this around interrupts (maskable vs. non-maskable) and for figuring out peripheral timings for external serial devices. We were doing some pretty hard-core stuff at the time. I recall a device we borrowed from one of the hardware manufacturers that sat in between the CPU and the socket. It allowed you to debug at the CPU level and single step through code. I remember we called it the skateboard because it resembled one but have not seen or heard of one since. Hardcore.

It's worth noting that many of the folks I worked with during this time are still friends. We were young and thought anything was possible. On top of that we had a leader who inspired us to live on the edge and accomplish amazing things.

Be a Student of Management

My experience is being a manager is hard. More specifically it's hard to do well. In my career I have tried being a manager and decided it was not for me and gone back to being an individual contributor. I liked the idea of being a manager but could tell I was not ready to be a good manager.

It wasn't until I became a manager where I was also considered an officer of the company and the company enrolled me in a year-long training. Oh wait, there is training for this? This is when I became a student of management. The assumption that the skills that made me a great individual contributor were the same as the ones I needed to be an equally great manager were quickly dispelled. I needed to grow some new muscles.

I am a naturally curious person, so this was something I wanted to learn more about. That is after I swallowed a little humility. I started off reading a handful of books recommended as part of the development program. These were helpful and gave me a foundation for the type of manager I wanted to be. But it was only through trying (and at times failing) that I learned where my weaknesses were. The biggest one was to not try and control the outcomes of using my technical expertise. This is where I came up with the pattern I called controlled failure - a topic for another post.

The key for me has been to acknowledge that as good of a manager I am, that I need to remain humble. There is always something new to learn. I have learned things at the best and worst of times. My goal is never stop learning how to be the best manager I can be - to always be a student.

Adam Grant is someone I have learned a bunch from. 

Kindness Factor

Kindness is something that has been on my mind lately. What does kindness look like at work? I was thinking on this and believe kindness starts with myself then extends to others. 

Am I the only one that has an easier time extending kindness to others than myself? Self analysis aside, I can't help but smile when being kind to others. Either way, it's me expressing the kindness - I am "in control".

The challenge is - I am not in control when looking to receive kindness. It's not something I can simply request. But it is something that when it's missing, I know it. I learned a long time ago that it's foolish to think that by modeling kindness - something will change. If it does, great, but it's not likely.

Kindness at work is happiness. Satisfaction. Feeling appreciated. All sound like good things to have in the work place - right? So why do I get the sense that there are others that see this idea as foreign / incompatible with the workplace? How about you - do you see kindness where you work? 

Yiddish for IT Leaders

I have come to the conclusion that I need to know more yiddish.  Can I use it as a code to hide what I am really thinking?  Help bypass any email filters?  Just make me feel better.  Here is my arsenal.

bupkis - As in - you don't know bupkis.

chutzpah - There is always one team member with too much of this.

glitch - Things are late again?  It must be another glitch.

kibitz - What we should call reviews.

klutz - You don't want this and a programming to go together.

kvetch - What I do when I get home from work.

nudnik - In management speak these are the team members you manage out.

schmuck - What you call someone who changes something directly in production - first.

schtik - A little off topic, but I think of Benji Bronk on the radio on my way into work.

shpiel - My weekly team briefings have at least one of these.

yutz - Yiddish has lots of fun words for describing people that bum you out.