Very Sparkly (Evolving Thinking on GenAI)

Tired of my GenX references yet? No? Good, here is another one.

Let me explain. As you may notice from my more recent blog entries, "AI" is something I have been frequently thinking about. Especially trying to work through the hype, where GenAI is very sparkly and attention grabbing, to a place where I see how AI makes things better. Further, what is the intersection of my years of engineering and building AI solutions? Is there one? Take for instance this entry from a year ago where this journey started to gain traction for me. Or me thinking about Vibe coding back in July.

I’ve been thinking about what “engineering” even means now that anyone can spin up an LLM and build something that looks like it works. It’s too easy to confuse “the demo runs” with “the system works.” So here’s where I’ve landed - four realities that to consider. These are not my original work (thanks Nate), but landed in my conciseness at the exact right time.

1. If you can’t write the invariant, you haven’t engineered it.
This is the difference between vibe coding and engineering. “Make it work” is not the same as “define what working means.” AI gives you plausibility, not correctness. The invariant is what keeps you from gambling with probabilities and calling it architecture.

2. If you can’t measure it in production, you didn’t build it.
Anyone can make a demo now. Real engineering starts when actual users show up and start breaking things in creative ways. Observability isn’t a bonus -it’s the only way to know if the magic still works after it leaves the notebook.

3. If you can’t explain why it failed to a regulator, you haven’t owned it.
“The AI did it” won’t fly with the FDA, SEC, or anyone with a pulse. If you can’t walk a smart non-engineer through what went wrong, you don’t understand your own system. Ownership means accountability, not just implementation.

4. Good system design still matters.
Data is still the asset. Models are temporary. The LLM you love today will be obsolete in six months. Build so you can swap the engine without rebuilding the car. The fundamentals haven’t changed - only the excuses have. The data represents your value / asset, the LLM is a processor to help you realize the value of your data.

Bottom line: these form a lifecycle - specify, verify, and own. The tools changed. The responsibility didn’t.

 

Can You Feel the Vibe?

“Vibe coding,” if we have to name it, is the instinct to follow an idea straight into a running system before the bureaucracy wakes up. It’s intoxicating. It’s useful. And left alone, it’s shadow IT with better syntax highlighting.

So: keep the spark, lose the shrapnel.

The Instinct Is Fine. The Invisibility Isn’t.

People aren’t vibe coding because they hate process; they’re doing it because process is slow at the exact moment their brain is fast. You know this dance:

• The idea shows up fully formed (or close enough).

• You need a REPL, a repo, and a data source—now.

• Every extra form to fill is a chance the idea evaporates.

We can wag our fingers about “proper intake” or we can design a place for this energy to land safely. The behavior isn’t going away. Pretending it will just pushes it deeper underground.

Design a Sandbox, Not a Maze

Imagine a paved path that feels like a dirt trail. That’s the job.

I want a sandbox where anyone can mash ideas into code, hit “deploy,” get real feedback… and where we still capture an audit trail, enforce basic guardrails, and promote winners to the grown-up pipeline automatically. Freedom up front, governance in the substrate.

Call it Spark → Prove → Promote:

1. Spark (Fast & Loose, but Logged):

   • Spin up a temp repo/branch from a template (vibe-template), auto-tagged as experimental.

   • Ephemeral env with pre-approved IAM role, read-only test data, and a 24-hour TTL.

   • Everything auto-logs: who did what, when, and which data they touched.

2. Prove (Make It Real Enough):

   • Smoke tests or a tiny checklist: “Does it run? Is there a README? Any secrets in code?”

   • Observability baked in (stdout goes to a sandboxed log store; errors page you, not the on-call team).

   • A lightweight “promote” button that kicks off a review workflow.

3. Promote (Graduate or Archive):

   • If it’s useful, a pipeline migrates code to the main org, assigns owners, and wires alerting.

   • If it’s junk, auto-archive and clean up the infra. No guilt, no clutter.

A win is a win. A miss evaporates without leaving landmines.

Guardrails Without Killing the Vibe

Here are the rails I care about. Most are invisible at run time, very visible at audit time.

• Identity Everywhere: Every scratch environment is tied to an actual person and a ticketable artifact (even if the ticket is created behind the scenes). “Anonymous Lambda” is not a vibe.

• Data Diets: Only synthetic or scrubbed data in Spark. Production data is opt-in, logged, and red-flagged. Boring but mandatory.

• Ephemeral by Default: Time-to-live on resources. If no one renews, it self-destructs (with polite warnings).

• Policy-as-Code, Quietly: Template repos include OPA/Snyk/whatever scans that fail hard on secrets and license issues. People shouldn’t have to think about it; it just works.

• Observability from Line One: Drop a logger, get a dashboard. If your experiment crashes, you get the ping, not PagerDuty at 2 a.m.

• Clear Promotion Path: Don’t make success painful. One click (or simple PR label) should kick off “make this production-ready” automations: code owners, IaC modules, secrets rotation, etc.

Tiny Example: The “Vibe Template” Repo

# vibe-template/.github/workflows/ci.yml
name: vibe-ci
on: [push]
jobs:
  test-n-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest --maxfail=1
      - name: secret-scan
        uses: trufflesecurity/trufflehog-actions-scan@v3
      - name: opa-policy
        run: opa eval --fail-defined -i policy.rego -d src

Nobody asked for a pipeline. It’s just there—fast, opinionated, and loud only when something’s wrong.

Three Questions Before You Push “Run”

If you’re about to vibe code (or bless someone who is), ask:

1. Can this leak anything we’d be fired for leaking? No? Proceed.

2. Will anyone be paged if this melts? Yes? Rethink. Make the blast radius yours.

3. If it works, do we know the next step? Success without a path is just future pain.

If you can’t answer those in 30 seconds, your sandbox isn’t sand enough.

What This Looks Like for Real Orgs (i.e., Us)

• Platform Team: Builds and owns the vibe platform: templates, ephemeral envs, guardrails code. Think “internal Heroku for experiments.”

• Security/AppSec: Codifies “thou shalt not” as policies embedded in the template, reviews the promote stage, not every commit.

• Engineers/Analysts/Whoever Codes: Use it guilt-free. The rule: if you’re outside the sandbox, you’re in grown-up land and the rules apply.

• Leadership: Measures outcomes: # of experiments, # promoted, lead time from idea to prod, incidents caused by experiments (ideally zero). If the metrics go the wrong way, adjust guardrails—not the concept.

Why Bother?

Because the alternative is either (a) slow innovation or (b) silent innovation. Option (b) feels faster until the audit hits, or the clever Lambda starts costing $20k/month in egress fees.

We can have speed and safety. We just have to engineer it. (Shocking, I know.)

---

TL;DR (Because I Know You Skipped)

• Vibe coding is happening. Engrave it in policy or pretend it isn’t—your choice.

• Build a paved sandbox: Spark → Prove → Promote.

• Guardrails = identity, data boundaries, ephemeral infra, policy-as-code, built-in observability, smooth promotion.

• Success path must be easier than stealth path. That’s the only lever that actually works.

Keep the spark. Lose the shrapnel.

Exploring DORA

We are considering implementing DORA metrics to help track and improve our outcomes. I had never heard of the term DORA, but most of the metrics rolled into this moniker I have a history with. As part of the discussion around DORA, we met with a few vendors whose products integrate with GitHub, Jira, etc., to surface metrics. While we didn't end up purchasing a product to help here, it wasn't because we needed to see the value of DORA. On the contrary, we value the outcomes that DORA metrics (et al.) are after; it is more about timing. Through acquisitions, mergers, and growth, we have a diverse group of teams, each at different maturity levels and mechanisms around SDLC. Starting to track DORA-esque metrics when a team may be working to climb up on the maturity curve seems premature for a few reasons. First, we should focus on helping teams up-level their SDLC. For instance, CI/CD is good, so everyone should do that. Second, most mechanisms have some metrics as outputs that we can leverage. If you're doing Scrum, are you tracking the key metrics or meeting every two weeks to plan work? Lastly, we need to be careful when comparing across teams that don't do the same thing or have similar needs. The apples and oranges analogy exists because it works.

As a team leader looking to uplevel the entire engineering organization, DORA and products that capture related metrics sound attractive from a centralized visibility perspective. Being new to the organization, I know relatively little about each team's maturity level, but many feel there is room to improve. Does that mean we are not going to track any DORA metrics? Of course not; we will track the ones with shared mechanisms and leverage that data. Measure, identify, improve, repeat.

P.S. I was pre-Dora The Explorer. But my kids weren't. Geography interests me, so this Dora is fabulous in my book.

A Brief History of Integrating

I was recently asked how I would integrate systems with similar functionality but different code bases and infrastructure. It got me thinking about how I feel I have been solving the integration problem throughout my career and how it (not surprisingly) keeps coming up. My answer was: It depends on what you have and where you want to land. I have many questions: how much of the data is the same? How reliable is the data? Is there an appetite to reduce any duplication? What is the desired recency (age) of the data? Is there a goal to become more homogenous? Regardless of the answers, I have employed several strategies/technologies over my career.

One of the first implementations I worked on was a practical solution for a PC-based application to leverage COBOL code on the mainframe. The company had invested years in the code and was not ready (or able) to rewrite it. We built an LU6.2 gateway to call the mainframe and pass data to the COBOL file's copybook. This was such a common need that later, Microsoft wrote an LU6.2 connector to create a code proxy from the copybook for you. Here we are 24 years after Y2K, and I am willing to bet that the banking and finance sectors still have a bunch of COBOL being used.

During the Object-Oriented boom, there was a big push to build an Enterprise Service Bus (ESB). We talked about technologies like MQSeries, CORBA, and RPC as ways to implement an ESB. I never saw an ESB I liked, probably because so many were never finished. Instead, I watched enterprise architects trying to resolve the enterprises' requirements into something for everyone. It's hard enough to nail this down for a single business line in an enterprise, much less many/all of them.

To some extent, we used databases as the point of integration. One system would be declared the data owner and copies of that data would be made for use by others. At the time, I saw this strategy fail more often than it succeeded because the copy would become stale and not meet our customer's expectations. Nowadays, we make copies, but it's less about integration and more about scale. To succeed, we had to figure out how to work in the world of eventual consistency.

As I shared these examples, I couldn't help but think of a few other topics. But these are the ones I wanted to share. If you've been in this industry for any time, I'm sure you have your own integration history. Integration is a persistent problem, a challenge that we all face, and it's safe to say that it's not going away anytime soon.

Do or do not. There is no try.

I recently dreamed I was in a large audience where an unidentified leader asked for a volunteer to lead a big project. They looked at me, and I said something like I will try, but I couldn't do it alone. Someone else stood up and said they would take on the project, and I could feel the audience staring at me with disdain. So much so that it woke me up.

I am not the type of person who looks for a lot of meaning in my dreams, mostly because I don't remember them. But this one woke me up, and I spent the rest of the early morning lying in bed, considering why it bothered me.

This was similar to talking to others and asking them how we get something done, and their answer is something like, "We don't have the resources." That wasn't the question; the question was, what would it take? My dream was similar because the question was not, "Would you deliver this project by yourself." It's not a question of ensuring a successful outcome; it's a question of pulling together enough for a reasonable discussion on feasibility.

The response I would guide my dream self to give is: Sure, I will lead that big project for you. Then, I would document what it's going to take to get it done and present that to the stakeholders. 

This dream bothered me because I was settling for trying when I knew better.

My Experience of APIs

I watched Nate Jones's TicTok last week. He talked about how APIs are becoming the interface to data. As I noodled on his words, I agreed with him, and it got me thinking about my history with APIs - more generally than his point.

I have been developing or calling APIs for a long time, and this got me thinking: weren't the MSDOS interrupts (related post) a form of API? We all know the right answer, so we will not say it out loud and move on.

But seriously. I remember creating an API for the DOS terminate and stay resident (TSR) module I wrote as the proxy for the server-based DBMS we wrote. This API used INT2F as the interface to the TSR.  We didn't expect CBASIC programmers to code to INT2F, so I had a callable C library that abstracted the internals. What we built was closer to something like ODBC, a generic API for retrieving data requiring you to write the query.

There was a period when our systems had a business layer and a data layer; the business layer represented business semantics, and the data layer represented the sproc/view/query/table structure. Weren't these all APIs? During the object-oriented period (which sounds like the Jurassic period), I didn't refer to them as APIs then; they were methods on an object.

We went from RPC to SOAP to REST, each an evolution to build distributed systems. All these are forms of APIs as well. Why? APIs are about creating a dependable contract that callers can depend on to create an outcome—not about the specific implementation technology. The "interesting" part of the API design process was how to design for reusability. And by interesting, I mean the part we spent a lot of time discussing. In the Enterprise IT space, we had enterprise architects who created taxonomies, libraries, and APIs that modeled the data and/or the business processes. Building models/services for the entire company didn't work, and a "single responsibility model" and "separation of concerns" were needed. Even now, we have engineers who have API design as a skill we depend on and use at some level of governance.

From these evolutions (and more) of APIs, I have learned they are hard. Why? Because API design is hard. Why? Because reuse is hard. This is why when we talk about APIs today, we don't think of them as static. Instead, we build them to be extensible as the needs evolve. Even designing for evolution, I have had more than my fair share of painful API deprecations.

I still love a good API design session.

Secure Apps Take a Village

I reflect on recent experiences and the tension between rolling out features and ensuring the product is secure. Features are flashy. They make good keynotes and/or demos. Features get customers excited and make money.

Security is about the backend. It's not flashy. It's hard to demo security; you probably wouldn't want to even if you could. But when something goes sideways, a security issue will lose you customers. It's like giving your competition bullets—"Hey, did you hear about that security breach at Acme?"

Security is getting more complex and more challenging to do well. Sure, we are smarter and better understand vulnerabilities. However, applications are also becoming more complicated (e.g., log4j and versioning). Complexity makes security harder—it just does. The bad actors are as determined as ever, and their methods continue to evolve.

I attribute much of this tension to the organization's culture. A typical culture pattern is when leadership compresses engineering estimates, and as time runs out, security verification takes place. This leads to pressure to complete verification and remedy errors without impacting the date. Regardless of the number of times that leaders tell people how important security is - actions speak louder. We must ship at this point in the project, leaving uncomfortable conversations and hard choices. 

The tension between shipping and being secure starts early, when most of the time is spent discussing/planning customer-facing features. Sure, security comes up, but in my experience, it gets a different in-depth consideration. If we are lucky,  the challenging security conversations/decisions are before we launch - and not after. Talking about security feels more like a bad thing when it's this late. Further contributing to this tension is when leaders push to make a date, a security issue is identified, and in response, they make statements akin to - "Security is always a priority, so when I push on the date, my expectation is that the system is still secure."

Imagine a product where this culture goes on for a long time; the product is very successful and lucky that lurking security issues are private. With some certainty, this culture will catch up with you, and a critical mass of security issues will arise, requiring engineering resources. Sometimes, the number and/or severity of the problems is so great - that feature delivery is paused, and most of the team is focused on remediation. I bet that for every case like this in the press, many more are not public.

Too often, I have seen teams rely on the hero model: people who, early on, take the risk of spending time on security—which, in my opinion, is under-acknowledged/appreciated. Or the heroes who scramble to fix issues after they are found—typically in a severe time crunch. The hero model is not sustainable and is unreliable for ensuring secure systems. Further, the hero model is often a symptom of something "wrong."

So, let's stop relying on heroes, best intentions, hollow statements, and punishing the bearers of bad news. In a fiercely competitive world, features are prioritized, and security often gets less attention, sometimes even becoming an afterthought. This is a culture war that needs to be waged. 

We must start by considering security not just as a necessity but as a feature as important as any other. We need to talk about it early. We need to do security reviews early and often. For instance, start a threat model on the first day, keep it up to date, and assess design decisions in the context of how it would change the threat model. Activities like this involve staffing security-oriented teams and embedding them into the project teams. Then, empower them to ensure positive security-oriented outcomes. A culture of security is independent of volunteers who take on additional responsibility to ensure a secure system.  We need mechanisms to help people do the "right" thing. Then, acknowledge (early) security wins, such as actions to avoid an event.

A culture of security is hard to change, and as leaders, we need to be honest with ourselves about it — are we backing up our words in ways that will lead to the outcome we want? Look in the mirror. Yes, the words mean something, but it takes more than words.

PS: If you're unfamiliar with the African phrase "it takes a village" - see here.

Classic Tech Books I Still Have My Shelf

Relating some of the books I still have on my bookshelf. They are totally obsolete but found myself reminiscing around what they represented from my past and why they still took up space on my shelf.

One thing to know about me is that by default, I am a book collector. I once had a collection of over 200 technical books on my shelf. That was before the great purge where I donated most of the books and hung onto 30 or so. The great book purge is a story for another time. Suffice it to say that the books that survived were either timeless or ones that I had a fondness for. Here are a few of the fond ones.

The Pink Shirt Book. What would I have done without this book? If you were programming to the "IBM PC" at the time, this was a must have. I pulled it off the shelf and it flopped open the chapter on INT 21. This was the system services interrupt the keyboard, files, etc. I also found a dog-eared page for the INT 27 terminate and stay resident (TSR) details as well. TSRs were the services of the time. This was mostly assembler, then bumping up to C as soon as I could.

The Genie Book. I used this book in conjunction with the Norton Pink Shirt book. It came out after the Norton book, but had more details on writing TSRs, multitasking, and expanded memory. I pulled this book off the shelf and there is a page that was used so much that it was sticking out - having come off the binding. Now that is one well used page. When grabbing the image for this post, I found that this book was available on the Internet Archive (archive.org). E-versions of technical books is one reason I was comfortable getting rid of so many of my technical books (see future purge post).

The Microsoft C 5.0 Quick Reference Guide. This is an actual picture of my copy because I could not find a stock photo. Note the cup stains on the cover. It was a C reference, the page I used the most for C were the escape sequences for printf (e.g. \n, \r) and the language type ranges (e.g. difference between long and float). I still use this reference occasionally for the last table on ASCII codes - the decimal, hex, and character representation of 0 to 255. If you have seen The Martian, then you know.

The Intel 386 Microprocessor Hardware Reference Manual. Yikes, I am not sure why I am hanging onto this one, I have not used it since the time when I needed it. I have flipped through it a couple of time since and seen all the timing and logic diagrams and thought - we actually did this stuff. I primarily used this around interrupts (maskable vs. non-maskable) and for figuring out peripheral timings for external serial devices. We were doing some pretty hard-core stuff at the time. I recall a device we borrowed from one of the hardware manufacturers that sat in between the CPU and the socket. It allowed you to debug at the CPU level and single step through code. I remember we called it the skateboard because it resembled one but have not seen or heard of one since. Hardcore.

It's worth noting that many of the folks I worked with during this time are still friends. We were young and thought anything was possible. On top of that we had a leader who inspired us to live on the edge and accomplish amazing things.

Builders gotta build

When I talk to people about what I like about what I do - there are usually two things that I tell them. First that I like to build stuff and second to solve problems. I am a builder - both in my professional and personal life. Building software is what I get paid to do because I have demonstrated that I am good at it. Building in the physical world is just fun.

Take for instance "building" at home. Fixed things in the house. Upgrades. Or creating something that I need. The one that I wished I had yesterday was a yard tool holder. I recently moved and left the yard tool holder behind. It was something that I built out of extra lumber I had around the house. It looked something like the picture I attached, but custom made for the tools and the materials I had available. I knew what I wanted / needed but was constrained by what I had available.

Sounds like a parallel to building software. There are the requirements and then there is what you can do. The components available and time are both constraints to what you can complete. This is one place where the problem solving comes into play - putting it all together to complete (deliver) something.

There is something rewarding about completing something that I built. Yesterday it was waxing poetic about my yard tool holder. Sounds like an opportunity to consider version 2.

New Coding Problems = Old Coding Problems?

Late last year and into this year (2024) I needed to take time off to recover from surgery. As my mind started to clear but my body was still healing, I started poking around some code. At the time I was playing Pokémon Go (PG) and wanted to build my own mapping application. Well not actually from scratch but build it using a bunch of Python that others had assembled. I was not successful, and it the reason why got me thinking about libraries vs. binaries. 

Library vs. Binary reuse is one of those problems that has been around as long as I can remember. The first time I ran into it was when coding in C (pre-C++) and considering an #inlcude file (.h), a library (.LIB), or a dynamic link library (.DLL). I don't plan on rehashing that thinking now because we all know the right answer and so we can just move on.

The Python code / repositories I was trying to assemble for this purpose suffered greatly from the conflicting version problem. Where code in the same namespace was dependent on different versions of the same library. This creates a Gordian Knot of mixed versions that is very difficult to work out. I am no Python expert, but my understanding from those that are is that this can be a common thing in the world of Python.

Not to give Python a bad name, I know it would happen in Java too. Yes, there are ways to work around this, but I was more focused on getting the solution running that having to unravel dependency tree conflicts.

It was a walk down memory lane that got me thinking that a lot of the old problems I had as a developer are still around. I know the tools have gotten better at helping us manage these and lessen the pain of the problem, but the problem still exists.

The Easy Fallacy

I was talking to one of the younger engineers at work recently and he said that he appreciated my experience. After I realized that it wasn't one of those - your so old moments. I laughed and told him that is probably more true than he realized. For instance, we were talking about an a fix that he labeled as "easy". I told him that in my experience easy changes can be a fallacy. I could see in his expression that he knew what I saying but he didn't know what I was saying.

I can't count the number of times when me, my colleagues, and/or team fell for this one. It looks something like - the change is a small one, often innocuous - like a configuration change. If you're nodding you're head right now, then you too have fallen or been witness to this one.

This happened recently a couple of times. One looked like a change that when merged to to the main branch, inadvertently picked up an unrelated change. This change was not something that was caught until it was promoted to production and resulted in a performance impact. As a result we had a week delay two weeks before the larger launch because we had to do even more work to earn trust that it would not happen a second time.

A contributor to falling for this fallacy is that making simple changes in a complex system isn't about how small the change is - it's about everything else after the change.

Which begs the question, how do you not fall for this one? I feel like this one hurts the most when your under a time crunch and your belief in the fallacy convinces you that you should make "easy" last minute changes. Ouch. Or that you let your guard down on the "easy" change and miss something. The answer for these two is - don't do this. I am sure you have your version of this. 

It works on my machine

One truth I had to come to terms with earlier in my career was the "it works on my machine" anti-pattern. It's a close cousin to the name of my blog - sometimes words DO mean something.

This was back in the days before Windows (yes there was such a thing) and I was writing a custom TSR (terminate and stay resident) module that we used to communicate with the custom database engine we wrote on NetWare (starting to get a sense of the decade yet?). Long story short there was a problem with the code that prevented it from working on one machine - just one. And it wasn't like it always didn't work, it only didn't work sporadically.

In my arrogance at the time, I was slow to take a look at this issue. My arrogant thinking went like (1) this is the only machine to have the issue (2) I spent a lot of time on this code and especially testing it (3) this code has been running on hundreds of other machines without an issue (4) so it must be something particular to the machine or user. Fast forward a couple of weeks and surprise a couple more people report having the issue. It was then that a colleague and I were talking, and I woke up from the spell - "this is something I need to look into" I finally told myself.

Turns out the issue was an interrupt conflict with a badly behaved driver that was slowly be rolled out in a pilot and very soon would release across the entire org and would impact my, now thousands, of customers. Once I figured this out (which was actually pretty hard to do) - the fix was simple; I just switched to a different interrupt. There were other times when I fell under the spell of my own perfume, each time learning something and making me humbler.

Computers and software have grown much more complex since then. developers have much less control over all the code they are dependent upon. In the example above, I was dependent on the CPU microcode, the BIOS and MSDOS - all of which I had the source code for. I could usually discount the microcode (I only recycled the manual last year) which meant that any issue was either with my code or the few dependencies I had. The lack of dependencies and illusion of control was part of where my arrogance came from. But today there are dependencies on dependencies; it's such a complex ecosystem. And yet developers still have quite a bit of arrogance. 

Why am I writing about this now? This time I was the one user for which the application did not work - last year (mid-2020) it started, and I would constantly send logs and traces to the development team, and nothing would happen. I told them that my fear is that my experience was the "canary in the coal mine" and that either (1) other people are also having the issue and not reporting it OR (2) that at some time in the not-too-distant future it would become more urgent. 

The later was this week.

It now works on my machine.

Keep It Simple (kiss) Revisited

I have a calendar from a vendor we use that has some of the classic coding and design principles - one for each month.  I was rubbing my chin staring at it this morning and I wanted to share what popped into my head...

While I am sure that the KISS principle has been written about (perhaps to death) I had another instance of this today as it applies to operations and infrastructure.

Quick background - I recently inherited an Operational group.  Operations is the clean up crew of development here.  While I understand the rationale of separating them, I think I like the idea of developers supporting their own code so that they better understand the impact of what they do.  What a great teaching tool - you want to not get up in the middle of the night - fix the code, do a better job in the first place, write a utility to help you out.  We have a bunch of applications that have been around for years and over time the developers who maintain many of these have moved on.  So today I asked the question of someone about two AD groups and what they are used for.  In both cases the answer was initially I don't know - and later the answer became these are not used anymore.

Part of keeping systems simple is getting rid of the things that are not used anymore.  We have all these extraneous moving parts that we don't need.  This just creates system bloat that should be easy to remove.

Granted you cannot get to everything - right now.  But this stuff has to get cleaned up over time.  Putting it into some sort of maintenance, wish list or Kaizen log seems like an easy thing to do.

All it takes is discipline.