Christopher Hendrix

Question for engineering leaders (managers, directors, VP's, etc): I'm curious about the current landscape of developer productivity analysis + management tools - things like LinearB and Jellyfish. Which tool, if any, are you using? Any opinions/pros/cons on the tool you're using (or the entire domain overall)?

21

Tech debt causally harms cost-of-change and meantime to adapt to market. You release a defect free, fully functioning piece of software that will *never change* but its code design is a mess. Would you care about tech debt? NO! Why? Because you don't need to change it. It works. It could be written in Whitespace or BrainF**k. Who cares. Nobody is going to look at it. What if it had a tiny change maybe once a year? How much would you care then? A bit more. Maybe Whitespace is ruled out but if it took a couple of days to work out what was happening would there be any business value in paying down the debt? 1 week of refactoring would take nearly 3 years before you break even. Not worth it. This isn't a hypothetical, I have built and maintained software which fits this exact scenario. When does the debt start to have a negative affect on cost-of-change and meantime to adapt to market? Once a quarter? Once a month? Now imagine that you are making hundreds of changes a day. How much is that tech debt costing your business now? And this is assuming that the software is 100% bug free and perfectly functioning. Now extend that, you are making hundreds of changes a day and the internal quality issues are causing you to create bugs which has the knock on effect of harming the user and customer's productivity. Devaluing your product. What's the cost of that debt to your business now? As engineers we need to talk more about how we are protecting and investing in technology in order to reduce cost-of-change and meantime to adapt to market, rather than talk technical debt. Because if we don't need to change or change rarely, then technical debt is not a problem.

39

11 Comments

We're looking for a senior-staff PM now at Honeycomb, who will drive and own a multi-year vision for a net-new, permanent and significant expansion of Honeycomb's product surface area: https://lnkd.in/g2sM_TfC I’ve sat in this role for 3.5 years now, on and off, so you’ll be inheriting a lot of my regretful decisions. But I’m here to help you navigate that! It involves big data (for real), deep OSS involvement, lots of customers who love the product, and a great team to boot. If you like the idea of thinking through how you’ll turn 10M events per second into 100k, without breaking the bank, and making it as semi-automatic as possible, without losing signal on that data, and building a compelling UI that lets people build and monitor this stuff, this might be up your alley! Wanna talk with tons of customers and spicy sales engineers who have to sometimes commit code crimes to make stuff work, and then tell you about it? Wanna make painful prioritization decisions? Wanna find a way to execute on a multi-year vision and not get lost along the way? Click the link above. Wanna eat shit in front of a customer whose exact set of constraints violates every tradeoff decision you made? And then go back to them 4 months later with the good news that you’ve fixed it all? Come say hi! Wanna try to wrap your head around someone's perceived notions of how observability works, and find creative ways to reorient their worldview? Hi hello please apply. Wanna get deeply involved in the #opentelemetry community, become a core contributor, attend lots of community events, and be continuously infused with customer problems and feedback and also overall industry directions? You know what to do. Wanna work with a very product-minded engineering team who will push you to get stuff done? Wanna be a part of a group that is usually fast, usually right on the first go of things, and isn't afraid to try things and maybe pull them back if they're not working? Wanna have fun too? Clicky clicky.

59

4 Comments

When engineers say "no," it reveals flaws in your pitch or opportunity, helping refine your approach Here are a few common mistakes 👉 𝗖𝗼𝗻𝘁𝗮𝗰𝘁𝗶𝗻𝗴 𝗽𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝗰𝗮𝗻𝗱𝗶𝗱𝗮𝘁𝗲𝘀 - 1. Not including basics things like work location, comp range, and tech stack 2. Not highlighting what makes the role/company relevant for them 3. Targeting the wrong (or not enough) people 𝗙𝗶𝗿𝘀𝘁 𝗶𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄𝘀 - 4. Jumping right into technical questions before creating buy-in 5. Not pitching your company's vision 6. Not understanding why the candidate is interviewing and what they're looking for in their next role 𝗟𝗮𝘁𝗲𝗿 𝘀𝘁𝗮𝗴𝗲𝘀 - 7. Assuming candidates are interested, without understanding their decision framework 8. Not debriefing and addressing concerns after every interview 9. Extending offers without pre-closing Good recruitment involves a lot more than just finding top talent Have a great weekend!

8

1 Comment

The best use case for overseas development I've heard: We have customers in the eastern hemisphere. To support those customers better, we're standing up product, dev, and support teams to help serve those customers needs. Two key ingredients in making that successful: commensurate pay and the same level of performance and accountability as US peers. #software #international #teammanagement #overseasdevelopment #startup #leadership #coaching

6

After numerous recent conversations with engineering leaders, it's clearer than ever to me that when it comes to software development, we're caught in a reactive app performance management cycle. Teams prioritize rapid feature shipping, inadvertently accumulating performance debt with each release. The pattern is frustratingly predictable: 1. Ship features quickly 2. Performance gradually degrades 3. Users complain or infrastructure costs spike 4. Briefly optimize performance 5. Repeat the cycle This approach is fundamentally broken. Performance ideally should be a first-class citizen in our development process, not an afterthought triggered by crisis. With Codeflash, I'm exploring a more proactive approach to performance engineering. The goal is to build performance consciousness into our development workflows, preventing technical debt before it becomes a systemic problem. I'm eager to hear from other engineers: How are you addressing performance challenges in your organizations? What strategies have you found effective in maintaining software efficiency without sacrificing development speed?

30

1 Comment

If you really need multiple backend repos before you have multiple teams question whether you should be multiple companies. If not you’re probably better off as a monorepo if so you’ve got bigger concerns. At UpContent we’re actively retreating from the architectural complexity of micro-repos, as a smaller team it’s just unnecessary and most tech these days support multiple separately deployable components/services in a mono repo. Exceptions might be tiny services that don’t hit your main db and for whatever reason are in another language. People always forget that part of microservice architecture principles were separate databases and no shared code. People also forget they were a way for large companies to segregate duties with fully autonomous teams each owning a service not a strategy for startups with one team. Multi-repo is a very valuable tool in the tool box but it is one for after your first team split and when you process in place to handle cross team coordination.

1

Every leader of a Tech Organization should strive for making that org better: More productive, more customer-focused, more fun to work in. 🌟 There is tons of things you can do (and equally many to avoid) when it comes to: ➡ structuring your org, ➡ aligning its teams around common goals, ➡ dealing with technical challenges, ➡ getting ready for leveraging AI, ➡ leading your org through changes. As also leadership is most fun when it's done together in a group, I will host a full-day workshop "How to Build and Run a Successful Tech Organization". 😃 In a group of like-minded tech leads we will discuss the above topics at Digitale Leute Summit 2024 in Cologne. A few spots are left. Get yours now at https://lnkd.in/dXyFnvGi (The discount code "DLS24JAN20" gives you 20% off. 😉) 🗓 14.11.2024 📍 STARTPLATZ Cologne ⌚ 9:00 - 17:00 Have a look at the several other amazingly valuable workshops as well: - Design Sprint for AI solutions with Markus Maass and Lars Becker - Product Discovery workshop with Frederik Vosberg and Timothy Krechel - OKR workshop with Christina Lange - Effective Product Strategy workshop with Tim Herbig - Masterclass on Team Topologies with Matthew Skelton and Manuel Pais (Team Topologies) 🇺🇦 🇵🇸 from Team Topologies #dlsummit

17

7 Comments

Prompting Strategy of the Day coach LLMs like someone who reports to you: Good engineering managers know that if you have given an engineer requirements and have even clarified them and expectations aren't being met the solution often falls out of asking the engineer to walk you through their approach step by step in words not code. Sometimes being able to identify just 10% of what they are missing unlocks the whole rest of the problem. Austin: The test is still failing.   To be clear the issue with  <the input string we are testing and want our method to handle gracefully> is that  <how the small addition I made to the input string is expected to cause a failure in a dependency we can’t control> Before generating more code why don’t you walk me through in words how you are you are breaking up our input string to be able to analyze it. You don't need to tell me how you are analyzing it. I just want to hear the description of how and what you broke out at each step so that, maybe I can help you better. LLM: <explanation> Austin: Ah I see, but are you conscious that, <something incredibly obvious being described that is wrong but that isn't so obvious in the code> If you have completed step 1 then at step 2 you should check first for the presence of  <delimiters that are optional and can appear in most any order>  if neither are present then we are done. But if one or both exist I would check for <the one easy exception to ordering rule that makes finding the solution easier> LLM: <provides code fixes the problem at hand but also that exceeds expectations elsewhere>

5

Cut Down Your Platform Engineering Org By 50%. Your Business Will Be OK! If you are a large enough tech company, you have a bloated platform engineering team! A couple of decades ago, Joel Spolsky coined the term 'Architectural Astronauts'—engineers who create grandiose, abstract designs that often fail to address real-world problems. Unfortunately, this issue has only worsened, especially in platform engineering. One of my biggest career regrets? Not fighting hard enough against these platform architects. Forced top-down architectures could have been replaced by simple open-source frameworks. I remember it taking over a year to build a self-service API because of a top-down, inversion-of-control framework forced upon all engineering orgs. Read about what killed popular products like Digg, Orkut, NetScape etc.  You can trace it back to bad engineering thought leadership. The amount of platform engineering waste is staggering. Most large tech companies are a graveyard of abandoned platforms and frameworks. Most in-production frameworks have one tenant and are on the deprecation path. The engineering 'thought leaders' who build these systems got promoted and will still rationalize their decisions by passionately talking about edge cases that will never happen. Almost every tech stack begins with off-the-shelf standard frameworks and platforms, mostly open source. But no two businesses are the same. And companies often start building their own platform pieces. As these platform teams grow, they become out of touch with the actual needs of the business. Led by powerful, disconnected engineering leaders, these teams wield more power than they should. Their solution to an inadequate platform? Build a new one, rather than improving the existing one. Unfortunately most of your IP & patents are from these platform teams. Most of your top engineering brain power is working on these platform teams. You will be warned how you will lose the top talent when you question these investments. If you have a 100+ people platform engineering org, cut it down by 50%. Your business will be OK.

30

10 Comments

on hiring an ea While companies like Athena and Wing are popular options, they come with challenges I didn't like: * Time zone differences * Cultural gaps * High initial costs ($3000+/month) Instead, leverage your network. Tell everyone you know that you're looking for an EA. You'll be surprised how many qualified candidates are already in your circle. Plus, working with someone who understands your culture and timezone is invaluable. Two crucial tips for success: Document everything. Create clear SOPs using tools like Notion or Loom. This makes onboarding smoother and sets clear expectations. Start small and scale up. Begin with a capped number of hours at a set rate. This lets you build trust and understand your needs. (Trust me - once you find someone great like my EA Maria, you'll wish you had more time with them!) Bottom line: Start with who you know, really just ask everyone for referrals first, document well, and scale gradually

6

When Klaviyo recruited me three years ago to help architect a system that would deliver hundreds of millions of email messages per hour on behalf of their customers, I thought, "Wow. That's a lot of mail. How could I *not* build that?" Now that we're doing it for real, I still think, "Wow. That's a lot of mail. We *built* that." $3 billion+ in Klaviyo-attributed value. 420 million messages sent in a single hour on #BlackFriday. And over 15,000 customers having their best day ever (🙌) on Klaviyo.  Read our full BFCM By The Numbers report to learn how Klaviyo customers achieved recording-breaking results this #BFCM.

35

9 Comments

This resonates so much. No, you can't do it all, especially not at a startup. Since COVID I see so many people try to balance unaided parenting with full-time careers, and no, I don't see anyone doing it well. My kids are now 17, 13 and 10 and just now, when they trek to most of their activities on their own, and do their HW on their own, and cook their own meal or snack, we have finally let go of the outside help ( and this is with both parents working from home for last 5 years, outside of peak COVID lockdown, we had always relied on a designated childminder outside of school hours). Yes, working remotely does mean that I have more time and flexibility in my life, it means I can walk my kids to school sometimes, or have breakfast with them instead of rushing to the office, it means I can go and see their midweek concert at school, or be there to grab them from school on a sick day in 10 mins. But it also means that on EVERY day basis we all need clear boundaries, work hours, dedicated spaces that are not continuously invaded by children, or feelings of guilt that our children are unattended. One thing that a lot of the arrangments lack is transparency, being explicit with ourselves, our partners, our managers and colleagues on when we are making trade-offs. What helps is verbalising our plans, putting explicit blocks on calendars for pickups, doctor's appointments and school activities, letting others know when you will have periods of diminished focus due to a school holiday or a sick kid at home, and ruthlessly prioritising. Very few of us can claim to be at Julius Ceasar level of multi-tasking. Flexibility of remote work ≠ combining full time work with full time parenting

25

Sup fam!! Relationships are everything!! They're the core of your engineering platform. Your team's design docs, software patterns, pull requests, and GitOps workflows all need to pass through a complex network of relationships before becoming meaningful to stakeholders. Before that, they're just bytes sitting on Jeff Bezos' disks. How are your relationships doing? Which relationships need some light touching up? Which ones need some serious investment? How about that one person on that one team? How about on _your_ team? Are you tracking these gaps with the same effectiveness you're tracking tech debt? It takes an intentional effort to build meaningful connections and to show peers you're a stakeholder in _their_ success. It's worth your time -- not to mention you'll feel great!! Want to optimize your engineering platform? Start by investing in your relationships!

38

Why asking your customers what they want doesn't work: 𝗛𝗲𝗿𝗲’𝘀 𝗮 𝗰𝗹𝗮𝘀𝘀𝗶𝗰 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗼𝗳 𝘁𝗿𝘆𝗶𝗻𝗴 𝘁𝗼 𝗴𝗶𝘃𝗲 𝘁𝗵𝗲 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝘄𝗮𝗻𝘁, from Intuit (the $170B financial giant): With the well-intended effort of giving customers what they ask for, Cook [Intuit’s CEO] recalls, Intuit’s development teams would extensively survey customers about what new features they’d like to see in Intuit products. And customers had a lot to say. 𝗧𝗵𝗲𝘆’𝗱 𝗿𝗮𝘁𝘁𝗹𝗲 𝗼𝗳𝗳 𝗮𝗻 𝗲𝘅𝗽𝗮𝗻𝘀𝗶𝘃𝗲 𝘄𝗶𝘀𝗵 𝗹𝗶𝘀𝘁. “They’d ask for 150 features,” Cook says. So the team jumped on that feedback. Development teams would spend weeks arguing and debating which of the list of potential new features were most important to provide. Everybody, Cook says, was guided by what he or she thought was right for the customer. But in reality, it offered no guidance at all. “We got into feature chase,” Cook says. “Too often we’d go look at what customers were asking for and build it.” But absent a clear understanding of 𝘁𝗵𝗲 𝗷𝗼𝗯 𝘁𝗵𝗲 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 𝘄𝗲𝗿𝗲 𝗵𝗶𝗿𝗶𝗻𝗴 𝘁𝗵𝗮𝘁 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝘁𝗼 𝗱𝗼, “there was simply no way to differentiate which features were the right ones. It’s like navigating without a compass.” ----------------------------------------------- The story is from Clay Christensen's "𝘾𝙤𝙢𝙥𝙚𝙩𝙞𝙣𝙜 𝘼𝙜𝙖𝙞𝙣𝙨𝙩 𝙇𝙪𝙘𝙠: 𝙏𝙝𝙚 𝙎𝙩𝙤𝙧𝙮 𝙤𝙛 𝙄𝙣𝙣𝙤𝙫𝙖𝙩𝙞𝙤𝙣 𝙖𝙣𝙙 𝘾𝙪𝙨𝙩𝙤𝙢𝙚𝙧 𝘾𝙝𝙤𝙞𝙘𝙚". If you want to know the solution, read the book :) Or check out the article (and find out how milkshakes are related to this!): https://lnkd.in/d9NuXMcx I'll wrap up with this great quote by W. Edwards Deming: (thanks Paulo for the reference!) “𝘐𝘧 𝘺𝘰𝘶 𝘥𝘰 𝘯𝘰𝘵 𝘬𝘯𝘰𝘸 𝘩𝘰𝘸 𝘵𝘰 𝘢𝘴𝘬 𝘵𝘩𝘦 𝘳𝘪𝘨𝘩𝘵 𝘲𝘶𝘦𝘴𝘵𝘪𝘰𝘯, 𝘺𝘰𝘶 𝘥𝘪𝘴𝘤𝘰𝘷𝘦𝘳 𝘯𝘰𝘵𝘩𝘪𝘯𝘨. 𝘈𝘧𝘵𝘦𝘳 𝘥𝘦𝘤𝘢𝘥𝘦𝘴 𝘰𝘧 𝘸𝘢𝘵𝘤𝘩𝘪𝘯𝘨 𝘨𝘳𝘦𝘢𝘵 𝘤𝘰𝘮𝘱𝘢𝘯𝘪𝘦𝘴 𝘧𝘢𝘪𝘭 𝘳𝘦𝘱𝘦𝘢𝘵𝘦𝘥𝘭𝘺, 𝘐’𝘷𝘦 𝘤𝘰𝘯𝘤𝘭𝘶𝘥𝘦𝘥 𝘵𝘩𝘢𝘵 𝘵𝘩𝘦𝘳𝘦 𝘪𝘴, 𝘪𝘯𝘥𝘦𝘦𝘥, 𝘢 𝘣𝘦𝘵𝘵𝘦𝘳 𝘲𝘶𝘦𝘴𝘵𝘪𝘰𝘯 𝘵𝘰 𝘢𝘴𝘬: 𝘞𝘩𝘢𝘵 𝘫𝘰𝘣 𝘥𝘪𝘥 𝘺𝘰𝘶 𝘩𝘪𝘳𝘦 𝘵𝘩𝘢𝘵 𝘱𝘳𝘰𝘥𝘶𝘤𝘵 𝘵𝘰 𝘥𝘰?"

9

4 Comments

How do you tackle technical debt? Every software company has technical debt, and the majority struggle to pay it back. After reading Shape Up, we've been trialing 8-week cycles of a 6-week build phase followed by a 2-week cool down. The cool down phase allows us to catch our breath and prepare for the next cycle. More importantly, it also allows us to focus on technical debt. The 2-week window gives us plenty of dedicated time to focus on large issues that will have significant benefits for future cycles.

12

5 Comments

On the flip side, many teams configure an excessive number of alerts, often with overly strict thresholds. A significant portion of these alerts are non-actionable, leading to alert fatigue. The common response becomes: 'These alerts are expected,' or 'They’ll self-resolve.' However, effective monitoring is only valuable when alerts are actionable and tied to clear resolution playbooks. Without this, alerts become noise rather than signals, undermining operational efficiency.

8

Last Tuesday, Malsgufa, our legacy CMS backend, was shut down. This milestone for BI's Tech and Product team culminated years of planning, effort, and execution. Malsgufa's first commit was on July 25, 2014, leading to 5,186 commits across nearly 30 repositories. Built and maintained by scores of engineers, its deprecation represents significant cost savings (20 production replicas, 3 instances of Solr, and an insane number of Zookeeper instances!), reduced security risk, and the elimination of an often-blamed boogeyman. The journey began in January 2020. Our challenge: moving traffic off Malsgufa without rebuilding everything in Muse. Initially, we tried GraphQL as an intermediary between Viking and our services, but progress was limited by Viking's complex Angular/Vue architecture. In January 2023, we pivoted to the strangler fig pattern with a shim. This approach proved far more effective – all Malsgufa traffic went through the shim first, with verbose logging to track unhandled requests. Meanwhile, we built Story Versioning API (SVAPI), incorporating lessons learned from Malsgufa's limitations around history and publishing states. We ran four major migrations (stories, custom pages, slideshows, and videos), building support in Muse and shimmed compatibility for Viking. Traffic gradually decreased from a torrent to a trickle, leading to the gloriously uneventful shutdown. Next, we will deprecate Viking by Q2 2025, unifying our users in the Muse ecosystem to create the industry's most innovative CMS. Special thanks to the entire Story Creation team: Chase Gruber, Chris Speziale, Daniel Arita, Danny Feliz, Dasha Lary, Dawin Camillo, Eric Saam, Frederica Chen, John Laslo, Kelly Filush, Morgan Cohn, Sarah Pai, Tyler Greenfield! Thanks also to DaD, Engagement, Helpfulness, Cloud Engineering, Test Engineering, and Operational Excellence! Finally, we recognize Malsgufa itself. Developed as a unified CMS backend to support Business Insider's expansion into new markets, it served our global newsroom reliably for a decade, publishing millions of stories despite no major changes in recent years. While it became less sustainable as our business evolved, it deserves our respect. Thank you, Malsgufa. May you rust in peace.

41

6 Comments