← CatZep/Engineering Guides/Management & Leadership/Cross-functional Partnerships

Collaborating During Major Incidents

Running a cross-team incident with a single commander, comms discipline, and follow-through that outlives the SEV.

Student edition · understand, apply, verify

Coordinate incident response around safe mitigation, role clarity, communication, and sustained learning.

What you should be able to do: Restore safely while preserving evidence and keeping uncertain cause/impact visible.

Context-dependent collaboration guidance. Examples/numbers are illustrative; first-person practice drafts are not verified personal history. Respect decision rights, safety/legal obligations, privacy, and the partners' actual capacity. No framework guarantees agreement or delivery.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

This question tests how you lead when the system is on fire and the fix spans org boundaries. A major incident that touches many teams fails not for lack of talent but for lack of coordination — five people debugging in parallel, no one deciding, leadership asking for status in the same channel engineers are trying to work in. The interviewer wants to see that you know the incident-command model, that you prioritize mitigation over root-cause in the moment, that you keep communication disciplined, and that you drive a blameless postmortem whose action items actually get done across every org involved. The answer follows the CARL shape.

What this question is really testing

Can you impose order on a multi-team crisis — establish command, restore service fast, communicate cleanly — and then convert the incident into durable fixes that span organizations, without hunting for someone to blame?

How to answer

• Establish a single incident commander. One person owns coordination and decisions — not the debugging, the direction. In a cross-team SEV the biggest failure mode is diffuse ownership; naming one commander (who may not be the most senior person) is the first move.
• Assign clear roles. Separate the commander from the operations lead doing the hands-on work, the communications lead handling stakeholders, and a scribe keeping the timeline. Roles prevent the two classic failures: everyone doing the same thing, or a critical thing no one owns.
• Enforce comms discipline. One incident channel, a regular status cadence, and a communications lead who shields the responders from the leadership firehose. Stakeholders get updates on a clock; engineers get to focus.
• Mitigate before root-cause. The first job is to stop the bleeding — roll back, fail over, shed load, flip the flag — and understand why afterward. Restore the user experience first; diagnose second.
• Run a blameless postmortem and drive action items to zero. Grade the system, not the person; find the real root cause; and — the part most orgs drop — track every action item to completion across all the teams involved, because the incident isn’t closed until the class can’t recur.

What the interviewer is looking for

• Fluency with the incident-command model — single commander, distinct roles.
• Mitigation instinct — restore service before chasing the root cause.
• Communication treated as a first-class workstream, shielding responders.
• A blameless culture — the reflex to fix the system, not assign fault.
• Relentless follow-through on cross-org action items, not just a nice writeup.

A worked example (CARL)

Practice draft: first-person narratives below are not independently verified autobiographical evidence. Use their structure, not their claimed results; replace with your genuine safe-to-share experience.

Context. A schema change in an upstream producer combined with a config push in our ads events storage tier caused write failures that cascaded into two downstream teams — measurement dashboards went stale and a billing pipeline started dropping events. Three teams were paged at once. Within ten minutes there were four separate debugging threads, a director asking for status in the main channel, and no single person actually deciding what to do. I was the most senior engineer online across the affected teams.

Actions. My first move was not to debug — it was to impose structure. I declared a single SEV, took the incident commander role explicitly, and said so in the channel: “I’m IC, all coordination here.” Then I assigned roles rather than letting people self-organize into chaos: I named an operations lead on my storage team to drive the hands-on mitigation, an operations lead on the producer team since the trigger was theirs, a communications lead to own stakeholder updates and get the director out of the working channel and onto a status cadence, and asked one person to scribe the timeline. I made the priority explicit and unpopular: we would mitigate before we understood. Several engineers wanted to root-cause the schema interaction first; I ruled that we’d roll back the config push and the schema change immediately to restore writes, and diagnose the interaction afterward — stop the bleeding first. The comms lead posted status every 15 minutes on a clock so stakeholders stopped interrupting responders. Once writes recovered, I kept the SEV open until we’d confirmed the downstream billing pipeline had caught up with no permanent event loss, because “our part is fixed” isn’t “the incident is resolved” when it spans teams. Afterward I ran the postmortem blameless — the root cause was that the producer’s schema change and our config push were individually safe but unsafe together, with no gate that could see the combination. The action items spanned three orgs, which is exactly where follow-through usually dies, so I created a shared tracker, assigned each item a single owner with a date, and reviewed it in the weekly ops meeting until every item was closed.

Results. We cut time-to-mitigation to about 25 minutes by rolling back before diagnosing, and confirmed zero permanent event loss in billing. The blameless postmortem produced a cross-team compatibility check that catches unsafe producer-plus-storage change combinations in CI, and every one of the cross-org action items was closed within the quarter — the same class of incident hasn’t recurred since.

Learnings. In a multi-team incident, coordination is the scarce resource, not engineering skill — naming one commander and clear roles is worth more than another debugger. Mitigate-before-root-cause is a discipline you have to enforce against smart people’s instinct to understand first. And a cross-org postmortem is only real if someone owns driving the action items to zero after the adrenaline fades.

Common follow-ups

Evidence to collect with your approved tools

• A decision/review record with the problem, constraints, options, and your contribution.
• Relevant tests, dashboards, or measurements with definitions, denominators, dates, and uncertainty.
• Incident or delivery records showing actual outcomes and unresolved limitations.
• Stakeholder agreements and follow-up actions with owners and review dates.
• Only authorized, minimized, safe-to-share evidence. Do not copy confidential personnel notes into a public story.

---

Worked example: three teams debug the same outage

Use the following scenario to practice making goals, uncertainty, responsibility, and evidence explicit.

Step	Evidence, decision, and expected result
Assign coordination	Name an incident commander and useful operations/comms/scribe roles under the response policy.
Identify safe mitigations	Compare rollback, reduced load, failover, and access safeguards; do not blindly change state just to act fast.
Communicate evidence and uncertainty	Record observed impact, actions, results, next update, and actual user recovery; avoid declaring root cause prematurely.
Track follow-through	Review contributing factors and prioritize owned improvements. Action closure reduces known risks but does not prove the failure class can never recur.

Check your understanding: Should root-cause certainty always precede mitigation?

No. Choose justified safe mitigation from available evidence while preserving investigation and recovery correctness.

Practice with your own situation

Choose a real cross-team decision. Record the shared problem, each party's constraints, options, assumptions, authority, agreed actions, owner/date, observed outcome, and reconsideration trigger. Use authorized evidence and distinguish contribution from attribution.

Use genuine, safe-to-share evidence. Example numbers here illustrate reasoning; they are not claims about your work or a guaranteed outcome.

Sources and further verification

• Google SRE Workbook: alerting on SLOs
• Google SRE books