Incident: A senior engineer’s promotion packet claims “led the migration that cut p99 latency by 40%” and “drove adoption of the new service mesh across three teams.” The packet is well-written. The committee is split. Nobody can find the migration RFC, the latency dashboard, or the adoption tracker. The packet is the only artifact.
Status: Open. Severity: Medium. Impact: The committee is now evaluating writing quality, not engineering impact. The engineer may be promoted or denied for reasons unrelated to their work. Both outcomes are wrong.
Symptom
The packet reads like a postmortem with the timeline removed. Claims are stated as facts. Numbers appear without units or baselines. Verbs are transitive but the objects are vague: “drove,” “led,” “owned,” “partnered with.” There is no artifact trail — no RFC link, no PR series, no dashboard snapshot, no incident review, no design doc with the engineer’s name in the revision history.
This is not a lying problem. It is a missing-artifact problem. The engineer did work. The work left traces. The traces are not in the packet. The committee is asked to reconstruct impact from prose.
Timeline
Promotion packets are usually assembled under deadline. The manager asks for a draft. The engineer writes it from memory, often late at night, often after a sprint. The manager edits for tone. The committee reads it cold, without the engineer in the room. The packet becomes the primary source.
If the packet cites artifacts, the committee can verify. If it does not, the committee either trusts the prose or discounts it. Neither is a system for evaluating engineering work. It is a system for evaluating packet-writing.
Contributing factors
1. The artifact trail is not part of the promotion process
Most engineering organizations have a career ladder and a promotion committee. Few have a requirement that impact claims link to artifacts. The ladder describes levels. The packet describes the engineer. The artifacts sit in git, CI, dashboards, and chat, unlinked.
This is a process gap, not a personal failing. The engineer is asked to prove impact in a format that does not require proof.
2. Impact is measured in outcomes, but artifacts record activity
Git history records commits. CI logs record builds. Incident channels record pages. None of these directly record “cut p99 latency by 40%.” That number lives in a dashboard, a query, or a before/after measurement. If the measurement was never saved, the claim is unverifiable.
This is the core mismatch: promotion packets claim outcomes, but the operational evidence teams generate records activity. The bridge between them — a saved measurement, a linked dashboard, a written before/after — is usually missing.
3. The committee has no shared standard for evidence
One committee member wants a design doc. Another wants a dashboard. Another wants peer feedback. Without a defined evidence standard, the committee defaults to prose quality and familiarity. This is how promotion becomes a writing contest with a social graph.
4. Retry storm: the same claims get re-litigated
When a packet lacks artifacts, the committee revisits the same claims in every calibration session. The claims do not resolve because there is nothing to resolve them against. This is a retry storm: the same request, the same failure, repeated until someone gives up or the deadline passes.
5. Normalization of deviance: everyone knows packets are unverifiable
The organization has learned that packets are mostly prose. The committee has learned to read between the lines. The manager has learned to write confidently. The engineer has learned to claim broadly. This is normalization of deviance: a known-bad practice that persists because it has not yet caused a visible failure. The visible failure is a bad promotion decision. It happens. It is attributed to the individual, not the process.
What the artifacts actually show
Pull the artifacts the team already generates. They tell a different story than the packet.
- CODEOWNERS: Who is actually required to review changes in the affected area? If the engineer is not an owner, “led the migration” needs a different artifact.
- Git history: Commit volume is not impact. But a series of commits across a migration window, with review approvals from other teams, is a trace. So is the absence of commits during the claimed period.
- CI retry logs: If the migration introduced flaky tests that were retried for weeks, the packet’s “cut latency” claim may be true while the operational cost is omitted.
- RFC threads: Who wrote the RFC? Who approved it? Who raised the blocking concern? The thread records the decision and the dissent.
- Incident channels: Did the migration cause an incident? Was the engineer on call? Did they write the postmortem? The channel records the response, not the narrative.
- On-call swap logs: If the engineer swapped out of on-call during the migration, that is a fact. It may be justified. It is still a fact.
- Exit interview notes: If the migration drove someone out, the notes may say so. This is uncomfortable and relevant.
- Career ladder docs: What does the ladder actually require at the next level? If it requires “impact across teams,” the packet needs cross-team artifacts, not a list of meetings.
- PIP timelines: If a teammate was on a PIP during the migration, the packet’s “led the team” claim needs context.
- Onboarding docs: If the migration required new onboarding steps, the doc is an artifact. If it did not, the migration may not have changed the system’s surface area.
None of these artifacts prove impact by themselves. Together, they form a trail. The trail either supports the packet’s claims, contradicts them, or is silent. Silence is a finding.
Why this gets misread as a personal failing
When the committee cannot verify a claim, it does one of two things. It trusts the engineer, or it discounts the engineer. Both are personal judgments. Neither is a system judgment.
The engineer who writes well and has a good reputation gets trusted. The engineer who writes plainly and has less visibility gets discounted. The difference is not impact. The difference is artifact trail and narrative skill.
This is the system failure: the promotion process rewards packet-writing and reputation, then attributes the outcome to engineering ability. The engineer who is denied is told to “build more visibility.” The engineer who is promoted is told they “demonstrated impact.” Neither feedback addresses the missing artifact trail.
Action items
For the committee: require artifact links for every impact claim
A claim without an artifact is a hypothesis. The committee can still consider it, but it should be labeled as unverified. This is not a punishment. It is a calibration. The committee’s job is to evaluate evidence, not prose.
Define the acceptable artifact types. A dashboard link with a saved query is an artifact. A PR series with review approvals is an artifact. A design doc with revision history is an artifact. A Slack thread is weaker but still an artifact. A verbal claim is not.
For managers: build the artifact trail before the packet deadline
The packet is a summary, not a discovery process. If the manager is writing the packet from scratch at deadline, the artifacts are already lost. Keep a running document per engineer: links to RFCs, dashboards, incident reviews, and peer feedback. This is not surveillance. It is the same evidence the committee needs.
For engineers: save the measurement, not just the outcome
When you run an experiment, save the before and after. When you ship a migration, link the dashboard. When you write an RFC, keep the thread. When you respond to an incident, write the postmortem. These are the artifacts that make impact legible.
This is not self-promotion. It is record-keeping. The organization already generates the evidence. The engineer’s job is to not lose it.
For the organization: define the evidence standard once
Publish the artifact types the committee accepts. Publish the format for linking them. Publish the rule that unverified claims are labeled. This removes the ambiguity that turns promotion into a writing contest.
HM Treasury’s Magenta Book provides a useful external reference for evaluation design: it emphasizes that evaluation should be built into an intervention’s design and delivery from the earliest stages, and that results need to be interpreted and presented with care. The same principle applies here. If impact evaluation is bolted on at packet time, the evidence is weak. If it is built into the work, the evidence is already there.
One reversible experiment
Pick one upcoming promotion packet. Before the committee reads it, ask the engineer and manager to add artifact links to every impact claim. If a claim has no artifact, mark it “unverified.” Run the committee session as usual. Compare the discussion to the previous cycle.
Rollback plan: If the artifact requirement slows the process without improving decisions, revert to the previous format for the next cycle. The experiment is reversible because it changes the packet format, not the promotion criteria.
What to measure: Did the committee spend less time debating unverifiable claims? Did the packet’s claims survive contact with the artifacts? Did any claim get withdrawn or narrowed? These are signals, not proof. But they are more useful than another cycle of prose review.
FAQ
Doesn’t requiring artifacts disadvantage engineers who work on invisible infrastructure?
It disadvantages engineers whose work leaves no artifacts. Infrastructure work leaves artifacts: dashboards, runbooks, incident reviews, migration RFCs, on-call logs. The problem is not invisibility. The problem is that the artifacts are not linked to the packet. If the work truly leaves no trace, that is a finding about the work, not the engineer.
Isn’t this just bureaucracy?
It is record-keeping. The organization already generates the artifacts. The packet currently ignores them. Linking them is less work than re-litigating the same claims in every calibration session.
What if the engineer did the work but the artifacts are owned by someone else?
Then the packet should say so and link the artifact. Ownership of the artifact is not the same as contribution to the outcome. The committee can evaluate contribution from the artifact trail: review approvals, RFC comments, incident response, commit history. The artifact does not need to be authored by the engineer. It needs to be linked.
How does this relate to the career ladder?
The ladder defines the expected level of impact. The artifacts define whether the impact occurred. If the ladder says “impact across teams,” the packet needs cross-team artifacts. If the ladder says “technical leadership,” the packet needs design docs, RFCs, or incident reviews. The ladder is the standard. The artifacts are the evidence.
What about qualitative impact, like mentoring or culture?
Qualitative impact still leaves artifacts: onboarding docs, peer feedback, exit interview notes, promotion packets of people the engineer mentored. These are weaker than dashboards but stronger than prose. The committee can weigh them accordingly.
Closing note
The packet is not the problem. The missing artifact trail is the problem. The packet is a symptom. The system failure is that promotion decisions are made on prose when the organization already generates evidence. Fix the trail, and the packet becomes a summary instead of a mystery.
