[tamird]

Ordinary query preprocessing hashes each rule definition, then drops every attribute except tags and visibility. When neither survives, Vec::retain leaves the empty vector holding its allocation until graph hashing releases the rule. This change drops that allocation immediately. The new test checks both the definition digest and the empty vector’s capacity.

I replayed a synthetic ordinary query containing 160,000 rules. Every rule has three discardable attributes, and 16,000 also have tags. Both optimized binaries ran on macOS arm64 with eight Rayon threads. After one warm-up per binary, I ran four pairs, with each binary going first twice. The complete hash and edge files matched in all ten runs.

Median over four measured runs per binary	Upstream	This change
Peak RSS from wait4	623.4 MiB	452.5 MiB
Elapsed time	1.009 s	1.017 s
User + system CPU	1.156 s	1.075 s

Peak RSS fell in all four pairs. The 170.9 MiB difference between medians exceeds the larger within-binary range of 27.3 MiB. Elapsed time varied by up to 0.523 s within a binary, so this measurement supports a memory improvement.

Reproduce the input and comparison

Build //:bazel-diff with bazel build -c opt //:bazel-diff at each revision. Save bazel-bin/src/bazel-diff as stock for the parent and candidate for the PR in a temporary directory. The query replay uses the public fixture encoder and Bazel stand-in from the earlier ordinary-query benchmark. Parent revision · Fixture helper

From that directory, download the helper, save the Python below as reproduce.py, and run it with Python 3.11 or newer.

curl -fsSLo perf_workload.py https://raw.githubusercontent.com/Tinder/bazel-diff/bb27eef6718addbc097d15d6fae8f75fec26ada6/tools/perf_workload.py
python3 reproduce.py ./stock ./candidate

"""Run from a temporary directory alongside the pinned public perf_workload.py."""
import hashlib
import json
import os
import statistics
import sys
import time
from pathlib import Path

from perf_workload import (ATTRIBUTE_STRING, ATTRIBUTE_STRING_LIST, encode_attribute,
                           encode_rule, encode_target, encode_varint, write_replay_bazel)


root = Path(__file__).resolve().parent
wire = root / "targets.pb"
ordinary = tuple(encode_attribute(n, ATTRIBUTE_STRING, string_value=v) for n, v in (
    ("generator_name", "generator"), ("mode", "release"), ("$internal_attr_hash", "hash")
))
tags = encode_attribute("tags", ATTRIBUTE_STRING_LIST, string_list_value=("public",))
with wire.open("wb") as f:
    for i in range(160_000):
        rule = encode_rule(
            f"//bench:t{i:06d}", "bench_rule", location=f"bench/BUILD:{i+1}:1",
            attributes=(*ordinary, tags) if i % 10 == 0 else ordinary,
            rule_inputs=("//bench:t000000",) if i else (), rule_outputs=(),
            starlark_environment_hash="", instantiation_stack=(),
        )
        msg = encode_target(1, rule)
        f.write(encode_varint(len(msg)))
        f.write(msg)
shim = write_replay_bazel(root / "bazel-replay", wire, root)


def digest(path):
    with path.open("rb") as f:
        return hashlib.file_digest(f, "sha256").hexdigest()


before, after = (str(Path(x).resolve()) for x in sys.argv[1:])
records = []
oracle = None
for pair, order in enumerate(((before, after), (before, after), (after, before), (after, before), (before, after))):
    for binary in order:
        hashes, edges = root / "hashes.json", root / "edges.json"
        args = [binary, "generate-hashes", f"--workspacePath={root}", f"--bazelPath={shim}",
                "--excludeExternalTargets", "--includeTargetType", f"--depEdgesFile={edges}", str(hashes)]
        start = time.monotonic()
        pid = os.posix_spawn(binary, args, os.environ | {"RAYON_NUM_THREADS": "8"})
        _, status, usage = os.wait4(pid, 0)
        elapsed = time.monotonic() - start
        if os.waitstatus_to_exitcode(status):
            raise RuntimeError((binary, status))
        outputs = [(path.stat().st_size, digest(path)) for path in (hashes, edges)]
        if oracle is None:
            oracle = outputs
        assert oracle == outputs
        peak = usage.ru_maxrss / (1024**2 if sys.platform == "darwin" else 1024)
        records.append(dict(pair=pair, arm="stock" if binary == before else "candidate",
                            wall=elapsed, cpu=usage.ru_utime+usage.ru_stime, peakMiB=peak,
                            outputs=outputs))
for arm in ("stock", "candidate"):
    measured = [r for r in records if r["pair"] and r["arm"] == arm]
    print(arm, {k:round(statistics.median(r[k] for r in measured), 3) for k in ("wall","cpu","peakMiB")})
print("outputs", oracle)
(root / "results.json").write_text(json.dumps(dict(wire=(wire.stat().st_size,digest(wire)),rows=records),indent=2)+"\n")

The generated query is 26,448,878 bytes and has SHA-256 4e77fb01093e9bdfd2a0b9d73157dc5bbf3b1fe20db3a908640f3164489f28a4. The hash file is 24,800,001 bytes with SHA-256 92e15eff56d4cb4c2d7839575242fbac6b870aa3dbbb86f723903410f4dfeb79. The edge file is 6,079,984 bytes with SHA-256 c2724f4c722097492251d4917eb8ac1a075571bcb642ee8bc783e31ab793bc0a.

Codex for Tamir.