From 146344e283e07f99ebd9cece6cb3ea76f247310f Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:16:04 +0400 Subject: [PATCH 1/2] Record the Cloud Armor rules that refuse requests without a credential Adds the exported snapshots of pol-api-lb from 2026-11-17: after rules 770-772 and log-only rule 800 were added, and after rule 800 was enforced. Rule 800 answers 403 at the load balancer to requests with no Authorization header, or a blank one, outside the Public API's public routes, which rules 770-772 allow first. These are requests the Public API refuses with 401 since PUBLIC_API_SERVER_CREDENTIAL_REQUIRED=1. The runbook gets a section on these rules: what callers through api.policyengine.org now see (Google's 403 page, not the Public API's 401), keeping rules 770-772 matched to the Public API's public routes, using rule 800 only while enforcement is on, preflights leaving the per-address limits, the accepted blank-header pattern, the 2-hour preview, and how to check and roll it back. It also corrects the summary table, where the calculate throttle (rule 1100) has been enforced since the 2026-11-10 snapshot, and updates the rule 900 notes that described the Public API serving requests without a credential. Co-Authored-By: Claude Opus 5.5 --- .../cloud-armor-edge-refusal.changed.md | 1 + ...5602Z-pol-api-lb-edge-refusal-preview.yaml | 99 +++++++++++++++++++ ...329Z-pol-api-lb-edge-refusal-enforced.yaml | 99 +++++++++++++++++++ docs/migration/lb-cloud-armor-runbook.md | 82 ++++++++++++--- 4 files changed, 270 insertions(+), 11 deletions(-) create mode 100644 changelog.d/cloud-armor-edge-refusal.changed.md create mode 100644 docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml create mode 100644 docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml diff --git a/changelog.d/cloud-armor-edge-refusal.changed.md b/changelog.d/cloud-armor-edge-refusal.changed.md new file mode 100644 index 000000000..61fc297ca --- /dev/null +++ b/changelog.d/cloud-armor-edge-refusal.changed.md @@ -0,0 +1 @@ +Requests to api.policyengine.org without an Authorization header, outside the Public API's public routes, are refused at the load balancer with 403 instead of reaching the Public API, which answers them with 401; the Cloud Armor runbook records the rules. diff --git a/docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml b/docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml new file mode 100644 index 000000000..4afa36373 --- /dev/null +++ b/docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml @@ -0,0 +1,99 @@ +creationTimestamp: '2026-07-21T07:16:39.177-07:00' +description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md +fingerprint: w1MANeduEPA= +id: '200433714284642728' +kind: compute#securityPolicy +labelFingerprint: 42WmSpB8rSM= +name: pol-api-lb +rules: +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/metadata$') + preview: false + priority: 1000 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 30 + intervalSec: 60 +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$') + preview: false + priority: 1100 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 75 + intervalSec: 60 +- action: allow + description: default rule + kind: compute#securityPolicyRule + match: + config: + srcIpRanges: + - '*' + versionedExpr: SRC_IPS_V1 + preview: false + priority: 2147483647 +- action: allow + description: Public route GET/HEAD /health served without a credential (API lockdown + 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/health' && (request.method == 'GET' || request.method + == 'HEAD') + preview: false + priority: 770 +- action: allow + description: Public route GET /v2/openapi.json served without a credential (API + lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/v2/openapi.json' && request.method == 'GET' + preview: false + priority: 771 +- action: allow + description: CORS preflights served without a credential (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.method == 'OPTIONS' && has(request.headers['origin']) && + has(request.headers['access-control-request-method']) + preview: false + priority: 772 +- action: deny(403) + description: Refuse requests without Authorization outside the public routes; API + v1 refuses them too (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: "!has(request.headers['authorization']) || request.headers['authorization'].matches('^[[:blank:]]*$')" + preview: true + priority: 800 +- action: allow + description: Requests carrying a bearer token skip the per-address /metadata and + /calculate limits; API v1 refuses any that fails validation (API lockdown 2.2 + · 5). See docs/migration/lb-cloud-armor-runbook.md + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/(?:metadata|calculate(?:-full)?)$') + && has(request.headers['authorization']) && request.headers['authorization'].matches('^[Bb]earer + [A-Za-z0-9._~+/-]+=*$') + preview: false + priority: 900 +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb +type: CLOUD_ARMOR diff --git a/docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml b/docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml new file mode 100644 index 000000000..5fb775362 --- /dev/null +++ b/docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml @@ -0,0 +1,99 @@ +creationTimestamp: '2026-07-21T07:16:39.177-07:00' +description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md +fingerprint: JyTEO8FdBD8= +id: '200433714284642728' +kind: compute#securityPolicy +labelFingerprint: 42WmSpB8rSM= +name: pol-api-lb +rules: +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/metadata$') + preview: false + priority: 1000 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 30 + intervalSec: 60 +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$') + preview: false + priority: 1100 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 75 + intervalSec: 60 +- action: allow + description: default rule + kind: compute#securityPolicyRule + match: + config: + srcIpRanges: + - '*' + versionedExpr: SRC_IPS_V1 + preview: false + priority: 2147483647 +- action: allow + description: Public route GET/HEAD /health served without a credential (API lockdown + 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/health' && (request.method == 'GET' || request.method + == 'HEAD') + preview: false + priority: 770 +- action: allow + description: Public route GET /v2/openapi.json served without a credential (API + lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/v2/openapi.json' && request.method == 'GET' + preview: false + priority: 771 +- action: allow + description: CORS preflights served without a credential (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.method == 'OPTIONS' && has(request.headers['origin']) && + has(request.headers['access-control-request-method']) + preview: false + priority: 772 +- action: deny(403) + description: Refuse requests without Authorization outside the public routes; API + v1 refuses them too (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: "!has(request.headers['authorization']) || request.headers['authorization'].matches('^[[:blank:]]*$')" + preview: false + priority: 800 +- action: allow + description: Requests carrying a bearer token skip the per-address /metadata and + /calculate limits; API v1 refuses any that fails validation (API lockdown 2.2 + · 5). See docs/migration/lb-cloud-armor-runbook.md + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/(?:metadata|calculate(?:-full)?)$') + && has(request.headers['authorization']) && request.headers['authorization'].matches('^[Bb]earer + [A-Za-z0-9._~+/-]+=*$') + preview: false + priority: 900 +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb +type: CLOUD_ARMOR diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index 5e52c3f09..ec01ea423 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -11,8 +11,8 @@ record of what is currently deployed is the newest exported snapshot in | | | |---|---| | Policy | `pol-api-lb`, attached to the Cloud Run backend service of the public API load balancer | -| Rules | Per-IP throttles on the metadata and calculate path families; thresholds in the snapshot | -| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **stays in preview at 75/60s** — observed legit/partner clients run 39–88/min there, so enforcing would 429 real use (incl. the partner API fallback) | +| Rules | Per-IP throttles on the metadata and calculate path families (1000, 1100; thresholds in the snapshot); a bearer-token exemption from them (900); a 403 for requests without `Authorization` outside the Public API's public routes (770–800) | +| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **ENFORCED** by the 2026-11-10 snapshot (it was still in preview on 2026-08-30, when observed legit/partner clients ran 39–88/min there); rules 900 and 770–800 enforced | Note on preview validation: Cloud Armor **throttle** rules in preview only ever log `CONFORM` (never `EXCEEDED`), so the `previewSecurityPolicy` outcome field @@ -47,7 +47,8 @@ Origin: overnight bot waves saturated backends / churned autoscaling ## Known caveat Cloud Armor only sees traffic that crosses the load balancer. Serverless -default URLs (`*.run.app`) bypass it entirely. This is +default URLs (`*.run.app`) bypass it entirely, including rule 800's refusal of +requests without a credential; the Public API refuses those itself. This is acceptable while ingress must stay open for CI smoke tests; revisit when ingress is locked down to the LB. @@ -68,15 +69,16 @@ client. Those routes apply their own per-browser limits on Vercel. made-up token is not served. Apply rule 900 only while that check is deployed. - **Why the exact pattern:** the Public API treats a blank `Authorization` - header as no credential and serves it, so a rule that only checked the - header's presence would let blank headers skip the throttle and still be - served. Two headers reach Cloud Armor joined by a comma, which the pattern + header as no credential, so a rule that only checked the header's presence + would let blank headers skip the throttle (and, before 2026-11-17, be + served). Two headers reach Cloud Armor joined by a comma, which the pattern rejects. -- **What it gives up:** until the Public API refuses requests without a - server credential, a signed-in user's own valid token in `Authorization` - skips the throttle and is served; requests with made-up tokens are not - capped at the load balancer, though each costs the Public API one - signature check. +- **What it gives up:** requests with made-up tokens are not capped at the + load balancer, though each costs the Public API one signature check. Before + the Public API refused requests without a server credential (2026-11-17), a + signed-in user's own valid token in `Authorization` also skipped the + throttle and was served; since then the Public API refuses it outside its + public routes. Check it with requests of your own: 40 requests a minute to `/us/metadata` with `Authorization: Bearer made-up` should all get 401 and @@ -85,6 +87,64 @@ none 429, and the load balancer log should show `gcloud compute security-policies rules update 900 --preview` (back to log-only) or `rules delete 900`, then export a snapshot. +## Requests without a credential (rules 770–800) + +Since 2026-11-17 the Public API runs with +`PUBLIC_API_SERVER_CREDENTIAL_REQUIRED=1`: it refuses every request without a +server credential with 401 before routing, except its public routes. Since the +same day, rule 800 refuses at the load balancer, with 403, the requests among +those that carry no `Authorization` header or a blank one (only spaces and +tabs). Rules 770–772 allow the public routes first: + +| Priority | Action | Matches | +|---|---|---| +| 770 | allow | `GET` and `HEAD /health` | +| 771 | allow | `GET /v2/openapi.json` | +| 772 | allow | CORS preflights: `OPTIONS` with `Origin` and `Access-Control-Request-Method` | +| 800 | `deny(403)` | `!has(request.headers['authorization']) \|\| request.headers['authorization'].matches('^[[:blank:]]*$')` | + +Rule 800 refuses only requests the Public API would refuse anyway, so they +no longer reach Cloud Run. + +- **What callers see:** Cloud Armor can only deny with 403, 404 or 502, and + its 403 is Google's HTML page. Through `api.policyengine.org`, a request + without a credential gets that page, not the Public API's 401 with + `WWW-Authenticate: Bearer` and a JSON body. Requests sent to the `*.run.app` + URLs skip the load balancer and get the Public API's 401. +- **Tokens:** Cloud Armor can't validate a token, so any non-blank + `Authorization` header passes rule 800 and the Public API checks it. A + made-up token still gets the Public API's 401. +- **Keep the allow rules matched to the Public API.** Rules 770–772 copy + `PUBLIC_ROUTES` and `is_public_request` in + `policyengine_api/request_caller.py`. A route made public there must be + allowed here before rule 800, or the load balancer refuses it. +- **Apply rule 800 only while enforcement is on.** With + `PUBLIC_API_SERVER_CREDENTIAL_REQUIRED` off, the Public API serves requests + without a credential, and rule 800 would refuse them. +- **Preflights and the throttles:** rule 772 matches before rules 1000 and + 1100, so CORS preflights to the metadata and calculate paths no longer count + toward those per-address limits. The Public API's CORS middleware answers a + preflight without running a route. +- **The blank pattern:** Cloud Armor refused `'^[ \t]*$'` with "Invalid + Argument"; the POSIX class `'^[[:blank:]]*$'` (space or tab) is accepted. + +Rule 800 ran in preview for 2 hours before it was enforced, shorter than the +24 hours under "Preview-to-enforcement checks", by decision. It logged 264 +would-be refusals, and the Public API had answered every one of them with 401. +Unlike throttle rules, a deny rule in preview logs `DENY` in +`jsonPayload.previewSecurityPolicy.outcome`, so preview hits can be counted +directly. + +Check it with requests of your own, without a credential, to +`api.policyengine.org`: `/`, `/us/metadata`, `POST /us/calculate` and `/v2/me` +get 403 with `text/html`; `GET` and `HEAD /health`, `/v2/openapi.json` and a +CORS preflight get 200; `Authorization: Bearer made-up` gets the Public API's +401. The load balancer log shows `jsonPayload.enforcedSecurityPolicy.priority=800` +with outcome `DENY` for the refused requests. Roll it back with +`gcloud compute security-policies rules update 800 --preview` (back to +log-only, within minutes) or `rules delete 800`, then export a snapshot. Rules +770–772 can stay; deleting 772 puts preflights back under the throttles. + ## Procedures Set once per shell: From 55fa5e8367819ab855bb1c37e378c2812210dab9 Mon Sep 17 00:00:00 2001 From: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:19:11 +0400 Subject: [PATCH 2/2] Record that the calculate throttle is enforced and preview logs exceeds The runbook's summary table said rule 1100 (the calculate throttle) stays in preview; it was enforced on 2026-11-09 at 15:01 UTC, after the partner API fallback that kept it in preview was removed in July 2026. The note on preview validation said throttle rules in preview only log CONFORM; rule 1100's preview logged RATE_LIMIT_THRESHOLD_EXCEED for 29,037 of 162,011 matching requests in the 30 days before it was enforced. Co-Authored-By: Claude Opus 5.5 --- docs/migration/lb-cloud-armor-runbook.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index ec01ea423..bd91e087f 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -12,13 +12,16 @@ record of what is currently deployed is the newest exported snapshot in |---|---| | Policy | `pol-api-lb`, attached to the Cloud Run backend service of the public API load balancer | | Rules | Per-IP throttles on the metadata and calculate path families (1000, 1100; thresholds in the snapshot); a bearer-token exemption from them (900); a 403 for requests without `Authorization` outside the Public API's public routes (770–800) | -| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **ENFORCED** by the 2026-11-10 snapshot (it was still in preview on 2026-08-30, when observed legit/partner clients ran 39–88/min there); rules 900 and 770–800 enforced | +| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **ENFORCED** 2026-11-09 15:01 UTC (it had stayed in preview because observed legit/partner clients ran 39–88/min there, including a partner API fallback that was removed in July 2026); rules 900 and 770–800 enforced | -Note on preview validation: Cloud Armor **throttle** rules in preview only ever -log `CONFORM` (never `EXCEEDED`), so the `previewSecurityPolicy` outcome field -cannot be used to count would-be-throttled requests. Judge a throttle rule's -enforcement readiness from **raw per-IP request-rate analysis of LB logs**, not -from preview outcome counts. +Note on preview validation: Cloud Armor **throttle** rules in preview do log +would-be throttling. `jsonPayload.previewSecurityPolicy.outcome` is +`RATE_LIMIT_THRESHOLD_EXCEED` for a request over the threshold: in the 30 days +before rule 1100 was enforced, its preview logged that outcome for 29,037 of +162,011 matching requests. (This note once said preview logs only `CONFORM`.) +Count those outcomes, and still judge a throttle rule's enforcement readiness +from **raw per-IP request-rate analysis of LB logs**, which shows who would be +throttled and by how much. Origin: overnight bot waves saturated backends / churned autoscaling (2026-07-13 → 2026-07-21 incidents; details in the cutover execution plan). @@ -131,9 +134,9 @@ no longer reach Cloud Run. Rule 800 ran in preview for 2 hours before it was enforced, shorter than the 24 hours under "Preview-to-enforcement checks", by decision. It logged 264 would-be refusals, and the Public API had answered every one of them with 401. -Unlike throttle rules, a deny rule in preview logs `DENY` in -`jsonPayload.previewSecurityPolicy.outcome`, so preview hits can be counted -directly. +A deny rule in preview logs `DENY` in +`jsonPayload.previewSecurityPolicy.outcome`; group those entries by +`httpRequest.status` to see whether the Public API served any of them. Check it with requests of your own, without a credential, to `api.policyengine.org`: `/`, `/us/metadata`, `POST /us/calculate` and `/v2/me`