diff --git a/changelog.d/cloud-armor-edge-refusal.changed.md b/changelog.d/cloud-armor-edge-refusal.changed.md new file mode 100644 index 000000000..61fc297ca --- /dev/null +++ b/changelog.d/cloud-armor-edge-refusal.changed.md @@ -0,0 +1 @@ +Requests to api.policyengine.org without an Authorization header, outside the Public API's public routes, are refused at the load balancer with 403 instead of reaching the Public API, which answers them with 401; the Cloud Armor runbook records the rules. diff --git a/docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml b/docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml new file mode 100644 index 000000000..4afa36373 --- /dev/null +++ b/docs/migration/armor/20261117T105602Z-pol-api-lb-edge-refusal-preview.yaml @@ -0,0 +1,99 @@ +creationTimestamp: '2026-07-21T07:16:39.177-07:00' +description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md +fingerprint: w1MANeduEPA= +id: '200433714284642728' +kind: compute#securityPolicy +labelFingerprint: 42WmSpB8rSM= +name: pol-api-lb +rules: +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/metadata$') + preview: false + priority: 1000 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 30 + intervalSec: 60 +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$') + preview: false + priority: 1100 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 75 + intervalSec: 60 +- action: allow + description: default rule + kind: compute#securityPolicyRule + match: + config: + srcIpRanges: + - '*' + versionedExpr: SRC_IPS_V1 + preview: false + priority: 2147483647 +- action: allow + description: Public route GET/HEAD /health served without a credential (API lockdown + 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/health' && (request.method == 'GET' || request.method + == 'HEAD') + preview: false + priority: 770 +- action: allow + description: Public route GET /v2/openapi.json served without a credential (API + lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/v2/openapi.json' && request.method == 'GET' + preview: false + priority: 771 +- action: allow + description: CORS preflights served without a credential (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.method == 'OPTIONS' && has(request.headers['origin']) && + has(request.headers['access-control-request-method']) + preview: false + priority: 772 +- action: deny(403) + description: Refuse requests without Authorization outside the public routes; API + v1 refuses them too (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: "!has(request.headers['authorization']) || request.headers['authorization'].matches('^[[:blank:]]*$')" + preview: true + priority: 800 +- action: allow + description: Requests carrying a bearer token skip the per-address /metadata and + /calculate limits; API v1 refuses any that fails validation (API lockdown 2.2 + · 5). See docs/migration/lb-cloud-armor-runbook.md + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/(?:metadata|calculate(?:-full)?)$') + && has(request.headers['authorization']) && request.headers['authorization'].matches('^[Bb]earer + [A-Za-z0-9._~+/-]+=*$') + preview: false + priority: 900 +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb +type: CLOUD_ARMOR diff --git a/docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml b/docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml new file mode 100644 index 000000000..5fb775362 --- /dev/null +++ b/docs/migration/armor/20261117T130329Z-pol-api-lb-edge-refusal-enforced.yaml @@ -0,0 +1,99 @@ +creationTimestamp: '2026-07-21T07:16:39.177-07:00' +description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md +fingerprint: JyTEO8FdBD8= +id: '200433714284642728' +kind: compute#securityPolicy +labelFingerprint: 42WmSpB8rSM= +name: pol-api-lb +rules: +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/metadata$') + preview: false + priority: 1000 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 30 + intervalSec: 60 +- action: throttle + description: '' + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$') + preview: false + priority: 1100 + rateLimitOptions: + conformAction: allow + enforceOnKey: IP + exceedAction: deny(429) + rateLimitThreshold: + count: 75 + intervalSec: 60 +- action: allow + description: default rule + kind: compute#securityPolicyRule + match: + config: + srcIpRanges: + - '*' + versionedExpr: SRC_IPS_V1 + preview: false + priority: 2147483647 +- action: allow + description: Public route GET/HEAD /health served without a credential (API lockdown + 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/health' && (request.method == 'GET' || request.method + == 'HEAD') + preview: false + priority: 770 +- action: allow + description: Public route GET /v2/openapi.json served without a credential (API + lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.path == '/v2/openapi.json' && request.method == 'GET' + preview: false + priority: 771 +- action: allow + description: CORS preflights served without a credential (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: request.method == 'OPTIONS' && has(request.headers['origin']) && + has(request.headers['access-control-request-method']) + preview: false + priority: 772 +- action: deny(403) + description: Refuse requests without Authorization outside the public routes; API + v1 refuses them too (API lockdown 5.1 · 4) + kind: compute#securityPolicyRule + match: + expr: + expression: "!has(request.headers['authorization']) || request.headers['authorization'].matches('^[[:blank:]]*$')" + preview: false + priority: 800 +- action: allow + description: Requests carrying a bearer token skip the per-address /metadata and + /calculate limits; API v1 refuses any that fails validation (API lockdown 2.2 + · 5). See docs/migration/lb-cloud-armor-runbook.md + kind: compute#securityPolicyRule + match: + expr: + expression: request.path.matches('^/[a-z]{2}/(?:metadata|calculate(?:-full)?)$') + && has(request.headers['authorization']) && request.headers['authorization'].matches('^[Bb]earer + [A-Za-z0-9._~+/-]+=*$') + preview: false + priority: 900 +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb +type: CLOUD_ARMOR diff --git a/docs/migration/lb-cloud-armor-runbook.md b/docs/migration/lb-cloud-armor-runbook.md index 5e52c3f09..bd91e087f 100644 --- a/docs/migration/lb-cloud-armor-runbook.md +++ b/docs/migration/lb-cloud-armor-runbook.md @@ -11,14 +11,17 @@ record of what is currently deployed is the newest exported snapshot in | | | |---|---| | Policy | `pol-api-lb`, attached to the Cloud Run backend service of the public API load balancer | -| Rules | Per-IP throttles on the metadata and calculate path families; thresholds in the snapshot | -| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **stays in preview at 75/60s** — observed legit/partner clients run 39–88/min there, so enforcing would 429 real use (incl. the partner API fallback) | - -Note on preview validation: Cloud Armor **throttle** rules in preview only ever -log `CONFORM` (never `EXCEEDED`), so the `previewSecurityPolicy` outcome field -cannot be used to count would-be-throttled requests. Judge a throttle rule's -enforcement readiness from **raw per-IP request-rate analysis of LB logs**, not -from preview outcome counts. +| Rules | Per-IP throttles on the metadata and calculate path families (1000, 1100; thresholds in the snapshot); a bearer-token exemption from them (900); a 403 for requests without `Authorization` outside the Public API's public routes (770–800) | +| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **ENFORCED** 2026-11-09 15:01 UTC (it had stayed in preview because observed legit/partner clients ran 39–88/min there, including a partner API fallback that was removed in July 2026); rules 900 and 770–800 enforced | + +Note on preview validation: Cloud Armor **throttle** rules in preview do log +would-be throttling. `jsonPayload.previewSecurityPolicy.outcome` is +`RATE_LIMIT_THRESHOLD_EXCEED` for a request over the threshold: in the 30 days +before rule 1100 was enforced, its preview logged that outcome for 29,037 of +162,011 matching requests. (This note once said preview logs only `CONFORM`.) +Count those outcomes, and still judge a throttle rule's enforcement readiness +from **raw per-IP request-rate analysis of LB logs**, which shows who would be +throttled and by how much. Origin: overnight bot waves saturated backends / churned autoscaling (2026-07-13 → 2026-07-21 incidents; details in the cutover execution plan). @@ -47,7 +50,8 @@ Origin: overnight bot waves saturated backends / churned autoscaling ## Known caveat Cloud Armor only sees traffic that crosses the load balancer. Serverless -default URLs (`*.run.app`) bypass it entirely. This is +default URLs (`*.run.app`) bypass it entirely, including rule 800's refusal of +requests without a credential; the Public API refuses those itself. This is acceptable while ingress must stay open for CI smoke tests; revisit when ingress is locked down to the LB. @@ -68,15 +72,16 @@ client. Those routes apply their own per-browser limits on Vercel. made-up token is not served. Apply rule 900 only while that check is deployed. - **Why the exact pattern:** the Public API treats a blank `Authorization` - header as no credential and serves it, so a rule that only checked the - header's presence would let blank headers skip the throttle and still be - served. Two headers reach Cloud Armor joined by a comma, which the pattern + header as no credential, so a rule that only checked the header's presence + would let blank headers skip the throttle (and, before 2026-11-17, be + served). Two headers reach Cloud Armor joined by a comma, which the pattern rejects. -- **What it gives up:** until the Public API refuses requests without a - server credential, a signed-in user's own valid token in `Authorization` - skips the throttle and is served; requests with made-up tokens are not - capped at the load balancer, though each costs the Public API one - signature check. +- **What it gives up:** requests with made-up tokens are not capped at the + load balancer, though each costs the Public API one signature check. Before + the Public API refused requests without a server credential (2026-11-17), a + signed-in user's own valid token in `Authorization` also skipped the + throttle and was served; since then the Public API refuses it outside its + public routes. Check it with requests of your own: 40 requests a minute to `/us/metadata` with `Authorization: Bearer made-up` should all get 401 and @@ -85,6 +90,64 @@ none 429, and the load balancer log should show `gcloud compute security-policies rules update 900 --preview` (back to log-only) or `rules delete 900`, then export a snapshot. +## Requests without a credential (rules 770–800) + +Since 2026-11-17 the Public API runs with +`PUBLIC_API_SERVER_CREDENTIAL_REQUIRED=1`: it refuses every request without a +server credential with 401 before routing, except its public routes. Since the +same day, rule 800 refuses at the load balancer, with 403, the requests among +those that carry no `Authorization` header or a blank one (only spaces and +tabs). Rules 770–772 allow the public routes first: + +| Priority | Action | Matches | +|---|---|---| +| 770 | allow | `GET` and `HEAD /health` | +| 771 | allow | `GET /v2/openapi.json` | +| 772 | allow | CORS preflights: `OPTIONS` with `Origin` and `Access-Control-Request-Method` | +| 800 | `deny(403)` | `!has(request.headers['authorization']) \|\| request.headers['authorization'].matches('^[[:blank:]]*$')` | + +Rule 800 refuses only requests the Public API would refuse anyway, so they +no longer reach Cloud Run. + +- **What callers see:** Cloud Armor can only deny with 403, 404 or 502, and + its 403 is Google's HTML page. Through `api.policyengine.org`, a request + without a credential gets that page, not the Public API's 401 with + `WWW-Authenticate: Bearer` and a JSON body. Requests sent to the `*.run.app` + URLs skip the load balancer and get the Public API's 401. +- **Tokens:** Cloud Armor can't validate a token, so any non-blank + `Authorization` header passes rule 800 and the Public API checks it. A + made-up token still gets the Public API's 401. +- **Keep the allow rules matched to the Public API.** Rules 770–772 copy + `PUBLIC_ROUTES` and `is_public_request` in + `policyengine_api/request_caller.py`. A route made public there must be + allowed here before rule 800, or the load balancer refuses it. +- **Apply rule 800 only while enforcement is on.** With + `PUBLIC_API_SERVER_CREDENTIAL_REQUIRED` off, the Public API serves requests + without a credential, and rule 800 would refuse them. +- **Preflights and the throttles:** rule 772 matches before rules 1000 and + 1100, so CORS preflights to the metadata and calculate paths no longer count + toward those per-address limits. The Public API's CORS middleware answers a + preflight without running a route. +- **The blank pattern:** Cloud Armor refused `'^[ \t]*$'` with "Invalid + Argument"; the POSIX class `'^[[:blank:]]*$'` (space or tab) is accepted. + +Rule 800 ran in preview for 2 hours before it was enforced, shorter than the +24 hours under "Preview-to-enforcement checks", by decision. It logged 264 +would-be refusals, and the Public API had answered every one of them with 401. +A deny rule in preview logs `DENY` in +`jsonPayload.previewSecurityPolicy.outcome`; group those entries by +`httpRequest.status` to see whether the Public API served any of them. + +Check it with requests of your own, without a credential, to +`api.policyengine.org`: `/`, `/us/metadata`, `POST /us/calculate` and `/v2/me` +get 403 with `text/html`; `GET` and `HEAD /health`, `/v2/openapi.json` and a +CORS preflight get 200; `Authorization: Bearer made-up` gets the Public API's +401. The load balancer log shows `jsonPayload.enforcedSecurityPolicy.priority=800` +with outcome `DENY` for the refused requests. Roll it back with +`gcloud compute security-policies rules update 800 --preview` (back to +log-only, within minutes) or `rules delete 800`, then export a snapshot. Rules +770–772 can stay; deleting 772 puts preflights back under the throttles. + ## Procedures Set once per shell: