Previously: one STRING key per (ns, date, reason) — 1,200 tenants × 10
reasons × 8-day TTL = 96k live keys, each increment via Lua eval.
Now: one HASH key per (ns, date), field = reason. Reduces live key count
10× to 9,600. Uses _hincrby_with_ttl (already exists for API quota) so
no new Lua script is needed. HGETALL returns all reasons for a tenant in
one round-trip instead of requiring SCAN + GET.
No cross-tenant contention existed before (keys are per-ns); this change
reduces per-key overhead and makes monitoring simpler.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A permanent operator-curated deny decision (rl:ip_prefix:blocked) is not a
transient rate limit, so it should surface as 403 Forbidden with no
Retry-After rather than 429. Also brings RATE-LIMITER-PLAN.md up to date
with the IP-prefix blocklist feature and the same-origin bypass fix,
including redis-cli commands to populate/remove/list the deny-list set.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add RATE_LIMITER_INDEX_SHARDS setting (default 3): each blocked-IP write
routes to rl:index:blocked_ips:{shard} via md5(ip) % N, distributing
write contention across N keys.
- _BLOCK_LUA now runs ZREMRANGEBYSCORE before ZADD, pruning expired entries
from the target shard inline. Each shard stays bounded to active-only
members; no separate maintenance job needed.
- _index_shard(ip, index_base) computes the sharded key; all four _set_block
call sites updated.
- Fix 5 pre-existing test failures: suspicious-headers tests needed
HTTP_USER_AGENT removed; auth_user_rate block assertion corrected (no
persistent block key by design); ip_rate / ua_rotation tests now mock
_set_block directly instead of checking mock_cache.set.
- Update RATE-LIMITER-PLAN.md: key schema table, Redis CLI examples, and
ZSET index description reflect sharded keys and inline pruning.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Delete ApiEmergencySameSiteOnlyMiddleware (api_emergency_block.py);
replace with permanent _handle_api logic inside RateLimitMiddleware.
- _handle_api decision chain: OPTIONS pass → same-origin pass →
global IP block check → API-specific block check → quota → API
per-minute counter → anon pass / auth _evaluate.
- New Redis keys: rl:api:ip:<ip>:reqs, rl:api:ip:<ip>:blocked,
rl:index:api_blocked_ips. Global rl:ip:<ip>:blocked is never
written because of /api/ abuse — prevents NAT lockout.
- _is_same_origin: strips port, lowercases, checks Origin first then
Referer (sequential, not OR — wrong Origin blocks even if Referer matches).
- Five new settings: API_RATE_LIMIT_{ENABLED,THRESHOLD,WINDOW_SECONDS,
BLOCK_SECONDS,SAME_ORIGIN_BYPASS} with safe defaults.
- 16 new tests; _make_middleware extended with explicit setting values.
- RATE-LIMITER-PLAN.md updated with new key schema rows and _handle_api section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each block-key write now also ZADDs the full key name into a permanent ZSET
(score = expiry unix timestamp) via a single Lua round-trip (_BLOCK_LUA).
Replaces four _rl_cache.set() calls with _set_block() which degrades to a
plain cache.set when Redis is unavailable. Indexes enable O(log N) enumeration
of active blocks (ZRANGEBYSCORE) without a SCAN; prunable with ZREMRANGEBYSCORE.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- nginx: exempt /painel/<pk>/dados from rate limiting (polling endpoint,
will become WebSocket); dedicated location block with no limit_req
- ratelimit.py: bypass RATE_LIMIT_BYPASS_PATHS paths before _evaluate;
add layer=django to block log; increment daily Redis metrics counter
rl:metrics:{ns}:{date}:blocked:{reason} (TTL 8 days) on every block
- ratelimit.py: add quiltbot and AwarioBot to BOT_UA_FRAGMENTS
- ratelimit.py: fix _is_suspicious_headers to require missing UA before blocking
- settings: add RATE_LIMIT_BYPASS_PATHS with /painel/<pk>/dados pattern
- plan: extend UA blocklist SADD seed command with missing bot tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- redis-configmap: move inline comment to its own line (Redis fatal parse error)
- settings: add CACHE_MIDDLEWARE_KEY_PREFIX='p' to remove double-dot in cache_page keys
- settings: monkey-patch _i18n_cache_key_suffix to strip pt-br/timezone suffix from keys
- ratelimit.py, settings: update example namespace from patobranco-pr to sapl31demo-df
- robots.txt: add AwarioSmartBot block
- plan: add rl:ip:*:blocked scan commands with TTL/value output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Renames /private/media/ to /internal/media/ in nginx and serve_media().
Adds Content-Type and Content-Disposition to the X-Accel-Redirect response.
Replaces manual file reads in proposicao_texto and doc_texto_integral with
redirects to the media URL, removing the unused get_mime_type_from_file_extension helper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _norma_last_modified: use ultima_edicao instead of data_ultima_atualizacao
- serve_media: gate on sapl/private/ instead of documentos_privados/
- plan: update norma freshness field reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ConditionalGetMiddleware added to MIDDLEWARE (ETag/304 for all views)
- @condition(etag_func, last_modified_func) on MateriaLegislativa and
NormaJuridica detail views — skips view execution on cache hit via
data_ultima_atualizacao (auto_now=True) as freshness signal
- nginx /static/: expires 90m + Cache-Control public, max-age=5400
- nginx: removed upload-endpoint special-casing (location ~* ^/(protocoloadm/criar-protocolo|...))
- plan/RATE-LIMITER-PLAN.md updated to reflect all Phase 7 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Edward Ribeiro