SPDT/sapl - sapl - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Edward Ribeiro	a40d1dfdf4	Update RATE-LIMITER-PLAN.md with 2026-05-11 changes - Decision log: add 5 entries (_handle_api, auth parity, threshold 60→35, quota recalibration, key namespace) - Key schema: remove auth quota rows; update API key patterns to ns-scoped form; rename index key; update thresholds (35/min, 1 000/day · 7 000/week) - _handle_api section: update decision chain (no step 7 for auth; ns-scoped block key); update settings table; update files changed - Enforcement stack table: collapse /api/ anon/auth rows into one - NAT constraint mitigations box: reflect unified /api/ treatment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	3 weeks ago
Edward Ribeiro	a33fcc2fe6	Add API-specific rate limiter and remove emergency block middleware - Delete ApiEmergencySameSiteOnlyMiddleware (api_emergency_block.py); replace with permanent _handle_api logic inside RateLimitMiddleware. - _handle_api decision chain: OPTIONS pass → same-origin pass → global IP block check → API-specific block check → quota → API per-minute counter → anon pass / auth _evaluate. - New Redis keys: rl:api:ip:<ip>:reqs, rl:api:ip:<ip>:blocked, rl:index:api_blocked_ips. Global rl:ip:<ip>:blocked is never written because of /api/ abuse — prevents NAT lockout. - _is_same_origin: strips port, lowercases, checks Origin first then Referer (sequential, not OR — wrong Origin blocks even if Referer matches). - Five new settings: API_RATE_LIMIT_{ENABLED,THRESHOLD,WINDOW_SECONDS, BLOCK_SECONDS,SAME_ORIGIN_BYPASS} with safe defaults. - 16 new tests; _make_middleware extended with explicit setting values. - RATE-LIMITER-PLAN.md updated with new key schema rows and _handle_api section. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	3 weeks ago
Edward Ribeiro	425519a967	Add rate limiting architecture diagrams to RATE-LIMITER-PLAN.md New section: Rate Limiting — Architecture Diagrams, covering: - NAT thundering herd before/after session bypass - nginx zone split before/after - Anonymous /api/ NAT problem before/after - Authenticated rate breach before/after - Per-path enforcement stack and trade-off table - The fundamental NAT constraint and mitigations Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	3 weeks ago
Edward Ribeiro	405ba55d32	Update RATE-LIMITER-PLAN.md with 2026-05-06/07 changes - Decision log: zone split, session bypass, soft auth rate limiting, threshold increases, API quota increases - nginx Layer 1: 4 zones replacing 2; updated burst table - Django Layer 2: new thresholds, auth breach no longer writes block key - Decision flow and mermaid diagram: updated thresholds and auth path - Key schema: updated all thresholds; user blocked marker noted as dead for rate breach (window resets naturally) - Session/voting bypass: new section with root cause, fix, and rationale - Enforcement order, test script examples: updated thresholds throughout Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	3 weeks ago
Edward Ribeiro	81cff6141c	Add RL_INDEX_BLOCKED_IPS/USERS ZSET indexes and atomic block writes via Lua Each block-key write now also ZADDs the full key name into a permanent ZSET (score = expiry unix timestamp) via a single Lua round-trip (_BLOCK_LUA). Replaces four _rl_cache.set() calls with _set_block() which degrades to a plain cache.set when Redis is unavailable. Indexes enable O(log N) enumeration of active blocks (ZRANGEBYSCORE) without a SCAN; prunable with ZREMRANGEBYSCORE. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	4 weeks ago
Edward Ribeiro	3d0b0ab2fb	Add per-consumer API daily/weekly quota and fix CORS headers on 429 responses - ratelimit.py: add _check_api_quota() with per-consumer Redis keys (quota:{ns}:daily/weekly:{period}:user\|ip); 50/350 anon, 1000/7000 auth - ratelimit.py: add X-RateLimit-Reason header to all 429 responses - settings.py: add API_QUOTA_ANON/AUTH_DAILY/WEEKLY config() settings - nginx sapl.conf: add `always` to CORS add_header directives so 429 responses carry CORS headers; remove invalid Credentials+wildcard combo; expose X-RateLimit-Reason and quota headers to browser JS - plan/RATE-LIMITER-PLAN.md: update decision flow, diagram, key schema, and enforcement table with quota entries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	4 weeks ago
Edward Ribeiro	725fb90225	Rate limiter: move scanner probes to nginx, fix NAT IP block for authenticated users, add 404-scan counter; remove dead painel sub-views - nginx sapl.conf: return 444 for scanner extension probes (.php, .asp, .jsp, .env, etc.) before requests reach Gunicorn — zero Python cost - ratelimit.py: remove check 2b (scanner_probe) — dead code now that nginx handles it; remove unused `import os` - ratelimit.py: authenticated users skip the rl:ip:blocked check (check 2) to prevent anonymous NAT traffic from blocking legislative house staff - ratelimit.py: add _handle_not_found — post-response 404 counter per anonymous IP; blocks after RATE_LIMIT_404_THRESHOLD (default 10) hits in the anon window, catching path probes without known extensions - settings.py: replace RATE_LIMIT_SCANNER_EXTENSIONS with RATE_LIMIT_404_THRESHOLD; add RL_IP_404S Redis key constant - painel: remove dead mensagem/parlamentares/votacao templates, views, and URL entries — unreachable from any menu or template - RATE-LIMITER-PLAN.md: update decision flow, mermaid diagram, enforcement graduation table, and key schema to reflect all changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	6cbd9316c2	Add painel/dados bypass, Django block metrics, and layer tracking in logs - nginx: exempt /painel/<pk>/dados from rate limiting (polling endpoint, will become WebSocket); dedicated location block with no limit_req - ratelimit.py: bypass RATE_LIMIT_BYPASS_PATHS paths before _evaluate; add layer=django to block log; increment daily Redis metrics counter rl:metrics:{ns}:{date}:blocked:{reason} (TTL 8 days) on every block - ratelimit.py: add quiltbot and AwarioBot to BOT_UA_FRAGMENTS - ratelimit.py: fix _is_suspicious_headers to require missing UA before blocking - settings: add RATE_LIMIT_BYPASS_PATHS with /painel/<pk>/dados pattern - plan: extend UA blocklist SADD seed command with missing bot tokens Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	74d5873426	Fix Redis configmap inline comment, clean cache key format, add blocked-IP scan to plan - redis-configmap: move inline comment to its own line (Redis fatal parse error) - settings: add CACHE_MIDDLEWARE_KEY_PREFIX='p' to remove double-dot in cache_page keys - settings: monkey-patch _i18n_cache_key_suffix to strip pt-br/timezone suffix from keys - ratelimit.py, settings: update example namespace from patobranco-pr to sapl31demo-df - robots.txt: add AwarioSmartBot block - plan: add rl:ip:*:blocked scan commands with TTL/value output Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	a3af6a0147	Rename nginx internal media path and simplify file serving in views Renames /private/media/ to /internal/media/ in nginx and serve_media(). Adds Content-Type and Content-Disposition to the X-Accel-Redirect response. Replaces manual file reads in proposicao_texto and doc_texto_integral with redirects to the media URL, removing the unused get_mime_type_from_file_extension helper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	93c1a21e72	Fix norma etag field and media private path - _norma_last_modified: use ultima_edicao instead of data_ultima_atualizacao - serve_media: gate on sapl/private/ instead of documentos_privados/ - plan: update norma freshness field reference Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	a4d7da3e7e	Phase 7: HTTP conditional requests, static caching, nginx cleanup - ConditionalGetMiddleware added to MIDDLEWARE (ETag/304 for all views) - @condition(etag_func, last_modified_func) on MateriaLegislativa and NormaJuridica detail views — skips view execution on cache hit via data_ultima_atualizacao (auto_now=True) as freshness signal - nginx /static/: expires 90m + Cache-Control public, max-age=5400 - nginx: removed upload-endpoint special-casing (location ~* ^/(protocoloadm/criar-protocolo\|...)) - plan/RATE-LIMITER-PLAN.md updated to reflect all Phase 7 changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	917e7e4106	Phase 6: scanner probe blocking, plan consolidation, and flow diagram Code: - Block IPs dynamically on scanner extension probes (.php, .asp, .aspx, .jsp, .cgi, .env) — writes rl:ip:{ip}:blocked on first hit; subsequent requests short-circuit at check 2 with zero counting overhead - Add RATE_LIMIT_SCANNER_EXTENSIONS setting (space-separated, env-overridable) - Import os in ratelimit.py for os.path.splitext Plan (RATE_LIMITER_PLAN.md → RATE-LIMITER-PLAN.md): - Rename to kebab-case for consistency with rate-limiter-v2.md - Merge missing content from rate-limiter-v2.md: context & problem statement, component diagram (DB0/DB1 split), decision log, Gunicorn tuning, nginx real-IP fixes, upload settings, N+1 fix (synced to actual implementation), enforcement graduation order, decorator migration table, file serving decision matrix, dynamic page caching guidelines, open questions - Add Mermaid decision flow diagram for RateLimitMiddleware._evaluate() - Add rationale section for rl:{ns}:ip:{ip}:w:{bucket} namespace scoping (5 arguments covering attack pattern match, gaming resistance, key orthogonality, multi-portal fairness, and isolation contract) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago
Edward Ribeiro	bea9d84373	Phase 5: X-Accel-Redirect for /media/, UA Redis deny list, per-path counters nginx: - /media/ proxied through Gunicorn (sapl_general rate limit) instead of direct alias — Django middleware now runs on every media request - /_accel/media/ internal location serves file bytes via X-Accel-Redirect sapl/base/media.py (new): - serve_media() gate: path traversal guard, auth redirect for documentos_privados/, per-path Redis counter, content-type metadata cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG sapl/middleware/ratelimit.py: - RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants - _incr_with_ttl() extracted to module level (reused by media.py) - Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH); _is_redis_blocked_ua() tokenises UA and checks sha256 of each token sapl/settings.py: - RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL, MEDIA_FILE_CACHE_TTL added (all env-tunable via config()) plan/RATE_LIMITER_PLAN.md: - Key schema table updated; media file serving section added; decision flow documented; UA deny list seed section expanded Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	1 month ago

14 Commits (079dd47df2c9a940f10487795c95dfc7ebd35bb6)