diff --git a/.gitignore b/.gitignore index 0013e9542..798da4154 100644 --- a/.gitignore +++ b/.gitignore @@ -110,3 +110,5 @@ media/* !media/.gitkeep restauracoes/* + +.claude diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..b3a1102eb --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,145 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +SAPL (Sistema de Apoio ao Processo Legislativo) is a Django-based legislative management system used by Brazilian municipal and state legislative houses. It manages bills, parliamentary sessions, committees, norms, protocols, and related legislative workflows. + +## Commands + +### Development + +```bash +# Run dev server +python manage.py runserver + +# Docker (dev, without bundled DB) +docker-compose -f docker/docker-compose-dev.yml up + +# Docker (dev, with PostgreSQL container) +docker-compose -f docker/docker-compose-dev-db.yml up +``` + +### Database Setup (local PostgreSQL) + +```bash +sudo -u postgres psql -c "CREATE ROLE sapl LOGIN ENCRYPTED PASSWORD 'sapl' NOSUPERUSER INHERIT CREATEDB NOCREATEROLE NOREPLICATION;" +sudo -u postgres psql -c "CREATE DATABASE sapl WITH OWNER=sapl ENCODING='UTF8' LC_COLLATE='pt_BR.UTF-8' LC_CTYPE='pt_BR.UTF-8' CONNECTION LIMIT=-1 TEMPLATE template0;" +python manage.py migrate +``` + +### Testing + +```bash +# All tests (reuses DB by default for speed) +pytest + +# Single test file or test function +pytest sapl/materia/tests/test_materia.py +pytest sapl/materia/tests/test_materia.py::test_function_name + +# Force DB recreation +pytest --create-db + +# With coverage +pytest --cov=sapl +``` + +Tests require `DJANGO_SETTINGS_MODULE=sapl.settings` (set in `pytest.ini`). All tests must be marked with `@pytest.mark.django_db`. The `conftest.py` root fixture provides an `app` fixture (WebTest `DjangoTestApp`). + +### Linting / Formatting + +```bash +flake8 . +isort . +autopep8 --in-place +``` + +### Restore Database from Backup + +```bash +./scripts/restore_db.sh -f /path/to/dump +./scripts/restore_db.sh -f /path/to/dump -p 5433 # Docker port +``` + +## Architecture + +### Django Apps + +Apps are under `sapl/` and follow domain boundaries: + +| App | Domain | +|-----|--------| +| `base` | `CasaLegislativa` (legislative house config), `AppConfig`, `Autor` (authorship) | +| `parliamentary` | `Parlamentar`, `Legislatura`, `SessaoLegislativa`, `Coligacao` | +| `materia` | Bills (`MateriaLegislativa`), types, tracking, annexes | +| `norma` | Laws/norms (`NormaJuridica`) and hierarchies | +| `sessao` | Plenary sessions, agenda, attendance, voting | +| `comissoes` | Committees (`Comissao`) and meetings (`Reuniao`) | +| `protocoloadm` | Administrative protocols and document intake | +| `compilacao` | Structured/articulated texts (LexML-like tree structure) | +| `lexml` | LexML XML standard integration | +| `audiencia` | Public hearings | +| `painel` | Real-time session display panel | +| `relatorios` | PDF report generation | +| `api` | REST API entry point (auto-generated ViewSets) | +| `crud` | Generic CRUD base views | +| `rules` | Business rules and permission definitions | + +### REST API + +The API uses a custom `drfautoapi` package (`drfautoapi/drfautoapi.py`) that auto-generates DRF ViewSets, Serializers, and FilterSets from Django models. Authentication is Token + Session. Permissions use a custom `SaplModelPermissions` class that maps HTTP methods to Django model permissions. + +OpenAPI 3.0 docs are generated by drf-spectacular. + +### Caching + +- **Default:** File-based (`/var/tmp/django_cache`) +- **Production:** Redis via django-redis; configured at startup by `configure_redis_cache()` in `sapl/settings.py` +- **Cache key prefix:** `cache:{POD_NAMESPACE}:` (namespace-isolated for multi-tenant k8s) +- **Rate limiter state** is shared via Redis keys + +### Feature Flags + +django-waffle is used for feature flags. Switches (global on/off) can be toggled via: + +```bash +python manage.py waffle_switch on|off +``` + +### Key Environment Variables + +| Variable | Purpose | +|----------|---------| +| `DATABASE_URL` | PostgreSQL connection string | +| `SECRET_KEY` | Django secret key | +| `DEBUG` | Debug mode | +| `REDIS_URL` | Redis host:port | +| `CACHE_BACKEND` | `file` or `redis` | +| `POD_NAMESPACE` | K8s namespace (used in cache key prefix) | +| `USE_SOLR` | Enable Haystack/Solr full-text search | +| `SOLR_URL` / `SOLR_COLLECTION` | Solr connection | + +### Docker Build + +The production build requires a MaxMind GeoLite2-ASN license key (for nginx ASN-based bot blocking): + +```bash +docker build --secret id=maxmind_key,src=.env -f docker/Dockerfile -t sapl:local . +``` + +Optional build args: `WITH_NGINX`, `WITH_GRAPHVIZ`, `WITH_POPPLER`, `WITH_PSQL_CLIENT`. + +### Key File Locations + +| File | Purpose | +|------|---------| +| `sapl/settings.py` | All Django settings, including cache/rate-limit setup | +| `pytest.ini` | Test configuration (DJANGO_SETTINGS_MODULE, addopts) | +| `conftest.py` | Root pytest fixtures | +| `drfautoapi/drfautoapi.py` | Auto-API generation logic | +| `docker/startup_scripts/start.sh` | Container entrypoint (migrations, waffle, gunicorn) | +| `requirements/requirements.txt` | Production deps | +| `requirements/test-requirements.txt` | Test deps | +| `requirements/dev-requirements.txt` | Dev/lint deps | diff --git a/docker/Dockerfile b/docker/Dockerfile index 831627ec8..2ead47a14 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -51,13 +51,13 @@ ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 \ RUN set -eux; \ apt-get update; \ apt-get install -y --no-install-recommends \ - curl jq bash tzdata fontconfig tini libmagic1 \ + curl jq bash tzdata fontconfig tini libmagic1 gettext-base \ libcairo2 libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf-2.0-0 \ libharfbuzz0b libfreetype6 libjpeg62-turbo zlib1g fonts-dejavu-core; \ if [ "$WITH_GRAPHVIZ" = "1" ]; then apt-get install -y --no-install-recommends graphviz; fi; \ if [ "$WITH_POPPLER" = "1" ]; then apt-get install -y --no-install-recommends poppler-utils; fi; \ if [ "$WITH_PSQL_CLIENT" = "1" ]; then apt-get install -y --no-install-recommends postgresql-client; fi; \ - if [ "$WITH_NGINX" = "1" ]; then apt-get install -y --no-install-recommends nginx; fi; \ + if [ "$WITH_NGINX" = "1" ]; then apt-get install -y --no-install-recommends nginx libnginx-mod-http-geoip2 libnginx-mod-http-ndk libnginx-mod-http-lua libmaxminddb0; fi; \ rm -rf /var/lib/apt/lists/* # Usuários/grupos (idempotente) @@ -67,7 +67,13 @@ RUN useradd --system --no-create-home --shell /usr/sbin/nologin sapl || true \ && usermod -aG nginx sapl || true # Estrutura de diretórios -RUN mkdir -p /var/interlegis/sapl /var/interlegis/sapl/data /var/interlegis/sapl/media /var/interlegis/sapl/run \ +RUN mkdir -p \ + /var/interlegis/sapl \ + /var/interlegis/sapl/data \ + /var/interlegis/sapl/media \ + /var/interlegis/sapl/run \ + /var/interlegis/sapl/tmp \ + /etc/nginx/geoip \ && chown -R root:nginx /var/interlegis/sapl /var/interlegis/sapl/run \ && chmod -R g+rwX /var/interlegis/sapl \ && chmod 2775 /var/interlegis/sapl /var/interlegis/sapl/run \ @@ -81,11 +87,27 @@ COPY --from=builder ${VENV_DIR} ${VENV_DIR} # Código da aplicação (depois do venv para aproveitar cache) COPY . /var/interlegis/sapl/ -# Nginx (somente se instalado) +# OpenResty config + GeoLite2-ASN database (somente se instalado). +# +# GeoLite2-ASN.mmdb is NOT downloaded at build time. +# Run docker/geoip/update_geoip.sh before each build to refresh it. +# The .mmdb file lives at docker/geoip/GeoLite2-ASN.mmdb (git-ignored binary). +# If the file is absent the build FAILS — run update_geoip.sh first. RUN if [ "$WITH_NGINX" = "1" ]; then \ rm -f /etc/nginx/conf.d/*; \ - cp docker/config/nginx/sapl.conf /etc/nginx/conf.d/sapl.conf; \ - cp docker/config/nginx/nginx.conf /etc/nginx/nginx.conf; \ + cp docker/config/nginx/sapl.conf /etc/nginx/conf.d/sapl.conf.template; \ + cp docker/config/nginx/nginx.conf /etc/nginx/nginx.conf; \ + cp docker/config/nginx/blocklist.lua /etc/nginx/blocklist.lua; \ + mkdir -p /usr/lib/lua/resty; \ + cp docker/config/nginx/resty_redis.lua /usr/lib/lua/resty/redis.lua; \ + if [ -f "docker/geoip/GeoLite2-ASN.mmdb" ]; then \ + cp docker/geoip/GeoLite2-ASN.mmdb /etc/nginx/geoip/GeoLite2-ASN.mmdb; \ + echo "[geoip] GeoLite2-ASN.mmdb installed."; \ + else \ + echo "[geoip] ERROR: docker/geoip/GeoLite2-ASN.mmdb not found."; \ + echo "[geoip] Run docker/geoip/update_geoip.sh then rebuild."; \ + exit 1; \ + fi; \ fi # Scripts + gunicorn.conf no diretório da app diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..9f4776ad7 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,107 @@ +# SAPL Docker Build + +## Building locally + +### 1. Prerequisites + +- Docker 23+ with BuildKit enabled (default since Docker 23) +- A free [MaxMind account](https://www.maxmind.com/en/geolite2/signup) with a license key + +### 2. Set your MaxMind license key + +Add the key to the project root `.env` file (already gitignored): + +``` +MAXMIND_LICENSE_KEY=your_key_here +``` + +The key is used **only at build time** to download the `GeoLite2-ASN.mmdb` database for +nginx ASN-based bot blocking. It is injected via a BuildKit secret and is **never stored +in any image layer** — it will not appear in `docker history` or any registry push. + +### 3. Build the image + +```bash +docker build \ + --secret id=maxmind_key,src=.env \ + -f docker/Dockerfile \ + -t sapl:local \ + . +``` + +Run from the **project root** (not from inside `docker/`), so the build context includes +the full source tree. + +#### Optional build args + +| Arg | Default | Description | +|---|---|---| +| `WITH_NGINX` | `1` | Include nginx in the image | +| `WITH_GRAPHVIZ` | `1` | Include Graphviz | +| `WITH_POPPLER` | `1` | Include Poppler (PDF utilities) | +| `WITH_PSQL_CLIENT` | `1` | Include `psql` client | + +Example — build without Graphviz: + +```bash +docker build \ + --secret id=maxmind_key,src=.env \ + --build-arg WITH_GRAPHVIZ=0 \ + -f docker/Dockerfile \ + -t sapl:local \ + . +``` + +### 4. If the MaxMind key is not provided + +The build will succeed but nginx will log an error on startup because +`/etc/nginx/geoip/GeoLite2-ASN.mmdb` will be missing. ASN-based bot blocking will +be inactive. All other Phase 0 mitigations (UA blocklist, rate limits, ETags) still apply. + +You can mount the database file at runtime as a workaround: + +```bash +docker run \ + -v /path/to/GeoLite2-ASN.mmdb:/etc/nginx/geoip/GeoLite2-ASN.mmdb:ro \ + sapl:local +``` + +--- + +## Production — Harbor + +Official images are built and pushed through **Harbor**. Before the next release, configure +the MaxMind license key as a build secret in the Harbor / CI pipeline: + +1. Add `MAXMIND_LICENSE_KEY` as a **masked CI/CD secret** in the Harbor build project + (do not put it in any Helm values file or ConfigMap). +2. Pass it to the build step: + ```bash + docker build \ + --secret id=maxmind_key,env=MAXMIND_LICENSE_KEY \ + -f docker/Dockerfile \ + -t harbor.your-registry/sapl/sapl:$VERSION \ + . + ``` + Note: `env=` variant reads the secret from an environment variable instead of a file — + useful in CI where `.env` files are not present. +3. Push as normal — the key will not be present in the pushed image. + +### Keeping GeoLite2-ASN up to date + +MaxMind updates the database every Tuesday. On production hosts, install the weekly refresh +cron (run as root): + +```bash +cat > /etc/cron.weekly/update-geoip << 'EOF' +#!/bin/bash +MAXMIND_KEY="$(kubectl get secret sapl-build-secrets -n interlegis-infra \ + -o jsonpath='{.data.MAXMIND_LICENSE_KEY}' | base64 -d)" +curl -fsSL \ + "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=${MAXMIND_KEY}&suffix=tar.gz" \ + | tar -xz -C /tmp --wildcards '*.mmdb' +mv /tmp/GeoLite2-ASN_*/GeoLite2-ASN.mmdb /etc/nginx/geoip/GeoLite2-ASN.mmdb +nginx -s reload +EOF +chmod +x /etc/cron.weekly/update-geoip +``` diff --git a/docker/config/nginx/blocklist.lua b/docker/config/nginx/blocklist.lua new file mode 100644 index 000000000..02a0526d8 --- /dev/null +++ b/docker/config/nginx/blocklist.lua @@ -0,0 +1,60 @@ +-- blocklist.lua: Redis-backed early IP rejection before Gunicorn. +-- ASN and User-Agent blocking are handled upstream by nginx if() blocks. +-- +-- Checks (Redis DB 1, read-only): +-- 1. ngx.shared.ip_prefix_blocked — in-process prefix cache (60s refresh, no Redis I/O) +-- 2. GET rl:ip:{ip}:blocked — global IP block +-- 3. GET rl:api:ns:{ns}:ip:{ip}:blocked — per-tenant API block (/api/ only) +-- +-- Checks 2+3 are pipelined in one Redis round trip. +-- On Redis failure: fail-open (request passes to Django). + +local url = os.getenv("REDIS_URL") or "redis://127.0.0.1:6379" +local REDIS_HOST, port_str = url:match("redis://([^:/]+):(%d+)") +if not REDIS_HOST then REDIS_HOST = url:match("redis://([^:/]+)") or "127.0.0.1" end +local REDIS_PORT = tonumber(port_str) or 6379 + +local POD_NS = os.getenv("POD_NAMESPACE") or "" +local ip = ngx.var.remote_addr +local is_api = ngx.var.uri:sub(1, 5) == "/api/" + +-- Build 4 prefix candidates for ip e.g. '203.0.113.42': +-- '203.', '203.0.', '203.0.113.', '203.0.113.42' +-- Mirrors Django's _is_ip_prefix_blocked normalisation. +local parts = {} +for p in ip:gmatch("[^.]+") do parts[#parts+1] = p end +local p1 = parts[1] .. "." +local p2 = parts[1] .. "." .. parts[2] .. "." +local p3 = parts[1] .. "." .. parts[2] .. "." .. parts[3] .. "." + +local function return_429() + ngx.status = 429 + ngx.header["Retry-After"] = "300" + ngx.header["Content-Type"] = "application/json" + ngx.say('{"detail":"Too Many Requests"}') + return ngx.exit(429) +end + +-- 1. Prefix check (shared dict — zero Redis I/O per request). +local dict = ngx.shared.ip_prefix_blocked +if dict:get(p1) or dict:get(p2) or dict:get(p3) or dict:get(ip) then + return return_429() +end + +-- 2+3. Pipeline both STRING block checks in one Redis round trip. +local red = require("resty.redis"):new() +red:set_timeout(200) +local ok = red:connect(REDIS_HOST, REDIS_PORT) +if not ok then return end -- fail-open + +red:select(1) +red:init_pipeline() +red:get("rl:ip:" .. ip .. ":blocked") +red:get("rl:api:ns:" .. POD_NS .. ":ip:" .. ip .. ":blocked") +local res = red:commit_pipeline() +red:set_keepalive(10000, 1) + +if not res then return end -- fail-open on pipeline error + +if res[1] == "1" then return return_429() end +if is_api and res[2] == "1" then return return_429() end diff --git a/docker/config/nginx/nginx.conf b/docker/config/nginx/nginx.conf index e002a6905..375453324 100644 --- a/docker/config/nginx/nginx.conf +++ b/docker/config/nginx/nginx.conf @@ -1,3 +1,11 @@ +load_module modules/ndk_http_module.so; +load_module modules/ngx_http_geoip2_module.so; +load_module modules/ngx_http_lua_module.so; + +# Make POD_NAMESPACE and Redis URL available to Lua. +env POD_NAMESPACE; +env REDIS_URL; + user www-data nginx; worker_processes 1; @@ -14,20 +22,148 @@ http { include /etc/nginx/mime.types; default_type application/octet-stream; + # ---------------------------------------------------------------- + # Real client IP extracted from X-Forwarded-For set by K8s Ingress. + # ---------------------------------------------------------------- + real_ip_header X-Forwarded-For; + real_ip_recursive on; + set_real_ip_from 10.0.0.0/8; + set_real_ip_from 172.16.0.0/12; + set_real_ip_from 192.168.0.0/16; + set_real_ip_from 127.0.0.1; + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' - '"$http_user_agent" "$http_x_forwarded_for"'; + '"$http_user_agent" "$http_x_forwarded_for" ' + 'rt=$request_time'; access_log /var/log/nginx/access.log main; - sendfile off; - #tcp_nopush on; + # ---------------------------------------------------------------- + # FIX: kernel bypass — was off (disables zero-copy file serving) + # ---------------------------------------------------------------- + sendfile on; + tcp_nopush on; + tcp_nodelay on; + + # ---------------------------------------------------------------- + # Timeouts — reduced from 300s to prevent bots holding threads. + # Per-location overrides in sapl.conf handle legitimate slow ops. + # ---------------------------------------------------------------- + keepalive_timeout 75; # was 300 + proxy_connect_timeout 10s; + proxy_read_timeout 120s; # was 300s — overridden per-location + proxy_send_timeout 120s; + + # ---------------------------------------------------------------- + # Rate limiting zones (effective once real_ip is resolved). + # sapl_general : 120 req/min — aligned with Django anon rate (RATE_LIMITER_RATE) + # sapl_media : 240 req/min — aligned with Django auth rate (RATE_LIMITER_RATE_AUTHENTICATED) + # sapl_api : 120 req/min — aligned with Django rate limiter threshold + # sapl_heavy : 10 req/min — PDF generation; slow by design + # Burst values are env-var configurable at container start (start.sh). + # ---------------------------------------------------------------- + limit_req_log_level warn; + + limit_req_zone $binary_remote_addr zone=sapl_general:20m rate=120r/m; + limit_req_zone $binary_remote_addr zone=sapl_media:20m rate=240r/m; + limit_req_zone $binary_remote_addr zone=sapl_api:20m rate=120r/m; + limit_req_zone $binary_remote_addr zone=sapl_heavy:10m rate=10r/m; + + # ---------------------------------------------------------------- + # ASN-Based Blocking (datacenter / scraper ASNs). + # Requires libnginx-mod-http-geoip2 and GeoLite2-ASN.mmdb. + # ---------------------------------------------------------------- + geoip2 /etc/nginx/geoip/GeoLite2-ASN.mmdb { + $geoip2_asn_number autonomous_system_number; + $geoip2_asn_org autonomous_system_organization; + } + + map $geoip2_asn_number $bot_asn { + default 0; + 16509 1; # Amazon AWS + 14618 1; # Amazon AWS us-east + 8075 1; # Microsoft Azure + 396982 1; # Google Cloud + 20473 1; # Vultr + 24940 1; # Hetzner + 16276 1; # OVH + 36352 1; # ColoCrossing + 63949 1; # Linode / Akamai + } + + # ---------------------------------------------------------------- + # Bot blocking by User-Agent. + # Chrome/98.0.4758 is a confirmed scraper (no real user runs a + # 2022 browser version in 2026). Googlebot excluded for SEO. + # ---------------------------------------------------------------- + map $http_user_agent $bot_ua_blocked { + default 0; + "~*GPTBot" 1; + "~*ClaudeBot" 1; + "~*PerplexityBot" 1; + "~*Bytespider" 1; + "~*AhrefsBot" 1; + "~*SemrushBot" 1; + "~*DotBot" 1; + "~*meta-externalagent" 1; + "~*OAI-SearchBot" 1; + "~*bingbot" 1; + "~*SERankingBacklinksBot" 1; + "~*Chrome/98\.0\.4758" 1; + } + + # ---------------------------------------------------------------- + # Lua: search path for vendored resty.* libraries. + # lua-resty-redis is not in Debian repos; vendored at docker/config/nginx/resty_redis.lua + # and copied to /usr/lib/lua/resty/redis.lua at image build time. + # ---------------------------------------------------------------- + lua_package_path '/usr/lib/lua/?.lua;;'; - keepalive_timeout 300; + # ---------------------------------------------------------------- + # Shared dict for IP-prefix deny list (refreshed every 60s). + # 1 MB holds ~10,000 prefix entries with overhead to spare. + # ---------------------------------------------------------------- + lua_shared_dict ip_prefix_blocked 1m; - proxy_connect_timeout 75s; - proxy_read_timeout 300s; + # ---------------------------------------------------------------- + # Background timer: populates ip_prefix_blocked from Redis DB 1. + # Runs once per worker at startup, then every 60s. + # ---------------------------------------------------------------- + init_worker_by_lua_block { + local url = os.getenv("REDIS_URL") or "redis://127.0.0.1:6379" + local REDIS_HOST, port_str = url:match("redis://([^:/]+):(%d+)") + if not REDIS_HOST then REDIS_HOST = url:match("redis://([^:/]+)") or "127.0.0.1" end + local REDIS_PORT = tonumber(port_str) or 6379 + local INTERVAL = 60 + local function refresh(premature) + if premature then return end + local ok, red = pcall(function() + local r = require("resty.redis"):new() + r:set_timeout(500) + assert(r:connect(REDIS_HOST, REDIS_PORT)) + r:select(1) + return r + end) + if ok then + local members = red:smembers("rl:ip_prefix:blocked") + red:set_keepalive(10000, 1) + if members and type(members) == "table" then + local dict = ngx.shared.ip_prefix_blocked + dict:flush_all() + for _, m in ipairs(members) do + local stripped = m:gsub("%.$", "") + local key = (select(2, stripped:gsub("%.", "")) < 3) + and (stripped .. ".") or stripped + dict:set(key, 1) + end + end + end + ngx.timer.at(INTERVAL, refresh) + end + ngx.timer.at(0, refresh) + } gzip on; gzip_disable "MSIE [1-6]\\.(?!.*SV1)"; diff --git a/docker/config/nginx/resty_redis.lua b/docker/config/nginx/resty_redis.lua new file mode 100644 index 000000000..b0c72f92b --- /dev/null +++ b/docker/config/nginx/resty_redis.lua @@ -0,0 +1,741 @@ +-- Copyright (C) Yichun Zhang (agentzh) + + +local sub = string.sub +local byte = string.byte +local tab_insert = table.insert +local tab_remove = table.remove +local tcp = ngx.socket.tcp +local null = ngx.null +local ipairs = ipairs +local type = type +local pairs = pairs +local unpack = unpack +local setmetatable = setmetatable +local tonumber = tonumber +local tostring = tostring +local rawget = rawget +local select = select +local tb_clear = require "table.clear" +--local error = error + + +local ok, new_tab = pcall(require, "table.new") +if not ok or type(new_tab) ~= "function" then + new_tab = function (narr, nrec) return {} end +end + +local tab_pool_len = 0 +local tab_pool = new_tab(16, 0) +local _M = new_tab(0, 55) + +_M._VERSION = '0.32' + + +local common_cmds = { + "get", "set", "mget", "mset", + "del", "incr", "decr", -- Strings + "llen", "lindex", "lpop", "lpush", + "lrange", "linsert", -- Lists + "hexists", "hget", "hset", "hmget", + --[[ "hmset", ]] "hdel", -- Hashes + "smembers", "sismember", "sadd", "srem", + "sdiff", "sinter", "sunion", -- Sets + "zrange", "zrangebyscore", "zrank", "zadd", + "zrem", "zincrby", -- Sorted Sets + "auth", "eval", "expire", "script", + "sort" -- Others +} + + +local sub_commands = { + "subscribe", "psubscribe" +} + +local blocking_commands = { + "blpop", "brpop" +} + +local unsub_commands = { + "unsubscribe", "punsubscribe" +} + + +local mt = { __index = _M } + + +local function get_tab_from_pool() + if tab_pool_len > 0 then + tab_pool_len = tab_pool_len - 1 + return tab_pool[tab_pool_len + 1] + end + + return new_tab(24, 0) -- one field takes 5 slots +end + + +local function put_tab_into_pool(tab) + if tab_pool_len >= 32 then + return + end + + tb_clear(tab) + tab_pool_len = tab_pool_len + 1 + tab_pool[tab_pool_len] = tab +end + + +function _M.new(self) + local sock, err = tcp() + if not sock then + return nil, err + end + local redis = setmetatable({ _sock = sock, + _subscribed = false, + _n_channel = { + unsubscribe = 0, + punsubscribe = 0, + }, + }, mt) + return redis +end + + +function _M.register_module_prefix(mod) + _M[mod] = function(self) + self._module_prefix = mod + return self + end +end + + +function _M.set_timeout(self, timeout) + local sock = rawget(self, "_sock") + if not sock then + error("not initialized", 2) + return + end + + sock:settimeout(timeout) +end + + +function _M.set_timeouts(self, connect_timeout, send_timeout, read_timeout) + local sock = rawget(self, "_sock") + if not sock then + error("not initialized", 2) + return + end + + sock:settimeouts(connect_timeout, send_timeout, read_timeout) +end + + +function _M.connect(self, host, port_or_opts, opts) + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + local unix + + do + local typ = type(host) + if typ ~= "string" then + error("bad argument #1 host: string expected, got " .. typ, 2) + end + + if sub(host, 1, 5) == "unix:" then + unix = true + end + + if unix then + typ = type(port_or_opts) + if port_or_opts ~= nil and typ ~= "table" then + error("bad argument #2 opts: nil or table expected, got " .. + typ, 2) + end + + else + typ = type(port_or_opts) + if typ ~= "number" then + port_or_opts = tonumber(port_or_opts) + if port_or_opts == nil then + error("bad argument #2 port: number expected, got " .. + typ, 2) + end + end + + if opts ~= nil then + typ = type(opts) + if typ ~= "table" then + error("bad argument #3 opts: nil or table expected, got " .. + typ, 2) + end + end + end + + end + + self._subscribed = false + + local ok, err + + if unix then + -- second argument of sock:connect() cannot be nil + if port_or_opts ~= nil then + ok, err = sock:connect(host, port_or_opts) + opts = port_or_opts + else + ok, err = sock:connect(host) + end + else + ok, err = sock:connect(host, port_or_opts, opts) + end + + if not ok then + return ok, err + end + + if opts and opts.ssl then + ok, err = sock:sslhandshake(false, opts.server_name, opts.ssl_verify) + if not ok then + return ok, "failed to do ssl handshake: " .. err + end + end + + return ok, err +end + + +function _M.set_keepalive(self, ...) + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + if rawget(self, "_subscribed") then + return nil, "subscribed state" + end + + return sock:setkeepalive(...) +end + + +function _M.get_reused_times(self) + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + return sock:getreusedtimes() +end + + +local function close(self) + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + return sock:close() +end +_M.close = close + + +local function _read_reply(self, sock) + local line, err = sock:receive() + if not line then + if err == "timeout" and not rawget(self, "_subscribed") and not rawget(self, "_blocking") then + sock:close() + end + return nil, err + end + + local prefix = byte(line) + + if prefix == 36 then -- char '$' + -- print("bulk reply") + + local size = tonumber(sub(line, 2)) + if size < 0 then + return null + end + + local data, err = sock:receive(size) + if not data then + if err == "timeout" then + sock:close() + end + return nil, err + end + + local dummy, err = sock:receive(2) -- ignore CRLF + if not dummy then + if err == "timeout" then + sock:close() + end + return nil, err + end + + return data + + elseif prefix == 43 then -- char '+' + -- print("status reply") + + return sub(line, 2) + + elseif prefix == 42 then -- char '*' + local n = tonumber(sub(line, 2)) + + -- print("multi-bulk reply: ", n) + if n < 0 then + return null + end + + local vals = new_tab(n, 0) + local nvals = 0 + for i = 1, n do + local res, err = _read_reply(self, sock) + if res then + nvals = nvals + 1 + vals[nvals] = res + + elseif res == nil then + return nil, err + + else + -- be a valid redis error value + nvals = nvals + 1 + vals[nvals] = {false, err} + end + end + + return vals + + elseif prefix == 58 then -- char ':' + -- print("integer reply") + return tonumber(sub(line, 2)) + + elseif prefix == 45 then -- char '-' + -- print("error reply: ", n) + + return false, sub(line, 2) + + else + -- when `line` is an empty string, `prefix` will be equal to nil. + return nil, "unknown prefix: \"" .. tostring(prefix) .. "\"" + end +end + + +local function _gen_req(args) + local nargs = #args + + local req = get_tab_from_pool() + req[1] = "*" + req[2] = nargs + req[3] = "\r\n" + local nbits = 4 + + for i = 1, nargs do + local arg = args[i] + if type(arg) ~= "string" then + arg = tostring(arg) + end + + req[nbits] = "$" + req[nbits + 1] = #arg + req[nbits + 2] = "\r\n" + req[nbits + 3] = arg + req[nbits + 4] = "\r\n" + + nbits = nbits + 5 + end + + -- it is much faster to do string concatenation on the C land + -- in real world (large number of strings in the Lua VM) + return req +end + + +local function _check_msg(self, res) + return rawget(self, "_subscribed") and + type(res) == "table" and (res[1] == "message" or res[1] == "pmessage") +end + + +local function _do_cmd(self, ...) + local args = {...} + + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + local req = _gen_req(args) + + local reqs = rawget(self, "_reqs") + if reqs then + reqs[#reqs + 1] = req + return + end + + -- print("request: ", table.concat(req)) + + local bytes, err = sock:send(req) + put_tab_into_pool(req) + + if not bytes then + return nil, err + end + + local res, err = _read_reply(self, sock) + while _check_msg(self, res) do + if rawget(self, "_buffered_msg") == nil then + self._buffered_msg = new_tab(1, 0) + end + + tab_insert(self._buffered_msg, res) + res, err = _read_reply(self, sock) + end + + return res, err +end + + +local function _check_unsubscribed(self, res) + if type(res) == "table" + and (res[1] == "unsubscribe" or res[1] == "punsubscribe") + then + self._n_channel[res[1]] = self._n_channel[res[1]] - 1 + + local buffered_msg = rawget(self, "_buffered_msg") + if buffered_msg then + -- remove messages of unsubscribed channel + local msg_type = + (res[1] == "punsubscribe") and "pmessage" or "message" + local j = 1 + for _, msg in ipairs(buffered_msg) do + if msg[1] == msg_type and msg[2] ~= res[2] then + -- move messages to overwrite the removed ones + buffered_msg[j] = msg + j = j + 1 + end + end + + -- clear remain messages + for i = j, #buffered_msg do + buffered_msg[i] = nil + end + + if #buffered_msg == 0 then + self._buffered_msg = nil + end + end + + if res[3] == 0 then + -- all channels are unsubscribed + self._subscribed = false + end + end +end + + +local function _check_subscribed(self, res) + if type(res) == "table" + and (res[1] == "subscribe" or res[1] == "psubscribe") + then + if res[1] == "subscribe" then + self._n_channel.unsubscribe = self._n_channel.unsubscribe + 1 + + elseif res[1] == "psubscribe" then + self._n_channel.punsubscribe = self._n_channel.punsubscribe + 1 + end + end +end + + +function _M.read_reply(self) + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + if not rawget(self, "_subscribed") then + return nil, "not subscribed" + end + + local buffered_msg = rawget(self, "_buffered_msg") + if buffered_msg then + local msg = buffered_msg[1] + tab_remove(buffered_msg, 1) + + if #buffered_msg == 0 then + self._buffered_msg = nil + end + + return msg + end + + local res, err = _read_reply(self, sock) + _check_unsubscribed(self, res) + + return res, err +end + + +local function do_cmd(self, cmd, ...) + local module_prefix = rawget(self, "_module_prefix") + if module_prefix then + self._module_prefix = nil + return _do_cmd(self, module_prefix .. "." .. cmd, ...) + end + + return _do_cmd(self, cmd, ...) +end + + +for i = 1, #common_cmds do + local cmd = common_cmds[i] + + _M[cmd] = + function (self, ...) + return do_cmd(self, cmd, ...) + end +end + +for i = 1, #blocking_commands do + local cmd = blocking_commands[i] + + _M[cmd] = + function (self, ...) + if not rawget(self, "_blocking") then + self._blocking = true + end + return do_cmd(self, cmd, ...) + end +end + +local function handle_subscribe_result(self, cmd, nargs, res) + local err + _check_subscribed(self, res) + + if nargs <= 1 then + return res + end + + local results = new_tab(nargs, 0) + results[1] = res + local sock = rawget(self, "_sock") + + for i = 2, nargs do + res, err = _read_reply(self, sock) + if not res then + return nil, err + end + + _check_subscribed(self, res) + results[i] = res + end + + return results +end + +for i = 1, #sub_commands do + local cmd = sub_commands[i] + + _M[cmd] = + function (self, ...) + if not rawget(self, "_subscribed") then + self._subscribed = true + end + + local nargs = select("#", ...) + + local res, err = _do_cmd(self, cmd, ...) + if not res then + return nil, err + end + + return handle_subscribe_result(self, cmd, nargs, res) + end +end + + +local function handle_unsubscribe_result(self, cmd, nargs, res) + local err + _check_unsubscribed(self, res) + + if self._n_channel[cmd] == 0 or nargs == 1 then + return res + end + + local results = new_tab(nargs, 0) + results[1] = res + local sock = rawget(self, "_sock") + local i = 2 + + while nargs == 0 or i <= nargs do + res, err = _read_reply(self, sock) + if not res then + return nil, err + end + + results[i] = res + i = i + 1 + + _check_unsubscribed(self, res) + if self._n_channel[cmd] == 0 then + -- exit the loop for unsubscribe() call + break + end + end + + return results +end + +for i = 1, #unsub_commands do + local cmd = unsub_commands[i] + + _M[cmd] = + function (self, ...) + -- assume all channels are unsubscribed by only one time + if not rawget(self, "_subscribed") then + return nil, "not subscribed" + end + + local nargs = select("#", ...) + + local res, err = _do_cmd(self, cmd, ...) + if not res then + return nil, err + end + + return handle_unsubscribe_result(self, cmd, nargs, res) + end +end + + +function _M.hmset(self, hashname, ...) + if select('#', ...) == 1 then + local t = select(1, ...) + + local n = 0 + for k, v in pairs(t) do + n = n + 2 + end + + local array = new_tab(n, 0) + + local i = 0 + for k, v in pairs(t) do + array[i + 1] = k + array[i + 2] = v + i = i + 2 + end + -- print("key", hashname) + return _do_cmd(self, "hmset", hashname, unpack(array)) + end + + -- backwards compatibility + return _do_cmd(self, "hmset", hashname, ...) +end + + +function _M.init_pipeline(self, n) + self._reqs = new_tab(n or 4, 0) +end + + +function _M.cancel_pipeline(self) + self._reqs = nil +end + + +function _M.commit_pipeline(self) + local reqs = rawget(self, "_reqs") + if not reqs then + return nil, "no pipeline" + end + + self._reqs = nil + + local sock = rawget(self, "_sock") + if not sock then + return nil, "not initialized" + end + + local bytes, err = sock:send(reqs) + for _, req in ipairs(reqs) do + put_tab_into_pool(req) + end + + if not bytes then + return nil, err + end + + local nvals = 0 + local nreqs = #reqs + local vals = new_tab(nreqs, 0) + for i = 1, nreqs do + local res, err = _read_reply(self, sock) + if res then + nvals = nvals + 1 + vals[nvals] = res + + elseif res == nil then + if err == "timeout" then + close(self) + end + return nil, err + + else + -- be a valid redis error value + nvals = nvals + 1 + vals[nvals] = {false, err} + end + end + + return vals +end + + +function _M.array_to_hash(self, t) + local n = #t + -- print("n = ", n) + local h = new_tab(0, n / 2) + for i = 1, n, 2 do + h[t[i]] = t[i + 1] + end + return h +end + + +-- this method is deperate since we already do lazy method generation. +function _M.add_commands(...) + local cmds = {...} + for i = 1, #cmds do + local cmd = cmds[i] + _M[cmd] = + function (self, ...) + return _do_cmd(self, cmd, ...) + end + end +end + + +setmetatable(_M, {__index = function(self, cmd) + local method = + function (self, ...) + return do_cmd(self, cmd, ...) + end + + -- cache the lazily generated method in our + -- module table + _M[cmd] = method + return method +end}) + + +return _M diff --git a/docker/config/nginx/sapl.conf b/docker/config/nginx/sapl.conf index 015538c96..9364650f4 100644 --- a/docker/config/nginx/sapl.conf +++ b/docker/config/nginx/sapl.conf @@ -1,10 +1,8 @@ upstream sapl_server { - - server unix:/var/interlegis/sapl/run/gunicorn.sock fail_timeout=0; - + server unix:/var/interlegis/sapl/run/gunicorn.sock fail_timeout=0; } -# Se o cliente já manda X-Request-ID, reaproveita; senão, usa $request_id (nginx) +# Reuse X-Request-ID from ingress if present; otherwise generate one. map $http_x_request_id $req_id { default $http_x_request_id; "" $request_id; @@ -18,52 +16,188 @@ server { client_max_body_size 4G; + # ---------------------------------------------------------------- + # Block known scraper ASNs (datacenter traffic) — zero Python cost. + # ---------------------------------------------------------------- + if ($bot_asn = 1) { + return 429 "Too Many Requests"; + } + + # ---------------------------------------------------------------- + # Block known bots by User-Agent — zero Python cost. + # ---------------------------------------------------------------- + if ($bot_ua_blocked = 1) { + return 429 "Too Many Requests"; + } + + # ---------------------------------------------------------------- + # Redis-backed IP blocklist (Lua): prefix SET, global IP block, + # and per-tenant API block — checked before reaching Gunicorn. + # ---------------------------------------------------------------- + access_by_lua_file /etc/nginx/blocklist.lua; + + # ---------------------------------------------------------------- + # robots.txt served directly by nginx. + # ---------------------------------------------------------------- + location = /robots.txt { + alias /var/interlegis/sapl/collected_static/robots.txt; + } + + # ---------------------------------------------------------------- + # Static files — no rate limiting, no proxy. + # ---------------------------------------------------------------- + location /static/ { + alias /var/interlegis/sapl/collected_static/; + expires 90m; + add_header Cache-Control "public, max-age=5400"; + } + + # ---------------------------------------------------------------- + # Media files — routed through Django for auth, rate counting, + # and content-type caching; served from disk via X-Accel-Redirect. + # ---------------------------------------------------------------- + location /media/ { + limit_req zone=sapl_media burst=${NGINX_BURST_MEDIA} nodelay; + limit_req_status 429; + + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; + } + + # Internal location used exclusively by X-Accel-Redirect responses + # from serve_media(). Not reachable by external clients. + location /internal/media/ { + internal; + alias /var/interlegis/sapl/media/; + sendfile on; + etag on; + } + + # ---------------------------------------------------------------- + # /relatorios/ — heaviest endpoint (PDF generation). + # Tighter rate limit; extended timeout for uncached generation. + # ---------------------------------------------------------------- + location /relatorios/ { + limit_req zone=sapl_heavy burst=${NGINX_BURST_HEAVY} nodelay; + limit_req_status 429; + + proxy_read_timeout 180s; + proxy_send_timeout 180s; + + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; + } + + # ---------------------------------------------------------------- + # /api/ — rate limited, CORS maintained from original config. + # ---------------------------------------------------------------- location /api/ { - add_header 'Access-Control-Allow-Origin' '*'; - add_header 'Access-Control-Allow-Credentials' 'true'; - add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, HEAD, OPTIONS'; - add_header 'Access-Control-Allow-Headers' 'Access-Control-Allow-Origin,XMLHttpRequest,Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Mx-ReqToken,X-Requested-With'; - add_header 'Access-Control-Expose-Headers' 'Access-Control-Allow-Origin,XMLHttpRequest,Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Mx-ReqToken,X-Requested-With'; - -# handle the browser's preflight steps - if ($request_method = 'OPTIONS') { - add_header 'Access-Control-Allow-Origin' '*'; - add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, HEAD, OPTIONS'; - add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range'; - add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range'; - add_header 'Access-Control-Max-Age' 1728000; - add_header 'Content-Type' 'text/plain; charset=utf-8'; - add_header 'Content-Length' 0; - return 204; + limit_req zone=sapl_api burst=${NGINX_BURST_API} nodelay; + limit_req_status 429; + + add_header 'Access-Control-Allow-Origin' '*' always; + add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, HEAD, OPTIONS' always; + add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always; + add_header 'Access-Control-Expose-Headers' 'Content-Type,X-RateLimit-Reason,Retry-After,X-Quota-Daily-Remaining,X-Quota-Weekly-Remaining' always; + + if ($request_method = 'OPTIONS') { + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, HEAD, OPTIONS'; + add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With'; + add_header 'Access-Control-Max-Age' 1728000; + add_header 'Content-Type' 'text/plain; charset=utf-8'; + add_header 'Content-Length' 0; + return 204; } - proxy_set_header X-Request-ID $req_id; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header Host $http_host; + proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://sapl_server; } - location /static/ { - alias /var/interlegis/sapl/collected_static/; + # ---------------------------------------------------------------- + # /painel//dados — high-frequency polling endpoint (will become + # WebSocket). No rate limiting at either layer (Django middleware + # also bypasses via RATE_LIMIT_BYPASS_PATHS). + # ---------------------------------------------------------------- + location ~ ^/painel/\d+/dados$ { + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; } - location /media/ { - alias /var/interlegis/sapl/media/; + # ---------------------------------------------------------------- + # Session voting paths — high-frequency during live votes; exempt + # from rate limiting. Multiple authenticated users share NAT IPs. + # Covers: /voto-individual/, /sessao//ordemdia, /sessao//expediente, + # /sessao//matordemdia/*, /sessao/pauta-sessao//, etc. + # ---------------------------------------------------------------- + location ~ ^/voto-individual/ { + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; } + location ~ ^/sessao/\d+ { + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; + } + + + # ---------------------------------------------------------------- + # Scanner extension probes (.php, .asp, etc.) — SAPL never serves + # these. Drop the connection before reaching Gunicorn. + # ---------------------------------------------------------------- + location ~* \.(php|asp|aspx|jsp|cgi|env|htaccess|htpasswd|bak|sql|sh|bash|py|rb|pl)$ { + return 444; + } + + # ---------------------------------------------------------------- + # General traffic — moderate rate limit. + # ---------------------------------------------------------------- location / { - proxy_set_header X-Request-ID $req_id; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + limit_req zone=sapl_general burst=${NGINX_BURST_GENERAL} nodelay; + limit_req_status 429; + + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header Host $http_host; + proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://sapl_server; } + error_page 429 /429.html; + location = /429.html { + add_header Retry-After 60 always; + root /var/interlegis/sapl/sapl/static/; + internal; + } + error_page 500 502 503 504 /500.html; location = /500.html { root /var/interlegis/sapl/sapl/static/; + internal; } } diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml index dc8559812..25d33159f 100644 --- a/docker/docker-compose.yaml +++ b/docker/docker-compose.yaml @@ -18,6 +18,7 @@ services: - "5433:5432" networks: - sapl-net + saplsolr: image: solr:8.11 restart: always @@ -32,11 +33,39 @@ services: - "8983:8983" networks: - sapl-net + + saplredis: + image: redis:7-alpine + restart: always + container_name: redis + labels: + NAME: "redis" + command: > + redis-server + --save "" + --appendonly no + --maxmemory 512mb + --maxmemory-policy allkeys-lru + --maxmemory-samples 10 + --maxclients 1000 + --timeout 300 + --tcp-keepalive 60 + --hz 20 + --lazyfree-lazy-eviction yes + --lazyfree-lazy-expire yes + --lazyfree-lazy-server-del yes + --databases 4 + --protected-mode no + ports: + - "6379:6379" + networks: + - sapl-net + sapl: - image: interlegis/sapl:3.1.165-RC2 -# build: -# context: ../ -# dockerfile: ./docker/Dockerfile +# image: interlegis/sapl:3.1.165-RC2 + build: + context: ../ + dockerfile: ./docker/Dockerfile container_name: sapl labels: NAME: "sapl" @@ -57,20 +86,25 @@ services: IS_ZK_EMBEDDED: 'True' ENABLE_SAPN: 'False' TZ: America/Sao_Paulo + REDIS_URL: redis://saplredis:6379 + CACHE_BACKEND: redis volumes: - sapl_data:/var/interlegis/sapl/data - sapl_media:/var/interlegis/sapl/media depends_on: - sapldb - saplsolr + - saplredis ports: - "80:80" networks: - sapl-net + networks: sapl-net: name: sapl-net driver: bridge + volumes: sapldb_data: sapl_data: diff --git a/docker/geoip/GeoLite2-ASN.mmdb b/docker/geoip/GeoLite2-ASN.mmdb new file mode 100644 index 000000000..c7b58b936 Binary files /dev/null and b/docker/geoip/GeoLite2-ASN.mmdb differ diff --git a/docker/geoip/update_geoip.sh b/docker/geoip/update_geoip.sh new file mode 100755 index 000000000..0e9e0a1f5 --- /dev/null +++ b/docker/geoip/update_geoip.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# update_geoip.sh — download / refresh GeoLite2-ASN.mmdb +# +# Run this script before building a new Docker image so the image bundles +# an up-to-date MaxMind ASN database. The .mmdb binary is git-ignored; +# only this script is tracked. +# +# Usage: +# # Option 1 — key in environment +# MAXMIND_LICENSE_KEY=your_key bash docker/geoip/update_geoip.sh +# +# # Option 2 — key in project .env file +# bash docker/geoip/update_geoip.sh +# +# The script writes GeoLite2-ASN.mmdb to the same directory as itself so +# the Dockerfile COPY step can find it at docker/geoip/GeoLite2-ASN.mmdb. +# +# Suggested automation: run via a host cron job or CI pipeline step +# before triggering a docker build, e.g.: +# +# # /etc/cron.weekly/update-sapl-geoip +# #!/bin/bash +# cd /path/to/sapl && bash docker/geoip/update_geoip.sh + +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OUT_FILE="$SCRIPT_DIR/GeoLite2-ASN.mmdb" + +# ── Resolve the license key ──────────────────────────────────────────────── +if [[ -z "${MAXMIND_LICENSE_KEY:-}" ]]; then + # Try the project .env (two directories up from docker/geoip/) + ENV_FILE="$(dirname "$(dirname "$SCRIPT_DIR")")/.env" + if [[ -f "$ENV_FILE" ]]; then + MAXMIND_LICENSE_KEY="$(grep -E '^MAXMIND_LICENSE_KEY=' "$ENV_FILE" 2>/dev/null \ + | cut -d= -f2- | tr -d '[:space:]' || true)" + fi +fi + +if [[ -z "${MAXMIND_LICENSE_KEY:-}" ]]; then + echo "ERROR: MAXMIND_LICENSE_KEY is not set." >&2 + echo " Set it in the environment or add MAXMIND_LICENSE_KEY= to .env" >&2 + exit 1 +fi + +# ── Download ─────────────────────────────────────────────────────────────── +URL="https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=${MAXMIND_LICENSE_KEY}&suffix=tar.gz" + +echo "[geoip] Downloading GeoLite2-ASN from MaxMind..." +tmpdir="$(mktemp -d)" +trap 'rm -rf "$tmpdir"' EXIT + +curl -fsSL --max-time 60 "$URL" | tar -xz --strip-components=1 -C "$tmpdir" +mv "$tmpdir"/GeoLite2-ASN.mmdb "$OUT_FILE" + +echo "[geoip] Saved: $OUT_FILE" +echo "[geoip] Build date: $(python3 -c " +import struct, datetime, pathlib +data = pathlib.Path('$OUT_FILE').read_bytes() +# MaxMind DB build epoch is in the last 16 bytes of the metadata section +marker = b'\xab\xcd\xefMaxMind.com' +idx = data.rfind(marker) +if idx >= 0: + # search for 'build_epoch' key nearby + chunk = data[idx:idx+512] + pos = chunk.find(b'build_epoch') + if pos >= 0: + val_start = pos + len(b'build_epoch') + 1 + epoch = struct.unpack('>Q', chunk[val_start+1:val_start+9])[0] + print(datetime.datetime.utcfromtimestamp(epoch).strftime('%Y-%m-%d')) + exit() +print('unknown') +" 2>/dev/null || echo "unknown")" diff --git a/docker/k8s/redis/redis-configmap.yaml b/docker/k8s/redis/redis-configmap.yaml new file mode 100644 index 000000000..4c2b8fe1a --- /dev/null +++ b/docker/k8s/redis/redis-configmap.yaml @@ -0,0 +1,36 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: redis-config + namespace: sapl-redis +data: + redis.conf: | + save "" + appendonly no + + maxmemory 5gb + maxmemory-policy allkeys-lru + maxmemory-samples 10 + + maxclients 20000 + tcp-backlog 511 + timeout 300 + tcp-keepalive 60 + + hz 20 + lazyfree-lazy-eviction yes + lazyfree-lazy-expire yes + lazyfree-lazy-server-del yes + + slowlog-log-slower-than 10000 + slowlog-max-len 256 + latency-monitor-threshold 10 + + bind 0.0.0.0 + protected-mode no + # DB0: cache DB1: rate limiter DB2: channels (future) + databases 2 + + activedefrag yes + active-defrag-ignore-bytes 100mb + active-defrag-threshold-lower 10 diff --git a/docker/k8s/redis/redis-deployment.yaml b/docker/k8s/redis/redis-deployment.yaml new file mode 100644 index 000000000..7a1a401cc --- /dev/null +++ b/docker/k8s/redis/redis-deployment.yaml @@ -0,0 +1,48 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sapl-redis + namespace: sapl-redis + labels: + app: sapl-redis +spec: + replicas: 1 + selector: + matchLabels: + app: sapl-redis + template: + metadata: + labels: + app: sapl-redis + spec: + containers: + - name: redis + image: redis:7-alpine + command: ["redis-server", "/etc/redis/redis.conf"] + resources: + requests: + memory: "1Gi" + cpu: "250m" + limits: + memory: "6Gi" + cpu: "1000m" + ports: + - containerPort: 6379 + livenessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 10 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 5 + periodSeconds: 10 + volumeMounts: + - name: redis-config + mountPath: /etc/redis + volumes: + - name: redis-config + configMap: + name: redis-config diff --git a/docker/k8s/redis/redis-service.yaml b/docker/k8s/redis/redis-service.yaml new file mode 100644 index 000000000..75307ff24 --- /dev/null +++ b/docker/k8s/redis/redis-service.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: redis + namespace: sapl-redis + labels: + app: sapl-redis +spec: + selector: + app: sapl-redis + ports: + - name: redis + port: 6379 + targetPort: 6379 + type: ClusterIP diff --git a/docker/k8s/sapl-k8s.yaml b/docker/k8s/sapl-k8s.yaml index 8a45e015e..15a15b7a5 100644 --- a/docker/k8s/sapl-k8s.yaml +++ b/docker/k8s/sapl-k8s.yaml @@ -189,16 +189,9 @@ spec: image: eribeiro/sapl:debug-k8s-1 ports: - containerPort: 80 - volumeMounts: - - name: data - mountPath: /var/interlegis/sapl/data - readOnly: true # secrets are always mounted read-only - volumes: - - name: data - secret: - secretName: sapl-secretkey - defaultMode: 0440 # ensures read-only env: + - name: REDIS_URL + value: "redis://redis.sapl-redis.svc.cluster.local:6379" - name: ADMIN_PASSWORD value: "interlegis" - name: ADMIN_EMAIL @@ -214,5 +207,12 @@ spec: - name: EMAIL_HOST_USER value: "usuariosmtp" - name: EMAIL_SEND_USER - - + volumeMounts: + - name: data + mountPath: /var/interlegis/sapl/data + readOnly: true # secrets are always mounted read-only + volumes: + - name: data + secret: + secretName: sapl-secretkey + defaultMode: 0440 # ensures read-only diff --git a/docker/scripts/redis_inject_test_data.py b/docker/scripts/redis_inject_test_data.py new file mode 100644 index 000000000..94dc0a1fe --- /dev/null +++ b/docker/scripts/redis_inject_test_data.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +""" +redis_inject_test_data.py — inject synthetic rate-limiter entries into Redis. + +Purpose: validate that RateLimitMiddleware reads the expected key schema, +that Redis CLI / RedisInsight shows the right structure, and that blocking +logic fires correctly without waiting for real traffic. + +Usage: + # Against docker-compose Redis (default) + python3 docker/scripts/redis_inject_test_data.py + + # Against a different host/port + REDIS_URL=redis://localhost:6379 python3 docker/scripts/redis_inject_test_data.py + + # Clear all synthetic keys written by a previous run + CLEAR=1 python3 docker/scripts/redis_inject_test_data.py + +Key schema (DB 1 — rate limiter): + rl:ip:{ip}:reqs INCR counter — anonymous request count (TTL 60s) + rl:ip:{ip}:blocked string "1" — IP hard-blocked (TTL 300s) + rl:{ns}:user:{uid}:reqs INCR counter — auth user request count (TTL 60s) + rl:{ns}:user:{uid}:blocked string "1" — user hard-blocked (TTL 300s) + rl:{ns}:ip:{ip}:w:{bucket} INCR — namespace/IP sliding window (TTL 120s) +""" + +import os +import sys +import time +from decouple import config + +# ── dependency check ────────────────────────────────────────────────────── +try: + import redis +except ImportError: + print("ERROR: redis-py not installed. Run: pip install redis", file=sys.stderr) + sys.exit(1) + +# ── config ──────────────────────────────────────────────────────────────── +REDIS_URL = config("REDIS_URL", default="redis://localhost:6379") +RATELIMIT_DB = 1 # DB1 is the rate-limiter database +CLEAR = config("CLEAR", default="0").lower() in ("1", "true", "yes") + +# Synthetic values — tweak to exercise different code paths +NAMESPACE = "sapl" # POD_NAMESPACE value (hostname or k8s namespace) +ANON_WINDOW = 60 # seconds — must match settings.RATE_LIMITER_RATE period +AUTH_WINDOW = 60 +BLOCK_TTL = 300 + +TEST_IPS = [ + "203.0.113.1", # below threshold (20 reqs) + "203.0.113.2", # AT threshold (35 reqs — should trigger block) + "203.0.113.3", # already blocked + "203.0.113.4", # namespace/window counter near threshold +] + +TEST_USERS = [ + {"uid": "42", "reqs": 50, "blocked": False}, # normal auth user + {"uid": "99", "reqs": 120, "blocked": False}, # AT auth threshold + {"uid": "7", "reqs": 10, "blocked": True}, # pre-blocked user +] + +# ── helpers ─────────────────────────────────────────────────────────────── + +def key_ip_reqs(ip): + return f"rl:ip:{ip}:reqs" + +def key_ip_blocked(ip): + return f"rl:ip:{ip}:blocked" + +def key_user_reqs(ns, uid): + return f"rl:{ns}:user:{uid}:reqs" + +def key_user_blocked(ns, uid): + return f"rl:{ns}:user:{uid}:blocked" + +def key_ns_window(ns, ip, bucket): + return f"rl:{ns}:ip:{ip}:w:{bucket}" + + +def write(r, key, value, ttl, label): + if isinstance(value, int): + pipe = r.pipeline() + pipe.set(key, value, ex=ttl) + pipe.execute() + else: + r.set(key, value, ex=ttl) + print(f" SET {key!r} = {value!r} EX {ttl}s ({label})") + + +def delete_pattern(r, pattern): + keys = r.keys(pattern) + if keys: + r.delete(*keys) + print(f" DEL {len(keys)} keys matching {pattern!r}") + else: + print(f" (no keys matching {pattern!r})") + + +# ── main ────────────────────────────────────────────────────────────────── + +def main(): + r = redis.from_url(REDIS_URL, db=RATELIMIT_DB, decode_responses=True) + try: + r.ping() + except redis.ConnectionError as exc: + print(f"ERROR: cannot connect to Redis at {REDIS_URL}: {exc}", file=sys.stderr) + sys.exit(1) + + print(f"Redis: {REDIS_URL} DB={RATELIMIT_DB} clear={CLEAR}") + print() + + # ── clear mode ──────────────────────────────────────────────────────── + if CLEAR: + print("=== Clearing synthetic test keys ===") + for ip in TEST_IPS: + delete_pattern(r, f"rl:ip:{ip}:*") + delete_pattern(r, f"rl:{NAMESPACE}:ip:{ip}:*") + for u in TEST_USERS: + delete_pattern(r, f"rl:{NAMESPACE}:user:{u['uid']}:*") + print("Done.") + return + + # ── anonymous IP counters ───────────────────────────────────────────── + print("=== Anonymous IP request counters (DB1) ===") + write(r, key_ip_reqs(TEST_IPS[0]), 20, ANON_WINDOW, "below threshold") + write(r, key_ip_reqs(TEST_IPS[1]), 35, ANON_WINDOW, "AT threshold → middleware will block on next req") + write(r, key_ip_reqs(TEST_IPS[3]), 30, ANON_WINDOW, "below threshold") + print() + + # ── blocked IPs ─────────────────────────────────────────────────────── + print("=== Blocked IPs (DB1) ===") + write(r, key_ip_blocked(TEST_IPS[2]), "1", BLOCK_TTL, "hard-blocked") + print() + + # ── namespace/IP sliding window ─────────────────────────────────────── + print("=== Namespace/IP sliding window (DB1) ===") + bucket = int(time.time() // ANON_WINDOW) + write(r, key_ns_window(NAMESPACE, TEST_IPS[3], bucket), 34, ANON_WINDOW * 2, + "near window threshold (next req triggers ua_rotation block)") + print() + + # ── authenticated user counters ─────────────────────────────────────── + print("=== Authenticated user request counters (DB1) ===") + for u in TEST_USERS: + if not u["blocked"]: + write(r, key_user_reqs(NAMESPACE, u["uid"]), u["reqs"], AUTH_WINDOW, + f"uid={u['uid']} reqs={u['reqs']}") + print() + + # ── blocked users ───────────────────────────────────────────────────── + print("=== Blocked users (DB1) ===") + for u in TEST_USERS: + if u["blocked"]: + write(r, key_user_blocked(NAMESPACE, u["uid"]), "1", BLOCK_TTL, + f"uid={u['uid']} hard-blocked") + print() + + # ── summary ─────────────────────────────────────────────────────────── + all_keys = r.keys("rl:*") + print(f"=== DB{RATELIMIT_DB} now contains {len(all_keys)} rl:* keys ===") + for k in sorted(all_keys): + ttl = r.ttl(k) + val = r.get(k) + print(f" {k!r:55s} val={val!r:5} ttl={ttl}s") + + +if __name__ == "__main__": + main() diff --git a/docker/startup_scripts/gunicorn.conf.py b/docker/startup_scripts/gunicorn.conf.py index 6bdcacb02..795a1d817 100644 --- a/docker/startup_scripts/gunicorn.conf.py +++ b/docker/startup_scripts/gunicorn.conf.py @@ -10,9 +10,9 @@ DJANGODIR = "/var/interlegis/sapl" SOCKFILE = f"unix:{DJANGODIR}/run/gunicorn.sock" USER = "sapl" GROUP = "nginx" -NUM_WORKERS = int(os.getenv("WEB_CONCURRENCY", "3")) -THREADS = int(os.getenv("GUNICORN_THREADS", "8")) -TIMEOUT = int(os.getenv("GUNICORN_TIMEOUT", "300")) +NUM_WORKERS = int(os.getenv("WEB_CONCURRENCY", "2")) # was 3 +THREADS = int(os.getenv("GUNICORN_THREADS", "4")) # was 8 +TIMEOUT = int(os.getenv("GUNICORN_TIMEOUT", "120")) # was 300 MAX_REQUESTS = 1000 WORKER_CLASS = "gthread" DJANGO_SETTINGS = "sapl.settings" @@ -36,7 +36,7 @@ chdir = DJANGODIR wsgi_app = WSGI_APP # Logs -loglevel = "debug" +loglevel = "info" # was debug — reduces log I/O accesslog = "/var/log/sapl/access.log" errorlog = "/var/log/sapl/error.log" # errorlog = "-" # send to stderr (so you see it in docker logs or terminal) @@ -53,7 +53,7 @@ keepalive = 10 backlog = 2048 max_requests = MAX_REQUESTS max_requests_jitter = 200 -worker_max_memory_per_child = 300 * 1024 * 1024 # 300 MB cap +worker_max_memory_per_child = 400 * 1024 * 1024 # 400 MB — was 300 MB # Environment (same as exporting before running) raw_env = [ diff --git a/docker/startup_scripts/start.sh b/docker/startup_scripts/start.sh index bd98bdfc0..b2fc83f11 100755 --- a/docker/startup_scripts/start.sh +++ b/docker/startup_scripts/start.sh @@ -104,6 +104,16 @@ write_env_file() { : "${RF:=1}" : "${MAX_SHARDS_PER_NODE:=1}" : "${ENABLE_SAPN:=False}" + : "${REDIS_URL:=}" + : "${CACHE_BACKEND:=file}" + : "${POD_NAMESPACE:=sapl}" + # nginx burst defaults — 2× each zone's sustained rate. + # general=90r/m media=180r/m api=60r/m heavy=10r/m + : "${NGINX_BURST_GENERAL:=180}" + : "${NGINX_BURST_MEDIA:=180}" + : "${NGINX_BURST_API:=120}" + : "${NGINX_BURST_HEAVY:=20}" + export NGINX_BURST_GENERAL NGINX_BURST_MEDIA NGINX_BURST_API NGINX_BURST_HEAVY tmp="$(mktemp)" { @@ -126,6 +136,13 @@ write_env_file() { printf 'RF=%s\n' "$RF" printf 'MAX_SHARDS_PER_NODE=%s\n' "$MAX_SHARDS_PER_NODE" printf 'ENABLE_SAPN=%s\n' "$ENABLE_SAPN" + printf 'REDIS_URL=%s\n' "$REDIS_URL" + printf 'CACHE_BACKEND=%s\n' "$CACHE_BACKEND" + printf 'POD_NAMESPACE=%s\n' "$POD_NAMESPACE" + printf 'NGINX_BURST_GENERAL=%s\n' "$NGINX_BURST_GENERAL" + printf 'NGINX_BURST_MEDIA=%s\n' "$NGINX_BURST_MEDIA" + printf 'NGINX_BURST_API=%s\n' "$NGINX_BURST_API" + printf 'NGINX_BURST_HEAVY=%s\n' "$NGINX_BURST_HEAVY" } > "$tmp" chmod 600 "$tmp" @@ -135,6 +152,7 @@ write_env_file() { wait_for_pg() { : "${DATABASE_URL:=postgresql://sapl:sapl@sapldb:5432/sapl}" + export DATABASE_URL log "Waiting for Postgres..." /bin/bash wait-for-pg.sh "$DATABASE_URL" } @@ -256,19 +274,94 @@ setup_cache_dir() { umask 0007 } +# --------------------------------------------------------------------------- +# Tenant namespace — resolved once at startup, written into .env +# --------------------------------------------------------------------------- + +resolve_pod_namespace() { + # 1. Already set by K8s Downward API (fieldRef: metadata.namespace) + [[ -n "${POD_NAMESPACE:-}" ]] && { log "POD_NAMESPACE=${POD_NAMESPACE} (from env)."; return 0; } + + # 2. K8s service-account namespace file (present in every in-cluster pod) + local ns_file="/var/run/secrets/kubernetes.io/serviceaccount/namespace" + if [[ -f "$ns_file" ]]; then + export POD_NAMESPACE="$(<"$ns_file")" + log "POD_NAMESPACE=${POD_NAMESPACE} (from service-account file)." + return 0 + fi + + # 3. Fallback for local development + export POD_NAMESPACE="sapl" + log "POD_NAMESPACE not found — using fallback '${POD_NAMESPACE}'." +} + +# --------------------------------------------------------------------------- +# Redis — check URL from deployment env, waffle switch, connectivity +# --------------------------------------------------------------------------- + +# 1. Log whether REDIS_URL was provided via the deployment env. +resolve_redis_url() { + if [[ -n "${REDIS_URL:-}" ]]; then + log "REDIS_URL set: $REDIS_URL" + else + log "REDIS_URL not set — file-based cache will be used." + fi +} + +# 2. Create/reset the REDIS_CACHE waffle switch; set CACHE_BACKEND accordingly. +configure_redis_cache() { + ./manage.py waffle_switch REDIS_CACHE off --create || true + if [[ -z "${REDIS_URL:-}" ]]; then + log "REDIS_URL not set — REDIS_CACHE switch OFF." + return 0 + fi + ./manage.py waffle_switch REDIS_CACHE on --create || true + export CACHE_BACKEND="redis" + log "REDIS_URL set — REDIS_CACHE switch ON." +} + +# 4. Block until Redis is reachable (or give up gracefully). +wait_for_redis() { + [[ -z "${REDIS_URL:-}" ]] && return 0 + [[ "${CACHE_BACKEND:-file}" != "redis" ]] && return 0 + log "Checking Redis connectivity..." + local host port retries=10 + host=$(python3 -c "from urllib.parse import urlparse; u=urlparse('${REDIS_URL}'); print(u.hostname or 'localhost')") + port=$(python3 -c "from urllib.parse import urlparse; u=urlparse('${REDIS_URL}'); print(u.port or 6379)") + until python3 -c "import socket; s=socket.create_connection(('$host',$port),2); s.close()" 2>/dev/null; do + retries=$((retries - 1)) + if [[ $retries -eq 0 ]]; then + log "WARNING: Redis unreachable after retries — falling back to file cache." + export CACHE_BACKEND="file" + return 0 + fi + log "Waiting for Redis at $host:$port... ($retries retries left)" + sleep 2 + done + log "Redis reachable at $host:$port." +} + start_services() { log "Starting gunicorn..." gunicorn -c gunicorn.conf.py & + log "Applying nginx config (burst: general=${NGINX_BURST_GENERAL} media=${NGINX_BURST_MEDIA} api=${NGINX_BURST_API} heavy=${NGINX_BURST_HEAVY})..." + envsubst '${NGINX_BURST_GENERAL} ${NGINX_BURST_MEDIA} ${NGINX_BURST_API} ${NGINX_BURST_HEAVY}' \ + < /etc/nginx/conf.d/sapl.conf.template \ + > /etc/nginx/conf.d/sapl.conf log "Starting nginx..." exec /usr/sbin/nginx -g "daemon off;" } main() { create_secret - write_env_file + resolve_pod_namespace + resolve_redis_url wait_for_pg configure_pg_timezone migrate_db + configure_redis_cache + wait_for_redis + write_env_file # writes resolved REDIS_URL + CACHE_BACKEND into .env configure_solr || true configure_sapn create_admin diff --git a/docs/rate-limiter-incidents.md b/docs/rate-limiter-incidents.md new file mode 100644 index 000000000..9171612f9 --- /dev/null +++ b/docs/rate-limiter-incidents.md @@ -0,0 +1,503 @@ +# Rate Limiter Incidents + +This document records real rate-limiting incidents, the root-cause analysis performed for each, the fixes applied, and the architectural discussion that followed. New incidents should be appended under their own section. + +--- + +## PatoBranco-PR — 2026-05-06 + +### Symptom + +Councilmembers reported being unable to access the voting interface during a live plenary session. The error was HTTP 429. Two blocking events occurred: + +| Event | Start | Recovery | Duration | +|-------|-------|----------|----------| +| 1 | 13:51:23 | ~14:01 | ~10 min | +| 2 | 14:22:30 | ~14:25 | ~3 min | + +Both recovered **before** the Django `BLOCK_TTL` of 300 seconds, which was the first diagnostic clue. + +### Environment + +- NAT IP: `200.175.17.66` (reported range `200.175.17.66/29`) +- Secondary range: `187.109.99.234/30` +- Peak observed: **24 requests/second** from that IP (confirmed in OpenSearch, 14:22:31) +- Paths involved: `/voto-individual/`, `/sessao/pauta-sessao/2600/`, `/sessao/2600/ordemdia` + +### Root Cause + +Multiple councilmembers share a single public IP via NAT. When a vote opened, all of them reloaded their browser simultaneously. Nginx saw the combined traffic as a single client exhausting its burst bucket and returned 429 — before any request reached Django. + +``` + ┌─────────────────────────────────────────┐ + │ nginx │ + │ │ + Councilmember A ──►│ IP: 200.175.17.66 │ + Councilmember B ──►│ IP: 200.175.17.66 ──► burst bucket │──► 429 (bucket full) + Councilmember C ──►│ IP: 200.175.17.66 exhausted │ + ... │ │ + └─────────────────────────────────────────┘ + │ + │ (never reached) + ▼ + ┌─────────────────────────────────────────┐ + │ Django middleware │ + │ │ + │ rl:ip::reqs (never incremented)│ + │ rl:user::reqs (never incremented)│ + │ Redis block key (never written) │ + └─────────────────────────────────────────┘ +``` + +### Why Recovery Was Faster Than 300 Seconds + +Django's block mechanism (`_set_block()`) was **never triggered**. The NAT IP was never written to Redis. The 429s came entirely from nginx's token bucket being exhausted. + +Recovery happened when the synchronized burst subsided (vote ended, users stopped reloading). The nginx bucket refilled at its configured rate. No TTL expiry was involved — recovery time was variable because it depended on how depleted the bucket was at the end of each burst, not on a fixed timer. + +Had Django's block fired, the outage would have been exactly 300 seconds both times. The variable durations (10 min vs 3 min) confirm nginx was the sole actor. + +### The Polling Source + +`voto_individual.html` contains a `setTimeout(location.reload, 30000)` — the page reloads itself every 30 seconds. When councilmembers opened the voting page at roughly the same time (vote announcement), their reload timers aligned. Each 30-second tick fired a synchronized burst from all clients behind the NAT. + +`/sessao//ordemdia` and `/sessao/pauta-sessao//` are not polled by JavaScript — they are normal page navigations. They appeared in the burst because councilmembers navigated to them at the same moment as the vote opened. + +### Two-Layer Rate Limiting Architecture + +``` + ┌──────────────────────────────────────────────────────┐ + Incoming request │ nginx │ + ─────────────────────►│ │ + │ limit_req zone=sapl_general ← IP-only, no auth │ + │ burst=${NGINX_BURST_GENERAL} nodelay │ + │ │ + │ If bucket full → 429 immediately │ + │ Redis: nothing written │ + └───────────────────────┬──────────────────────────────┘ + │ (only if bucket has room) + ▼ + ┌──────────────────────────────────────────────────────┐ + │ Django RateLimitMiddleware │ + │ │ + │ 1. Bypass check ← RATE_LIMIT_BYPASS_PATHS │ + │ 2. API quota check (if /api/) │ + │ 3. _evaluate() │ + │ a. IP block check (Redis rl:ip::blocked) │ + │ b. User block check (Redis rl:user::blocked) │ + │ c. Rate counter (rl:ip::reqs) │ + │ d. User counter (rl:user::reqs) │ + │ │ + │ If rate exceeded → SET block key (TTL=300s) │ + │ → ZADD rl:index:blocked_ips │ + └──────────────────────────────────────────────────────┘ +``` + +**The core mismatch:** Django tracks per-user buckets (`rl:user::reqs`) which are NAT-safe. Nginx tracks per-IP buckets which collapse all users behind a NAT into one. Nginx fires first, so Django's smarter per-user accounting is never consulted during a burst. + +### Fix Applied + +Added nginx `location` blocks for session and voting paths that pass requests through **without** `limit_req`. These regex locations take priority over the catch-all `location /` by nginx matching rules. + +**`docker/config/nginx/sapl.conf`:** +```nginx +location ~ ^/voto-individual/ { + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; +} + +location ~ ^/sessao/\d+ { + proxy_set_header X-Request-ID $req_id; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; +} +``` + +**`sapl/settings.py`** — `RATE_LIMIT_BYPASS_PATHS` extended to match: +```python +RATE_LIMIT_BYPASS_PATHS = [ + r'^/painel/\d+/dados$', + r'^/voto-individual/', + r'^/sessao/\d+', + r'^/sessao/pauta-sessao/\d+/', +] +``` + +These paths are safe to exempt because: +- They require an authenticated session cookie to perform any meaningful action. +- Django's per-user rate counter still runs as a backstop. +- The cost of a false-positive block (councilmember unable to vote) outweighs the risk of abuse on these URLs. + +--- + +## Architectural Discussion + +### Why Increasing Burst Rates Is Not the Right Fix + +`burst` controls how many requests above the sustained rate are allowed in a spike before 429 fires. A larger burst absorbs thundering herds from NAT but also allows more requests from burst-style attackers (scanners, credential stuffers) before the throttle engages. It is the same knob, pulled in opposite directions. + +The correct dimensioning formula for a legislative house would be: + +``` +burst ≥ users_behind_NAT × tabs_per_user × requests_per_page_load +``` + +For a large state assembly (90 deputies, 2 tabs each) this exceeds 180 — a burst value that renders rate limiting ineffective against automated tools. + +**Conclusion:** burst tuning is not the right tool for this problem. Exemption of known-safe high-frequency paths is. + +### The Multi-Tab Problem + +Staff members commonly open multiple SAPL tabs simultaneously. Each tab has its own reload timer. If 10 staff members have 3 tabs open, a synchronized event generates 30+ requests from the NAT IP even before councilmembers are counted. This means the bucket can be pre-exhausted before the voting burst even starts. + +This further reinforces that IP-based rate limiting is the wrong unit for authenticated traffic in a shared-office environment. + +### Why Per-User Rate Limiting Does Not Fully Solve This + +Django already increments both `rl:ip::reqs` and `rl:user::reqs`. The per-user counter is NAT-safe. But it never runs during a burst because nginx drops the request first. + +Moving rate limiting for authenticated users to the Django layer only (removing nginx `limit_req` for authenticated paths) would make per-user counting the effective control. The obstacle is that nginx cannot distinguish authenticated from anonymous requests without reading and resolving the session cookie — which requires a database or Redis lookup nginx cannot perform natively. + +### Architectural Solutions + +| Approach | Effort | Solves NAT problem | Protects against bots | +|----------|--------|-------------------|----------------------| +| Nginx bypass for known session paths *(done)* | Low | Yes, for bypassed paths | Yes, general paths still rate-limited | +| Increase `NGINX_BURST_GENERAL` | Trivial | Partially | Weakens bot protection | +| Nginx `limit_req_zone` keyed on session cookie string | Medium | Yes (per-session token, not per-IP) | Yes, each session has own bucket | +| Move all auth-path rate limiting to Django only | Medium | Yes | Depends on Django rate correctly tuned | +| Replace `setTimeout(location.reload)` with WebSocket/SSE push | High | Yes — eliminates synchronized reloads entirely | N/A | + +### WebSocket / SSE Consideration + +The 30-second self-reload in `voto_individual.html` is the synchronization mechanism that creates the thundering herd. If the server pushed state changes (vote opened, result published) instead of the client polling by reloading, the synchronized burst would not exist regardless of how many users or tabs are open. + +A full WebSocket rewrite (Django Channels + Redis pub/sub channel layer) would: +- Eliminate polling bursts on session/voting paths +- Make vote-state updates instantaneous instead of up to 30 seconds late +- Require nginx configuration for `Upgrade: websocket` proxying + +The nginx bypass is the correct operational fix for now. The WebSocket rewrite is the correct architectural fix for the future. They are not substitutes — the bypass would remain useful for the initial WebSocket handshake, which is still an HTTP request subject to burst limits. + +--- + +## Diagrams — Issues, Solutions, and Trade-offs (2026-05-06/07) + +--- + +### 1. NAT Thundering Herd — Before the Fix + +During a live vote all councilmembers reload simultaneously. nginx sees one +IP, exhausts its bucket, and returns 429 before Django is ever involved. +Django's per-user counter (NAT-safe) is never consulted. + +``` + Office / Chamber — behind one NAT IP (200.175.17.66) + ┌──────────────────────────────────────────────────────┐ + │ Councilmember A browser reload ──┐ │ + │ Councilmember B browser reload ──┤ │ + │ Councilmember C browser reload ──┤ ~24 req/s │ + │ Staff tab 1 browser reload ──┤ same public IP │ + │ Staff tab 2 browser reload ──┘ │ + └────────────────────────────┬─────────────────────────┘ + │ all requests look identical to nginx + ▼ + ┌─────────────────────────────────────┐ + │ nginx sapl_general │ + │ rate=30r/m burst=60 nodelay │ + │ │ + │ token bucket: 0 tokens remaining │ + │ → 429 returned immediately │ + └──────────────────┬──────────────────┘ + │ + ╳ Django never reached + ╳ rl:ip:{ip}:reqs never incremented + ╳ rl:user:{uid}:reqs never incremented + ╳ per-user NAT-safe counter never consulted + │ + ▼ + 429 for all N users in the org + recovery: wait for nginx bucket refill + (~3–10 min depending on depletion) + NOT a Django 300s block (Redis never written) +``` + +--- + +### 2. NAT Thundering Herd — After the Session Bypass Fix + +Session and voting paths have dedicated nginx `location` blocks with no +`limit_req`. Regex locations take priority over `location /`. + +``` + Office / Chamber — behind one NAT IP + ┌──────────────────────────────────────────────────────┐ + │ Councilmember A /voto-individual/ reload ──┐ │ + │ Councilmember B /voto-individual/ reload ──┤ │ + │ Councilmember C /sessao/2600/ordemdia ───────┤ │ + │ Staff tab /sessao/pauta-sessao/2600/ ──┘ │ + └────────────────────────────┬─────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ nginx │ + │ │ + │ location ~ ^/voto-individual/ ─┐ │ + │ location ~ ^/sessao/\d+ ─┤ │ no limit_req + │ location ~ ^/painel/\d+/dados ─┘ │ pass through + └──────────────────┬──────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ Django RateLimitMiddleware │ + │ │ + │ RATE_LIMIT_BYPASS_PATHS match? │ + │ → yes: return get_response() │ + │ (no counter, no block check) │ + └──────────────────┬──────────────────┘ + │ + ▼ + ✓ View served + All N users get their page +``` + +--- + +### 3. nginx Zone Architecture — Before vs After + +**Before (single shared zone)** + +``` + All traffic (HTML + media + API) + │ + ▼ + ┌───────────────────────────────┐ + │ sapl_general │ ← one bucket per IP + │ rate=30r/m burst=60 │ + │ │ + │ /media/page.pdf ──────────┐ │ ← media request drains + │ /materia/123/ ──────────┤ │ the same bucket as + │ /api/materia/? ──────────┘ │ the HTML page + └───────────────────────────────┘ + + Problem: a page with 20 media attachments + burns 20 tokens from the page-load budget +``` + +**After (four independent zones)** + +``` + ┌─────────────────┐ location / + │ sapl_general │ rate=90r/m burst=180 — HTML page requests + └─────────────────┘ + + ┌─────────────────┐ location /media/ + │ sapl_media │ rate=180r/m burst=180 — media downloads + └─────────────────┘ own bucket, never + drains page quota + ┌─────────────────┐ location /api/ + │ sapl_api │ rate=60r/m burst=120 — API calls + └─────────────────┘ quota layer is + real constraint + ┌─────────────────┐ location /relatorios/ + │ sapl_heavy │ rate=10r/m burst=20 — PDF generation + └─────────────────┘ nodelay tight by design + + Session/voting paths: NO zone — exempt from limit_req entirely + Static files /static/: NO zone — served directly from disk +``` + +--- + +### 4. Anonymous /api/ NAT Problem — Before vs After + +**Before** + +``` + Office — 10 staff, JS polling /api/ every 5s = 120 req/min combined + │ + ▼ + ┌─────────────────────────────────────┐ + │ nginx sapl_general (was shared) │ + │ burst not yet exhausted → pass │ + └──────────────────┬──────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ Django _evaluate_anonymous │ + │ │ + │ INCR rl:ip:{ip}:reqs │ + │ count = 120 ≥ threshold (120/m) │ + │ │ + │ SET rl:ip:{ip}:blocked EX 300 │◄── block key written + │ ZADD rl:index:blocked_ips │ affects ALL paths + └──────────────────┬──────────────────┘ + │ + ▼ + Next request to /materia/, /sessao/, /voto-individual/ ... + → 429 ip_blocked (300s) + Entire org locked out of ALL SAPL pages + because of JS polling the API +``` + +**After** + +``` + Office — 10 staff, JS polling /api/ every 5s = 120 req/min combined + │ + ▼ + ┌─────────────────────────────────────┐ + │ nginx sapl_api │ + │ rate=60r/m burst=120 nodelay │ + │ throttles burst, passes remainder │ + └──────────────────┬──────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ Django: quota check │ + │ 500 req/day not exceeded → pass │ + └──────────────────┬──────────────────┘ + │ + ▼ + ┌─────────────────────────────────────┐ + │ Anonymous /api/ early return │ + │ │ + │ rl:ip:{ip}:reqs NOT incremented │◄── no counter + │ rl:ip:{ip}:blocked NOT written │◄── no block key + └──────────────────┬──────────────────┘ + │ + ▼ + ✓ View served + Page requests from same IP unaffected +``` + +--- + +### 5. Authenticated Rate Breach — Before vs After + +**Before** + +``` + Authenticated user clicks rapidly: 241 requests in 60s + │ + ▼ + ┌─────────────────────────────────────┐ + │ _evaluate_authenticated │ + │ INCR rl:user:{uid}:reqs → 241 │ + │ count ≥ 240 (auth threshold) │ + │ │ + │ SET rl:user:{uid}:blocked EX 300 │◄── 5-minute lockout + │ ZADD rl:index:blocked_users │ + └──────────────────┬──────────────────┘ + │ + ▼ + All requests for 300s → 429 user_blocked + User must wait 5 minutes to do anything + No way to self-recover sooner +``` + +**After** + +``` + Authenticated user clicks rapidly: 241 requests in 60s + │ + ▼ + ┌─────────────────────────────────────┐ + │ _evaluate_authenticated │ + │ INCR rl:user:{uid}:reqs → 241 │ + │ count ≥ 240 (auth threshold) │ + │ │ + │ return 429 auth_user_rate │◄── this request only + │ (no SET, no ZADD) │◄── no block key written + └──────────────────┬──────────────────┘ + │ + Counter TTL = 60s (auth_window) + │ + ▼ + T+60s: rl:user:{uid}:reqs expires + User automatically recovers + No admin intervention needed +``` + +--- + +### 6. Enforcement Stack Per Path — Trade-off Summary + +``` +Path nginx zone Django counter Block written? Notes +────────────────────── ─────────────── ──────────────── ────────────── ────────────────────────────── +/static/* none none — disk-served, zero Django cost +/painel//dados none none — bypass: high-freq polling +/voto-individual/* none none — bypass: live vote +/sessao//* none none — bypass: live session +/sessao/pauta-sessao/* none none — bypass: live session +/media/* sapl_media anon IP / auth anon: yes auth gate in serve_media() + 180r/m b=180 counter runs auth: no +/api/* (anonymous) sapl_api quota only no ← key change: no IP counter, + 60r/m b=120 500/day — no collateral NAT block +/api/* (authenticated) sapl_api per-user 240/m no (soft) per-user, NAT-safe + 60r/m b=120 counter runs +/relatorios/* sapl_heavy anon/auth runs anon: yes tight rate — PDF generation + 10r/m b=20 at Django auth: no +/* (everything else) sapl_general anon/auth runs anon: yes normal page navigation + 90r/m b=180 at Django auth: no auth gets 240/m soft limit +``` + +**Legend:** +- `anon: yes` — anonymous IP gets a 300s block key on breach +- `auth: no` — authenticated users get 429 for that request, window resets in 60s, no persistent block +- `none` — no rate limiting at either layer (path is exempt) + +--- + +### 7. The Fundamental NAT Constraint + +``` + IP-based rate limiting cannot distinguish these two scenarios: + + Scenario A — Legitimate (15 users, 1 tab each, vote opens) + ┌──────────────────────────────────────────────────────┐ + │ User 1 ──► GET /voto-individual/ │ + │ User 2 ──► GET /voto-individual/ 15 req/s │ + │ ... 1 public IP │ + │ User 15 ──► GET /sessao/2600/ordemdia │ + └──────────────────────────────────────────────────────┘ + + Scenario B — Bot (1 process, 15 threads, scraping) + ┌──────────────────────────────────────────────────────┐ + │ Thread 1 ──► GET /materia/1/ │ + │ Thread 2 ──► GET /materia/2/ 15 req/s │ + │ ... 1 public IP │ + │ Thread 15 ──► GET /materia/15/ │ + └──────────────────────────────────────────────────────┘ + + To nginx and an IP-based counter: identical. + + Resolution strategies applied: + ┌──────────────────────────────────────────────────────────────────┐ + │ Known safe high-freq paths → nginx bypass + Django bypass │ + │ Authenticated users → per-user counter (uid), NAT-safe │ + │ Anonymous /api/ → quota only, no IP counter │ + │ Everything else (anon) → IP counter + 300s block on breach │ + └──────────────────────────────────────────────────────────────────┘ + + Long-term: APP_ACCESS_KEYs per tenant → quota per org, not per IP + WebSocket push for voting → eliminates polling bursts +``` + +--- + +## Pending Investigations + +The following incidents may or may not share the same root cause as the PatoBranco-PR event. Each should be investigated using the OpenSearch query patterns established above and documented here. + +- [ ] Other houses reporting intermittent 429s during session hours +- [ ] Azure crawler bot (`52.167.144.162`) — 2 × 429 observed on 2026-05-06 at patobranco-pr; appears to be a legitimate Microsoft indexer hitting non-session paths; confirm it is correctly rate-limited and not causing collateral blocks on shared IPs +- [ ] Investigate whether `187.109.99.234/30` (patobranco secondary NAT range) experienced any blocks independently of `200.175.17.66/29` diff --git a/plan/RATE-LIMITER-PLAN.md b/plan/RATE-LIMITER-PLAN.md new file mode 100644 index 000000000..adfc27cb7 --- /dev/null +++ b/plan/RATE-LIMITER-PLAN.md @@ -0,0 +1,1671 @@ +# SAPL — Rate Limiter & Redis Operations + +> **Scope**: Django / Gunicorn / nginx / Kubernetes fleet of 1,200+ pods. +> Each pod has a dedicated PostgreSQL instance. A K8s Ingress sits in front of all tenants. +> **This document is canonical** — all earlier session notes are consolidated here. + +--- + +## Context & Problem Statement + +### Fleet + +| Item | Detail | +|------|--------| +| System | SAPL — Django 2.2, legislative management for Brazilian municipal chambers | +| Fleet | ~1,200 Kubernetes pods, each with a dedicated PostgreSQL pod | +| Pod limits | 1 core CPU (limit) / 35m (request) · 1600Mi RAM (limit) / 800Mi (request) | +| Users | Legislative house staff, often behind NAT (many users, one public IP) | +| Workloads | PDF generation (synchronous, ReportLab), file uploads up to 150 MB, WebSocket voting panel | + +### OOM Kill Pattern + +Workers grow from ~35 MB at birth to 800–900 MB within 2–3 minutes, then are killed and replaced in a continuous cycle. + +Root causes: +- Bot scraping triggers synchronous PDF generation — entire document built in RAM (ReportLab) +- `worker_max_memory_per_child` only checks **between requests**; workers blocked on long requests are never recycled +- `TIMEOUT=300` lets bots hold threads for up to 5 minutes while memory accumulates +- 3 workers × 300 MB each = ~900 MB — breaching the 800Mi request threshold + +### Bot Traffic Profile (Barueri pod, 16 days, 662 k requests) + +| Actor | Requests | % of total | +|-------|----------|-----------| +| Googlebot | ~154,000 | 23.2% | +| Chrome/98.0.4758 (spoofed scraper) | 90,774 | 13.7% | +| kube-probe (healthcheck) | 69,065 | 10.4% | +| meta-externalagent | 28,325 | 4.3% | +| GPTBot | 11,489 | 1.7% | +| bingbot | 7,639 | 1.1% | +| OAI-SearchBot + Applebot | 6,681 | 1.0% | +| **Total identified bots** | **~377,000** | **~56.9%** | + +**Botnet fingerprint:** +- Rotates User-Agents (Chrome/121, Chrome/122, Firefox/123, Safari/17…) across requests +- Crawls all sub-endpoints of the same matéria within 1 second from different IPs +- Distributes crawling across tenants — each pod stays under the per-pod rate limit, never triggering it +- Primary targets: `/relatorios/{id}/etiqueta-materia-legislativa` (~40 KB PDF) and all `/materia/{id}/*` sub-endpoints + +### Static File Traffic (from CSV analysis) + +| Category | Requests | Transfers | +|----------|----------|----------| +| Logos / images | 62,776 | ~24 GB | +| PDFs | 8,869 | 5.1 GB | +| Parliamentarian photos | 11,856 | ~0.5 GB | +| **Total** | **83,501** | **~30 GB** | + +Top offender: `Brasão - Foz do Iguaçu.png` — 14,512 requests, 5.6 GB from a single 392 KB file. + +### Hard Constraints + +| Constraint | Impact | +|------------|--------| +| Per-pod PostgreSQL | Rate-limit counters not shared across pods | +| NAT environments | IP-based rate limiting causes false positives | +| `TIMEOUT=300` / uploads to 150 MB | Must not be broken — intentional for slow workflows | + +--- + +## Architecture Overview + +### Component Diagram + +```mermaid +graph TD + Client([Bot / Human Client]) + nginx[nginx] + gunicorn[Gunicorn\n2 workers / 4 threads] + mw[Django Middleware\nRateLimitMiddleware] + view[View Layer\nCBV + decorators] + db0[(Redis DB0\npage cache)] + db1[(Redis DB1\nrate limiter)] + pg[(PostgreSQL\nper-pod)] + fs[Filesystem\nPDFs / media] + + Client -->|HTTP| nginx + nginx -->|proxy_pass| gunicorn + gunicorn --> mw + mw -->|pass| view + mw -->|429| nginx + view --> pg + view --> fs + view -->|read/write cached pages| db0 + mw -->|counters + blocked markers| db1 +``` + +> DB2 is reserved for Django Channels (WebSocket — future). + +### Redis Memory Budget + +| Key type | Key schema | TTL | DB | Est. size | +|----------|-----------|-----|----|----------| +| Page / view cache | `cache:{ns}:*` | 60–600 s | 0 | ~0.5 GB | +| Static cache (images/logos) | `static:{ns}:{sha256}` | 3–24 h | 0 | ~2.4 GB | +| IP request counter | `rl:ip:{ip}:reqs` | 60 s | 1 | ~0.6 MB | +| IP blocked marker | `rl:ip:{ip}:blocked` | 300 s | 1 | ~0.06 MB | +| Blocked-IP index | `rl:index:blocked_ips:{0..N-1}` | self-pruning ZSET (N=3) | 1 | ~0.01 MB | +| User request counter | `rl:{ns}:user:{uid}:reqs` | 60 s | 1 | negligible | +| User blocked marker | `rl:{ns}:user:{uid}:blocked` | 300 s | 1 | negligible | +| Blocked-user index | `rl:index:blocked_users` | permanent ZSET | 1 | negligible | +| Path counter | `rl:{ns}:path:{sha256}:reqs` | 60 s | 1 | ~0.3 MB | +| UA deny list | `rl:bot:ua:blocked` | permanent SET | 1 | ~0.03 MB | +| IP-prefix deny list | `rl:ip_prefix:blocked` | permanent SET | 1 | ~0.01 MB | +| NS/IP/window counter | `rl:{ns}:ip:{ip}:w:{bucket}` | 120 s | 1 | ~0.6 MB | +| API daily quota (all callers, by IP) | `quota:{ns}:daily:{date}` (HASH, field=ip) | 24 h | 1 | ~32 MB at 1 200 tenants | +| API weekly quota (all callers, by IP) | `quota:{ns}:weekly:{week}` (HASH, field=ip) | 7 d | 1 | ~158 MB at 1 200 tenants | +| API rate counter (ns-scoped) | `rl:api:ns:{ns}:ip:{ip}:reqs` | 60 s | 1 | negligible | +| API block marker (ns-scoped) | `rl:api:ns:{ns}:ip:{ip}:blocked` | 60 s | 1 | negligible | +| Block metrics (HASH) | `rl:metrics:{ns}:{date}` (HASH, field=reason) | 8 d | 1 | ~0.5 MB | +| Redis overhead (× 1.5) | | | | ~1.6 GB | +| **Total ceiling** | | | | **~5 GB** | + +--- + +## Decision Log + +| Decision | Chosen | Rationale | +|----------|--------|-----------| +| Redis topology | **Single pod** (no Sentinel, no Cluster) | 65 MB of active data fits comfortably; cluster complexity not justified | +| PDF caching in Redis | **No** — ETags + sendfile are sufficient | Once rate limiting + ETags are active, repeat requests become 304s with zero bytes transferred | +| HTTP conditional requests | **`ConditionalGetMiddleware` + `@condition` decorator** | `ConditionalGetMiddleware` handles ETag/304 for all views; `@condition(etag_func, last_modified_func)` on materia/norma detail views skips view execution entirely on cache hit | +| Upload endpoint special-casing (nginx) | **Removed** — fall through to `location /` | No justification for separate `limit_req` zone; `location /` with `sapl_general` covers it | +| Static asset cache policy | **90 min** (`expires 90m`, `max-age=5400`) | Conservative — safe with `collectstatic` content-hashed filenames; `immutable` not used (would require verified forever-hashed URLs) | +| Rate-limit enforcement | **Django middleware** with shared Redis | No nginx image changes required; solves cross-pod consistency immediately | +| `worker_max_memory_per_child` | **400 MB** | Pod limit 1600Mi, 2 workers × 400 MB = 800 MB — leaves 800 Mi headroom | +| `sendfile off` → `on` | **Bug** — flip to `on` | No valid production reason found; disabling userspace copy is always correct | +| `/media/` serving | **X-Accel-Redirect** | Routes all `/media/` through Gunicorn so Django middleware runs; nginx serves bytes via internal location | +| Cache backend switch | **At pod startup** via `start.sh` + waffle switch | Pod restart is acceptable; avoids per-request runtime overhead | +| nginx zone splitting (2026-05-07) | **4 zones**: general / media / api / heavy | `/media/` and `/api/` requests were draining the same bucket as HTML page loads, causing false 429s on heavy pages | +| Session/voting nginx bypass (2026-05-06) | **No `limit_req`** on `/voto-individual/` and `/sessao//` | Multiple councilmembers behind a NAT IP exhausted the nginx burst during live votes (PatoBranco-PR incident) | +| Auth rate breach: no persistent block (2026-05-07) | **429 per-request only**, window resets after 60 s | A 300 s lockout is the wrong penalty for a logged-in user who clicked too fast; persistent block is appropriate for anonymous/bot traffic only | +| Raise rate thresholds (2026-05-07) | anon 35→120/m · auth 120→240/m · 404 threshold 10→20 | SAPL pages fire 12–45 parallel requests; old thresholds blocked normal navigation for users in offices with multiple open tabs | +| API quota increase (2026-05-07) | anon 50→500/day · auth 1 000→5 000/day | Previous anon quota of 50/day was exhausted by a developer testing the API before lunch | +| Auth not exempt from `/api/` rate limit (2026-05-11) | **All callers keyed by IP** — auth status not checked | Authenticating must not bypass the per-minute cap; `_evaluate` (240/min per-user) still governs non-`/api/` paths | +| Auth-specific API quotas removed (2026-05-11) | **Single `API_QUOTA_DAILY/WEEKLY`** for all callers by IP | Per-user quota added false precision; IP-based cap is sufficient alongside the per-minute block | +| API rate limit keys namespaced (2026-05-11) | `rl:api:ns:{ns}:ip:{ip}:reqs/blocked` | Without `{ns}` a block in one k8s pod namespace leaked into all tenants sharing the same Redis instance | +| API threshold raised (2026-05-11) | 60→120 req/min | Aligns with legitimate integration patterns; slow-drip abuse is caught by the daily quota | +| API block TTL reduced (2026-05-11) | 300→60 s | Shorter cooldown reduces false-positive lockout duration for shared IPs | +| API quota raised (2026-05-11) | 1 000→100 000/day · 7 000→700 000/week | Quota serves as outer envelope for all-day slow scrapers; 1 000/day was exhausted too quickly for legitimate integrations | +| IP-prefix deny list added (2026-05-11) | `rl:ip_prefix:blocked` permanent SET, dot-boundary-anchored prefix match (`df2f5ee30`) | Operators need to block whole ranges (e.g. `103.124.225.*`) at runtime by curating prefixes, not individual IPs; mirrors the existing `rl:bot:ua:blocked` runtime-deny-list pattern | +| Same-origin bypass moved after block checks (2026-05-11) | `_is_same_origin` now runs *after* the IP-prefix / global-IP / API-IP block checks in `_handle_api` (`5f71354f5`) | `Origin`/`Referer` are client-controlled and trivially spoofable; the bypass was short-circuiting before `ip = get_client_ip(request)` and before every block lookup, letting a forged `Origin` header defeat an operator-set `rl:ip::blocked` key entirely | +| Metrics key format consolidated to HASH (2026-06-09) | `rl:metrics:{ns}:{date}` HASH (field = reason) instead of `rl:metrics:{ns}:{date}:blocked:{reason}` STRING per reason (`d0eb02d27`) | With 1,200 tenants sharing one Redis instance, the old design produced up to 96 k live STRING keys (1,200 × ~10 reasons × 8-day TTL); each increment issued a Lua eval whose event-loop lock adds latency to all Redis commands when many tenants are under simultaneous attack. HASH consolidation reduces to 9,600 keys (10× fewer), reuses the existing `_hincrby_with_ttl` function, and makes monitoring simpler (`HGETALL` returns all reasons in one round-trip) | + +--- + +## Directory layout + +``` +docker/k8s/ +└── redis/ + ├── redis-configmap.yaml # redis.conf — no persistence, allkeys-lru, 5 GB ceiling + ├── redis-deployment.yaml # Deployment (1 replica, redis:7-alpine) + └── redis-service.yaml # ClusterIP service on port 6379 +``` + +--- + +## Prerequisites + +- `kubectl` configured to talk to the target cluster. +- A `sapl-redis` namespace (created below if it doesn't exist). + +--- + +## Deploy + +```bash +# 1. Create the namespace (idempotent) +rancher kubectl create namespace sapl-redis --dry-run=client -o yaml | rancher kubectl apply -f - + +# 2. Apply all three manifests +rancher kubectl apply -f docker/k8s/redis/redis-configmap.yaml +rancher kubectl apply -f docker/k8s/redis/redis-deployment.yaml +rancher kubectl apply -f docker/k8s/redis/redis-service.yaml + +# 3. Verify the pod is Running +rancher kubectl -n sapl-redis get pods -l app=sapl-redis +``` + +Expected output: +``` +NAME READY STATUS RESTARTS AGE +sapl-redis-6d9f8b7c4d-xk2lm 1/1 Running 0 30s +``` + +--- + +## Verify the rate limiter + +### Canary tenants + +Current canary namespaces receiving the `rate-limiter-2026` image: + +``` +joaopessoa-pb patobranco-pr al-am al-pi al-ro divinopolis-mg +``` + +Verify image digest, `imagePullPolicy: Always`, and `REDIS_URL` for all six at once: + +```bash +# From monitoring_metrics-2025-2026/logs/cluster-prod/ +bash check-canary-tenants.sh +``` + +Expected: all checks green and the same `sha256` digest across all pods. + +--- + +### Functional test + +`scripts/test_ratelimiter.py` fires repeated GET requests at a SAPL URL and reports +when the first 429 is returned. + +### Usage + +``` +python scripts/test_ratelimiter.py [-n NUM] [-d DELAY] [-t TIMEOUT] +``` + +| Flag | Default | Meaning | +|------|---------|---------| +| `url` | *(required)* | Full URL including scheme, e.g. `http://localhost` | +| `-n`, `--num-requests` | `50` | Maximum requests to send | +| `-d`, `--delay` | `0.1` | Seconds between requests | +| `-t`, `--timeout` | `10` | Per-request timeout in seconds | + +The script stops and prints a summary as soon as a 429 is received. + +### Examples + +```bash +# Hit the anonymous threshold (120 req/min) — fire 130 requests with minimal delay +python scripts/test_ratelimiter.py http://localhost -n 130 -d 0.05 + +# Slower fire — check that legitimate traffic is not rate-limited +python scripts/test_ratelimiter.py http://localhost -n 20 -d 2 + +# Test against a staging pod via port-forward +rancher kubectl port-forward -n deploy/sapl 8080:80 & +python scripts/test_ratelimiter.py http://localhost:8080 -n 40 -d 0.05 +``` + +### Reading the output + +``` +Request 1: Status 200 | Time: 0.045s +... +Request 36: Status 429 | Time: 0.038s + -> Rate limited on request 36 + +Summary: + Total requests attempted: 36 + Successful (200): 35 + Rate limited (429): 1 + First 429 occurred at request: 36 +``` + +A first-429 near the configured anonymous threshold (120 req/min) confirms the +middleware is wired correctly. A first-429 much earlier points to nginx `limit_req` +firing before Django sees the request. + +--- + +## Inject REDIS_URL into SAPL instances + +`REDIS_URL` points at the shared instance: + +``` +redis://redis.sapl-redis.svc.cluster.local:6379 + ^^^^^ ^^^^^^^^^^ + svc namespace +``` + +`start.sh` picks it up on every pod startup and sets the `REDIS_CACHE` waffle switch +automatically — no further intervention needed. + +### Fleet-wide rollout + +Uses the `app.kubernetes.io/name=sapl` pod label to discover every SAPL namespace +automatically — onboarding a new municipality requires no script changes. + +```bash +for ns in $(rancher kubectl get pods -A -l app.kubernetes.io/name=sapl \ + -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do + rancher kubectl set env deployment/sapl \ + REDIS_URL=redis://redis.sapl-redis.svc.cluster.local:6379 \ + -n $ns +done +``` + +### Roll back + +```bash +for ns in $(rancher kubectl get pods -A -l app.kubernetes.io/name=sapl \ + -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do + rancher kubectl set env deployment/sapl REDIS_URL- -n $ns +done +``` + +`kubectl set env deployment/sapl REDIS_URL-` (trailing `-`) removes the variable. +`start.sh` then falls back to file-based cache automatically. + +--- + +## Monitor + +### Pod and events + +```bash +# Pod status +rancher kubectl -n sapl-redis get pods -l app=sapl-redis -o wide + +# Deployment events (useful right after apply) +rancher kubectl -n sapl-redis describe deployment sapl-redis + +# Pod events (OOMKill, restarts, etc.) +rancher kubectl -n sapl-redis describe pod -l app=sapl-redis +``` + +### Logs + +```bash +# Tail live logs +rancher kubectl -n sapl-redis logs -f deploy/sapl-redis + +# Last 100 lines +rancher kubectl -n sapl-redis logs deploy/sapl-redis --tail=100 +``` + +### Redis INFO + +```bash +# Memory usage +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ + redis-cli info memory \ + | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio' + +# Connection pressure +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ + redis-cli info stats \ + | grep -E 'rejected_connections|instantaneous_ops_per_sec' + +# Key distribution per DB +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli info keyspace + +# Recent slow queries +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli slowlog get 10 + +# Live command sampling (1-second window) +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli --latency-history -i 1 +``` + +### Rate-limiter keys (DB 1) + +```bash +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ + redis-cli -n 1 dbsize + +# All rate-limiter keys for an IP prefix +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ + redis-cli -n 1 --scan --pattern 'rl:ip:*' | head -20 + +# All currently blocked IPs (legacy SCAN — use ZSET index below instead) +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ + redis-cli -n 1 --scan --pattern 'rl:ip:*:blocked' +``` + +Via port-forward (local machine — run `kubectl port-forward svc/redis -n sapl-redis 6379:6379` first): + +```bash +# All active blocked IPs via sharded ZSET index (O(log N), no SCAN) +# IPs are distributed across N shards (default 3) via md5(ip) % N. +NOW=$(date +%s) +for i in 0 1 2; do + redis-cli -n 1 ZRANGEBYSCORE rl:index:blocked_ips:$i $NOW +inf WITHSCORES +done + +# All active blocked users via ZSET index +redis-cli -n 1 ZRANGEBYSCORE rl:index:blocked_users $NOW +inf WITHSCORES + +# Count of currently active blocked IPs (sum across shards) +for i in 0 1 2; do + redis-cli -n 1 ZCOUNT rl:index:blocked_ips:$i $NOW +inf +done + +# Pruning is automatic — each _set_block write prunes expired entries from its +# shard inline (ZREMRANGEBYSCORE inside _BLOCK_LUA). Manual pruning no longer needed. +# To prune the legacy unsharded key (harmless; expires within one BLOCK_TTL after deploy): +redis-cli -n 1 DEL rl:index:blocked_ips rl:index:api_blocked_ips + +# Legacy: blocked IPs with value and remaining TTL (still works; slower on large key spaces) +redis-cli -n 1 --scan --pattern 'rl:ip:*:blocked' | while read key; do + echo "$key → $(redis-cli -n 1 GET $key) (TTL: $(redis-cli -n 1 TTL $key)s)" +done +``` + +--- + +## Seed the UA deny list (once after first deploy) + +`rl:bot:ua:blocked` is a permanent Redis SET in DB 1. Each member is the +SHA-256 of a **UA token** — the identifying fragment extracted after splitting +on `/`, spaces, `;`, `(`, `)`, e.g.: + +``` +UA string: "GPTBot/1.1 (+https://openai.com/gptbot)" +Tokens: GPTBot 1.1 +https: ... +Hash stored: sha256("GPTBot") +``` + +The middleware (`_is_redis_blocked_ua`) tokenises the incoming UA the same +way and checks each token hash against the cached set. The SET is fetched +from Redis at most once per `RATE_LIMITER_UA_BLOCKLIST_REFRESH` seconds (default 60) +per worker process. + +The bots in `BOT_UA_FRAGMENTS` (Python list, always active) and this Redis +SET are **independent** — the Python list provides the baseline and the Redis +SET allows adding new offenders at runtime **without a code deploy**. + +```bash +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:bot:ua:blocked \ + "$(echo -n 'GPTBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'ClaudeBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'PerplexityBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'Bytespider' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'AhrefsBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'meta-externalagent' | sha256sum | cut -d' ' -f1)" + "$(echo -n 'OAI-SearchBot' | sha256sum | cut -d' ' -f1)" + "$(echo -n 'quiltbot' | sha256sum | cut -d' ' -f1)" + "$(echo -n 'Googlebot' | sha256sum | cut -d' ' -f1)" + "$(echo -n 'Applebot' | sha256sum | cut -d' ' -f1)" + "$(echo -n 'meta-webindexer' | sha256sum | cut -d' ' -f1)" + "$(echo -n 'AwarioBot' | sha256sum | cut -d' ' -f1)" + +# Add a new offender at runtime (picked up within RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds) +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:bot:ua:blocked "$(echo -n 'NewBot' | sha256sum | cut -d' ' -f1)" +``` + +--- + +## Manage the IP-prefix deny list (`rl:ip_prefix:blocked`) + +`rl:ip_prefix:blocked` (constant `RL_IP_PREFIX_BLOCKLIST`, added in `df2f5ee30`) is a +**permanent Redis SET in DB 1**. Each member is a dotted-decimal **prefix string** +such as `103.124.225` or `45.177.154` — or a complete dotted-quad address such as +`201.23.71.13` to block one specific IP. Any client IP that starts with a stored +prefix on a **dot boundary** is blocked, e.g. `103.124.225` matches `103.124.225.7` +but not `103.124.2250.1` or `103.124.2255` (an octet-aligned, /24-ish range match — +not a raw string prefix). A stored entry that is already a full 4-octet address +(no trailing dot) is matched by equality only. + +This check runs **before everything else** — both at the top of `_evaluate` (non-`/api/` +paths) and at the top of `_handle_api` (`/api/` paths) — and applies universally to +**all traffic**: anonymous, authenticated, `/api/` and regular paths alike. It mirrors +the `rl:bot:ua:blocked` runtime-deny-list pattern: the SET is fetched via `SMEMBERS` +at most once per `RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH` seconds (default `60`) +per worker process, so additions/removals take effect fleet-wide without a code deploy +or restart — only the in-process cache refresh delay. + +### Populate — block a range or a single IP + +```bash +# Block an entire /24-ish range by prefix (matches 103.124.225.0 – 103.124.225.255) +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:ip_prefix:blocked 103.124.225 + +# Block multiple ranges in one call +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:ip_prefix:blocked 103.124.225 45.177.154 + +# Block one specific IP (matched by equality only — no range semantics) +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:ip_prefix:blocked 201.23.71.13 +``` + +### Remove — unblock a range or a single IP + +```bash +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SREM rl:ip_prefix:blocked 103.124.225 + +# Remove multiple entries in one call +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SREM rl:ip_prefix:blocked 103.124.225 45.177.154 201.23.71.13 +``` + +### List — inspect the current deny list + +```bash +# All prefixes/addresses currently blocked +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SMEMBERS rl:ip_prefix:blocked + +# Count of entries +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SCARD rl:ip_prefix:blocked + +# Check whether a specific prefix/address is already in the list +rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ + SISMEMBER rl:ip_prefix:blocked 103.124.225 +``` + +Via port-forward (local machine): + +```bash +redis-cli -n 1 SADD rl:ip_prefix:blocked 103.124.225 +redis-cli -n 1 SREM rl:ip_prefix:blocked 103.124.225 +redis-cli -n 1 SMEMBERS rl:ip_prefix:blocked +``` + +> **Note**: changes are picked up by each worker within `RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH` +> seconds (default `60`) — there is no immediate fleet-wide invalidation, by design +> (mirrors `rl:bot:ua:blocked`; avoids a Redis round-trip on every request). + +--- + +## Local standalone Redis (development / testing) + +No Kubernetes? Run Redis directly with Docker: + +```bash +sudo docker run --rm -p 6379:6379 redis:7-alpine \ + redis-server --save "" --appendonly no +``` + +Then point Django at it by exporting the env var before starting the dev server: + +```bash +export REDIS_URL="redis://localhost:6379" +export CACHE_BACKEND="redis" +python manage.py runserver +``` + +Or add them to your local `.env` file: + +``` +REDIS_URL=redis://localhost:6379 +CACHE_BACKEND=redis +``` + +> **Note**: the waffle switch `REDIS_CACHE` must also be `on` in your local +> database for `start.sh` to activate the Redis backend. Run: +> ```bash +> python manage.py waffle_switch REDIS_CACHE on --create +> ``` + +--- + +## Update `redis.conf` without redeploying + +```bash +# Edit the ConfigMap +rancher kubectl -n sapl-redis edit configmap redis-config + +# Restart the pod to pick up the new config +rancher kubectl -n sapl-redis rollout restart deployment/sapl-redis +``` + +--- + +## Gunicorn tuning + +`docker/startup_scripts/gunicorn.conf.py` — resolved values for the current pod budget (1600Mi RAM, 1 CPU): + +```python +NUM_WORKERS = int(os.getenv("WEB_CONCURRENCY", "2")) # was 3 +THREADS = int(os.getenv("GUNICORN_THREADS", "4")) # was 8 +TIMEOUT = int(os.getenv("GUNICORN_TIMEOUT", "120")) # was 300 + +max_requests = 1000 +max_requests_jitter = 200 +worker_max_memory_per_child = 400 * 1024 * 1024 # 400 MB — was 300 MB +``` + +**Per-location timeout strategy** — nginx overrides the global Gunicorn timeout per-path: + +| Operation | Timeout | Rationale | +|-----------|---------|-----------| +| Normal page rendering | 60 s | No legitimate page should take > 60 s | +| API endpoints | 30 s | Stateless, fast by design | +| PDF download (cached / nginx) | 30 s | nginx serves from disk, worker not involved | +| PDF generation (uncached) | 180 s | Kept high — addressed in a future phase | +| Large file upload | 180 s | nginx buffers upload; worker processes after | + +--- + +## nginx real-IP and core fixes + +Added to `docker/config/nginx/nginx.conf` (http {} block): + +```nginx +# Kernel bypass — was off (bug) +sendfile on; +tcp_nopush on; +tcp_nodelay on; + +# Real client IP from X-Forwarded-For set by K8s Ingress +real_ip_header X-Forwarded-For; +real_ip_recursive on; +set_real_ip_from 10.0.0.0/8; +set_real_ip_from 172.16.0.0/12; +set_real_ip_from 192.168.0.0/16; +``` + +Without `real_ip_recursive on`, `$remote_addr` inside the pod would always be the Ingress IP, making IP-based rate limiting and blocking meaningless. + +--- + +## Django upload settings + +Added to `sapl/settings.py` — files above 2 MB are streamed to disk rather than held in worker RAM. Critical for 150 MB upload support without OOM pressure: + +```python +FILE_UPLOAD_MAX_MEMORY_SIZE = 2 * 1024 * 1024 # 2 MB +DATA_UPLOAD_MAX_MEMORY_SIZE = 10 * 1024 * 1024 # 10 MB +MAX_DOC_UPLOAD_SIZE = 150 * 1024 * 1024 # 150 MB +FILE_UPLOAD_TEMP_DIR = '/var/interlegis/sapl/tmp' +``` + +--- + +## N+1 fix — `get_etiqueta_protocolos` + +`sapl/relatorios/views.py` — previously called `MateriaLegislativa.objects.filter()` inside a loop over protocols. Fixed to **three queries total** regardless of volume (one for protocols, one for materias, one for documentos): + +```python +# sapl/relatorios/views.py +def get_etiqueta_protocolos(prots): + prot_list = list(prots) + if not prot_list: + return [] + + # Pre-fetch MateriaLegislativa for all protocols in one query. + materia_query = Q() + for p in prot_list: + materia_query |= Q(numero_protocolo=p.numero, ano=p.ano) + materias_map = { + (m.numero_protocolo, m.ano): m + for m in MateriaLegislativa.objects.filter( + materia_query).select_related('tipo') + } + + # Pre-fetch DocumentoAdministrativo for all protocols in one query. + documentos_map = { + doc.protocolo_id: doc + for doc in DocumentoAdministrativo.objects.filter( + protocolo__in=prot_list).select_related('tipo') + } + + protocolos = [] + for p in prot_list: + dic = {} + dic['titulo'] = str(p.numero) + '/' + str(p.ano) + # ... timestamp / assunto / interessado / autor fields ... + + materia = materias_map.get((p.numero, p.ano)) + dic['num_materia'] = ( + materia.tipo.sigla + ' ' + str(materia.numero) + '/' + str(materia.ano) + if materia else '' + ) + + documento = documentos_map.get(p.pk) + dic['num_documento'] = ( + documento.tipo.sigla + ' ' + str(documento.numero) + '/' + str(documento.ano) + if documento else '' + ) + + dic['ident_processo'] = dic['num_materia'] or dic['num_documento'] + protocolos.append(dic) + return protocolos +``` + +--- + +## Rate limiting — two layers, two jobs + +SAPL enforces rate limits at two independent layers. They use different +algorithms and protect different things; their thresholds must be tuned +separately. + +### Layer 1 — nginx `limit_req` (leaky bucket) + +Defined in `docker/config/nginx/nginx.conf` (zones) and `sapl.conf` (burst). + +``` +sapl_general rate=90r/m # 1 token every 0.67 s (HTML page requests) +sapl_media rate=180r/m # 1 token every 0.33 s (/media/ — own bucket) +sapl_api rate=60r/m # 1 token every 1 s (/api/ — own bucket) +sapl_heavy rate=10r/m # 1 token every 6 s (PDF/report endpoints) +``` + +Each path has its own zone so media downloads and API calls cannot exhaust +the page-load bucket for a user navigating normally. + +`burst=N nodelay` means nginx accepts up to N requests instantly above the +current token level, then enforces the drip rate. Requests beyond the burst +cap return 429 before reaching Gunicorn — **zero Python cost**. + +Burst values are set at container startup via env vars: + +| Env var | Default | Location | +|---------|---------|----------| +| `NGINX_BURST_GENERAL` | `180` | `location /` | +| `NGINX_BURST_MEDIA` | `180` | `location /media/` | +| `NGINX_BURST_API` | `120` | `location /api/` | +| `NGINX_BURST_HEAVY` | `20` | `location /relatorios/` (nodelay kept) | + +Defaults are 2× the zone's per-minute rate, so a user can spend a full +minute's quota in a single burst before the leaky bucket takes over. + +**Session and voting paths are fully exempt from `limit_req`** — they have +dedicated location blocks with no rate zone. See §Session/voting bypass below. + +### Layer 2 — Django `RateLimitMiddleware` (sliding window) + +Defined in `sapl/middleware/ratelimit.py`, backed by Redis DB 1. + +Requests that pass nginx reach Python. The middleware counts them in a +60-second sliding window per IP (anonymous) or per user (authenticated): + +| Env var | Default | Scope | +|---------|---------|-------| +| `RATE_LIMITER_RATE` | `120/m` | Anonymous IP | +| `RATE_LIMITER_RATE_AUTHENTICATED` | `240/m` | Authenticated user (keyed by user pk — NAT-safe) | +| `RATE_LIMITER_RATE_BOT` | `5/m` | *(reserved — bots are currently blocked outright, not counted)* | +| `RATE_LIMITER_UA_BLOCKLIST_REFRESH` | `60` s | How often each worker re-fetches `rl:bot:ua:blocked` from Redis | + +**Anonymous breach** — when the window count hits the threshold the IP block key +is written atomically (Lua: `SET key 1 EX 300` + `ZADD index score key`) with a +300 s TTL. Subsequent requests from that IP return 429 without touching the +database. + +**Authenticated breach** — returns 429 for the over-limit request only; **no +persistent block key is written**. The counter expires after 60 s (the window +TTL) and the user can proceed again automatically. A 300 s lockout is the wrong +penalty for a logged-in user who clicked too fast; that severity is reserved for +anonymous/bot traffic. + +Decision flow inside `RateLimitMiddleware.__call__()` / `_evaluate()`: + +``` +-1. IP starts with a prefix in rl:ip_prefix:blocked? → 403 reason=ip_prefix_blocked + (checked first, before everything else; universal — applies to + authenticated users too; 403 + no Retry-After, since this is a + permanent operator-curated deny decision, not a transient rate limit) +0. /api/ + path AND consumer daily/weekly quota exceeded? → 429 reason=quota_daily / quota_weekly + (per-consumer: auth users by pk, anon by masked IP; fail-open when Redis unavailable) +1a. UA matches BOT_UA_FRAGMENTS list? → 429 reason=known_ua +1b. UA token hash in rl:bot:ua:blocked SET? → 429 reason=redis_ua +2. Anonymous AND IP in rl:ip:{ip}:blocked? → 429 reason=ip_blocked + (authenticated users skip — they have independent per-user limiting at 3c) + (scanner extension probes are rejected at nginx before reaching Django — see sapl.conf) +3. Authenticated user? + 3a. User in rl:{ns}:user:{uid}:blocked? → 429 reason=user_blocked + 3b. Suspicious headers (no Accept/AL)? → 429 reason=suspicious_headers_auth + 3c. User request count ≥ auth threshold? → 429 (no block key) reason=auth_user_rate +4. Anonymous: + 4a. Suspicious headers? → 429 reason=suspicious_headers + 4b. IP request count ≥ anon threshold? → SET blocked, 429 reason=ip_rate + 4c. NS/IP window count ≥ anon threshold? → SET blocked, 429 reason=ua_rotation + → pass +``` + +### Decision flow diagram + +```mermaid +flowchart TD + REQ([Request]) --> CM1 + + CM1{"IP starts with a prefix in\nrl:ip_prefix:blocked?"} + CM1 -- "yes — operator-curated permanent deny list" --> R_PREFIX([403\nip_prefix_blocked\nno Retry-After]) + CM1 -- no --> C0 + + C0{"/api/ path AND\ndaily/weekly quota exceeded?"} + C0 -- "yes — quota:{ns}:daily/weekly:{period}:user/ip exceeded" --> R_QUOTA([429\nquota_daily / quota_weekly]) + C0 -- no --> C1 + + C1{"Known bot UA?"} + C1 -- "yes — substring in BOT_UA_FRAGMENTS" --> R_UA([429\nknown_ua]) + C1 -- no --> C1B + + C1B{"Redis UA deny list?"} + C1B -- "yes — token hash in rl:bot:ua:blocked" --> R_RUA([429\nredis_ua]) + C1B -- no --> C2 + + C2{"Authenticated?"} + C2 -- yes --> C2B + C2 -- no --> C2_ANON + + C2_ANON{"IP blocked?\nrl:ip:IP:blocked"} + C2_ANON -- yes --> R_IPB([429\nip_blocked]) + C2_ANON -- no --> C3 + + C3{"Authenticated?"} + C3 -- yes --> C3A + C3 -- "no (anonymous)" --> C4A + + subgraph AUTH ["Authenticated"] + C3A{"User blocked?"} + C3A -- "yes — rl:ns:user:UID:blocked" --> R_UB([429\nuser_blocked]) + C3A -- no --> C3B + C3B{"Suspicious headers?\nno Accept-Language + no Accept"} + C3B -- yes --> R_SH([429\nsuspicious_headers_auth]) + C3B -- no --> C3C + C3C{"User rate ≥ 240/min?"} + C3C -- "yes — no block key written;\nwindow resets after 60 s" --> R_AUR([429\nauth_user_rate]) + C3C -- no --> PASS_A([✓ pass]) + end + + subgraph ANON ["Anonymous"] + C4A{"Suspicious headers?\nno Accept-Language + no Accept"} + C4A -- yes --> R_ASH([429\nsuspicious_headers]) + C4A -- no --> C4B + C4B{"IP rate ≥ 120/min?"} + C4B -- yes --> SIPR["SET rl:ip:IP:blocked TTL 300 s"] + SIPR --> R_IPR([429\nip_rate]) + C4B -- no --> C4C + C4C{"NS/IP window hit\n≥ 120 in bucket?"} + C4C -- yes --> SUAR["SET rl:ip:IP:blocked TTL 300 s"] + SUAR --> R_UAR([429\nua_rotation]) + C4C -- no --> PASS_N([✓ pass]) + end +``` + +### Enforcement graduation order + +Roll out to canary pods first; promote check-by-check in order of false-positive risk: + +| Order | Check | Reason | Risk | Condition to promote | +|-------|-------|--------|------|---------------------| +| nginx | scanner extensions | `return 444` in `sapl.conf` for `.php`/`.asp`/etc. | Zero | Gunicorn never sees these requests | +| 0th | `quota_daily` / `quota_weekly` | Per-consumer daily/weekly cap on `/api/` paths | Low | Limits set well above per-minute rate (500/day anon, 5 000/day auth) | +| 1st | `known_ua` | Substring in hardcoded `BOT_UA_FRAGMENTS` list | Zero | UA strings are deterministic | +| 2nd | `redis_ua` | Token hash in `rl:bot:ua:blocked` SET | Zero | Keys only set manually by operators | +| 3rd | `ip_blocked` | Marker set by prior proven-bad requests | Zero | Fast-path only, no new blocks created | +| 4th | `ip_rate` | Rolling IP counter ≥ 120/min | Low | Threshold calibrated from canary logs | +| 5th | `suspicious_headers` | No Accept-Language **and** no Accept | Medium | Confirmed no legitimate clients omit both headers | +| 6th | `ua_rotation` (ns/window) | NS/IP clock-aligned bucket ≥ 120 | Medium | | +| 7th | `404_scan` | Anonymous IP accumulates ≥ 20 404s/min | Low | Catches path probes without known extensions | + +### Decorator migration + +For views where `django-ratelimit` decorators already exist: + +| Endpoint type | Action | Reason | +|---------------|--------|--------| +| List views (GET) | Remove after middleware stable | Middleware covers equivalent threshold | +| Detail views (GET) | Remove after middleware stable | Middleware covers equivalent threshold | +| Search / filter views | Remove last | Expensive queries — keep stricter per-view limit until traffic data confirms safety | +| PDF / file generation | **Keep permanently** | Most expensive endpoint; per-view limit tighter than global | +| Write endpoints (POST/PUT/DELETE) | **Keep permanently** | Different abuse surface | +| Auth endpoints (login, reset) | **Keep permanently** | Credential stuffing; must be independent of IP rate | + +### Why they are not the same number + +| | nginx burst | Django threshold | +|-|------------|-----------------| +| **Algorithm** | Leaky bucket — token refills over time | Sliding window — hard count per 60 s | +| **Protects** | Gunicorn workers from being flooded | Per-client fairness, business policy | +| **Tuned by** | Capacity of the server | Acceptable request volume per client | +| **Failure mode** | Workers overwhelmed | Legitimate user over-browsing | + +A SAPL page fires 12–45 parallel requests — most are `/static/` served +directly by nginx (zero Django cost), but 5–15 may reach Gunicorn. +With `rate=90r/m` and `burst=180` a user can load several heavy pages back-to-back +before the leaky bucket takes over. The Django threshold (120/m fixed window +for anonymous, 240/m for authenticated) catches sustained automated traffic that +arrives slowly enough to pass the nginx burst cap. + +Note: nginx rates are hardcoded in `nginx.conf` (rebuild to change); burst values +are env-var configurable at container start via `start.sh` defaults. + +--- + +## Rate Limiting — Architecture Diagrams + +### NAT Thundering Herd — Before the Fix + +During a live vote all councilmembers reload simultaneously. nginx sees one +IP, exhausts its bucket, and returns 429 before Django is ever involved. +Django's per-user counter (NAT-safe) is never consulted. + +``` + Office / Chamber — behind one NAT IP + ┌──────────────────────────────────────────────────────┐ + │ Councilmember A browser reload ──┐ │ + │ Councilmember B browser reload ──┤ │ + │ Councilmember C browser reload ──┤ ~24 req/s │ + │ Staff tab 1 browser reload ──┤ same public IP │ + │ Staff tab 2 browser reload ──┘ │ + └────────────────────────────┬─────────────────────────┘ + │ all requests look identical to nginx + ▼ + ┌─────────────────────────────────────┐ + │ nginx sapl_general │ + │ rate=30r/m burst=60 nodelay │ + │ │ + │ token bucket: 0 tokens remaining │ + │ → 429 returned immediately │ + └──────────────────┬──────────────────┘ + │ + ╳ Django never reached + ╳ rl:ip:{ip}:reqs never incremented + ╳ rl:user:{uid}:reqs never consulted + │ + ▼ + 429 for all N users in the org + recovery: nginx bucket refill (~3–10 min) + NOT a Django 300s block — Redis never written +``` + +--- + +### NAT Thundering Herd — After the Session Bypass Fix + +``` + Office / Chamber — behind one NAT IP + ┌──────────────────────────────────────────────────────┐ + │ Councilmember A /voto-individual/ reload ──┐ │ + │ Councilmember B /voto-individual/ reload ──┤ │ + │ Councilmember C /sessao/2600/ordemdia ───────┤ │ + │ Staff tab /sessao/pauta-sessao/2600/ ──┘ │ + └────────────────────────────┬─────────────────────────┘ + ▼ + ┌─────────────────────────────────────┐ + │ nginx │ + │ │ + │ location ~ ^/voto-individual/ ─┐ │ + │ location ~ ^/sessao/\d+ ─┤ │ no limit_req + │ location ~ ^/painel/\d+/dados ─┘ │ pass through + └──────────────────┬──────────────────┘ + ▼ + ┌─────────────────────────────────────┐ + │ Django RateLimitMiddleware │ + │ RATE_LIMIT_BYPASS_PATHS match? │ + │ → yes: return get_response() │ + └──────────────────┬──────────────────┘ + ▼ + ✓ View served +``` + +--- + +### nginx Zone Architecture — Before vs After + +**Before** — all traffic sharing one bucket per IP: + +``` + /media/page.pdf ──┐ + /materia/123/ ───┤──► sapl_general rate=30r/m burst=60 + /api/materia/? ───┘ + + Problem: 20 media attachments on a page burn 20 tokens + from the same budget as the HTML page load +``` + +**After** — four independent buckets: + +``` + location / ──► sapl_general rate=90r/m burst=180 + location /media/ ──► sapl_media rate=180r/m burst=180 + location /api/ ──► sapl_api rate=60r/m burst=120 + location /relatorios/ ──► sapl_heavy rate=10r/m burst=20 (nodelay) + location /sessao/\d+ ──► (no zone) exempt + location /voto-indiv.. ──► (no zone) exempt + location /static/ ──► (no zone) disk-served, no Django +``` + +--- + +### Anonymous /api/ NAT Problem — Before vs After + +**Before** — anonymous API hits polluted the global IP counter: + +``` + 10 staff, JS polling /api/ → 120 req/min from NAT IP + │ + ▼ + Django _evaluate_anonymous + INCR rl:ip:{ip}:reqs → 120 ≥ threshold + SET rl:ip:{ip}:blocked EX 300 ◄── global block + │ + ▼ + Next GET /materia/ → 429 ip_blocked + Next GET /sessao/ → 429 ip_blocked + Entire org locked out of ALL paths for 300s +``` + +**After** — anonymous API skips the IP counter entirely: + +``` + 10 staff, JS polling /api/ → 120 req/min from NAT IP + │ + ▼ + nginx sapl_api rate=60r/m burst=120 + (throttles sustained traffic) + │ + ▼ + Django quota check: 500/day not exceeded → pass + Anonymous /api/: early return, no _evaluate() + rl:ip:{ip}:reqs NOT incremented + rl:ip:{ip}:blocked NOT written + │ + ▼ + Page requests from same IP: unaffected ✓ + Worst case: 500 API req/day quota exhausted + → only API access blocked, pages still work +``` + +--- + +### Authenticated Rate Breach — Before vs After + +``` + BEFORE AFTER + ────────────────────────────────── ────────────────────────────────── + User clicks fast: 241 req in 60s User clicks fast: 241 req in 60s + │ │ + ▼ ▼ + count ≥ 240 (auth threshold) count ≥ 240 (auth threshold) + │ │ + ▼ ▼ + SET rl:user:{uid}:blocked EX 300 return 429 for this request only + ZADD rl:index:blocked_users (no SET, no ZADD) + │ │ + ▼ ▼ + All requests for 300s → 429 T+60s: counter key expires + User locked out for 5 minutes User recovers automatically + No self-recovery possible No admin intervention needed +``` + +--- + +### Enforcement Stack Per Path — Trade-off Summary + +``` +Path nginx zone Django Block key? Notes +───────────────────── ───────────────── ────────────── ────────── ────────────────────── +/static/* none none — disk-served +/painel//dados none (bypass) none (bypass) — high-freq polling +/voto-individual/* none (bypass) none (bypass) — live vote +/sessao//* none (bypass) none (bypass) — live session +/media/* sapl_media anon counter anon: yes auth gate in serve_media + 180r/m b=180 runs auth: no +/api/* (anonymous) sapl_api quota only no ← no IP counter; no + 60r/m b=120 500/day collateral NAT block +/api/* (auth) sapl_api per-user 240/m no (soft) per-uid, NAT-safe + 60r/m b=120 counter runs +/relatorios/* sapl_heavy counter runs anon: yes tight — PDF generation + 10r/m b=20 +/* (everything else) sapl_general counter runs anon: yes normal browsing + 90r/m b=180 auth: no auth: 429, resets in 60s +``` + +`anon: yes` — anonymous IP gets a 300s block key on breach (all paths locked) +`auth: no` — authenticated users get 429 for that request; counter expires in 60s + +--- + +### The Fundamental NAT Constraint + +``` + IP-based rate limiting cannot distinguish these two scenarios: + + Legitimate (15 users, vote opens simultaneously) + ┌─────────────────────────────────────────────┐ + │ User 1 ──► GET /voto-individual/ │ + │ User 2 ──► GET /voto-individual/ 15 req/s │ + │ ... 1 IP │ + │ User 15 ──► GET /sessao/2600/ │ + └─────────────────────────────────────────────┘ + + Bot (1 process, 15 threads, scraping) + ┌─────────────────────────────────────────────┐ + │ Thread 1 ──► GET /materia/1/ │ + │ Thread 2 ──► GET /materia/2/ 15 req/s │ + │ ... 1 IP │ + │ Thread 15 ──► GET /materia/15/ │ + └─────────────────────────────────────────────┘ + + To nginx and an IP counter: identical. + + Mitigations applied + ┌──────────────────────────────────────────────────────────────────┐ + │ Known safe high-freq paths → bypass at both layers │ + │ Authenticated users → per-user counter (uid), NAT-safe │ + │ Anonymous /api/ → quota only, no IP counter │ + │ Everything else (anon) → IP counter + 300s block │ + └──────────────────────────────────────────────────────────────────┘ + + Long-term + ┌──────────────────────────────────────────────────────────────────┐ + │ APP_ACCESS_KEYs per tenant → quota per org, not per IP │ + │ WebSocket push for voting → eliminates polling bursts │ + └──────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Session/voting bypass (2026-05-06) + +### Problem + +Multiple councilmembers behind a shared NAT IP were receiving 429 errors during +live plenary votes. Root cause: nginx's `limit_req` fires before any request +reaches Django, so Django's per-user counters (which are NAT-safe) were never +consulted. When a vote opened, 15+ users simultaneously reloaded their voting +pages, exhausting the shared IP's nginx burst bucket. + +The `voto_individual.html` template contains `setTimeout(location.reload, 30000)` +— the page reloads itself every 30 seconds. When councilmembers open the page at +roughly the same time (vote announcement), their reload timers align and all fire +in the same second. + +See `docs/rate-limiter-incidents.md` — PatoBranco-PR 2026-05-06 for full analysis. + +### Fix + +Dedicated nginx `location` blocks with **no `limit_req`** for session and voting +paths. These regex locations take priority over `location /` by nginx matching +rules. Mirrored in `RATE_LIMIT_BYPASS_PATHS` so Django's middleware also skips +counting (defense-in-depth). + +```nginx +# sapl.conf — no rate limiting on session/voting paths +location ~ ^/painel/\d+/dados$ { proxy_pass http://sapl_server; } +location ~ ^/voto-individual/ { proxy_pass http://sapl_server; } +location ~ ^/sessao/\d+ { proxy_pass http://sapl_server; } +``` + +```python +# settings.py +RATE_LIMIT_BYPASS_PATHS = [ + r'^/painel/\d+/dados$', + r'^/voto-individual/', + r'^/sessao/\d+', + r'^/sessao/pauta-sessao/\d+/', +] +``` + +### Why these paths are safe to exempt + +- All meaningful actions require an authenticated session cookie. +- Django's per-user counter (240/m, keyed by user pk) still applies as a backstop. +- The real abuse vectors (scrapers, credential stuffing) target different URL patterns. +- The cost of a false-positive block (councilmember unable to vote) far outweighs + the risk of abuse on these paths. + +### Long-term fix + +Replace `setTimeout(location.reload, 30000)` in `voto_individual.html` with +server-push (WebSocket or SSE). Removes the synchronisation mechanism entirely — +the thundering herd cannot occur if the server pushes vote-open events instead of +clients polling by reloading. + +--- + +## Request routing — how nginx reaches Django + +`proxy_pass http://sapl_server` forwards the HTTP request — with the original +path intact — to the Gunicorn Unix socket. Django doesn't know or care that +nginx is in front; it sees a standard HTTP request. + +``` +GET /media/foo.pdf + │ + ▼ + nginx (sapl.conf) + location /media/ → proxy_pass to Unix socket + │ + ▼ + Gunicorn (WSGI server) + receives raw HTTP, calls Django WSGI application + │ + ▼ + Django middleware stack (settings.MIDDLEWARE) + RateLimitMiddleware → pass or 429 + │ + ▼ + Django URL router (sapl/urls.py) + r'^media/(?P.*)$' → serve_media + │ + ▼ + serve_media(request, path='foo.pdf') + returns HttpResponse with X-Accel-Redirect: /internal/media/foo.pdf + │ + ▼ + nginx sees X-Accel-Redirect header + /internal/media/ internal location → reads file from disk → sends to client +``` + +nginx does no routing beyond picking a `location` block. The mapping from +URL path to Python function lives entirely in `sapl/urls.py`. `proxy_pass` is +just a pipe. + +--- + +## Media file serving — `serve_media` and X-Accel-Redirect + +All `/media/` requests (public and private) are routed through Gunicorn so that +Django middleware runs on every hit. Nginx serves the file bytes via +`X-Accel-Redirect` — the Gunicorn worker is freed as soon as it sends the +response headers. + +### nginx locations (`docker/config/nginx/sapl.conf`) + +```nginx +# Static files — no rate limiting, no proxy; 90-minute browser cache. +location /static/ { + alias /var/interlegis/sapl/collected_static/; + expires 90m; + add_header Cache-Control "public, max-age=5400"; +} + +# Proxied to Gunicorn — Django middleware + serve_media() run here. +# Own zone so media downloads don't drain the general page-load bucket. +location /media/ { + limit_req zone=sapl_media burst=${NGINX_BURST_MEDIA} nodelay; + proxy_pass http://sapl_server; +} + +# Internal — only reachable via X-Accel-Redirect, not by external clients. +location /internal/media/ { + internal; + alias /var/interlegis/sapl/media/; + sendfile on; + etag on; +} +``` + +Upload endpoints (`/protocoloadm/criar-protocolo`, `/materia/.*upload`, `/norma/.*upload`) no longer have a dedicated `location` block — they fall through to `location /` which applies the `sapl_general` zone. + +### Django view (`sapl/base/media.py`) + +`serve_media(request, path)` — registered at `^media/(?P.*)$` in `sapl/urls.py`. + +Per-request steps: + +1. **Path traversal guard** — `os.path.abspath` check; raises 404 on escape. +2. **Auth gate** — `documentos_privados/` paths require an authenticated session; redirects to login otherwise. +3. **Path counter** — increments `rl:{ns}:path:{sha256}:reqs` in Redis DB 1 (TTL = `MEDIA_PATH_COUNTER_TTL`). +4. **Serve** — in DEBUG: `django.views.static.serve` directly. In production: `X-Accel-Redirect: /internal/media/`. Nginx sets `Content-Type` from its own `mime.types`. + +### Settings + +| Setting | Default | Purpose | +|---------|---------|---------| +| `MEDIA_PATH_COUNTER_TTL` | `60` s | TTL for both URL-path and storage-path counters (DB 1) | + +### File serving decision matrix + +| File type | Size | Strategy | +|-----------|------|----------| +| Logos / images | Any | nginx `alias` + `sendfile` + ETag + `Cache-Control` | +| Small PDFs | ≤ 360 KB | nginx direct + ETag | +| Medium PDFs | 360 KB – 2 MB | nginx direct + ETag + rate limit | +| Large PDFs | > 2 MB | nginx direct + strict rate limit; never Redis | +| LGPD-restricted | Any | Django `serve_media` → `X-Accel-Redirect` → nginx (access control enforced) | +| Public `/media/` | Any | Django `serve_media` → `X-Accel-Redirect` → nginx (middleware runs; path counter written) | + +### Why Redis is not needed for PDFs + +With the full mitigation stack active: +- **ASN blocking** drops datacenter bot traffic at nginx (zero Python cost) +- **UA blocking** drops known-UA bots at nginx (zero Python cost) +- **Shared Redis rate counters** enforce limits across all pods +- **ETags** convert repeat requests to 304 responses with zero bytes transferred +- **`sendfile on`** means disk reads bypass userspace entirely + +Redis PDF caching would solve "high request volume reaching the file layer" — but that problem no longer exists once the above stack is active. For `Brasão - Foz do Iguaçu.png` (392 KB × 14,512 requests = 5.6 GB), a 50% conditional-request hit rate saves ~2.8 GB immediately — without any Redis. + +--- + +## Key schema reference + +| DB | Use case | Key pattern | TTL | Threshold | Constant | +|----|----------|-------------|-----|-----------|----------| +| 0 | Page / view cache | `cache:{ns}:*` | 300 s (default) | — | `CACHES['default']` KEY_PREFIX | +| 0 | Static file cache (logos) | `static:{ns}:{sha256}` | 3 – 24 h | — | *Future* (requires OpenResty/Lua) | +| 0 | File content cache (≤ 360 KB) | `file:{ns}:{sha256}` | 1 h | — | *Future* | +| 1 | IP rate-limit counter | `rl:ip:{ip}:reqs` | 60 s | 120 (`RATE_LIMITER_RATE`) | `RL_IP_REQUESTS` | +| 1 | IP 404 counter | `rl:ip:{ip}:404s` | 60 s | 20 (`RATE_LIMIT_404_THRESHOLD`) | `RL_IP_404S` | +| 1 | IP blocked marker | `rl:ip:{ip}:blocked` | 300 s | — | `RL_IP_BLOCKED` | +| 1 | Blocked-IP ZSET index | `rl:index:blocked_ips:{0..N-1}` | self-pruning ZSET, score=expiry ts, N=`RATE_LIMITER_INDEX_SHARDS` (default 3) | — | `RL_INDEX_BLOCKED_IPS` | +| 1 | User rate-limit counter | `rl:{ns}:user:{uid}:reqs` | 60 s | 240 (`RATE_LIMITER_RATE_AUTHENTICATED`) | `RL_USER_REQUESTS` | +| 1 | User blocked marker | `rl:{ns}:user:{uid}:blocked` | 300 s | — *(not written on rate breach; window resets naturally)* | `RL_USER_BLOCKED` | +| 1 | Blocked-user ZSET index | `rl:index:blocked_users` | permanent ZSET, score=expiry ts | — *(not written on rate breach)* | `RL_INDEX_BLOCKED_USERS` | +| 1 | Namespace/IP sliding window | `rl:{ns}:ip:{ip}:w:{bucket}` | 120 s | 120 (`RATE_LIMITER_RATE`) | `RL_NS_WINDOW` | +| 1 | Path counter (`/media/`) | `rl:{ns}:path:{sha256}:reqs` | 60 s | — (observability only) | `RL_PATH_REQUESTS` | +| 1 | Path counter (`/static/`) | `rl:{ns}:path:{sha256}:reqs` | 60 s | — | *Future* (requires OpenResty/Lua) | +| 1 | UA deny list | `rl:bot:ua:blocked` | permanent SET | — (block on match) | `RL_UA_BLOCKLIST` | +| 1 | IP-prefix deny list | `rl:ip_prefix:blocked` | permanent SET | — (block on prefix match) | `RL_IP_PREFIX_BLOCKLIST` | +| 1 | API daily quota (all callers, by IP) | `quota:{ns}:daily:{date}` HASH, field=ip | 24 h | 100 000 (`API_QUOTA_DAILY`) | `QUOTA_DAILY_HASH` | +| 1 | API weekly quota (all callers, by IP) | `quota:{ns}:weekly:{week}` HASH, field=ip | 7 d | 700 000 (`API_QUOTA_WEEKLY`) | `QUOTA_WEEKLY_HASH` | +| 1 | API IP rate counter (all callers, ns-scoped) | `rl:api:ns:{ns}:ip:{ip}:reqs` | 60 s (`API_RATE_LIMIT_WINDOW_SECONDS`) | 120 (`API_RATE_LIMIT_THRESHOLD`) | `RL_API_IP_REQUESTS` | +| 1 | API IP block marker (ns-scoped) | `rl:api:ns:{ns}:ip:{ip}:blocked` | 60 s (`API_RATE_LIMIT_BLOCK_SECONDS`) | — | `RL_API_IP_BLOCKED` | +| 1 | API blocked-IP ZSET index | `rl:index:api_blocked_ips:{0..N-1}` | self-pruning ZSET, score=expiry ts, N=`RATE_LIMITER_INDEX_SHARDS` (default 3) | — | `RL_INDEX_API_BLOCKED_IPS` | +| 1 | Block metrics counter | `rl:metrics:{ns}:{date}` HASH, field=reason | 8 d | — (observability only) | `RL_METRICS_BLOCKED` | +| 2 | Django Channels | `channels:*` | session TTL | — | *Future* | + +### What each counter catches — and misses + +**`rl:ip:{ip}:reqs` — global rolling IP counter** + +Catches: any sustained anonymous volume from a single IP regardless of namespace, +path, or User-Agent — pure request rate. + +Misses: a user legitimately accessing several municipality SAPLs simultaneously; +their requests accumulate across namespaces into one global count and may trip the +threshold even though no individual SAPL is being abused. Also misses a +timing-aware scraper that paces exactly 34 req/min: the 60 s TTL resets from the +first request, so the attacker can safely send 34, wait for reset, repeat forever. + +--- + +**`rl:ip:{ip}:blocked` — IP short-circuit marker** + +Written when `rl:ip:{ip}:reqs` hits the anonymous threshold (step 4b) or when the +namespace/IP bucket hits the threshold (step 4c). Checked at step 2 — before any +counting — so a blocked IP never increments any counter on subsequent requests. + +Catches: saves Redis INCR + EXPIRE calls for every request from an already-blocked +IP; the 300 s TTL is a hard cooldown regardless of how many requests arrive. + +Misses: the TTL is fixed — a persistent attacker simply waits 300 s and gets +another full window quota. Also, because the key is global (no namespace), an IP +blocked for one municipal SAPL is blocked for all SAPLs on the same pod — +collateral effect for shared IPs. + +--- + +**`rl:{ns}:ip:{ip}:w:{bucket}` — namespace-scoped clock-aligned bucket** + +Catches: sustained scraping against a *specific* municipal SAPL that stays just +under the global threshold; a scraper pacing 34 req/min globally across namespaces +still accumulates in the per-namespace bucket. Clock alignment (bucket = +`time() // 60`) means a burst straddling a minute boundary still contributes to +the *next* bucket for 120 s (2× TTL), making precise timing attacks harder. + +Misses: an IP that floods one namespace to exactly 34 req/min: it never reaches 35 +in the bucket either. Cross-namespace legitimate traffic that happens to land +within the same clock minute — same blind spot as `rl:ip:*` but scoped lower. + +**Why this key is namespace-scoped** + +Five arguments for `rl:{ns}:ip:{ip}:w:{bucket}` over a global `rl:ip:{ip}:w:{bucket}`: + +1. **Matches the observed attack pattern.** The botnet in §Bot Traffic Profile targets one SAPL at a time, not the fleet evenly. A scraper hammering `fortaleza-ce` at 34 req/min has a namespace counter of 34 and a global counter of 34. Without the namespace the two keys are redundant — the window adds no new signal. With it, a scraper that legitimately distributes across 5 SAPLs (7 req/min each, 35 globally) is caught globally but *not* per-SAPL — correct behaviour, since no single SAPL is being abused. + +2. **Two counters defeat two different gaming strategies.** `rl:ip:{ip}:reqs` uses a rolling TTL (starts on the first INCR). A scraper that knows this can send 34 requests, wait ~61 s for the key to expire, and repeat indefinitely. The clock-aligned window resets at wall-clock minute boundaries. To game *both* simultaneously the attacker must time bursts to expire the rolling key *and* land entirely within one clock window — two independent constraints that are hard to satisfy together. + +3. **Without the namespace it duplicates the global counter.** All pods share the same Redis. A global `rl:ip:{ip}:w:{bucket}` would aggregate that IP's traffic from every pod — exactly what `rl:ip:{ip}:reqs` already does, just with different reset timing. Two keys measuring the same dimension is wasted INCR overhead with no added signal. + +4. **Multi-SAPL legitimate IPs are not penalised.** Municipal IT departments, ISP shared exit nodes, and Googlebot all produce high global request rates while being individually harmless to any one SAPL. A namespaced window lets them access 10 SAPLs at 3 req/min each without triggering a per-SAPL block, while the global counter still catches them if their total rate is abusive. + +5. **Consistent with the established `{ns}` isolation contract.** All user-keyed (`rl:{ns}:user:{uid}:*`) and path-keyed (`rl:{ns}:path:{sha256}:reqs`) entries are namespace-scoped. A global window key would break the invariant that per-tenant data is isolated — complicating key-space inspection, `SCAN`-based dashboards, and future per-tenant rate adjustments. + +--- + +**`rl:{ns}:user:{uid}:reqs` — authenticated user counter** + +Catches: an authenticated account being used as a scraping credential — even if +the requests come from many different IPs (e.g., distributed proxy pool), all +requests share the same `uid` and accumulate in one counter. + +Misses: a credential that is shared across multiple legitimate users in the same +office; all their activity adds up to one counter and can trip the 240/min +threshold during a busy session. + +--- + +**`rl:{ns}:user:{uid}:blocked` — authenticated user short-circuit marker** + +Written when `rl:{ns}:user:{uid}:reqs` hits the authenticated threshold (step 3c). +Checked at step 3a — before counting — so a blocked user never increments their +counter on subsequent requests during the 300 s cooldown. + +Previously caught: credential-stuffing or runaway automation using a valid session — +once the 240/min threshold was hit the account was locked out for 300 s. + +**Changed (2026-05-07):** `_set_block` is no longer called on authenticated rate +breach. The 429 is returned for the over-limit request; the counter expires after +60 s and the user proceeds automatically. The `rl:{ns}:user:{uid}:blocked` marker +and `rl:index:blocked_users` ZSET are therefore **not written on rate breach** — +only legacy entries from before this change may exist. A 300 s lockout is wrong +for a logged-in user who clicked too fast; that penalty is reserved for +anonymous/bot traffic. + +--- + +**`rl:{ns}:path:{sha256}:reqs` — per-media-file URL counter** + +Currently observability-only (no threshold enforced). Intended for future +hot-file detection: a single document being hammered by many IPs would show +a spike in this counter even if no individual IP exceeds the IP threshold. + +Misses: nothing is blocked today. Once a threshold is added, it will miss +distributed access where many IPs each download the file once (legitimate CDN +pre-warming or public interest event). + +--- + +**`rl:index:blocked_ips:{0..N-1}` / `rl:index:blocked_users` — ZSET enumeration indexes** + +Written atomically alongside every block-key write via `_BLOCK_LUA` (Lua: `SET key 1 EX ttl` + `ZREMRANGEBYSCORE index -inf now-1` + `ZADD index expire_ts key`). Score = unix expiry timestamp. IPs are routed to a shard via `md5(ip) % N` (default N=3, configurable via `RATE_LIMITER_INDEX_SHARDS`). + +Catches: gives monitoring and admin tooling an O(log N) view of all active blocks — `ZRANGEBYSCORE index: +inf` across all shards — without a fleet-wide `SCAN`. Distributes write contention across N keys. Inline `ZREMRANGEBYSCORE` keeps each shard bounded to active-only entries (no unbounded growth). + +Misses: the fallback path (Redis unavailable) skips the ZADD — the actual block key is still set via `cache.set`, but the index entry is lost for that event. Querying all blocked IPs requires iterating all N shards. + +--- + +**`rl:bot:ua:blocked` — runtime UA deny list** + +Catches: new bot UA tokens added at runtime via `redis-cli SADD` without a code +deploy; picked up within `RATE_LIMITER_UA_BLOCKLIST_REFRESH` seconds (default 60) +per worker. Complements the hardcoded `BOT_UA_FRAGMENTS` Python list. + +Misses: bots that rotate UA tokens on every request (no single token accumulates); +bots that impersonate a valid browser UA completely (no known fragment to match). + +--- + +## /api/ rate limiting — `_handle_api` (2026-05-11) + +### Problem + +Three concurrent problems made the old anonymous-/api/-passes-immediately design +insufficient: + +1. **SAPL itself polls /api/** from the browser (same-origin). It must not be counted or blocked. +2. **Legitimate user scripts behind NAT** poll `/api/` aggressively. The old design had no per-minute cap; quota (daily/weekly) was the only gate, and quota exhaustion wrote `rl:ip::blocked`, locking every person behind the same NAT out of the entire application. +3. **Bots** hammer `/api/` with no constraint other than the nginx `sapl_api` zone (60 r/m burst 120). + +### Solution — `_handle_api` + +`RateLimitMiddleware.__call__` delegates all `/api/` requests to `_handle_api`, which applies a separate, scoped decision chain (current order, **post-`5f71354f5` fix** — see §Security fix below for why block checks now precede the same-origin bypass): + +| Step | Condition | Action | +|------|-----------|--------| +| 1 | `OPTIONS` method | Pass — CORS preflight must never be blocked | +| 2 | `ip = get_client_ip(request)` | (no action — IP resolved up front so every check below can use it) | +| 3 | IP-prefix block (`_is_ip_prefix_blocked`) | **403** `ip_prefix_blocked` — operator-curated range block; applies to everyone. 403 (not 429) and no `Retry-After`: this is a permanent deny decision, not a transient rate limit | +| 4 | `rl:ip::blocked` exists | 429 `global_ip_blocked` — global block also covers `/api/` | +| 5 | `rl:api:ns::ip::blocked` exists | 429 `api_ip_blocked` — API-only, tenant-scoped block | +| 6 | Same-origin (`_is_same_origin`) | Pass, skipping steps 7-8 only — SAPL's own browser polling is exempt from quota/rate-limit *accounting*, never from the block checks above | +| 7 | Daily/weekly quota exceeded | 429 `quota_daily` / `quota_weekly` | +| 8 | API counter ≥ threshold (all callers) | Write `rl:api:ns::ip::blocked`; 429 `api_threshold_exceeded` | +| — | Under threshold | Pass | + +Auth status is **not checked**. Authenticated and anonymous callers are treated identically — both keyed by IP, both subject to the same threshold and quota. `_evaluate` (240/min per-user) still governs all non-`/api/` paths. + +**Key invariant**: `rl:ip::blocked` is **never written** because of `/api/` abuse. +`rl:api:ns::ip::blocked` is tenant-scoped and blocks only `/api/` — page requests from the same NAT continue, and a block in one k8s namespace does not affect other tenants. + +### Security fix — same-origin bypass let blocked IPs through (`5f71354f5`, 2026-05-11) + +**Symptom**: an operator set `rl:ip:201.23.71.13:blocked` (the global `RL_IP_BLOCKED` key) with a large TTL, expecting that IP to be blocked everywhere — per the middleware's own documented decision flow, "global IP block also covers `/api/`". Requests from that IP to `/api/` endpoints kept getting through. + +**Root cause**: in the original `_handle_api`, the same-origin check ran — and could `return` — *before* `ip = get_client_ip(request)` and before any of the block-key lookups: + +```python +def _handle_api(self, request): + if request.method == 'OPTIONS': + return self.get_response(request) + if self.api_same_origin_bypass and _is_same_origin(request): + return self.get_response(request) # <-- returned HERE, nothing below ever ran + ip = get_client_ip(request) + # ...IP-prefix block, global IP block, API block, quota, rate limit +``` + +`_is_same_origin` decides "same origin" purely by parsing the client-supplied `Origin`/`Referer` headers and comparing their host to `request.get_host()` — **both headers are entirely client-controlled**, with no CSRF-token or session/cookie verification backing them. Any bot can send `Origin: https://` and be treated as "SAPL's own browser polling," getting a complete free pass on the IP-prefix blocklist, the global IP block, the API-specific IP block, daily/weekly quota, and the per-minute API rate limit. `201.23.71.13` was simply sending requests with a spoofed `Origin`/`Referer` matching the SAPL host. + +**Fix**: reorder `_handle_api` so block checks — decisions already made by the system or an operator about an IP — always run first and can never be bypassed by spoofable headers. The same-origin bypass now sits *after* the IP-prefix, global-IP, and API-IP block checks (step 6 above) and exempts a request from quota/rate-limit *accounting* only, never from an active block. See the updated decision-chain table above for the full new order. + +```bash +# Manual verification (against local Redis, ratelimit cache / DB 1): +redis-cli -n 1 SET rl:ip:201.23.71.13:blocked 1 EX 3600 +curl -H 'X-Forwarded-For: 201.23.71.13' -H 'Origin: https://' http://localhost:8000/api/materia/ +# Before the fix: 200 (bypassed). After the fix: 429 with X-RateLimit-Reason: global_ip_blocked +``` + +### Same-origin detection — `_is_same_origin` + +Replaces `ApiEmergencySameSiteOnlyMiddleware._came_from_same_host` (deleted). + +| Aspect | Emergency block | `_is_same_origin` | +|--------|-----------------|-------------------| +| Normalization | strip port, lowercase (both sides) | same | +| Origin + Referer | `origin_match OR referer_match` | sequential: Origin first, Referer only if Origin absent | +| Wrong Origin with matching Referer | pass (Referer wins) | block (explicit wrong Origin = cross-origin) | +| Both absent | block | block | + +The sequential check is stricter and matches the spec: an explicit wrong `Origin` +header means the browser knows this is cross-origin, regardless of what `Referer` says. + +### Settings + +| Setting | Env var | Default | Purpose | +|---------|---------|---------|---------| +| `API_RATE_LIMIT_ENABLED` | same | `True` | Master switch; set False to revert to quota-only | +| `API_RATE_LIMIT_THRESHOLD` | same | `120` | Requests per window before API block | +| `API_RATE_LIMIT_WINDOW_SECONDS` | same | `60` | Counter TTL (seconds) | +| `API_RATE_LIMIT_BLOCK_SECONDS` | same | `60` | `rl:api:ns::ip::blocked` TTL | +| `API_RATE_LIMIT_SAME_ORIGIN_BYPASS` | same | `True` | Disable to test without same-origin pass | +| `API_QUOTA_DAILY` | same | `100 000` | Daily call cap per IP (all callers) | +| `API_QUOTA_WEEKLY` | same | `700 000` | Weekly call cap per IP (7× daily) | + +### Files changed + +| File | Change | +|------|--------| +| `sapl/middleware/api_emergency_block.py` | Deleted | +| `sapl/settings.py` | Removed `ApiEmergencySameSiteOnlyMiddleware` from `MIDDLEWARE`; added `API_RATE_LIMIT_*` and `API_QUOTA_*` settings; auth-specific quota settings removed; threshold 60→120; block TTL 300→60 s; quota 1 000→100 000/day; added `RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH` (default `60`) | +| `sapl/middleware/ratelimit.py` | Added `RL_API_IP_REQUESTS`, `RL_API_IP_BLOCKED` (both ns-scoped), `RL_INDEX_API_BLOCKED_IPS` constants; added `_is_same_origin`; extended `__init__`; added `_handle_api`, `_api_block_response`; auth check removed from `_handle_api` and `_check_api_quota` — all callers keyed by IP. Later (`df2f5ee30`): added `RL_IP_PREFIX_BLOCKLIST` constant, `_ip_prefix_blocklist`/`_ip_prefix_blocklist_fetched_at` class state, `_refresh_ip_prefix_blocklist`, `_is_ip_prefix_blocked`, and a top-of-`_evaluate` + top-of-`_handle_api` check. Later still (`5f71354f5`): reordered `_handle_api` so `ip = get_client_ip(request)` and all block-key lookups (IP-prefix, global IP, API IP) run *before* the same-origin bypass — closing the spoofed-`Origin` bypass described above | +| `sapl/middleware/test_ratelimiter.py` | Extended `_make_middleware`; added 17 new tests for `_handle_api`/quota/same-origin. Later: added `_seed_prefix_blocklist` fixture and 11 tests for the IP-prefix blocklist; added 5 regression tests proving same-origin headers can no longer bypass an active block (`test_api_same_origin_does_not_bypass_global_ip_block`, `..._api_ip_block`, `..._ip_prefix_block`, `test_api_same_origin_still_skips_quota_and_rate_limit_when_not_blocked`). Later (`d0eb02d27`): added `test_inc_block_metric_uses_hash_key` verifying metrics writes use `_hincrby_with_ttl` with the HASH key and reason as field | + +--- + +## Dynamic page caching + +**Goal**: Eliminate ORM queries for anonymous bot requests on list views. +**Prerequisite**: Phase 1 (shared Redis, `CACHE_BACKEND=redis`). + +Many SAPL list views (`pesquisar-materia`, `norma`, etc.) are not truly dynamic for anonymous users between edits. A bot hammering `?page=1` through `?page=100` triggers 100 ORM queries per pod. With Redis page cache, each unique URL is queried once per TTL across the entire fleet. + +```python +# Apply to anonymous list views only — AnonCachePageMixin already wired to materia/sessao detail views. +from django.views.decorators.cache import cache_page +from django.utils.decorators import method_decorator + +@method_decorator(cache_page(60 * 5), name='dispatch') # 5-minute TTL +class PesquisarMateriaView(FilterView): + ... +``` + +> **Safety check**: `cache_page` sets `Cache-Control: private` for authenticated sessions automatically. +> Verify this is working before deploying — accidentally caching a session-aware response is a data leak. + +### Cache TTL guidelines + +| View type | TTL | Reasoning | +|-----------|-----|-----------| +| Matéria list (anonymous) | 300 s | Changes infrequently between sessions | +| Norma list (anonymous) | 300 s | Same | +| Parlamentar list | 3600 s | Changes rarely | +| Search results | 60 s | Query-dependent; shorter TTL safer | +| Authenticated views | Never | `cache_page` respects this automatically | +| PDF generation | Never | Too large — serve from disk via nginx | + +--- + +## HTTP Conditional Requests + +Two complementary mechanisms eliminate redundant work for unchanged content. + +### `ConditionalGetMiddleware` (all views) + +Added to `MIDDLEWARE` in `sapl/settings.py` (after `CommonMiddleware`). For every +Django response it: + +1. Generates a weak `ETag` from an MD5 of the response body if none is set. +2. Compares against the client's `If-None-Match` / `If-Modified-Since`. +3. Returns `304 Not Modified` (no body) on a match. +4. Handles `HEAD` requests by stripping the body and keeping headers. + +**Caveat**: the view still executes and renders before the check fires. The saving +is bandwidth, not CPU/DB work. + +### `@condition` decorator — materia and norma detail views + +For `MateriaLegislativaCrud.DetailView` and `NormaCrud.DetailView` a cheap +freshness function runs *before* the view body: + +```python +# sapl/materia/views.py +def _materia_last_modified(request, *args, **kwargs): + return MateriaLegislativa.objects.filter( + pk=kwargs['pk'] + ).values_list('data_ultima_atualizacao', flat=True).first() + +def _materia_etag(request, *args, **kwargs): + ts = _materia_last_modified(request, *args, **kwargs) + return f'{kwargs["pk"]}-{ts.timestamp()}' if ts else None + +@method_decorator(condition(etag_func=_materia_etag, last_modified_func=_materia_last_modified), name='get') +class DetailView(AnonCachePageMixin, Crud.DetailView): + ... +``` + +`NormaCrud.DetailView` follows the same pattern with `_norma_last_modified` / +`_norma_etag` querying `NormaJuridica.ultima_edicao`. + +**On a cache hit**: one `VALUES` query fires, Django returns `304` — view body, +template render, and ORM work are all skipped. + +**Signal used**: `data_ultima_atualizacao` (`auto_now=True`) — updated by Django +on every `save()`, so the ETag is invalidated automatically whenever the record +changes. + +--- + +## nginx Lua layer — implementation notes + +### Current stack: nginx + libnginx-mod-http-lua + +`blocklist.lua` uses Debian's `libnginx-mod-http-lua` + `lua-resty-redis` packages. These are built against the same `nginx` version as `libnginx-mod-http-geoip2`, so all three modules are ABI-compatible and available for both `amd64` and `arm64`. + +What the Lua layer does: + +| Check | Mechanism | Redis I/O | +|-------|-----------|-----------| +| IP-prefix block | `ngx.shared.ip_prefix_blocked` dict (60 s refresh) | 0 per request | +| Global IP block | `GET rl:ip:{ip}:blocked` | pipelined | +| Per-tenant API block | `GET rl:api:ns:{ns}:ip:{ip}:blocked` | pipelined (1 round trip for both GETs) | + +ASN blocking and UA blocking remain in nginx `if()` blocks (rewrite phase, before Lua access phase) — no Redis involved. + +### Why OpenResty was not used + +OpenResty was evaluated and rejected for this deployment: + +1. **No arm64 Debian packages.** The `openresty.org/package/debian` repo publishes only `amd64`. Local Mac builds (Apple Silicon) and any future ARM node pools would need `--platform linux/amd64` + QEMU emulation. + +2. **No functional difference for this use case.** The only OpenResty features beyond `libnginx-mod-http-lua` that were considered: + - *Bundled resty.* libraries* — all needed ones (`resty.redis`, `resty.core`, `resty.lrucache`) are in Debian packages. + - *OPM* — used only to install `lua-resty-maxminddb` for ASN lookups in Lua. Dropped when ASN blocking moved back to the GeoIP2 C module, which requires no OPM. + - *Slightly newer LuaJIT* — irrelevant at this workload. + +### Future opportunity: migrate to OpenResty + +Migration would be worth considering **only if** one of the following becomes a requirement: + +| Trigger | Why it needs OpenResty | +|---------|----------------------| +| ASN lookup in Lua (remove GeoIP2 C module dependency) | Needs `lua-resty-maxminddb` via OPM; not in Debian repos | +| Per-request rate counting in nginx (move counters out of Django) | `resty.limit.req` / `resty.limit.conn` — not in Debian's lua packages | +| Cosocket-based upstream health checks or dynamic upstreams | OpenResty's `resty.upstream.*` ecosystem | +| JWT validation at nginx layer | `lua-resty-jwt` via OPM | + +Migration steps when the time comes: +1. Add `--platform linux/amd64` to `docker build` (or use a multi-arch CI builder). +2. Replace `nginx libnginx-mod-http-geoip2 libnginx-mod-http-lua lua-resty-redis` with `openresty libmaxminddb0` + `opm get anjia0532/lua-resty-maxminddb`. +3. Update config paths: `/etc/nginx/` → `/usr/local/openresty/nginx/conf/`; binary `/usr/sbin/nginx` → `/usr/local/openresty/nginx/sbin/nginx`. +4. Replace `geoip2` C-module directives + `$bot_asn` map with `mmdb.lookup(ip)` in `blocklist.lua`. +5. Remove `lua_package_path` directive (OpenResty bundles all resty.* paths automatically). + +--- + +## Open Questions + +| # | Question | Status | Blocks | +|---|----------|--------|--------| +| 1 | Does Chrome/98.0.4758 impersonator appear consistently in nginx access logs? | Needs investigation | UA block safety | + +| 3 | `CONN_MAX_AGE` tuning | Currently **300 s** (`sapl/settings.py`). Evaluate whether to reduce given worker recycling at 400 MB. | Gunicorn tuning | +| 4 | WebSocket voting panel priority | Separate project. Resumes after Redis is on k8s, bot siege addressed, and OOM pressure reduced. | Phase 5 sequencing | diff --git a/plan/rate-limiter-v2.md b/plan/rate-limiter-v2.md new file mode 100644 index 000000000..6236663c8 --- /dev/null +++ b/plan/rate-limiter-v2.md @@ -0,0 +1,1224 @@ +# SAPL — OOM Investigation & Remediation Plan (v2) + +> **Scope**: Django 2.2 / Gunicorn / nginx / Kubernetes fleet of 1,200+ pods. +> Each pod has a dedicated PostgreSQL instance. A K8s Ingress sits in front of all tenants. +> **This document is canonical** — all earlier session notes are consolidated here. + +--- + +## Table of Contents + +1. [Architecture Overview](#0-architecture-overview) +2. [Context & Problem Statement](#1-context--problem-statement) +3. [Decision Log](#2-decision-log) +4. [Phase 0 — Immediate Hardening (No New Infra)](#3-phase-0--immediate-hardening-no-new-infra) +5. [Phase 1 — Shared Redis (Single Pod)](#4-phase-1--shared-redis-single-pod) +6. [Phase 2 — Rate Limiting & Bot Mitigation](#5-phase-2--rate-limiting--bot-mitigation) +7. [Phase 3 — File Serving Corrections](#6-phase-3--file-serving-corrections) +8. [Phase 4 — Dynamic Page Caching](#7-phase-4--dynamic-page-caching) +9. [Open Questions](#8-open-questions) + +--- + +## 0. Architecture Overview + +### 0.1 Component Diagram + +```mermaid +graph TD + Client([Bot / Human Client]) + nginx[nginx\nDebian pkg] + gunicorn[Gunicorn\n2 workers / 4 threads] + mw[Django Middleware\nRateLimitMiddleware] + view[View Layer\nCBV + decorators] + redis[(Redis\nDB0: cache\nDB1: rate limiter)] + pg[(PostgreSQL\nper-pod)] + fs[Filesystem\nPDFs / media] + + Client -->|HTTP| nginx + nginx -->|proxy_pass| gunicorn + gunicorn --> mw + mw -->|pass| view + mw -->|429| nginx + view --> pg + view --> fs + view --> redis + mw --> redis + nginx -->|SISMEMBER / GET| redis +``` + +> DB2 is reserved for Django Channels (WebSocket — future Phase 5). + +### 0.2 Redis Memory Budget and Key Layout + +| Key type | Key schema | TTL | DB | Est. size | +|---|---|---|---|---| +| Page / view cache | `cache:{ns}:` | 60–600 s | 0 | ~0.5 GB | +| Static cache (images/logos) | `cache:{ns}:static:{sha256}` | 3–24 h | 0 | ~2.4 GB | +| PDF cache (≤ 360 KB) | `cache:{ns}:file:{sha256}` | 1 h | 0 | ~0.9 GB | +| IP request counter | `rl:ip:{ip}:reqs` | 60 s | 1 | ~0.6 MB | +| IP blocked marker | `rl:ip:{ip}:blocked` | 300 s | 1 | ~0.06 MB | +| User request counter | `rl:{ns}:user:{id}:reqs` | 60 s | 1 | negligible | +| User blocked marker | `rl:{ns}:user:{id}:blocked` | 300 s | 1 | negligible | +| Path counter | `rl:{ns}:path:{sha256}:reqs` | 60 s | 1 | ~0.3 MB | +| UA deny list | `rl:bot:ua:blocked` | permanent SET | 1 | ~0.03 MB | +| NS/IP/window counter | `rl:ns:{ns}:ip:{ip}:w:{bucket}` | 60 s × 2 | 1 | ~0.6 MB | +| **Redis overhead (× 1.5)** | | | | ~1.6 GB | +| **Total ceiling** | | | | **~5 GB** | + +**Key conventions:** +- `{ns}` = Kubernetes namespace (tenant identifier). All path and user keys include it. +- `{user}` / `{id}` = normalized user PK: `str(user.pk).lower().strip()`. +- Django `CACHES` uses `KEY_PREFIX: "cache:{ns}"` (e.g. `cache:sapl:`) to namespace all DB0 cache keys. + DB1 (rate limiter) uses raw `rl:*` keys — no prefix — for compatibility with the Lua / middleware INCR scripts. +- DB2 is reserved for Django Channels; allocate separately when WebSocket work resumes. + +--- + +## 1. Context & Problem Statement + +### Fleet + +| Item | Detail | +|---|---| +| System | SAPL — Django 2.2, legislative management for Brazilian municipal chambers | +| Fleet | ~1,200 Kubernetes pods, each with a dedicated PostgreSQL pod | +| Pod limits | 1 core CPU (limit) / 35m (request) · 1600Mi RAM (limit) / 800Mi (request) | +| Users | Legislative house staff, often behind NAT (many users, one public IP) | +| Workloads | PDF generation (synchronous, ReportLab), file uploads up to 150 MB, WebSocket voting panel | + +### OOM Kill Pattern + +Workers grow from ~35 MB at birth to 800–900 MB within 2–3 minutes, then are killed and replaced in a continuous cycle. + +Root causes: +- Bot scraping triggers synchronous PDF generation — entire document built in RAM (ReportLab) +- `worker_max_memory_per_child` only checks **between requests**; workers blocked on long requests are never recycled +- `TIMEOUT=300` lets bots hold threads for up to 5 minutes while memory accumulates +- 3 workers × 300 MB each = ~900 MB — breaching the 800Mi request threshold + +### Bot Traffic Profile (Barueri pod, 16 days, 662k requests) + +| Actor | Requests | % of total | +|---|---|---| +| Googlebot | ~154,000 | 23.2% | +| Chrome/98.0.4758 (spoofed scraper) | 90,774 | 13.7% | +| kube-probe (healthcheck) | 69,065 | 10.4% | +| meta-externalagent | 28,325 | 4.3% | +| GPTBot | 11,489 | 1.7% | +| bingbot | 7,639 | 1.1% | +| OAI-SearchBot + Applebot | 6,681 | 1.0% | +| **Total identified bots** | **~377,000** | **~56.9%** | + +**Botnet fingerprint:** +- Rotates User-Agents (Chrome/121, Chrome/122, Firefox/123, Safari/17…) across requests +- Crawls all sub-endpoints of the same matéria within 1 second from different IPs +- Distributes crawling across tenants — each pod stays under the per-pod rate limit, never triggering it +- Primary targets: `/relatorios/{id}/etiqueta-materia-legislativa` (~40 KB PDF) and all `/materia/{id}/*` sub-endpoints + +### Static File Traffic (from CSV analysis) + +| Category | Requests | Transfers | +|---|---|---| +| Logos / images | 62,776 | ~24 GB | +| PDFs | 8,869 | 5.1 GB | +| Parliamentarian photos | 11,856 | ~0.5 GB | +| **Total** | **83,501** | **~30 GB** | + +Top offender: `Brasão - Foz do Iguaçu.png` — 14,512 requests, 5.6 GB from a single 392 KB file. + +### Confirmed Bugs + +```nginx +# nginx.conf — WRONG (disables kernel bypass) +sendfile off; + +# sapl.conf — missing on /media/ location +location /media/ { + alias /var/interlegis/sapl/media/; + # no ETag, no Cache-Control, no X-Robots-Tag +} +``` + +```python +# settings.py — per-pod cache, not shared +CACHES = { + 'default': { + 'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache', + 'LOCATION': '/var/tmp/django_cache', + 'OPTIONS': {"MAX_ENTRIES": 10000}, + } +} +``` + +Django rate limiter (`django-ratelimit` at 35/m) uses `FileBasedCache` — counters are isolated per pod, making rate limiting completely ineffective at fleet scale. + +### Hard Constraints + +| Constraint | Impact | +|---|---| +| Per-pod PostgreSQL | Rate-limit counters not shared across pods | +| No Redis initially | No shared state for rate limiting or caching | +| NAT environments | IP-based rate limiting causes false positives | +| `TIMEOUT=300` / uploads to 150 MB | Must not be broken — intentional for slow workflows | + +--- + +## 2. Decision Log + +| Decision | Chosen | Rationale | Session | +|---|---|---|---| +| Redis topology | **Single pod** (no Sentinel, no Cluster) | 65 MB of active data fits comfortably on one node; cluster complexity not justified at this data volume | v2 | +| PDF caching in Redis | **No** — ETags + sendfile are sufficient | Once rate limiting + ETags are active, repeat requests become 304s with zero bytes transferred | Session 4 | +| nginx rate-limit end state | **Django middleware** with shared Redis | No nginx image changes required; solves cross-pod consistency immediately | Session 5 | +| `worker_max_memory_per_child` | **400 MB** | Pod limit 1600Mi, 2 workers × 400 MB = 800 MB — leaves 800 Mi headroom; previous 300 MB was OOMKilled before recycling could act | v2 | +| `sendfile off` | **Bug** — flip to `on` | No valid production reason found in uploaded config; disabling userspace copy is always correct | Session 5 | +| nginx serves `/media/` directly | Confirmed via `alias` in `sapl.conf` | `X-Accel-Redirect` only needed for LGPD-restricted documents | Session 5 | +| Cache backend switch timing | **At pod startup** via `start.sh` + waffle switch | Pod restart is acceptable; avoids per-request runtime overhead | Session 5 | +| Secret injection | Per-namespace Secret with `optional: true` | Enables gradual rollout; pod starts on file cache if Secret is absent | Session 5 | +| Redis k8s files location | `$PROJECT_ROOT/docker/k8s/` | Consistent with existing Docker artifacts in the repo | v2 | + +--- + +## 3. Phase 0 — Immediate Hardening (No New Infra) + +**Goal**: Stop the OOM kill cycle and reduce bot load with zero infrastructure additions. +**Risk**: Low — all changes are config-only. + +### 3.1 Gunicorn Tuning + +The core tension: reducing workers protects memory but reduces concurrency. The fix is to reduce the **number** of workers (from 3 to 2) and raise the per-worker **ceiling** so the recycling mechanism has time to act. + +```python +# docker/startup_scripts/gunicorn.conf.py +import os +import pathlib + +NAME = "SAPL" +DJANGODIR = "/var/interlegis/sapl" +SOCKFILE = f"unix:{DJANGODIR}/run/gunicorn.sock" +USER = "sapl" +GROUP = "nginx" + +NUM_WORKERS = int(os.getenv("WEB_CONCURRENCY", "2")) # was 3 +THREADS = int(os.getenv("GUNICORN_THREADS", "4")) # was 8 +TIMEOUT = int(os.getenv("GUNICORN_TIMEOUT", "120")) # was 300 +WORKER_CLASS = "gthread" +DJANGO_SETTINGS = "sapl.settings" +WSGI_APP = "sapl.wsgi:application" + +proc_name = NAME +bind = SOCKFILE +umask = 0o007 +user = USER +group = GROUP +chdir = DJANGODIR +wsgi_app = WSGI_APP + +loglevel = "info" # was debug — reduces log I/O +accesslog = "/var/log/sapl/access.log" +errorlog = "/var/log/sapl/error.log" +capture_output = True + +workers = NUM_WORKERS +worker_class = WORKER_CLASS +threads = THREADS +timeout = TIMEOUT +graceful_timeout = 30 +keepalive = 10 +backlog = 2048 + +max_requests = 1000 +max_requests_jitter = 200 +worker_max_memory_per_child = 400 * 1024 * 1024 # 400 MB — was 300 MB + +raw_env = [f"DJANGO_SETTINGS_MODULE={DJANGO_SETTINGS}"] +preload_app = False + +def on_starting(server): + pathlib.Path(SOCKFILE).parent.mkdir(parents=True, exist_ok=True) + +def post_fork(server, worker): + try: + from django import db + db.connections.close_all() + except Exception: + pass +``` + +**Per-location timeout strategy** — replace the one-size-fits-all 300s: + +| Operation | Previous | Recommended | Rationale | +|---|---|---|---| +| Normal page rendering | 300 s | 60 s | No legitimate page should take > 60 s | +| API endpoints | 300 s | 30 s | Stateless, fast by design | +| PDF download (cached / nginx) | 300 s | 30 s | nginx serves from disk, worker not involved | +| PDF generation (uncached) | 300 s | 180 s | Kept high — addressed in Phase 5 | +| Large file upload | 300 s | 180 s | nginx buffers upload, worker processes after | + +--- + +### 3.2 nginx Fixes + +Three confirmed bugs in the uploaded config — all fixed here. + +```nginx +# /etc/nginx/nginx.conf — http {} block + +# FIX 1: kernel bypass (was off — CRITICAL) +sendfile on; +tcp_nopush on; +tcp_nodelay on; + +# FIX 2: reduced timeouts (was 300s everywhere) +keepalive_timeout 75; +proxy_read_timeout 120s; # overridden per-location for slow ops +proxy_connect_timeout 10s; +proxy_send_timeout 120s; + +# Real client IP from X-Forwarded-For set by K8s Ingress +real_ip_header X-Forwarded-For; +real_ip_recursive on; +set_real_ip_from 10.0.0.0/8; +set_real_ip_from 172.16.0.0/12; +set_real_ip_from 192.168.0.0/16; +``` + +```nginx +# sapl.conf — FIX 3: add caching headers to /media/ +location /media/ { + alias /var/interlegis/sapl/media/; + sendfile on; + etag on; + add_header Cache-Control "public, max-age=86400, stale-while-revalidate=3600"; + add_header X-Robots-Tag "noindex" always; +} +``` + +**Upload endpoints** — keep `proxy_request_buffering on` so nginx absorbs slow uploads before handing off to Gunicorn: + +```nginx +location ~* ^/(protocoloadm/criar-protocolo|materia/.*upload|norma/.*upload) { + proxy_request_buffering on; + proxy_read_timeout 180s; + proxy_send_timeout 180s; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $http_host; + proxy_redirect off; + proxy_pass http://sapl_server; +} +``` + +--- + +### 3.3 Bot UA Blocklist in nginx + +Blocks known bots at nginx — before any Gunicorn worker is allocated. + +```nginx +# nginx.conf — http {} block +map $http_user_agent $bot_ua_blocked { + default 0; + "~*GPTBot" 1; + "~*ClaudeBot" 1; + "~*PerplexityBot" 1; + "~*Bytespider" 1; + "~*AhrefsBot" 1; + "~*SemrushBot" 1; + "~*DotBot" 1; + "~*meta-externalagent" 1; + "~*OAI-SearchBot" 1; + "~*Chrome/98\.0\.4758" 1; # confirmed scraper — no real user runs a 2022 browser in 2026 +} + +# sapl.conf — server {} block (before any location) +if ($bot_ua_blocked = 1) { + return 429 "Too Many Requests"; +} +``` + +**Limitation**: Bots with rotating or spoofed UAs are not caught here. They are handled by Django middleware in Phase 2 (checks 3–5). This is intentional — nginx handles the cheap deterministic case; Django handles the expensive probabilistic case. + +--- + +### 3.4 ASN-Based Blocking (Mandatory) + +Blocks bot traffic by datacenter ASN — before UA parsing, before any Python process is touched. + +**Step 1 — install the GeoIP2 module and database:** + +```bash +# Debian / Ubuntu +apt install libnginx-mod-http-geoip2 libmaxminddb0 mmdb-bin + +# Download GeoLite2-ASN (free MaxMind account required) +mkdir -p /etc/nginx/geoip +curl -sL "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=YOUR_KEY&suffix=tar.gz" \ + | tar -xz --strip-components=1 --wildcards '*.mmdb' -C /etc/nginx/geoip/ +``` + +**Step 2 — configure nginx:** + +```nginx +# nginx.conf — top-level (outside http {}) +load_module modules/ngx_http_geoip2_module.so; + +# nginx.conf — http {} block +geoip2 /etc/nginx/geoip/GeoLite2-ASN.mmdb { + $geoip2_asn_number autonomous_system_number; + $geoip2_asn_org autonomous_system_organization; +} + +map $geoip2_asn_number $bot_asn { + default 0; + 16509 1; # Amazon AWS + 14618 1; # Amazon AWS us-east + 8075 1; # Microsoft Azure + 396982 1; # Google Cloud + 20473 1; # Vultr + 24940 1; # Hetzner + 16276 1; # OVH + 36352 1; # ColoCrossing + 63949 1; # Linode / Akamai +} + +# sapl.conf — server {} block (before bot_ua_blocked check) +if ($bot_asn = 1) { + return 429 "Too Many Requests"; +} +``` + +**Step 3 — keep the database fresh** (host cron — no k8s CronJob): + +```bash +# /etc/cron.weekly/update-geoip +#!/bin/bash +curl -sL "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=${MAXMIND_KEY}&suffix=tar.gz" \ + | tar -xz -C /tmp --wildcards '*.mmdb' +mv /tmp/GeoLite2-ASN_*/GeoLite2-ASN.mmdb /etc/nginx/geoip/GeoLite2-ASN.mmdb +nginx -s reload +``` + +**Tradeoff**: Blocks datacenter ASNs where bots originate. May over-block VPN users and developers on cloud instances. + +--- + +### 3.5 robots.txt + +Passive mitigation — effective over days/weeks for compliant bots. The spoofed Chrome/98 botnet ignores it; handled by nginx UA blocking above. + +``` +# Place at /var/interlegis/sapl/collected_static/robots.txt +User-agent: GPTBot +Disallow: / +Crawl-delay: 10 + +User-agent: ClaudeBot +Disallow: / +Crawl-delay: 10 + +User-agent: meta-externalagent +Disallow: / +Crawl-delay: 10 + +User-agent: OAI-SearchBot +Disallow: / +Crawl-delay: 10 + +User-agent: * +Disallow: /relatorios/ +Crawl-delay: 10 +``` + +Serve directly from nginx (no Django involvement): + +```nginx +# sapl.conf +location = /robots.txt { + alias /var/interlegis/sapl/collected_static/robots.txt; +} +``` + +--- + +### 3.6 N+1 Fix in `get_etiqueta_protocolos` + +Confirmed in `sapl/protocoloadm/utils.py` — `MateriaLegislativa.objects.filter()` called inside a loop over protocols. Two queries total regardless of volume: + +```python +# BEFORE — one query per protocol (N+1) +def get_etiqueta_protocolos(prots): + protocolos = [] + for p in prots: + dic = {} + for materia in MateriaLegislativa.objects.filter( + numero_protocolo=p.numero, ano=p.ano): + dic['num_materia'] = ( + materia.tipo.sigla + ' ' + + str(materia.numero) + '/' + str(materia.ano) + ) + protocolos.append(dic) + return protocolos + + +# AFTER — two queries total regardless of volume +def get_etiqueta_protocolos(prots): + from django.db.models import Q + import functools, operator + + prot_list = list(prots) + if not prot_list: + return [] + + query = functools.reduce( + operator.or_, + [Q(numero_protocolo=p.numero, ano=p.ano) for p in prot_list] + ) + materias_map = { + (m.numero_protocolo, m.ano): m + for m in MateriaLegislativa.objects.filter(query).select_related('tipo') + } + + protocolos = [] + for p in prot_list: + dic = {} + materia = materias_map.get((p.numero, p.ano)) + dic['num_materia'] = ( + f"{materia.tipo.sigla} {materia.numero}/{materia.ano}" + if materia else '' + ) + # ... rest of existing loop body unchanged + protocolos.append(dic) + return protocolos +``` + +--- + +### 3.7 ETags / 304 Responses + +Adding `etag on` and `Cache-Control` to the `/media/` location (§3.2) converts repeat bot requests from full downloads to 304 responses with empty bodies. + +For `Brasão - Foz do Iguaçu.png` (392 KB × 14,512 requests = **5.6 GB**), even a 50% conditional hit rate saves ~2.8 GB immediately — without any Redis. + +**Why this is sufficient for PDFs**: See Phase 3 §6.2. + +--- + +### 3.8 Django Upload Settings + +```python +# sapl/settings.py +# Files above 2 MB are streamed to a temp file on disk rather than +# held in worker RAM. Critical for 150 MB upload support. +FILE_UPLOAD_MAX_MEMORY_SIZE = 2 * 1024 * 1024 # 2 MB +DATA_UPLOAD_MAX_MEMORY_SIZE = 10 * 1024 * 1024 # 10 MB +MAX_DOC_UPLOAD_SIZE = 150 * 1024 * 1024 # 150 MB +FILE_UPLOAD_TEMP_DIR = '/var/interlegis/sapl/tmp' +``` + +--- + +## 4. Phase 1 — Shared Redis (Single Pod) + +**Goal**: Deploy Redis so all subsequent phases have shared state. +**Risk**: Medium — new stateful infrastructure. Non-fatal fallback to file cache if Redis is unreachable. + +### 4.1 Redis Kubernetes Manifests + +Files live under `$PROJECT_ROOT/docker/k8s/`. + +```yaml +# docker/k8s/redis-configmap.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: redis-config + namespace: redis +data: + redis.conf: | + save "" + appendonly no + + maxmemory 5gb + maxmemory-policy allkeys-lru + maxmemory-samples 10 + + maxclients 20000 + tcp-backlog 511 + timeout 300 + tcp-keepalive 60 + + hz 20 + lazyfree-lazy-eviction yes + lazyfree-lazy-expire yes + lazyfree-lazy-server-del yes + + slowlog-log-slower-than 10000 + slowlog-max-len 256 + latency-monitor-threshold 10 + + bind 0.0.0.0 + protected-mode no + databases 4 # DB0: cache, DB1: rate limiter, DB2: channels (future) + + activedefrag yes + active-defrag-ignore-bytes 100mb + active-defrag-threshold-lower 10 +``` + +```yaml +# docker/k8s/redis-pod.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sapl-redis + namespace: redis +spec: + replicas: 1 + selector: + matchLabels: + app: sapl-redis + template: + metadata: + labels: + app: sapl-redis + spec: + containers: + - name: redis + image: redis:7-alpine + command: ["redis-server", "/etc/redis/redis.conf"] + resources: + requests: + memory: "1Gi" + cpu: "250m" + limits: + memory: "6Gi" + cpu: "1000m" + ports: + - containerPort: 6379 + volumeMounts: + - name: redis-config + mountPath: /etc/redis + volumes: + - name: redis-config + configMap: + name: redis-config +``` + +```yaml +# docker/k8s/redis-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: sapl-redis + namespace: redis +spec: + selector: + app: sapl-redis + ports: + - port: 6379 + targetPort: 6379 +``` + +**Pod budget rationale:** + +| Data type | Estimated memory | +|---|---| +| Rate limit counters (all pods, all IPs) | ~50–110 MB | +| View / template cache | ~300–600 MB | +| Small file cache (logos, etiquetas) | ~500 MB–1 GB | +| Redis overhead (× 1.5) | ~1.6 GB | +| **Total ceiling** | **~5 GB** | + +--- + +### 4.2 Use-Case / Key-Prefix Mapping + +| Use case | Key prefix | DB | TTL | Notes | +|---|---|---|---|---| +| Page / view cache | `cache:{ns}:*` | 0 | 60–600 s | `KEY_PREFIX=cache:{ns}` in Django CACHES | +| Static file cache (logos) | `cache:{ns}:static:{sha256}` | 0 | 3–24 h | ns = POD_NAMESPACE | +| PDF cache (≤ 360 KB) | `cache:{ns}:file:{sha256}` | 0 | 1 h | ns required | +| Rate limiter counters | `rl:*` | 1 | 60–300 s | Raw keys, no prefix | +| UA deny list | `rl:bot:ua:blocked` | 1 | permanent SET | Seed once after deploy | +| WebSocket / Channels | `channels:*` | 2 | session TTL | **Future — Phase 5** | + +--- + +### 4.3 Django Settings — Startup-Time Backend Selection + +```python +# sapl/settings.py +REDIS_URL = config('REDIS_URL', default='') +CACHE_BACKEND = config('CACHE_BACKEND', default='file') + +_redis_ready = CACHE_BACKEND == 'redis' and bool(REDIS_URL) + +CACHES = { + 'default': { + 'BACKEND': ( + 'django_redis.cache.RedisCache' if _redis_ready + else 'django.core.cache.backends.filebased.FileBasedCache' + ), + 'LOCATION': REDIS_URL + '/0' if _redis_ready else '/var/tmp/django_cache', + 'KEY_PREFIX': f'cache:{POD_NAMESPACE}', # e.g. "cache:sapl:" or "cache:sapl31demo-df:" + **( + { + 'OPTIONS': { + 'CLIENT_CLASS': 'django_redis.client.DefaultClient', + 'CONNECTION_POOL_KWARGS': { + # 1,200 pods × 2 workers × 6 = 14,400 peak connections + # maxclients=20,000 gives 40% headroom + 'max_connections': 6, + 'socket_timeout': 0.5, + 'socket_connect_timeout': 0.5, + }, + 'IGNORE_EXCEPTIONS': True, # cache miss on Redis failure — app degrades gracefully + }, + 'TIMEOUT': 300, + } if _redis_ready else { + 'OPTIONS': {'MAX_ENTRIES': 10000}, + } + ), + }, + 'ratelimit': { + 'BACKEND': 'django_redis.cache.RedisCache', + 'LOCATION': REDIS_URL + '/1' if _redis_ready else '', + 'OPTIONS': { + 'CLIENT_CLASS': 'django_redis.client.DefaultClient', + 'CONNECTION_POOL_KWARGS': { + 'max_connections': 6, + 'socket_timeout': 0.5, + 'socket_connect_timeout': 0.5, + }, + 'IGNORE_EXCEPTIONS': True, + }, + } if _redis_ready else { + 'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache', + 'LOCATION': '/var/tmp/django_ratelimit_cache', + 'OPTIONS': {'MAX_ENTRIES': 5000}, + }, +} + +RATELIMIT_USE_CACHE = 'ratelimit' +``` + +`start.sh` additions — resolve URL and read waffle switch before Gunicorn starts: + +```bash +resolve_redis_url() { + # 1. Already set by local Secret via envFrom — highest precedence + [[ -n "${REDIS_URL:-}" ]] && { log "REDIS_URL from local secret."; return 0; } + + # 2. Try global cluster Secret via k8s API + local api="https://kubernetes.default.svc" + local token ca + token="$(<'/var/run/secrets/kubernetes.io/serviceaccount/token')" + ca="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" + + local url + url=$(curl -sf --cacert "$ca" \ + -H "Authorization: Bearer $token" \ + "${api}/api/v1/namespaces/interlegis-infra/secrets/sapl-global-redis" \ + | python3 -c " +import sys, json, base64 +d = json.load(sys.stdin).get('data', {}) +v = d.get('REDIS_URL', '') +print(base64.b64decode(v).decode() if v else '') +" 2>/dev/null || echo "") + + if [[ -n "$url" ]]; then + export REDIS_URL="$url" + log "REDIS_URL from global cluster secret." + return 0 + fi + log "No REDIS_URL found — file-based cache will be used." +} + +resolve_cache_backend() { + [[ -z "${REDIS_URL:-}" ]] && return 0 + log "REDIS_URL set — checking REDIS_CACHE waffle switch..." + local active + active=$(psql "$DATABASE_URL" -At -v ON_ERROR_STOP=0 -c \ + "SELECT active FROM waffle_switch WHERE name='REDIS_CACHE' LIMIT 1;" \ + 2>/dev/null || echo "") + if [[ "$active" == "t" ]]; then + log "REDIS_CACHE switch ON — activating Redis cache backend." + export CACHE_BACKEND="redis" + else + log "REDIS_CACHE switch OFF — using file-based cache." + export CACHE_BACKEND="file" + fi +} + +wait_for_redis() { + [[ -z "${REDIS_URL:-}" ]] && return 0 + log "Checking Redis connectivity..." + local host port + host=$(python3 -c "from urllib.parse import urlparse; u=urlparse('${REDIS_URL}'); print(u.hostname or 'localhost')") + port=$(python3 -c "from urllib.parse import urlparse; u=urlparse('${REDIS_URL}'); print(u.port or 6379)") + local retries=10 + until python3 -c "import socket; s=socket.create_connection(('${host}',${port}),2); s.close()" 2>/dev/null; do + retries=$((retries-1)) + [[ $retries -eq 0 ]] && { log "WARNING: Redis unreachable — continuing on file cache."; return 0; } + log "Waiting for Redis... ($retries retries left)" + sleep 2 + done + log "Redis reachable at ${host}:${port}." +} + +configure_redis_cache() { + [[ -z "${REDIS_URL:-}" ]] && return 0 + log "Creating REDIS_CACHE waffle switch (default: off)" + python3 manage.py waffle_switch REDIS_CACHE off --create +} +``` + +--- + +### 4.4 Rollout Sequence + +```bash +# Enable Redis for one namespace +kubectl create secret generic sapl-redis \ + --namespace=fortaleza-ce \ + --from-literal=REDIS_URL="redis://sapl-redis.redis.svc.cluster.local:6379" \ + --dry-run=client -o yaml | kubectl apply -f - + +kubectl exec -n fortaleza-ce deploy/sapl -- \ + python manage.py waffle_switch REDIS_CACHE on --create + +kubectl rollout restart deployment/sapl -n fortaleza-ce + +# Disable without removing secret +kubectl exec -n fortaleza-ce deploy/sapl -- \ + python manage.py waffle_switch REDIS_CACHE off +kubectl rollout restart deployment/sapl -n fortaleza-ce + +# Fleet-wide rollout (parallel) +kubectl get namespaces -l app=sapl -o name | sed 's|namespace/||' | \ + xargs -P 10 -I{} kubectl exec -n {} deploy/sapl -- \ + python manage.py waffle_switch REDIS_CACHE on --create + +kubectl get namespaces -l app=sapl -o name | sed 's|namespace/||' | \ + xargs -P 5 -I{} kubectl rollout restart deployment/sapl -n {} +``` + +**Seed the UA deny list once after Redis is deployed:** + +```bash +kubectl exec -n redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:bot:ua:blocked \ + "$(echo -n 'GPTBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'ClaudeBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'PerplexityBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'Bytespider' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'AhrefsBot' | sha256sum | cut -d' ' -f1)" \ + "$(echo -n 'meta-externalagent' | sha256sum | cut -d' ' -f1)" + +# Add new offenders at runtime without restart +kubectl exec -n redis deploy/sapl-redis -- redis-cli -n 1 \ + SADD rl:bot:ua:blocked "$(echo -n 'NewBot/1.0' | sha256sum | cut -d' ' -f1)" +``` + +**Production monitoring commands:** + +```bash +# Memory usage +kubectl exec -n redis deploy/sapl-redis -- redis-cli info memory \ + | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio' + +# Connection pressure +kubectl exec -n redis deploy/sapl-redis -- redis-cli info stats \ + | grep -E 'rejected_connections|instantaneous_ops_per_sec' + +# Key distribution per DB +kubectl exec -n redis deploy/sapl-redis -- redis-cli info keyspace + +# Slow log +kubectl exec -n redis deploy/sapl-redis -- redis-cli slowlog get 25 +``` + +--- + +### 4.5 Inspecting Redis State + +#### CLI quick-reference (redis-cli or `kubectl exec`) + +```bash +# ── Connection ───────────────────────────────────────────────────────────── +# docker-compose +redis-cli -h localhost -p 6379 + +# k8s pod +kubectl exec -n deploy/sapl-redis -- redis-cli + +# ── DB selection (always specify -n for rate-limiter work) ───────────────── +# DB0 = page cache DB1 = rate limiter DB2 = channels (future) +redis-cli -n 1 # select DB1 + +# ── Key inspection ───────────────────────────────────────────────────────── +# List all rate-limiter keys +redis-cli -n 1 KEYS "rl:*" + +# Request counter for a specific IP +redis-cli -n 1 GET "rl:ip:203.0.113.1:reqs" + +# Remaining TTL on a counter +redis-cli -n 1 TTL "rl:ip:203.0.113.1:reqs" + +# Check if an IP is hard-blocked +redis-cli -n 1 EXISTS "rl:ip:203.0.113.1:blocked" + +# Authenticated user counter (ns = POD_NAMESPACE, uid = user pk) +redis-cli -n 1 GET "rl:sapl:user:42:reqs" + +# Namespace/IP sliding window (bucket = epoch // 60) +redis-cli -n 1 KEYS "rl:sapl:ip:203.0.113.1:w:*" + +# ── Manual block / unblock ───────────────────────────────────────────────── +# Block an IP for 5 minutes +redis-cli -n 1 SET "rl:ip:1.2.3.4:blocked" 1 EX 300 + +# Immediately unblock an IP +redis-cli -n 1 DEL "rl:ip:1.2.3.4:blocked" + +# Unblock a user +redis-cli -n 1 DEL "rl:sapl:user:42:blocked" + +# ── Aggregate stats ──────────────────────────────────────────────────────── +# Count all blocked IPs right now +redis-cli -n 1 KEYS "rl:ip:*:blocked" | wc -l + +# Count all blocked users +redis-cli -n 1 KEYS "rl:*:user:*:blocked" | wc -l + +# Total DB1 key count +redis-cli -n 1 DBSIZE + +# Memory used by DB1 +redis-cli INFO keyspace | grep "db1" + +# ── Cache DB inspection (DB0) ─────────────────────────────────────────────── +# Count cached page responses (KEY_PREFIX = cache:{ns}, e.g. "cache:sapl:") +redis-cli -n 0 KEYS "cache:sapl:*" | wc -l + +# Memory used by DB0 +redis-cli INFO keyspace | grep "db0" +``` + +#### RedisInsight + +Open `http://localhost:5540` (or whatever port you mapped) and connect to: +- **Host**: `localhost` (or the k8s service name) +- **Port**: `6379` +- **Database**: switch between DB0 (cache) and DB1 (rate limiter) using the database selector + +Filter keys by prefix `rl:ip:` to see all anonymous IP counters, `rl:*:user:` for authenticated users. + +#### Populate synthetic test data + +```bash +# Inject test entries to validate key schema and blocking thresholds +python3 docker/scripts/redis_inject_test_data.py + +# Point at a non-default Redis +REDIS_URL=redis://sapl-redis.redis.svc:6379 python3 docker/scripts/redis_inject_test_data.py + +# Clear all synthetic entries written by the script +CLEAR=1 python3 docker/scripts/redis_inject_test_data.py +``` + +--- + +## 5. Phase 2 — Rate Limiting & Bot Mitigation + +**Goal**: Effective cross-pod throttling using shared Redis. +**Prerequisite**: Phase 1 (Redis deployed and `CACHE_BACKEND=redis`). + +### 5.1 Middleware Architecture + +```mermaid +flowchart TD + A([Request arrives at nginx]) --> B{SISMEMBER\nrl:bot:ua:blocked} + B -->|hit| Z1[429 — zero Django cost] + B -->|miss| C{GET\nrl:ip:blocked} + C -->|exists| Z2[429 — zero Django cost] + C -->|nil| D[proxy_pass to Gunicorn] + D --> E{authenticated?} + E -->|yes| F{INCR\nrl:{ns}:user:{id}:reqs\n>= 120/min?} + E -->|no| G{suspicious\nheaders?} + F -->|yes| Z3[SET user:blocked\n429] + F -->|no| H[call view] + G -->|yes| Z4[429] + G -->|no| I{INCR\nrl:ip:reqs\n>= 30/min?} + I -->|yes| Z5[SET ip:blocked\n429] + I -->|no| J{INCR\nrl:ns:ip:window\n>= 30/min?} + J -->|yes| Z6[SET ip:blocked\n429] + J -->|no| H + H --> K[Filesystem / ORM / Response] +``` + +### 5.2 RateLimitMiddleware Implementation + +```python +# sapl/middleware/ratelimit.py +import hashlib +import logging +import time + +from django.conf import settings +from django.core.cache import caches +from django.http import HttpResponse + +logger = logging.getLogger('sapl.ratelimit') + +BOT_UA_FRAGMENTS = [ + 'GPTBot', 'ClaudeBot', 'PerplexityBot', + 'Bytespider', 'AhrefsBot', 'meta-externalagent', + 'Chrome/98.0.4758', +] + + +def _sha256(s: str) -> str: + return hashlib.sha256(s.encode()).hexdigest() + + +def _is_suspicious_headers(request) -> bool: + # Real browsers send all three; bots frequently omit them + missing = sum([ + not request.META.get('HTTP_ACCEPT_LANGUAGE'), + not request.META.get('HTTP_ACCEPT'), + not request.META.get('HTTP_REFERER'), + ]) + return missing >= 2 + + +def _get_ip(request) -> str: + return ( + request.META.get('HTTP_X_FORWARDED_FOR', '').split(',')[0].strip() + or request.META.get('REMOTE_ADDR', '') + ) + + +class RateLimitMiddleware: + ANON_IP_THRESHOLD = 30 # req/min + AUTH_USER_THRESHOLD = 120 # req/min + BLOCK_TTL = 300 # seconds + + def __init__(self, get_response): + self.get_response = get_response + self._rl_cache = caches['ratelimit'] + + def __call__(self, request): + decision = self._evaluate(request) + if decision['action'] == 'block': + logger.warning('ratelimit_block', extra={ + 'ip': decision['ip'], + 'reason': decision['reason'], + 'ua': request.META.get('HTTP_USER_AGENT', ''), + 'path': request.path, + 'namespace': getattr(request, 'tenant', 'unknown'), + }) + return HttpResponse(status=429) + return self.get_response(request) + + def _evaluate(self, request): + ip = _get_ip(request) + + # Check 1: known UA (all requests) + ua = request.META.get('HTTP_USER_AGENT', '') + for fragment in BOT_UA_FRAGMENTS: + if fragment.lower() in ua.lower(): + return {'action': 'block', 'reason': 'known_ua', 'ip': ip} + + # Check 2: IP blocked marker + if self._rl_cache.get(f'rl:ip:{ip}:blocked'): + if not getattr(request, 'user', None) or not request.user.is_authenticated: + return {'action': 'block', 'reason': 'ip_blocked', 'ip': ip} + + if getattr(request, 'user', None) and request.user.is_authenticated: + return self._evaluate_authenticated(request, ip) + return self._evaluate_anonymous(request, ip) + + def _evaluate_authenticated(self, request, ip): + user_id = str(request.user.pk).lower().strip() + ns = getattr(request, 'tenant', 'global') + + if self._rl_cache.get(f'rl:{ns}:user:{user_id}:blocked'): + return {'action': 'block', 'reason': 'user_blocked', 'ip': ip} + + if _is_suspicious_headers(request): + return {'action': 'block', 'reason': 'suspicious_headers_auth', 'ip': ip} + + count = self._incr_with_ttl(f'rl:{ns}:user:{user_id}:reqs', ttl=60) + if count >= self.AUTH_USER_THRESHOLD: + self._rl_cache.set(f'rl:{ns}:user:{user_id}:blocked', 1, + timeout=self.BLOCK_TTL) + return {'action': 'block', 'reason': 'auth_user_rate', 'ip': ip} + + return {'action': 'pass', 'ip': ip} + + def _evaluate_anonymous(self, request, ip): + # Check 3: suspicious headers + if _is_suspicious_headers(request): + return {'action': 'block', 'reason': 'suspicious_headers', 'ip': ip} + + # Check 4: IP request rate + count = self._incr_with_ttl(f'rl:ip:{ip}:reqs', ttl=60) + if count >= self.ANON_IP_THRESHOLD: + self._rl_cache.set(f'rl:ip:{ip}:blocked', 1, timeout=self.BLOCK_TTL) + return {'action': 'block', 'reason': 'ip_rate', 'ip': ip} + + # Check 5: per-ns/ip/window (catches UA rotators) + ns = getattr(request, 'tenant', 'global') + bucket = int(time.time() // 60) + count = self._incr_with_ttl(f'rl:ns:{ns}:ip:{ip}:w:{bucket}', ttl=120) + if count >= self.ANON_IP_THRESHOLD: + self._rl_cache.set(f'rl:ip:{ip}:blocked', 1, timeout=self.BLOCK_TTL) + return {'action': 'block', 'reason': 'ua_rotation', 'ip': ip} + + return {'action': 'pass', 'ip': ip} + + def _incr_with_ttl(self, key: str, ttl: int) -> int: + """Atomic INCR + EXPIRE — TTL only set on key creation.""" + lua = """ + local n = redis.call('INCR', KEYS[1]) + if n == 1 then redis.call('EXPIRE', KEYS[1], ARGV[1]) end + return n + """ + client = self._rl_cache._cache.get_client() + return client.eval(lua, 1, key, ttl) +``` + +--- + +### 5.3 Settings Reference + +```python +# sapl/settings.py +MIDDLEWARE = [ + 'sapl.middleware.ratelimit.RateLimitMiddleware', # before session/auth + 'django.contrib.sessions.middleware.SessionMiddleware', + # ... rest unchanged +] + +RATE_LIMITER_RATE = config('RATE_LIMITER_RATE', default='35/m') +RATE_LIMITER_RATE_AUTHENTICATED = config('RATE_LIMITER_RATE_AUTHENTICATED', default='120/m') +RATE_LIMITER_RATE_BOT = config('RATE_LIMITER_RATE_BOT', default='5/m') + +``` + +--- + +### 5.4 Enforcement Graduation Order + +Roll out to canary pods first; promote check-by-check in order of false-positive risk: + +| Order | Check | Risk | Condition to promote | +|---|---|---|---| +| 1st | `known_ua` | Zero | UA strings are deterministic | +| 2nd | `ip_blocked` | Zero | Key only set by prior proven-bad requests | +| 3rd | `ip_rate` | Low | Threshold calibrated from canary logs | +| 4th | `suspicious_headers` | Medium | Confirmed no legitimate clients omit all 3 headers | +| 5th | `ua_rotation` (ns/window) | Medium | | + +--- + +### 5.5 Decorator Migration + +For views where `django-ratelimit` decorators already exist: + +| Endpoint type | Action | Reason | +|---|---|---| +| List views (GET) | Remove after Phase 2 stable | Middleware covers equivalent threshold | +| Detail views (GET) | Remove after Phase 2 stable | Middleware covers equivalent threshold | +| Search / filter views | Remove last | Expensive queries — keep stricter per-view limit | +| PDF / file generation | **Keep permanently** | Most expensive; per-view limit tighter than global | +| Write endpoints (POST/PUT/DELETE) | **Keep permanently** | Different abuse surface | +| Auth endpoints (login, reset) | **Keep permanently** | Credential stuffing; must be independent | + +--- + +## 6. Phase 3 — File Serving Corrections + +**Goal**: Ensure nginx serves files correctly with kernel bypass and caching headers. +**Risk**: Low — config changes only. + +### 6.1 Confirmed Architecture + +nginx already serves `/media/` directly via `alias` — **Django is not involved in file serving for public media**. `X-Accel-Redirect` is only needed for LGPD-restricted documents that must pass through Django for access control. + +The corrected `nginx.conf` and `sapl.conf` are shown in Phase 0 §3.2. No additional changes needed here. + +### 6.2 Why Redis is NOT Needed for PDFs + +With the full mitigation stack active: +- **ASN blocking** (Phase 0) drops datacenter bot traffic at nginx +- **UA blocking** (Phase 0) drops known-UA bots at nginx +- **Shared Redis rate counters** (Phase 2) enforce limits across all pods +- **ETags** (Phase 0 §3.2) convert repeat requests to 304 with zero bytes transferred +- **`sendfile on`** (Phase 0 §3.2) means disk reads bypass userspace entirely + +Redis PDF caching would solve "high request volume reaching the file layer" — but that problem no longer exists once the above stack is active. Redis memory is better reserved for rate counters, page cache, and sessions. + +### 6.3 File Serving Decision Matrix + +| File type | Size | Strategy | +|---|---|---| +| Logos / images | Any | nginx `alias` + `sendfile` + ETag + `Cache-Control` | +| Small PDFs | ≤ 360 KB | nginx direct + ETag | +| Medium PDFs | 360 KB – 2 MB | nginx direct + ETag + rate limit | +| Large PDFs | > 2 MB | nginx + strict rate limit; never Redis | +| LGPD-restricted | Any | Django view → `X-Accel-Redirect` → nginx (access control enforced) | + +--- + +## 7. Phase 4 — Dynamic Page Caching + +**Goal**: Eliminate ORM queries for anonymous bot requests on list views. +**Prerequisite**: Phase 1 (shared Redis, `CACHE_BACKEND=redis`). + +### 7.1 The Key Insight + +Many SAPL list views (`pesquisar-materia`, `norma`, etc.) are not truly dynamic for anonymous users between edits. A bot hammering `?page=1` through `?page=100` triggers 100 ORM queries per pod. With Redis page cache, each unique URL is queried once per TTL across the entire fleet. + +```python +# views.py — apply to anonymous list views only +from django.views.decorators.cache import cache_page +from django.utils.decorators import method_decorator + +@method_decorator(cache_page(60 * 5), name='dispatch') # 5-minute TTL +class PesquisarMateriaView(FilterView): + ... +``` + +> **Critical safety check**: `cache_page` sets `Cache-Control: private` for authenticated sessions automatically. Verify this is working before deploying — accidentally caching a session-aware response is a data leak. + +### 7.2 Cache TTL Guidelines + +| View type | TTL | Reasoning | +|---|---|---| +| Matéria list (anonymous) | 300 s | Changes infrequently between sessions | +| Norma list (anonymous) | 300 s | Same | +| Parlamentar list | 3600 s | Changes rarely | +| Search results | 60 s | Query-dependent, shorter TTL safer | +| Authenticated views | Never | `cache_page` respects this automatically | +| PDF generation | Never | Too large — serve from disk via nginx | + +--- + +## 8. Open Questions + +| # | Question | Status | Blocks | +|---|---|---|---| +| 1 | Does Chrome/98.0.4758 impersonator appear consistently in nginx access logs? | Needs investigation | Phase 0 UA block safety | +| 3 | Dockerfile scope | Single image for all tenants (confirmed). All path-based Redis keys include `{ns}`. | — | +| 4 | WebSocket voting panel priority | Separate project. Resumes after Redis is on k8s, bot siege addressed, and OOM pressure reduced. | Phase 5 sequencing | +| 5 | `CONN_MAX_AGE` tuning | Currently **300 s** (`sapl/settings.py:272`). Evaluate whether to reduce given worker recycling at 400 MB. | Phase 0 tuning | +| 6 | k8s Redis manifests | Development artifacts go under `$PROJECT_ROOT/docker/k8s/` (redis-pod.yaml, redis-service.yaml, redis-configmap.yaml). | Phase 1 delivery | + +--- + +*Document consolidated from multi-session architecture review — Edward / Interlegis SAPL infrastructure.* diff --git a/requirements/requirements.txt b/requirements/requirements.txt index ca05c71b7..9db02dc49 100644 --- a/requirements/requirements.txt +++ b/requirements/requirements.txt @@ -42,3 +42,8 @@ XlsxWriter==3.2.0 setuptools==80.9.0 git+https://github.com/interlegis/django-admin-bootstrapped + +# Redis cache backend (Phase 1 — shared rate-limiter state). +# 4.12.1 is the last release that explicitly supports Django 2.2. +# Upgrade to 5.x when the project moves to Django 3.2+. +django-redis==4.12.1 diff --git a/sapl/audiencia/views.py b/sapl/audiencia/views.py index d5ea7f710..c9df5cf8c 100755 --- a/sapl/audiencia/views.py +++ b/sapl/audiencia/views.py @@ -9,11 +9,7 @@ from sapl.crud.base import RP_DETAIL, RP_LIST, Crud, MasterDetailCrud from .forms import AudienciaForm, AnexoAudienciaPublicaForm from .models import AudienciaPublica, AnexoAudienciaPublica -from ratelimit.decorators import ratelimit -from django.utils.decorators import method_decorator - -from ..settings import RATE_LIMITER_RATE -from ..utils import ratelimit_ip +from sapl.middleware.page_cache import AnonCachePageMixin def index(request): @@ -28,8 +24,9 @@ class AudienciaCrud(Crud): list_field_names = ['numero', 'nome', 'tipo', 'materia', 'data'] ordering = '-ano', '-numero', '-data', 'nome', 'tipo' - class ListView(Crud.ListView): + class ListView(AnonCachePageMixin, Crud.ListView): paginate_by = 10 + anon_cache_ttl = 120 # PAGE_CACHE_TTL_LIST — hearings are added infrequently def get_context_data(self, **kwargs): context = super().get_context_data(**kwargs) diff --git a/sapl/base/media.py b/sapl/base/media.py new file mode 100644 index 000000000..058dc87ea --- /dev/null +++ b/sapl/base/media.py @@ -0,0 +1,93 @@ +""" +serve_media — X-Accel-Redirect gate for all /media/ files. + +Production flow (nginx proxies /media/ to Gunicorn): + 1. Django middleware runs (IP rate-limit, bot UA check, etc.). + 2. serve_media() runs auth check for sapl/private/, writes + URL-path counter to Redis DB 1, then returns X-Accel-Redirect. + Nginx serves the bytes directly from disk — Gunicorn worker freed immediately. + +Development flow (DEBUG=True, nginx absent): + Falls back to django.views.static.serve for live file serving. + +Redis side-effects per request (DB 1, TTL=MEDIA_PATH_COUNTER_TTL): + rl:{ns}:path:{sha256('/media/')}:reqs — URL-path access counter +""" + +import hashlib +import mimetypes +import os +from urllib.parse import quote + +from django.conf import settings +from django.http import Http404, HttpResponse +from django.views.static import serve + +from sapl import settings as sapl_settings +from sapl.middleware.ratelimit import ( + _NAMESPACE, + RL_PATH_REQUESTS, + _incr_with_ttl, +) + + +def _safe_resolve(rel_path): + """ + Return the absolute path of rel_path inside MEDIA_ROOT. + Raises Http404 if the resolved path would escape the root + (path traversal guard). + """ + abs_root = os.path.abspath(settings.MEDIA_ROOT) + abs_path = os.path.abspath(os.path.join(abs_root, rel_path)) + if not abs_path.startswith(abs_root + os.sep) and abs_path != abs_root: + raise Http404 + return abs_path + + +def _is_public_docadm(path): + # Documentos Administrativos são sempre salvos na pasta /private, + # mas podem ter acesso público liberado como OSTENSIVO ou requerer + # autenticacao se for DOC_ADM_RESTRITIVO + from sapl.base.models import AppConfig, DOC_ADM_OSTENSIVO + return 'documentoadministrativo' in path and \ + AppConfig.attr('documentos_administrativos') == DOC_ADM_OSTENSIVO + + +def serve_media(request, path): + """ + Registered in sapl/urls.py for both DEBUG and production. + Route: ^media/(?P.*)$ + """ + # Path traversal guard — raises Http404 on escape attempt. + abs_path = _safe_resolve(path) + + # Auth gate for private documents — redirect to login if anonymous. + if path.startswith('sapl/private/') and not _is_public_docadm(path): + user = getattr(request, 'user', None) + if user is None or not user.is_authenticated: + from django.contrib.auth.views import redirect_to_login + return redirect_to_login(request.get_full_path()) + + # 404 before writing any counters. + if not os.path.isfile(abs_path): + raise Http404 + + # URL-path counter (DB 1). + _incr_with_ttl( + RL_PATH_REQUESTS.format(ns=_NAMESPACE, sha256=hashlib.sha256(f'/media/{path}'.encode()).hexdigest()), + ttl=sapl_settings.MEDIA_PATH_COUNTER_TTL, + ) + + if settings.DEBUG: + # Development: no nginx present; serve the file directly. + return serve(request, path, document_root=settings.MEDIA_ROOT) + + # Production: tell nginx to serve the file from the internal location. + filename = os.path.basename(path) + content_type, _ = mimetypes.guess_type(filename) + response = HttpResponse(content_type=content_type or 'application/octet-stream') + response['X-Accel-Redirect'] = f'/internal/media/{path}' + response['Content-Disposition'] = f"inline; filename*=UTF-8''{quote(filename)}" + response['Cache-Control'] = 'public, max-age=86400, stale-while-revalidate=3600' + response['X-Robots-Tag'] = 'noindex' + return response diff --git a/sapl/base/views.py b/sapl/base/views.py index 2b53aa57c..c85106322 100644 --- a/sapl/base/views.py +++ b/sapl/base/views.py @@ -48,10 +48,11 @@ from sapl.parlamentares.models import ( from sapl.protocoloadm.models import (Anexado, Protocolo) from sapl.relatorios.views import (relatorio_estatisticas_acesso_normas) from sapl.sessao.models import (Bancada, SessaoPlenaria) -from sapl.settings import EMAIL_SEND_USER, RATE_LIMITER_RATE +from sapl.settings import EMAIL_SEND_USER from sapl.utils import (gerar_hash_arquivo, intervalos_tem_intersecao, mail_service_configured, SEPARADOR_HASH_PROPOSICAO, show_results_filter_set, google_recaptcha_configured, - get_client_ip, sapn_is_enabled, is_weak_password, ratelimit_ip) + sapn_is_enabled, is_weak_password) +from sapl.middleware.ratelimit import smart_key, smart_rate from .forms import (AlterarSenhaForm, CasaLegislativaForm, ConfiguracoesAppForm, EstatisticasAcessoNormasForm) from .models import AppConfig, CasaLegislativa @@ -67,8 +68,8 @@ class IndexView(TemplateView): return TemplateView.get(self, request, *args, **kwargs) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, method=ratelimit.UNSAFE, block=True), name='dispatch') @@ -1400,8 +1401,8 @@ class SaplSearchView(SearchView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class PesquisarAuditLogView(PermissionRequiredMixin, FilterView): @@ -1457,12 +1458,9 @@ class PesquisarAuditLogView(PermissionRequiredMixin, FilterView): data = self.filterset.data - url = '' - - if data: - url = '&' + str(self.request.META["QUERY_STRING"]) - if url.startswith("&page"): - url = '' + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' resultados = self.object_list # if 'page' in self.request.META['QUERY_STRING']: diff --git a/sapl/comissoes/views.py b/sapl/comissoes/views.py index 6017ae2b3..1d7194d3d 100644 --- a/sapl/comissoes/views.py +++ b/sapl/comissoes/views.py @@ -28,7 +28,9 @@ from sapl.crud.base import (Crud, CrudAux, MasterDetailCrud, RP_LIST) from sapl.materia.models import (MateriaEmTramitacao, MateriaLegislativa, PautaReuniao, Tramitacao) -from sapl.utils import show_results_filter_set, ratelimit_ip +from sapl.middleware.page_cache import AnonCachePageMixin +from sapl.middleware.ratelimit import smart_key, smart_rate +from sapl.utils import show_results_filter_set from .models import (CargoComissao, Comissao, Composicao, DocumentoAcessorio, Participacao, Periodo, Reuniao, TipoComissao) @@ -36,7 +38,6 @@ from .models import (CargoComissao, Comissao, Composicao, DocumentoAcessorio, from ratelimit.decorators import ratelimit from django.utils.decorators import method_decorator -from ..settings import RATE_LIMITER_RATE def pegar_url_composicao(pk): @@ -173,6 +174,10 @@ class ComissaoCrud(Crud): 'data_criacao', 'data_extincao', 'ativa'] ordering = '-ativa', 'sigla' + class ListView(AnonCachePageMixin, Crud.ListView): + # Committee lists change rarely; 5-minute cache is conservative. + anon_cache_ttl = 300 # PAGE_CACHE_TTL_DETAIL + class CreateView(Crud.CreateView): form_class = ComissaoForm @@ -338,8 +343,8 @@ class RemovePautaView(PermissionRequiredMixin, CreateView): return HttpResponseRedirect(success_url) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class AdicionaPautaView(PermissionRequiredMixin, FilterView): diff --git a/sapl/crud/base.py b/sapl/crud/base.py index fd091ec52..a81549d2d 100644 --- a/sapl/crud/base.py +++ b/sapl/crud/base.py @@ -26,8 +26,8 @@ from sapl.crispy_layout_mixin import CrispyLayoutFormMixin, get_field_display from sapl.crispy_layout_mixin import SaplFormHelper from sapl.rules import (RP_ADD, RP_CHANGE, RP_DELETE, RP_DETAIL, RP_LIST) -from sapl.settings import RATE_LIMITER_RATE -from sapl.utils import normalize, ratelimit_ip +from sapl.middleware.ratelimit import smart_key, smart_rate +from sapl.utils import normalize from ratelimit.decorators import ratelimit from django.utils.decorators import method_decorator @@ -390,8 +390,8 @@ class CrudBaseMixin(CrispyLayoutFormMixin): return self.model._meta.verbose_name_plural -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class CrudListView(PermissionRequiredContainerCrudMixin, ListView): @@ -730,8 +730,8 @@ class CrudCreateView(PermissionRequiredContainerCrudMixin, return super().form_valid(form) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class CrudDetailView(PermissionRequiredContainerCrudMixin, @@ -1188,8 +1188,8 @@ class MasterDetailCrud(Crud): context['title'] = title return context - @method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, + @method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class ListView(Crud.ListView): @@ -1424,8 +1424,8 @@ class MasterDetailCrud(Crud): else: return self.resolve_url(ACTION_LIST, args=(pk,)) - @method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, + @method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class DetailView(Crud.DetailView): diff --git a/sapl/materia/forms.py b/sapl/materia/forms.py index 3615fee47..979e96c3b 100644 --- a/sapl/materia/forms.py +++ b/sapl/materia/forms.py @@ -42,7 +42,8 @@ from sapl.utils import (autor_label, autor_modal, timing, models_with_gr_for_model, qs_override_django_filter, SEPARADOR_HASH_PROPOSICAO, validar_arquivo, YES_NO_CHOICES, - GoogleRecapthaMixin, get_client_ip) + GoogleRecapthaMixin) +from sapl.middleware.ratelimit import get_client_ip from .models import (AcompanhamentoMateria, Anexada, Autoria, DespachoInicial, DocumentoAcessorio, Numeracao, diff --git a/sapl/materia/migrations/0088_fix_view_materiaemtramitacao.py b/sapl/materia/migrations/0088_fix_view_materiaemtramitacao.py new file mode 100644 index 000000000..1f3ca52d8 --- /dev/null +++ b/sapl/materia/migrations/0088_fix_view_materiaemtramitacao.py @@ -0,0 +1,62 @@ +from django.db import migrations, models + +_OLD_VIEW = """ + create or replace view materia_materiaemtramitacao as + select m.id as id, + m.id as materia_id, + t.id as tramitacao_id, + t.unidade_tramitacao_destino_id as unidade_tramitacao_atual_id + from materia_materialegislativa m + inner join materia_tramitacao t on (m.id = t.materia_id) + where t.id = (select max(id) from materia_tramitacao where materia_id = m.id) + order by m.id DESC +""" + +_NEW_VIEW = """ + create or replace view materia_materiaemtramitacao as + select distinct on (m.id) + m.id as id, + m.id as materia_id, + t.id as tramitacao_id, + t.unidade_tramitacao_destino_id as unidade_tramitacao_atual_id + from materia_materialegislativa m + inner join materia_tramitacao t on t.materia_id = m.id + order by m.id desc, t.id desc +""" + + +class Migration(migrations.Migration): + # CREATE INDEX CONCURRENTLY cannot run inside a transaction. + atomic = False + + dependencies = [ + ('materia', '0087_update_viewdb_materiaemtramitacao'), + ] + + operations = [ + migrations.RunSQL(sql=_NEW_VIEW, reverse_sql=_OLD_VIEW), + migrations.SeparateDatabaseAndState( + database_operations=[ + migrations.RunSQL( + sql=""" + CREATE INDEX CONCURRENTLY IF NOT EXISTS + tram_materia_id_desc + ON materia_tramitacao (materia_id, id DESC) + """, + reverse_sql=""" + DROP INDEX CONCURRENTLY IF EXISTS + tram_materia_id_desc + """, + ), + ], + state_operations=[ + migrations.AddIndex( + model_name='tramitacao', + index=models.Index( + fields=['materia', '-id'], + name='tram_materia_id_desc', + ), + ), + ], + ), + ] diff --git a/sapl/materia/models.py b/sapl/materia/models.py index bdeb402e3..fb0df6ee5 100644 --- a/sapl/materia/models.py +++ b/sapl/materia/models.py @@ -1350,6 +1350,9 @@ class Tramitacao(models.Model): verbose_name = _('Tramitação') verbose_name_plural = _('Tramitações') ordering = ('-data_tramitacao', '-id') + indexes = [ + models.Index(fields=['materia', '-id'], name='tram_materia_id_desc'), + ] def __str__(self): return _('%(materia)s | %(status)s | %(data)s') % { diff --git a/sapl/materia/views.py b/sapl/materia/views.py index 698ad6d33..cbdb11718 100644 --- a/sapl/materia/views.py +++ b/sapl/materia/views.py @@ -34,6 +34,7 @@ import weasyprint from ratelimit.decorators import ratelimit from django.utils.decorators import method_decorator +from django.views.decorators.http import condition import sapl from sapl.base.email_utils import do_envia_email_confirmacao @@ -52,12 +53,14 @@ from sapl.materia.forms import (AnexadaForm, AutoriaForm, AutoriaMultiCreateForm from sapl.norma.models import LegislacaoCitada from sapl.parlamentares.models import Legislatura from sapl.protocoloadm.models import Protocolo -from sapl.settings import MAX_DOC_UPLOAD_SIZE, MEDIA_ROOT, RATE_LIMITER_RATE +from sapl.settings import MAX_DOC_UPLOAD_SIZE, MEDIA_ROOT from sapl.utils import (autor_label, autor_modal, gerar_hash_arquivo, get_base_url, - get_client_ip, get_mime_type_from_file_extension, lista_anexados, + lista_anexados, mail_service_configured, montar_row_autor, SEPARADOR_HASH_PROPOSICAO, show_results_filter_set, get_tempfile_dir, - google_recaptcha_configured, MultiFormatOutputMixin, ratelimit_ip) + google_recaptcha_configured, MultiFormatOutputMixin) +from sapl.middleware.ratelimit import get_client_ip, smart_key, smart_rate +from sapl.middleware.page_cache import AnonCachePageMixin from .forms import (AcessorioEmLoteFilterSet, AcompanhamentoMateriaForm, AnexadaEmLoteFilterSet, AdicionarVariasAutoriasFilterSet, @@ -119,24 +122,14 @@ def proposicao_texto(request, pk): return redirect(reverse('sapl.materia:proposicao_detail', kwargs={'pk': pk})) - arquivo = proposicao.texto_original - - mime = get_mime_type_from_file_extension(arquivo.name) - - with open(arquivo.path, 'rb') as f: - data = f.read() - - response = HttpResponse(data, content_type='%s' % mime) - response['Content-Disposition'] = ( - 'inline; filename="%s"' % arquivo.name.split('/')[-1]) - return response + return redirect(proposicao.texto_original.url) logger.error('user=' + username + '. Objeto Proposicao com pk={} não encontrado.'.format(pk)) raise Http404 -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class AdicionarVariasAutorias(PermissionRequiredForAppCrudMixin, FilterView): @@ -361,11 +354,11 @@ class StatusTramitacaoCrud(CrudAux): return reverse('sapl.materia:pesquisar_statustramitacao') -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class PesquisarStatusTramitacaoView(FilterView): +class PesquisarStatusTramitacaoView(AnonCachePageMixin, FilterView): model = StatusTramitacao filterset_class = StatusTramitacaoFilterSet paginate_by = 20 @@ -403,12 +396,9 @@ class PesquisarStatusTramitacaoView(FilterView): data = self.filterset.data - url = '' - - if data: - url = '&' + str(self.request.META["QUERY_STRING"]) - if url.startswith("&page"): - url = '' + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' if 'descricao' in self.request.META['QUERY_STRING'] or\ 'page' in self.request.META['QUERY_STRING']: @@ -1796,6 +1786,17 @@ class MateriaAssuntoCrud(MasterDetailCrud): return initial +def _materia_last_modified(request, *args, **kwargs): + return MateriaLegislativa.objects.filter( + pk=kwargs['pk'] + ).values_list('data_ultima_atualizacao', flat=True).first() + + +def _materia_etag(request, *args, **kwargs): + ts = _materia_last_modified(request, *args, **kwargs) + return f'{kwargs["pk"]}-{ts.timestamp()}' if ts else None + + class MateriaLegislativaCrud(Crud): model = MateriaLegislativa help_topic = 'materia_legislativa' @@ -1881,7 +1882,12 @@ class MateriaLegislativaCrud(Crud): def get_success_url(self): return self.search_url - class DetailView(Crud.DetailView): + @method_decorator(condition(etag_func=_materia_etag, last_modified_func=_materia_last_modified), name='get') + class DetailView(AnonCachePageMixin, Crud.DetailView): + # Materia detail pages are public, read-only, and change infrequently + # once published. Cache anonymous responses for 5 minutes to absorb + # bot and search-engine traffic without hitting PostgreSQL. + anon_cache_ttl = 300 # PAGE_CACHE_TTL_DETAIL layout_key = 'MateriaLegislativaDetail' template_name = "materia/materialegislativa_detail.html" @@ -2013,11 +2019,11 @@ class AcompanhamentoExcluirView(TemplateView): return HttpResponseRedirect(self.get_success_url()) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class MateriaLegislativaPesquisaView(MultiFormatOutputMixin, FilterView): +class MateriaLegislativaPesquisaView(AnonCachePageMixin, MultiFormatOutputMixin, FilterView): model = MateriaLegislativa filterset_class = MateriaLegislativaFilterSet paginate_by = 50 @@ -2278,8 +2284,8 @@ class AcompanhamentoMateriaView(CreateView): kwargs={'pk': self.kwargs['pk']}) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class DocumentoAcessorioEmLoteView(PermissionRequiredMixin, FilterView): @@ -2394,8 +2400,8 @@ class DocumentoAcessorioEmLoteView(PermissionRequiredMixin, FilterView): return self.get(request, self.kwargs) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class MateriaAnexadaEmLoteView(PermissionRequiredMixin, FilterView): @@ -2522,8 +2528,8 @@ class MateriaAnexadaEmLoteView(PermissionRequiredMixin, FilterView): return HttpResponseRedirect(success_url) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class PrimeiraTramitacaoEmLoteView(PermissionRequiredMixin, FilterView): diff --git a/sapl/middleware/page_cache.py b/sapl/middleware/page_cache.py new file mode 100644 index 000000000..553f9632a --- /dev/null +++ b/sapl/middleware/page_cache.py @@ -0,0 +1,75 @@ +""" +AnonCachePageMixin — anonymous-only Django view response caching. + +Why anonymous-only? + - Authenticated responses include CSRF tokens and user-specific UI fragments + (edit/delete action buttons injected by SAPL's CRUD framework). Caching + those would serve stale or wrong data to other users. + - Bot traffic is entirely anonymous. A 2-minute cache converts hundreds of + identical list-view DB queries into a single one — exactly the workload + that triggers OOM in the fleet. + +How it works: + - `dispatch()` short-circuits to the normal (uncached) path for any + authenticated request. + - For anonymous GET/HEAD requests, the response is stored in the 'default' + Redis cache under a key that includes the full URL (scheme + host + path + + query string). The Django cache framework handles key construction and + TTL expiry automatically. + - HTTPS and HTTP requests are stored under separate keys (Django default). + +Usage: + + from sapl.middleware.page_cache import AnonCachePageMixin + + class MyListView(AnonCachePageMixin, ListView): + anon_cache_ttl = settings.PAGE_CACHE_TTL_LIST # 120 s + + class MyDetailView(AnonCachePageMixin, DetailView): + anon_cache_ttl = settings.PAGE_CACHE_TTL_DETAIL # 300 s + +TTL reference (see settings.PAGE_CACHE_TTL_*): + + View type Default TTL + ───────────────────────────────────────────────────── + Public list (norma, materia, sessao…) 120 s (PAGE_CACHE_TTL_LIST) + Public detail (norma, materia, sessao…) 300 s (PAGE_CACHE_TTL_DETAIL) + Stable detail (parlamentar, comissão) 600 s (PAGE_CACHE_TTL_STABLE) + +Invalidation: + The cache TTL is intentionally short (≤ 10 min) so stale content expires + on its own. Explicit invalidation is not implemented — legislative data + changes infrequently and short TTLs are acceptable. +""" + +from django.conf import settings +from django.views.decorators.cache import cache_page, never_cache +from django.utils.decorators import method_decorator + + +class AnonCachePageMixin: + """ + Cache the full view response for anonymous (unauthenticated) requests. + + Set `anon_cache_ttl` on the subclass to override the default TTL. + Authenticated requests always bypass the cache. + """ + + # Override per view class. Use settings.PAGE_CACHE_TTL_* for consistency. + anon_cache_ttl = getattr(settings, 'PAGE_CACHE_TTL_LIST', 120) + + def dispatch(self, request, *args, **kwargs): + if getattr(request, 'user', None) and request.user.is_authenticated: + # Authenticated: skip cache entirely — response may contain + # user-specific controls (CSRF token, edit/delete buttons). + handler = never_cache( + lambda req, *a, **kw: super(AnonCachePageMixin, self).dispatch(req, *a, **kw) + ) + return handler(request, *args, **kwargs) + + # Anonymous: wrap the parent dispatch in cache_page so Django stores + # the rendered response in the 'default' cache for anon_cache_ttl seconds. + handler = cache_page(self.anon_cache_ttl)( + lambda req, *a, **kw: super(AnonCachePageMixin, self).dispatch(req, *a, **kw) + ) + return handler(request, *args, **kwargs) diff --git a/sapl/middleware/ratelimit.py b/sapl/middleware/ratelimit.py new file mode 100644 index 000000000..5be632cef --- /dev/null +++ b/sapl/middleware/ratelimit.py @@ -0,0 +1,774 @@ +""" +RateLimitMiddleware — cross-pod rate limiting backed by shared Redis. + +Decision flow (per request): + Both /api/ and non-/api/ paths — checked first, before all else: + -1. IP in rl:ip_prefix:blocked (prefix match)? → 403 (Forbidden — not a rate + limit; an operator-curated permanent deny list, no Retry-After) + (universal — applies to authenticated users too, like the UA bot checks) + /api/ paths — handled by _handle_api: + 0a. OPTIONS? → pass (CORS preflight must never be blocked) + 0b. rl:ip_prefix:blocked? → 403 (prefix match; see -1 above) + 0c. rl:ip::blocked? → 429 (global block also covers /api/) + 0d. rl:api:ip::blocked? → 429 (API-only block) + Block checks 0b-0d always run before the same-origin check below — + Origin/Referer are client-controlled and trivially spoofable, so they + must never be able to override an already-made block decision. + 0e. Same-origin? → pass, skipping 0f-0g only (SAPL's own polling + is exempt from quota/rate-limit accounting, never from active blocks) + 0f. Daily/weekly quota exceeded? → 429 + 0g. Anon + API threshold exceeded? → SET rl:api:ip::blocked, 429 + (never writes rl:ip::blocked) + 0h. Auth: falls through to _evaluate (per-user counter) + Non-/api/ paths: + 1. Known bot UA? → 429 (Python list — substring match) + 1b. Redis UA deny list? → 429 (runtime SET — token hash match, refreshed every 60 s) + 2. Anonymous AND IP in blocked set? → 429 (authenticated users skip — have per-user limit at 3c) + 3. Authenticated user? + a. User blocked? → 429 + b. Suspicious hdrs? → 429 + c. User rate ≥ 240? → 429 (no persistent block; window resets after 60 s) + 4. Anonymous: + a. Suspicious hdrs? → 429 + b. IP rate ≥ 120/min? → SET RL_IP_BLOCKED, 429 + c. NS/IP window hit? → SET RL_IP_BLOCKED, 429 + +Degrades gracefully to non-atomic counting when Redis is unavailable. + +_NAMESPACE is settings.POD_NAMESPACE, resolved once at startup: + - K8s: start.sh reads the k8s namespace from the Downward API env var + or the service-account namespace file, writes it to .env as POD_NAMESPACE. + - Bare-metal / VM / docker-compose: defaults to the machine hostname + (socket.gethostbyname_ex result computed in settings.py). +Since a deployment serves exactly one tenant, this is a startup constant — +no per-request lookup is needed or correct. +""" + +import hashlib +import logging +import re +import time +from datetime import date + +from sapl import settings +from django.core.cache import caches +from django.http import HttpResponse + +logger = logging.getLogger('sapl.ratelimit') + +# --------------------------------------------------------------------------- +# Tenant namespace — resolved once at startup from settings.POD_NAMESPACE. +# On K8s: the k8s namespace (e.g. "sapl31demo-df"), set by start.sh. +# On bare-metal / VM / docker-compose: the machine hostname (default). +# --------------------------------------------------------------------------- + +_NAMESPACE = settings.POD_NAMESPACE + +# --------------------------------------------------------------------------- +# Redis key templates — module-level constants, never inline strings +# --------------------------------------------------------------------------- + +RL_IP_REQUESTS = 'rl:ip:{ip}:reqs' +RL_IP_BLOCKED = 'rl:ip:{ip}:blocked' +RL_IP_404S = 'rl:ip:{ip}:404s' +RL_USER_REQUESTS = 'rl:{ns}:user:{uid}:reqs' +RL_USER_BLOCKED = 'rl:{ns}:user:{uid}:blocked' +RL_NS_WINDOW = 'rl:{ns}:ip:{ip}:w:{bucket}' +RL_PATH_REQUESTS = 'rl:{ns}:path:{sha256}:reqs' +RL_UA_BLOCKLIST = 'rl:bot:ua:blocked' # permanent SET — runtime UA deny list +RL_IP_PREFIX_BLOCKLIST = 'rl:ip_prefix:blocked' # permanent SET — runtime IP-prefix deny list +RL_METRICS_BLOCKED = 'rl:metrics:{ns}:{date}' # HASH: field = reason, value = daily count + +# ZSET indexes — members are full block-key strings, score = expiry unix timestamp. +# Lets admin/monitoring tools enumerate active blocks with a single ZRANGEBYSCORE +# without scanning all keys. Prunable via: ZREMRANGEBYSCORE 0 . +RL_INDEX_BLOCKED_IPS = 'rl:index:blocked_ips' +RL_INDEX_BLOCKED_USERS = 'rl:index:blocked_users' + +# API-specific rate limit keys — scope limited to /api/, never written by non-/api/ paths. +RL_API_IP_REQUESTS = 'rl:api:ns:{ns}:ip:{ip}:reqs' +RL_API_IP_BLOCKED = 'rl:api:ns:{ns}:ip:{ip}:blocked' +RL_INDEX_API_BLOCKED_IPS = 'rl:index:api_blocked_ips' + +# --------------------------------------------------------------------------- +# API quota keys — per-tenant HASH, one key per period. +# Structure: HASH key = api_quota:{ns}:daily:{date} field = {ip} value = counter +# HASH key = api_quota:{ns}:weekly:{week} field = {ip} value = counter +# Weekly key uses ISO week notation (yyyy-Www) — unambiguous, Monday-anchored. +# TTL set once on hash creation (Lua TTL guard); resets are implicit in the +# date/week embedded in the key name. Fields are IPs; no per-field TTL. +# Memory: ~63 bytes/field vs ~148 bytes for per-IP STRING keys (~57% saving). +# --------------------------------------------------------------------------- +API_QUOTA_DAILY_HASH = 'api_quota:{ns}:daily:{date}' +API_QUOTA_WEEKLY_HASH = 'api_quota:{ns}:weekly:{week}' + +# --------------------------------------------------------------------------- +# Bot UA fragments +# --------------------------------------------------------------------------- + +BOT_UA_FRAGMENTS = [ + 'GPTBot', + 'ClaudeBot', + 'PerplexityBot', + 'Bytespider', + 'AhrefsBot', + 'meta-externalagent', + 'OAI-SearchBot', + 'bingbot', + 'SERankingBacklinksBot', + 'Chrome/98.0.4758', # known scraper impersonating an old Chrome + 'quiltbot', + 'AwarioBot', +] + +_INCR_LUA = """ + local n = redis.call('INCR', KEYS[1]) + if n == 1 then redis.call('EXPIRE', KEYS[1], ARGV[1]) end + return n +""" + +# Atomically increment a HASH field and set TTL on the hash key the first time +# it is created (TTL < 0 covers both -1 = no expire and -2 = key just created). +# KEYS[1] = hash key ARGV[1] = field (IP) ARGV[2] = TTL seconds +_HINCRBY_LUA = """ + local n = redis.call('HINCRBY', KEYS[1], ARGV[1], 1) + if redis.call('TTL', KEYS[1]) < 0 then + redis.call('EXPIRE', KEYS[1], ARGV[2]) + end + return n +""" + +# Atomically write a block key and record it in the sharded ZSET index. +# Prunes expired entries from the target shard before inserting the new one, +# keeping each shard bounded to only active blocks (amortised O(1) cleanup). +# KEYS[1] = block key KEYS[2] = shard index key +# ARGV[1] = ttl (seconds) ARGV[2] = expiry unix timestamp (now + ttl) +# ARGV[3] = current unix timestamp (for pruning: remove score < now) +_BLOCK_LUA = """ + local now = tonumber(ARGV[3]) + redis.call('SET', KEYS[1], '1', 'EX', ARGV[1]) + redis.call('ZREMRANGEBYSCORE', KEYS[2], '-inf', now - 1) + redis.call('ZADD', KEYS[2], ARGV[2], KEYS[1]) + return 1 +""" + + +def _index_shard(ip, index_base): + """ + Return the sharded ZSET key for an IP. + + IPs are distributed across RATE_LIMITER_INDEX_SHARDS shards using + md5(ip) % N, spreading write contention and bounding each shard's size. + The mapping is deterministic: the same IP always routes to the same shard. + """ + n = settings.RATE_LIMITER_INDEX_SHARDS + shard = int(hashlib.md5(ip.encode()).hexdigest(), 16) % n + return f'{index_base}:{shard}' + + +def make_ratelimit_cache_key(key, key_prefix, version): + """ + Pass-through cache key function for the 'ratelimit' Django cache backend. + + Django's default key function produces '{KEY_PREFIX}:{VERSION}:{key}', + which turns django-ratelimit's own keys (already prefixed 'rl:{hash}') + into ':1:rl:{hash}' — an ugly leading colon and version number that does + not match the clean 'rl:*' keys written directly by RateLimitMiddleware. + + Setting KEY_FUNCTION to this function makes both key namespaces consistent: + django-ratelimit decorator keys → rl:{hash} + RateLimitMiddleware keys → rl:ip:{ip}:reqs / rl:{ns}:user:{uid}:reqs / … + """ + return key + + +def _sha256(s): + return hashlib.sha256(s.encode()).hexdigest() + + +def get_client_ip(request): + """ + Return the real client IP, applying django-ratelimit's ip_mask so that + IPv6 /64 subnets are collapsed to a single key (prevents per-address + rotation attacks). Also checks HTTP_X_REAL_IP for nginx setups that + use that header instead of X-Forwarded-For. + + Canonical source — imported from here by other SAPL modules. + """ + from ratelimit.core import ip_mask + x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') + if x_forwarded_for: + ip = x_forwarded_for.split(',')[0].strip() + else: + ip = ( + request.META.get('HTTP_X_REAL_IP') + or request.META.get('REMOTE_ADDR') + or '0.0.0.0' + ) + return ip_mask(ip) + + +def ratelimit_ip(group, request): + """Key function for django-ratelimit decorators (group param is ignored).""" + return get_client_ip(request) + + +def smart_key(group, request): + """ + Auth-aware key for @ratelimit decorators. + + Authenticated users are keyed by user pk so that office workers sharing + a NAT IP don't count against each other. Anonymous requests fall back to + the masked IP (IPv6 /64 collapsed via ip_mask). + """ + user = getattr(request, 'user', None) + if user is not None and user.is_authenticated: + return str(user.pk) + return ratelimit_ip(group, request) + + +def smart_rate(group, request): + """ + Auth-aware rate string for @ratelimit decorators. + + Returns RATE_LIMITER_RATE_AUTHENTICATED for authenticated users, + RATE_LIMITER_RATE for anonymous users — mirrors the thresholds applied + by RateLimitMiddleware so view-level and middleware-level limits agree. + """ + user = getattr(request, 'user', None) + if user is not None and user.is_authenticated: + return settings.RATE_LIMITER_RATE_AUTHENTICATED + return settings.RATE_LIMITER_RATE + + +def _is_same_origin(request): + """ + Return True if Origin or Referer header matches the current SAPL host. + Strips port and lowercases both sides before comparing — DNS is case-insensitive + and reverse proxies may expose a different port than the browser sees. + Checks Origin first; falls back to Referer only when Origin is absent. + Returns False when both headers are absent. + """ + from urllib.parse import urlparse + + def _normalize(host): + return host.lower().split(':', 1)[0].strip() + + try: + host = _normalize(request.get_host()) + except Exception: + return False + + origin = request.META.get('HTTP_ORIGIN', '') + if origin: + try: + return _normalize(urlparse(origin).netloc) == host + except ValueError: + return False + + referer = request.META.get('HTTP_REFERER', '') + if referer: + try: + return _normalize(urlparse(referer).netloc) == host + except ValueError: + return False + + return False + + +def _is_suspicious_headers(request): + """Real browsers send Accept-Language + Accept; bots frequently omit them.""" + missing = sum([ + not request.META.get('HTTP_ACCEPT_LANGUAGE'), + not request.META.get('HTTP_ACCEPT'), + ]) + # Também considera User-Agent antes de bloquear + has_ua = bool(request.META.get('HTTP_USER_AGENT')) + return missing >= 2 and not has_ua + + +def _parse_rate(rate_str): + """Parse '35/m' or '120/m' into (count, seconds).""" + count, period = rate_str.split('/') + count = int(count) + seconds = {'s': 1, 'm': 60, 'h': 3600}.get(period.lower(), 60) + return count, seconds + + +def _incr_with_ttl(key, ttl): + """ + Atomic INCR + EXPIRE via Redis Lua script (ratelimit cache, DB 1). + Falls back to non-atomic cache get/set when Redis is unavailable. + Exported at module level so sapl.base.media can reuse it for path counters. + """ + try: + from django_redis import get_redis_connection + client = get_redis_connection('ratelimit') + return client.eval(_INCR_LUA, 1, key, ttl) + except Exception: + rl_cache = caches['ratelimit'] + count = (rl_cache.get(key) or 0) + 1 + rl_cache.set(key, count, timeout=ttl) + return count + + +def _hincrby_with_ttl(hash_key, field, ttl): + """ + Atomic HINCRBY + conditional EXPIRE via Redis Lua script (ratelimit cache, DB 1). + + Increments the counter for `field` inside `hash_key` and sets a TTL on the + hash key if it does not already have one (i.e. on first creation). + Falls back to a composite STRING key `hash_key:field` when Redis is unavailable. + """ + try: + from django_redis import get_redis_connection + client = get_redis_connection('ratelimit') + return client.eval(_HINCRBY_LUA, 1, hash_key, field, ttl) + except Exception: + rl_cache = caches['ratelimit'] + fallback_key = f'{hash_key}:{field}' + count = (rl_cache.get(fallback_key) or 0) + 1 + rl_cache.set(fallback_key, count, timeout=ttl) + return count + + +def _set_block(block_key, index_key, ttl): + """ + Atomically set a block key (with TTL) and record it in a sharded ZSET index. + Score = expiry unix timestamp. Prunes expired entries from the target shard + before inserting (inline cleanup — no separate maintenance job needed). + Falls back to a plain cache.set when Redis is unavailable (index skipped). + """ + now = int(time.time()) + expire_at = now + ttl + try: + from django_redis import get_redis_connection + client = get_redis_connection('ratelimit') + client.eval(_BLOCK_LUA, 2, block_key, index_key, ttl, expire_at, now) + except Exception: + caches['ratelimit'].set(block_key, 1, timeout=ttl) + + +class RateLimitMiddleware: + BLOCK_TTL = 300 # seconds an IP/user stays blocked after threshold breach + + # In-process cache for the Redis UA deny list. + # Shared across all instances in the same worker process (one per worker). + # Refreshed every RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds via SMEMBERS. + _ua_blocklist: set = set() + _ua_blocklist_fetched_at: float = 0.0 + + # In-process cache for the Redis IP-prefix deny list (operator-curated SET + # of dotted-decimal prefixes). Normalized to trailing-dot form on refresh so + # checking is O(1) per candidate via set membership. + _ip_prefix_blocklist: set = set() + _ip_prefix_blocklist_fetched_at: float = 0.0 + + def __init__(self, get_response): + self.get_response = get_response + self.anon_threshold, self.anon_window = _parse_rate(settings.RATE_LIMITER_RATE) + self.auth_threshold, self.auth_window = _parse_rate(settings.RATE_LIMITER_RATE_AUTHENTICATED) + self._rl_cache = caches['ratelimit'] + self.not_found_threshold = settings.RATE_LIMIT_404_THRESHOLD + self._bypass_paths = [ + re.compile(p) for p in getattr(settings, 'RATE_LIMIT_BYPASS_PATHS', []) + ] + self.api_quota_daily = settings.API_QUOTA_DAILY + self.api_quota_weekly = settings.API_QUOTA_WEEKLY + self.api_rate_limit_enabled = getattr(settings, 'API_RATE_LIMIT_ENABLED', True) + self.api_threshold = getattr(settings, 'API_RATE_LIMIT_THRESHOLD', 60) + self.api_window = getattr(settings, 'API_RATE_LIMIT_WINDOW_SECONDS', 60) + self.api_block_seconds = getattr(settings, 'API_RATE_LIMIT_BLOCK_SECONDS', 300) + self.api_same_origin_bypass = getattr(settings, 'API_RATE_LIMIT_SAME_ORIGIN_BYPASS', True) + logger.info( + '[RATELIMIT] anon=%s auth=%s bot=%s bypass_paths=%s', + settings.RATE_LIMITER_RATE, + settings.RATE_LIMITER_RATE_AUTHENTICATED, + settings.RATE_LIMITER_RATE_BOT, + [p.pattern for p in self._bypass_paths] or '(none)', + ) + logger.info( + '[API QUOTAS] daily=%s weekly=%s (all callers keyed by IP)', + settings.API_QUOTA_DAILY, + settings.API_QUOTA_WEEKLY, + ) + logger.info( + '[API RATE LIMIT] enabled=%s threshold=%s window=%ss block=%ss same_origin_bypass=%s', + self.api_rate_limit_enabled, self.api_threshold, self.api_window, + self.api_block_seconds, self.api_same_origin_bypass, + ) + + def __call__(self, request): + if any(p.match(request.path) for p in self._bypass_paths): + return self.get_response(request) + + # Reject parameter pollution: the `page` param must appear at most once. + # Duplicate page= values (e.g. ?page=3&page=2&page=3) are a bot + # fingerprint — no legitimate browser or script sends this. + if len(request.GET.getlist('page')) > 1: + logger.warning( + 'ratelimit_block layer=django reason=param_pollution ip=%s path=%s', + get_client_ip(request), request.path, + ) + self._inc_block_metric('param_pollution') + response = HttpResponse(status=400) + response['X-RateLimit-Reason'] = 'param_pollution' + return response + + if request.path.startswith('/api/'): + return self._handle_api(request) + + decision = self._evaluate(request) + if decision['action'] == 'block': + logger.warning( + 'ratelimit_block layer=django reason=%s ip=%s path=%s namespace=%s', + decision['reason'], + decision['ip'], + request.path, + _NAMESPACE, + extra={'ua': request.META.get('HTTP_USER_AGENT', '')}, + ) + self._inc_block_metric(decision['reason']) + if decision['reason'] == 'ip_prefix_blocked': + response = HttpResponse(status=403) + else: + response = HttpResponse(status=429) + response['Retry-After'] = self.BLOCK_TTL + response['X-RateLimit-Reason'] = decision['reason'] + return response + logger.debug( + 'ratelimit_pass ip=%s path=%s user=%s namespace=%s', + decision['ip'], + request.path, + getattr(getattr(request, 'user', None), 'pk', 'anon'), + _NAMESPACE, + ) + response = self.get_response(request) + if response.status_code == 404: + self._handle_not_found(request, decision['ip']) + return response + + # ------------------------------------------------------------------ + # /api/ handling + # ------------------------------------------------------------------ + + def _api_block_response(self, reason, retry_after=None): + from django.http import JsonResponse + if reason == 'ip_prefix_blocked': + resp = JsonResponse({'detail': 'Forbidden.'}, status=403) + resp['X-RateLimit-Reason'] = reason + return resp + if retry_after is None: + retry_after = self.api_block_seconds + resp = JsonResponse( + {'detail': 'API rate limit exceeded. Please reduce polling frequency.', + 'retry_after_seconds': retry_after}, + status=429, + ) + resp['Retry-After'] = retry_after + resp['X-RateLimit-Reason'] = reason + return resp + + def _handle_api(self, request): + # 1. OPTIONS preflight — CORS must never be blocked + if request.method == 'OPTIONS': + return self.get_response(request) + + ip = get_client_ip(request) + + # 2. IP-prefix block — operator-curated deny list, applies to everyone. + # Block checks (2-4) must run before the same-origin bypass below: + # Origin/Referer are client-controlled and trivially spoofable, so + # they must never be able to override an already-made block decision. + if self._is_ip_prefix_blocked(ip): + logger.warning( + 'api_rate_limit_block reason=ip_prefix_blocked ip=%s path=%s user_agent=%s', + ip, request.path, request.META.get('HTTP_USER_AGENT', ''), + ) + self._inc_block_metric('api_ip_prefix_blocked') + return self._api_block_response('ip_prefix_blocked') + + # 3. Global IP block also covers /api/ + if self._rl_cache.get(RL_IP_BLOCKED.format(ip=ip)): + logger.warning( + 'api_rate_limit_block reason=global_ip_blocked ip=%s path=%s user_agent=%s', + ip, request.path, request.META.get('HTTP_USER_AGENT', ''), + ) + self._inc_block_metric('api_global_ip_blocked') + return self._api_block_response('global_ip_blocked') + + # 4. API-specific block (blocks /api/ only, never set by non-/api/ paths) + if self._rl_cache.get(RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip)): + logger.warning( + 'api_rate_limit_block reason=api_ip_blocked ip=%s path=%s user_agent=%s', + ip, request.path, request.META.get('HTTP_USER_AGENT', ''), + ) + self._inc_block_metric('api_ip_blocked') + return self._api_block_response('api_ip_blocked') + + # 5. Same-origin (SAPL's own polling) — exempt from quota/rate-limit + # accounting only (steps 6-7 below). Must come after the block + # checks above, never before them. + if self.api_same_origin_bypass and _is_same_origin(request): + return self.get_response(request) + + # 6. Daily/weekly quota (existing logic, preserved) + exceeded = self._check_api_quota(request) + if exceeded: + logger.warning( + 'quota_exceeded window=%s ip=%s path=%s namespace=%s', + exceeded, ip, request.path, _NAMESPACE, + extra={'ua': request.META.get('HTTP_USER_AGENT', '')}, + ) + self._inc_block_metric(f'quota_{exceeded}') + response = HttpResponse(status=429) + response['Retry-After'] = 86400 + response['X-RateLimit-Reason'] = f'quota_{exceeded}' + return response + + # 7. Per-minute rate limit — 60/min for all callers (anon and auth). + # Auth is not exempt: authenticating must not bypass this cap. + # Writes rl:api:ip::blocked only — never rl:ip::blocked. + if self.api_rate_limit_enabled: + count = self._incr_with_ttl(RL_API_IP_REQUESTS.format(ns=_NAMESPACE, ip=ip), self.api_window) + if count >= self.api_threshold: + _set_block(RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip), _index_shard(ip, RL_INDEX_API_BLOCKED_IPS), self.api_block_seconds) + logger.warning( + 'api_rate_limit_block reason=api_threshold_exceeded ' + 'ip=%s path=%s user_agent=%s count=%s threshold=%s', + ip, request.path, request.META.get('HTTP_USER_AGENT', ''), + count, self.api_threshold, + ) + self._inc_block_metric('api_threshold_exceeded') + return self._api_block_response('api_threshold_exceeded') + return self.get_response(request) + + # ------------------------------------------------------------------ + # Evaluation + # ------------------------------------------------------------------ + + def _evaluate(self, request): + ip = get_client_ip(request) + + # Check 0: IP-prefix block — operator-curated deny list, applies to everyone + if self._is_ip_prefix_blocked(ip): + return {'action': 'block', 'reason': 'ip_prefix_blocked', 'ip': ip} + + # Check 1: known bad UA (hardcoded Python list — substring match) + ua = request.META.get('HTTP_USER_AGENT', '') + for fragment in BOT_UA_FRAGMENTS: + if fragment.lower() in ua.lower(): + return {'action': 'block', 'reason': 'known_ua', 'ip': ip} + + # Check 1b: runtime UA deny list (Redis SET — token hash match) + if self._is_redis_blocked_ua(ua): + return {'action': 'block', 'reason': 'redis_ua', 'ip': ip} + + # Check 2: IP already blocked — authenticated users are exempt since they + # have independent per-user limiting at check 3c; IP blocks target anonymous traffic. + user = getattr(request, 'user', None) + if not (user and user.is_authenticated) and self._rl_cache.get(RL_IP_BLOCKED.format(ip=ip)): + return {'action': 'block', 'reason': 'ip_blocked', 'ip': ip} + + if user is not None and user.is_authenticated: + return self._evaluate_authenticated(request, ip) + return self._evaluate_anonymous(request, ip) + + def _evaluate_authenticated(self, request, ip): + uid = str(request.user.pk) + + # Check 3a: user already blocked + if self._rl_cache.get(RL_USER_BLOCKED.format(ns=_NAMESPACE, uid=uid)): + return {'action': 'block', 'reason': 'user_blocked', 'ip': ip} + + # Check 3b: suspicious headers + if _is_suspicious_headers(request): + return {'action': 'block', 'reason': 'suspicious_headers_auth', 'ip': ip} + + # Check 3c: authenticated request rate — return 429 for this request only; + # no persistent block key so the window resets naturally after auth_window + # seconds. A 300s lockout is wrong for a logged-in user who clicked fast. + count = self._incr_with_ttl( + RL_USER_REQUESTS.format(ns=_NAMESPACE, uid=uid), ttl=self.auth_window + ) + if count >= self.auth_threshold: + return {'action': 'block', 'reason': 'auth_user_rate', 'ip': ip} + + return {'action': 'pass', 'ip': ip} + + def _evaluate_anonymous(self, request, ip): + # Check 4a: suspicious headers + if _is_suspicious_headers(request): + return {'action': 'block', 'reason': 'suspicious_headers', 'ip': ip} + + # Check 4b: IP request rate + count = self._incr_with_ttl(RL_IP_REQUESTS.format(ip=ip), ttl=self.anon_window) + if count >= self.anon_threshold: + _set_block(RL_IP_BLOCKED.format(ip=ip), _index_shard(ip, RL_INDEX_BLOCKED_IPS), self.BLOCK_TTL) + return {'action': 'block', 'reason': 'ip_rate', 'ip': ip} + + # Check 4c: per-namespace/IP/window (catches UA rotators behind NAT) + bucket = int(time.time() // self.anon_window) + count = self._incr_with_ttl( + RL_NS_WINDOW.format(ns=_NAMESPACE, ip=ip, bucket=bucket), + ttl=self.anon_window * 2, + ) + if count >= self.anon_threshold: + _set_block(RL_IP_BLOCKED.format(ip=ip), _index_shard(ip, RL_INDEX_BLOCKED_IPS), self.BLOCK_TTL) + return {'action': 'block', 'reason': 'ua_rotation', 'ip': ip} + + return {'action': 'pass', 'ip': ip} + + # ------------------------------------------------------------------ + # Helpers — delegate to module-level so media.py can reuse them + # ------------------------------------------------------------------ + + def _handle_not_found(self, request, ip): + """ + Block IPs that accumulate too many 404s in one window — catches scanner + probes that use paths without recognised extensions (e.g. /wp-login, + /.git/HEAD, /xmlrpc) and bypass check 2b entirely. + Only anonymous requests are counted; authenticated users have their own + per-user rate limit and may legitimately hit stale bookmarks. + """ + user = getattr(request, 'user', None) + if user and user.is_authenticated: + return + count = self._incr_with_ttl(RL_IP_404S.format(ip=ip), ttl=self.anon_window) + if count >= self.not_found_threshold: + _set_block(RL_IP_BLOCKED.format(ip=ip), _index_shard(ip, RL_INDEX_BLOCKED_IPS), self.BLOCK_TTL) + logger.warning( + 'ratelimit_block layer=django reason=404_scan ip=%s path=%s namespace=%s', + ip, request.path, _NAMESPACE, + extra={'ua': request.META.get('HTTP_USER_AGENT', '')}, + ) + self._inc_block_metric('404_scan') + + def _check_api_quota(self, request): + """ + Increment daily and weekly API quota counters for all /api/ callers. + All callers are keyed by IP — auth status is not checked. + + Keys are HASH structures: one hash per tenant per period, field = IP. + This keeps the global keyspace at O(tenants) instead of O(unique IPs), + reducing Redis memory ~57% vs. per-IP STRING keys at scale. + + Fails open (returns None) if Redis/cache is unavailable. + """ + today = date.today() + iso = today.isocalendar() + date_str = today.isoformat() + week_str = f'{iso[0]}-W{iso[1]:02d}' + + ip = get_client_ip(request) + api_d_hash = API_QUOTA_DAILY_HASH.format(ns=_NAMESPACE, date=date_str) + api_w_hash = API_QUOTA_WEEKLY_HASH.format(ns=_NAMESPACE, week=week_str) + + try: + if _hincrby_with_ttl(api_d_hash, ip, 86400) > self.api_quota_daily: + return 'daily' + if _hincrby_with_ttl(api_w_hash, ip, 7 * 86400) > self.api_quota_weekly: + return 'weekly' + except Exception: + pass # fail open — quota not enforced when Redis unavailable + return None + + def _incr_with_ttl(self, key, ttl): + return _incr_with_ttl(key, ttl) + + def _inc_block_metric(self, reason): + """Increment daily per-reason block counter in Redis DB 1 (TTL 8 days).""" + hash_key = RL_METRICS_BLOCKED.format(ns=_NAMESPACE, date=date.today().isoformat()) + try: + _hincrby_with_ttl(hash_key, reason, ttl=8 * 86400) + except Exception: + pass + + def _refresh_ua_blocklist(self): + """ + Fetch the full UA deny list from Redis DB 1 (SMEMBERS). + Stores sha256 hex-strings in the class-level set. + Falls back silently — an empty set means no runtime blocks. + """ + try: + from django_redis import get_redis_connection + client = get_redis_connection('ratelimit') + raw = client.smembers(RL_UA_BLOCKLIST) + RateLimitMiddleware._ua_blocklist = { + m.decode() if isinstance(m, bytes) else m for m in raw + } + RateLimitMiddleware._ua_blocklist_fetched_at = time.time() + logger.debug('[RATELIMIT] ua_blocklist refreshed entries=%d', len(raw)) + except Exception as exc: + logger.debug('[RATELIMIT] ua_blocklist refresh skipped: %s', exc) + + def _is_redis_blocked_ua(self, ua): + """ + Return True if any slash/space/semicolon token in `ua` has a sha256 + that appears in the Redis UA deny list. + + The SET stores sha256(fragment) — e.g. sha256('GPTBot'). + Tokenising by common UA separators means 'GPTBot/1.1 (OpenAI)' + produces token 'GPTBot' whose hash matches the seeded entry. + Degrades to False when Redis is unavailable. + """ + if time.time() - self._ua_blocklist_fetched_at > settings.RATE_LIMITER_UA_BLOCKLIST_REFRESH: + self._refresh_ua_blocklist() + if not self._ua_blocklist: + return False + tokens = re.split(r'[\s/;()+,]+', ua) + return any( + hashlib.sha256(t.encode()).hexdigest() in self._ua_blocklist + for t in tokens if t + ) + + def _refresh_ip_prefix_blocklist(self): + """ + Fetch the full IP-prefix deny list from Redis DB 1 (SMEMBERS) and + normalise entries to trailing-dot form so membership checks are O(1). + + Normalisation: strip trailing dot, then re-add if fewer than 3 dots + (i.e. it's a prefix, not a full dotted-quad). Examples: + '103.124.225' → '103.124.225.' + '103.124.225.' → '103.124.225.' + '103.124.225.7'→ '103.124.225.7' (exact IP, no trailing dot) + """ + try: + from django_redis import get_redis_connection + client = get_redis_connection('ratelimit') + raw = client.smembers(RL_IP_PREFIX_BLOCKLIST) + normalized = set() + for m in raw: + entry = m.decode() if isinstance(m, bytes) else m + stripped = entry.rstrip('.') + normalized.add(stripped + '.' if stripped.count('.') < 3 else stripped) + RateLimitMiddleware._ip_prefix_blocklist = normalized + RateLimitMiddleware._ip_prefix_blocklist_fetched_at = time.time() + logger.debug('[RATELIMIT] ip_prefix_blocklist refreshed entries=%d', len(normalized)) + except Exception as exc: + logger.debug('[RATELIMIT] ip_prefix_blocklist refresh skipped: %s', exc) + + def _is_ip_prefix_blocked(self, ip): + """ + Return True if `ip` or any of its dot-anchored prefixes is in the + local IP-prefix deny set. + + Generates up to 4 candidates for '203.0.113.42': + '203.', '203.0.', '203.0.113.', '203.0.113.42' + Each lookup is O(1) against the normalised in-process set. + Degrades to False when Redis is unavailable (empty set). + """ + if time.time() - self._ip_prefix_blocklist_fetched_at > settings.RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH: + self._refresh_ip_prefix_blocklist() + if not self._ip_prefix_blocklist: + return False + parts = ip.split('.') + if len(parts) != 4: + return False + candidates = ( + parts[0] + '.', + parts[0] + '.' + parts[1] + '.', + parts[0] + '.' + parts[1] + '.' + parts[2] + '.', + ip, + ) + return any(c in self._ip_prefix_blocklist for c in candidates) diff --git a/sapl/middleware/test_ratelimiter.py b/sapl/middleware/test_ratelimiter.py new file mode 100644 index 000000000..e0a4fe5b6 --- /dev/null +++ b/sapl/middleware/test_ratelimiter.py @@ -0,0 +1,926 @@ +""" +Unit tests for sapl/middleware/ratelimit.py. + +No database access is needed — all tests use RequestFactory and mocks. +Redis is never contacted; _incr_with_ttl is either mocked directly on the +middleware instance or the fallback non-atomic path is exercised via the +mock cache. +""" + +import time + +import pytest +from unittest.mock import MagicMock, patch +from django.test import RequestFactory + +from sapl.middleware.ratelimit import ( + _NAMESPACE, + _hincrby_with_ttl, + _index_shard, + _is_same_origin, + _is_suspicious_headers, + _parse_rate, + get_client_ip, + make_ratelimit_cache_key, + API_QUOTA_DAILY_HASH, + API_QUOTA_WEEKLY_HASH, + RateLimitMiddleware, + RL_API_IP_BLOCKED, + RL_API_IP_REQUESTS, + RL_INDEX_BLOCKED_IPS, + RL_IP_BLOCKED, + RL_IP_PREFIX_BLOCKLIST, + RL_USER_BLOCKED, + smart_key, + smart_rate, +) + +# --------------------------------------------------------------------------- +# Shared test helpers +# --------------------------------------------------------------------------- + +_factory = RequestFactory() + +# Headers that a normal browser would send — used as the default baseline. +_NORMAL_HEADERS = { + 'HTTP_ACCEPT': 'text/html,application/xhtml+xml', + 'HTTP_ACCEPT_LANGUAGE': 'pt-BR,pt;q=0.9', +} + + +def _req(ip='1.2.3.4', ua='Mozilla/5.0', path='/', extra_meta=None): + """GET request with sensible defaults and browser-like headers.""" + request = _factory.get(path) + request.META.update({'REMOTE_ADDR': ip, 'HTTP_USER_AGENT': ua, **_NORMAL_HEADERS}) + if extra_meta: + request.META.update(extra_meta) + return request + + +def _anon_req(**kwargs): + r = _req(**kwargs) + r.user = MagicMock(is_authenticated=False) + return r + + +def _auth_req(uid=7, **kwargs): + r = _req(**kwargs) + r.user = MagicMock(is_authenticated=True, pk=uid) + return r + + +def _make_middleware( + anon_rate='35/m', + auth_rate='120/m', + api_rate_limit_enabled=True, + api_threshold=60, + api_window=60, + api_block_seconds=300, + api_same_origin_bypass=True, +): + """ + Return (middleware, mock_cache). + + The ratelimit cache is replaced with a MagicMock whose .get() returns None + by default (nothing blocked, no counters set). Tests may replace + mock_cache.get.side_effect or mock mw._incr_with_ttl directly. + + sapl.middleware.ratelimit imports settings as `from sapl import settings` + (a direct module reference), so django.test.override_settings has no effect + on it. We patch the name in the ratelimit module's namespace instead. + """ + mock_cache = MagicMock() + mock_cache.get.return_value = None + get_response = MagicMock(return_value=MagicMock(status_code=200)) + + mock_settings = MagicMock() + mock_settings.RATE_LIMITER_RATE = anon_rate + mock_settings.RATE_LIMITER_RATE_AUTHENTICATED = auth_rate + mock_settings.RATE_LIMITER_RATE_BOT = '5/m' + mock_settings.RATE_LIMIT_404_THRESHOLD = 20 + mock_settings.RATE_LIMIT_BYPASS_PATHS = [] + mock_settings.POD_NAMESPACE = _NAMESPACE # keep module-level _NAMESPACE consistent + mock_settings.API_QUOTA_DAILY = 999999 + mock_settings.API_QUOTA_WEEKLY = 999999 + mock_settings.RATE_LIMITER_UA_BLOCKLIST_REFRESH = 60 + mock_settings.RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH = 60 + mock_settings.API_RATE_LIMIT_ENABLED = api_rate_limit_enabled + mock_settings.API_RATE_LIMIT_THRESHOLD = api_threshold + mock_settings.API_RATE_LIMIT_WINDOW_SECONDS = api_window + mock_settings.API_RATE_LIMIT_BLOCK_SECONDS = api_block_seconds + mock_settings.API_RATE_LIMIT_SAME_ORIGIN_BYPASS = api_same_origin_bypass + mock_settings.RATE_LIMITER_INDEX_SHARDS = 3 + + with ( + patch('sapl.middleware.ratelimit.caches') as mock_caches, + patch('sapl.middleware.ratelimit.settings', mock_settings), + ): + mock_caches.__getitem__.return_value = mock_cache + mw = RateLimitMiddleware(get_response) + # __init__ already set mw._rl_cache = caches['ratelimit'] == mock_cache, + # but reassign explicitly so tests have a direct handle on the same object. + mw._rl_cache = mock_cache + return mw, mock_cache + + +# --------------------------------------------------------------------------- +# _parse_rate +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize('rate_str,expected', [ + ('35/m', (35, 60)), + ('120/m', (120, 60)), + ('10/s', (10, 1)), + ('5/h', (5, 3600)), + ('1/M', (1, 60)), # period is case-insensitive +]) +def test_parse_rate(rate_str, expected): + assert _parse_rate(rate_str) == expected + + +# --------------------------------------------------------------------------- +# make_ratelimit_cache_key — pass-through, no mangling +# --------------------------------------------------------------------------- + +def test_make_ratelimit_cache_key_passthrough(): + assert make_ratelimit_cache_key('rl:ip:1.2.3.4:reqs', 'some_prefix', 1) == 'rl:ip:1.2.3.4:reqs' + assert make_ratelimit_cache_key('rl:abc123', '', 99) == 'rl:abc123' + + +# --------------------------------------------------------------------------- +# _is_suspicious_headers +# --------------------------------------------------------------------------- + +def test_suspicious_both_headers_missing(): + r = _factory.get('/') + r.META.pop('HTTP_ACCEPT', None) + r.META.pop('HTTP_ACCEPT_LANGUAGE', None) + assert _is_suspicious_headers(r) is True + + +def test_suspicious_one_header_missing_is_not_suspicious(): + """Only flagged when *both* headers are absent.""" + r = _factory.get('/') + r.META['HTTP_ACCEPT'] = 'text/html' + r.META.pop('HTTP_ACCEPT_LANGUAGE', None) + assert _is_suspicious_headers(r) is False + + +def test_suspicious_both_headers_present(): + r = _factory.get('/') + r.META['HTTP_ACCEPT'] = 'text/html' + r.META['HTTP_ACCEPT_LANGUAGE'] = 'pt-BR' + assert _is_suspicious_headers(r) is False + + +# --------------------------------------------------------------------------- +# get_client_ip — header priority and XFF chain +# --------------------------------------------------------------------------- + +def test_get_client_ip_remote_addr(): + r = _factory.get('/') + r.META['REMOTE_ADDR'] = '10.0.0.1' + assert get_client_ip(r) == '10.0.0.1' + + +def test_get_client_ip_xff_single(): + r = _factory.get('/') + r.META['HTTP_X_FORWARDED_FOR'] = '203.0.113.5' + assert get_client_ip(r) == '203.0.113.5' + + +def test_get_client_ip_xff_chain_uses_leftmost(): + """The leftmost IP in XFF is the real client; the rest are proxies.""" + r = _factory.get('/') + r.META['HTTP_X_FORWARDED_FOR'] = '203.0.113.5, 10.0.0.1, 10.0.0.2' + assert get_client_ip(r) == '203.0.113.5' + + +def test_get_client_ip_x_real_ip_used_when_no_xff(): + r = _factory.get('/') + r.META['REMOTE_ADDR'] = '127.0.0.1' + r.META['HTTP_X_REAL_IP'] = '203.0.113.9' + assert get_client_ip(r) == '203.0.113.9' + + +def test_get_client_ip_xff_preferred_over_x_real_ip(): + r = _factory.get('/') + r.META['HTTP_X_FORWARDED_FOR'] = '203.0.113.1' + r.META['HTTP_X_REAL_IP'] = '203.0.113.2' + assert get_client_ip(r) == '203.0.113.1' + + +# --------------------------------------------------------------------------- +# smart_key / smart_rate +# --------------------------------------------------------------------------- + +def test_smart_key_anon_returns_masked_ip(): + r = _anon_req(ip='5.5.5.5') + assert smart_key(None, r) == '5.5.5.5' + + +def test_smart_key_auth_returns_pk_string(): + r = _auth_req(uid=42, ip='5.5.5.5') + assert smart_key(None, r) == '42' + + +def test_smart_rate_anon_returns_anon_rate(): + with patch('sapl.middleware.ratelimit.settings') as mock_s: + mock_s.RATE_LIMITER_RATE = '35/m' + mock_s.RATE_LIMITER_RATE_AUTHENTICATED = '120/m' + assert smart_rate(None, _anon_req()) == '35/m' + + +def test_smart_rate_auth_returns_auth_rate(): + with patch('sapl.middleware.ratelimit.settings') as mock_s: + mock_s.RATE_LIMITER_RATE = '35/m' + mock_s.RATE_LIMITER_RATE_AUTHENTICATED = '120/m' + assert smart_rate(None, _auth_req()) == '120/m' + + +# --------------------------------------------------------------------------- +# _index_shard — sharded ZSET key routing +# --------------------------------------------------------------------------- + +def test_index_shard_is_deterministic(): + """Same IP always maps to the same shard.""" + from sapl.middleware.ratelimit import _index_shard + with patch('sapl.middleware.ratelimit.settings') as mock_s: + mock_s.RATE_LIMITER_INDEX_SHARDS = 3 + key1 = _index_shard('1.2.3.4', 'rl:index:blocked_ips') + key2 = _index_shard('1.2.3.4', 'rl:index:blocked_ips') + assert key1 == key2 + + +def test_index_shard_stays_within_range(): + """Shard suffix is always 0 … N-1.""" + from sapl.middleware.ratelimit import _index_shard + import re + with patch('sapl.middleware.ratelimit.settings') as mock_s: + mock_s.RATE_LIMITER_INDEX_SHARDS = 3 + ips = [f'10.0.0.{i}' for i in range(50)] + for ip in ips: + key = _index_shard(ip, 'rl:index:blocked_ips') + m = re.search(r':(\d+)$', key) + assert m and 0 <= int(m.group(1)) < 3, f'out-of-range shard for {ip}: {key}' + + +def test_index_shard_distributes_across_shards(): + """With enough IPs, all 3 shards are used.""" + from sapl.middleware.ratelimit import _index_shard + with patch('sapl.middleware.ratelimit.settings') as mock_s: + mock_s.RATE_LIMITER_INDEX_SHARDS = 3 + shards_seen = { + _index_shard(f'192.168.{i}.{j}', 'rl:index:blocked_ips').split(':')[-1] + for i in range(5) for j in range(10) + } + assert shards_seen == {'0', '1', '2'} + + +# --------------------------------------------------------------------------- +# Check 0 — IP-prefix blocklist (operator-curated SET, dot-anchored matching) +# --------------------------------------------------------------------------- + +@pytest.fixture +def _seed_prefix_blocklist(): + """ + Seed RateLimitMiddleware._ip_prefix_blocklist for the test and restore the + previous class-level state afterwards (it's shared across instances, like + the UA blocklist). + """ + saved_list = RateLimitMiddleware._ip_prefix_blocklist + saved_fetched_at = RateLimitMiddleware._ip_prefix_blocklist_fetched_at + + def _seed(prefixes): + RateLimitMiddleware._ip_prefix_blocklist = set(prefixes) + RateLimitMiddleware._ip_prefix_blocklist_fetched_at = time.time() + + yield _seed + + RateLimitMiddleware._ip_prefix_blocklist = saved_list + RateLimitMiddleware._ip_prefix_blocklist_fetched_at = saved_fetched_at + + +def test_is_ip_prefix_blocked_matches_dot_boundary(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['103.124.225']) + assert mw._is_ip_prefix_blocked('103.124.225.7') is True + # must not match on raw substring — only on a full-octet boundary + assert mw._is_ip_prefix_blocked('103.124.2250.1') is False + assert mw._is_ip_prefix_blocked('103.124.2255') is False + + +def test_is_ip_prefix_blocked_exact_match(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['103.124.225']) + assert mw._is_ip_prefix_blocked('103.124.225') is True + + +def test_is_ip_prefix_blocked_full_address_matches_only_exactly(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['103.124.225.7']) + assert mw._is_ip_prefix_blocked('103.124.225.7') is True + # a full dotted-quad entry must not be treated as a prefix of a longer string + assert mw._is_ip_prefix_blocked('103.124.225.70') is False + + +def test_is_ip_prefix_blocked_trailing_dot_in_stored_prefix(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['103.124.225.']) + # anchoring must not double the dot ('103.124.225..') and break the match + assert mw._is_ip_prefix_blocked('103.124.225.7') is True + + +def test_is_ip_prefix_blocked_empty_list_passes(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist([]) + assert mw._is_ip_prefix_blocked('1.2.3.4') is False + + +def test_is_ip_prefix_blocked_no_match_passes(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['103.124.225', '45.177.154']) + assert mw._is_ip_prefix_blocked('1.2.3.4') is False + + +def test_evaluate_blocks_on_ip_prefix_before_other_checks(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['1.2.3']) + result = mw._evaluate(_anon_req(ip='1.2.3.4', ua='GPTBot')) + # would otherwise match the known_ua check — prefix block must win + assert result == {'action': 'block', 'reason': 'ip_prefix_blocked', 'ip': '1.2.3.4'} + + +def test_evaluate_passes_through_when_ip_not_in_prefix_list(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['9.9.9']) + mw._incr_with_ttl = MagicMock(return_value=1) + result = mw._evaluate(_anon_req(ip='1.2.3.4')) + assert result['action'] == 'pass' + + +def test_refresh_ip_prefix_blocklist_populates_set(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist([]) # start empty so the refresh result is observable + + mock_client = MagicMock() + mock_client.smembers.return_value = {b'103.124.225', b'45.177.154'} + with patch('django_redis.get_redis_connection', return_value=mock_client): + mw._refresh_ip_prefix_blocklist() + + mock_client.smembers.assert_called_once_with(RL_IP_PREFIX_BLOCKLIST) + assert RateLimitMiddleware._ip_prefix_blocklist == {'103.124.225', '45.177.154'} + + +# --------------------------------------------------------------------------- +# Check 1 — known bot User-Agent +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize('ua', [ + 'GPTBot/1.0', + 'Mozilla/5.0 (compatible; ClaudeBot/1.0)', + 'PerplexityBot', + 'Bytespider', + 'AhrefsBot/7.0', + 'meta-externalagent/1.1', + 'OAI-SearchBot', + 'Mozilla/5.0 (compatible; bingbot/2.0)', + 'SERankingBacklinksBot/1.0', + 'Mozilla/5.0 AppleWebKit Chrome/98.0.4758.80', +]) +def test_known_bot_ua_blocked(ua): + mw, _ = _make_middleware() + result = mw._evaluate(_anon_req(ua=ua)) + assert result == {'action': 'block', 'reason': 'known_ua', 'ip': '1.2.3.4'} + + +def test_bot_ua_check_is_case_insensitive(): + mw, _ = _make_middleware() + result = mw._evaluate(_anon_req(ua='gptbot/2.0')) + assert result['reason'] == 'known_ua' + + +# --------------------------------------------------------------------------- +# Check 2 — IP already blocked in cache +# --------------------------------------------------------------------------- + +def test_ip_blocked_in_cache(): + mw, mock_cache = _make_middleware() + ip = '1.2.3.4' + mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None + result = mw._evaluate(_anon_req(ip=ip)) + assert result == {'action': 'block', 'reason': 'ip_blocked', 'ip': ip} + + +# --------------------------------------------------------------------------- +# Check 3a — authenticated user blocked in cache +# --------------------------------------------------------------------------- + +def test_auth_user_blocked_in_cache(): + mw, mock_cache = _make_middleware() + uid = '7' + mock_cache.get.side_effect = lambda key: ( + 1 if key == RL_USER_BLOCKED.format(ns=_NAMESPACE, uid=uid) else None + ) + result = mw._evaluate(_auth_req(uid=int(uid))) + assert result == {'action': 'block', 'reason': 'user_blocked', 'ip': '1.2.3.4'} + + +# --------------------------------------------------------------------------- +# Check 3b — authenticated + suspicious headers +# --------------------------------------------------------------------------- + +def test_auth_suspicious_headers_blocked(): + mw, _ = _make_middleware() + r = _auth_req() + r.META.pop('HTTP_ACCEPT', None) + r.META.pop('HTTP_ACCEPT_LANGUAGE', None) + r.META.pop('HTTP_USER_AGENT', None) + result = mw._evaluate(r) + assert result == {'action': 'block', 'reason': 'suspicious_headers_auth', 'ip': '1.2.3.4'} + + +# --------------------------------------------------------------------------- +# Check 3c — authenticated request rate +# --------------------------------------------------------------------------- + +def test_auth_rate_exceeded_blocks_and_marks_user_blocked(): + mw, mock_cache = _make_middleware(auth_rate='5/m') + mw._incr_with_ttl = MagicMock(return_value=5) # exactly at threshold + result = mw._evaluate(_auth_req(uid=7)) + # auth_user_rate has no persistent block key — the window resets naturally + assert result == {'action': 'block', 'reason': 'auth_user_rate', 'ip': '1.2.3.4'} + mock_cache.set.assert_not_called() + + +def test_auth_under_rate_passes(): + mw, mock_cache = _make_middleware(auth_rate='5/m') + mw._incr_with_ttl = MagicMock(return_value=4) # one below threshold + result = mw._evaluate(_auth_req(uid=7)) + assert result == {'action': 'pass', 'ip': '1.2.3.4'} + mock_cache.set.assert_not_called() + + +# --------------------------------------------------------------------------- +# Check 4a — anonymous + suspicious headers +# --------------------------------------------------------------------------- + +def test_anon_suspicious_headers_blocked(): + mw, _ = _make_middleware() + r = _anon_req() + r.META.pop('HTTP_ACCEPT', None) + r.META.pop('HTTP_ACCEPT_LANGUAGE', None) + r.META.pop('HTTP_USER_AGENT', None) + result = mw._evaluate(r) + assert result == {'action': 'block', 'reason': 'suspicious_headers', 'ip': '1.2.3.4'} + + +# --------------------------------------------------------------------------- +# Check 4b — anonymous IP request rate +# --------------------------------------------------------------------------- + +def test_anon_ip_rate_exceeded_blocks_and_marks_ip_blocked(): + mw, _ = _make_middleware(anon_rate='5/m') + mw._incr_with_ttl = MagicMock(return_value=5) # first call (IP counter) hits threshold + with patch('sapl.middleware.ratelimit._set_block') as mock_set_block: + result = mw._evaluate(_anon_req()) + assert result == {'action': 'block', 'reason': 'ip_rate', 'ip': '1.2.3.4'} + mock_set_block.assert_called_once_with( + RL_IP_BLOCKED.format(ip='1.2.3.4'), + _index_shard('1.2.3.4', RL_INDEX_BLOCKED_IPS), + RateLimitMiddleware.BLOCK_TTL, + ) + + +# --------------------------------------------------------------------------- +# Check 4c — per-namespace/IP/window (UA rotation detection) +# --------------------------------------------------------------------------- + +def test_anon_ua_rotation_detected_blocks_and_marks_ip_blocked(): + mw, _ = _make_middleware(anon_rate='5/m') + # First call (IP counter) is under threshold; second (window counter) hits it. + mw._incr_with_ttl = MagicMock(side_effect=[4, 5]) + with patch('sapl.middleware.ratelimit._set_block') as mock_set_block: + result = mw._evaluate(_anon_req()) + assert result == {'action': 'block', 'reason': 'ua_rotation', 'ip': '1.2.3.4'} + mock_set_block.assert_called_once_with( + RL_IP_BLOCKED.format(ip='1.2.3.4'), + _index_shard('1.2.3.4', RL_INDEX_BLOCKED_IPS), + RateLimitMiddleware.BLOCK_TTL, + ) + + +def test_anon_under_all_thresholds_passes(): + mw, mock_cache = _make_middleware(anon_rate='5/m') + mw._incr_with_ttl = MagicMock(return_value=4) # both counters below threshold + result = mw._evaluate(_anon_req()) + assert result == {'action': 'pass', 'ip': '1.2.3.4'} + mock_cache.set.assert_not_called() + + +# --------------------------------------------------------------------------- +# __call__ — block returns 429, pass forwards to get_response +# --------------------------------------------------------------------------- + +def test_call_block_returns_429_with_retry_after_header(): + mw, _ = _make_middleware() + mw._evaluate = MagicMock(return_value={'action': 'block', 'reason': 'known_ua', 'ip': '1.2.3.4'}) + response = mw(_factory.get('/')) + assert response.status_code == 429 + assert response['Retry-After'] == str(RateLimitMiddleware.BLOCK_TTL) + mw.get_response.assert_not_called() + + +def test_call_ip_prefix_block_returns_403_without_retry_after_header(): + mw, _ = _make_middleware() + mw._evaluate = MagicMock(return_value={'action': 'block', 'reason': 'ip_prefix_blocked', 'ip': '1.2.3.4'}) + response = mw(_factory.get('/')) + assert response.status_code == 403 + assert 'Retry-After' not in response + assert response['X-RateLimit-Reason'] == 'ip_prefix_blocked' + mw.get_response.assert_not_called() + + +def test_call_pass_forwards_request_to_get_response(): + mw, _ = _make_middleware() + mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) + request = _anon_req() + mw(request) + mw.get_response.assert_called_once_with(request) + + +def test_call_rejects_duplicate_page_param_with_400(): + mw, _ = _make_middleware() + request = _factory.get('/sessao/pesquisar-sessao', data={'page': ['3', '2', '3']}) + response = mw(request) + assert response.status_code == 400 + assert response['X-RateLimit-Reason'] == 'param_pollution' + mw.get_response.assert_not_called() + + +def test_call_allows_single_page_param(): + mw, _ = _make_middleware() + mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) + request = _factory.get('/sessao/pesquisar-sessao', data={'page': '2'}) + mw(request) + mw.get_response.assert_called_once_with(request) + + +def test_call_allows_no_page_param(): + mw, _ = _make_middleware() + mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) + request = _factory.get('/sessao/pesquisar-sessao') + mw(request) + mw.get_response.assert_called_once_with(request) + + +# --------------------------------------------------------------------------- +# _is_same_origin +# --------------------------------------------------------------------------- + +def test_is_same_origin_no_headers_returns_false(): + r = _factory.get('/api/materia/') + r.META['SERVER_NAME'] = 'sapl.example.com' + r.META['SERVER_PORT'] = '80' + r.META.pop('HTTP_ORIGIN', None) + r.META.pop('HTTP_REFERER', None) + assert _is_same_origin(r) is False + + +def test_is_same_origin_matching_origin(): + r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') + r.META['HTTP_ORIGIN'] = 'https://sapl.example.com' + assert _is_same_origin(r) is True + + +def test_is_same_origin_mismatched_origin(): + r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') + r.META['HTTP_ORIGIN'] = 'https://other.example.com' + assert _is_same_origin(r) is False + + +def test_is_same_origin_wrong_origin_blocks_even_if_referer_matches(): + """If Origin is present and wrong, Referer must not be consulted.""" + r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') + r.META['HTTP_ORIGIN'] = 'https://evil.com' + r.META['HTTP_REFERER'] = 'https://sapl.example.com/page/' + assert _is_same_origin(r) is False + + +def test_is_same_origin_referer_used_when_no_origin(): + r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') + r.META.pop('HTTP_ORIGIN', None) + r.META['HTTP_REFERER'] = 'https://sapl.example.com/page/?q=1' + assert _is_same_origin(r) is True + + +def test_is_same_origin_port_stripped_from_both_sides(): + """Host with port and Origin without port must match after normalization.""" + r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='8000') + r.META['HTTP_HOST'] = 'sapl.example.com:8000' + r.META['HTTP_ORIGIN'] = 'http://sapl.example.com' + assert _is_same_origin(r) is True + + +def test_is_same_origin_case_insensitive(): + r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') + r.META['HTTP_ORIGIN'] = 'https://SAPL.EXAMPLE.COM' + assert _is_same_origin(r) is True + + +# --------------------------------------------------------------------------- +# _handle_api — OPTIONS and same-origin bypass +# --------------------------------------------------------------------------- + +def _api_req(ip='1.2.3.4', ua='Mozilla/5.0', path='/api/materia/', method='GET', extra_meta=None): + """Anonymous /api/ request with browser headers.""" + request = _factory.generic(method, path) + request.META.update({'REMOTE_ADDR': ip, 'HTTP_USER_AGENT': ua, **_NORMAL_HEADERS}) + if extra_meta: + request.META.update(extra_meta) + request.user = MagicMock(is_authenticated=False) + return request + + +def test_api_options_passes_without_counting(): + mw, _ = _make_middleware() + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(method='OPTIONS') + mw(request) + mw.get_response.assert_called_once_with(request) + mw._incr_with_ttl.assert_not_called() + + +def test_api_same_origin_passes_without_counting(): + mw, _ = _make_middleware() + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(extra_meta={ + 'SERVER_NAME': 'sapl.example.com', + 'SERVER_PORT': '80', + 'HTTP_HOST': 'sapl.example.com', + 'HTTP_ORIGIN': 'https://sapl.example.com', + }) + mw(request) + mw.get_response.assert_called_once_with(request) + mw._incr_with_ttl.assert_not_called() + + +def test_api_malicious_origin_is_not_same_origin(): + mw, _ = _make_middleware(api_threshold=999) + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock(return_value=1) + request = _api_req(extra_meta={ + 'SERVER_NAME': 'sapl.example.com', + 'SERVER_PORT': '80', + 'HTTP_HOST': 'sapl.example.com', + 'HTTP_ORIGIN': 'https://evil.com?x=sapl.example.com', + }) + mw(request) + # Must reach the counter (not short-circuit as same-origin) + mw._incr_with_ttl.assert_called_once() + + +# Same-origin headers a spoofing client can trivially forge — Origin/Referer +# carry no authentication, so they must never override an active block. +_SAME_ORIGIN_META = { + 'SERVER_NAME': 'sapl.example.com', + 'SERVER_PORT': '80', + 'HTTP_HOST': 'sapl.example.com', + 'HTTP_ORIGIN': 'https://sapl.example.com', +} + + +def test_api_same_origin_does_not_bypass_global_ip_block(): + mw, mock_cache = _make_middleware() + ip = '1.2.3.4' + mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(ip=ip, extra_meta=_SAME_ORIGIN_META) + + response = mw(request) + + mw.get_response.assert_not_called() + assert response.status_code == 429 + assert response['X-RateLimit-Reason'] == 'global_ip_blocked' + + +def test_api_same_origin_does_not_bypass_api_ip_block(): + mw, mock_cache = _make_middleware() + ip = '1.2.3.4' + blocked_key = RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip) + mock_cache.get.side_effect = lambda key: 1 if key == blocked_key else None + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(ip=ip, extra_meta=_SAME_ORIGIN_META) + + response = mw(request) + + mw.get_response.assert_not_called() + assert response.status_code == 429 + assert response['X-RateLimit-Reason'] == 'api_ip_blocked' + + +def test_api_same_origin_does_not_bypass_ip_prefix_block(_seed_prefix_blocklist): + mw, _ = _make_middleware() + _seed_prefix_blocklist(['1.2.3']) + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(ip='1.2.3.4', extra_meta=_SAME_ORIGIN_META) + + response = mw(request) + + mw.get_response.assert_not_called() + assert response.status_code == 403 + assert response['X-RateLimit-Reason'] == 'ip_prefix_blocked' + + +def test_api_same_origin_still_skips_quota_and_rate_limit_when_not_blocked(): + """Legitimate same-origin polling keeps its original exemption from accounting.""" + mw, _ = _make_middleware() + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(extra_meta=_SAME_ORIGIN_META) + + response = mw(request) + + mw.get_response.assert_called_once_with(request) + mw._check_api_quota.assert_not_called() + mw._incr_with_ttl.assert_not_called() + assert response.status_code == 200 + + +# --------------------------------------------------------------------------- +# _handle_api — Check 3a: IP-prefix block (operator-curated deny list) +# --------------------------------------------------------------------------- + +def test_api_blocks_on_ip_prefix_before_quota_checks(_seed_prefix_blocklist): + mw, mock_cache = _make_middleware() + _seed_prefix_blocklist(['1.2.3']) + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + request = _api_req(ip='1.2.3.4') + + response = mw(request) + + mw.get_response.assert_not_called() + mw._check_api_quota.assert_not_called() + mw._incr_with_ttl.assert_not_called() + assert response.status_code == 403 + assert response['X-RateLimit-Reason'] == 'ip_prefix_blocked' + + +def test_api_passes_through_when_ip_not_in_prefix_list(_seed_prefix_blocklist): + mw, _ = _make_middleware(api_threshold=999) + _seed_prefix_blocklist(['9.9.9']) + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock(return_value=1) + request = _api_req(ip='1.2.3.4') + + mw(request) + + mw.get_response.assert_called_once_with(request) + + +# --------------------------------------------------------------------------- +# _handle_api — rate limiting and block key isolation +# --------------------------------------------------------------------------- + +def test_api_external_request_increments_api_counter(): + mw, _ = _make_middleware(api_threshold=10) + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock(return_value=5) # under threshold + request = _api_req() + response = mw(request) + mw.get_response.assert_called_once_with(request) + call_args = mw._incr_with_ttl.call_args[0] + assert call_args[0] == RL_API_IP_REQUESTS.format(ns=_NAMESPACE, ip='1.2.3.4') + + +def test_api_threshold_exceeded_creates_api_block_not_global_block(): + mw, _ = _make_middleware(api_threshold=5) + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock(return_value=5) # at threshold + request = _api_req() + with patch('sapl.middleware.ratelimit._set_block') as mock_set_block: + response = mw(request) + assert response.status_code == 429 + mock_set_block.assert_called_once() + block_key = mock_set_block.call_args[0][0] + assert block_key == RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip='1.2.3.4') + assert block_key != RL_IP_BLOCKED.format(ip='1.2.3.4') + + +def test_api_global_block_also_blocks_api(): + mw, mock_cache = _make_middleware() + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + ip = '1.2.3.4' + mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None + response = mw(_api_req(ip=ip)) + assert response.status_code == 429 + assert response['X-RateLimit-Reason'] == 'global_ip_blocked' + mw._incr_with_ttl.assert_not_called() + + +def test_api_specific_block_blocks_api_only(): + mw, mock_cache = _make_middleware() + mw._check_api_quota = MagicMock(return_value=None) + mw._incr_with_ttl = MagicMock() + ip = '1.2.3.4' + mock_cache.get.side_effect = lambda key: 1 if key == RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip) else None + response = mw(_api_req(ip=ip)) + assert response.status_code == 429 + assert response['X-RateLimit-Reason'] == 'api_ip_blocked' + mw._incr_with_ttl.assert_not_called() + + +def test_api_block_response_is_json_with_retry_after(): + mw, _ = _make_middleware(api_block_seconds=120) + resp = mw._api_block_response('api_threshold_exceeded') + assert resp.status_code == 429 + assert 'application/json' in resp['Content-Type'] + assert resp['Retry-After'] == '120' + assert resp['X-RateLimit-Reason'] == 'api_threshold_exceeded' + + +def test_api_auth_user_daily_quota_exceeded_returns_429(): + """Auth users are subject to the same daily quota as anon callers (keyed by IP).""" + mw, _ = _make_middleware() + request = _api_req(ip='10.0.0.1') + request.user = MagicMock(is_authenticated=True, pk=42) + mw.api_quota_daily = 1 + + with patch('sapl.middleware.ratelimit._hincrby_with_ttl', return_value=2): + resp = mw(request) + + assert resp.status_code == 429 + assert resp['X-RateLimit-Reason'] == 'quota_daily' + + +def test_api_weekly_quota_exceeded_returns_429(): + """Weekly quota block fires when daily passes but weekly counter exceeds limit.""" + mw, _ = _make_middleware() + request = _api_req(ip='10.0.0.2') + mw.api_quota_daily = 999999 + mw.api_quota_weekly = 1 + + # daily returns 1 (under limit), weekly returns 2 (over limit) + with patch('sapl.middleware.ratelimit._hincrby_with_ttl', side_effect=[1, 2]): + resp = mw(request) + + assert resp.status_code == 429 + assert resp['X-RateLimit-Reason'] == 'quota_weekly' + + +def test_api_quota_uses_hash_keys(): + """_check_api_quota calls _hincrby_with_ttl with HASH keys (no IP in key name).""" + from datetime import date + mw, _ = _make_middleware() + request = _api_req(ip='10.0.0.3') + mw.api_quota_daily = 999999 + mw.api_quota_weekly = 999999 + + with patch('sapl.middleware.ratelimit._hincrby_with_ttl', return_value=1) as mock_h: + mw._check_api_quota(request) + + today = date.today() + iso = today.isocalendar() + expected_daily_hash = API_QUOTA_DAILY_HASH.format(ns=_NAMESPACE, date=today.isoformat()) + expected_weekly_hash = API_QUOTA_WEEKLY_HASH.format( + ns=_NAMESPACE, week=f'{iso[0]}-W{iso[1]:02d}' + ) + calls = mock_h.call_args_list + assert calls[0][0][0] == expected_daily_hash # first arg of first call = hash key + assert calls[1][0][0] == expected_weekly_hash # first arg of second call = hash key + # IP is the field (second positional arg), not embedded in the key + assert '10.0.0.3' not in calls[0][0][0] + assert '10.0.0.3' not in calls[1][0][0] + + +def test_inc_block_metric_uses_hash_key(): + """_inc_block_metric calls _hincrby_with_ttl with a HASH key; reason is the field.""" + from datetime import date + mw, _ = _make_middleware() + today_str = date.today().isoformat() + expected_key = f'rl:metrics:{_NAMESPACE}:{today_str}' + + with patch('sapl.middleware.ratelimit._hincrby_with_ttl') as mock_h: + mw._inc_block_metric('ip_rate') + + mock_h.assert_called_once() + args = mock_h.call_args[0] + assert args[0] == expected_key, f'hash key mismatch: {args[0]!r}' + assert args[1] == 'ip_rate', f'field (reason) mismatch: {args[1]!r}' + assert 'ip_rate' not in args[0], 'reason must be HASH field, not embedded in key' + + +def test_non_api_path_uses_global_evaluate_not_api_handler(): + mw, _ = _make_middleware() + mw._handle_api = MagicMock() + mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) + mw(_anon_req(path='/')) + mw._handle_api.assert_not_called() + mw._evaluate.assert_called_once() diff --git a/sapl/norma/views.py b/sapl/norma/views.py index 38f4eb1b0..b0f5a3e40 100644 --- a/sapl/norma/views.py +++ b/sapl/norma/views.py @@ -21,6 +21,7 @@ import weasyprint from ratelimit.decorators import ratelimit from django.utils.decorators import method_decorator +from django.views.decorators.http import condition from sapl import settings import sapl @@ -30,15 +31,17 @@ from sapl.compilacao.views import IntegracaoTaView from sapl.crud.base import (RP_DETAIL, RP_LIST, Crud, CrudAux, MasterDetailCrud, make_pagination) from sapl.materia.models import Orgao -from sapl.utils import show_results_filter_set, get_client_ip, \ - sapn_is_enabled, MultiFormatOutputMixin, ratelimit_ip +from sapl.middleware.ratelimit import get_client_ip, smart_key, smart_rate +from sapl.middleware.page_cache import AnonCachePageMixin +from sapl.utils import show_results_filter_set, \ + sapn_is_enabled, MultiFormatOutputMixin from .forms import (AnexoNormaJuridicaForm, NormaFilterSet, NormaJuridicaForm, NormaPesquisaSimplesForm, NormaRelacionadaForm, AutoriaNormaForm, AssuntoNormaFilterSet) from .models import (AnexoNormaJuridica, AssuntoNorma, NormaJuridica, NormaRelacionada, TipoNormaJuridica, TipoVinculoNormaJuridica, AutoriaNorma, NormaEstatisticas) -from ..settings import RATE_LIMITER_RATE + # LegislacaoCitadaCrud = Crud.build(LegislacaoCitada, '') TipoNormaCrud = CrudAux.build( @@ -60,11 +63,11 @@ class AssuntoNormaCrud(CrudAux): return reverse('sapl.norma:pesquisar_assuntonorma') -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class PesquisarAssuntoNormaView(FilterView): +class PesquisarAssuntoNormaView(AnonCachePageMixin, FilterView): model = AssuntoNorma filterset_class = AssuntoNormaFilterSet paginate_by = 20 @@ -102,12 +105,9 @@ class PesquisarAssuntoNormaView(FilterView): data = self.filterset.data - url = '' - - if data: - url = '&' + str(self.request.META["QUERY_STRING"]) - if url.startswith("&page"): - url = '' + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' if 'assunto' in self.request.META['QUERY_STRING'] or\ 'page' in self.request.META['QUERY_STRING']: @@ -154,11 +154,11 @@ class NormaRelacionadaCrud(MasterDetailCrud): layout_key = 'NormaRelacionadaDetail' -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class NormaPesquisaView(MultiFormatOutputMixin, FilterView): +class NormaPesquisaView(AnonCachePageMixin, MultiFormatOutputMixin, FilterView): model = NormaJuridica filterset_class = NormaFilterSet paginate_by = 50 @@ -276,6 +276,17 @@ class NormaTaView(IntegracaoTaView): return self.get_redirect_deactivated() +def _norma_last_modified(request, *args, **kwargs): + return NormaJuridica.objects.filter( + pk=kwargs['pk'] + ).values_list('ultima_edicao', flat=True).first() + + +def _norma_etag(request, *args, **kwargs): + ts = _norma_last_modified(request, *args, **kwargs) + return f'{kwargs["pk"]}-{ts.timestamp()}' if ts else None + + class NormaCrud(Crud): model = NormaJuridica help_topic = 'norma_juridica' @@ -291,6 +302,7 @@ class NormaCrud(Crud): namespace = self.model._meta.app_config.name return reverse('%s:%s' % (namespace, 'norma_pesquisa')) + @method_decorator(condition(etag_func=_norma_etag, last_modified_func=_norma_last_modified), name='get') class DetailView(Crud.DetailView): def get(self, request, *args, **kwargs): estatisticas_acesso_normas = AppConfig.objects.first().estatisticas_acesso_normas diff --git a/sapl/painel/urls.py b/sapl/painel/urls.py index 0795d0a35..b645c7bbe 100644 --- a/sapl/painel/urls.py +++ b/sapl/painel/urls.py @@ -1,8 +1,7 @@ from django.conf.urls import url from .apps import AppConfig -from .views import (cronometro_painel, get_dados_painel, painel_mensagem_view, - painel_parlamentar_view, painel_view, painel_votacao_view, +from .views import (cronometro_painel, get_dados_painel, painel_view, switch_painel, verifica_painel, votante_view) app_name = AppConfig.name @@ -11,12 +10,8 @@ urlpatterns = [ url(r'^painel-principal/(?P\d+)$', painel_view, name="painel_principal"), url(r'^painel/(?P\d+)/dados$', get_dados_painel, name='dados_painel'), - url(r'^painel/mensagem$', painel_mensagem_view, name="painel_mensagem"), - url(r'^painel/parlamentar$', painel_parlamentar_view, - name='painel_parlamentar'), url(r'^painel/switch-painel$', switch_painel, name="switch_painel"), - url(r'^painel/votacao$', painel_votacao_view, name='painel_votacao'), url(r'^painel/verifica-painel$', verifica_painel, name="verifica_painel"), url(r'^painel/cronometro$', cronometro_painel, name='cronometro_painel'), diff --git a/sapl/painel/views.py b/sapl/painel/views.py index bfe9df2c1..351328b18 100644 --- a/sapl/painel/views.py +++ b/sapl/painel/views.py @@ -24,7 +24,8 @@ from sapl.sessao.models import (ExpedienteMateria, OradorExpediente, OrdemDia, PresencaOrdemDia, RegistroVotacao, SessaoPlenaria, SessaoPlenariaPresenca, VotoParlamentar, RegistroLeitura) -from sapl.utils import filiacao_data, get_client_ip, sort_lista_chave +from sapl.middleware.ratelimit import get_client_ip +from sapl.utils import filiacao_data, sort_lista_chave from .models import Cronometro @@ -327,20 +328,6 @@ def verifica_painel(request): return resposta -@user_passes_test(check_permission) -def painel_mensagem_view(request): - return render(request, 'painel/mensagem.html') - - -@user_passes_test(check_permission) -def painel_parlamentar_view(request): - return render(request, 'painel/parlamentares.html') - - -@user_passes_test(check_permission) -def painel_votacao_view(request): - return render(request, 'painel/votacao.html') - @user_passes_test(check_permission) def cronometro_painel(request): diff --git a/sapl/parlamentares/views.py b/sapl/parlamentares/views.py index a46ce2b84..075cccc0e 100644 --- a/sapl/parlamentares/views.py +++ b/sapl/parlamentares/views.py @@ -33,7 +33,9 @@ from sapl.materia.models import Autoria, Proposicao, Relatoria from sapl.norma.models import AutoriaNorma, NormaJuridica from sapl.parlamentares.apps import AppConfig from sapl.rules import SAPL_GROUP_VOTANTE -from sapl.utils import (parlamentares_ativos, show_results_filter_set, ratelimit_ip) +from sapl.middleware.page_cache import AnonCachePageMixin +from sapl.middleware.ratelimit import smart_key, smart_rate +from sapl.utils import (parlamentares_ativos, show_results_filter_set) from .forms import (ColigacaoFilterSet, FiliacaoForm, FrenteForm, LegislaturaForm, MandatoForm, ParlamentarCreateForm, ParlamentarForm, VotanteForm, @@ -48,7 +50,7 @@ from .models import (CargoMesa, Coligacao, ComposicaoColigacao, ComposicaoMesa, from ratelimit.decorators import ratelimit from django.utils.decorators import method_decorator -from ..settings import RATE_LIMITER_RATE + FrenteCargoCrud = CrudAux.build(FrenteCargo, 'frente_cargo') BlocoCargoCrud = CrudAux.build(BlocoCargo, 'bloco_cargo') @@ -188,11 +190,11 @@ class ProposicaoParlamentarCrud(CrudBaseForListAndDetailExternalAppView): _('Texto Eletrônico')) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class PesquisarParlamentarView(FilterView): +class PesquisarParlamentarView(AnonCachePageMixin, FilterView): model = Parlamentar filterset_class = ParlamentarFilterSet paginate_by = 20 @@ -230,11 +232,9 @@ class PesquisarParlamentarView(FilterView): super(PesquisarParlamentarView, self).get(request) data = self.filterset.data - url = '' - if data: - url = "&" + str(self.request.META['QUERY_STRING']) - if url.startswith("&page"): - url = '' + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' if 'nome_parlamentar' in self.request.META['QUERY_STRING'] or\ 'page' in self.request.META['QUERY_STRING']: @@ -254,11 +254,11 @@ class PesquisarParlamentarView(FilterView): return self.render_to_response(context) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class PesquisarColigacaoView(FilterView): +class PesquisarColigacaoView(AnonCachePageMixin, FilterView): model = Coligacao filterset_class = ColigacaoFilterSet paginate_by = 20 @@ -290,11 +290,9 @@ class PesquisarColigacaoView(FilterView): super(PesquisarColigacaoView, self).get(request) data = self.filterset.data - url = '' - if data: - url = "&" + str(self.request.META['QUERY_STRING']) - if url.startswith("&page"): - url = '' + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' if 'nome' in self.request.META['QUERY_STRING'] or\ 'page' in self.request.META['QUERY_STRING']: @@ -314,11 +312,11 @@ class PesquisarColigacaoView(FilterView): return self.render_to_response(context) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class PesquisarPartidoView(FilterView): +class PesquisarPartidoView(AnonCachePageMixin, FilterView): model = Partido filterset_class = PartidoFilterSet paginate_by = 20 @@ -349,11 +347,9 @@ class PesquisarPartidoView(FilterView): super(PesquisarPartidoView, self).get(request) data = self.filterset.data - url = '' - if data: - url = "&" + str(self.request.META['QUERY_STRING']) - if url.startswith("&page"): - url = '' + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' if 'nome' in self.request.META['QUERY_STRING'] or\ 'page' in self.request.META['QUERY_STRING']: @@ -749,7 +745,9 @@ class ParlamentarCrud(Crud): 'filiacao_atual', 'ativo'] - class DetailView(Crud.DetailView): + class DetailView(AnonCachePageMixin, Crud.DetailView): + # Parlamentar profiles change only at term boundaries — 10-minute cache. + anon_cache_ttl = 600 # PAGE_CACHE_TTL_STABLE def get_template_names(self): if self.request.user.has_perm(self.permission(RP_CHANGE)): @@ -788,10 +786,12 @@ class ParlamentarCrud(Crud): """ return super(Crud.CreateView, self).form_valid(form) - class ListView(Crud.ListView): + class ListView(AnonCachePageMixin, Crud.ListView): template_name = "parlamentares/parlamentares_list.html" paginate_by = None logger = logging.getLogger(__name__) + # Full list changes only when a mandato starts/ends — 10-minute cache. + anon_cache_ttl = 600 # PAGE_CACHE_TTL_STABLE @xframe_options_exempt def get(self, request, *args, **kwargs): diff --git a/sapl/protocoloadm/views.py b/sapl/protocoloadm/views.py index 345382438..0cd724efb 100755 --- a/sapl/protocoloadm/views.py +++ b/sapl/protocoloadm/views.py @@ -44,10 +44,11 @@ from sapl.protocoloadm.forms import VinculoDocAdminMateriaForm,\ from sapl.protocoloadm.models import Protocolo, DocumentoAdministrativo,\ VinculoDocAdminMateria from sapl.relatorios.views import relatorio_doc_administrativos -from sapl.utils import (create_barcode, get_base_url, get_client_ip, - get_mime_type_from_file_extension, lista_anexados, +from sapl.middleware.ratelimit import get_client_ip, smart_key, smart_rate +from sapl.utils import (create_barcode, get_base_url, + lista_anexados, show_results_filter_set, mail_service_configured, from_date_to_datetime_utc, - google_recaptcha_configured, get_tempfile_dir, MultiFormatOutputMixin, ratelimit_ip) + google_recaptcha_configured, get_tempfile_dir, MultiFormatOutputMixin) from .forms import (AcompanhamentoDocumentoForm, AnexadoEmLoteFilterSet, AnexadoForm, AnularProtocoloAdmForm, compara_tramitacoes_doc, @@ -62,7 +63,7 @@ from .forms import (AcompanhamentoDocumentoForm, AnexadoEmLoteFilterSet, Anexado from .models import (Anexado, AcompanhamentoDocumento, DocumentoAcessorioAdministrativo, DocumentoAdministrativo, StatusTramitacaoAdministrativo, TipoDocumentoAdministrativo, TramitacaoAdministrativo) -from ..settings import MEDIA_ROOT, RATE_LIMITER_RATE +from ..settings import MEDIA_ROOT from ratelimit.decorators import ratelimit from django.utils.decorators import method_decorator @@ -101,28 +102,10 @@ def recuperar_materia_protocolo(request): def doc_texto_integral(request, pk): - can_see = True - - if not request.user.is_authenticated: - app_config = AppConfig.objects.last() - if app_config and app_config.documentos_administrativos == 'R': - can_see = False - - if can_see: - documento = DocumentoAdministrativo.objects.get(pk=pk) - if documento.texto_integral: - arquivo = documento.texto_integral - - mime = get_mime_type_from_file_extension(arquivo.name) - - with open(arquivo.path, 'rb') as f: - data = f.read() - - response = HttpResponse(data, content_type='%s' % mime) - response['Content-Disposition'] = ( - 'inline; filename="%s"' % arquivo.name.split('/')[-1]) - return response - raise Http404 + documento = get_object_or_404(DocumentoAdministrativo, pk=pk) + if not documento.texto_integral: + raise Http404 + return redirect(documento.texto_integral.url) def get_pdf_docacessorios(request, pk): @@ -538,8 +521,8 @@ class StatusTramitacaoAdministrativoCrud(CrudAux): ordering = 'sigla' -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class ProtocoloPesquisaView(PermissionRequiredMixin, FilterView): @@ -587,10 +570,9 @@ class ProtocoloPesquisaView(PermissionRequiredMixin, FilterView): # Então a ordem da URL está diferente data = self.filterset.data if data and data.get('numero') is not None: - url = "&" + str(self.request.environ['QUERY_STRING']) - if url.startswith("&page"): - ponto_comeco = url.find('numero=') - 1 - url = url[ponto_comeco:] + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' else: url = '' @@ -1039,8 +1021,8 @@ class ProtocoloMateriaTemplateView(PermissionRequiredMixin, TemplateView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class PesquisarDocumentoAdministrativoView(DocumentoAdministrativoMixin, @@ -1117,10 +1099,9 @@ class PesquisarDocumentoAdministrativoView(DocumentoAdministrativoMixin, # Então a ordem da URL está diferente data = self.filterset.data if data and data.get('tipo') is not None: - url = "&" + str(self.request.environ['QUERY_STRING']) - if url.startswith("&page"): - ponto_comeco = url.find('tipo=') - 1 - url = url[ponto_comeco:] + qr = self.request.GET.copy() + qr.pop('page', None) + url = ('&' + qr.urlencode()) if qr else '' else: url = '' self.filterset.form.fields['o'].label = _('Ordenação') @@ -1176,8 +1157,8 @@ class AnexadoCrud(MasterDetailCrud): return 'AnexadoDetail' -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class DocumentoAnexadoEmLoteView(PermissionRequiredMixin, FilterView): @@ -1657,8 +1638,8 @@ class FichaSelecionaAdmView(PermissionRequiredMixin, FormView): 'materia/impressos/ficha_adm_pdf.html') -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class PrimeiraTramitacaoEmLoteAdmView(PermissionRequiredMixin, FilterView): @@ -1897,8 +1878,8 @@ class VinculoDocAdminMateriaCrud(MasterDetailCrud): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class VinculoDocAdminMateriaEmLoteView(PermissionRequiredMixin, FilterView): diff --git a/sapl/relatorios/forms.py b/sapl/relatorios/forms.py index f659c254c..726501524 100644 --- a/sapl/relatorios/forms.py +++ b/sapl/relatorios/forms.py @@ -543,9 +543,7 @@ class RelatorioMateriasTramitacaoFilterSet(django_filters.FilterSet): @property def qs(self): parent = super(RelatorioMateriasTramitacaoFilterSet, self).qs - return parent.distinct().order_by( - '-materia__ano', 'materia__tipo', '-materia__numero' - ) + return parent.order_by('-materia__ano', 'materia__tipo', '-materia__numero') class Meta: model = MateriaEmTramitacao diff --git a/sapl/relatorios/views.py b/sapl/relatorios/views.py index 60ba1635e..9ba13536f 100755 --- a/sapl/relatorios/views.py +++ b/sapl/relatorios/views.py @@ -48,10 +48,10 @@ from sapl.sessao.views import (get_identificacao_basica, get_mesa_diretora, get_oradores_explicacoes_pessoais, get_consideracoes_finais, get_ocorrencias_da_sessao, get_assinaturas, get_correspondencias) -from sapl.settings import MEDIA_URL, RATE_LIMITER_RATE +from sapl.settings import MEDIA_URL from sapl.settings import STATIC_ROOT from sapl.utils import LISTA_DE_UFS, TrocaTag, filiacao_data, create_barcode, show_results_filter_set, \ - num_materias_por_tipo, parlamentares_ativos, MultiFormatOutputMixin, ratelimit_ip + num_materias_por_tipo, parlamentares_ativos, MultiFormatOutputMixin, is_report_allowed from .templates import (pdf_capa_processo_gerar, pdf_documento_administrativo_gerar, pdf_espelho_gerar, pdf_etiqueta_protocolo_gerar, pdf_materia_gerar, @@ -59,8 +59,6 @@ from .templates import (pdf_capa_processo_gerar, pdf_protocolo_gerar, pdf_sessao_plenaria_gerar) from sapl.crud.base import make_pagination -from ratelimit.decorators import ratelimit -from django.utils.decorators import method_decorator def get_kwargs_params(request, fields): @@ -1141,8 +1139,29 @@ def relatorio_etiqueta_protocolo(request, nro, ano): def get_etiqueta_protocolos(prots): + prot_list = list(prots) + if not prot_list: + return [] + + # Pre-fetch MateriaLegislativa for all protocols in one query. + materia_query = Q() + for p in prot_list: + materia_query |= Q(numero_protocolo=p.numero, ano=p.ano) + materias_map = { + (m.numero_protocolo, m.ano): m + for m in MateriaLegislativa.objects.filter( + materia_query).select_related('tipo') + } + + # Pre-fetch DocumentoAdministrativo for all protocols in one query. + documentos_map = { + doc.protocolo_id: doc + for doc in DocumentoAdministrativo.objects.filter( + protocolo__in=prot_list).select_related('tipo') + } + protocolos = [] - for p in prots: + for p in prot_list: dic = {} dic['titulo'] = str(p.numero) + '/' + str(p.ano) @@ -1159,11 +1178,11 @@ def get_etiqueta_protocolos(prots): dic['nom_autor'] = str(p.autor or ' ') - dic['num_materia'] = '' - for materia in MateriaLegislativa.objects.filter( - numero_protocolo=p.numero, ano=p.ano): - dic['num_materia'] = materia.tipo.sigla + ' ' + \ - str(materia.numero) + '/' + str(materia.ano) + materia = materias_map.get((p.numero, p.ano)) + dic['num_materia'] = ( + materia.tipo.sigla + ' ' + str(materia.numero) + '/' + str(materia.ano) + if materia else '' + ) dic['natureza'] = '' if p.tipo_processo == 0: @@ -1171,11 +1190,11 @@ def get_etiqueta_protocolos(prots): if p.tipo_processo == 1: dic['natureza'] = 'Legislativo' - dic['num_documento'] = '' - for documento in DocumentoAdministrativo.objects.filter( - protocolo=p): - dic['num_documento'] = documento.tipo.sigla + ' ' + \ - str(documento.numero) + '/' + str(documento.ano) + documento = documentos_map.get(p.pk) + dic['num_documento'] = ( + documento.tipo.sigla + ' ' + str(documento.numero) + '/' + str(documento.ano) + if documento else '' + ) dic['ident_processo'] = dic['num_materia'] or dic['num_documento'] @@ -1828,26 +1847,18 @@ class RelatoriosListView(TemplateView): class RelatorioMixin: # TODO: verificar se todos os relatorios de sistema/relatorios extendem esse Mixin def get(self, request, *args, **kwargs): - super(RelatorioMixin, self).get(request) - - # TODO: import as global - from sapl.utils import is_report_allowed if not is_report_allowed(request): raise Http404() - is_relatorio = request.GET.get('relatorio') - context = self.get_context_data(filter=self.filterset) - - if is_relatorio: + if request.GET.get('relatorio'): + filterset_class = self.get_filterset_class() + self.filterset = self.get_filterset(filterset_class) + context = self.get_context_data(filter=self.filterset) return self.relatorio(request, context) - else: - return self.render_to_response(context) + + return super(RelatorioMixin, self).get(request, *args, **kwargs) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioDocumentosAcessoriosView(RelatorioMixin, FilterView): model = DocumentoAcessorio filterset_class = RelatorioDocumentosAcessoriosFilterSet @@ -1892,10 +1903,6 @@ class RelatorioDocumentosAcessoriosView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioVotacoesNominaisView(RelatorioMixin, MultiFormatOutputMixin, FilterView): model = VotoParlamentar filterset_class = RelatorioVotacoesNominaisFilterSet @@ -1965,10 +1972,6 @@ class RelatorioVotacoesNominaisView(RelatorioMixin, MultiFormatOutputMixin, Filt return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioAtasView(RelatorioMixin, FilterView): model = SessaoPlenaria filterset_class = RelatorioAtasFilterSet @@ -1994,10 +1997,6 @@ class RelatorioAtasView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioPresencaSessaoView(RelatorioMixin, FilterView): logger = logging.getLogger(__name__) model = SessaoPlenaria @@ -2232,10 +2231,6 @@ class RelatorioPresencaSessaoView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioHistoricoTramitacaoView(RelatorioMixin, FilterView): model = MateriaLegislativa filterset_class = RelatorioHistoricoTramitacaoFilterSet @@ -2293,10 +2288,6 @@ class RelatorioHistoricoTramitacaoView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioDataFimPrazoTramitacaoView(RelatorioMixin, FilterView): model = MateriaEmTramitacao filterset_class = RelatorioDataFimPrazoTramitacaoFilterSet @@ -2360,10 +2351,6 @@ class RelatorioDataFimPrazoTramitacaoView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioReuniaoView(RelatorioMixin, FilterView): model = Reuniao filterset_class = RelatorioReuniaoFilterSet @@ -2398,10 +2385,6 @@ class RelatorioReuniaoView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioAudienciaView(RelatorioMixin, FilterView): model = AudienciaPublica filterset_class = RelatorioAudienciaFilterSet @@ -2436,10 +2419,6 @@ class RelatorioAudienciaView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioMateriasTramitacaoView(RelatorioMixin, FilterView): model = MateriaEmTramitacao filterset_class = RelatorioMateriasTramitacaoFilterSet @@ -2554,10 +2533,6 @@ class RelatorioMateriasTramitacaoView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioMateriasPorAnoAutorTipoView(RelatorioMixin, FilterView): model = MateriaLegislativa filterset_class = RelatorioMateriasPorAnoAutorTipoFilterSet @@ -2637,10 +2612,6 @@ class RelatorioMateriasPorAnoAutorTipoView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioMateriasPorAutorView(RelatorioMixin, FilterView): model = MateriaLegislativa filterset_class = RelatorioMateriasPorAutorFilterSet @@ -2711,10 +2682,6 @@ class RelatorioMateriaAnoAssuntoView(ListView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioNormasPublicadasMesView(RelatorioMixin, FilterView): model = NormaJuridica filterset_class = RelatorioNormasMesFilterSet @@ -2755,10 +2722,6 @@ class RelatorioNormasPublicadasMesView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioNormasVigenciaView(RelatorioMixin, FilterView): model = NormaJuridica filterset_class = RelatorioNormasVigenciaFilterSet @@ -2823,10 +2786,6 @@ class RelatorioNormasVigenciaView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioHistoricoTramitacaoAdmView(RelatorioMixin, FilterView): model = DocumentoAdministrativo filterset_class = RelatorioHistoricoTramitacaoAdmFilterSet @@ -2877,10 +2836,6 @@ class RelatorioHistoricoTramitacaoAdmView(RelatorioMixin, FilterView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, - block=True), - name='dispatch') class RelatorioNormasPorAutorView(RelatorioMixin, FilterView): model = NormaJuridica filterset_class = RelatorioNormasPorAutorFilterSet diff --git a/sapl/sessao/views.py b/sapl/sessao/views.py index ab30c2c6a..52971691a 100755 --- a/sapl/sessao/views.py +++ b/sapl/sessao/views.py @@ -9,9 +9,9 @@ from django.contrib import messages from django.contrib.auth.decorators import permission_required from django.contrib.auth.mixins import PermissionRequiredMixin from django.core.exceptions import ObjectDoesNotExist -from django.db.models import Max, Q +from django.db.models import Max, Prefetch, Q from django.http import JsonResponse -from django.http.response import Http404, HttpResponseRedirect +from django.http.response import Http404, HttpResponseBadRequest, HttpResponseRedirect from django.urls import reverse from django.urls.base import reverse_lazy from django.utils import timezone @@ -47,9 +47,11 @@ from sapl.sessao.apps import AppConfig from sapl.sessao.forms import ExpedienteMateriaForm, OrdemDiaForm, OrdemExpedienteLeituraForm, \ CorrespondenciaForm, CorrespondenciaEmLoteFilterSet from sapl.sessao.models import Correspondencia -from sapl.settings import TIME_ZONE, RATE_LIMITER_RATE -from sapl.utils import show_results_filter_set, remover_acentos, get_client_ip, \ - MultiFormatOutputMixin, PautaMultiFormatOutputMixin, ratelimit_ip +from sapl.settings import TIME_ZONE +from sapl.middleware.ratelimit import get_client_ip, smart_key, smart_rate +from sapl.middleware.page_cache import AnonCachePageMixin +from sapl.utils import show_results_filter_set, remover_acentos, \ + MultiFormatOutputMixin, PautaMultiFormatOutputMixin from .forms import (AdicionarVariasMateriasFilterSet, AdicionarVariasMateriasForm, BancadaForm, ExpedienteForm, JustificativaAusenciaForm, OcorrenciaSessaoForm, ListMateriaForm, @@ -172,7 +174,7 @@ def verifica_sessao_iniciada(request, spk, is_leitura=False): aux_text = 'leitura' if is_leitura else 'votação' logger.info('user=' + username + '. Não é possível abrir matérias para {}. ' 'Esta SessaoPlenaria (id={}) não foi iniciada ou está finalizada.'.format( - aux_text, spk)) + aux_text, spk)) msg = _('Não é possível abrir matérias para {}. ' 'Esta Sessão Plenária não foi iniciada ou está finalizada.' ' Vá em "Abertura"->"Dados Básicos" e altere os valores dos campos necessários.'.format(aux_text)) @@ -224,38 +226,43 @@ def abrir_votacao(request, pk, spk): def customize_link_materia(context, pk, has_permission, is_expediente): + # sessao_plenaria is the same for every row — resolve once + object_list = context['object_list'] + if object_list: + sessao_plenaria = object_list[0].sessao_plenaria + else: + sessao_plenaria = SessaoPlenaria.objects.get(id=pk) + data_sessao = sessao_plenaria.data_fim or sessao_plenaria.data_inicio + for i, row in enumerate(context['rows']): - materia = context['object_list'][i].materia - obj = context['object_list'][i] + obj = object_list[i] + materia = obj.materia # already select_related + url_materia = reverse( 'sapl.materia:materialegislativa_detail', kwargs={'pk': materia.id}) - numeracao = materia.numeracao_set.first() if materia.numeracao_set.first() else "-" - todos_autoria = materia.autoria_set.all() - autoria = todos_autoria.filter(primeiro_autor=True) + + numeracao = materia._numeracao_prefetch[0] if materia._numeracao_prefetch else "-" + + todos_autoria = materia._autoria_prefetch + autoria = [a for a in todos_autoria if a.primeiro_autor] autor = ', '.join([str(a.autor) for a in autoria]) if autoria else "-" + todos_autores = ', '.join([str(a.autor) for a in todos_autoria]) if autoria else "-" - todos_autores = ', '.join([str(a.autor) - for a in todos_autoria]) if autoria else "-" + num_protocolo = materia.numero_protocolo or "-" - num_protocolo = materia.numero_protocolo if materia.numero_protocolo else "-" - sessao_plenaria = SessaoPlenaria.objects.get(id=pk) - data_sessao = sessao_plenaria.data_fim if sessao_plenaria.data_fim else sessao_plenaria.data_inicio - tramitacao = Tramitacao.objects \ - .select_related('materia', 'status', 'materia__tipo') \ - .filter(materia=materia, turno__isnull=False, data_tramitacao__lte=data_sessao) \ - .exclude(turno__exact='') \ - .order_by('-data_tramitacao', '-id') \ - .first() + tramitacao = next( + (t for t in materia._tramitacao_prefetch if t.data_tramitacao <= data_sessao), + None, + ) turno = '-' if tramitacao: for t in Tramitacao.TURNO_CHOICES: if t[0] == tramitacao.turno: turno = t[1] break - materia_em_tramitacao = MateriaEmTramitacao.objects \ - .select_related("materia", "tramitacao") \ - .filter(materia=materia) \ - .first() + + materia_em_tramitacao = materia._met_prefetch[0] if materia._met_prefetch else None + # idUnica para cada materia idAutor = "autor" + str(i) idAutores = "autores" + str(i) @@ -281,12 +288,9 @@ def customize_link_materia(context, pk, has_permission, is_expediente): # url em toda a string de title_materia context['rows'][i][1] = (title_materia, None) - exist_resultado = obj.registrovotacao_set.filter( - materia=obj.materia).exists() - exist_retirada = obj.retiradapauta_set.filter( - materia=obj.materia).exists() - exist_leitura = obj.registroleitura_set.filter( - materia=obj.materia).exists() + exist_resultado = bool(obj._votacao_prefetch) + exist_retirada = bool(obj._retirada_prefetch) + exist_leitura = bool(obj._leitura_prefetch) if (obj.tipo_votacao != LEITURA and not exist_resultado and not exist_retirada) or \ (obj.tipo_votacao == LEITURA and not exist_leitura): @@ -408,8 +412,7 @@ def customize_link_materia(context, pk, has_permission, is_expediente): resultado = '''Não há resultado''' elif exist_retirada: - retirada = obj.retiradapauta_set.filter( - materia_id=obj.materia_id).last() + retirada = obj._retirada_prefetch[-1] retirada_descricao = retirada.tipo_de_retirada.descricao retirada_observacao = retirada.observacao url = reverse('sapl.sessao:retiradapauta_detail', @@ -421,13 +424,11 @@ def customize_link_materia(context, pk, has_permission, is_expediente): else: if obj.tipo_votacao == LEITURA: - resultado = obj.registroleitura_set.filter( - materia_id=obj.materia_id).last() + resultado = obj._leitura_prefetch[-1] resultado_descricao = "Matéria lida" resultado_observacao = resultado.observacao else: - resultado = obj.registrovotacao_set.filter( - materia_id=obj.materia_id).last() + resultado = obj._votacao_prefetch[-1] resultado_descricao = resultado.tipo_resultado_votacao.nome resultado_observacao = resultado.observacao @@ -486,11 +487,11 @@ def customize_link_materia(context, pk, has_permission, is_expediente): 'mid': obj.materia_id}) resultado = ( - '%s

%s' % ( - url, - context.get('page', 1), - resultado_descricao, - resultado_observacao)) + '%s

%s' % ( + url, + context.get('page', 1), + resultado_descricao, + resultado_observacao)) else: if obj.tipo_votacao == NOMINAL: @@ -501,7 +502,7 @@ def customize_link_materia(context, pk, has_permission, is_expediente): 'pk': obj.sessao_plenaria_id, 'oid': obj.pk, 'mid': obj.materia_id}) + \ - '?&materia=expediente' + '?&materia=expediente' else: url = reverse( 'sapl.sessao:votacao_nominal_transparencia', @@ -509,7 +510,7 @@ def customize_link_materia(context, pk, has_permission, is_expediente): 'pk': obj.sessao_plenaria_id, 'oid': obj.pk, 'mid': obj.materia_id}) + \ - '?&materia=ordem' + '?&materia=ordem' resultado = ('%s
%s' % (url, @@ -524,7 +525,7 @@ def customize_link_materia(context, pk, has_permission, is_expediente): 'pk': obj.sessao_plenaria_id, 'oid': obj.pk, 'mid': obj.materia_id}) + \ - '?&materia=expediente' + '?&materia=expediente' else: url = reverse( 'sapl.sessao:votacao_simbolica_transparencia', @@ -532,7 +533,7 @@ def customize_link_materia(context, pk, has_permission, is_expediente): 'pk': obj.sessao_plenaria_id, 'oid': obj.pk, 'mid': obj.materia_id}) + \ - '?&materia=ordem' + '?&materia=ordem' resultado = ('%s
%s' % (url, @@ -796,7 +797,7 @@ class MateriaOrdemDiaCrud(MasterDetailCrud): sessao_plenaria=self.kwargs['pk']).aggregate( Max('numero_ordem'))['numero_ordem__max'] self.initial['numero_ordem'] = ( - max_numero_ordem if max_numero_ordem else 0) + 1 + max_numero_ordem if max_numero_ordem else 0) + 1 return self.initial def get_success_url(self): @@ -833,20 +834,58 @@ class MateriaOrdemDiaCrud(MasterDetailCrud): layout_key = 'OrdemDiaDetail' class ListView(MasterDetailCrud.ListView): - paginate_by = None + paginate_by = 100 ordering = ['numero_ordem', 'materia', 'resultado'] def get_context_data(self, **kwargs): - if self.get_queryset().count() > 500: - self.paginate_by = 50 - else: - self.paginate_by = None - context = super().get_context_data(**kwargs) - has_permition = self.request.user.has_module_perms(AppConfig.label) return customize_link_materia(context, self.kwargs['pk'], has_permition, False) + def get_queryset(self): + return super().get_queryset().select_related( + 'materia', 'materia__tipo', 'sessao_plenaria', + ).prefetch_related( + Prefetch( + 'materia__materiaemtramitacao_set', + to_attr='_met_prefetch', + ), + Prefetch( + 'materia__numeracao_set', + to_attr='_numeracao_prefetch', + ), + Prefetch( + 'materia__autoria_set', + queryset=Autoria.objects.select_related('autor'), + to_attr='_autoria_prefetch', + ), + Prefetch( + 'materia__tramitacao_set', + queryset=Tramitacao.objects.filter( + turno__isnull=False, + ).exclude(turno='').order_by('-data_tramitacao', '-id'), + to_attr='_tramitacao_prefetch', + ), + Prefetch( + 'registrovotacao_set', + queryset=RegistroVotacao.objects.select_related( + 'tipo_resultado_votacao', + ), + to_attr='_votacao_prefetch', + ), + Prefetch( + 'retiradapauta_set', + queryset=RetiradaPauta.objects.select_related( + 'tipo_de_retirada', + ), + to_attr='_retirada_prefetch', + ), + Prefetch( + 'registroleitura_set', + to_attr='_leitura_prefetch', + ), + ) + def recuperar_materia(request): tipo = TipoMateriaLegislativa.objects.get(pk=request.GET['tipo_materia']) @@ -905,24 +944,60 @@ class ExpedienteMateriaCrud(MasterDetailCrud): 'resultado'] class ListView(MasterDetailCrud.ListView): - paginate_by = None + paginate_by = 100 ordering = ['numero_ordem', 'materia', 'resultado'] def get_context_data(self, **kwargs): - - if self.get_queryset().count() > 500: - self.paginate_by = 50 - else: - self.paginate_by = None - context = super().get_context_data(**kwargs) - if self.request.GET.get('page'): context['page'] = self.request.GET.get('page') - has_permition = self.request.user.has_module_perms(AppConfig.label) return customize_link_materia(context, self.kwargs['pk'], has_permition, True) + def get_queryset(self): + return super().get_queryset().select_related( + 'materia', 'materia__tipo', 'sessao_plenaria', + ).prefetch_related( + Prefetch( + 'materia__materiaemtramitacao_set', + to_attr='_met_prefetch', + ), + Prefetch( + 'materia__numeracao_set', + to_attr='_numeracao_prefetch', + ), + Prefetch( + 'materia__autoria_set', + queryset=Autoria.objects.select_related('autor'), + to_attr='_autoria_prefetch', + ), + Prefetch( + 'materia__tramitacao_set', + queryset=Tramitacao.objects.filter( + turno__isnull=False, + ).exclude(turno='').order_by('-data_tramitacao', '-id'), + to_attr='_tramitacao_prefetch', + ), + Prefetch( + 'registrovotacao_set', + queryset=RegistroVotacao.objects.select_related( + 'tipo_resultado_votacao', + ), + to_attr='_votacao_prefetch', + ), + Prefetch( + 'retiradapauta_set', + queryset=RetiradaPauta.objects.select_related( + 'tipo_de_retirada', + ), + to_attr='_retirada_prefetch', + ), + Prefetch( + 'registroleitura_set', + to_attr='_leitura_prefetch', + ), + ) + class CreateView(MasterDetailCrud.CreateView): form_class = ExpedienteMateriaForm @@ -939,7 +1014,7 @@ class ExpedienteMateriaCrud(MasterDetailCrud): sessao_plenaria=self.kwargs['pk']).aggregate( Max('numero_ordem'))['numero_ordem__max'] initial['numero_ordem'] = ( - max_numero_ordem if max_numero_ordem else 0) + 1 + max_numero_ordem if max_numero_ordem else 0) + 1 return initial def get_success_url(self): @@ -973,7 +1048,6 @@ class ExpedienteMateriaCrud(MasterDetailCrud): return initial class DetailView(MasterDetailCrud.DetailView): - layout_key = 'ExpedienteMateriaDetail' @@ -1346,7 +1420,12 @@ class SessaoCrud(Crud): {'subnav_template_name': 'sessao/subnav-solene.yaml'}) return context - class DetailView(Crud.DetailView): + class DetailView(AnonCachePageMixin, Crud.DetailView): + # Session plenary detail pages are public and read-only during the + # session. Cache anonymous responses for 2 minutes — short enough + # that real-time voting tallies visible in the detail page stay + # reasonably fresh for public observers. + anon_cache_ttl = 120 # PAGE_CACHE_TTL_LIST @property def layout_key(self): @@ -1419,7 +1498,7 @@ class PresencaView(FormMixin, PresencaMixin, DetailView): # Id dos parlamentares presentes marcados = request.POST.getlist('presenca_ativos') \ - + request.POST.getlist('presenca_inativos') + + request.POST.getlist('presenca_inativos') # Deletar os que foram desmarcados deletar = set(presentes_banco) - set(marcados) @@ -1534,7 +1613,7 @@ class PresencaOrdemDiaView(FormMixin, PresencaMixin, DetailView): # Id dos parlamentares presentes marcados = request.POST.getlist('presenca_ativos') \ - + request.POST.getlist('presenca_inativos') + + request.POST.getlist('presenca_inativos') # Deletar os que foram desmarcados deletar = set(presentes_banco) - set(marcados) @@ -1796,7 +1875,7 @@ def insere_parlamentar_composicao(request): username = request.user.username if request.user.has_perm( '%s.add_%s' % ( - AppConfig.label, IntegranteMesa._meta.model_name)): + AppConfig.label, IntegranteMesa._meta.model_name)): composicao = IntegranteMesa() @@ -1860,7 +1939,7 @@ def remove_parlamentar_composicao(request): username = request.user.username if request.POST and request.user.has_perm( '%s.delete_%s' % ( - AppConfig.label, IntegranteMesa._meta.model_name)): + AppConfig.label, IntegranteMesa._meta.model_name)): if 'composicao_mesa' in request.POST: try: @@ -2914,7 +2993,7 @@ class VotacaoView(SessaoPermissionMixin): username = request.user.username self.logger.error('user=' + username + '. Problemas ao salvar RegistroVotacao da materia de id={} ' 'e da ordem de id={}. '.format(materia_id, ordem_id) + str( - e)) + e)) return self.form_invalid(form) else: ordem = OrdemDia.objects.get(id=ordem_id) @@ -3797,8 +3876,8 @@ class SessaoListView(ListView): return context -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class PautaSessaoView(TemplateView): @@ -3816,8 +3895,8 @@ class PautaSessaoView(TemplateView): reverse('sapl.sessao:pauta_sessao_detail', kwargs={'pk': sessao.pk})) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class PautaSessaoDetailView(PautaMultiFormatOutputMixin, DetailView): @@ -3984,7 +4063,8 @@ class PautaSessaoDetailView(PautaMultiFormatOutputMixin, DetailView): 'resultado_observacao': resultado_observacao, 'situacao': ultima_tramitacao.status if ultima_tramitacao else _("Não informada"), 'processo': f'{str(numeracao.numero_materia)}/{str(numeracao.ano_materia)}' if numeracao else '-', - 'autor': [str(x.autor) for x in Autoria.objects.select_related("autor").filter(materia_id=o.materia_id)], + 'autor': [str(x.autor) for x in + Autoria.objects.select_related("autor").filter(materia_id=o.materia_id)], 'turno': get_turno(ultima_tramitacao.turno) if ultima_tramitacao else '', 'periodo': 'ordem dia', }) @@ -4001,11 +4081,11 @@ class PautaSessaoDetailView(PautaMultiFormatOutputMixin, DetailView): return self.render_to_response(context) -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') -class PesquisarSessaoPlenariaView(MultiFormatOutputMixin, FilterView): +class PesquisarSessaoPlenariaView(AnonCachePageMixin, MultiFormatOutputMixin, FilterView): model = SessaoPlenaria filterset_class = SessaoPlenariaFilterSet paginate_by = 10 @@ -4064,17 +4144,17 @@ class PesquisarSessaoPlenariaView(MultiFormatOutputMixin, FilterView): data = self.filterset.data if data and data.get('data_inicio__year') is not None: - url = "&" + str(self.request.META['QUERY_STRING']) - if url.startswith("&page"): - ponto_comeco = url.find('data_inicio__year=') - 1 - url = url[ponto_comeco:] - context['filter_url'] = url + qr = self.request.GET.copy() + qr.pop('page', None) + context['filter_url'] = ('&' + qr.urlencode()) if qr else '' context['numero_res'] = len(self.object_list) return context def get(self, request, *args, **kwargs): + if len(request.GET.getlist('page')) > 1: + return HttpResponseBadRequest() r = super().get(request) @@ -4092,7 +4172,6 @@ class PesquisarSessaoPlenariaView(MultiFormatOutputMixin, FilterView): return r - class PesquisarPautaSessaoView(PesquisarSessaoPlenariaView): filterset_class = PautaSessaoFilterSet template_name = 'sessao/pauta_sessao_filter.html' @@ -5331,7 +5410,7 @@ class CorrespondenciaCrud(MasterDetailCrud): sessao_plenaria=self.kwargs['pk']).aggregate( Max('numero_ordem'))['numero_ordem__max'] initial['numero_ordem'] = ( - max_numero_ordem if max_numero_ordem else 0) + 1 + max_numero_ordem if max_numero_ordem else 0) + 1 return initial @@ -5393,8 +5472,8 @@ class CorrespondenciaCrud(MasterDetailCrud): return obj -@method_decorator(ratelimit(key=ratelimit_ip, - rate=RATE_LIMITER_RATE, +@method_decorator(ratelimit(key=smart_key, + rate=smart_rate, block=True), name='dispatch') class CorrespondenciaEmLoteView(PermissionRequiredMixin, FilterView): diff --git a/sapl/settings.py b/sapl/settings.py index 4be58ab0c..8fcd1be2c 100644 --- a/sapl/settings.py +++ b/sapl/settings.py @@ -143,9 +143,11 @@ MIDDLEWARE = [ 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.middleware.common.CommonMiddleware', + 'django.middleware.http.ConditionalGetMiddleware', 'sapl.middleware.endpoint_restriction.EndpointRestrictionMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'sapl.middleware.ratelimit.RateLimitMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', @@ -201,13 +203,93 @@ SPECTACULAR_SETTINGS = { 'VERSION': '1.0.0', } -CACHES = { - 'default': { - 'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache', - 'LOCATION': '/var/tmp/django_cache', - 'OPTIONS': {"MAX_ENTRIES": 10000}, +# --------------------------------------------------------------------------- +# Tenant namespace — used as Redis cache KEY_PREFIX (cache:{ns}:*) and +# as the rate-limiter scope for per-namespace keys. +# Defaults to the machine hostname so self-hosted (bare-metal / VM / +# docker-compose) deployments work without any extra config. +# On Kubernetes, POD_NAMESPACE is set by start.sh via the Downward API or +# the service-account namespace file (e.g. "sapl31demo-df"). +# --------------------------------------------------------------------------- +POD_NAMESPACE = config('POD_NAMESPACE', default=host) + +# --------------------------------------------------------------------------- +# Cache — switches between file-based (default) and Redis at pod startup. +# REDIS_URL and CACHE_BACKEND are resolved by start.sh before Gunicorn +# starts; settings.py reads them as env vars (written into .env). +# --------------------------------------------------------------------------- +REDIS_URL = config('REDIS_URL', default='') +CACHE_BACKEND = config('CACHE_BACKEND', default='file') + +# Avoid the double-dot in cache_page keys that occurs when this is '' (default). +# Namespace isolation is already handled by CACHES['default']['KEY_PREFIX']; +# this 'p' just fills the empty slot in Django's key format string. +CACHE_MIDDLEWARE_KEY_PREFIX = 'p' + + +def _build_cache_layer(pod_namespace, cache_backend, redis_url): + """ + Return the CACHES dict for the given runtime environment. + + Two backends are always defined: + default — DB 0: page/view/static-file cache, KEY_PREFIX isolates tenants. + ratelimit — DB 1: rate-limiter counters; pass-through KEY_FUNCTION keeps + raw 'rl:*' keys consistent between RateLimitMiddleware + (get_redis_connection) and @ratelimit decorator paths. + + Redis path: both backends share the same connection-pool settings so a + single pool object is created once and referenced by both caches. + File path: used in development and as a fallback when Redis is absent. + """ + if cache_backend == 'redis' and bool(redis_url): + _pool = { + 'max_connections': 3, # 1,200 pods × 2 workers × 3 = 7,200 peak (headroom for nginx connections) + 'socket_timeout': 0.5, + 'socket_connect_timeout': 0.5, + } + return { + 'default': { + 'BACKEND': 'django_redis.cache.RedisCache', + 'LOCATION': f'{redis_url}/0', + 'KEY_PREFIX': f'cache:{pod_namespace}', + 'TIMEOUT': 300, + 'OPTIONS': { + 'CLIENT_CLASS': 'django_redis.client.DefaultClient', + 'CONNECTION_POOL_KWARGS': _pool, + 'IGNORE_EXCEPTIONS': True, # degrades to cache miss on Redis failure + }, + }, + 'ratelimit': { + 'BACKEND': 'django_redis.cache.RedisCache', + 'LOCATION': f'{redis_url}/1', + 'KEY_FUNCTION': 'sapl.middleware.ratelimit.make_ratelimit_cache_key', + 'OPTIONS': { + 'CLIENT_CLASS': 'django_redis.client.DefaultClient', + 'CONNECTION_POOL_KWARGS': _pool, + 'IGNORE_EXCEPTIONS': True, + }, + }, + } + + return { + 'default': { + 'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache', + 'LOCATION': '/var/tmp/django_cache', + 'KEY_PREFIX': f'cache:{pod_namespace}', + 'OPTIONS': {'MAX_ENTRIES': 10000}, + }, + 'ratelimit': { + 'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache', + 'LOCATION': '/var/tmp/django_ratelimit_cache', + 'KEY_FUNCTION': 'sapl.middleware.ratelimit.make_ratelimit_cache_key', + 'OPTIONS': {'MAX_ENTRIES': 5000}, + }, } -} + + +CACHES = _build_cache_layer(POD_NAMESPACE, CACHE_BACKEND, REDIS_URL) + +RATELIMIT_USE_CACHE = 'ratelimit' ROOT_URLCONF = 'sapl.urls' @@ -315,8 +397,81 @@ WAFFLE_ENABLE_ADMIN_PAGES = True MAX_DOC_UPLOAD_SIZE = 150 * 1024 * 1024 # 150MB MAX_IMAGE_UPLOAD_SIZE = 2 * 1024 * 1024 # 2MB DATA_UPLOAD_MAX_MEMORY_SIZE = 10 * 1024 * 1024 # 10MB +# Files above 2 MB are streamed to a temp file on disk rather than held in +# worker RAM. Critical for large upload support without memory blowup. +FILE_UPLOAD_MAX_MEMORY_SIZE = 2 * 1024 * 1024 # 2MB +FILE_UPLOAD_TEMP_DIR = '/var/interlegis/sapl/tmp' + +# --------------------------------------------------------------------------- +# Rate limiting — RateLimitMiddleware (sapl/middleware/ratelimit.py) +# --------------------------------------------------------------------------- +RATE_LIMITER_RATE = config('RATE_LIMITER_RATE', default='120/m') +RATE_LIMITER_RATE_AUTHENTICATED = config('RATE_LIMITER_RATE_AUTHENTICATED', default='240/m') +RATE_LIMITER_RATE_BOT = config('RATE_LIMITER_RATE_BOT', default='5/m') + +# Seconds between re-fetches of the runtime UA deny list from Redis DB 1. +# Lower values pick up new blocked UAs faster; higher values reduce Redis round-trips. +RATE_LIMITER_UA_BLOCKLIST_REFRESH = config('RATE_LIMITER_UA_BLOCKLIST_REFRESH', default=60, cast=int) + +# Seconds between re-fetches of the IP-prefix deny list (rl:ip_prefix:blocked SET). +RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH = config('RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH', default=60, cast=int) + +# Number of shards for the blocked-IP ZSET indexes. +# Each shard receives IPs deterministically via md5(ip) % N, distributing +# write contention across N keys. Increase for high-throughput deployments. +RATE_LIMITER_INDEX_SHARDS = config('RATE_LIMITER_INDEX_SHARDS', default=3, cast=int) + +# Maximum 404 responses from one anonymous IP in one anon window before the IP +# is blocked. Catches path-probing scanners that don't use recognised extensions. +RATE_LIMIT_404_THRESHOLD = config('RATE_LIMIT_404_THRESHOLD', default=20, cast=int) + +# Paths exempt from rate limiting at the Django layer. +# Regex strings matched against request.path. +# /painel//dados is a high-frequency polling endpoint (will become WebSocket); +# it is also exempt at the nginx layer (location block with no limit_req). +RATE_LIMIT_BYPASS_PATHS = [ + r'^/painel/\d+/dados$', + r'^/voto-individual/', + r'^/sessao/\d+', + r'^/sessao/pauta-sessao/\d+/', +] -RATE_LIMITER_RATE = config('RATE_LIMITER_RATE', default='35/m') +# API quota — daily and weekly call caps for all /api/ callers (anon and auth). +# All callers are keyed by IP — auth status is not checked. +# Weekly default is 7× the daily cap. +API_QUOTA_DAILY = config('API_QUOTA_DAILY', default=100000, cast=int) +API_QUOTA_WEEKLY = config('API_QUOTA_WEEKLY', default=700000, cast=int) + +# API-specific per-minute rate limit for external (non-same-origin) anonymous calls. +# Abuse writes rl:api:ip::blocked only — never rl:ip::blocked. +API_RATE_LIMIT_ENABLED = config('API_RATE_LIMIT_ENABLED', default=True, cast=bool) +API_RATE_LIMIT_THRESHOLD = config('API_RATE_LIMIT_THRESHOLD', default=120, cast=int) +API_RATE_LIMIT_WINDOW_SECONDS = config('API_RATE_LIMIT_WINDOW_SECONDS', default=60, cast=int) +API_RATE_LIMIT_BLOCK_SECONDS = config('API_RATE_LIMIT_BLOCK_SECONDS', default=60, cast=int) +API_RATE_LIMIT_SAME_ORIGIN_BYPASS = config('API_RATE_LIMIT_SAME_ORIGIN_BYPASS', default=True, cast=bool) + +# Media file serving — serve_media (sapl/base/media.py) via X-Accel-Redirect. +# TTL for both URL-path and storage-path access counters (DB 1). +MEDIA_PATH_COUNTER_TTL = config('MEDIA_PATH_COUNTER_TTL', default=60, cast=int) + +# --------------------------------------------------------------------------- +# Anonymous page caching — AnonCachePageMixin (sapl/middleware/page_cache.py) +# TTLs apply only to anonymous (unauthenticated) GET responses. +# Authenticated users always bypass the cache (see AnonCachePageMixin). +# --------------------------------------------------------------------------- +# Public list views (norma, materia, sessao, parlamentares…) +PAGE_CACHE_TTL_LIST = config('PAGE_CACHE_TTL_LIST', default=120, cast=int) +# Public detail views — rarely mutated once published +PAGE_CACHE_TTL_DETAIL = config('PAGE_CACHE_TTL_DETAIL', default=300, cast=int) +# High-stability detail views (parlamentar, comissão) — change only each term +PAGE_CACHE_TTL_STABLE = config('PAGE_CACHE_TTL_STABLE', default=600, cast=int) + +logger.info( + '[PAGE_CACHE] list=%ds detail=%ds stable=%ds', + PAGE_CACHE_TTL_LIST, + PAGE_CACHE_TTL_DETAIL, + PAGE_CACHE_TTL_STABLE, +) # Internationalization # https://docs.djangoproject.com/en/1.8/topics/i18n/ @@ -334,7 +489,6 @@ USE_I18N = True USE_L10N = True USE_TZ = True - ## ## Monkey patch of the Django 2.2 because latest version of psycopg2 returns DB time zone as UTC, ## but Django 2.2 requires an int! This should be removed once we are able to upgrade to Django >= 4 @@ -357,6 +511,16 @@ def _compat_utc_tzinfo_factory(offset): pg_utils.utc_tzinfo_factory = _compat_utc_tzinfo_factory +## +## Strip the language/timezone suffix from Django page-cache keys. +## Django's _i18n_cache_key_suffix appends ".pt-br.America/Sao_Paulo" when +## USE_I18N and USE_TZ are True. SAPL is monolingual (always pt-br / +## America/Sao_Paulo), so the suffix is constant noise that bloats key names. +## +import django.utils.cache as _dj_cache + +_dj_cache._i18n_cache_key_suffix = lambda request, cache_key: cache_key + # DATE_FORMAT = 'N j, Y' DATE_FORMAT = 'd/m/Y' SHORT_DATE_FORMAT = 'd/m/Y' @@ -409,7 +573,7 @@ CRISPY_FAIL_SILENTLY = not DEBUG FILTERS_HELP_TEXT_FILTER = False LOGGING_CONSOLE_VERBOSE = config( - 'LOGGING_CONSOLE_VERBOSE', cast=bool, default=False) + 'LOGGING_CONSOLE_VERBOSE', cast=bool, default=True) LOGGING = { 'version': 1, @@ -429,7 +593,7 @@ LOGGING = { 'formatters': { 'verbose': { 'format': '%(levelname)s %(asctime)s [%(request_id)s] ' + host + '%(pathname)s %(name)s:%(funcName)s:%(' - 'lineno)d %(message)s ' + 'lineno)d %(message)s ' }, 'simple': { 'format': '%(levelname)s %(asctime)s [%(request_id)s] - %(message)s' diff --git a/sapl/static/429.html b/sapl/static/429.html new file mode 100644 index 000000000..2f1bc8ead --- /dev/null +++ b/sapl/static/429.html @@ -0,0 +1,42 @@ + + + + + + 429 – Muitas Requisições + + + +
+
429
+

Muitas Requisições

+

Você realizou muitas requisições em um curto período. Aguarde um momento e tente novamente.

+

Se o problema persistir, entre em contato com o suporte da sua Câmara Municipal.

+
+ + diff --git a/sapl/static/500.html b/sapl/static/500.html new file mode 100644 index 000000000..6dd771897 --- /dev/null +++ b/sapl/static/500.html @@ -0,0 +1,42 @@ + + + + + + 500 – Erro Interno do Servidor + + + +
+
500
+

Erro Interno do Servidor

+

Ocorreu um erro inesperado. Nossa equipe foi notificada e trabalhará para resolver o problema.

+

Tente novamente em alguns instantes. Se o problema persistir, entre em contato com o suporte da sua Câmara Municipal.

+
+ + diff --git a/sapl/static/robots.txt b/sapl/static/robots.txt new file mode 100644 index 000000000..4b791ce0b --- /dev/null +++ b/sapl/static/robots.txt @@ -0,0 +1,35 @@ +User-agent: GPTBot +Disallow: / +Crawl-delay: 10 + +User-agent: ClaudeBot +Disallow: / +Crawl-delay: 10 + +User-agent: meta-externalagent +Disallow: / +Crawl-delay: 10 + +User-agent: OAI-SearchBot +Disallow: / +Crawl-delay: 10 + +User-agent: bingbot +Disallow: / +Crawl-delay: 10 + +User-agent: SERankingBacklinksBot +Disallow: / +Crawl-delay: 10 + +User-agent: anthropic-python +Disallow: / +Crawl-delay: 10 + +User-agent: AwarioSmartBot +Disallow: / +Crawl-delay: 10 + +User-agent: * +Disallow: /relatorios/ +Crawl-delay: 10 diff --git a/sapl/templates/painel/index.html b/sapl/templates/painel/index.html index d5dc8dc4c..816ffbde8 100644 --- a/sapl/templates/painel/index.html +++ b/sapl/templates/painel/index.html @@ -297,7 +297,7 @@ $('#tema_solene_div').show(); } - if (data["brasao"] != null) + if (data["brasao"] != null && $("#logo-painel").attr("src") !== data["brasao"]) $("#logo-painel").attr("src", data["brasao"]); var presentes = $("#parlamentares"); diff --git a/sapl/templates/painel/mensagem.html b/sapl/templates/painel/mensagem.html deleted file mode 100644 index da8e355d4..000000000 --- a/sapl/templates/painel/mensagem.html +++ /dev/null @@ -1,120 +0,0 @@ -{% load i18n %} -{% load common_tags %} -{% load render_bundle from webpack_loader %} -{% load webpack_static from webpack_loader %} - - - - - - - - - - - {% block head_title %}{% trans 'SAPL - Sistema de Apoio ao Processo Legislativo' %}{% endblock %} - - - {% render_chunk_vendors 'css' %} - {% render_bundle 'global' 'css' %} - {% render_bundle 'painel' 'css' %} - - - - - - -

{{ context.title }}

- -

Ajax refresh counter:

-

-

-

-

-




-
- - - - - {% render_chunk_vendors 'js' %} - {% render_bundle 'global' 'js' %} - {% render_bundle 'painel' 'js' %} - - - diff --git a/sapl/templates/painel/parlamentares.html b/sapl/templates/painel/parlamentares.html deleted file mode 100644 index 3dbe6e740..000000000 --- a/sapl/templates/painel/parlamentares.html +++ /dev/null @@ -1,128 +0,0 @@ -{% load i18n %} -{% load common_tags %} - -{% load render_bundle from webpack_loader %} -{% load webpack_static from webpack_loader %} - - - - - - - - - - - {% block head_title %}{% trans 'SAPL - Sistema de Apoio ao Processo Legislativo' %}{% endblock %} - - - {% render_chunk_vendors 'css' %} - {% render_bundle 'global' 'css' %} - {% render_bundle 'painel' 'css' %} - - - - - - - -

{{ context.title }}

- -

-

-

-

-

- - - - -
-
    -
-
- - - - {% render_chunk_vendors 'js' %} - {% render_bundle 'global' 'js' %} - {% render_bundle 'painel' 'js' %} - - - diff --git a/sapl/templates/painel/votacao.html b/sapl/templates/painel/votacao.html deleted file mode 100644 index e73551160..000000000 --- a/sapl/templates/painel/votacao.html +++ /dev/null @@ -1,123 +0,0 @@ -{% load i18n %} -{% load render_bundle from webpack_loader %} -{% load webpack_static from webpack_loader %} - - - - - - - - - - - {% block head_title %}{% trans 'SAPL - Sistema de Apoio ao Processo Legislativo' %}{% endblock %} - - - {% render_chunk_vendors 'css' %} - {% render_bundle 'global' 'css' %} - {% render_bundle 'painel' 'css' %} - - - - -

{{ context.title }}

- -

-

-

-

-

- - - - -
-
    -
-
-
- - - - {% render_chunk_vendors 'js' %} - {% render_bundle 'global' 'js' %} - {% render_bundle 'painel' 'js' %} - - - diff --git a/sapl/urls.py b/sapl/urls.py index d2967e927..dcb3010b1 100644 --- a/sapl/urls.py +++ b/sapl/urls.py @@ -82,6 +82,13 @@ urlpatterns += [ ] +# Media files — served via X-Accel-Redirect in production, directly in DEBUG. +from sapl.base.media import serve_media # noqa: E402 + +urlpatterns += [ + url(r'^media/(?P.*)$', serve_media, name='serve_media'), +] + # Fix a static asset finding error on Django 1.9 + gunicorn: # http://stackoverflow.com/questions/35510373/ @@ -95,11 +102,7 @@ if settings.DEBUG: urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) - urlpatterns += [ - url(r'^media/(?P.*)$', view_static_server, { - 'document_root': settings.MEDIA_ROOT, - }), - ] + # media/ is handled by serve_media below (works in DEBUG too) # Make the rate limiter return 429 (Too Many Requests) instead of 403 (Forbidden Access) diff --git a/sapl/utils.py b/sapl/utils.py index 42d4b2c6e..b4be99653 100644 --- a/sapl/utils.py +++ b/sapl/utils.py @@ -1,6 +1,6 @@ import csv import string -from functools import wraps +from functools import lru_cache, wraps import hashlib import io from itertools import groupby, chain @@ -401,23 +401,6 @@ def xstr(s): return '' if s is None else str(s) -def get_client_ip(request): - from ratelimit.core import ip_mask - x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') - if x_forwarded_for: - ip = x_forwarded_for.split(',')[0] - else: - ip = request.META.get('HTTP_X_REAL_IP') or request.META.get('REMOTE_ADDR') or '0.0.0.0' - return ip_mask(ip) - - -def ratelimit_ip(group, request): - """ - Ignore group param in django-ratelimit==3.0.1 - """ - return get_client_ip(request) - - def get_base_url(request): # TODO substituir por Site.objects.get_current().domain # from django.contrib.sites.models import Site @@ -976,13 +959,15 @@ def parlamentares_ativos(data_inicio, data_fim=None): return Parlamentar.objects.filter(id__in=parlamentares_id) -def show_results_filter_set(qr): - query_params = set(qr.keys()) - if ((len(query_params) == 1 and 'iframe' in query_params) or - len(query_params) == 0): - return False +_IGNORED_PARAMS = frozenset({'iframe', 'pesquisar', 'csrfmiddlewaretoken'}) - return True + +def show_results_filter_set(qr): + meaningful = { + k for k, v in qr.items() + if k not in _IGNORED_PARAMS and v and v.strip() + } + return bool(meaningful) def sort_lista_chave(lista, chave): @@ -1120,6 +1105,7 @@ def cached_call(key, timeout=300): return result return wrap + return cache_decorator @@ -1265,7 +1251,7 @@ class GoogleRecapthaMixin: return cd -# TODO: cache this map and invalidate on each update +@lru_cache(maxsize=None) def get_report_urls_map(): from django.urls import get_resolver from django.urls.base import reverse @@ -1294,13 +1280,11 @@ def get_report_urls_map(): def is_report_allowed(request, url_path=None): - from sapl.utils import get_report_urls_map # TODO: import global - url_map = get_report_urls_map() # TODO: cache this!!! Globally - + url_map = get_report_urls_map() path = url_path if url_path else request.path - authenticated = True if request.user.is_authenticated else False + authenticated = request.user.is_authenticated - if path in url_map.keys(): + if path in url_map: path_metadata = url_map[path] if not authenticated and path_metadata['public']: return True @@ -1340,7 +1324,7 @@ def get_path_to_name_report_map(): class Row: - def __init__(self, cols, is_header = False): + def __init__(self, cols, is_header=False): self.cols = cols self.is_header = is_header @@ -1349,7 +1333,7 @@ class Row: class Table: - def __init__(self, header = [], rows = []): + def __init__(self, header=[], rows=[]): self.header = header self.rows = rows @@ -1551,7 +1535,7 @@ class MultiFormatOutputMixin: verbose_name = [] - if hasattr(self, f'hook_header_{fname}'): # suporta extensao de funcionalidade + if hasattr(self, f'hook_header_{fname}'): # suporta extensao de funcionalidade h = getattr(self, f'hook_header_{fname}')() yield h continue diff --git a/scripts/test_ratelimiter.py b/scripts/test_ratelimiter.py new file mode 100644 index 000000000..44801956f --- /dev/null +++ b/scripts/test_ratelimiter.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +""" +Script to test rate limiting of an endpoint. +""" + +import argparse +import time +import requests +from collections import defaultdict +from urllib.parse import urlparse + + +def test_rate_limiter(url, num_requests=50, delay=0.1, timeout=10): + """Send multiple requests and analyze rate limiting behavior.""" + parsed = urlparse(url) + if not parsed.scheme or not parsed.netloc: + raise ValueError( + "URL must include a protocol and host, e.g. http://localhost or https://example.com" + ) + if parsed.scheme not in {"http", "https"}: + raise ValueError("Unsupported URL scheme: %s. Use http or https." % parsed.scheme) + + status_counts = defaultdict(int) + response_times = [] + first_rate_limited_at = None + attempted_requests = 0 + + print(f"Testing rate limiter on: {url}") + print(f"Number of requests: {num_requests}") + print(f"Delay between requests: {delay}s") + print("-" * 50) + + for i in range(num_requests): + attempted_requests += 1 + try: + start_time = time.time() + response = requests.get(url, timeout=timeout) + elapsed = time.time() - start_time + + status_counts[response.status_code] += 1 + response_times.append(elapsed) + + print(f"Request {i+1:3d}: Status {response.status_code} | Time: {elapsed:.3f}s") + + if response.status_code == 429: + if first_rate_limited_at is None: + first_rate_limited_at = i + 1 + print(f" -> Rate limited on request {i+1}") + break + + except requests.exceptions.RequestException as e: + print(f"Request {i+1:3d}: Error - {e}") + status_counts['ERROR'] += 1 + + if i < num_requests - 1: + time.sleep(delay) + + print("-" * 50) + print("\nSummary:") + print(f" Total requests attempted: {attempted_requests}") + print(f" Successful (200): {status_counts.get(200, 0)}") + print(f" Rate limited (429): {status_counts.get(429, 0)}") + if first_rate_limited_at is not None: + print(f" First 429 occurred at request: {first_rate_limited_at}") + print(f" Other errors: {sum(v for k, v in status_counts.items() if k not in [200, 429, 'ERROR'])}") + + if response_times: + avg_time = sum(response_times) / len(response_times) + print(f"\nAverage response time: {avg_time:.3f}s") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Test rate limiter of a URL") + parser.add_argument( + "url", + help="URL to test, including protocol (http:// or https://)", + ) + parser.add_argument("-n", "--num-requests", type=int, default=50, help="Number of requests") + parser.add_argument("-d", "--delay", type=float, default=0.1, help="Delay between requests (seconds)") + parser.add_argument("-t", "--timeout", type=int, default=10, help="Request timeout (seconds)") + + args = parser.parse_args() + test_rate_limiter(args.url, args.num_requests, args.delay, args.timeout) \ No newline at end of file diff --git a/scripts/test_ratelimiter.sh b/scripts/test_ratelimiter.sh deleted file mode 100755 index 57e243db2..000000000 --- a/scripts/test_ratelimiter.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/bash - -#URL=http://localhost:8000/materia/4379 -#URL=http://localhost:8000/norma/pesquisar -#URL=http://localhost/norma/pesquisar -#URL=https://sapl31demo.interlegis.leg.br/docadm/45 -#URL=https://sapl.joaopessoa.pb.leg.br/materia/186300 -#URL=http://localhost:8000/materia/4379/materiaassunto -#URL=http://localhost:8000/sessao/4984 -URL="http://localhost:8000/docadm/pesq-doc-adm?tipo=&o=&numero=&complemento=&ano=&protocolo__numero=&numero_externo=&data_0=&data_1=&interessado=&assunto=&tramitacao=&tramitacaoadministrativo__status=&tramitacaoadministrativo__unidade_tramitacao_destino=&pesquisar=Pesquisar" - -for i in $(seq 1 12); do - curl -sS -o /dev/null -w "req=$i http=%{http_code} time=%{time_total}\n" "$URL" -done diff --git a/work_queues.md b/work_queues.md new file mode 100644 index 000000000..8d62176a1 --- /dev/null +++ b/work_queues.md @@ -0,0 +1,173 @@ +# SAPL — Work Queues & Real-Time: Async PDF + WebSocket Voting + +> **Status**: Planned follow-up mini-project. +> **Prerequisite**: Redis A (cache + rate-limiter pod, `rate-limiter-2026` branch) must be +> deployed to production, stable, and OOM pressure confirmed reduced before starting this work. +> **Scope**: Django 2.2 / Gunicorn / Celery / Django Channels — same fleet of 1,200+ pods. + +--- + +## Table of Contents + +1. [Context & Motivation](#1-context--motivation) +2. [Redis Topology for Work Queues](#2-redis-topology-for-work-queues) +3. [Phase 1 — Async PDF via Celery](#3-phase-1--async-pdf-via-celery) +4. [Phase 2 — Django Channels (WebSocket Voting Panel)](#4-phase-2--django-channels-websocket-voting-panel) +5. [Open Questions](#5-open-questions) + +--- + +## 1. Context & Motivation + +After `rate-limiter-2026` ships: + +| Remaining pain point | Current behaviour | Target | +|---|---|---| +| PDF generation | Holds a Gunicorn worker thread for the full build duration (10–60 s). Workers are at 400 MB cap — a PDF request burns one slot for up to a minute | Enqueue via Celery; respond 202 immediately; worker is freed | +| WebSocket voting panel | Not implemented; councillors use a polling page | Persistent connection via Django Channels backed by Redis | + +--- + +## 2. Redis Topology for Work Queues + +> **Critical constraint**: Celery broker **must** be a **separate** Redis instance (Redis B) +> with `noeviction` policy. +> Redis A (cache + rate-limiter) uses `allkeys-lru` — tasks enqueued there would be silently +> evicted under memory pressure, causing jobs to vanish without error. + +| Instance | Role | Eviction policy | Persistence | +|---|---|---|---| +| **Redis A** (existing) | Page cache (DB0), rate limiter (DB1), Django Channels (DB2) | `allkeys-lru` | none | +| **Redis B** (new) | Celery broker + result backend | `noeviction` | AOF + RDB snapshot | + +```yaml +# docker/k8s/redis-celery-configmap.yaml +data: + redis.conf: | + maxmemory-policy noeviction # never evict tasks + appendonly yes # AOF persistence ON + save "900 1" # RDB snapshot every 15 min if ≥1 change + databases 2 # DB0 = broker queue, DB1 = result backend +``` + +--- + +## 3. Phase 1 — Async PDF via Celery + +### 3.1 Current (synchronous) flow + +Holds worker memory for the entire PDF build: + +```mermaid +sequenceDiagram + participant B as Browser + participant G as Gunicorn worker + participant ORM as PostgreSQL + participant RL as ReportLab + + B->>G: GET /pdf/materia/12345 + G->>ORM: N+1 queries (get_etiqueta_protocolos) + ORM-->>G: data + G->>RL: build entire PDF in RAM + RL-->>G: PDF bytes (held in worker memory) + G-->>B: stream response + note over G: worker blocked for full duration +``` + +### 3.2 Target (async) flow + +Worker freed immediately after enqueueing: + +```mermaid +sequenceDiagram + participant B as Browser + participant G as Gunicorn worker + participant Q as Redis B (Celery queue) + participant W as Celery worker + participant D as Disk / nginx + + B->>G: POST /pdf/materia/12345 + G->>Q: enqueue task + G-->>B: 202 Accepted + task_id + W->>W: build PDF (out of band) + W->>D: write PDF to /media/pdf/task_id.pdf + B->>G: GET /pdf/status/task_id + G-->>B: 302 → nginx /media/pdf/task_id.pdf +``` + +### 3.3 Celery settings + +```python +# sapl/settings.py additions +CELERY_BROKER_URL = config('CELERY_BROKER_URL', default='') +CELERY_RESULT_BACKEND = config('CELERY_RESULT_BACKEND', default='') + +# Soft limit: warn at 350 MB; hard limit: kill+restart at 450 MB. +# Keeps Celery workers inside the same memory envelope as Gunicorn workers. +CELERY_WORKER_MAX_MEMORY_PER_CHILD = 400 * 1024 # KB +CELERY_TASK_SOFT_TIME_LIMIT = 120 # seconds — warn +CELERY_TASK_TIME_LIMIT = 180 # seconds — SIGKILL +``` + +### 3.4 k8s manifests + +New files to be created under `docker/k8s/`: + +- `redis-celery-configmap.yaml` — Redis B config (noeviction, AOF) +- `redis-celery-deployment.yaml` — single-replica Redis B pod +- `redis-celery-service.yaml` — ClusterIP service +- `celery-deployment.yaml` — Celery worker deployment (same image as SAPL) + +### 3.5 Environment variables (per-namespace Secret) + +| Variable | Example value | Notes | +|---|---|---| +| `CELERY_BROKER_URL` | `redis://sapl-redis-celery.redis.svc:6379/0` | Redis B, DB0 | +| `CELERY_RESULT_BACKEND` | `redis://sapl-redis-celery.redis.svc:6379/1` | Redis B, DB1 | + +--- + +## 4. Phase 2 — Django Channels (WebSocket Voting Panel) + +Uses **Redis A DB2** (reserved in the existing key-layout table — no new infra needed beyond +what ships in `rate-limiter-2026`). + +### 4.1 Channel layer settings + +```python +# sapl/settings.py additions +CHANNEL_LAYERS = { + "default": { + "BACKEND": "channels_redis.core.RedisChannelLayer", + "CONFIG": { + "hosts": [("sapl-redis.redis.svc.cluster.local", 6379)], + "db": 2, # DB2 reserved for channels (see rate-limiter-v2.md §0.2) + "capacity": 1500, + "expiry": 10, + }, + } +} +``` + +### 4.2 Prerequisites before starting + +- [ ] Redis A stable in production (rate limiter + cache confirmed working) +- [ ] OOM kill rate reduced to near-zero +- [ ] Bot siege resolved (Phase 0–2 metrics reviewed) +- [ ] Decision on ASGI server (Daphne vs Uvicorn + channels) — Gunicorn alone cannot serve WebSockets + +--- + +## 5. Open Questions + +| # | Question | Blocks | +|---|---|---| +| 1 | Which PDF endpoints are highest priority for async migration? (`/relatorios/`, `/materia/pdf/`, other)? | Phase 1 scope | +| 2 | Should the Celery worker run in the same pod as Gunicorn (sidecar) or a dedicated deployment? | Phase 1 k8s design | +| 3 | Result backend TTL — how long should generated PDFs be retained before cleanup? | Phase 1 storage design | +| 4 | ASGI server selection for Channels (Daphne vs uvicorn + channels) | Phase 2 | +| 5 | WebSocket voting panel: is per-session or per-pod state acceptable? | Phase 2 architecture | + +--- + +*Planned work — begins after `rate-limiter-2026` is stable in production.*