mirror of https://github.com/interlegis/sapl.git
Browse Source
Implements a multi-layer DDoS/abuse mitigation stack across nginx, Django
middleware, and Redis:
**nginx hardening (Phase 0)**
- GeoIP2 ASN-based bot blocking (MaxMind GeoLite2-ASN, offline build)
- Request-rate and connection-rate limit zones; scanner probe zones
- Gunicorn worker/timeout tuning; N+1 query fix in relatorio views
- X-Accel-Redirect for private /media/ files; nginx internal media path
**Redis infrastructure (Phase 1)**
- Shared Redis pod for cross-pod state (k8s manifests: Deployment, Service, ConfigMap)
- Django dual-backend cache: default (file/Redis) + dedicated `ratelimit` Redis DB 1
- configure_redis_cache() wired into settings startup; KEY_PREFIX namespace isolation
**RateLimitMiddleware (Phase 2)**
- Per-IP request rate counter (60/min sliding window) → block key in Redis DB 1
- Per-IP 404-scan counter and UA Redis deny list
- NAT-aware: blocks authenticated users behind a shared blocked IP
- Scoped API rate limit (60/min) with daily/weekly quota per consumer
- IP-prefix blocklist (SET `rl:ip_prefix:blocked`) with 4-candidate SMEMBERS check
- Blocked-IP ZSET indexes with inline pruning; atomic block writes via Lua script
- Returns 403 for prefix-list hits, 429 for rate/quota hits
- Django block metrics tracked per layer; CORS headers on 429 responses
- Collapse rl:metrics STRING keys into a HASH per tenant per day
**Anonymous page caching (Phase 4)**
- AnonCachePageMixin applied to 8 public search/filter views (materia, sessao,
pesquisar-sessao, comissoes, norma, audiencia, relatorios)
- Mitigates pesquisar-sessao DDoS; sanitises page= param pollution
- ETag/Last-Modified conditional request support on norma and media views
**nginx Lua early-rejection layer**
- `blocklist.lua`: per-request access_by_lua_file hook — checks shared dict for
prefix matches (zero Redis I/O), then pipelines GET rl:ip:{ip}:blocked +
GET rl:api:ns:{ns}:ip:{ip}:blocked; returns 429 before Gunicorn is reached
- `init_worker_by_lua_block`: 60s timer refreshes `ngx.shared.ip_prefix_blocked`
from Redis SMEMBERS; fail-open on Redis error
- Uses `libnginx-mod-http-lua` + `libnginx-mod-http-ndk` (Debian Bookworm dynamic
modules); load order in nginx.conf: NDK → GeoIP2 → Lua
- `resty_redis.lua` vendored (lua-resty-redis is not in Debian Bookworm repos)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rate-limiter-2026
55 changed files with 8156 additions and 850 deletions
@ -0,0 +1,145 @@ |
|||
# CLAUDE.md |
|||
|
|||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
|||
|
|||
## Project Overview |
|||
|
|||
SAPL (Sistema de Apoio ao Processo Legislativo) is a Django-based legislative management system used by Brazilian municipal and state legislative houses. It manages bills, parliamentary sessions, committees, norms, protocols, and related legislative workflows. |
|||
|
|||
## Commands |
|||
|
|||
### Development |
|||
|
|||
```bash |
|||
# Run dev server |
|||
python manage.py runserver |
|||
|
|||
# Docker (dev, without bundled DB) |
|||
docker-compose -f docker/docker-compose-dev.yml up |
|||
|
|||
# Docker (dev, with PostgreSQL container) |
|||
docker-compose -f docker/docker-compose-dev-db.yml up |
|||
``` |
|||
|
|||
### Database Setup (local PostgreSQL) |
|||
|
|||
```bash |
|||
sudo -u postgres psql -c "CREATE ROLE sapl LOGIN ENCRYPTED PASSWORD 'sapl' NOSUPERUSER INHERIT CREATEDB NOCREATEROLE NOREPLICATION;" |
|||
sudo -u postgres psql -c "CREATE DATABASE sapl WITH OWNER=sapl ENCODING='UTF8' LC_COLLATE='pt_BR.UTF-8' LC_CTYPE='pt_BR.UTF-8' CONNECTION LIMIT=-1 TEMPLATE template0;" |
|||
python manage.py migrate |
|||
``` |
|||
|
|||
### Testing |
|||
|
|||
```bash |
|||
# All tests (reuses DB by default for speed) |
|||
pytest |
|||
|
|||
# Single test file or test function |
|||
pytest sapl/materia/tests/test_materia.py |
|||
pytest sapl/materia/tests/test_materia.py::test_function_name |
|||
|
|||
# Force DB recreation |
|||
pytest --create-db |
|||
|
|||
# With coverage |
|||
pytest --cov=sapl |
|||
``` |
|||
|
|||
Tests require `DJANGO_SETTINGS_MODULE=sapl.settings` (set in `pytest.ini`). All tests must be marked with `@pytest.mark.django_db`. The `conftest.py` root fixture provides an `app` fixture (WebTest `DjangoTestApp`). |
|||
|
|||
### Linting / Formatting |
|||
|
|||
```bash |
|||
flake8 . |
|||
isort . |
|||
autopep8 --in-place <file.py> |
|||
``` |
|||
|
|||
### Restore Database from Backup |
|||
|
|||
```bash |
|||
./scripts/restore_db.sh -f /path/to/dump |
|||
./scripts/restore_db.sh -f /path/to/dump -p 5433 # Docker port |
|||
``` |
|||
|
|||
## Architecture |
|||
|
|||
### Django Apps |
|||
|
|||
Apps are under `sapl/` and follow domain boundaries: |
|||
|
|||
| App | Domain | |
|||
|-----|--------| |
|||
| `base` | `CasaLegislativa` (legislative house config), `AppConfig`, `Autor` (authorship) | |
|||
| `parliamentary` | `Parlamentar`, `Legislatura`, `SessaoLegislativa`, `Coligacao` | |
|||
| `materia` | Bills (`MateriaLegislativa`), types, tracking, annexes | |
|||
| `norma` | Laws/norms (`NormaJuridica`) and hierarchies | |
|||
| `sessao` | Plenary sessions, agenda, attendance, voting | |
|||
| `comissoes` | Committees (`Comissao`) and meetings (`Reuniao`) | |
|||
| `protocoloadm` | Administrative protocols and document intake | |
|||
| `compilacao` | Structured/articulated texts (LexML-like tree structure) | |
|||
| `lexml` | LexML XML standard integration | |
|||
| `audiencia` | Public hearings | |
|||
| `painel` | Real-time session display panel | |
|||
| `relatorios` | PDF report generation | |
|||
| `api` | REST API entry point (auto-generated ViewSets) | |
|||
| `crud` | Generic CRUD base views | |
|||
| `rules` | Business rules and permission definitions | |
|||
|
|||
### REST API |
|||
|
|||
The API uses a custom `drfautoapi` package (`drfautoapi/drfautoapi.py`) that auto-generates DRF ViewSets, Serializers, and FilterSets from Django models. Authentication is Token + Session. Permissions use a custom `SaplModelPermissions` class that maps HTTP methods to Django model permissions. |
|||
|
|||
OpenAPI 3.0 docs are generated by drf-spectacular. |
|||
|
|||
### Caching |
|||
|
|||
- **Default:** File-based (`/var/tmp/django_cache`) |
|||
- **Production:** Redis via django-redis; configured at startup by `configure_redis_cache()` in `sapl/settings.py` |
|||
- **Cache key prefix:** `cache:{POD_NAMESPACE}:` (namespace-isolated for multi-tenant k8s) |
|||
- **Rate limiter state** is shared via Redis keys |
|||
|
|||
### Feature Flags |
|||
|
|||
django-waffle is used for feature flags. Switches (global on/off) can be toggled via: |
|||
|
|||
```bash |
|||
python manage.py waffle_switch <switch_name> on|off |
|||
``` |
|||
|
|||
### Key Environment Variables |
|||
|
|||
| Variable | Purpose | |
|||
|----------|---------| |
|||
| `DATABASE_URL` | PostgreSQL connection string | |
|||
| `SECRET_KEY` | Django secret key | |
|||
| `DEBUG` | Debug mode | |
|||
| `REDIS_URL` | Redis host:port | |
|||
| `CACHE_BACKEND` | `file` or `redis` | |
|||
| `POD_NAMESPACE` | K8s namespace (used in cache key prefix) | |
|||
| `USE_SOLR` | Enable Haystack/Solr full-text search | |
|||
| `SOLR_URL` / `SOLR_COLLECTION` | Solr connection | |
|||
|
|||
### Docker Build |
|||
|
|||
The production build requires a MaxMind GeoLite2-ASN license key (for nginx ASN-based bot blocking): |
|||
|
|||
```bash |
|||
docker build --secret id=maxmind_key,src=.env -f docker/Dockerfile -t sapl:local . |
|||
``` |
|||
|
|||
Optional build args: `WITH_NGINX`, `WITH_GRAPHVIZ`, `WITH_POPPLER`, `WITH_PSQL_CLIENT`. |
|||
|
|||
### Key File Locations |
|||
|
|||
| File | Purpose | |
|||
|------|---------| |
|||
| `sapl/settings.py` | All Django settings, including cache/rate-limit setup | |
|||
| `pytest.ini` | Test configuration (DJANGO_SETTINGS_MODULE, addopts) | |
|||
| `conftest.py` | Root pytest fixtures | |
|||
| `drfautoapi/drfautoapi.py` | Auto-API generation logic | |
|||
| `docker/startup_scripts/start.sh` | Container entrypoint (migrations, waffle, gunicorn) | |
|||
| `requirements/requirements.txt` | Production deps | |
|||
| `requirements/test-requirements.txt` | Test deps | |
|||
| `requirements/dev-requirements.txt` | Dev/lint deps | |
|||
@ -0,0 +1,107 @@ |
|||
# SAPL Docker Build |
|||
|
|||
## Building locally |
|||
|
|||
### 1. Prerequisites |
|||
|
|||
- Docker 23+ with BuildKit enabled (default since Docker 23) |
|||
- A free [MaxMind account](https://www.maxmind.com/en/geolite2/signup) with a license key |
|||
|
|||
### 2. Set your MaxMind license key |
|||
|
|||
Add the key to the project root `.env` file (already gitignored): |
|||
|
|||
``` |
|||
MAXMIND_LICENSE_KEY=your_key_here |
|||
``` |
|||
|
|||
The key is used **only at build time** to download the `GeoLite2-ASN.mmdb` database for |
|||
nginx ASN-based bot blocking. It is injected via a BuildKit secret and is **never stored |
|||
in any image layer** — it will not appear in `docker history` or any registry push. |
|||
|
|||
### 3. Build the image |
|||
|
|||
```bash |
|||
docker build \ |
|||
--secret id=maxmind_key,src=.env \ |
|||
-f docker/Dockerfile \ |
|||
-t sapl:local \ |
|||
. |
|||
``` |
|||
|
|||
Run from the **project root** (not from inside `docker/`), so the build context includes |
|||
the full source tree. |
|||
|
|||
#### Optional build args |
|||
|
|||
| Arg | Default | Description | |
|||
|---|---|---| |
|||
| `WITH_NGINX` | `1` | Include nginx in the image | |
|||
| `WITH_GRAPHVIZ` | `1` | Include Graphviz | |
|||
| `WITH_POPPLER` | `1` | Include Poppler (PDF utilities) | |
|||
| `WITH_PSQL_CLIENT` | `1` | Include `psql` client | |
|||
|
|||
Example — build without Graphviz: |
|||
|
|||
```bash |
|||
docker build \ |
|||
--secret id=maxmind_key,src=.env \ |
|||
--build-arg WITH_GRAPHVIZ=0 \ |
|||
-f docker/Dockerfile \ |
|||
-t sapl:local \ |
|||
. |
|||
``` |
|||
|
|||
### 4. If the MaxMind key is not provided |
|||
|
|||
The build will succeed but nginx will log an error on startup because |
|||
`/etc/nginx/geoip/GeoLite2-ASN.mmdb` will be missing. ASN-based bot blocking will |
|||
be inactive. All other Phase 0 mitigations (UA blocklist, rate limits, ETags) still apply. |
|||
|
|||
You can mount the database file at runtime as a workaround: |
|||
|
|||
```bash |
|||
docker run \ |
|||
-v /path/to/GeoLite2-ASN.mmdb:/etc/nginx/geoip/GeoLite2-ASN.mmdb:ro \ |
|||
sapl:local |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Production — Harbor |
|||
|
|||
Official images are built and pushed through **Harbor**. Before the next release, configure |
|||
the MaxMind license key as a build secret in the Harbor / CI pipeline: |
|||
|
|||
1. Add `MAXMIND_LICENSE_KEY` as a **masked CI/CD secret** in the Harbor build project |
|||
(do not put it in any Helm values file or ConfigMap). |
|||
2. Pass it to the build step: |
|||
```bash |
|||
docker build \ |
|||
--secret id=maxmind_key,env=MAXMIND_LICENSE_KEY \ |
|||
-f docker/Dockerfile \ |
|||
-t harbor.your-registry/sapl/sapl:$VERSION \ |
|||
. |
|||
``` |
|||
Note: `env=` variant reads the secret from an environment variable instead of a file — |
|||
useful in CI where `.env` files are not present. |
|||
3. Push as normal — the key will not be present in the pushed image. |
|||
|
|||
### Keeping GeoLite2-ASN up to date |
|||
|
|||
MaxMind updates the database every Tuesday. On production hosts, install the weekly refresh |
|||
cron (run as root): |
|||
|
|||
```bash |
|||
cat > /etc/cron.weekly/update-geoip << 'EOF' |
|||
#!/bin/bash |
|||
MAXMIND_KEY="$(kubectl get secret sapl-build-secrets -n interlegis-infra \ |
|||
-o jsonpath='{.data.MAXMIND_LICENSE_KEY}' | base64 -d)" |
|||
curl -fsSL \ |
|||
"https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=${MAXMIND_KEY}&suffix=tar.gz" \ |
|||
| tar -xz -C /tmp --wildcards '*.mmdb' |
|||
mv /tmp/GeoLite2-ASN_*/GeoLite2-ASN.mmdb /etc/nginx/geoip/GeoLite2-ASN.mmdb |
|||
nginx -s reload |
|||
EOF |
|||
chmod +x /etc/cron.weekly/update-geoip |
|||
``` |
|||
@ -0,0 +1,60 @@ |
|||
-- blocklist.lua: Redis-backed early IP rejection before Gunicorn. |
|||
-- ASN and User-Agent blocking are handled upstream by nginx if() blocks. |
|||
-- |
|||
-- Checks (Redis DB 1, read-only): |
|||
-- 1. ngx.shared.ip_prefix_blocked — in-process prefix cache (60s refresh, no Redis I/O) |
|||
-- 2. GET rl:ip:{ip}:blocked — global IP block |
|||
-- 3. GET rl:api:ns:{ns}:ip:{ip}:blocked — per-tenant API block (/api/ only) |
|||
-- |
|||
-- Checks 2+3 are pipelined in one Redis round trip. |
|||
-- On Redis failure: fail-open (request passes to Django). |
|||
|
|||
local url = os.getenv("REDIS_URL") or "redis://127.0.0.1:6379" |
|||
local REDIS_HOST, port_str = url:match("redis://([^:/]+):(%d+)") |
|||
if not REDIS_HOST then REDIS_HOST = url:match("redis://([^:/]+)") or "127.0.0.1" end |
|||
local REDIS_PORT = tonumber(port_str) or 6379 |
|||
|
|||
local POD_NS = os.getenv("POD_NAMESPACE") or "" |
|||
local ip = ngx.var.remote_addr |
|||
local is_api = ngx.var.uri:sub(1, 5) == "/api/" |
|||
|
|||
-- Build 4 prefix candidates for ip e.g. '203.0.113.42': |
|||
-- '203.', '203.0.', '203.0.113.', '203.0.113.42' |
|||
-- Mirrors Django's _is_ip_prefix_blocked normalisation. |
|||
local parts = {} |
|||
for p in ip:gmatch("[^.]+") do parts[#parts+1] = p end |
|||
local p1 = parts[1] .. "." |
|||
local p2 = parts[1] .. "." .. parts[2] .. "." |
|||
local p3 = parts[1] .. "." .. parts[2] .. "." .. parts[3] .. "." |
|||
|
|||
local function return_429() |
|||
ngx.status = 429 |
|||
ngx.header["Retry-After"] = "300" |
|||
ngx.header["Content-Type"] = "application/json" |
|||
ngx.say('{"detail":"Too Many Requests"}') |
|||
return ngx.exit(429) |
|||
end |
|||
|
|||
-- 1. Prefix check (shared dict — zero Redis I/O per request). |
|||
local dict = ngx.shared.ip_prefix_blocked |
|||
if dict:get(p1) or dict:get(p2) or dict:get(p3) or dict:get(ip) then |
|||
return return_429() |
|||
end |
|||
|
|||
-- 2+3. Pipeline both STRING block checks in one Redis round trip. |
|||
local red = require("resty.redis"):new() |
|||
red:set_timeout(200) |
|||
local ok = red:connect(REDIS_HOST, REDIS_PORT) |
|||
if not ok then return end -- fail-open |
|||
|
|||
red:select(1) |
|||
red:init_pipeline() |
|||
red:get("rl:ip:" .. ip .. ":blocked") |
|||
red:get("rl:api:ns:" .. POD_NS .. ":ip:" .. ip .. ":blocked") |
|||
local res = red:commit_pipeline() |
|||
red:set_keepalive(10000, 1) |
|||
|
|||
if not res then return end -- fail-open on pipeline error |
|||
|
|||
if res[1] == "1" then return return_429() end |
|||
if is_api and res[2] == "1" then return return_429() end |
|||
@ -0,0 +1,741 @@ |
|||
-- Copyright (C) Yichun Zhang (agentzh) |
|||
|
|||
|
|||
local sub = string.sub |
|||
local byte = string.byte |
|||
local tab_insert = table.insert |
|||
local tab_remove = table.remove |
|||
local tcp = ngx.socket.tcp |
|||
local null = ngx.null |
|||
local ipairs = ipairs |
|||
local type = type |
|||
local pairs = pairs |
|||
local unpack = unpack |
|||
local setmetatable = setmetatable |
|||
local tonumber = tonumber |
|||
local tostring = tostring |
|||
local rawget = rawget |
|||
local select = select |
|||
local tb_clear = require "table.clear" |
|||
--local error = error |
|||
|
|||
|
|||
local ok, new_tab = pcall(require, "table.new") |
|||
if not ok or type(new_tab) ~= "function" then |
|||
new_tab = function (narr, nrec) return {} end |
|||
end |
|||
|
|||
local tab_pool_len = 0 |
|||
local tab_pool = new_tab(16, 0) |
|||
local _M = new_tab(0, 55) |
|||
|
|||
_M._VERSION = '0.32' |
|||
|
|||
|
|||
local common_cmds = { |
|||
"get", "set", "mget", "mset", |
|||
"del", "incr", "decr", -- Strings |
|||
"llen", "lindex", "lpop", "lpush", |
|||
"lrange", "linsert", -- Lists |
|||
"hexists", "hget", "hset", "hmget", |
|||
--[[ "hmset", ]] "hdel", -- Hashes |
|||
"smembers", "sismember", "sadd", "srem", |
|||
"sdiff", "sinter", "sunion", -- Sets |
|||
"zrange", "zrangebyscore", "zrank", "zadd", |
|||
"zrem", "zincrby", -- Sorted Sets |
|||
"auth", "eval", "expire", "script", |
|||
"sort" -- Others |
|||
} |
|||
|
|||
|
|||
local sub_commands = { |
|||
"subscribe", "psubscribe" |
|||
} |
|||
|
|||
local blocking_commands = { |
|||
"blpop", "brpop" |
|||
} |
|||
|
|||
local unsub_commands = { |
|||
"unsubscribe", "punsubscribe" |
|||
} |
|||
|
|||
|
|||
local mt = { __index = _M } |
|||
|
|||
|
|||
local function get_tab_from_pool() |
|||
if tab_pool_len > 0 then |
|||
tab_pool_len = tab_pool_len - 1 |
|||
return tab_pool[tab_pool_len + 1] |
|||
end |
|||
|
|||
return new_tab(24, 0) -- one field takes 5 slots |
|||
end |
|||
|
|||
|
|||
local function put_tab_into_pool(tab) |
|||
if tab_pool_len >= 32 then |
|||
return |
|||
end |
|||
|
|||
tb_clear(tab) |
|||
tab_pool_len = tab_pool_len + 1 |
|||
tab_pool[tab_pool_len] = tab |
|||
end |
|||
|
|||
|
|||
function _M.new(self) |
|||
local sock, err = tcp() |
|||
if not sock then |
|||
return nil, err |
|||
end |
|||
local redis = setmetatable({ _sock = sock, |
|||
_subscribed = false, |
|||
_n_channel = { |
|||
unsubscribe = 0, |
|||
punsubscribe = 0, |
|||
}, |
|||
}, mt) |
|||
return redis |
|||
end |
|||
|
|||
|
|||
function _M.register_module_prefix(mod) |
|||
_M[mod] = function(self) |
|||
self._module_prefix = mod |
|||
return self |
|||
end |
|||
end |
|||
|
|||
|
|||
function _M.set_timeout(self, timeout) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
error("not initialized", 2) |
|||
return |
|||
end |
|||
|
|||
sock:settimeout(timeout) |
|||
end |
|||
|
|||
|
|||
function _M.set_timeouts(self, connect_timeout, send_timeout, read_timeout) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
error("not initialized", 2) |
|||
return |
|||
end |
|||
|
|||
sock:settimeouts(connect_timeout, send_timeout, read_timeout) |
|||
end |
|||
|
|||
|
|||
function _M.connect(self, host, port_or_opts, opts) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
local unix |
|||
|
|||
do |
|||
local typ = type(host) |
|||
if typ ~= "string" then |
|||
error("bad argument #1 host: string expected, got " .. typ, 2) |
|||
end |
|||
|
|||
if sub(host, 1, 5) == "unix:" then |
|||
unix = true |
|||
end |
|||
|
|||
if unix then |
|||
typ = type(port_or_opts) |
|||
if port_or_opts ~= nil and typ ~= "table" then |
|||
error("bad argument #2 opts: nil or table expected, got " .. |
|||
typ, 2) |
|||
end |
|||
|
|||
else |
|||
typ = type(port_or_opts) |
|||
if typ ~= "number" then |
|||
port_or_opts = tonumber(port_or_opts) |
|||
if port_or_opts == nil then |
|||
error("bad argument #2 port: number expected, got " .. |
|||
typ, 2) |
|||
end |
|||
end |
|||
|
|||
if opts ~= nil then |
|||
typ = type(opts) |
|||
if typ ~= "table" then |
|||
error("bad argument #3 opts: nil or table expected, got " .. |
|||
typ, 2) |
|||
end |
|||
end |
|||
end |
|||
|
|||
end |
|||
|
|||
self._subscribed = false |
|||
|
|||
local ok, err |
|||
|
|||
if unix then |
|||
-- second argument of sock:connect() cannot be nil |
|||
if port_or_opts ~= nil then |
|||
ok, err = sock:connect(host, port_or_opts) |
|||
opts = port_or_opts |
|||
else |
|||
ok, err = sock:connect(host) |
|||
end |
|||
else |
|||
ok, err = sock:connect(host, port_or_opts, opts) |
|||
end |
|||
|
|||
if not ok then |
|||
return ok, err |
|||
end |
|||
|
|||
if opts and opts.ssl then |
|||
ok, err = sock:sslhandshake(false, opts.server_name, opts.ssl_verify) |
|||
if not ok then |
|||
return ok, "failed to do ssl handshake: " .. err |
|||
end |
|||
end |
|||
|
|||
return ok, err |
|||
end |
|||
|
|||
|
|||
function _M.set_keepalive(self, ...) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
if rawget(self, "_subscribed") then |
|||
return nil, "subscribed state" |
|||
end |
|||
|
|||
return sock:setkeepalive(...) |
|||
end |
|||
|
|||
|
|||
function _M.get_reused_times(self) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
return sock:getreusedtimes() |
|||
end |
|||
|
|||
|
|||
local function close(self) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
return sock:close() |
|||
end |
|||
_M.close = close |
|||
|
|||
|
|||
local function _read_reply(self, sock) |
|||
local line, err = sock:receive() |
|||
if not line then |
|||
if err == "timeout" and not rawget(self, "_subscribed") and not rawget(self, "_blocking") then |
|||
sock:close() |
|||
end |
|||
return nil, err |
|||
end |
|||
|
|||
local prefix = byte(line) |
|||
|
|||
if prefix == 36 then -- char '$' |
|||
-- print("bulk reply") |
|||
|
|||
local size = tonumber(sub(line, 2)) |
|||
if size < 0 then |
|||
return null |
|||
end |
|||
|
|||
local data, err = sock:receive(size) |
|||
if not data then |
|||
if err == "timeout" then |
|||
sock:close() |
|||
end |
|||
return nil, err |
|||
end |
|||
|
|||
local dummy, err = sock:receive(2) -- ignore CRLF |
|||
if not dummy then |
|||
if err == "timeout" then |
|||
sock:close() |
|||
end |
|||
return nil, err |
|||
end |
|||
|
|||
return data |
|||
|
|||
elseif prefix == 43 then -- char '+' |
|||
-- print("status reply") |
|||
|
|||
return sub(line, 2) |
|||
|
|||
elseif prefix == 42 then -- char '*' |
|||
local n = tonumber(sub(line, 2)) |
|||
|
|||
-- print("multi-bulk reply: ", n) |
|||
if n < 0 then |
|||
return null |
|||
end |
|||
|
|||
local vals = new_tab(n, 0) |
|||
local nvals = 0 |
|||
for i = 1, n do |
|||
local res, err = _read_reply(self, sock) |
|||
if res then |
|||
nvals = nvals + 1 |
|||
vals[nvals] = res |
|||
|
|||
elseif res == nil then |
|||
return nil, err |
|||
|
|||
else |
|||
-- be a valid redis error value |
|||
nvals = nvals + 1 |
|||
vals[nvals] = {false, err} |
|||
end |
|||
end |
|||
|
|||
return vals |
|||
|
|||
elseif prefix == 58 then -- char ':' |
|||
-- print("integer reply") |
|||
return tonumber(sub(line, 2)) |
|||
|
|||
elseif prefix == 45 then -- char '-' |
|||
-- print("error reply: ", n) |
|||
|
|||
return false, sub(line, 2) |
|||
|
|||
else |
|||
-- when `line` is an empty string, `prefix` will be equal to nil. |
|||
return nil, "unknown prefix: \"" .. tostring(prefix) .. "\"" |
|||
end |
|||
end |
|||
|
|||
|
|||
local function _gen_req(args) |
|||
local nargs = #args |
|||
|
|||
local req = get_tab_from_pool() |
|||
req[1] = "*" |
|||
req[2] = nargs |
|||
req[3] = "\r\n" |
|||
local nbits = 4 |
|||
|
|||
for i = 1, nargs do |
|||
local arg = args[i] |
|||
if type(arg) ~= "string" then |
|||
arg = tostring(arg) |
|||
end |
|||
|
|||
req[nbits] = "$" |
|||
req[nbits + 1] = #arg |
|||
req[nbits + 2] = "\r\n" |
|||
req[nbits + 3] = arg |
|||
req[nbits + 4] = "\r\n" |
|||
|
|||
nbits = nbits + 5 |
|||
end |
|||
|
|||
-- it is much faster to do string concatenation on the C land |
|||
-- in real world (large number of strings in the Lua VM) |
|||
return req |
|||
end |
|||
|
|||
|
|||
local function _check_msg(self, res) |
|||
return rawget(self, "_subscribed") and |
|||
type(res) == "table" and (res[1] == "message" or res[1] == "pmessage") |
|||
end |
|||
|
|||
|
|||
local function _do_cmd(self, ...) |
|||
local args = {...} |
|||
|
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
local req = _gen_req(args) |
|||
|
|||
local reqs = rawget(self, "_reqs") |
|||
if reqs then |
|||
reqs[#reqs + 1] = req |
|||
return |
|||
end |
|||
|
|||
-- print("request: ", table.concat(req)) |
|||
|
|||
local bytes, err = sock:send(req) |
|||
put_tab_into_pool(req) |
|||
|
|||
if not bytes then |
|||
return nil, err |
|||
end |
|||
|
|||
local res, err = _read_reply(self, sock) |
|||
while _check_msg(self, res) do |
|||
if rawget(self, "_buffered_msg") == nil then |
|||
self._buffered_msg = new_tab(1, 0) |
|||
end |
|||
|
|||
tab_insert(self._buffered_msg, res) |
|||
res, err = _read_reply(self, sock) |
|||
end |
|||
|
|||
return res, err |
|||
end |
|||
|
|||
|
|||
local function _check_unsubscribed(self, res) |
|||
if type(res) == "table" |
|||
and (res[1] == "unsubscribe" or res[1] == "punsubscribe") |
|||
then |
|||
self._n_channel[res[1]] = self._n_channel[res[1]] - 1 |
|||
|
|||
local buffered_msg = rawget(self, "_buffered_msg") |
|||
if buffered_msg then |
|||
-- remove messages of unsubscribed channel |
|||
local msg_type = |
|||
(res[1] == "punsubscribe") and "pmessage" or "message" |
|||
local j = 1 |
|||
for _, msg in ipairs(buffered_msg) do |
|||
if msg[1] == msg_type and msg[2] ~= res[2] then |
|||
-- move messages to overwrite the removed ones |
|||
buffered_msg[j] = msg |
|||
j = j + 1 |
|||
end |
|||
end |
|||
|
|||
-- clear remain messages |
|||
for i = j, #buffered_msg do |
|||
buffered_msg[i] = nil |
|||
end |
|||
|
|||
if #buffered_msg == 0 then |
|||
self._buffered_msg = nil |
|||
end |
|||
end |
|||
|
|||
if res[3] == 0 then |
|||
-- all channels are unsubscribed |
|||
self._subscribed = false |
|||
end |
|||
end |
|||
end |
|||
|
|||
|
|||
local function _check_subscribed(self, res) |
|||
if type(res) == "table" |
|||
and (res[1] == "subscribe" or res[1] == "psubscribe") |
|||
then |
|||
if res[1] == "subscribe" then |
|||
self._n_channel.unsubscribe = self._n_channel.unsubscribe + 1 |
|||
|
|||
elseif res[1] == "psubscribe" then |
|||
self._n_channel.punsubscribe = self._n_channel.punsubscribe + 1 |
|||
end |
|||
end |
|||
end |
|||
|
|||
|
|||
function _M.read_reply(self) |
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
if not rawget(self, "_subscribed") then |
|||
return nil, "not subscribed" |
|||
end |
|||
|
|||
local buffered_msg = rawget(self, "_buffered_msg") |
|||
if buffered_msg then |
|||
local msg = buffered_msg[1] |
|||
tab_remove(buffered_msg, 1) |
|||
|
|||
if #buffered_msg == 0 then |
|||
self._buffered_msg = nil |
|||
end |
|||
|
|||
return msg |
|||
end |
|||
|
|||
local res, err = _read_reply(self, sock) |
|||
_check_unsubscribed(self, res) |
|||
|
|||
return res, err |
|||
end |
|||
|
|||
|
|||
local function do_cmd(self, cmd, ...) |
|||
local module_prefix = rawget(self, "_module_prefix") |
|||
if module_prefix then |
|||
self._module_prefix = nil |
|||
return _do_cmd(self, module_prefix .. "." .. cmd, ...) |
|||
end |
|||
|
|||
return _do_cmd(self, cmd, ...) |
|||
end |
|||
|
|||
|
|||
for i = 1, #common_cmds do |
|||
local cmd = common_cmds[i] |
|||
|
|||
_M[cmd] = |
|||
function (self, ...) |
|||
return do_cmd(self, cmd, ...) |
|||
end |
|||
end |
|||
|
|||
for i = 1, #blocking_commands do |
|||
local cmd = blocking_commands[i] |
|||
|
|||
_M[cmd] = |
|||
function (self, ...) |
|||
if not rawget(self, "_blocking") then |
|||
self._blocking = true |
|||
end |
|||
return do_cmd(self, cmd, ...) |
|||
end |
|||
end |
|||
|
|||
local function handle_subscribe_result(self, cmd, nargs, res) |
|||
local err |
|||
_check_subscribed(self, res) |
|||
|
|||
if nargs <= 1 then |
|||
return res |
|||
end |
|||
|
|||
local results = new_tab(nargs, 0) |
|||
results[1] = res |
|||
local sock = rawget(self, "_sock") |
|||
|
|||
for i = 2, nargs do |
|||
res, err = _read_reply(self, sock) |
|||
if not res then |
|||
return nil, err |
|||
end |
|||
|
|||
_check_subscribed(self, res) |
|||
results[i] = res |
|||
end |
|||
|
|||
return results |
|||
end |
|||
|
|||
for i = 1, #sub_commands do |
|||
local cmd = sub_commands[i] |
|||
|
|||
_M[cmd] = |
|||
function (self, ...) |
|||
if not rawget(self, "_subscribed") then |
|||
self._subscribed = true |
|||
end |
|||
|
|||
local nargs = select("#", ...) |
|||
|
|||
local res, err = _do_cmd(self, cmd, ...) |
|||
if not res then |
|||
return nil, err |
|||
end |
|||
|
|||
return handle_subscribe_result(self, cmd, nargs, res) |
|||
end |
|||
end |
|||
|
|||
|
|||
local function handle_unsubscribe_result(self, cmd, nargs, res) |
|||
local err |
|||
_check_unsubscribed(self, res) |
|||
|
|||
if self._n_channel[cmd] == 0 or nargs == 1 then |
|||
return res |
|||
end |
|||
|
|||
local results = new_tab(nargs, 0) |
|||
results[1] = res |
|||
local sock = rawget(self, "_sock") |
|||
local i = 2 |
|||
|
|||
while nargs == 0 or i <= nargs do |
|||
res, err = _read_reply(self, sock) |
|||
if not res then |
|||
return nil, err |
|||
end |
|||
|
|||
results[i] = res |
|||
i = i + 1 |
|||
|
|||
_check_unsubscribed(self, res) |
|||
if self._n_channel[cmd] == 0 then |
|||
-- exit the loop for unsubscribe() call |
|||
break |
|||
end |
|||
end |
|||
|
|||
return results |
|||
end |
|||
|
|||
for i = 1, #unsub_commands do |
|||
local cmd = unsub_commands[i] |
|||
|
|||
_M[cmd] = |
|||
function (self, ...) |
|||
-- assume all channels are unsubscribed by only one time |
|||
if not rawget(self, "_subscribed") then |
|||
return nil, "not subscribed" |
|||
end |
|||
|
|||
local nargs = select("#", ...) |
|||
|
|||
local res, err = _do_cmd(self, cmd, ...) |
|||
if not res then |
|||
return nil, err |
|||
end |
|||
|
|||
return handle_unsubscribe_result(self, cmd, nargs, res) |
|||
end |
|||
end |
|||
|
|||
|
|||
function _M.hmset(self, hashname, ...) |
|||
if select('#', ...) == 1 then |
|||
local t = select(1, ...) |
|||
|
|||
local n = 0 |
|||
for k, v in pairs(t) do |
|||
n = n + 2 |
|||
end |
|||
|
|||
local array = new_tab(n, 0) |
|||
|
|||
local i = 0 |
|||
for k, v in pairs(t) do |
|||
array[i + 1] = k |
|||
array[i + 2] = v |
|||
i = i + 2 |
|||
end |
|||
-- print("key", hashname) |
|||
return _do_cmd(self, "hmset", hashname, unpack(array)) |
|||
end |
|||
|
|||
-- backwards compatibility |
|||
return _do_cmd(self, "hmset", hashname, ...) |
|||
end |
|||
|
|||
|
|||
function _M.init_pipeline(self, n) |
|||
self._reqs = new_tab(n or 4, 0) |
|||
end |
|||
|
|||
|
|||
function _M.cancel_pipeline(self) |
|||
self._reqs = nil |
|||
end |
|||
|
|||
|
|||
function _M.commit_pipeline(self) |
|||
local reqs = rawget(self, "_reqs") |
|||
if not reqs then |
|||
return nil, "no pipeline" |
|||
end |
|||
|
|||
self._reqs = nil |
|||
|
|||
local sock = rawget(self, "_sock") |
|||
if not sock then |
|||
return nil, "not initialized" |
|||
end |
|||
|
|||
local bytes, err = sock:send(reqs) |
|||
for _, req in ipairs(reqs) do |
|||
put_tab_into_pool(req) |
|||
end |
|||
|
|||
if not bytes then |
|||
return nil, err |
|||
end |
|||
|
|||
local nvals = 0 |
|||
local nreqs = #reqs |
|||
local vals = new_tab(nreqs, 0) |
|||
for i = 1, nreqs do |
|||
local res, err = _read_reply(self, sock) |
|||
if res then |
|||
nvals = nvals + 1 |
|||
vals[nvals] = res |
|||
|
|||
elseif res == nil then |
|||
if err == "timeout" then |
|||
close(self) |
|||
end |
|||
return nil, err |
|||
|
|||
else |
|||
-- be a valid redis error value |
|||
nvals = nvals + 1 |
|||
vals[nvals] = {false, err} |
|||
end |
|||
end |
|||
|
|||
return vals |
|||
end |
|||
|
|||
|
|||
function _M.array_to_hash(self, t) |
|||
local n = #t |
|||
-- print("n = ", n) |
|||
local h = new_tab(0, n / 2) |
|||
for i = 1, n, 2 do |
|||
h[t[i]] = t[i + 1] |
|||
end |
|||
return h |
|||
end |
|||
|
|||
|
|||
-- this method is deperate since we already do lazy method generation. |
|||
function _M.add_commands(...) |
|||
local cmds = {...} |
|||
for i = 1, #cmds do |
|||
local cmd = cmds[i] |
|||
_M[cmd] = |
|||
function (self, ...) |
|||
return _do_cmd(self, cmd, ...) |
|||
end |
|||
end |
|||
end |
|||
|
|||
|
|||
setmetatable(_M, {__index = function(self, cmd) |
|||
local method = |
|||
function (self, ...) |
|||
return do_cmd(self, cmd, ...) |
|||
end |
|||
|
|||
-- cache the lazily generated method in our |
|||
-- module table |
|||
_M[cmd] = method |
|||
return method |
|||
end}) |
|||
|
|||
|
|||
return _M |
|||
Binary file not shown.
@ -0,0 +1,73 @@ |
|||
#!/usr/bin/env bash |
|||
# update_geoip.sh — download / refresh GeoLite2-ASN.mmdb |
|||
# |
|||
# Run this script before building a new Docker image so the image bundles |
|||
# an up-to-date MaxMind ASN database. The .mmdb binary is git-ignored; |
|||
# only this script is tracked. |
|||
# |
|||
# Usage: |
|||
# # Option 1 — key in environment |
|||
# MAXMIND_LICENSE_KEY=your_key bash docker/geoip/update_geoip.sh |
|||
# |
|||
# # Option 2 — key in project .env file |
|||
# bash docker/geoip/update_geoip.sh |
|||
# |
|||
# The script writes GeoLite2-ASN.mmdb to the same directory as itself so |
|||
# the Dockerfile COPY step can find it at docker/geoip/GeoLite2-ASN.mmdb. |
|||
# |
|||
# Suggested automation: run via a host cron job or CI pipeline step |
|||
# before triggering a docker build, e.g.: |
|||
# |
|||
# # /etc/cron.weekly/update-sapl-geoip |
|||
# #!/bin/bash |
|||
# cd /path/to/sapl && bash docker/geoip/update_geoip.sh |
|||
|
|||
set -Eeuo pipefail |
|||
|
|||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
|||
OUT_FILE="$SCRIPT_DIR/GeoLite2-ASN.mmdb" |
|||
|
|||
# ── Resolve the license key ──────────────────────────────────────────────── |
|||
if [[ -z "${MAXMIND_LICENSE_KEY:-}" ]]; then |
|||
# Try the project .env (two directories up from docker/geoip/) |
|||
ENV_FILE="$(dirname "$(dirname "$SCRIPT_DIR")")/.env" |
|||
if [[ -f "$ENV_FILE" ]]; then |
|||
MAXMIND_LICENSE_KEY="$(grep -E '^MAXMIND_LICENSE_KEY=' "$ENV_FILE" 2>/dev/null \ |
|||
| cut -d= -f2- | tr -d '[:space:]' || true)" |
|||
fi |
|||
fi |
|||
|
|||
if [[ -z "${MAXMIND_LICENSE_KEY:-}" ]]; then |
|||
echo "ERROR: MAXMIND_LICENSE_KEY is not set." >&2 |
|||
echo " Set it in the environment or add MAXMIND_LICENSE_KEY=<key> to .env" >&2 |
|||
exit 1 |
|||
fi |
|||
|
|||
# ── Download ─────────────────────────────────────────────────────────────── |
|||
URL="https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=${MAXMIND_LICENSE_KEY}&suffix=tar.gz" |
|||
|
|||
echo "[geoip] Downloading GeoLite2-ASN from MaxMind..." |
|||
tmpdir="$(mktemp -d)" |
|||
trap 'rm -rf "$tmpdir"' EXIT |
|||
|
|||
curl -fsSL --max-time 60 "$URL" | tar -xz --strip-components=1 -C "$tmpdir" |
|||
mv "$tmpdir"/GeoLite2-ASN.mmdb "$OUT_FILE" |
|||
|
|||
echo "[geoip] Saved: $OUT_FILE" |
|||
echo "[geoip] Build date: $(python3 -c " |
|||
import struct, datetime, pathlib |
|||
data = pathlib.Path('$OUT_FILE').read_bytes() |
|||
# MaxMind DB build epoch is in the last 16 bytes of the metadata section |
|||
marker = b'\xab\xcd\xefMaxMind.com' |
|||
idx = data.rfind(marker) |
|||
if idx >= 0: |
|||
# search for 'build_epoch' key nearby |
|||
chunk = data[idx:idx+512] |
|||
pos = chunk.find(b'build_epoch') |
|||
if pos >= 0: |
|||
val_start = pos + len(b'build_epoch') + 1 |
|||
epoch = struct.unpack('>Q', chunk[val_start+1:val_start+9])[0] |
|||
print(datetime.datetime.utcfromtimestamp(epoch).strftime('%Y-%m-%d')) |
|||
exit() |
|||
print('unknown') |
|||
" 2>/dev/null || echo "unknown")" |
|||
@ -0,0 +1,36 @@ |
|||
apiVersion: v1 |
|||
kind: ConfigMap |
|||
metadata: |
|||
name: redis-config |
|||
namespace: sapl-redis |
|||
data: |
|||
redis.conf: | |
|||
save "" |
|||
appendonly no |
|||
|
|||
maxmemory 5gb |
|||
maxmemory-policy allkeys-lru |
|||
maxmemory-samples 10 |
|||
|
|||
maxclients 20000 |
|||
tcp-backlog 511 |
|||
timeout 300 |
|||
tcp-keepalive 60 |
|||
|
|||
hz 20 |
|||
lazyfree-lazy-eviction yes |
|||
lazyfree-lazy-expire yes |
|||
lazyfree-lazy-server-del yes |
|||
|
|||
slowlog-log-slower-than 10000 |
|||
slowlog-max-len 256 |
|||
latency-monitor-threshold 10 |
|||
|
|||
bind 0.0.0.0 |
|||
protected-mode no |
|||
# DB0: cache DB1: rate limiter DB2: channels (future) |
|||
databases 2 |
|||
|
|||
activedefrag yes |
|||
active-defrag-ignore-bytes 100mb |
|||
active-defrag-threshold-lower 10 |
|||
@ -0,0 +1,48 @@ |
|||
apiVersion: apps/v1 |
|||
kind: Deployment |
|||
metadata: |
|||
name: sapl-redis |
|||
namespace: sapl-redis |
|||
labels: |
|||
app: sapl-redis |
|||
spec: |
|||
replicas: 1 |
|||
selector: |
|||
matchLabels: |
|||
app: sapl-redis |
|||
template: |
|||
metadata: |
|||
labels: |
|||
app: sapl-redis |
|||
spec: |
|||
containers: |
|||
- name: redis |
|||
image: redis:7-alpine |
|||
command: ["redis-server", "/etc/redis/redis.conf"] |
|||
resources: |
|||
requests: |
|||
memory: "1Gi" |
|||
cpu: "250m" |
|||
limits: |
|||
memory: "6Gi" |
|||
cpu: "1000m" |
|||
ports: |
|||
- containerPort: 6379 |
|||
livenessProbe: |
|||
exec: |
|||
command: ["redis-cli", "ping"] |
|||
initialDelaySeconds: 10 |
|||
periodSeconds: 15 |
|||
failureThreshold: 3 |
|||
readinessProbe: |
|||
exec: |
|||
command: ["redis-cli", "ping"] |
|||
initialDelaySeconds: 5 |
|||
periodSeconds: 10 |
|||
volumeMounts: |
|||
- name: redis-config |
|||
mountPath: /etc/redis |
|||
volumes: |
|||
- name: redis-config |
|||
configMap: |
|||
name: redis-config |
|||
@ -0,0 +1,15 @@ |
|||
apiVersion: v1 |
|||
kind: Service |
|||
metadata: |
|||
name: redis |
|||
namespace: sapl-redis |
|||
labels: |
|||
app: sapl-redis |
|||
spec: |
|||
selector: |
|||
app: sapl-redis |
|||
ports: |
|||
- name: redis |
|||
port: 6379 |
|||
targetPort: 6379 |
|||
type: ClusterIP |
|||
@ -0,0 +1,169 @@ |
|||
#!/usr/bin/env python3 |
|||
""" |
|||
redis_inject_test_data.py — inject synthetic rate-limiter entries into Redis. |
|||
|
|||
Purpose: validate that RateLimitMiddleware reads the expected key schema, |
|||
that Redis CLI / RedisInsight shows the right structure, and that blocking |
|||
logic fires correctly without waiting for real traffic. |
|||
|
|||
Usage: |
|||
# Against docker-compose Redis (default) |
|||
python3 docker/scripts/redis_inject_test_data.py |
|||
|
|||
# Against a different host/port |
|||
REDIS_URL=redis://localhost:6379 python3 docker/scripts/redis_inject_test_data.py |
|||
|
|||
# Clear all synthetic keys written by a previous run |
|||
CLEAR=1 python3 docker/scripts/redis_inject_test_data.py |
|||
|
|||
Key schema (DB 1 — rate limiter): |
|||
rl:ip:{ip}:reqs INCR counter — anonymous request count (TTL 60s) |
|||
rl:ip:{ip}:blocked string "1" — IP hard-blocked (TTL 300s) |
|||
rl:{ns}:user:{uid}:reqs INCR counter — auth user request count (TTL 60s) |
|||
rl:{ns}:user:{uid}:blocked string "1" — user hard-blocked (TTL 300s) |
|||
rl:{ns}:ip:{ip}:w:{bucket} INCR — namespace/IP sliding window (TTL 120s) |
|||
""" |
|||
|
|||
import os |
|||
import sys |
|||
import time |
|||
from decouple import config |
|||
|
|||
# ── dependency check ────────────────────────────────────────────────────── |
|||
try: |
|||
import redis |
|||
except ImportError: |
|||
print("ERROR: redis-py not installed. Run: pip install redis", file=sys.stderr) |
|||
sys.exit(1) |
|||
|
|||
# ── config ──────────────────────────────────────────────────────────────── |
|||
REDIS_URL = config("REDIS_URL", default="redis://localhost:6379") |
|||
RATELIMIT_DB = 1 # DB1 is the rate-limiter database |
|||
CLEAR = config("CLEAR", default="0").lower() in ("1", "true", "yes") |
|||
|
|||
# Synthetic values — tweak to exercise different code paths |
|||
NAMESPACE = "sapl" # POD_NAMESPACE value (hostname or k8s namespace) |
|||
ANON_WINDOW = 60 # seconds — must match settings.RATE_LIMITER_RATE period |
|||
AUTH_WINDOW = 60 |
|||
BLOCK_TTL = 300 |
|||
|
|||
TEST_IPS = [ |
|||
"203.0.113.1", # below threshold (20 reqs) |
|||
"203.0.113.2", # AT threshold (35 reqs — should trigger block) |
|||
"203.0.113.3", # already blocked |
|||
"203.0.113.4", # namespace/window counter near threshold |
|||
] |
|||
|
|||
TEST_USERS = [ |
|||
{"uid": "42", "reqs": 50, "blocked": False}, # normal auth user |
|||
{"uid": "99", "reqs": 120, "blocked": False}, # AT auth threshold |
|||
{"uid": "7", "reqs": 10, "blocked": True}, # pre-blocked user |
|||
] |
|||
|
|||
# ── helpers ─────────────────────────────────────────────────────────────── |
|||
|
|||
def key_ip_reqs(ip): |
|||
return f"rl:ip:{ip}:reqs" |
|||
|
|||
def key_ip_blocked(ip): |
|||
return f"rl:ip:{ip}:blocked" |
|||
|
|||
def key_user_reqs(ns, uid): |
|||
return f"rl:{ns}:user:{uid}:reqs" |
|||
|
|||
def key_user_blocked(ns, uid): |
|||
return f"rl:{ns}:user:{uid}:blocked" |
|||
|
|||
def key_ns_window(ns, ip, bucket): |
|||
return f"rl:{ns}:ip:{ip}:w:{bucket}" |
|||
|
|||
|
|||
def write(r, key, value, ttl, label): |
|||
if isinstance(value, int): |
|||
pipe = r.pipeline() |
|||
pipe.set(key, value, ex=ttl) |
|||
pipe.execute() |
|||
else: |
|||
r.set(key, value, ex=ttl) |
|||
print(f" SET {key!r} = {value!r} EX {ttl}s ({label})") |
|||
|
|||
|
|||
def delete_pattern(r, pattern): |
|||
keys = r.keys(pattern) |
|||
if keys: |
|||
r.delete(*keys) |
|||
print(f" DEL {len(keys)} keys matching {pattern!r}") |
|||
else: |
|||
print(f" (no keys matching {pattern!r})") |
|||
|
|||
|
|||
# ── main ────────────────────────────────────────────────────────────────── |
|||
|
|||
def main(): |
|||
r = redis.from_url(REDIS_URL, db=RATELIMIT_DB, decode_responses=True) |
|||
try: |
|||
r.ping() |
|||
except redis.ConnectionError as exc: |
|||
print(f"ERROR: cannot connect to Redis at {REDIS_URL}: {exc}", file=sys.stderr) |
|||
sys.exit(1) |
|||
|
|||
print(f"Redis: {REDIS_URL} DB={RATELIMIT_DB} clear={CLEAR}") |
|||
print() |
|||
|
|||
# ── clear mode ──────────────────────────────────────────────────────── |
|||
if CLEAR: |
|||
print("=== Clearing synthetic test keys ===") |
|||
for ip in TEST_IPS: |
|||
delete_pattern(r, f"rl:ip:{ip}:*") |
|||
delete_pattern(r, f"rl:{NAMESPACE}:ip:{ip}:*") |
|||
for u in TEST_USERS: |
|||
delete_pattern(r, f"rl:{NAMESPACE}:user:{u['uid']}:*") |
|||
print("Done.") |
|||
return |
|||
|
|||
# ── anonymous IP counters ───────────────────────────────────────────── |
|||
print("=== Anonymous IP request counters (DB1) ===") |
|||
write(r, key_ip_reqs(TEST_IPS[0]), 20, ANON_WINDOW, "below threshold") |
|||
write(r, key_ip_reqs(TEST_IPS[1]), 35, ANON_WINDOW, "AT threshold → middleware will block on next req") |
|||
write(r, key_ip_reqs(TEST_IPS[3]), 30, ANON_WINDOW, "below threshold") |
|||
print() |
|||
|
|||
# ── blocked IPs ─────────────────────────────────────────────────────── |
|||
print("=== Blocked IPs (DB1) ===") |
|||
write(r, key_ip_blocked(TEST_IPS[2]), "1", BLOCK_TTL, "hard-blocked") |
|||
print() |
|||
|
|||
# ── namespace/IP sliding window ─────────────────────────────────────── |
|||
print("=== Namespace/IP sliding window (DB1) ===") |
|||
bucket = int(time.time() // ANON_WINDOW) |
|||
write(r, key_ns_window(NAMESPACE, TEST_IPS[3], bucket), 34, ANON_WINDOW * 2, |
|||
"near window threshold (next req triggers ua_rotation block)") |
|||
print() |
|||
|
|||
# ── authenticated user counters ─────────────────────────────────────── |
|||
print("=== Authenticated user request counters (DB1) ===") |
|||
for u in TEST_USERS: |
|||
if not u["blocked"]: |
|||
write(r, key_user_reqs(NAMESPACE, u["uid"]), u["reqs"], AUTH_WINDOW, |
|||
f"uid={u['uid']} reqs={u['reqs']}") |
|||
print() |
|||
|
|||
# ── blocked users ───────────────────────────────────────────────────── |
|||
print("=== Blocked users (DB1) ===") |
|||
for u in TEST_USERS: |
|||
if u["blocked"]: |
|||
write(r, key_user_blocked(NAMESPACE, u["uid"]), "1", BLOCK_TTL, |
|||
f"uid={u['uid']} hard-blocked") |
|||
print() |
|||
|
|||
# ── summary ─────────────────────────────────────────────────────────── |
|||
all_keys = r.keys("rl:*") |
|||
print(f"=== DB{RATELIMIT_DB} now contains {len(all_keys)} rl:* keys ===") |
|||
for k in sorted(all_keys): |
|||
ttl = r.ttl(k) |
|||
val = r.get(k) |
|||
print(f" {k!r:55s} val={val!r:5} ttl={ttl}s") |
|||
|
|||
|
|||
if __name__ == "__main__": |
|||
main() |
|||
@ -0,0 +1,503 @@ |
|||
# Rate Limiter Incidents |
|||
|
|||
This document records real rate-limiting incidents, the root-cause analysis performed for each, the fixes applied, and the architectural discussion that followed. New incidents should be appended under their own section. |
|||
|
|||
--- |
|||
|
|||
## PatoBranco-PR — 2026-05-06 |
|||
|
|||
### Symptom |
|||
|
|||
Councilmembers reported being unable to access the voting interface during a live plenary session. The error was HTTP 429. Two blocking events occurred: |
|||
|
|||
| Event | Start | Recovery | Duration | |
|||
|-------|-------|----------|----------| |
|||
| 1 | 13:51:23 | ~14:01 | ~10 min | |
|||
| 2 | 14:22:30 | ~14:25 | ~3 min | |
|||
|
|||
Both recovered **before** the Django `BLOCK_TTL` of 300 seconds, which was the first diagnostic clue. |
|||
|
|||
### Environment |
|||
|
|||
- NAT IP: `200.175.17.66` (reported range `200.175.17.66/29`) |
|||
- Secondary range: `187.109.99.234/30` |
|||
- Peak observed: **24 requests/second** from that IP (confirmed in OpenSearch, 14:22:31) |
|||
- Paths involved: `/voto-individual/`, `/sessao/pauta-sessao/2600/`, `/sessao/2600/ordemdia` |
|||
|
|||
### Root Cause |
|||
|
|||
Multiple councilmembers share a single public IP via NAT. When a vote opened, all of them reloaded their browser simultaneously. Nginx saw the combined traffic as a single client exhausting its burst bucket and returned 429 — before any request reached Django. |
|||
|
|||
``` |
|||
┌─────────────────────────────────────────┐ |
|||
│ nginx │ |
|||
│ │ |
|||
Councilmember A ──►│ IP: 200.175.17.66 │ |
|||
Councilmember B ──►│ IP: 200.175.17.66 ──► burst bucket │──► 429 (bucket full) |
|||
Councilmember C ──►│ IP: 200.175.17.66 exhausted │ |
|||
... │ │ |
|||
└─────────────────────────────────────────┘ |
|||
│ |
|||
│ (never reached) |
|||
▼ |
|||
┌─────────────────────────────────────────┐ |
|||
│ Django middleware │ |
|||
│ │ |
|||
│ rl:ip:<ip>:reqs (never incremented)│ |
|||
│ rl:user:<id>:reqs (never incremented)│ |
|||
│ Redis block key (never written) │ |
|||
└─────────────────────────────────────────┘ |
|||
``` |
|||
|
|||
### Why Recovery Was Faster Than 300 Seconds |
|||
|
|||
Django's block mechanism (`_set_block()`) was **never triggered**. The NAT IP was never written to Redis. The 429s came entirely from nginx's token bucket being exhausted. |
|||
|
|||
Recovery happened when the synchronized burst subsided (vote ended, users stopped reloading). The nginx bucket refilled at its configured rate. No TTL expiry was involved — recovery time was variable because it depended on how depleted the bucket was at the end of each burst, not on a fixed timer. |
|||
|
|||
Had Django's block fired, the outage would have been exactly 300 seconds both times. The variable durations (10 min vs 3 min) confirm nginx was the sole actor. |
|||
|
|||
### The Polling Source |
|||
|
|||
`voto_individual.html` contains a `setTimeout(location.reload, 30000)` — the page reloads itself every 30 seconds. When councilmembers opened the voting page at roughly the same time (vote announcement), their reload timers aligned. Each 30-second tick fired a synchronized burst from all clients behind the NAT. |
|||
|
|||
`/sessao/<pk>/ordemdia` and `/sessao/pauta-sessao/<pk>/` are not polled by JavaScript — they are normal page navigations. They appeared in the burst because councilmembers navigated to them at the same moment as the vote opened. |
|||
|
|||
### Two-Layer Rate Limiting Architecture |
|||
|
|||
``` |
|||
┌──────────────────────────────────────────────────────┐ |
|||
Incoming request │ nginx │ |
|||
─────────────────────►│ │ |
|||
│ limit_req zone=sapl_general ← IP-only, no auth │ |
|||
│ burst=${NGINX_BURST_GENERAL} nodelay │ |
|||
│ │ |
|||
│ If bucket full → 429 immediately │ |
|||
│ Redis: nothing written │ |
|||
└───────────────────────┬──────────────────────────────┘ |
|||
│ (only if bucket has room) |
|||
▼ |
|||
┌──────────────────────────────────────────────────────┐ |
|||
│ Django RateLimitMiddleware │ |
|||
│ │ |
|||
│ 1. Bypass check ← RATE_LIMIT_BYPASS_PATHS │ |
|||
│ 2. API quota check (if /api/) │ |
|||
│ 3. _evaluate() │ |
|||
│ a. IP block check (Redis rl:ip:<ip>:blocked) │ |
|||
│ b. User block check (Redis rl:user:<id>:blocked) │ |
|||
│ c. Rate counter (rl:ip:<ip>:reqs) │ |
|||
│ d. User counter (rl:user:<id>:reqs) │ |
|||
│ │ |
|||
│ If rate exceeded → SET block key (TTL=300s) │ |
|||
│ → ZADD rl:index:blocked_ips │ |
|||
└──────────────────────────────────────────────────────┘ |
|||
``` |
|||
|
|||
**The core mismatch:** Django tracks per-user buckets (`rl:user:<id>:reqs`) which are NAT-safe. Nginx tracks per-IP buckets which collapse all users behind a NAT into one. Nginx fires first, so Django's smarter per-user accounting is never consulted during a burst. |
|||
|
|||
### Fix Applied |
|||
|
|||
Added nginx `location` blocks for session and voting paths that pass requests through **without** `limit_req`. These regex locations take priority over the catch-all `location /` by nginx matching rules. |
|||
|
|||
**`docker/config/nginx/sapl.conf`:** |
|||
```nginx |
|||
location ~ ^/voto-individual/ { |
|||
proxy_set_header X-Request-ID $req_id; |
|||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
|||
proxy_set_header X-Forwarded-Proto $scheme; |
|||
proxy_set_header Host $http_host; |
|||
proxy_redirect off; |
|||
proxy_pass http://sapl_server; |
|||
} |
|||
|
|||
location ~ ^/sessao/\d+ { |
|||
proxy_set_header X-Request-ID $req_id; |
|||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
|||
proxy_set_header X-Forwarded-Proto $scheme; |
|||
proxy_set_header Host $http_host; |
|||
proxy_redirect off; |
|||
proxy_pass http://sapl_server; |
|||
} |
|||
``` |
|||
|
|||
**`sapl/settings.py`** — `RATE_LIMIT_BYPASS_PATHS` extended to match: |
|||
```python |
|||
RATE_LIMIT_BYPASS_PATHS = [ |
|||
r'^/painel/\d+/dados$', |
|||
r'^/voto-individual/', |
|||
r'^/sessao/\d+', |
|||
r'^/sessao/pauta-sessao/\d+/', |
|||
] |
|||
``` |
|||
|
|||
These paths are safe to exempt because: |
|||
- They require an authenticated session cookie to perform any meaningful action. |
|||
- Django's per-user rate counter still runs as a backstop. |
|||
- The cost of a false-positive block (councilmember unable to vote) outweighs the risk of abuse on these URLs. |
|||
|
|||
--- |
|||
|
|||
## Architectural Discussion |
|||
|
|||
### Why Increasing Burst Rates Is Not the Right Fix |
|||
|
|||
`burst` controls how many requests above the sustained rate are allowed in a spike before 429 fires. A larger burst absorbs thundering herds from NAT but also allows more requests from burst-style attackers (scanners, credential stuffers) before the throttle engages. It is the same knob, pulled in opposite directions. |
|||
|
|||
The correct dimensioning formula for a legislative house would be: |
|||
|
|||
``` |
|||
burst ≥ users_behind_NAT × tabs_per_user × requests_per_page_load |
|||
``` |
|||
|
|||
For a large state assembly (90 deputies, 2 tabs each) this exceeds 180 — a burst value that renders rate limiting ineffective against automated tools. |
|||
|
|||
**Conclusion:** burst tuning is not the right tool for this problem. Exemption of known-safe high-frequency paths is. |
|||
|
|||
### The Multi-Tab Problem |
|||
|
|||
Staff members commonly open multiple SAPL tabs simultaneously. Each tab has its own reload timer. If 10 staff members have 3 tabs open, a synchronized event generates 30+ requests from the NAT IP even before councilmembers are counted. This means the bucket can be pre-exhausted before the voting burst even starts. |
|||
|
|||
This further reinforces that IP-based rate limiting is the wrong unit for authenticated traffic in a shared-office environment. |
|||
|
|||
### Why Per-User Rate Limiting Does Not Fully Solve This |
|||
|
|||
Django already increments both `rl:ip:<ip>:reqs` and `rl:user:<id>:reqs`. The per-user counter is NAT-safe. But it never runs during a burst because nginx drops the request first. |
|||
|
|||
Moving rate limiting for authenticated users to the Django layer only (removing nginx `limit_req` for authenticated paths) would make per-user counting the effective control. The obstacle is that nginx cannot distinguish authenticated from anonymous requests without reading and resolving the session cookie — which requires a database or Redis lookup nginx cannot perform natively. |
|||
|
|||
### Architectural Solutions |
|||
|
|||
| Approach | Effort | Solves NAT problem | Protects against bots | |
|||
|----------|--------|-------------------|----------------------| |
|||
| Nginx bypass for known session paths *(done)* | Low | Yes, for bypassed paths | Yes, general paths still rate-limited | |
|||
| Increase `NGINX_BURST_GENERAL` | Trivial | Partially | Weakens bot protection | |
|||
| Nginx `limit_req_zone` keyed on session cookie string | Medium | Yes (per-session token, not per-IP) | Yes, each session has own bucket | |
|||
| Move all auth-path rate limiting to Django only | Medium | Yes | Depends on Django rate correctly tuned | |
|||
| Replace `setTimeout(location.reload)` with WebSocket/SSE push | High | Yes — eliminates synchronized reloads entirely | N/A | |
|||
|
|||
### WebSocket / SSE Consideration |
|||
|
|||
The 30-second self-reload in `voto_individual.html` is the synchronization mechanism that creates the thundering herd. If the server pushed state changes (vote opened, result published) instead of the client polling by reloading, the synchronized burst would not exist regardless of how many users or tabs are open. |
|||
|
|||
A full WebSocket rewrite (Django Channels + Redis pub/sub channel layer) would: |
|||
- Eliminate polling bursts on session/voting paths |
|||
- Make vote-state updates instantaneous instead of up to 30 seconds late |
|||
- Require nginx configuration for `Upgrade: websocket` proxying |
|||
|
|||
The nginx bypass is the correct operational fix for now. The WebSocket rewrite is the correct architectural fix for the future. They are not substitutes — the bypass would remain useful for the initial WebSocket handshake, which is still an HTTP request subject to burst limits. |
|||
|
|||
--- |
|||
|
|||
## Diagrams — Issues, Solutions, and Trade-offs (2026-05-06/07) |
|||
|
|||
--- |
|||
|
|||
### 1. NAT Thundering Herd — Before the Fix |
|||
|
|||
During a live vote all councilmembers reload simultaneously. nginx sees one |
|||
IP, exhausts its bucket, and returns 429 before Django is ever involved. |
|||
Django's per-user counter (NAT-safe) is never consulted. |
|||
|
|||
``` |
|||
Office / Chamber — behind one NAT IP (200.175.17.66) |
|||
┌──────────────────────────────────────────────────────┐ |
|||
│ Councilmember A browser reload ──┐ │ |
|||
│ Councilmember B browser reload ──┤ │ |
|||
│ Councilmember C browser reload ──┤ ~24 req/s │ |
|||
│ Staff tab 1 browser reload ──┤ same public IP │ |
|||
│ Staff tab 2 browser reload ──┘ │ |
|||
└────────────────────────────┬─────────────────────────┘ |
|||
│ all requests look identical to nginx |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ nginx sapl_general │ |
|||
│ rate=30r/m burst=60 nodelay │ |
|||
│ │ |
|||
│ token bucket: 0 tokens remaining │ |
|||
│ → 429 returned immediately │ |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
╳ Django never reached |
|||
╳ rl:ip:{ip}:reqs never incremented |
|||
╳ rl:user:{uid}:reqs never incremented |
|||
╳ per-user NAT-safe counter never consulted |
|||
│ |
|||
▼ |
|||
429 for all N users in the org |
|||
recovery: wait for nginx bucket refill |
|||
(~3–10 min depending on depletion) |
|||
NOT a Django 300s block (Redis never written) |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### 2. NAT Thundering Herd — After the Session Bypass Fix |
|||
|
|||
Session and voting paths have dedicated nginx `location` blocks with no |
|||
`limit_req`. Regex locations take priority over `location /`. |
|||
|
|||
``` |
|||
Office / Chamber — behind one NAT IP |
|||
┌──────────────────────────────────────────────────────┐ |
|||
│ Councilmember A /voto-individual/ reload ──┐ │ |
|||
│ Councilmember B /voto-individual/ reload ──┤ │ |
|||
│ Councilmember C /sessao/2600/ordemdia ───────┤ │ |
|||
│ Staff tab /sessao/pauta-sessao/2600/ ──┘ │ |
|||
└────────────────────────────┬─────────────────────────┘ |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ nginx │ |
|||
│ │ |
|||
│ location ~ ^/voto-individual/ ─┐ │ |
|||
│ location ~ ^/sessao/\d+ ─┤ │ no limit_req |
|||
│ location ~ ^/painel/\d+/dados ─┘ │ pass through |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ Django RateLimitMiddleware │ |
|||
│ │ |
|||
│ RATE_LIMIT_BYPASS_PATHS match? │ |
|||
│ → yes: return get_response() │ |
|||
│ (no counter, no block check) │ |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
✓ View served |
|||
All N users get their page |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### 3. nginx Zone Architecture — Before vs After |
|||
|
|||
**Before (single shared zone)** |
|||
|
|||
``` |
|||
All traffic (HTML + media + API) |
|||
│ |
|||
▼ |
|||
┌───────────────────────────────┐ |
|||
│ sapl_general │ ← one bucket per IP |
|||
│ rate=30r/m burst=60 │ |
|||
│ │ |
|||
│ /media/page.pdf ──────────┐ │ ← media request drains |
|||
│ /materia/123/ ──────────┤ │ the same bucket as |
|||
│ /api/materia/? ──────────┘ │ the HTML page |
|||
└───────────────────────────────┘ |
|||
|
|||
Problem: a page with 20 media attachments |
|||
burns 20 tokens from the page-load budget |
|||
``` |
|||
|
|||
**After (four independent zones)** |
|||
|
|||
``` |
|||
┌─────────────────┐ location / |
|||
│ sapl_general │ rate=90r/m burst=180 — HTML page requests |
|||
└─────────────────┘ |
|||
|
|||
┌─────────────────┐ location /media/ |
|||
│ sapl_media │ rate=180r/m burst=180 — media downloads |
|||
└─────────────────┘ own bucket, never |
|||
drains page quota |
|||
┌─────────────────┐ location /api/ |
|||
│ sapl_api │ rate=60r/m burst=120 — API calls |
|||
└─────────────────┘ quota layer is |
|||
real constraint |
|||
┌─────────────────┐ location /relatorios/ |
|||
│ sapl_heavy │ rate=10r/m burst=20 — PDF generation |
|||
└─────────────────┘ nodelay tight by design |
|||
|
|||
Session/voting paths: NO zone — exempt from limit_req entirely |
|||
Static files /static/: NO zone — served directly from disk |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### 4. Anonymous /api/ NAT Problem — Before vs After |
|||
|
|||
**Before** |
|||
|
|||
``` |
|||
Office — 10 staff, JS polling /api/ every 5s = 120 req/min combined |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ nginx sapl_general (was shared) │ |
|||
│ burst not yet exhausted → pass │ |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ Django _evaluate_anonymous │ |
|||
│ │ |
|||
│ INCR rl:ip:{ip}:reqs │ |
|||
│ count = 120 ≥ threshold (120/m) │ |
|||
│ │ |
|||
│ SET rl:ip:{ip}:blocked EX 300 │◄── block key written |
|||
│ ZADD rl:index:blocked_ips │ affects ALL paths |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
Next request to /materia/, /sessao/, /voto-individual/ ... |
|||
→ 429 ip_blocked (300s) |
|||
Entire org locked out of ALL SAPL pages |
|||
because of JS polling the API |
|||
``` |
|||
|
|||
**After** |
|||
|
|||
``` |
|||
Office — 10 staff, JS polling /api/ every 5s = 120 req/min combined |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ nginx sapl_api │ |
|||
│ rate=60r/m burst=120 nodelay │ |
|||
│ throttles burst, passes remainder │ |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ Django: quota check │ |
|||
│ 500 req/day not exceeded → pass │ |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ Anonymous /api/ early return │ |
|||
│ │ |
|||
│ rl:ip:{ip}:reqs NOT incremented │◄── no counter |
|||
│ rl:ip:{ip}:blocked NOT written │◄── no block key |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
✓ View served |
|||
Page requests from same IP unaffected |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### 5. Authenticated Rate Breach — Before vs After |
|||
|
|||
**Before** |
|||
|
|||
``` |
|||
Authenticated user clicks rapidly: 241 requests in 60s |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ _evaluate_authenticated │ |
|||
│ INCR rl:user:{uid}:reqs → 241 │ |
|||
│ count ≥ 240 (auth threshold) │ |
|||
│ │ |
|||
│ SET rl:user:{uid}:blocked EX 300 │◄── 5-minute lockout |
|||
│ ZADD rl:index:blocked_users │ |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
▼ |
|||
All requests for 300s → 429 user_blocked |
|||
User must wait 5 minutes to do anything |
|||
No way to self-recover sooner |
|||
``` |
|||
|
|||
**After** |
|||
|
|||
``` |
|||
Authenticated user clicks rapidly: 241 requests in 60s |
|||
│ |
|||
▼ |
|||
┌─────────────────────────────────────┐ |
|||
│ _evaluate_authenticated │ |
|||
│ INCR rl:user:{uid}:reqs → 241 │ |
|||
│ count ≥ 240 (auth threshold) │ |
|||
│ │ |
|||
│ return 429 auth_user_rate │◄── this request only |
|||
│ (no SET, no ZADD) │◄── no block key written |
|||
└──────────────────┬──────────────────┘ |
|||
│ |
|||
Counter TTL = 60s (auth_window) |
|||
│ |
|||
▼ |
|||
T+60s: rl:user:{uid}:reqs expires |
|||
User automatically recovers |
|||
No admin intervention needed |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### 6. Enforcement Stack Per Path — Trade-off Summary |
|||
|
|||
``` |
|||
Path nginx zone Django counter Block written? Notes |
|||
────────────────────── ─────────────── ──────────────── ────────────── ────────────────────────────── |
|||
/static/* none none — disk-served, zero Django cost |
|||
/painel/<pk>/dados none none — bypass: high-freq polling |
|||
/voto-individual/* none none — bypass: live vote |
|||
/sessao/<pk>/* none none — bypass: live session |
|||
/sessao/pauta-sessao/* none none — bypass: live session |
|||
/media/* sapl_media anon IP / auth anon: yes auth gate in serve_media() |
|||
180r/m b=180 counter runs auth: no |
|||
/api/* (anonymous) sapl_api quota only no ← key change: no IP counter, |
|||
60r/m b=120 500/day — no collateral NAT block |
|||
/api/* (authenticated) sapl_api per-user 240/m no (soft) per-user, NAT-safe |
|||
60r/m b=120 counter runs |
|||
/relatorios/* sapl_heavy anon/auth runs anon: yes tight rate — PDF generation |
|||
10r/m b=20 at Django auth: no |
|||
/* (everything else) sapl_general anon/auth runs anon: yes normal page navigation |
|||
90r/m b=180 at Django auth: no auth gets 240/m soft limit |
|||
``` |
|||
|
|||
**Legend:** |
|||
- `anon: yes` — anonymous IP gets a 300s block key on breach |
|||
- `auth: no` — authenticated users get 429 for that request, window resets in 60s, no persistent block |
|||
- `none` — no rate limiting at either layer (path is exempt) |
|||
|
|||
--- |
|||
|
|||
### 7. The Fundamental NAT Constraint |
|||
|
|||
``` |
|||
IP-based rate limiting cannot distinguish these two scenarios: |
|||
|
|||
Scenario A — Legitimate (15 users, 1 tab each, vote opens) |
|||
┌──────────────────────────────────────────────────────┐ |
|||
│ User 1 ──► GET /voto-individual/ │ |
|||
│ User 2 ──► GET /voto-individual/ 15 req/s │ |
|||
│ ... 1 public IP │ |
|||
│ User 15 ──► GET /sessao/2600/ordemdia │ |
|||
└──────────────────────────────────────────────────────┘ |
|||
|
|||
Scenario B — Bot (1 process, 15 threads, scraping) |
|||
┌──────────────────────────────────────────────────────┐ |
|||
│ Thread 1 ──► GET /materia/1/ │ |
|||
│ Thread 2 ──► GET /materia/2/ 15 req/s │ |
|||
│ ... 1 public IP │ |
|||
│ Thread 15 ──► GET /materia/15/ │ |
|||
└──────────────────────────────────────────────────────┘ |
|||
|
|||
To nginx and an IP-based counter: identical. |
|||
|
|||
Resolution strategies applied: |
|||
┌──────────────────────────────────────────────────────────────────┐ |
|||
│ Known safe high-freq paths → nginx bypass + Django bypass │ |
|||
│ Authenticated users → per-user counter (uid), NAT-safe │ |
|||
│ Anonymous /api/ → quota only, no IP counter │ |
|||
│ Everything else (anon) → IP counter + 300s block on breach │ |
|||
└──────────────────────────────────────────────────────────────────┘ |
|||
|
|||
Long-term: APP_ACCESS_KEYs per tenant → quota per org, not per IP |
|||
WebSocket push for voting → eliminates polling bursts |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Pending Investigations |
|||
|
|||
The following incidents may or may not share the same root cause as the PatoBranco-PR event. Each should be investigated using the OpenSearch query patterns established above and documented here. |
|||
|
|||
- [ ] Other houses reporting intermittent 429s during session hours |
|||
- [ ] Azure crawler bot (`52.167.144.162`) — 2 × 429 observed on 2026-05-06 at patobranco-pr; appears to be a legitimate Microsoft indexer hitting non-session paths; confirm it is correctly rate-limited and not causing collateral blocks on shared IPs |
|||
- [ ] Investigate whether `187.109.99.234/30` (patobranco secondary NAT range) experienced any blocks independently of `200.175.17.66/29` |
|||
File diff suppressed because it is too large
File diff suppressed because it is too large
@ -0,0 +1,93 @@ |
|||
""" |
|||
serve_media — X-Accel-Redirect gate for all /media/ files. |
|||
|
|||
Production flow (nginx proxies /media/ to Gunicorn): |
|||
1. Django middleware runs (IP rate-limit, bot UA check, etc.). |
|||
2. serve_media() runs auth check for sapl/private/, writes |
|||
URL-path counter to Redis DB 1, then returns X-Accel-Redirect. |
|||
Nginx serves the bytes directly from disk — Gunicorn worker freed immediately. |
|||
|
|||
Development flow (DEBUG=True, nginx absent): |
|||
Falls back to django.views.static.serve for live file serving. |
|||
|
|||
Redis side-effects per request (DB 1, TTL=MEDIA_PATH_COUNTER_TTL): |
|||
rl:{ns}:path:{sha256('/media/<path>')}:reqs — URL-path access counter |
|||
""" |
|||
|
|||
import hashlib |
|||
import mimetypes |
|||
import os |
|||
from urllib.parse import quote |
|||
|
|||
from django.conf import settings |
|||
from django.http import Http404, HttpResponse |
|||
from django.views.static import serve |
|||
|
|||
from sapl import settings as sapl_settings |
|||
from sapl.middleware.ratelimit import ( |
|||
_NAMESPACE, |
|||
RL_PATH_REQUESTS, |
|||
_incr_with_ttl, |
|||
) |
|||
|
|||
|
|||
def _safe_resolve(rel_path): |
|||
""" |
|||
Return the absolute path of rel_path inside MEDIA_ROOT. |
|||
Raises Http404 if the resolved path would escape the root |
|||
(path traversal guard). |
|||
""" |
|||
abs_root = os.path.abspath(settings.MEDIA_ROOT) |
|||
abs_path = os.path.abspath(os.path.join(abs_root, rel_path)) |
|||
if not abs_path.startswith(abs_root + os.sep) and abs_path != abs_root: |
|||
raise Http404 |
|||
return abs_path |
|||
|
|||
|
|||
def _is_public_docadm(path): |
|||
# Documentos Administrativos são sempre salvos na pasta /private, |
|||
# mas podem ter acesso público liberado como OSTENSIVO ou requerer |
|||
# autenticacao se for DOC_ADM_RESTRITIVO |
|||
from sapl.base.models import AppConfig, DOC_ADM_OSTENSIVO |
|||
return 'documentoadministrativo' in path and \ |
|||
AppConfig.attr('documentos_administrativos') == DOC_ADM_OSTENSIVO |
|||
|
|||
|
|||
def serve_media(request, path): |
|||
""" |
|||
Registered in sapl/urls.py for both DEBUG and production. |
|||
Route: ^media/(?P<path>.*)$ |
|||
""" |
|||
# Path traversal guard — raises Http404 on escape attempt. |
|||
abs_path = _safe_resolve(path) |
|||
|
|||
# Auth gate for private documents — redirect to login if anonymous. |
|||
if path.startswith('sapl/private/') and not _is_public_docadm(path): |
|||
user = getattr(request, 'user', None) |
|||
if user is None or not user.is_authenticated: |
|||
from django.contrib.auth.views import redirect_to_login |
|||
return redirect_to_login(request.get_full_path()) |
|||
|
|||
# 404 before writing any counters. |
|||
if not os.path.isfile(abs_path): |
|||
raise Http404 |
|||
|
|||
# URL-path counter (DB 1). |
|||
_incr_with_ttl( |
|||
RL_PATH_REQUESTS.format(ns=_NAMESPACE, sha256=hashlib.sha256(f'/media/{path}'.encode()).hexdigest()), |
|||
ttl=sapl_settings.MEDIA_PATH_COUNTER_TTL, |
|||
) |
|||
|
|||
if settings.DEBUG: |
|||
# Development: no nginx present; serve the file directly. |
|||
return serve(request, path, document_root=settings.MEDIA_ROOT) |
|||
|
|||
# Production: tell nginx to serve the file from the internal location. |
|||
filename = os.path.basename(path) |
|||
content_type, _ = mimetypes.guess_type(filename) |
|||
response = HttpResponse(content_type=content_type or 'application/octet-stream') |
|||
response['X-Accel-Redirect'] = f'/internal/media/{path}' |
|||
response['Content-Disposition'] = f"inline; filename*=UTF-8''{quote(filename)}" |
|||
response['Cache-Control'] = 'public, max-age=86400, stale-while-revalidate=3600' |
|||
response['X-Robots-Tag'] = 'noindex' |
|||
return response |
|||
@ -0,0 +1,62 @@ |
|||
from django.db import migrations, models |
|||
|
|||
_OLD_VIEW = """ |
|||
create or replace view materia_materiaemtramitacao as |
|||
select m.id as id, |
|||
m.id as materia_id, |
|||
t.id as tramitacao_id, |
|||
t.unidade_tramitacao_destino_id as unidade_tramitacao_atual_id |
|||
from materia_materialegislativa m |
|||
inner join materia_tramitacao t on (m.id = t.materia_id) |
|||
where t.id = (select max(id) from materia_tramitacao where materia_id = m.id) |
|||
order by m.id DESC |
|||
""" |
|||
|
|||
_NEW_VIEW = """ |
|||
create or replace view materia_materiaemtramitacao as |
|||
select distinct on (m.id) |
|||
m.id as id, |
|||
m.id as materia_id, |
|||
t.id as tramitacao_id, |
|||
t.unidade_tramitacao_destino_id as unidade_tramitacao_atual_id |
|||
from materia_materialegislativa m |
|||
inner join materia_tramitacao t on t.materia_id = m.id |
|||
order by m.id desc, t.id desc |
|||
""" |
|||
|
|||
|
|||
class Migration(migrations.Migration): |
|||
# CREATE INDEX CONCURRENTLY cannot run inside a transaction. |
|||
atomic = False |
|||
|
|||
dependencies = [ |
|||
('materia', '0087_update_viewdb_materiaemtramitacao'), |
|||
] |
|||
|
|||
operations = [ |
|||
migrations.RunSQL(sql=_NEW_VIEW, reverse_sql=_OLD_VIEW), |
|||
migrations.SeparateDatabaseAndState( |
|||
database_operations=[ |
|||
migrations.RunSQL( |
|||
sql=""" |
|||
CREATE INDEX CONCURRENTLY IF NOT EXISTS |
|||
tram_materia_id_desc |
|||
ON materia_tramitacao (materia_id, id DESC) |
|||
""", |
|||
reverse_sql=""" |
|||
DROP INDEX CONCURRENTLY IF EXISTS |
|||
tram_materia_id_desc |
|||
""", |
|||
), |
|||
], |
|||
state_operations=[ |
|||
migrations.AddIndex( |
|||
model_name='tramitacao', |
|||
index=models.Index( |
|||
fields=['materia', '-id'], |
|||
name='tram_materia_id_desc', |
|||
), |
|||
), |
|||
], |
|||
), |
|||
] |
|||
@ -0,0 +1,75 @@ |
|||
""" |
|||
AnonCachePageMixin — anonymous-only Django view response caching. |
|||
|
|||
Why anonymous-only? |
|||
- Authenticated responses include CSRF tokens and user-specific UI fragments |
|||
(edit/delete action buttons injected by SAPL's CRUD framework). Caching |
|||
those would serve stale or wrong data to other users. |
|||
- Bot traffic is entirely anonymous. A 2-minute cache converts hundreds of |
|||
identical list-view DB queries into a single one — exactly the workload |
|||
that triggers OOM in the fleet. |
|||
|
|||
How it works: |
|||
- `dispatch()` short-circuits to the normal (uncached) path for any |
|||
authenticated request. |
|||
- For anonymous GET/HEAD requests, the response is stored in the 'default' |
|||
Redis cache under a key that includes the full URL (scheme + host + path + |
|||
query string). The Django cache framework handles key construction and |
|||
TTL expiry automatically. |
|||
- HTTPS and HTTP requests are stored under separate keys (Django default). |
|||
|
|||
Usage: |
|||
|
|||
from sapl.middleware.page_cache import AnonCachePageMixin |
|||
|
|||
class MyListView(AnonCachePageMixin, ListView): |
|||
anon_cache_ttl = settings.PAGE_CACHE_TTL_LIST # 120 s |
|||
|
|||
class MyDetailView(AnonCachePageMixin, DetailView): |
|||
anon_cache_ttl = settings.PAGE_CACHE_TTL_DETAIL # 300 s |
|||
|
|||
TTL reference (see settings.PAGE_CACHE_TTL_*): |
|||
|
|||
View type Default TTL |
|||
───────────────────────────────────────────────────── |
|||
Public list (norma, materia, sessao…) 120 s (PAGE_CACHE_TTL_LIST) |
|||
Public detail (norma, materia, sessao…) 300 s (PAGE_CACHE_TTL_DETAIL) |
|||
Stable detail (parlamentar, comissão) 600 s (PAGE_CACHE_TTL_STABLE) |
|||
|
|||
Invalidation: |
|||
The cache TTL is intentionally short (≤ 10 min) so stale content expires |
|||
on its own. Explicit invalidation is not implemented — legislative data |
|||
changes infrequently and short TTLs are acceptable. |
|||
""" |
|||
|
|||
from django.conf import settings |
|||
from django.views.decorators.cache import cache_page, never_cache |
|||
from django.utils.decorators import method_decorator |
|||
|
|||
|
|||
class AnonCachePageMixin: |
|||
""" |
|||
Cache the full view response for anonymous (unauthenticated) requests. |
|||
|
|||
Set `anon_cache_ttl` on the subclass to override the default TTL. |
|||
Authenticated requests always bypass the cache. |
|||
""" |
|||
|
|||
# Override per view class. Use settings.PAGE_CACHE_TTL_* for consistency. |
|||
anon_cache_ttl = getattr(settings, 'PAGE_CACHE_TTL_LIST', 120) |
|||
|
|||
def dispatch(self, request, *args, **kwargs): |
|||
if getattr(request, 'user', None) and request.user.is_authenticated: |
|||
# Authenticated: skip cache entirely — response may contain |
|||
# user-specific controls (CSRF token, edit/delete buttons). |
|||
handler = never_cache( |
|||
lambda req, *a, **kw: super(AnonCachePageMixin, self).dispatch(req, *a, **kw) |
|||
) |
|||
return handler(request, *args, **kwargs) |
|||
|
|||
# Anonymous: wrap the parent dispatch in cache_page so Django stores |
|||
# the rendered response in the 'default' cache for anon_cache_ttl seconds. |
|||
handler = cache_page(self.anon_cache_ttl)( |
|||
lambda req, *a, **kw: super(AnonCachePageMixin, self).dispatch(req, *a, **kw) |
|||
) |
|||
return handler(request, *args, **kwargs) |
|||
@ -0,0 +1,774 @@ |
|||
""" |
|||
RateLimitMiddleware — cross-pod rate limiting backed by shared Redis. |
|||
|
|||
Decision flow (per request): |
|||
Both /api/ and non-/api/ paths — checked first, before all else: |
|||
-1. IP in rl:ip_prefix:blocked (prefix match)? → 403 (Forbidden — not a rate |
|||
limit; an operator-curated permanent deny list, no Retry-After) |
|||
(universal — applies to authenticated users too, like the UA bot checks) |
|||
/api/ paths — handled by _handle_api: |
|||
0a. OPTIONS? → pass (CORS preflight must never be blocked) |
|||
0b. rl:ip_prefix:blocked? → 403 (prefix match; see -1 above) |
|||
0c. rl:ip:<ip>:blocked? → 429 (global block also covers /api/) |
|||
0d. rl:api:ip:<ip>:blocked? → 429 (API-only block) |
|||
Block checks 0b-0d always run before the same-origin check below — |
|||
Origin/Referer are client-controlled and trivially spoofable, so they |
|||
must never be able to override an already-made block decision. |
|||
0e. Same-origin? → pass, skipping 0f-0g only (SAPL's own polling |
|||
is exempt from quota/rate-limit accounting, never from active blocks) |
|||
0f. Daily/weekly quota exceeded? → 429 |
|||
0g. Anon + API threshold exceeded? → SET rl:api:ip:<ip>:blocked, 429 |
|||
(never writes rl:ip:<ip>:blocked) |
|||
0h. Auth: falls through to _evaluate (per-user counter) |
|||
Non-/api/ paths: |
|||
1. Known bot UA? → 429 (Python list — substring match) |
|||
1b. Redis UA deny list? → 429 (runtime SET — token hash match, refreshed every 60 s) |
|||
2. Anonymous AND IP in blocked set? → 429 (authenticated users skip — have per-user limit at 3c) |
|||
3. Authenticated user? |
|||
a. User blocked? → 429 |
|||
b. Suspicious hdrs? → 429 |
|||
c. User rate ≥ 240? → 429 (no persistent block; window resets after 60 s) |
|||
4. Anonymous: |
|||
a. Suspicious hdrs? → 429 |
|||
b. IP rate ≥ 120/min? → SET RL_IP_BLOCKED, 429 |
|||
c. NS/IP window hit? → SET RL_IP_BLOCKED, 429 |
|||
|
|||
Degrades gracefully to non-atomic counting when Redis is unavailable. |
|||
|
|||
_NAMESPACE is settings.POD_NAMESPACE, resolved once at startup: |
|||
- K8s: start.sh reads the k8s namespace from the Downward API env var |
|||
or the service-account namespace file, writes it to .env as POD_NAMESPACE. |
|||
- Bare-metal / VM / docker-compose: defaults to the machine hostname |
|||
(socket.gethostbyname_ex result computed in settings.py). |
|||
Since a deployment serves exactly one tenant, this is a startup constant — |
|||
no per-request lookup is needed or correct. |
|||
""" |
|||
|
|||
import hashlib |
|||
import logging |
|||
import re |
|||
import time |
|||
from datetime import date |
|||
|
|||
from sapl import settings |
|||
from django.core.cache import caches |
|||
from django.http import HttpResponse |
|||
|
|||
logger = logging.getLogger('sapl.ratelimit') |
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Tenant namespace — resolved once at startup from settings.POD_NAMESPACE. |
|||
# On K8s: the k8s namespace (e.g. "sapl31demo-df"), set by start.sh. |
|||
# On bare-metal / VM / docker-compose: the machine hostname (default). |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
_NAMESPACE = settings.POD_NAMESPACE |
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Redis key templates — module-level constants, never inline strings |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
RL_IP_REQUESTS = 'rl:ip:{ip}:reqs' |
|||
RL_IP_BLOCKED = 'rl:ip:{ip}:blocked' |
|||
RL_IP_404S = 'rl:ip:{ip}:404s' |
|||
RL_USER_REQUESTS = 'rl:{ns}:user:{uid}:reqs' |
|||
RL_USER_BLOCKED = 'rl:{ns}:user:{uid}:blocked' |
|||
RL_NS_WINDOW = 'rl:{ns}:ip:{ip}:w:{bucket}' |
|||
RL_PATH_REQUESTS = 'rl:{ns}:path:{sha256}:reqs' |
|||
RL_UA_BLOCKLIST = 'rl:bot:ua:blocked' # permanent SET — runtime UA deny list |
|||
RL_IP_PREFIX_BLOCKLIST = 'rl:ip_prefix:blocked' # permanent SET — runtime IP-prefix deny list |
|||
RL_METRICS_BLOCKED = 'rl:metrics:{ns}:{date}' # HASH: field = reason, value = daily count |
|||
|
|||
# ZSET indexes — members are full block-key strings, score = expiry unix timestamp. |
|||
# Lets admin/monitoring tools enumerate active blocks with a single ZRANGEBYSCORE |
|||
# without scanning all keys. Prunable via: ZREMRANGEBYSCORE <index> 0 <now>. |
|||
RL_INDEX_BLOCKED_IPS = 'rl:index:blocked_ips' |
|||
RL_INDEX_BLOCKED_USERS = 'rl:index:blocked_users' |
|||
|
|||
# API-specific rate limit keys — scope limited to /api/, never written by non-/api/ paths. |
|||
RL_API_IP_REQUESTS = 'rl:api:ns:{ns}:ip:{ip}:reqs' |
|||
RL_API_IP_BLOCKED = 'rl:api:ns:{ns}:ip:{ip}:blocked' |
|||
RL_INDEX_API_BLOCKED_IPS = 'rl:index:api_blocked_ips' |
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# API quota keys — per-tenant HASH, one key per period. |
|||
# Structure: HASH key = api_quota:{ns}:daily:{date} field = {ip} value = counter |
|||
# HASH key = api_quota:{ns}:weekly:{week} field = {ip} value = counter |
|||
# Weekly key uses ISO week notation (yyyy-Www) — unambiguous, Monday-anchored. |
|||
# TTL set once on hash creation (Lua TTL guard); resets are implicit in the |
|||
# date/week embedded in the key name. Fields are IPs; no per-field TTL. |
|||
# Memory: ~63 bytes/field vs ~148 bytes for per-IP STRING keys (~57% saving). |
|||
# --------------------------------------------------------------------------- |
|||
API_QUOTA_DAILY_HASH = 'api_quota:{ns}:daily:{date}' |
|||
API_QUOTA_WEEKLY_HASH = 'api_quota:{ns}:weekly:{week}' |
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Bot UA fragments |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
BOT_UA_FRAGMENTS = [ |
|||
'GPTBot', |
|||
'ClaudeBot', |
|||
'PerplexityBot', |
|||
'Bytespider', |
|||
'AhrefsBot', |
|||
'meta-externalagent', |
|||
'OAI-SearchBot', |
|||
'bingbot', |
|||
'SERankingBacklinksBot', |
|||
'Chrome/98.0.4758', # known scraper impersonating an old Chrome |
|||
'quiltbot', |
|||
'AwarioBot', |
|||
] |
|||
|
|||
_INCR_LUA = """ |
|||
local n = redis.call('INCR', KEYS[1]) |
|||
if n == 1 then redis.call('EXPIRE', KEYS[1], ARGV[1]) end |
|||
return n |
|||
""" |
|||
|
|||
# Atomically increment a HASH field and set TTL on the hash key the first time |
|||
# it is created (TTL < 0 covers both -1 = no expire and -2 = key just created). |
|||
# KEYS[1] = hash key ARGV[1] = field (IP) ARGV[2] = TTL seconds |
|||
_HINCRBY_LUA = """ |
|||
local n = redis.call('HINCRBY', KEYS[1], ARGV[1], 1) |
|||
if redis.call('TTL', KEYS[1]) < 0 then |
|||
redis.call('EXPIRE', KEYS[1], ARGV[2]) |
|||
end |
|||
return n |
|||
""" |
|||
|
|||
# Atomically write a block key and record it in the sharded ZSET index. |
|||
# Prunes expired entries from the target shard before inserting the new one, |
|||
# keeping each shard bounded to only active blocks (amortised O(1) cleanup). |
|||
# KEYS[1] = block key KEYS[2] = shard index key |
|||
# ARGV[1] = ttl (seconds) ARGV[2] = expiry unix timestamp (now + ttl) |
|||
# ARGV[3] = current unix timestamp (for pruning: remove score < now) |
|||
_BLOCK_LUA = """ |
|||
local now = tonumber(ARGV[3]) |
|||
redis.call('SET', KEYS[1], '1', 'EX', ARGV[1]) |
|||
redis.call('ZREMRANGEBYSCORE', KEYS[2], '-inf', now - 1) |
|||
redis.call('ZADD', KEYS[2], ARGV[2], KEYS[1]) |
|||
return 1 |
|||
""" |
|||
|
|||
|
|||
def _index_shard(ip, index_base): |
|||
""" |
|||
Return the sharded ZSET key for an IP. |
|||
|
|||
IPs are distributed across RATE_LIMITER_INDEX_SHARDS shards using |
|||
md5(ip) % N, spreading write contention and bounding each shard's size. |
|||
The mapping is deterministic: the same IP always routes to the same shard. |
|||
""" |
|||
n = settings.RATE_LIMITER_INDEX_SHARDS |
|||
shard = int(hashlib.md5(ip.encode()).hexdigest(), 16) % n |
|||
return f'{index_base}:{shard}' |
|||
|
|||
|
|||
def make_ratelimit_cache_key(key, key_prefix, version): |
|||
""" |
|||
Pass-through cache key function for the 'ratelimit' Django cache backend. |
|||
|
|||
Django's default key function produces '{KEY_PREFIX}:{VERSION}:{key}', |
|||
which turns django-ratelimit's own keys (already prefixed 'rl:{hash}') |
|||
into ':1:rl:{hash}' — an ugly leading colon and version number that does |
|||
not match the clean 'rl:*' keys written directly by RateLimitMiddleware. |
|||
|
|||
Setting KEY_FUNCTION to this function makes both key namespaces consistent: |
|||
django-ratelimit decorator keys → rl:{hash} |
|||
RateLimitMiddleware keys → rl:ip:{ip}:reqs / rl:{ns}:user:{uid}:reqs / … |
|||
""" |
|||
return key |
|||
|
|||
|
|||
def _sha256(s): |
|||
return hashlib.sha256(s.encode()).hexdigest() |
|||
|
|||
|
|||
def get_client_ip(request): |
|||
""" |
|||
Return the real client IP, applying django-ratelimit's ip_mask so that |
|||
IPv6 /64 subnets are collapsed to a single key (prevents per-address |
|||
rotation attacks). Also checks HTTP_X_REAL_IP for nginx setups that |
|||
use that header instead of X-Forwarded-For. |
|||
|
|||
Canonical source — imported from here by other SAPL modules. |
|||
""" |
|||
from ratelimit.core import ip_mask |
|||
x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') |
|||
if x_forwarded_for: |
|||
ip = x_forwarded_for.split(',')[0].strip() |
|||
else: |
|||
ip = ( |
|||
request.META.get('HTTP_X_REAL_IP') |
|||
or request.META.get('REMOTE_ADDR') |
|||
or '0.0.0.0' |
|||
) |
|||
return ip_mask(ip) |
|||
|
|||
|
|||
def ratelimit_ip(group, request): |
|||
"""Key function for django-ratelimit decorators (group param is ignored).""" |
|||
return get_client_ip(request) |
|||
|
|||
|
|||
def smart_key(group, request): |
|||
""" |
|||
Auth-aware key for @ratelimit decorators. |
|||
|
|||
Authenticated users are keyed by user pk so that office workers sharing |
|||
a NAT IP don't count against each other. Anonymous requests fall back to |
|||
the masked IP (IPv6 /64 collapsed via ip_mask). |
|||
""" |
|||
user = getattr(request, 'user', None) |
|||
if user is not None and user.is_authenticated: |
|||
return str(user.pk) |
|||
return ratelimit_ip(group, request) |
|||
|
|||
|
|||
def smart_rate(group, request): |
|||
""" |
|||
Auth-aware rate string for @ratelimit decorators. |
|||
|
|||
Returns RATE_LIMITER_RATE_AUTHENTICATED for authenticated users, |
|||
RATE_LIMITER_RATE for anonymous users — mirrors the thresholds applied |
|||
by RateLimitMiddleware so view-level and middleware-level limits agree. |
|||
""" |
|||
user = getattr(request, 'user', None) |
|||
if user is not None and user.is_authenticated: |
|||
return settings.RATE_LIMITER_RATE_AUTHENTICATED |
|||
return settings.RATE_LIMITER_RATE |
|||
|
|||
|
|||
def _is_same_origin(request): |
|||
""" |
|||
Return True if Origin or Referer header matches the current SAPL host. |
|||
Strips port and lowercases both sides before comparing — DNS is case-insensitive |
|||
and reverse proxies may expose a different port than the browser sees. |
|||
Checks Origin first; falls back to Referer only when Origin is absent. |
|||
Returns False when both headers are absent. |
|||
""" |
|||
from urllib.parse import urlparse |
|||
|
|||
def _normalize(host): |
|||
return host.lower().split(':', 1)[0].strip() |
|||
|
|||
try: |
|||
host = _normalize(request.get_host()) |
|||
except Exception: |
|||
return False |
|||
|
|||
origin = request.META.get('HTTP_ORIGIN', '') |
|||
if origin: |
|||
try: |
|||
return _normalize(urlparse(origin).netloc) == host |
|||
except ValueError: |
|||
return False |
|||
|
|||
referer = request.META.get('HTTP_REFERER', '') |
|||
if referer: |
|||
try: |
|||
return _normalize(urlparse(referer).netloc) == host |
|||
except ValueError: |
|||
return False |
|||
|
|||
return False |
|||
|
|||
|
|||
def _is_suspicious_headers(request): |
|||
"""Real browsers send Accept-Language + Accept; bots frequently omit them.""" |
|||
missing = sum([ |
|||
not request.META.get('HTTP_ACCEPT_LANGUAGE'), |
|||
not request.META.get('HTTP_ACCEPT'), |
|||
]) |
|||
# Também considera User-Agent antes de bloquear |
|||
has_ua = bool(request.META.get('HTTP_USER_AGENT')) |
|||
return missing >= 2 and not has_ua |
|||
|
|||
|
|||
def _parse_rate(rate_str): |
|||
"""Parse '35/m' or '120/m' into (count, seconds).""" |
|||
count, period = rate_str.split('/') |
|||
count = int(count) |
|||
seconds = {'s': 1, 'm': 60, 'h': 3600}.get(period.lower(), 60) |
|||
return count, seconds |
|||
|
|||
|
|||
def _incr_with_ttl(key, ttl): |
|||
""" |
|||
Atomic INCR + EXPIRE via Redis Lua script (ratelimit cache, DB 1). |
|||
Falls back to non-atomic cache get/set when Redis is unavailable. |
|||
Exported at module level so sapl.base.media can reuse it for path counters. |
|||
""" |
|||
try: |
|||
from django_redis import get_redis_connection |
|||
client = get_redis_connection('ratelimit') |
|||
return client.eval(_INCR_LUA, 1, key, ttl) |
|||
except Exception: |
|||
rl_cache = caches['ratelimit'] |
|||
count = (rl_cache.get(key) or 0) + 1 |
|||
rl_cache.set(key, count, timeout=ttl) |
|||
return count |
|||
|
|||
|
|||
def _hincrby_with_ttl(hash_key, field, ttl): |
|||
""" |
|||
Atomic HINCRBY + conditional EXPIRE via Redis Lua script (ratelimit cache, DB 1). |
|||
|
|||
Increments the counter for `field` inside `hash_key` and sets a TTL on the |
|||
hash key if it does not already have one (i.e. on first creation). |
|||
Falls back to a composite STRING key `hash_key:field` when Redis is unavailable. |
|||
""" |
|||
try: |
|||
from django_redis import get_redis_connection |
|||
client = get_redis_connection('ratelimit') |
|||
return client.eval(_HINCRBY_LUA, 1, hash_key, field, ttl) |
|||
except Exception: |
|||
rl_cache = caches['ratelimit'] |
|||
fallback_key = f'{hash_key}:{field}' |
|||
count = (rl_cache.get(fallback_key) or 0) + 1 |
|||
rl_cache.set(fallback_key, count, timeout=ttl) |
|||
return count |
|||
|
|||
|
|||
def _set_block(block_key, index_key, ttl): |
|||
""" |
|||
Atomically set a block key (with TTL) and record it in a sharded ZSET index. |
|||
Score = expiry unix timestamp. Prunes expired entries from the target shard |
|||
before inserting (inline cleanup — no separate maintenance job needed). |
|||
Falls back to a plain cache.set when Redis is unavailable (index skipped). |
|||
""" |
|||
now = int(time.time()) |
|||
expire_at = now + ttl |
|||
try: |
|||
from django_redis import get_redis_connection |
|||
client = get_redis_connection('ratelimit') |
|||
client.eval(_BLOCK_LUA, 2, block_key, index_key, ttl, expire_at, now) |
|||
except Exception: |
|||
caches['ratelimit'].set(block_key, 1, timeout=ttl) |
|||
|
|||
|
|||
class RateLimitMiddleware: |
|||
BLOCK_TTL = 300 # seconds an IP/user stays blocked after threshold breach |
|||
|
|||
# In-process cache for the Redis UA deny list. |
|||
# Shared across all instances in the same worker process (one per worker). |
|||
# Refreshed every RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds via SMEMBERS. |
|||
_ua_blocklist: set = set() |
|||
_ua_blocklist_fetched_at: float = 0.0 |
|||
|
|||
# In-process cache for the Redis IP-prefix deny list (operator-curated SET |
|||
# of dotted-decimal prefixes). Normalized to trailing-dot form on refresh so |
|||
# checking is O(1) per candidate via set membership. |
|||
_ip_prefix_blocklist: set = set() |
|||
_ip_prefix_blocklist_fetched_at: float = 0.0 |
|||
|
|||
def __init__(self, get_response): |
|||
self.get_response = get_response |
|||
self.anon_threshold, self.anon_window = _parse_rate(settings.RATE_LIMITER_RATE) |
|||
self.auth_threshold, self.auth_window = _parse_rate(settings.RATE_LIMITER_RATE_AUTHENTICATED) |
|||
self._rl_cache = caches['ratelimit'] |
|||
self.not_found_threshold = settings.RATE_LIMIT_404_THRESHOLD |
|||
self._bypass_paths = [ |
|||
re.compile(p) for p in getattr(settings, 'RATE_LIMIT_BYPASS_PATHS', []) |
|||
] |
|||
self.api_quota_daily = settings.API_QUOTA_DAILY |
|||
self.api_quota_weekly = settings.API_QUOTA_WEEKLY |
|||
self.api_rate_limit_enabled = getattr(settings, 'API_RATE_LIMIT_ENABLED', True) |
|||
self.api_threshold = getattr(settings, 'API_RATE_LIMIT_THRESHOLD', 60) |
|||
self.api_window = getattr(settings, 'API_RATE_LIMIT_WINDOW_SECONDS', 60) |
|||
self.api_block_seconds = getattr(settings, 'API_RATE_LIMIT_BLOCK_SECONDS', 300) |
|||
self.api_same_origin_bypass = getattr(settings, 'API_RATE_LIMIT_SAME_ORIGIN_BYPASS', True) |
|||
logger.info( |
|||
'[RATELIMIT] anon=%s auth=%s bot=%s bypass_paths=%s', |
|||
settings.RATE_LIMITER_RATE, |
|||
settings.RATE_LIMITER_RATE_AUTHENTICATED, |
|||
settings.RATE_LIMITER_RATE_BOT, |
|||
[p.pattern for p in self._bypass_paths] or '(none)', |
|||
) |
|||
logger.info( |
|||
'[API QUOTAS] daily=%s weekly=%s (all callers keyed by IP)', |
|||
settings.API_QUOTA_DAILY, |
|||
settings.API_QUOTA_WEEKLY, |
|||
) |
|||
logger.info( |
|||
'[API RATE LIMIT] enabled=%s threshold=%s window=%ss block=%ss same_origin_bypass=%s', |
|||
self.api_rate_limit_enabled, self.api_threshold, self.api_window, |
|||
self.api_block_seconds, self.api_same_origin_bypass, |
|||
) |
|||
|
|||
def __call__(self, request): |
|||
if any(p.match(request.path) for p in self._bypass_paths): |
|||
return self.get_response(request) |
|||
|
|||
# Reject parameter pollution: the `page` param must appear at most once. |
|||
# Duplicate page= values (e.g. ?page=3&page=2&page=3) are a bot |
|||
# fingerprint — no legitimate browser or script sends this. |
|||
if len(request.GET.getlist('page')) > 1: |
|||
logger.warning( |
|||
'ratelimit_block layer=django reason=param_pollution ip=%s path=%s', |
|||
get_client_ip(request), request.path, |
|||
) |
|||
self._inc_block_metric('param_pollution') |
|||
response = HttpResponse(status=400) |
|||
response['X-RateLimit-Reason'] = 'param_pollution' |
|||
return response |
|||
|
|||
if request.path.startswith('/api/'): |
|||
return self._handle_api(request) |
|||
|
|||
decision = self._evaluate(request) |
|||
if decision['action'] == 'block': |
|||
logger.warning( |
|||
'ratelimit_block layer=django reason=%s ip=%s path=%s namespace=%s', |
|||
decision['reason'], |
|||
decision['ip'], |
|||
request.path, |
|||
_NAMESPACE, |
|||
extra={'ua': request.META.get('HTTP_USER_AGENT', '')}, |
|||
) |
|||
self._inc_block_metric(decision['reason']) |
|||
if decision['reason'] == 'ip_prefix_blocked': |
|||
response = HttpResponse(status=403) |
|||
else: |
|||
response = HttpResponse(status=429) |
|||
response['Retry-After'] = self.BLOCK_TTL |
|||
response['X-RateLimit-Reason'] = decision['reason'] |
|||
return response |
|||
logger.debug( |
|||
'ratelimit_pass ip=%s path=%s user=%s namespace=%s', |
|||
decision['ip'], |
|||
request.path, |
|||
getattr(getattr(request, 'user', None), 'pk', 'anon'), |
|||
_NAMESPACE, |
|||
) |
|||
response = self.get_response(request) |
|||
if response.status_code == 404: |
|||
self._handle_not_found(request, decision['ip']) |
|||
return response |
|||
|
|||
# ------------------------------------------------------------------ |
|||
# /api/ handling |
|||
# ------------------------------------------------------------------ |
|||
|
|||
def _api_block_response(self, reason, retry_after=None): |
|||
from django.http import JsonResponse |
|||
if reason == 'ip_prefix_blocked': |
|||
resp = JsonResponse({'detail': 'Forbidden.'}, status=403) |
|||
resp['X-RateLimit-Reason'] = reason |
|||
return resp |
|||
if retry_after is None: |
|||
retry_after = self.api_block_seconds |
|||
resp = JsonResponse( |
|||
{'detail': 'API rate limit exceeded. Please reduce polling frequency.', |
|||
'retry_after_seconds': retry_after}, |
|||
status=429, |
|||
) |
|||
resp['Retry-After'] = retry_after |
|||
resp['X-RateLimit-Reason'] = reason |
|||
return resp |
|||
|
|||
def _handle_api(self, request): |
|||
# 1. OPTIONS preflight — CORS must never be blocked |
|||
if request.method == 'OPTIONS': |
|||
return self.get_response(request) |
|||
|
|||
ip = get_client_ip(request) |
|||
|
|||
# 2. IP-prefix block — operator-curated deny list, applies to everyone. |
|||
# Block checks (2-4) must run before the same-origin bypass below: |
|||
# Origin/Referer are client-controlled and trivially spoofable, so |
|||
# they must never be able to override an already-made block decision. |
|||
if self._is_ip_prefix_blocked(ip): |
|||
logger.warning( |
|||
'api_rate_limit_block reason=ip_prefix_blocked ip=%s path=%s user_agent=%s', |
|||
ip, request.path, request.META.get('HTTP_USER_AGENT', ''), |
|||
) |
|||
self._inc_block_metric('api_ip_prefix_blocked') |
|||
return self._api_block_response('ip_prefix_blocked') |
|||
|
|||
# 3. Global IP block also covers /api/ |
|||
if self._rl_cache.get(RL_IP_BLOCKED.format(ip=ip)): |
|||
logger.warning( |
|||
'api_rate_limit_block reason=global_ip_blocked ip=%s path=%s user_agent=%s', |
|||
ip, request.path, request.META.get('HTTP_USER_AGENT', ''), |
|||
) |
|||
self._inc_block_metric('api_global_ip_blocked') |
|||
return self._api_block_response('global_ip_blocked') |
|||
|
|||
# 4. API-specific block (blocks /api/ only, never set by non-/api/ paths) |
|||
if self._rl_cache.get(RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip)): |
|||
logger.warning( |
|||
'api_rate_limit_block reason=api_ip_blocked ip=%s path=%s user_agent=%s', |
|||
ip, request.path, request.META.get('HTTP_USER_AGENT', ''), |
|||
) |
|||
self._inc_block_metric('api_ip_blocked') |
|||
return self._api_block_response('api_ip_blocked') |
|||
|
|||
# 5. Same-origin (SAPL's own polling) — exempt from quota/rate-limit |
|||
# accounting only (steps 6-7 below). Must come after the block |
|||
# checks above, never before them. |
|||
if self.api_same_origin_bypass and _is_same_origin(request): |
|||
return self.get_response(request) |
|||
|
|||
# 6. Daily/weekly quota (existing logic, preserved) |
|||
exceeded = self._check_api_quota(request) |
|||
if exceeded: |
|||
logger.warning( |
|||
'quota_exceeded window=%s ip=%s path=%s namespace=%s', |
|||
exceeded, ip, request.path, _NAMESPACE, |
|||
extra={'ua': request.META.get('HTTP_USER_AGENT', '')}, |
|||
) |
|||
self._inc_block_metric(f'quota_{exceeded}') |
|||
response = HttpResponse(status=429) |
|||
response['Retry-After'] = 86400 |
|||
response['X-RateLimit-Reason'] = f'quota_{exceeded}' |
|||
return response |
|||
|
|||
# 7. Per-minute rate limit — 60/min for all callers (anon and auth). |
|||
# Auth is not exempt: authenticating must not bypass this cap. |
|||
# Writes rl:api:ip:<ip>:blocked only — never rl:ip:<ip>:blocked. |
|||
if self.api_rate_limit_enabled: |
|||
count = self._incr_with_ttl(RL_API_IP_REQUESTS.format(ns=_NAMESPACE, ip=ip), self.api_window) |
|||
if count >= self.api_threshold: |
|||
_set_block(RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip), _index_shard(ip, RL_INDEX_API_BLOCKED_IPS), self.api_block_seconds) |
|||
logger.warning( |
|||
'api_rate_limit_block reason=api_threshold_exceeded ' |
|||
'ip=%s path=%s user_agent=%s count=%s threshold=%s', |
|||
ip, request.path, request.META.get('HTTP_USER_AGENT', ''), |
|||
count, self.api_threshold, |
|||
) |
|||
self._inc_block_metric('api_threshold_exceeded') |
|||
return self._api_block_response('api_threshold_exceeded') |
|||
return self.get_response(request) |
|||
|
|||
# ------------------------------------------------------------------ |
|||
# Evaluation |
|||
# ------------------------------------------------------------------ |
|||
|
|||
def _evaluate(self, request): |
|||
ip = get_client_ip(request) |
|||
|
|||
# Check 0: IP-prefix block — operator-curated deny list, applies to everyone |
|||
if self._is_ip_prefix_blocked(ip): |
|||
return {'action': 'block', 'reason': 'ip_prefix_blocked', 'ip': ip} |
|||
|
|||
# Check 1: known bad UA (hardcoded Python list — substring match) |
|||
ua = request.META.get('HTTP_USER_AGENT', '') |
|||
for fragment in BOT_UA_FRAGMENTS: |
|||
if fragment.lower() in ua.lower(): |
|||
return {'action': 'block', 'reason': 'known_ua', 'ip': ip} |
|||
|
|||
# Check 1b: runtime UA deny list (Redis SET — token hash match) |
|||
if self._is_redis_blocked_ua(ua): |
|||
return {'action': 'block', 'reason': 'redis_ua', 'ip': ip} |
|||
|
|||
# Check 2: IP already blocked — authenticated users are exempt since they |
|||
# have independent per-user limiting at check 3c; IP blocks target anonymous traffic. |
|||
user = getattr(request, 'user', None) |
|||
if not (user and user.is_authenticated) and self._rl_cache.get(RL_IP_BLOCKED.format(ip=ip)): |
|||
return {'action': 'block', 'reason': 'ip_blocked', 'ip': ip} |
|||
|
|||
if user is not None and user.is_authenticated: |
|||
return self._evaluate_authenticated(request, ip) |
|||
return self._evaluate_anonymous(request, ip) |
|||
|
|||
def _evaluate_authenticated(self, request, ip): |
|||
uid = str(request.user.pk) |
|||
|
|||
# Check 3a: user already blocked |
|||
if self._rl_cache.get(RL_USER_BLOCKED.format(ns=_NAMESPACE, uid=uid)): |
|||
return {'action': 'block', 'reason': 'user_blocked', 'ip': ip} |
|||
|
|||
# Check 3b: suspicious headers |
|||
if _is_suspicious_headers(request): |
|||
return {'action': 'block', 'reason': 'suspicious_headers_auth', 'ip': ip} |
|||
|
|||
# Check 3c: authenticated request rate — return 429 for this request only; |
|||
# no persistent block key so the window resets naturally after auth_window |
|||
# seconds. A 300s lockout is wrong for a logged-in user who clicked fast. |
|||
count = self._incr_with_ttl( |
|||
RL_USER_REQUESTS.format(ns=_NAMESPACE, uid=uid), ttl=self.auth_window |
|||
) |
|||
if count >= self.auth_threshold: |
|||
return {'action': 'block', 'reason': 'auth_user_rate', 'ip': ip} |
|||
|
|||
return {'action': 'pass', 'ip': ip} |
|||
|
|||
def _evaluate_anonymous(self, request, ip): |
|||
# Check 4a: suspicious headers |
|||
if _is_suspicious_headers(request): |
|||
return {'action': 'block', 'reason': 'suspicious_headers', 'ip': ip} |
|||
|
|||
# Check 4b: IP request rate |
|||
count = self._incr_with_ttl(RL_IP_REQUESTS.format(ip=ip), ttl=self.anon_window) |
|||
if count >= self.anon_threshold: |
|||
_set_block(RL_IP_BLOCKED.format(ip=ip), _index_shard(ip, RL_INDEX_BLOCKED_IPS), self.BLOCK_TTL) |
|||
return {'action': 'block', 'reason': 'ip_rate', 'ip': ip} |
|||
|
|||
# Check 4c: per-namespace/IP/window (catches UA rotators behind NAT) |
|||
bucket = int(time.time() // self.anon_window) |
|||
count = self._incr_with_ttl( |
|||
RL_NS_WINDOW.format(ns=_NAMESPACE, ip=ip, bucket=bucket), |
|||
ttl=self.anon_window * 2, |
|||
) |
|||
if count >= self.anon_threshold: |
|||
_set_block(RL_IP_BLOCKED.format(ip=ip), _index_shard(ip, RL_INDEX_BLOCKED_IPS), self.BLOCK_TTL) |
|||
return {'action': 'block', 'reason': 'ua_rotation', 'ip': ip} |
|||
|
|||
return {'action': 'pass', 'ip': ip} |
|||
|
|||
# ------------------------------------------------------------------ |
|||
# Helpers — delegate to module-level so media.py can reuse them |
|||
# ------------------------------------------------------------------ |
|||
|
|||
def _handle_not_found(self, request, ip): |
|||
""" |
|||
Block IPs that accumulate too many 404s in one window — catches scanner |
|||
probes that use paths without recognised extensions (e.g. /wp-login, |
|||
/.git/HEAD, /xmlrpc) and bypass check 2b entirely. |
|||
Only anonymous requests are counted; authenticated users have their own |
|||
per-user rate limit and may legitimately hit stale bookmarks. |
|||
""" |
|||
user = getattr(request, 'user', None) |
|||
if user and user.is_authenticated: |
|||
return |
|||
count = self._incr_with_ttl(RL_IP_404S.format(ip=ip), ttl=self.anon_window) |
|||
if count >= self.not_found_threshold: |
|||
_set_block(RL_IP_BLOCKED.format(ip=ip), _index_shard(ip, RL_INDEX_BLOCKED_IPS), self.BLOCK_TTL) |
|||
logger.warning( |
|||
'ratelimit_block layer=django reason=404_scan ip=%s path=%s namespace=%s', |
|||
ip, request.path, _NAMESPACE, |
|||
extra={'ua': request.META.get('HTTP_USER_AGENT', '')}, |
|||
) |
|||
self._inc_block_metric('404_scan') |
|||
|
|||
def _check_api_quota(self, request): |
|||
""" |
|||
Increment daily and weekly API quota counters for all /api/ callers. |
|||
All callers are keyed by IP — auth status is not checked. |
|||
|
|||
Keys are HASH structures: one hash per tenant per period, field = IP. |
|||
This keeps the global keyspace at O(tenants) instead of O(unique IPs), |
|||
reducing Redis memory ~57% vs. per-IP STRING keys at scale. |
|||
|
|||
Fails open (returns None) if Redis/cache is unavailable. |
|||
""" |
|||
today = date.today() |
|||
iso = today.isocalendar() |
|||
date_str = today.isoformat() |
|||
week_str = f'{iso[0]}-W{iso[1]:02d}' |
|||
|
|||
ip = get_client_ip(request) |
|||
api_d_hash = API_QUOTA_DAILY_HASH.format(ns=_NAMESPACE, date=date_str) |
|||
api_w_hash = API_QUOTA_WEEKLY_HASH.format(ns=_NAMESPACE, week=week_str) |
|||
|
|||
try: |
|||
if _hincrby_with_ttl(api_d_hash, ip, 86400) > self.api_quota_daily: |
|||
return 'daily' |
|||
if _hincrby_with_ttl(api_w_hash, ip, 7 * 86400) > self.api_quota_weekly: |
|||
return 'weekly' |
|||
except Exception: |
|||
pass # fail open — quota not enforced when Redis unavailable |
|||
return None |
|||
|
|||
def _incr_with_ttl(self, key, ttl): |
|||
return _incr_with_ttl(key, ttl) |
|||
|
|||
def _inc_block_metric(self, reason): |
|||
"""Increment daily per-reason block counter in Redis DB 1 (TTL 8 days).""" |
|||
hash_key = RL_METRICS_BLOCKED.format(ns=_NAMESPACE, date=date.today().isoformat()) |
|||
try: |
|||
_hincrby_with_ttl(hash_key, reason, ttl=8 * 86400) |
|||
except Exception: |
|||
pass |
|||
|
|||
def _refresh_ua_blocklist(self): |
|||
""" |
|||
Fetch the full UA deny list from Redis DB 1 (SMEMBERS). |
|||
Stores sha256 hex-strings in the class-level set. |
|||
Falls back silently — an empty set means no runtime blocks. |
|||
""" |
|||
try: |
|||
from django_redis import get_redis_connection |
|||
client = get_redis_connection('ratelimit') |
|||
raw = client.smembers(RL_UA_BLOCKLIST) |
|||
RateLimitMiddleware._ua_blocklist = { |
|||
m.decode() if isinstance(m, bytes) else m for m in raw |
|||
} |
|||
RateLimitMiddleware._ua_blocklist_fetched_at = time.time() |
|||
logger.debug('[RATELIMIT] ua_blocklist refreshed entries=%d', len(raw)) |
|||
except Exception as exc: |
|||
logger.debug('[RATELIMIT] ua_blocklist refresh skipped: %s', exc) |
|||
|
|||
def _is_redis_blocked_ua(self, ua): |
|||
""" |
|||
Return True if any slash/space/semicolon token in `ua` has a sha256 |
|||
that appears in the Redis UA deny list. |
|||
|
|||
The SET stores sha256(fragment) — e.g. sha256('GPTBot'). |
|||
Tokenising by common UA separators means 'GPTBot/1.1 (OpenAI)' |
|||
produces token 'GPTBot' whose hash matches the seeded entry. |
|||
Degrades to False when Redis is unavailable. |
|||
""" |
|||
if time.time() - self._ua_blocklist_fetched_at > settings.RATE_LIMITER_UA_BLOCKLIST_REFRESH: |
|||
self._refresh_ua_blocklist() |
|||
if not self._ua_blocklist: |
|||
return False |
|||
tokens = re.split(r'[\s/;()+,]+', ua) |
|||
return any( |
|||
hashlib.sha256(t.encode()).hexdigest() in self._ua_blocklist |
|||
for t in tokens if t |
|||
) |
|||
|
|||
def _refresh_ip_prefix_blocklist(self): |
|||
""" |
|||
Fetch the full IP-prefix deny list from Redis DB 1 (SMEMBERS) and |
|||
normalise entries to trailing-dot form so membership checks are O(1). |
|||
|
|||
Normalisation: strip trailing dot, then re-add if fewer than 3 dots |
|||
(i.e. it's a prefix, not a full dotted-quad). Examples: |
|||
'103.124.225' → '103.124.225.' |
|||
'103.124.225.' → '103.124.225.' |
|||
'103.124.225.7'→ '103.124.225.7' (exact IP, no trailing dot) |
|||
""" |
|||
try: |
|||
from django_redis import get_redis_connection |
|||
client = get_redis_connection('ratelimit') |
|||
raw = client.smembers(RL_IP_PREFIX_BLOCKLIST) |
|||
normalized = set() |
|||
for m in raw: |
|||
entry = m.decode() if isinstance(m, bytes) else m |
|||
stripped = entry.rstrip('.') |
|||
normalized.add(stripped + '.' if stripped.count('.') < 3 else stripped) |
|||
RateLimitMiddleware._ip_prefix_blocklist = normalized |
|||
RateLimitMiddleware._ip_prefix_blocklist_fetched_at = time.time() |
|||
logger.debug('[RATELIMIT] ip_prefix_blocklist refreshed entries=%d', len(normalized)) |
|||
except Exception as exc: |
|||
logger.debug('[RATELIMIT] ip_prefix_blocklist refresh skipped: %s', exc) |
|||
|
|||
def _is_ip_prefix_blocked(self, ip): |
|||
""" |
|||
Return True if `ip` or any of its dot-anchored prefixes is in the |
|||
local IP-prefix deny set. |
|||
|
|||
Generates up to 4 candidates for '203.0.113.42': |
|||
'203.', '203.0.', '203.0.113.', '203.0.113.42' |
|||
Each lookup is O(1) against the normalised in-process set. |
|||
Degrades to False when Redis is unavailable (empty set). |
|||
""" |
|||
if time.time() - self._ip_prefix_blocklist_fetched_at > settings.RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH: |
|||
self._refresh_ip_prefix_blocklist() |
|||
if not self._ip_prefix_blocklist: |
|||
return False |
|||
parts = ip.split('.') |
|||
if len(parts) != 4: |
|||
return False |
|||
candidates = ( |
|||
parts[0] + '.', |
|||
parts[0] + '.' + parts[1] + '.', |
|||
parts[0] + '.' + parts[1] + '.' + parts[2] + '.', |
|||
ip, |
|||
) |
|||
return any(c in self._ip_prefix_blocklist for c in candidates) |
|||
@ -0,0 +1,926 @@ |
|||
""" |
|||
Unit tests for sapl/middleware/ratelimit.py. |
|||
|
|||
No database access is needed — all tests use RequestFactory and mocks. |
|||
Redis is never contacted; _incr_with_ttl is either mocked directly on the |
|||
middleware instance or the fallback non-atomic path is exercised via the |
|||
mock cache. |
|||
""" |
|||
|
|||
import time |
|||
|
|||
import pytest |
|||
from unittest.mock import MagicMock, patch |
|||
from django.test import RequestFactory |
|||
|
|||
from sapl.middleware.ratelimit import ( |
|||
_NAMESPACE, |
|||
_hincrby_with_ttl, |
|||
_index_shard, |
|||
_is_same_origin, |
|||
_is_suspicious_headers, |
|||
_parse_rate, |
|||
get_client_ip, |
|||
make_ratelimit_cache_key, |
|||
API_QUOTA_DAILY_HASH, |
|||
API_QUOTA_WEEKLY_HASH, |
|||
RateLimitMiddleware, |
|||
RL_API_IP_BLOCKED, |
|||
RL_API_IP_REQUESTS, |
|||
RL_INDEX_BLOCKED_IPS, |
|||
RL_IP_BLOCKED, |
|||
RL_IP_PREFIX_BLOCKLIST, |
|||
RL_USER_BLOCKED, |
|||
smart_key, |
|||
smart_rate, |
|||
) |
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Shared test helpers |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
_factory = RequestFactory() |
|||
|
|||
# Headers that a normal browser would send — used as the default baseline. |
|||
_NORMAL_HEADERS = { |
|||
'HTTP_ACCEPT': 'text/html,application/xhtml+xml', |
|||
'HTTP_ACCEPT_LANGUAGE': 'pt-BR,pt;q=0.9', |
|||
} |
|||
|
|||
|
|||
def _req(ip='1.2.3.4', ua='Mozilla/5.0', path='/', extra_meta=None): |
|||
"""GET request with sensible defaults and browser-like headers.""" |
|||
request = _factory.get(path) |
|||
request.META.update({'REMOTE_ADDR': ip, 'HTTP_USER_AGENT': ua, **_NORMAL_HEADERS}) |
|||
if extra_meta: |
|||
request.META.update(extra_meta) |
|||
return request |
|||
|
|||
|
|||
def _anon_req(**kwargs): |
|||
r = _req(**kwargs) |
|||
r.user = MagicMock(is_authenticated=False) |
|||
return r |
|||
|
|||
|
|||
def _auth_req(uid=7, **kwargs): |
|||
r = _req(**kwargs) |
|||
r.user = MagicMock(is_authenticated=True, pk=uid) |
|||
return r |
|||
|
|||
|
|||
def _make_middleware( |
|||
anon_rate='35/m', |
|||
auth_rate='120/m', |
|||
api_rate_limit_enabled=True, |
|||
api_threshold=60, |
|||
api_window=60, |
|||
api_block_seconds=300, |
|||
api_same_origin_bypass=True, |
|||
): |
|||
""" |
|||
Return (middleware, mock_cache). |
|||
|
|||
The ratelimit cache is replaced with a MagicMock whose .get() returns None |
|||
by default (nothing blocked, no counters set). Tests may replace |
|||
mock_cache.get.side_effect or mock mw._incr_with_ttl directly. |
|||
|
|||
sapl.middleware.ratelimit imports settings as `from sapl import settings` |
|||
(a direct module reference), so django.test.override_settings has no effect |
|||
on it. We patch the name in the ratelimit module's namespace instead. |
|||
""" |
|||
mock_cache = MagicMock() |
|||
mock_cache.get.return_value = None |
|||
get_response = MagicMock(return_value=MagicMock(status_code=200)) |
|||
|
|||
mock_settings = MagicMock() |
|||
mock_settings.RATE_LIMITER_RATE = anon_rate |
|||
mock_settings.RATE_LIMITER_RATE_AUTHENTICATED = auth_rate |
|||
mock_settings.RATE_LIMITER_RATE_BOT = '5/m' |
|||
mock_settings.RATE_LIMIT_404_THRESHOLD = 20 |
|||
mock_settings.RATE_LIMIT_BYPASS_PATHS = [] |
|||
mock_settings.POD_NAMESPACE = _NAMESPACE # keep module-level _NAMESPACE consistent |
|||
mock_settings.API_QUOTA_DAILY = 999999 |
|||
mock_settings.API_QUOTA_WEEKLY = 999999 |
|||
mock_settings.RATE_LIMITER_UA_BLOCKLIST_REFRESH = 60 |
|||
mock_settings.RATE_LIMITER_IP_PREFIX_BLOCKLIST_REFRESH = 60 |
|||
mock_settings.API_RATE_LIMIT_ENABLED = api_rate_limit_enabled |
|||
mock_settings.API_RATE_LIMIT_THRESHOLD = api_threshold |
|||
mock_settings.API_RATE_LIMIT_WINDOW_SECONDS = api_window |
|||
mock_settings.API_RATE_LIMIT_BLOCK_SECONDS = api_block_seconds |
|||
mock_settings.API_RATE_LIMIT_SAME_ORIGIN_BYPASS = api_same_origin_bypass |
|||
mock_settings.RATE_LIMITER_INDEX_SHARDS = 3 |
|||
|
|||
with ( |
|||
patch('sapl.middleware.ratelimit.caches') as mock_caches, |
|||
patch('sapl.middleware.ratelimit.settings', mock_settings), |
|||
): |
|||
mock_caches.__getitem__.return_value = mock_cache |
|||
mw = RateLimitMiddleware(get_response) |
|||
# __init__ already set mw._rl_cache = caches['ratelimit'] == mock_cache, |
|||
# but reassign explicitly so tests have a direct handle on the same object. |
|||
mw._rl_cache = mock_cache |
|||
return mw, mock_cache |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _parse_rate |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
@pytest.mark.parametrize('rate_str,expected', [ |
|||
('35/m', (35, 60)), |
|||
('120/m', (120, 60)), |
|||
('10/s', (10, 1)), |
|||
('5/h', (5, 3600)), |
|||
('1/M', (1, 60)), # period is case-insensitive |
|||
]) |
|||
def test_parse_rate(rate_str, expected): |
|||
assert _parse_rate(rate_str) == expected |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# make_ratelimit_cache_key — pass-through, no mangling |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_make_ratelimit_cache_key_passthrough(): |
|||
assert make_ratelimit_cache_key('rl:ip:1.2.3.4:reqs', 'some_prefix', 1) == 'rl:ip:1.2.3.4:reqs' |
|||
assert make_ratelimit_cache_key('rl:abc123', '', 99) == 'rl:abc123' |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _is_suspicious_headers |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_suspicious_both_headers_missing(): |
|||
r = _factory.get('/') |
|||
r.META.pop('HTTP_ACCEPT', None) |
|||
r.META.pop('HTTP_ACCEPT_LANGUAGE', None) |
|||
assert _is_suspicious_headers(r) is True |
|||
|
|||
|
|||
def test_suspicious_one_header_missing_is_not_suspicious(): |
|||
"""Only flagged when *both* headers are absent.""" |
|||
r = _factory.get('/') |
|||
r.META['HTTP_ACCEPT'] = 'text/html' |
|||
r.META.pop('HTTP_ACCEPT_LANGUAGE', None) |
|||
assert _is_suspicious_headers(r) is False |
|||
|
|||
|
|||
def test_suspicious_both_headers_present(): |
|||
r = _factory.get('/') |
|||
r.META['HTTP_ACCEPT'] = 'text/html' |
|||
r.META['HTTP_ACCEPT_LANGUAGE'] = 'pt-BR' |
|||
assert _is_suspicious_headers(r) is False |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# get_client_ip — header priority and XFF chain |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_get_client_ip_remote_addr(): |
|||
r = _factory.get('/') |
|||
r.META['REMOTE_ADDR'] = '10.0.0.1' |
|||
assert get_client_ip(r) == '10.0.0.1' |
|||
|
|||
|
|||
def test_get_client_ip_xff_single(): |
|||
r = _factory.get('/') |
|||
r.META['HTTP_X_FORWARDED_FOR'] = '203.0.113.5' |
|||
assert get_client_ip(r) == '203.0.113.5' |
|||
|
|||
|
|||
def test_get_client_ip_xff_chain_uses_leftmost(): |
|||
"""The leftmost IP in XFF is the real client; the rest are proxies.""" |
|||
r = _factory.get('/') |
|||
r.META['HTTP_X_FORWARDED_FOR'] = '203.0.113.5, 10.0.0.1, 10.0.0.2' |
|||
assert get_client_ip(r) == '203.0.113.5' |
|||
|
|||
|
|||
def test_get_client_ip_x_real_ip_used_when_no_xff(): |
|||
r = _factory.get('/') |
|||
r.META['REMOTE_ADDR'] = '127.0.0.1' |
|||
r.META['HTTP_X_REAL_IP'] = '203.0.113.9' |
|||
assert get_client_ip(r) == '203.0.113.9' |
|||
|
|||
|
|||
def test_get_client_ip_xff_preferred_over_x_real_ip(): |
|||
r = _factory.get('/') |
|||
r.META['HTTP_X_FORWARDED_FOR'] = '203.0.113.1' |
|||
r.META['HTTP_X_REAL_IP'] = '203.0.113.2' |
|||
assert get_client_ip(r) == '203.0.113.1' |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# smart_key / smart_rate |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_smart_key_anon_returns_masked_ip(): |
|||
r = _anon_req(ip='5.5.5.5') |
|||
assert smart_key(None, r) == '5.5.5.5' |
|||
|
|||
|
|||
def test_smart_key_auth_returns_pk_string(): |
|||
r = _auth_req(uid=42, ip='5.5.5.5') |
|||
assert smart_key(None, r) == '42' |
|||
|
|||
|
|||
def test_smart_rate_anon_returns_anon_rate(): |
|||
with patch('sapl.middleware.ratelimit.settings') as mock_s: |
|||
mock_s.RATE_LIMITER_RATE = '35/m' |
|||
mock_s.RATE_LIMITER_RATE_AUTHENTICATED = '120/m' |
|||
assert smart_rate(None, _anon_req()) == '35/m' |
|||
|
|||
|
|||
def test_smart_rate_auth_returns_auth_rate(): |
|||
with patch('sapl.middleware.ratelimit.settings') as mock_s: |
|||
mock_s.RATE_LIMITER_RATE = '35/m' |
|||
mock_s.RATE_LIMITER_RATE_AUTHENTICATED = '120/m' |
|||
assert smart_rate(None, _auth_req()) == '120/m' |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _index_shard — sharded ZSET key routing |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_index_shard_is_deterministic(): |
|||
"""Same IP always maps to the same shard.""" |
|||
from sapl.middleware.ratelimit import _index_shard |
|||
with patch('sapl.middleware.ratelimit.settings') as mock_s: |
|||
mock_s.RATE_LIMITER_INDEX_SHARDS = 3 |
|||
key1 = _index_shard('1.2.3.4', 'rl:index:blocked_ips') |
|||
key2 = _index_shard('1.2.3.4', 'rl:index:blocked_ips') |
|||
assert key1 == key2 |
|||
|
|||
|
|||
def test_index_shard_stays_within_range(): |
|||
"""Shard suffix is always 0 … N-1.""" |
|||
from sapl.middleware.ratelimit import _index_shard |
|||
import re |
|||
with patch('sapl.middleware.ratelimit.settings') as mock_s: |
|||
mock_s.RATE_LIMITER_INDEX_SHARDS = 3 |
|||
ips = [f'10.0.0.{i}' for i in range(50)] |
|||
for ip in ips: |
|||
key = _index_shard(ip, 'rl:index:blocked_ips') |
|||
m = re.search(r':(\d+)$', key) |
|||
assert m and 0 <= int(m.group(1)) < 3, f'out-of-range shard for {ip}: {key}' |
|||
|
|||
|
|||
def test_index_shard_distributes_across_shards(): |
|||
"""With enough IPs, all 3 shards are used.""" |
|||
from sapl.middleware.ratelimit import _index_shard |
|||
with patch('sapl.middleware.ratelimit.settings') as mock_s: |
|||
mock_s.RATE_LIMITER_INDEX_SHARDS = 3 |
|||
shards_seen = { |
|||
_index_shard(f'192.168.{i}.{j}', 'rl:index:blocked_ips').split(':')[-1] |
|||
for i in range(5) for j in range(10) |
|||
} |
|||
assert shards_seen == {'0', '1', '2'} |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 0 — IP-prefix blocklist (operator-curated SET, dot-anchored matching) |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
@pytest.fixture |
|||
def _seed_prefix_blocklist(): |
|||
""" |
|||
Seed RateLimitMiddleware._ip_prefix_blocklist for the test and restore the |
|||
previous class-level state afterwards (it's shared across instances, like |
|||
the UA blocklist). |
|||
""" |
|||
saved_list = RateLimitMiddleware._ip_prefix_blocklist |
|||
saved_fetched_at = RateLimitMiddleware._ip_prefix_blocklist_fetched_at |
|||
|
|||
def _seed(prefixes): |
|||
RateLimitMiddleware._ip_prefix_blocklist = set(prefixes) |
|||
RateLimitMiddleware._ip_prefix_blocklist_fetched_at = time.time() |
|||
|
|||
yield _seed |
|||
|
|||
RateLimitMiddleware._ip_prefix_blocklist = saved_list |
|||
RateLimitMiddleware._ip_prefix_blocklist_fetched_at = saved_fetched_at |
|||
|
|||
|
|||
def test_is_ip_prefix_blocked_matches_dot_boundary(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['103.124.225']) |
|||
assert mw._is_ip_prefix_blocked('103.124.225.7') is True |
|||
# must not match on raw substring — only on a full-octet boundary |
|||
assert mw._is_ip_prefix_blocked('103.124.2250.1') is False |
|||
assert mw._is_ip_prefix_blocked('103.124.2255') is False |
|||
|
|||
|
|||
def test_is_ip_prefix_blocked_exact_match(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['103.124.225']) |
|||
assert mw._is_ip_prefix_blocked('103.124.225') is True |
|||
|
|||
|
|||
def test_is_ip_prefix_blocked_full_address_matches_only_exactly(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['103.124.225.7']) |
|||
assert mw._is_ip_prefix_blocked('103.124.225.7') is True |
|||
# a full dotted-quad entry must not be treated as a prefix of a longer string |
|||
assert mw._is_ip_prefix_blocked('103.124.225.70') is False |
|||
|
|||
|
|||
def test_is_ip_prefix_blocked_trailing_dot_in_stored_prefix(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['103.124.225.']) |
|||
# anchoring must not double the dot ('103.124.225..') and break the match |
|||
assert mw._is_ip_prefix_blocked('103.124.225.7') is True |
|||
|
|||
|
|||
def test_is_ip_prefix_blocked_empty_list_passes(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist([]) |
|||
assert mw._is_ip_prefix_blocked('1.2.3.4') is False |
|||
|
|||
|
|||
def test_is_ip_prefix_blocked_no_match_passes(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['103.124.225', '45.177.154']) |
|||
assert mw._is_ip_prefix_blocked('1.2.3.4') is False |
|||
|
|||
|
|||
def test_evaluate_blocks_on_ip_prefix_before_other_checks(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['1.2.3']) |
|||
result = mw._evaluate(_anon_req(ip='1.2.3.4', ua='GPTBot')) |
|||
# would otherwise match the known_ua check — prefix block must win |
|||
assert result == {'action': 'block', 'reason': 'ip_prefix_blocked', 'ip': '1.2.3.4'} |
|||
|
|||
|
|||
def test_evaluate_passes_through_when_ip_not_in_prefix_list(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['9.9.9']) |
|||
mw._incr_with_ttl = MagicMock(return_value=1) |
|||
result = mw._evaluate(_anon_req(ip='1.2.3.4')) |
|||
assert result['action'] == 'pass' |
|||
|
|||
|
|||
def test_refresh_ip_prefix_blocklist_populates_set(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist([]) # start empty so the refresh result is observable |
|||
|
|||
mock_client = MagicMock() |
|||
mock_client.smembers.return_value = {b'103.124.225', b'45.177.154'} |
|||
with patch('django_redis.get_redis_connection', return_value=mock_client): |
|||
mw._refresh_ip_prefix_blocklist() |
|||
|
|||
mock_client.smembers.assert_called_once_with(RL_IP_PREFIX_BLOCKLIST) |
|||
assert RateLimitMiddleware._ip_prefix_blocklist == {'103.124.225', '45.177.154'} |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 1 — known bot User-Agent |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
@pytest.mark.parametrize('ua', [ |
|||
'GPTBot/1.0', |
|||
'Mozilla/5.0 (compatible; ClaudeBot/1.0)', |
|||
'PerplexityBot', |
|||
'Bytespider', |
|||
'AhrefsBot/7.0', |
|||
'meta-externalagent/1.1', |
|||
'OAI-SearchBot', |
|||
'Mozilla/5.0 (compatible; bingbot/2.0)', |
|||
'SERankingBacklinksBot/1.0', |
|||
'Mozilla/5.0 AppleWebKit Chrome/98.0.4758.80', |
|||
]) |
|||
def test_known_bot_ua_blocked(ua): |
|||
mw, _ = _make_middleware() |
|||
result = mw._evaluate(_anon_req(ua=ua)) |
|||
assert result == {'action': 'block', 'reason': 'known_ua', 'ip': '1.2.3.4'} |
|||
|
|||
|
|||
def test_bot_ua_check_is_case_insensitive(): |
|||
mw, _ = _make_middleware() |
|||
result = mw._evaluate(_anon_req(ua='gptbot/2.0')) |
|||
assert result['reason'] == 'known_ua' |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 2 — IP already blocked in cache |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_ip_blocked_in_cache(): |
|||
mw, mock_cache = _make_middleware() |
|||
ip = '1.2.3.4' |
|||
mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None |
|||
result = mw._evaluate(_anon_req(ip=ip)) |
|||
assert result == {'action': 'block', 'reason': 'ip_blocked', 'ip': ip} |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 3a — authenticated user blocked in cache |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_auth_user_blocked_in_cache(): |
|||
mw, mock_cache = _make_middleware() |
|||
uid = '7' |
|||
mock_cache.get.side_effect = lambda key: ( |
|||
1 if key == RL_USER_BLOCKED.format(ns=_NAMESPACE, uid=uid) else None |
|||
) |
|||
result = mw._evaluate(_auth_req(uid=int(uid))) |
|||
assert result == {'action': 'block', 'reason': 'user_blocked', 'ip': '1.2.3.4'} |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 3b — authenticated + suspicious headers |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_auth_suspicious_headers_blocked(): |
|||
mw, _ = _make_middleware() |
|||
r = _auth_req() |
|||
r.META.pop('HTTP_ACCEPT', None) |
|||
r.META.pop('HTTP_ACCEPT_LANGUAGE', None) |
|||
r.META.pop('HTTP_USER_AGENT', None) |
|||
result = mw._evaluate(r) |
|||
assert result == {'action': 'block', 'reason': 'suspicious_headers_auth', 'ip': '1.2.3.4'} |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 3c — authenticated request rate |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_auth_rate_exceeded_blocks_and_marks_user_blocked(): |
|||
mw, mock_cache = _make_middleware(auth_rate='5/m') |
|||
mw._incr_with_ttl = MagicMock(return_value=5) # exactly at threshold |
|||
result = mw._evaluate(_auth_req(uid=7)) |
|||
# auth_user_rate has no persistent block key — the window resets naturally |
|||
assert result == {'action': 'block', 'reason': 'auth_user_rate', 'ip': '1.2.3.4'} |
|||
mock_cache.set.assert_not_called() |
|||
|
|||
|
|||
def test_auth_under_rate_passes(): |
|||
mw, mock_cache = _make_middleware(auth_rate='5/m') |
|||
mw._incr_with_ttl = MagicMock(return_value=4) # one below threshold |
|||
result = mw._evaluate(_auth_req(uid=7)) |
|||
assert result == {'action': 'pass', 'ip': '1.2.3.4'} |
|||
mock_cache.set.assert_not_called() |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 4a — anonymous + suspicious headers |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_anon_suspicious_headers_blocked(): |
|||
mw, _ = _make_middleware() |
|||
r = _anon_req() |
|||
r.META.pop('HTTP_ACCEPT', None) |
|||
r.META.pop('HTTP_ACCEPT_LANGUAGE', None) |
|||
r.META.pop('HTTP_USER_AGENT', None) |
|||
result = mw._evaluate(r) |
|||
assert result == {'action': 'block', 'reason': 'suspicious_headers', 'ip': '1.2.3.4'} |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 4b — anonymous IP request rate |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_anon_ip_rate_exceeded_blocks_and_marks_ip_blocked(): |
|||
mw, _ = _make_middleware(anon_rate='5/m') |
|||
mw._incr_with_ttl = MagicMock(return_value=5) # first call (IP counter) hits threshold |
|||
with patch('sapl.middleware.ratelimit._set_block') as mock_set_block: |
|||
result = mw._evaluate(_anon_req()) |
|||
assert result == {'action': 'block', 'reason': 'ip_rate', 'ip': '1.2.3.4'} |
|||
mock_set_block.assert_called_once_with( |
|||
RL_IP_BLOCKED.format(ip='1.2.3.4'), |
|||
_index_shard('1.2.3.4', RL_INDEX_BLOCKED_IPS), |
|||
RateLimitMiddleware.BLOCK_TTL, |
|||
) |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# Check 4c — per-namespace/IP/window (UA rotation detection) |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_anon_ua_rotation_detected_blocks_and_marks_ip_blocked(): |
|||
mw, _ = _make_middleware(anon_rate='5/m') |
|||
# First call (IP counter) is under threshold; second (window counter) hits it. |
|||
mw._incr_with_ttl = MagicMock(side_effect=[4, 5]) |
|||
with patch('sapl.middleware.ratelimit._set_block') as mock_set_block: |
|||
result = mw._evaluate(_anon_req()) |
|||
assert result == {'action': 'block', 'reason': 'ua_rotation', 'ip': '1.2.3.4'} |
|||
mock_set_block.assert_called_once_with( |
|||
RL_IP_BLOCKED.format(ip='1.2.3.4'), |
|||
_index_shard('1.2.3.4', RL_INDEX_BLOCKED_IPS), |
|||
RateLimitMiddleware.BLOCK_TTL, |
|||
) |
|||
|
|||
|
|||
def test_anon_under_all_thresholds_passes(): |
|||
mw, mock_cache = _make_middleware(anon_rate='5/m') |
|||
mw._incr_with_ttl = MagicMock(return_value=4) # both counters below threshold |
|||
result = mw._evaluate(_anon_req()) |
|||
assert result == {'action': 'pass', 'ip': '1.2.3.4'} |
|||
mock_cache.set.assert_not_called() |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# __call__ — block returns 429, pass forwards to get_response |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_call_block_returns_429_with_retry_after_header(): |
|||
mw, _ = _make_middleware() |
|||
mw._evaluate = MagicMock(return_value={'action': 'block', 'reason': 'known_ua', 'ip': '1.2.3.4'}) |
|||
response = mw(_factory.get('/')) |
|||
assert response.status_code == 429 |
|||
assert response['Retry-After'] == str(RateLimitMiddleware.BLOCK_TTL) |
|||
mw.get_response.assert_not_called() |
|||
|
|||
|
|||
def test_call_ip_prefix_block_returns_403_without_retry_after_header(): |
|||
mw, _ = _make_middleware() |
|||
mw._evaluate = MagicMock(return_value={'action': 'block', 'reason': 'ip_prefix_blocked', 'ip': '1.2.3.4'}) |
|||
response = mw(_factory.get('/')) |
|||
assert response.status_code == 403 |
|||
assert 'Retry-After' not in response |
|||
assert response['X-RateLimit-Reason'] == 'ip_prefix_blocked' |
|||
mw.get_response.assert_not_called() |
|||
|
|||
|
|||
def test_call_pass_forwards_request_to_get_response(): |
|||
mw, _ = _make_middleware() |
|||
mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) |
|||
request = _anon_req() |
|||
mw(request) |
|||
mw.get_response.assert_called_once_with(request) |
|||
|
|||
|
|||
def test_call_rejects_duplicate_page_param_with_400(): |
|||
mw, _ = _make_middleware() |
|||
request = _factory.get('/sessao/pesquisar-sessao', data={'page': ['3', '2', '3']}) |
|||
response = mw(request) |
|||
assert response.status_code == 400 |
|||
assert response['X-RateLimit-Reason'] == 'param_pollution' |
|||
mw.get_response.assert_not_called() |
|||
|
|||
|
|||
def test_call_allows_single_page_param(): |
|||
mw, _ = _make_middleware() |
|||
mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) |
|||
request = _factory.get('/sessao/pesquisar-sessao', data={'page': '2'}) |
|||
mw(request) |
|||
mw.get_response.assert_called_once_with(request) |
|||
|
|||
|
|||
def test_call_allows_no_page_param(): |
|||
mw, _ = _make_middleware() |
|||
mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) |
|||
request = _factory.get('/sessao/pesquisar-sessao') |
|||
mw(request) |
|||
mw.get_response.assert_called_once_with(request) |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _is_same_origin |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_is_same_origin_no_headers_returns_false(): |
|||
r = _factory.get('/api/materia/') |
|||
r.META['SERVER_NAME'] = 'sapl.example.com' |
|||
r.META['SERVER_PORT'] = '80' |
|||
r.META.pop('HTTP_ORIGIN', None) |
|||
r.META.pop('HTTP_REFERER', None) |
|||
assert _is_same_origin(r) is False |
|||
|
|||
|
|||
def test_is_same_origin_matching_origin(): |
|||
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') |
|||
r.META['HTTP_ORIGIN'] = 'https://sapl.example.com' |
|||
assert _is_same_origin(r) is True |
|||
|
|||
|
|||
def test_is_same_origin_mismatched_origin(): |
|||
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') |
|||
r.META['HTTP_ORIGIN'] = 'https://other.example.com' |
|||
assert _is_same_origin(r) is False |
|||
|
|||
|
|||
def test_is_same_origin_wrong_origin_blocks_even_if_referer_matches(): |
|||
"""If Origin is present and wrong, Referer must not be consulted.""" |
|||
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') |
|||
r.META['HTTP_ORIGIN'] = 'https://evil.com' |
|||
r.META['HTTP_REFERER'] = 'https://sapl.example.com/page/' |
|||
assert _is_same_origin(r) is False |
|||
|
|||
|
|||
def test_is_same_origin_referer_used_when_no_origin(): |
|||
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') |
|||
r.META.pop('HTTP_ORIGIN', None) |
|||
r.META['HTTP_REFERER'] = 'https://sapl.example.com/page/?q=1' |
|||
assert _is_same_origin(r) is True |
|||
|
|||
|
|||
def test_is_same_origin_port_stripped_from_both_sides(): |
|||
"""Host with port and Origin without port must match after normalization.""" |
|||
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='8000') |
|||
r.META['HTTP_HOST'] = 'sapl.example.com:8000' |
|||
r.META['HTTP_ORIGIN'] = 'http://sapl.example.com' |
|||
assert _is_same_origin(r) is True |
|||
|
|||
|
|||
def test_is_same_origin_case_insensitive(): |
|||
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80') |
|||
r.META['HTTP_ORIGIN'] = 'https://SAPL.EXAMPLE.COM' |
|||
assert _is_same_origin(r) is True |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _handle_api — OPTIONS and same-origin bypass |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def _api_req(ip='1.2.3.4', ua='Mozilla/5.0', path='/api/materia/', method='GET', extra_meta=None): |
|||
"""Anonymous /api/ request with browser headers.""" |
|||
request = _factory.generic(method, path) |
|||
request.META.update({'REMOTE_ADDR': ip, 'HTTP_USER_AGENT': ua, **_NORMAL_HEADERS}) |
|||
if extra_meta: |
|||
request.META.update(extra_meta) |
|||
request.user = MagicMock(is_authenticated=False) |
|||
return request |
|||
|
|||
|
|||
def test_api_options_passes_without_counting(): |
|||
mw, _ = _make_middleware() |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(method='OPTIONS') |
|||
mw(request) |
|||
mw.get_response.assert_called_once_with(request) |
|||
mw._incr_with_ttl.assert_not_called() |
|||
|
|||
|
|||
def test_api_same_origin_passes_without_counting(): |
|||
mw, _ = _make_middleware() |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(extra_meta={ |
|||
'SERVER_NAME': 'sapl.example.com', |
|||
'SERVER_PORT': '80', |
|||
'HTTP_HOST': 'sapl.example.com', |
|||
'HTTP_ORIGIN': 'https://sapl.example.com', |
|||
}) |
|||
mw(request) |
|||
mw.get_response.assert_called_once_with(request) |
|||
mw._incr_with_ttl.assert_not_called() |
|||
|
|||
|
|||
def test_api_malicious_origin_is_not_same_origin(): |
|||
mw, _ = _make_middleware(api_threshold=999) |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock(return_value=1) |
|||
request = _api_req(extra_meta={ |
|||
'SERVER_NAME': 'sapl.example.com', |
|||
'SERVER_PORT': '80', |
|||
'HTTP_HOST': 'sapl.example.com', |
|||
'HTTP_ORIGIN': 'https://evil.com?x=sapl.example.com', |
|||
}) |
|||
mw(request) |
|||
# Must reach the counter (not short-circuit as same-origin) |
|||
mw._incr_with_ttl.assert_called_once() |
|||
|
|||
|
|||
# Same-origin headers a spoofing client can trivially forge — Origin/Referer |
|||
# carry no authentication, so they must never override an active block. |
|||
_SAME_ORIGIN_META = { |
|||
'SERVER_NAME': 'sapl.example.com', |
|||
'SERVER_PORT': '80', |
|||
'HTTP_HOST': 'sapl.example.com', |
|||
'HTTP_ORIGIN': 'https://sapl.example.com', |
|||
} |
|||
|
|||
|
|||
def test_api_same_origin_does_not_bypass_global_ip_block(): |
|||
mw, mock_cache = _make_middleware() |
|||
ip = '1.2.3.4' |
|||
mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(ip=ip, extra_meta=_SAME_ORIGIN_META) |
|||
|
|||
response = mw(request) |
|||
|
|||
mw.get_response.assert_not_called() |
|||
assert response.status_code == 429 |
|||
assert response['X-RateLimit-Reason'] == 'global_ip_blocked' |
|||
|
|||
|
|||
def test_api_same_origin_does_not_bypass_api_ip_block(): |
|||
mw, mock_cache = _make_middleware() |
|||
ip = '1.2.3.4' |
|||
blocked_key = RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip) |
|||
mock_cache.get.side_effect = lambda key: 1 if key == blocked_key else None |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(ip=ip, extra_meta=_SAME_ORIGIN_META) |
|||
|
|||
response = mw(request) |
|||
|
|||
mw.get_response.assert_not_called() |
|||
assert response.status_code == 429 |
|||
assert response['X-RateLimit-Reason'] == 'api_ip_blocked' |
|||
|
|||
|
|||
def test_api_same_origin_does_not_bypass_ip_prefix_block(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware() |
|||
_seed_prefix_blocklist(['1.2.3']) |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(ip='1.2.3.4', extra_meta=_SAME_ORIGIN_META) |
|||
|
|||
response = mw(request) |
|||
|
|||
mw.get_response.assert_not_called() |
|||
assert response.status_code == 403 |
|||
assert response['X-RateLimit-Reason'] == 'ip_prefix_blocked' |
|||
|
|||
|
|||
def test_api_same_origin_still_skips_quota_and_rate_limit_when_not_blocked(): |
|||
"""Legitimate same-origin polling keeps its original exemption from accounting.""" |
|||
mw, _ = _make_middleware() |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(extra_meta=_SAME_ORIGIN_META) |
|||
|
|||
response = mw(request) |
|||
|
|||
mw.get_response.assert_called_once_with(request) |
|||
mw._check_api_quota.assert_not_called() |
|||
mw._incr_with_ttl.assert_not_called() |
|||
assert response.status_code == 200 |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _handle_api — Check 3a: IP-prefix block (operator-curated deny list) |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_api_blocks_on_ip_prefix_before_quota_checks(_seed_prefix_blocklist): |
|||
mw, mock_cache = _make_middleware() |
|||
_seed_prefix_blocklist(['1.2.3']) |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
request = _api_req(ip='1.2.3.4') |
|||
|
|||
response = mw(request) |
|||
|
|||
mw.get_response.assert_not_called() |
|||
mw._check_api_quota.assert_not_called() |
|||
mw._incr_with_ttl.assert_not_called() |
|||
assert response.status_code == 403 |
|||
assert response['X-RateLimit-Reason'] == 'ip_prefix_blocked' |
|||
|
|||
|
|||
def test_api_passes_through_when_ip_not_in_prefix_list(_seed_prefix_blocklist): |
|||
mw, _ = _make_middleware(api_threshold=999) |
|||
_seed_prefix_blocklist(['9.9.9']) |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock(return_value=1) |
|||
request = _api_req(ip='1.2.3.4') |
|||
|
|||
mw(request) |
|||
|
|||
mw.get_response.assert_called_once_with(request) |
|||
|
|||
|
|||
# --------------------------------------------------------------------------- |
|||
# _handle_api — rate limiting and block key isolation |
|||
# --------------------------------------------------------------------------- |
|||
|
|||
def test_api_external_request_increments_api_counter(): |
|||
mw, _ = _make_middleware(api_threshold=10) |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock(return_value=5) # under threshold |
|||
request = _api_req() |
|||
response = mw(request) |
|||
mw.get_response.assert_called_once_with(request) |
|||
call_args = mw._incr_with_ttl.call_args[0] |
|||
assert call_args[0] == RL_API_IP_REQUESTS.format(ns=_NAMESPACE, ip='1.2.3.4') |
|||
|
|||
|
|||
def test_api_threshold_exceeded_creates_api_block_not_global_block(): |
|||
mw, _ = _make_middleware(api_threshold=5) |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock(return_value=5) # at threshold |
|||
request = _api_req() |
|||
with patch('sapl.middleware.ratelimit._set_block') as mock_set_block: |
|||
response = mw(request) |
|||
assert response.status_code == 429 |
|||
mock_set_block.assert_called_once() |
|||
block_key = mock_set_block.call_args[0][0] |
|||
assert block_key == RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip='1.2.3.4') |
|||
assert block_key != RL_IP_BLOCKED.format(ip='1.2.3.4') |
|||
|
|||
|
|||
def test_api_global_block_also_blocks_api(): |
|||
mw, mock_cache = _make_middleware() |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
ip = '1.2.3.4' |
|||
mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None |
|||
response = mw(_api_req(ip=ip)) |
|||
assert response.status_code == 429 |
|||
assert response['X-RateLimit-Reason'] == 'global_ip_blocked' |
|||
mw._incr_with_ttl.assert_not_called() |
|||
|
|||
|
|||
def test_api_specific_block_blocks_api_only(): |
|||
mw, mock_cache = _make_middleware() |
|||
mw._check_api_quota = MagicMock(return_value=None) |
|||
mw._incr_with_ttl = MagicMock() |
|||
ip = '1.2.3.4' |
|||
mock_cache.get.side_effect = lambda key: 1 if key == RL_API_IP_BLOCKED.format(ns=_NAMESPACE, ip=ip) else None |
|||
response = mw(_api_req(ip=ip)) |
|||
assert response.status_code == 429 |
|||
assert response['X-RateLimit-Reason'] == 'api_ip_blocked' |
|||
mw._incr_with_ttl.assert_not_called() |
|||
|
|||
|
|||
def test_api_block_response_is_json_with_retry_after(): |
|||
mw, _ = _make_middleware(api_block_seconds=120) |
|||
resp = mw._api_block_response('api_threshold_exceeded') |
|||
assert resp.status_code == 429 |
|||
assert 'application/json' in resp['Content-Type'] |
|||
assert resp['Retry-After'] == '120' |
|||
assert resp['X-RateLimit-Reason'] == 'api_threshold_exceeded' |
|||
|
|||
|
|||
def test_api_auth_user_daily_quota_exceeded_returns_429(): |
|||
"""Auth users are subject to the same daily quota as anon callers (keyed by IP).""" |
|||
mw, _ = _make_middleware() |
|||
request = _api_req(ip='10.0.0.1') |
|||
request.user = MagicMock(is_authenticated=True, pk=42) |
|||
mw.api_quota_daily = 1 |
|||
|
|||
with patch('sapl.middleware.ratelimit._hincrby_with_ttl', return_value=2): |
|||
resp = mw(request) |
|||
|
|||
assert resp.status_code == 429 |
|||
assert resp['X-RateLimit-Reason'] == 'quota_daily' |
|||
|
|||
|
|||
def test_api_weekly_quota_exceeded_returns_429(): |
|||
"""Weekly quota block fires when daily passes but weekly counter exceeds limit.""" |
|||
mw, _ = _make_middleware() |
|||
request = _api_req(ip='10.0.0.2') |
|||
mw.api_quota_daily = 999999 |
|||
mw.api_quota_weekly = 1 |
|||
|
|||
# daily returns 1 (under limit), weekly returns 2 (over limit) |
|||
with patch('sapl.middleware.ratelimit._hincrby_with_ttl', side_effect=[1, 2]): |
|||
resp = mw(request) |
|||
|
|||
assert resp.status_code == 429 |
|||
assert resp['X-RateLimit-Reason'] == 'quota_weekly' |
|||
|
|||
|
|||
def test_api_quota_uses_hash_keys(): |
|||
"""_check_api_quota calls _hincrby_with_ttl with HASH keys (no IP in key name).""" |
|||
from datetime import date |
|||
mw, _ = _make_middleware() |
|||
request = _api_req(ip='10.0.0.3') |
|||
mw.api_quota_daily = 999999 |
|||
mw.api_quota_weekly = 999999 |
|||
|
|||
with patch('sapl.middleware.ratelimit._hincrby_with_ttl', return_value=1) as mock_h: |
|||
mw._check_api_quota(request) |
|||
|
|||
today = date.today() |
|||
iso = today.isocalendar() |
|||
expected_daily_hash = API_QUOTA_DAILY_HASH.format(ns=_NAMESPACE, date=today.isoformat()) |
|||
expected_weekly_hash = API_QUOTA_WEEKLY_HASH.format( |
|||
ns=_NAMESPACE, week=f'{iso[0]}-W{iso[1]:02d}' |
|||
) |
|||
calls = mock_h.call_args_list |
|||
assert calls[0][0][0] == expected_daily_hash # first arg of first call = hash key |
|||
assert calls[1][0][0] == expected_weekly_hash # first arg of second call = hash key |
|||
# IP is the field (second positional arg), not embedded in the key |
|||
assert '10.0.0.3' not in calls[0][0][0] |
|||
assert '10.0.0.3' not in calls[1][0][0] |
|||
|
|||
|
|||
def test_inc_block_metric_uses_hash_key(): |
|||
"""_inc_block_metric calls _hincrby_with_ttl with a HASH key; reason is the field.""" |
|||
from datetime import date |
|||
mw, _ = _make_middleware() |
|||
today_str = date.today().isoformat() |
|||
expected_key = f'rl:metrics:{_NAMESPACE}:{today_str}' |
|||
|
|||
with patch('sapl.middleware.ratelimit._hincrby_with_ttl') as mock_h: |
|||
mw._inc_block_metric('ip_rate') |
|||
|
|||
mock_h.assert_called_once() |
|||
args = mock_h.call_args[0] |
|||
assert args[0] == expected_key, f'hash key mismatch: {args[0]!r}' |
|||
assert args[1] == 'ip_rate', f'field (reason) mismatch: {args[1]!r}' |
|||
assert 'ip_rate' not in args[0], 'reason must be HASH field, not embedded in key' |
|||
|
|||
|
|||
def test_non_api_path_uses_global_evaluate_not_api_handler(): |
|||
mw, _ = _make_middleware() |
|||
mw._handle_api = MagicMock() |
|||
mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'}) |
|||
mw(_anon_req(path='/')) |
|||
mw._handle_api.assert_not_called() |
|||
mw._evaluate.assert_called_once() |
|||
@ -0,0 +1,42 @@ |
|||
<!DOCTYPE html> |
|||
<html lang="pt-BR"> |
|||
<head> |
|||
<meta charset="UTF-8"> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|||
<title>429 – Muitas Requisições</title> |
|||
<style> |
|||
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; } |
|||
body { |
|||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; |
|||
background: #f5f5f5; |
|||
color: #333; |
|||
display: flex; |
|||
align-items: center; |
|||
justify-content: center; |
|||
min-height: 100vh; |
|||
} |
|||
.card { |
|||
background: #fff; |
|||
border-top: 4px solid #e74c3c; |
|||
border-radius: 4px; |
|||
padding: 2.5rem 3rem; |
|||
max-width: 480px; |
|||
width: 90%; |
|||
text-align: center; |
|||
box-shadow: 0 2px 8px rgba(0,0,0,.08); |
|||
} |
|||
.code { font-size: 4rem; font-weight: 700; color: #e74c3c; line-height: 1; } |
|||
h1 { font-size: 1.25rem; margin: .75rem 0 1rem; } |
|||
p { font-size: .95rem; line-height: 1.6; color: #555; } |
|||
.retry { margin-top: 1.5rem; font-size: .85rem; color: #888; } |
|||
</style> |
|||
</head> |
|||
<body> |
|||
<div class="card"> |
|||
<div class="code">429</div> |
|||
<h1>Muitas Requisições</h1> |
|||
<p>Você realizou muitas requisições em um curto período. Aguarde um momento e tente novamente.</p> |
|||
<p class="retry">Se o problema persistir, entre em contato com o suporte da sua Câmara Municipal.</p> |
|||
</div> |
|||
</body> |
|||
</html> |
|||
@ -0,0 +1,42 @@ |
|||
<!DOCTYPE html> |
|||
<html lang="pt-BR"> |
|||
<head> |
|||
<meta charset="UTF-8"> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|||
<title>500 – Erro Interno do Servidor</title> |
|||
<style> |
|||
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; } |
|||
body { |
|||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; |
|||
background: #f5f5f5; |
|||
color: #333; |
|||
display: flex; |
|||
align-items: center; |
|||
justify-content: center; |
|||
min-height: 100vh; |
|||
} |
|||
.card { |
|||
background: #fff; |
|||
border-top: 4px solid #e67e22; |
|||
border-radius: 4px; |
|||
padding: 2.5rem 3rem; |
|||
max-width: 480px; |
|||
width: 90%; |
|||
text-align: center; |
|||
box-shadow: 0 2px 8px rgba(0,0,0,.08); |
|||
} |
|||
.code { font-size: 4rem; font-weight: 700; color: #e67e22; line-height: 1; } |
|||
h1 { font-size: 1.25rem; margin: .75rem 0 1rem; } |
|||
p { font-size: .95rem; line-height: 1.6; color: #555; } |
|||
.retry { margin-top: 1.5rem; font-size: .85rem; color: #888; } |
|||
</style> |
|||
</head> |
|||
<body> |
|||
<div class="card"> |
|||
<div class="code">500</div> |
|||
<h1>Erro Interno do Servidor</h1> |
|||
<p>Ocorreu um erro inesperado. Nossa equipe foi notificada e trabalhará para resolver o problema.</p> |
|||
<p class="retry">Tente novamente em alguns instantes. Se o problema persistir, entre em contato com o suporte da sua Câmara Municipal.</p> |
|||
</div> |
|||
</body> |
|||
</html> |
|||
@ -0,0 +1,35 @@ |
|||
User-agent: GPTBot |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: ClaudeBot |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: meta-externalagent |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: OAI-SearchBot |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: bingbot |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: SERankingBacklinksBot |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: anthropic-python |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: AwarioSmartBot |
|||
Disallow: / |
|||
Crawl-delay: 10 |
|||
|
|||
User-agent: * |
|||
Disallow: /relatorios/ |
|||
Crawl-delay: 10 |
|||
@ -1,120 +0,0 @@ |
|||
{% load i18n %} |
|||
{% load common_tags %} |
|||
{% load render_bundle from webpack_loader %} |
|||
{% load webpack_static from webpack_loader %} |
|||
|
|||
<!DOCTYPE HTML> |
|||
<!--[if IE 8]> <html class="no-js lt-ie9" lang="pt-br"> <![endif]--> |
|||
<!--[if gt IE 8]><!--> |
|||
<html lang="pt-br"> |
|||
<!--<![endif]--> |
|||
|
|||
<head> |
|||
<meta charset="UTF-8"> |
|||
<!-- TODO: does it need this head_title here? --> |
|||
<title>{% block head_title %}{% trans 'SAPL - Sistema de Apoio ao Processo Legislativo' %}{% endblock %}</title> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|||
|
|||
{% render_chunk_vendors 'css' %} |
|||
{% render_bundle 'global' 'css' %} |
|||
{% render_bundle 'painel' 'css' %} |
|||
|
|||
|
|||
<STYLE type="text/css"> |
|||
@media screen { |
|||
body {font-size: medium; color: white; line-height: 1em; background: black;} |
|||
} |
|||
</STYLE> |
|||
|
|||
</head> |
|||
<body> |
|||
<h1>{{ context.title }}</h1> |
|||
<input id="json_url" type="hidden" value="{% url 'sapl.painel:dados_painel' %}"> |
|||
<h2>Ajax refresh counter: <span id="counter"></span></h2> |
|||
<h3> |
|||
<span id="sessao_plenaria"></span><br/><br/> |
|||
<span id="sessao_plenaria_data"></span><br/><br/> |
|||
<span id="sessao_plenaria_hora_inicio"></span></br><br/> |
|||
<h2><span id="relogio"></span></h2></br><br/><br/> |
|||
<span id="materia_legislativa_texto"></span><br/> |
|||
<span id="observacao_materia"></span> |
|||
</h3> |
|||
</body> |
|||
|
|||
{% render_chunk_vendors 'js' %} |
|||
{% render_bundle 'global' 'js' %} |
|||
{% render_bundle 'painel' 'js' %} |
|||
|
|||
<script type="text/javascript"> |
|||
$(document).ready(function() { |
|||
|
|||
//TODO: replace by a fancy jQuery clock |
|||
function checkTime(i) { |
|||
if (i<10) {i = "0" + i}; // add zero in front of numbers < 10 |
|||
return i; |
|||
} |
|||
function startTime() { |
|||
var today=new Date(); |
|||
var h=today.getHours(); |
|||
var m=today.getMinutes(); |
|||
var s=today.getSeconds(); |
|||
m = checkTime(m); |
|||
s = checkTime(s); |
|||
$("#relogio").text(h+":"+m+":"+s) |
|||
var t = setTimeout(function(){ |
|||
startTime() |
|||
},500); |
|||
} |
|||
|
|||
startTime(); |
|||
|
|||
var counter = 1; |
|||
(function poll() { |
|||
$.ajax({ |
|||
url: $("#json_url").val(), |
|||
type: "GET", |
|||
success: function(data) { |
|||
|
|||
//TODO: json spitted out is very complex, have to simplify/flat it |
|||
//TODO: probably building it by hand on REST side |
|||
|
|||
console.debug(data) |
|||
|
|||
var presentes = $("#parlamentares"); |
|||
presentes.children().remove(); |
|||
|
|||
presentes_ordem_dia = data.presentes_ordem_dia |
|||
$.each(presentes_ordem_dia, function(index, parlamentar) { |
|||
$('<li />', {text: parlamentar.nome + '/' + parlamentar.partido + ' ' + parlamentar.voto }).appendTo(presentes); |
|||
}); |
|||
|
|||
var votacao = $("#votacao") |
|||
votacao.children().remove() |
|||
votacao.append("<li>Sim: " + data["numero_votos_sim"] + "</li>") |
|||
votacao.append("<li>Não: " + data["numero_votos_nao"] + "</li>") |
|||
votacao.append("<li>Abstenções: " + data["numero_abstencoes"] + "</li>") |
|||
votacao.append("<li>Presentes: " + data["presentes"] + "</li>") |
|||
votacao.append("<li>Total votos: " + data["total_votos"] + "</li>") |
|||
|
|||
$("#sessao_plenaria").text(data["sessao_plenaria"]) |
|||
$("#sessao_plenaria_data").text("Data Início: " + data["sessao_plenaria_data"]) |
|||
$("#sessao_plenaria_hora_inicio").text("Hora Início: " + data["sessao_plenaria_hora_inicio"]) |
|||
|
|||
$("#materia_legislativa_texto").text(data["materia_legislativa_texto"]) |
|||
$("#observacao_materia").text(data["observacao_materia"]) |
|||
$("#resultado_votacao").text(data["tipo_resultado"]) |
|||
|
|||
$("#counter").text(counter); |
|||
counter++; |
|||
}, |
|||
error: function(err) { |
|||
console.error(err); |
|||
}, |
|||
dataType: "json", |
|||
//complete: setTimeout(function() {poll()}, 5000), |
|||
timeout: 20000 // TODO: decrease |
|||
}) |
|||
})(); |
|||
}); |
|||
</script> |
|||
</html> |
|||
@ -1,128 +0,0 @@ |
|||
{% load i18n %} |
|||
{% load common_tags %} |
|||
|
|||
{% load render_bundle from webpack_loader %} |
|||
{% load webpack_static from webpack_loader %} |
|||
|
|||
<!DOCTYPE HTML> |
|||
<!--[if IE 8]> <html class="no-js lt-ie9" lang="pt-br"> <![endif]--> |
|||
<!--[if gt IE 8]><!--> |
|||
<html lang="pt-br"> |
|||
<!--<![endif]--> |
|||
|
|||
<head> |
|||
<meta charset="UTF-8"> |
|||
<!-- TODO: does it need this head_title here? --> |
|||
<title>{% block head_title %}{% trans 'SAPL - Sistema de Apoio ao Processo Legislativo' %}{% endblock %}</title> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|||
|
|||
{% render_chunk_vendors 'css' %} |
|||
{% render_bundle 'global' 'css' %} |
|||
{% render_bundle 'painel' 'css' %} |
|||
|
|||
|
|||
|
|||
<STYLE type="text/css"> |
|||
@media screen { |
|||
body {font-size: medium; color: white; line-height: 1em; background: black;} |
|||
} |
|||
</STYLE> |
|||
|
|||
</head> |
|||
<body> |
|||
<h1>{{ context.title }}</h1> |
|||
<input id="json_url" type="hidden" value="{% url 'sapl.painel:dados_painel' %}"> |
|||
<h3> |
|||
<span id="sessao_plenaria"></span><br/><br/> |
|||
<span id="sessao_plenaria_data"></span><br/><br/> |
|||
<span id="sessao_plenaria_hora_inicio"></span></br><br/> |
|||
<h2><span id="relogio"></span></h2> |
|||
<table> |
|||
<tr> |
|||
<td> |
|||
<ul id="parlamentares"> |
|||
</ul> |
|||
</td> |
|||
</tr> |
|||
</table> |
|||
</h3> |
|||
</body> |
|||
|
|||
{% render_chunk_vendors 'js' %} |
|||
{% render_bundle 'global' 'js' %} |
|||
{% render_bundle 'painel' 'js' %} |
|||
|
|||
<script type="text/javascript"> |
|||
$(document).ready(function() { |
|||
|
|||
//TODO: replace by a fancy jQuery clock |
|||
function checkTime(i) { |
|||
if (i<10) {i = "0" + i}; // add zero in front of numbers < 10 |
|||
return i; |
|||
} |
|||
function startTime() { |
|||
var today=new Date(); |
|||
var h=today.getHours(); |
|||
var m=today.getMinutes(); |
|||
var s=today.getSeconds(); |
|||
m = checkTime(m); |
|||
s = checkTime(s); |
|||
$("#relogio").text(h+":"+m+":"+s) |
|||
var t = setTimeout(function(){ |
|||
startTime() |
|||
},500); |
|||
} |
|||
|
|||
startTime(); |
|||
|
|||
var counter = 1; |
|||
(function poll() { |
|||
$.ajax({ |
|||
url: $("#json_url").val(), |
|||
type: "GET", |
|||
success: function(data) { |
|||
|
|||
//TODO: json spitted out is very complex, have to simplify/flat it |
|||
//TODO: probably building it by hand on REST side |
|||
|
|||
console.debug(data) |
|||
|
|||
var presentes = $("#parlamentares"); |
|||
presentes.children().remove(); |
|||
|
|||
presentes_ordem_dia = data.presentes_ordem_dia |
|||
$.each(presentes_ordem_dia, function(index, parlamentar) { |
|||
$('<li />', {text: parlamentar.nome + '/' + parlamentar.partido }).appendTo(presentes); |
|||
/*$('<li />', {text: parlamentar.nome + '/' + parlamentar.partido + ' ' + parlamentar.voto }).appendTo(presentes);*/ |
|||
}); |
|||
|
|||
var votacao = $("#votacao") |
|||
votacao.children().remove() |
|||
votacao.append("<li>Sim: " + data["numero_votos_sim"] + "</li>") |
|||
votacao.append("<li>Não: " + data["numero_votos_nao"] + "</li>") |
|||
votacao.append("<li>Abstenções: " + data["numero_abstencoes"] + "</li>") |
|||
votacao.append("<li>Presentes: " + data["presentes"] + "</li>") |
|||
votacao.append("<li>Total votos: " + data["total_votos"] + "</li>") |
|||
|
|||
$("#sessao_plenaria").text(data["sessao_plenaria"]) |
|||
$("#sessao_plenaria_data").text("Data Início: " + data["sessao_plenaria_data"]) |
|||
$("#sessao_plenaria_hora_inicio").text("Hora Início: " + data["sessao_plenaria_hora_inicio"]) |
|||
|
|||
$("#materia_legislativa_texto").text(data["materia_legislativa_texto"]) |
|||
$("#observacao_materia").text(data["observacao_materia"]) |
|||
$("#resultado_votacao").text(data["tipo_resultado"]) |
|||
|
|||
$("#counter").text(counter); |
|||
counter++; |
|||
}, |
|||
error: function(err) { |
|||
console.error(err); |
|||
}, |
|||
dataType: "json", |
|||
//complete: setTimeout(function() {poll()}, 5000), |
|||
timeout: 20000 // TODO: decrease |
|||
}) |
|||
})(); |
|||
}); |
|||
</script> |
|||
</html> |
|||
@ -1,123 +0,0 @@ |
|||
{% load i18n %} |
|||
{% load render_bundle from webpack_loader %} |
|||
{% load webpack_static from webpack_loader %} |
|||
|
|||
<!DOCTYPE HTML> |
|||
<!--[if IE 8]> <html class="no-js lt-ie9" lang="pt-br"> <![endif]--> |
|||
<!--[if gt IE 8]><!--> |
|||
<html lang="pt-br"> |
|||
<!--<![endif]--> |
|||
|
|||
<head> |
|||
<meta charset="UTF-8"> |
|||
<!-- TODO: does it need this head_title here? --> |
|||
<title>{% block head_title %}{% trans 'SAPL - Sistema de Apoio ao Processo Legislativo' %}{% endblock %}</title> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|||
|
|||
{% render_chunk_vendors 'css' %} |
|||
{% render_bundle 'global' 'css' %} |
|||
{% render_bundle 'painel' 'css' %} |
|||
|
|||
<STYLE type="text/css"> |
|||
@media screen { |
|||
body {font-size: medium; color: white; line-height: 1em; background: black;} |
|||
} |
|||
</STYLE> |
|||
</head> |
|||
<body> |
|||
<h1>{{ context.title }}</h1> |
|||
<input id="json_url" type="hidden" value="{% url 'sapl.painel:dados_painel' %}"> |
|||
<h3> |
|||
<span id="sessao_plenaria"></span><br/><br/> |
|||
<span id="sessao_plenaria_data"></span><br/><br/> |
|||
<span id="sessao_plenaria_hora_inicio"></span></br><br/> |
|||
<h2><span id="relogio"></span></h2> |
|||
<table> |
|||
<tr> |
|||
<td> |
|||
<ul id="votacao"> |
|||
</ul> |
|||
</td> |
|||
</tr> |
|||
</table> |
|||
<span id="resultado_votacao"></span><br/> |
|||
</h3> |
|||
</body> |
|||
|
|||
{% render_chunk_vendors 'js' %} |
|||
{% render_bundle 'global' 'js' %} |
|||
{% render_bundle 'painel' 'js' %} |
|||
|
|||
<script type="text/javascript"> |
|||
$(document).ready(function() { |
|||
|
|||
//TODO: replace by a fancy jQuery clock |
|||
function checkTime(i) { |
|||
if (i<10) {i = "0" + i}; // add zero in front of numbers < 10 |
|||
return i; |
|||
} |
|||
function startTime() { |
|||
var today=new Date(); |
|||
var h=today.getHours(); |
|||
var m=today.getMinutes(); |
|||
var s=today.getSeconds(); |
|||
m = checkTime(m); |
|||
s = checkTime(s); |
|||
$("#relogio").text(h+":"+m+":"+s) |
|||
var t = setTimeout(function(){ |
|||
startTime() |
|||
},500); |
|||
} |
|||
|
|||
startTime(); |
|||
|
|||
var counter = 1; |
|||
(function poll() { |
|||
$.ajax({ |
|||
url: $("#json_url").val(), |
|||
type: "GET", |
|||
success: function(data) { |
|||
|
|||
//TODO: json spitted out is very complex, have to simplify/flat it |
|||
//TODO: probably building it by hand on REST side |
|||
|
|||
console.debug(data) |
|||
|
|||
var presentes = $("#parlamentares"); |
|||
presentes.children().remove(); |
|||
|
|||
presentes_ordem_dia = data.presentes_ordem_dia |
|||
$.each(presentes_ordem_dia, function(index, parlamentar) { |
|||
$('<li />', {text: parlamentar.nome + '/' + parlamentar.partido + ' ' + parlamentar.voto }).appendTo(presentes); |
|||
}); |
|||
|
|||
var votacao = $("#votacao") |
|||
votacao.children().remove() |
|||
votacao.append("<li>Sim: " + data["numero_votos_sim"] + "</li>") |
|||
votacao.append("<li>Não: " + data["numero_votos_nao"] + "</li>") |
|||
votacao.append("<li>Abstenções: " + data["numero_abstencoes"] + "</li>") |
|||
votacao.append("<li>Presentes: " + data["presentes"] + "</li>") |
|||
votacao.append("<li>Total votos: " + data["total_votos"] + "</li>") |
|||
|
|||
$("#sessao_plenaria").text(data["sessao_plenaria"]) |
|||
$("#sessao_plenaria_data").text("Data Início: " + data["sessao_plenaria_data"]) |
|||
$("#sessao_plenaria_hora_inicio").text("Hora Início: " + data["sessao_plenaria_hora_inicio"]) |
|||
|
|||
$("#materia_legislativa_texto").text(data["materia_legislativa_texto"]) |
|||
$("#observacao_materia").text(data["observacao_materia"]) |
|||
$("#resultado_votacao").text(data["tipo_resultado"]) |
|||
|
|||
$("#counter").text(counter); |
|||
counter++; |
|||
}, |
|||
error: function(err) { |
|||
console.error(err); |
|||
}, |
|||
dataType: "json", |
|||
//complete: setTimeout(function() {poll()}, 5000), |
|||
timeout: 20000 // TODO: decrease |
|||
}) |
|||
})(); |
|||
}); |
|||
</script> |
|||
</html> |
|||
@ -0,0 +1,83 @@ |
|||
#!/usr/bin/env python3 |
|||
""" |
|||
Script to test rate limiting of an endpoint. |
|||
""" |
|||
|
|||
import argparse |
|||
import time |
|||
import requests |
|||
from collections import defaultdict |
|||
from urllib.parse import urlparse |
|||
|
|||
|
|||
def test_rate_limiter(url, num_requests=50, delay=0.1, timeout=10): |
|||
"""Send multiple requests and analyze rate limiting behavior.""" |
|||
parsed = urlparse(url) |
|||
if not parsed.scheme or not parsed.netloc: |
|||
raise ValueError( |
|||
"URL must include a protocol and host, e.g. http://localhost or https://example.com" |
|||
) |
|||
if parsed.scheme not in {"http", "https"}: |
|||
raise ValueError("Unsupported URL scheme: %s. Use http or https." % parsed.scheme) |
|||
|
|||
status_counts = defaultdict(int) |
|||
response_times = [] |
|||
first_rate_limited_at = None |
|||
attempted_requests = 0 |
|||
|
|||
print(f"Testing rate limiter on: {url}") |
|||
print(f"Number of requests: {num_requests}") |
|||
print(f"Delay between requests: {delay}s") |
|||
print("-" * 50) |
|||
|
|||
for i in range(num_requests): |
|||
attempted_requests += 1 |
|||
try: |
|||
start_time = time.time() |
|||
response = requests.get(url, timeout=timeout) |
|||
elapsed = time.time() - start_time |
|||
|
|||
status_counts[response.status_code] += 1 |
|||
response_times.append(elapsed) |
|||
|
|||
print(f"Request {i+1:3d}: Status {response.status_code} | Time: {elapsed:.3f}s") |
|||
|
|||
if response.status_code == 429: |
|||
if first_rate_limited_at is None: |
|||
first_rate_limited_at = i + 1 |
|||
print(f" -> Rate limited on request {i+1}") |
|||
break |
|||
|
|||
except requests.exceptions.RequestException as e: |
|||
print(f"Request {i+1:3d}: Error - {e}") |
|||
status_counts['ERROR'] += 1 |
|||
|
|||
if i < num_requests - 1: |
|||
time.sleep(delay) |
|||
|
|||
print("-" * 50) |
|||
print("\nSummary:") |
|||
print(f" Total requests attempted: {attempted_requests}") |
|||
print(f" Successful (200): {status_counts.get(200, 0)}") |
|||
print(f" Rate limited (429): {status_counts.get(429, 0)}") |
|||
if first_rate_limited_at is not None: |
|||
print(f" First 429 occurred at request: {first_rate_limited_at}") |
|||
print(f" Other errors: {sum(v for k, v in status_counts.items() if k not in [200, 429, 'ERROR'])}") |
|||
|
|||
if response_times: |
|||
avg_time = sum(response_times) / len(response_times) |
|||
print(f"\nAverage response time: {avg_time:.3f}s") |
|||
|
|||
|
|||
if __name__ == "__main__": |
|||
parser = argparse.ArgumentParser(description="Test rate limiter of a URL") |
|||
parser.add_argument( |
|||
"url", |
|||
help="URL to test, including protocol (http:// or https://)", |
|||
) |
|||
parser.add_argument("-n", "--num-requests", type=int, default=50, help="Number of requests") |
|||
parser.add_argument("-d", "--delay", type=float, default=0.1, help="Delay between requests (seconds)") |
|||
parser.add_argument("-t", "--timeout", type=int, default=10, help="Request timeout (seconds)") |
|||
|
|||
args = parser.parse_args() |
|||
test_rate_limiter(args.url, args.num_requests, args.delay, args.timeout) |
|||
@ -1,14 +0,0 @@ |
|||
#!/bin/bash |
|||
|
|||
#URL=http://localhost:8000/materia/4379 |
|||
#URL=http://localhost:8000/norma/pesquisar |
|||
#URL=http://localhost/norma/pesquisar |
|||
#URL=https://sapl31demo.interlegis.leg.br/docadm/45 |
|||
#URL=https://sapl.joaopessoa.pb.leg.br/materia/186300 |
|||
#URL=http://localhost:8000/materia/4379/materiaassunto |
|||
#URL=http://localhost:8000/sessao/4984 |
|||
URL="http://localhost:8000/docadm/pesq-doc-adm?tipo=&o=&numero=&complemento=&ano=&protocolo__numero=&numero_externo=&data_0=&data_1=&interessado=&assunto=&tramitacao=&tramitacaoadministrativo__status=&tramitacaoadministrativo__unidade_tramitacao_destino=&pesquisar=Pesquisar" |
|||
|
|||
for i in $(seq 1 12); do |
|||
curl -sS -o /dev/null -w "req=$i http=%{http_code} time=%{time_total}\n" "$URL" |
|||
done |
|||
@ -0,0 +1,173 @@ |
|||
# SAPL — Work Queues & Real-Time: Async PDF + WebSocket Voting |
|||
|
|||
> **Status**: Planned follow-up mini-project. |
|||
> **Prerequisite**: Redis A (cache + rate-limiter pod, `rate-limiter-2026` branch) must be |
|||
> deployed to production, stable, and OOM pressure confirmed reduced before starting this work. |
|||
> **Scope**: Django 2.2 / Gunicorn / Celery / Django Channels — same fleet of 1,200+ pods. |
|||
|
|||
--- |
|||
|
|||
## Table of Contents |
|||
|
|||
1. [Context & Motivation](#1-context--motivation) |
|||
2. [Redis Topology for Work Queues](#2-redis-topology-for-work-queues) |
|||
3. [Phase 1 — Async PDF via Celery](#3-phase-1--async-pdf-via-celery) |
|||
4. [Phase 2 — Django Channels (WebSocket Voting Panel)](#4-phase-2--django-channels-websocket-voting-panel) |
|||
5. [Open Questions](#5-open-questions) |
|||
|
|||
--- |
|||
|
|||
## 1. Context & Motivation |
|||
|
|||
After `rate-limiter-2026` ships: |
|||
|
|||
| Remaining pain point | Current behaviour | Target | |
|||
|---|---|---| |
|||
| PDF generation | Holds a Gunicorn worker thread for the full build duration (10–60 s). Workers are at 400 MB cap — a PDF request burns one slot for up to a minute | Enqueue via Celery; respond 202 immediately; worker is freed | |
|||
| WebSocket voting panel | Not implemented; councillors use a polling page | Persistent connection via Django Channels backed by Redis | |
|||
|
|||
--- |
|||
|
|||
## 2. Redis Topology for Work Queues |
|||
|
|||
> **Critical constraint**: Celery broker **must** be a **separate** Redis instance (Redis B) |
|||
> with `noeviction` policy. |
|||
> Redis A (cache + rate-limiter) uses `allkeys-lru` — tasks enqueued there would be silently |
|||
> evicted under memory pressure, causing jobs to vanish without error. |
|||
|
|||
| Instance | Role | Eviction policy | Persistence | |
|||
|---|---|---|---| |
|||
| **Redis A** (existing) | Page cache (DB0), rate limiter (DB1), Django Channels (DB2) | `allkeys-lru` | none | |
|||
| **Redis B** (new) | Celery broker + result backend | `noeviction` | AOF + RDB snapshot | |
|||
|
|||
```yaml |
|||
# docker/k8s/redis-celery-configmap.yaml |
|||
data: |
|||
redis.conf: | |
|||
maxmemory-policy noeviction # never evict tasks |
|||
appendonly yes # AOF persistence ON |
|||
save "900 1" # RDB snapshot every 15 min if ≥1 change |
|||
databases 2 # DB0 = broker queue, DB1 = result backend |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## 3. Phase 1 — Async PDF via Celery |
|||
|
|||
### 3.1 Current (synchronous) flow |
|||
|
|||
Holds worker memory for the entire PDF build: |
|||
|
|||
```mermaid |
|||
sequenceDiagram |
|||
participant B as Browser |
|||
participant G as Gunicorn worker |
|||
participant ORM as PostgreSQL |
|||
participant RL as ReportLab |
|||
|
|||
B->>G: GET /pdf/materia/12345 |
|||
G->>ORM: N+1 queries (get_etiqueta_protocolos) |
|||
ORM-->>G: data |
|||
G->>RL: build entire PDF in RAM |
|||
RL-->>G: PDF bytes (held in worker memory) |
|||
G-->>B: stream response |
|||
note over G: worker blocked for full duration |
|||
``` |
|||
|
|||
### 3.2 Target (async) flow |
|||
|
|||
Worker freed immediately after enqueueing: |
|||
|
|||
```mermaid |
|||
sequenceDiagram |
|||
participant B as Browser |
|||
participant G as Gunicorn worker |
|||
participant Q as Redis B (Celery queue) |
|||
participant W as Celery worker |
|||
participant D as Disk / nginx |
|||
|
|||
B->>G: POST /pdf/materia/12345 |
|||
G->>Q: enqueue task |
|||
G-->>B: 202 Accepted + task_id |
|||
W->>W: build PDF (out of band) |
|||
W->>D: write PDF to /media/pdf/task_id.pdf |
|||
B->>G: GET /pdf/status/task_id |
|||
G-->>B: 302 → nginx /media/pdf/task_id.pdf |
|||
``` |
|||
|
|||
### 3.3 Celery settings |
|||
|
|||
```python |
|||
# sapl/settings.py additions |
|||
CELERY_BROKER_URL = config('CELERY_BROKER_URL', default='') |
|||
CELERY_RESULT_BACKEND = config('CELERY_RESULT_BACKEND', default='') |
|||
|
|||
# Soft limit: warn at 350 MB; hard limit: kill+restart at 450 MB. |
|||
# Keeps Celery workers inside the same memory envelope as Gunicorn workers. |
|||
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 400 * 1024 # KB |
|||
CELERY_TASK_SOFT_TIME_LIMIT = 120 # seconds — warn |
|||
CELERY_TASK_TIME_LIMIT = 180 # seconds — SIGKILL |
|||
``` |
|||
|
|||
### 3.4 k8s manifests |
|||
|
|||
New files to be created under `docker/k8s/`: |
|||
|
|||
- `redis-celery-configmap.yaml` — Redis B config (noeviction, AOF) |
|||
- `redis-celery-deployment.yaml` — single-replica Redis B pod |
|||
- `redis-celery-service.yaml` — ClusterIP service |
|||
- `celery-deployment.yaml` — Celery worker deployment (same image as SAPL) |
|||
|
|||
### 3.5 Environment variables (per-namespace Secret) |
|||
|
|||
| Variable | Example value | Notes | |
|||
|---|---|---| |
|||
| `CELERY_BROKER_URL` | `redis://sapl-redis-celery.redis.svc:6379/0` | Redis B, DB0 | |
|||
| `CELERY_RESULT_BACKEND` | `redis://sapl-redis-celery.redis.svc:6379/1` | Redis B, DB1 | |
|||
|
|||
--- |
|||
|
|||
## 4. Phase 2 — Django Channels (WebSocket Voting Panel) |
|||
|
|||
Uses **Redis A DB2** (reserved in the existing key-layout table — no new infra needed beyond |
|||
what ships in `rate-limiter-2026`). |
|||
|
|||
### 4.1 Channel layer settings |
|||
|
|||
```python |
|||
# sapl/settings.py additions |
|||
CHANNEL_LAYERS = { |
|||
"default": { |
|||
"BACKEND": "channels_redis.core.RedisChannelLayer", |
|||
"CONFIG": { |
|||
"hosts": [("sapl-redis.redis.svc.cluster.local", 6379)], |
|||
"db": 2, # DB2 reserved for channels (see rate-limiter-v2.md §0.2) |
|||
"capacity": 1500, |
|||
"expiry": 10, |
|||
}, |
|||
} |
|||
} |
|||
``` |
|||
|
|||
### 4.2 Prerequisites before starting |
|||
|
|||
- [ ] Redis A stable in production (rate limiter + cache confirmed working) |
|||
- [ ] OOM kill rate reduced to near-zero |
|||
- [ ] Bot siege resolved (Phase 0–2 metrics reviewed) |
|||
- [ ] Decision on ASGI server (Daphne vs Uvicorn + channels) — Gunicorn alone cannot serve WebSockets |
|||
|
|||
--- |
|||
|
|||
## 5. Open Questions |
|||
|
|||
| # | Question | Blocks | |
|||
|---|---|---| |
|||
| 1 | Which PDF endpoints are highest priority for async migration? (`/relatorios/`, `/materia/pdf/`, other)? | Phase 1 scope | |
|||
| 2 | Should the Celery worker run in the same pod as Gunicorn (sidecar) or a dedicated deployment? | Phase 1 k8s design | |
|||
| 3 | Result backend TTL — how long should generated PDFs be retained before cleanup? | Phase 1 storage design | |
|||
| 4 | ASGI server selection for Channels (Daphne vs uvicorn + channels) | Phase 2 | |
|||
| 5 | WebSocket voting panel: is per-session or per-pod state acceptable? | Phase 2 architecture | |
|||
|
|||
--- |
|||
|
|||
*Planned work — begins after `rate-limiter-2026` is stable in production.* |
|||
Loading…
Reference in new issue