Sistema de Apoio ao Processo Legislativo

41 KiB

Raw Blame History

SAPL — Rate Limiter & Redis Operations

Scope: Django / Gunicorn / nginx / Kubernetes fleet of 1,200+ pods. Each pod has a dedicated PostgreSQL instance. A K8s Ingress sits in front of all tenants. This document is canonical — all earlier session notes are consolidated here.

Context & Problem Statement

Fleet

Item	Detail
System	SAPL — Django 2.2, legislative management for Brazilian municipal chambers
Fleet	~1,200 Kubernetes pods, each with a dedicated PostgreSQL pod
Pod limits	1 core CPU (limit) / 35m (request) · 1600Mi RAM (limit) / 800Mi (request)
Users	Legislative house staff, often behind NAT (many users, one public IP)
Workloads	PDF generation (synchronous, ReportLab), file uploads up to 150 MB, WebSocket voting panel

OOM Kill Pattern

Workers grow from ~35 MB at birth to 800–900 MB within 2–3 minutes, then are killed and replaced in a continuous cycle.

Root causes:

Bot scraping triggers synchronous PDF generation — entire document built in RAM (ReportLab)
worker_max_memory_per_child only checks between requests; workers blocked on long requests are never recycled
TIMEOUT=300 lets bots hold threads for up to 5 minutes while memory accumulates
3 workers × 300 MB each = ~900 MB — breaching the 800Mi request threshold

Bot Traffic Profile (Barueri pod, 16 days, 662 k requests)

Actor	Requests	% of total
Googlebot	~154,000	23.2%
Chrome/98.0.4758 (spoofed scraper)	90,774	13.7%
kube-probe (healthcheck)	69,065	10.4%
meta-externalagent	28,325	4.3%
GPTBot	11,489	1.7%
bingbot	7,639	1.1%
OAI-SearchBot + Applebot	6,681	1.0%
Total identified bots	~377,000	~56.9%

Botnet fingerprint:

Rotates User-Agents (Chrome/121, Chrome/122, Firefox/123, Safari/17…) across requests
Crawls all sub-endpoints of the same matéria within 1 second from different IPs
Distributes crawling across tenants — each pod stays under the per-pod rate limit, never triggering it
Primary targets: /relatorios/{id}/etiqueta-materia-legislativa (~40 KB PDF) and all /materia/{id}/* sub-endpoints

Static File Traffic (from CSV analysis)

Category	Requests	Transfers
Logos / images	62,776	~24 GB
PDFs	8,869	5.1 GB
Parliamentarian photos	11,856	~0.5 GB
Total	83,501	~30 GB

Top offender: Brasão - Foz do Iguaçu.png — 14,512 requests, 5.6 GB from a single 392 KB file.

Hard Constraints

Constraint	Impact
Per-pod PostgreSQL	Rate-limit counters not shared across pods
NAT environments	IP-based rate limiting causes false positives
`TIMEOUT=300` / uploads to 150 MB	Must not be broken — intentional for slow workflows

Architecture Overview

Component Diagram

graph TD
    Client([Bot / Human Client])
    nginx[nginx]
    gunicorn[Gunicorn\n2 workers / 4 threads]
    mw[Django Middleware\nRateLimitMiddleware]
    view[View Layer\nCBV + decorators]
    db0[(Redis DB0\npage cache)]
    db1[(Redis DB1\nrate limiter)]
    pg[(PostgreSQL\nper-pod)]
    fs[Filesystem\nPDFs / media]

    Client -->|HTTP| nginx
    nginx -->|proxy_pass| gunicorn
    gunicorn --> mw
    mw -->|pass| view
    mw -->|429| nginx
    view --> pg
    view --> fs
    view -->|read/write cached pages| db0
    mw -->|counters + blocked markers| db1

DB2 is reserved for Django Channels (WebSocket — future).

Redis Memory Budget

Key type	Key schema	TTL	DB	Est. size
Page / view cache	`cache:{ns}:*`	60–600 s	0	~0.5 GB
Static cache (images/logos)	`static:{ns}:{sha256}`	3–24 h	0	~2.4 GB
IP request counter	`rl:ip:{ip}:reqs`	60 s	1	~0.6 MB
IP blocked marker	`rl:ip:{ip}:blocked`	300 s	1	~0.06 MB
User request counter	`rl:{ns}:user:{uid}:reqs`	60 s	1	negligible
User blocked marker	`rl:{ns}:user:{uid}:blocked`	300 s	1	negligible
Path counter	`rl:{ns}:path:{sha256}:reqs`	60 s	1	~0.3 MB
UA deny list	`rl:bot:ua:blocked`	permanent SET	1	~0.03 MB
NS/IP/window counter	`rl:{ns}:ip:{ip}:w:{bucket}`	120 s	1	~0.6 MB
Redis overhead (× 1.5)				~1.6 GB
Total ceiling				~5 GB

Decision Log

Decision	Chosen	Rationale
Redis topology	Single pod (no Sentinel, no Cluster)	65 MB of active data fits comfortably; cluster complexity not justified
PDF caching in Redis	No — ETags + sendfile are sufficient	Once rate limiting + ETags are active, repeat requests become 304s with zero bytes transferred
HTTP conditional requests	`ConditionalGetMiddleware` + `@condition` decorator	`ConditionalGetMiddleware` handles ETag/304 for all views; `@condition(etag_func, last_modified_func)` on materia/norma detail views skips view execution entirely on cache hit
Upload endpoint special-casing (nginx)	Removed — fall through to `location /`	No justification for separate `limit_req` zone; `location /` with `sapl_general` covers it
Static asset cache policy	90 min (`expires 90m`, `max-age=5400`)	Conservative — safe with `collectstatic` content-hashed filenames; `immutable` not used (would require verified forever-hashed URLs)
Rate-limit enforcement	Django middleware with shared Redis	No nginx image changes required; solves cross-pod consistency immediately
`worker_max_memory_per_child`	400 MB	Pod limit 1600Mi, 2 workers × 400 MB = 800 MB — leaves 800 Mi headroom
`sendfile off` → `on`	Bug — flip to `on`	No valid production reason found; disabling userspace copy is always correct
`/media/` serving	X-Accel-Redirect	Routes all `/media/` through Gunicorn so Django middleware runs; nginx serves bytes via internal location
Cache backend switch	At pod startup via `start.sh` + waffle switch	Pod restart is acceptable; avoids per-request runtime overhead

Directory layout

docker/k8s/
└── redis/
    ├── redis-configmap.yaml    # redis.conf — no persistence, allkeys-lru, 5 GB ceiling
    ├── redis-deployment.yaml   # Deployment (1 replica, redis:7-alpine)
    └── redis-service.yaml      # ClusterIP service on port 6379

Prerequisites

kubectl configured to talk to the target cluster.
A sapl-redis namespace (created below if it doesn't exist).

Deploy

# 1. Create the namespace (idempotent)
rancher kubectl create namespace sapl-redis --dry-run=client -o yaml | rancher kubectl apply -f -

# 2. Apply all three manifests
rancher kubectl apply -f docker/k8s/redis/redis-configmap.yaml
rancher kubectl apply -f docker/k8s/redis/redis-deployment.yaml
rancher kubectl apply -f docker/k8s/redis/redis-service.yaml

# 3. Verify the pod is Running
rancher kubectl -n sapl-redis get pods -l app=sapl-redis

Expected output:

NAME                          READY   STATUS    RESTARTS   AGE
sapl-redis-6d9f8b7c4d-xk2lm   1/1     Running   0          30s

Verify the rate limiter

scripts/test_ratelimiter.py fires repeated GET requests at a SAPL URL and reports when the first 429 is returned.

Usage

python scripts/test_ratelimiter.py <URL> [-n NUM] [-d DELAY] [-t TIMEOUT]

Flag	Default	Meaning
`url`	(required)	Full URL including scheme, e.g. `http://localhost`
`-n`, `--num-requests`	`50`	Maximum requests to send
`-d`, `--delay`	`0.1`	Seconds between requests
`-t`, `--timeout`	`10`	Per-request timeout in seconds

The script stops and prints a summary as soon as a 429 is received.

Examples

# Hit the anonymous threshold (35 req/min) — fire 40 requests with minimal delay
python scripts/test_ratelimiter.py http://localhost -n 40 -d 0.05

# Slower fire — check that legitimate traffic is not rate-limited
python scripts/test_ratelimiter.py http://localhost -n 20 -d 2

# Test against a staging pod via port-forward
rancher kubectl port-forward -n <NAMESPACE> deploy/sapl 8080:80 &
python scripts/test_ratelimiter.py http://localhost:8080 -n 40 -d 0.05

Reading the output

Request   1: Status 200 | Time: 0.045s
...
Request  36: Status 429 | Time: 0.038s
  -> Rate limited on request 36

Summary:
  Total requests attempted: 36
  Successful (200):          35
  Rate limited (429):        1
  First 429 occurred at request: 36

A first-429 near the configured anonymous threshold (35 req/min) confirms the middleware is wired correctly. A first-429 much earlier points to nginx limit_req firing before Django sees the request.

Inject REDIS_URL into SAPL instances

REDIS_URL points at the shared instance:

redis://redis.sapl-redis.svc.cluster.local:6379
         ^^^^^  ^^^^^^^^^^
         svc    namespace

start.sh picks it up on every pod startup and sets the REDIS_CACHE waffle switch automatically — no further intervention needed.

Fleet-wide rollout

Uses the app.kubernetes.io/name=sapl pod label to discover every SAPL namespace automatically — onboarding a new municipality requires no script changes.

for ns in $(rancher kubectl get pods -A -l app.kubernetes.io/name=sapl \
  -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do
  rancher kubectl set env deployment/sapl \
    REDIS_URL=redis://redis.sapl-redis.svc.cluster.local:6379 \
    -n $ns
done

Roll back

for ns in $(rancher kubectl get pods -A -l app.kubernetes.io/name=sapl \
  -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do
  rancher kubectl set env deployment/sapl REDIS_URL- -n $ns
done

kubectl set env deployment/sapl REDIS_URL- (trailing -) removes the variable. start.sh then falls back to file-based cache automatically.

Monitor

Pod and events

# Pod status
rancher kubectl -n sapl-redis get pods -l app=sapl-redis -o wide

# Deployment events (useful right after apply)
rancher kubectl -n sapl-redis describe deployment sapl-redis

# Pod events (OOMKill, restarts, etc.)
rancher kubectl -n sapl-redis describe pod -l app=sapl-redis

Logs

# Tail live logs
rancher kubectl -n sapl-redis logs -f deploy/sapl-redis

# Last 100 lines
rancher kubectl -n sapl-redis logs deploy/sapl-redis --tail=100

Redis INFO

# Memory usage
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \
  redis-cli info memory \
  | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'

# Connection pressure
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \
  redis-cli info stats \
  | grep -E 'rejected_connections|instantaneous_ops_per_sec'

# Key distribution per DB
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli info keyspace

# Recent slow queries
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli slowlog get 10

# Live command sampling (1-second window)
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli --latency-history -i 1

Rate-limiter keys (DB 1)

rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \
  redis-cli -n 1 dbsize

rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \
  redis-cli -n 1 --scan --pattern 'rl:ip:*' | head -20

Seed the UA deny list (once after first deploy)

rl:bot:ua:blocked is a permanent Redis SET in DB 1. Each member is the SHA-256 of a UA token — the identifying fragment extracted after splitting on /, spaces, ;, (, ), e.g.:

UA string:  "GPTBot/1.1 (+https://openai.com/gptbot)"
Tokens:      GPTBot  1.1  +https:  ...
Hash stored: sha256("GPTBot")

The middleware (_is_redis_blocked_ua) tokenises the incoming UA the same way and checks each token hash against the cached set. The SET is fetched from Redis at most once per RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds (default 60) per worker process.

The bots in BOT_UA_FRAGMENTS (Python list, always active) and this Redis SET are independent — the Python list provides the baseline and the Redis SET allows adding new offenders at runtime without a code deploy.

rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \
  SADD rl:bot:ua:blocked \
    "$(echo -n 'GPTBot'             | sha256sum | cut -d' ' -f1)" \
    "$(echo -n 'ClaudeBot'          | sha256sum | cut -d' ' -f1)" \
    "$(echo -n 'PerplexityBot'      | sha256sum | cut -d' ' -f1)" \
    "$(echo -n 'Bytespider'         | sha256sum | cut -d' ' -f1)" \
    "$(echo -n 'AhrefsBot'          | sha256sum | cut -d' ' -f1)" \
    "$(echo -n 'meta-externalagent' | sha256sum | cut -d' ' -f1)"

# Add a new offender at runtime (picked up within RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds)
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \
  SADD rl:bot:ua:blocked "$(echo -n 'NewBot' | sha256sum | cut -d' ' -f1)"

Local standalone Redis (development / testing)

No Kubernetes? Run Redis directly with Docker:

sudo docker run --rm -p 6379:6379 redis:7-alpine \
  redis-server --save "" --appendonly no

Then point Django at it by exporting the env var before starting the dev server:

export REDIS_URL="redis://localhost:6379"
export CACHE_BACKEND="redis"
python manage.py runserver

Or add them to your local .env file:

REDIS_URL=redis://localhost:6379
CACHE_BACKEND=redis

Note: the waffle switch REDIS_CACHE must also be on in your local database for start.sh to activate the Redis backend. Run:
python manage.py waffle_switch REDIS_CACHE on --create

Update `redis.conf` without redeploying

# Edit the ConfigMap
rancher kubectl -n sapl-redis edit configmap redis-config

# Restart the pod to pick up the new config
rancher kubectl -n sapl-redis rollout restart deployment/sapl-redis

Gunicorn tuning

docker/startup_scripts/gunicorn.conf.py — resolved values for the current pod budget (1600Mi RAM, 1 CPU):

NUM_WORKERS = int(os.getenv("WEB_CONCURRENCY", "2"))      # was 3
THREADS     = int(os.getenv("GUNICORN_THREADS", "4"))      # was 8
TIMEOUT     = int(os.getenv("GUNICORN_TIMEOUT", "120"))    # was 300

max_requests                = 1000
max_requests_jitter         = 200
worker_max_memory_per_child = 400 * 1024 * 1024  # 400 MB — was 300 MB

Per-location timeout strategy — nginx overrides the global Gunicorn timeout per-path:

Operation	Timeout	Rationale
Normal page rendering	60 s	No legitimate page should take > 60 s
API endpoints	30 s	Stateless, fast by design
PDF download (cached / nginx)	30 s	nginx serves from disk, worker not involved
PDF generation (uncached)	180 s	Kept high — addressed in a future phase
Large file upload	180 s	nginx buffers upload; worker processes after

nginx real-IP and core fixes

Added to docker/config/nginx/nginx.conf (http {} block):

# Kernel bypass — was off (bug)
sendfile    on;
tcp_nopush  on;
tcp_nodelay on;

# Real client IP from X-Forwarded-For set by K8s Ingress
real_ip_header     X-Forwarded-For;
real_ip_recursive  on;
set_real_ip_from   10.0.0.0/8;
set_real_ip_from   172.16.0.0/12;
set_real_ip_from   192.168.0.0/16;

Without real_ip_recursive on, $remote_addr inside the pod would always be the Ingress IP, making IP-based rate limiting and blocking meaningless.

Django upload settings

Added to sapl/settings.py — files above 2 MB are streamed to disk rather than held in worker RAM. Critical for 150 MB upload support without OOM pressure:

FILE_UPLOAD_MAX_MEMORY_SIZE = 2 * 1024 * 1024       # 2 MB
DATA_UPLOAD_MAX_MEMORY_SIZE = 10 * 1024 * 1024      # 10 MB
MAX_DOC_UPLOAD_SIZE         = 150 * 1024 * 1024     # 150 MB
FILE_UPLOAD_TEMP_DIR        = '/var/interlegis/sapl/tmp'

N+1 fix — `get_etiqueta_protocolos`

sapl/relatorios/views.py — previously called MateriaLegislativa.objects.filter() inside a loop over protocols. Fixed to three queries total regardless of volume (one for protocols, one for materias, one for documentos):

# sapl/relatorios/views.py
def get_etiqueta_protocolos(prots):
    prot_list = list(prots)
    if not prot_list:
        return []

    # Pre-fetch MateriaLegislativa for all protocols in one query.
    materia_query = Q()
    for p in prot_list:
        materia_query |= Q(numero_protocolo=p.numero, ano=p.ano)
    materias_map = {
        (m.numero_protocolo, m.ano): m
        for m in MateriaLegislativa.objects.filter(
            materia_query).select_related('tipo')
    }

    # Pre-fetch DocumentoAdministrativo for all protocols in one query.
    documentos_map = {
        doc.protocolo_id: doc
        for doc in DocumentoAdministrativo.objects.filter(
            protocolo__in=prot_list).select_related('tipo')
    }

    protocolos = []
    for p in prot_list:
        dic = {}
        dic['titulo'] = str(p.numero) + '/' + str(p.ano)
        # ... timestamp / assunto / interessado / autor fields ...

        materia = materias_map.get((p.numero, p.ano))
        dic['num_materia'] = (
            materia.tipo.sigla + ' ' + str(materia.numero) + '/' + str(materia.ano)
            if materia else ''
        )

        documento = documentos_map.get(p.pk)
        dic['num_documento'] = (
            documento.tipo.sigla + ' ' + str(documento.numero) + '/' + str(documento.ano)
            if documento else ''
        )

        dic['ident_processo'] = dic['num_materia'] or dic['num_documento']
        protocolos.append(dic)
    return protocolos

Rate limiting — two layers, two jobs

SAPL enforces rate limits at two independent layers. They use different algorithms and protect different things; their thresholds must be tuned separately.

Layer 1 — nginx `limit_req` (leaky bucket)

Defined in docker/config/nginx/nginx.conf (zones) and sapl.conf (burst).

sapl_general  rate=30r/m   # 1 token every 2 s
sapl_heavy    rate=10r/m   # 1 token every 6 s  (PDF/report endpoints)

burst=N nodelay means nginx accepts up to N requests instantly above the current token level, then enforces the drip rate. Requests beyond the burst cap return 429 before reaching Gunicorn — zero Python cost.

Burst values are set at container startup via env vars:

Env var	Default	Location
`NGINX_BURST_GENERAL`	`60`	`location /`, `location /media/`
`NGINX_BURST_API`	`60`	`location /api/`
`NGINX_BURST_HEAVY`	`20`	`location /relatorios/`

Defaults are 2× the zone's per-minute rate, so a user can spend a full minute's quota in a single burst before the leaky bucket takes over.

Layer 2 — Django `RateLimitMiddleware` (sliding window)

Defined in sapl/middleware/ratelimit.py, backed by Redis DB 1.

Requests that pass nginx reach Python. The middleware counts them in a 60-second sliding window per IP (anonymous) or per user (authenticated):

Env var	Default	Scope
`RATE_LIMITER_RATE`	`35/m`	Anonymous IP
`RATE_LIMITER_RATE_AUTHENTICATED`	`120/m`	Authenticated user
`RATE_LIMITER_RATE_BOT`	`5/m`	(reserved — bots are currently blocked outright, not counted)
`RATE_LIMITER_UA_BLOCKLIST_REFRESH`	`60` s	How often each worker re-fetches `rl:bot:ua:blocked` from Redis

When the window count hits the threshold the IP/user is written to a Redis blocked-set with a 300 s TTL and subsequent requests return 429 with Retry-After: 300 — without touching the database.

Decision flow inside RateLimitMiddleware._evaluate():

1.  IP in whitelist?                          → pass (no further checks)
1a. UA matches BOT_UA_FRAGMENTS list?         → 429  reason=known_ua
1b. UA token hash in rl:bot:ua:blocked SET?   → 429  reason=redis_ua
2.  IP in rl:ip:{ip}:blocked?                 → 429  reason=ip_blocked
2b. Path extension in RATE_LIMIT_SCANNER_EXTENSIONS? → SET blocked, 429  reason=scanner_probe
3.  Authenticated user?
    3a. User in rl:{ns}:user:{uid}:blocked?   → 429  reason=user_blocked
    3b. Suspicious headers (no Accept/AL)?    → 429  reason=suspicious_headers_auth
    3c. User request count ≥ auth threshold?  → SET blocked, 429  reason=auth_user_rate
4.  Anonymous:
    4a. Suspicious headers?                   → 429  reason=suspicious_headers
    4b. IP request count ≥ anon threshold?    → SET blocked, 429  reason=ip_rate
    4c. NS/IP window count ≥ anon threshold?  → SET blocked, 429  reason=ua_rotation
    → pass

Decision flow diagram

flowchart TD
    REQ([Request]) --> C1

    C1{"Known bot UA?"}
    C1 -- "yes — substring in BOT_UA_FRAGMENTS" --> R_UA([429\nknown_ua])
    C1 -- no --> C1B

    C1B{"Redis UA deny list?"}
    C1B -- "yes — token hash in rl:bot:ua:blocked" --> R_RUA([429\nredis_ua])
    C1B -- no --> C2

    C2{"IP blocked?"}
    C2 -- "yes — rl:ip:IP:blocked exists" --> R_IPB([429\nip_blocked])
    C2 -- no --> C2B

    C2B{"Scanner extension?\n.php .asp .aspx …"}
    C2B -- yes --> SIPB["SET rl:ip:IP:blocked  TTL 300 s"]
    SIPB --> R_SCN([429\nscanner_probe])
    C2B -- no --> C3

    C3{"Authenticated?"}
    C3 -- yes --> C3A
    C3 -- no --> C4A

    subgraph AUTH ["Authenticated"]
        C3A{"User blocked?"}
        C3A -- "yes — rl:ns:user:UID:blocked" --> R_UB([429\nuser_blocked])
        C3A -- no --> C3B
        C3B{"Suspicious headers?\nno Accept-Language + no Accept"}
        C3B -- yes --> R_SH([429\nsuspicious_headers_auth])
        C3B -- no --> C3C
        C3C{"User rate ≥ 120/min?"}
        C3C -- yes --> SUB["SET rl:ns:user:UID:blocked  TTL 300 s"]
        SUB --> R_AUR([429\nauth_user_rate])
        C3C -- no --> PASS_A([✓ pass])
    end

    subgraph ANON ["Anonymous"]
        C4A{"Suspicious headers?\nno Accept-Language + no Accept"}
        C4A -- yes --> R_ASH([429\nsuspicious_headers])
        C4A -- no --> C4B
        C4B{"IP rate ≥ 35/min?"}
        C4B -- yes --> SIPR["SET rl:ip:IP:blocked  TTL 300 s"]
        SIPR --> R_IPR([429\nip_rate])
        C4B -- no --> C4C
        C4C{"NS/IP window hit\n≥ 35 in bucket?"}
        C4C -- yes --> SUAR["SET rl:ip:IP:blocked  TTL 300 s"]
        SUAR --> R_UAR([429\nua_rotation])
        C4C -- no --> PASS_N([✓ pass])
    end

Enforcement graduation order

Roll out to canary pods first; promote check-by-check in order of false-positive risk:

Order	Check	Reason	Risk	Condition to promote
1st	`known_ua`	Substring in hardcoded `BOT_UA_FRAGMENTS` list	Zero	UA strings are deterministic
2nd	`redis_ua`	Token hash in `rl:bot:ua:blocked` SET	Zero	Keys only set manually by operators
3rd	`ip_blocked`	Marker set by prior proven-bad requests	Zero	Fast-path only, no new blocks created
4th	`scanner_probe`	Path ext in `RATE_LIMIT_SCANNER_EXTENSIONS`	Zero	Django never legitimately serves `.php`/`.asp`/etc.
5th	`ip_rate`	Rolling IP counter ≥ 35/min	Low	Threshold calibrated from canary logs
6th	`suspicious_headers`	No Accept-Language and no Accept	Medium	Confirmed no legitimate clients omit both headers
7th	`ua_rotation` (ns/window)	NS/IP clock-aligned bucket ≥ 35	Medium	NAT IP whitelist in place (see Open Questions)

Decorator migration

For views where django-ratelimit decorators already exist:

Endpoint type	Action	Reason
List views (GET)	Remove after middleware stable	Middleware covers equivalent threshold
Detail views (GET)	Remove after middleware stable	Middleware covers equivalent threshold
Search / filter views	Remove last	Expensive queries — keep stricter per-view limit until traffic data confirms safety
PDF / file generation	Keep permanently	Most expensive endpoint; per-view limit tighter than global
Write endpoints (POST/PUT/DELETE)	Keep permanently	Different abuse surface
Auth endpoints (login, reset)	Keep permanently	Credential stuffing; must be independent of IP rate

Why they are not the same number

	nginx burst	Django threshold
Algorithm	Leaky bucket — token refills over time	Sliding window — hard count per 60 s
Protects	Gunicorn workers from being flooded	Per-client fairness, business policy
Tuned by	Capacity of the server	Acceptable request volume per client
Failure mode	Workers overwhelmed	Legitimate user over-browsing

A user loading a page quickly may fire 5–10 Django requests in two seconds. With rate=30r/m (1 token/2 s) and burst=60 they absorb that fine; the leaky bucket refills before they click the next link. The Django threshold (35/m sliding window) catches sustained automated traffic from a single IP that looks like scraping even if it arrives slowly enough to beat the nginx burst cap.

Request routing — how nginx reaches Django

proxy_pass http://sapl_server forwards the HTTP request — with the original path intact — to the Gunicorn Unix socket. Django doesn't know or care that nginx is in front; it sees a standard HTTP request.

GET /media/foo.pdf
      │
      ▼
   nginx (sapl.conf)
   location /media/ → proxy_pass to Unix socket
      │
      ▼
   Gunicorn (WSGI server)
   receives raw HTTP, calls Django WSGI application
      │
      ▼
   Django middleware stack (settings.MIDDLEWARE)
   RateLimitMiddleware → pass or 429
      │
      ▼
   Django URL router (sapl/urls.py)
   r'^media/(?P<path>.*)$' → serve_media
      │
      ▼
   serve_media(request, path='foo.pdf')
   returns HttpResponse with X-Accel-Redirect: /internal/media/foo.pdf
      │
      ▼
   nginx sees X-Accel-Redirect header
   /internal/media/ internal location → reads file from disk → sends to client

nginx does no routing beyond picking a location block. The mapping from URL path to Python function lives entirely in sapl/urls.py. proxy_pass is just a pipe.

Media file serving — `serve_media` and X-Accel-Redirect

All /media/ requests (public and private) are routed through Gunicorn so that Django middleware runs on every hit. Nginx serves the file bytes via X-Accel-Redirect — the Gunicorn worker is freed as soon as it sends the response headers.

nginx locations (`docker/config/nginx/sapl.conf`)

# Static files — no rate limiting, no proxy; 90-minute browser cache.
location /static/ {
    alias /var/interlegis/sapl/collected_static/;
    expires 90m;
    add_header Cache-Control "public, max-age=5400";
}

# Proxied to Gunicorn — Django middleware + serve_media() run here.
location /media/ {
    limit_req zone=sapl_general burst=${NGINX_BURST_GENERAL} nodelay;
    proxy_pass http://sapl_server;
}

# Internal — only reachable via X-Accel-Redirect, not by external clients.
location /internal/media/ {
    internal;
    alias /var/interlegis/sapl/media/;
    sendfile on;
    etag on;
}

Upload endpoints (/protocoloadm/criar-protocolo, /materia/.*upload, /norma/.*upload) no longer have a dedicated location block — they fall through to location / which applies the sapl_general zone.

Django view (`sapl/base/media.py`)

serve_media(request, path) — registered at ^media/(?P<path>.*)$ in sapl/urls.py.

Per-request steps:

Path traversal guard — os.path.abspath check; raises 404 on escape.
Auth gate — documentos_privados/ paths require an authenticated session; redirects to login otherwise.
Path counter — increments rl:{ns}:path:{sha256}:reqs in Redis DB 1 (TTL = MEDIA_PATH_COUNTER_TTL).
Serve — in DEBUG: django.views.static.serve directly. In production: X-Accel-Redirect: /internal/media/<path>. Nginx sets Content-Type from its own mime.types.

Settings

Setting	Default	Purpose
`MEDIA_PATH_COUNTER_TTL`	`60` s	TTL for both URL-path and storage-path counters (DB 1)

File serving decision matrix

File type	Size	Strategy
Logos / images	Any	nginx `alias` + `sendfile` + ETag + `Cache-Control`
Small PDFs	≤ 360 KB	nginx direct + ETag
Medium PDFs	360 KB – 2 MB	nginx direct + ETag + rate limit
Large PDFs	> 2 MB	nginx direct + strict rate limit; never Redis
LGPD-restricted	Any	Django `serve_media` → `X-Accel-Redirect` → nginx (access control enforced)
Public `/media/`	Any	Django `serve_media` → `X-Accel-Redirect` → nginx (middleware runs; path counter written)

Why Redis is not needed for PDFs

With the full mitigation stack active:

ASN blocking drops datacenter bot traffic at nginx (zero Python cost)
UA blocking drops known-UA bots at nginx (zero Python cost)
Shared Redis rate counters enforce limits across all pods
ETags convert repeat requests to 304 responses with zero bytes transferred
sendfile on means disk reads bypass userspace entirely

Redis PDF caching would solve "high request volume reaching the file layer" — but that problem no longer exists once the above stack is active. For Brasão - Foz do Iguaçu.png (392 KB × 14,512 requests = 5.6 GB), a 50% conditional-request hit rate saves ~2.8 GB immediately — without any Redis.

Key schema reference

DB	Use case	Key pattern	TTL	Threshold	Constant
0	Page / view cache	`cache:{ns}:*`	300 s (default)	—	`CACHES['default']` KEY_PREFIX
0	Static file cache (logos)	`static:{ns}:{sha256}`	3 – 24 h	—	Future (requires OpenResty/Lua)
0	File content cache (≤ 360 KB)	`file:{ns}:{sha256}`	1 h	—	Future
1	IP rate-limit counter	`rl:ip:{ip}:reqs`	60 s	35 (`RATE_LIMITER_RATE`)	`RL_IP_REQUESTS`
1	IP blocked marker	`rl:ip:{ip}:blocked`	300 s	—	`RL_IP_BLOCKED`
1	User rate-limit counter	`rl:{ns}:user:{uid}:reqs`	60 s	120 (`RATE_LIMITER_RATE_AUTHENTICATED`)	`RL_USER_REQUESTS`
1	User blocked marker	`rl:{ns}:user:{uid}:blocked`	300 s	—	`RL_USER_BLOCKED`
1	Namespace/IP sliding window	`rl:{ns}:ip:{ip}:w:{bucket}`	120 s	35 (`RATE_LIMITER_RATE`)	`RL_NS_WINDOW`
1	Path counter (`/media/`)	`rl:{ns}:path:{sha256}:reqs`	60 s	— (observability only)	`RL_PATH_REQUESTS`
1	Path counter (`/static/`)	`rl:{ns}:path:{sha256}:reqs`	60 s	—	Future (requires OpenResty/Lua)
1	UA deny list	`rl:bot:ua:blocked`	permanent SET	— (block on match)	`RL_UA_BLOCKLIST`
2	Django Channels	`channels:*`	session TTL	—	Future

What each counter catches — and misses

rl:ip:{ip}:reqs — global rolling IP counter

Catches: any sustained anonymous volume from a single IP regardless of namespace, path, or User-Agent — pure request rate.

Misses: a user legitimately accessing several municipality SAPLs simultaneously; their requests accumulate across namespaces into one global count and may trip the threshold even though no individual SAPL is being abused. Also misses a timing-aware scraper that paces exactly 34 req/min: the 60 s TTL resets from the first request, so the attacker can safely send 34, wait for reset, repeat forever.

rl:ip:{ip}:blocked — IP short-circuit marker

Written when rl:ip:{ip}:reqs hits the anonymous threshold (step 4b) or when the namespace/IP bucket hits the threshold (step 4c). Checked at step 2 — before any counting — so a blocked IP never increments any counter on subsequent requests.

Catches: saves Redis INCR + EXPIRE calls for every request from an already-blocked IP; the 300 s TTL is a hard cooldown regardless of how many requests arrive.

Misses: the TTL is fixed — a persistent attacker simply waits 300 s and gets another full window quota. Also, because the key is global (no namespace), an IP blocked for one municipal SAPL is blocked for all SAPLs on the same pod — collateral effect for shared IPs.

rl:{ns}:ip:{ip}:w:{bucket} — namespace-scoped clock-aligned bucket

Catches: sustained scraping against a specific municipal SAPL that stays just under the global threshold; a scraper pacing 34 req/min globally across namespaces still accumulates in the per-namespace bucket. Clock alignment (bucket = time() // 60) means a burst straddling a minute boundary still contributes to the next bucket for 120 s (2× TTL), making precise timing attacks harder.

Misses: an IP that floods one namespace to exactly 34 req/min: it never reaches 35 in the bucket either. Cross-namespace legitimate traffic that happens to land within the same clock minute — same blind spot as rl:ip:* but scoped lower.

Why this key is namespace-scoped

Five arguments for rl:{ns}:ip:{ip}:w:{bucket} over a global rl:ip:{ip}:w:{bucket}:

Matches the observed attack pattern. The botnet in §Bot Traffic Profile targets one SAPL at a time, not the fleet evenly. A scraper hammering fortaleza-ce at 34 req/min has a namespace counter of 34 and a global counter of 34. Without the namespace the two keys are redundant — the window adds no new signal. With it, a scraper that legitimately distributes across 5 SAPLs (7 req/min each, 35 globally) is caught globally but not per-SAPL — correct behaviour, since no single SAPL is being abused.
Two counters defeat two different gaming strategies. rl:ip:{ip}:reqs uses a rolling TTL (starts on the first INCR). A scraper that knows this can send 34 requests, wait ~61 s for the key to expire, and repeat indefinitely. The clock-aligned window resets at wall-clock minute boundaries. To game both simultaneously the attacker must time bursts to expire the rolling key and land entirely within one clock window — two independent constraints that are hard to satisfy together.
Without the namespace it duplicates the global counter. All pods share the same Redis. A global rl:ip:{ip}:w:{bucket} would aggregate that IP's traffic from every pod — exactly what rl:ip:{ip}:reqs already does, just with different reset timing. Two keys measuring the same dimension is wasted INCR overhead with no added signal.
Multi-SAPL legitimate IPs are not penalised. Municipal IT departments, ISP shared exit nodes, and Googlebot all produce high global request rates while being individually harmless to any one SAPL. A namespaced window lets them access 10 SAPLs at 3 req/min each without triggering a per-SAPL block, while the global counter still catches them if their total rate is abusive.
Consistent with the established {ns} isolation contract. All user-keyed (rl:{ns}:user:{uid}:*) and path-keyed (rl:{ns}:path:{sha256}:reqs) entries are namespace-scoped. A global window key would break the invariant that per-tenant data is isolated — complicating key-space inspection, SCAN-based dashboards, and future per-tenant rate adjustments.

rl:{ns}:user:{uid}:reqs — authenticated user counter

Catches: an authenticated account being used as a scraping credential — even if the requests come from many different IPs (e.g., distributed proxy pool), all requests share the same uid and accumulate in one counter.

Misses: a credential that is shared across multiple legitimate users in the same office; all their activity adds up to one counter and can trip the 120/min threshold during a busy session.

rl:{ns}:user:{uid}:blocked — authenticated user short-circuit marker

Written when rl:{ns}:user:{uid}:reqs hits the authenticated threshold (step 3c). Checked at step 3a — before counting — so a blocked user never increments their counter on subsequent requests during the 300 s cooldown.

Catches: credential-stuffing or runaway automation using a valid session — once the 120/min threshold is hit, the account is locked out immediately for 300 s. Unlike the IP marker, the block is namespace-scoped, so the same user account can be blocked on one SAPL but still active on another.

Misses: same fixed-TTL weakness as the IP marker — a persistent attacker resumes after 300 s. An account shared by multiple legitimate users (e.g., a departmental login) can be locked out during peak collaborative use.

rl:{ns}:path:{sha256}:reqs — per-media-file URL counter

Currently observability-only (no threshold enforced). Intended for future hot-file detection: a single document being hammered by many IPs would show a spike in this counter even if no individual IP exceeds the IP threshold.

Misses: nothing is blocked today. Once a threshold is added, it will miss distributed access where many IPs each download the file once (legitimate CDN pre-warming or public interest event).

rl:bot:ua:blocked — runtime UA deny list

Catches: new bot UA tokens added at runtime via redis-cli SADD without a code deploy; picked up within RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds (default 60) per worker. Complements the hardcoded BOT_UA_FRAGMENTS Python list.

Misses: bots that rotate UA tokens on every request (no single token accumulates); bots that impersonate a valid browser UA completely (no known fragment to match).

Dynamic page caching

Goal: Eliminate ORM queries for anonymous bot requests on list views. Prerequisite: Phase 1 (shared Redis, CACHE_BACKEND=redis).

Many SAPL list views (pesquisar-materia, norma, etc.) are not truly dynamic for anonymous users between edits. A bot hammering ?page=1 through ?page=100 triggers 100 ORM queries per pod. With Redis page cache, each unique URL is queried once per TTL across the entire fleet.

# Apply to anonymous list views only — AnonCachePageMixin already wired to materia/sessao detail views.
from django.views.decorators.cache import cache_page
from django.utils.decorators import method_decorator

@method_decorator(cache_page(60 * 5), name='dispatch')  # 5-minute TTL
class PesquisarMateriaView(FilterView):
    ...

Safety check: cache_page sets Cache-Control: private for authenticated sessions automatically. Verify this is working before deploying — accidentally caching a session-aware response is a data leak.

Cache TTL guidelines

View type	TTL	Reasoning
Matéria list (anonymous)	300 s	Changes infrequently between sessions
Norma list (anonymous)	300 s	Same
Parlamentar list	3600 s	Changes rarely
Search results	60 s	Query-dependent; shorter TTL safer
Authenticated views	Never	`cache_page` respects this automatically
PDF generation	Never	Too large — serve from disk via nginx

HTTP Conditional Requests

Two complementary mechanisms eliminate redundant work for unchanged content.

`ConditionalGetMiddleware` (all views)

Added to MIDDLEWARE in sapl/settings.py (after CommonMiddleware). For every Django response it:

Generates a weak ETag from an MD5 of the response body if none is set.
Compares against the client's If-None-Match / If-Modified-Since.
Returns 304 Not Modified (no body) on a match.
Handles HEAD requests by stripping the body and keeping headers.

Caveat: the view still executes and renders before the check fires. The saving is bandwidth, not CPU/DB work.

`@condition` decorator — materia and norma detail views

For MateriaLegislativaCrud.DetailView and NormaCrud.DetailView a cheap freshness function runs before the view body:

# sapl/materia/views.py
def _materia_last_modified(request, *args, **kwargs):
    return MateriaLegislativa.objects.filter(
        pk=kwargs['pk']
    ).values_list('data_ultima_atualizacao', flat=True).first()

def _materia_etag(request, *args, **kwargs):
    ts = _materia_last_modified(request, *args, **kwargs)
    return f'{kwargs["pk"]}-{ts.timestamp()}' if ts else None

@method_decorator(condition(etag_func=_materia_etag, last_modified_func=_materia_last_modified), name='get')
class DetailView(AnonCachePageMixin, Crud.DetailView):
    ...

NormaCrud.DetailView follows the same pattern with _norma_last_modified / _norma_etag querying NormaJuridica.ultima_edicao.

On a cache hit: one VALUES query fires, Django returns 304 — view body, template render, and ORM work are all skipped.

Signal used: data_ultima_atualizacao (auto_now=True) — updated by Django on every save(), so the ETag is invalidated automatically whenever the record changes.

Open Questions

#	Question	Status	Blocks
1	Does Chrome/98.0.4758 impersonator appear consistently in nginx access logs?	Needs investigation	UA block safety
2	Which legislative house IPs can be pre-whitelisted in `RATE_LIMIT_WHITELIST_IPS`?	No list yet — obtain in the future. Setting is optional / future.	Enforcement safety for NAT users
3	`CONN_MAX_AGE` tuning	Currently 300 s (`sapl/settings.py`). Evaluate whether to reduce given worker recycling at 400 MB.	Gunicorn tuning
4	WebSocket voting panel priority	Separate project. Resumes after Redis is on k8s, bot siege addressed, and OOM pressure reduced.	Phase 5 sequencing

41 KiB Raw Blame History