SPDT
/
sapl
mirror of https://github.com/interlegis/sapl.git
5
0
Fork 0
Code Issues Releases Wiki Activity
Browse Source

Add API-specific rate limiter and remove emergency block middleware 

- Delete ApiEmergencySameSiteOnlyMiddleware (api_emergency_block.py);
  replace with permanent _handle_api logic inside RateLimitMiddleware.
- _handle_api decision chain: OPTIONS pass → same-origin pass →
  global IP block check → API-specific block check → quota → API
  per-minute counter → anon pass / auth _evaluate.
- New Redis keys: rl:api:ip:<ip>:reqs, rl:api:ip:<ip>:blocked,
  rl:index:api_blocked_ips. Global rl:ip:<ip>:blocked is never
  written because of /api/ abuse — prevents NAT lockout.
- _is_same_origin: strips port, lowercases, checks Origin first then
  Referer (sequential, not OR — wrong Origin blocks even if Referer matches).
- Five new settings: API_RATE_LIMIT_{ENABLED,THRESHOLD,WINDOW_SECONDS,
  BLOCK_SECONDS,SAME_ORIGIN_BYPASS} with safe defaults.
- 16 new tests; _make_middleware extended with explicit setting values.
- RATE-LIMITER-PLAN.md updated with new key schema rows and _handle_api section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rate-limiter-2026
Edward Ribeiro 3 weeks ago
parent
425519a967
commit
a33fcc2fe6
4 changed files with 433 additions and 28 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 67
      plan/RATE-LIMITER-PLAN.md
  2. 180
      sapl/middleware/ratelimit.py
  3. 205
      sapl/middleware/test_ratelimiter.py
  4. 9
      sapl/settings.py

67
plan/RATE-LIMITER-PLAN.md
View File

@ -1207,6 +1207,9 @@ Redis PDF caching would solve "high request volume reaching the file layer" —
| 1 | API weekly quota (anon) | `quota:{ns}:weekly:{week}:ip:{ip}` | 7 d | 3 500 (`API_QUOTA_ANON_WEEKLY`) | `QUOTA_IP_WEEKLY` | | 1 | API weekly quota (anon) | `quota:{ns}:weekly:{week}:ip:{ip}` | 7 d | 3 500 (`API_QUOTA_ANON_WEEKLY`) | `QUOTA_IP_WEEKLY` |
| 1 | API daily quota (auth) | `quota:{ns}:daily:{date}:user:{uid}` | 24 h | 5 000 (`API_QUOTA_AUTH_DAILY`) | `QUOTA_USER_DAILY` | | 1 | API daily quota (auth) | `quota:{ns}:daily:{date}:user:{uid}` | 24 h | 5 000 (`API_QUOTA_AUTH_DAILY`) | `QUOTA_USER_DAILY` |
| 1 | API weekly quota (auth) | `quota:{ns}:weekly:{week}:user:{uid}` | 7 d | 35 000 (`API_QUOTA_AUTH_WEEKLY`) | `QUOTA_USER_WEEKLY` | | 1 | API weekly quota (auth) | `quota:{ns}:weekly:{week}:user:{uid}` | 7 d | 35 000 (`API_QUOTA_AUTH_WEEKLY`) | `QUOTA_USER_WEEKLY` |
| 1 | API IP rate counter (anon external) | `rl:api:ip:{ip}:reqs` | 60 s (`API_RATE_LIMIT_WINDOW_SECONDS`) | 60 (`API_RATE_LIMIT_THRESHOLD`) | `RL_API_IP_REQUESTS` |
| 1 | API IP block marker (anon external) | `rl:api:ip:{ip}:blocked` | 300 s (`API_RATE_LIMIT_BLOCK_SECONDS`) | — | `RL_API_IP_BLOCKED` |
| 1 | API blocked-IP ZSET index | `rl:index:api_blocked_ips` | permanent ZSET, score=expiry ts | — | `RL_INDEX_API_BLOCKED_IPS` |
| 2 | Django Channels | `channels:*` | session TTL | — | *Future* | | 2 | Django Channels | `channels:*` | session TTL | — | *Future* |
### What each counter catches — and misses ### What each counter catches — and misses
@ -1332,6 +1335,70 @@ bots that impersonate a valid browser UA completely (no known fragment to match)
--- ---
## /api/ rate limiting — `_handle_api` (2026-05-11)
### Problem
Three concurrent problems made the old anonymous-/api/-passes-immediately design
insufficient:
1. **SAPL itself polls /api/** from the browser (same-origin). It must not be counted or blocked.
2. **Legitimate user scripts behind NAT** poll `/api/` aggressively. The old design had no per-minute cap; quota (daily/weekly) was the only gate, and quota exhaustion wrote `rl:ip:<ip>:blocked`, locking every person behind the same NAT out of the entire application.
3. **Bots** hammer `/api/` with no constraint other than the nginx `sapl_api` zone (60 r/m burst 120).
### Solution — `_handle_api`
`RateLimitMiddleware.__call__` delegates all `/api/` requests to `_handle_api`, which applies a separate, scoped decision chain:
| Step | Condition | Action |
|------|-----------|--------|
| 1 | `OPTIONS` method | Pass — CORS preflight must never be blocked |
| 2 | Same-origin (`_is_same_origin`) | Pass — SAPL's own browser polling; no counter |
| 3 | `rl:ip:<ip>:blocked` exists | 429 `global_ip_blocked` — global block also covers `/api/` |
| 4 | `rl:api:ip:<ip>:blocked` exists | 429 `api_ip_blocked` — API-only block |
| 5 | Daily/weekly quota exceeded | 429 `quota_daily` / `quota_weekly` (unchanged) |
| 6 | Anon: API counter ≥ threshold | Write `rl:api:ip:<ip>:blocked`; 429 `api_threshold_exceeded` |
| 6 | Anon: under threshold | Pass |
| 7 | Authenticated | Delegate to `_evaluate` (per-user counter unchanged) |
**Key invariant**: `rl:ip:<ip>:blocked` is **never written** because of `/api/` abuse.
`rl:api:ip:<ip>:blocked` blocks only `/api/` — page requests from the same NAT continue.
### Same-origin detection — `_is_same_origin`
Replaces `ApiEmergencySameSiteOnlyMiddleware._came_from_same_host` (deleted).
| Aspect | Emergency block | `_is_same_origin` |
|--------|-----------------|-------------------|
| Normalization | strip port, lowercase (both sides) | same |
| Origin + Referer | `origin_match OR referer_match` | sequential: Origin first, Referer only if Origin absent |
| Wrong Origin with matching Referer | pass (Referer wins) | block (explicit wrong Origin = cross-origin) |
| Both absent | block | block |
The sequential check is stricter and matches the spec: an explicit wrong `Origin`
header means the browser knows this is cross-origin, regardless of what `Referer` says.
### Settings
| Setting | Env var | Default | Purpose |
|---------|---------|---------|---------|
| `API_RATE_LIMIT_ENABLED` | same | `True` | Master switch; set False to revert to quota-only |
| `API_RATE_LIMIT_THRESHOLD` | same | `60` | Requests per window before API block |
| `API_RATE_LIMIT_WINDOW_SECONDS` | same | `60` | Counter TTL (seconds) |
| `API_RATE_LIMIT_BLOCK_SECONDS` | same | `300` | `rl:api:ip:<ip>:blocked` TTL |
| `API_RATE_LIMIT_SAME_ORIGIN_BYPASS` | same | `True` | Disable to test without same-origin pass |
### Files changed
| File | Change |
|------|--------|
| `sapl/middleware/api_emergency_block.py` | Deleted |
| `sapl/settings.py` | Removed `ApiEmergencySameSiteOnlyMiddleware` from `MIDDLEWARE`; added 5 new `API_RATE_LIMIT_*` settings |
| `sapl/middleware/ratelimit.py` | Added `RL_API_IP_REQUESTS`, `RL_API_IP_BLOCKED`, `RL_INDEX_API_BLOCKED_IPS` constants; added `_is_same_origin`; extended `__init__`; added `_handle_api`, `_api_block_response`; refactored `__call__` |
| `sapl/middleware/test_ratelimiter.py` | Extended `_make_middleware`; added 16 new tests |
---
## Dynamic page caching ## Dynamic page caching
**Goal**: Eliminate ORM queries for anonymous bot requests on list views. **Goal**: Eliminate ORM queries for anonymous bot requests on list views.

180
sapl/middleware/ratelimit.py
View File

@ -2,9 +2,16 @@
RateLimitMiddleware — cross-pod rate limiting backed by shared Redis. RateLimitMiddleware — cross-pod rate limiting backed by shared Redis.
Decision flow (per request): Decision flow (per request):
0. /api/ path AND consumer daily/weekly quota exceeded? → 429 /api/ paths — handled by _handle_api:
Anonymous /api/ (quota not exceeded): pass immediately — no IP counter, 0a. OPTIONS? → pass (CORS preflight must never be blocked)
no block key. nginx sapl_api zone (60r/m) is the burst gate. 0b. Same-origin? → pass (SAPL's own browser polling)
0c. rl:ip:<ip>:blocked? → 429 (global block also covers /api/)
0d. rl:api:ip:<ip>:blocked? → 429 (API-only block)
0e. Daily/weekly quota exceeded? → 429
0f. Anon + API threshold exceeded? → SET rl:api:ip:<ip>:blocked, 429
(never writes rl:ip:<ip>:blocked)
0g. Auth: falls through to _evaluate (per-user counter)
Non-/api/ paths:
1. Known bot UA? → 429 (Python list — substring match) 1. Known bot UA? → 429 (Python list — substring match)
1b. Redis UA deny list? → 429 (runtime SET — token hash match, refreshed every 60 s) 1b. Redis UA deny list? → 429 (runtime SET — token hash match, refreshed every 60 s)
2. Anonymous AND IP in blocked set? → 429 (authenticated users skip — have per-user limit at 3c) 2. Anonymous AND IP in blocked set? → 429 (authenticated users skip — have per-user limit at 3c)
@ -68,6 +75,11 @@ RL_METRICS_BLOCKED = 'rl:metrics:{ns}:{date}:blocked:{reason}' # daily counter
RL_INDEX_BLOCKED_IPS = 'rl:index:blocked_ips' RL_INDEX_BLOCKED_IPS = 'rl:index:blocked_ips'
RL_INDEX_BLOCKED_USERS = 'rl:index:blocked_users' RL_INDEX_BLOCKED_USERS = 'rl:index:blocked_users'
# API-specific rate limit keys — scope limited to /api/, never written by non-/api/ paths.
RL_API_IP_REQUESTS = 'rl:api:ip:{ip}:reqs'
RL_API_IP_BLOCKED = 'rl:api:ip:{ip}:blocked'
RL_INDEX_API_BLOCKED_IPS = 'rl:index:api_blocked_ips'
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# API quota keys — per-consumer, per-day/week, tenant-scoped. # API quota keys — per-consumer, per-day/week, tenant-scoped.
# Consumer identity: authenticated users by uid, anonymous by masked IP. # Consumer identity: authenticated users by uid, anonymous by masked IP.
@ -190,6 +202,41 @@ def smart_rate(group, request):
return settings.RATE_LIMITER_RATE return settings.RATE_LIMITER_RATE
def _is_same_origin(request):
"""
Return True if Origin or Referer header matches the current SAPL host.
Strips port and lowercases both sides before comparing — DNS is case-insensitive
and reverse proxies may expose a different port than the browser sees.
Checks Origin first; falls back to Referer only when Origin is absent.
Returns False when both headers are absent.
"""
from urllib.parse import urlparse
def _normalize(host):
return host.lower().split(':', 1)[0].strip()
try:
host = _normalize(request.get_host())
except Exception:
return False
origin = request.META.get('HTTP_ORIGIN', '')
if origin:
try:
return _normalize(urlparse(origin).netloc) == host
except ValueError:
return False
referer = request.META.get('HTTP_REFERER', '')
if referer:
try:
return _normalize(urlparse(referer).netloc) == host
except ValueError:
return False
return False
def _is_suspicious_headers(request): def _is_suspicious_headers(request):
"""Real browsers send Accept-Language + Accept; bots frequently omit them.""" """Real browsers send Accept-Language + Accept; bots frequently omit them."""
missing = sum([ missing = sum([
@ -265,6 +312,11 @@ class RateLimitMiddleware:
self.api_quota_anon_weekly = settings.API_QUOTA_ANON_WEEKLY self.api_quota_anon_weekly = settings.API_QUOTA_ANON_WEEKLY
self.api_quota_auth_daily = settings.API_QUOTA_AUTH_DAILY self.api_quota_auth_daily = settings.API_QUOTA_AUTH_DAILY
self.api_quota_auth_weekly = settings.API_QUOTA_AUTH_WEEKLY self.api_quota_auth_weekly = settings.API_QUOTA_AUTH_WEEKLY
self.api_rate_limit_enabled = getattr(settings, 'API_RATE_LIMIT_ENABLED', True)
self.api_threshold = getattr(settings, 'API_RATE_LIMIT_THRESHOLD', 60)
self.api_window = getattr(settings, 'API_RATE_LIMIT_WINDOW_SECONDS', 60)
self.api_block_seconds = getattr(settings, 'API_RATE_LIMIT_BLOCK_SECONDS', 300)
self.api_same_origin_bypass = getattr(settings, 'API_RATE_LIMIT_SAME_ORIGIN_BYPASS', True)
logger.info( logger.info(
'[RATELIMIT] anon=%s auth=%s bot=%s allowlist=%s bypass_paths=%s', '[RATELIMIT] anon=%s auth=%s bot=%s allowlist=%s bypass_paths=%s',
settings.RATE_LIMITER_RATE, settings.RATE_LIMITER_RATE,
@ -280,35 +332,18 @@ class RateLimitMiddleware:
settings.API_QUOTA_AUTH_DAILY, settings.API_QUOTA_AUTH_DAILY,
settings.API_QUOTA_AUTH_WEEKLY, settings.API_QUOTA_AUTH_WEEKLY,
) )
logger.info(
'[API RATE LIMIT] enabled=%s threshold=%s window=%ss block=%ss same_origin_bypass=%s',
self.api_rate_limit_enabled, self.api_threshold, self.api_window,
self.api_block_seconds, self.api_same_origin_bypass,
)
def __call__(self, request): def __call__(self, request):
if any(p.match(request.path) for p in self._bypass_paths): if any(p.match(request.path) for p in self._bypass_paths):
return self.get_response(request) return self.get_response(request)
if request.path.startswith('/api/'): if request.path.startswith('/api/'):
exceeded = self._check_api_quota(request) return self._handle_api(request)
if exceeded:
ip = get_client_ip(request)
logger.warning(
'quota_exceeded window=%s ip=%s path=%s namespace=%s',
exceeded, ip, request.path, _NAMESPACE,
extra={'ua': request.META.get('HTTP_USER_AGENT', '')},
)
self._inc_block_metric(f'quota_{exceeded}')
response = HttpResponse(status=429)
response['Retry-After'] = 86400
response['X-RateLimit-Reason'] = f'quota_{exceeded}'
return response
# Anonymous /api/ requests: quota + nginx sapl_api zone are the only
# controls. Skip _evaluate so anonymous API traffic never increments
# the global IP counter or writes a block key — a misbehaving script
# behind a NAT must not lock out the org's page requests.
# Authenticated /api/ falls through to _evaluate_authenticated normally
# (per-user counter, NAT-safe).
user = getattr(request, 'user', None)
if not (user and user.is_authenticated):
return self.get_response(request)
decision = self._evaluate(request) decision = self._evaluate(request)
if decision['action'] == 'block': if decision['action'] == 'block':
@ -337,6 +372,99 @@ class RateLimitMiddleware:
self._handle_not_found(request, decision['ip']) self._handle_not_found(request, decision['ip'])
return response return response
# ------------------------------------------------------------------
# /api/ handling
# ------------------------------------------------------------------
def _api_block_response(self, reason, retry_after=None):
from django.http import JsonResponse
if retry_after is None:
retry_after = self.api_block_seconds
resp = JsonResponse(
{'detail': 'API rate limit exceeded. Please reduce polling frequency.',
'retry_after_seconds': retry_after},
status=429,
)
resp['Retry-After'] = retry_after
resp['X-RateLimit-Reason'] = reason
return resp
def _handle_api(self, request):
# 1. OPTIONS preflight — CORS must never be blocked
if request.method == 'OPTIONS':
return self.get_response(request)
# 2. Same-origin (SAPL's own polling) — no counter, no block
if self.api_same_origin_bypass and _is_same_origin(request):
return self.get_response(request)
ip = get_client_ip(request)
# 3. Global IP block also covers /api/
if self._rl_cache.get(RL_IP_BLOCKED.format(ip=ip)):
logger.warning(
'api_rate_limit_block reason=global_ip_blocked ip=%s path=%s user_agent=%s',
ip, request.path, request.META.get('HTTP_USER_AGENT', ''),
)
self._inc_block_metric('api_global_ip_blocked')
return self._api_block_response('global_ip_blocked')
# 4. API-specific block (blocks /api/ only, never set by non-/api/ paths)
if self._rl_cache.get(RL_API_IP_BLOCKED.format(ip=ip)):
logger.warning(
'api_rate_limit_block reason=api_ip_blocked ip=%s path=%s user_agent=%s',
ip, request.path, request.META.get('HTTP_USER_AGENT', ''),
)
self._inc_block_metric('api_ip_blocked')
return self._api_block_response('api_ip_blocked')
# 5. Daily/weekly quota (existing logic, preserved)
exceeded = self._check_api_quota(request)
if exceeded:
logger.warning(
'quota_exceeded window=%s ip=%s path=%s namespace=%s',
exceeded, ip, request.path, _NAMESPACE,
extra={'ua': request.META.get('HTTP_USER_AGENT', '')},
)
self._inc_block_metric(f'quota_{exceeded}')
response = HttpResponse(status=429)
response['Retry-After'] = 86400
response['X-RateLimit-Reason'] = f'quota_{exceeded}'
return response
user = getattr(request, 'user', None)
if not (user and user.is_authenticated):
# 6. API-specific per-minute rate limit for anonymous external callers.
# Writes rl:api:ip:<ip>:blocked only — never rl:ip:<ip>:blocked.
if self.api_rate_limit_enabled:
count = self._incr_with_ttl(RL_API_IP_REQUESTS.format(ip=ip), self.api_window)
if count >= self.api_threshold:
_set_block(RL_API_IP_BLOCKED.format(ip=ip), RL_INDEX_API_BLOCKED_IPS, self.api_block_seconds)
logger.warning(
'api_rate_limit_block reason=api_threshold_exceeded '
'ip=%s path=%s user_agent=%s count=%s threshold=%s',
ip, request.path, request.META.get('HTTP_USER_AGENT', ''),
count, self.api_threshold,
)
self._inc_block_metric('api_threshold_exceeded')
return self._api_block_response('api_threshold_exceeded')
return self.get_response(request)
# 7. Authenticated /api/ — existing per-user rate limiter unchanged
decision = self._evaluate(request)
if decision['action'] == 'block':
logger.warning(
'ratelimit_block layer=django reason=%s ip=%s path=%s namespace=%s',
decision['reason'], decision['ip'], request.path, _NAMESPACE,
extra={'ua': request.META.get('HTTP_USER_AGENT', '')},
)
self._inc_block_metric(decision['reason'])
response = HttpResponse(status=429)
response['Retry-After'] = self.BLOCK_TTL
response['X-RateLimit-Reason'] = decision['reason']
return response
return self.get_response(request)
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Evaluation # Evaluation
# ------------------------------------------------------------------ # ------------------------------------------------------------------

205
sapl/middleware/test_ratelimiter.py
View File

@ -13,11 +13,14 @@ from django.test import RequestFactory
from sapl.middleware.ratelimit import ( from sapl.middleware.ratelimit import (
_NAMESPACE, _NAMESPACE,
_is_same_origin,
_is_suspicious_headers, _is_suspicious_headers,
_parse_rate, _parse_rate,
get_client_ip, get_client_ip,
make_ratelimit_cache_key, make_ratelimit_cache_key,
RateLimitMiddleware, RateLimitMiddleware,
RL_API_IP_BLOCKED,
RL_API_IP_REQUESTS,
RL_IP_BLOCKED, RL_IP_BLOCKED,
RL_USER_BLOCKED, RL_USER_BLOCKED,
smart_key, smart_key,
@ -58,7 +61,16 @@ def _auth_req(uid=7, **kwargs):
return r return r
def _make_middleware(allowlist=None, anon_rate='35/m', auth_rate='120/m'): def _make_middleware(
allowlist=None,
anon_rate='35/m',
auth_rate='120/m',
api_rate_limit_enabled=True,
api_threshold=60,
api_window=60,
api_block_seconds=300,
api_same_origin_bypass=True,
):
""" """
Return (middleware, mock_cache). Return (middleware, mock_cache).
@ -79,7 +91,19 @@ def _make_middleware(allowlist=None, anon_rate='35/m', auth_rate='120/m'):
mock_settings.RATE_LIMITER_RATE_AUTHENTICATED = auth_rate mock_settings.RATE_LIMITER_RATE_AUTHENTICATED = auth_rate
mock_settings.RATE_LIMITER_RATE_BOT = '5/m' mock_settings.RATE_LIMITER_RATE_BOT = '5/m'
mock_settings.RATE_LIMIT_ALLOWLIST_IPS = allowlist or [] mock_settings.RATE_LIMIT_ALLOWLIST_IPS = allowlist or []
mock_settings.RATE_LIMIT_404_THRESHOLD = 20
mock_settings.RATE_LIMIT_BYPASS_PATHS = []
mock_settings.POD_NAMESPACE = _NAMESPACE # keep module-level _NAMESPACE consistent mock_settings.POD_NAMESPACE = _NAMESPACE # keep module-level _NAMESPACE consistent
mock_settings.API_QUOTA_ANON_DAILY = 999999
mock_settings.API_QUOTA_ANON_WEEKLY = 999999
mock_settings.API_QUOTA_AUTH_DAILY = 999999
mock_settings.API_QUOTA_AUTH_WEEKLY = 999999
mock_settings.RATE_LIMITER_UA_BLOCKLIST_REFRESH = 60
mock_settings.API_RATE_LIMIT_ENABLED = api_rate_limit_enabled
mock_settings.API_RATE_LIMIT_THRESHOLD = api_threshold
mock_settings.API_RATE_LIMIT_WINDOW_SECONDS = api_window
mock_settings.API_RATE_LIMIT_BLOCK_SECONDS = api_block_seconds
mock_settings.API_RATE_LIMIT_SAME_ORIGIN_BYPASS = api_same_origin_bypass
with ( with (
patch('sapl.middleware.ratelimit.caches') as mock_caches, patch('sapl.middleware.ratelimit.caches') as mock_caches,
@ -383,3 +407,182 @@ def test_call_pass_forwards_request_to_get_response():
request = _anon_req() request = _anon_req()
mw(request) mw(request)
mw.get_response.assert_called_once_with(request) mw.get_response.assert_called_once_with(request)
# ---------------------------------------------------------------------------
# _is_same_origin
# ---------------------------------------------------------------------------
def test_is_same_origin_no_headers_returns_false():
r = _factory.get('/api/materia/')
r.META['SERVER_NAME'] = 'sapl.example.com'
r.META['SERVER_PORT'] = '80'
r.META.pop('HTTP_ORIGIN', None)
r.META.pop('HTTP_REFERER', None)
assert _is_same_origin(r) is False
def test_is_same_origin_matching_origin():
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80')
r.META['HTTP_ORIGIN'] = 'https://sapl.example.com'
assert _is_same_origin(r) is True
def test_is_same_origin_mismatched_origin():
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80')
r.META['HTTP_ORIGIN'] = 'https://other.example.com'
assert _is_same_origin(r) is False
def test_is_same_origin_wrong_origin_blocks_even_if_referer_matches():
"""If Origin is present and wrong, Referer must not be consulted."""
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80')
r.META['HTTP_ORIGIN'] = 'https://evil.com'
r.META['HTTP_REFERER'] = 'https://sapl.example.com/page/'
assert _is_same_origin(r) is False
def test_is_same_origin_referer_used_when_no_origin():
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80')
r.META.pop('HTTP_ORIGIN', None)
r.META['HTTP_REFERER'] = 'https://sapl.example.com/page/?q=1'
assert _is_same_origin(r) is True
def test_is_same_origin_port_stripped_from_both_sides():
"""Host with port and Origin without port must match after normalization."""
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='8000')
r.META['HTTP_HOST'] = 'sapl.example.com:8000'
r.META['HTTP_ORIGIN'] = 'http://sapl.example.com'
assert _is_same_origin(r) is True
def test_is_same_origin_case_insensitive():
r = _factory.get('/api/materia/', SERVER_NAME='sapl.example.com', SERVER_PORT='80')
r.META['HTTP_ORIGIN'] = 'https://SAPL.EXAMPLE.COM'
assert _is_same_origin(r) is True
# ---------------------------------------------------------------------------
# _handle_api — OPTIONS and same-origin bypass
# ---------------------------------------------------------------------------
def _api_req(ip='1.2.3.4', ua='Mozilla/5.0', path='/api/materia/', method='GET', extra_meta=None):
"""Anonymous /api/ request with browser headers."""
request = _factory.generic(method, path)
request.META.update({'REMOTE_ADDR': ip, 'HTTP_USER_AGENT': ua, **_NORMAL_HEADERS})
if extra_meta:
request.META.update(extra_meta)
request.user = MagicMock(is_authenticated=False)
return request
def test_api_options_passes_without_counting():
mw, _ = _make_middleware()
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock()
request = _api_req(method='OPTIONS')
mw(request)
mw.get_response.assert_called_once_with(request)
mw._incr_with_ttl.assert_not_called()
def test_api_same_origin_passes_without_counting():
mw, _ = _make_middleware()
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock()
request = _api_req(extra_meta={
'SERVER_NAME': 'sapl.example.com',
'SERVER_PORT': '80',
'HTTP_HOST': 'sapl.example.com',
'HTTP_ORIGIN': 'https://sapl.example.com',
})
mw(request)
mw.get_response.assert_called_once_with(request)
mw._incr_with_ttl.assert_not_called()
def test_api_malicious_origin_is_not_same_origin():
mw, _ = _make_middleware(api_threshold=999)
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock(return_value=1)
request = _api_req(extra_meta={
'SERVER_NAME': 'sapl.example.com',
'SERVER_PORT': '80',
'HTTP_HOST': 'sapl.example.com',
'HTTP_ORIGIN': 'https://evil.com?x=sapl.example.com',
})
mw(request)
# Must reach the counter (not short-circuit as same-origin)
mw._incr_with_ttl.assert_called_once()
# ---------------------------------------------------------------------------
# _handle_api — rate limiting and block key isolation
# ---------------------------------------------------------------------------
def test_api_external_request_increments_api_counter():
mw, _ = _make_middleware(api_threshold=10)
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock(return_value=5) # under threshold
request = _api_req()
response = mw(request)
mw.get_response.assert_called_once_with(request)
call_args = mw._incr_with_ttl.call_args[0]
assert call_args[0] == RL_API_IP_REQUESTS.format(ip='1.2.3.4')
def test_api_threshold_exceeded_creates_api_block_not_global_block():
mw, _ = _make_middleware(api_threshold=5)
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock(return_value=5) # at threshold
request = _api_req()
with patch('sapl.middleware.ratelimit._set_block') as mock_set_block:
response = mw(request)
assert response.status_code == 429
mock_set_block.assert_called_once()
block_key = mock_set_block.call_args[0][0]
assert block_key == RL_API_IP_BLOCKED.format(ip='1.2.3.4')
assert block_key != RL_IP_BLOCKED.format(ip='1.2.3.4')
def test_api_global_block_also_blocks_api():
mw, mock_cache = _make_middleware()
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock()
ip = '1.2.3.4'
mock_cache.get.side_effect = lambda key: 1 if key == RL_IP_BLOCKED.format(ip=ip) else None
response = mw(_api_req(ip=ip))
assert response.status_code == 429
assert response['X-RateLimit-Reason'] == 'global_ip_blocked'
mw._incr_with_ttl.assert_not_called()
def test_api_specific_block_blocks_api_only():
mw, mock_cache = _make_middleware()
mw._check_api_quota = MagicMock(return_value=None)
mw._incr_with_ttl = MagicMock()
ip = '1.2.3.4'
mock_cache.get.side_effect = lambda key: 1 if key == RL_API_IP_BLOCKED.format(ip=ip) else None
response = mw(_api_req(ip=ip))
assert response.status_code == 429
assert response['X-RateLimit-Reason'] == 'api_ip_blocked'
mw._incr_with_ttl.assert_not_called()
def test_api_block_response_is_json_with_retry_after():
mw, _ = _make_middleware(api_block_seconds=120)
resp = mw._api_block_response('api_threshold_exceeded')
assert resp.status_code == 429
assert 'application/json' in resp['Content-Type']
assert resp['Retry-After'] == '120'
assert resp['X-RateLimit-Reason'] == 'api_threshold_exceeded'
def test_non_api_path_uses_global_evaluate_not_api_handler():
mw, _ = _make_middleware()
mw._handle_api = MagicMock()
mw._evaluate = MagicMock(return_value={'action': 'pass', 'ip': '1.2.3.4'})
mw(_anon_req(path='/'))
mw._handle_api.assert_not_called()
mw._evaluate.assert_called_once()

9
sapl/settings.py
View File

@ -151,7 +151,6 @@ MIDDLEWARE = [
'django.contrib.messages.middleware.MessageMiddleware', 'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware', 'django.middleware.security.SecurityMiddleware',
'sapl.middleware.api_emergency_block.ApiEmergencySameSiteOnlyMiddleware', # TODO: REMOVE AFTER RL WORKS!
'whitenoise.middleware.WhiteNoiseMiddleware', 'whitenoise.middleware.WhiteNoiseMiddleware',
'waffle.middleware.WaffleMiddleware', 'waffle.middleware.WaffleMiddleware',
'sapl.middleware.check_password.CheckWeakPasswordMiddleware', 'sapl.middleware.check_password.CheckWeakPasswordMiddleware',
@ -448,6 +447,14 @@ API_QUOTA_ANON_WEEKLY = config('API_QUOTA_ANON_WEEKLY', default=3500, cast=int)
API_QUOTA_AUTH_DAILY = config('API_QUOTA_AUTH_DAILY', default=5000, cast=int) API_QUOTA_AUTH_DAILY = config('API_QUOTA_AUTH_DAILY', default=5000, cast=int)
API_QUOTA_AUTH_WEEKLY = config('API_QUOTA_AUTH_WEEKLY', default=35000, cast=int) API_QUOTA_AUTH_WEEKLY = config('API_QUOTA_AUTH_WEEKLY', default=35000, cast=int)
# API-specific per-minute rate limit for external (non-same-origin) anonymous calls.
# Abuse writes rl:api:ip:<ip>:blocked only — never rl:ip:<ip>:blocked.
API_RATE_LIMIT_ENABLED = config('API_RATE_LIMIT_ENABLED', default=True, cast=bool)
API_RATE_LIMIT_THRESHOLD = config('API_RATE_LIMIT_THRESHOLD', default=60, cast=int)
API_RATE_LIMIT_WINDOW_SECONDS = config('API_RATE_LIMIT_WINDOW_SECONDS', default=60, cast=int)
API_RATE_LIMIT_BLOCK_SECONDS = config('API_RATE_LIMIT_BLOCK_SECONDS', default=300, cast=int)
API_RATE_LIMIT_SAME_ORIGIN_BYPASS = config('API_RATE_LIMIT_SAME_ORIGIN_BYPASS', default=True, cast=bool)
# Media file serving — serve_media (sapl/base/media.py) via X-Accel-Redirect. # Media file serving — serve_media (sapl/base/media.py) via X-Accel-Redirect.
# TTL for both URL-path and storage-path access counters (DB 1). # TTL for both URL-path and storage-path access counters (DB 1).
MEDIA_PATH_COUNTER_TTL = config('MEDIA_PATH_COUNTER_TTL', default=60, cast=int) MEDIA_PATH_COUNTER_TTL = config('MEDIA_PATH_COUNTER_TTL', default=60, cast=int)

Write Preview
Loading…
Cancel
Save