Kafka Design

Async Communication — Kafka Design

1. Why Kafka¶

System Design v2 originally specified RabbitMQ with topic exchanges for async communication. That design was superseded before any service beyond auth/audit implemented messaging — the platform moved to Kafka instead, and this page is the canonical design for it. Two reasons drove the switch:

Replay. Kafka retains events on a topic for a configurable retention period instead of deleting them on consumption. A new consumer (or a consumer rebuilding state after a bug) can replay history instead of only seeing events from the moment it started listening.
Ordering per entity. Partitioning by aggregate ID (see below) guarantees all events for a given fine, payment, or vehicle are processed in the order they were published — something a classic AMQP queue does not guarantee once you have more than one consumer instance.

The tradeoff is that Kafka has no per-message routing logic (no exchange types, no routing key patterns) — a consumer subscribes to a whole topic and filters in application code if it only cares about some event types on that topic.

2. Topics & Partitioning¶

Naming convention: trafficgrid.<domain>, one topic per bounded-context domain (not one topic per event type — that would explode the topic count and break ordering across related events for the same entity).

audit.events is the one exception to the naming convention: a shared audit topic (core-common's KafkaConfig.AUDIT_EVENTS_TOPIC) for services whose changes are only audited, not consumed by any domain service.

Partition key: the domain aggregate's ID (vehicle_id, fine_id, payment_id, notification_id). Kafka guarantees ordering only within a partition, and all messages with the same key land on the same partition — so all events for one fine are always processed in the order they were published, even under concurrent producers.

Topic	Partition Key	Partitions (dev)	Description
trafficgrid.vehicles	entity id	3	Vehicle, vehicle document and citizen-vehicle (claim) events
trafficgrid.fines	fine_id	3	Fine and parking ticket events
trafficgrid.payments	payment_id	3	Payment transaction events
trafficgrid.notifications	notification_id	3	Notification dispatch commands (internal)
trafficgrid.users	user_id / role_id	3	User, role and permission management events
audit.events	entity id	3	Audit-only events from system-parameters-service

Partition counts above are for local/dev (docker-compose, single broker). Production partition counts are set from expected throughput per topic and are not fixed by this doc — see System Design § 9.2.

3. Events By Topic¶

Topic	Event Type	Publisher	Consumers	Trigger
trafficgrid.fines	fine.created	payments-service	audit-service, notifications-service	New fine issued against a vehicle
trafficgrid.fines	fine.status.updated	payments-service	audit-service, notifications-service	Fine cleared or cancelled by an admin
trafficgrid.fines	fine.settled	payments-service	audit-service, notifications-service	A successful payment marks the fine `PAID` (`SETTLE_FINE`)
trafficgrid.fines	fine.court.outcome	payments-service	audit-service, notifications-service	Court sets a penalty or dismisses a referred fine (`COURT_OUTCOME_RECORDED`)
trafficgrid.fines	fine.dispute.opened	payments-service	audit-service (notifications-service ignores it)	Citizen disputes a fine (fine → `DISPUTED`)
trafficgrid.fines	fine.dispute.withdrawn	payments-service	audit-service (notifications-service ignores it)	Citizen withdraws their dispute (fine → `PENDING`)
trafficgrid.fines	fine.dispute.decided	payments-service	audit-service, notifications-service	Reviewer upholds (fine → `CLEARED`) or rejects (fine → `PENDING`) a dispute
trafficgrid.payments	payment.success	payments-service	audit-service, notifications-service	Payment confirmed by provider
trafficgrid.payments	payment.failed	payments-service	audit-service, notifications-service	Payment declined or failed
trafficgrid.vehicles	vehicle.claim.verified	vehicle-service	audit-service, notifications-service	CVR verifies a citizen's claim to a vehicle (`VERIFY_CITIZEN_VEHICLE`)
trafficgrid.vehicles	vehicle.claim.rejected	vehicle-service	audit-service, notifications-service	CVR rejects a citizen's claim to a vehicle (`REJECT_CITIZEN_VEHICLE`)
trafficgrid.vehicles	vehicle.document.expiry	vehicle-service	notifications-service (planned)	Vehicle document within 30 days of expiry (planned)
trafficgrid.users	user.created	auth-service	audit-service	User account created
trafficgrid.users	user.deactivated	auth-service	audit-service	User account deactivated
trafficgrid.users	role.created	auth-service	audit-service	Role created
trafficgrid.users	role.updated	auth-service	audit-service	Role updated
trafficgrid.users	role.deleted	auth-service	audit-service	Role deleted
trafficgrid.users	role.permissions.assigned	auth-service	audit-service	Permissions assigned to a role

audit-service consumes the domain topics (trafficgrid.users, trafficgrid.fines, trafficgrid.payments, trafficgrid.vehicles) and the shared audit.events topic in its own consumer group. Login, logout and access-token refresh publish no events. fines-service was retired; payments-service owns the fines domain and publishes the trafficgrid.fines events.

As implemented¶

Every event on these topics is an audit-event message (see §6) whose action names what happened. The message key is the affected entity's id.

Topic	Publisher	Actions
trafficgrid.users	auth-service	User, role and permission actions, e.g. `CREATE_USER`, `UPDATE_USER_STATUS`, `CREATE_ROLE`, `ASSIGN_ROLE_PERMISSIONS`
trafficgrid.fines	payments-service	`CREATE_FINE`, `UPDATE_FINE_STATUS`, `SETTLE_FINE`, `COURT_OUTCOME_RECORDED`, and for disputes `OPEN_FINE_DISPUTE` (`fine.dispute.opened`), `WITHDRAW_FINE_DISPUTE` (`fine.dispute.withdrawn`), `DECIDE_FINE_DISPUTE` (`fine.dispute.decided`)
trafficgrid.payments	payments-service	`SUCCEED_PAYMENT`, `FAIL_PAYMENT` — payments-service's own `PaymentEventListener` also consumes `SUCCEED_PAYMENT` to settle the covered fines to `PAID`
audit.events	system-parameters-service	`CREATE_`, `UPDATE_`, `ACTIVATE_` and `DEACTIVATE_` + `FINE_CATEGORY`, `ORGANIZATION` or `CURRENCY`; `CREATE_`, `ACTIVATE_` and `DEACTIVATE_` + `PAYMENT_METHOD` or `NOTIFICATION_CHANNEL`
trafficgrid.vehicles	vehicle-service	`CREATE_VEHICLE`, `CREATE_VEHICLE_DOCUMENT`, `UPDATE_VEHICLE_DOCUMENT`, `LINK_CITIZEN_VEHICLE` (a claim, created `PENDING_VERIFICATION`), `UNLINK_CITIZEN_VEHICLE`, `VERIFY_CITIZEN_VEHICLE` (`vehicle.claim.verified`), `REJECT_CITIZEN_VEHICLE` (`vehicle.claim.rejected`)

A dispute action publishes one message, with entityType Fine, the actor, and the fine (carrying its dispute) as beforeState/afterState (beforeState is null when a dispute is opened). The fine status change a dispute causes is inside that message: no separate UPDATE_FINE_STATUS is published. Dispute notifications are no longer deferred: notification-service tells the vehicle owner when a dispute is decided. It ignores a dispute being opened or withdrawn, which the citizen did themselves.

Vehicle events. vehicle-service publishes on its own domain topic, trafficgrid.vehicles. It no longer uses the deprecated audit.events, which now carries only system-parameters-service. Each event goes out after its change commits. A citizen-vehicle action's entityType is CitizenVehicle, its key and entityId are the link id, and its state is the link: { id, citizenId, vehicleId, numberPlate, isPrimary, verificationStatus }. The two CVR outcomes are published by vehicle-service's poller, with no user behind them (actorId is null):

VERIFY_CITIZEN_VEHICLE: the claim's verificationStatus goes from PENDING_VERIFICATION to VERIFIED
REJECT_CITIZEN_VEHICLE: it goes to REJECTED, and afterState adds CVR's reason

Fine events after commit. payments-service publishes every fine action only once the change has committed, so a fine that rolls back is never announced. createFine is deliberately not @Transactional: its vehicle, offence and organisation lookups are remote calls that must finish before anything is written, so the save commits on its own and CREATE_FINE is published straight after it. The other fine actions run in a transaction and publish from an after-commit hook.

The payer on SETTLE_FINE. SETTLE_FINE has no actor, as no user settles a fine. Its afterState is the fine plus two top-level fields: paymentId, the settling payment, and paidBy, that payment's payer id (null for walk-in cash, which has no account). A consumer can therefore tell an owner who paid their own fine from one whose fine someone else paid.

system-parameters-service, vehicle-service and payments-service publish without blocking or failing the change: a send failure is logged.

Notifications from events¶

notification-service is the live consumer of trafficgrid.fines, trafficgrid.payments and trafficgrid.vehicles, in consumer group notifications-service (one listener per topic: FineEventListener, PaymentEventListener, VehicleEventListener). The publishers don't know notifications exist: notification-service reads each event's action and afterState and decides who to tell. It creates one notification per event per recipient (see Data Model.md §2.11) and pushes it (see Failure Handling.md).

Topic	Action	Condition	Notification type	Recipient
trafficgrid.vehicles	`VERIFY_CITIZEN_VEHICLE`	none	`VEHICLE_VERIFIED`	the claiming citizen (`afterState.citizenId`)
trafficgrid.vehicles	`REJECT_CITIZEN_VEHICLE`	none	`VEHICLE_VERIFICATION_FAILED` (with CVR's `reason`)	the claiming citizen
trafficgrid.fines	`CREATE_FINE`	fine status `PENDING` or `REFERRED_TO_COURT`	`FINE_ISSUED` ("Fine referred to court" wording when referred)	vehicle owner
trafficgrid.fines	`SETTLE_FINE`	the owner is not the payer (`afterState.paidBy`; walk-in cash has none, so the owner is told)	`FINE_PAID`	vehicle owner
trafficgrid.fines	`DECIDE_FINE_DISPUTE`	dispute `UPHELD` or `REJECTED`	`FINE_DISPUTE_DECIDED` (amount and due date when rejected, plus the decision note)	vehicle owner
trafficgrid.fines	`COURT_OUTCOME_RECORDED`	fine `PENDING` (penalty set) or `CANCELLED` (dismissed)	`FINE_COURT_OUTCOME`	vehicle owner
trafficgrid.fines	`UPDATE_FINE_STATUS`	to `CLEARED` or `CANCELLED`	`FINE_STATUS_CHANGED` (with the reason)	vehicle owner
trafficgrid.payments	`SUCCEED_PAYMENT`	`afterState.paidBy.id` present	`PAYMENT_SUCCESS` (lists the covered fines' references)	the payer
trafficgrid.payments	`FAIL_PAYMENT`	`afterState.paidBy.id` present	`PAYMENT_FAILED` (with the provider's `details.provider.statusMessage`)	the payer

Every other action is ignored, including OPEN_FINE_DISPUTE, WITHDRAW_FINE_DISPUTE, LINK_CITIZEN_VEHICLE, UNLINK_CITIZEN_VEHICLE and CREATE_VEHICLE: nobody is notified about something they just did themselves.
Vehicle owner is the citizen holding the fine's vehicle's VERIFIED link, looked up when the event is processed through vehicle-service's internal GET /internal/v1/vehicles/{id}/owner (called directly, not through the gateway, with the shared X-Service-Key; see API Contracts.md §6.2). The lookup happens only for an action that calls for a notification. A vehicle with no verified owner notifies nobody; the skip is logged at info level. Because the owner is resolved at processing time, a fine processed after its vehicle changed hands notifies the new owner.
A payment with no payer account (walk-in cash) notifies nobody. Because SETTLE_FINE skips an owner who paid themselves, a citizen who pays their own fine gets only PAYMENT_SUCCESS, not a separate FINE_PAID.
Each notification's payload carries the deep-link target and display data (API Contracts.md §6.9): fineId, referenceNumber, numberPlate (plus amount, currency, dueDate where the text names them) for fines; paymentId, referenceNumber, fineIds for payments; vehicleId, linkId, numberPlate for vehicles.

All three topics carry the same audit envelope, so one default target type serves every listener; messages are read by shape with no type headers, as audit-service does.

4. Consumer Groups¶

Each consumer service — not each instance — has its own consumer group (payments-service, notifications-service, audit-service). This is what makes Kafka's pub/sub model work for this platform:

Every consumer group gets its own full copy of every event on the topics it subscribes to. notifications-service and audit-service can both consume trafficgrid.fines independently — one lagging or reprocessing does not affect the other.
Within one consumer group, partitions are divided across running instances. If notifications-service scales to 3 instances and trafficgrid.fines has 3 partitions, each instance owns one partition and no two instances ever process the same message. Scaling past the partition count leaves instances idle — partition count is the ceiling on a topic's consumer parallelism.
Adding a new consumer for an existing topic is a config-only change: register a new consumer group and subscribe. No change to the publisher, and no re-delivery risk to existing consumers.

5. Delivery Semantics & Idempotency¶

Kafka in this design is configured for at-least-once delivery: producers use acks=all with enable.idempotence=true (prevents duplicate writes from producer-side retries), and consumers commit offsets manually, only after the message is fully processed. This means a consumer that crashes after processing a message but before committing its offset will receive that message again on restart.

Every consumer must therefore be idempotent — processing the same event twice must produce the same end state as processing it once. This is the same principle the Data Model idempotency_key column applies to payments; see Failure Handling § Retries and Idempotency for the general pattern.

As implemented in notification-service: the notifications table has a unique constraint on (event_id, user_id), where event_id is the event's eventId (§6). Each notification is inserted in its own transaction before anything is pushed. When the same event is delivered again, the insert hits the constraint, the duplicate counts as already handled, and nothing is inserted or pushed a second time. Publishing the same event twice therefore yields one notification. The key includes the recipient, so one event could notify several users, each once.

6. Event Envelope¶

Every event published to any trafficgrid.* topic uses the same envelope shape, published from core-service so all producers/consumers share one contract:

{
  "eventId": "uuid",
  "eventType": "fine.created",
  "occurredAt": "2026-09-22T10:15:00Z",
  "producer": "payments-service",
  "payload": { }
}

As implemented, services publish core-common's AuditEvent fields directly as the JSON value, plus one extra property, actorName:

{
  "eventId": "uuid",
  "actorId": "uuid",
  "actorRole": "ZRP_OFFICER",
  "actorName": "Const. J. Dube",
  "organizationId": "uuid",
  "action": "CREATE_FINE",
  "entityType": "Fine",
  "entityId": "uuid",
  "beforeState": null,
  "afterState": "{\"id\":\"...\",\"status\":\"PENDING\"}"
}

actorName is the actor's full name at the time (from the gateway's X-User-Name header, see API Contracts § 2); audit-service stores it as audit_logs.actor_name. beforeState/afterState are JSON strings of the record as the service returned it. Each producer sends its own local message class, and producers send no Java type headers: audit-service binds every message by shape into its own event class, ignoring type headers and unknown properties. A new producer must do the same.

eventId is a UUID generated when the event is published, unique per event and distinct from the Kafka message key. A consumer uses it to recognise a redelivery: notification-service de-duplicates on it (see §5). It is set on every event on trafficgrid.fines, trafficgrid.payments and trafficgrid.vehicles:

core-common's AuditEvent has an eventId, generated by its builder; the no-args constructor used when reading leaves it null for older messages.
payments-service's message builder generates one too, and its publisher fills it in for a message built any other way.
vehicle-service generates one for each event it publishes.

auth-service (trafficgrid.users) and system-parameters-service (audit.events) don't send one yet. Their only consumer, audit-service, doesn't need it.

The fuller envelope below (eventType, occurredAt, producer, payload) is the target design and is not yet implemented; of its fields, only eventId exists today.

In the target design, eventId is a producer-generated UUID, distinct from the Kafka message key — it is what a consumer uses as the idempotency check (e.g. "have I already processed this eventId"), independent of Kafka offset mechanics. payload is event-type-specific and documented alongside the relevant service's contract in API Contracts. JSON is used for Phase 1 for simplicity; Avro + Schema Registry is a future-backlog item once cross-service payload evolution needs stricter contracts (see Implementation Tracker).

7. Retry & Dead Letter Topics¶

A failed message is not retried in place — Kafka has no per-message NACK/requeue. Instead, consumers use Spring Kafka's non-blocking retry (@RetryableTopic), which republishes a failed message to a retry topic with an increasing backoff, then to a dead letter topic (DLT) once retries are exhausted:

trafficgrid.fines                  ← original topic
trafficgrid.fines-retry-0          ← 10s backoff
trafficgrid.fines-retry-1          ← 1m backoff
trafficgrid.fines-retry-2          ← 5m backoff
trafficgrid.fines-dlt              ← dead letter, after retries exhausted

This keeps the main topic's consumer lag unaffected by a single bad message — retries happen off to the side.

As implemented, notification-service (like payments-service) uses Spring Kafka's blocking DefaultErrorHandler rather than retry topics: a failed message is retried in place 3 times, 1 second apart, then published to <topic>.DLT (for example trafficgrid.fines.DLT). A message that can't be deserialized goes straight there as its original bytes. Only a failure before the notification is recorded, such as the vehicle-owner lookup being unavailable, fails a message; a push failure never does (see Failure Handling). An alert fires on any message landing in a *-dlt topic (see Failure Handling § Dead Letter Queues); a developer inspects it, fixes the root cause, and replays it back onto the original topic manually.

8. Infrastructure¶

Local/dev runs a single Kafka broker in KRaft mode (no ZooKeeper) via apache/kafka:3.7.0 in docker-compose — see System Design § 9.1. Production runs a multi-broker Kafka cluster with topic replication factor ≥ 3 and min.insync.replicas=2, so the cluster tolerates a broker loss without becoming unavailable or losing acknowledged writes — see System Design § 9.2.

Services connect via spring.kafka.bootstrap-servers, sourced from config-server like every other datastore connection property.