¶ In this lesson you will
- Open Discover on the right index and time range
- Search by application number, event and traceId
- Get a traceId from the browser
¶ Where the logs are
Every engine (application, service, payment, notification, document generation, officer…) writes its logs to Kibana. Open Kibana (staging) → Discover, choose the index pattern elastic*, and set the time range to cover the moment of the problem. Each line is shipped twice, so expect duplicate rows.
The application engine prefixes its lines with <application number> -> <EVENT> -> <service id>, so the application number is the best starting point.
Start from what you have
| When | KQL (paste in Discover) | What you get |
|---|---|---|
| Everything about an application | message:"C260930154908NNDJPX" |
All lines across engines (743 for a fully processed application) |
| Only its errors | message:"<APP_NO>" and level:ERROR |
The failure lines |
| Only the workflow engine | service:"application-mgmt-engine" and message:"<APP_NO>" |
No notification or document noise |
| One event | message:"<APP_NO> -> APPROVE" |
Every line of that event |
| The event timeline | message:"<APP_NO>" and message:"Event Execution Time" |
One line per event executed |
| What the applicant submitted | message:"<APP_NO>" and message:"Application Payload" |
The form payload the engine received |
| A payment | message:"<BILL_ID>" |
Payment and notification lines |
| Every error of a service | service:"application-mgmt-engine" and level:ERROR and message:"<SERVICE_ID>" |
All applications of the service |
| One request end to end | traceId:"<TRACE_ID>" |
Everything that one click caused (392 lines for a submit) |
The submitted payload of the demo application
- KQL query: message:"C260930154908NNDJPX" and message:"Application Payload"
- Time range: Last 2 hours
- Hit count: 18 hits
- service: application-mgmt-engine
- message: ... - Application Payload: { "changeRequestId" ...
The event timeline: PAY, ASSIGN_OFFICE, APPROVE
- KQL query: message:"C260930154908NNDJPX" and message:"Event Execution Time"
- Hit count: 8 hits
- APPROVE event (15:54:11)
- ASSIGN_OFFICE event (15:50:22)
- PAY event (15:50:20)
¶ Get a traceId from the browser
A failed click shows an error, but the details are in the logs. Open DevTools (F12 or Cmd+Opt+I) before clicking: Network only records while it is open. Filter submit-application (applicant) or process-application (officer), click the row, open Preview and copy traceId.
Network, filtered on submit-application
- Network tab
- Filter: submit-application
- Preflight request (ignore)
- submit-application xhr: click to inspect
Preview: responseCode and traceId
- Selected request: submit-application
- Preview tab
- data.applicationId: 1499b76f-...
- responseCode: APPLICATION_CREATED
- traceId: copy this into Kibana
Or paste this snippet in the Console first; every response with a traceId is printed as it happens:
Console snippet: print every traceId
(() => {
const open = XMLHttpRequest.prototype.open, send = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.open = function (m, u) { this.__req = m + ' ' + u; return open.apply(this, arguments); };
XMLHttpRequest.prototype.send = function () {
this.addEventListener('loadend', () => {
try {
const j = JSON.parse(this.responseText);
if (j && j.traceId) console.log(this.status, this.__req.split('?')[0], '\n responseCode:', j.responseCode, '\n traceId:', j.traceId);
} catch (e) { /* not JSON */ }
});
return send.apply(this, arguments);
};
console.log('traceId logger is on: now click Submit / Approve');
})();
- Paste in DevTools → Console, press Enter, then click Submit / Approve
The snippet printing the submit's traceId
- submit-application: 200, APPLICATION_CREATED
- traceId: 6abd26dec6a51fc73ac8bbbc4830f013
- SyntaxError: Unexpected identifier 'submission' (condition string)
- SyntaxError: Unexpected identifier 'approval'
- get-application-draft 404: APPLICATION_DRAFT_NOT_FOUND (harmless)
The traceId in Kibana: the whole request
- KQL query: traceId:"6abd26dec6a51fc73ac8bbbc4830f013"
- Time range: Last 30 minutes
- Hit count: 392 hits for one submit
- Application ... sent to audit log
¶ Officer actions and running events
Officer actions and running events
| When | KQL (paste in Discover) | What you get |
|---|---|---|
| Who clicked, from which state | traceId:"<TRACE_ID>" |
"Set initiating user ID…", "Current event APPROVE, current application state: …" |
| Failed user clicks | message:"PROCESSING_FAILURE_TRACKER" and message:"PROCESSING_ENDPOINT" |
Officer or applicant actions that failed |
| Role problems | message:"UNAUTHORIZED_ROLE_ON_TRANSITION" |
The event and start state refused |
| Missing state | message:"EXCEPTION_OCCURRED" and message:"getApplicationState" |
Search the method name: class names are one token |
| Failed system events | message:"PROCESSING_FAILURE_TRACKER" and message:"INTERNAL" |
ASSIGN_OFFICE, nextEvent… after their retries |
| Which breaking action failed | message:"WORKFLOW_ENGINE_ERROR" and message:"<APP_NO>" |
"Breaking action: {actionType: OFFICE_ASSIGNMENT…}" |
| A chained event | message:"next event" and message:"<APP_NO>" |
Failures of nextEvent requests |
| Notifications | message:"NOTIFICATION_ACTION" and message:"<APP_NO>" |
SMS / email actions |
| Certificate | message:"CERTIFICATE_GENERATION" and message:"<APP_NO>" |
The parameters sent to the template |
Tip · noise you can ignore
CACHE_VALUE_NOT_FOUNDfrom ServiceClientCacheManagerImpl is a harmless cache miss.APPLICATION_DRAFT_NOT_FOUNDin the portal console just means there was no draft.
¶ Check yourself
You have a 500 error on Reject and a traceId from DevTools. What is the fastest query?
- A. message:"500"
- B. level:ERROR (last 7 days)
- C. traceId:"6abd1baceb7eeb2030e42443c2391c10" and level:ERROR (your traceId)
- D. service:"officer-portal"
Show the answer
C. The traceId narrows the logs to that one click; adding level:ERROR leaves only the failure lines.
← D4 · Operational gotchas and error codes · D6 · Debug drills: four real failures →