# C2B Callback Processing - Async Queue Implementation

## Overview
This document reflects the C2B non-STK callback flow that is now implemented in LendCore. The current design uses a MySQL-backed database queue, an inline queue drain for shared-hosting environments, duplicate callback protection, suspense fallback, and automatic cleanup after successful allocation.

## Problem Statement
- M-Pesa sends C2B non-STK confirmations to relay-box at relay.sportinoti.com.
- relay-box then relays the non-STK confirmation to the LendCore endpoint at `/api/payment/c2b/nonstk/confirmation`.
- The previous pattern processed confirmations synchronously, which made the callback endpoint slow and vulnerable to timeouts under load.
- The callback needed a safer approach that would not depend on a long-running Linux or Windows background worker process.
- The system also needed a reliable fallback when a payment could not be matched immediately.
- The design should also allow queued or failed payloads to be retried later even if LendCore is temporarily offline, using the existing relay-box flow rather than introducing new scripts.

## What Was Implemented

### 1. Shared-hosting-friendly queue processing
Primary file: `app/Http/Controllers/Api/MpesaCallbackController.php`

The callback endpoint now:
- receives the relayed `POST /api/payment/c2b/nonstk/confirmation` request from relay-box
- validates the payload minimally
- writes a suspense backup immediately
- dispatches the queued job
- triggers a small inline queue drain from the same request so the callback does not depend on a separate background worker

This makes the flow practical for cPanel-style hosting where a persistent worker process may not be available.

### 2. Database queue backed by MySQL
Primary file: `config/queue.php`

The queue uses Laravel's database driver with the MySQL connection configured for the application. This keeps the solution simple and compatible with the existing LendCore stack without requiring Redis or another external service.

### 3. Queue job for payment processing
Primary file: `app/Jobs/ProcessC2bNonStkConfirmation.php`

The queued job now:
- implements `ShouldQueue`
- validates the transaction payload
- checks whether the transaction was already processed
- resolves the user from the `BillRefNumber`
- records the payment against active loans when a matching user is found
- removes the matching suspense row after a successful allocation
- stores the transaction in suspense if the user is missing or the job fails after retries

### 4. Suspense backup and cleanup
Primary file: `app/Services/SuspenseService.php`

The suspense service now:
- stores unmatched callback payloads safely
- prevents duplicate suspense rows for the same `transaction_id`
- removes the corresponding suspense row when the payment is successfully allocated

This ensures that successful transactions do not remain in the unmatched backlog indefinitely.

### 5. Duplicate and concurrency protection
Implemented in the controller and queue flow:
- duplicate callbacks are ignored when a pending job for the same `TransID` already exists
- a short-lived lock prevents two web requests from draining the same queue batch at the same time

This reduces duplicate processing and race conditions.

### 6. Logging and observability
Implemented via queue event logging and callback/job logs.

The system now logs:
- callback receipt
- queue dispatch
- queue job start and completion
- successful allocation
- suspense backup creation and cleanup
- failures and retries

These logs are visible in `storage/logs/laravel.log` and are useful for debugging production traffic.

### 7. Regression coverage
Primary file: `tests/Feature/WebQueueControllerTest.php`

A feature test now verifies that the callback endpoint queues and processes the job immediately through the web-friendly queue path.

### 8. Retry behavior when LendCore is offline
The current relay-box flow is designed to keep the payloads moving without introducing new infrastructure. The existing recovery and manual queue scripts remain the mechanism for retrying payloads later when LendCore is unavailable or the endpoint is temporarily failing.

In practice:
- payloads that are queued are retried by the existing relay-box recovery flow
- failed or stalled payloads are not discarded
- the same scripts continue to re-submit the payloads to LendCore when the service becomes reachable again
- no new worker script is introduced; the current relay-box scripts remain the operational entry point

## End-to-end sequence

1. M-Pesa sends a non-STK confirmation to relay-box at relay.sportinoti.com.
2. relay-box relays the confirmation to `/api/payment/c2b/nonstk/confirmation` on LendCore.
3. The controller validates the payload and creates a suspense backup.
4. The controller dispatches the job and triggers an inline queue drain.
5. The queued job processes the confirmation and allocates the payment when the user is found.
6. On success, the matching suspense row is removed.
7. If the user is missing or processing fails, the transaction remains in suspense for follow-up.

## Recommendations Implemented

### 1. Use a database queue for shared hosting
Implemented: the solution now uses Laravel's database queue driver with the existing MySQL database.

Why it was implemented:
- it works in a shared-hosting environment
- it avoids requiring Redis or a dedicated background worker process

### 2. Process the queue inline from the callback request
Implemented: the callback endpoint triggers a very small queue drain immediately after dispatching the job.

Why it was implemented:
- it reduces the dependency on a continuously running worker
- it still acknowledges the callback quickly while processing the first job inline

### 3. Prevent duplicate processing
Implemented: the callback controller checks for an existing pending job by `TransID` before dispatching a new one.

Why it was implemented:
- duplicate or retried callbacks no longer create duplicate business processing

### 4. Protect against concurrent queue draining
Implemented: a short-lived lock prevents overlapping web-based queue drains from processing the same batch at the same time.

Why it was implemented:
- it avoids race conditions during bursts of callback traffic

### 5. Preserve fallback transactions in suspense
Implemented: the callback writes a backup suspense row before processing, and failed or unmatched transactions stay there for later review or replay.

Why it was implemented:
- the system does not lose the payload if the matching user is not found or the job fails

### 6. Use explicit MySQL settings for queue and suspense access
Implemented: the queue and suspense logic now target the MySQL connection explicitly so the callback path and the worker path use the same database.

Why it was implemented:
- it removes ambiguity caused by environment overrides and SQLite fallback issues during runtime

## Recommendations Still Worth Considering for Production

### 1. Move to Redis or SQS later
This is still a good future upgrade if callback volume grows significantly.

Benefits:
- faster queue performance
- better scaling with many workers
- lower contention than the database queue

### 2. Use a dedicated queue name for C2B traffic
A separate queue such as `nonstk` would help isolate callback traffic from other background jobs.

### 3. Add a supervisor or service wrapper
A proper worker supervisor is still recommended for production reliability, especially when the system is handling sustained traffic.

### 4. Add dead-letter handling and replay tooling
The current suspense table is already the fallback, but a dedicated replay workflow and dead-letter handling would make recovery and operations easier. For now, the existing relay-box recovery flow remains the mechanism for replaying payloads without adding new scripts.

### 5. Add monitoring and alerting
Queue depth, failed jobs, worker availability, and suspense backlog should still be monitored.

## Monitoring and Metrics

Useful checks:
- queue size: `SELECT COUNT(*) FROM jobs WHERE queue = 'c2b';`
- unmatched suspense backlog: `SELECT COUNT(*) FROM suspense_transactions WHERE status = 'unmatched';`
- failed jobs: `SELECT COUNT(*) FROM failed_jobs;`
- recent application logs: `tail -f storage/logs/laravel.log`

## Deployment Notes

### Local verification
```bash
php artisan test --filter=WebQueueControllerTest
php artisan queue:work --queue=c2b --tries=3 --timeout=120
```

### Production guidance
- keep the callback endpoint returning `200 OK` quickly
- ensure the MySQL queue tables exist and are accessible
- monitor the suspense table for unmatched transactions
- rely on the existing relay-box recovery and manual queue scripts for replay when LendCore is offline
- consider a supervisor for long-running queue processing if traffic grows

## Current Status
The implementation is now in place and the core callback-to-payment flow is working through the queue path with backup suspense handling and duplicate protection.

## Capacity Under a Large Burst of Payloads
Assuming a large sequence of payloads hits `/api/payment/c2b/nonstk/confirmation`, the current structure is reasonably capable of absorbing the traffic in a controlled way, but it is not a high-throughput enterprise queue system by itself.

### What it handles well
- The endpoint responds quickly because it does not do full loan allocation inline.
- The callback is acknowledged fast, which helps avoid timeouts from the relay-box side.
- The queue and suspense flow allow the system to keep processing without blocking the HTTP request thread for long.
- Duplicate callbacks are reduced because pending jobs are detected by `TransID`.

### Practical limits
- The queue is backed by MySQL, so it is suitable for moderate volume and shared-hosting environments, but it will not match the throughput of Redis or SQS under very heavy load.
- The inline queue drain helps during bursts, but it is still bounded by request time, memory, and the database performance of the host.
- If the incoming rate is very high, the `jobs` table can grow temporarily and processing may lag behind the incoming callback rate.
- If the host is slow or MySQL becomes a bottleneck, the queue backlog may increase and suspense entries may accumulate.

### Expected behavior under sustained pressure
- Small to medium bursts should be handled well.
- Large bursts will usually be accepted and queued rather than failing outright.
- Processing may become delayed, but the payloads should still be preserved in the queue and/or suspense layer.
- If traffic remains high for an extended period, the system will begin to show queue backlog, slower processing, and potentially more unmatched suspense records until the backlog clears.

### Bottom line
The current architecture is a solid shared-hosting-friendly solution for reliable callback handling, but it is best viewed as a practical middle-ground implementation rather than a high-scale event-processing platform. For large or continuously growing traffic, Redis/SQS-backed workers and a dedicated supervisor process would provide better throughput and resilience.

---

**Last updated:** 2026-06-27
**Scope:** LendCore C2B non-STK async processing, shared-hosting queue flow, suspense fallback, and implemented recommendations
## Monitoring & Metrics

### Key Metrics to Track
- Queue size: `SELECT COUNT(*) FROM jobs WHERE queue='default';`
- Processing rate: `SELECT COUNT(*) FROM relay_transactions WHERE status='forwarded' AND forwarded_at > DATE_SUB(NOW(), INTERVAL 1 HOUR);`
- Error rate: `SELECT COUNT(*) FROM relay_transactions WHERE status='failed';`
- Response times: Parse application logs for endpoint response times

### Alerts to Set Up
- Queue size > 1000 (processing backlog)
- Error rate > 5% (systematic failures)
- Jobs with failed attempts = 3 (max retries exceeded)

## Future Enhancements

1. **Dead Letter Queue:** Jobs with all retries exhausted → separate DLQ table
2. **Batch Processing:** Combine multiple small confirmations into batch API call
3. **Circuit Breaker:** If LendCore unavailable > 10 retries, disable forwarding temporarily
4. **Metrics Dashboard:** Real-time queue depth, processing rate, error trends
5. **Webhook Callbacks:** Notify sender when transaction fully processed (not just queued)

---

**Implementation Date:** [INSERT DATE]
**Modified By:** System Update - C2B Async Queueing
**Status:** Ready for Production Deployment
