# Data Manager

## Overview

The admin Data Manager provides two controlled upload workflows:

- **Import JSON Payments**: validates relay-box payment records, then imports them in PHP chunks with live JavaScript progress.
- **Upload Legacy User Data**: validates usable legacy user and loan rows, then imports them through `MasterLegacyImportService`.

The section is available in the admin sidebar and is protected by the existing `auth` and `admin` middleware.

## Legacy User Data

### Workflow

1. Open **Admin > Data Manager > Upload Legacy User Data**.
2. Select a `.csv` or `.txt` file.
3. Select **Validate CSV**.
4. Review the validation result and number of usable data rows.
5. Select **Import validated data** when validation passes.
6. Review the completion summary and row-level results.

### Required data

The legacy CSV must contain normalized equivalents of:

- `ID Number` or `national_id`
- `Principal`

The existing legacy import service also supports fields such as `Member`, `Address`, `Reference`, `Phone`, `Guarantor`, `Disbursed`, `Cycle`, `Payable`, `Paid`, `Outstanding`, `Due`, `Arrears`, `Age`, `Day`, and `Officer ID`.

### Ignored rows

Legacy exports may contain blank rows, totals, notes, or other incomplete rows. A row is outside the import set when it has either:

- no ID Number; or
- no positive Principal value.

These rows are ignored during the Data Manager validation phase. They do not block valid rows and are not displayed as validation errors. Rows with a usable ID and positive Principal are passed to the existing `MasterLegacyImportService`, which applies the established import rules.

### Import results

After processing, the completion page reports:

- total rows read;
- users created;
- loans created;
- historic payments applied;
- skipped rows; and
- row-level status, national ID, and message.

Legacy imports preserve the existing service behavior for tiers, loan defaults, repayment schedules, legacy payment values, officer assignments, and historical payment allocation.

## JSON Payments

### Workflow

1. Open **Admin > Data Manager > Import JSON Payments**.
2. Select a `.json` relay-box export.
3. Select **Validate and import**.
4. Monitor the live progress marker and review the production result.

### Required data

Each JSON payment record must contain these fields inside `payload`:

- User reference: `BillRefNumber`.
- Amount: `TransAmount`.
- Unique transaction ID: `TransID`.

Amounts may contain currency symbols or comma separators, for example `KSh 2,500.00`. The importer parses the numeric value before processing.

### Validation and import

The validation phase checks that:

- the file is valid JSON;
- each record has `payload.BillRefNumber`;
- each record has a positive `payload.TransAmount`; and
- each record has a `payload.TransID`.

A valid JSON upload is stored temporarily and processed by `JsonPaymentImportService` in bounded PHP chunks. The service follows the established production rules for matching, allocation, suspense, guarantee deposits, and duplicate protection.

### Production behavior

After validation, the Data Manager runs the native payment service. The established payment procedure handles:

- national ID and BillRefNumber matching;
- active-loan selection with positive repayment balances;
- oldest-first repayment allocation;
- guarantee-deposit routing for valid excess payments where applicable;
- suspense handling for unmatched or invalid transactions; and
- duplicate protection for source transaction IDs.

The admin flow accepts relay-box JSON directly. Callback-shaped CSV payloads and queue behavior remain documented for the standalone command-line importer, but are not used by this admin JSON workflow.

### Payment result states

The production result shows each transaction, including matched users, selected loans, allocations, suspense decisions, errors, and summary totals. The import is marked complete only after every JSON record has been processed.

## Progress States

The Data Manager presents the workflow in explicit stages:

1. **Upload**: the file is received and stored temporarily.
2. **Validate**: headers and row values are checked.
3. **Process**: payment records are handled in bounded PHP chunks.
4. **Apply**: each valid payment is committed through the native payment service.
5. **Complete**: summary counts and row-level results are displayed.

Invalid uploads are rejected before import, and temporary files from invalid or completed workflows are removed.

Legacy imports are processed in small PHP chunks driven by JavaScript requests. The browser requests the next chunk after the previous request completes, so no queue daemon, cron job, or operating-system process is required. This keeps the feature compatible with shared hosting.

Each request processes 10 JSON records by default. The status page displays the number of records processed and the percentage complete. JavaScript automatically requests the next chunk while the status is `running`, then displays the completed summary. HTTP or server failures are shown to the administrator and the page retries after a short delay.

Older imports that were left in the `queued` state are also resumed by the same chunk-processing flow.

### Import status lifecycle

The status record is stored temporarily in the configured local filesystem under `storage/app/private/data-manager`:

1. `running` is created after the JSON file passes validation.
2. JavaScript calls the chunk endpoint and processes the next 10 payment records.
3. PHP stores the cumulative `rows_processed`, `row_count`, `percentage`, results, and statistics after each chunk.
4. The browser refreshes the status page and continues until all rows are complete.
5. The final state becomes `completed` with `percentage` set to `100`.
6. A processing exception becomes `failed` with a visible error message.

The uploaded JSON and temporary status files are removed after completion or failure. The storage directory must be writable by the web server. The workflow uses the configured Laravel local disk, so it does not depend on a system-specific path.

### Troubleshooting

If the page remains in progress, inspect the saved status file and `storage/logs/laravel.log`. A failed HTTP chunk response is displayed in the browser and retried. Common causes are an expired upload file or insufficient write permissions for `storage/app/private/data-manager`.

## Routes

All routes are inside the admin middleware group:

| Method | Route name | Purpose |
|---|---|---|
| `GET` | `admin.data-manager.payments` | Payment upload page |
| `POST` | `admin.data-manager.payments.preview` | Validate JSON and start payment import |
| `GET` | `admin.data-manager.payments.status` | Display payment import progress and result |
| `GET` | `admin.data-manager.payments.process` | Process the next JSON payment chunk |
| `GET` | `admin.data-manager.legacy-users` | Legacy user upload page |
| `POST` | `admin.data-manager.legacy-users.validate` | Validate legacy CSV |
| `POST` | `admin.data-manager.legacy-users.import` | Import validated legacy CSV |
| `GET` | `admin.data-manager.legacy-users.status` | Display import progress and result |
| `GET` | `admin.data-manager.legacy-users.process` | Process the next legacy CSV chunk |

## Related Procedures

- Payment allocation and suspense policy: [`apply_CSV_Payments.md`](apply_CSV_Payments.md)
- Legacy CSV comparison and field mapping: [`CSV_LEGACY_IMPORT_VALIDATION_ROADMAP.md`](CSV_LEGACY_IMPORT_VALIDATION_ROADMAP.md)
- Legacy validation script: [`CSV_LEGACY_LOAN_IMPORT_VALIDATION_SCRIPT.md`](CSV_LEGACY_LOAN_IMPORT_VALIDATION_SCRIPT.md)
- Legacy loan accounting: [`LEGACY_IMPORT_LOAN_FLAG_UPDATE.md`](LEGACY_IMPORT_LOAN_FLAG_UPDATE.md)
