feat: process import from pre-processed chunks (#12527)

Fixes: FRMW-2968

In this PR we have done two major things.

- First, we remove storing CSV contents within the workflow storage and neither store the JSON payloads to be created/updated in workflows. Earlier, they all were workflow inputs, hence were stored in the workflow
- Introduce a naive concept of chunks and process chunks one by one. The next PR making chunking a bit more robust while using streams, adding ability to resume from the failed chunk and so on.

> [!IMPORTANT]  
> The new endpoint `/admin/product/imports` is not in use yet. But it will be after the next (final) PR.

## Old context in workflow storage

![CleanShot 2025-05-19 at 17 11 08@2x](https://github.com/user-attachments/assets/798bdcc9-a368-4c1f-afdd-2a77f5ce43e0)

## New context in workflow storage

![CleanShot 2025-05-19 at 17 15 08@2x](https://github.com/user-attachments/assets/0463d035-403f-4600-a9cd-5af24d5fee7c)
This commit is contained in:
Harminder Virk
2025-05-20 12:33:18 +00:00
committed by GitHub
parent 3e5794d657
commit fca5ad77b4
13 changed files with 868 additions and 12 deletions
+1
View File
@@ -65,6 +65,7 @@ module.exports = defineConfig({
options: {
// This is the directory where we can reliably write in CI environments
upload_dir: path.join(os.tmpdir(), "uploads"),
private_upload_dir: path.join(os.tmpdir(), "static"),
},
},
],