Fixes: FRMW-2960
This PR adds support for processing large CSV files by breaking them into chunks and processing one chunk at a time. This is how it works in nutshell.
- The CSV file is read as a stream and each chunk of the stream is one CSV row.
- We read upto 1000 rows (plus a few more to ensure product variants of a product are not split into multiple chunks).
- Each chunk is then normalized using the `CSVNormalizer` and validated using zod schemas. If there is an error, the entire process will be aborted and the existing chunks will be deleted.
- Each chunk is written to a JSON file, so that we can process them later (after user confirms) without re-processing or validating the CSV file.
- The confirmation process will start consuming one chunk at a time and create/update products using the `batchProducts` workflow.
## Resume or not to resume processing of chunks
Let's imagine during processing of chunks, we find that chunk 3 leads to a database error. However, till this time we have processed the first two chunks already. How do we deal with this situation? Options are:
- We store at which chunk we failed and then during the re-upload we ignore chunks before the failed one. In my conversation with @olivermrbl we discovered that resuming will have to work with certain assumptions if we decide to implement it.
- What if a user updates the CSV rows which are part of the already processed chunks? These changes will be ignored and they will never notice it.
- Resuming works if the file name is still the same. What if they made changes and saved the file with "Save as - New name". In that case we will anyways process the entire file.
- We will have to fetch the old workflow from the workflow engine using some `ilike` search, so that we can see at which chunk the last run failed for the given file.
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
Fixes: FRMW-2974
Currently during the product imports, we create multiple chunks that must be deleted after the import has finished (either successfully or with an error). Deleting files one by one leads to multiple network calls and slows down everything.
The `bulkDelete` method deletes multiple files (with their fileKey) in one go
The `auth.login` method of the JS SDK allows passing custom, which is useful for custom authentication providers. For example:
```ts
const response = await sdk.auth.login("customer", "phone-auth", {
phone
})
```
However, the `auth.register` method doesn't allow that, so we can't do the following:
```ts
const response = await sdk.auth.register("customer", "phone-auth", {
phone
})
```
Instead, we'd have to use the `client.fetch` method.
This PR fixes the input type of the payload passed to the `register` method to be similar to that of `login`, which would allow using it with custom authentication providers
Fixes: FRMW-2968
In this PR we have done two major things.
- First, we remove storing CSV contents within the workflow storage and neither store the JSON payloads to be created/updated in workflows. Earlier, they all were workflow inputs, hence were stored in the workflow
- Introduce a naive concept of chunks and process chunks one by one. The next PR making chunking a bit more robust while using streams, adding ability to resume from the failed chunk and so on.
> [!IMPORTANT]
> The new endpoint `/admin/product/imports` is not in use yet. But it will be after the next (final) PR.
## Old context in workflow storage
## New context in workflow storage
* feat: Add an analytics module and local and posthog providers
* fix: Add tests and wire up in missing places
* fix: Address feedback and add missing module typing
* fix: Address feedback and add missing module typing
---------
Co-authored-by: Adrien de Peretti <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
**What**
- Add missing index for query that falls in select in strategy, specifically the heaviest one with variant IN filtering
- Allow to force the strategy to be sent to the entry point module when using the graph API. It is useful with big dataset where filtering is enough without pagination and select in can offer better performances
e.g
```ts
await query.graph({
entity: 'product',
fields,
filters,
strategy: 'select-in' // <-- this will force the module to use receive this value and apply it accordingly
})
```
Fixes: FRMW-2965
In this PR we replace/remove the existing step to normalize a CSV file with the newly written CSV normalizer and also we validate the file contents further using a Zod schema.
I have duplicated the schema for now. But it is makes sense to re-use the schema for CSV validating and `/admin/products/batch`, then I can keep one source of truth under utils and re-export it. WDYT?
**Screenshots of some errors after validating the file strictly**
Change the signature of the `update` method in `IAuthProvider` to match other methods. this is necessary as in the generated references `update` was being inferred as a property rather than a method.
* add failing test for upsertWithReplace order
* reproduce prices update shuffling issue
* fix: fix order of returned updates in updateMany
* fix: fix order of returned updates in ProductService
* fix: reset test count to 1
* Create tame-insects-marry.md
---------
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
**What**
Currently, runAsStep keep reference of the workflow context that is being run as step, except that the step is composed for the current workflow composition and not the workflow being run as a step. Therefore, the context are currently miss matched leading to wrong configuration being used in case of async workflows.
**BUG**
This fix allow the runAsStep to use the current composition context to configure the step for the sub workflow to be run
**BUG BREAKING**
fix the step config wrongly used to wrap async step handlers. Now steps configured async through .config that returns a new step response will indeed marked itself as success without the need for background execution or calling setStepSuccess (as it was expected originally)
**FEATURE**
This pr also add support for cancelling running transaction, the transaction will be marked as being cancelled, once the current step finished, it will cancel the transaction to start compensating all previous steps including itself
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
* fix: Plugin admin folder loading with backslash on Windows
* fix: Plugin admin folder loading with backslash on Windows - Add changeset
---------
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
* fix(core-flows): use prduct title for line item title
* fix: module tests
* fix: http tests
* fix: display item subtitle instead of prod title as secondary text in line item
* fix: claim/exchange items
