* chore(): Shard unit tests jobs
* pass with no tests
* try artifacts
* Revert "try artifacts"
This reverts commit 43663b46ea1fb2d3131182b733ab6e202ea83f1d.
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* test(): test dynamic max workers
* Clarify test description and improve CI
**What**
- Remove overserialization withing transaction and rely more on internal service or protected method instead which does not serialize and work with Infered entity type
- Slightly rework loop complexity
Overall, this gives a good spare of resources and time spent for serialization
* chore(): Upgrade mikro orm
* handle 'null' value for big number props
* 6.5.2
* remove only
* fix pricing module rule value
* switch select in strategy for balances
* revert to select in strategy for order module
* fix defining DML ManyToOne
* fix define relationship
* test fix
* more fixes
* change order strategy to balanced
* change order strategy to balanced
* prevent unnecessary manager fork
* revert generated www changes
* remove unnecessary changes
* Create real-cobras-deny.md
* address feedback
---------
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
RESOLVES CORE-1171
**What**
- Reduced async overhead for objects with mixed sync/async properties
- Lower memory pressure from eliminated promise allocations
- Faster primitive value processing with early returns
- Better concurrency through selective batching of truly async operations
- Event loop friendly behavior preventing artificial delays
- Reduced memory allocation from eliminated duplicate processing
- Decreased GC pressure from WeakMap-based caching instead of map
**note**
Now, `resolveValue`always treat every resoltion as sync operation unless it is not, meaning we do not create promise overhead when not necessary and only when actually treating with promises
RESOLVES CORE-1163
RESOLVES CORE-1164
**What**
### Add support for non auto retryable steps.
When marking a step with `maxRetries`, when it will fail it will be marked as temporary failure and then retry itself automatically. Thats the default behaviour, if you now add `autoRetry: false`, when the step will fail it will be marked as temporary failure but not retry automatically. you can now call the workflow engine run to resume the workflow from the failing step to be retried.
### Add support for `maxAwaitingRetries`
When setting `retyIntervalAwaiting` a step that is awaiting will be retried after the specified interval without maximun retry. Now you can set `maxAwaitingRetries` to force a maximum awaiting retry number
### Add support to manually retry an awaiting step
In some scenario, either a machine dies while a step is executing or a step is taking longer than expected, you can now call `retryStep` on the workflow engine to force a retry of the step that is supposedly stucked
RESOLVES CORE-1155
**What**
- prevent update payment collection worklow to fetch data and call steps when there is no payment collection
- prevent refresh Cart Shipping Methods Workflow to fetch data and call steps when there is no shipping methods
- update promotion step to remove extraneous module call
* chore(): only execute race execution checks for async workflows
* chore(): workflow redis publish only for async flows
* Create cyan-gorillas-poke.md
* chore(): workflow redis publish only for async flows
* fix negative check
---------
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
* fix(orchestration): Use the step definition max retries on set step failure
* Create sweet-turkeys-wait.md
* allow to force permanent failure
* update changeset
* chore(pricing): Fix excessive db queries during price sets update
* chore(pricing): Fix excessive db queries during price sets update
* finalize upsert with replace of the rules
* fix limit
* Create quiet-pumpkins-hang.md
---------
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
**What**
Currently, when cancelling async workflows, the step will get rescheduled while the current worker try to continue the execution leading to concurrency failure on compensation. This pr prevent the current worker from executing while an async step gets rescheduled
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
- Fix missing `ON DELETE CASCADE` constraint on order credit lines
- Fix `receiveReturn` miss usage
- Make all order integration tests to run and rename them all to `*.spec.ts`
- Fix package.json typo
* feat(create-medusa-app): Allow to create project with specific medusa version
* feat(create-medusa-app): Allow to create project with specific medusa version
* feat(create-medusa-app): Allow to create project with specific medusa version
* naming
* Create smart-singers-dress.md
**What**
Currently, filtering data providing a `deleted_at` value will automatically apply the `withDeleted` flag which in turns remove the default constraint apply to all queries `deleted_at: null`. The problem is that it does not account for the value assign to `deleted_at` leading to inconsistent behaviour depending on the value. e.g filtering with `deleted_at: { $eq: null }` where the expectation is to only filter the non deleted record will end up returning deleted record as well by applying the `withDeleted` filters.
This pr revert this auto detection if favor of the user providing `withDeleted` explicitly, as it is already supported , plus the filters.
Further more, some integration tests demonstrate how to filter deleted records (e.g product) from the api. While the api did not properly support it, this pr adds support to pass with_deleted flags to the query and being handled accordingly to our api support. Validators have been updated and product list end point benefit from it. Also, the list config type was already accepting such value which I have translated to the remote query config.
Also, since the previous pr was adjusting the product types, I ve adjusted them to match the expectation
RESOLVES FRMW-2978
**What**
Add retry mechanism to database connection management to prevent failing when the server start faster than what makes the connection available
FIXES SUP-1824
**What**
The medusa internal service update should always return the data in the expected shape described by the interface. The medusa service should not have to handle the reshapre
FIXES CLO-524
**What**
Add hidden stepDefinition object as part of the step argument and ensure the runAsStep handlers rely on the latest definition when config is being used on the returned step in order to ensure async configuration propagation and nested configuration
**What**
- Adjust lock duration, it is in seconds and not in ms
- Log on lock release
- Renew lock separately from the other promises
- Add more logs
- Log when lock can't be acquired. It can be expected in case two processes try to sync the same entity and in that case it can be ignored. But at least it gives some information in case it happens for another reason
- Log when the release lock failed, the lock will remain in the locking provider for 1 minute before being removed. But it wont prevent other entities to be synced
**What**
- Add missing index for query that falls in select in strategy, specifically the heaviest one with variant IN filtering
- Allow to force the strategy to be sent to the entry point module when using the graph API. It is useful with big dataset where filtering is enough without pagination and select in can offer better performances
e.g
```ts
await query.graph({
entity: 'product',
fields,
filters,
strategy: 'select-in' // <-- this will force the module to use receive this value and apply it accordingly
})
```
**What**
Currently, runAsStep keep reference of the workflow context that is being run as step, except that the step is composed for the current workflow composition and not the workflow being run as a step. Therefore, the context are currently miss matched leading to wrong configuration being used in case of async workflows.
**BUG**
This fix allow the runAsStep to use the current composition context to configure the step for the sub workflow to be run
**BUG BREAKING**
fix the step config wrongly used to wrap async step handlers. Now steps configured async through .config that returns a new step response will indeed marked itself as success without the need for background execution or calling setStepSuccess (as it was expected originally)
**FEATURE**
This pr also add support for cancelling running transaction, the transaction will be marked as being cancelled, once the current step finished, it will cancel the transaction to start compensating all previous steps including itself
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
Make sure there is no open handles left and that the shutdown function are properly called. Refactor and improve the medusa test runner. Make sure all modules instances are released and cleaned up
**NOTE:**
On a separate PR we can continue the investigation for the memory growing over time while the tests execute
* test(workflow-engine): Ensure failed subscriber does not compensate workflows
* test(workflow-engine): Ensure failed subscriber does not compensate workflows
* fix(medusa): sales_channel_id middleware manipulation leading to lost of the sc
* fix(medusa): sales_channel_id middleware manipulation leading to lost of the sc
* add unit tests
* add unit tests
* improve
* integration tests
**What**
Now that all events management are fixed in the workflows life cycle, the run as step needs to leverage the workflow engine if present (which should always be the case for async workflows) in order to ensure the continuation and the ability to mark parent step in parent workflow as success or failure
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
We found out that the pricing context from the cart always contains the entire cart, even though it is kind of wrong. The issue is that even though we improve the performances of the query, it will cost a lot to have hundreds of constraint for nothing potentially. For that reason, we cache the attributes in memory with the best possible query we can do to gather them and we renew them when we perform a calculate prices if it has been reset. That way, we ensure we don't have unnecessary checks on attributes that does not have rules.
Since we don't have the type table anymore which was doing that for us and until we have a proper caching layer it would do IMO. But the rules type table was very useful for these attributes findings
**What**
Reduce database queries when possible and use proper data structure and aggregation when possible in order to reduce performance decrease overall
**What**
I have removed the check for the context key where it was fetching all attributes available and then stripping out the one that does not exists.. On big dataset these would remove multiple hundreds of ms of query execution
**What**
Currently the util await for event infinitely, this can lead to chain crashes in the jest tests suites leading to too much noise to investigate proper issues.
We now have a default time out raced against the promise that is configurable to prevent from waiting for an excessive amount of time
* fix(workflow-engine-*): Prevent passing shared context reference
* fix(workflow-engine-*): Prevent passing shared context reference
* prevent tests from hanging
* fix event handling
* add integration tests
* use interval for scheduled in tests
* skip tests for now
* Create silent-glasses-enjoy.md
* fix cancel
* changeset
* push multiple aliases
* test multiple field alias
* increase wait time to index on test
---------
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
Co-authored-by: Carlos R. L. Rodrigues <rodrigolr@gmail.com>
**What**
- Use the resource id filtering when possible instead of relying on programmatic intersection checks over potential hundreds thousands of resources from the link when in fact it is not necessary to fetch everything to check in memory but instead check in the db
- Also fix normalizeDataForContext middlewares
**What**
First iteration to prevent events from overwhelming the systems.
- Group emitted event ids when possible instead of creating a message per id which leads to reduced amount of events to process massively in cases of import for example
- Update the index engine to process event data in batches of 100
- Update event handling by the index engine to be able to upsert by batch as well
- Fix index engine build config for intermediate listeners inferrence
**What**
Jest is patching the event emitter meaning that sometimes it can lead to flacky behaviors and block the test execution if the done callback is never reached. To prevent that from happening, the fail trap will call the done callback after a given time and warn that the test could not be concluded because of jest blocking it
**What**
Currently only cron pattern are supported by scheduled jobs, this can lead to issue. for example you set the pattern to execute every hours at minute 0 and second 0 (as it is expected to execute at exactly this constraint) but due to the moment it gets executed we our out of the second 0 then the job wont get executed until the next scheduled cron table execution.
With this pr we introduce the `interval` configuration which allows you the specify a delay between execution in ms (e.g every minute -> 60 * 1000 ms) which ensure that once a job is executed another one is scheduled for a minute later.
**Usage**
```ts
// jobs/job-1.ts
const thirtySeconds = 30 * 1000
export const config = {
name: "job-1",
schedule: {
interval: thirtySeconds
},
}
```
FIXES SUP-1101
also fixes https://github.com/medusajs/medusa/issues/11743
**What**
The store cors is currently applied after the ensure publishable API key middleware, in this pr the order is fixed and the cors will be applied first.
**What**
Currently, the workflow engine redis does not make any distinction between worker modes, when starting as server, the engine listen to the queue which contains everything and try to execute the corresponding workflow which does not exists since job workflows are not loaded in server mode. Now, a dedicated queue is created for jobs and the worker is only started if the instance is not in server mode. In order to clean up the old queue, if the old queue trigger a scheduled job then it gets removed from the queue since it will get re added to the new queue by the new worker instances
**What**
Currently, we are potentially providing an array of selector/data leading to fetching data sequentially before running on update which will fetch data again in batch and perform the update. Now we can pass the data directly which includes the id already and only perform one bulk fetch + one bulk update.
This pr also include a fix on the inventory validation, currently, only the item to update inventory is being checked, with this pr we also check the inventory for the items that needs to be created
**What**
- Prevent event release when a workflow is run as step and finish
- Use `unlink` instead of `del` when removing keys from redist to push the execution to async thread
**What**
- Fix soft deletion and restoration emitted events
- Improve soft deleted/restore algorithm
- Fix big number field handling null value during partial hydration from mikro orm
**What**
Some steps were calling the modules even when nothing was needed which for some operations would create transaction for nothing leading to extra execution time that add up very quickly on cloud network
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
- Move create product to use native create by structuring the data appropriately, it means no more `upsertWithReplace` being very poorly performant and got 20x better performances on staging
- Improvements in `upsertWithReplace` to still get performance boost for places that still relies on it. Mostly bulking the operations when possible
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
Adrien de PerettiandGitHub
