FIXES CLO-524
**What**
Add hidden stepDefinition object as part of the step argument and ensure the runAsStep handlers rely on the latest definition when config is being used on the returned step in order to ensure async configuration propagation and nested configuration
**What**
- Adjust lock duration, it is in seconds and not in ms
- Log on lock release
- Renew lock separately from the other promises
- Add more logs
- Log when lock can't be acquired. It can be expected in case two processes try to sync the same entity and in that case it can be ignored. But at least it gives some information in case it happens for another reason
- Log when the release lock failed, the lock will remain in the locking provider for 1 minute before being removed. But it wont prevent other entities to be synced
**What**
- Add missing index for query that falls in select in strategy, specifically the heaviest one with variant IN filtering
- Allow to force the strategy to be sent to the entry point module when using the graph API. It is useful with big dataset where filtering is enough without pagination and select in can offer better performances
e.g
```ts
await query.graph({
entity: 'product',
fields,
filters,
strategy: 'select-in' // <-- this will force the module to use receive this value and apply it accordingly
})
```
**What**
Currently, runAsStep keep reference of the workflow context that is being run as step, except that the step is composed for the current workflow composition and not the workflow being run as a step. Therefore, the context are currently miss matched leading to wrong configuration being used in case of async workflows.
**BUG**
This fix allow the runAsStep to use the current composition context to configure the step for the sub workflow to be run
**BUG BREAKING**
fix the step config wrongly used to wrap async step handlers. Now steps configured async through .config that returns a new step response will indeed marked itself as success without the need for background execution or calling setStepSuccess (as it was expected originally)
**FEATURE**
This pr also add support for cancelling running transaction, the transaction will be marked as being cancelled, once the current step finished, it will cancel the transaction to start compensating all previous steps including itself
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
Make sure there is no open handles left and that the shutdown function are properly called. Refactor and improve the medusa test runner. Make sure all modules instances are released and cleaned up
**NOTE:**
On a separate PR we can continue the investigation for the memory growing over time while the tests execute
* test(workflow-engine): Ensure failed subscriber does not compensate workflows
* test(workflow-engine): Ensure failed subscriber does not compensate workflows
* fix(medusa): sales_channel_id middleware manipulation leading to lost of the sc
* fix(medusa): sales_channel_id middleware manipulation leading to lost of the sc
* add unit tests
* add unit tests
* improve
* integration tests
**What**
Now that all events management are fixed in the workflows life cycle, the run as step needs to leverage the workflow engine if present (which should always be the case for async workflows) in order to ensure the continuation and the ability to mark parent step in parent workflow as success or failure
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
We found out that the pricing context from the cart always contains the entire cart, even though it is kind of wrong. The issue is that even though we improve the performances of the query, it will cost a lot to have hundreds of constraint for nothing potentially. For that reason, we cache the attributes in memory with the best possible query we can do to gather them and we renew them when we perform a calculate prices if it has been reset. That way, we ensure we don't have unnecessary checks on attributes that does not have rules.
Since we don't have the type table anymore which was doing that for us and until we have a proper caching layer it would do IMO. But the rules type table was very useful for these attributes findings
**What**
Reduce database queries when possible and use proper data structure and aggregation when possible in order to reduce performance decrease overall
**What**
I have removed the check for the context key where it was fetching all attributes available and then stripping out the one that does not exists.. On big dataset these would remove multiple hundreds of ms of query execution
**What**
Currently the util await for event infinitely, this can lead to chain crashes in the jest tests suites leading to too much noise to investigate proper issues.
We now have a default time out raced against the promise that is configurable to prevent from waiting for an excessive amount of time
* fix(workflow-engine-*): Prevent passing shared context reference
* fix(workflow-engine-*): Prevent passing shared context reference
* prevent tests from hanging
* fix event handling
* add integration tests
* use interval for scheduled in tests
* skip tests for now
* Create silent-glasses-enjoy.md
* fix cancel
* changeset
* push multiple aliases
* test multiple field alias
* increase wait time to index on test
---------
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
Co-authored-by: Carlos R. L. Rodrigues <rodrigolr@gmail.com>
**What**
- Use the resource id filtering when possible instead of relying on programmatic intersection checks over potential hundreds thousands of resources from the link when in fact it is not necessary to fetch everything to check in memory but instead check in the db
- Also fix normalizeDataForContext middlewares
**What**
First iteration to prevent events from overwhelming the systems.
- Group emitted event ids when possible instead of creating a message per id which leads to reduced amount of events to process massively in cases of import for example
- Update the index engine to process event data in batches of 100
- Update event handling by the index engine to be able to upsert by batch as well
- Fix index engine build config for intermediate listeners inferrence
**What**
Jest is patching the event emitter meaning that sometimes it can lead to flacky behaviors and block the test execution if the done callback is never reached. To prevent that from happening, the fail trap will call the done callback after a given time and warn that the test could not be concluded because of jest blocking it
**What**
Currently only cron pattern are supported by scheduled jobs, this can lead to issue. for example you set the pattern to execute every hours at minute 0 and second 0 (as it is expected to execute at exactly this constraint) but due to the moment it gets executed we our out of the second 0 then the job wont get executed until the next scheduled cron table execution.
With this pr we introduce the `interval` configuration which allows you the specify a delay between execution in ms (e.g every minute -> 60 * 1000 ms) which ensure that once a job is executed another one is scheduled for a minute later.
**Usage**
```ts
// jobs/job-1.ts
const thirtySeconds = 30 * 1000
export const config = {
name: "job-1",
schedule: {
interval: thirtySeconds
},
}
```
FIXES SUP-1101
also fixes https://github.com/medusajs/medusa/issues/11743
**What**
The store cors is currently applied after the ensure publishable API key middleware, in this pr the order is fixed and the cors will be applied first.
**What**
Currently, the workflow engine redis does not make any distinction between worker modes, when starting as server, the engine listen to the queue which contains everything and try to execute the corresponding workflow which does not exists since job workflows are not loaded in server mode. Now, a dedicated queue is created for jobs and the worker is only started if the instance is not in server mode. In order to clean up the old queue, if the old queue trigger a scheduled job then it gets removed from the queue since it will get re added to the new queue by the new worker instances
**What**
Currently, we are potentially providing an array of selector/data leading to fetching data sequentially before running on update which will fetch data again in batch and perform the update. Now we can pass the data directly which includes the id already and only perform one bulk fetch + one bulk update.
This pr also include a fix on the inventory validation, currently, only the item to update inventory is being checked, with this pr we also check the inventory for the items that needs to be created
**What**
- Prevent event release when a workflow is run as step and finish
- Use `unlink` instead of `del` when removing keys from redist to push the execution to async thread
**What**
- Fix soft deletion and restoration emitted events
- Improve soft deleted/restore algorithm
- Fix big number field handling null value during partial hydration from mikro orm
**What**
Some steps were calling the modules even when nothing was needed which for some operations would create transaction for nothing leading to extra execution time that add up very quickly on cloud network
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
- Move create product to use native create by structuring the data appropriately, it means no more `upsertWithReplace` being very poorly performant and got 20x better performances on staging
- Improvements in `upsertWithReplace` to still get performance boost for places that still relies on it. Mostly bulking the operations when possible
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
* fix(): Allow to query deleted records from graph API
* fix(): Allow to query deleted records from graph API
* handle empty q value
* update staled at sync
* rename integration tests file
* Create strong-houses-marry.md
* try to fix flacky tests
* fix pricing context
* update changeset
* update changeset
* fix import
* skip test for now
---------
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
**What**
Currently, the dashboard is fetching way more data than needed to display the products. This can lead to issues for complex catalog. this pr aim to reduce the data to be fetched to exactly what is needed.
<img width="2553" alt="Screenshot 2025-02-19 at 15 08 16" src="https://github.com/user-attachments/assets/b00e856c-e040-4f05-9f15-41fb8d299387" />
**What**
- Add index engine feature flag
- apply it to the `store/products` end point as well as `admin/products`
- Query builder various fixes
- search capabilities on full data of every entities. The `q` search will be applied to all involved joined table for selection/where clauses
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
* feat(framework): Improve production structured logging with more valuable information as well
* feat(framework): Improve production structured logging with more valuable information as well
* feat(framework): Improve production structured logging with more valuable information as well
* feat(framework): Improve production structured logging with more valuable information as well
* fix typo
* update trust proxy
* Create new-ligers-heal.md
* revert trust proxy
FIXES FRMW-2915
**What**
Backport metadata management. all the metadata get preserved unless a specific empty string is provided for the key which in turn would remove that key from the metadata
FIXES FRMW-2914
**What**
Currently, when the instrumentation is enabled, some instrument catches the errors in order to set the span status and info, unfortunately, these errors are not re throw leading to swallow them and return nothing in the end leading to potential breaks in flows
Closes: FRMW-2892, FRMW-2893
**What**
Wired up the building block that we merged previously in order to manage data synchronization. The flow is as follow
- On application start
- Build schema object representation from configuration
- Check configuration changes
- if new entities configured
- Data synchronizer initialize orchestrator and start sync
- for each entity
- acquire lock
- mark existing data as staled
- sync all data by batch
- marked them not staled anymore
- acknowledge each processed batch and renew lock
- update metadata with last synced cursor for entity X
- release lock
- remove all remaining staled data
- if any entities removed from last configuration
- remove the index data and relations
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
FIXES FRMW-2888
**What**
Since the migration scripts runner run as part of the migrate script, we were cleaning up the resources but we didn't take into consideration that links are loading automagically by just importing the links. The issue is that once a link have been imported, the next import will rely on the cached module and therefore when cleaning the resources, re importing those links is not enough. Instead, we are now treating the run-script as a separate command that we run during running the migration so that resources are isolated and loaded according to the need of each script themselves
FIXES FRMW-2878
**What**
Currently, the `one-to-one` unique constraints does not account for deleted record. This prevents from inserting a new record wth the same fk if another one is deleted.
**Caveat**
`hasOne` with FK option is meant to be a special case, for example a many to one - one to many without defining the other side of the relation. In that case we don't handle this behaviour and keep it as it is
PARTIALLY RESOLVES FRMW-2876
**What**
Fix wrong usage of the `subscriberId` in the event bus. It happens that the subscriber id coming from the context was not used at all. This issue lead to duplicated event subscriber with the same subscriber id, it also prevent unsubscribing from event since rand id will be assigned.
**NOTE**
This PR does not handle overide strategy for subscribers with the same id. this still needs to be discussed
FIXES FRMW-2879
**What**
The exclusion regexp was broken and instead use the ignore options of the glob sync function to properly ignore definition files
RESOLVES FRMW-2875
**What**
Allow to generate migration for plugins. Migration generation defer from project migration generation and therefore we choose to separate responsibility entirely.
The flow is fairly simple, the user run `npx medusa plugin:db:generate` and the script will scan for all available modules in the plugins, gather their models information and generate the appropriate migrations and snapshot (for later generation)
Co-authored-by: Harminder Virk <1706381+thetutlage@users.noreply.github.com>
RESOLVES FRMW-2862
**What**
This PR enable the `create-medusa-app` CLI to accept a new `--plugin` option to scaffold a plugin. This is complementary to all the plugin commands being created/adjusted separately to that pr.
Also, this pr brings a little refactoring around resource scaffolding, the idea was to contain the refactoring to a little area and not expend it to the entire create-medusa-app package to not disrupt and expand the scope for which the purpose was to introduce the plugin scaffolding capabilities
**Addition**
- medusa project will get their package.json name changed to the project name
- Remove build step from medusa project creation
**Plugin flow**
- in the plugin
- `npx create-medsa-app --plugin`
- `yarn dev`
- in the project
- `yalc add plugin-name`
- `yarn dev`
Any changes on the plugin will publish, push in the local registry which will fire the hot reload of the app and include the new changes from the plugin
FIXES SUP-560
**What**
Currently, no matter the error when looking for a provider to exists in the container we are throwing a normalized error stating that the provider does not exists in the container. The issue is that the first initialization of the provider occurs the first time we resolve it, and the error can be that the provider failed to be instanciated for any reason.
In this pr, we are explicitly checking for the error to be an awilix resolution error to throw the classic error and otherwise we provide the issue why the provider failed to be resolved.
RESOLVES FRMW-2835
In this PR, we trace HTTP requests using the route pattern and not the request URL. This allows us to aggregate all HTTP requests under a single route.
In terms of implementation, we have to self find the route for a given request, since there is no API in express to do the same and we begin tracing even before the request is handed over to Express.
Since, the route matching happens via RegExp matches, we ensure that this does not add much performance overhead. The matching takes between 0.8ms-1.5ms for various different routes
Co-authored-by: Harminder Virk <1706381+thetutlage@users.noreply.github.com>
FIXES FRMW-2852
**What**
A workflow distributed transaction expect any response and error to be serializable. When it is not the case, the distributed transaction might fail during the save checkpoint that occurs for async steps. This can lead to unexpected behaviour.
With this pr, we introduce a way to handle non serialazable object in a more sustainable manner, this means the following:
- If a workflow throw any non serialazable error (e.g AWS error that contains full IncomingMessage object that related to network communication, think of req/res) then we identify that this object is not serialzable and we clean up the object to make it serializable without loosing the main information, add a new error to the workflow to informed of this issue and can be handled by the user.
- If a response is not serializable (which should not happen at this point because it is handled before by the value resolver), in that case, we wont be able to reuse that response to continue the workflow which means that the workflow is in a non runnable state. In that case we throw a specific error stating that a non serializable context is being provided
**second what**
This pr refactor the `runAsStep` to add better support for workflow cancelation, especially async ones
**What**
- Allow to provide `foreignKeyName` option for hasOne and belongsTo relationships
- `model.hasOne(() => OtherEntity, { foreignKey: true, foreignKeyName: 'other_entity_something_id' })`
- The above will also output a generated type that takes into consideration the custom fk name 🔽
- Update types to account for defined custom foreign key name
- Fix joiner config linkable generation to account for custom linkable keys that provide a public API for their model but are not part of the list of the models included in the MedusaService
- This was supposed to be handled correctly but the implementation was not considering that custom linkable keys could reference models not part of the one provided to medusa service
- Migrate fulfillment module to DML
- Fix has one with fk behaviour and hooks (the relation should be assigned but not the fk)
- Fix has one belongsTo hooks (the relation should be assigned but not the fk)
- Fix hasOneWithFk and belongsTo non persisted fk to be selectable
- Allow to define `belongsTo` without other side definition for `ManyToOne` with no counter part defined
- Meaning that if a user defined `belongsTo` on one side andnot mapped by and no counter part on the other entity it will be considered as a `ManyToOne`
- `orphanRemoval` on `OneToOne` have been removed, this means that when assigning a new object relation to an entity, the previous one gets deconected but not deleted automatically. This prevent removing data un volountarely
**NOTE**
As per our convention here are some information to keep in mind
**HasOne <> BelongsTo**
Define `OneToOne`, The foreign key is owned by the belongs to and the relation needs to be provided to cascade if wanted
**HasMany <> BelongsTo**
Define `OneToMane` <> `ManyToOne`, the foreign key is owned by the many to one and for those relation no cascade will be performed, the foreign key must be provided. For the `HasMany` the cascade is available
**HasOne (with FK)**
Will act similarly to belongs to with **HasOne <> BelongsTo**
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
**What**
Keep the autocompletion for the fields API in Query APIs while being more flexible in order to prevent limitation from our depth limit but also to be more flexible when assigning string[] coming from the API for example.
RESOLVES FRMW-2832
RESOLVES FRMW-2833
**What**
Migrate workflow engines to DML. Alos includes and update to the linkable generation which now takes into account id and primary keys to generate the linkable instead of only primary keys
FIXES CMRC-722
**What**
- It should not be allowed to delete a default sales channel
- The admin does not allow to delete a sales channel use as the default for the store
RESOLVES FRMW-2826
**What**
We have many places where we define only one to one on one side of the relation. In that case it is a one to one that does not mapped by to the other side relation since it is not defined. This pr allow to create a one to one without mapped by. In order to remove breaking changes, in that case we ask the user to explicitly define the `mappedBy` as `undefined`
**What**
DML class name and mikro orm entity name are different, it was ok and the idea was to accept that and do the transformation when needed. But since class name are usually upper case first we decided to re integrate that convention in the DML class name as well to simplify consumption of the dml name without having to worry about the normal case
RESOLVES FRMW-2813
**What**
Extract what we have done in the order module DML migration.
This includes all the changes we discussed around default value and nullable management
From now on, when the default is being set through the DML on a property, no default initializer will be created, this means that when a mikro orm class is being instantiated, no default will be applied at instantiation time. This fixes the issue when retrieving data and not selecting fields that have a default. The default initializer was taking the relay making you think the data was present except that it was the value of the default initializer. From now on, the default value is applied as follows
- db column default
- before flush
- on persist
The same applies for nullable properties/columns
Co-authored-by: Harminder Virk <1706381+thetutlage@users.noreply.github.com>
RESOLVES SUP-185
**What**
The cart fields to be fetched for the promotion workflow was not including the sales_channel_id, making any rules being constraint to that fields not being validated against this property.
RESOLVES SUP-188
**What**
Two changes are happening here
- In the stripe payment provider, idempotent cancellation action, if not id is provided then return the existing data unchanged
- Payment module should not try to cancel a session that have failed to be created in the first place
FIXES CMRC-691
**What**
`Date` is something that get executed, since workflows are meant to compose the definition of what will be executed, the date where always having the same value as they was executed once during composition.
Instead wrap those into transformer that will be executed when needed and fix the Date issues
**What**
Currently, when setting a `belongsTo` relationship on the DML with the otherside being `hasMany` it result in a wrongly generated gql output making the belongs to being a collection of the relation instead of the relation directly. This pr fixes this issue
**What**
The values are wrongly set which can lead to infinite depth iteration
**Note**
This will change on how we manage those cases in the future but currently it fixes an issue
Adrien de PerettiandGitHub
