Commit Graph

33 Commits

Author SHA1 Message Date
Adrien de Peretti
fc4d5f0ac9 chore(): Workflow engine timers and notification improvements (#13434)
RESOLVES CORE-1177

**What**
main changes are:
- not blocking execution when notifying
- timers management
- race condition checks improvements
2025-09-08 18:19:55 +00:00
Adrien de Peretti
d7692100e7 chore(orchestration): add support for autoRetry, maxAwaitingRetries, retryStep (#13391)
RESOLVES CORE-1163
RESOLVES CORE-1164

**What**

### Add support for non auto retryable steps.

When marking a step with `maxRetries`, when it will fail it will be marked as temporary failure and then retry itself automatically. Thats the default behaviour, if you now add `autoRetry: false`, when the step will fail it will be marked as temporary failure but not retry automatically. you can now call the workflow engine run to resume the workflow from the failing step to be retried.

### Add support for `maxAwaitingRetries`

When setting `retyIntervalAwaiting` a step that is awaiting will be retried after the specified interval without maximun retry. Now you can set `maxAwaitingRetries` to force a maximum awaiting retry number

### Add support to manually retry an awaiting step

In some scenario, either a machine dies while a step is executing or a step is taking longer than expected, you can now call `retryStep` on the workflow engine to force a retry of the step that is supposedly stucked
2025-09-08 12:46:30 +00:00
Adrien de Peretti
2b89510df3 chore(): only execute race execution checks and publish message only for async workflows (#13396)
* chore(): only execute race execution checks for async workflows

* chore(): workflow redis publish only for async flows

* Create cyan-gorillas-poke.md

* chore(): workflow redis publish only for async flows

* fix negative check

---------

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-09-04 09:12:09 +02:00
Adrien de Peretti
bd206cb250 chore(): improve workflow engine storage (#13345)
* chore(workflow-engines): Improve race condition management

* cleanup

* cleanup

* chore(workflow-engines): Improve race condition management

* chore(workflow-engines): Improve race condition management

* chore(workflow-engines): heartbeat extend TTL

* Refactor chore title for workflow engine improvements

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* chore(): Improve workflow execution db interaction

* update tests

* revert idempotent

* add run_id index + await deletion

* improve saving

* comment

* remove only

---------

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-09-02 11:18:12 +02:00
Carlos R. L. Rodrigues
9412669e65 chore: idempotent cart operations (#13236)
* chore(core-flows): idempotent cart operations

* changeset

* add tests

* revert

* revert route

* promo test

* skip bugs

* fix test

* tests

* avoid workflow name conflict

* prevent nested workflow from being deleted until the top level parent finishes

* remove unused setTimeout

* update changeset

* rm comments

---------

Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
2025-08-28 15:04:00 +02:00
Adrien de Peretti
95d282e8ef fix: test utils events + workflow storage (#12834)
* feat(test-utils): Make event subscriber waiter robust and concurrent

* feat(test-utils): Make event subscriber waiter robust and concurrent

* fix workflows storage

* remove timeout

* Create gentle-teachers-doubt.md

* revert timestamp

* update changeset

* fix execution loop

* exit if no steps to await

* typo

* check next

* check next

* changeset

* skip when async steps

* wait workflow executions utils

* wait workflow executions utils

* wait workflow executions utils

* increase timeout

* break loop

---------

Co-authored-by: Carlos R. L. Rodrigues <rodrigolr@gmail.com>
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-06-30 13:34:08 +02:00
Adrien de Peretti
316a325b63 fix(workflow-engine-*): Cleanup expired executions and reduce redis storage usage (#12795) 2025-06-24 13:32:10 +02:00
Adrien de Peretti
7fdbf2a965 fix(workflows-sdk): Miss match context usage within run as step (#12449)
**What**

Currently, runAsStep keep reference of the workflow context that is being run as step, except that the step is composed for the current workflow composition and not the workflow being run as a step. Therefore, the context are currently miss matched leading to wrong configuration being used in case of async workflows.

**BUG**
This fix allow the runAsStep to use the current composition context to configure the step for the sub workflow to be run

**BUG BREAKING**
fix the step config wrongly used to wrap async step handlers. Now steps configured async through .config that returns a new step response will indeed marked itself as success without the need for background execution or calling setStepSuccess (as it was expected originally)

**FEATURE**
This pr also add support for cancelling running transaction, the transaction will be marked as being cancelled, once the current step finished, it will cancel the transaction to start compensating all previous steps including itself

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-05-14 13:28:16 +00:00
Adrien de Peretti
80007f3afd feat(workflows-*): Allow to re run non idempotent but stored workflow with the same transaction id if considered done (#12362) 2025-05-06 17:17:49 +02:00
Adrien de Peretti
28958b2e26 Chore/orchestration storage improvements (#12178)
**What**
Cleanup and improve workflow storage utility
2025-04-18 08:35:23 +00:00
Adrien de Peretti
8618e6ee38 fix(): Properly handle workflow as step now that events are fixed entirely (#12196)
**What**
Now that all events management are fixed in the workflows life cycle, the run as step needs to leverage the workflow engine if present (which should always be the case for async workflows) in order to ensure the continuation and the ability to mark parent step in parent workflow as success or failure

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-04-17 09:34:19 +00:00
Adrien de Peretti
13e159d8ad fix(workflow-engine-*): Prevent passing shared context reference (#11873)
* fix(workflow-engine-*): Prevent passing shared context reference

* fix(workflow-engine-*): Prevent passing shared context reference

* prevent tests from hanging

* fix event handling

* add integration tests

* use interval for scheduled in tests

* skip tests for now

* Create silent-glasses-enjoy.md

* fix cancel

* changeset

* push multiple aliases

* test multiple field alias

* increase wait time to index on test

---------

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
Co-authored-by: Carlos R. L. Rodrigues <rodrigolr@gmail.com>
2025-04-09 10:39:29 +02:00
Adrien de Peretti
fc652ea51e fix(workflow-engine-*): scheduled jobs interval (#11800)
**What**
Currently only cron pattern are supported by scheduled jobs, this can lead to issue. for example you set the pattern to execute every hours at minute 0 and second 0 (as it is expected to execute at exactly this constraint) but due to the moment it gets executed we our out of the second 0 then the job wont get executed until the next scheduled cron table execution.

With this pr we introduce the `interval` configuration which allows you the specify a delay between execution in ms (e.g every minute -> 60 * 1000 ms) which ensure that once a job is executed another one is scheduled for a minute later.

**Usage**
```ts
// jobs/job-1.ts
const thirtySeconds = 30 * 1000

export const config = {
  name: "job-1",
  schedule: {
    interval: thirtySeconds
  },
}
```
2025-03-13 15:05:13 +00:00
Carlos R. L. Rodrigues
6053ec3976 chore(workflow-engine-redis): remove repeatable jobs from old queue (#11822)
What:
  * Old  deployments have repeatable jobs registered in a wrong queue. When the `server` instance picks that job, the workflow doesn't exist, it calls to remove the job, which then removes the job from the new queue.
  * This PR cleans up any repeatable job from the queue that is exclusive to handle workflows.
2025-03-12 14:54:10 +00:00
Adrien de Peretti
72d2cf9207 fix(workflow-engines): race condition when retry interval is used (#11771) 2025-03-12 09:53:34 -03:00
Adrien de Peretti
cc8422d3a1 fix(workflow-engine-redis): Split the queues and respect worker mode for job executions (#11740)
**What**
Currently, the workflow engine redis does not make any distinction between worker modes, when starting as server, the engine listen to the queue which contains everything and try to execute the corresponding workflow which does not exists since job workflows are not loaded in server mode. Now, a dedicated queue is created for jobs and the worker is only started if the instance is not in server mode. In order to clean up the old queue, if the old queue trigger a scheduled job then it gets removed from the queue since it will get re added to the new queue by the new worker instances
2025-03-06 11:52:52 +00:00
Carlos R. L. Rodrigues
c8376a9f15 chore(medusa): clear workflow execution (#11200)
CLOSES: SUP-704
2025-02-03 10:47:32 +00:00
Carlos R. L. Rodrigues
90ae187e09 fix(workflows-sdk): name for when/then step (#10459) 2024-12-05 15:47:42 -03:00
Adrien de Peretti
34d57870ad chore: workflow internals improvementss (#9455) 2024-10-10 09:11:56 +02:00
Adrien de Peretti
6937d74252 chore(): Move dependencies around (#9278)
RESOLVES FRMW-2716 [1]

**What**
First stab at re organising dependencies in the modules and their usage.
2024-09-24 14:58:04 +00:00
Harminder Virk
9e711720dd chore: upgrade moduleResolution to Node16 (#9269) 2024-09-24 17:19:20 +05:30
Carlos R. L. Rodrigues
ef8dc4087e feat: run nested async workflows (#9119) 2024-09-16 13:06:45 +00:00
Stevche Radevski
9ba52021d0 fix: Log on error by default in async workflow executions (#8723) 2024-08-22 13:33:11 +02:00
Harminder Virk
19d30df624 refactor: Run migrations and links sync after loading the app (#8582) 2024-08-14 14:09:24 +05:30
Stevche Radevski
4155d0354f fix: Use a global reference to distributed transaction and scheduler (#8462)
* fix: Add storage cleanup on scheduled jobs

* fix: Use a global reference to distributed transaction and scheduler classes
2024-08-06 14:14:51 +02:00
Stevche Radevski
5c09f0813d fix: Upgrade bullmq and adjust to breaking changes in repeatable jobs (#8446) 2024-08-05 13:00:27 +02:00
Stevche Radevski
bcba5b3a6c chore: Make scheduled job tests less likely to fail (#8241) 2024-07-23 16:59:27 +02:00
Stevche Radevski
26d600b6db feat: Ensure async workflow executions have access to shared container (#8157)
* feat: Ensure async workflow executions have access to shared container

* fix: Register workflow worker on application start
2024-07-17 12:17:48 +02:00
Carlos R. L. Rodrigues
5600e58b7f chore(orchestration): idempotent (#7771) 2024-06-25 10:34:00 -03:00
Adrien de Peretti
d2a5201eeb chore: Refactor and improve abstract module service factory (#7688)
* chore: Refactor and improve abstract module service factory

* align naming

* clean up some template args and tests

* partially migrate modules

* partially migrate modules

* migrate more modules

* migrate last modules

* fix typings

* rename interface

* rename interface

* fixes

* fixes

* rm local plain tests
2024-06-13 13:12:37 +02:00
Stevche Radevski
3d72002c28 fix: Remove scheduled workflows if they no longer exist when executed (#7673) 2024-06-11 12:57:25 +02:00
Stevche Radevski
69410162f6 feat: Add support for scheduled workflows (#7651)
We still need to:
But wanted to open the PR for early feedback on the approach
2024-06-10 14:49:52 +00:00
Adrien de Peretti
4eae25e1ef chore(): Reorganize modules (#7210)
**What**
Move all modules to the modules directory
2024-05-02 15:33:34 +00:00