Commit Graph

123 Commits

Author SHA1 Message Date
Adrien de Peretti
1347698876 feat(): improve module typings in medusa config (#14478)
* feat(events): Allow priority configuration

* feat(events): Allow priority configuration

* wip

* wip

* wip

* fix typings

* Create cold-lamps-search.md

* implement priority config usage

* comment out #1

* fixes

* fixes
2026-01-14 18:39:57 +01:00
Carlos R. L. Rodrigues
8426fca710 chore(medusa): middleware policies (#14521) 2026-01-13 16:46:53 -03:00
Shahed Nasser
1ca3516a5c feat(deps,framework): add zod as framework dependency (#14441)
## Summary

**What** — What changes are introduced in this PR?

Export Zod as a dependency of `@medusajs/framework`.

Closes DX-2414

**Why** — Why are these changes relevant or necessary?  

Zod is an essential part of Medusa development. We use it in the core and developers use it in their customizations.

Developers using pnpm won't have access to Zod, as it's not a top-level dependency. While they can install any version, since Zod is an essential aspect of our framework, it's more convenient that we export it and make it accessible to developers.

**How** — How have these changes been implemented?

1. Add Zod as a dependency in `@medusajs/deps` and export it in `@medusajs/framework`
2. Change imports of Zod across projects to import from `@medusajs/framework` and remove the Zod dependency.

> Note: this change doesn't cover admin extensions (and our related packages), as they're not related to the Medusa framework and using Zod in them isn't part of the conventions we document.

Developers can import Zod like this now:

```ts
import { z } from "@medusajs/framework/zod"
```

**Testing** — How have these changes been tested, or how can the reviewer test the feature?

Use the following import in a Medusa project to create an validate zod schemas:

```bash
import { MedusaRequest, MedusaResponse } from "@medusajs/framework/http";
import { z } from "@medusajs/framework/zod"

export const PostCustomSchema = z.object({
  name: z.string(),
})

type PostCustomSchema = z.infer<typeof PostCustomSchema>

export async function POST(
  req: MedusaRequest<PostCustomSchema>,
  res: MedusaResponse
) {
  res.json({
    message: `Hello, ${req.validatedBody.name}`
  })
}

// in middleware
import { defineMiddlewares, validateAndTransformBody } from "@medusajs/framework/http"
import { PostCustomSchema } from "./admin/custom/route"

export default defineMiddlewares({
  routes: [
    {
      matcher: "/custom",
      middlewares: [validateAndTransformBody(PostCustomSchema)],
    },
  ],
})
```

---

## Examples

-

---

## Checklist

Please ensure the following before requesting a review:

- [x] I have added a **changeset** for this PR
    - Every non-breaking change should be marked as a **patch**
    - To add a changeset, run `yarn changeset` and follow the prompts
- [ ] The changes are covered by relevant **tests**
- [x] I have verified the code works as intended locally
- [x] I have linked the related issue(s) if applicable

---

## Additional Context

-
2026-01-09 13:20:01 +00:00
Adrien de Peretti
233ec261be fix: Add schema only flag on Medusa app loader (#14502)
* fix(build): Introduce a schema only flag for more heavy light weight loading

* fix(build): cleanup and tests

* Create shy-snails-raise.md

* fix(build): cleanup
2026-01-09 12:21:28 +01:00
Carlos R. L. Rodrigues
b2245cc672 chore(rbac): user link and utils (#14320) 2026-01-07 10:40:15 -03:00
Carlos R. L. Rodrigues
1bfde8dc57 feat(rbac): role-based access control module (#14310) 2026-01-07 05:36:39 -03:00
Pedro Guzman
001923da2b feat: add Medusa Cloud OAuth provider (#14395)
* feat: add Medusa Cloud OAuth provider

* add Cloud login button

* fetch whether cloud auth is enabled through api

* allow unregistered to get session

* handle existing users

* address PR comments

* prevent double execution

* a few more fixes

* fix callback url

* fix spelling

* refresh session

* 200 instead of 201

* only allow cloud identities to create user

* fix condition
2025-12-30 17:30:10 +01:00
Carlos R. L. Rodrigues
d347e369ce fix(medusa,framework): HMR file watch (#14390) 2025-12-27 15:52:38 +01:00
Adrien de Peretti
181d5fa671 feat(): Auto generated types generation upon build (#14337)
* feat(): Auto generated types generation upon build

* Create happy-steaks-cheer.md

* test(): default types true

* improvements

* improvements

* improvements

* improvements

* improvements
2025-12-19 10:13:17 +01:00
Adrien de Peretti
c8a7122ba9 chore(): Update locale header usage (#14318)
* chore(): Update locale header usage

* Create three-kiwis-shave.md

* chore(): Update locale header usage
2025-12-16 10:39:40 +01:00
Adrien de Peretti
7b4dda5a17 chore(): apply locale middleware to all store routes (#14305)
* chore(): apply locale middleware to all store routes

* Create pink-papayas-brush.md
2025-12-15 13:36:01 +01:00
Nicolas Gorga
8964a03fa1 chore(): Remove default_locale from StoreLocale (#14300)
## Summary

**What** — What changes are introduced in this PR?

Remove `default_locale` from Store <> Locale relation

**Why** — Why are these changes relevant or necessary?  

*Please provide answer here*

**How** — How have these changes been implemented?

*Please provide answer here*

**Testing** — How have these changes been tested, or how can the reviewer test the feature?

*Please provide answer here*

---

## Examples

Provide examples or code snippets that demonstrate how this feature works, or how it can be used in practice.  
This helps with documentation and ensures maintainers can quickly understand and verify the change.

```ts
// Example usage
```

---

## Checklist

Please ensure the following before requesting a review:

- [ ] I have added a **changeset** for this PR
    - Every non-breaking change should be marked as a **patch**
    - To add a changeset, run `yarn changeset` and follow the prompts
- [ ] The changes are covered by relevant **tests**
- [ ] I have verified the code works as intended locally
- [ ] I have linked the related issue(s) if applicable

---

## Additional Context

Add any additional context, related issues, or references that might help the reviewer understand this PR.


Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-12-15 09:05:46 +00:00
Adrien de Peretti
356283c359 chore(): Accept an extra agument 'all-or-nothing' on the migrate command (#14262)
* chore(): Accept an extra agument 'all-or-nothing' on the migrate command

* Create rich-camels-brush.md

* chore(): Accept an extra agument 'all-or-nothing' on the migrate command

* chore(): Accept an extra agument 'all-or-nothing' on the migrate command

* chore(): Accept an extra agument 'all-or-nothing' on the migrate command

* chore(): fix broken down migrations

* chore(): update changeset
2025-12-10 09:23:41 +01:00
Adrien de Peretti
6dc0b8bed8 feat(): Introduce translation module and preliminary application of them (#14189)
* feat(): Translation first steps

* feat(): locale middleware

* feat(): readonly links

* feat(): feature flag

* feat(): modules sdk

* feat(): translation module re export

* start adding workflows

* update typings

* update typings

* test(): Add integration tests

* test(): centralize filters preparation

* test(): centralize filters preparation

* remove unnecessary importy

* fix workflows

* Define StoreLocale inside Store Module

* Link definition to extend Store with supported_locales

* store_locale migration

* Add supported_locales handling in Store Module

* Tests

* Accept supported_locales in Store endpoints

* Add locales to js-sdk

* Include locale list and default locale in Store Detail section

* Initialize local namespace in js-sdk

* Add locales route

* Make code primary key of locale table to facilitate upserts

* Add locales routes

* Show locale code as is

* Add list translations api route

* Batch endpoint

* Types

* New batchTranslationsWorkflow and various updates to existent ones

* Edit default locale UI

* WIP

* Apply translation agnostically

* middleware

* Apply translation agnostically

* fix Apply translation agnostically

* apply translations to product list

* Add feature flag

* fetch translations by batches of 250 max

* fix apply

* improve and test util

* apply to product list

* dont manage translations if no locale

* normalize locale

* potential todo

* Protect translations routes with feature flag

* Extract normalize locale util to core/utils

* Normalize locale on write

* Normalize locale for read

* Use feature flag to guard translations UI across the board

* Avoid throwing incorrectly when locale_code not present in partial updates

* move applyTranslations util

* remove old tests

* fix util tests

* fix(): product end points

* cleanup

* update lock

* remove unused var

* cleanup

* fix apply locale

* missing new dep for test utils

* Change entity_type, entity_id to reference, reference_id

* Remove comment

* Avoid registering translations route if ff not enabled

* Prevent registering express handler for disabled route via defineFileConfig

* Add tests

* Add changeset

* Update test

* fix integration tests, module and internals

* Add locale id plus fixed

* Allow to pass array of reference_id

* fix unit tests

* fix link loading

* fix store route

* fix sales channel test

* fix tests

---------

Co-authored-by: Nicolas Gorga <nicogorga11@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-12-08 19:33:08 +01:00
Adrien de Peretti
fe49b567d6 chore: Backend HMR (expriemental) (#14074)
**What**

 This PR introduces experimental Hot Module Replacement (HMR) for the Medusa backend, enabling developers to see code changes reflected immediately without restarting the server. This significantly improves the development experience by reducing iteration time.

### Key Features

  - Hot reload support for:
    - API Routes
    - Workflows & Steps
    - Scheduled Jobs
    - Event Subscribers
    - Modules
  - IPC-based architecture: The dev server runs in a child process, communicating with the parent watcher via IPC. When HMR fails, the child process is killed and restarted, ensuring
  clean resource cleanup.
  - Recovery mechanism: Automatically recovers from broken module states without manual intervention.
  - Graceful fallback: When HMR cannot handle a change (e.g., medusa-config.ts, .env), the server restarts completely.


### Architecture
```mermaid
  flowchart TB
      subgraph Parent["develop.ts (File Watcher)"]
          W[Watch Files]
      end

      subgraph Child["start.ts (HTTP Server)"]
          R[reloadResources]
          R --> MR[ModuleReloader]
          R --> WR[WorkflowReloader]
          R --> RR[RouteReloader]
          R --> SR[SubscriberReloader]
          R --> JR[JobReloader]
      end

      W -->|"hmr-reload"| R
      R -->|"hmr-result"| W
```

### How to enable it

Backend HMR is behind a feature flag. Enable it by setting:

```ts
  // medusa-config.ts
  module.exports = defineConfig({
    featureFlags: {
      backend_hmr: true
    }
  })
```

or

```bash
export MEDUSA_FF_BACKEND_HMR=true
```

or

```
// .env
MEDUSA_FF_BACKEND_HMR=true
```

Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-12-08 08:48:36 +00:00
Adrien de Peretti
9f7846ae0b chore(): revert route loading parallelization (#14204)
**What**
To prevent undeterministic route loading, remove route loading concurrency
2025-12-03 13:54:31 +00:00
Adrien de Peretti
7e3eb6e413 chore: cleanup/improve bootstrap (#14023)
* chore: cleanup/improve bootstrap

* chore: cleanup/improve bootstrap

* Create few-guests-visit.md
2025-11-10 14:35:20 +01:00
Pedro Guzman
28c3ea68f5 chore: fix opentelemetry-resources dependency typo (#13948) 2025-11-04 08:22:41 +01:00
Adrien de Peretti
afb40d437b chore(): Cleanup and organize deps (#13940)
* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* Create lucky-poets-scream.md

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* chore(): Cleanup and organize deps

* dedupe snapshot this build

* split into 4 shard

* re configure packages integration tests

* re configure packages integration tests

* re configure packages integration tests

* re configure packages integration tests

* update scripts

* update scripts

* update scripts

* update scripts

* update scripts

* update scripts

* update scripts

* update scripts

* reduce shard for packages
2025-11-03 19:06:37 +01:00
Adrien de Peretti
37563987b8 chore(): Fix dependencies (#13932) 2025-11-02 17:46:46 +01:00
Adrien de Peretti
224ab39a81 chore(): Update dependencies usage (#13910)
* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* fix for wxios 1.6

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* push scripts

* update build concurrency

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* chore(): Update dependencies usage

* fixes

* update yarn

* fixes

* fix script

* Create heavy-suns-tickle.md

* update changeset

---------

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-10-31 16:44:14 +01:00
Carlos R. L. Rodrigues
13d7d15be5 chore(modules-sdk): parallel migrations (#13898) 2025-10-31 11:05:53 -03:00
Adrien de Peretti
85b1f3d43a feat(index): Add http/module api to interact with the index engine (#13869)
* feat(index): Add support to trigger sync manually

* feat(index): Add API route to interact with

* feat(index): Add API route to interact with

* feat(index): Add API route to interact with

* test(): Add http integration tests

* Create weak-elephants-reply.md
2025-10-28 20:31:39 +01:00
Adrien de Peretti
02b6d01382 chore(): Add instrumentation to deps (#13646)
* chore(): Add instrumentation to deps

* deps

* Create two-bikes-compare.md
2025-10-02 11:29:26 +02:00
Adrien de Peretti
b9d6f73320 Feat(): distributed caching (#13435)
RESOLVES CORE-1153

**What**
- This pr mainly lay the foundation the caching layer. It comes with a modules (built in memory cache) and a redis provider.
- Apply caching to few touch point to test

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-09-30 16:19:06 +00:00
Adrien de Peretti
12a96a7c70 chore(): Move peer deps into a single package and re export from framework (#13439)
* chore(): Move peer deps into a single package and re export from framework

* WIP

* update core packages

* update cli and deps

* update medusa

* update exports path

* remove analyze

* update modules deps

* finalise changes

* fix yarn

* fix import

* Refactor peer dependencies into a single package

Consolidate peer dependencies into one package and re-export from the framework.

* update changeset

* Update .changeset/brown-cows-sleep.md

Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>

* rm deps

* fix deps

* increase timeout

* upgrade version

* update versions

* update versions

* fixes

* update lock

* fix missing import

* fix missing import

---------

Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-09-22 18:36:22 +02:00
Adrien de Peretti
92d30b28f4 chore(): remove ssl_mode from url and also use sslmode (#13568)
* chore(): remove ssl_mode from url and also use sslmode

* improve regexp

* chore(): remove ssl_mode from url and also use sslmode

* chore(): remove ssl_mode from url and also use sslmode

* Update SSL mode configuration in changeset

Removed 'ssl_mode' from URL and replaced it with 'sslmode'.
2025-09-22 12:11:43 +02:00
Adrien de Peretti
a4b72f9a21 fix(): prepare list query for list (#13476)
* fix(): prepare list query for list

* Create two-bulldogs-scream.md
2025-09-11 11:50:33 +02:00
Carlos R. L. Rodrigues
b4c0f131b7 chore(framework,medusa): load custom flags before medusa config (#13312)
* chore(framework,medusa): load custom flags before medusa config

* test

* test runner

* changeset

* check manager featureFlags

* discover and register flags

* rm comments

* update changeset

* changeset

* use local cli

* execute from local medusa command

---------

Co-authored-by: Adrien de Peretti <adrien.deperetti@gmail.com>
2025-09-01 16:04:43 +02:00
Carlos R. L. Rodrigues
e2213448ac feat: custom logger (#13156)
* feat: custom logger

* mock log

* unit test

* FF and jobs loader

* unit test

* add to ResourceLoader

* get from container

* mock

* rm log

* default logger mock

* link loaders, express

* comments

* initialize container as first step

* db conn

* test

* initialize start

* plugin build using default logger

* ignore .medusa

* revert ignroe

---------

Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-08-28 15:31:19 +02:00
Carlos R. L. Rodrigues
e413cfefc2 feat(utils): define file config (#13283)
** What
 - Allow auto-loaded Medusa files to export a config object.
 - Currently supports isDisabled to control loading.
 - new instance `FeatureFlag` exported by `@medusajs/framework/utils`
 - `feature-flags` is now a supported folder for medusa projects, modules, providers and plugins. They will be loaded and added to `FeatureFlag`

** Why
 - Enables conditional loading of routes, migrations, jobs, subscribers, workflows, and other files based on feature flags.

```ts
// /src/feature-flags

import { FlagSettings } from "@medusajs/framework/feature-flags"

const CustomFeatureFlag: FlagSettings = {
  key: "custom_feature",
  default_val: false,
  env_key: "FF_MY_CUSTOM_FEATURE",
  description: "Enable xyz",
}

export default CustomFeatureFlag
```

```ts
// /src/modules/my-custom-module/migration/Migration20250822135845.ts

import { FeatureFlag } from "@medusajs/framework/utils"

export class Migration20250822135845 extends Migration {
  override async up(){ }
  override async down(){ }
}

defineFileConfig({
  isDisabled: () => !FeatureFlag.isFeatureEnabled("custom_feature")
})
```
2025-08-26 12:22:30 +00:00
William Bouchard
67d3660abf feat(dashboard, core-flows): associate shipping option to type (#13226)
* feat(dashboard, core-flows): allow associating shipping option type to a shipping option

* edit as well

* fix translation schema

* fix some tests

* changeset

* add new test to update shipping option type

* add new test to create shipping option with shipping option type

* pr comments

* pr comments

* rename variable

* make zod great again
2025-08-19 11:02:36 -04:00
Sebastian Rindom
12a38bcd2b feat(api): add view configuration API routes (#13177)
* feat: add view_configurations feature flag

  - Add feature flag provider and hooks to admin dashboard
  - Add backend API endpoint for feature flags
  - Create view_configurations feature flag (disabled by default)
  - Update order list table to use legacy version when flag is disabled
  - Can be enabled with MEDUSA_FF_VIEW_CONFIGURATIONS=true env var

* fix: naming

* fix: feature flags unauthenticated

* fix: add test

* feat: add settings module

* fix: deps

* fix: cleanup

* fix: add more tetsts

* fix: rm changelog

* fix: deps

* fix: add settings module to default modules list

* feat(api): add view configuration API routes

- Add CRUD endpoints for view configurations
- Add active view configuration management endpoints
- Add feature flag middleware for view config routes
- Add comprehensive integration tests
- Add HTTP types for view configuration payloads and responses
- Support system defaults and user-specific configurations
- Enable setting views as active during create/update operations

* fix: test

* fix: test

* fix: test

* fix: change view configuration path

* fix: tests

* fix: remove manual settings module config from integration tests

* fix: container typing

* fix: workflows
2025-08-15 13:17:52 +02:00
Shahed Nasser
57ad8fc755 chore: use since instead of version tag in TSDocs (#13126) 2025-08-01 17:52:52 +03:00
Pedro Guzman
1f2cefa68e fix: fix script migrations order 2025-07-21 19:33:24 +02:00
Pedro Guzman
c50e63aa17 fix: log client errors as info 2025-07-15 10:12:44 +02:00
Pedro Guzman
a2b065b3ae fix: only log server errors 2025-07-14 17:33:44 +02:00
Riqwan Thamir
d5a2cfbc5a chore: upgrade to latest zod v3 (#12938)
* chore: upgrade to latest zod v3

* chore: set explicit versions

* chore: remove v3 scope

---------

Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-07-14 14:23:47 +02:00
Riqwan Thamir
820a936b98 feat: add cookie options (#12720)
* feat: add cookie options

* feat: allow configuring hmr server port via the HMR_PORT env var

* support configuring HMR host and proto

* allow configuring the hmr client_port

* cleanup

* cleanup

---------

Co-authored-by: Harminder Virk <virk.officials@gmail.com>
Co-authored-by: Salvador Gironès <salvadorgirones@gmail.com>
Co-authored-by: Adrien de Peretti <adrien.deperetti@gmail.com>
2025-06-25 11:18:28 +02:00
Adrien de Peretti
d517dbd66a feat(): Add support for jwt asymetric keys (#12813)
* feat(): Add support for jwt asymetric keys

* Create early-chefs-chew.md

* fix unit tests

* Add verify options support

* feedback

* fix unit tests
2025-06-25 10:29:32 +02:00
Adrien de Peretti
a833c3c98c fix(utils): build query withDeleted remove auto detection (#12788)
**What**
Currently, filtering data providing a `deleted_at` value will automatically apply the `withDeleted` flag which in turns remove the default constraint apply to all queries `deleted_at: null`. The problem is that it does not account for the value assign to `deleted_at` leading to inconsistent behaviour depending on the value. e.g filtering with `deleted_at: { $eq: null }` where the expectation is to only filter the non deleted record will end up returning deleted record as well by applying the `withDeleted` filters.

This pr revert this auto detection if favor of the user providing `withDeleted` explicitly, as it is already supported , plus the filters.

Further more, some integration tests demonstrate how to filter deleted records (e.g product) from the api. While the api did not properly support it, this pr adds support to pass with_deleted flags to the query and being handled accordingly to our api support. Validators have been updated and product list end point benefit from it. Also, the list config type was already accepting such value which I have translated to the remote query config.

Also, since the previous pr was adjusting the product types, I ve adjusted them to match the expectation
2025-06-25 07:51:37 +00:00
juanzgc
3826bde591 fix(medusa): Query Config update Order By filter (#12781)
**What**
Fixed a bug in the prepareListQuery function where nested field ordering was not properly building the expected nested object structure. The function was returning flat objects like { "employee.first_name": "ASC" } instead of the correct nested structure { "employee": { "first_name": "ASC" } }.

**Why**
The buildOrder function is designed to create nested objects from dot-notation field paths, which is essential for proper query building in the Medusa framework. When this functionality was broken, it prevented correct ordering of related fields and caused queries to fail or return unexpected results.

**How**
- Root cause: The `prepareListQuery` function was not properly utilizing the `buildOrder` utility function to transform dot-notation field paths into nested objects
- Before: order = "employee.first_name" → { "employee.first_name": "ASC" }
- After: order = "employee.first_name" → { "employee": { "first_name": "ASC" } }
- Added comprehensive tests: Created detailed unit tests for the prepareListQuery function focusing on buildOrder functionality, covering various scenarios including:
  - Simple ascending/descending order
  - Nested field ordering (e.g., product.title)
  - Deeply nested ordering (e.g., product.variants.prices.amount)
  - Multiple nesting levels (up to 5 levels deep)
- Added integration tests: Created integration tests in `product.spec.ts` to verify the full end-to-end functionality of nested ordering with variant titles

The fix ensures that the buildOrder function properly transforms dot-notation field paths into the expected nested object structure, enabling correct query building for related field ordering throughout the Medusa framework.

Resolves SUP-1868
2025-06-24 15:45:54 +00:00
Harminder Virk
94b62c6724 fix: initiate request container before other express middleware (#12761) 2025-06-19 07:23:15 +00:00
Adrien de Peretti
9bdd5429af fix(core, medusa-test-utils): Fix medusa test runner plugin modules loading (#12753) 2025-06-17 19:43:31 +02:00
Adrien de Peretti
cbf3644eb7 chore(): Add retry strategy to database connection (#12713)
RESOLVES FRMW-2978

**What**
Add retry mechanism to database connection management to prevent failing when the server start faster than what makes the connection available
2025-06-13 07:18:49 +00:00
Harminder Virk
cf0297f74a feat: implement stream based processing of the files (#12574)
Fixes: FRMW-2960

This PR adds support for processing large CSV files by breaking them into chunks and processing one chunk at a time. This is how it works in nutshell.

- The CSV file is read as a stream and each chunk of the stream is one CSV row.
- We read upto 1000 rows (plus a few more to ensure product variants of a product are not split into multiple chunks).
- Each chunk is then normalized using the `CSVNormalizer` and validated using zod schemas. If there is an error, the entire process will be aborted and the existing chunks will be deleted.
- Each chunk is written to a JSON file, so that we can process them later (after user confirms) without re-processing or validating the CSV file.
- The confirmation process will start consuming one chunk at a time and create/update products using the `batchProducts` workflow.

## Resume or not to resume processing of chunks

Let's imagine during processing of chunks, we find that chunk 3 leads to a database error. However, till this time we have processed the first two chunks already. How do we deal with this situation? Options are:

- We store at which chunk we failed and then during the re-upload we ignore chunks before the failed one. In my conversation with @olivermrbl we discovered that resuming will have to work with certain assumptions if we decide to implement it.
   - What if a user updates the CSV rows which are part of the already processed chunks? These changes will be ignored and they will never notice it.
   - Resuming works if the file name is still the same. What if they made changes and saved the file with "Save as - New name". In that case we will anyways process the entire file.
   - We will have to fetch the old workflow from the workflow engine using some `ilike` search, so that we can see at which chunk the last run failed for the given file.

Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
2025-05-29 05:42:16 +00:00
Harminder Virk
1f5f50010a fix: expand method ALL for bodyparser config and additional validator (#12612)
Fixes: FRMW-2969
2025-05-26 10:22:26 +00:00
Adrien de Peretti
7685d66c07 fix(framework): Fix compiler to create the dist after clean-up (#12582)
* fix(framework): Fix compiler to create the dist after clean-up

* Create angry-stingrays-fetch.md
2025-05-22 15:30:39 +02:00
Stevche Radevski
b9a51e217d feat: Add an analytics module and local and posthog providers (#12505)
* feat: Add an analytics module and local and posthog providers

* fix: Add tests and wire up in missing places

* fix: Address feedback and add missing module typing

* fix: Address feedback and add missing module typing

---------

Co-authored-by: Adrien de Peretti <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-05-19 19:57:13 +02:00
Carlos R. L. Rodrigues
b868a4ef4d feat(index): add filterable fields to link definition (#11898)
* feat(index): add filterable fields to link definition

* rm comment

* break recursion

* validate read only links

* validate filterable

* gql schema array

* link parents

* isInverse

* push id when not present

* Fix ciruclar relationships and add tests to ensure proper behaviour (part 1)

* log and fallback to entity.alias

* cleanup and fixes

* cleanup and fixes

* cleanup and fixes

* fix get attributes

* gql type

* unit test

* array inference

* rm only

* package.json

* pacvkage.json

* fix link retrieval on duplicated entity type and aliases + tests

* link parents as array

* Match only parent entity

* rm comment

* remove hard coded schema

* extend types

* unit test

* test

* types

* pagination type

* type

* fix integration tests

* Improve performance of in selection

* use @@ to filter property

* escape jsonPath

* add Event Bus by default

* changeset

* rm postgres analyze

* estimate count

* new query

* parent aliases

* inner query w/ filter and sort relations

* address comments

---------

Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
2025-04-29 12:10:31 +02:00