Glob 7 uses the `inflight` module, which leaks memory. Also, all other Medusa packages are using glob 10+. So upgraded the one used by the framework too.
Fixes: FRMW-2972
Fixes: FRMW-2960
This PR adds support for processing large CSV files by breaking them into chunks and processing one chunk at a time. This is how it works in nutshell.
- The CSV file is read as a stream and each chunk of the stream is one CSV row.
- We read upto 1000 rows (plus a few more to ensure product variants of a product are not split into multiple chunks).
- Each chunk is then normalized using the `CSVNormalizer` and validated using zod schemas. If there is an error, the entire process will be aborted and the existing chunks will be deleted.
- Each chunk is written to a JSON file, so that we can process them later (after user confirms) without re-processing or validating the CSV file.
- The confirmation process will start consuming one chunk at a time and create/update products using the `batchProducts` workflow.
## Resume or not to resume processing of chunks
Let's imagine during processing of chunks, we find that chunk 3 leads to a database error. However, till this time we have processed the first two chunks already. How do we deal with this situation? Options are:
- We store at which chunk we failed and then during the re-upload we ignore chunks before the failed one. In my conversation with @olivermrbl we discovered that resuming will have to work with certain assumptions if we decide to implement it.
- What if a user updates the CSV rows which are part of the already processed chunks? These changes will be ignored and they will never notice it.
- Resuming works if the file name is still the same. What if they made changes and saved the file with "Save as - New name". In that case we will anyways process the entire file.
- We will have to fetch the old workflow from the workflow engine using some `ilike` search, so that we can see at which chunk the last run failed for the given file.
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
Fixes: FRMW-2974
Currently during the product imports, we create multiple chunks that must be deleted after the import has finished (either successfully or with an error). Deleting files one by one leads to multiple network calls and slows down everything.
The `bulkDelete` method deletes multiple files (with their fileKey) in one go
Since the runtime of the `@medusajs/analytics-posthog` relies on `posthog-node` package. It should be either installed as a dependency or a peerDependency that will be satisfied by the user project.
In this PR, I have added it as a peer dependency
Fixes: FRMW-2968
In this PR we have done two major things.
- First, we remove storing CSV contents within the workflow storage and neither store the JSON payloads to be created/updated in workflows. Earlier, they all were workflow inputs, hence were stored in the workflow
- Introduce a naive concept of chunks and process chunks one by one. The next PR making chunking a bit more robust while using streams, adding ability to resume from the failed chunk and so on.
> [!IMPORTANT]
> The new endpoint `/admin/product/imports` is not in use yet. But it will be after the next (final) PR.
## Old context in workflow storage
## New context in workflow storage
Fixes: FRMW-2965
In this PR we replace/remove the existing step to normalize a CSV file with the newly written CSV normalizer and also we validate the file contents further using a Zod schema.
I have duplicated the schema for now. But it is makes sense to re-use the schema for CSV validating and `/admin/products/batch`, then I can keep one source of truth under utils and re-export it. WDYT?
**Screenshots of some errors after validating the file strictly**
* feat: implement direct upload
* feat: add direct-upload endpoint
* refactor: implement feedback
* refactor: have a dedicated endpoint for direct uploads
* refactor: convert responses to snakecase
* refactor: rename method to createImport
* test: add tests for the presigned-urls endpoint
* fix: oas CLI to convert Windows paths to unix
* Create tidy-pigs-heal.md
* refactor: revert to isolate the issue
* refactor: add back the change with logs
* fix: remove external dependency
* feat: add createMultiple flag to enforce inApp link uniqueness
* changes
* mocks
* default
* many to many
---------
Co-authored-by: Carlos R. L. Rodrigues <rodrigolr@gmail.com>
Fixes: FRMW-2930
This PR updates the MikroORM config to use the module name as the snapshot name when generating migration files. Otherwise, MikroORM defaults to the database name and every time you update the database name, it will create a new snapshot.
Also, we migrate existing snapshot files to be same the new file name to avoid breaking changes.
Fixes: FRMW-2920
The feature is called `middleware-file-loader`, because it does more than just collecting middlewares. It also collects the bodyParser config for routes
Fixes: FRMW-2919
This PR adds a new routes loader with a single responsibility of scanning the filesystem and collecting routes. Sorting of routes, merging middleware and registering them with express are going to separate implementations.
The new `RoutesLoader` class allows overriding routes as-well (not recommended though) and this is how routes are de-duplicated.
- When two routes for the exact route pattern/matcher are discovered, the routes loader will only keep the last one.
- Routes files can also override handlers for specific HTTP methods. For example, the original route file exported handlers for `GET` and `POST`, but the overriding one only defines `GET`. In that case, we will continue using the original implementation for the `POST` handler.
- If an overriding route file exports additional configuration like `export const AUTHENTICATION=false`, then this will only impact the handlers exported from this file and not the original handlers.
Routes sorting has been already been implemented in a separate PR and you can visualize it using this URL. https://routes-visualizer.fly.dev/
In the framework package we are directly importing many uninstalled dependencies. The issue does not occur for transitive dependencies. However, the `glob` package is not a transitive dependency of any production dependency and hence it fails.
Fixes: #11044
Fixes: FRMW-2877
Fixes: FRMW-2863
Adds the `plugin:build` command that is used to compile the source code of a plugin for publishing it to a package registry. The command is similar to the `build` command, except it does not copy the `package.json` and the `lock` files to the build output
Fixes: FRMW-2865
In this PR we add support for developing a plugin in watch mode. During the file change, we re-compile the source code (incrementally), publishes the package, and updates the installations of the plugin.
We are using `yalc` under the hood and it must be installed as a dev dependency in the plugin project and the main Medusa app.
Fixes: FRMW-2858
This PR merge the modules exported by the plugins with the modules defined within the user config. As a result, all modules get loaded without changing the internals of the loader.
However, you cannot disable the module of a plugin by re-adding it to the `modules` array. That is something we should handle separately.
We've added the breaking change label because of the following fix:
We did broke the ability to completely disable modules in the past pr's, in this pr we re add the ability to disable a module and that this modules does not get loaded at all. ([here](https://github.com/medusajs/medusa/pull/10895/commits/6dd164f783d632c927e0a48c50090dcf7387ccdd))
Co-authored-by: Adrien de Peretti <25098370+adrien2p@users.noreply.github.com>
* feat: convert MikroORM entities to DML entities
* feat: wip on repository changes
* continue repositories and types rework
* fix order repository usage
* continue to update product category repository
* Add foreign key as part of the inferred DML type
* ../../core/types/src/dml/index.ts
* ../../core/types/src/dml/index.ts
* fix: relationships mapping
* handle nullable foreign keys types
* handle nullable foreign keys types
* handle nullable foreign keys types
* continue to update product category repository
* fix all product category repositories issues
* fix product category service types
* fix product module service types
* fix product module service types
* fix repository template type
* refactor: use a singleton DMLToMikroORM factory instance
Since the MikroORM MetadataStorage is global, we will also have to turn DML
to MikroORM entities conversion use a global bucket as well
* refactor: update product module to use DML in tests
* wip: tests
* WIP product linkable fixes
* continue type fixing and start test fixing
* test: fix more tests
* fix repository
* fix pivot table computaion + fix mikro orm repository
* fix many to many management and configuration
* fix many to many management and configuration
* fix many to many management and configuration
* update product tag relation configuration
* Introduce experimental dml hooks to fix some issues with categories
* more fixes
* fix product tests
* add missing id prefixes
* fix product category handle management
* test: fix more failing tests
* test: make it all green
* test: fix breaking tests
* fix: build issues
* fix: build issues
* fix: more breaking tests
* refactor: fix issues after merge
* refactor: fix issues after merge
* refactor: surpress types error
* test: fix DML failing tests
* improve many to many inference + tests
* Wip fix columns from product entity
* remove product model before create hook and manage handle validation and transformation at the service level
* test: fix breaking unit tests
* fix: product module service to not update handle on product update
* fix define link and joiner config
* test: fix joiner config test
* test: fix joiner config test
* fix joiner config primary keys
* Fix joiner config builder
* Fix joiner config builder
* test: remove only modifier from test
* refactor: remove hooks usage from product collection
* refactor: remove hooks usage from product-option
* refactor: remove hooks usage for computing category handle
* refactor: remove hooks usage from productCategory model
* refactor: remove hooks from DML
* refactor: remove cruft
* cleanup
* re add foerign key indexes
* chore: remove unused types
* refactor: cleanup
* migration and models configuration adjustments
* cleanup
* fix random ordering
* fix
* test: fix product-category tests
* test: update breaking DML tests
* test: array assertion to not care about ordering
* fix: temporarily apply id ordering for products
* fix ordering
* fix ordering remove logs
---------
Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
Initially I thought that we will have to add special checks to allow relationships referencing itself. However, it turned out that not to be the case. Instead, it had more to do with how self relationships are defined in Mikro ORM.
In case of `belongsTo` relationship we have to define the other side as well either as a `hasMany` or `hasOne`. For example:
**❌ The following code will fail, because no children are defined for the parent**
```ts
const user = model.define("user", {
id: model.number(),
username: model.text(),
parent: model.belongsTo(() => user)
})
```
**✅ Addition of children relationship will make things work**
```ts
const user = model.define("user", {
id: model.number(),
username: model.text(),
parent: model.belongsTo(() => user, { mappedBy: "children" }),
children: model.hasMany(() => user, { mappedBy: "parent" }),
})
```
We can see the similar setup here with our `ProductCategory` MikroORM entity. https://github.com/medusajs/medusa/blob/develop/packages/modules/product/src/models/product-category.ts#L87-L94
@adrien2p Correct me if I am wrong. But I have added the tests for now so that we know the behavior of self relationships
Fixes bug introduced by https://github.com/medusajs/medusa/pull/9912
By default the Node.js server listens on the `0.0.0.0` host. However, the related PR changes the host to `localhost` and hence resulted in an unwanted breaking change. This PR reverts the default value assignment and let Node.js decide the host when not explicitly specified.
Fixes: FRMW-2762
This PR fixes the error reporting to include the original error message and the error cause when logging it using the JSON formatter of Winston.
> Side note: There is still a lot of room for improvements in our logging. But that require a overhaul cleanup in multiple places and will require dedicated loggers for commands and APP and special treatment for pretty printing errors.
## Error output in JSON format
**Before**
**After**
Harminder VirkandGitHub
