RESOLVES FRMW-2978
**What**
Add retry mechanism to database connection management to prevent failing when the server start faster than what makes the connection available
Fixes: FRMW-2960
This PR adds support for processing large CSV files by breaking them into chunks and processing one chunk at a time. This is how it works in nutshell.
- The CSV file is read as a stream and each chunk of the stream is one CSV row.
- We read upto 1000 rows (plus a few more to ensure product variants of a product are not split into multiple chunks).
- Each chunk is then normalized using the `CSVNormalizer` and validated using zod schemas. If there is an error, the entire process will be aborted and the existing chunks will be deleted.
- Each chunk is written to a JSON file, so that we can process them later (after user confirms) without re-processing or validating the CSV file.
- The confirmation process will start consuming one chunk at a time and create/update products using the `batchProducts` workflow.
## Resume or not to resume processing of chunks
Let's imagine during processing of chunks, we find that chunk 3 leads to a database error. However, till this time we have processed the first two chunks already. How do we deal with this situation? Options are:
- We store at which chunk we failed and then during the re-upload we ignore chunks before the failed one. In my conversation with @olivermrbl we discovered that resuming will have to work with certain assumptions if we decide to implement it.
- What if a user updates the CSV rows which are part of the already processed chunks? These changes will be ignored and they will never notice it.
- Resuming works if the file name is still the same. What if they made changes and saved the file with "Save as - New name". In that case we will anyways process the entire file.
- We will have to fetch the old workflow from the workflow engine using some `ilike` search, so that we can see at which chunk the last run failed for the given file.
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
* feat: Add an analytics module and local and posthog providers
* fix: Add tests and wire up in missing places
* fix: Address feedback and add missing module typing
* fix: Address feedback and add missing module typing
---------
Co-authored-by: Adrien de Peretti <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
* feat(index): add filterable fields to link definition
* rm comment
* break recursion
* validate read only links
* validate filterable
* gql schema array
* link parents
* isInverse
* push id when not present
* Fix ciruclar relationships and add tests to ensure proper behaviour (part 1)
* log and fallback to entity.alias
* cleanup and fixes
* cleanup and fixes
* cleanup and fixes
* fix get attributes
* gql type
* unit test
* array inference
* rm only
* package.json
* pacvkage.json
* fix link retrieval on duplicated entity type and aliases + tests
* link parents as array
* Match only parent entity
* rm comment
* remove hard coded schema
* extend types
* unit test
* test
* types
* pagination type
* type
* fix integration tests
* Improve performance of in selection
* use @@ to filter property
* escape jsonPath
* add Event Bus by default
* changeset
* rm postgres analyze
* estimate count
* new query
* parent aliases
* inner query w/ filter and sort relations
* address comments
---------
Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
**What**
- Use the resource id filtering when possible instead of relying on programmatic intersection checks over potential hundreds thousands of resources from the link when in fact it is not necessary to fetch everything to check in memory but instead check in the db
- Also fix normalizeDataForContext middlewares
**What**
Currently only cron pattern are supported by scheduled jobs, this can lead to issue. for example you set the pattern to execute every hours at minute 0 and second 0 (as it is expected to execute at exactly this constraint) but due to the moment it gets executed we our out of the second 0 then the job wont get executed until the next scheduled cron table execution.
With this pr we introduce the `interval` configuration which allows you the specify a delay between execution in ms (e.g every minute -> 60 * 1000 ms) which ensure that once a job is executed another one is scheduled for a minute later.
**Usage**
```ts
// jobs/job-1.ts
const thirtySeconds = 30 * 1000
export const config = {
name: "job-1",
schedule: {
interval: thirtySeconds
},
}
```
**What**
- Reworks how admin extensions are loaded from plugins.
- Reworks how extensions are managed internally in the dashboard project.
**Why**
- Previously we loaded extensions from plugins the same way we do for extension found in a users application. This being scanning the source code for possible extensions in `.medusa/server/src/admin`, and including any extensions that were discovered in the final virtual modules.
- This was causing issues with how Vite optimizes dependencies, and would lead to CJS/ESM issues. Not sure of the exact cause of this, but the issue was pinpointed to Vite not being able to register correctly which dependencies to optimize when they were loaded through the virtual module from a plugin in `node_modules`.
**What changed**
- To circumvent the above issue we have changed to a different strategy for loading extensions from plugins. The changes are the following:
- We now build plugins slightly different, if a plugin has admin extensions we now build those to `.medusa/server/src/admin/index.mjs` and `.medusa/server/src/admin/index.js` for a ESM and CJS build.
- When determining how to load extensions from a source we follow these rules:
- If the source has a `medusa-plugin-options.json` or is the root application we determine that it is a `local` extension source, and load extensions as previously through a virtual module.
- If it has neither of the above, but has a `./admin` export in its package.json then we determine that it is a `package` extension, and we update the entry point for the dashboard to import the package and pass its extensions a long to the dashboard manager.
**Changes required by plugin authors**
- The change has no breaking changes, but requires plugin authors to update the `package.json` of their plugins to also include a `./admin` export. It should look like this:
```json
{
"name": "@medusajs/plugin",
"version": "0.0.1",
"description": "A starter for Medusa plugins.",
"author": "Medusa (https://medusajs.com)",
"license": "MIT",
"files": [
".medusa/server"
],
"exports": {
"./package.json": "./package.json",
"./workflows": "./.medusa/server/src/workflows/index.js",
"./.medusa/server/src/modules/*": "./.medusa/server/src/modules/*/index.js",
"./modules/*": "./.medusa/server/src/modules/*/index.js",
"./providers/*": "./.medusa/server/src/providers/*/index.js",
"./*": "./.medusa/server/src/*.js",
"./admin": {
"import": "./.medusa/server/src/admin/index.mjs",
"require": "./.medusa/server/src/admin/index.js",
"default": "./.medusa/server/src/admin/index.js"
}
},
}
```
FIXES SUP-1101
also fixes https://github.com/medusajs/medusa/issues/11743
**What**
The store cors is currently applied after the ensure publishable API key middleware, in this pr the order is fixed and the cors will be applied first.
Fixes: FRMW-2920
The feature is called `middleware-file-loader`, because it does more than just collecting middlewares. It also collects the bodyParser config for routes
Fixes: FRMW-2919
This PR adds a new routes loader with a single responsibility of scanning the filesystem and collecting routes. Sorting of routes, merging middleware and registering them with express are going to separate implementations.
The new `RoutesLoader` class allows overriding routes as-well (not recommended though) and this is how routes are de-duplicated.
- When two routes for the exact route pattern/matcher are discovered, the routes loader will only keep the last one.
- Routes files can also override handlers for specific HTTP methods. For example, the original route file exported handlers for `GET` and `POST`, but the overriding one only defines `GET`. In that case, we will continue using the original implementation for the `POST` handler.
- If an overriding route file exports additional configuration like `export const AUTHENTICATION=false`, then this will only impact the handlers exported from this file and not the original handlers.
Routes sorting has been already been implemented in a separate PR and you can visualize it using this URL. https://routes-visualizer.fly.dev/
The `storefrontUrl` is defined in the `packages/core/types/src/common/config-module.ts` file, but for many purposes, including the documentation, we use the types defined in `packages/core/framework/src/config/types.ts`. This PR adds the `storefrontUrl` to the framework type
* feat(framework): Improve production structured logging with more valuable information as well
* feat(framework): Improve production structured logging with more valuable information as well
* feat(framework): Improve production structured logging with more valuable information as well
* feat(framework): Improve production structured logging with more valuable information as well
* fix typo
* update trust proxy
* Create new-ligers-heal.md
* revert trust proxy
**What**
- Bumps the versions of Vite across the entire stack, to prevent an issue similar to what is described here: https://github.com/vitejs/vite/discussions/18271
Not entirely sure what was happening as I couldn't reproduce the issue, but Adrien faced the issue yesterday when working with local versions of our packages. It does appear as if the range we had before could lead to a version of Vite to be installed with said bug.
* ../../core/types/src/dml/index.ts
* ../../core/types/src/dml/index.ts
* fix: relationships mapping
* handle nullable foreign keys types
* handle nullable foreign keys types
* handle nullable foreign keys types
* continue to update product category repository
* fix all product category repositories issues
* fix product category service types
* fix product module service types
* fix product module service types
* fix repository template type
* refactor: use a singleton DMLToMikroORM factory instance
Since the MikroORM MetadataStorage is global, we will also have to turn DML
to MikroORM entities conversion use a global bucket as well
* refactor: update product module to use DML in tests
* wip: tests
* WIP product linkable fixes
* continue type fixing and start test fixing
* test: fix more tests
* fix repository
* fix pivot table computaion + fix mikro orm repository
* fix many to many management and configuration
* fix many to many management and configuration
* fix many to many management and configuration
* update product tag relation configuration
* Introduce experimental dml hooks to fix some issues with categories
* more fixes
* fix product tests
* add missing id prefixes
* fix product category handle management
* test: fix more failing tests
* test: make it all green
* test: fix breaking tests
* fix: build issues
* fix: build issues
* fix: more breaking tests
* refactor: fix issues after merge
* refactor: fix issues after merge
* refactor: surpress types error
* test: fix DML failing tests
* improve many to many inference + tests
* Wip fix columns from product entity
* remove product model before create hook and manage handle validation and transformation at the service level
* test: fix breaking unit tests
* fix: product module service to not update handle on product update
* fix define link and joiner config
* test: fix joiner config test
* test: fix joiner config test
* fix joiner config primary keys
* Fix joiner config builder
* Fix joiner config builder
* test: remove only modifier from test
* refactor: remove hooks usage from product collection
* refactor: remove hooks usage from product-option
* refactor: remove hooks usage for computing category handle
* refactor: remove hooks usage from productCategory model
* refactor: remove hooks from DML
* refactor: remove cruft
* order dml
* cleanup
* re add foerign key indexes
* wip
* chore: remove unused types
* wip
* changes
* rm raw
* autoincrement
* wip
* rel
* refactor: cleanup
* migration and models configuration adjustments
* cleanup
* number searchable
* fix random ordering
* fix
* test: fix product-category tests
* test: update breaking DML tests
* test: array assertion to not care about ordering
* fix: temporarily apply id ordering for products
* types
* wip
* WIP type improvements
* update order models
* partially fix types temporarely
* rel
* fix: recursive type issue
* improve type inference breaks
* improve type inference breaks
* update models
* rm nullable
* default value
* repository
* update default value handling
* fix unit tests
* WIP
* toMikroORM
* fix relations
* cascades
* fix
* experimental dml hooks
* rm migration
* serial
* nullable autoincrement
* fix model
* model changes
* fix one to one DML
* order test
* fix addresses
* fix unit tests
* Re align dml entity name inference
* update model table name config
* update model table name config
* revert
* update return relation
* WIP
* hasOne
* models
* fix
* model
* initial commit
* cart service
* order module
* utils unit test
* index engine
* changeset
* merge
* fix hasOne with fk
* update
* free text filter per entity
* tests
* prod category
* property string many to many
* fix big number
* link modules migration set names
* merge
* shipping option rules
* serializer
* unit test
* fix test mikro orm init
* fix test mikro orm init
* Maintain merge object properties
* fix test mikro orm init
* prevent unit test from connecting to db
* wip
* fix test
* fix test
* link test
* schema
* models
* auto increment
* hook
* model hooks
* order
* wip
* orm version
* request return field
* fix return configuration on order model
* workflows
* core flows
* unit test
* test
* base repo
* test
* base repo
* test fix
* inventory move
* locking inventory
* test
* free text fix
* rm timeout mock
* migrate fulfillment values
* v6.4.3
* cleanup
* link-modules update sql
* revert test
* remove fake timers
---------
Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
Co-authored-by: Harminder Virk <virk.officials@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
FIXES FRMW-2879
**What**
The exclusion regexp was broken and instead use the ignore options of the glob sync function to properly ignore definition files
In the framework package we are directly importing many uninstalled dependencies. The issue does not occur for transitive dependencies. However, the `glob` package is not a transitive dependency of any production dependency and hence it fails.
Fixes: #11044
Fixes: FRMW-2877
