**What**
- Adjust lock duration, it is in seconds and not in ms
- Log on lock release
- Renew lock separately from the other promises
- Add more logs
- Log when lock can't be acquired. It can be expected in case two processes try to sync the same entity and in that case it can be ignored. But at least it gives some information in case it happens for another reason
- Log when the release lock failed, the lock will remain in the locking provider for 1 minute before being removed. But it wont prevent other entities to be synced
* feat(index): add filterable fields to link definition
* rm comment
* break recursion
* validate read only links
* validate filterable
* gql schema array
* link parents
* isInverse
* push id when not present
* Fix ciruclar relationships and add tests to ensure proper behaviour (part 1)
* log and fallback to entity.alias
* cleanup and fixes
* cleanup and fixes
* cleanup and fixes
* fix get attributes
* gql type
* unit test
* array inference
* rm only
* package.json
* pacvkage.json
* fix link retrieval on duplicated entity type and aliases + tests
* link parents as array
* Match only parent entity
* rm comment
* remove hard coded schema
* extend types
* unit test
* test
* types
* pagination type
* type
* fix integration tests
* Improve performance of in selection
* use @@ to filter property
* escape jsonPath
* add Event Bus by default
* changeset
* rm postgres analyze
* estimate count
* new query
* parent aliases
* inner query w/ filter and sort relations
* address comments
---------
Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
**What**
First iteration to prevent events from overwhelming the systems.
- Group emitted event ids when possible instead of creating a message per id which leads to reduced amount of events to process massively in cases of import for example
- Update the index engine to process event data in batches of 100
- Update event handling by the index engine to be able to upsert by batch as well
- Fix index engine build config for intermediate listeners inferrence
**What**
- Add index engine feature flag
- apply it to the `store/products` end point as well as `admin/products`
- Query builder various fixes
- search capabilities on full data of every entities. The `q` search will be applied to all involved joined table for selection/where clauses
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
What:
- `query.index` helper. It queries the index module, and aggregate the rest of requested fields/relations if needed like `query.graph`.
Not covered in this PR:
- Hydrate only sub entities returned by the query. Example: 1 out of 5 variants have returned, it should only hydrate the data of the single entity, currently it will merge all the variants of the product.
- Generate types of indexed data
example:
```ts
const query = container.resolve(ContainerRegistrationKeys.QUERY)
await query.index({
entity: "product",
fields: [
"id",
"description",
"status",
"variants.sku",
"variants.barcode",
"variants.material",
"variants.options.value",
"variants.prices.amount",
"variants.prices.currency_code",
"variants.inventory_items.inventory.sku",
"variants.inventory_items.inventory.description",
],
filters: {
"variants.sku": { $like: "%-1" },
"variants.prices.amount": { $gt: 30 },
},
pagination: {
order: {
"variants.prices.amount": "DESC",
},
},
})
```
This query return all products where at least one variant has the title ending in `-1` and at least one price bigger than `30`.
The Index Module only hold the data used to paginate and filter, and the returned object is:
```json
{
"id": "prod_01JKEAM2GJZ14K64R0DHK0JE72",
"title": null,
"variants": [
{
"id": "variant_01JKEAM2HC89GWS95F6GF9C6YA",
"sku": "extra-variant-1",
"prices": [
{
"id": "price_01JKEAM2JADEWWX72F8QDP6QXT",
"amount": 80,
"currency_code": "USD"
}
]
}
]
}
```
All the rest of the fields will be hydrated from their respective modules, and the final result will be:
```json
{
"id": "prod_01JKEAY2RJTF8TW9A23KTGY1GD",
"description": "extra description",
"status": "draft",
"variants": [
{
"sku": "extra-variant-1",
"barcode": null,
"material": null,
"id": "variant_01JKEAY2S945CRZ6X4QZJ7GVBJ",
"options": [
{
"value": "Red"
}
],
"prices": [
{
"amount": 20,
"currency_code": "CAD",
"id": "price_01JKEAY2T2EEYSWZHPGG11B7W7"
},
{
"amount": 80,
"currency_code": "USD",
"id": "price_01JKEAY2T2NJK2E5468RK84CAR"
}
],
"inventory_items": [
{
"variant_id": "variant_01JKEAY2S945CRZ6X4QZJ7GVBJ",
"inventory_item_id": "iitem_01JKEAY2SNY2AWEHPZN0DDXVW6",
"inventory": {
"sku": "extra-variant-1",
"description": "extra variant 1",
"id": "iitem_01JKEAY2SNY2AWEHPZN0DDXVW6"
}
}
]
}
]
}
```
Co-authored-by: Adrien de Peretti <25098370+adrien2p@users.noreply.github.com>
Closes: FRMW-2892, FRMW-2893
**What**
Wired up the building block that we merged previously in order to manage data synchronization. The flow is as follow
- On application start
- Build schema object representation from configuration
- Check configuration changes
- if new entities configured
- Data synchronizer initialize orchestrator and start sync
- for each entity
- acquire lock
- mark existing data as staled
- sync all data by batch
- marked them not staled anymore
- acknowledge each processed batch and renew lock
- update metadata with last synced cursor for entity X
- release lock
- remove all remaining staled data
- if any entities removed from last configuration
- remove the index data and relations
Co-authored-by: Carlos R. L. Rodrigues <37986729+carlos-r-l-rodrigues@users.noreply.github.com>
* ../../core/types/src/dml/index.ts
* ../../core/types/src/dml/index.ts
* fix: relationships mapping
* handle nullable foreign keys types
* handle nullable foreign keys types
* handle nullable foreign keys types
* continue to update product category repository
* fix all product category repositories issues
* fix product category service types
* fix product module service types
* fix product module service types
* fix repository template type
* refactor: use a singleton DMLToMikroORM factory instance
Since the MikroORM MetadataStorage is global, we will also have to turn DML
to MikroORM entities conversion use a global bucket as well
* refactor: update product module to use DML in tests
* wip: tests
* WIP product linkable fixes
* continue type fixing and start test fixing
* test: fix more tests
* fix repository
* fix pivot table computaion + fix mikro orm repository
* fix many to many management and configuration
* fix many to many management and configuration
* fix many to many management and configuration
* update product tag relation configuration
* Introduce experimental dml hooks to fix some issues with categories
* more fixes
* fix product tests
* add missing id prefixes
* fix product category handle management
* test: fix more failing tests
* test: make it all green
* test: fix breaking tests
* fix: build issues
* fix: build issues
* fix: more breaking tests
* refactor: fix issues after merge
* refactor: fix issues after merge
* refactor: surpress types error
* test: fix DML failing tests
* improve many to many inference + tests
* Wip fix columns from product entity
* remove product model before create hook and manage handle validation and transformation at the service level
* test: fix breaking unit tests
* fix: product module service to not update handle on product update
* fix define link and joiner config
* test: fix joiner config test
* test: fix joiner config test
* fix joiner config primary keys
* Fix joiner config builder
* Fix joiner config builder
* test: remove only modifier from test
* refactor: remove hooks usage from product collection
* refactor: remove hooks usage from product-option
* refactor: remove hooks usage for computing category handle
* refactor: remove hooks usage from productCategory model
* refactor: remove hooks from DML
* refactor: remove cruft
* order dml
* cleanup
* re add foerign key indexes
* wip
* chore: remove unused types
* wip
* changes
* rm raw
* autoincrement
* wip
* rel
* refactor: cleanup
* migration and models configuration adjustments
* cleanup
* number searchable
* fix random ordering
* fix
* test: fix product-category tests
* test: update breaking DML tests
* test: array assertion to not care about ordering
* fix: temporarily apply id ordering for products
* types
* wip
* WIP type improvements
* update order models
* partially fix types temporarely
* rel
* fix: recursive type issue
* improve type inference breaks
* improve type inference breaks
* update models
* rm nullable
* default value
* repository
* update default value handling
* fix unit tests
* WIP
* toMikroORM
* fix relations
* cascades
* fix
* experimental dml hooks
* rm migration
* serial
* nullable autoincrement
* fix model
* model changes
* fix one to one DML
* order test
* fix addresses
* fix unit tests
* Re align dml entity name inference
* update model table name config
* update model table name config
* revert
* update return relation
* WIP
* hasOne
* models
* fix
* model
* initial commit
* cart service
* order module
* utils unit test
* index engine
* changeset
* merge
* fix hasOne with fk
* update
* free text filter per entity
* tests
* prod category
* property string many to many
* fix big number
* link modules migration set names
* merge
* shipping option rules
* serializer
* unit test
* fix test mikro orm init
* fix test mikro orm init
* Maintain merge object properties
* fix test mikro orm init
* prevent unit test from connecting to db
* wip
* fix test
* fix test
* link test
* schema
* models
* auto increment
* hook
* model hooks
* order
* wip
* orm version
* request return field
* fix return configuration on order model
* workflows
* core flows
* unit test
* test
* base repo
* test
* base repo
* test fix
* inventory move
* locking inventory
* test
* free text fix
* rm timeout mock
* migrate fulfillment values
* v6.4.3
* cleanup
* link-modules update sql
* revert test
* remove fake timers
---------
Co-authored-by: adrien2p <adrien.deperetti@gmail.com>
Co-authored-by: Harminder Virk <virk.officials@gmail.com>
Co-authored-by: Oli Juhl <59018053+olivermrbl@users.noreply.github.com>
Fixes: FRMW-2742
In this PR, we fix the build output of the backend source code, which eliminates a lot of magic between the development and production environments.
Right now, we only compile the source files from the `src` directory and write them within the `dist` directory.
**Here's how the `src` directory with a custom module looks like**
```
src
├── modules
│ └── hello
│ ├── index.ts
```
**Here's the build output**
```
dist
├── modules
│ └── hello
│ ├── index.js
```
Let's imagine a file at the root of your project (maybe the `medusa-config.js` file) that wants to import the `modules/hello/index` file. How can we ensure that the import will work in both the development and production environments?
If we write the import targeting the `src` directory, it will break in production because it should target the `dist` directory.
## Solution
The solution is to compile everything within the project and mimic the file structure in the build output, not just the `src` directory.
**Here's how the fixed output should look like**
```
dist
├── src
│ ├── modules
│ │ └── hello
│ │ ├── index.js
├── medusa-config.js
├── yarn.lock
├── package.json
```
If you notice carefully, we also have `medusa-config.js`, `yarn.lock`, and `package.json` within the `dist` directory. We do so to create a standalone built application, something you can copy/paste to your server and run without relying on the original source code.
- This results in small containers since you are not copying unnecessary files.
- Clear distinction between the development and the production code. If you want to run the production server, then `cd` into the `dist` directory and run it from there.
## Changes in the PR
- Breaking: Remove the `dist` and `build` folders. Instead, write them production artefacts within the `.medusa` directory as `.medusa/admin` and `.medusa/server`.
- Breaking: Change the output of the `.medusa/server` folder to mimic the root project structure.
- Refactor: Remove `Symbol.for("ts-node.register.instance")]` check to find from where to load the source code.
- Refactor: Use `tsc` for creating the production build. This ensures we respect `tsconfig` settings when creating the build and also perform type-checking.
Co-authored-by: Adrien de Peretti <25098370+adrien2p@users.noreply.github.com>
**What**
Align the index engine API to be similar to the Query API
## Example
```ts
// Benefit from the same level of typing like the remote query
const { data, metadata } = await indexEngine.query<'product'>({
fields: [
"product.*",
"product.variants.*",
"product.variants.prices.*",
],
filters: {
product: {
variants: {
prices: {
amount: { $gt: 50 },
},
},
},
},
pagination: {
order: {
product: {
variants: {
prices: {
amount: "DESC",
},
},
},
},
},
})
```
