feat: product import strategy (#1706)
* init: copy PI files * feat: add subscribers, refactor strategies folder * wip: strategies integration tests package * fix: rename * wip: use redis * wip: use redis deps, redis setup in local tests * fix: naming collision, medusa config * fix: typing, update apply changes for new event ordering and reimplement interface * feat: make redis container run in integration tests * fix: missing yarn lock * feat: redis setup v2 * fix: setup server imports * fix: a lot of integration issues * fix: a lot of integration issues v2, transform tags, fix `ops` object parsing * wip: parsing product options * feat: ✨creating product and variants works, processing product/variant options, update schema * fix: query keys, logic for finding existing variant * fix: types * feat: update product variant's options * feat: parse MA records * feat: creating/updating MA records, region detection, error handling * feat: throw an error when creating an MA for nonexistent region * refactor: remove unused methods * refactor: use provided ids to track records, extract a couple of methods * refactor: remove unused method * refactor/wip: add initial comment for main methods * refactor: replace usage of RedisJSON functionality with basic k/v api * feat: async progress report * types: define more precise types, cleanup * feat: error handling * feat: unit testing preprocessing * feat: integration testing for CI, fix legacy bug where user is unable to create a variant if regional price is also sent as payload, add csv for integration tests * fix: error throw for logs * feat: add product endpoint snap * refactor: remove log * feat: add snaps, rebase * refactor: add comments * feat: snap update * refactor: typo * refactor: change error handler * feat: Redis cleanup after the job is done * testing :fix product unit test, remove integration snap, add inline object matcher * testing: fix obsolete snaps * refactor: update comments * fix: rebase issue * fix: rebase issue v2, remove log form an integration test * fix: try reverting setup server * fix: insert variants test * refactor: don't pass tx manager, refactor methods * refactor: don't use regionRepo, add `retrieveByName` to region repo * refactor: don't use productRepo * refactor: don't use `productVariantRepo` * refactor: remove repo mocks from unit tests * fix: product import unit tests * feat: file cleanup on finalize, kill test logs * wip: use files to persist ops instead of redis, move strategy class into `batch-job` folder * fix: minio delete method, add file cleanup method to import, fix promise coordination * fix: replace redis methods * feat: store import ops as a file instead of Redis * feat: test cleanup * fix: change unit tests after Redis logic removal * feat: use `results` for progress reporting, add `stat_descriptors` info after preprocessing, remove redis mentions * feat: extract to other files, use directory from property, fix strategy loader to allow other files in `strategies` directory * feat: fix instance progress counter * fix: mock services types * fix: update snaps * fix: error handling stream, fix test file service name generation * fix: remove dir with tmp files after testing * fix: new yarn.lock after rebase * fix: remove log, change object shape * fix: add DI types * refactor: remove container as a csv parser dep * fix: remove seeder, change typings * refactor: reimplement `retrieveByName` in the region service * fix: unit tests typings * fix: remove ts-ignore, complete typings for csv parser validators * fix: don't keep track of progress since it is redundant and only keep track of `advancement_count` * fix: return of the batch job seeder * fix: update find region by name method * fix: update types for service typings * fix: update redis type usage * fix: update unit tests file * fix: unit tests * fix: remove redis from integration test * feat: refactor region retrieval by name * feat: refactor product option update * fix: remove repo import * fix: return redis in test * fix: handle stream error * fix: tmp data cleanup Co-authored-by: fPolic <frane@medusajs.com>
This commit is contained in:
Frane Polić
fPolic
@@ -0,0 +1,255 @@
|
||||
const fs = require("fs")
|
||||
const path = require("path")
|
||||
|
||||
const setupServer = require("../../../../helpers/setup-server")
|
||||
const { useApi } = require("../../../../helpers/use-api")
|
||||
const { initDb, useDb } = require("../../../../helpers/use-db")
|
||||
|
||||
const adminSeeder = require("../../../helpers/admin-seeder")
|
||||
const batchJobSeeder = require("../../../helpers/batch-job-seeder")
|
||||
const userSeeder = require("../../../helpers/user-seeder")
|
||||
|
||||
const adminReqConfig = {
|
||||
headers: {
|
||||
Authorization: "Bearer test_token",
|
||||
},
|
||||
}
|
||||
|
||||
jest.setTimeout(1000000)
|
||||
|
||||
function cleanTempData() {
|
||||
// cleanup tmp ops files
|
||||
const opsFiles = path.resolve("__tests__", "batch-jobs", "product", "imports")
|
||||
|
||||
fs.rmSync(opsFiles, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
describe("Product import batch job", () => {
|
||||
let medusaProcess
|
||||
let dbConnection
|
||||
|
||||
beforeAll(async () => {
|
||||
const cwd = path.resolve(path.join(__dirname, "..", "..", ".."))
|
||||
dbConnection = await initDb({ cwd })
|
||||
|
||||
cleanTempData() // cleanup if previous process didn't manage to do it
|
||||
|
||||
medusaProcess = await setupServer({
|
||||
cwd,
|
||||
redisUrl: "redis://127.0.0.1:6379",
|
||||
uploadDir: __dirname,
|
||||
verbose: false,
|
||||
})
|
||||
})
|
||||
|
||||
afterAll(async () => {
|
||||
const db = useDb()
|
||||
await db.shutdown()
|
||||
|
||||
cleanTempData()
|
||||
|
||||
medusaProcess.kill()
|
||||
})
|
||||
|
||||
beforeEach(async () => {
|
||||
try {
|
||||
await batchJobSeeder(dbConnection)
|
||||
await adminSeeder(dbConnection)
|
||||
await userSeeder(dbConnection)
|
||||
} catch (e) {
|
||||
console.log(e)
|
||||
throw e
|
||||
}
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
const db = useDb()
|
||||
await db.teardown()
|
||||
})
|
||||
|
||||
it("should import a csv file", async () => {
|
||||
jest.setTimeout(1000000)
|
||||
const api = useApi()
|
||||
|
||||
const response = await api.post(
|
||||
"/admin/batch-jobs",
|
||||
{
|
||||
type: "product_import",
|
||||
context: {
|
||||
fileKey: "product-import.csv",
|
||||
},
|
||||
},
|
||||
adminReqConfig
|
||||
)
|
||||
|
||||
const batchJobId = response.data.batch_job.id
|
||||
|
||||
expect(batchJobId).toBeTruthy()
|
||||
|
||||
// Pull to check the status until it is completed
|
||||
let batchJob
|
||||
let shouldContinuePulling = true
|
||||
while (shouldContinuePulling) {
|
||||
const res = await api.get(
|
||||
`/admin/batch-jobs/${batchJobId}`,
|
||||
adminReqConfig
|
||||
)
|
||||
|
||||
await new Promise((resolve, _) => {
|
||||
setTimeout(resolve, 1000)
|
||||
})
|
||||
|
||||
batchJob = res.data.batch_job
|
||||
|
||||
shouldContinuePulling = !(
|
||||
batchJob.status === "completed" || batchJob.status === "failed"
|
||||
)
|
||||
}
|
||||
|
||||
expect(batchJob.status).toBe("completed")
|
||||
|
||||
const productsResponse = await api.get("/admin/products", adminReqConfig)
|
||||
|
||||
expect(productsResponse.data.count).toBe(2)
|
||||
expect(productsResponse.data.products).toEqual([
|
||||
expect.objectContaining({
|
||||
id: "O6S1YQ6mKm",
|
||||
title: "Test product",
|
||||
description: "test-product-description-1",
|
||||
handle: "test-product-product-1",
|
||||
is_giftcard: false,
|
||||
status: "draft",
|
||||
thumbnail: "test-image.png",
|
||||
variants: [
|
||||
expect.objectContaining({
|
||||
title: "Test variant",
|
||||
product_id: "O6S1YQ6mKm",
|
||||
sku: "test-sku-1",
|
||||
barcode: "test-barcode-1",
|
||||
ean: null,
|
||||
upc: null,
|
||||
inventory_quantity: 10,
|
||||
prices: [
|
||||
expect.objectContaining({
|
||||
currency_code: "eur",
|
||||
amount: 100,
|
||||
region_id: "region-product-import-0",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
currency_code: "usd",
|
||||
amount: 110,
|
||||
}),
|
||||
expect.objectContaining({
|
||||
currency_code: "dkk",
|
||||
amount: 130,
|
||||
region_id: "region-product-import-1",
|
||||
}),
|
||||
],
|
||||
options: [
|
||||
expect.objectContaining({
|
||||
value: "option 1 value red",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
value: "option 2 value 1",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
],
|
||||
images: [
|
||||
expect.objectContaining({
|
||||
url: "test-image.png",
|
||||
}),
|
||||
],
|
||||
options: [
|
||||
expect.objectContaining({
|
||||
title: "test-option-1",
|
||||
product_id: "O6S1YQ6mKm",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
title: "test-option-2",
|
||||
product_id: "O6S1YQ6mKm",
|
||||
}),
|
||||
],
|
||||
tags: [
|
||||
expect.objectContaining({
|
||||
value: "123_1",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
expect.objectContaining({
|
||||
id: "5VxiEkmnPV",
|
||||
title: "Test product",
|
||||
description: "test-product-description",
|
||||
handle: "test-product-product-2",
|
||||
is_giftcard: false,
|
||||
status: "draft",
|
||||
thumbnail: "test-image.png",
|
||||
profile_id: expect.any(String),
|
||||
variants: [
|
||||
expect.objectContaining({
|
||||
title: "Test variant",
|
||||
product_id: "5VxiEkmnPV",
|
||||
sku: "test-sku-2",
|
||||
barcode: "test-barcode-2",
|
||||
ean: null,
|
||||
upc: null,
|
||||
inventory_quantity: 10,
|
||||
allow_backorder: false,
|
||||
manage_inventory: true,
|
||||
prices: [
|
||||
expect.objectContaining({
|
||||
currency_code: "dkk",
|
||||
amount: 110,
|
||||
region_id: "region-product-import-2",
|
||||
}),
|
||||
],
|
||||
options: [
|
||||
expect.objectContaining({
|
||||
value: "Option 1 value 1",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
expect.objectContaining({
|
||||
title: "Test variant",
|
||||
product_id: "5VxiEkmnPV",
|
||||
sku: "test-sku-3",
|
||||
barcode: "test-barcode-3",
|
||||
ean: null,
|
||||
upc: null,
|
||||
inventory_quantity: 10,
|
||||
allow_backorder: false,
|
||||
manage_inventory: true,
|
||||
prices: [
|
||||
expect.objectContaining({
|
||||
currency_code: "usd",
|
||||
amount: 120,
|
||||
region_id: null,
|
||||
}),
|
||||
],
|
||||
options: [
|
||||
expect.objectContaining({
|
||||
value: "Option 1 Value blue",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
],
|
||||
images: [
|
||||
expect.objectContaining({
|
||||
url: "test-image.png",
|
||||
}),
|
||||
],
|
||||
options: [
|
||||
expect.objectContaining({
|
||||
title: "test-option",
|
||||
product_id: "5VxiEkmnPV",
|
||||
}),
|
||||
],
|
||||
tags: [
|
||||
expect.objectContaining({
|
||||
value: "123",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,4 @@
|
||||
Product id,Product Handle,Product Title,Product Subtitle,Product Description,Product Status,Product Thumbnail,Product Weight,Product Length,Product Width,Product Height,Product HS Code,Product Origin Country,Product Mid Code,Product Material,Product Collection Title,Product Collection Handle,Product Type,Product Tags,Product Discountable,Product External ID,Product Profile Name,Product Profile Type,Variant id,Variant Title,Variant SKU,Variant Barcode,Variant Inventory Quantity,Variant Allow backorder,Variant Manage inventory,Variant Weight,Variant Length,Variant Width,Variant Height,Variant HS Code,Variant Origin Country,Variant Mid Code,Variant Material,Price ImportLand [EUR],Price USD,Price denmark [DKK],Price Denmark [DKK],Option 1 Name,Option 1 Value,Option 2 Name,Option 2 Value,Image 1 Url
|
||||
O6S1YQ6mKm,test-product-product-1,Test product,,test-product-description-1,draft,,,,,,,,,,Test collection 1,test-collection1,test-type-1,123_1,TRUE,,profile_1,profile_type_1,,Test variant,test-sku-1,test-barcode-1,10,FALSE,TRUE,,,,,,,,,100,110,130,,test-option-1,option 1 value red,test-option-2,option 2 value 1,test-image.png
|
||||
5VxiEkmnPV,test-product-product-2,Test product,,test-product-description,draft,,,,,,,,,,Test collection,test-collection2,test-type,123,TRUE,,profile_2,profile_type_2,,Test variant,test-sku-2,test-barcode-2,10,FALSE,TRUE,,,,,,,,,,,,110,test-option,Option 1 value 1,,,test-image.png
|
||||
5VxiEkmnPV,test-product-product-2,Test product,,test-product-description,draft,,,,,,,,,,Test collection,test-collection2,test-type,123,TRUE,,profile_2,profile_type_2,,Test variant,test-sku-3,test-barcode-3,10,FALSE,TRUE,,,,,,,,,,120,,,test-option,Option 1 Value blue,,,test-image.png
|
||||
|
Reference in New Issue
Block a user