mage2alokai

Stay connected

For those who would love to work with Magento on backend but use NoSQL power on the frontend. Two way / real time data synchronizer.

It's part of alokai project - first Progressive Web App for eCommerce with Magento2 support. Some details about the rationale and our goals here

It synchronizes all the products, attributes, taxrules, categories and links between products and categories.

This is multi-process data synchronizer between Magento to Alokai ElasticSearch database.

At this point synchronization works with following entities:

• Products
• Categories
• Taxrules
• Attributes
• Product-to-categories
• Reviews (require custom module Divante/ReviewApi to work)
• Cms Blocks & Pages (require custom module SnowdogApps/magento2-cms-api)

Categories and Product-to-categories links are additionaly stored in Redis cache for rapid-requests (for example from your WebAPI). Our other project alokai-api exposes this database to be used in PWA/JS webapps.

Datasync uses oauth + magento2 rest API to get the data. KUE is used for job queueing and multi-process/multi-tenant processing is enabled by default ElasticSearch is used for NoSQL database Redis is used for KUE queue backend

By default all services are used without authorization and on default ports (check out config.js or ENV variables for change of this behavior).

Tutorial on installation / integration manual for Alokai connectivity

How to perform full / initial import for Alokai

To get started with VS we must start with some very basics about the architecture; the project is backed by three separate Node.js applications

Alokai Architecture

alokai (Github) — is the main project where you can also find most of the documentation, issues mapped to further releases and other resources to start with — Vue.js on webpack.

alokai-api (Github) — is the API layer which provides the data to vue-storefront app — Node.js, Express; This project consist of docker instances for Redis and ElasticSearch required by mage2vuestorefront and pimcore2vuestorefront

mage2alokai — THIS project -data bridges which are in charge of moving data back from Magento2 to Alokai data store.

You must install alokai-api locally. You may install it using the Alokai installer - see details. Or manually by executing the sequence of commands:

git clone https://github.com/DivanteLtd/vue-storefront-api
cd vue-storefront-api
npm install
npm run migrate
docker-compose up -d
npm run dev

The key command is docker-compose up -d which runs the ElasticSearch and Redis instances - both required by mage2vuestorefront

Elastic 7 Support

By default, Alokai API docker files and config are based on Elastic 5.6. We plan to change the default Elastic version to 7 with the 1.11 stable release. As for now, the Elastic 7 support is marked as experimental.

In order to index data to Elastic 7 please make sure you set the proper apiVersion in the config.js:

  elasticsearch: {
    apiVersion: process.env.ELASTICSEARCH_API_VERSION || '7.1'
  },

or just use the env variable:

export ELASTICSEARCH_API_VERSION=7.1

Starting from Elasitc 6 and 7 we can have just single document type per single index. Alokai used to have product, category ... types defined in the vue_storefront_catalog.

From now on, we're using the separate indexes per each entity type. The convention is: ${indexName}_${entityType}. If your' logical index name is vue_storefront_catalog then it will be mapped to the physical indexes of: vue_storefront_catalog_product, vue_storefront_catalog_category ...

Initial Alokai import

Now, You're ready to run the importer. Please check the config file. You may setup the Magento access data and URLs by config values or ENV variables.

We'll use in the following example - the ENV variables. The simplest command sequence to perform full reindex is:

export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9

echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
export INDEX_NAME=vue_storefront_catalog

node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
node --harmony cli.js reviews

After installing the 3rd party Magneto module (SnowdogApps/magento2-cms-api) there are two additional imports available:

node --harmony cli.js blocks
node --harmony cli.js pages

Please note:

• --removeNonExistent option means - all records that were found in the index but currently don't exist in the API feed - will be removed. Please use this option ONLY for the full reindex!
• INDEX_NAME by default is set to the vue_storefront_catalog but You may set it to any other elastic search index name.
• The categories importer option --generateUniqueUrlKeys is by default set to true. This is due the fact that in Magento2, the category.url_key field is not mandatory unique and from v. 1.7 Alokai uses the category.url_key to display the category details without any client's side modification.
• PRODUCTS_EXCLUDE_DISABLED by default is set to false. To only import enabled products set this to true.

Cache invalidation: Recent version of Alokai do support output caching. Output cache is being tagged with the product and categories id (products and categories used on specific page). Mage2vuestorefront can invalidate cache of product and category pages if You set the following ENV variables:

export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1

• VS_INVALIDATE_CACHE_URL is a cache to the Alokai instance - used as a webhook to clear the output cache.

Please note: After data import - especially when You're not sure about the product attributes data types - please reindex ElasticSearch to establish the correct / current database schema. You may do this using Database tool in the vue-storefront-api folder:

cd vue-storefront-api
npm run db rebuild -- --indexName=vue_storefront_catalog

If You like to create a new, empty index please run:

cd vue-storefront-api
npm run db new -- --indexName=vue_storefront_catalog

Checking indexed data

If you want to see how many products were stored into Elastic data store, you can use Kibana to do so. Kibana is part of alokai-api. Once you start docker containers of alokai-api you can access it on http://localhost:5601/.

To see count of indexed products go to DEV tools and run following query:

GET vue_storefront_catalog/product/_count

See https://www.elastic.co/guide/en/kibana/current/console-kibana.html to find out more.

Delta indexer

After initial setup and full-reindex You may want to add indexer to the crontab to index only modified product records. This is fairly easy - You just need to add the following command to crontab:

node --harmony cli.js productsdelta --partitions=1

This command will execute full reindex at first call - and then will be storing the last index date in the .lastIndex.json and downloading only these products which have updated_at > last index date.

If you have a multistore setup and would like to use the delta indexer for each storeview you can not use the delta timestamp from .lastIndex.json for all stores; instead you will need to set the INDEX_META_PATH to a unique value for each store you are indexing. For instance:

export INDEX_META_PATH=.lastIndex-UK.json && node --harmony cli.js productsdelta --partitions=1

Please note: Magento2 has a bug with altering updated_at field. Please install a fix for that before using this method:

composer require codepeak/magento2-productfix
php bin/magento cache:flush

Parent products updates

Please note if there is a simple product update request coming from Delta Indexer or On Demand indexer mage2vuestorefront will - by default - check and update the configurable/parent product as well. The parent product update is scheduled in the productsworker mo