Besides writing code, the daily work of developers concerns data backups, migration, and deployment. Fed up with obstacles concerning a long media data migration, our Java developers involved in Hybris e-commerce B2B projects for over 10 years came up to me once and said they couldn’t do it anymore. An exhaustive manual routine hammered them totally. A few weeks later, they presented me a proof of concept of a module that automated high volume data migration.
We call it a Cloud Backup and Migration v. 2.0 for SAP® Commerce (the CBM module in short). The module can be installed from Aimprosoft GitHub or SAPAppCenter.
The module was created to provide our peers with a way of accelerated backup management and the data migration on running servers.
I nostalgically recall the time of FTP servers. My fingers and brain were burning because of endless data extracts and savings. Someone could connect so-called ‘backup’ via the FTP server, make changes, and disappear invisibly. And you were left alone with changes without any backup. The backup was an archive in the same folder that was made manually at best. And you need to pray that the hard drive isn’t corrupted.
If only we could imagine with guys drinking local spirits, we would be able to make backups on hot servers.
The module did well among the first adapters using real-life business scenarios of customers facilitating migration and deployment on hot servers in real-time. That’s a pretty good perk considering the effort-effectiveness of backup management and flexibility to juggle with Amazon S3 and Dropbox storages due to module.
The OOTB approach of migration, backup, and data management is hardly configurable in SAP Hybris websites. It consumed tons of gold programming hours.
Recommended: SAP Certified Application Associate – Master Data Governance C_MDG_1909
Contents
Challenges developers face while migrating media files
The first challenge that we tackled to solve was a media migration in real-time. Instead of setting up separate extensions for Amazon and migrating data via backoffice, developers can migrate media between a local machine and the most usable by our clients’ cloud storage Dropbox and Amazon S3.
It requires only two clicks to transfer files from premises to the cloud storage and back without loss of quality and downtime. The Commerce CBM works for it regardless of the technical parameters of the e-commerce website.
The second challenge relates to the migration of alarming amounts of item lists in the e-commerce website selling apparel and glasses of our B2C client. Our guys did backup management in all its complexity manually because SAP didn’t have an automated solution for quick backup management. They generated scripts, restricted permissions to archived data, filtered media content before backing up.
Media migration and backup management in such a way delayed us significantly. As a result, it ignited us to create an open-source tool that can reduce delivery time for several hours.
The third challenge we faced repeatedly made us work thoroughly on the connection to the accounts of cloud storage. Dropbox has no typical connection to SAP Commerce Cloud. We streamlined the Dropbox connection flow for the end user, making it look like the AWS S3 connection. It is not necessary, but agree, very nice to enable data Import/Export between cloud and local storage seamlessly.
You can also connect to the unlimited saved and active Amazon S3 Storage accounts through the CBM module, which empowers you to migrate medias in any volumes as quickly as you need.
Features of the Commerce CBM
Server space optimization
Since media files occupy 90% of total storage space in e-commerce websites, we developed a three-step approach in the CBM module to juggle storages favoring free or less expensive ones for data migration. The website keeps operating, also.
We’ve tried to cover the needs of content managers’ and developers’ daily tasks. We went out with the idea that users can access the server and use media files in real-time. You do not need to switch off the website, download and transfer data, deploy, launch the website, and only then get round to the files for the purpose.
Besides, the CBM module automatically transfers the massive files stored on the website to your local environment. One reference link replaces another one within seconds. The module generates and sets up an URL in the strategy that will reference the new storage while medias migrating. Voilà!
Intuitive perspective for the backoffice
Code is beautiful. A product of code ought to be human-oriented. The Commerce CBM has a custom user-friendly UI for intuitively understandable usage.
Automated backups and archiving
.
Users can use stored and restored ImpEx, CSV, or media files backed up archived. Zipped content with all exporting data is created automatically in the SAP Commerce Cloud 1808/1811 (on B2C Accelerator). The whole website’s content or its part, e.g., product or content catalogs, can be managed during usual backup activities in one click.
Unlim cloud storage accounts
The CBM module integrates with Amazon S3 and Dropbox, the most popular cloud services among our clients. Users can avoid double work with manual efforts: cloud integration allows transfer data there and back within any number of cloud storage. You can use the external storage to simplify the management of backed-up content exporting and importing files without resorting to a local machine.
The next versions will have a vast number of clouds for seamless backup migration. Now developers can leverage an unlimited number of previously created storage accounts.
Independent contribution
The backup facilitation is not covered with automation. Parallel task progress is the thing the CBM enables also. In teamwork as a natural process, each developer can export, backup, and restore files independently. Distributed project teams can access the backup copies of the latest content components and modifications quickly. Developers can apply created backups by the CBM module on any server.
Scenario 1. Media migration from Dropbox
The main thing that pushed us to create the CBM Commerce is the lack of basic media migration capabilities. The typical media migration with the CBM module looks like the following.
Getting your project server on SAP Commerce 1808/1811 stopped, you can install the downloaded module. That is probably the only time when you need to stop. Further, automation will do most of the work for you.
After you have updated the system with the installed CBM extension, you can proceed with the login credentials of Dropbox or Amazon S3 storages to choose active accounts for data migration. A drop-down list shows you an extending number of available storage for import and export in the Migration Section.
For example, you can choose Dropbox as source storage and Amazon S3 as target storage, or any other combination you favor. The popped up verification message will invite you to confirm the migration as well as lead you during the processing. After completion, you will be able to use your backups up to you.
Source and target storage can’t match. The CBM is smart and tolerant to give you notice of it if you did it mistakenly. Or probably nimble are its creators. Haha, anyway, even a newbie that has never backed up can cope with it.
Scenario 2. Data import from the Amazon S3 account
The module has a number of connections available with credentials to make a direct data transfer between storages. Data import with the module requires access to the list of connectors to choose import storage. Amazon S3 accounts are added to the drop-down list with connection credentials on the import page.
The chosen files will be imported after pressing the button ‘START IMPORT TASK.’ The runtime may vary depending on the size of the files. The module speaks a user language, so you will be accompanied by small talk messages to inform you about successful data downloading and applying.
You can check the user’s name who performed the import task, a finished date, a task status, and code in the ‘History’ section.
We implemented connection providers for every storage type that can check required credentials and support connections to storages. Successful data import can be performed if the backed up website or catalog is restored in the state before backing up.
As for export scenarios, the CBM performs them in the same simple way in the hot server mode as well.
Additional advantages
The Commerce CBM saves developers from fruitless routine and business owners from ruin caused by manual iterations. We counted 24 hours saved for one backup due to automated creation and deployment. E-commerce projects on SAP Commerce 1808/1811 can become the first adapters of streamlined business process automation.
We added additional data protection to the Commerce CBM with role-based access control. It is rather a business-oriented feature to minimize security risks with discretionary access control. You might face it with dynamically changing business data required to be used by different users in parallel securely.
Summing up
We got the hang of efficient data migration and backup management with the Commerce CBM and are happy to share it with the anticipant development community.
Peers, if you’ve got an idea on how to improve the media migration or backup management, or you struggle with an annoying task, ping me up. Our guys are ready to be challenged more to contribute to the community and help businesses.