Carveout, rollout, outsourcing

The project of SAP system introduction in the Bakalland group was a complex undertaking. It involved a lot of hard work. The project can be divided into two main stages: carveout and rollout. The first stage of carveout is the acquisition of the SAP system from an original installation in Norway to a location in Poland. The second intensive stage – rollout – required a huge commitment from the employees of Bakalland and Delecta. A migration from the currently-used ERP system in Bakalland to a SAP installation of Delecta was necessary. During the functionality analysis and the difference identification in the processes, over 100 gaps and functional differences were identified requiring adaptation. The rollout stage will be presented in the subsequent article. Below, we describe the course of work associated with the first stage – carveout. More about system migration and hosting at All for One Data Centers see on article “Bakalland and Delecta: Migration, new location, and SAP maintenance”.

Business background of the project

In the 25 years of its operation, Bakalland (initially Uno Tradex) has grown into a leader of the nuts and dried fruit market in Poland from a three-man company situated in a studio flat. The success can be attributed to a consistent development of the distribution network and the product range. Only six years after the creation of the company, an acquisition of other companies begun, which ‘as a dowry’’, introduced new outlets and allowed to expand the product portfolio, among others. The mergers with other companies gave great opportunities for synergistic use of their strong sides. This skill has been very well mastered at Bakalland.

Another opportunity to adopt good practices of a new entity in the group appeared with the acquisition of the Delecta Company, which initially occurred at the beginning of 2015.

The expansion of the portfolio with complete products and a higher distribution development potential in the country as well as abroad form the most important business benefits from the merger of the two entities. In addition, an essential benefit, but rarely mentioned, is the added value to the purchase, which, for Bakalland, is the SAP ERP system implemented and adapted for business needs.

The SAP installation was an essential part of the Delecta purchase. However, before the system could support business processes, an IT project had to be conducted consisting of: the isolation (carveout) of a part of the system, supporting Delecta’s processes from the SAP installation of the Orkla syndicate (the previous owner); and then adapting it to business requirements of Bakalland, and finally, its boot-up in consecutive partnerships of the capital group.

From July of this year, the Bakalland Group is using a SAP system fully adapted to its needs, in a broad functional scope (finance and controlling, material and storage management, production planning, quality management, maintenance management and HR management). The system functions in an efficient and a doubly secured All for One Data Centers infrastructure, administrated by BCC specialists. BCC service teams support the employees of the group in their everyday work with the system.

We are introducing the objectives and methodology of BCC (now All for One Poland), according to which the project for the Bakalland Company was realized.

Targets and conditionings

Like any project, including carveout, different from operational activities, it has two basic attributes: time frames and a measurable product/result. These two features are mostly well defined by carveouts. The goal is to take over a part of the existing SAP system and to transfer it to a new hardware infrastructure. Timeframes are determined by the complexity of the system and other untechnical factors.

The scope of the Delecta system included the modules: FI, CO, SD, MM, WM, PP, PM, QM as well as HR, so it was very broad and required a high involvement of resources.

On the other hand, emphasis was placed on the fastest possible project completion. This was dictated by budgetary issues: the system was ultimately meant to be taken over by All for One Data Centers, which meant significant lower maintenance costs of the installation in relation to the parent company.

Such a situation is typical for SAP separation projects because the systems usually follow the acquired subsidiary companies to a cheaper infrastructure located in Poland.

Here, we have another essential factor: access to the source system. The parent companies are usually reluctant to grant access to central systems for consulting companies that do not have a permanent working contract.

This is why it is worth to encapsulate appropriate provisions in the acquisition contract. Otherwise, the available options for project the execution may be reduce to only using the Company Code Deletion (CCD) service indicated directly through SAP, or project realization through a central IT team. Both options are usually substantially more expensive (even over twofold) than entrusting the implementation to an experienced consulting company.

Apart from “political" issues, data organization has an impact on the choice of tools used for project execution. We have three possible scenarios: data of an assigned partnership are stored on a separate system, on a separate mandate or are integrated with the remaining organizational units of the corporation.

The first two cases are unusual because they do not provide synergic effects of mutual data processing for all organizational units. Nevertheless, it is worth verifying the situation in this area because unit separation in these scenarios comes down to the adoption of a finished system or to the deletion of irrelevant mandates.

The third option occurs 99% of the times, as was the case of Delecta, so further consideration will be devoted to this variant.

Action plan

Before we get to the details, it is worth to first answer the question – what in fact is SAP carveout?

It is a project, within which a copy of the central and productive SAP system is performed, and then irrelevant data is deleted and in the final step, such prepared environment is transferred to a new hardware infrastructure.

There is also an alternate possibility to create an empty final environment and a gradual transfer of configurations, extensions and, in the end, also data. However, because of the fact that, usually, one of the main goals is the adoption of transactional data, it results in the necessity for their migration. This is not a simple task, so this approach is rarely used in practice, only in specific situations. In real conditions, we aim for risk minimizing related to the introduction of changes in systems critical for operational activity, which is also true for carveout projects. A specific action methodology is used for this. It may seem surprising at first, but a standard BCC Go Forward methodology was used for the Delecta system assignment, developed to perform SAP implementations. A division into five phases was introduced:

• Project preparation,
• concept,
• realization,
• boot-up preparation,
• boot-up and post-boot-up support.

In so far as the project phases coincide with other types of SAP projects, the other actions realized in specific phases have a different character.

The concept does not focus on business processes and system settings, but on the identification of objects and the scope of deleted data, as well as the tools used for this.

As part of the realization, we do not build solutions, but delete data. However, there is one common element left – system tests. After data deletion, a verification is desired to see if the system still functions correctly. This is an essential step because a range of settings depend on correct basic data, which are modified during the realization. The realization and system verification is conducted on a SAP test instance.

A positive reception allows for a transfer to the production boot-up preparation phase. In contrast to standard SAP projects, data migration is not performed in carveout and users are not trained. Instead, the previously-developed tools are used for data deletion on the production system, which results in a green light for system migration from an organization that controls the source system.

Essentially, the production boot-up comes down to performing system migration to a new data center and creating a final SAP environment from a single copy.

Generally, actions in a carveout-type project (also in the project realized for Bakalland-Delecta) are in the following order:

• preparing the SAP working environment – SAP productive environment snapshot,
• identification of objects and the scope of irrelevant data deletion,
• data deletion in the test environment,
• performing verification tests of the environmental integrity,
• creating a productive system copy and transferring operational activity of the partnership to a copied system,
• data deletion on a new productive system,
• migration of the ‘’cleaned’’ system to a new hardware infrastructure,
• commencing operational work on a new system.

Małgorzata Gołuchowska, Financial Director, Bakalland

After a phase of intensive growth in the Bakalland company, time has come to tidy up processes, a larger integration and group consolidation, as well as providing our business actions with frameworks, which will give access to reliable and up-to-date information, and at the same time, allow for a better control and support in the making of good managerial decisions.The necessity to encompass our system into an integrated IT system framework has been obvious to us for some time, and recently, it became searing.The incorporation of the Delecta Company into the Bakalland Capital Group with a functional SAP system became a great opportunity to achieve this goal at minimal costs – financial and organizational – in the SAP project. We have acquired a functional system that has serviced a company in a related trade, which in a relatively short time, but with very intensive work of the team, managed to be adapted to the needs of Bakalland.The SAP project posed special challenges in the capital group with ongoing organizational changes for all participants of the undertaking. Knowing the challenges before us, we were looking for a partner with versatile expertise. We have chosen BCC (now All for One Poland) because of its large and competent team of consultants as well as extensive experience in SAP projects related to mergers and acquisitions. And also for the complete SAP maintenance offer, from application service for users, to hosting and administration.
Małgorzata Gołuchowska, Financial Director, Bakalland

Concept

This is an especially important stage of the project because mistakes made in this phase can create large delays in further phases of the project.

In the first step of the concept, the tools that will be used for data deletion must be defined. There are two options here:

• SAP Company Code Deletion (CCD) service,
• Data backup tools.

The first option is recommended by the SAP Company for obvious reasons, but its cost may prove very high.

The second option is data backup tools used mostly for decreasing table sizes, and as a result, increasing the system speed. Originally, they are not intended to perform carveouts, but practice shows that they are excellent at this. In Delecta’s case, the second option was used and further argument will be introduced in this context (independent from the accepted tool, the steps described in the previous step remain identical).

The carveout concept is specific and, basically, it only concentrates around data (transactional and basic).

Object identification alone is not enough, it is essential to also state its attributes. These attributes define the way in which the scope of data for the chosen object meant for deletion will be determined.

The determination of attributes should begin with the presentation of organizational structures implemented in SAP, along with pointing out the branches representing the assigned part of the organization.

At first, it might seem that indicating the business unit, which should be kept, is enough, but nothing could be more wrong.

In the case of Delecta, as part of its business unit, a plant was defined that did not concern the partnership. It was the remains of a former organizational structure that should have been deleted before the migration.

The fact of the existence of an additional plant had an impact on the method of data identification (e.g. it became impossible to use only the sales organization assigned to Delecta for SD objects: individual documents related to the discarded plant had to be identified).

To avoid annoying surprises, it is essential to analyze the entire organizational structure (business units, plants, warehouses, sales organizations, purchasing organizations, the areas of cost accounting, etc.)

The second set of data for verification is the module range that can differ for the assigned partnership and the whole organization. In the Delecta system assignment project, it turned out that the central organization used the PM module, which remained out of the reach of Delecta’s activity range (so the module was not considered in the offering stage, but it was included into the project range after its inauguration).

Important notice: because of the fact that the green light for the system migration is given by a central organization and it has full knowledge on the whole of the used system range, the concept should be confirmed by a central organization, and not by the assigned partnership.

The last step after object backup and attributes identification (criteria identifying deleted data) is the preparation of the backup map. It presents the dependencies between given objects and determines the order in which the backup is performed.

Realization

The carveout realization, as mentioned before, comes down to performing a backup in accordance with the prepared concept. Unfortunately, this process almost always brings problems. Trouble arises most frequently in two areas:

• Unfinished processes – backup tools only work on data generated by properly completed processes. It is basically impossible to backup any open documents (production orders, sales orders, deliveries that were not confirmed with an invoice, etc.). This means that all open positions have to be closed. SAP tools exist for large amounts of objects that allow introducing mass changes, but unfortunately not for all. It is important to register all changes in a previously prepared log. All actions performed in the realization stage will have to be repeated in the boot-up preparation phase on a new productive system. In the Rieber Foods system, we unfortunately come across a large number of open order processes, of which their handling affected the carveout realization time.
• Very large/small efficient data objects – depending on the age and the intensity of system use, it can reach very large sizes. Deleting a large amount of data is unfortunately very time consuming. For very large, but simple projects (e.g. material movement in WM), the backup process can last even several days, which is acceptable with a proper action order. Unfortunately, apart from large and simple objects, there are also large and complicated ones. Complicated in the sense that prior to deleting a single record, a large amount (can reach several thousands) of additional verification questions are performed, if the given object can be deleted (e.g. MM_SPSTOCK object). The situation becomes very difficult, as the standard backup program for such an object can work for even several dozen days (without a break), which is rather disqualifying from the project perspective. In this case, all that is left is to write your own program that imitates the actions of the standard one, but optimized for the deletion of the specific range of data. This is accompanied with a large risk, which can be reduced with experience and in-depth tests by the consulting company. At Delecta, the process of MM_SPSTOCK object backup was sped up, from the predicted 150 days, to only around 8 hours.

The above potential problems generate risk that, after data deletion, the system will not function correctly. In such cases, the moderation action is to always perform full acceptation tests. Such tests should be supported with appropriate documentation, in other words – filled with test scenarios. Most often, carveout projects are conducted on “old" systems, for which up-to-date test scenarios may not exist. With this in mind, it is worth beginning a documentation quality verification at the earliest possible stage of the project (incomplete documentation should be documented at the stage of concept building).

Apart from the acceptation tests conducted by the assigned unit, system verification conducted by a central organization is also essential. Usually, business units for which data was deleted are involved in the process. During the project planning, the fact that the availability of people from given units may be limited, should be considered, so it is worth ensuring adequate support in this regard beforehand, preferably in agreement with the central organization.

The time of the realization process largely depends on the size of the system as well as the hardware infrastructure that supports it. The test data deletion took less than two months at Delecta on a rather large system.

A good practice in the realization phase is to keep a backup log. In the log, we register the backup time of given objects. Thanks to which we will be able to rather precisely estimate the time essential to conduct the boot-up preparation phase.

Start-up preparation

Armed with a concept and an activity log deriving from its use in practice and the entirety being verified by tests, we can proceed to data deletion on the productive system. Of course, the aim is not to deprive the central organization of all data except for those belonging to the unit that is being assigned. This forces the creation of an alternate productive environment for the assigned unit. This environment is created during ‘’mini go-live’’ where:

• access is temporarily blocked to the current productive system,
• its copy is created,
• it is migrated to a new hardware platform (under constant control of the central organization),
• all interfaces are transferred,
• authorization changes are performed (access to the new system remains enabled only for the members of the separated unit; but they lose access to the current productive system),
• SAP GUI is transferred to the new system.

At this point, it is worth considering a few details.

The process described above is almost entirely performed by the central organization, so it is essential to guarantee appropriate resources in the contract. “Mini go-live" can be successfully performed at the weekend, subject to prior preparation of the appropriate infrastructure.

Hardware resources assigned to the new system cannot differ much from the resources that support the current productive system. This is associated with the fact that, despite a lower number of users straining the new productive system, a data backup will be performed in parallel resulting in high system stress. Therefore, it is also advisable to define these issues in the contract.

At the beginning of the boot-up preparation stage, the project already has a test system with deleted data, confirmed by the central organization. This is a fantastic opportunity to perform the system’s migration test. Such a migration allows verifying two incredibly important issues: data connection and target platform compatibility. As a result of licensing resolutions and the SAP HANA platform as a target environment, Bakalland-Delecta decided on MaxDb as a database layer. The original Orkli system was supported by Oracle. In this case, only heterogeneous migration test came into play, which ended in success.

However, it turned out that the connection speed was too low – the test base transfer took four days (we will mention the consequences further on).

Considering the previously-acquired experience, the boot-up preparation phase takes much shorter than realization. In the project for Bakalland-Delecta, a productive data deletion was realized in three weeks. The phase ends in with a final audit of the system by the central organization (specific business units verify if sensitive data are not left in the system). Only a positive audit result allows to proceed to the last phase of the project.

Go-live

Just like any SAP production boot-up, carveout requires a certain amount of system downtime for operational actions. Naturally, it is kept at a minimum. In carveout, the downtime is determined by the length of the three basic activities:

• system export to a file,
• a physical transfer of the generated file,
• SAP system reproduction from a file (due to limited time, it is unusual to reproduce the entirety of the environment, only the productive system; the remaining systems are run independently in the following days after the boot-up).

After data deletion, the Delecta system took up 1.5 TB of disc space. The export of such a large system took over 8 hours. Further 14 hours were needed for system recreation along with settings adaptation and interface reconfiguration, which brought it to a total of nearly 24 hours.

As mentioned previously, the test base transfer took 4 days, so the whole boot-up window should take 5 full days (without any buffers), which was unacceptable from a business stand point. An alternate plan was introduced: data was transported by plane on external discs by a Delecta employee. The whole physical transfer operation was shortened to 8 hours.

The system was launched in a predicted time frame without major problems. In the next step, system development could be commenced for the needs of the Bakalland Capital Group. This stage, which includes rollout solutions to the remaining entities in the group, will also took place with the support of BCC (now All for One Poland).