PayUp’s Prepaid transaction processing system is a system that requires high availability. It is a distributed system that is responsible for the settlement of transactions for the sale of recharges for subscription-free cell phones, as well as for the settlement of other financial services, such as bill payment and payment card acceptance using POS terminals distributed in geographically dispersed sales locations. It is the most important tool and the foundation of the company’s operations.
The system includes several thousand POS terminals, a wide area network connecting telecommunications operators, business partners and servers, which are maintained at BCC’s data center (currently All for One Poland). Since the beginning of its operations on the Polish market, the company has been using the services of the BCC Outsourcing Center, which is responsible for maintaining the availability and performance of the client’s systems, and since January 2011 has also provided hosting and administration of the transaction system for the company’s Romanian branch.
Technology
High availability of services in the financial industry is crucial to running a business. This poses a major challenge for the IT service provider. For the client’s needs, BCC prepared a redundant server platform built on the basis of IT virtualization technology. It was decided to choose Red Hat Linux as the operating system and MySQL database.
Several thousand remote terminals from Poland and Romania currently connect to the Prepaid system. Connections to the system are established around the clock, as many terminals are located at points open 24 hours a day. During “peak hours," the number of connections reaches several hundred per minute. Connections are handled by three application servers, and data is stored in three database servers. With 24/7/365 operation requirements, a high availability environment is a must.
One of the most important features of the solution proposed and launched by BCC IT consultants is the transfer of virtual servers between physical machines online, without the need to shut down or reboot the operating system. This eliminates the need to shut down the system in connection with, for example, the expansion or repair of a single server.
Connections
Connections of POS terminals to the prepaid system are made using various technologies: GPRS, WAN, IPsec VPN site-to-site and dial-up telephone connections. In turn, connections between the BCC data center and telecom operators and business partners use IPSec site-to-site tunnels and, in addition, dedicated connections.
The Prepaid system itself automatically generates and distributes reports in the form of e-mails and files sent to the client.
Based on the requirements presented by PayUp, an isolated network environment was prepared in the BCC data center. The backbone of the network is formed by fully redundant switches which have the ability to automatically reconfigure in case of failure of any node. Connections to the “outside world" are handled by a cluster of Juniper firewalls operating in active-passive mode.
Internet access is provided by several independent connections provided by different operators and built using different technologies (fiber optics, radio links). The entire configuration of routing routes is supervised by the BGP protocol, thanks to which link failure at individual operators automatically transfers traffic to other links.
Proactive monitoring
Of course, the environment in which the Prepaid system operates must be monitored in detail. Basic parameters are checked continuously: from the availability of servers and the services running on them, to such detailed areas as disk space occupancy or CPU and memory load. Due to the type of service, network connections are also checked in detail, as well as the availability of key hosts on the part of telecom operators and partners.
Any threat detected by the monitoring system (e.g., exceeding the defined security threshold for the use of server CPU power) is signaled by displaying an alarm on the monitors in the BCC monitoring center and generating an audible alarm. In addition, an SMS text message is sent to the cell phone numbers of those who are currently responsible for the availability of BCC data center services. At the BCC monitoring center, administrators are on duty 24/7, who observe the indications of the monitoring systems and, depending on the situation, can immediately take action appropriate to the situation. Support from qualified specialists is thus provided around the clock, every day of the year.
However, watching the indicators on the monitors is not the only task of the consultants at the monitoring center. Every day routine checks are performed on the servers, according to a set procedure. Among other things, logs of operating systems, applications or databases are verified. Also, the infrastructure of the data center itself is inspected every day, allowing proactive catching of emerging threats that could disrupt the system in the future.
Safety and standards
The security of systems and data maintained at the BCC Outsourcing Center (currently All for One Poland) is ensured by implemented ISO standards. The handling of systems within BCC’s IT outsourcing services is based on the Information Security Management System, compliant with ISO/IEC-27001, the most restrictive standard, indicating requirements for confidentiality, integrity and availability of information.
The ISO/IEC-27001 standard covers all issues related to the protection of information created, stored and processed in a company. Its purpose is to verify and strengthen safeguards taking into account the sources of threats, which can be people, business processes and technologies. The standard is part of the procedures related to building Business Continuity Planning, which enables efficient response to crisis situations and even unpredictable disasters, such as fire, flood or construction disaster.
Each production system has its own detailed business continuity plan (PCD), which specifies what steps need to be taken to restore its availability in the event of a disaster. PCD plans are regularly tested and updated by administrators. Testing the PCD plan involves completely restoring the system to a backup server and verifying that it is working properly. With regular testing, we are able to determine exactly how much time will be needed to restore the system in case of a disaster.
The BCC Outsourcing Center also has an IT Service Management System that complies with the ISO/IEC-20000 (ITIL) standard. The ISO/IEC-20000 IT service management standard gives specific guidelines on how to effectively handle customer requests, respond to incidents or resolve problems. The standard defines a set of key IT processes – including service level management and reporting, availability and business continuity management, information security management, and configuration and change management.
Flexible environment
Hosting of the Prepaid system is an example of a “tailor-made" service. Initially, the system consisted of several servers and included terminals in Poland. In subsequent years, the installation grew as the client’s business grew. The implementation of new services and the expansion of coverage to another country, Romania, increased the number of serving machines. The number of connections to more telecommunications operators outside Poland also increased. Cooperation with the BCC Outsourcing Center allowed the client to reduce IT maintenance costs at the beginning of its business operations. The fee covered only the computing power that was used.
Recipients of services in Romania are unaware that authorization of their transactions travels a geographically very long route – to Zlotniki near Poznan in Poland, to the GSM operator appropriate for the transaction in Romania, and back again – to a terminal standing somewhere in a store in their country.
In addition, the great flexibility of the technology on which the solution is based allowed for smooth expansion of the system’s resources. Subsequent development work is carried out without any downtime in the operation of the production infrastructure. This is a very important advantage, not only in the financial services industry.
Prepaid system hosting is an example of a “tailor-made" service. Initially, the system consisted of a few servers, and in subsequent years the installation grew as the client’s business grew
Competencies for hire
Hardware costs are only part of the expenses in IT projects. Often a much bigger problem – and a significant cost – is acquiring the right staff. Recruiting and then implementing and training new IT administrators and network administrators with the right skills is a time-consuming and costly process. Another challenge is organizing the work in accordance with best practices for the IT industry and ISO/ITIL standards.
Such a demanding and complex system as Prepaid requires the care of experienced and competent administrators. BCC’s team, which is responsible for the ongoing maintenance and development of the systems, includes more than a dozen administrators. Their competence and specialization in the respective fields are confirmed by many years of experience in implementation projects and service, as well as certificates, including in database technology (Oracle, MSSQL, DB2), operating systems (Windows, Unix, Linux, i5/OS) and network technologies (Juniper, Cisco).
In addition to purely technical competence, language competence is no less important. People of different nationalities collaborated on the implementation of the Prepaid system. In day-to-day work concerning the maintenance of the Prepaid system or changes in the configuration of technical parameters, BCC administrators contact specialists from Portugal and Romania, among others, in English.
BCC Outsourcing Center provides a comprehensive service for the maintenance of the IT environment: from hosting and colocation in the BCC data center, to systems administration. BCC works with the largest hardware and software suppliers, which allows it to offer the best technical solutions tailor-made. The needs of each client are treated individually.
Marek Strzelczyk, Director of IT and Operations, Member of the Management Board, PayUp Poland
Currently, the systems located at the BCC Outsourcing Center (currently All for One Poland) support about 7,000 POS terminals (about 4,500 in Poland and 2,500 in Romania). In both countries, the terminals are distributed throughout their territories.
From the beginning of PayUp’s operations, the company’s concept was to outsource many services, including IT system hosting services. BCC was chosen as our partner because of its professionalism and competitive pricing. The fact that we decided to transfer to BCC also the systems from Romania, which were initially serviced by a local provider, is an expression of our confidence in the competence presented by the employees of the BCC Outsourcing Center and in the quality of the services provided.
The IT team of PayUp Poland consists of six people. Their competencies focus on knowledge of POS terminals, their applications and the central applications of the Prepaid system. Issues related to the management of servers, operating systems and databases are left to our partner.
In cooperation with BCC, we managed to develop the concept of Disaster Recovery – a redundant hardware and software solution that ensures parallel operation of PayUp’s systems in two geographically distant locations and taking over all transaction traffic by one of these locations in case of failure in the other location. BCC staff served with their knowledge and experience in the development of this solution.
In their day-to-day interactions, the Outsourcing Center’s staff demonstrates a high degree of availability and flexibility, a friendly approach to customers and far-reaching assistance, such as hosting our colleagues and partners from other countries for reference visits.
Marek Strzelczyk, Director of IT and Operations, Member of the Management Board, PayUp Poland