Resolving Scalability Issues with AWS Serverless

OVERVIEW

As a well-established and respected information and analytics provider, they rely on many core systems to service both their internal and client needs. Over the years, CoreLogic has continued to grow with customers expecting faster, more in-depth insights and systems being required to cope with ever-increasing demand.

Specialising in property, one of their core systems used throughout the New Zealand business was their property matching solution, held in an on premise Oracle database. Data Army were employed to redesign this solution to operate within the AWS cloud, removing issues that had arisen from lack of scalability, visibility, reporting and deployment.

See CoreLogic website: corelogic.com.au

BUSINESS CHALLENGES

• Scaling limitations of the legacy system meant client and internal requests could not be serviced as required
• Legacy system incurred large vendor licence fees
• Slow response to change due to lack of continuous integration/deployment
• Limited visibility into system metrics and client usage

PROJECT OUTCOMES

Using AWS Serverless architecture and the components detailed below, Data Army were able to migrate the current CoreLogic matching system into the AWS cloud, and achieve the following business outcomes:

• Improved customer satisfaction, system is always available, sub-millisecond response times to calls now achieved
• De-coupled serverless architecture allowing for independent scaling of compute components, as well as resilience to quickly recover from failure
• Potential 45% reduction in monthly fees (average AWS monthly charge versus Oracle standard edition licence cost) + reduction in internal resources

THE DATA ARMY SOLUTION

Data Army decided on an AWS serverless architecture to meet the scaling requirements identified. API calls to the matching service were unpredictable and sporadic, and in some cases required the matching of many millions of records. The solution had to be scaled at speed to allow for many concurrent calls to be made and results returned, without incurring large costs of managing pre-warmed servers that could be left idle for long periods.

AWS Lambda was chosen for its ability to scale at speed and be hyper-ready to respond without any of the costly overheads of managing a fleet of servers. Adding SQS allowed for a de-coupling of the application components to increase fault tolerance and allow for continuous deployment with reduced risk of loss of data. DynamoDB was selected as a further serverless component to run the matching, whilst S3 provided the backbone storage.