Resolving Scalability Issues with AWS Serverless
CoreLogic is the largest provider of property information, analytics and property-related risk management services in Australia and New Zealand.
AWS, Migration, Serverless, DevOps
As a well-established and respected information and analytics provider, they rely on many core systems to service both their internal and client needs. Over the years, CoreLogic has continued to grow with customers expecting faster, more in-depth insights and systems being required to cope with ever-increasing demand.
Specialising in property, one of their core systems used throughout the New Zealand business was their property matching solution, held in an on premise Oracle database. Data Army were employed to redesign this solution to operate within the AWS cloud, removing issues that had arisen from lack of scalability, visibility, reporting and deployment.
See CoreLogic website to understand what they do: corelogic.com.au
- Scaling limitations of the legacy system meant client and internal requests could not be serviced as required
- Legacy system incurred large vendor licence fees
- Slow response to change due to lack of continuous integration/deployment
- Limited visibility into system metrics and client usage
“Data Army came in and understood the problem that we were trying to solve and some of the scalablity issues that we were having. Data Army was able to build out a solution in some of Amazon’s services that more than met our needs.”
— Glynn Willis, CoreLogic Australia
Using AWS Serverless architecture and the components detailed below, Data Army were able to migrate the current CoreLogic matching system into the AWS cloud, and achieve the following business outcomes:
- Solution elastically scales to manage unpredictable usage whilst improving latency – improved customer experience, greater system resilience, reduced internal resource, reduced risk
- Full system logging to improve visibility into system usage to aid development and enhance internal analytics
- CI/CD pipeline for improved business agility – quickly deploy changes with reduced risk
- Improved customer satisfaction, system is always available, sub-millisecond response times to calls now achieved
- De-coupled serverless architecture allowing for independent scaling of compute components, as well as resilience to quickly recover from failure
- Potential 45% reduction in monthly fees (average AWS monthly charge versus Oracle standard edition licence cost) + reduction in internal resources
THE DATA ARMY SOLUTION
Data Army decided on an AWS serverless architecture to meet the scaling requirements identified. API calls to the matching service were unpredictable and sporadic, and in some cases required the matching of many millions of records. The solution had to be scaled at speed to allow for many concurrent calls to be made and results returned, without incurring large costs of managing pre-warmed servers that could be left idle for long periods.
AWS Lambda was chosen for its ability to scale at speed and be hyper-ready to respond without any of the costly overheads of managing a fleet of servers. Adding SQS allowed for a de-coupling of the application components to increase fault tolerance and allow for continuous deployment with reduced risk of loss of data. DynamoDB was selected as a further serverless component to run the matching, whilst S3 provided the backbone storage.