Legacy Software Modernization For a US Top Market Research Holding

How shifting from an outdated system to Apache Spark and AWS cloud services allowed a leading market research company to speed up data processing threefold and decrease software maintenance costs by 34%.

Challenge

Succeeding in today’s market is tough. Innovation is where breakthrough opportunities reside. But an innovative approach cannot be implemented without using up-to-date technologies.

Our customer embraces innovation just as we do. They needed to jump on the new technology bandwagon to operate optimally. That’s why they decided to migrate from legacy technologies to modern cloud infrastructure.

Tasks

  • Migration to Spark. Spark is used in a wide range of organizations for large-scale data processing. It allows running workloads faster, writing applications quickly in Java, Scala, Python, R, and SQL, combining SQL, streaming and complex analytics. Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. It can access diverse data sources.
  • Validation of business logic, which is critical to keep things running smoothly. Publishing data is actually pretty easy. There are a lot of ways to do that. But securing and organizing your data is an important task that requires some thought.
  • Data correctness verification. Accurate data is the backbone of any database. Rigorous, objective and transparent verification processes are vital to establishing and maintaining high-quality data.
  • Performance improvement in general and reducing data processing time in particular, which, by the way, speak for themselves.

Solution

The solution involved taking the data prepared by the company’s different teams from the cloud storage (S3) and executing the project code to message, consolidate and filter data to perform further calculations. Code execution results are recorded in a file that can be used either for the company’s other projects or for making reports.

The solution includes:

  • orchestration of all jobs (tasks);
  • migration to AWS cloud services;
  • refactoring of the code to handle increased data volume;
  • the possibility of sharing the transformed data with different teams and using it within various company projects.

Key features

44A67049-CAEF-4DF3-87DB-A1DEDDB1514DCreated with sketchtool.

Data verification

(data qualitatively and clearly coincides with what is expected)

7F23E3B4-9AD3-4792-B359-A53D00BFD643Created with sketchtool.

Each job (task) gives the expected result

C9B9E0C9-4E91-4340-B471-6F667306DE6ECreated with sketchtool.

Runtime reduction

EE6D800B-4722-480E-876F-D61820151CD6Created with sketchtool.

Maintenance cost reduction

B897A2A3-5514-4C5F-B533-1BEC4DF9429DCreated with sketchtool.

Modifiability improvement

Value

Thanks to the solution implemented by *instinctools, the users got an opportunity to manage the revisions of documents and correlate them with all the necessary regulations in a moreefficient way. It allowed our customer to attract more clients, which increased the business profitability

Technologies

Scala
HDFS
AWS EMR
AWS Lambda
AWS RDS
AWS SQS
Spark
Grafana
S3
Kubernetes
Airflow

Legacy software modernization

is the key to adapt, grow and remain competitive for any organization. In close cooperation with your company, we will work out the best strategy for this process!

Do you have a similar project idea?

Anna Vasilevskaya
Anna Vasilevskaya Account Executive

Get in touch

Drop us a line about your project at contact@instinctools.com or via the contact form below, and we will contact you soon.