In this project, we were hired to help our Client (a multi-site international pharma) mirror in-house in vitro assay result and annotation databases to a data warehouse for data analytics and advanced visualization.
As with most in vitro (primary and secondary screening) assay result repositories, data analytics and visualization applications require data transformation in order to run efficiently. Connecting to raw data does not work well since the transformations, joins, or lookups required to summarize data do take time, slowing down user queries. Collaboratively with our Client's research informatics team, we designed and implemented an automated ETL workflow which mirrors the source data (transforming it along the way) into a warehouse used by endpoint visualization and analytics applications.
In order to guarantee data availability and database integrity during the database updates which occur across the Amazon cloud and our Client's on-premises servers, we implemented the workflow in a single database transaction with auto-rollback or final commit. We also put in place detailed logging to help monitor the workflow over time.
Our Client's scientists can now focus on their data analytics tasks without having to wait on every click. We helped improve productivity, saving our Client many hours of valuable work time.
Contact us (sales_at_saberinformatics_dot_com) or call us to discuss in confidence the challenges your organization is facing and how we can address them together.
