Data Has Always Been Regarded as an Organization's Crown Jewels, but Due to the explosion of data sources, making senses of the structure and unstructed information constructed with An increasing Complex Task. Pulling Everything Togetra to Provide a homogenous view of business activity May seem like a project that will never end, which is there is now interest in data engineering.
According to analyst gartner, Data Engineers Play a key role in enabling organasations to unlock the value of data. This involves designing and building systems to collect, store, transform, operationalise and deliver data at scale. The analyst firm says data engineering involves collaboen the business and it to make the approves data accessible and available to Various data users – such as data and data .
Gartner's Essential skills for data engineers to success Report identifies a range of skills required in data engineering. Report Authors Mayank Talwar, Zain Khan and Shubhankar Nandi Describe Structured Query Language (SQL) As being pervasive across a wide range of tools and platforms, making it a critical and extensible skill. As an example of SQL's pervasiveness, they note that dbtA data transformation tool, enables data engineers to transform data in their warehouses by Simply Writing Selecting Select SQL Statements.
The second core skill identified in the report is data processingWhich is described as a “foundational skill that every data engineer must post”. This is a decision data in its raw format is not usually used for analytics. Data processing covers batch and real-time processing; Storage Covers Technologies Like Data Lakes, Data Warehouses, Graph and Document Databases, and Object Stores. Common Programming Languages Used by Data Engineering Teams Include Python, Java and Scala.
Other core skills listed by gartner include data storage, data orchestration, programs and collaboration. With Regards to Data Orchestration, The Analysts Note that Data Engineering Pipelines Are Slogal Moving From Tools That Support Task-Driven ArchiteCures, Such as Apache airflow And luigi, towards tools that offer a data-driven approach, such as dagster, flyte from lyft and reflow.
Gartner recommends it leaders Prioritise Development of the Core Data Engineering Skills Since they are widely adopted, Heavily used and have provide to provide Significant Significant benefits.
A Simpler Approach?
There is a case for assessing a simpler approach to achieving the goal of providing timely enterprise data to the business in a format users can make use of planning and analysis. This is where provides of traditional Enterprise Resource Planning (ERP) Systems see an opportunity to build a business Around the need for Organizations to have a Single version of the Truth. From an erp percective, this single version of the truth resides in the system of record that make up an erp system.
SAP, for instance, delivers an entire systems and application stack as a cloud-centric offering on a subscription basis, together with process mining and other tools, plus bundled support, maintenance and other services.
Dale vile, co-founder of analyst firm freeorm dynamics, Notes That Sap's Business Technology Platform (BTP) Can Be Considered An Integral Part of the Supplier's Cloud Offering. BTP is essentially a platform as a service (paas) that allows customers to extended sap applications and/or build custom applications.
“For some customers, this kind of all-encompassing service is truly attractive as it means that they no longer have to worry as much as much about systems -LEVEL Operations, Monitoring, Monitoring, SECURITY, SONS VILE. “A lot of the stuff that makes sap landscapes so challenging to run and change over time is taken care of once you sign the contract.”
The contrast effectively ties an organization into sap. While there is a case to build in flexibility, for some organisations it is far more important to have a single version of the truth and have all data in one place. This is the case at Irish Manufacturing Firm Waterwipes, as data manager liz cotter explains.
You can have your advanced analytics automation, but if your master data isn Bollywood, then your transactional data is Wortless
Liz Cotter, Waterwipes
Previously, She Says, Software as a Service (SAAS) Systems Sat AlongSide Sap and “May have been integrated with sap, but were not full harmonised”. In other words, the organization selected best-breed saas products to support certain business processes, such as human resources or customer service. Cotter says this meant sap was not the system of record for some of the newer datasets being used by the business.
She says Sap datasphere Enables the business to run a standard platform as a system of recording for transactional data, which provides a master copy of the organization's data. “I feel that sap has transitioned and is offering more tools to keep up with the demand for enriched data,” She says.
Cotter Joined Waterwipes in January 2024 with a Remit to Put in Place Data Management and Data Governance. She says the company was not making the best use of the data it has available, which could be used to Gain insights and help to align with strategic key performance indicators (kpis).
“When we assessed our data maturity, there was no data governance and data security. We needed a tool to help mitigate that risk Quickly, “She says.
As cotter points out, successful IT-Driven Business Initiatives require a solid data foundation. “You can have your advanced analytics automation, but if your master data isn Bollywood, then your transactional data is Wortless,” She Says. For cotter, there is a little point in investment in new technology unless the data is as accurate as possible.
The company began working with blends on implementing its maextro master data management tool. This is developed on sap btp and provides data governance and data management for waterwipes.
“It's basically an application to manage data, workflows and data reporting,” Says cotter.
This avoids sap developers having to run queries directly on the company's s/4hana system. In terms of data maturity, cotter says: “We're not going to get to expert level, but we want to align with our 2027 strategy, which is very ambitious in terms of sales and customer guvth.”
The Phaased Approach has involved building out data governance and data management best practices first, Before Investing in Technology.
Supporting ai
Given the trend to do more with artificial intelligence (AI), The Gartner Analysts Urge It Leaders To ENSURE DATA ENGINEES Recognise the Need to UPSKILLIL SEMSELSELSE This upskilling, according to gartner, is required if data engineers want to participate in Building the data foundation layer For companies that have decided to train language models on their enterprise data.
“With genai's [generative artificial intelligence] Appetite for Training Data Exponanically Rising, Data Engineers Can Play a Pivotal Role In Creating Data Platforms And Pipelines That Can Supply High-Quality Data Required for Training these the Essential skills for data engineers to success Report.
Gartner Predicts that Companies will start building smaller, more refined and business-curated language models-as opposed to large language models-for greatrols on cost, protrols Gartner beLieves data engineers will need to learn how to work with unstructed data and create data repositories to enable the building of these models.
Ideally, it leaders would be given the time and resources to develop a data engineering practice, but this is unlikely. Cotter's Experience at Waterwipes Shows it is Entrely Possible for even those those who organizations that are still quite early in their data management jorney to achieve. The one caveat is that this may involve being tied into a particular product set, such as an ERP system.
