"Generative Artificial Intelligence and Machine Learning: Pioneers of the Data Quality Revolution"

2023-06-05

"Generating AI and ML Fueling a Revolution in Data Quality" "By automating and simplifying data management tasks in unprecedented ways, generative artificial intelligence technology is set to completely transform the way organizations handle data. A new revolution in data quality is underway, supported by models that utilize generative artificial intelligence and machine learning technologies such as ChatGPT. While AI has primarily been applied to predictive analytics in the data industry, we are now entering a creative era of generative AI, where powerful natural language processing, data analysis, and automation tools will shape the future of data management and data quality. AI programs in the data industry have been in use since the 1950s and 1960s, employing rule-based systems, symbolic reasoning, and expert systems to infer and generate insights from data. Today, the use of AI is accelerating: according to the Data and Analytics Leadership Annual Survey 2023, 80.5% of data executives believe that AI/ML will be the area of investment growth in data and analytics in 2023, with 16.3% of organizations considering it their top investment priority, followed by 10.6% of organizations focusing on data quality. "Data quality is a crucial aspect of data management," explains Davide Pelosi, Solution Engineering Manager at Talend, a leader in data integration and data integrity. "It ensures that enterprises can make informed decisions based on accurate, complete, and consistent information. Poor data quality can lead to decision-making errors, revenue loss, and damage to brand reputation. "Fortunately, software solution providers are leading the way with innovative data quality tools and technologies that help enterprises quickly and efficiently identify and address data quality issues," he says. "However, there is still much work to be done. In a recent survey, 97% of Talend respondents stated that they face challenges when using data, with data quality being their top concern. This indicates a significant opportunity for improvement, and the rewards are substantial for companies that excel in this area." "The Future of Data Quality Workflow" According to a report by Gartner, at least 50% of data management tasks will be automated by 2025. Most of this automation will be driven by AI/ML, such as generative language models, signaling the need for traditional data management techniques to take a back seat as ChatGPT and other generative language applications are poised to disrupt the market. These technologies have already made waves in the business world, from automating content creation to development tasks, and their impact on data management and data quality initiatives is exciting. "By automating and simplifying data management tasks in unprecedented ways, these technologies have the potential to completely change the way organizations handle data," comments Pelosi. "The prospects of further automation and efficiency mean that enterprises can ensure the accuracy, completeness, and consistency of their data more easily than ever before. Let's take the data quality workflow as an example. First, technical data quality assessments can be performed using machine learning algorithms to identify anomalies and quantify the severity of issues. Based on the assessment results, generative language models can suggest data quality rules and transformations in natural language text, which can be easily understood by business stakeholders. From there, these proposed rules can be reviewed and validated by data quality experts and business stakeholders, who can accept or reject the rules or propose modifications to better align with their business requirements. "Enterprises can also create additional business rules using natural language queries without the need for complex UI," adds Pelosi. "For example, enterprise users might ask, 'Please raise the acceptable age for drinking to 18 and flag all non-compliant individuals as not eligible for the spring marketing campaign,' just like what we do with Alexa today. Once the rules are accepted, they can be transformed into executable code, such as Python or SQL, using a similar template-based approach. Of course, testing and validation with data samples are necessary before deploying the code to a production environment to ensure that the rules work as expected and meet data quality metrics. However, once completed, the cleansed data can be used for various downstream tasks, from data analysis and visualization to machine learning and business intelligence. "Imagine a world where data management and quality undergo a significant transformation, and we can witness it firsthand. While the use of generative language models in this field is still in its early stages and being researched by industry experts, there have already been some astonishing research projects and prototypes showcasing the incredible potential of this technology." "Cautions in Addressing Current Limitations" With its human-level performance in various professional and academic benchmark tests, the latest version of OpenAI's GPT technology, GPT-4, is impressive. However, GPT-4 shares similar limitations with earlier GPT models. The OpenAI team states, "Most importantly, it is still not entirely reliable. Extreme caution should be exercised when using language model outputs, especially in high-stakes situations, and exact protocols (such as human review, contextual grounding, or complete avoidance of high-risk uses) should be matched to the specific needs of the use case." Aaron Kalb, Chief Strategy Officer and Co-founder of Alation, reiterates this point: tools like GPT should not be trusted to provide recommendations for important decisions. "This is because it is designed to generate content that appears correct and has great flexibility and fluency, which creates a false sense of credibility and may lead to the so-called 'illusion' of artificial intelligence. As Kalb (a founding team member behind Apple's groundbreaking Siri voice assistant) explains, the very authenticity and ease of use that make GPT so enticing are also its most apparent limitations: "GPT sounds smart and impressive. The problem is that it still doesn't know what it's talking about. It doesn't try to express knowledge in language. It's really good at knowing what word comes next after previous words 'feel right' because it has effectively read and memorized the entire internet. It often gets the right answers because for many questions, humans have collectively posted the answers online." Source: https://aimagazine.com/articles/generative-ai-and-ml-fuelling-a-revolution-in-data-quality"

RECENT AI NEWS

RECENT AI TOOLS