Artificial Intelligence Alignment May Not Be Necessary

2023-08-02

In many cases, uncensored models that have not undergone the RLHF phase actually perform better than aligned models. OpenAI has recognized that one of the most important reasons behind the success of ChatGPT is reinforcement learning from human feedback (RLHF). Everyone agrees on this point. Since then, RLHF has been used to build models. By training LLM through interaction with human evaluators, RLHF attempts to improve the performance of AI models in real-world applications. However, it can introduce biases and reduce the robustness of the models. A recent paper titled "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback" by researchers from Harvard University, Stanford University, Massachusetts Institute of Technology, University of California, Berkeley, and many other universities discusses the issues with RLHF methods. Not bad, but not the best The paper points out that obtaining high-quality feedback from human evaluators is one of the main challenges of RLHF. While humans can provide valuable feedback, they are also susceptible to various limitations and biases. Inconsistent evaluators may struggle to understand the context or objectives of AI models, resulting in suboptimal feedback. The complexity of supervision, especially in long conversations, also hinders accurate evaluation of model performance. Furthermore, data quality is another key issue. Due to limited attention, time constraints, and cognitive biases, human evaluators may unintentionally provide inconsistent or inaccurate feedback. Even diligent evaluators can have disagreements due to subjective interpretations and different perspectives. The feedback forms used in RLHF can further exacerbate these challenges. Depending on the evaluation method, evaluators may provide binary judgments, rankings, or comparisons, each with its own advantages and disadvantages. Choosing the most suitable feedback form for a specific AI task is complex, leading to potential variations in the training process. A fundamental problem of RLHF is accurately representing human values through a reward function. Human preferences are context-dependent, dynamic, and often influenced by social and cultural factors. Designing a reward function that encompasses the complexity of human values is a daunting task. Incorrect assumptions about human decision-making or the use of reward models that ignore individuality and contextual dependence can lead to inconsistent AI models. Why so many adjustments? The diversity of human evaluators further complicates the reward modeling process. Different evaluators may have different preferences, expertise, and cultural backgrounds. Attempting to integrate their feedback into a single reward model may overlook important divergences and result in biased AI models that favor majority opinions. This can be detrimental to underrepresented groups and perpetuate existing social biases. To address these challenges, researchers must explore techniques that capture preferences in a more nuanced and context-aware manner. Utilizing comprehensive reward models that consider feedback from multiple evaluators or personalized reward models that cater to individual preferences can help capture the diversity of human values. Addressing potential biases in the data collection process in a transparent manner and conducting thorough evaluations to identify and mitigate harmful biases are necessary steps for responsible AI development. To overcome these data limitations, researchers should explore economically efficient data collection methods that do not compromise data quality and diversity. Training on GPT output data to align faster has become a new trend, but ultimately, it can introduce the same biases to other models. So far, there is no better method. The fundamental challenges of RLHF have significant implications for AI alignment. Some problems may have feasible solutions through technological advancements, but others may require alternative approaches. Researchers must approach AI alignment with caution when relying solely on RLHF, as this method may not fully address certain issues. Essentially, RLHF can lead to excessive fine-tuning of models, potentially hindering their functionality. This phenomenon is known as the alignment tax of AI models. When a model undergoes several benchmark tests, humans in the loop try to make the model as consistent and "politically correct" as possible, resulting in a significant reduction in performance. The alignment tax is the additional cost that AI systems must incur to maintain consistency, at the expense of missing the opportunity to build an unaligned or uncensored model, which can ultimately hinder its performance. That's why, in many cases, uncensored models that have not undergone the RLHF phase actually perform better than aligned models. Source: https://analyticsindiamag.com/ai-alignment-is-a-joke/

RECENT AI NEWS

RECENT AI TOOLS