In many cases, uncensored models that have not undergone the RLHF phase actually perform better than aligned models.
OpenAI has recognized that one of the most important reasons behind the success of ChatGPT is reinforcement learning from human feedback (RLHF). Everyone agrees on this point. Since then, RLHF has been used to build models.
By training LLM through interaction with human evaluators, RLHF attempts to improve the performance of AI models in real-world applications. However, it can introduce biases and reduce the robustness of the models. A recent paper titled "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback" by researchers from Harvard University, Stanford University, Massachusetts Institute of Technology, University of California, Berkeley, and many other universities discusses the issues with RLHF methods.
Not bad, but not the best
The paper points out that obtaining high-quality feedback from human evaluators is one of the main challenges of RLHF. While humans can provide valuable feedback, they are also susceptible to various limitations and biases. Inconsistent evaluators may struggle to understand the context or objectives of AI models, resulting in suboptimal feedback. The complexity of supervision, especially in long conversations, also hinders accurate evaluation of model performance.
Furthermore, data quality is another key issue. Due to limited attention, time constraints, and cognitive biases, human evaluators may unintentionally provide inconsistent or inaccurate feedback. Even diligent evaluators can have disagreements due to subjective interpretations and different perspectives.
The feedback forms used in RLHF can further exacerbate these challenges. Depending on the evaluation method, evaluators may provide binary judgments, rankings, or comparisons, each with its own advantages and disadvantages. Choosing the most suitable feedback form for a specific AI task is complex, leading to potential variations in the training process.
A fundamental problem of RLHF is accurately representing human values through a reward function. Human preferences are context-dependent, dynamic, and often influenced by social and cultural factors. Designing a reward function that encompasses the complexity of human values is a daunting task. Incorrect assumptions about human decision-making or the use of reward models that ignore individuality and contextual dependence can lead to inconsistent AI models.
Why so many adjustments?
The diversity of human evaluators further complicates the reward modeling process. Different evaluators may have different preferences, expertise, and cultural backgrounds. Attempting to integrate their feedback into a single reward model may overlook important divergences and result in biased AI models that favor majority opinions. This can be detrimental to underrepresented groups and perpetuate existing social biases.
To address these challenges, researchers must explore techniques that capture preferences in a more nuanced and context-aware manner. Utilizing comprehensive reward models that consider feedback from multiple evaluators or personalized reward models that cater to individual preferences can help capture the diversity of human values.
Addressing potential biases in the data collection process in a transparent manner and conducting thorough evaluations to identify and mitigate harmful biases are necessary steps for responsible AI development.
To overcome these data limitations, researchers should explore economically efficient data collection methods that do not compromise data quality and diversity. Training on GPT output data to align faster has become a new trend, but ultimately, it can introduce the same biases to other models. So far, there is no better method.
The fundamental challenges of RLHF have significant implications for AI alignment. Some problems may have feasible solutions through technological advancements, but others may require alternative approaches. Researchers must approach AI alignment with caution when relying solely on RLHF, as this method may not fully address certain issues.
Essentially, RLHF can lead to excessive fine-tuning of models, potentially hindering their functionality. This phenomenon is known as the alignment tax of AI models. When a model undergoes several benchmark tests, humans in the loop try to make the model as consistent and "politically correct" as possible, resulting in a significant reduction in performance.
The alignment tax is the additional cost that AI systems must incur to maintain consistency, at the expense of missing the opportunity to build an unaligned or uncensored model, which can ultimately hinder its performance. That's why, in many cases, uncensored models that have not undergone the RLHF phase actually perform better than aligned models.
Source: https://analyticsindiamag.com/ai-alignment-is-a-joke/
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.