[AINews] Chameleon: Meta's (unreleased) GPT4o-like Omnimodal Model • ButtondownTwitterTwitter

This section provides a recap of discussions and developments in the AI Discord community. It covers a range of topics including investments in GPUs by Hugging Face, departures from OpenAI's alignment team, debates on GPT-4o capabilities, challenges presented by the NIAN benchmark, and discussions on model advancements and multimodal capabilities. The section also highlights the anticipation for Meta's multimodal model in Discord communities and conversations around funding GPU usage, boosting models with the OpenHermes dataset, effects of removing refusal mechanisms from Language Models, and solutions for troubleshooting Llama 3 errors.

The tech community reacts to the FTC's decision to ban noncompetes, potentially reshaping the industry. Engineers debate the pros and cons of open source versus proprietary employment, exploring the impact on contributions and salaries. It's a vibrant discussion reflecting the evolving dynamics within the tech sector.

The Discord channels are buzzing with a variety of updates and discussions in the AI community. From issues with rich text translation to the launch of new datasets, from concerns about data privacy to the development of next-gen AI fusion models, the conversations cover a wide array of topics. Various platform updates, API limitations, and technical challenges are being shared and discussed among members. The community is also excited about upcoming events like the SF Generative AI Summit and Snowflake Dev Day. Good-natured banter and shared resources add a sense of camaraderie and collaboration among the participants.

Unsloth AI (Daniel Han) Channel Summaries

• General Discussions: Members discuss various topics including delay frustrations, hardware requirements, skepticism coping mechanisms, training resources, and off-topic debates.

• Random Channel: Discussions on training losses with Llama3, RAM issues with ShareGPT dataset, finding similarly formatted text tools, and opinions on Sam Altman's leadership.

• Help Channel: Troubleshooting an AttributeError while training Llama 3 on custom data, a driver's manual for RAG, PyPDF2 vs. PyPDF, GGUF model conversion issues, and CUDA compatibility challenges on Databricks platform.

GPT-4o Slows Over Time:

Members observed that as conversations with GPT-4o get longer, the inference speed significantly drops, sometimes resulting in the model halting mid-inference. This issue was noted by users on different platforms including the Mac app and the website, with discrepancies in performance.

GPT-4o Tops Rankings:

The updated LMSYS arena rankings show that GPT-4o has claimed the top position. One user enthusiastically noted, 'gpt4o top 1'.

Image Input to GPT-4o:

Users discussed how to send images to GPT-4o, confirming that it's possible through API or by sending the image directly. Instructions and detailed documentation can be found here.

Custom GPTs Upgrading to GPT-4o:

Some users realized that their custom GPTs had already transitioned from GPT-4 Turbo to GPT-4o, evident from the improved response speed. This change appears to be in the rollout stage with varying availability.

Free vs. Plus Access to GPT-4o:

The rollout of GPT-4o is not region-specific and is gradually becoming available to more users, with Plus users receiving priority. Despite its enhanced capabilities, the transition and access limits have caused some confusion and mixed experiences among users.

Ontological Drill Help Sought:

A user asked for a powerful ontological drill but felt their current one lacked enough 'oomph.' They shared a detailed example of their prompt structure.

Markdown in AI Prompts:

A user inquired about using markdown in prompts for AI, and another confirmed that the model responds well to markdown, emphasizing that guiding the AI's attention is crucial.

Dynamic Character Roles in AI:

Techniques for programming multiple character roles within AI using markdown were discussed. A user shared a comprehensive prompt example involving various characters in a theater-like scenario. Prompt Example

Troubleshooting Function Calls with GPT-3.5:

Issues with GPT-3.5's function calls returning random data were discussed. The solution proposed involved reframing instructions to focus on using actual provided data only.

Issues with GPT-4o and Rewriting:

Several users noted that GPT-4o tends to rewrite original prompts rather than adjust according to feedback. Discussions included guidance on clear positive instruction and avoiding negatives like prohibitions against calculations.

• In this section, a user shared a YouTube video showcasing their efforts to create a business advisor using LangChain and Gemini AI startup.
• Another user discussed challenges with GPT-4o and grid puzzle generation, seeking an open-source model for better results.
• Additionally, a user linked a LinkedIn post about building a study companion with GenAI to innovate in education.
• Finally, one user explored controlnet training and shared a linked image related to their project.

Nous Research AI

• EOS token not stopping fine-tuning model: Members discussed issues with the eos_token_id not stopping the generation on a fine-tuned Qwen 4B model.
• Nous Hermes model replies in Chinese: Users reported the Nous-Hermes-2-Mixtral-8x7B-DPO model returning responses in Chinese instead of English summaries.
• Regex vs. semantic search for text patterns: Discussion on finding text with specific formatting efficiently using semantic search or regex.
• GPT-4o for symbolic language in algebra: Utilizing GPT-4o to create a symbolic language for integrals and derivatives.

Automated Knowledge Graphs with DSPy and Neo4j

• An LLM-driven project on automated knowledge graph construction from text shared.
• GitHub repository: chrisammon3000/dspy-neo4j-knowledge-graph

Users discuss troubleshooting different issues in LM Studio, including glibc problems during installation, embedding models for RAG in Pinecone, and false antivirus warnings. Discussions also cover comparing model performance, quantization challenges, and memory overclocking limits. Additionally, users seek recommendations for medical LLMs and explore options for reducing compile times.

Interconnects (Nathan Lambert): Lectures, Projects, and Updates

• Channel Renamed to 'Lectures and Projects': The channel has been renamed to better reflect its focus areas.
• New Lecture Video Released: Nathan Lambert shared a YouTube video on 'Stanford CS25: V4 I Aligning Open Language Models' released on April 18, 2024.
• Upcoming Technical Project 'Life after DPO': Lambert announced a new technical project titled 'Life after DPO,' with further details pending.

Different Discussions in the AI Community

• Members discussed training on data from previous tasks as a state-of-the-art approach.
• Criticism was made on semantic text similarity metrics and a proposal for a Hierarchical Memory Transformer was highlighted.
• Ideas were shared on audio and video tokenization strategies.
• Conversations took place on seeking MLP-based attention approximations, compute costs in data preprocessing, and critiquing papers lacking hyperparameter search.
• Other topics included rich text translation challenges, Hugging Face's GPU donation, concerns over Slack's data usage, emerging multimodal models, and resignation of OpenAI's head of alignment.

GPT-4o & LlamaParse shine in document parsing:

• Introducing GPT-4o, a state-of-the-art model for multimodal understanding, showcasing superior document parsing capabilities. LlamaParse utilizes LLMs to extract documents efficiently.

Revamped LlamaParse UI offers more options:

• The LlamaParse user interface has been significantly revamped to display an expanded array of options.

First-ever in-person meetup announced:

• LlamaIndex announced their first-ever meetup at their new San Francisco office in collaboration with Activeloop and Tryolabs to discuss the latest in generative AI and the advancements in retrieval augmented generation engines.

Structured Image Extraction with GPT-4o:

• A full cookbook demonstrates how to extract structured JSONs from images using GPT-4o, which outperforms GPT-4V in integrating image and text understanding.

Handling large tables without hallucinations:

• Addressing the issue of LLMs hallucinating over complex tables, the example of the Caltrain schedule illustrates poor parsing and the ongoing challenge.

The sections highlighted various discussions and topics related to AI in different Discord channels. From advancements in language models to issues with specific AI technologies, many members shared their insights and experiences. These ranged from implementing memory in chatbots and addressing indexing problems in databases to challenges with streaming output. Additionally, announcements about new projects like the AI Reality TV platform and the sharing of helpful resources like the Adrenaline app for learning repositories were also featured. Moreover, the conversation delved into technical aspects such as CUDA kernel optimization, symbolic algebraic functions, compute graph operations, and the permissibility of modifying implementations in frameworks like Tinygrad.

• Riley Goodside calls out GPT-4o shortcomings: Riley Goodside showcased GPT-4o's failures on ChatGPT, noting it didn't meet expectations set by OpenAI’s demo.
• Google's AI stumbles despite announcements: During Google I/O, several hallucinations occurred during the keynote, contradicting Google's claims.
• A Plea for Sober AI: A blog post advocated for a grounded approach toward AI. 0xgrrr mentioned their product Alter, focusing on practical AI solutions.
• Community echoes sentiments: Multiple members appreciated the blog post for articulating frustrations with current AI hype.