Close Menu
Beverly Hills Examiner

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Biden says he wouldnt have run for a second term in 2024 if he knew he had cancer

    November 16, 2026

    John Summit Says Hes Never Submitting to the Grammys Again, Calls Dance Category Nominations A F–king Joke

    November 16, 2026

    Trump Slams 60 Minutes After Norah ODonnells Shocking Iran War Report

    November 16, 2026
    Facebook X (Twitter) Instagram
    Beverly Hills Examiner
    • Home
    • US News
    • Politics
    • Business
    • Science
    • Technology
    • Lifestyle
    • Music
    • Television
    • Film
    • Books
    • Contact
      • About
      • Amazon Disclaimer
      • DMCA / Copyrights Disclaimer
      • Terms and Conditions
      • Privacy Policy
    Beverly Hills Examiner
    Home»Technology»Apple Engineers Show How Flimsy AI ‘Reasoning’ Can Be
    Technology

    Apple Engineers Show How Flimsy AI ‘Reasoning’ Can Be

    By AdminOctober 16, 2024
    Facebook Twitter Pinterest LinkedIn WhatsApp Email Reddit Telegram
    Apple Engineers Show How Flimsy AI ‘Reasoning’ Can Be

    [ad_1]

    For a while now, companies like OpenAI and Google have been touting advanced “reasoning” capabilities as the next big step in their latest artificial intelligence models. Now, though, a new study from six Apple engineers shows that the mathematical “reasoning” displayed by advanced large language models can be extremely brittle and unreliable in the face of seemingly trivial changes to common benchmark problems.

    The fragility highlighted in these new results helps support previous research suggesting that LLMs’ use of probabilistic pattern matching is missing the formal understanding of underlying concepts needed for truly reliable mathematical reasoning capabilities. “Current LLMs are not capable of genuine logical reasoning,” the researchers hypothesize based on these results. “Instead, they attempt to replicate the reasoning steps observed in their training data.”

    Mix It Up

    In “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models”—currently available as a preprint paper—the six Apple researchers start with GSM8K’s standardized set of more than 8,000 grade-school level mathematical word problems, which is often used as a benchmark for modern LLMs’ complex reasoning capabilities. They then take the novel approach of modifying a portion of that testing set to dynamically replace certain names and numbers with new values—so a question about Sophie getting 31 building blocks for her nephew in GSM8K could become a question about Bill getting 19 building blocks for his brother in the new GSM-Symbolic evaluation.

    This approach helps avoid any potential “data contamination” that can result from the static GSM8K questions being fed directly into an AI model’s training data. At the same time, these incidental changes don’t alter the actual difficulty of the inherent mathematical reasoning at all, meaning models should theoretically perform just as well when tested on GSM-Symbolic as GSM8K.

    Instead, when the researchers tested more than 20 state-of-the-art LLMs on GSM-Symbolic, they found average accuracy reduced across the board compared to GSM8K, with performance drops between 0.3 percent and 9.2 percent, depending on the model. The results also showed high variance across 50 separate runs of GSM-Symbolic with different names and values. Gaps of up to 15 percent accuracy between the best and worst runs were common within a single model and, for some reason, changing the numbers tended to result in worse accuracy than changing the names.

    This kind of variance—both within different GSM-Symbolic runs and compared to GSM8K results—is more than a little surprising since, as the researchers point out, “the overall reasoning steps needed to solve a question remain the same.” The fact that such small changes lead to such variable results suggests to the researchers that these models are not doing any “formal” reasoning but are instead “attempt[ing] to perform a kind of in-distribution pattern-matching, aligning given questions and solution steps with similar ones seen in the training data.”

    Don’t Get Distracted

    Still, the overall variance shown for the GSM-Symbolic tests was often relatively small in the grand scheme of things. OpenAI’s ChatGPT-4o, for instance, dropped from 95.2 percent accuracy on GSM8K to a still-impressive 94.9 percent on GSM-Symbolic. That’s a pretty high success rate using either benchmark, regardless of whether or not the model itself is using “formal” reasoning behind the scenes (though total accuracy for many models dropped precipitously when the researchers added just one or two additional logical steps to the problems).

    The tested LLMs fared much worse, though, when the Apple researchers modified the GSM-Symbolic benchmark by adding “seemingly relevant but ultimately inconsequential statements” to the questions. For this “GSM-NoOp” benchmark set (short for “no operation”), a question about how many kiwis someone picks across multiple days might be modified to include the incidental detail that “five of them [the kiwis] were a bit smaller than average.”

    Adding in these red herrings led to what the researchers termed “catastrophic performance drops” in accuracy compared to GSM8K, ranging from 17.5 percent to a whopping 65.7 percent, depending on the model tested. These massive drops in accuracy highlight the inherent limits in using simple “pattern matching” to “convert statements to operations without truly understanding their meaning,” the researchers write.

    [ad_2]

    Original Source Link

    Share. Facebook Twitter Pinterest LinkedIn WhatsApp Email Reddit Telegram
    Previous Article6G phone networks could be 9000 times faster than 5G
    Next Article Aaron Judge snaps postseason slump, Yankees take 2-0 lead over Guardians in ALCS

    RELATED POSTS

    Apple launches new app to study how people use AI for health | TechCrunch

    November 16, 2026

    TechCrunch Mobility: The Nvidia backup plan for Chinas self-driving industry | TechCrunch

    November 15, 2026

    Where was Nvidias security team? | TechCrunch

    November 15, 2026

    Asian governments react to Nvidia CEO Jensen Huangs exit | TechCrunch

    November 14, 2026

    Amazons new Kindle Colorsoft is a premium e-reader with a premium price | TechCrunch

    November 14, 2026

    ChatGPTs new Wallet lets AI agents shop and check out for you | TechCrunch

    November 13, 2026
    latest posts

    Biden says he wouldnt have run for a second term in 2024 if he knew he had cancer

    ATLANTA (AP) — Joe Biden says he did not know he had cancer while he…

    John Summit Says Hes Never Submitting to the Grammys Again, Calls Dance Category Nominations A F–king Joke

    November 16, 2026

    Trump Slams 60 Minutes After Norah ODonnells Shocking Iran War Report

    November 16, 2026

    Brave Circle-K Worker Fired for Defending Himself, Co-Worker as Video Shows 4 Would-Be Robbers Ramming Store with SUV * The Gateway Pundit * by Brooklynn Robinson, The Western Journal

    November 16, 2026

    Dystopian Thriller The Flaw Is a Cinematic Puzzle About Totalitarianism: Each One of Us Is a Regime

    November 16, 2026

    Peanut the squirrels former owner has been detained by ICE and could be deported

    November 16, 2026

    Apple launches new app to study how people use AI for health | TechCrunch

    November 16, 2026
    Categories
    • Books (1,611)
    • Business (6,516)
    • Cover Story (16)
    • Film (6,450)
    • Lifestyle (4,496)
    • Music (6,525)
    • Politics (6,500)
    • Science (5,856)
    • Technology (6,452)
    • Television (6,149)
    • Uncategorized (3)
    • US News (6,500)
    popular posts

    Sam Calls Out Meghan For Response To Him Kissing Jordana

    [ad_1] Siesta Key star Sam Logan called out his former fling Meghan Biscoff after she…

    Adam Schiff Is Running For The Senate

    January 26, 2023

    Home price correction’s latest shift, as told by 6 housing market charts

    March 14, 2023

    Anduril CEO Brian Schimpf says economic warfare is the ‘new normal’ for military conflicts

    June 9, 2026
    Archives
    Browse By Category
    • Books (1,611)
    • Business (6,516)
    • Cover Story (16)
    • Film (6,450)
    • Lifestyle (4,496)
    • Music (6,525)
    • Politics (6,500)
    • Science (5,856)
    • Technology (6,452)
    • Television (6,149)
    • Uncategorized (3)
    • US News (6,500)
    About Us

    We are a creativity led international team with a digital soul. Our work is a custom built by the storytellers and strategists with a flair for exploiting the latest advancements in media and technology.

    Most of all, we stand behind our ideas and believe in creativity as the most powerful force in business.

    What makes us Different

    We care. We collaborate. We do great work. And we do it with a smile, because we’re pretty damn excited to do what we do. If you would like details on what else we can do visit out Contact page.

    Our Picks

    Peanut the squirrels former owner has been detained by ICE and could be deported

    November 16, 2026

    Apple launches new app to study how people use AI for health | TechCrunch

    November 16, 2026

    Angels & Demons: Ken Russell's The Devils

    November 15, 2026
    © 2026 Beverly Hills Examiner. All rights reserved. All articles, images, product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Terms & Conditions and Privacy Policy.

    Type above and press Enter to search. Press Esc to cancel.

    We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
    Cookie SettingsAccept All
    Manage consent

    Privacy Overview

    This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
    Necessary
    Always Enabled
    Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
    CookieDurationDescription
    cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
    cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
    cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
    cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
    cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
    viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
    Functional
    Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
    Performance
    Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
    Analytics
    Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
    Advertisement
    Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
    Others
    Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
    SAVE & ACCEPT