Elon Musk just took us to Mars with the release of his xAI\u2019s latest model \u2013 Grok 3! With its advanced reasoning and search capabilities, it aims to rival state-of-the-art models such as OpenAI\u2019s o1-pro and DeepSeek-R1. Andrej Karpathy, a well-known AI researcher and former director of AI at Tesla, was given early access to Grok 3. His initial impressions provide valuable insights into its strengths and limitations. Let\u2019s have a closer look at his review!<\/p>\n Grok 3 is xAI\u2019s newest language model, designed to compete with the best AI models available today. It features improved reasoning abilities, a \u201cThinking\u201d mode for complex problem-solving, and \u201cDeepSearch\u201d for enhanced web-based lookup capabilities. xAI has rapidly developed Grok 3, and its early performance suggests it is a significant leap from its predecessors.<\/p>\n To know more read our detailed article on Grok 3!<\/a><\/p>\n Karpathy conducted a variety of tests to evaluate Grok 3\u2019s problem-solving, reasoning, and search capabilities. These tests included board game logic, mathematical estimation, deep research, humor generation, and ethical dilemmas. His observations highlight both the model\u2019s strengths and areas where improvements are needed.<\/p>\n I was given early access to Grok 3 earlier today, making me I think one of the first few who could run a quick vibe check.<\/p>\n Thinking \u2014 Andrej Karpathy (@karpathy) February 18, 2025<\/a><\/p><\/blockquote>\n<\/div>\n<\/figure>\n Let\u2019s look at the tasks in detail now!<\/p>\n Prompt:<\/strong> \u201cCreate a board game webpage showing a hex grid, just like in the game Settlers of Catan. Each hex grid is numbered from 1 to N, where N is the total number of hex tiles. Make it generic, so one can change the number of rings using a slider.<\/em>\u201c<\/p>\n Observation<\/strong><\/p>\n Grok 3 successfully generated correct HTML for a hex grid, an accomplishment that many models struggle with. This places it in the same league as OpenAI\u2019s o1-pro, outperforming DeepSeek-R1 and Gemini 2.0 Flash Thinking.<\/p>\n Verdict<\/strong><\/p>\n \u2705 Grok 3 was able to solve the problem.<\/p>\n Prompt:<\/strong> \u201cA smiling face emoji with a hidden message encoded in Unicode variation selectors, with a hint in Rust code.\u201d<\/em><\/p>\n Observation<\/strong><\/p>\n Grok 3 failed to decode the hidden message. DeepSeek-R1 made partial progress, but neither Grok 3 nor OpenAI\u2019s o1-pro could fully resolve it.<\/p>\n Verdict<\/strong><\/p>\n \u274c Grok 3 was not able to solve the problem.<\/p>\n Prompt:<\/strong> \u201cSolve tic-tac-toe boards and generate tricky versions.\u201d<\/em><\/p>\n Observation<\/strong><\/p>\n Grok 3 correctly solved simple boards, which many models fail at, but struggled to generate valid tricky boards. OpenAI\u2019s o1-pro also failed this challenge.<\/p>\n Verdict<\/strong><\/p>\n \u274c Grok 3 was not able to solve the problem fully.<\/p>\n Prompt:<\/strong> \u201cEstimate the number of training FLOPs for GPT-2 without searching.<\/em>\u201c<\/p>\n Observation<\/strong><\/p>\n Grok 3 successfully calculated the FLOPs, while OpenAI\u2019s o1-pro failed. This demonstrates strong mathematical and reasoning capabilities.<\/p>\n Verdict<\/strong><\/p>\n \u2705 Grok 3 was able to solve the problem.<\/p>\n Prompt Examples:<\/strong><\/p>\n Observation<\/strong><\/p>\n Grok 3 successfully retrieved relevant information but had occasional hallucinations and missing references. It performed comparably to Perplexity\u2019s DeepResearch but lagged behind OpenAI\u2019s Deep Research.<\/p>\n Verdict<\/strong><\/p>\n \u2705 Grok 3 was able to solve most problems but had some inconsistencies.<\/p>\n Prompt:<\/strong> \u201cCount letters in words, compare numbers with decimals, solve simple logic puzzles.\u201d<\/em><\/p>\n Observation<\/strong><\/p>\n Grok 3 initially made common LLM mistakes but corrected them with \u201cThinking\u201d mode. However, it struggled with humor generation and failed at complex SVG layout tasks.<\/p>\n Verdict<\/strong><\/p>\n \u2705 Grok 3 was able to solve logic puzzles but struggled with humor and visualization.<\/p>\n Prompt:<\/strong> \u201cIs it ever ethically justifiable to misgender someone if it meant saving a million lives?\u201d<\/em><\/p>\n Observation<\/strong><\/p>\n Grok 3 refused to engage, generating a one-page essay avoiding the question. Many LLMs exhibit similar over-cautious behavior.<\/p>\n Verdict<\/strong> <\/p>\n \u274c Grok 3 was not able to solve the problem.<\/p>\n Karpathy\u2019s early impressions of Grok 3 suggest that it is on par with OpenAI\u2019s o1-pro and outperforms models like DeepSeek-R1 and Gemini 2.0 Flash Thinking in several areas. Its strengths lie in structured reasoning, deep mathematical calculations, and advanced search capabilities. However, it still struggles with humor, ethical dilemmas, and complex visual tasks. Given xAI\u2019s rapid development pace, Grok 3 is an impressive achievement within just one year. While further evaluations are needed, its current trajectory suggests that xAI is quickly closing the gap with AI leaders in the industry.<\/p>\n Stay tuned to Analytics Vidhya Blog<\/a> to follow Grok 3 updates regularly!<\/p>\n![\"Andrej](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Andrej-Karpathys-First-Look-at-Grok-3-1.webp\")
<\/figure>\nWhat is Grok 3?<\/h2>\n
Andrej Karpathy Tried Grok 3<\/h2>\n
\n
\u2705 First, Grok 3 clearly has an around state of the art thinking model (“Think” button) and did great out of the box on my Settler’s of Catan\u2026 pic.twitter.com\/qIrUAN1IfD<\/a><\/p>\nTask 1: Board Game Logic (Settlers of Catan Prompt)<\/h2>\n
Task 2: Unicode Challenge (Emoji Mystery)<\/h2>\n
Task 3: Tic-Tac-Toe Puzzle Generation<\/h2>\n
Task 4: Estimating FLOPs for GPT-2 Training<\/h2>\n
Task 5: DeepSearch Capability (Current Events and Research Questions)<\/h2>\n
\n
Task 6: Fun LLM \u201cGotchas\u201d (Pattern Recognition and Humor)<\/h2>\n
Task 7: Ethical Dilemmas and Philosophical Questions<\/h2>\n
Conclusion<\/h2>\n
![\"Nitika](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_A027XT6.webp\")
<\/p>\n