Benchmarks Discover ‘DeepSeek-V3-0324 Is Extra Weak Than Qwen2.5-Max’

With the most recent steady launch dated January 28, 2025, Qwen2.5-Max is assessed as a Combination-of-Specialists (MoE) language mannequin developed by Alibaba. Like different language fashions, Qwen2.5-Max is able to producing textual content, understanding completely different languages, and performing superior logic. Based on latest benchmarks, it’s also safer than DeepSeek-V3-0324.

Utilizing Recon to scan for vulnerabilities

A workforce of analysts with Shield AI, the corporate behind a purple teaming and safety vulnerability scanning software often known as Recon, not too long ago used their platform to check the safety of Qwen2.5-Max towards that of DeepSeek-V3.

The workforce’s evaluation reads, partially: “We noticed that DeepSeek-V3-0324 is extra weak than Qwen2.5-Max, with Recon attaining an nearly 25% larger assault success charge (ASR).”

Whereas it could be safer than its competitors, Qwen2.5-Max isn’t precisely good. Based on their assessments, the AI mannequin is most prone to immediate injection assaults, as these represented nearly 48% of all profitable cyberattacks towards Qwen2.5-Max. Evasion and jailbreak assaults proved to be much less profitable with an approximate ASR of 40% for each.

Exposing vulnerabilities in DeepSeek-V3

Recon makes use of a complete Assault Library to scan current-gen AI fashions and establish vulnerabilities throughout six particular classes:

• Evasion methods
• System immediate leaks
• Immediate injection assaults
• AI jailbreak makes an attempt
• Normal security controls
• Adversarial suffix resistance

Along with simulated cyberattacks, Recon additionally assesses the AI fashions’ resistance to producing probably dangerous or unlawful content material. For instance, throughout adversarial suffix resistance assessments, Recon makes an attempt to control the AI mannequin into producing dangerous or unlawful content material.

The Shield AI workforce ran Recon towards each Qwen2.5-Max and DeepSeek-V3, with the previous boasting a decrease assault success charge (ASR) throughout a wide range of assaults; together with jailbreaks, immediate injection, and evasion methods.

Whereas Qwen2.5-Max had a 47% ASR towards immediate injection assaults, in comparison with DeepSeek-V3’s notably larger 77%. Towards evasion methods, Qwen2.5-Max scored a 39.4% ASR towards evasion methods, whereas DeepSeek-V3 scored 69.2%. Each AI fashions displayed related outcomes throughout different simulated cyberattacks.

Analyzing DeepSeek-V3’s strengths

Regardless of its safety weaknesses, DeepSeek-V3-0324 nonetheless outperforms Qwen2.5-Max in a number of completely different benchmarks. Not like the ASR, a better rating in these assessments truly signifies higher efficiency.

Based on these benchmarks, DeepSeek-V3-0324’s strengths embody normal language understanding (MMLU-Professional), superior subjects akin to biology, physics, and chemistry (GPQA Diamond), arithmetic (MATH-500, AI in drugs (AIME 2024), and coding (LiveCodeBench).