Reposted by M M
Benchmarks for cybersecurity are everywhere and mostly measuring the wrong thing.
We reviewed evals from Microsoft, Meta and academia and found they don't measure what matters for defenders in real IR situations. 🧵
s1.ai/benchmk1
s1.ai
LLMs in the SOC (Part 1) | Why Benchmarks Fail Security Operations Teams
LLM cybersecurity benchmarks fail to measure what defenders need: faster detection, reduced containment time, and better decisions under pressure.
![[tumblr post from user brainstatic (brain static)]
I want a Santa movie where he's actually Saint Nicholaos of Myra. Three kids run away from home and find a portal to the Council Of Nicaea in 325 AD. Then the kids watch him debate the promulgation of canon law. In the end they return home and tell their parents that presents aren't important so long as Pentecost is counted 50 days after Easter on the Roman solar calendar and not the Jewish lunar calendar.
And the kids tell their parents the whole story and the dad is like "those kids and their imaginations."
But then he sees a piece of paper in the dog's mouth and goes "whatcha got there boy? Is that... orders from the Holy See that the schismatic bishop Meletius is to remain in Lycopolis and keep his episcopal title but the ecclesiastics ordained by him must again receive the laying of hands?
But…how?" And then he gazes off to the sky and you hear a light jingling of bells and then a voice over goes "Ho ho ho! The baptisms performed by the Paulian heretics are invalid! Merry Christmas!"](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:sedw2ag3rnmh2vwo75wy5flb/bafkreianvdytcr537nmqny2sawvip4smcmh3fcbavysrucegabnofcmr7y)
