You may never hear me say this again because I am a gruff cynic who is not sure that hanging out here is good for his psyche, but thank goodness for HN.
While I have been chasing around today, trying to resolve this on different servers, it turns out a three line nginx mitigation (blocking multipart uploads) I wrote in the wee small hours GMT has been quietly protecting a vulnerable machine I could not shut down (from attacks powered by every single mainstream agentic AI tool — Anthropic, OpenAI, Qwen cloud, Baidu, seemingly also AWS, Meta)
Including an hour before the PHP patch actually shipped.
So without the HN discussion I wouldn't have found enough people pointing at possible mitigations and I would have lost a server to this, and I thank everyone who pointed out that second mitigation in particular.
> from attacks powered by every single mainstream agentic AI
Is it so bad? Have you measured the number of attacks?
> So without the HN discussion I wouldn't have found enough people pointing at possible mitigations and I would have lost a server to this
I run a tiny server. I like this server a lot. I only patched an hour after the Debian package was ready and didn't think about it again. Do you mean to imply there's a real chance my little server has been compromised?
[Edit:] My server is quite public too. It hosts four SLDs and many subdomains and receives both human and bot traffic. Only one or two subdomains use PHP, but a reasonable attacker surely tries all of them.
Agreed. It made my ancient old man face tear pores by smiling when I saw people organizing in the comments to throw sandbags against the wall while waiting to be rescued.
The vulnerability isn't mitigated by disabling multipart uploads. You're working under a false sense of security and encouraging others to do the same. Just stop it and patch.
Is it just me, or is it the truly brave that decide to deploy and protect PHP instances? Especially if WordPress or Drupal are involve?
I finished moving the remainder of my stuff off of PHP and WordPress last year and haven't missed it at all. Backwards compatibility was typically pretty bad.
It seemed like maintaining existing working PHP code was always a cycle of: upgrade to newer PHP, that version has breaking changes, fix that code or update to a new WordPress (plus all of the plugins that would inevitably need updating too).
> As a result of the leak and pre-announcement around 20:00 UTC, Calif, one of the security firms that had reported the issues mistakenly published a video.
This is the most surprising part to me. It sounds like Calif saw the leaks, interpreted that as embargo being broken, and then raced to publish their [anime dancing] marketing video? I'd have expected much better.
This reads like some guy tripped and fell on top of his mouse and in so doing, accidentally turned on screen recording, performed the exploit a few times, got the anime girl asset and soundtrack, dropped all of them into Premiere, added the weird effects, rendered the video, and published the video to Twitter. "Ow. What did I fall on? Oh noooo, I broke the embargo!"
Your comment is funny. But obviously, "mistakenly" refers to the mistake of believing the embargo being broken, and then, very intentionally and not mistakenly, publishing a stupid video.
The video was posted hours early, quickly deleted (but not before being reshared), and then posted again at the proper time. The link in your comment is not the first release of the video
What are these "rules" you speak of? There's usually nothing binding in these agreements. It's just groups either trying to do the right thing by end users or trying not to look like assholes, get a better payout, or some combination thereof.
Finding an exploit and publishing is is not against the law. In some cases, publishing immediately is the responsible thing to do. For example, when it's actively being exploited in the wild, and waiting just puts people at risk that don't have any recourse to protect themselves, since they don't know they're at risk. There can be nuance to that as well, but in general, as soon as it's being exploited, anything that makes it harder for end users to a) know there is an exploit, or b) tell if they're susceptible to it (e.g. through a POC) is just taking away people's ability to take the action they deem appropriate.
That sounds oddly threatening, especially given that chainguard is in the right here. Embargoes are stupid in 2026 and make no sense. The rumor of this issue was enough for it to be independently discovered by several white hats and almost certainly many threat actors.
> Embargoes are stupid in 2026 and make no sense. The rumor of this issue was enough for it to be independently discovered by several white hats and almost certainly many threat actors.
The point of the embargo was to prevent all rumors? But it did not hold.
Releasing a statement that a critical security bug exists with no more information, does the opposite of preventing rumors. I think one of the most surefire ways of generating rumors is to say "something big will happen tomorrow, please don't speculate about it".
Anyway, GPs point is that even if nobody leaked anything, and the only thing publicly known was the existence of the bugs, people would anyway have been able to find them independently. So what is the purpose of waiting 24h before releasing the patches?
I think the PHP security team and the whole PHP team could have been more communicative and keep releasing information, adapting to the news, suspected exploits and mitigations.
If a video is released showcasing the exploit and following mitigation recommendation, PHP should clarify what it is and if the mitigations are recommended or not. From the pre-announcement:
> We reserve the right to release earlier should details of either vulnerability become public, or should evidence of active exploitation appear in the wild.
If no info is released by them, no other party should have released anything. If info is released or leaked from partners or involved parties, you should notify and explain what's going on.
But no, complete silence from PHP news or any PHP community member.
I get that it was a complicated situation, and I think PHP is the "less wrong" party involved, but I think this constructive and external criticism could help in the future. It has been an interesting day (GMT+2 here) and a rollercoaster of monitoring and mitigating without any reliable source.
No meaningful info was released. A lot of incorrect info was speculated on.
I have a lot of sympathy for php team - it would be a lot of work to respond to every conspiracy theory on twitter, while trying to get everything ready, in the middle of the night no less.
> Are there structurally much safer ecosystems out there?
Several attributes make a safer ecosystem. An important one is diversity - technical diversity.
For an interpreted language, the runtime is important. Python has several runtimes, as does Ruby. Javascript also has several, from the few browsers, to node, deno and bun. PHP has for all practical purposes, only one. So if there's a security incident in PHP, it affects almost all users. But when e.g. JRuby has a vulnerability, it will affect only a fraction of the installed ruby applications.
For compiled languages, the tooling, libraries and runtimes is where diversity is important. With e.g. rust, the vast majority of CVEs come from libraries (crates). But since there often are several crates for a single concept, diversity helps mitigate here too.
Garbage collection is also an option. Lots of backends on the JVM!
(Of course there's Log4shell, but that could've happened to a Rust-based stack too, and might be more likely to, considering the proliferation of deeply-nested dependency trees full of tiny libraries in that ecosystem.)
While the main Java and .NET virtual machine implementations are much more secure than PHP, they were still written at least partially in unsafe C++. I won't be surprised if frontier LLMs eventually find more vulnerabilities in those platforms.
I presume that leaving an "anyone can run any command on over 40% of the web servers in the world" hole in PHP for 14 years means the entire team has been relieved of their responsibilities.
I was asking myself today; how to ensure that you don't miss a cve? I mean you'll sooner or later stumble across it if you're monitoring your communities but I thought that maybe there exists some kind of central solution, like a service were you could subscribe for your tech stack and get notified?
Big companies usually use some sort of vulnerability response management software which is connected to your CMDB, ideally containing SBOMs of everything you use, and subscribed to various providers of CVEs.
Those tools then map the CVEs to your SBOMs and you'll get alerts, security incidents and so on.
I was thinking about this as well today. And while I'm sure there are many cybersec companies that send out alerts, I trust this forum the most.
Maybe it would be worth it to have an agent that knows your stack, reads the top threads on hn all day long and messages you when it thinks there is something serious going on that could impact you.
Several tech stacks have this built in or available.
For my rust applications, I use "cargo audit" which checks all dependencies against the CVEs. For my typescript and javascript applications, deno audit and/or yarn npm audit.
For the rest of the stack `debsecan` does exactly what you want on Ubuntu: report CVEs for installed software.
I don't think there's any robust method. I lurk HN, subscribe to security lists, run our code through scanning tools regularly. All options suck, all options have gaps, and all options mean you have to stay on the hamster wheel.
Lots of licensed solutions have this (typically SAST or DAST tooling like Snyk, etc...). Given your build manifest/SBOM they cross-reference CVEs for vulnerabilities during/after build time.
I may be misunderstanding this but think the hard part about building a threat intel feed like this is just trusting that the underlying integration would surface something you care about... especially if you want to DIY it. You can scrape HN, GitHub advisories, NIST, whatever you want... but you may give yourself a false sense of security after seeing the "all clear" checkbox each morning
I hope that someone has some sort of a reasonable approach here / I would love to help work on something open source if not
Any tool to scan dependencies takes various inputs like repos, container stores or already built SBOM and gives you warnings. It doesn't much matter if you have actual code and repos as long as you know how to point it at some. The fremium ones like snyk would love to send you an email report on growing CVE counts on your last run but only until hell freezes over..
> On October 21st at 07:41 UTC, more than six hours before the coordinated release, Chainguard unilaterally broke the embargo and published built images containing the fix.
This is incredibly damming of Chainguard. I struggle to see justification for this, even with the chaos approaching the coordinated release.
Let's hope so. Shall we create a new norm? A 365 day timeout, for companies like Chainguard that deliberately break embargoes.
Very irresponsible behaviour in my opinion; and if it's a decision made unilaterally by their CEO, probably a breach of their SOC 2 certification. What's next? "$X isn't patching this CVE fast enough, here's a binary blob of our fix, don't worry, we're making this free!"
Nah, Chainguard should be praised for this. It was a different company that leaked the POC early, and then PHP, Wiz, OpenAI, and Calif sat on their hands doing nothing while the vuln was being actively exploited in the wild. Chainguard did the right thing by getting the fix out as quick as possible to stop the bleeding (or gushing flow of blood, really) rather than sitting on it so they could make a coordinated marketing post, like the others did.
That's a reasonably take. Embargos only work until the information's leaked, and once that's happened, however it happened, the bad guys have the exploit information and the clock starts ticking.
PHP’s view is that Chaingiard started the clock because the video was insufficiently detailed. Not sure if that is the case, but just sharing I had a different reading of the statement.
They released some binaries, built for their own docker image based on their own Linux distro. Other people couldn't use Chainguard's "fix" to patch their own systems. The only people who seem to have benefited from the stunt are the hackers who used AI to reverse-engineer the binaries and write exploits.
That was a giant fuck-you to the rest of the PHP ecosystem, nothing more.
Chainguard effectively held everyone hostage, and forced everyone hosting container images and wanting to be online, to trust their builds and then saying they did it for "free" to get free advertisement. Incredibly irresponsible behavior. They used this as a big fat advertisement board.
I dunno, the post is awfully critical of Chainguard for leaking a fixed binary 6 hours early, but it was the research firm "partner" that "mistakenly" posted the dancing anime POC video over 24 hours early that created the mass panic (not to mention spawning millions of agents crawling the php codebase and trying to brute force a POC and probably using more energy than a lot of countries in the process).
wow, okay. yes, that really does look like a "my first PoC" video made by an early teenager. hard to believe that this was produced by a company that wants to be taken seriously.
There do seem to be the perpetual problem with the field of computer security that only insufferable and immature kids matter, due to discrete math adjacent nature of computer science, and yet they also legitimately can't be trusted, or does it not... unless LLMs do change everything.
I wish people would name this idiotic company, I can't see Twitter here and all people are talking about is a childish video they made. I don't ever want to do business with said company, ever.
They're apparently not trustworthy, and behave as children.
>As a result of the leak and pre-announcement around 20:00 UTC, Calif, one of the security firms that had reported the issues mistakenly published a video. It showed their proof of concept working against a number of targets and revealed a few additional, though limited, details about the vulnerabilities.
Calif more than Chainguard, but still Chainguard too.
Calif was obviously twitching to claim credit for their findings, and what I find troubling is this eagerness was more important to them than giving users a chance to update their systems.
Even if they ended up waiting only a day post release I would still say it seems a bit too soon. Maybe a week after. Not everyone’s system have aggressive patching schedules and PHP is soo heavily used.
This is probably the future, tho. Being first to disclose a vuln is a Big Deal in some quarters. And now, the velocity of AI driven pentest swarms mean every researcher who finds something worthwhile has to think "how close is the next droog's swarm to finding what I found" and the temptation to jump the gun only gets stronger.
I don't know, it's showing most sites being pwnd in less than 20s with some taking just under 30s. Of course, that's on homeboy's local Mac. Not sure how long it takes to gain control with PHP2Shell to run the exploit though. Just started looking at that part myself
Everything i saw on twitter prior to the release turned out to be incorrect. At most people figured out it had something to do with session upload handler and UAF, but nobody knew what.
I want to go back to talks by bored defcon guys or gals calmly stating how they obtained private keys of a globally trusted CA with just a bit of math. Which, mind you, ate the other part of the teams time.
Now I know how much I hustled today likely due to that atrocity of a video. I think I will take a vacation day and sit in a forest near a river on friday.
dopamine junkies crave stimuli, and realistic details of humans are just noise for the data. anime humans are simplified to extremis which alone makes some people think it's about kids
I kinda get Chainguard's position here. They were part of the embargo, saw exploits hitting in the wild already and decided to release rather than hold things back.
Embargos make sense if no one is exploiting it yet, but if its out in the wild, the best path forward is releaseing as soon as possible.
The person who wrote the widely-referred-to tweet last night (claiming to "have recreated & lab/field tested CVE-2026-109618 / CVE-2026-109619") followed up a couple hours later and said that their POC had not been proven to work "in the wild"
"Cloudflare successfully deployed multiple mitigations for all its customers while the vulnerability was leaked before the official release date and we observed attack traffic targeting our customers."
If that's not an in-the-wild exploit, it literally couldn't have been further away than one or two more iterations.
It sounds like there was no "co-ordination" behind any of this co-ordinated disclosure. You'd have expected there to be an IRC channel, a mailing list, a Slack chat, or something for something of this scale.
Was there even an incident co-ordinator here? Was any of the parties talking to each other?
Many people actively following would have 403'ed multiparts so people seeing those blocked might have assumed it started being exploited early instead of probes / background noise.
if their comment in true (linked in sibling comments), then it's really not damning. typically an embargo is ended when someone leaks early or it's being exploited in the wild.
so, if they saw evidence of exploits in the wild, it is not unusual at all to release the fix.
If they're gonna unilaterally release an exploit patch 6 hours before an agreed upon embargo for a CVSS 9.2 that affects like ~70% of the internet then they better have reams of evidence showing this was being exploited in a widespread manner and not just "we saw a dancing anime girl edit of a theoretical exploit on twitter, so we had to release it to the world"
And evidence they at least gave a heads up to everyone in on the embargo, eg "I have to release you all have 1 hour to get on board".
There is a MASSIVE difference between a small number of people exploiting in the wild, and a few hours after a release a bazillion script kiddies doing so. The move to release and break an embargo was childish, absurd, self-serving, and lacking in all common sense. Not to mention, breaking trust.
I agree with those in this thread who say, never going to do business with them. Ever.
As a customer, I'd immediately assume no one in the security community would trust them again, and seek someone else to give my money to. Why pay a) an untrustworthy company who b) now is not trusted by anyone and c) won't ever be able to early release again anyhow? And if anything, are left out of important conversations.
Way to burn bridges. You only get to be an asshat once.
It's the age of LLMs. Exploits go from theoretical to real fairly quickly. PHP's open source, it's not impossible to conceive of someone seeing that video, getting the general idea of what to do, and working the rest out with a decent model.
Because if there are already publicly available exploits seen, it makes no more sense to keep holding patches back just so that OpenAI etc. can get a nice press release
2. Calif sees leak, goes and releases dancing anime girl video before pulling it, goes viral
3. Chainguard sees dancing anime video, doesn't recognize Calif, and thinks the exploit is in the wild
[Edit 3a]: Or alternatively, Calif's leaked video causes the PoC to be found by others by providing enough of a lead
4. Chainguard releases updated binaries thinking it doesn't matter, causing the real exploit to then be found early
5. Which then causes PHP Foundation to basically have a meltdown
Personally I blame Calif for the unprofessional video which absolutely completely looks like some script kiddie and for being too detailed [edit 2: assuming Calif produced it and the anime wasn't a prank edit]. In this case it wasn't just unprofessional fun but may have literally messed up the responsible disclosure.
> 3. Chainguard sees dancing anime video, doesn't recognize Calif, and thinks the exploit is in the wild
I'm not sure about this part. I saw https://jackalsyn.com/posts/php-up-2026.html going around a bit over 12 hours ago, before the chainguard images went out, and plenty of other chatter of exploitation happening in the wild.
Not really sure if 3 is correct. Might have been Calif leaks the video -> it goes viral -> people make PoCs based on the video -> Chainguard releases the binary because there seem to be credible PoCs out there (that are not just the Shigure Ui video).
Climate change that keep getting worse, rising political extremism, the wealth gap getting bigger and bigger, I can keep going but you get the point. You just have to open any news media once really.
I can empathize with them, this is actually quite difficult when you're an open source project.
You have your source code public, so you can't commit/push because then you've leaked info.
Your build pipeline is also likely public, so you can't just build from another source code... source.
You can create a private repo, but now you also need to build a private build system.
You don't get the same generous build resources for private repos, so you either:
a) simplify the build to just the essentials
b) pay
Unless you have significant funding, you'll probably go for a).
Now you have to maintain two systems, also with some differences between them.
Plus you need a way to get the build artifacts from the private system to the public one, typically at a fixed point in time.
Any failure/delay (e.g. less powerful build on private repo, errors because you exercise it rarely) causes people to get upset that you didn't respect the announced time.
Maybe you're also using things like image attestation, another can of worms.
Those are all challenges that surely need to be overcome!
But they had _two months_ to figure out how to overcome them. For them to not have figured out a solution and have pre-built images ready to push as soon as the embargo lifted is ridiculous.
I don't know about PHP, but at work our artifacts are signed, and you certainly cannot have the signing certificate on your laptop.
It's stored in some secure server, and only accessible from some allowlisted build nodes, for specific builds (release branch, ...).
Now of course, there are ways around that if required, but they are, by design, manual, cumbersome and slow. I could not create a release build locally in a few minutes. A few hours maybe if the whole organisation dedicated itself to it.
I'd say that for someone like PHP, in this era of AI-driven vulnerability/patch race, having a private CI/CD pipeline is a necessity.
PHP is written in C, I'd be surprised if similar problems didn't lurk in other places.
Maybe they will construct the private pipeline after this crisis. Millions of websites are probably now waiting for all the downstream Docker images to be built and pushed.
The images everyone is waiting for are built by Docker though? So now if you're Docker you have to have a private CI/CD pipeline with all library projects?
It would be great, but it's also something to manage. Just keeping it working with all the changes that happen across all projects would be something.
Making a cloud CI/build/release system whose behavior can't be reproduced locally requires some kind of brain damage. I can't believe people tolerate this.
The problems you describe in your first post are only problems when you can't reproduce your build process locally. With a local build you can always do it in private for free.
The problems you bring up here are eminently solvable too. Yes there are many solutions for building for multiple architectures and versions. Cross builds, containers, VMs, or just having more than one computer locally. You can address building the wrong thing with reproducible builds, but for a one off patch like this it's not the most likely problem and relying on a trusted maintainer temporarily during an emergency is better than waiting for a glacially slow cloud build while exploits spread. Pushing builds to the cloud is inherently a cloud step, so it's fine to leave keys for that in the cloud, but there should be a process for publishing local builds and it should be essentially instant when the build is already done.
You should always have a local build process that replicates CI. There is no excuse.
There's over 3 million downloads per month for the official PHP image. That may certainly not be "most PHP users". But still is a a lot of users.
I'm quite certain that your exact sentiment is why the images weren't updated immediately: I'm afraid PHP foundation doesn't see docker images as a particularly important distribution system.
> I'm quite certain that your exact sentiment is why the images weren't updated immediately:
I wouldn't be that certain.
I think it's likely that the vast majority of PHP Docker image downloads are for local development, for one thing.
But nevertheless, it would not come close to the number of websites running PHP from the Ubuntu Server packages. It is absolutely right that the effort was focussed on the distribution route. Because PHP is not "expert" hosting territory; it is still more often whole cloud VMs running the distro packages.
The PHP Foundation should manage it, the Docker ""Official" Images" approach clearly just doesn't work properly for responding to vulnerability patches.
I find it incredibly naive that they’d hope it would not leak after weeks/months of embargo.
Being given advance notice of an upcoming security fix is an immense privilege, organisations should turn around quickly when given such information, they shouldn’t be given more than a couple of days to update IMHO, it’s not like the rest of the world can wait.
Keep in mind that some of the organizations are presumably patching ancient versions of PHP like 5.6. I am sure this is a process which takes considerable time, especially as upstream PHP has washed their hands of such old versions and the downstreams are effectively on their own.
(It is often much cheaper for an enterprise to just get a long term support contract than to properly upgrade to a modern PHP stack.)
> (It is often much cheaper for an enterprise to just get a long term support contract than to properly upgrade to a modern PHP stack.)
I've experienced similar to this. I know I shouldn't be, but I still get amazed every time I hear that some small company is still using an old DOS machine running core software based on dBase that nobody has any clue on how works and the developer has since moved on to a higher plane. One person told me that they sweat every time the power goes out longer than the attached UPS can run holding their breath that the machine powers back up.
PHP foundation also doesn't "Run" the apt or rpm packages, yet they certainly coordinated with debian, and other big distros to push the updates out timely.
Now I don't know what happened here. Maybe PHP foundation reached out to docker but got no help, maybe they simply forgot about docker as distributor of their software.
> The PHP security team closely monitored and inspected appearing PoCs after the leak but did not identify any functional exploits at the time of release/public disclosure
Okay, but how close did they get? Were they close enough that a determined actor could've (perhaps using an LLM) managed to get to a functioning exploit?
I know of a php exploited about 4 hours before their release time 1400 UTC, using the session bug in an exploit that was passed around. I'm really not very much in the loop about cyber security things.
There should be more interpreters or runtimes. Regardless of the language they are written in. Indeed a rustPHP, maybe a JPHP, cPHP, etc etc. There is zend, but AFAIK that's still using the "official" PHP under the hood. And there is or was hiphop. Not sure how that is going, though.
Because now a vulnerability in PHP affects practically 100% of the users. But with more diversity, it would affect only a part. And if one runtime proves insecure more often than another one, organizations have a choice to move to an interpreter that better fits their security needs.
As long as programmers trust 3rd-party-provided content-length input, it will be insecure. At this point, I'm not sure why we even use content-length anymore since you're going to need to size your buffers based on the actual content.
I'm generally bullish on migrating C projects to Rust, but that will only reduce a particular class of vulnerabilities. There will still be other vulnerabilities which remain discoverable by LLMs, and for those, the problem of accelerated exploit creation will remain.
I'm really not sure how you mitigate this issue, other than serving static HTML and reducing the attack surface to cut out dependencies like PHP.
So instead of using a memory safe language or harden the C code they hope LLMs will find bugs in their code? Neither do they mention how their way to patch Exceptions into C makes It very difficult for Fuzzers to find those bugs? Good luck this statement should tell everyone to migrate away from that horrible language and runtime.
This is what I thought as well. How is this still so popular if it has such fundemental flaws? With the increasing power of LLM, more vulnerabilities will definitely surface sooner or later.
> The PHP security team, operating on consensus, struggled to choose between these and feared the repercussions of shipping the wrong choice.
This is the real story. You ship one of them (the breaking change) with —fuck-my-shit-up or something as a compile time option to switch to the other fix.
>On Tuesday, October 20th, at around 15:00 UTC, the two CVE IDs began circulating on online forums and social media. They came with a wrong release time and a wrong CVSS score of 10.0 (the actual score is 9.2). No technical details leaked.
This was partly me. I take responsibility for posting it on HN. I am truly sorry for the disruption this has likely caused.
My contact had informed me there was no embargo, and told me her contact had told her the same. It transpired this morning that was a lie.
>On October 21st at 07:41 UTC, more than six hours before the coordinated release, Chainguard unilaterally broke the embargo and published built images containing the fix. Like other distributors, Chainguard had received the patch in advance so its users could update the moment the vulnerabilities were disclosed. Any attacker could now study the fix in binary form while every other PHP user was still waiting for the release. This disregarded weeks of work by many parties toward a coordinated disclosure.
On the plus side, at least I'm not Chainguard. Yikes.
>Chainguard unilaterally broke the embargo and published built images containing the fix.
>This disregarded weeks of work by many parties toward a coordinated disclosure.
Good. Coordinated disclosure meant that people couldn't protect themselves. That's not a good thing. It meant sites were forced to shut down instead of simply updating.
No... Uncoordinated disclosure meant that attackers were suddenly armed with detailed information about an already-hyped severe vuln hours before the majority of downstream consumers had the means to patch their systems.
Maybe your calculus could hold if Chainguard's registry were the primary distribution mechanism for PHP in use worldwide (such that the benefit of an updated image would at least be widespread compared to the sitting ducks relying on the other repositories), but it isn't.
If the sites are taking no action they will be vulnerable no matter when the patch is released. For the people who do take action before chainguard's release the safe option was to just turn off your server (and let any big competitor site stay up and take your traffic since they got the patch ahead of time). People taking this option will be offline until a fix is available for them to use. They are not in danger of being hacked.
Trying to say that uncordinated disclosure is bad because people might get hacked yesterday instead of today just is not a convincing argument to me. I prefer the solution that results in the minimum amount of downtime for sites and sites provide value to society.
That is incredibly damaging to the trust that coordinated vulnerability disclosure depends upon. The obvious consequence is that Chainguard should never again receive embargoed fixes in advance. You don’t get invited back into the room after demonstrating that you can’t be trusted with the information.
They didn’t get to unilaterally decide that the embargo was inconvenient and publish early.
Why do you place a higher priority on maintaining coordinated vulnerability disclosure than of the safety of actual users of the software being protected?
While I have been chasing around today, trying to resolve this on different servers, it turns out a three line nginx mitigation (blocking multipart uploads) I wrote in the wee small hours GMT has been quietly protecting a vulnerable machine I could not shut down (from attacks powered by every single mainstream agentic AI tool — Anthropic, OpenAI, Qwen cloud, Baidu, seemingly also AWS, Meta)
Including an hour before the PHP patch actually shipped.
So without the HN discussion I wouldn't have found enough people pointing at possible mitigations and I would have lost a server to this, and I thank everyone who pointed out that second mitigation in particular.
Is it so bad? Have you measured the number of attacks?
> So without the HN discussion I wouldn't have found enough people pointing at possible mitigations and I would have lost a server to this
I run a tiny server. I like this server a lot. I only patched an hour after the Debian package was ready and didn't think about it again. Do you mean to imply there's a real chance my little server has been compromised?
[Edit:] My server is quite public too. It hosts four SLDs and many subdomains and receives both human and bot traffic. Only one or two subdomains use PHP, but a reasonable attacker surely tries all of them.
I absolutely loved being here today.
So many brilliant minds screaming and hating together, trying to survive the tide.
There aren’t many places like this left on the commercial internet, especially in this brave new AI-driven world.
I felt a great disturbance in the Force. But in a good way.
Together, we are strong. May the Force be with us all!
I finished moving the remainder of my stuff off of PHP and WordPress last year and haven't missed it at all. Backwards compatibility was typically pretty bad.
It seemed like maintaining existing working PHP code was always a cycle of: upgrade to newer PHP, that version has breaking changes, fix that code or update to a new WordPress (plus all of the plugins that would inevitably need updating too).
This is the most surprising part to me. It sounds like Calif saw the leaks, interpreted that as embargo being broken, and then raced to publish their [anime dancing] marketing video? I'd have expected much better.
This reads like some guy tripped and fell on top of his mouse and in so doing, accidentally turned on screen recording, performed the exploit a few times, got the anime girl asset and soundtrack, dropped all of them into Premiere, added the weird effects, rendered the video, and published the video to Twitter. "Ow. What did I fall on? Oh noooo, I broke the embargo!"
Heck of a mistake.
The video was posted 44 seconds before that time: https://x.com/lcfr_hax/status/2112906567662399830
This can't possibly be interpreted as some major violation of the embargo, so what are people talking about?
PHP 8.2.35 - https://news.ycombinator.com/item?id=50180873 - Oct 2026 (439 comments)
PHP2Shell: AI uncovers critical RCE in PHP - https://news.ycombinator.com/item?id=50180836 - Oct 2026 (146 comments)
PHP session.upload_progress heap UAF (CVE-2026-109618 / -109619) - https://news.ycombinator.com/item?id=50176077 - Oct 2026 (61 comments)
Security advisory: Critical remote code execution vulnerabilities, update now - https://news.ycombinator.com/item?id=50172041 - Oct 2026 (631 comments)
Tell HN: 2x PHP RCE CVEs with CVSS 10.0 Incoming - https://news.ycombinator.com/item?id=50167754 - Oct 2026 (60 comments)
> Chainguard unilaterally broke the embargo and published built images ...
Won't be the last time this happens. If there is no consequence of breaking the rules, people will keep doing that.
Finding an exploit and publishing is is not against the law. In some cases, publishing immediately is the responsible thing to do. For example, when it's actively being exploited in the wild, and waiting just puts people at risk that don't have any recourse to protect themselves, since they don't know they're at risk. There can be nuance to that as well, but in general, as soon as it's being exploited, anything that makes it harder for end users to a) know there is an exploit, or b) tell if they're susceptible to it (e.g. through a POC) is just taking away people's ability to take the action they deem appropriate.
The point of the embargo was to prevent all rumors? But it did not hold.
Anyway, GPs point is that even if nobody leaked anything, and the only thing publicly known was the existence of the bugs, people would anyway have been able to find them independently. So what is the purpose of waiting 24h before releasing the patches?
If a video is released showcasing the exploit and following mitigation recommendation, PHP should clarify what it is and if the mitigations are recommended or not. From the pre-announcement: > We reserve the right to release earlier should details of either vulnerability become public, or should evidence of active exploitation appear in the wild.
If no info is released by them, no other party should have released anything. If info is released or leaked from partners or involved parties, you should notify and explain what's going on.
But no, complete silence from PHP news or any PHP community member.
I get that it was a complicated situation, and I think PHP is the "less wrong" party involved, but I think this constructive and external criticism could help in the future. It has been an interesting day (GMT+2 here) and a rollercoaster of monitoring and mitigating without any reliable source.
I have a lot of sympathy for php team - it would be a lot of work to respond to every conspiracy theory on twitter, while trying to get everything ready, in the middle of the night no less.
> PHP security team, operating on consensus, struggled to choose
What an unfortunate waste of a security audit. (and consensus is an obvious flaw in decision making)
> PHP runs a large share of the web... Yet security work on PHP itself still depends on a small group ...
Another unfortunate big structural issue that's unlikely to change.
Are there structurally much safer ecosystems out there?
When a corporation sponsors the language, such as Java, Go, .Net, there's an inherent economic component which acts as a forcing function.
PHP being a foundation sounds nice. But consensus wasn't strong enough to affect the key outcomes.
Several attributes make a safer ecosystem. An important one is diversity - technical diversity.
For an interpreted language, the runtime is important. Python has several runtimes, as does Ruby. Javascript also has several, from the few browsers, to node, deno and bun. PHP has for all practical purposes, only one. So if there's a security incident in PHP, it affects almost all users. But when e.g. JRuby has a vulnerability, it will affect only a fraction of the installed ruby applications.
For compiled languages, the tooling, libraries and runtimes is where diversity is important. With e.g. rust, the vast majority of CVEs come from libraries (crates). But since there often are several crates for a single concept, diversity helps mitigate here too.
(Of course there's Log4shell, but that could've happened to a Rust-based stack too, and might be more likely to, considering the proliferation of deeply-nested dependency trees full of tiny libraries in that ecosystem.)
I presume that leaving an "anyone can run any command on over 40% of the web servers in the world" hole in PHP for 14 years means the entire team has been relieved of their responsibilities.
Also, blameless post-mortems exist for a reason.
Maybe it would be worth it to have an agent that knows your stack, reads the top threads on hn all day long and messages you when it thinks there is something serious going on that could impact you.
For my rust applications, I use "cargo audit" which checks all dependencies against the CVEs. For my typescript and javascript applications, deno audit and/or yarn npm audit.
For the rest of the stack `debsecan` does exactly what you want on Ubuntu: report CVEs for installed software.
PHP2SHELL, in particular, made me expand my weekly security review to include my external network resources.
The natural conclusion for this seems to be realtime monitoring CVE and pre-CVE vulnerability indications in exposed stacks.
Some have free licenses, many don't.
Sometimes build time is too slow (e.g., a repo that doesn't receive updates every day or so).
I hope that someone has some sort of a reasonable approach here / I would love to help work on something open source if not
But, certainly do your own threat assessment and develop and implement whatever plan works for you.
This is incredibly damming of Chainguard. I struggle to see justification for this, even with the chaos approaching the coordinated release.
In their place I would worry that this was the last time anyone shared an embargoed build with them.
Very irresponsible behaviour in my opinion; and if it's a decision made unilaterally by their CEO, probably a breach of their SOC 2 certification. What's next? "$X isn't patching this CVE fast enough, here's a binary blob of our fix, don't worry, we're making this free!"
What was Wiz/OpenAI supposed to do? PHP was already aware of the issue.
They released some binaries, built for their own docker image based on their own Linux distro. Other people couldn't use Chainguard's "fix" to patch their own systems. The only people who seem to have benefited from the stunt are the hackers who used AI to reverse-engineer the binaries and write exploits.
That was a giant fuck-you to the rest of the PHP ecosystem, nothing more.
[citation needed]
By which i mean, this seems untrue. If you mean the anime video, that is not a leak of a PoC.
https://www.chainguard.dev/athena
That can be for marketing purposes or by mistake. Not sure how to deal with that.
Dont trust us with information lol.
Song is Petite Meller - High Forever (Marlon Hoffstadt Remix). Which is annoyingly catchy
thanks.
This industry is full of immature children who cannot keep their mouths shut for 24 hours. They have no clue what massive coordination this requires.
They're apparently not trustworthy, and behave as children.
>As a result of the leak and pre-announcement around 20:00 UTC, Calif, one of the security firms that had reported the issues mistakenly published a video. It showed their proof of concept working against a number of targets and revealed a few additional, though limited, details about the vulnerabilities.
You can see the same demo video on their site: https://calif.io/php2shell/#php-5-4-0-rce
1: https://youtu.be/Ci_zad39Uhw?t=15
Calif more than Chainguard, but still Chainguard too.
Calif was obviously twitching to claim credit for their findings, and what I find troubling is this eagerness was more important to them than giving users a chance to update their systems.
Even if they ended up waiting only a day post release I would still say it seems a bit too soon. Maybe a week after. Not everyone’s system have aggressive patching schedules and PHP is soo heavily used.
It's maybe not professional to release an omg panic video, but not even remotely the same thing as what chainguard did.
I dont know why the timing of the bug is surprising to you.
They took it down but someone (Kirill Firsov: https://x.com/k_firsov/status/2112641845926989944) added the anime stuff and reposted it. Totally irresponsible.
Edit: okay he didn't add the anime, but he reposted it.
I want to go back to talks by bored defcon guys or gals calmly stating how they obtained private keys of a globally trusted CA with just a bit of math. Which, mind you, ate the other part of the teams time.
Now I know how much I hustled today likely due to that atrocity of a video. I think I will take a vacation day and sit in a forest near a river on friday.
Embargos make sense if no one is exploiting it yet, but if its out in the wild, the best path forward is releaseing as soon as possible.
[0] https://x.com/J4ck3LSyN/status/2112703096443097263
[1] https://x.com/J4ck3LSyN/status/2112753673235501137
"Cloudflare successfully deployed multiple mitigations for all its customers while the vulnerability was leaked before the official release date and we observed attack traffic targeting our customers."
If that's not an in-the-wild exploit, it literally couldn't have been further away than one or two more iterations.
From Cloudflare’s perspective, there is no downside to saying what they observed, and deploying defenses.
We may never know if this was actually being successfully exploited prior to patch availability.
Was there even an incident co-ordinator here? Was any of the parties talking to each other?
so, if they saw evidence of exploits in the wild, it is not unusual at all to release the fix.
There is a MASSIVE difference between a small number of people exploiting in the wild, and a few hours after a release a bazillion script kiddies doing so. The move to release and break an embargo was childish, absurd, self-serving, and lacking in all common sense. Not to mention, breaking trust.
I agree with those in this thread who say, never going to do business with them. Ever.
I highly doubt their paying customers feel the same way
I would let Chainguard deal with the patches post official release like all other unreliable parties.
As a customer, I'd immediately assume no one in the security community would trust them again, and seek someone else to give my money to. Why pay a) an untrustworthy company who b) now is not trusted by anyone and c) won't ever be able to early release again anyhow? And if anything, are left out of important conversations.
Way to burn bridges. You only get to be an asshat once.
I have seen no evidence to back up this claim. Quite the contrary actually.
TIL observing an exploit in the wild, after the details leaked during the embargo and pushing a fix for your customer is "being an asshat".
Because if there are already publicly available exploits seen, it makes no more sense to keep holding patches back just so that OpenAI etc. can get a nice press release
[1] https://news.ycombinator.com/item?id=50176735
1. There was a leak of the pre-announcement
2. Calif sees leak, goes and releases dancing anime girl video before pulling it, goes viral
3. Chainguard sees dancing anime video, doesn't recognize Calif, and thinks the exploit is in the wild
[Edit 3a]: Or alternatively, Calif's leaked video causes the PoC to be found by others by providing enough of a lead
4. Chainguard releases updated binaries thinking it doesn't matter, causing the real exploit to then be found early
5. Which then causes PHP Foundation to basically have a meltdown
Personally I blame Calif for the unprofessional video which absolutely completely looks like some script kiddie and for being too detailed [edit 2: assuming Calif produced it and the anime wasn't a prank edit]. In this case it wasn't just unprofessional fun but may have literally messed up the responsible disclosure.
I'm not sure about this part. I saw https://jackalsyn.com/posts/php-up-2026.html going around a bit over 12 hours ago, before the chainguard images went out, and plenty of other chatter of exploitation happening in the wild.
Still not great though.
https://calif.io/php2shell/#php-5-4-0-rce
You have your source code public, so you can't commit/push because then you've leaked info.
Your build pipeline is also likely public, so you can't just build from another source code... source.
You can create a private repo, but now you also need to build a private build system.
You don't get the same generous build resources for private repos, so you either:
a) simplify the build to just the essentials
b) pay
Unless you have significant funding, you'll probably go for a).
Now you have to maintain two systems, also with some differences between them.
Plus you need a way to get the build artifacts from the private system to the public one, typically at a fixed point in time.
Any failure/delay (e.g. less powerful build on private repo, errors because you exercise it rarely) causes people to get upset that you didn't respect the announced time.
Maybe you're also using things like image attestation, another can of worms.
But they had _two months_ to figure out how to overcome them. For them to not have figured out a solution and have pre-built images ready to push as soon as the embargo lifted is ridiculous.
Now of course, there are ways around that if required, but they are, by design, manual, cumbersome and slow. I could not create a release build locally in a few minutes. A few hours maybe if the whole organisation dedicated itself to it.
PHP is written in C, I'd be surprised if similar problems didn't lurk in other places.
Maybe they will construct the private pipeline after this crisis. Millions of websites are probably now waiting for all the downstream Docker images to be built and pushed.
It would be great, but it's also something to manage. Just keeping it working with all the changes that happen across all projects would be something.
But the top 1 percent by usage/downloads, I can imagine, and PHP probably qualifies into that top slice.
But let's go with that scenario, now you have all sorts of problems, from practical to security, like:
- accidentally/maliciously building the wrong thing? there's a patch, people agreed to it, but the wrong commit is built
- building for multiple architectures/versions?
- signing keys on someone's laptop?
- worse, image push/write keys on someone's laptop?
And so on.
The problems you bring up here are eminently solvable too. Yes there are many solutions for building for multiple architectures and versions. Cross builds, containers, VMs, or just having more than one computer locally. You can address building the wrong thing with reproducible builds, but for a one off patch like this it's not the most likely problem and relying on a trusted maintainer temporarily during an emergency is better than waiting for a glacially slow cloud build while exploits spread. Pushing builds to the cloud is inherently a cloud step, so it's fine to leave keys for that in the cloud, but there should be a process for publishing local builds and it should be essentially instant when the build is already done.
You should always have a local build process that replicates CI. There is no excuse.
For all the long-tail users, PHP is a distro package. That route needed the most work instead, and it got it.
I'm quite certain that your exact sentiment is why the images weren't updated immediately: I'm afraid PHP foundation doesn't see docker images as a particularly important distribution system.
I wouldn't be that certain.
I think it's likely that the vast majority of PHP Docker image downloads are for local development, for one thing.
But nevertheless, it would not come close to the number of websites running PHP from the Ubuntu Server packages. It is absolutely right that the effort was focussed on the distribution route. Because PHP is not "expert" hosting territory; it is still more often whole cloud VMs running the distro packages.
Being given advance notice of an upcoming security fix is an immense privilege, organisations should turn around quickly when given such information, they shouldn’t be given more than a couple of days to update IMHO, it’s not like the rest of the world can wait.
(It is often much cheaper for an enterprise to just get a long term support contract than to properly upgrade to a modern PHP stack.)
I've experienced similar to this. I know I shouldn't be, but I still get amazed every time I hear that some small company is still using an old DOS machine running core software based on dBase that nobody has any clue on how works and the developer has since moved on to a higher plane. One person told me that they sweat every time the power goes out longer than the attached UPS can run holding their breath that the machine powers back up.
Now I don't know what happened here. Maybe PHP foundation reached out to docker but got no help, maybe they simply forgot about docker as distributor of their software.
Okay, but how close did they get? Were they close enough that a determined actor could've (perhaps using an LLM) managed to get to a functioning exploit?
It sounds cliche but yes, they should rewrite PHP in rust or other memory safe language.
Because now a vulnerability in PHP affects practically 100% of the users. But with more diversity, it would affect only a part. And if one runtime proves insecure more often than another one, organizations have a choice to move to an interpreter that better fits their security needs.
I bet that takes a week at most.
I'm really not sure how you mitigate this issue, other than serving static HTML and reducing the attack surface to cut out dependencies like PHP.
PHP2Shell: AI uncovers critical RCE in PHP
https://news.ycombinator.com/item?id=50180836
Patch the Planet: global response coordination as agents uncover critical vulnerabilities
https://news.ycombinator.com/item?id=50181175
Upcoming critical PHP security release (CVE-2026-109618 / -109619)
https://news.ycombinator.com/item?id=50172041
This is the real story. You ship one of them (the breaking change) with —fuck-my-shit-up or something as a compile time option to switch to the other fix.
for legacy systems, how difficult would it be for an attacker assisted by "AI" to stage the conditions
on a scale of 1 (easiest) to 10 (hardest)
This was partly me. I take responsibility for posting it on HN. I am truly sorry for the disruption this has likely caused.
My contact had informed me there was no embargo, and told me her contact had told her the same. It transpired this morning that was a lie.
>On October 21st at 07:41 UTC, more than six hours before the coordinated release, Chainguard unilaterally broke the embargo and published built images containing the fix. Like other distributors, Chainguard had received the patch in advance so its users could update the moment the vulnerabilities were disclosed. Any attacker could now study the fix in binary form while every other PHP user was still waiting for the release. This disregarded weeks of work by many parties toward a coordinated disclosure.
On the plus side, at least I'm not Chainguard. Yikes.
>This disregarded weeks of work by many parties toward a coordinated disclosure.
Good. Coordinated disclosure meant that people couldn't protect themselves. That's not a good thing. It meant sites were forced to shut down instead of simply updating.
Maybe your calculus could hold if Chainguard's registry were the primary distribution mechanism for PHP in use worldwide (such that the benefit of an updated image would at least be widespread compared to the sitting ducks relying on the other repositories), but it isn't.
Trying to say that uncordinated disclosure is bad because people might get hacked yesterday instead of today just is not a convincing argument to me. I prefer the solution that results in the minimum amount of downtime for sites and sites provide value to society.
They didn’t get to unilaterally decide that the embargo was inconvenient and publish early.
https://news.ycombinator.com/item?id=50176735
The guide to PHP security:
Use something else.
The end.