> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.
What a weak minded statement. Just admit you messed up and call it a day.
Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work
on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
I would consider OP's comment as written to be born of bigotry and not a genuine examination of how Indian culture within Emudhra is influencing their decision to pull out. Too bad because I definitely think there is a component.
> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
Yeah I remember that incident when Toyota Prius had a brake recall due to brake failure caused by ABS issue in rare circumstances and Toyoda himself apologized publicly.
This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.
What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.
It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until to experience situations in which they currently suck.
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
My mistake. English is not my first language. In my native language, the word racism is used for discrimination or prejudice against anything foreign (nation, race, language, ...).
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
Today "racism" is used as a synonym for "bigotry". Would you agree it was bigotry? Or do you only have a problem with people using words wrong, and not bigotry itself?
Whynot use the correct word for it, which is prejudice? Racism is prejudice against a race. You can be prejudiced to any group of people, doesnt mean that you are biggoted.
> Whynot use the correct word for it, which is prejudice?
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Also modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."
So they didn't even use their own certificates? :)
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
Wow this press release is a real PR disaster. It's really just about their business value.
Starting the statement with "a global digital trust company" is so ironic on that context.
> There were no end entity certificates issued out of the affected CA’s.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc.
2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.
True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.
now the question for the media for the next week :
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.
What a weak minded statement. Just admit you messed up and call it a day.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
* https://en.wikipedia.org/wiki/Izzat
* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)
Also, this is the response from official Indian Ministry of Electronics & IT:
…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”
Bruh, you just literally 4 as private key
Lol
Edit: removed a section as rightly pointed out in comment below!
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
People like you are the ones make the world place to live in
It raises the question why these CAs agree to participate on this forum and then complain about doing so.
1: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144
You apparently don't understand Indians.
e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
For a something this big they'd probably have execs making a visit to a certain forest.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)
I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."
A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.
Some people just follow a spec and don't stop to think what they spec is supposed to accomplish.
Good that we are now investing thousands of engineering hours globally to fix this.
Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.
Sorry, it's been a long week.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until to experience situations in which they currently suck.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
We've had to rework part of our application and certificate use because of this.
My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.
Let's call it "Let's Decrypt!"
No, this is not even remotely sensible.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
house collapses and people get injured.
concrete manufacture says we no longer build houses cause it’s too small of a business.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
The key needs to mention which curve it is using.
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
https://en.wikipedia.org/wiki/Endianness
I had a bunch of openwrt routers based on the ADM5120 SoC, some of them were of different endianness, totally wth:
https://linux-mips.org/wiki/Adm5120
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Also modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
Yes, that's literally what it means.
This seems like a weird statement given that anyone can issue certificates out of the affected CAs.
They just didn't mention that now anyone can be their own Certificate Authority.
So they didn't even use their own certificates? :)
Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.
Congratulations to those who wrote or prompted this statement.
lol, I didn't realize the latest comic was also topical.
https://xkcd.com/3314/
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
Popcorn recommended for the read!
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
Is 14 hours a diligent response time for something like this?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
https://security.googleblog.com/2014/07/maintaining-digital-...
Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?
From everything that came out they seem terribly unqualified.
https://news.ycombinator.com/item?id=50194909
My trust in those vetting the CAs is completely broken now.
First one I clicked on?
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
We have established he could at least count to 4 :)
Are they liable?
Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.
Don't these people pay any attention to politics?
You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.