LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance, and debugging what's wrong is not trivial.
So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!
There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do.
This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this.
> I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them.
I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.
1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them.
2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime - but it doesn't work if it's at the cost of shitty performance or failing toolcalls.
> 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them.
OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable.
> 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
In theory, yes. In practice, no, there are differences. Ollama, llama.cpp, vLLM and SGLang all say "ChatCompletionRequest" compatible, but the devil is in the details, they don't have 100% the same request/response schema across all compatible models.
> 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
Yup, makes sense! If you're unable to find models when you use OpenRouter, it makes zero sense to continue to use OpenRouter.
> You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime
Huh? That's not how it works or does it make sense, nor have I've seen anyone use OpenRouter like that.
if you want to control the routing, you'd lock down providers
I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate.
If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models.
This is quite literally the point of OpenRouter.
Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.
> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate.
Why do you care about the public benchmarks at all?
The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it.
Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up.
I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices.
For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out.
If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity.
I don't think anyone serious is just willy-nilly making individual requests against OpenRouter
Despite your confidence, that is indeed the basis of this massive corporations entire business plan.
If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRoute
If OpenRouter is only for bad accuracy, they should say as much and fade into deserved obscurity.
I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible.
For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
> to the point I wonder if I should just use the providers directly.
How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful.
Personally I pin a single vetted provider in the interest of minimizing risk.
This is very useful information and comes at a perfect time! I use Openrouter for my newly released running tracker (I use it for live coaching and post-run debriefs). I've benchmarked a bunch of models over time to evaluate their aptitude for this specific task, and have noticed that sometimes a model can underperform for seemingly no reason. I'll be sure to include model providers in my benchmarking suite going forward!
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
> Providers probably serve quantized versions without disclosing it
It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers.
In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
I mean that is addressed as well later, even when they do declare the quantization, doesn't mean you'll get better performance than the lower quantized one. I'm guessing they're doing something similar to what stadia was doing, saying you are playing games in 4k, because you're getting a 4k stream, but the game itself is running in 1080p
The best part about OpenRouter is 200 OK is probably hardcoded into their responses.
I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
Under privacy, ignored, I put providers that always suck. Digital Ocean, OpenInference, DeepInfra, AtlasCloud, and Alibaba (that one is different, it doesn’t suck in the same way as the others with a dumb model, but instead it’s heavily censored and doesn’t like being used as RP model in Skyrim).
Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.
I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.
I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.
I've used Requesty (https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.
There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time.
RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term.
How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in.
The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor.
As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please.
Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
Very informative article, but I'd say all of this is in favor of OpenRouter, not against it. It gives you full flexibility, not just of models, but of providers. That's the offer!
I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
This is actually expected behavior. No content and no tool call is the same as content-only: the agent decided it's done. Anthropic has done this for a while.
Yea I can believe all these. I've personally have been having issues on these points:
"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.
"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
I believe OpenRouter addresses this with sticky routing and keeps a conversation on the same provider and falls back if that provider becomes unavailable via an explicit session_id to keep requests together. You’re right though that upon a failure cache is not portable and incurs additional costs.
I use Fireworks for a production app. Much lower volumes though. My only gripe is their serverless offering for reranking has only 1 model which is Qwen 8b and it's quite expensive.
I never tested this but always suspected it. When I learned that providers differentiate themselves on how they optimize and host the model (otherwise, why would you choose one over another?), I figured some were less "give me the best possible experience" than others.
Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.
The way that OpenRouter frames it with having calls routed to whatever provider is handy encourages you to think that they’re all the same in basic functionality. Really sours me on OpenRouter.
could not agree more one pet peeve of mine is that using strict json output also does not work on all the endpoints of the models so I have to go through them one by one try it out and then only whitelist them.
Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs.
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
Just chiming in that I've never had this experience with them. I set, hit, and depend on those limits regularly. I would be wildly interested in reading what their support analysis had to say about this situation, if such a postmortem was done.
great post, I just put $100 in credits on open router to try different models, I think maybe using the "real" provider is the way forward once I've spent my credits and decided which one I want (for now).
Be sure to spend them! Another great feature of OpenRouter is that they will take your credits after a year. They just delete them. A ToS-legalized theft if you ask me. (Even if you are still actively using those credits.)
As of right now, they only expire credits if you don't use the account at all for a year. This is different from, say, OpenAI, where your credits expire unconditionally. That said, they do deserve the right to expire them regardless, so this could change in the future.
I noticed some strange behavior when I was setting it up to avoid providers that collect data, hence blocking default providers. Definitely it is worth filtering well tested providers, which is a feature OpenRouter provides.
The article that I read addressed this point specifically. "Trusted" providers do not maintain consistent performance, and do not have the same performance across different models.
Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more providers", it just shows a clearly broken modal. And the the next two graphs are even worse. Really all of these graphs are pretty and completely 100% useless.
I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important.
As expected vibecoding bros cannot even read the manual properly.
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.
The article mentioned they pinned the provider and model and the result was bad as well with their own shenanigans. That part was in the end of the article so maybe you missed it.
It's good to quantify the extent to which a lot of the stuff on there is just vibe-hosted. But I think it was always pretty apparent that this was the case. You've got numerous providers all running the gamut of:
So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!
There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this.
> I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them.
I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.
2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime - but it doesn't work if it's at the cost of shitty performance or failing toolcalls.
OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable.
> 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite?
In theory, yes. In practice, no, there are differences. Ollama, llama.cpp, vLLM and SGLang all say "ChatCompletionRequest" compatible, but the devil is in the details, they don't have 100% the same request/response schema across all compatible models.
> 3. Great example, if amazon does not lead me to good products, I will stop using it, and this is why I dislike amazon. There are so many crap products, and the search seems to try to push crap products instead of what i'm actually looking for at a good price.
Yup, makes sense! If you're unable to find models when you use OpenRouter, it makes zero sense to continue to use OpenRouter.
> You take a bunch of providers with not great uptime, put them in a pool and now you get great uptime
Huh? That's not how it works or does it make sense, nor have I've seen anyone use OpenRouter like that.
If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models.
Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.Why do you care about the public benchmarks at all?
The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it.
Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up.
I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices.
For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out.
If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
Hmm, yeah, good and condense version of what my previous comment said. I'm much impressed by your reading ability.
For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
> Fireworks scored 46% on TAU, a 30 point gap
Another surprise was DigitalOcean being bottom of barrel too.
Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.
How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful.
Personally I pin a single vetted provider in the interest of minimizing risk.
Android app called "AudioRun"
https://play.google.com/store/apps/details?id=com.audiorun.a...
https://apkpure.com/audiorun/com.audiorun.app
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers. In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.
It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time.
RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term.
I feel like I must be missing something?
The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor.
As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please.
Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
https://newsletter.semianalysis.com/p/clustermax-20-the-indu...
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
This is actually expected behavior. No content and no tool call is the same as content-only: the agent decided it's done. Anthropic has done this for a while.
Is it, though?
"200 OK, no answer" - insane that openrouter's main feature is literally a fallback and streaming doesn't support 200 no content to fallback to another provider or smth.
"rate-limit by IP"... now it kinda makes sense why deepseek v4.1 flash rate limits me on prod but never seems to happen on local. Makes you have to basically pin Deepseek as provider, since I've never had 429 error on them
If that's the case, I think it's quite unreasonable to complain about a service that has no cost, but I could be wrong.
Noob question: do good harnesses automatically optimize for bad tool calling behavior automatically?
On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...
https://app.answerhq.co/openrouter-ai/articles/credits/credi...
I was so disappointed that I won't consider any of them for at least a few years.
I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important.
What am I missing? Is this really true?
It is pretty trivial to pin a single provider for a model. Better yet, instead of calling the model directly, use presets instead. You can easily change the setting on openrouter without having to update your app every time.
I feel there is still a lot of progress to be made before we can really trust LLM providers.
* Hardware availability
* Competency
* Scruples