Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way

This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth

Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.

For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.

I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..

2020s: ha! with some advanced probabilistic models, we may be able to deduce something about what is being typed behind one of our layers of encryption, let's sent 100 packets per keystroke to mitigate that

https://www.brendangregg.com/sshanalysis.html

The 2023 patch should finally fix that 2004 issue.

Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.

1) I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing. (Or rsync'ing a file, which ditto).

2) I'm not in a high-security environment where nation-states have an interest in sniffing my keystrokes.

3) I often open SSH connections to servers in other continents. Those underwater cables have massive bandwidth, but they're also in constant use by thousands upon thousands of people. So anything I can do to reduce my bandwidth by 100x is probably worth doing.

Any reason you can think of why I should not be setting ObscureKeystrokeTiming=no in my ~/.ssh/config?

Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.

For example, "nc" (netcat) is pre-installed on all platforms where ssh is.

I am working on a high-performance game that runs over ssh.

Found your problem.

But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.

That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier!

Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"

Granted... it would increase the cost (since you're adding reverse proxies) but it would be a quick way to get acceptable latency, rudimentary DDoS protection, and you could try different connection options independent of the main app's logic.

It would be hard to estimate how much latency you're adding with a SSH2 reverse proxy in this case, but it's probably lower than one might think.

The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] But I loved this write-up. It was readable and explained the problem the OP was encountering and proposed solutions well.

Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!

I am working on a high-performance game that runs over ssh. The TUI for the game is created in bubbletea 1 and sent over ssh via wish.

The game is played in an 80x60 window that I update 10 times a second. I’m targeting at least 2,000 concurrent players, which means updating ~100 million cells a second. I care about performance.

High performance with ssh and wish? For sure not. Rather use UDP over secure sockets. Or just normal sockets. Even Claude would come up with much faster code than the ssh/wish nonsense. Or mosh, but this also too complicated.

When you say “LLMs did not fully solve this problem” some people tend to respond with “you’re holding it wrong!” > > I think they’re sometimes right! Interacting with LLMs is a new skill, and it feels pretty weird if you’re used to writing software like it’s 2020. A more talented user of LLMs may have trivially solved this problem.

So one thing I only recently figured out is that using ChatGPT via the web browser chat is massively different from using OpenAI's code-focused Codex model / interface. Once I switched to using Codex (via the VS Code extension + my own ChatGPT subscription) the quality of answers I got improved massively.

So if you're trying to use LLM to help with debug, make sure you're using the right model!! There are apparently massive differences between models of the same generation from the same company

I am working on a high-performance game that runs over ssh.

Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead Step two, tune TCP/IP stack Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)

I did add a trackpad monitor though. It shows my raw MacBook trackpad data.

https://varun.ch/contact/

In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.

Why not just add random "jitter" to the keystroke packets, but keeping just the 1 actual packet?

And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?

>> That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical.

Switching to telnet instead of SSH might be an option.

I am working on a high-performance game that runs over ssh.

By 'ssh', you mean 'ssh' (library/program + protocol + encryption + decryption) on top of TCP/IP, on top of the Internet, right?

OK, I'm not against it... but you do understand that there are all kinds of ways for that to slow things down, right?

Your issues may (or may not!) include such things as:

o Nagle's algorithm AKA buffering AKA packets not being sent until N bytes (where N > 1) ready to send, as other posters have suggested;

o Slower encryption/decryption on older hardware (if users with older hardware is a target market, and if the added loss in speed makes an impact in gameplay, depending on the game, this may or may not be the case...)

o The fact that TCP/IP (as opposed to UDP / Datagrams / "Raw" sockets) imposes a connection-oriented abstraction, requiring additional round trips of ACK ("I got the packet") RESEND ("I didn't get the packet") on top of the connectionless architecture that is the Internet (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...), which adds additonal latency, so, for example if a rural user in Australia experiences a 350ms delay for a raw packet to get to a U.S. server (or vice versa), then TCP/IP might make this 700ms or more, depending on the quality of the connection!

o The speed of the game limited to both the bandwidth and latency of the slowest user (if a multi-player game, and if the game must not update until that user "moves"... again, game architecture will determine this, and it wouldn't be applicable to all games...)

Now, you could use UDP, as other posters have suggested, but then you must manually manage connections and encryption...

That may be the right choice for some types programmers, some types of games/applications -- but equal-and-oppositely it may be the wrong choice for others...

Anyway, wishing you well with your game development!

I haven't used SSH (at least, not in a debug capacity), so I'm not sure what SSH debugging options exist -- but it would be nice if SSH had a full logging debug mode, which would explain exactly WHY it chose to send any given packet that it did along with related helpful information, such as latency/time/other metrics, etc., if it doesn't have this/these feature(s) already...

If you get clever and write a client to ensure sensitive data like passwords or email are sent in a burst you could just use an encryption library just for that data instead.

Keystroke obfuscation can be disabled client-side.

please never do that (in production)

if anyone half way serious tries they _will_ be able to break you encryption end find what you typed

this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.

One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.

Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.

It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?

It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.

very confidently told me that my tcpdump output was normal ssh behavior:

I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.

Vibe coders man...

I could vibecode an SSH zmq daemon in an afternoon.

I am working on a high-performance game that runs over ssh.

WAT. Please no.