SSH / SCP Instability

[Thorgear]

I'm running several ASUS routers, some with Merlin, one with stock FW, and one with Tomato. I have a problem that occurs when I have an SSH session open (via PuTTy) and an SCP session open (via WinSCP). The problem occurs on all three FWs.

Both connections will get dropped intermittently, but often, and I can't get a new SSH or SCP connection for a few minutes. I'm pretty sure the problem is triggered by writing to files through either connection, but it's not clearly reproducible. Restarting Dropbear (via Telnet) allows me to get a new SSH or SCP connection immediately.

 

[sfx2000]

Restarting Dropbear (via Telnet) allows me to get a new SSH or SCP connection immediately.

Well one - why are you running telnetd in the first place?

Second - what are you doing that causes Dropbear to run out of memory (e.g. the SSH/SCP comment)?

Maybe you're just exceeding the capability of a dual-core ARM (or a single core MIPS, can't make assumptions here), but generally, the kernel, to protect the OS, will kill processes that run out of memory...

 

[Thorgear]

Sfx please, please stay out of my threads.

 

[sfx2000]

Sfx please, please stay out of my threads.

I fail to see the flames here...

1) Asked why you're running telnetd - this is a security concern for you - telnet is pretty broken
2) You're crashing dropbear - what are you doing to crash it, because it should not crash - so that's good info for the community.

So you can see this as being one thing, but these are kind inquiries as to what you're doing to get in to this post.

not understanding your position here...

 

[sfx2000]

Sfx please, please stay out of my threads.

You edited your thread - nice... you accused me of being inflammatory, misdirecting, and generally not helpful.

Doesn't solve the issue - I asked the question as to why/what? In any event, I'm here... and I'll be on threads that I find interesting...

I'm asking you thing think about the why/what/how - and whether this is a should - just because it's there doesn't mean you should use it...

And one might see this as misdirection - I'm asking for a level of critical thought whether one should be in that position or not.

 

[sfx2000]

Now that we have that out of the way - what are you doing to crash dropbear?

 

[Thorgear]

My strong reaction is based on this and other times you've posted in my threads. Your comments often seem inflammatory. Maybe I'm overreacting. I appreciate your help, and I definitely want to give back to the community. OK, let me start over.

I need telnetd as a fallback. I don't have security concerns.

I don't see Dropbear running out of memory or getting restarted by the OS. I'm looking at a few files and making very few, very small changes. Restarting Dropbear fixes the problem, but so does waiting a few min. What makes you think it's a memory problem?

This problem occurs on four N66s running Merlin, one N66 running Tomato, and one AC68 running Merlin. The common denominators are dropbear, WinSCP, and PuTTy.

Because this is an intermittent problem, I think it's too difficult to diagnose. I'm looking more for the "Oh, yes, I've seen that before." kind of response. Otherwise, I'm wasting other people's time. In the absence of that kind of "Ah, ha!" response, I'm looking for a work-around.

Thanks.

 

[sfx2000]

I need telnetd as a fallback. I don't have security concerns.

on the lan - that's probably ok, but on the WAN side, I do suggest you reconsider...

I don't see Dropbear running out of memory or getting restarted by the OS. I'm looking at a few files and making very few, very small changes. Restarting Dropbear fixes the problem, but so does waiting a few min. What makes you think it's a memory problem?

Because it doesn't crash - the kernel just stops the process and terminates it - it's the OOM (out of memory) and this might not be in the kernel config to print this out to syslog...

The point that you can start this up says something - the question now is why?

This problem occurs on four N66s running Merlin, one N66 running Tomato, and one AC68 running Merlin. The common denominators are dropbear, WinSCP, and PuTTy.

yep - and now the question why dropbear is killed in the first place - it's one thing to just do VPN to copy files, and it's another to login in a remote server and open up a few files - just opening files, depending on the file type and application - this can generate a lot of socket requests, and to dropbear (or openssh) each one takes a bit of memory...

Because this is an intermittent problem, I think it's too difficult to diagnose. I'm looking more for the "Oh, yes, I've seen that before." kind of response. Otherwise, I'm wasting other people's time. In the absence of that kind of "Ah, ha!" response, I'm looking for a work-around.

Intermittent issues like this - well, that's the problem, eh? And there is likely no "ahah" moment that's going to solve that one.

So let's look at the use case first, rather that the symptoms after the fact - maybe it's how things are done, and dropbear is running off the pier and the kernel kills it out of kindness..

(repost/edit to clean up quote thread markup)

 

[sfx2000]

So let's drill down - Dropbear is fairly robust, but it's a single process, and will request memory

Making some assumptions - which are probably incorrect

The remote file we're trying to open - a word file, bless microsoft, can easy take up dozens of sockets, which on the LAN over samba is probably not an issue

Then we have the remove VPN processes - OpenVPN and the TUN/TAP interface - not sure what you're using here, but that takes up RAM, and AsusWRT as a transport, does try very hard to keep memory sane here.

So we have an application, going over Dropbear perhaps, over OpenVPN on something that is memory tight in the first place - we might be exceeding capability here..

Crazy stuff, and tough to work through - opening a file remote over VPN, with a router that is doing VPN and working hard in the first place, it's a bit of an ask...

 

[System Error Message]

i dont see the issue of @sfx2000 being in your thread, he offers advice and jokes just as i do and no one seems to tell me off. Though my jokes usually require more intelligence to decipher. He has quite a lot of experience and age so you should at least show some respect for him and hear him out.

 

[Thorgear]

on the lan - that's probably ok, but on the WAN side, I do suggest you reconsider...

Only the Tomato router faces the Internet, which is why I don't run telnetd on that router. None of the routers have open services on the WAN ports.

Because it doesn't crash - the kernel just stops the process and terminates it - it's the OOM (out of memory) and this might not be in the kernel config to print this out to syslog...

If it can't be seen by ps (top, etc), and it isn't in the syslog, and I don't see a memory leak, then how do you know it's happening?[/QUOTE]

yep - and now the question why dropbear is killed in the first place - it's one thing to just do VPN to copy files, and it's another to login in a remote server and open up a few files - just opening files, depending on the file type and application - this can generate a lot of socket requests, and to dropbear (or openssh) each one takes a bit of memory...

I don't see how you're certain that Dropbear is getting killed. However, I'll take it on faith.

Intermittent issues like this - well, that's the problem, eh? And there is likely no "ahah" moment that's going to solve that one.

That's my point. If this isn't a know problem, and clearly it isn't, then it's not worth wasting time trying to diagnose it. It just isn't important enough to me, and I have a fine work-around. I'm happy to replace the SCP connection with FTP, as soon as I can get vsftpd to let me access the root dir.

See: http://www.snbforums.com/threads/ftp-access-to-root-dir.33873/

 

[sfx2000]

I don't see how you're certain that Dropbear is getting killed. However, I'll take it on faith.

That you mentioned that kicking Dropbear, and things start to work again - that's what is bugging me - and perhaps this might be a hint - you're in the middle of that mess, lol... and it's pretty darn odd if you ask me...

It's pretty obvious this is an edge case, considering capabilities of the devices/builds at hand - but as you mention earlier - there's a couple of constants - and dropbear sticks out...

 

[Thorgear]

The remote file we're trying to open - a word file, bless microsoft, can easy take up dozens of sockets, which on the LAN over samba is probably not an issue

I'm opening only small text files like scripts and configs. I generally only read the files in WinSCP. I edit them in vi from the SSH shell.

Then we have the remove VPN processes - OpenVPN and the TUN/TAP interface - not sure what you're using here, but that takes up RAM, and AsusWRT as a transport, does try very hard to keep memory sane here.

So we have an application, going over Dropbear perhaps, over OpenVPN on something that is memory tight in the first place - we might be exceeding capability here..

I deliberately don't run anything CPU or memory intensive on these routers. No VPN, Samba, guest networks, AiProtection, QoS, BL, other than FTP, no USB apps, AiCloud. On all but one of these routers, I don't even have a firewall. Depending on how you measure memory, I run with 65% to 75% free RAM.

Crazy stuff, and tough to work through - opening a file remote over VPN, with a router that is doing VPN and working hard in the first place, it's a bit of an ask...

Depending on how you measure memory, I run with 65% to 75% free RAM. Again, depending on how you measure it, nominal CPU is under 10%. Fully loaded, I rarely go over 60% on the AC68, lower on the N66s.

 

[Thorgear]

That you mentioned that kicking Dropbear, and things start to work again - that's what is bugging me - and perhaps this might be a hint - you're in the middle of that mess, lol... and it's pretty darn odd if you ask me...

It's pretty obvious this is an edge case, considering capabilities of the devices/builds at hand - but as you mention earlier - there's a couple of constants - and dropbear sticks out...

OK, I get that. Makes sense. Thanks.

 

[sfx2000]

If possible - don't put it off or put up with it - seriously...and it's good that you brought it up.

Hard enough when it's intermittent, but I'll agree with you - something is happening there - just takes some time to sort the use case.

Find that use case - and someone can fix it, and the community benefits - which is a win/win

 

[Thorgear]

i dont see the issue of @sfx2000 being in your thread, he offers advice and jokes just as i do and no one seems to tell me off. Though my jokes usually require more intelligence to decipher. He has quite a lot of experience and age so you should at least show some respect for him and hear him out.

I goes back to prior threads. I feel like there's so much time spent going back and forth on things that have nothing to do with the problem at hand, or defending my position. For example, in a recent thread, it felt like he was insisting that a well-formed password would solve any wireless security problem. I had to prove that that wasn't true in this case. We went back and forth about several things. It was so frustrating. I had asked a simple question, which in the end, had a simple answer.

As far as age and experience goes, I'm realtively new to routers, but I've been a programmer since before the PC existed. Having age and experience myself, I offer tremendous admiration and respect for people who have the same. I gave Sfx a lot of respect and appreciation in the beginning. This frustration built up over the course of several threads.

 

[Thorgear]

OK, at this point, I feel like I've reached the point where I've become completely irrational. This is exactly the kind of thread I was trying to avoid. We are technical people, in a technical forum. It's not the place for group therapy.

I felt like I was about to run into a very frustrating thread, and I wanted to nip it in the bud. That's why I edited the message down to one sentence. My problems with Sfx is my problem. If he's not frustrating anyone else, then it's most definitely my problem.

I really wish there were some way to get out of this thing without anyone, including me, losing face.

 

[sfx2000]

As far as age and experience goes, I'm realtively new to routers, but I've been a programmer since before the PC existed. Having age and experience myself, I offer tremendous admiration and respect for people who have the same. I gave Sfx a lot of respect and appreciation in the beginning. This frustration built up over the course of several threads.

Hehe - explains why we're both bothered by this situation...

We're both #greybeards - and we have different approaches - still in the biz with DevOps/Agile, blah... Slack, give me a break - I participate here because a) it's fun, and b) I learn a thing or two..

 

[sfx2000]

I really wish there were some way to get out of this thing without anyone, including me, losing face.

Let's try to find that edge case - ok?

 

[Thorgear]

Hehe - explains why we're both bothered by this situation...

I get it. I don't know what's up with me. Maybe it's that the forum approach is so slow for me. It would be so much faster and easier to just talk.

We're both #greybeards - and we have different approaches - still in the biz with DevOps/Agile, blah... Slack, give me a break - I participate here because a) it's fun, and b) I learn a thing or two..

We've both seen a lot of change. A technology revolution really. I switched careers about three years ago. Now I write math books, of all things.

Well, in any case, I do apologize. Thank you all for putting up with my "episode".