Actions

Dev/Qubes Remote Support

From Whonix

< Dev

Development notes: QubesOS to be remotely manageable from on-demand, ephemeral, hidden onion service to dom0/AdminVM.

user documentation: Qubes - SSH or VNC into Qubes dom0

challenges[edit]

  • There are no Open Source remote support tools which are usable by laymen.
    • There is teamviewer but it's proprietary and therefore the teamviewer company could take over any users who use it.
  • The existing Open Source tools do not account for that most users nowadays are behind common NAT routers which make it hard to receive unsolicited incoming connections, i.e. hard to run servers. Would require port forwardings which are difficult for users. The internet is full of users seeking help how and failing to create port forwardings. Users are also often on networks (3/4G) that do not even support incoming (server) connections.
  • Even if some remote support instructions use Open Source software and work on Linux, these won't work easily in Qubes dom0 since non-networked by default.
  • Most if not all existing remote support Open Source software does not use encryption by default.
    • Therefore any man-in-the-middle could take over the connection and compromise the remote support receiver.
      • It's possible to run VNC over encrypted/authenticated SSH but this is not something which laymen users that need remote support are capable to use.

goals[edit]

  • functional for users with reasonable slow internet connections
  • reasonable security
  • Open Source
  • work behind NAT

desired security properties[edit]

  • encrypted
  • authenticated
  • clear and reliable signaling to the user whether remote admin is enabled and whether it is currently in use
  • Even after sys-whonix compromise the attacker should not be able to access dom0 through VNC.

Implementation Draft[edit]

  • Command line tools (which can be called by any GUI).
  • Whonix-Workstation script. (GUI and command line tools would run here.)
  • Whonix-Gateway script. (Tor onion service)
  • dom0 package (Qubes dom0 configuration, SSH/VNC server)
  • Packaging of a (python) GUI [1] (the GUI itself to be written by Marek)

sys-whonix GUI[edit]

able to see already existing hidden services per sys-whonix in a widget.

Adding remote support thought requirements:

  • List configured onion services.
  • Delete them.
  • Create new remote support related hidden onion service. When done, output onion address, port and shared secret for the user to copy paste to chosen secured communication channel.
  • create time based or persistent remote support
  • multiple remote support onions at the same time remove?
  • optionally a feature to send access info to the support agent from the GUI

Missing parts[edit]

Avoidable?

  • Qubes dom0 salt
  • Qubes dom0 GUI

Idea[edit]

general[edit]

  • x2go supports SSH.
    • (Needless to say that SSH or any extra encryption complicates things.)
    • TODO: test if x2go can work in acceptable speed over a Tor onion. FreeNX (maybe outdated), SPICE, x2go... x2go [in packages.debian.org: yes] seems most promising.
  • Run an SSH server in dom0.
  • magic-wormhole?
    • [in packages.debian.org: yes]
    • (File_Sharing#Magic-Wormhole)
    • These code words are quite usable. Example: 7-crossover-clockwork We'd need two of these.

questions:

  • SSH server running in dom0 is OK?
  • I guess running VNC (and perhaps SSH) in dom0 is OK?
  • Things can always be more compartmentalized, secure but also getting more and more complicated, fragile.

I am considering to script this entirely from dom0 using `qvm-run`. Only expectation: installed (use salt?) and non-broken sys-whonix

Draft Concept[edit]

remote-support-provider[edit]

On remote support provider machine.

  • dom0 qvm-start sys-whonix
  • dom0 qvm-run sys-whonix create .onion service config file in /usr/local/etc/torrc.d/40_remote_support.conf
  • dom0 qvm-run sys-whonix restart Tor [1]
  • dom0 qvm-start remote-support-provider-server VM
  • dom0 qvm-run remote-support-provider-server VM install openssh-server
  • make openssh-server reachable through .onion running inside sys-whonix VM
  • dom0 wormhole send connection.zip
    • write wormhole code into script variable
  • connection.zip contains
    • .onion hostname of remote support provider
    • ~/.ssh/known_hosts
    • .auth_private file (private key) for authenticated Tor onion service
    • ssh key
  • tell remote support receiver out of band (telephone, chat, e-mail) the wormhole code
  • wait for remote support receiver to start the remote support receiver tool and to enter the wormhole code
  • inside remote-support-provider-server VM: x2go connect to remote support receiver localhost:port [reverse ssh]

remote-support-receiver[edit]

On remote support receiver machine.

  • dom0 sudo dnf install x2go-server
  • dom0 qvm-start sys-whonix
  • dom0 wormhole receive connection.zip
    • Must be done in dom0. Not sys-whonix. Because goal is sys-whonix to be untrusted.
  • dom0 install ~/.ssh/known_hosts
  • dom0 qvm-run sys-whonix install .auth_private file (private key)
    • sudo sourcefile=~/user.auth_private anon-server-to-client-install (adjust path)
  • dom0 qvm-run sys-whonix test connection to .onion
  • dom0 test connection to .onion
  • dom0 reverse ssh connect to remote support provider .onion
  • There is now an open port on remote support receiver machine over ssh and over .onion available to remote support provider.

qubes-remote-support-provider-gui[edit]

Not needed. Remote support provider can use CLI tools for connection setup. Would be encapsulated into a simple CLI call and telling the remote support receiver the wormhole code. Once the remote support receiver connected to the remote support provider (reverse SSH), the remote support provider can use a $VNC viewer (which is graphical) to connect.)

notes[edit]

  • Can use two wormhole codes if not avoidable.
  • Automatically re-connect if reverse SSH is down.

ssh keys[edit]

challenges:

  • Have to manage file /root/.ssh/authorized_keys using scripts.
    • Use secure, long, script generated passwords instead?

send access into to gpg agent[edit]

> optionally a feature to send access info to the support agent from the GUI (for example GPG encrypted email)

Patrick said: I would discourage that. Challanges:

  • calling e-mail / gpg from command line
  • calling e-mail / gpg from dom0 or adminVM?

Preliminary Work Done[edit]

See also[edit]

x2go[edit]

pros[edit]

  • Easy to use, built-in support for using over SSH.
  • Connection settings for slow/high latency connections (i.e. hopefully useful over Tor).

x2go bugs[edit]

Happening with hardened SSH config file.

https://bugs.x2go.org/cgi-bin/bugreport.cgi?bug=1349 [archive] 

NXPROXY - Version 3.5.99.19

Copyright (c) 2001, 2011 NoMachine (http://www.nomachine.com)
Copyright (c) 2008-2014 Oleksandr Shneyder <o.shneyder@phoca-gmbh.de>
Copyright (c) 2014-2016 Ulrich Sibiller <uli42@gmx.de>
Copyright (c) 2014-2016 Mihai Moldovan <ionic@ionic.de>
Copyright (c) 2011-2016 Mike Gabriel <mike.gabriel@das-netzwerkteam.de>
Copyright (c) 2015-2016 Qindel Group (http://www.qindel.com)

NXCOMP, NX protocol compression and NX extensions to this software
are copyright of the aforementioned persons and companies.

Redistribution and use of the present software is allowed according
to terms specified in the file LICENSE.nxcomp which comes in the
source distribution.

All rights reserved.

NOTE: This software has received contributions from various other
contributors, only the core maintainers and supporters are listed as
copyright holders. Please contact us, if you feel you should be listed
as copyright holder, as well.

NX protocol compression is derived from DXPC project.

Copyright (c) 1995,1996 Brian Pane
Copyright (c) 1996,1997 Zachary Vonler and Brian Pane
Copyright (c) 1999 Kevin Vigor and Brian Pane
Copyright (c) 2000,2003 Gian Filippo Pinzari and Brian Pane

All rights reserved.

See https://github.com/ArcticaProject/nx-libs for more information.

Info: Proxy running in server mode with pid '5072'.
Session: Starting session at 'Mon Jul  6 18:19:31 2020'.
Info: Using errors file '/home/user/.x2go/S-user-50-1594059610_stDXFCE_dp24/sessions'.
Info: Using stats file '/home/user/.x2go/S-50/stats'.
Loop: WARNING! Overriding auxiliary X11 port with new value '1'.
Warning: Overriding auxiliary X11 port with new value '1'.
Info: Using abstract X11 socket in kernel namespace for accessing DISPLAY=:0.
Info: Connecting to remote host 'localhost:51683'.
Info: Connected to remote proxy on FD#5.
Loop: WARNING! Non printable character decimal '10' received in remote data from FD#5.
Warning: Non printable character decimal '10' received in remote data from FD#5.
Connection timeout, abortingInfo: Aborting the procedure due to signal '15'.
Session: Session terminated at 'Mon Jul  6 18:20:01 2020'.

Unable to execute: rdesktop  -g 800x600

magic-wormhole[edit]

A challenge is to get magic-wormhole into Qubes R4.0 dom0 since Fedora 25 (which Qubes R4.0 dom0 is based on) has no package for magic-wormhole and Fedora does not offer backports.

CLI GUI Communication Design[edit]

TODO: plan how this CLI script would interact with the GUI

idea:

1) stdout could just show the wormhole password (currently implemented)

2) stderr could contain debug information (the bash xtrace) (currently implemented)

3) Or better communicate with the GUI thorugh the file system? The GUI could watch for a file /path/to/wormhole_password and once created, show that to the user. Also /path/to/error_output could contain human readable error summaries. Then stdout/stderr could be shown in GUI when a user clicks "show debug information". (Suggested method.)

CLI GUI Timeout Handling Design[edit]

How should timeouts be handled?

  • One way would be using a lot command `timeout` inside this script. That would make this script look more complex.
  • This script could stay as is and another wrapper script could handle timeouts.
  • Or the GUI could handle timeouts. If timeout is reached, just show debug info. This is hopefully only required in development.

Proof of Concept[edit]

ssh and VNC into Qubes dom0[edit]

Using the term VNC loosely here as these instructions are using x2go.

remote-support-receiver sys-whonix setup[edit]

[2]

1. sys-whonix qvm-connect-tcp listener setup

Inside sys-whonix VM. Start socat listener. [3] [4] [5] 

qvm-connect-tcp 22:dom0:22

Should show:

Binding TCP 'dom0:22' to 'localhost:22'...

3. setup authenticated onion v3

[6]

Inside sys-whonix VM. 

sudo virtport=22 hsport=22 hsname=remote_support client=1 anon-auth-autogen

Will show.

INFO: Created torconffile '/usr/local/etc/torrc.d/43_remote_support_hs_autogen.conf'.
INFO: Reloading Tor.
INFO: Giving Tor 5 seconds to create hidden service file.
INFO: Installed ".auth" file (public key) '/var/lib/tor_autogen/remote_support/1.auth' to '/var/lib/tor_autogen/remote_support/1.auth' to allow client '1' to access hsname 'remote_support' onion_url 'xxx.onion'.
INFO: Reloading Tor again to activate ".auth" (public key) file for client '1'.
INFO: You need to provide client '1' with ".auth_private" file (private key) '/var/lib/tor_autogen/remote_support/1.auth_private'.
INFO: Visitors that use Whonix could store '/var/lib/tor_autogen/remote_support/1.auth_private' in '/home/user/1.auth_private' and then run 'sudo sourcefile=/home/user/1.auth_private anon-server-to-client-install'.

4. restart Tor

Inside sys-whonix VM.

[1] 

sudo systemctl restart tor@default

5. get onion v3 domain name

Inside sys-whonix VM. 

sudo cat /var/lib/tor/remote_support/hostname

remote-support-receiver dom0 setup[edit]

1. In dom0. Install packages.

Optional. Some tool to simplify copying files from a VM to dom0 will later greatly simplify the setup. 

mkdir -p ~/bin

nano ~/bin/qubes-copy-from-vm 

#!/bin/bash
qvm-run --user root --pass-io "$1" "cat $2" > "$3"

chmod +x ~/bin/qubes-copy-from-vm

2. In dom0. Install packages. [7] 

sudo qubes-dom0-update openssh-server x2goserver x2goserver-xsession

[8]

3. dom0 SSH server configuration hardening

In dom0.

Optional.

[9] [10]

4. ssh PasswordAuthentication No

At least the following should be set.

/etc/ssh/sshd_config 

PasswordAuthentication No

5. enable sshd

In dom0. 

sudo systemctl enable sshd

6. start sshd

In dom0. 

sudo systemctl start sshd

7. reload sshd

In dom0.

[11] 

sudo systemctl reload sshd

8. check sshd status

In dom0. 

sudo systemctl status sshd

9. dom0 x2go setup 

sudo x2godbadmin --createdb

sudo systemctl enable x2gocleansessions.service

sudo systemctl start x2gocleansessions.service

10. dom0 qrexec policy setup

In dom0.

[12]

Open file /etc/qubes-rpc/policy/qubes.ConnectTCP in an editor with root rights.

Append. 

{{gateway_product_name_vm}} dom0 allow,target=dom0

Save.

11. dom0 ssh authorized_keys setup

Contents of remote-support-provider file /home/user/.ssh/id_ed25519.pub need to be transferred to dom0 /home/user/.ssh/id_ed25519.pub.

Inside remote-support-provider VM.

In dom0. As user. Not root! 

qubes-copy-from-vm {{gateway_product_name_vm}} /home/user/.ssh/id_ed25519.pub

In dom0. As user. Not root! 

mkdir -p ~/.ssh

In dom0. As user. Not root!

WARNING: If you used ssh previously, you might want to append instead. The following command would overwrite your ~/.ssh/authorized_keys. 

mv id_ed25519.pub ~/.ssh/authorized_keys

In dom0. As user. Not root! [13] 

sudo chmod --recursive og-rwx ~/.ssh

remote-support-provider setup[edit]

remote-support-provider sys-whonix setup

1. get 1.auth_private

Get /var/lib/tor_autogen/remote_support/1.auth_private from remote-support-receiver sys-whonix and copy to remote-support-provider sys-whonix.

Could use for example in remote-support-provider sys-whonix sudo wormhole send /var/lib/tor_autogen/remote_support/1.auth_private and in remote-support-provider sys-whonix wormhole receive.

[14]

2. authenticated onion v3 key installation

Move 1.auth_private key to Tor onion v3 private key folder.

Inside remote-support-provider sys-whonix VM. 

sudo sourcefile=/home/user/1.auth_private anon-server-to-client-install

remote-support-provider AppVM setup

3. remote-support-provider VM creation.

Create a VM named remote-support-provider (or any other name) based on whonix-ws-15 TemplateVM with networking set to sys-whonix. Usual instructions for Multiple Whonix-Workstation. Could even skip VM creation and use an already existing VM anon-whonix instead.

4. Install SSH client and x2go client.

Inside remote-support-provider VM. 

sudo apt install --no-install-recommends openssh-client x2goclient

5. Create SSH key.

Inside remote-support-provider VM. 

ssh-keygen -t ed25519

6. View id_ed25519 public key.

Inside remote-support-provider VM. 

cat /home/user/.ssh/id_ed25519.pub

7. ssh client configuration hardening

Inside remote-support-provider VM.

Optional. [15] [10]

Usage[edit]

Introduction[edit]

After making an SSH server available in Qubes dom0 as per above instructions it will be accessible through an anonymous onion service.

ssh[edit]

Test this first.

Inside remote-support-provider VM.

Qubes dom0 requires the SSH key being added to Qubes dom0 ~/.ssh/authorized_keys as documented earlier on this page. 

ssh xxx.onion

x2goclient[edit]

Moved to Remote_Administration#x2goclient.

Test Results[edit]

All times are created with an external stopwatch and should have +/- 2 seconds human caused inaccuracy.

  • Qubes R4.0
    • ssh:
      • ssh connection initialization takes 12 seconds.
      • There is a 2 second delay after each simple command such as ls.
      • Running a simple command such as ls -la in a folder with 60 files takes 2 seconds until results are drawn.
      • Comparable to a SSH connection over clearnet.
    • x2go session type terminal:
      • connection speed setting "modem":
        • Establishing a connection takes 90 seconds.
        • There is an two seconds delay between a keypress on inside remote-support-provider VM until it gets visible in the remote terminal.
        • Running a simple command such as ls -la in a folder with 60 files takes 18 seconds until results are drawn.
        • from opening the x2go viewer to having the full desktop contents drawn it takes 180 seconds
      • connection speed setting "WAN":
        • time to open x2go viewer: 100 seconds
        • delay from clicking menu bar to actually seeing menu bar: 6 seconds
        • Running a simple command such as ls -la in a folder with 60 files takes 18 seconds until results are drawn.
        • Redraw screen after pressing enter: 18 seconds
    • This is much faster with Non-Qubes-Whonix when following instructions for SSH into Whonix with x2go. This might be because Qubes R4.0 dom0 is based Fedora 25 which ships an older version of x2go. It could also be because the x2go version of Qubes dom0 Fedora 25 is older than the x2go version in Debian buster based Whonix 15. The speed issues might be fixed with the release of Qubes R4.1 which will come with a more newer dom0 version of Fedora and thereby with a newer version of x2go.
  • Qubes R4.1

Debugging[edit]

  • Perhaps Exercise setting up a simple onion service web server first.
  • SSH into Whonix-Gateway (conflicts with above Tor configuration)
  • Could copy remote-support-provider folder ~/.ssh key to sys-whonix and then use ssh 127.0.0.1 directly inside sys-whonix to test if the SSH connection is functional. This was successfully tested during development. Useful to exclude any potential Tor connectivity issues.
    • Same as above but can also be done using x2goclient.
  • dom0: sudo journalctl -f -u sshd
  • dom0: journalctl to watch qrexec logs

Implementation[edit]

Debug Scripts[edit]

  • As a connectivity test run whonixcheck --leak-tests in sys-whonix of remote-support-receiver and remote-support-provider due to Tor network issues. [16]
  • /var/lib/tor/remote_support
  • /var/lib/tor_autogen/remote_support

Proof of Concept Documentation[edit]

Moved to Remote_Administration#Qubes_-_SSH_or_VNC_into_Qubes_dom0.

(Latest version on this page was https://www.whonix.org/w/index.php?title=Dev/Qubes_Remote_Support&oldid=60584#Proof_of_Concept_Documentation [archive]].)

Archive[edit]

Footnotes[edit]

  1. ↑ 1.0 1.1 reload should be sufficient but probably some bug in Tor required restart instead.
  2. ↑ Useful for debugging. x) sys-whonix software installation Inside sys-whonix VM. Install ssh and x2go. 
    sudo apt --no-install-recommends install openssh-client x2goclient
  3. ↑ Previously functional socat method. 
    sudo socat TCP-LISTEN:22,fork EXEC:"qrexec-client-vm dom0 local.ssh"

    Last tested, functional version of this page using socat: https://www.whonix.org/w/index.php?title=Dev/Qubes_Remote_Support&oldid=58559 [archive]

  4. ↑ 
    user@host:~$ qvm-connect-tcp
Usage: /usr/bin/qvm-connect-tcp [localport]:[vmname]:[port]
Bind localport to another VM port using the qubes.ConnectTCP RPC service.
  5. ↑ Symptom inside sys-whonix VM if refused/missing dom0 qrexec policy in dom0. 
    2020/07/11 13:20:25 socat[2807] E waitpid(): child 2808 exited with status 126
  6. ↑ See archive for unauthenticated onion v3.
  7. ↑ Initial proof of concept had socat added here too.
  8. ↑ sftp-server required. 
    Unable to find the sftp-server binary. Neither bundled, nor found in $PATH nor additional directories.

If you are using a Linux-based operating system, please ask your system administrator to install the package containing the sftp-server binary. Common names are openssh, openssh-server or openssh-sftp-server depending upon distribution. 

If the sftp-server binary is installed on your system, please report a bug mentioning its path on: 

https://wiki.x2go.org/doku.php/wiki:bugs
  9. ↑ Currently broken. Use this file as /etc/ssh/sshd_config.
  10. ↑ 10.0 10.1 Currently broken due to x2go bugs.
  11. ↑ This should not be required only in cases where sshd was previously running.
  12. ↑ x) dom0 qrexec rpc setup In dom0. Setup qrexec rpc. Open file /etc/qubes-rpc/local.ssh in an editor with root rights. Add. 
    #!/bin/sh
exec socat STDIO TCP-CONNECT:localhost:22

    Save.

    Make executable.

    (Symptom if qubes-rpc is not executable.) 

    2020/07/10 14:00:35 socat[14462] E waitpid(): child 14463 exited with status 127

    sudo chmod +x /etc/qubes-rpc/local.ssh

    Previous functional example using socat.

    Open file /etc/qubes-rpc/policy/local.ssh in an editor with root rights.

    This box uses sudoedit for better security [archive]. This is an example and other tools can also achieve the same goal. If this example does not work for you or if you are not using Whonix ™, please refer to this link. 

    sudoedit /etc/qubes-rpc/policy/local.ssh

    Add. 

    {{gateway_product_name_vm}} dom0 allow
  13. ↑ chmod og-rwx ~/.ssh/authorized_keys was insufficient, sshd would refuse with insecure permissions error message. chmod --recursive og-rwx ~/.ssh was insufficient probably because file was owned by root, not user.
  14. ↑ During testing in remote-support-receiver sys-whonix VM. 
    qvm-copy /var/lib/tor_autogen/remote_support/1.auth_private

    In another sys-whonix-two remote-support-provider VM. 

    mv QubesIncoming/{{gateway_product_name_vm}}/1.auth_private ~/
  15. ↑ Currently broken. Harden /etc/ssh/ssh_config as per here.
  16. ↑

    [NOTICE] Guard guard-name ($fingerprint) is failing more circuits than usual. Most likely this means the Tor network is overloaded. Success counts are 200/300. Use counts are 0/0. 294 circuits completed, 0 were unusable, 100 collapsed, and 280 timed out. For reference, your timeout cutoff is 60 seconds.

Retrieved from "https://www.whonix.org/w/index.php?title=Dev/Qubes_Remote_Support&oldid=71258"
By using our website, you acknowledge that you have read, understood and agreed to our Privacy Policy, Cookie Policy, Terms of Service, and E-Sign Consent.
More information