[tor-dev] (no subject)

[Sambuddho Chakravarty]

Hi All

We (myself and my student, cc'ed here) have some questions regarding Tor
relay bandwidth. We created a small testbed (private Tor network)
involving relays, clients and servers over virtual machines. The client
uses the relays to create two hop circtuits to communicate to the server.
The link bandwidths between the virtual machines is otherwise ~ 10 Gbps
(tested via end-to-end measurement tools like iperf and via downloading
large files of sizes ~2 GB through https downloads).

When not using Tor, the client achieve a download rate of approximately 8
Gbit/s while downloading a file of approximately 2 Gbytes. The same
process, when repeated over Tor circuit (through relays of the private Tor
network, that used virtual machines), p*rovides a maxmium download rate **of
about 700 Mbit/s for the client (while the CPU utilization was over 80%).* This
is much less compared to the expected throughput of 8Gbps achieved when
client communicates directly
to the server (without using Tor circuits).

We tinkered around with the Tor config and discovered that when changed the
config fields of BandwidthBurst and BandwidthRate to 100 MBytes, we
achieved end-to-end throughput of about 100 Mbit/s (while the CPU
Utilization was less than 20%) (while the underlying network link capacity
is 1 Gbit/s). The moment
the BandwidthBurst and BandwidthRate values are increased to 1 GB (which
should effectively be 8 Gbit/s), the achieved throughput is about 650-700
Mbps (which is still 8 times slower than expected).

Further, removing all bandwidth restrictions, the inter VM bandwidth (for
the relays in the private Tor network) is around 8 Gbit/s (measured both
via Iperf and through downloading a file > 2 GBytes using encrypted HTTP
traffic). While this is as per the expected behavior of TCP, to utilize
all the available bandwidth, the same does not happen for Tor which
continues to ramp upto a maximum of 700 Mbit/s

We are confused as to why Tor's link bandwidth utilization is not higher
(Even on a private Tor network), even when using a high capacity VMs in the
private Tor network -- each one having about 4 cores and 8 GBytes of RAM. I
understand
after speaking from some Tor developers and maintainers that Tor does not
utilize all the available CPU cores. Our VMs however do not have any other
process running either. We run the Tor processes over Debian servers with
little no other programs. More strangely, while download via HTTPS achieves
a very large fraction of the link capacity (>80%), Tor traffic happens to
achieve
a throughput of around 700 Mbit/s (which is around 90% lower than the max
link bandwidth).

We tried the same with 2-hop Tor circuits, hoping to see some improvement.
But
to no avail. We continue to observe the same poor bandwidth utilization.

It would be great if you shed some light as to where we may be going wrong.

Regards
Sambuddho

[Matt Traudt]

Post by Sambuddho Chakravarty
When not using Tor, the client achieve a download rate of approximately
8 Gbit/s while downloading a file of approximately 2 Gbytes. The same
process, when repeated over Tor circuit (through relays of the private
Tor network, that used virtual machines), p*rovides a maxmium download
rate **of about 700 Mbit/s for the client (while the CPU utilization was
over 80%).* This is much less compared to the expected throughput of
8Gbps achieved when client communicates directly
to the server (without using Tor circuits).
We tinkered around with the Tor config and discovered that when changed
the config fields of BandwidthBurst and BandwidthRate to 100 MBytes, we
achieved end-to-end throughput of about 100 Mbit/s (while the CPU
Utilization was less than 20%) (while the underlying network link
capacity is 1 Gbit/s). The moment 
the   BandwidthBurst and BandwidthRate values are increased to 1 GB
(which should effectively be 8 Gbit/s), the achieved throughput is about
650-700 Mbps (which is still 8 times slower than expected). 

So you have two data points when using Tor.

1. Limit Tor to 100 MBytes --> only get 100 Mbps
2. Limit Tor to 1 GByte --> only get 700 Mbps

(1) sounds like it could be a real bug.

Both are approximately 8x slow downs, but I think that is coincidental.
I would guess that if you limited Tor to 7 Gbits, 5 Gbits, etc. you
would continue to see 700 Mbps. It would be interesting to see the
result of this.

As I hypothesized to either you or your student on IRC, I believe your
Tor relays as limited by CPU capacity (and its inability to use many
cores effectively); therefore, as Tor currently exists, is unable to
achieve a throughput of higher than 700 Mbps. Tor does a lot of crypto.
Perhaps it is way less efficient at doing its crypto than the TLS
library you were using for HTTPS downloads is at doing its crypto.

As another piece of evidence supporting my "700 Mbps is the max Tor can
do" theory, see the list of the fastest Tor relays[0] in the real
network. IPredator has likely done some heavy kernel tweaking, but the
remaining relays are all rather close to 700 Mbps.

Thank you for your interest in Tor.

Matt

[0]: https://metrics.torproject.org/rs.html#toprelays