[tor-dev] Proposal: Expose raw bwauth votes

[Tom Ritter]

I'm not sure, but I think
https://trac.torproject.org/projects/tor/ticket/21377 needed a
proposal so I tried to write one up.

-tom

[teor]

Post by Tom Ritter
I'm not sure, but I think
https://trac.torproject.org/projects/tor/ticket/21377 needed a
proposal so I tried to write one up.

Thanks Tom!

Proposals are great for new directory authority features, particularly
when we're changing the directory specification.

Post by Tom Ritter
Filename: xxx-expose-bwauth_votes.txt
Title: Have Directory Authorities expose raw bwauth vote documents
Author: Tom Ritter
Created: 11-December-2017
Status: Open
1. Introduction
Bandwidth Authorities (bwauths) perform scanning of the Tor Network
and calculate observed speeds for each relay. They produce a 'bwauth
vote file' that is given to a Directory Authority. The Directory
Authority uses the speed value from this file in its vote file
denoting its view of the speed of the relay.
After collecting all of the votes from other Authorities, a consensus
is calculated, and the consensus's view of a relay's speed is
determined by choosing the low-median value [or is it high-median?]

low-median

Post by Tom Ritter
of all the authorities' values for each relay.
Only a single metric from the bwauth vote file is exposed by a
Directory Authority's vote, however the original file contains
considerably more diagnostic information about how the bwauth arrives
at that measurement for that relay.
2. Motivation
The bwauth vote file contains more information that is exposed in the
overall vote file. This information is useful to debug anomalies in
relays' utilization and suspected bugs in the (decrepit) bwauth code.
Currently, all bwauths expose the raw vote file through various (non-
standard) means, and that file is downloaded (hourly) by a single person
(as long as his home internet connection and home server is working)
and archived (with a small amount of robustness.)
It would be preferable to have this exposed in a standard manner.
Doing so would no longer require bwauths to run HTTP servers to expose
the file, no longer require them to take additional manual steps to
provide it, and would enable public consumption by any interested
parties. We hope that Collector will begin archiving the files.
3. Specification
An authority SHOULD publish the bwauth vote used to calculate its
current vote. It should make the bwauth vote file available at the

SHOULD

Post by Tom Ritter
same time as its normal vote file.

Since the file is created by an external process, and we never store a
copy of any old files, I suggest that we make it available at the URL at
all times, as "the file that the authority has right now".

(And return a 404 when the file does not exist, is not readable, etc.)

Post by Tom Ritter
It should make the file available
at
http://<hostname>/tor/status-vote/next/bwauth.z

We shouldn't use next/ unless we're willing to cache a copy of the file
we actually used to vote. If we do, we should serve it from next/ as
soon as we vote using it, then serve it from current/ as soon as the
consensus is created.

If we don't store a copy of the file, we should use a different URL,
like status-vote/now/bwauth, and recommend that the file is fetched at
hh:50, when the votes are created. This would allow us to implement
current/bwauth and next/bwauth in a future version.

Have you thought about versioning the URL if we have multiple flavours
of bwauth file? We could use bwauth-<FLAVOR> like consensuses.

Also, Tor has new compression options for zstd and lzma.

Given that this is an externally-controlled file, we could stream it
from disk and compress it on the fly with something cheap like gzip
or zstd.

Post by Tom Ritter
It MUST NOT attempt to send its bwauth vote file in a HTTP POST to
other authorities and it SHOULD NOT make bwauth vote files from other
authorities available.
4. Security Implications
The raw bwauth vote file does not [really: is not believed to] expose
any sensitive information. All authorities currently make this
document public already, an example is at
https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

How large is the file?
Maybe we should pre-compress it, to avoid CPU exhaustion attacks.
If we did this, we could say that it's safe, as long as it is smaller
than the full consensus or full set of descriptors.

Post by Tom Ritter
5. Compatibility
Exposing the document presents no compatibility concerns.
The compatibility concern is with applications that want to consume
the document. The bwauth vote file has no specification, and has been
extended in ad-hoc ways. Applications that merely wish to archive the
document (e.g. Collector) won't have a problems. Applications that
want to parse it may encounter errors if a new (unexpected) field is
added, or assumptions are made about the text encoding or formatting
of the document.

Parsers may also encounter issues if we transition to another bwauth
implementation. We could remove fields if this happens.

T

[Tom Ritter]

Sending two replies, with an updated proposal in the second.

Post by teor

Post by Tom Ritter
It should make the file available
at
http://<hostname>/tor/status-vote/next/bwauth.z

We shouldn't use next/ unless we're willing to cache a copy of the file
we actually used to vote. If we do, we should serve it from next/ as
soon as we vote using it, then serve it from current/ as soon as the
consensus is created.
If we don't store a copy of the file, we should use a different URL,
like status-vote/now/bwauth, and recommend that the file is fetched at
hh:50, when the votes are created. This would allow us to implement
current/bwauth and next/bwauth in a future version.

Sure.

Post by teor
Have you thought about versioning the URL if we have multiple flavours
of bwauth file? We could use bwauth-<FLAVOR> like consensuses.

For lack of a better name I'll propose bwauth-legacy?

Post by teor
Also, Tor has new compression options for zstd and lzma.
Given that this is an externally-controlled file, we could stream it
from disk and compress it on the fly with something cheap like gzip
or zstd.

I haven't seen any indicated in dir-spec how to handle those? Or how I
should change the proposal to accommodate them? Should I make the url
.gz and say that the DirAuth should compress it and stream it from
disk?

Post by teor

Post by Tom Ritter
The raw bwauth vote file does not [really: is not believed to] expose
any sensitive information. All authorities currently make this
document public already, an example is at
https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

How large is the file?
Maybe we should pre-compress it, to avoid CPU exhaustion attacks.
If we did this, we could say that it's safe, as long as it is smaller
than the full consensus or full set of descriptors.

~2.6MB. See above.

-tom

[teor]

Post by Tom Ritter

Post by teor
Also, Tor has new compression options for zstd and lzma.
Given that this is an externally-controlled file, we could stream it
from disk and compress it on the fly with something cheap like gzip
or zstd.

I haven't seen any indicated in dir-spec how to handle those? Or how I
should change the proposal to accommodate them? Should I make the url
.gz and say that the DirAuth should compress it and stream it from
disk?

There is this section about gzip-compression, the extension is ".z":
https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3777

I have opened this ticket to document how other compression formats
work in tor-spec:

https://trac.torproject.org/projects/tor/ticket/24909

T

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------

[isis agora lovecruft]

Post by Tom Ritter
I'm not sure, but I think
https://trac.torproject.org/projects/tor/ticket/21377 needed a
proposal so I tried to write one up.
-tom

Hi tom, thanks for the proposal!

Post by Tom Ritter
Filename: xxx-expose-bwauth_votes.txt
Title: Have Directory Authorities expose raw bwauth vote documents
Author: Tom Ritter
Created: 11-December-2017
Status: Open

I changed things recently, you'll need a "Ticket:" field if your proposal is
in state {OPEN,ACCEPTED,CLOSED,FINISHED}. [0]

(Although, maybe we shouldn't require "Ticket:" for state OPEN, so as not to
hinder calling it OPEN and discussing it even for those things which don't
yet have tickets?)

Post by Tom Ritter
1. Introduction
Bandwidth Authorities (bwauths) perform scanning of the Tor Network
and calculate observed speeds for each relay. They produce a 'bwauth
vote file' that is given to a Directory Authority. The Directory
Authority uses the speed value from this file in its vote file
denoting its view of the speed of the relay.
After collecting all of the votes from other Authorities, a consensus
is calculated, and the consensus's view of a relay's speed is
determined by choosing the low-median value [or is it high-median?]
of all the authorities' values for each relay.
Only a single metric from the bwauth vote file is exposed by a
Directory Authority's vote, however the original file contains
considerably more diagnostic information about how the bwauth arrives
at that measurement for that relay.
2. Motivation
The bwauth vote file contains more information that is exposed in the

/s/that/than/ ???

Post by Tom Ritter
overall vote file. This information is useful to debug anomalies in
relays' utilization and suspected bugs in the (decrepit) bwauth code.
Currently, all bwauths expose the raw vote file through various (non-
standard) means, and that file is downloaded (hourly) by a single person
(as long as his home internet connection and home server is working)
and archived (with a small amount of robustness.)
It would be preferable to have this exposed in a standard manner.
Doing so would no longer require bwauths to run HTTP servers to expose
the file, no longer require them to take additional manual steps to
provide it, and would enable public consumption by any interested
parties. We hope that Collector will begin archiving the files.
3. Specification
An authority SHOULD publish the bwauth vote used to calculate its
current vote. It should make the bwauth vote file available at the
same time as its normal vote file. It should make the file available
at
http://<hostname>/tor/status-vote/next/bwauth.z

If it's "next", how is it possible to expose it at the same time as its vote
which is based upon it? Maybe we should change the URL to be "current"?

Post by Tom Ritter
It MUST NOT attempt to send its bwauth vote file in a HTTP POST to
other authorities and it SHOULD NOT make bwauth vote files from other
authorities available.
4. Security Implications
The raw bwauth vote file does not [really: is not believed to] expose
any sensitive information. All authorities currently make this
document public already, an example is at
https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

Maybe we want to think about resource exhaustion attacks if we're making a
standarised interface available to it? The response after all is going
likely always be much larger than the request.

Post by Tom Ritter
5. Compatibility
Exposing the document presents no compatibility concerns.
The compatibility concern is with applications that want to consume
the document. The bwauth vote file has no specification, and has been
extended in ad-hoc ways. Applications that merely wish to archive the
document (e.g. Collector) won't have a problems. Applications that
want to parse it may encounter errors if a new (unexpected) field is
added, or assumptions are made about the text encoding or formatting
of the document.

A specification for the documents that BWAuths produce would be an extremely
welcome contribution! but probably shouldn't be prerequisite to accepting
and implementing this proposal.

Thanks again for the proposal, tom!

[0]: https://gitweb.torproject.org/torspec.git/commit/?id=8be6722e8d9

Best regards,

[Tom Ritter]

Updated proposal attached.

Post by isis agora lovecruft

Post by Tom Ritter
Status: Open

I changed things recently, you'll need a "Ticket:" field if your proposal is
in state {OPEN,ACCEPTED,CLOSED,FINISHED}. [0]
(Although, maybe we shouldn't require "Ticket:" for state OPEN, so as not to
hinder calling it OPEN and discussing it even for those things which don't
yet have tickets?)

I added Ticket: https://trac.torproject.org/projects/tor/ticket/21377

Post by isis agora lovecruft

Post by Tom Ritter
An authority SHOULD publish the bwauth vote used to calculate its
current vote. It should make the bwauth vote file available at the
same time as its normal vote file. It should make the file available
at
http://<hostname>/tor/status-vote/next/bwauth.z

If it's "next", how is it possible to expose it at the same time as its vote
which is based upon it? Maybe we should change the URL to be "current"?

teor suggested 'now'? I'll make it whichever you think it should be =)

Post by isis agora lovecruft

Post by Tom Ritter
The raw bwauth vote file does not [really: is not believed to] expose
any sensitive information. All authorities currently make this
document public already, an example is at
https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

Maybe we want to think about resource exhaustion attacks if we're making a
standarised interface available to it? The response after all is going
likely always be much larger than the request.

teor suggested compressing and streaming from disk?

-tom

[teor]

Hi,

It looks like this proposal was lost, and never committed to torspec.
Sorry about that!

Juga and I have updated the proposal.
Please see the full text inline below.

We have also written the corresponding dir-spec patch:
https://github.com/torproject/torspec/pull/26/files

We are tracking the spec and proposal in:
https://trac.torproject.org/projects/tor/ticket/26694

The Tor implementation in:
https://trac.torproject.org/projects/tor/ticket/21377

And the Collector implementation in:
https://trac.torproject.org/projects/tor/ticket/21378

Post by Tom Ritter

Post by isis agora lovecruft

Post by Tom Ritter
An authority SHOULD publish the bwauth vote used to calculate its
current vote. It should make the bwauth vote file available at the
same time as its normal vote file. It should make the file available
at
http://<hostname>/tor/status-vote/next/bwauth.z

If it's "next", how is it possible to expose it at the same time as its vote
which is based upon it? Maybe we should change the URL to be "current"?

teor suggested 'now'? I'll make it whichever you think it should be =)

We changed the scheme so the bandwidth file at
status-vote/(next|current)/bandwidth.z

is the bandwidth file used to produce:
status-vote/(next|current)/authority.z

We publish the bandwidth file to next/ when the vote is published to next/.
And the same with current.

It should be relatively easy to implement this publishing in the tor code.
And to implement downloads in Collector.

Post by Tom Ritter

Post by isis agora lovecruft

Post by Tom Ritter
The raw bwauth vote file does not [really: is not believed to] expose
any sensitive information. All authorities currently make this
document public already, an example is at
https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

Maybe we want to think about resource exhaustion attacks if we're making a
standarised interface available to it? The response after all is going
likely always be much larger than the request.

teor suggested compressing and streaming from disk?

We’ll need to take a copy of the file anyway, to avoid race conditions.
We can use the same code we use for the rest of the directory documents.

T

Filename: xxx-expose-bandwidth-files.txt
Title: Have Directory Authorities expose raw bandwidth list files
Author: Tom Ritter
Created: 11-December-2017
Status: Open
Ticket: https://trac.torproject.org/projects/tor/ticket/21377

1. Introduction

Bandwidth Authorities (bwauths) perform scanning of the Tor Network
and calculate observed bandwidths for each relay. They produce a bandwidth
list file that is given to a Directory Authority. The Directory
Authority uses the bw (bandwidth) value from this file in its vote file
denoting its view of the bandwidth of the relay.

After collecting all of the votes from other Authorities, a consensus
is calculated, and the consensus's view of a relay's speed is
determined by choosing the low-median value of all the authorities'
values for each relay.

Only a single metric from the bandwidth list file is exposed by a
Directory Authority's vote, however the original file contains
considerably more diagnostic information about how the bwauth arrives
at that measurement for that relay.

For more details, see the bandwidth list file specification in
bandwidth-file-spec.txt.

2. Motivation

The bandwidth list file contains more information than is exposed in the
overall vote file. This information is useful to debug:
* anomalies in relays' utilization,
* suspected bugs in the (decrepit) bwauth code, and
* the transition to a replacement bwauth implementation.

Currently, all bwauths expose the bandwidth list file through various (non-
standard) means, and that file is downloaded (hourly) by a single person
(as long as his home internet connection and home server is working)
and archived (with a small amount of robustness.)

It would be preferable to have this exposed in a standard manner.
Doing so would no longer require bwauths to run HTTP servers to expose
the file, no longer require them to take additional manual steps to
provide it, and would enable public consumption by any interested
parties. We hope that Collector will begin archiving the files.

3. Specification

An authority SHOULD publish the bandwidth list file used to calculate its
next vote. It SHOULD make the bandwidth list file available whenever the
corresponding vote is available, at the corresponding URL. (See
dir-spec for the exact details.)

It SHOULD make the file available at
http://<hostname>/tor/status-vote/next/bandwidth.z
http://<hostname>/tor/status-vote/current/bandwidth.z

It MUST NOT attempt to send its bandwidth list file in a HTTP POST to
other authorities and it SHOULD NOT make bandwidth list files from other
authorities available.

Clients interested in consuming these documents should download them from
each authority's:
* next URL when votes are created. (In the public Tor network, this is
after HH:50 during normal operation, and after HH:20 during a
consensus failure.)
* current URL after the valid-after time in the consensus.
(After HH:00, and HH:30 during consensus failure.)

4. Security Implications

The raw bandwidth list file does not [really: is not believed to] expose
any sensitive information. All authorities currently make this
document public already, an example is at
https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

5. Compatibility

Exposing the document presents no compatibility concerns.

Applications that parse the document should follow the bandwidth list file
specification in bandwidth-file-spec.txt.
If a new bandwidth list format version is added, the applications MAY need
to upgrade to that version.

[Iain Learmonth]

Hi,

Post by teor
It MUST NOT attempt to send its bandwidth list file in a HTTP POST to
other authorities and it SHOULD NOT make bandwidth list files from other
authorities available.

There is no mechanism specified that could be used to send this file via
an HTTP POST. There is also no mechanism to make available bandwidth
list files from other authorities specified. I wonder if this paragraph
has to be here.

I can also see a future where, to support more robust metrics
collection, we might want to have authorities also provide mirrors of
other authorities' documents.

Thanks,
Iain.

[teor]

Post by Iain Learmonth

Post by teor
It MUST NOT attempt to send its bandwidth list file in a HTTP POST to
other authorities and it SHOULD NOT make bandwidth list files from other
authorities available.

There is no mechanism specified that could be used to send this file via
an HTTP POST.

But there is a mechanism for sending votes via HTTP POST:

An authority SHOULD publish its vote immediately at the start of each voting
period (minus VoteSeconds+DistSeconds). It does this by making it
available at
http://<hostname>/tor/status-vote/next/authority.z
and sending it in an HTTP POST request to each other authority at the URL
http://<hostname>/tor/post/vote

https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n1586

Post by Iain Learmonth
There is also no mechanism to make available bandwidth
list files from other authorities specified.

Once an authority has a vote from another authority, it makes it available
at
http://<hostname>/tor/status-vote/next/<fp>.z
where <fp> is the fingerprint of the other authority's identity key.
And at
http://<hostname>/tor/status-vote/next/d/<d>.z
where <d> is the digest of the vote document.

https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n1597

Post by Iain Learmonth
I wonder if this paragraph
has to be here.

We included this paragraph to avoid implementing unnecessary features.
We wanted a minimal implementation to start with.

Post by Iain Learmonth
I can also see a future where, to support more robust metrics
collection, we might want to have authorities also provide mirrors of
other authorities' documents.

I think this would be a good feature, but it’s outside the scope of a
minimal implementation.

If someone would like to help out:
* I would happily merge a spec or proposal patch
* Please open tickets for the POST and GET features
* See "Authorities should put a hash of the bandwidth file in their votes"
https://trac.torproject.org/projects/tor/ticket/26698

T