[tor-dev] [tor-project] Intent to Minimise Effort: Fallback Directory Mirrors

[teor]

Hi all,

I'd like to move this thread to the tor-dev list. I made a list of
Core Tor tickets at the end of this email. Let's talk about them
on tor-dev.

Here's some background:

Rebuilding the fallback list takes a lot of effort!

So I'm picking up this tor-project thread while the latest fallback rebuild
is fresh in my mind.

The rebuild took a day or two of my time for regular changes.
And another day or two for once-off changes.

We made a few changes to streamline the process:
* atagar and I wrote scripts that make it easier to generate fallback lines,
and get operator contact details [0]
* pastly, atagar and irl helped out with code reviews and list format changes,
which should make the list easier to read and parse
* pastly and nickm helped out with fallback list updates and backport

Having this help meant that I had time to make the process better.
Thanks!

So I can't see any reasons to keep running opt-ins.

Are we listing non-exits?

Yes. Non-exits and under-used exits are the best fallback directory
mirrors, because they are under less load.

I suppose those might see a radical change in
their "IP reputation" when they end up on the blocklists after the next
ransomware attack. Exits are already in the lists.

Yes, there is a higher risk that a fallback will end up on one of these
lists. But all public Tor relays are at risk of ending up on a list like
this.

Because of the higher risk, we will continue to ask relay operators to
opt-in [1]. So we need to find people to help with this process every
6-12 months. And make it more efficient.

use ContactInfo
as a way to collect the intention of relay operators to opt-in/opt-out
of the fallback directory mirrors list.

This is a good idea, *if* we still think an opt-in is necessary.

Isn't that way to collect operator intentions also useful in opt-out
mode? (instead of continuing to use emails) Or how do you plan to
collect the opt-out info?

Why are emails preferred over other methods? Another options would be
wrapping a script that ops can run to create a cyberpunk trac ticket and
opt-out. This is probably easier than merging emails. But maybe we do
not want to handle this through trac.
I'd be happy to help w/ that eventually, and the fallback script.

This is a good idea.
At the moment, I take emails sent to me or tor-relays, and turn them
into trac comments or trac tickets. If we put it directly on trac,
anyone could handle the ticket.

The most time-consuming part of the rebuild is the opt-in process.

Here's what we do at the moment: [2]

0. Send an email to tor-relays to ask operators to opt-in
1. Run a script to find fallbacks whose details have changed, and email
tor-relays to ask operators if their details are permanent

Here are the most time-consuming steps:

2a. Receive personal emails and emails to the list with fallback details
2b. Generate an updated fallback line for each relay [0]
2c. Update the whitelist

And these steps are quick:

3. Run the update script to generate a new list
4. Write a changes file
5. Announce the new list
6. Backport the list to all supported Tor versions

Here are some things I'd like to change:

Just have relay fingerprints in the fallback whitelist [3]

The fallback script checks relay details, but it can just rely on Onionoo
to say when the address last changed.

This removes step 1 and step 2b.


Add a torrc option and descriptor line to opt-in as a FallbackDir [4]

This is inspired by nusenu's "ContactInfo" idea, and Hiro's "bot" idea.
Having a descriptor line and torrc option gets us a more reliable
implementation, and it's easier for relay operators to use. And
there's no need for an email bot or trac bot.

This removes step 2a and step 2c, for tor versions with this feature.
For tor versions without this feature, we can use the simpler fingerprint
list.


Make the hard-coded authorities into a separate include file with a standard format [5]

We'll also change the fallback file format (again, sorry!) to make
them consistent. This makes it easier for stem, metrics-lib, and
other Tor implementations to use the same authorities and fallbacks.

This makes step 5 easier (for libraries that use fallbacks).


Let's discuss these changes on the tor-dev list!

T

[0]: https://gitweb.torproject.org/tor.git/tree/scripts/maint/generateFallbackDirLine.py
https://gitweb.torproject.org/tor.git/tree/scripts/maint/lookupFallbackDirContact.py
[1]: https://trac.torproject.org/projects/tor/ticket/24789
[2]: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors
[3]: https://trac.torproject.org/projects/tor/ticket/24838
[4]: https://trac.torproject.org/projects/tor/ticket/24839
[5]: https://trac.torproject.org/projects/tor/ticket/24818

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------

[Tom Ritter]

Post by teor
Add a torrc option and descriptor line to opt-in as a FallbackDir [4]

Setting a config entry is easy and requires no thought. It's easy to
set without understanding the requirements or implications. Getting a
personal email and request for one's relay to keep a stable IP lends a
lot of gravity to the importance of it.

Are you worried about that?

Work-wise, the worst thing that would happen is that the list just
needs to be regenerated more frequently - I think? But user-wise
people on those older versions will have significantly increased
startup times if only 20% of their FallbackDirs are working...

-tom

[teor]

Post by Tom Ritter

Post by teor
Add a torrc option and descriptor line to opt-in as a FallbackDir [4]

Setting a config entry is easy and requires no thought. It's easy to
set without understanding the requirements or implications. Getting a
personal email and request for one's relay to keep a stable IP lends a
lot of gravity to the importance of it.
Are you worried about that?

My experience is that personal emails don't actually make much
difference. Operators try hard to keep things stable, but sometimes
their provider makes changes, or their circumstances change.

Here's what I've tried in the past:
* I sent personal emails the first few times,
* I sent list emails,
* I sent no emails at all, because we wanted to get seized relays
off the list quickly.

Every time, we rebuilt the list after 6-12 months.
So this time, we only selected fallbacks that had the same address for
at least 3 months. I'm hoping that makes a difference.

Even if it doesn't change fallback stability, it's still good to tell people.

I would be happy if someone else sent personal emails to:
* potential operators of high-bandwidth relays
* every fallback operator when there is a new list
* any fallback operator when their relay details change, or their
fallback goes down, asking them to fix it if they can

I think a bot or a human could do a very good job of these things.
(But I'm not that human.)

Hiro, did you want to write an email bot?
Or anyone else?
(The bot would need to work out how to deal with obfuscated email
addresses. We might need a human to code this part, and a list of
any particularly difficult ones.)

Post by Tom Ritter
Work-wise, the worst thing that would happen is that the list just
needs to be regenerated more frequently - I think?

I want to reduce the time it takes to rebuild the list.
It takes days right now. If I'm going to keep on doing it, I'd like
to cut that down to hours.

The features I suggested let me do that.
And they make it easier for others to rebuild the list.
(I don't want to be the only one who can rebuild the list.)

And they don't stop people sending personal emails.
They just mean that the process doesn't rely on them.

Post by Tom Ritter
But user-wise
people on those older versions will have significantly increased
startup times if only 20% of their FallbackDirs are working...

Tor is designed to bootstrap if 0% of the fallbacks and 50% of the
authorities are working.

Recent tor versions wait 6 seconds before trying an authority.
(Older versions try an authority and a fallback straight away.)
They will try 5 fallbacks and 3 authorities in the first 30 seconds.

And we backport the new fallback list to all supported versions.

T