[tor-dev] New Fallback Directory File Format

Discussion:

teor

2017-12-22 14:53:35 UTC

Hi all,

Some time in the next few weeks, the Tor fallback directory mirror file
format will change. This affects stem and Relay Search, which parse this
file.

Change Description

Here is a list of changes to the file format:
* the "weight" line has been removed, and replaced with a Tor config
default (#24679, #24681)
* the comma that separates fallback C strings is now on its own line
* a "nickname" comment has been added (#24600)
* an optional "extrainfo" comment has been added (#22759)

The added fields will be populated with placeholders until the list is
rebuilt (#22271). This will hopefully happen some time in the next few
weeks.

Requesting More Extra Info Caches

There are only a few fallbacks that cache extra-info documents.
I checked 67, and only 4 cached extra-info documents.

atagar, do you want me to ask some fallback operators to set
DownloadExtraInfo 1?

What number or proportion would you like?

(We allow approximately 25% of fallbacks to go down before we start to rebuild
the list. In the worst case, this can mean that ~40% are down at some point.)

Example Entries

A sample entry in the new format, using actual relay info:

"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB"
" ipv6=[2a01:4f8:162:51e2::2]:9001"
/* nickname=rueckgrat */
/* extrainfo=1 */
,

The current fallback file in the new format, with placeholders:
https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.inc

A small sample fallback file in the new format, with actual relay info:
https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format.inc

Please let me know if you would like any changes to the format.

T

--
Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
xmpp: teor at torproject dot org
------------------------------------------------------------------------

Tyler Durden

2017-12-22 22:16:00 UTC

Permalink

Hi

I just added the DownloadExtraInfo in the config file for the nodes
"chulak" and "aurora".

Greetings

Post by teor
Hi all,
Some time in the next few weeks, the Tor fallback directory mirror file
format will change. This affects stem and Relay Search, which parse this
file.
Change Description
* the "weight" line has been removed, and replaced with a Tor config
default (#24679, #24681)
* the comma that separates fallback C strings is now on its own line
* a "nickname" comment has been added (#24600)
* an optional "extrainfo" comment has been added (#22759)
The added fields will be populated with placeholders until the list is
rebuilt (#22271). This will hopefully happen some time in the next few
weeks.
Requesting More Extra Info Caches
There are only a few fallbacks that cache extra-info documents.
I checked 67, and only 4 cached extra-info documents.
atagar, do you want me to ask some fallback operators to set
DownloadExtraInfo 1?
What number or proportion would you like?
(We allow approximately 25% of fallbacks to go down before we start to rebuild
the list. In the worst case, this can mean that ~40% are down at some point.)
Example Entries
"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB"
" ipv6=[2a01:4f8:162:51e2::2]:9001"
/* nickname=rueckgrat */
/* extrainfo=1 */
,
https://github.com/teor2345/tor/blob/ticket22759_tree/src/or/fallback_dirs.inc
https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format.inc
Please let me know if you would like any changes to the format.
T
--
Tim Wilson-Brown (teor)
teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
xmpp: teor at torproject dot org
------------------------------------------------------------------------
_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Damian Johnson

2017-12-23 21:16:57 UTC

Permalink

Hi Tim, added preliminary Stem parsing support for the v2 fallback format...

https://gitweb.torproject.org/stem.git/commit/?id=ea55eaa

Few thoughts are...

* It would be nice if the 'extrainfo=' lines were mandatory since I
need a delimiter between the entries.
* Maybe we should start the document with a format version number?

Cheers! -Damian

teor

2017-12-23 22:23:21 UTC

Permalink

Post by Damian Johnson
Hi Tim, added preliminary Stem parsing support for the v2 fallback format...
https://gitweb.torproject.org/stem.git/commit/?id=ea55eaa
Few thoughts are...
* It would be nice if the 'extrainfo=' lines were mandatory since I
need a delimiter between the entries.

We have a few alternatives here:

C requires a comma as the delimiter between entries.
I can guarantee there will always be a comma delimiter after every entry,
Including the last entry. I will also make sure that any comment fields
come before this delimiter. I can guarantee there will never be a comma
inside the C string or comments in the entry.

Alternately, I can make extrainfo mandatory, and if I can't fetch a relay's
descriptor (a rare case?), I will mark is as 0.

But I'd like to be able to add extra fields in future without breaking
parsers, so I don't want parsers relying on field order.

Do you want me to add an explicit end of record comment, or is the
comma sufficient?

Is there a delimiter you'd like me to add before the first entry?

Post by Damian Johnson
* Maybe we should start the document with a format version number?

I thought of that, too. Thanks for reminding me.

https://trac.torproject.org/projects/tor/ticket/24725

Post by Damian Johnson
Cheers! -Damian

_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

Damian Johnson

2017-12-24 00:14:25 UTC

Permalink

Post by teor
Do you want me to add an explicit end of record comment, or is the
comma sufficient?

Hi Tim. I'd rather not rely on just a comma. I can easily see us
tweaking the layout so 'expect a line with only a comma' breaks.

I actually like both of the other suggested options: having
'extrainfo=0' so that's explicit *and* a delimiter. For instance...

/* ============================== */

... or whatever. That would both help with parsing and make the file
nicer to read.

Cheers! -Damian

teor

2017-12-24 02:26:23 UTC

Permalink

Post by Damian Johnson

Post by teor
Do you want me to add an explicit end of record comment, or is the
comma sufficient?

Hi Tim. I'd rather not rely on just a comma. I can easily see us
tweaking the layout so 'expect a line with only a comma' breaks.
I actually like both of the other suggested options: having
'extrainfo=0' so that's explicit *and* a delimiter. For instance...
/* ============================== */
... or whatever. That would both help with parsing and make the file
nicer to read.

Done!

* the file now starts with a type and a version line:
/* type=fallback */
/* version=2.0.0 */
* extrainfo is mandatory (occasionally we won't get a descriptor, so
we'll warn and mark the relay extrainfo=0)
* each fallback entry ends with /* ===== */

Two remaining questions:
* is 6 extra info caches (up from 4) enough in a list of 150?
* do you want the delimiter before the first fallback entry as well?

Sample entry:

"5.9.110.236:9030 orport=9001 id=0756B7CD4DFC8182BE23143FAC0642F515182CEB"
" ipv6=[2a01:4f8:162:51e2::2]:9001"
/* nickname=rueckgrat */
/* extrainfo=1 */
/* ===== */
,

Sample file:

https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format_version.inc

T

--
Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
xmpp: teor at torproject dot org
------------------------------------------------------------------------

Damian Johnson

2017-12-24 20:13:43 UTC

Permalink

Post by teor
Done!
/* type=fallback */
/* version=2.0.0 */
* extrainfo is mandatory (occasionally we won't get a descriptor, so
we'll warn and mark the relay extrainfo=0)
* each fallback entry ends with /* ===== */

Sweet, thanks Tim!

Post by teor
* is 6 extra info caches (up from 4) enough in a list of 150?

Hmmm. Can't say on Stem's side I have an opinion on this. It doesn't
rely on fallback directories for anything so extrainfo caches aren't a
concern.

Post by teor
* do you want the delimiter before the first fallback entry as well?

Stem doesn't care about a delimiter before the first entry but that
seems like a good idea so we have a clear separation between comments
and the start of the machine readable section.

A detail Stem does care about is that the last entry ends with a
delimiter. If it doesn't that's fine, but code is a tad simpler if we
ensure it does. :)

Cheers! -Damian

Iain Learmonth

2017-12-24 20:26:46 UTC

Permalink

Hi,

On 24/12/17 20:13, Damian Johnson wrote.

As we are planning to also add a parser to metrics-lib (#24434), would
it be possible to get a full description of the format of the file
possibly in RFC5234 format so that we can check that the generator and
parsers all match up to that specification?

Thanks,
Iain.

teor

2017-12-26 07:47:44 UTC

Permalink

Post by Damian Johnson

Sweet, thanks Tim!

Post by teor
* is 6 extra info caches (up from 4) enough in a list of 150?

Hmmm. Can't say on Stem's side I have an opinion on this. It doesn't
rely on fallback directories for anything so extrainfo caches aren't a
concern.

So stem just uses a mix of fallbacks and authorities?
Good, that's what Tor does.
I think we will be fine then.

Post by Damian Johnson

Post by teor
* do you want the delimiter before the first fallback entry as well?

Stem doesn't care about a delimiter before the first entry but that
seems like a good idea so we have a clear separation between comments
and the start of the machine readable section.

I made the header end with a delimiter, and I made the list of entries
start with a delimiter:

https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format_version.3.inc

This means that parsers can (and should) ingore the human-readable
second section.

Post by Damian Johnson
A detail Stem does care about is that the last entry ends with a
delimiter. If it doesn't that's fine, but code is a tad simpler if we
ensure it does. :)

It does and it will continue to.

Post by Damian Johnson
As we are planning to also add a parser to metrics-lib (#24434), would
it be possible to get a full description of the format of the file
possibly in RFC5234 format so that we can check that the generator and
parsers all match up to that specification?

I have written up a format in the standard torspec style:

https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt

It is deliberately under-specified, please let me know if this causes
any trouble when writing the parser, and I will tighten it up.

It's not ABNF/RFC5234, it's rather restrictive, and strict ABNF is
unreadable for case sensitive strings. I am happy to put an ABNF spec in
an appendix, if someone wants to write one.

T

--
Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
xmpp: teor at torproject dot org
------------------------------------------------------------------------

teor

2017-12-27 10:53:42 UTC

Permalink

Post by teor

Post by Iain Learmonth
As we are planning to also add a parser to metrics-lib (#24434), would
it be possible to get a full description of the format of the file
possibly in RFC5234 format so that we can check that the generator and
parsers all match up to that specification?

https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt
It is deliberately under-specified, please let me know if this causes
any trouble when writing the parser, and I will tighten it up.
It's not ABNF/RFC5234, it's rather restrictive, and strict ABNF is
unreadable for case sensitive strings. I am happy to put an ABNF spec in
an appendix, if someone wants to write one.

We've added separators to each section, and a timestamp field in the header.

The revised spec is here:

https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt

The revised sample file is here:

https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format_version.4.inc

We're tracking the details in this ticket:

https://trac.torproject.org/projects/tor/ticket/24742

T

--
Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
xmpp: teor at torproject dot org
------------------------------------------------------------------------

teor

2018-01-05 22:33:32 UTC

Permalink

Hi,

Post by teor

We've added separators to each section, and a timestamp field in the header.
https://github.com/teor2345/torspec/blob/fallback-format-2/fallback-spec.txt
https://trac.torproject.org/projects/tor/attachment/ticket/22759/fallback_dirs_new_format_version.4.inc
https://trac.torproject.org/projects/tor/ticket/24742

We have merged the list format change to Tor's master branch.

The fallback 2.0.0 spec hasn't been merged yet, but atagar has reviewed it.
It's at:
https://github.com/teor2345/torspec/blob/fallback-format-2-v2/fallback-spec.txt

The fallback list is now in the 2.0.0 format, but it has exactly the same fallbacks in it:
https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc

Some time over the next few days, we will generate a new list of fallbacks in
the 2.0.0 format, and backport it to Tor 0.2.9 and later.

T

Iain Learmonth

2018-01-06 15:52:45 UTC

Permalink

Hi,

Post by teor
The fallback 2.0.0 spec hasn't been merged yet, but atagar has reviewed it.
https://github.com/teor2345/torspec/blob/fallback-format-2-v2/fallback-spec.txt

I've added this link to the Metrics ticket #24434, I started work on
this but ran out of time after not getting very far at all.

Post by teor
https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc
Some time over the next few days, we will generate a new list of fallbacks in
the 2.0.0 format, and backport it to Tor 0.2.9 and later.

Ok, I'll see if I can do a quick hack workaround to generate the list
for Relay Search until the data is available through Tor Metrics.

Thanks,
Iain.

Iain Learmonth

2018-01-06 15:57:18 UTC

Permalink

Hi,

Post by Iain Learmonth
Ok, I'll see if I can do a quick hack workaround to generate the list
for Relay Search until the data is available through Tor Metrics.

In fact, my original quick hack workaround still works, as it's only
extracting fingerprints. (Tested that there is no lines changed in a
diff against the same file in the old format).

I'm ready to update as soon as there's a new list available.

Thanks,
Iain.

teor

2018-01-10 11:57:37 UTC

Permalink

Hi,

We've merged a new list of fallbacks in the version 2.0.0 format.

But we also want to keep a separate list of authorities, rather than hiding
them in the middle of config.c:

https://trac.torproject.org/projects/tor/ticket/24818

So we are changing the current authority and fallback list formats,
to make them consistent with each other. (Sorry about doing the two changes
separately, we had to do fallbacks for the 0.3.2.9 release deadline.)

Here's what will change in version 3.0.0:

https://github.com/teor2345/torspec/commits/dir-list

Hopefully, you can use the same parser for both file formats.

Post by Iain Learmonth
Hi,

Post by teor
The fallback 2.0.0 spec hasn't been merged yet, but atagar has reviewed it.
https://github.com/teor2345/torspec/blob/fallback-format-2-v2/fallback-spec.txt

I've added this link to the Metrics ticket #24434, I started work on
this but ran out of time after not getting very far at all.

The fallback list spec version 2.0.0 has been merged, it's now at:

https://gitweb.torproject.org/torspec.git/tree/dir-list-spec.txt

Post by Iain Learmonth

Post by teor
The fallback list is now in the 2.0.0 format, but it has exactly the
https://gitweb.torproject.org/tor.git/tree/src/or/fallback_dirs.inc
Some time over the next few days, we will generate a new list of fallbacks in
the 2.0.0 format, and backport it to Tor 0.2.9 and later.

Ok, I'll see if I can do a quick hack workaround to generate the list
for Relay Search until the data is available through Tor Metrics.

I'm glad this worked out!
Thanks for doing this update.

T

--
Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------

Damian Johnson

2018-01-12 19:45:00 UTC

Permalink

Post by teor
We've merged a new list of fallbacks in the version 2.0.0 format.

Hi Tim. Last couple weeks I've been working on a Fallback and Endosome
branch for Stem. Just finished and merged the former...

https://gitweb.torproject.org/stem.git/commit/?id=ea2752c

Stem now supports the v2 format, has additional test coverage, and is
synced with Tor's current fallback file.

Cheers! -Damian