Discussion:
[tor-dev] Ready to Integrate/Review New Marionette Version into Tor
John Helmsen
2018-07-20 20:12:21 UTC
Permalink
Gentlemen and Ladies,

We are now ready to integrate Marionette, or at least have it evaluated, as
a bridge for the Tor browser in its Pythonless form.

At the Tor meeting in March, we successfully operated Marionette as a
bridge by implementing the PT v2.0 specification (Thanks ahf!).

Now we have a new version of Marionette which operates as a stand-alone
binary (NO PYTHON!). I checked that it still forms a bridge, like at the
Tor meeting. We also have a wider variety of transports enabled.

We are in the process of writing the documentation for Marionette, but the
documentation on the web page should be sufficient for at least getting a
full evaluation started. We'd like to have the evaluation complete by the
end of next month, hopefully the middle of next month, and stand ready to
make any and all changes necessary.

A full set of documentation will also be written for designing your own
protocols. This is in process.

Please let us know what you need.

John Helmsen
David Fifield
2018-07-24 02:05:29 UTC
Permalink
Post by John Helmsen
We are in the process of writing the documentation for Marionette, but the
documentation on the web page should be sufficient for at least getting a full
evaluation started.  We'd like to have the evaluation complete by the end of
next month, hopefully the middle of next month, and stand ready to make any and
all changes necessary.
A full set of documentation will also be written for designing your own
protocols.  This is in process.
Please let us know what you need.
The Tor Browser developers may have more specific requests, but I can
suggest some steps to get started.

Open a ticket at https://trac.torproject.org/ for discussion and to
track progress.
Type: project
Component: Applications/Tor Browser
Keywords: marionette
The old ticket for FTE is a good reference: https://bugs.torproject.org/10362

And then it would help if you port your build process to the Tor Browser
build system. General information:
https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking
First, just build
git clone https://git.torproject.org/builders/tor-browser-build.git
cd tor-browser-build
git checkout tbb-8.0a9-build3
make testbuild # or, e.g., testbuild-linux-x86_64
Then you'll have to add a new project (consisting of a "build" and
"config" file) for Marionette and each of its dependencies. You can copy
from existing projects as templates. Here is the meek project, for
example:
https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/meek
You'll also need to add bridge lines to:
https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js
To build just one project, not an entire release, do e.g.:
rbm/rbm build gmp --target testbuild --target torbrowser-linux-x86_64
rbm/rbm build marionette --target testbuild --target torbrowser-linux-x86_64
John Helmsen
2018-07-24 15:42:08 UTC
Permalink
David,

Thank you, I have created the ticket as #26920.
https://trac.torproject.org/projects/tor/ticket/26920#ticket. Having
downloaded the git project, it seems that this work cannot be performed on
a Mac, since it doesn't run 'runc'. Is that right?

Ben,

I am currently trying to create a virtual machine using Ubuntu 16.04 for
development. Unless I am mistaken, this work cannot be done on a Mac.
Please do the same, so that we can put this thing to bed.
Post by John Helmsen
Post by John Helmsen
We are in the process of writing the documentation for Marionette, but
the
Post by John Helmsen
documentation on the web page should be sufficient for at least getting
a full
Post by John Helmsen
evaluation started. We'd like to have the evaluation complete by the
end of
Post by John Helmsen
next month, hopefully the middle of next month, and stand ready to make
any and
Post by John Helmsen
all changes necessary.
A full set of documentation will also be written for designing your own
protocols. This is in process.
Please let us know what you need.
The Tor Browser developers may have more specific requests, but I can
suggest some steps to get started.
Open a ticket at https://trac.torproject.org/ for discussion and to
track progress.
Type: project
Component: Applications/Tor Browser
Keywords: marionette
The old ticket for FTE is a good reference: https://bugs.torproject.org/
10362
And then it would help if you port your build process to the Tor Browser
https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking
First, just build
git clone https://git.torproject.org/
builders/tor-browser-build.git
cd tor-browser-build
git checkout tbb-8.0a9-build3
make testbuild # or, e.g., testbuild-linux-x86_64
Then you'll have to add a new project (consisting of a "build" and
"config" file) for Marionette and each of its dependencies. You can copy
from existing projects as templates. Here is the meek project, for
https://gitweb.torproject.org/builders/tor-browser-build.
git/tree/projects/meek
https://gitweb.torproject.org/builders/tor-browser-build.
git/tree/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js
rbm/rbm build gmp --target testbuild --target
torbrowser-linux-x86_64
rbm/rbm build marionette --target testbuild --target
torbrowser-linux-x86_64
--
John Helmsen
***@redjack.com
C: (240) 899-5676
David Fifield
2018-07-24 16:46:50 UTC
Permalink
Thank you, I have created the ticket as #26920. https://trac.torproject.org/
projects/tor/ticket/26920#ticket.  Having downloaded the git project, it seems
that this work cannot be performed on a Mac, since it doesn't run 'runc'.  Is
that right?
Right, the README says "To build Tor Browser, you need a Linux
distribution that has support for runc (such as Debian jessie, Ubuntu
16.04, Fedora 20, etc ...)."

The tag I suggested, tbb-8.0a9-build3, is the tag of the most recent
alpha release (I think). I haven't tried building it myself--if it
doesn't work, you may just need to try a different tag. One of the Tor
Browser devs could suggest an alternative if it doesn't work on the
first try.
John Helmsen
2018-07-24 17:57:36 UTC
Permalink
Okay, I have generated a VM using VirtualBox of Ubuntu version 16. I've
had to restart the build process a couple of times, since the hard drive
was 10GB, then 20GB. Now I am using a 50GB box, so it may work this time.

Should we create the branch using the -b command? ('git checkout -b
tbb-8.0a9-build3'?) Otherwise, it complains about being headless.
Post by David Fifield
Thank you, I have created the ticket as #26920. https://trac.
torproject.org/
projects/tor/ticket/26920#ticket. Having downloaded the git project,
it seems
that this work cannot be performed on a Mac, since it doesn't run
'runc'. Is
that right?
Right, the README says "To build Tor Browser, you need a Linux
distribution that has support for runc (such as Debian jessie, Ubuntu
16.04, Fedora 20, etc ...)."
The tag I suggested, tbb-8.0a9-build3, is the tag of the most recent
alpha release (I think). I haven't tried building it myself--if it
doesn't work, you may just need to try a different tag. One of the Tor
Browser devs could suggest an alternative if it doesn't work on the
first try.
--
John Helmsen
***@redjack.com
C: (240) 899-5676
David Fifield
2018-07-24 18:09:45 UTC
Permalink
Okay, I have generated a VM using VirtualBox of Ubuntu version 16.  I've had to
restart the build process a couple of times, since the hard drive was 10GB,
then 20GB.  Now I am using a 50GB box, so it may work this time.
Should we create the branch using the -b command? ('git checkout -b
tbb-8.0a9-build3'?) Otherwise, it complains about being headless.
It doesn't matter, just for the sake of doing a testbuild and priming
the cache of dependencies (you'll notice the git_clones directory get
filled in among others). But yes, you'll want a branch for integration,
something like
git checkout -b marionette-integration tbb-8.0a9-build3
John Helmsen
2018-07-26 15:47:28 UTC
Permalink
David,

Ben hit the following error while running 'make testbuild':

--2018-07-26 08:33:09--
https://downloads.sourceforge.net/stixfonts/STIXv1.1.1-latex.zip
Resolving downloads.sourceforge.net (downloads.sourceforge.net)...
216.105.38.13
Connecting to downloads.sourceforge.net
(downloads.sourceforge.net)|216.105.38.13|:443...
connected.
HTTP request sent, awaiting response... 404 Not Found
2018-07-26 08:33:09 ERROR 404: Not Found.

When looking for the file, I found this on the website:

As of April 2018 STIX Fonts has moved to GitHub (
https://github.com/stipub/stixfonts/). All releases of STIX are available
only through GitHub and any new issues should be reported on the new site.
Post by John Helmsen
Post by John Helmsen
Okay, I have generated a VM using VirtualBox of Ubuntu version 16. I've
had to
Post by John Helmsen
restart the build process a couple of times, since the hard drive was
10GB,
Post by John Helmsen
then 20GB. Now I am using a 50GB box, so it may work this time.
Should we create the branch using the -b command? ('git checkout -b
tbb-8.0a9-build3'?) Otherwise, it complains about being headless.
It doesn't matter, just for the sake of doing a testbuild and priming
the cache of dependencies (you'll notice the git_clones directory get
filled in among others). But yes, you'll want a branch for integration,
something like
git checkout -b marionette-integration tbb-8.0a9-build3
--
John Helmsen
***@redjack.com
C: (240) 899-5676
Sukhbir Singh
2018-07-26 16:26:53 UTC
Permalink
Post by John Helmsen
David,
--2018-07-26 08:33:09--
https://downloads.sourceforge.net/stixfonts/STIXv1.1.1-latex.zip
Resolving downloads.sourceforge.net (downloads.sourceforge.net)...
216.105.38.13
Connecting to downloads.sourceforge.net
(downloads.sourceforge.net)|216.105.38.13|:443...
connected.
HTTP request sent, awaiting response... 404 Not Found
2018-07-26 08:33:09 ERROR 404: Not Found.
(https://trac.torproject.org/projects/tor/ticket/26949 tracks this issue.)

If you want to fix it for now, you can apply this and it should work:

diff --git a/projects/fonts/build b/projects/fonts/build
index 9b33da9..a1e066b 100644
--- a/projects/fonts/build
+++ b/projects/fonts/build
@@ -13,8 +13,8 @@ mkdir -p $distdir
END; %]

[% IF c("var/linux") || c("var/osx") %]
- unzip -o STIXv1.1.1-latex.zip -d STIX
- cp "STIX/Fonts/fonts/opentype/public/stix/STIXMath-Regular.otf" $distdir/
+ unzip -o 2.0.0.zip -d STIX
+ cp "STIX/stixfonts-2.0.0/archive/STIXv1.1.1/Fonts/STIX-Word/STIXMath-Regular.otf" $distdir/
[% END %]
[% IF c("var/linux") %]
cp NotoEmoji-Regular.ttf $distdir/
diff --git a/projects/fonts/config b/projects/fonts/config
index 1547403..9d11d2c 100644
--- a/projects/fonts/config
+++ b/projects/fonts/config
@@ -102,6 +102,6 @@ input_files:
- URL: https://github.com/googlei18n/noto-cjk/raw/f36eda03dfa5582a6d49abbfb5c83d0209584158/NotoSansTC-Regular.otf
sha256sum: e6b82f7d3dab605c428161124ceb5e169cde93de632d800297b167cdd88e7baa
enable: '[% c("var/linux") %]'
- - URL: https://downloads.sourceforge.net/stixfonts/STIXv1.1.1-latex.zip
- sha256sum: e3b0f712e2644438eee2d0dcd2b10b2d54f1b972039de95b2f8e800bae1adbd8
+ - URL: https://github.com/stipub/stixfonts/archive/2.0.0.zip
+ sha256sum: 4327a16797dabebedce28a9075671730e22c7f74831b24b1fb91e27faec5a235
enable: '[% c("var/linux") || c("var/osx") %]'
David Fifield
2018-07-26 16:32:54 UTC
Permalink
Post by Sukhbir Singh
Post by John Helmsen
--2018-07-26 08:33:09--
https://downloads.sourceforge.net/stixfonts/STIXv1.1.1-latex.zip
Resolving downloads.sourceforge.net (downloads.sourceforge.net)...
216.105.38.13
Connecting to downloads.sourceforge.net
(downloads.sourceforge.net)|216.105.38.13|:443...
connected.
HTTP request sent, awaiting response... 404 Not Found
2018-07-26 08:33:09 ERROR 404: Not Found.
(https://trac.torproject.org/projects/tor/ticket/26949 tracks this issue.)
Thanks, Sukhbir.

I think you can also work around it by downloading STIXv1.1.1-latex.zip
manually and copying it to out/fonts/STIXv1.1.1-latex.zip.

Georg Koppen
2018-07-26 08:10:00 UTC
Permalink
Post by David Fifield
Post by John Helmsen
We are in the process of writing the documentation for Marionette, but the
documentation on the web page should be sufficient for at least getting a full
evaluation started.  We'd like to have the evaluation complete by the end of
next month, hopefully the middle of next month, and stand ready to make any and
all changes necessary.
A full set of documentation will also be written for designing your own
protocols.  This is in process.
Please let us know what you need.
The Tor Browser developers may have more specific requests, but I can
suggest some steps to get started.
Open a ticket at https://trac.torproject.org/ for discussion and to
track progress.
Type: project
Component: Applications/Tor Browser
Keywords: marionette
The old ticket for FTE is a good reference: https://bugs.torproject.org/10362
And then it would help if you port your build process to the Tor Browser
https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking
First, just build
git clone https://git.torproject.org/builders/tor-browser-build.git
cd tor-browser-build
git checkout tbb-8.0a9-build3
make testbuild # or, e.g., testbuild-linux-x86_64
Then you'll have to add a new project (consisting of a "build" and
"config" file) for Marionette and each of its dependencies. You can copy
from existing projects as templates. Here is the meek project, for
https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/meek
https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js
rbm/rbm build gmp --target testbuild --target torbrowser-linux-x86_64
rbm/rbm build marionette --target testbuild --target torbrowser-linux-x86_64
Thanks David. Yes, those are good first steps to get started. Experience
shows that getting things running for Linux is the easiest part, thus
I'd suggest to start with that one first.

There is no need to have everything ready for all supported platforms to
get your PT integrated. It's fine to ship it for the alpha for some
platforms only to further shake out bugs and make it more robust. Thus,
don't hesitate to put things into review state early on.

Finally, don't hesitate as well to ask in #tor-dev or by mail or on the
trac ticket (I'll have it on my radar) in case you are stuck or are
running into issues, we are happy to help.

Georg
Loading...