Saptak's Blog Posts
Making my first OnionShare release
Posted: 2024-02-29T18:11:14+05:30One of the biggest bottlenecks in maintaining the OnionShare desktop application has been packaging and releasing the tool. Since OnionShare is a cross platform tool, we need to ensure that release works in most different desktop Operating Systems. To know more about the pain that goes through in making an OnionShare release, read the blogs[1][2][3] that Micah Lee wrote on this topic.
However, one other big bottleneck in our release process apart from all the technical difficulties is that Micah has always been the one making the releases, and even though the other maintainers are aware of process, we have never actually made a release. Hence, to mitigate that, we decided that I will be making the OnionShare 2.6.1 release.
PS: Since Micah has written pretty detailed blogs with code snippets, I am not going to include much code snippets (unless I made significant changes) to not lengthen this already long code further. I am going to keep this blog more like a narrative of my experience.
Getting the hardwares ready
Firstly, given the threat model of OnionShare, we decided that it is always good to have a clean machine to do the OnionShare release works, especially the signing part of things. Micah has already automated a lot of the release process using GitHub Actions over the years, but we still need to build the Apple Silicon versions of OnionShare manually and then merge with the Intel version to create a univeral2 app bundle.
Also, in general, it's a good practise to have and use the signing keys in a clean machine for a projective as sensitive as OnionShare that is used by people with high threat models. So I decided to get a new Macbook for the same. This would help me build the Apple Silicon version as well as sign the packages for the other Operating Systems.
Also, I received the HARICA signing keys from Glenn Sorrentino that is needed for signing the Windows releases.
Fixing the bugs, merging the PRs
After the 2.6.1-dev release was created, we noticed some bugs that we wanted to fix before making the 2.6.1. We fixed, reviewed and merged most of those bugs. Also, there were few older PRs and documentation changes from contributors that I wanted to be merged before making the release.
Translations
Localization is an important part of OnionShare since it enables users to use OnionShare in the language they are most comfortable with. There were quite some translation PRs. Also, emmapeel2 who always helps us with weblate wizardry, made certain changes in the setup, which I also wanted to include in this release.
After creating the release PR, I also need to check which languages are greater than 90% translated, and make a push to hopefully making some more languages pass that threshold, and finally make the OnionShare release with only the languages that cross that threshold.
Making the Release PR
And, then I started making the release PR. I was almost sure that since Micah had just made a dev release, most things would go smoothly. But my big mistake was not learning from the pain in Micah's blog.
Updating dependencies in Snapcraft
Updating the poetry dependencies went pretty smoothly.
There was nothing much to update in the pluggable transport scripts as well.
But then I started updating and packaging for Snapcraft and Flatpak. Updating tor versions to the latest went pretty smoothly. In snapcraft, the python dependencies needed to be compared manually with the pyproject.toml. I definitely feel like we should automate this process in future, but for now, it wasn't too bad.
But trying to build snap with snapcraft locally just was not working for me in my system. I kept getting lxd errors that I was not fully sure what to do about. I decided to move ahead with flatpak packaging and wait to discuss the snapcraft issue with Micah later. I was satisfied that at least it was building through GitHub Actions.
Updating dependencies in Flatpak
Even though I read about the hardship that Micah had to go through with updating pluggable transports and python dependencies in flatpak packaging, I didn't learn my lesson. I decided, let's give it a try. I tried updating the pluggable transports and faced the same issue that Micah did. I tried modifying the tool, even manually updating the commits, but something or the other failed.
Then, I moved on to updating the python dependencies for flatpak. The generator code that Micah wrote for desktops worked perfectly, but the cli gave me pain. The format in which the dependencies were getting generated and the existing formats were not matching. And I didn't want to be too brave and change the format, since flatpak isn't my area of expertise. But, python kind of is. So I decided to check if I can update the flatpak-poetry-generator.py files to work. And I managed to fix that!
That helped me update the dependencies in flatpak.
MacOS and Windows Signing fun!
Creating Apple Silicon app bundle
As mentioned before, we still need to create an Apple Silicon bundle and then merge it with the Intel build generated from CI to get the universal2 app bundle. Before doing that, need to install the poetry dependencies, tor dependencies and the pluggable transport dependencies.
And I hit an issue again: our get-tor.py script is not working.
The script failed to verify the Tor Browser version that we were downloading. This has happened before, and I kind of doubted that Tor PGP script must have expired. I tried verifying manually and seems like that was the case. The subkey used for signing had expired. So I downloaded the new Tor Browser Developers signing keys, created a PR, and seems like I could download tor now.
Once that was done, I just needed to run:
/Library/Frameworks/Python.framework/Versions/3.11/bin/poetry run python ./setup-freeze.py bdist_mac
rm -rf build/OnionShare.app/Contents/Resources/lib
mv build/exe.macosx-10.9-universal2-3.11/lib build/OnionShare.app/Contents/Resources/
/Library/Frameworks/Python.framework/Versions/3.11/bin/poetry run python ./scripts/build-macos.py cleanup-build
And amazingly, it built successfully in the very first try! That was easy! Now I just need to merge the Intel app bundle and the Silicon app bundle and everything should work (Spoiler alert: It doesn't!).
Once the app bundle was created, it was time to sign and notarize. However, the process was a little difficult for me to do since Micah had previously used an individual account. So I passed on the universal2 bundle to him and moved on to signing work in Windows.
Signing the Windows package
I had to boot into my Windows 11 VM to finish the signing and making the windows release. Since this was the first time I was doing the release, I had to first get my VM ready by installing all the dependencies needed for signing and packaging. I am not super familiar with Windows development environment so had to figure out adding PATH and other such things to make all the dependencies work. The next thing to do was setting up the HARICA smart card.
Setting up the HARICA smart card
Thankfully, Micah had already done this before so he was able to help me out a bit. I had to log into the control panel, download and import certificates to my smart card and change the token password and administrator password for my smart card. Apart from the UI of the SafeNet client not being the best, everything else went mostly smoothly.
Since Micah had already made some changes to fix the code signing and packaging stuff, it went pretty smooth for me and I didn't face much obstructions. Science & Design, founded by Glenn Sorrentino (who designed the beautiful OnionShare UX!), has taken on the role of fiscal sponsor for OnionShare and hence the package now gets signed under the name of Science and Design Inc.
Meanwhile, Micah had got back to me saying that the universal2 bundle didn't work.
So, the Apple Silicon bundle didn't work
One of the mistakes that I made was I didn't test my Apple Silicon build. I thought I will test it once it is signed and notarized. However, Micah confirmed that even after signing and notarizing, the universal2 build is not working. It kept giving segmentation fault. Time to get back to debugging.
Downgrading cx-freeze to 6.15.9
The first thought that came to my mind was, Micah had made a dev build in October 2023. So the cx-freeze release from that time should still be building correctly. So I decided to try and do build (instead of bdist_mac) with the cx-freeze version at that time (which was 6.15.9) and check if the binary created works. And thankfully, that did work. I tried with 6.15.10 and it didn't. So I decided to stick to 6.15.9.
So let's try now running bdist_mac, create a .app bundle and hopefully everything will work perfectly! But nope! The command failed with:
OnionShare.app/Contents/MacOS/frozen_application_license.txt: No such file or directory
So now I had a decision to make, should I try to monkey-patch this and just figure out how to fix this or try to make the latest cx-freeze work. I decided to give the latest cx-freeze (version 6.15.15) another try.
Trying zip_include_packages
So, one thing I noticed we were doing differently than what cx-freeze documentation and examples for PySide6 mentioned was we put our dependencies in packages, instead of zip_include_packages in the setup options.
"build_exe": {
"packages": [
"cffi",
"engineio",
"engineio.async_drivers.gevent",
"engineio.async_drivers.gevent_uwsgi",
"gevent",
"jinja2.ext",
"onionshare",
"onionshare_cli",
"PySide6",
"PySide6.QtCore",
"PySide6.QtGui",
"PySide6.QtWidgets",
],
"excludes": [
"test",
"tkinter",
...
],
...
}
So I thought, let's try moving all of the depencies into zip_include_packages from packages. Basically zip_include_packages includes the dependencies in the zip file, whereas packages place them in the file system and not the zip file. My guess was, the Apple Silicon configuration of how a .app bundle should be structured has changed. So the new options looked something like this:
"build_exe": {
"zip_include_packages": [
"cffi",
"engineio",
"engineio.async_drivers.gevent",
"engineio.async_drivers.gevent_uwsgi",
"gevent",
"jinja2.ext",
"onionshare",
"onionshare_cli",
"PySide6",
"PySide6.QtCore",
"PySide6.QtGui",
"PySide6.QtWidgets",
],
"excludes": [
"test",
"tkinter",
...
],
...
}
So I created a build using that, ran the binary, and it gave an error. But I was happy, because it wasn't segmentation fault. The error mainly because it was not able to import some functions from onionshare_cli. So as a next step, I decided to move everything apart from onionshare and onionshare_cli to zip_include_packages. It looked something like this:
"build_exe": {
"packages": [
"onionshare",
"onionshare_cli",
],
"zip_include_packages": [
"cffi",
"engineio",
"engineio.async_drivers.gevent",
"engineio.async_drivers.gevent_uwsgi",
"gevent",
"jinja2.ext",
"PySide6",
"PySide6.QtCore",
"PySide6.QtGui",
"PySide6.QtWidgets",
],
"excludes": [
"test",
"tkinter",
...
],
...
}
This almost worked. Problem was, PySide 6.4 had changed how they deal with ENUMs and we were still using deprecated code. Now, fixing the deprecations would take a lot of time, so I decided to create an issue for the same and decided to deal with it after the release.
At this point, I was pretty frustrated, so I decided to do, what I didn't want to do. Just have both packages and zip_include_packages. So I did that, build the binary and it worked. I decided to make the .app bundle. It worked perfectly as well! Great!
I was a little worried that adding the dependencies in both packages and zip_include_packages might increase the size of the bundle, but surprisingle, it actually decreased the size compared to the dev build. So that's nice! I also realized that I don't need to replace the lib directory inside the .app bundle anymore. I ran the cleanup code, hit some FileNotFoundError, tried to find if the files were now in a different location, couldn't find them, decided to put them in a try-except block.
After that, I merged the silicon bundle with Intel bundle to create the universal2 bundle again, sent to Micah for signing, and seems like everything worked!
Creating PGP signature for all the builds
Now that we had all the build files ready, I tried installing and running them all, and seems like everything is working fine. Next, I needed to generate PGP signature for each of the build files and then create a GitHub release. However, Micah is the one who has always created signatures. So the options for us now were:
- create an OnionShare GPG key that everyone uses
- sign with my GPG and update documentations to reflect the same
The issue with creating a new OnionShare GPG key was distribution. The maintainers of OnionShare are spread across timezones and continents. So we decided to create signature with my GPG and update the documentation on how to verify the downloads.
Concluding the release
Once the signatures were done, the next steps were mostly straightforward:
- Create a GitHub release
- Publish onionshare-cli on PyPi
- Push the build and signatures to the onionshare.org servers and update the website and docs
- Create PRs in Flathub and Homebrew cask
- Make the snapcraft edge to stable
The above went pretty smooth without much difficulty. Once everything was merged, it was time to make an announcement. Since Micah has been doing the announcements, we decided to stick with that for the release so that it reaches to more people.
Anonymous Chat using OnionShare
Posted: 2021-02-26T15:22:50+05:30
So the new OnionShare is out and it has a bunch of exciting new features and some improvements in the UI/UX designs of the tool. One of the main new features that I helped build was the anonymous chat feature in OnionShare. Just like the other modes (share, receive, and website), there is now a chat mode. So if you want to start a chat service, you just start the chat server, share the onion address of the server with people you want to chat with, everyone opens this onion address in Tor Browser and voila! You have an anonymous chat.
Let's dive in a little deeper into the feature.
Why do we need an anonymous chat?
A common question that we got during developing this feature is what's the use of an anonymous chat room since we already have end-to-end encrypted messaging apps. It leaves a lot fewer traces.
The way we achieve this is very simple. There is no form of storage whatsoever in OnionShare chat mode. The chat is not persistent. The chat server stores no information at all (not even the usernames of people chatting). So once the chat server is closed, and the Tor Browser tab with the chat client is closed, there is no data (or metadata) related to chat that remains, even in the person's system who started the server. Hence, it leaves much less trace compared to other chat applications.
A good example of the above as mentioned by Micah in his blog is:
If, for example, you send a message to a Signal group, a copy of your message ends up on each device (the devices, and computers if they set up Signal Desktop of each member of the group). Even if disappearing messages is turned on it’s hard to confirm all copies of the messages are actually deleted from all devices, and from any other places (like notifications databases) they may have been saved to. OnionShare chat rooms don’t store any messages anywhere, so the problem is reduced to a minimum.
Given that the OnionShare chat feature works over the onion network, so it also has the additional anonymity feature. Also, adding to the anonymity feature, OnionShare chat doesn't need any form of signing in. Hence, people chatting can stay anonymous, and everything happens inside the tor network. One can just start a chat server, share the link via some disposable way, and then wait for the other people to join while maintaining anonymity.
Because it's an onion service, there is no way for an attacker to eavesdrop on the messages. The closest they can get is if they run a malicious Tor rendezvous node that's chosen for the onion service, they'll be able to spy on encrypted onion traffic. So, there's no capturing ciphertext to decrypt later on.
So what happens under the hood?
The chat feature is dependent on flask-socketio and eventlet for the WebSocket server implementation, and socket.io client js for the frontend implementation of the chat client. So when a chat server is started, the WebSocket is started in a namespace "/chat". Whenever a new user joins the link, they are given a randomly generated username and they are added to the room "default". There is only one room, and the actual name of the room can be set from the OnionShare settings-related code, but it doesn't really impact anything in the implementation. Both the room name and the randomly generated username are stored in a flask session. But that information is also completely gone once the chat server is stopped. The room and username information are only there to emit the messages properly.
You can also change the randomly generated username to a username (or pseudo username) of your choice for that particular session.
There are two main types of messages:
- status messages - these are sent from the client to the server only when a new user joins or someone updates their username. The status message is then broadcasted to all the other connected clients, who will then see it as a form of a status message in the chat window.
- user messages - these are sent when a user sends a message. All messages are broadcasted, so in case you share the link to multiple users, there is no concept of private message and everyone connected to the room can view your messages. Hence, sharing the onion link securely is important.
All of these WebSocket communication happens over the Tor onion services. OnionShare in itself doesn't implement any encryption algorithm to the chat and heavily relies on the Tor onion service's encryptions for the same. The message from the client to the OnionShare server is E2EE as it goes via Tor's onion connection. Then the OnionShare server broadcasts the message to all the other clients connected to the chat room through their E2EE onion connection, over WebSockets.
So what now?
I feel, as of now, the OnionShare anonymous chat is great if you quickly want to have an anonymous, secure, non-persistent conversation with someone or a group of people. It is also great if a whistleblower wants to share some details over chat with a journalist and then remove all traces of that conversation completely. But I feel if someone needs to host a chat server for a long time where people can connect anonymously, this is probably not the best solution for that.
There are still some issues that we will hopefully improve in the next releases. Firstly, we need to try and make it a bit more asynchronous. Right now, if left inactive for a long time, the Tor connection over WebSocket is sometimes dropped which isn't a great user experience. We are working on improving that.
Also, we will improve the UI for the chat more to give a better user experience.
With the new tabs feature, one can have all different modes (even multiple servers of same mode) running at the same time. So you can have a chat server and share mode server running at the same time. All the modes are very independent of each other and hence don't affect one another in any way.
I hope you all enjoy the new chat feature and leave feedbacks/suggestions on how to improve it. You can also read more about this and other features at docs.onionshare.org