The results

The assets in scope for the audit can be found on Paul Miller's NIP 44 repository. Included is the full text of the spec, as well as implementations in Typescript, Rust, and Go.

Overall, here's what Cure53 had to say:

The Cure53 team succeeded in achieving very good coverage of the WP1-WP4 targets. All
ten findings spotted by the testers were classified as general weaknesses with limited
exploitation potential. In other words, no vulnerabilities were detected within the inspected
components.

In other words, no vulnerabilities were found, but there are things we can do to harden the implementations. Cure53 elaborates:

The fact that Cure53 was not able to identify any exploitable vulnerabilities can be
interpreted as a positive sign in regard to the security of the NIP44 specification and
implementations. Nevertheless, even though the spotted problem can all be seen as general
weaknesses and hardening advice, they should not be ignored. It is known that weaknesses
may serve as entry points for more severe vulnerabilities in the future. Cure53 strongly
recommends swift resolution of all reported flaws.

These weaknesses and recommendations each have their own code, prefixed by "NOS-01", which is our audit number. Here's a quick summary of the individual points:

• NOS-01-001 describes a weakness related to naive secp256k1 implementations, which may accept public keys with both x and y coordinates instead of just x and a sign-bit. This can result in the compromise of a sender's private key if the sender can be tricked into encrypting a message with an invalid public key. This is known as a "twist attack". Cure53 recommends some additional test vectors to make sure that uncompressed keys are not accepted.
• NOS-01-002 includes a handful of minor recommendations, including specifying the initial counter for ChaCha20, nailing down key formats, and a few other things which they go into more depth on elsewhere.
• NOS-01-003 provides some additional test vectors to prevent twist attacks and out of bound errors.
• NOS-01-004 suggests using a constant time equality function to prevent timing side-channel attacks, along with some code samples.
• NOS-01-005 identifies missing range checks in the Go implementation which should be addressed in order to avoid crashes.
• NOS-01-006 addresses the lack of provisions for forward secrecy in the NIP 44 spec. They suggest introducing sender-side forward secrecy by deriving the session key using ephemeral keys and a KDF.
• NOS-01-007 explains that using the same key for multiple purposes can lead to unexpected compromises when taken together. Best practice is to use a different key for signing and for encryption, which nostr violates. While the particular curves we use aren't vulnerable to this exploit, Cure53 recommends deriving an encryption key from a user's main key using a KDF in order to reduce the impact of a leaked encryption key.
• NOS-01-008 identifies an incorrect use of the salt parameter in calls to HKDF (an HMAC-based Key Derivation Function allows us to prove that an encrypted payload was not forged). Currently, the salt is generated randomly every time, but the security definition of HKDF specifies that it be static. This is likely not exploitable, but should be fixed. Cure53 suggests switching the nonce and salt when calling the HKDF, and deriving a static salt using something like SHA256(pubA, pubB, "nip-44-v2").
• NOS-01-009 demonstrates that the Message Authentication Code (MAC) tag is currently computed without a nonce. This doesn't compromise the encryption, and authentication is covered by event signatures as specified in NIP 01, but it does prevent the MAC from being useful on its own.
• NOS-01-010 suggests using clearer boundaries when constructing the HMAC payload.

The takeaway here is that with a few minor changes, the spec itself is sufficient to "provide a simple way for the users to communicate privately". There are, however, a number of edge cases which can compromise implementations, so for any developers out there porting the spec to new languages, be sure to take a look at Paul's test vectors.

In the end, the NIP44 specification and implementations appear to have already achieved the security goal of providing users with a way to communicate privately. Miscellaneous issues identified by Cure53 during NOS-01 correspond to further hardening of the current version and/or suggestions for future versions. Following these suggestions could provide more advanced security guarantees.

They also clarify that:

Importantly, the lack of certain security guarantees, such as forward secrecy, post-compromise security, deniability, and post-quantum, had been known to the development team prior to NOS-01.

Since this is the case, it's important that client developers be absolutely clear with their users about what NIP 44 is good for, and what it's not. This is a "simple" way to communicate privately — users with more advanced privacy requirements should use something like MLS or SimpleX.

What's next?

NIP 44 isn't quite ready for widespread use just yet, but it shouldn't take long to incorporate all the adjustments mentioned in the audit. Then there are several NIPs which rely on the encryption in NIP 04 which should eventually be migrated individually. Beyond that, a whole world of affordances for encrypted data becomes available on nostr.

First and most anticipated, we can finally deprecate NIP 04 and stop leaking DM metadata. Vitor's Sealed Gift Wraps are an excellent alternative to NIP 04 DMs, and add support for small group chats, which will be a big improvement to client UX.

Other more ambitious specifications targeting larger groups exist as well, including my Closed Communities draft spec. This NIP makes it possible to extend NIP 72 communities with a private component, which can be useful for publishers or real-life communities.

There are lots of other things NIP 44 might be used for as well, for example:

• The ability to share your location only with certain people
• Private calendar events, product listings, or streams
• Private tags — instead of encrypting an entire event, it should be possible to encrypt only a single tag's value. This allows senders to attach private data to public events.

Conclusion

There's lots of work still to be done, but I think this marks a huge milestone for nostr. So thank you to Paul Miller for his work on the spec and the audit, and to OpenSats for funding the audit!

This discussion gets complicated very quickly, so to begin, I thought I'd cover the two components that everyone thinks they understand first (relays and encryption), and move on to the one that combines the existing primitives in new and exciting ways (groups).

What even are relays, anyway?

If you saw my talk at Nostrasia, this section is just a re-iteration of it. In fact, I wrote this blog post before the conference to help me with my presentation. So hello, from the past.

Relays are dumb. They're just servers, which speak a particular protocol, and which implement some minimal logic to support that protocol. Relays have to be interoperable, which is why it's important to design new parts of the nostr protocol in such a way as to be backwards-compatible, and simple to implement.

As a result, relay development has historically focused on performance to the exclusion of new functionality. You can see this with the success of strfry and Will's adaptation of the underlying technique to nostrdb.

One of my favorite programming talks is Simple Made Easy by Rich Hickey, the author of Clojure. One of the points he makes in that talk is that object-oriented programming is such a mess because it "complects" behavior and state. Hickey's solution to this problem is to use functions and values rather than methods and state.

I think that this same mistake is one we are at risk of making with nostr relays. I'll come back to this in a bit, but keep that distinction in the back of your mind: separating data and behavior can lead to a much simpler and more robust system.

Despite the emphasis on interoperability, relays shouldn't all be exact clones. There are a few different ways relays can differentiate themselves, either by providing specific functionality, by storing different sets of events, or by exposing a different interface.

Added functionality

Some implementations have pushed features forward, for example rbr.bio which implemented the COUNT verb from NIP 45 early on. However, these efforts are usually either supported by special clients (like nostr.band or primal's caching service), or fail to gain wide adoption and are eventually abandoned. The result is that NIPs specifying optional functionality tend to die without sufficient user demand.

The reason for this is that if you're building a client that selects relays based on user preferences, it can't rely on functionality that doesn't exist on the majority of relays. Without widespread support, search results or feeds requested based on Primal's special syntax will be incomplete or fail entirely. As a result, clients often build proprietary solutions to these problems, increasing centralization risk.

Subtracted functionality

Relays are able to helpfully differentiate themselves however, by offering a subset of protocol functionality. For example, purplepag.es is a reliable source for profile data and relay preferences, but it doesn't require clients to do anything different in order to be useful. Instead, users can simply add purplepag.es to their relay list and instantly get a better experience. A similar approach many commodity relays take is blocking certain event kinds, for example 1059 gift wraps or NIP 94 image headers. This fortifies their particular purpose, which is to not support more exotic protocol features.

Other kinds of relays don't support this guarantee, for example relay.noswhere.com only supports search if the search term is not accompanied by a filter, responding to standard requests with "filters: not planned (non-search queries)". This can still be useful, but it is not a relay, because it requires clients to special case how it's used.

Data relevance

There is a different kind of differentiation relays can implement, based not on functionality, but on content. This is what I mean when I talk about "relay de-commodification". The value proposition this type of relay offers is not "we can do something no one else does", but "we have the information you're looking for".

There are two ways to accomplish this: content curation and access controls. The difference being that content curation is context-independent, and might be based on topic, keyword, or LLM analysis, while access controls only accepts content published by certain accounts. Several relay implementations and hosting providers (for example relay.tools) support both of these approaches, and of course it's easy to combine them.

Likewise, there are two ways for a relay to gain access to the desired data, leading to two different value propositions.

A relay that focuses on topical data might scrape the wider nostr network in order to ensure it is able to reliably serve everything relevant to its niche. This might be done using any number of heuristics, including filtering by pubkey, topic, or sentiment, but in any case the relay is not intended primarily to be written to, but to be read from. This is primarily an additive model, focusing on offering a complete view of the given topic.

Another approach is to let the data come to you instead of scraping it from the wider network by only accepting events published by certain people, or being the go-to relay for a certain topic. In order for this model to work though, there needs to be some way for the relay to keep this data to itself by preventing scraper-powered relays from harvesting its data. This effectively means that these relays will not only have policies for who can write to them, but also who can read from them. These might be the same set of people (for example with a community relay), or more people might be allowed to read from the relay than are able to write to it, as in the case of content producers who want a paywalled private social media enclave.

Keep relays weird

Most relay implementations expose their data over websockets in line with the spec, but this isn't actually inherent to the function of a relay. We've already seen several projects implement relays in-app for caching purposes. These relays don't usually go through the ceremony of spec adherence, but some (e.g. nostrdb) do.

Different transport mechanisms and hosting options are possible for relays, and would only require clients to use a specific adapter to get at them. Some examples:

• A relay running on your phone as a service that other nostr clients can share to reduce data storage requirements
• A relay running on a LAN, only accessible from a particular location
• A relay running via any other transport, for example radio or carrier pigeon

The idea I'm trying to get across here is that relays aren't necessarily websocket servers. More abstractly, they are a name attached to a set of events that can be accessed in a standard way.

So, what are relays?

To recap, here's my definition of a relay:

1. They correctly support standard functionality
2. They should only support additional functionality related to curating their data set
3. They may restrict the data types they accept and serve
4. They may restrict the data they accept and serve based on content or pubkey

One additional feature that isn't currently standard but ought to be is strfry's negentropy synchronization feature. This is the opposite of AUTH, which protects a relay's own dataset, in that it allows relays to more efficiently share their data with peers. A nice bonus is that it also allows clients to use their local cache more effectively as well.

Simple Made Easy

Going back to Rich Hickey's talk, separating behavior and state allows you to work directly with your data, freely implementing new behavior whenever you need to. This applies to sets of events stored on relays just as much as it applies to in-memory data.

One thing that makes this possible is cryptographic hash functions, which cleanly transform state into values - the difference being that "state" couples a name with data that may change, while a hash is a compact representation of immutable data. If two hashes of event ids don't match, they're not the same set of events.

How does this apply to relays? Instead of adding new behavior to relays directly ("methods"), we can instead call "functions" on a relay's data.

This function can be anything - a centralized server, client code, or a DVM - that takes REQ filters as input. The function gathers the needed data (maybe from a cache, maybe using a sync command, maybe from multiple relays for completeness), performs the transformation, and returns it. The result is that relays become simple, interoperable repositories for data, without limiting what server-side computation can be applied to events.

While huge search indexes and centralized servers have a cost in terms of centralization, I think DVMs solve this problem brilliantly - the common interface for this augmented behavior is open, and can admit anyone who wants to start competing for search, counting, recommendations, or anything else you might want to do with events.

As Alan Perlis said, "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures."

Encrypted Messages

Nostr DMs were a huge mistake, and I want to do it again.

NIP 04 has some obvious problems, most notably metadata leakage. The cryptography also hasn't been audited, and has some potential theoretical attack vectors. I've been working with some other nostr devs on a new cryptographic primitive, which will be audited and hopefully result in a better foundation for future work.

But fixing the cryptography still doesn't fix DMs. Nostr is not really the right architecture for secure private messaging, because ratchets require state. Nostr is designed such that there are no hard dependencies between events, which are replicated across multiple relays. This makes it very hard to manage the kind of state that protocols like SimpleX or Signal require.

Even apart from the difficulty of implementing stateful messaging protocols on nostr, we have two other problems to contend with: metadata leaks and spotty deliverability. Both of these problems come from the reliance of nostr on relays, which generally are not trustworthy or reliable.

Metadata Leakage

NIP 04 is really bad. Anyone can see who you corresponded with, when, how frequently, and how much you said just by looking at the event. Vitor's NIP 24 chat proposal aims to fix this with gift wraps, which use ephemeral keys to obscure the sender, padding to obscure message size, timestamp randomization to obscure timing, and double-wrapping to prevent leakage of the signed message.

This is a big improvement, but there are other forms of metadata that are still visible. To start, the recipient is still visible, along with how many messages they have received and their general size. So you can tell if someone sends lots of messages, or not a lot, but that's about it.

If we don't assume relays are trustworthy though, there are all kinds of implicit metadata that can be used to infer information about the sender, including:

• Client fingerprinting
• Relay selection for publishing and queries
• IP address collection
• First seen timestamp
• Identification of users by AUTH
• Correlation of the event with other messages sent/received during a session

These issues are much harder to solve, because they are part of the process of delivering the message, rather than just constructing it.

Transport and deliverability

The key feature of nostr that makes it work is relays. Relays provide storage redundancy and heterogeneity of purpose with a common interface shared by all instances. This is great for censorship resistance, but everything comes with a tradeoff.

Completeness and consistency are not guaranteed on nostr. This is ok in a social media context, because there's no way to keep up with everything everyone you follow says anyway, and you can rely on interactions and algorithmic tools to catch you up on anything important you missed.

But with private messages, every payload counts. Receiving messages out of order, with a delay, or not at all can severely disrupt conversations between two people, leading to misunderstandings and worse. This is a common experience on nostr; even fiatjaf has expressed skepticism about the viability of private messages on nostr.

This flakiness comes from a combination of low relay quality and poor relay selection on the part of clients. Relays can't deliver messages if they can't keep up with clients, go offline, or drop messages (accidentally or deliberately). Likewise, clients can't find messages if they look for them in the wrong place.

Relays fixes this

Let me just put things into perspective really quick. Twitter DMs are not e2e encrypted, and weren't encrypted at all until earlier this year. Mastodon messages are not encrypted - instance admins can read everything. Facebook has e2ee messages, but only between normal users, not in communities, groups, business chats, or marketplace chats.

So while nostr's architecture is not sufficient to make secure encrypted communication a value proposition of the protocol in itself, there is a lot we can do to improve on the status quo. Because nostr has no admins, the only option for privacy of any kind is end-to-end encryption. Metadata leakage notwithstanding, nostr messages can be "good enough" for some definition of that phrase.

It's important to not just punt on making DMs available in a standard way, because there is an expectation that a complete social media solution will have a way to establish a private communication channel with someone else in the network. Otherwise you're stuck with public correspondence and link sharing in order to hop to another protocol.

Let's just assume that you agree with me at this point, and that until there is a javascript SDK for SimpleX or MLS, we're stuck with nostr DMs of some kind. So how can we reduce metadata leakage and improve deliverability?

Fixing Flakiness

Let's start with deliverability first. NIP 65's inbox/outbox model is great. While it doesn't fix deliverability on its own, if every relay recommended was in fact reliable, and reliably followed by clients, there would be a very clear indication of where a particular user's messages could be found. Neither of these things is really true right now, but I do think clients are moving in this direction. Hopefully we can educate users better on how to pick good relays.

There are some limits to this model of course. For example, if you want to read notes from 1000 pubkeys at once, your client will likely end up connecting to hundreds of relays unless it has a cap, in which case it will miss a lot of notes.

One clean solution to this problem is relay proxies. There are a few implementations, but the one I run at mux.coracle.social is a stateless proxy that uses a wrapper protocol so clients can still select and dispatch to relays. I originally built this to improve mobile data use by deduplicating events on the server, and it does that pretty well, although it currently does it in the most naive way possible.

Multiplextr currently does not qualify as a "real" relay based on my four rules, since it uses a wrapper protocol. However, a smarter implementation would be able to route requests and events effectively by relying on client authentication and filter analysis, without changing the protocol. This would also allow clients to completely offload the complex business of relay selection to its proxy. Clients would also be able to combine multiple proxies just like they combine relays - because proxies are relays!

Proxying Privacy

Another downside of NIP 65 is that it encourages clients to connect indiscriminately to hundreds of different relays based on tag hints, other users' preferences, and relays baked into bech32 identifiers. This is not only bad for battery life and performance, but also allows attackers to easily trick your client into to connecting to their relay so they can start analyzing your traffic.

The best way to defend yourself against nosy relays is to not connect to them at all. And the easiest way to do that is to only connect to relays you control (or trust), and let them do the dirty work for you.

With a proxy, even if you don't control it, you can reduce the number of parties who can snoop on your traffic by orders of magnitude. And since other people use the proxy too, your traffic gets mixed with theirs, making it much harder to analyze. Of course, the downside of using a public proxy is that the proxy sees all messages you send and receive, and the relays you select for each request. Even worse, an untrusted multiplexer can hijack an authenticated session, exfiltrating private data from protected relays.

A proxy you control (or trust, either via social relationships or economic ones) is the best of both worlds. This proxy could also do some smart things for you to improve privacy as well, for example:

• Delayed publishing of gift wrapped messages so relays can't guess as easily when the event was first seen.
• Closing and re-opening connections to reduce the number of messages sent over an AUTH'd session.
• It could block suspicious messages on the fly, for example spam or notes with phishing links.
• If an abundance of options are available for relay selections, it could randomize which relays are used, or avoid using less reputable ones.

So I guess what I'm saying here, is "run a node". But not just any node - in the early days of nostr there were a lot of people spinning up relays because they thought it helped decentralization. Well, as it turns out, just like in bitcoin you have to use your node in order for it to be useful.

Beyond DMs

I'm not done yet, though. What's the point of all this? Why put all this effort into "good enough" DMs when we could spend the effort adopting SimpleX or MLS (said Semisol, who lives rent free in my head)?

Well, because there isn't a single "best" DM option. As I've dug into the topic, I've discovered that a ton of tradeoffs exist between complexity, privacy, and scale. The way I understand it, there are three basic options here:

• One-to-one encryption. This is how both SimpleX and Vitor's group message draft works. It allows for things like ratchets to be added on top, and is fairly simple - but fails to scale because every message has to be encrypted separately for every group member, resulting in n*m messages, where n is the number of group members, and m is the number of messages sent.
• Encryption via shared key. This is how WhatsApp works, and it scales really well because messages don't have to be duplicated. And while it does support forward secrecy, it does not support post-compromise security, so it is significantly less secure.
• MLS has a unique hierarchical approach to managing keys, which improves the complexity of one-to-one encryption so that it's logarithmic rather than exponential. However, MLS is highly stateful and still has scaling limitations.

My takeaway is that there is no perfect solution. Ratchets are the gold standard, but can't really be implemented using nostr relays, since they frequently require servers to have very specific characteristics, like the ability to deliver messages in order.

The fact remains though, that social media needs encrypted messaging of some kind, even if we encourage users to bail to a more complete solution if they want to be truly private. And encryption is useful beyond small group chats - think of Facebook groups, which scale to millions of people and whose users have a much lower expectation of privacy. This is what nostr is good at - social data, whether encrypted or in plaintext.

Belt and Suspenders

For a long time I have waffled between two visions of how private groups ought to work. Should they be encrypted messages sprinkled across the nostr network like confetti? Or should they live on one or two relays in plaintext, protected by AUTH?

Both of these options have issues. Sprinkling encrypted messages across the network almost guarantees deliverability problems, since already many commodity relays block gift wraps. And that's ok! They have no obligation to store your encrypted messages.

Storing group messages in plaintext is also sub-optimal, even if it's on a dedicated relay. Of course we're not going for military-grade secrecy here, but it would be trivially easy through a bug, or even a client that doesn't know how to deal with groups, to re-broadcast events that should stay private. We could do something wacky, like strip the signature from events hosted on a relay, but if we're going to do something non-standard, we might as well do the best we can.

So why not both? Group messages should be encrypted, and they should be stored on a particular set of relays. Again, assuming the relays used are trustworthy, this fixes both privacy and deliverability problems, because it's very clear where messages posted to the group should go.

Promoting relays to trusted status has some nice, unintended benefits too. One common objection to using encrypted content is that it becomes hard to filter. But if a relay is trusted to store a group's messages, it can also be trusted to be admitted as a member to the group. This would allow the relay to decrypt events and filter them to respond to requests by authenticated group members.

Of course, this would violate rule #1 for relays, since they would no longer be supporting standard functionality, but a guy can dream, can't he?

Conclusion

When I started working on nostr I had no idea it would end up being this complex to accomplish what I set out to do. Building a twitter clone was only a first step to an MVP, but I always had my eye on the prize of serving my local church community, which doesn't give a sat about social media per se.

Private groups are important to me not because I'm setting out to support political dissidents or journalists (although I hope they find nostr useful too), but because I want to get my friends off of Facebook, and the only way to do that is to create private marketplaces, calendars, communties, and more.

In order to make that happen though, a few things need to happen first. I need a better way to encrypt messages without leaking metadata, and (un)fortunately I'm no cryptographer. It's well known that NIP 04 has a number of issues, and there have been several proposals to fix it since as early as October 2022. Building on that foundation would be a waste of time.

My purpose in this post is to outline the problems with NIP 04 as it stands, and sketch out for you some possible solutions applicable not only to DMs, but to other encrypted use cases.

NIP 04 Considered Harmful

It's pretty well-known that nostr DMs leak metadata. What this means is that when you send a DM to someone else, anyone can see:

• Who you are
• Who you sent a message to
• When you sent the message
• How big the message was

And, of course, vice versa. Now, I personally don't take the cyanide pill on this like some do. DMs in centralized social platforms are far less private even than this, since the platform can read them anytime they want, share them with law enforcement, and potentially leak them to the entire world. It's my opinion that if applications were designed with the above properties in mind, it would be possible to gamify DMs, making metadata leakage a feature rather than a bug. Of course, no one has designed such a thing, so maybe it's a moot point.

The much bigger problem, as I learned last week, has to do with the cryptography used in NIP 04 (and several other places, including private mute lists and app-specific data). Here's a quote from Paul Miller's very helpful explanation:

1. Unhashed ECDH exposes some curves to Cheon's attack 1, which allows an attacker to learn private key structure when doing continuous diffie-hellman operations. Hashing converts decisional oracle into a computational oracle, which is harder.
2. CBC IVs should not only be unrepeatable, they must be unpredictable. Predictable IV leads to Chosen Plaintext Attack. There are real exploits.
3. CBC with reused key has 96-bit nonces and loses 6 bits of security for every magnitude. 10**4 (10K) messages would bring archive security of the scheme to 72 bits. 72 bits is very low. For example, BTC has been doing 2^67 hashes/sec as per early 2023. Advanced adversaries have a lot of resources to break the scheme
4. CBC has continuously been exploited by Padding Oracle attack. We can't expect all implementations to have proper padding check
5. AES was modeled as ideal cipher. Ideal ciphers don't care about non-random keys. However, AES is not an ideal cipher. Having a non-IND key, as in NIP4, exposes it to all kinds of unknown attacks, and reducing security of overall scheme

There is a lot going on here, very little of which I (and likely you) understand. But the really horrifying takeaway for me is in items 1 and 3. If you read it carefully, you'll notice that the more times you use your private key to encrypt data using NIP 04's encryption scheme, the easier it is for an attacker to brute-force your key.

I want to be careful not to overstate this — I don't think this means that everyone's private key is compromised; it would require many thousands of signatures to meaningfully reduce the security of your key. But it does mean that NIP 04's days are numbered — and if we don't replace it, we have a real existential threat to the entire protocol on our hands.

So much for motivation to put NIP 04 to rest. Where do we go from here?

The lay of the land

There are a few different components to making private messages work (whether we're talking about DMs or encrypted groups).

• Encryption scheme
• Explicit metadata hiding
• Implicit metadata hiding
• Transport and deliverability
• Identity and key exchange

Each of these components needs to be solved in order to create a robust messaging system on nostr.

Encryption scheme

As mentioned above, NIP 04 is borked. Luckily, there are a few folks in the nostr dev community who know a thing or two about encryption, most notably Paul Miller. Way back in May, Paul proposed a new encryption scheme based on his implementation of XChaCha20, which is the same algorithm used by SMP (more on that below).

To move the proposal forward, I created a different PR with some edits and adjustments, which is very near to being merged. The implementation already exists in nostr-tools, and there are PRs out for nos2x and Alby as well.

Explicit metadata hiding

As mentioned above, normal nostr messages share lots of information publicly, even if the content is encrypted, including:

• Kind
• Created timestamp
• Tags (topics, mentions, recipients, etc)
• Message size
• Author

Private messages have to hide this information, at the very minimum.

Implicit metadata hiding

The above level of security should be enough for most use cases. But it definitely doesn't get us to total secrecy yet. Some additional metadata includes:

• Client fingerprinting
• Relay selection for publishing and queries
• IP address collection
• First seen timestamp
• Identification of users by AUTH
• Correlation of the event with other messages sent/received during a session

These issues are much harder to solve, because they are part of the process of delivering the message, rather than just constructing it. These issues can be mitigated to some extent by using TOR, proxying connections with relays, using short-lived connections, randomizing publish time, and careful selection of trusted relays. But all of these are only techniques to reduce exposure, and require a prescriptive protocol to wrap them all up into a cohesive whole.

Transport and deliverability

In my view, the key feature of nostr that makes it work is relays. Relays provide storage redundancy and heterogeneity of purpose with a common interface shared by all instances. This is great for censorship resistance, but everything comes with a tradeoff.

Publishing notes is sort of like hiding easter eggs. If you want a kid to be able to find an egg, you have to put it somewhere they'll look for it. Putting an egg on the roof does not constitute "hiding" it, at least in a way consistent with the spirit of the game.

Historically, notes have been broadcast to whatever relays the user or client selects, without regard with someone else's preferences, resulting in apparently missing notes when author and recipient don't share a relay.

This problem has been partly solved by NIP 65, which encourages clients to publish notes where the recipient will look for them. There is some guesswork involved in selecting good relays, but the process is fairly straightforward for direct messages and group chats.

The situation can be improved further by DM-specific signaling, like what @fiatjaf has suggested as an addition to NIP 65. A user could then run his own relay that accepts messages from anyone, but only serves messages to himself. This would be the only relay well-behaved clients would ever write encrypted messages to, improving both privacy and deliverability.

Variants of this could also work for encrypted groups. For small groups, the same relay hints could be used as with DMs, and for larger groups with key sharing the admin could set up a dedicated relay for the group which members would write to.

This also creates a potential affordance for querying of encrypted events. If a relay is highly trusted, it could be added as a member to an encrypted group, gaining the ability to decrypt events. It could then receive REQ messages for those events, filter based on the encrypted content, and serve the encrypted version. This could greatly improve performance for larger groups where downloading all events isn't feasible.

Identity and key exchange

One dimension of privacy that deserves its own section is identity and key exchange. In nostr, this is simpler than in other protocols because in nostr your pubkey is your username. Arguably, this is a flaw that will need to be solved at some point, but for now it means that we get to skip a lot of the complexity involved with binding a name to a key.

However, the problem is still relevant, because one effective way to improve privacy is through key exchange. In the case of DMs, it would be nice to be able to request an alias key from someone you want to communicate with in order to hide the only piece of metadata that has to remain on a message in order to deliver it, that is, the recipient.

There's a lot of good stuff about this problem in the Messaging Layer Security protocol. The author of 0xChat has also put together a draft spec for executing simple key exchange, albeit with less information hiding than in the MLS protocol.

Key exchange between individual users can be considered an optional privacy enhancement, but is required for group chat based on a shared key. In either case, exchanging keys is fairly straightforward — the real challenge is achieving forward secrecy.

By default, if at any point a user's private key is compromised, any messages that were retained by an attacker can then be decrypted. This is a major vulnerability, and solving it requires key exchange messages to be reliably discarded.

What's cooking

So that's an overview of the problems that need to be solved for private DMs and group chats. Not all of these problems will be solved right away, just because the bar for creating a decentralized encrypted chat system on par (or better than) Signal is the kind of thing you get venture capital for.

So the question becomes, should we build a sub-par messaging system that is tightly integrated with nostr, or refer users to a better option for encrypted communication?

Well, why did Twitter create its own DM solution rather than integrating email? Was it to harvest user data, or to provide additional utility to their users by allowing the social graph to inform messaging? Native features simply create a better UX.

On the other hand, in Paul Miller's words:

I would like to remind that nostr DMs, if nostr gets a traction, would be used by all kinds of people, including dissidents in dangerous places. Would you be willing to risk someone getting killed just because you want to keep backwards compatibility with bad encryption?

There is no easy answer, but it's clear that as we continue to work on this problem it is very important that applications communicate clearly the privacy implications of all "private" features built on nostr.

So, assuming we can pull off an appropriate balance between privacy and convenience, let me outline three new developments in this area that I'm excited about.

Gift wrap

Way back in April, Kieran proposed a dumb, but effective, way of hiding metadata on DMs. The technique involved simply encrypting a regular nostr event and putting that in another event's content field. The neat thing about this is that not only does this hide some of the most revealing event metadata, it's not tied to a particular event kind — anything can be wrapped.

What this means is that we can build entire "nostr within nostr" sub-networks based on encryption — some between two users, others based on a shared group key. This is much more powerful than a messaging-specific encryption schema. Now we can make private reactions, private calendars, private client recommendations, private reviews. I find this very exciting!

This proposal was further developed by Vitor — the current version can be found here. The latest version introduces double-wrapping, which can help prevent the wrapped event from being leaked by stripping its signature — that "draft" event is encrypted, wrapped, and signed to verify authorship without revealing contents, and then that wrapper is in turn wrapped and signed with a burner key to avoid revealing the author.

Wrapping also makes it possible to randomize the created timestamp to avoid timing analysis, as well as to add padding to a message, reducing the possibility of payload size analysis.

New DMs and small groups

The main subject of Vitor's proposal was a new way to do DMs and small groups using double-wrapping to hide metadata. This works basically the same way NIP 04 did — a shared key is derived and the message is encrypted for the recipient, whose pubkey is revealed publicly so that it can be delivered.

However, Vitor was able to extend this to multiple recipients simply by wrapping the message multiple times, with multiple shared secrets. With the addition of some metadata on the inner event, this makes it possible to define a group where the members know who each other member is, but no one else does.

Large groups

This proposal has one important limitation though — the number of events that need to be signed and broadcast is equal to number of messages * number of members. This is fine with 5, 10, or even 100 members, but becomes infeasible when you reach groups of 1000+ members.

For that, we have two similar proposals, one by @simulx, and one by myself. The differences between the two are not important for this discussion — both use gift wrap as a way to exchange and rotate shared group keys, and both support a weak form of forward secrecy.

As a side note, this is the proposal I'm most excited about, and why I got into nostr in the first place. Private community groups which can support shared calendars, marketplaces, notes, etc., are what I'm most excited to see on nostr.

Shortcomings

The above proposals are great, but of course leave much to be desired. If these are built, nostr will be hugely improved in both security and functionality, with the following weaknesses:

• Deliverability is still an open question — are relay selections enough to ensure recipients get their messages, and no one else does?
• There is no way to create a group whose members cannot dox each other. You must always be able to trust the people you share a group with, and of course, the larger the group, the more people there are that can betray you.
• Content can always be shared outside a group, although in Vitor's small group proposal, the only way to verifiably leak messages would be to reveal your private key.
• Key rotation is not supported by the small groups proposal, but it may be possible to solve it for the entire protocol by building out a name resolution system for nostr.
• Forward secrecy for large groups is weak at best. A key rotation can be triggered at any time, including when a member leaves the group, but unless all relays and all members destroy the key rotation events, it may be possible to decrypt messages after the fact.
• Credential sharing still has to be bootstrapped, meaning at least one encrypted message for each participant may be publicly seen. Of course, credential sharing could also be done out of band, which would largely solve this problem.

What about SimpleX?

Lots of people, including Semisol and Will, have expressed interest in integrating nostr with SimpleX and using their cryptography and transport instead of nostr's.

The first question to answer though, is which protocol should we use? SimpleX is split into a few different pieces: there's the SimpleX Messaging Protocol, which describes setting up unidirectional channels, the SMP Agent Protocol which describes how to convert two unidirectional channels into a bidirectional channel, and the SimpleX Chat Protocol which describes a standard way to send chat messages across a bidirectional channel.

SimpleX Chat Protocol

So far, most proposals to fix messaging have focused simply on DMs, and sometimes on group chat. This is SimpleX's wheelhouse, so it could definitely make sense to communicate that kind of information over SimpleX's top-level chat protocol, and leave nostr out of it entirely.

The downside of this is that it requires clients to support two different protocols, and connect users to two categories of servers. This isn't a deal breaker, but it does increase the complexity involved in writing a full-featured nostr client, especially since there are no implementations of the SimpleX protocol other than (17k lines of) Haskell.

Using SimpleX's chat protocol would also limit the message types we'd be able to encrypt. Anything outside SimpleX's protocol (like zaps for example) would have to be sent using a custom namespace like nostr.kind.1234, which wouldn't be recognized by normal SimpleX clients. There are also areas of overlap where both protocols describe how to do something, for example profile data, requiring clients to choose which kind to support, or send both.

Another wrinkle is that the SimpleX protocol is AGPL v3 licensed, meaning proprietary software would not be able to interoperate, and MIT licensed products would have to abide by their license's terms.

SMP and Agent Protocols

The lower-level SMP and Agent protocols are more promising however, since their scope is narrower, and doesn't overlap as much with nostr's own core competencies. Both are quite simple, and very prescriptive, which should make it easy to create an implementation.

SMP is actually quite similar to some of the proposals mentioned above. Some of the same primitives are used (including XChaCha20 for encryption), and SMP is transport agnostic meaning it could be implemented over websockets and supported by nostr relays. The network topology is basically the same as nostr, except SimpleX doesn't make any assumptions about which servers should be selected, so in that regard nostr actually has an advantage.

The main incompatibility is that SimpleX requires the use of ed25519 and ed448 EdDSA algorithms for signatures, which means nostr's native secp256k1 signature scheme wouldn't be compatible. But this could be ok, since keys shouldn't be re-used anyway, and there are other ways to share a nostr identity.

Forward secrecy?

Whenever a server is involved, several attack vectors inherently exist that would not in peer-to-peer systems. Servers can threaten user privacy by storing messages that should be deleted, analyzing user behavior to infer identity, and leaking messages that should otherwise be kept secret.

This has been used by some to suggest that nostr relays are not a good way to transport messages from one user to another. However, SMP has the same attack vector:

Simplex messaging server implementations MUST NOT create, store or send to any other servers:

• Logs of the client commands and transport connections in the production environment.
• History of deleted queues, retrieved or acknowledged messages (deleted queues MAY be stored temporarily as part of the queue persistence implementation).
• Snapshots of the database they use to store queues and messages (instead simplex messaging clients must manage redundancy by using more than one simplex messaging server). In-memory persistence is recommended.
• Any other information that may compromise privacy or forward secrecy of communication between clients using simplex messaging servers.

So whether SimpleX servers or nostr relays are used, privacy guarantees are severely weakened if the server is not trustworthy. To maintain complete privacy in either scheme, users have to deploy their own servers.

Relays to the rescue

Because nostr servers are currently more commonly self-hosted, and more configurable, this makes nostr the better candidate for channel hosting. Users can choose which relays they send messages to, and relay admins can tune relays to only accept messages from and serve messages to certain parties, delete messages after a certain time, and implement policies for what kinds of messages they decide to relay.

All that to say, SMP itself or a similar protocol could easily be built into nostr relays. Relays are not the wrong tool for the job, in fact they're exactly the right tool! This is great news for us, since it means SMP is applicable to our problem, and we can learn from it, even if we don't strictly implement the protocol.

Bootstrapping over nostr

One assumption SMP makes is that channels will be bootstrapped out of band, and provides no mechanism for an initial connection. This is to ensure that channels can't be correlated with user identities. Out of band coordination is definitely the gold standard here, but coordination could also be done over the public nostr network with minimal metadata leakage — only that "someone sent {pubkey} something".

Unidirectional channels

Unidirectional channels are a great primitive, which can be composed to solve various problems. Direct messages between two parties are an obvious use case, but the SimpleX Chat Protocol also includes an extension for groups, defined as a set of bi-directional SimpleX connections with other group members.

This is similar to Vitor's small-group model, and likewise isn't conducive to supporting very large groups, since a copy of every message has to be sent to each group member.

However, it's possible to build large groups on top of unidirectional channels by sharing receiver information within the group and having all participants read from and write to a single channel. In fact, this is very similar to how our large group proposals work.

Conclusion

So, the takeaway here is that SimpleX's lower level protocols are great, and we should learn from them. It may be worth creating a NIP that describes the implementation of their unidirectional channels on nostr, and re-using that primitive in DM/group implementations, or simply following the same general process adapted to nostr.

I'm personally encouraged by reading SimpleX to see that we are headed in the right direction — although many of the details of our proposals have not been fully hammered out in the way SimpleX's has.

As for what the future holds, that's up to the builders. If you want to see tighter SimpleX integration, write a NIP and a reference implementation! I'll continue learning and fold in everything I can into my work going forward.