Digital signatures are essential to making Nostr work. The goal of Nostr is to break down walled gardens by subverting one of their key value propositions: content authentication. Or, in other words, the ability to know that a particular person said a particular thing.

This is a challenge in the digital world because information can be copied or fabricated at will. Simply saying that someone authored a particular piece of content doesn't make it so. When you go to Twitter and you load up a tweet, you only know that tweet is real because you trust Twitter. And if someone takes a screenshot of that or copies the text and emails it to you, then you have even less assurance that what's been presented to you is authentic.

What this means is that data that is not cryptographically signed is tightly coupled to custody. The only person who can reliably attest to the authenticity of a given piece of information is the person who can trace its provenance from the author, through storage, and to your device. This is very convenient for social media platforms — reliance on unsigned data means that they are needed. There has to be a single trustworthy custodian in order for unsigned data to work. The same is true of search results on Google; you don't know that search results are any good unless Google says they are.

What signed data gives us is the ability to know that something is true without having to trust anyone. If I create a note on Nostr and use my private key to sign it, anyone can verify the signature using the hash of the event and my public key (which is attached to the event). This lets them know that the event was created by the person who has access to my private key, i.e., me.

A Nostr event can thus be sent over an untrusted communication channel without the recipient losing the ability to know that it was me who signed it. As long as they know my public key, I can email a Nostr event, I can send a Nostr event over a peer-to-peer communication or over Bluetooth or over the LAN, or I can print it up and send it by mail. No intermediary can stop me without securing a monopoly on my communication.

Publicity Technology

The business model that fuels today's social media platforms is predicated on the capture of user data for their exclusive monetization. The user has become the product. Our data is used in a focused way to create targeted advertisements, or in the aggregate to understand and anticipate user behavior.

Signed data solves only half of this problem — it actually worsens the problem to the extent that data is public and accessible to anyone who wants to analyze it for patterns. Designing digital identity also has an incredible amount of complexity involved, and must be approached with caution. From Philip Sheldrake's essay, Human identity: the number one challenge in computer science:

Put starkly, many millions of people have been excluded, persecuted, and murdered with the assistance of prior identity architectures, and no other facet of information technology smashes into the human condition in quite the same way as digital identity[...] This should give anyone involved in digital identity cause to consider the emergent (i.e. unplanned) consequences of their work.

When designing systems that make use of digital identity, it's important to work from a conception of identity not as objective, but as subjective — that is, defined not by a set of static attributes, but by the dialectical contexts and relationships the person behind the identity participates in. The former kind of identity allows others to act upon the identity; the latter allows the person who own the identity to act.

Cryptographic identity doesn't automatically make this distinction, but can be used in either way. If the goal is user empowerment, a system of identity that is crafted to protect the digital freedoms of the user must be carefully designed.

Because identity is intended to be shared in a social setting, Nostr is not really "privacy technology". Rather, Nostr is "publicity technology".

When you create an event and you send it to untrusted custodians (particularly if left unprotected by access controls or encryption) you are advertising something about yourself to the entire world. All the data included in an event and all the metadata that can be harvested by observers and middlemen points back to you.

This is suitable for Twitter-like use cases (although user privacy is a concern even in a broadcast social media context), but always has to be considered when building products on Nostr. For users, it's best to use a VPN and Tor in combination with Nostr if you're concerned about privacy. Even so, in the aggregate signed data can still be collected and used to understand both individual users and entire social clusters.

Dis-intermediating Data

With that in mind, signed data does help reduce the capture of user attention by dis-intermediating content delivery. The current business model of social media platforms is predicated on the attention users give the platform, which is maximized by designs which stimulate "engagement", the creation and consumption of digital content.

The old way of doing this was through centralized content production. A business would create content — for example, movies, magazines, or podcasts — and present it to users for their consumption. Of course, it was a lot easier to directly monetize this content because it was both high quality and protected by intellectual property laws.

On social media, content is not produced by the platform, but by users. This introduces a second side to engagement — users not only consume, but also produce content. This keeps them even more engaged, and provides even more information about them to the platform.

When content is signed, it can no longer be captured by the platform (even if it is still visible to the platform). The result is that platforms lose the ability to enforce their monopoly on user attention. As a result of signed data, user attention can be diverted to other platforms that host a copy of the data. Nostr takes this effect even further by decoupling data storage and user interaction — relays store notes, but clients mediate user interactions.

On Nostr, clients can be more aligned with users, since they can only capture user attention to the extent that their functionality is what's valuable to the user, not the data they have access to.

The ability users have on an open network to leave a platform without losing all their data or their social graph is called credible exit. This is the opposite of "vendor lock-in", which occurs when platforms make it difficult to leave them. The export features social platforms offer are nearly useless because they break all the links in your social graph. But if all your social data was signed and the social graph was open, it would be quite easy to leave.

Social media companies can still exist in a world of signed data, but they will have to offer a real value proposition to their users in order to retain them. This means that they'll be more likely to serve their users rather than extract as much value as possible from them.

Whether open source software wins out or for-profit companies start building on Nostr, signed data weakens platforms' hold on their users and realigns the interests of social media platforms with those of their users. And while I think there's still room for skepticism about the effects of social media in general on people and communities, removing lock-in fixes a lot of existing perverse incentives in the system.

Dealing with other people's signed events is always going to require some adversarial thinking. Events can be malformed, either from a buggy implementation or from someone trying to attack software applications. A p tag might have an invalid or missing value, or a relay hint that is trying to spy on users. Event content might include HTML injection attacks, or illegal content.

Distinguishing between when to discard an event and when to try to handle it anyway is difficult. The most important thing when handling events is to protect the user from attacks. Everything else should be handled in a fault-tolerant way, assuming incompetence rather than hostility.

It's important to note that Postel's "be liberal in what you accept" has two different possible interpretations here. Fault tolerance does not mean repairing data that contradicts established conventions. A common example of this is when clients publish events with bech32-encoded data in tags instead of hex-encoded data.

This is of course a difficult judgment call, since as mentioned above the vocabulary of nostr is idiomatic and evolving. The important thing is not to allow a minority, non-standard data format to hijack a well-established content type. This can go some way toward taming the chaos of a decentralized system, as well as protect against organized attacks that would attempt to "embrace, extend, extinguish" the protocol.

Here are a few examples of malformed data and how it should be handled:

• If an event's signature is invalid, discard it — there's no way to prove it isn't forged. There are of course exceptions to this, like with NIP 59 wrapped events, which have no signature, but are still authenticated cryptographically.
• If an event contains code intended to crash, denial-of-service, or hijack the client, the client should protect itself using conventional defensive programming techniques — by sanitizing HTML before displaying it, not using regular expressions with user provided data, and by handling errors using null checks and fallbacks.
• If an event contains an adversarial relay hint or suspicious media URL, clients should have a policy in place, informed by user preferences, which blocks connections to unknown or untrustworthy hosts. This can be accomplished through domain white/black listing, or through trust assessments based on proof-of-work or the reputation of the event author.
• If an event contains data that doesn't adhere to the relevant specification, for example missing start or end times on a calendar event, clients might choose to show a fallback value, e.g. "not specified", or an error indicator.

One reason to prefer fault-tolerant handling over aggressive validation is that strict validation reduces interoperability unnecessarily by converting minor, recoverable errors into catastrophic exceptions that ultimately result in a poor UX. Care has to be taken to balance user protection, user experience, and long-term stewardship of the protocol when making these decisions.

An additional wrinkle to this problem involves writing rather than reading events. A number of bugs have emerged where data published by one client will be unwittingly deleted by another client. Here are a few examples:

• Some clients at one point started adding JSON-encoded relay selections to the content field of kind 3 follower list events. Other clients would drop these selections when updating user follows.
• Some clients started adding muted words to kind 10000 mute lists, in addition to muted pubkeys. Clients that weren't expecting this ended up dropping muted words when updating muted pubkeys.
• Similarly, some clients began adding private mutes to kind 100000 mute lists by JSON-encoding the tags, encrypting the result, and placing it in the event's content field. This resulted in mutes being ignored or overwritten on update.

In every case, these bugs are a result of poor design — overloading a single kind for multiple purposes is always a recipe for disaster. But in cases where poor designs become conventional, it can be helpful to handle events in such a way that unanticipated data doesn't get dropped.

One way I've found to do this is to write parsers for an event in order to turn it into a standard data structure that can be used elsewhere in my application with the original event attached. Then, when saving a new version of the event, I update the event directly, keeping tags and content intact as much as possible before re-publishing. Here's an example:

readProfile = e => {...decodeJson(e.content), e}
writeProfile = ({e, ...p}) => {kind: 0, content: encodeJson(p), tags: e.tags}

You can see that the original event is passed through to writeProfile, allowing us to avoid dropping any tags that may have been included.

This adds a certain amount of additional effort to implementations, but is important for avoiding disruption of user experience.

Backwards Compatibility

Backwards incompatibility is one of the big problems of spec design. When breaking backwards compatibility, not only do you break other existing implementations, but in a system where events can't be migrated to the new format you also break all historical data, even if it was published by your own app.

At the same time, breaking backwards compatibility can free you up to improve the protocol in important ways. In many cases, it also won't have much of an impact. Supporting historical data is important for some applications, like archival services, but many applications have a strong recency bias such that they don't necessarily mind throwing away old data.

I think there's a balance to this. I don't think we can be backwards compatibility maximalists, but we also shouldn't be careless about throwing away old formats for no reason. There is a perverse impulse among software developers to refactor for dumb reasons which should absolutely be avoided.

When introducing new backwards-compatible functionality to Nostr, it's usually best to enhance existing data formats as long as it doesn't result in overloading a single kind with multiple use cases.

When breaking backwards compatibility however, it's best to create an entirely new kind and evangelize for migrating to it. This is much harder than modifying the behavior of existing event kinds, but it's far more polite. The downside is it breaks network effects - and can still result in a poor UX for clients that don't adopt the new format if it results in missing content.

Over time, Nostr protocol development has gotten increasingly conservative. As use cases proliferate, it becomes more difficult to get feedback on new data formats. As implementations proliferate, it becomes more difficult to advocate for breaking changes. Nostr specifications are not sacred, but their effectiveness relies almost entirely on interoperability. Maintaining compatibility requires conscientiousness, communication, and contributions to other projects - there's no better way to get people to listen to you than writing code to improve their implementation.

The Economics of Censorship

Nostr was originally designed to be "censorship-resistant". But this is only one part of a bigger vision. Nostr is designed to promote the digital freedoms of users — things like privacy, access to information, anonymity, freedom of association, and credible exit. Because of how cryptography was adopted (primarily by institutions) in the early days, the internet is no longer aligned with the interests of its users, but has largely been captured by "platforms".

As a result, we have a state of rent-seeking in which the users who contribute all the value that the internet has to offer are exploited, undermined, manipulated, deceived, and sold by the custodians of that value. This is a far cry from the humanitarian dream of the early internet. In the words of Tim Berners-Lee,

We create the Web, by designing computer protocols and software; this process is completely under our control. We choose what properties we want it to have and not have[...] The goal of the Web is to serve humanity. We build it now so that those who come to it later will be able to create things that we cannot ourselves imagine.

In the fifteen years since this was written, the internet has only become more predatory, and proprietary. Everywhere you go, you're being tracked — maybe for the purposes of understanding your behavior to serve you ads, maybe for reasons more convoluted, but always in the interest of the ones doing the tracking.

This isn't to say the internet isn't a net positive to those who use it. The internet is the single most successful technology in human history for promoting access to information, alternative community, and trade. But it is not what it should be. Reforming the internet requires active investment from those who stand the gain or lose the most from how it evolves — its users. No one can do this for us; we have to hold companies and platforms accountable ourselves. And in order to do that, we need to understand why censorship happens.

Censorship is not the product of mere arbitrary exercise of power, but of incentives. In authoritarian regimes, censorship is implemented to protect the regime's power. In business, censorship is implemented to protect the business' revenue.

Traditionally, products are sold directly to customers, who then get access to the good or service they've purchased. This business model can transfer to some extent to the realm of information-based products, particularly in a business-to-business setting, where information can be justified in terms of greater productivity.

But consumer software is an entirely different story. If your users are only there to improve their quality of life through access to entertainment or communication, and the marginal cost of production trends toward zero for digital goods, then the price will also trend toward zero because competitors can always offer the same product at a lower price.

Combine that with the imperative for platforms to aggressively acquire users, and you get a strong downward pressure on the upfront pricing of software products. "Network effect" is the idea that a network's value does not grow linearly with the number of users, but quadratically with the number of connections between users.

Building a large network requires significant resources in order to fund growth or stay ahead of competitors. These resources in turn must be provided by people with capital. As a result, businesses that rely on scaling network effects are typically funded by investors and oriented at a return on investment. This includes social media platforms and similar businesses, such as search engines, marketplaces, and entertainment platforms.

Capitalists want to see the businesses they've invested in make a profit. The best way to make a profit in a market, according to Peter Thiel's Zero to One, is to establish a monopoly. Because user retention depends on network effects, market saturation is always the goal, and switching costs must be high — companies must "capture" their users.

The art of creating this moat seems to have been perfected. Facebook has been around for a long time, as has Google. In contrast to early internet companies like MySpace, Yahoo, and AOL, these second-generation internet giants have learned how to simultaneously retain and monetize their users.

Very little of these companies' revenue comes from direct monetization. Instead, these big tech companies monetize their users by making them the product. Users have two things that platform owners can sell: their attention and their data. Platforms can only sell these things because they have access to them by virtue of being intermediaries. Platforms subsist by inserting themselves into private or public relationships in order to siphon off these intangible goods.

This works because the majority of internet users either don't believe their attention or data are worth anything ("why do I need privacy? I have nothing to hide"), or feel helpless to opt out of the attention economy. Seeing a billboard in your peripheral vision or having cameras record your license plate seem hardly comparable to paying a monthly fee for road use. Whether you care about being recorded or not, you're still going to use the roads.

It's true that data and attention are not particularly valuable on an individual basis. But they're massively powerful in the aggregate. "Big data" is not mere information, but the ability to predict patterns of behavior, which can then be used at scale to manipulate people — to buy, believe, and act.

Advertising is not just about informing willing buyers about products and services, it's about conditioning people to act in predictable ways. Social engineering (like all engineering) is progressive and totalizing — unless restrained. The cost of giving away our attention and data to these internet intermediaries is, ultimately, the loss of our free will.

This is particularly true because users are progressively getting less and worse service in exchange for their time and attention. Because big internet companies have access to economies of scale that individuals don't, they have the ability to asymmetrically impose complexity costs on users who fall outside of the streamlined "ideal customer profile" they've built their business on. Quoting from this blog post:

Bigger players like corporations and governments gained a big systemic advantage over individuals: they make the rules but they don’t bear the costs of the rules breaking. They increase the complexity of everyday life and instead of being punished for it, they are rewarded for it.

A Path Forward

This slide isn't inevitable. But it's also human nature — everyone wants something for nothing, and so internet users have voluntarily given up their free will to gain connectivity, entertainment, and efficiency. Simply pushing back is not enough — the problem is structural, and time has allowed its beneficiaries to entrench their position.

This problem needs more than a technical solution — politics, culture, community, and capital all have roles to play here. Nostr's role is to offer tools that individuals can use according to their own values, and for their own interests. Nostr makes it possible for internet users to preserve their own digital sovereignty and hold tech platforms accountable.

My personal hope for Nostr is that it will aid in the restoration and cultivation of human flourishing and agency in a world increasingly mediated by digital technology. Nostr does not solve every problem. The ones it does solve, it sometimes solves poorly. Nostr is not a panacea, it is a toolset that can be applied to our benefit or to our detriment, but which has certain intrinsic qualities that the internet desperately needs right now.

I want to emphasize that it is only through people that any of the internet's ills can be cured. Nostr is not a comprehensive system for restraining digital evil (such a thing would be itself totalitarian), but a way of allowing individual and community agency to be exercised in solving problems germane to individuals' and communities' own particular circumstances.

An example of this is how for a while it was considered best practice to re-publish notes when replying to them. This practice emerged before the outbox model gained traction, and was a hacky way of attempting to ensure relays had the full context required for a given note. Over time though, pubkey hints emerged as a better way to ensure other clients could find required context.

Another one of these things is "relay-based groups", or as I prefer to call it "relays-as-groups" (RAG). Such a thing doesn't really exist - there's no spec for it (although some aspects of the concept are included in NIP 29), but at the same time there are two concrete implementations (Flotilla and Chachi) which leverage several different NIPs in order to create a cohesive system for groups on nostr.

This composability is one of the neat qualities of nostr. Not only would it be unhelpful to specify how different parts of the protocol should work together, it would be impossible because of the number of possible combinations possible just from applying a little bit of common sense to the NIPs repo. No one said it was ok to put t tags on a kind 0. But no one's stopping you! And the semantics are basically self-evident if you understand its component parts.

So, instead of writing a NIP that sets relay-based groups in stone, I'm writing this guide in order to document how I've combined different parts of the nostr protocol to create a compelling architecture for groups.

Relays

Relays already have a canonical identity, which is the relay's url. Events posted to a relay can be thought of as "posted to that group". This means that every relay is already a group. All nostr notes have already been posted to one or more groups.

One common objection to this structure is that identifying a group with a relay means that groups are dependent on the relay to continue hosting the group. In normal broadcast nostr (which forms organic permissionless groups based on user-centric social clustering), this is a very bad thing, because hosts are orthogonal to group identity. Communities are completely different. Communities actually need someone to enforce community boundaries, implement moderation, etc. Reliance on a host is a feature, not a bug (in contrast to NIP 29 groups, which tend to co-locate many groups on a single host, relays-as-groups tends to encourage one group, one host).

This doesn't mean that federation, mirrors, and migration can't be accomplished. In a sense, leaving this on the social layer is a good thing, because it adds friction to the dissolution/forking of a group. But the door is wide open to protocol additions to support those use cases for relay-based groups. One possible approach would be to follow this draft PR which specifies a "federation" event relays could publish on their own behalf.

Relay keys

This draft PR to NIP 11 specifies a self field which represents the relay's identity. Using this, relays can publish events on their own behalf. Currently, the pubkey field sort of does the same thing, but is overloaded as a contact field for the owner of the relay.

AUTH

Relays can control access using NIP 42 AUTH. There are any number of modes a relay can operate in:

1. No auth, fully public - anyone can read/write to the group.
2. Relays may enforce broad or granular access controls with AUTH.

Relays may deny EVENTs or REQs depending on user identity. Messages returned in AUTH, CLOSED, or OK messages should be human readable. It's crucial that clients show these error messages to users. Here's how Flotilla handles failed AUTH and denied event publishing:

LIMITS, PROBE, or some other reflection scheme could also be used in theory to help clients adapt their interface depending on user abilities and relay policy.

3. AUTH with implicit access controls.

In this mode, relays may exclude matching events from REQs if the user does not have permission to view them. This can be useful for multi-use relays that host hidden rooms. This mode should be used with caution, because it can result in confusion for the end user.

See Frith for a relay implementation that supports some of these auth policies.

Invite codes

If a user doesn't have access to a relay, they can request access using this draft NIP. This is true whether access has been explicitly or implicitly denied (although users will have to know that they should use an invite code to request access).

The above referenced NIP also contains a mechanism for users to request an invite code that they can share with other users.

The policy for these invite codes is entirely up to the relay. They may be single-use, multi-use, or require additional verification. Additional requirements can be communicated to the user in the OK message, for example directions to visit an external URL to register.

See Frith for a relay implementation that supports invite codes.

Content

Any kind of event can be published to a relay being treated as a group, unless rejected by the relay implementation. In particular, NIP 7D was added to support basic threads, and NIP C7 for chat messages.

Since which relay an event came from determines which group it was posted to, clients need to have a mechanism for keeping track of which relay they received an event from, and should not broadcast events to other relays (unless intending to cross-post the content).

Rooms

Rooms follow NIP 29. I wish NIP 29 wasn't called "relay based groups", which is very confusing when talking about "relays as groups". It's much better to think of them as sub-groups, or as Flotilla calls them, "rooms".

EDIT: Flotilla has migrated to exclusively use "managed rooms" — i.e., fully NIP 29 compliant rooms. Relays without NIP 29 support can still support chat, but all messages will be presented as sent to a single room. I've removed references to unmanaged rooms in what follows.

Rooms have two modes - managed and unmanaged. Managed rooms follow all the rules laid out in NIP 29 about metadata published by the relay and user membership. In either case, rooms are represented by a random room id, and are posted to by including the id in an event's h tag. This allows rooms to switch between managed and unmanaged modes without losing any content.

Managed room names come from kind 39000 room meta events, but unmanaged rooms don't have these. Instead, room names should come from members' NIP 51 kind 10009 membership lists. Tags on these lists should look like this: ["group", "groupid", "wss://group.example.com", "Cat lovers"]. If no name can be found for the room (i.e., there aren't any members), the room should be ignored by clients.

Rooms present a difficulty for publishing to the relay as a whole, since content with an h tag can't be excluded from requests. Currently, relay-wide posts are h-tagged with _ which works for "group" clients, but not more generally. I'm not sure how to solve this other than to ask relays to support negative filters. I have ideas on how to solve this in future iterations of relay-based groups, for example using virtual relays or just a better rooms spec.

Cross-posting

The simplest way to cross-post content from one group (or room) to another, is to quote the original note in whatever event kind is appropriate. For example, a blog post might be quoted in a kind 9 to be cross-posted to chat, or in a kind 11 to be cross-posted to a thread. kind 16 reposts can be used the same way if the reader's client renders reposts.

Posting the original event to multiple relays-as-groups is trivial, since all you have to do is send the event to the relay. Posting to multiple rooms simultaneously by appending multiple h tags is however not recommended, since group relays/clients are incentivised to protect themselves from spam by rejecting events with multiple h tags (similar to how events with multiple t tags are sometimes rejected).

Privacy

Currently, it's recommended to include a NIP 70 - tag on content posted to relays-as-groups to discourage replication of relay-specific content across the network.

Another slightly stronger approach would be for group relays to strip signatures in order to make events invalid (or at least deniable). For this approach to work, users would have to be able to signal that they trust relays to be honest. We could also use ZkSNARKS to validate signatures in bulk.

In any case, group posts should not be considered "private" in the same way E2EE groups might be. Relays-as-groups should be considered a good fit for low-stakes groups with many members (since trust deteriorates quickly as more people get involved).

Membership

There is currently no canonical member list published by relays (except for NIP 29 managed rooms). Instead, users keep track of their own relay and room memberships using kind 10009 lists. Relay-level memberships are represented by an r tag containing the relay url, and room-level memberships are represented using a group tag.

Users can choose to advertise their membership in a RAG by using unencrypted tags, or they may keep their membership private by using encrypted tags. Advertised memberships are useful for helping people find groups based on their social graph:

User memberships should not be trusted, since they can be published unilaterally by anyone, regardless of actual access, so it's better to think of them as "bookmarked groups" or "favorites". Possible improvements in this area would be the ability to provide proof of access:

• Relays could publish member lists (although this would sacrifice member privacy)
• Relays could support a new command that allows querying a particular member's access status
• Relays could provide a proof to the member that they could then choose to publish or not

Moderation

There are two parts to moderation: reporting and taking action based on these reports.

Reporting is already covered by NIP 56. Clients should be careful about encouraging users to post reports for illegal content under their own identity, since that can itself be illegal. Relays also should not serve reports to users, since that can be used to find rather than address objectionable content.

Reports are only one mechanism for flagging objectionable content. Relay operators and administrators can use whatever heuristics they like to identify and address objectionable content. This might be via automated policies that auto-ban based on reports from high-reputation people, a client that implements NIP 86 relay management API, or by some other admin interface.

There's currently no way for moderators of a given relay to be advertised, or for a moderator's client to know that the user is a moderator (so that they can enable UI elements for in-app moderation). This could be addressed via NIP 11, LIMITS, or some other mechanism in the future.

General best practices

In general, it's very important when developing a client to assume that the relay has no special support for any of the above features, instead treating all of this stuff as progressive enhancement.

For example, if a user enters an invite code, go ahead and send it to the relay using a kind 28934 event. If it's rejected, you know that it didn't work. But if it's accepted, you don't know that it worked - you only know that the relay allowed the user to publish that event. This is helpful, becaues it may imply that the user does indeed have access to the relay. But additional probing may be needed, and reliance on error messages down the road when something else fails unexpectedly is indispensable.

This paradigm may drive some engineers nuts, because it's basically equivalent to coding your clients to reverse-engineer relay support for every feature you want to use. But this is true of nostr as a whole - anyone can put whatever weird stuff in an event and sign it. Clients have to be extremely compliant with Postell's law - doing their absolute best to accept whatever weird data or behavior shows up and handle failure in any situation. Sure, it's annoying, but it's the cost of permissionless development. What it gets us is a completely open-ended protocol, in which anything can be built, and in which every solution is tested by the market.

But your experience has not been all you hoped. Before you've even had a chance to upload your AI-generated cyberpunk avatar or make up exploit codenames for your pseudonym's bio, you've been confronted with a new concept that has left you completely nonplussed.

It doesn't help that this new idea might be called by any number of strange names. You may have been asked to "paste your nsec", "generate a private key", "enter your seed words", "connect with a bunker", "sign in with extension", or even "generate entropy". Sorry about that.

All these terms are really referring to one concept under many different names: that of "cryptographic identity".

Now, you may have noticed that I just introduced yet another new term which explains exactly nothing. You're absolutely correct. And now I'm going to proceed to ignore your complaints and talk about something completely different. But bear with me, because the juice is worth the squeeze.

Identity

What is identity? There are many philosophical, political, or technical answers to this question, but for our purposes it's probably best to think of it this way:

Identity is the essence of a thing. Identity separates one thing from all others, and is itself indivisible.

This definition has three parts:

• Identity is "essential": a thing can change, but its identity cannot. I might re-paint my house, replace its components, sell it, or even burn it down, but its identity as something that can be referred to - "this house" - is durable, even outside the boundaries of its own physical existence.
• Identity is a unit: you can't break an identity into multiple parts. A thing might be composed of multiple parts, but that's only incidental to the identity of a thing, which is a concept, not a material thing.
• Identity is distinct: identity is what separates one thing from all others - the concept of an apple can't be mixed with that of an orange; the two ideas are distinct. In the same way, a single concrete apple is distinct in identity from another - even if the component parts of the apple decompose into compost used to grow more apples.

Identity is not a physical thing, but a metaphysical thing. Or, in simpler terms, identity is a "concept".

I (or someone more qualified) could at this point launch into a Scholastic tangent on what "is" is, but that is, fortunately, not necessary here. The kind of identities I want to focus on here are not our actual identities as people, but entirely fictional identities that we use to extend our agency into the digital world.

Think of it this way - your bank login does not represent you as a complete person. It only represents the access granted to you by the bank. This access is in fact an entirely new identity that has been associated with you, and is limited in what it's useful for.

Other examples of fictional identities include:

• The country you live in
• Your social media persona
• Your mortgage
• Geographical coordinates
• A moment in time
• A chess piece

Some of these identites are inert, for example points in space and time. Other identies have agency and so are able to act in the world - even as fictional concepts. In order to do this, they must "authenticate" themselves (which means "to prove they are real"), and act within a system of established rules.

For example, your D&D character exists only within the collective fiction of your D&D group, and can do anything the rules say. Its identity is authenticated simply by your claim as a member of the group that your character in fact exists. Similarly, a lawyer must prove they are a member of the Bar Association before they are allowed to practice law within that collective fiction.

"Cryptographic identity" is simply another way of authenticating a fictional identity within a given system. As we'll see, it has some interesting attributes that set it apart from things like a library card or your latitude and longitude. Before we get there though, let's look in more detail at how identities are authenticated.

Certificates

Merriam-Webster defines the verb "certify" as meaning "to attest authoritatively". A "certificate" is just a fancy way of saying "because I said so". Certificates are issued by a "certificate authority", someone who has the authority to "say so". Examples include your boss, your mom, or the Pope.

This method of authentication is how almost every institution authenticates the people who associate with it. Colleges issue student ID cards, governments issue passports, and websites allow you to "register an account".

In every case mentioned above, the "authority" creates a closed system in which a document (aka a "certificate") is issued which serves as a claim to a given identity. When someone wants to access some privileged service, location, or information, they present their certificate. The authority then validates it and grants or denies access. In the case of an international airport, the certificate is a little book printed with fancy inks. In the case of a login page, the certificate is a username and password combination.

This pattern for authentication is ubiquitous, and has some very important implications.

First of all, certified authentication implies that the issuer of the certificate has the right to exclusive control of any identity it issues. This identity can be revoked at any time, or its permissions may change. Your social credit score may drop arbitrarily, or money might disappear from your account. When dealing with certificate authorities, you have no inherent rights.

Second, certified authentication depends on the certificate authority continuing to exist. If you store your stuff at a storage facility but the company running it goes out of business, your stuff might disappear along with it.

Usually, authentication via certificate authority works pretty well, since an appeal can always be made to a higher authority (nature, God, the government, etc). Authorities also can't generally dictate their terms with impunity without losing their customers, alienating their constituents, or provoking revolt. But it's also true that certification by authority creates an incentive structure that frequently leads to abuse - arbitrary deplatforming is increasingly common, and the bigger the certificate authority, the less recourse the certificate holder (or "subject") has.

Certificates also put the issuer in a position to intermediate relationships that wouldn't otherwise be subject to their authority. This might take the form of selling user attention to advertisers, taking a cut of financial transactions, or selling surveillance data to third parties.

Proliferation of certificate authorities is not a solution to these problems. Websites and apps frequently often offer multiple "social sign-in" options, allowing their users to choose which certificate authority to appeal to. But this only piles more value into the social platform that issues the certificate - not only can Google shut down your email inbox, they can revoke your ability to log in to every website you used their identity provider to get into.

In every case, certificate issuance results in an asymmetrical power dynamic, where the issuer is able to exert significant control over the certificate holder, even in areas unrelated to the original pretext for the relationship between parties.

Self-Certification

But what if we could reverse this power dynamic? What if individuals could issue their own certificates and force institutions to accept them?

Ron Swanson's counterexample notwithstanding, there's a reason I can't simply write myself a parking permit and slip it under the windshield wiper. Questions about voluntary submission to legitimate authorities aside, the fact is that we don't have the power to act without impunity - just like any other certificate authority, we have to prove our claims either by the exercise of raw power or by appeal to a higher authority.

So the question becomes: which higher authority can we appeal to in order to issue our own certificates within a given system of identity?

The obvious answer here is to go straight to the top and ask God himself to back our claim to self-sovereignty. However, that's not how he normally works - there's a reason they call direct acts of God "miracles". In fact, Romans 13:1 explicitly says that "the authorities that exist have been appointed by God". God has structured the universe in such a way that we must appeal to the deputies he has put in place to govern various parts of the world.

Another tempting appeal might be to nature - i.e. the material world. This is the realm in which we most frequently have the experience of "self-authenticating" identities. For example, a gold coin can be authenticated by biting it or by burning it with acid. If it quacks like a duck, walks like a duck, and looks like a duck, then it probably is a duck.

In most cases however, the ability to authenticate using physical claims depends on physical access, and so appeals to physical reality have major limitations when it comes to the digital world. Captchas, selfies and other similar tricks are often used to bridge the physical world into the digital, but these are increasingly easy to forge, and hard to verify.

There are exceptions to this rule - an example of self-certification that makes its appeal to the physical world is that of a signature. Signatures are hard to forge - an incredible amount of data is encoded in physical signatures, from strength, to illnesses, to upbringing, to personality. These can even be scanned and used within the digital world as well. Even today, most contracts are sealed with some simulacrum of a physical signature. Of course, this custom is quickly becoming a mere historical curiosity, since the very act of digitizing a signature makes it trivially forgeable.

So: transcendent reality is too remote to subtantiate our claims, and the material world is too limited to work within the world of information. There is another aspect of reality remaining that we might appeal to: information itself.

Physical signatures authenticate physical identities by encoding unique physical data into an easily recognizable artifact. To transpose this idea to the realm of information, a "digital signature" might authenticate "digital identities" by encoding unique "digital data" into an easily recognizable artifact.

Unfortunately, in the digital world we have the additional challenge that the artifact itself can be copied, undermining any claim to legitimacy. We need something that can be easily verified and unforgeable.

Digital Signatures

In fact such a thing does exist, but calling it a "digital signature" obscures more than it reveals. We might just as well call the thing we're looking for a "digital fingerprint", or a "digital electroencephalogram". Just keep that in mind as we work our way towards defining the term - we are not looking for something looks like a physical signature, but for something that does the same thing as a physical signature, in that it allows us to issue ourselves a credential that must be accepted by others by encoding privileged information into a recognizable, unforgeable artifact.

With that, let's get into the weeds.

An important idea in computer science is that of a "function". A function is a sort of information machine that converts data from one form to another. One example is the idea of "incrementing" a number. If you increment 1, you get 2. If you increment 2, you get 3. Incrementing can be reversed, by creating a complementary function that instead subtracts 1 from a number.

A "one-way function" is a function that can't be reversed. A good example of a one-way function is integer rounding. If you round a number and get 5, what number did you begin with? It's impossible to know - 5.1, 4.81, 5.332794, in fact an infinite number of numbers can be rounded to the number 5. These numbers can also be infinitely long - for example rounding PI to the nearest integer results in the number 3.

A real-life example of a useful one-way function is sha256. This function is a member of a family of one-way functions called "hash functions". You can feed as much data as you like into sha256, and you will always get 256 bits of information out. Hash functions are especially useful because collisions between outputs are very rare - even if you change a single bit in a huge pile of data, you're almost certainly going to get a different output.

Taking this a step further, there is a whole family of cryptographic one-way "trapdoor" functions that act similarly to hash functions, but which maintain a specific mathematical relationship between the input and the output which allows the input/output pair to be used in a variety of useful applications. For example, in Elliptic Curve Cryptography, scalar multiplication on an elliptic curve is used to derive the output.

"Ok", you say, "that's all completely clear and lucidly explained" (thank you). "But what goes into the function?" You might expect that because of our analogy to physical signatures we would have to gather an incredible amount of digital information to cram into our cryptographic trapdoor function, mashing together bank statements, a record of our heartbeat, brain waves and cellular respiration. Well, we could do it that way (maybe), but there's actually a much simpler solution.

Let's play a quick game. What number am I thinking of? Wrong, it's 82,749,283,929,834. Good guess though.

The reason we use signatures to authenticate our identity in the physical world is not because they're backed by a lot of implicit physical information, but because they're hard to forge and easy to validate. Even so, there is a lot of variation in a single person's signature, even from one moment to the next.

Trapdoor functions solve the validation problem - it's trivially simple to compare one 256-bit number to another. And randomness solves the problem of forgeability.

Now, randomness (A.K.A. "entropy") is actually kind of hard to generate. Random numbers that don't have enough "noise" in them are known as "pseudo-random numbers", and are weirdly easy to guess. This is why Cloudflare uses a video stream of their giant wall of lava lamps to feed the random number generator that powers their CDN. For our purposes though, we can just imagine that our random numbers come from rolling a bunch of dice.

To recap, we can get a digital equivalent of a physical signature (or fingerprint, etc) by 1. coming up with a random number, and 2. feeding it into our chosen trapdoor function. The random number is called the "private" part. The output of the trapdoor function is called the "public" part. These two halves are often called "keys", hence the terms "public key" and "private key".

And now we come full circle - remember about 37 years ago when I introduced the term "cryptographic identity"? Well, we've finally arrived at the point where I explain what that actually is.

A "cryptographic identity" is identified by a public key, and authenticated by the ability to prove that you know the private key.

Notice that I didn't say "authenticated by the private key". If you had to reveal the private key in order to prove you know it, you could only authenticate a public key once without losing exclusive control of the key. But cryptographic identities can be authenticated any number of times because the certification is an algorithm that only someone who knows the private key can execute.

This is the super power that trapdoor functions have that hash functions don't. Within certain cryptosystems, it is possible to mix additional data with your private key to get yet another number in such a way that someone else who only knows the public key can prove that you know the private key.

For example, if my secret number is 12, and someone tells me the number 37, I can "combine" the two by adding them together and returning the number 49. This "proves" that my secret number is 12. Of course, addition is not a trapdoor function, so it's trivially easy to reverse, which is why cryptography is its own field of knowledge.

What's it for?

If I haven't completely lost you yet, you might be wondering why this matters. Who cares if I can prove that I made up a random number?

To answer this, let's consider a simple example: that of public social media posts.

Most social media platforms function by issuing credentials and verifying them based on their internal database. When you log in to your Twitter (ok, fine, X) account, you provide X with a phone number (or email) and password. X compares these records to the ones stored in the database when you created your account, and if they match they let you "log in" by issuing yet another credential, called a "session key".

Next, when you "say" something on X, you pass along your session key and your tweet to X's servers. They check that the session key is legit, and if it is they associate your tweet with your account's identity. Later, when someone wants to see the tweet, X vouches for the fact that you created it by saying "trust me" and displaying your name next to the tweet.

In other words, X creates and controls your identity, but they let you use it as long as you can prove that you know the secret that you agreed on when you registered (by giving it to them every time).

Now pretend that X gets bought by someone even more evil than Elon Musk (if such a thing can be imagined). The new owner now has the ability to control your identity, potentially making it say things that you didn't actually say. Someone could be completely banned from the platform, but their account could be made to continue saying whatever the owner of the platform wanted.

In reality, such a breach of trust would quickly result in a complete loss of credibility for the platform, which is why this kind of thing doesn't happen (at least, not that we know of).

But there are other ways of exploiting this system, most notably by censoring speech. As often happens, platforms are able to confiscate user identities, leaving the tenant no recourse except to appeal to the platform itself (or the government, but that doesn't seem to happen for some reason - probably due to some legalese in social platforms' terms of use). The user has to start completely from scratch, either on the same platform or another.

Now suppose that when you signed up for X instead of simply telling X your password you made up a random number and provided a cryptographic proof to X along with your public key. When you're ready to tweet (there's no need to issue a session key, or even to store your public key in their database) you would again prove your ownership of that key with a new piece of data. X could then publish that tweet or not, along with the same proof you provided that it really came from you.

What X can't do in this system is pretend you said something you didn't, because they don't know your private key.

X also wouldn't be able to deplatform you as effectively either. While they could choose to ban you from their website and refuse to serve your tweets, they don't control your identity. There's nothing they can do to prevent you from re-using it on another platform. Plus, if the system was set up in such a way that other users followed your key instead of an ID made up by X, you could switch platforms and keep your followers. In the same way, it would also be possible to keep a copy of all your tweets in your own database, since their authenticity is determined by your digital signature, not X's "because I say so".

This new power is not just limited to social media either. Here are some other examples of ways that self-issued cryptographic identites transform the power dynamic inherent in digital platforms:

• Banks sometimes freeze accounts or confiscate funds. If your money was stored in a system based on self-issued cryptographic keys rather than custodians, banks would not be able to keep you from accessing or moving your funds. This system exists, and it's called bitcoin.
• Identity theft happens when your identifying information is stolen and used to take out a loan in your name, and without your consent. The reason this is so common is because your credentials are not cryptographic - your name, address, and social security number can only be authenticated by being shared, and they are shared so often and with so many counterparties that they frequently end up in data breaches. If credit checks were authenticated by self-issued cryptographic keys, identity theft would cease to exist (unless your private key itself got stolen).
• Cryptographic keys allow credential issuers to protect their subjects' privacy better too. Instead of showing your ID (including your home address, birth date, height, weight, etc), the DMV could sign a message asserting that the holder of a given public key indeed over 21. The liquor store could then validate that claim, and your ownership of the named key, without knowing anything more about you. Zero-knowledge proofs take this a step further.

In each of these cases, the interests of the property owner, loan seeker, or customer are elevated over the interests of those who might seek to control their assets, exploit their hard work, or surveil their activity. Just as with personal privacy, freedom of speech, and Second Amendment rights the individual case is rarely decisive, but in the aggregate realigned incentives can tip the scale in favor of freedom.

Objections

Now, there are some drawbacks to digital signatures. Systems that rely on digital signatures are frequently less forgiving of errors than their custodial counterparts, and many of their strengths have corresponding weaknesses. Part of this is because people haven't yet developed an intuition for how to use cryptographic identities, and the tools for managing them are still being designed. Other aspects can be mitigated through judicious use of keys fit to the problems they are being used to solve.

Below I'll articulate some of these concerns, and explore ways in which they might be mitigated over time.

Key Storage

Keeping secrets is hard. "A lie can travel halfway around the world before the truth can get its boots on", and the same goes for gossip. Key storage has become increasingly important as more of our lives move online, to the extent that password managers have become almost a requirement for keeping track of our digital lives. But even with good password management, credentials frequently end up for sale on the dark web as a consequence of poorly secured infrastructure.

Apart from the fact that all of this is an argument for cryptographic identities (since keys are shared with far fewer parties), it's also true that the danger of losing a cryptographic key is severe, especially if that key is used in multiple places. Instead of hackers stealing your Facebook password, they might end up with access to all your other social media accounts too!

Keys should be treated with the utmost care. Using password managers is a good start, but very valuable keys should be stored even more securely - for example in a hardware signing device. This is a hassle, and something additional to learn, but is an indispensable part of taking advantage of the benefits associated with cryptographic identity.

There are ways to lessen the impact of lost or stolen secrets, however. Lots of different techniques exist for structuring key systems in such a way that keys can be protected, invalidated, or limited. Here are a few:

• Hierarchical Deterministic Keys allow for the creation of a single root key from which multiple child keys can be generated. These keys are hard to link to the parent, which provides additional privacy, but this link can also be proven when necessary. One limitation is that the identity system has to be designed with HD keys in mind.
• Key Rotation allows keys to become expendable. Additional credentials might be attached to a key, allowing the holder to prove they have the right to rotate the key. Social attestations can help with the process as well if the key is embedded in a web of trust.
• Remote Signing is a technique for storing a key on one device, but using it on another. This might take the form of signing using a hardware wallet and transferring an SD card to your computer for broadcasting, or using a mobile app like Amber to manage sessions with different applications.
• Key sharding takes this to another level by breaking a single key into multiple pieces and storing them separately. A coordinator can then be used to collaboratively sign messages without sharing key material. This dramatically reduces the ability of an attacker to steal a complete key.

Multi-Factor Authentication

One method for helping users secure their accounts that is becoming increasingly common is "multi-factor authentication". Instead of just providing your email and password, platforms send a one-time use code to your phone number or email, or use "time-based one time passwords" which are stored in a password manager or on a hardware device.

Again, MFA is a solution to a problem inherent in account-based authentication which would not be nearly so prevalent in a cryptographic identity system. Still, theft of keys does happen, and so MFA would be an important improvement - if not for an extra layer of authentication, then as a basis for key rotation.

In a sense, MFA is already being researched - key shards is one way of creating multiple credentials from a single key. However, this doesn't address the issue of key rotation, especially when an identity is tied to the public key that corresponds to a given private key. There are two possible solutions to this problem:

• Introduce a naming system. This would allow identities to use a durable name, assigning it to different keys over time. The downside is that this would require the introduction of either centralized naming authorities (back to the old model), or a blockchain in order to solve Zooko's trilemma.
• Establish a chain of keys. This would require a given key to name a successor key in advance and self-invalidate, or some other process like social recovery to invalidate an old key and assign the identity to a new one. This also would significantly increase the complexity of validating messages and associating them with a given identity.

Both solutions are workable, but introduce a lot of complexity that could cause more trouble than it's worth, depending on the identity system we're talking about.

Surveillance

One of the nice qualities that systems based on cryptographic identities have is that digitally signed data can be passed through any number of untrusted systems and emerge intact. This ability to resist tampering makes it possible to broadcast signed data more widely than would otherwise be the case in a system that relies on a custodian to authenticate information.

The downside of this is that more untrusted systems have access to data. And if information is broadcast publicly, anyone can get access to it.

This problem is compounded by re-use of cryptographic identities across multiple contexts. A benefit of self-issued credentials is that it becomes possible to bring everything attached to your identity with you, including social context and attached credentials. This is convenient and can be quite powerful, but it also means that more context is attached to your activity, making it easier to infer information about you for advertising or surveillance purposes. This is dangerously close to the dystopian ideal of a "Digital ID".

The best way to deal with this risk is to consider identity re-use an option to be used when desirable, but to default to creating a new key for every identity you create. This is no worse than the status quo, and it makes room for the ability to link identities when desired.

Another possible approach to this problem is to avoid broadcasting signed data when possible. This could be done by obscuring your cryptographic identity when data is served from a database, or by encrypting your signed data in order to selectively share it with named counterparties.

Still, this is a real risk, and should be kept in mind when designing and using systems based on cryptographic identity. If you'd like to read more about this, please see this blog post.

Making Keys Usable

You might be tempted to look at that list of trade-offs and get the sense that cryptographic identity is not for mere mortals. Key management is hard, and footguns abound - but there is a way forward. With nostr, some new things are happening in the world of key management that have never really happened before.

Plenty of work over the last 30 years has gone into making key management tractable, but none have really been widely adopted. The reason for this is simple: network effect.

Many of these older key systems only applied the thinnest veneer of humanity over keys. But an identity is much richer than a mere label. Having a real name, social connections, and a corpus of work to attach to a key creates a system of keys that humans care about.

By bootstrapping key management within a social context, nostr ensures that the payoff of key management is worth the learning curve. Not only is social engagement a strong incentive to get off the ground, people already on the network are eager to help you get past any roadblocks you might face.

So if I could offer an action item: give nostr a try today. Whether you're in it for the people and their values, or you just want to experiment with cryptographic identity, nostr is a great place to start. For a quick introduction and to securely generate keys, visit njump.me.

Thanks for taking the time to read this post. I hope it's been helpful, and I can't wait to see you on nostr!

For the uninitiated, DVM stands for "data vending machines". They're actually sort of hard to describe — in technical terms they act more like clients, since they simply read events from and publish events to relays. In most cases though, these events are part of a request/response flow initiated by users elsewhere on the network. In practice, DVMs are bots, but there's also nothing to prevent the work they do from being powered by human interaction. They're an amazingly flexible tool for building anything from custom feeds, to transcription services, to chatbots, to protocol gateways.

The hype cycle for DVMs seems to have reached escape velocity in a way few other things have - zaps being the possible exception. But what exactly DVMs are remains something of a mystery to many nostr developers - and how to build one may as well be written on clay tablets.

This blog post is designed to address that - below is a soup to nuts (no nutzaps though) guide to building a DVM flow, both from the client and the server side.

Here's what we'll be covering:

• Discovering DVM metadata
• Basic request/response flow
• Implementing a minimal example

Let's get started!

DVM Metadata

First of all, it's helpful to know how DVMs are reified on the nostr network. While not strictly necessary, this can be useful for discovering DVMs and presenting them to users, and for targeting specific DVMs we want a response from.

NIP 89 goes into this in more detail, but the basic idea is that anyone can create a kind 31990 "application handler" event and publish it to the network with their own (or a dedicated) public key. This handler was originally intended to advertise clients, but has been re-purposed for DVM listings as well.

Here's what the "Fluffy Frens" handler looks like:

{
    "content": "{\"name\": \"Fluffy Frens\", \"picture\": \"https://image.nostr.build/f609311532c470f663e129510a76c9a1912ae9bc4aaaf058e5ba21cfb512c88e.jpg\", \"about\": \"I show recent notes about animals\", \"lud16\": \"discovery_content_fluffy@nostrdvm.com\", \"supportsEncryption\": true, \"acceptsNutZaps\": false, \"personalized\": false, \"amount\": \"free\", \"nip90Params\": {\"max_results\": {\"required\": false, \"values\": [], \"description\": \"The number of maximum results to return (default currently 100)\"}}}",
    "created_at": 1738874694,
    "id": "0aa8d1f19cfe17e00ce55ca86fea487c83be39a1813601f56f869abdfa776b3c",
    "kind": 31990,
    "pubkey": "7b7373dd58554ff4c0d28b401b9eae114bd92e30d872ae843b9a217375d66f9d",
    "sig": "22403a7996147da607cf215994ab3b893176e5302a44a245e9c0d91214e4c56fae40d2239dce58ea724114591e8f95caed2ba1a231d09a6cd06c9f0980e1abd5",
    "tags": [
        ["k", "5300"],
        ["d", "198650843898570c"]
    ]
}

This event is rendered in various clients using the kind-0-style metadata contained in the content field, allowing users to browse DVMs and pick one for their use case. If a user likes using a particular DVM, they might publish a kind 31989 "application recommendation", which other users can use to find DVMs that are in use within their network.

Note the k tag in the handler event - this allows DVMs to advertise support only for specific job types. It's also important to note that even though the spec doesn't cover relay selection, most clients use the publisher's kind 10002 event to find out where the DVM listens for events.

If this looks messy to you, you're right. See this PR for a proposal to split DVMs out into their own handler kind, give them a dedicated pubkey along with dedicated metadata and relay selections, and clean up the data model a bit.

DVM Flow

Now that we know what a DVM looks like, we can start to address how they work. My explanation below will elide some of the detail involved in NIP 90 for simplicity, so I encourage you to read the complete spec.

The basic DVM flow can be a little (very) confusing to work with, because in essence it's a request/response paradigm, but it has some additional wrinkles.

First of all, the broker for the request isn't abstracted away as is usually the case with request/response flows. Regular HTTP requests involve all kinds of work in the background - from resolving domain names to traversing routers, VPNs, and ISP infrastructure. But developers don't generally have to care about all these intermediaries.

With DVMs, on the other hand, the essential complexity of relay selection can't simply be ignored. DVMs often advertise their own relay selections, which should be used rather than a hard-coded or randomly chosen relay to ensure messages are delivered. The benefit of this is that DVMs can avoid censorship, just as users can, by choosing relays that are willing to broker their activity. DVMs can even select multiple relays to broker requests, which means that clients might receive multiple copies of the same response.

Secondly, the DVM request/response model is far more fluid than is usually the case with request/response flows. There are a set of standard practices, but the flow is flexible enough to admit exceptions to these conventions for special use cases. Here are some examples:

• Normally, clients p-tag the DVM they wish to address. But if a client isn't picky about where a response comes from, they may choose to send an open request to the network and collect responses from multiple DVMs simultaneously.
• Normally, a client creates a request before collecting responses using a subscription with an e-tag filter matching the request event. But clients may choose to skip the request step entirely and collect responses from the network that have already been created. This can be useful for computationally intensive tasks or common queries, where a single result can be re-used multiple times.
• Sometimes, a DVM may respond with a kind 7000 job status event to let clients know they're working on the request. This is particularly useful for longer-running tasks, where feedback is useful for building a responsive UX.
• There are also some details in the spec regarding monetization, parameterization, error codes, encryption, etc.

Example DVM implementation

For the purposes of this blog post, I'll keep things simple by illustrating the most common kind of DVM flow: a kind 5300 content discovery request, addressed to a particular DVM. If you're interested in other use cases, please visit data-vending-machines.org for additional documented kinds.

The basic flow looks like this:

• The DVM starts by listening for kind 5300 job requests on some relays it has selected and advertised via NIP 89 (more on that later)
• A client creates a request event of kind 5300, p-tagged with the DVM's pubkey and sends it to the DVM's relay selections.
• The DVM receives the event and processes it, issuing optional kind 7000 job status events, and eventually issuing a kind 6300 job result event (job result event kinds are always 1000 greater than the request's kind).
• The client listens to the same relays for a response, and when it comes through does whatever it wants to with it.

Here's a swimlane diagram of that flow:

To avoid massive code samples, I'm going to implement our DVM entirely using nak (backed by the power of the human mind).

The first step is to start our DVM listening for requests that it wants to respond to. Nak's default pubkey is 79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798, so we'll only listen for requests sent to nak.

nak req -k 5300 -t p=79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798

This gives us the following filter:

["REQ","nak",{"kinds":[5300],"#p":["79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798"]}]

To open a subscription to nos.lol and stream job requests, add --stream wss://nos.lol to the previous request and leave it running.

Next, open a new terminal window for our "client" and create a job request. In this case, there's nothing we need to provide as input, but we'll include it just for illustration. It's also good practice to include an expiration tag so we're not asking relays to keep our ephemeral requests forever.

nak event -k 5300 -t p=79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798 -t expiration=$(( $(date +%s) + 30 )) -t input=hello

Here's what comes out:

{
  "kind": 5300,
  "id": "0e419d0b3c5d29f86d2132a38ca29cdfb81a246e1a649cb2fe1b9ed6144ebe30",
  "pubkey": "79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798",
  "created_at": 1739407684,
  "tags": [
    ["p", "79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798"],
    ["expiration", "1739407683"],
    ["input", "hello"]
  ],
  "content": "",
  "sig": "560807548a75779a7a68c0ea73c6f097583e2807f4bb286c39931e99a4e377c0a64af664fa90f43e01ddd1de2e9405acd4e268f1bf3bc66f0ed5a866ea093966"
}

Now go ahead and publish this event by adding nos.lol to the end of your nak command. If all goes well, you should see your event pop up in your "dvm" subscription. If so, great! That's half of the flow.

Next, we'll want our client to start listening for kind 6300 responses to the request. In your "client" terminal window, run:

nak req -k 6300 -t e=<your-eventid-here> --stream nos.lol

Note that if you only want to accept responses from the specified DVM (a good policy in general to avoid spam) you would include a p tag here. I've omitted it for brevity. Also notice the k tag specifies the request kind plus 1000 - this is just a convention for what kinds requests and responses use.

Now, according to data-vending-machines.org, kind 5300 responses are supposed to put a JSON-encoded list of e-tags in the content field of the response. Weird, but ok. Stop the subscription in your "dvm" terminal and respond to your "client" with a recommendation to read my first note:

nak event -k 6300 -t e=a65665a3a4ca2c0d7b7582f4f0d073cd1c83741c25a07e98d49a43e46d258caf -c '[["e","214f5898a7b75b7f95d9e990b706758ea525fe86db54c1a28a0f418c357f9b08","wss://nos.lol/"]]' nos.lol

Here's the response event we're sending:

{
  "kind": 6300,
  "id": "bb5f38920cbca15d3c79021f7d0051e82337254a84c56e0f4182578e4025232e",
  "pubkey": "79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798",
  "created_at": 1739408411,
  "tags": [
    ["e", "a65665a3a4ca2c0d7b7582f4f0d073cd1c83741c25a07e98d49a43e46d258caf"]
  ],
  "content": "[[\"e\",\"214f5898a7b75b7f95d9e990b706758ea525fe86db54c1a28a0f418c357f9b08\",\"wss://nos.lol/\"]]",
  "sig": "a0fe2c3419c5c54cf2a6d9a2a5726b2a5b766d3c9e55d55568140979354003aacb038e90bdead43becf5956faa54e3b60ff18c0ea4d8e7dfdf0c8dd97fb24ff9"
}

Notice the e tag targets our original request.

This should result in the job result event showing up in our "client" terminal. Success!

If something isn't working, I've also create a video of the full process with some commentary which you can find here.

Note that in practice, DVMs can be much more picky about the requests they will respond to, due to implementations failing to follow Postel's law. Hopefully that will improve over time. For now, here are a few resources that are useful when working with or developing DVMs:

• dvmdash
• data-vending-machines.org
• noogle
• nostrdvm

Conclusion

I started this post by hinting that DVMs might be as fundamental as relays are to making nostr work. But (apart from the fact that we'd end up with an acronym like DVMNOSTRZ+*, which would only exascerbate the pronounciation wars (if such a thing were possible)), that's not exactly true.

DVMs have emerged as a central paradigm in the nostr world because they're a generalization of a design pattern unique to nostr's architecture - but which exists in many other places, including NIP 46 signer flows and NIP 47 wallet connect. Each of these sub-protocols works by using relays as neutral brokers for requests in order to avoid coupling services to web addresses.

This approach has all kinds of neat benefits, not least of which is allowing service providers to host their software without having to accept incoming TCP connections. But it's really an emergent property of relays, which not only are useful for brokering communication between users (aka storing events), but also brokering communication between machines.

The possibilities of this architecture have only started to emerge, so be on the lookout for new applications, and don't be afraid to experiment - just please, don't serialize json inside json 🤦‍♂️

The result of the experiment was inconclusive, but as a side effect it produced a great case study in converting a NINO into a Real Nostr App.

Introducing Roz

Roz, a friendly notary for nostr events.

To use it, simply publish an event to relay.damus.io or nos.lol, and roz will make note of it. To find out when roz first saw a given event, just ask:

curl https://roz.coracle.social/notary/cb429632ae22557d677a11149b2d0ccd72a1cf66ac55da30e3534ed1a492765d

This will return a JSON payload with a seen key indicating when roz first saw the event. How (and whether) you use this is up to you!

De-NINO-fying roz

Roz is just a proof of concept, so don't rely on it being there forever. And anyway, roz is a NINO, since it provides value to nostr (potentially), but doesn't really do things in a nostr-native way. It also hard-codes its relays, and certainly doesn't use the outbox model or sign events. But that's ok, it's a proof of concept.

A much better way to do this would be to modify roz to properly leverage nostr's capabilities, namely:

• Use nostr-native data formats (i.e., draft a new kind)
• Use relays instead of proprietary servers for data storage
• Leverage nostr identities and signatures to decouple trust from storage, and allow trusted attestations to be discovered

Luckily, this is not hard at all. In fact, I've gone ahead and drafted a PR to the NIPs repo that adds timestamp annotations to NIP 03, as an alternative to OpenTimestamps. The trade-off is that while user attestations are far less reliable than OTS proofs, they're much easier to verify, and can reach a pretty high level of reliability by combining multiple attestation sources with other forms of reputation.

In other words, instead of going nuclear and embedding your attestations into The Time Chain, you can simply ask 5-10 relays or people you trust for their attestations for a given event.

This PR isn't terribly important on its own, but it does remove one small barrier between us and trusted key rotation events (or other types of event that require establishing a verifiable chain of causality).

In this post, I will be committing the "no true Scotsman" fallacy, in which I start with the most liberal definition I can come up with, and gradually refine it until all that is left is the purest, gleamingest, most imaginary and unattainable nostr app imaginable. As I write this, I wonder if anything built yet will actually qualify. In any case, here we go.

It uses nostr

The lowest bar for what a "nostr app" might be is an app ("application" - i.e. software, not necessarily a native app of any kind) that has some nostr-specific code in it, but which doesn't take any advantage of what makes nostr distinctive as a protocol.

Examples might include a scraper of some kind which fulfills its charter by fetching data from relays (regardless of whether it validates or retains signatures). Another might be a regular web 2.0 app which provides an option to "log in with nostr" by requesting and storing the user's public key.

In either case, the fact that nostr is involved is entirely neutral. A scraper can scrape html, pdfs, jsonl, whatever data source - nostr relays are just another target. Likewise, a user's key in this scenario is treated merely as an opaque identifier, with no appreciation for the super powers it brings along.

In most cases, this kind of app only exists as a marketing ploy, or less cynically, because it wants to get in on the hype of being a "nostr app", without the developer quite understanding what that means, or having the budget to execute properly on the claim.

It leverages nostr

Some of you might be wondering, "isn't 'leverage' a synonym for 'use'?" And you would be right, but for one connotative difference. It's possible to "use" something improperly, but by definition leverage gives you a mechanical advantage that you wouldn't otherwise have. This is the second category of "nostr app".

This kind of app gets some benefit out of the nostr protocol and network, but in an entirely selfish fashion. The intention of this kind of app is not to augment the nostr network, but to augment its own UX by borrowing some nifty thing from the protocol without really contributing anything back.

Some examples might include:

• Using nostr signers to encrypt or sign data, and then store that data on a proprietary server.
• Using nostr relays as a kind of low-code backend, but using proprietary event payloads.
• Using nostr event kinds to represent data (why), but not leveraging the trustlessness that buys you.

An application in this category might even communicate to its users via nostr DMs - but this doesn't make it a "nostr app" any more than a website that emails you hot deals on herbal supplements is an "email app". These apps are purely parasitic on the nostr ecosystem.

In the long-term, that's not necessarily a bad thing. Email's ubiquity is self-reinforcing. But in the short term, this kind of "nostr app" can actually do damage to nostr's reputation by over-promising and under-delivering.

It complements nostr

Next up, we have apps that get some benefit out of nostr as above, but give back by providing a unique value proposition to nostr users as nostr users. This is a bit of a fine distinction, but for me this category is for apps which focus on solving problems that nostr isn't good at solving, leaving the nostr integration in a secondary or supporting role.

One example of this kind of app was Mutiny (RIP), which not only allowed users to sign in with nostr, but also pulled those users' social graphs so that users could send money to people they knew and trusted. Mutiny was doing a great job of leveraging nostr, as well as providing value to users with nostr identities - but it was still primarily a bitcoin wallet, not a "nostr app" in the purest sense.

Other examples are things like Nostr Nests and Zap.stream, whose core value proposition is streaming video or audio content. Both make great use of nostr identities, data formats, and relays, but they're primarily streaming apps. A good litmus test for things like this is: if you got rid of nostr, would it be the same product (even if inferior in certain ways)?

A similar category is infrastructure providers that benefit nostr by their existence (and may in fact be targeted explicitly at nostr users), but do things in a centralized, old-web way; for example: media hosts, DNS registrars, hosting providers, and CDNs.

To be clear here, I'm not casting aspersions (I don't even know what those are, or where to buy them). All the apps mentioned above use nostr to great effect, and are a real benefit to nostr users. But they are not True Scotsmen.

It embodies nostr

Ok, here we go. This is the crème de la crème, the top du top, the meilleur du meilleur, the bee's knees. The purest, holiest, most chaste category of nostr app out there. The apps which are, indeed, nostr indigitate.

This category of nostr app (see, no quotes this time) can be defined by the converse of the previous category. If nostr was removed from this type of application, would it be impossible to create the same product?

To tease this apart a bit, apps that leverage the technical aspects of nostr are dependent on nostr the protocol, while apps that benefit nostr exclusively via network effect are integrated into nostr the network. An app that does both things is working in symbiosis with nostr as a whole.

An app that embraces both nostr's protocol and its network becomes an organic extension of every other nostr app out there, multiplying both its competitive moat and its contribution to the ecosystem:

• In contrast to apps that only borrow from nostr on the technical level but continue to operate in their own silos, an application integrated into the nostr network comes pre-packaged with existing users, and is able to provide more value to those users because of other nostr products. On nostr, it's a good thing to advertise your competitors.
• In contrast to apps that only market themselves to nostr users without building out a deep integration on the protocol level, a deeply integrated app becomes an asset to every other nostr app by becoming an organic extension of them through interoperability. This results in increased traffic to the app as other developers and users refer people to it instead of solving their problem on their own. This is the "micro-apps" utopia we've all been waiting for.

Credible exit doesn't matter if there aren't alternative services. Interoperability is pointless if other applications don't offer something your app doesn't. Marketing to nostr users doesn't matter if you don't augment their agency as nostr users.

If I had to choose a single NIP that represents the mindset behind this kind of app, it would be NIP 89 A.K.A. "Recommended Application Handlers", which states:

Nostr's discoverability and transparent event interaction is one of its most interesting/novel mechanics. This NIP provides a simple way for clients to discover applications that handle events of a specific kind to ensure smooth cross-client and cross-kind interactions.

These handlers are the glue that holds nostr apps together. A single event, signed by the developer of an application (or by the application's own account) tells anyone who wants to know 1. what event kinds the app supports, 2. how to link to the app (if it's a client), and (if the pubkey also publishes a kind 10002), 3. which relays the app prefers.

As a sidenote, NIP 89 is currently focused more on clients, leaving DVMs, relays, signers, etc somewhat out in the cold. Updating 89 to include tailored listings for each kind of supporting app would be a huge improvement to the protocol. This, plus a good front end for navigating these listings (sorry nostrapp.link, close but no cigar) would obviate the evil centralized websites that curate apps based on arbitrary criteria.

Examples of this kind of app obviously include many kind 1 clients, as well as clients that attempt to bring the benefits of the nostr protocol and network to new use cases - whether long form content, video, image posts, music, emojis, recipes, project management, or any other "content type".

To drill down into one example, let's think for a moment about forms. What's so great about a forms app that is built on nostr? Well,

• There is a spec for forms and responses, which means that...
• Multiple clients can implement the same data format, allowing for credible exit and user choice, even of...
• Other products not focused on forms, which can still view, respond to, or embed forms, and which can send their users via NIP 89 to a client that does...
• Cryptographically sign forms and responses, which means they are self-authenticating and can be sent to...
• Multiple relays, which reduces the amount of trust necessary to be confident results haven't been deliberately "lost".

Show me a forms product that does all of those things, and isn't built on nostr. You can't, because it doesn't exist. Meanwhile, there are plenty of image hosts with APIs, streaming services, and bitcoin wallets which have basically the same levels of censorship resistance, interoperability, and network effect as if they weren't built on nostr.

It supports nostr

Notice I haven't said anything about whether relays, signers, blossom servers, software libraries, DVMs, and the accumulated addenda of the nostr ecosystem are nostr apps. Well, they are (usually).

This is the category of nostr app that gets none of the credit for doing all of the work. There's no question that they qualify as beautiful nostrcorns, because their value propositions are entirely meaningless outside of the context of nostr. Who needs a signer if you don't have a cryptographic identity you need to protect? DVMs are literally impossible to use without relays. How are you going to find the blossom server that will serve a given hash if you don't know which servers the publishing user has selected to store their content?

In addition to being entirely contextualized by nostr architecture, this type of nostr app is valuable because it does things "the nostr way". By that I mean that they don't simply try to replicate existing internet functionality into a nostr context; instead, they create entirely new ways of putting the basic building blocks of the internet back together.

A great example of this is how Nostr Connect, Nostr Wallet Connect, and DVMs all use relays as brokers, which allows service providers to avoid having to accept incoming network connections. This opens up really interesting possibilities all on its own.

So while I might hesitate to call many of these things "apps", they are certainly "nostr".

Appendix: it smells like a NINO

So, let's say you've created an app, but when you show it to people they politely smile, nod, and call it a NINO (Nostr In Name Only). What's a hacker to do? Well, here's your handy-dandy guide on how to wash that NINO stench off and Become a Nostr.

You app might be a NINO if:

• There's no NIP for your data format (or you're abusing NIP 78, 32, etc by inventing a sub-protocol inside an existing event kind)
• There's a NIP, but no one knows about it because it's in a text file on your hard drive (or buried in your project's repository)
• Your NIP imposes an incompatible/centralized/legacy web paradigm onto nostr
• Your NIP relies on trusted third (or first) parties
• There's only one implementation of your NIP (yours)
• Your core value proposition doesn't depend on relays, events, or nostr identities
• One or more relay urls are hard-coded into the source code
• Your app depends on a specific relay implementation to work (ahem, relay29)
• You don't validate event signatures
• You don't publish events to relays you don't control
• You don't read events from relays you don't control
• You use legacy web services to solve problems, rather than nostr-native solutions
• You use nostr-native solutions, but you've hardcoded their pubkeys or URLs into your app
• You don't use NIP 89 to discover clients and services
• You haven't published a NIP 89 listing for your app
• You don't leverage your users' web of trust for filtering out spam
• You don't respect your users' mute lists
• You try to "own" your users' data

Now let me just re-iterate - it's ok to be a NINO. We need NINOs, because nostr can't (and shouldn't) tackle every problem. You just need to decide whether your app, as a NINO, is actually contributing to the nostr ecosystem, or whether you're just using buzzwords to whitewash a legacy web software product.

If you're in the former camp, great! If you're in the latter, what are you waiting for? Only you can fix your NINO problem. And there are lots of ways to do this, depending on your own unique situation:

• Drop nostr support if it's not doing anyone any good. If you want to build a normal company and make some money, that's perfectly fine.
• Build out your nostr integration - start taking advantage of webs of trust, self-authenticating data, event handlers, etc.
• Work around the problem. Think you need a special relay feature for your app to work? Guess again. Consider encryption, AUTH, DVMs, or better data formats.
• Think your idea is a good one? Talk to other devs or open a PR to the nips repo. No one can adopt your NIP if they don't know about it.
• Keep going. It can sometimes be hard to distinguish a research project from a NINO. New ideas have to be built out before they can be fully appreciated.
• Listen to advice. Nostr developers are friendly and happy to help. If you're not sure why you're getting traction, ask!

I sincerely hope this article is useful for all of you out there in NINO land. Maybe this made you feel better about not passing the totally optional nostr app purity test. Or maybe it gave you some actionable next steps towards making a great NINON (Nostr In Not Only Name) app. In either case, GM and PV.

This of course means that I broke compatibility with all the NIP 29 group clients out there, but I did it for good reasons. In this post I hope to explain those reasons, and speculate on how best to move forward with "relay-based" groups on nostr.

To give you some quick context, NIP 29 and my approach to groups (which I'll dub "relays-as-groups" for clarity) are very similar, with a fundamental difference. Both have chat, join requests, group metadata, membership, moderation, etc. However, the basic unit of a NIP 29 group is a random group id string, while the basic unit of a Flotilla group is a relay itself.

I believe this design difference emerged in part because of what we were attempting to build. NIP 29 groups tend to be more telegram-like, where groups function more like chat rooms, and users join each one individually. On flotilla, groups function more like Discord servers, or Slack workspaces, and users join an entire group of chat rooms at once.

"Relays as groups" has four major advantages over "groups hosted by relays".

Decentralization

First, using relays as groups supports decentralization better than hosting user-managed groups on relays.

(To be clear here, I'm not referring to client-managed groups, which is a whole different approach that we've experimented with in the past, both with NIP 72 communities and with NIP 87 encrypted groups. Both NIP 29 and relays-as-groups solve many of the consistency problems associated with attempting to have a linear, synchronous conversation across multiple relays. While both alternatives have a story for migrating or mirroring a group, both rely heavily on the host relay to not censor user messages.)

I know what you're thinking. How can reducing the number of relays improve decentralization? Relays were originally introduced in order to create redundancy and spread trust across many actors, creating censorship resistance. This model was difficult for many bitcoiners to wrap their heads around, because it's a very different kind of decentralization than what a blockchain delivers.

Communities are an essentially different use case from a broadcast network where content is delivered based on author or recipient relay selections. Every message to a community would have to be delivered to all members of the community. Sending each message to all members' personal inbox relays just doesn't make sense; there has to be some other inbox for the community to work off of.

Additionally, online communities almost always have moderators and admins. This is even true of very public, open types of communities, like subreddits. The only type of community that doesn't have moderators is one that emerges naturally from social clustering. And even in that case there is loose consensus about who is in and who isn't, based on the actions/follows/mutes of the participants, whether these clusters are huge or tiny. Socially-emergent groups are served well by chat applications or broadcast networks.

But the subset of online communities that do prefer to confer moderator status on certain members are essentially centralized. In other words, centralization and control is a feature, not a bug.

Now, that doesn't mean there don't need to be considerations around credible exit and removing/adding moderators over time. But the fact is that moderator-led communities are always under the oversight of the moderators at any given time, even if the identity of those moderators changes and their power is limited.

What this implies is that decentralization for moderator-led groups looks very different from decentralization for a broadcast network. There is nothing at all wrong with giving moderators full control over the group's communications (qua the group; DMs and public broadcast content between group members should happen outside the group's infrastructure, just as people also exist outside the communities they are a part of). What is important is that no one has control over groups that they aren't nominally the admin of.

In concrete terms, what this all means is that community moderators should self-host their infrastructure. This is the same principle as motivates self-custody and home servers, but applied to communities. If community leaders manage their own relays, this means that no hosting company or relay admin can de-platform their community. Centralization of network infrastructure in this case aligns with the trust structure of the group.

Applying this to our group dilemma, it's easy to see that NIP 29 groups are more vulnerable to censorship or data harvesting attacks by malicious relay admins, since many unrelated groups might live on a single relay. In contrast, if you treat relays as groups themselves, every group is forced to live on a separate relay, spreading risk across more hosting providers.

Now, this doesn't necessarily mean that many "relays" aren't "virtual relays" managed by the same hosting provider. So I'll admit that even "relay-based" groups don't completely solve this problem. But I think it will tend to nudge community organizers toward thinking about community infrastructure in a more self-sovereign (or community-sovereign) way.

Investment in Relays

While both NIP 29 and relays-as-groups rely heavily on relays to implement the features that support each specification, there's an important difference between the feature sets. In NIP 29, relay support is specific only to groups, and isn't applicable to other use cases. In contrast, every protocol feature added to support the "relays as groups" can be re-purposed for other types of relays.

Take join requests for example. NIP 29's kind 9021 events allow users to request access to a group, and that's all. Kind 28934 join requests on the other hand allow users to request access to relays. Which in the relays-as-groups model means group access, but it also means custom feed access, inbox relay access, maybe even blossom server access. In fact, kind 28934 was originally proposed at the beginning of this year in order to support a different version of hosted groups, but remains as relevant as it ever was despite iteration on groups.

The orthogonality of features added to relays to any specific use case will long-term result in simpler specs, and more interesting relay-based use cases being possible. Join requests are only one example. The same is true of 1984-based moderation, the proposed LIMITS command, AUTH, NIP 11 relay metadata, etc.

We already have web of trust relays, feed relays, archival relays, and many more. Being able to request access to closed versions of these is useful. Being able to signal federation between multiple instances of these, run by different people, is useful. And of course, relay metadata, reports, and LIMITS are self-evidently useful for normal relays, since they pre-date Flotilla.

I've always said that relays are some of the coolest and most under-appreciated parts of nostr. This doesn't mean that we should add every possible feature to them, but features related to data curation and access control fit really well with what relays are good for. For more on the role of relays and what features should be added to them, see my nostrasia talk Functional Relays.

Declarative vs Imperative

A common paradigm in programming is that of declarative vs imperative programming. Imperative programming focuses on "how" to achieve a given result, leaving "what" the code is doing to be inferred by the programmer. Declarative programming instead focuses on the "what", and allows some underlying implementation to solve the how. A good balance between these paradigms (and knowing when to use one over the other) allows programmers to work faster, make fewer mistakes, and produce less code.

Another way to look at this is that a specification should contain as much ambiguity as possible, but without compromising the system attributes the specification is supposed to guarantee. It can get complex when figuring out what attributes are core to the specification, since sometimes the "how" does actually matter a lot.

However, nostr in particular falls pretty far along the "declarative" end of this spectrum because of its decentralized nature. The only person who can say anything with any authority is the person who signs an event. This event is a "declaration", and any effects it has are necessarily up to the relays, clients, and people interpreting the event. However, what others do with an event is an expectation that must be taken into account by the publisher, forming a feedback loop. This dialectic is what creates stability in the protocol.

In more concrete terms, no one can "tell" anyone else what they have to do by publishing an event like you might in a traditional, centralized RPC-type system. Any event whose semantics are a "command" rather than a "fact" or "request" is broken unless the counter party is fully committed to carrying out the command. An example of a "command" scenario on nostr is NIP 46 remote signing, in which the bunker is the agent of the user making the request. If the bunker implementation fails to carry out a valid command initiated by the user, its interpretation of that event is objectively incorrect.

NIP 29 applies this same paradigm to relays, particularly in the area of moderation, membership edits, and group metadata. In other words, there are several "commands" which instruct the relay to do something.

This isn't necessarily a bad thing, but it does increase the number of things the interface between the client and relay have to agree on. A regular relay may accept an add-user request, but then do nothing with it, violating the contract it has implicitly accepted with the user. The solution to this is feature detection, which is a whole other API to be specified and implemented.

My ideal solution to this problem is to shift the semantics of events away from "commands" to "facts" - in other words, to make the interface more declarative.

In fact, we already have an interface for moderation that works like this. Many clients support kind 1984 "report" events. Users sending these reports have no expectations about how they will be used. They are a "fact", a declaration of opinion with certain semantics. Other actors in the network may choose whether or not to pay attention to these.

This same model is easily applied to communities. Without having to implement any feature detection (either for the relay's implementation, or for the user's role on that relay), anyone can simply send a "report". This goes into the black hole of the relay, and may subsequently be ignored, broadcasted, or acted on.

The really nice thing about this model is that because there is no expectation for "how" reports are to be interpreted, any approach to moderation can be used depending on relay policy or client implementation. In NIP 29, if you issue a delete-event, it either happens or it doesn't and if it doesn't, you have to explain the failure to the user somehow.

In the relays-as-groups model, e-tagging an event in a kind 1984 requires no user feedback, and therefore it can be interpreted however the relay prefer. This can result in insta-banning, manual review, thresholds based on number of reporters, a leaky-bucket social score algorithm, shadow banning, temporary banning, soft-moderation by allowing clients to request reports and respond to them by changing user interface elements, or anything else you can think of.

The reason I think this is important is that community moderation is a very hard problem, and baking certain semantics into the specification can result in the complete failure of the spec. NIP 72 should be considered an example of what not to do. Some NIP 72 communities have survived due to the dedication of the moderators, but many more have failed because of the rigid moderation model. We should try not to make the same mistake again.

Conclusion

Now, having said all that, I think there is actually a lot of value to NIP 29. What finally clicked for me this week after releasing Flotilla is that the two approaches are actually complementary to one another. One of the most common feature requests I've already heard for flotilla is to have more complete support for rooms, which are currently implemented as not much more than hashtags. Better rooms (i.e., "nested groups") would require: authentication, membership, moderation, and pretty much everything else that exists for the top-level group.

As much as I believe the relays-as-groups approach is superior to NIP 29 for top-level groups, it doesn't make any sense to try to "nest" relays to create sub-groups. Something like NIP 29 is needed in order to fully support rooms anyway, so I think the convergence of the two approaches is all but inevitable. In fact, fiatjaf has already merged a PR which will allow me to use the same event kinds in flotilla as exist already in NIP 29 clients.

There are just a few more changes that are necessary in order for me to fully adopt NIP 29 in Flotilla:

• NIP 29 feature detection
• Opaque ids for unmanaged groups prevent unmanaged groups from having human-readable names
• We need a mechanism for building membership lists without relay support
• Better handling for 9021 group join requests

I've opened PRs for each of these (linked above). Hopefully we can work through these issues and combine our powers to become the Captain Planet of group implementations.

Just to state the obvious, and to kick off this survey of nostr key management options, let me just state that asking users to paste their nsec into your app is a bad idea. However good your intentions, this opens your users up to all kinds of attack vectors, including clipboard hijacking attacks, exposing keys to insecure communication channels, exposing keys to many different apps, supply chain attacks, XSS attacks, and yes, bugs that cause your software to send keys to analytics or error reporting backends.

The era of nsec-pasting is over.

I've committed to embracing the pain and removing nsec login from Coracle, and I encourage other devs to do the same. The sooner we treat key management with the urgency it deserves, the sooner we can come up with a secure and convenient key management solution.

As an aside, ncryptsec is a great innovation for securely transporting keys, but it still doesn't protect against exposure to apps that need to use keys. It has its place though; in fact I'm of the opinion that nsec and seed words should be deprecated, and support for them should be removed. Giving friendly names and human-readable representations to data that is essentially private is a really bad idea (unless you're memorizing your key). But I digress.

Signer Comparisons

Let's go through a few existing options for key management, and compare their relative merits. I've tried to list them in the order they appeared on the scene, which also helps to clarify the logic of how signers have evolved. Throughout, I will be focusing on what kinds of user experience each approach unlocks for non-technical users, since my goal is to build products that work for regular people.

Extension Signers

The first signer application (that I know of) was nos2x, by fiatjaf. As I understand it, this was a proof-of-concept of how users might protect their keys without releasing custody of them. And it works really well! In fact, even though there have been many forks and imitators, I still use nos2x when using nostr on my desktop browser.

Extension signers offer a great user experience, along a narrow happy path. Setting up a browser extension is a relatively familiar process for normal users, and once it's done you don't really have to think about it. In theory, extensions can also include their own onboarding process and key backup strategies as well, allowing users to get started in a single place. Plus, there's very little latency involved in making calls to the signer extension.

This positive experience breaks down quickly though once a user wants to use a desktop or mobile application. When this happens, users have to start over essentially from scratch. Nothing they did to set up the extension helps them move to another signer application.

While it's technically possible to use extension signers on mobile via e.g. the Kiwi browser, this doesn't work for native apps or apps installed as PWAs. Instead, you either have to revert to pasting keys, or use some other solution.

One slight permutation of extension signers is browser signers, like Spring. Instead of adding a signer to your browser, Spring allows you to install a browser that holds your keys and allows you to use any nostr web application. But this has all the same basic limitations that extension signers do.

Hardware Signers

Hardware signers came around shortly after extension signers. I'm not going to spend much time talking about them here, because although they're about as far along the spectrum towards security as you can go, they're also not very convenient. Non-technical users aren't going to onboard by buying (or building) a device which they have to connect to their desktop via USB whenever they want to sign a message. Hardware signers have their place, but onboarding isn't it.

The only hardware signer I'm aware of (although I'm sure I've heard of others) is from LNBits, and is usually used via a browser extension like horse. This of course means that it has all the same limitations that browser extensions have, and then some (although mobile and desktop apps would likely be able to find a way to talk directly to the signer).

Hosted Signers

Remote signers (aka "bunkers") use the Nostr Connect protocol (also known as NIP 46) for remote signing.

Hosted signers in particular are one example of a NIP 46 remote signer, which lives on "somebody else's computer". Because they use a legacy web architecture, they can be built to be very familiar and convenient to users. It's trivial to build a hosted signer that offers email/password login along with 2FA, password resets, session revokation, the whole shebang. But they have one fatal flaw, which is that they are custodial. This means that not only do users have to relinquish exclusive control over their keys, but hosted signers also can become a target for hackers.

Desktop Signers

Several projects exist which allow users to run their own bunker, on their own hardware. These include nostr clients like Gossip, as well as command-line utilities like nak. This approach is mostly an improvement over extension signers, because it widens the scope of applications that can conveniently access the signer from those that run in the browser to those that run on the desktop computer the signer lives on. The downside is that they have to communicate via relays, which either introduces latency or requires an additional component to be running locally.

While it's technically possible to use desktop signers to log in on other computers or mobile apps, I don't think that's going to be very practical for most people. Mobile apps by definition are more portable than regular computers. Anyone who wants to access their nostr account on more than one device will have to either set up separate solutions, or go with another kind of remote signer. This isn't a huge obstacle for people highly invested in nostr, but it's a significant amount of friction for a new user.

Mobile Signers

Mobile signers solve the problem introduced by desktop signers of not always having access to your signer (or of your signer not having access to you, due to being powered down or disconnected from the internet). Mobile devices are generally more available than desktop devices, and also have better push notifications. This means that users can approve signer requests from any device as easily as tapping a notification.

Mobile signers on Android can also upgrade their UX by taking advantage of NIP 55 to avoid the round trip to relays, reducing latency and making it possible to sign things offline. Amber has been a pioneer in this area, and other projects like Nostrum and Alby's nostr-signer have been prototyped.

To date, there unfortunately haven't been any signer applications released for iOS, which leaves the mobile signer story incomplete. In my opinion, this is probably the most promising solution for end users, although it's currently under-developed.

Web Signers

One interesting alternative that combines the benefits of hosted, desktop, and mobile wallets is nsec.app. This is a web application frontend which keeps keys in the browser, so that they are never shared with a third party. Using web push notifications and a healthy sprinkle of black magic, nsec.app is able to respond to signer requests by opening itself in a browser window.

This works generally pretty well for desktop web applications, less well on android, still less well for android PWAs, and (to my understanding) not at all on iOS. Artur from nostr.band is working on these problems using a variety of approaches, one of which is embedding nsec.app in an iframe and communicating using postMessage.

This approach also makes it possible to sync keys between your phone and desktop, simulating a hosted UX by making them accessible from either location by signing in to nsec.app. This is done by encrypting user keys and storing them on the nsec.app server. In theory this should be secure, but it's something to consider.

I'm cautiously optimistic about this approach. If successful, it would enable a single brand to exist on every platform, which is important to reduce unnecessary configuration and cognitive overhead for users.

Multisig Signers

Another experimental approach is multi-sig. Promenade is a project by fiatjaf exploring this possibility. This would allow users to split their keys across different custodians and require all (or some majority of them) to approve an event signature before it would be valid.

The downsides of this are an increase in complexity (more moving parts for users to deal with) and latency (more parties to coordinate with to sign events). I'm also not clear on whether encryption is possible using multi-signature keys. If not, that would preclude not only existing direct messages (which will hopefully end up on MLS eventually anyway), but also things like private lists, mutes, and application settings. I think multi-signature signers are promising, but are definitely a long-term project.

Self-Hosted Signers

Coming nearly full circle, self-hosted signers are a special case of hosted signers, but, you know, self-hosted. These signers might live on a home server like a Start9 and be accessible for signer request approvals via tor, or they might live on a server run by the user (or an Uncle Jim). This would be an extremely convenient approach for anyone willing to deal with the complexities of hosting the infrastructure.

A good candidate for NIP 46 support might be AlbyHub, which is already one of the easiest self-hosted wallets to set up and use. Adding signer suppport to AlbyHub would allow users to have their wallet and nostr keys stored in the same place, and accessible anywhere either via the web interface or via AlbyGo.

Omniplatform Signers

This leads me to, finally, "omniplatform" signers. This isn't really a new architecture, but a combination of several. User choice is great, but nostr has a very tight complexity budget when onboarding new users. If a brand can manage to get new users set up with a very simple but sub-optimal solution, then grow them into a more complete integration into the nostr ecosystem, that would be a huge win.

I think Alby has a great shot at doing this, if it's something they want to prioritize. Bitwarden would also be a great candidate, since they already have apps on every platform, as well as a self-hosted option (Vaultwarden). If users could start with a mobile app, and incrementally set up a browser extension, self-hosted vault, and hardware signer as needed, that I think would be an ideal path.

Nostr Connect: broken, but promising

If you can't tell from the above comparison, I'm partial to NIP 46 as the best, most flexible way to build high-quality user experiences. Remote key management means a reduction in moving keys, hosting keys, and software installation and administration. If we can get users to the point where their keys live in only two places (their password manager and their signer), we'll be doing good.

There are however many ways to implement NIP 46. Implementing all of them in a single client (or signer) would be burdensome for developers, and introduce a lot of UI complexity to users. Here's a quick survey of flows that currently exist.

Signer -> Client

The simplest way to connect a client and a bunker is for a user to explicitly authorize the connection by copying a bunker:// URL from their signer application to their client. This allows the bunker to generate and validate a secret embedded in the URL without the client having to do anything other than pass it along in the initial connect request.

This is a great UX for people who know what they're doing, but isn't at all friendly to newcomers. Someone signing in for the first time isn't going to know what a bunker link is, and even if they do they're immediately confronted with the problem of picking a signer, setting it up, and finding out where in that app they can obtain a bunker link. This can be marginally smoothed out using things like protocol handlers and QR codes, but these won't apply in all (or even most) cases.

Client -> Signer

The reverse flow is similar. This relies on the user to explicitly authorize the connection by copying a nostrconnect:// url from the client into the signer app. In technical terms, this requires one fewer step, since in NIP 46 the connection is always initiated by the client. In this case, the pasting of the URL replaces the connect request. The client listens for a response containing a client-generated secret embedded in the nostrconnect:// url. This step isn't currently supported by all signer apps, some of which return an ack instead. This can result in session hijacking, where an attacker can intercept signing requests (although they can't do anything that would require the user's key, like decrypting messages).

While at first glance nostrconnect seems functionally identical to bunker links, the UX has the potential to be much better. The reason for this has to do with how people use which devices, and where a client or signer application is most likely to be run. This requires making some assumptions, but in my mind the most common scenario is that a user will want to host their signer on their phone, since that is the device that is most universally available for authorizations (apart from an always-online hosted signer on the open internet). In other words, users generally have their phones with them when they're using their computer, but often don't have a desktop available when using their phone. This idea is validated by (for example) the prevalence of SMS-based 2FA, which assumes the presence of a phone.

Assuming the signer is on the user's phone, QR-scan flows for client authorization make a lot more sense if the client is the one generating the link, since they can simply scan a code generated on another device with their camera, or copy/paste or use a protocol handler for a client on the same device. In contrast, when using a bunker link users might find themselves in the awkward position of having to copy a link from their phone to their desktop. Whether this is done via QR code or by sending yourself a link via DM/text/email, it's an awkward flow most people aren't really prepared for.

Auto-Connect

Some enhancements have been made to the bunker flow which allow clients to send an initial connect request without asking the user to copy links between apps. These allow clients to do away with opaque magic strings entirely and provide the idealized "just one click" flow. However, after trying to make this flow work over the course of a couple weeks, I've come to the opinion that the additional complexity involved in automating the flow just isn't worth it.

There are a few variants of this "auto-connect" flow:

• Signer NIP-05: Signers can register a NIP 05 address for a user's pubkey on their domain, allowing users to enter their address rather than their pubkey on login. Unfortunately, this address has no relation to their actual NIP 05 address, which can result in a lot of confusion.
• User NIP-05: To solve this problem, fiatjaf has proposed a new version which allows users to enter their own NIP 05 in at login instead of the one provided by the signer. The client would then look up the user's 10046 event and follow the signer pubkey listed there.
• Nostrconnect handler: Signers may publish a NIP 89 handler which includes a handler url that clients can send nostrconnect urls to. This isn't currently specified anywhere, but it is supported by nsec.app. This bypasses the NIP 05 address requirement entirely, allowing users to simply pick a signer and click a button.

Each of these flows have their own strengths and weaknesses, but all of them share a dependency on some external source of truth for routing a user to the correct bunker.

In the first case, this is done by remembering the NIP 05 address assigned by the signer, which relies on DNS and on users to not forget which address they're using.

In the second case, this is done by relying on the user having done a significant amount of configuration (setting up a NIP 05, adding it to their kind 0, and having published a 10046 event) which may or may not exist. This forces clients to gracefully degrade to some alternative login method anyway, and adds UX friction since users have to choose which interface will work for them.

The final method bypasses the need for users to remember their NIP 05 address, but it does require either the client or the user to select a trusted signer. If poorly implemented, this could result in users choosing an untrustworthy signer on signup (risking their keys), or the wrong signer on login resulting in a broken session.

For all these reasons, I've opted to go with the vanilla bunker/nostrconnect flow, which allows me to display a simple interface to users. Presenting a QR code without comment assumes that users know what to do with it, but the benefit is that it makes explicit the signer selection step which the auto-connect flows try to paper over. This is actually a good thing, because instead of using heuristics like addresses or lists of signers presented by a client to make the decision, users can choose based on which app they actually have installed, which is a richer mnemonic device.

Making NIP 46 Work

The bottom line here is that while NIP 46 is the best baseline for signer support, it doesn't currently work very well at all. There are a variety of reasons for this:

• The specification itself isn't clear, and is constantly changing. This leads to incompatibilities between apps and signers (or explosive complexity in trying to handle every case).
• Extensions to the basic bunker flow (both in terms of signer implementation and signer discovery) are worth researching, but each one creates another dimension of possible incompatibility. Signers will be incentivized to support every possible login flow, creating complexity for users and increasing attack surface area. Clients will have to implement fallbacks to their preferred signup flows, again resulting in UX complexity.
• Clients don't currently deal well with latency. In order for NIP 46 to work smoothly, clients will have to implement better loading, debouncing, optimistic updates, publish status, and "undo". There are downsides to this, but many of these features endu up being built by mature software products anyway, so supporting these patterns may actually improve rather than degrade UX.
• There's currently no easy and secure way for users to store keys in a single signer which they can access anywhere. This means that users have to set up multiple bunkers depending where they're sitting, or resort to alternative login methods like NIP 07 or 55. These are great upgrades, since they reduce latency and bandwidth use, but shouldn't be required for new users to learn.
• There's no unified experience across platforms. If a user signs up on their desktop, how do they safely transfer their keys to their Android signer app? If they're given seed words, how can they import them as an nsec? Consensus on best practices would be an improvement, but I think only a unified UX across platforms for a single signer can really solve this.
• As nice as it might be to bypass app stores and built-in push notifications, shunning traditional platforms drastically increases the friction for users. To my knowledge, no signer app currently exists in traditional app stores, or supports built-in push notifications. If we want nostr to be accessible to non-technical folks, we can't ask them to start by downloading Obtanium or zap.store and a UnifiedPush distributor for their platform.

As I mentioned above, I don't think NIP 46 will ever be the only solution for signers. But I do think it's a great baseline on which to build a kind of "progressive enhancement" approach. For example, clients should support at least nostrconnect/bunker links, and encourage users once they've logged in to upgrade to NIP 55 or NIP 07 signers. Signers should exist in the mainstream app store and use native push notifications, with an option to install elsewhere or opt-in to UnifiedPush.

The goal here is to balance user experience and security. The number one rule for this is to reduce attack vectors for obtaining user keys. This points to (ideally) a single non-custodial signer, easily accessible to the user, and a simple protocol for using that signer from any app. Progressive enhancement is fine, but we should always be able to fall back to this baseline.

In order to do this, users must be able to articulate "what" they are looking for, and clients must be able to articulate "how" to find that thing. This "how" is divided into two parts: building a request that will match the desired content (very easy), and selecting a relay that is able to serve that content to the user requesting it (very very hard).

Why guessing isn't good enough

As a concrete example, let's say the user wants to find everyone in their "network" who is using a particular topic. The process would look something like this:

1. The user clicks the "network" tab and types in the topic they want to browse. This is the "what".
2. The client then translates the term "network" to a list of public keys using whatever definition they prefer (Follows? WoT? Grapevine?), and builds a filter that might look something like this: [{"authors": pubkeys, "#t": ["mytopic"]}]. Any relay will happily accept, understand, and respond to that filter.
3. The client then has to decide which relays it should send that filter to. This is the ??? stage of the outbox model, which immediately precedes:
4. Profit

It may not be immediately obvious why selecting the correct relays might be difficult. Most people post to relay.damus.io, and most people read from relay.damus.io, so in most cases you should be good, right?

This approach to relay selection has historically worked "well enough", but it depends on a flawed definition of success. If you only want to find 90% of the content that matches your query, using the top 10 relays will suffice. But nostr is intended to be censorship-resistant. What if those 10 hubs have banned a particular public key? Nostr clients should (at least in theory) be 100% successful in retrieving requested content. Even if someone only posts to their self-hosted relay, you should be able to find their notes if their account is set up properly.

A naive solution to fixing the FOMO

A 90% hit rate results in a feeling of flakiness, even if users aren't completely aware of what isn't working. Feeds will be incomplete, quoted notes will be missing, replies will be orphaned, user profiles won't load. The natural response to the FOMO this creates is for users to "try harder" by adding more relays.

On the read side, this means clients open more connections, resulting in much higher data transfer requirements, with massively diminishing returns, since there's no reason to expect that a randomly chosen relay will have a substantially different data set.

One the publish side, this means that clients end up publishing more copies of their data to more relays. This approach has been automated in the past by services like Blastr, which don't store a copy of events published to the relay, but instead forward events to the top 300 relays in the network. This results in a two-orders-of-magnitude increase in storage required, and only makes the read side of the problem worse, since it reduces the uniqueness of the data set each relay stores. This in turn means that more duplicates are retrieved when querying relays.

Both halves of this approach are equivalent to guessing. On the read side, users are guessing which relays will have any arbitrary content they might ask for in the future. On the write side, users are guessing which relays other people might use to find their notes. It is a brute-force method for finding content.

Randomness results in centralization

In theory, random relay selection would result in a perfect distribution of content across all relays in the network. But in practice, this method of selection isn't random at all, but is strongly influenced by user bias in what constitutes a "good" relay. While some users may check nostr.watch for ping times, geographical proximity, or uptime, most will choose relays based on familiar names or other people's recommendations.

In either case, these biases are entirely orthogonal to achieving a higher content retrieval hit rate, except when bias in relay selection results in clustering — i.e., centralization. In other words, the kind of randomness exhibited by users when selecting relays actually results in pretty much everyone picking the same few relays. We see this same effect when people try to come up with passwords or seed phrases — human-provided randomness is anything but random.

Clustering improves the hit rate when requesting events (slightly), but it results in nearly as much centralization as if only a single relay was used — and a lot more duplicate events.

Something (anything) other than randomness

In early 2023, Mike Dilger introduced NIP 65 (now known as the "Outbox Model") with a problem statement in the spirit of the original description of nostr: "Nostr should scale better. People should be able to find what they want."

Historical note: NIP 65 was formerly known as the "Gossip Model", derived from the name of Mike's desktop nostr client, called "Gossip". This unfortunately created a lot of confusion, since gossip protocols work very differently from how nostr tends to work, hence the re-brand.

Before NIP 65, an informal standard existed in which kind 3 user contact lists also included a list of relays that clients could use as something similar to Mastodon's "home servers". This list included the option to only read or write from a given relay. Unfortunately, it wasn't really clear what the semantics of this relay list were, so different clients handled them differently (and many clients ignored them). Usually this amounted to user-provided static relay configurations, which resulted in the naive relay selection approach described above.

NIP 65 used a very similar format (a list of relay urls with optional "read" or "write" directives), but with a very important semantic difference: relays listed in a user's kind 10002 were intended to "advertise to others, not for configuring one's client." In other words, these relay selections were intended as a signal to other users that they should use certain relays when attempting to communicate with the author of the relay list.

I highly recommend reading the entire NIP, which is very short and easy to read. But the mechanics of the spec are very simple:

When seeking events from a user, Clients SHOULD use the WRITE relays of the user's kind:10002.

When seeking events about a user, where the user was tagged, Clients SHOULD use the READ relays of the user's kind:10002.

When broadcasting an event, Clients SHOULD:

• Broadcast the event to the WRITE relays of the author
• Broadcast the event to all READ relays of each tagged user

For the first time, we had a way to differentiate relays in terms of what content could be found where.

When looking for a note by a particular user, a client could now look up the author's write relays according to their kind 10002 event, and send its query there. The result is a much higher hit rate with much lower data transfer requirements, and fewer connections per query.

Making Outbox Work

There are of course some assumptions required to make this work.

First, the user must know which author they're looking for. This isn't always true when looking up a quote or parent note, but context and pubkey hints solve this difficulty in most cases.

The author must also publish a kind 10002 event. This may not always be the case, but clients should prompt users to set up their relay list correctly. This isn't really a flaw in the Outbox Model, just in implementations of it.

Additionally, the user's client must be able to find the author's kind 10002 event. This is the "bootstrapping" phase of the Outbox Model, during which the mechanisms the system provides for finding events aren't available. This requires us to fall back to randomly guessing which relays have the content we're looking for, which as we saw above doesn't work very well.

Other than guessing, there are a few different ways a client might find the relay selection event in question, each of which is applicable in different circumstances. In most cases, using one of a handful of indexer relays like purplepag.es or relay.nostr.band is a simple and efficient way to find user profiles and relay selections.

However, if an author's content has been aggressively purged from these indexers due to censorship, they obviously can't be relied upon. Even though the author in question hasn't been deplatformed from nostr itself (since he can always self-host a publicly accessible relay to store his content), he has been effectively shadow-banned.

To get around this, relay selections have to be communicated in some other way. Nostr has a few different mechanisms for this:

• If the author's NIP 05 address is known and properly configured (it may not be), clients can look up the author's NIP 05 endpoint to find some reasonable relay hints. Unfortunately, these are often neglected, and usually custodial, so they can run into the same problems.
• If the author's pubkey is found in another signed event found on nostr, relay hints can be a way to propagate relay selections through the network. This relies on implementations picking reliable relay hints which can be difficult, and hints do tend to become less reliable over time. However, this strategy is very effective in resisting censorship because it makes banning viral — if a relay wants to completely purge a particular pubkey from their database, they have to purge every event that references it, since events are tamper-proof.
• In extremis, relay recommendations can always be communicated out-of-band. This can be done using manual input, QR codes, DHTs, jsonl torrents full of kind 10002 events, or any other mechanism client developers choose to resort to.

Another, more technical assumption is that any given query can be fulfilled by few enough relays that a client can actually make all the connections needed, without running into resource limits. If you're trying to request content from 10,000 users across 1,000 relays, you're going to have a bad time. This was pointed out to me by Mazin of nostr.wine. He makes a good point, and it's definitely something to keep in mind. There are some mitigating factors though.

The first is that the current topology of the network probably won't persist forever. Because nostr is largely populated by self-hosting enthusiasts, the number of "tiny" relays is proportionally much higher than it will be if adoption picks up, even if the total number of relays grows. The trajectory is that nostr will drift toward fewer, larger relays, reducing the number of connections needed to fulfill any given query.

This is "centralizing", but it's important to understand that this isn't necessarily a bad thing. As long as there are more than one or two large hubs, there is user choice. And as long as it's possible to run a new relay, there is always an escape hatch. Nostr, like bitcoin, has no hard dependency on the biggest player in the network.

The other thing to consider is that there are lots of other techniques we can use to overcome the limits of the lowest-common denominator's limitations (mobile browser clients), including self hosted or third-party relay proxies. The trade-off here is that a little trust (aka centralization) can go a long way to reducing resource requirements needed to fulfill queries using the Outbox model.

If you're interested in more details on this topic, see this blog post.

That was a long digression, but there is one other thing that the Outbox model assumes to be the case. Even if the correct relays are found and connected to, they still may not return all desired content, either because they don't have it, or because they refuse to return it to the user requesting it.

This can happen if the publishing client isn't following the Outbox Model, if the author had migrated from one relay set to another without copying their notes over, or if the relay in question chose not to retain the author's content for some reason.

The first two issues can be fixed by improving implementations, but the question of policy is a little more interesting.

Relativistic relays

The Outbox Model is a mechanical process; it's only as useful as user relay selections are. In order for it to work, users have to be able to make intelligent relay selections.

Every relay has trade-offs, depending on its policy. 140.f7z.io would not be useful for long-form content, for example. Some relays might have a content retention policy that changes depending on whether you're a paying user. If you don't pay, you might find out too late that your content has been deleted from the relay.

So what makes a relay "good" for a particular use case? Well, it's complicated. Here are a few factors that go into that calculus:

• Is the relay in the same geographical as the user? Proximity reduces latency, but jurisdictional arbitrage might be desired. Users should probably have a variety of relays that fit different profiles.
• Will the relay ban the user? Do the operators have a history of good behavior? Is the relay focused on particular types of content? Is the relay's focus consistent with the user's goal in adding that relay to their list?
• What are the relay's retention policies? A user might want to set up an archival relay for her old content, or a multi-availability-zone relay so her notes are immediately accessible to the rest of the network.
• Does the relay require payment? Paid relays are more aligned with their users, but obviously come at a financial cost.
• Does the relay have policies for read-protecting content? If so, other users might not be able to find your posts published to that relay. On the other hand, some relays are configured to work as inboxes for direct messages, which can help preserve privacy.
• Does the relay request that users authenticate? Authentication can help manage spam, but it also allows relays to correlate content requests with users, reducing user privacy.
• Is the relay you use hosted by your client's developer? If so, you're in danger of getting banned from your client and your relay at the same time.
• Is the relay a hub? Using hubs can help smooth out rough areas in Outbox Model implementations, at the cost of centralization.
• Is the relay used by anyone else? One-off relays can be useful for archival purposes, but often won't be used by clients following the Outbox Model, depending on how they optimize requests.

There are lots of ways to approach the problem of helping users select relays, but it's an inherently complex problem which very few people will have the patience to properly address on their own. Relay selection is a multi-dimensional problem, and requires satisfying multiple constraints with a limited number of relay selections.

In the future, special-purpose clients might be used to help people build relay sets. Clients also might provide curated "relay kits" that users can choose and customize. Or, we might see an increase in hybrid solutions, like smarter relay proxies or client-local relays that synchronize using other protocols or platforms.

The Limitations of Outbox

Outbox is not a complete solution, not because of any of the caveats listed above, but because NIP 65 per se only addresses the question of how to index content by pubkey in a broadcast social media context. But there are many other scenarios for relay selection that Outbox does not solve:

• Community, chat, and group posts might be best posted to relays dedicated to that context.
• Direct messages shouldn't follow the same contours as public social media content.
• Topic-oriented relays, or relays serving a custom feed might be useful independent of who uses them.
• Relays focused on serving a particular kind of event, like music, long-form content, or relay selections, are useful independent of who reads from or writes to them.
• Certain clients might need to fulfill particular use cases by using relays that support certain protocol features, like search, count, or sync commands.
• Some events might not make sense to publish to relays, but should instead be shared only directly, out of band.

Some of these use cases might be solved by new specifications similar to Outbox that prescribe where certain data belongs — for example, NIP 17 requires users to publish a different relay list before they can receive direct messages, while NIP 72 places community relay recommendations directly into the group's metadata object. A reasonably complete list of different relay types can be found in this PR, very few of which have a canonical way to manage selections.

Other use cases might be supported more informally, either by relays advertising their own value proposition, or via third-party NIP 66 metadata. Still others might be supported by scoping the network down to only certain relays through explicit relay selection — this is how white-labeled Coracle instances work.

The basic idea here is that there are categories of events that don't have anything to do with where a particular person puts his or her "tweets". For every "what" on nostr, there should be a "how".

Keep nostr weird

Whatever additional systems we end up adopting for helping with relay selection, one thing is certain — people will continue to discover new, creative uses for relays, and we will always be playing catch up. This is one of the coolest things about nostr!

But it does mean that users will have to adapt their expectations to a network that partitions, re-configures, and evolves over time. Nostr is not a "worse" experience than legacy social media, but it is a version of social media that has itself been set free from the stagnant walled-garden model. Nostr is in many ways a living organism — we should be careful not to impose our expectations prematurely, leaving room to discover what this thing actually is, or can be.

If you enjoyed this post but want more take a look at the talk I gave at Nostrasia last year. I also wrote up a blog post at about the same time that addresses some of the same issues, but focuses more on privacy concerns around relays and nostr groups. Finally, I recently wrote this comment, which includes some details about challenges I've faced putting Outbox into Coracle.

A few weeks ago, Mike proposed the addition of a feature to nostr. The content of the proposal itself isn't important, but the resulting conversation illustrated something important about nostr development that I wanted to draw attention to.

If you're interested, you can find the issue here. The idea was basically a tag that disabled comments to a reply, for when you wanted to gracefully exit a conversation that had outlived its usefulness.

While I definitely sympathize with the experience of getting stuck in an unproductive argument and being unable to leave because you have to have the last word, I do think it's better to take responsibility for leaving the conversation, rather than make other people do it for you. You can either outlast your opponent, let them have the last word, or tell them "I don't want to talk about this any more, I'm not going to reply".

This is just my opinion, and it's really not important whether I'm right or not. What was interesting was how Vitor responded:

I am not sure if the NIP review process should consider "what's good for the user" in the discussion. That kind of nanny state thinking is what went wrong with regular social media in the first place.

Permaculture and Ethics

What it sounds like (although I have a hard time believing this is actually his position), is that Vitor is dismissing the relevance of an ethical framework in designing a protocol, preferring to stick to the mechanics of what is being suggested. As Vitor says, "Clients can do whatever they want, of course." This is true, but ignores the question of why a client developer might want to do any particular thing.

In a recent Thank God for Nostr episode, I interviewed Scott Mann of the Permaculture Podcast. When I asked him his opinion on how to manage decentralized protocol design and build effective consensus, here's what he had to say:

[You] don't need to frame it as a software development project, or even a protocol. I would look at it as a distributed community-based and community-supported project, whatever that is. Because permaculture is such a large umbrella, I like to go up to that 50,000 foot view and pull away from what the details are. Because the details are what we're going to build our solutions from.

And that's one of the things I didn't mention earlier — there's kind of a hierarchy within permaculture that goes, at the top are the ethics of permaculture: earth care, people care, fair share. Beneath that are the principles, then usually we have strategies and techniques.

But there's a dividing line between ethics and principles and the strategies and techniques, that we start at that top, and use the ethics to decide whether or not we're even going to launch a project.

He goes on to say:

I'm going to use the principles and see how can I apply those principles to my research and original design. To make sure that whatever I'm creating creates some kind of a surplus, to have a refinement process in place before I even launch, like what is that going to look like, even if I have to change it later, just having some of these building blocks in place.

And then once I've done that and have gone deep into my research into what this might look like, how I might launch it, that's where I would start going into strategies and going ok, how do I want to market this? How do I want to get this out to the people who are going to use it, how do I want to maintain this, how do I want to do distributed decision making, and then as I start to think about distributed decision making, looking at what is the form that I want to use for that?

Scott's thesis is that ethics and principles should be in the front of your mind both as you're considering a project, and as you continue to build it out. This not only makes sense as something to do if you want to succeed, it's categorically true. Action can't happen without agency, and it is your agency that informs what you choose to do, and how you plan to go about it.

The word "ethics" comes from the Greek ēthos, meaning "moral character". In other words, your ethic is who you are. Your values, hopes, preferences, faults all factor in to your actions.

This is not always clear, because in fiat-world, many people suspend their values in order to "get something done". If you want to protect your savings, you invest it in index funds, propping up the stock price of companies that hate you. If you want to make money, you go work at a job where you're berated quarterly for being racist. People think that they can exercise their own values in their private life, while actively undermining those values with the majority of their time and purchasing power.

But this is not how people with integrity act. And I think if you can say one thing about nostr — both its developers and the community at large — is that they have a very high level of integrity. In other words, their actions are clearly informed by their ēthos.

This is a very good thing. What is the point of building an entirely new internet if we're not going to impose our values upon it? What an absolute waste of effort.

A Nostr Manifesto

This is not to say that any one developer has the right to imposing his own vision on the protocol because of his own personal values and reasons for contributing, which is what I think Vitor was being cautious about. But there are lines I think we can draw as a community that can't really be crossed without excluding yourself from what I might call the "nostr group ethic".

So what are those boundaries? What is a nostrich? Here are some values I've observed to be generally shared among nostr developers and users. Not everyone would full agree with these (including myself), but I think they're a fair characterization of the community.

1. Free speech absolutism. No central entity should be able to globally censor any content. This comes with the trade-off of objectively evil content continuing to exist. This trade-off is acceptable, both because of the value of free speech, but also because evil will continue to exist regardless of attempts to suppress it.
2. Empower individuals over institutions. No centralized entity can be trusted to safeguard the interests of the individual. Institutional incentives are asymmetric and easily corrupted. Better to have many subjective views of the world, than a single, centrally managed view of the world.
3. Advertising-based business models should be viewed with great skepticism and caution. Advertising is a system of incentives that is central to the institutional corruption we see around us. Broadly, this includes paid ads, monetization of engagement, public/private partnerships, and "crypto".

There are others that are shared by many within nostr, although not as widely agreed upon. Two I can think of are:

• Economic activity should be voluntary. Software and content should be free, and creators should be amply rewarded via zaps or other value-for-value models.
• Social media should support real life community and relationships, not detract from them. We should all take time to touch grass.

I'm personally skeptical of the first of these, and strongly in support of the second. Much of my energy as a nostr developer has gone into attempting to subvert and reform traditional patterns of social media to not only support, but also resemble relationships that exist in the analog world God made, and placed us in.

This particular principle is the one at play in the conversation I linked to at the top of this post. My comments weren't an accusation that anyone was acting "unethically" in a universal sense, only that the proposed feature was incompatible with my vision for what nostr should be.

But of course, my vision is not shared by everyone, and the principle of "support real life" is clearly subservient to core ethic #2, which admits the value of a diverse set of opinions about the world. I have no right (or ability) to invalidate anyone else's core principles. But by the same token I'm free to express my own, and attempt to convince other people to share them. This is the basic value proposition of freedom of speech itself.

Ethical Cohesion

I would go further, and say that not only is it permissible to talk about ethical reasons for building one thing or another on nostr, it's essential. By having these conversations we fuse our individual ethics into a shared ethic. By calibrating our moral compasses to point in (roughly) the same direction, we also decrease the friction involved in getting something done.

I think this was a significant part of the idea behind Sovereign Engineering — get a bunch of people in a room together sharing meals and going on hikes, and the work will accelerate! This is also the way a church works. By meeting weekly together we strengthen our shared identity and build one another up through our activity. In fact, this is the basic definition of a community as Scott Mann puts it. In his words, a community can provide:

a series of connections, and a knowledge base, and a skillset that we can't fulfill as an individual, while having a social relationship with people in such a way that we can call on them for help.

So maybe, as I've said before, the real protocol is the friends we made along the way. Disagreement and discussion is a healthy thing for a community to have, and we should never stop asking "why?"

• Products with pubkeys: https://github.com/nostr-protocol/nips/pull/1225
• Shared-ownership events: https://github.com/nostr-protocol/nips/pull/1192 and https://github.com/nostr-protocol/nips/pull/1015
• Spreadsheets: https://github.com/nostr-protocol/nips/pull/1189
• Relational databases: https://github.com/nostr-protocol/nips/pull/1168
• Relay-specific notes: https://github.com/nostr-protocol/nips/pull/1146
• Editable notes: https://github.com/nostr-protocol/nips/pull/1090
• Restricted events: https://github.com/nostr-protocol/nips/pull/1083
• Relay-based access control: https://github.com/nostr-protocol/nips/pull/1030
• Protected events: https://github.com/nostr-protocol/nips/pull/1029
• Closed groups: https://github.com/nostr-protocol/nips/pull/875

Some of these I like, some I don't. But most of them go beyond adding new features to nostr (for example audio, video, speedruns, resumes, etc), and begin to change how nostr actually works.

Nostr can be an everything app, but I think that means something very specific. Nostr can represent data types from pretty much any domain, but it can't actually support all the semantics needed to build any arbitrary system.

I would suggest conservatism in what we build on nostr, but of course anyone can build whatever they want. But I do think it's possible to identify things that nostr is good at, and things it's bad at, and play to nostr's strengths.

Nostr's strengths:

• Being able to model any data type orthogonally to the rest
• Single-owner, self-authenticating, atomic data types
• Potential for robust content dispersal and retrieval if we can flesh out NIP 65 etc.

Nostr's weaknesses:

• Mutable state, non-atomic state
• Shared ownership, key delegation/rotation
• Privacy — metadata will always leak, lack of consistency makes key rotation harder
• Consistency — not everyone has the same view of the world
• Transactionality — nostr isn't good for updating multiple pieces of data in lockstep

It happens that the original use case of nostr — public broadcast social media — benefits greatly from nostr's strengths, and isn't bothered by any of nostr's weaknesses. Blob storage like what blossom is building also works well in this paradigm. A lot of the use cases @PABLOF7z​ has identified work beautifully well because of the single-owner public-read nature of nostr, which makes forks easy to model.

But things like access control, relational data, collaborative document creation, heavier datasets, or anything that requires a solution to the double-spend problem become very awkward (or impossible) to model on pure nostr. A simple example of this is lists. Not only is it common for a single user to mess up his follow lists because of a lack of consistency between clients or devices, but commonly requested features like shared ownership lists immediately result in a huge increase of complexity, either on the key management side or on the data structure side. Both of these problems are difficult to solve on nostr due to lack of consistency — keys can't always be reliably or safely shared, and linked data structures spanning multiple events by different authors can be hard to assemble reliably.

I think the danger here is that if we as a developer community fail to realize the limitations of nostr and try to adapt the protocol to fit every possible use case, on an ad-hoc basis, we're going to end up with a tragedy of the commons, where no developer can comprehend what must be done to get his work done, and all kinds of weird artifacts appear for end users that no one can explain.

Here are some suggestions I have for preventing this from happening. I realize no one is going to follow them. But maybe they can be helpful for avoidance of wasted time.

• Don't overload event kinds. Many people (including myself) have tried to extend kind 0 with attributes for forms, products, and groups, but that leads to madness. Instead, create a new metadata event signifying a different kind of agent.
• Don't model things as replaceable events if you can avoid it. This creates the problem of shared mutable state, which nostr doesn't have a good story for resolving. They also have a hard limit on how big they can be.
• Use different keys for different things. For domains where some kind of access control needs to be implemented, not tying everything to your main pubkey makes it possible to create and burn keys as needed. Incidentally, this can help users maintain better privacy. An example of this is private groups, which have a dedicated key separate from the group creator's own key.
• Event ownership should always be (is) single-key. If you need shared ownership, figure out a way to share keys. More work needs to go into key invalidation and rotation for this to really work.
• Explore the fork model — this is "my" version of the same thing, and coexists with rather than supersedes the original. This has potential not just for groups or wiki entries.
• Distinguish between different ways to use relays. Relays may be used as indexers (holding specific event types or supporting different features like search/count), repositories (holding many diverse events, to be used with filters), or curated feeds (to be used without filters, or with only a few filters).

These are just suggestions, and many of them may be wrong. Nostr development is hard, and getting harder. Keep it simple.

What's wrong with outbox?

It all started with a post from jack asking why more devs haven't implemented the outbox model. There are many answers to this question, not least having to do with there being two standards for user relay selections, and ongoing changes to NIP 65. But I don't want to talk about compatibility here.

nostr:nevent1qydhwumn8ghj7argv4nx7un9wd6zumn0wd68yvfwvdhk6tcprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszymhwden5te0wfjkccte9enrw73wd9hj7qpq2uf488j3uy084kpsn594xcef9g9x3lplx4xnglf0xwghyw2n3tfqqnrm02

Mazin responded with some numbers which estimate how many connections the outbox model requires. Too many connections can become expensive for low-power clients like mobile phones, not to mention some privacy issues stemming from nosy relays.

nevent1qythwumn8ghj76twvfhhstnwdaehgu3wwa5kuef0qyv8wumn8ghj7cm9d3kxzu3wdehhxarj9emkjmn99uq3samnwvaz7tmrwfjkzarj9ehx7um5wgh8w6twv5hsqgrn7l6zj7ht6ruyk76vvvtkfs4xrhyzc3tm64l3eyfvd40y26sz0gshmunh

I have some minor disagreements with Mazin's numbers, but I basically agree with his point — a purist outbox model, where a large proportion of nostr users run their own relays results in a high number of connections to different relays. I brought this question up late last year in my interview with Mike Dilger and in a conversation with fiatjaf, who convinced me that in practice, this doesn't matter — enough people will use a handful of larger hubs that there will be a good amount of overlap in relay selections between most pubkeys.

To articulate this more clearly: the goal is not "personal web nodes", which is a pipe dream the Farcasters and BlueSkys (BlueSkies?) of the world aim at, but a more pragmatic mix between large hubs and smaller purpose-built relays. These small relays might be outlets for large publishers, small groups, or nerds who also run their own SMTP servers and lightning nodes.

The point of the outbox model is that these small nodes be possible to run, and discoverable from the rest of the network so that we can preserve the censorship-resistant qualities of nostr that brought us here in the first place.

Blast It!

It's no secret that I've long been a critic of Mutiny's blastr relay implementation. My main objection is that the blastr approach doesn't account for the hard limits involved in scaling smaller relays. If the goal is to cross-pollinate notes across all relays in the network, all relays will require the same size database, and contain all notes in the network. This works right now (sort of), but as the network grows, the relays running on a $5 VPS are going to have their disks fill up and will inevitably fall over.

nevent1qyvhwumn8ghj76r0v3kxymmy9ehx7um5wgcjucm0d5hszxnhwden5te0wpuhyctdd9jzuenfv96x5ctx9e3k7mf0qythwumn8ghj7un9d3shjtnwdaehgu3wvfskuep0qqs07jr9qx49h53nhw76u7c3up2s72k7le2zj94h5fugmcgtyde4j9qfrnwxj

Not only that, but the content breakdown on any given relay by default becomes an undifferentiated soup of "GM", chinese notes, bots, bitcoin memes, and porn. Blastr makes it impossible to run an interesting relay without implementing write policies.

Which is actually fine! Because that's always been true — servers that allow anonymous uploads always get abused. Tony is just helpfully pointing out to us that this is no less true of nostr relays. I only wish he could have waited a little longer before mounting his attack on the network, because lots of hobbyists are interested in running interesting relays, but the tools don't yet exist to protect those servers from unsolicited notes.

One other note on blastr — Tony at one point described blastr as a relay proxy. This is an interesting perspective, which puts things in a different light. More on proxies later.

Ban Jack?

Here's a thought experiment: how might we actually "ban blastr"? @Pablof7z suggested to me in a conversation that you could configure your relay to check every note that gets published to your relay against the big nostr hubs, and if it exists on any of them to simply delete it. Of course, that would result in your relay being basically empty, and the hubs having all of your content. That's game theory for you I guess.

Another approach that was floated was to encourage users to only publish to small relays. In theory, this would force clients to implement outbox so users could still see the content they were subscribed to. Fiatjaf even posted two identical notes, one to his personal relay, and one to a hub to see which would get more engagement. The note posted to the mainstream relay got 10x more replies and likes than the more obscure note.

nostr:nevent1qyd8wumn8ghj7urewfsk66ty9enxjct5dfskvtnrdakj7qgmwaehxw309aex2mrp0yh8wetnw3jhymnzw33jucm0d5hszymhwden5te0wp6hyurvv4cxzeewv4ej7qpqdc2drrmdmlkcyna5kkcv8yls4f8zaj82jjl00xrh2tmmhw3ejsmsmp945r

Of course, this is thwarted by blastr, since blastr not only replicates notes posted to it, it also actively crawls the network as well. So the next logical step in this train of thought would be for hubs to encourage people to use small relays by actively blocking high-profile accounts.

nostr:nevent1qydhwumn8ghj7argv4nx7un9wd6zumn0wd68yvfwvdhk6tcpzdmhxue69uhhyetvv9ujue3h0ghxjme0qyd8wumn8ghj7urewfsk66ty9enxjct5dfskvtnrdakj7qpqpjhnn69lej55kde9l64jgmdkx2ngy2yk87trgjuzdte2skkwwnhqv5esfq

This would of course never happen (Damus is one client that hasn't implemented NIP 65, and they also run the biggest relay), but it was a fun thought experiment. At any rate, the silliness of the suggestion didn't stop certain people from getting offended that we would "disrupt the free market" by "forcing" our opinions on everyone else. Oh well.

Death to Blastr

In reality, even though blastr makes it a little harder to adopt outbox in the short term, its days are numbered. Eventually, relay operators will start to feel the pain of unsolicted notes, and will either shut their relays down or look for tools that will help them curate the content they host.

From my perspective, these tools take two forms — read protection and write protection. This is something I alluded to in my talk at Nostrasia last November.

Write protection is straightforward — already many relays have access control lists based on active subscriptions, invite codes, or just static whitelists that determine who is allowed to post to a given relay, or what event authors are represented there. This approach effectively prevents blastr from using relays as free storage, which is a huge improvement.

Read protection is more tricky, because anything publicly readable will be scraped by blastr and replicated to unauthenticated-write relays across the network. In most cases, this is ok, but there are use cases for relays to exist that host a unique collection of notes oriented around some organizing principle. Unfortunately, with blastr in action (or any scraper that might exist), the only way to do this is to actively protect proprietary content. There are a few approaches that can work to make this happen:

• IP-based access control lists
• AUTH-based access control lists
• Stripping signatures when serving events
• Storing and serving encrypted content

Each of these approaches has its own set of trade-offs. But depending on use case, any of them or a combination of them could work to allow relay operators to carve out their own piece of the nostr-verse. In fact, this is a big part of what Coracle is about — the white-labeled version of the product confines certain notes to proprietary relays, with optional encrypted group support.

Enough of my polemic against blastr. Let's talk about how to make the outbox model actually work.

Hints are pointless

Right now, clients that implement the outbox model rely pretty heavily on relay hints to find related notes — whether user profiles, reply parents, or community definitions. The problem with hints is that they are prone to link rot. Many of the relays that were set up a year ago when nostr took off are no longer online, and yet they persist in user relay lists, and in relay hints. These hints can't be updated — they are set in stone. What this means is that a different mechanism has to be used to find the notes the hints were supposed to help locate.

Because of this, I've come around to the position that hints are basically pointless. They are fine as a stopgap, and might be appropriate for certain obscure and ill-defined use cases where relay urls are the most durable address type available. But they provide basically no value in supporting the long-term robustness of the network.

What are durable, however, are pubkeys. Pubkeys are available pretty much everywhere, except in event id hints — and there is a proposal in the works to add a pubkey to those too. The cool thing about pubkeys as hints is that once you have a pubkey, all you need to do is find that person's kind 10002 inbox/outbox selections, and you should be able to find any note they have published.

This goes with the caveat that when users change their relay selections, or rotate their key, they (or their relays) should be sure to copy their notes to the new relay/pubkey.

The question then is: how do I find a given pubkey's relay selections?

There are already several mechanisms that make this reasonably easy. First of all, NIP 65 explicitly recommends publishing relay selections to a wide range of relays. This is a place where the blastr approach is appropriate. As a result, relay selections are usually available on the most popular public relays. Then there are special purpose relays like purplepag.es, which actively seek out these notes and index them.

These indexes are not confined to relays either. It would be trivial to create a DVM that you could ask for a pubkey's relay selections, optionally for a fee. Alex Gleason's proxy tag could also be used to indicate indexes that exist outside the nostr network — whether that be torrents, DHT keys, or what have you.

The best part is that this doesn't negatively impact the decentralization of the network because in principle these indexes are stateless — in other words, they're easily derived from the state of the public part of the nostr network.

Just do it for me

Looping back to where we started — the complexity and technical challenges of implementing the outbox model — there is a simple solution that many people have experimented with in different ways that could solve both issues at once: proxies.

As I mentioned above, Tony thinks of blastr as a proxy, and he's right. More specifically, it's a write-proxy. This is only part of its functionality (it also acts as an independent agent which crawls the network. EDIT: apparently this is not true!), but it is an essential part of how people use it.

Another kind of proxy is a read proxy. There are several implementations of these, including my own multiplextr proxy, which is outbox-compatible (although it requires a wrapper protocol for use). The advantage of a proxy like this is that it can reduce the number of connections a client has to open, and the number of duplicate events it has to download.

Proxies can do all kinds of fancy things in the background too, like managing the outbox model on behalf of the client, building an index of everything the user would be likely to ask for in advance to speed up response times, and more.

One interesting possibility is that a NIP 46 signer could double as a proxy, reducing the number of round trips needed. And since a signer already has access to your private key, this kind of proxy would not result in an escalation in permissions necessary for the proxy to work.

It's simple

The number of cool and creative solutions to the content replication and indexing problem is huge, and certainly doesn't end with blastr. Just to summarize the next steps I'm excited to see (to be honest, I want to build them myself, but we all know how that goes):

• More clients supporting outbox
• Outbox implementations maturing (Coracle's still has some issues that need to be worked out)
• A shift from relying on relay hints to relying on pubkey hints + relay selection indexes of some kind
• Proxy/signer combos which can take on some of the heavy lifting for clients of delivering events to the right inboxes, and pulling events from the right outboxes

Let's get building!

Coracle now has private groups.

This means you can now send almost any nostr event over an encrypted channel to the rest of the group's members. This is substantially different from group chats, in that it uses rotating shared keys to provide weak forward secrecy, better scaling, and dynamic member access. This more closely approximates one of the most popular social media products in existence - The Nostr is now a direct competitor of The Facebook.

I built this for my community. I wanted something "good enough" to entice people to leave the advertising-fueled surveillance honeypot that is Facebook. In order to work, it needed to at least support notes, events, and marketplace listings. Although support is still quite basic, Coracle has checked all three of these boxes.

Before I get into the details though, it's important to mention that these groups should not be considered "private" any more than Facebook groups or Mastodon servers are (although privacy is substantially better). A better analog might be WeChat, which uses encryption with the same set of trade-offs. So don't post anything to private groups that might get you in trouble!

With that said, it's possible to run a highly private group. The backbone of this spec is e2e encryption, but relay selection can play an important part in hiding metadata from the rest of the network. If you have a relay you trust to protect notes and not share metadata, your security is significantly increased.

Prior art

Nostr-compatible group products aren't a totally novel thing, as it turns out. In fact draft NIP 112 has been around since June, and is already implemented in ArcadeCity. So why am I creating a new standard? I'l get into the positive benefits of my approach more below, but the quick answers are:

• The new encryption standard is going to break compatibility anyway. If we can end up with a better spec, now is the time.
• ArcadeCity development seems to have stalled.
• NIP-72 communities already have a ton of traction, and match what I'm trying to achieve with encrypted channels.

Of course I'm highly indebted to the project, the design of which is still visible in my final design.

Another product that exists to do something similar in a nostr-compatible way is Soapbox by Alex Gleason. This is a great project, particularly since his Mostr project bridges the ActivityPub world and Nostr. ActivityPub works well for highly centralized communities, but the architecture suffers from this centralization too. In particular, not even DMs are e2e encrypted, and just like regular notes are protected only by authentication enforced by servers.

Finally, there's NIP 29, which is fiatjaf's competing groups project. This has some interesting properties, for example the ability to "fork" a group by linking events together. However, similar to ActivityPub it relies exclusively on relays to protect user privacy, and in a fairly non-standard way. You do get to take advantage of nostr's multi-master architecture though, and signatures are also stripped from events in order to discourage propagation through the network.

None of these solutions quite satisfied me, so I built my own.

How it works

One of the coolest things about a NIP 72 community-based group spec is that is supports a spectrum of privacy requirements. A group admin might choose to publish group metadata privately so that it's only visible to the group, publicly so that other people can find the group and ask to join, or leave off a private component entirely.

Likewise, since private groups are backwards-compatible with public communities, it's easy to add a private component to existing groups. This can be useful especially for groups run by a business or content publisher, since public exposure is a good thing but certain group members might have more or less access. This could be used to support a patreon-type model, automating group membership based on subscription tier, for example.

An important aspect of the design that makes automation possible is the concept of a dedicated administration key. By decoupling this key from the original creator of the group, ownership can be shared as simply as sharing the key. This allows multiple admins to manage the group simultaneously either manually or using automations built into the group relays or special purpose bot-clients.

This of course raises the issue of admin access revocation, which isn't possible - that is, until we have a solution for key rotation for normal accounts. Once that's in place, the same process can be used to rotate group admin keys.

In the meantime, it's also trivial to reduce the exposure an admin key gets. You wouldn't generally want to simply paste the key wherever it's needed, but luckily that problem has already been solved as well. Instead of giving every admin or admin bot the key, it's trivial to set up an nsecbunker that authorizes each admin client - and can revoke access as needed.

This level of administration is of course fairly complex, but I think it's important to think through the requirements businesses and other advanced users will eventually impose and anticipate them as we're able, not through over-engineering, but through simple concepts that can be reused.

One other neat feature of this NIP is the definition of invite codes, which are essential for running a private group at any kind of scale. When requesting access to a group, a user can send along a "claim", which can be anything - for example a static invite code, a payment receipt, or an explanation of why they want to join. This claim can be validated by hand by a human, or processed by a bot to instantly admit the new member to the group.

When a new member is admitted to the group, the admin can either share an existing access key with them, or they can rotate the key for the entire group. If relays expire access keys after a certain amount of time, this can create a weak form of forward secrecy, where attackers won't be able to access old content, even if they gain access to the admin key.

Limitations and Future Work

The bar for new nostr clients has risen significantly since I first put Coracle out there. The new groups component is far more mature than Coracle was for much of its early life, but it has its rough edges. Many of these just need to be smoothed out through further UX work, but some are more technical in nature.

• The groups spec relies on NIP 44, which isn't yet available in most signer extensions. That means that unless you log in with your private key (please don't), you won't be able to create or gain access to any private groups.
• Hybrid groups (public groups with a private area) aren't really tested yet, or fully supported in Coracle's UI. It's an open question whether this is even a good idea, since it becomes pretty hard for users to know if they're posting publicly or privately in every context.
• Moderation is not implemented, so if you're creating a public group there is currently no way in Coracle to approve posts. Also, groups created in Coracle don't show up in Satellite for some reason — this is something I'll be working on improving.
• Whether this approach actually scales is another question. It's very hard to build member lists of hundreds of thousands of people, and without a relay helping to filter events, it might become prohibitively expensive to download and analyze all the events posted to a group. We'll see what develops as the design matures and the implementation undergoes stress testing.

Conclusion

Something I like about both nostr and bitcoin is that it empowers the users of the software. The corollary of this of course is that it's important to exercise this power with care - real damage can be done with this group spec, just as real damage can be done to bitcoin holders through low entropy key generation or poor key handling practices. So please, if you're going to implement this spec, communicate clearly with your users its limitations, and encourage them to run their own relays.

Nevertheless, I am stoked to be another 1% closer to my goal of helping my community - and anyone else who uses nostr - to exercise individual sovereignty and protect their freedom and privacy. Let's keep at it.

The results

The assets in scope for the audit can be found on Paul Miller's NIP 44 repository. Included is the full text of the spec, as well as implementations in Typescript, Rust, and Go.

Overall, here's what Cure53 had to say:

The Cure53 team succeeded in achieving very good coverage of the WP1-WP4 targets. All
ten findings spotted by the testers were classified as general weaknesses with limited
exploitation potential. In other words, no vulnerabilities were detected within the inspected
components.

In other words, no vulnerabilities were found, but there are things we can do to harden the implementations. Cure53 elaborates:

The fact that Cure53 was not able to identify any exploitable vulnerabilities can be
interpreted as a positive sign in regard to the security of the NIP44 specification and
implementations. Nevertheless, even though the spotted problem can all be seen as general
weaknesses and hardening advice, they should not be ignored. It is known that weaknesses
may serve as entry points for more severe vulnerabilities in the future. Cure53 strongly
recommends swift resolution of all reported flaws.

These weaknesses and recommendations each have their own code, prefixed by "NOS-01", which is our audit number. Here's a quick summary of the individual points:

• NOS-01-001 describes a weakness related to naive secp256k1 implementations, which may accept public keys with both x and y coordinates instead of just x and a sign-bit. This can result in the compromise of a sender's private key if the sender can be tricked into encrypting a message with an invalid public key. This is known as a "twist attack". Cure53 recommends some additional test vectors to make sure that uncompressed keys are not accepted.
• NOS-01-002 includes a handful of minor recommendations, including specifying the initial counter for ChaCha20, nailing down key formats, and a few other things which they go into more depth on elsewhere.
• NOS-01-003 provides some additional test vectors to prevent twist attacks and out of bound errors.
• NOS-01-004 suggests using a constant time equality function to prevent timing side-channel attacks, along with some code samples.
• NOS-01-005 identifies missing range checks in the Go implementation which should be addressed in order to avoid crashes.
• NOS-01-006 addresses the lack of provisions for forward secrecy in the NIP 44 spec. They suggest introducing sender-side forward secrecy by deriving the session key using ephemeral keys and a KDF.
• NOS-01-007 explains that using the same key for multiple purposes can lead to unexpected compromises when taken together. Best practice is to use a different key for signing and for encryption, which nostr violates. While the particular curves we use aren't vulnerable to this exploit, Cure53 recommends deriving an encryption key from a user's main key using a KDF in order to reduce the impact of a leaked encryption key.
• NOS-01-008 identifies an incorrect use of the salt parameter in calls to HKDF (an HMAC-based Key Derivation Function allows us to prove that an encrypted payload was not forged). Currently, the salt is generated randomly every time, but the security definition of HKDF specifies that it be static. This is likely not exploitable, but should be fixed. Cure53 suggests switching the nonce and salt when calling the HKDF, and deriving a static salt using something like SHA256(pubA, pubB, "nip-44-v2").
• NOS-01-009 demonstrates that the Message Authentication Code (MAC) tag is currently computed without a nonce. This doesn't compromise the encryption, and authentication is covered by event signatures as specified in NIP 01, but it does prevent the MAC from being useful on its own.
• NOS-01-010 suggests using clearer boundaries when constructing the HMAC payload.

The takeaway here is that with a few minor changes, the spec itself is sufficient to "provide a simple way for the users to communicate privately". There are, however, a number of edge cases which can compromise implementations, so for any developers out there porting the spec to new languages, be sure to take a look at Paul's test vectors.

In the end, the NIP44 specification and implementations appear to have already achieved the security goal of providing users with a way to communicate privately. Miscellaneous issues identified by Cure53 during NOS-01 correspond to further hardening of the current version and/or suggestions for future versions. Following these suggestions could provide more advanced security guarantees.

They also clarify that:

Importantly, the lack of certain security guarantees, such as forward secrecy, post-compromise security, deniability, and post-quantum, had been known to the development team prior to NOS-01.

Since this is the case, it's important that client developers be absolutely clear with their users about what NIP 44 is good for, and what it's not. This is a "simple" way to communicate privately — users with more advanced privacy requirements should use something like MLS or SimpleX.

What's next?

NIP 44 isn't quite ready for widespread use just yet, but it shouldn't take long to incorporate all the adjustments mentioned in the audit. Then there are several NIPs which rely on the encryption in NIP 04 which should eventually be migrated individually. Beyond that, a whole world of affordances for encrypted data becomes available on nostr.

First and most anticipated, we can finally deprecate NIP 04 and stop leaking DM metadata. Vitor's Sealed Gift Wraps are an excellent alternative to NIP 04 DMs, and add support for small group chats, which will be a big improvement to client UX.

Other more ambitious specifications targeting larger groups exist as well, including my Closed Communities draft spec. This NIP makes it possible to extend NIP 72 communities with a private component, which can be useful for publishers or real-life communities.

There are lots of other things NIP 44 might be used for as well, for example:

• The ability to share your location only with certain people
• Private calendar events, product listings, or streams
• Private tags — instead of encrypting an entire event, it should be possible to encrypt only a single tag's value. This allows senders to attach private data to public events.

Conclusion

There's lots of work still to be done, but I think this marks a huge milestone for nostr. So thank you to Paul Miller for his work on the spec and the audit, and to OpenSats for funding the audit!

This discussion gets complicated very quickly, so to begin, I thought I'd cover the two components that everyone thinks they understand first (relays and encryption), and move on to the one that combines the existing primitives in new and exciting ways (groups).

What even are relays, anyway?

If you saw my talk at Nostrasia, this section is just a re-iteration of it. In fact, I wrote this blog post before the conference to help me with my presentation. So hello, from the past.

Relays are dumb. They're just servers, which speak a particular protocol, and which implement some minimal logic to support that protocol. Relays have to be interoperable, which is why it's important to design new parts of the nostr protocol in such a way as to be backwards-compatible, and simple to implement.

As a result, relay development has historically focused on performance to the exclusion of new functionality. You can see this with the success of strfry and Will's adaptation of the underlying technique to nostrdb.

One of my favorite programming talks is Simple Made Easy by Rich Hickey, the author of Clojure. One of the points he makes in that talk is that object-oriented programming is such a mess because it "complects" behavior and state. Hickey's solution to this problem is to use functions and values rather than methods and state.

I think that this same mistake is one we are at risk of making with nostr relays. I'll come back to this in a bit, but keep that distinction in the back of your mind: separating data and behavior can lead to a much simpler and more robust system.

Despite the emphasis on interoperability, relays shouldn't all be exact clones. There are a few different ways relays can differentiate themselves, either by providing specific functionality, by storing different sets of events, or by exposing a different interface.

Added functionality

Some implementations have pushed features forward, for example rbr.bio which implemented the COUNT verb from NIP 45 early on. However, these efforts are usually either supported by special clients (like nostr.band or primal's caching service), or fail to gain wide adoption and are eventually abandoned. The result is that NIPs specifying optional functionality tend to die without sufficient user demand.

The reason for this is that if you're building a client that selects relays based on user preferences, it can't rely on functionality that doesn't exist on the majority of relays. Without widespread support, search results or feeds requested based on Primal's special syntax will be incomplete or fail entirely. As a result, clients often build proprietary solutions to these problems, increasing centralization risk.

Subtracted functionality

Relays are able to helpfully differentiate themselves however, by offering a subset of protocol functionality. For example, purplepag.es is a reliable source for profile data and relay preferences, but it doesn't require clients to do anything different in order to be useful. Instead, users can simply add purplepag.es to their relay list and instantly get a better experience. A similar approach many commodity relays take is blocking certain event kinds, for example 1059 gift wraps or NIP 94 image headers. This fortifies their particular purpose, which is to not support more exotic protocol features.

Other kinds of relays don't support this guarantee, for example relay.noswhere.com only supports search if the search term is not accompanied by a filter, responding to standard requests with "filters: not planned (non-search queries)". This can still be useful, but it is not a relay, because it requires clients to special case how it's used.

Data relevance

There is a different kind of differentiation relays can implement, based not on functionality, but on content. This is what I mean when I talk about "relay de-commodification". The value proposition this type of relay offers is not "we can do something no one else does", but "we have the information you're looking for".

There are two ways to accomplish this: content curation and access controls. The difference being that content curation is context-independent, and might be based on topic, keyword, or LLM analysis, while access controls only accepts content published by certain accounts. Several relay implementations and hosting providers (for example relay.tools) support both of these approaches, and of course it's easy to combine them.

Likewise, there are two ways for a relay to gain access to the desired data, leading to two different value propositions.

A relay that focuses on topical data might scrape the wider nostr network in order to ensure it is able to reliably serve everything relevant to its niche. This might be done using any number of heuristics, including filtering by pubkey, topic, or sentiment, but in any case the relay is not intended primarily to be written to, but to be read from. This is primarily an additive model, focusing on offering a complete view of the given topic.

Another approach is to let the data come to you instead of scraping it from the wider network by only accepting events published by certain people, or being the go-to relay for a certain topic. In order for this model to work though, there needs to be some way for the relay to keep this data to itself by preventing scraper-powered relays from harvesting its data. This effectively means that these relays will not only have policies for who can write to them, but also who can read from them. These might be the same set of people (for example with a community relay), or more people might be allowed to read from the relay than are able to write to it, as in the case of content producers who want a paywalled private social media enclave.

Keep relays weird

Most relay implementations expose their data over websockets in line with the spec, but this isn't actually inherent to the function of a relay. We've already seen several projects implement relays in-app for caching purposes. These relays don't usually go through the ceremony of spec adherence, but some (e.g. nostrdb) do.

Different transport mechanisms and hosting options are possible for relays, and would only require clients to use a specific adapter to get at them. Some examples:

• A relay running on your phone as a service that other nostr clients can share to reduce data storage requirements
• A relay running on a LAN, only accessible from a particular location
• A relay running via any other transport, for example radio or carrier pigeon

The idea I'm trying to get across here is that relays aren't necessarily websocket servers. More abstractly, they are a name attached to a set of events that can be accessed in a standard way.

So, what are relays?

To recap, here's my definition of a relay:

1. They correctly support standard functionality
2. They should only support additional functionality related to curating their data set
3. They may restrict the data types they accept and serve
4. They may restrict the data they accept and serve based on content or pubkey

One additional feature that isn't currently standard but ought to be is strfry's negentropy synchronization feature. This is the opposite of AUTH, which protects a relay's own dataset, in that it allows relays to more efficiently share their data with peers. A nice bonus is that it also allows clients to use their local cache more effectively as well.

Simple Made Easy

Going back to Rich Hickey's talk, separating behavior and state allows you to work directly with your data, freely implementing new behavior whenever you need to. This applies to sets of events stored on relays just as much as it applies to in-memory data.

One thing that makes this possible is cryptographic hash functions, which cleanly transform state into values - the difference being that "state" couples a name with data that may change, while a hash is a compact representation of immutable data. If two hashes of event ids don't match, they're not the same set of events.

How does this apply to relays? Instead of adding new behavior to relays directly ("methods"), we can instead call "functions" on a relay's data.

This function can be anything - a centralized server, client code, or a DVM - that takes REQ filters as input. The function gathers the needed data (maybe from a cache, maybe using a sync command, maybe from multiple relays for completeness), performs the transformation, and returns it. The result is that relays become simple, interoperable repositories for data, without limiting what server-side computation can be applied to events.

While huge search indexes and centralized servers have a cost in terms of centralization, I think DVMs solve this problem brilliantly - the common interface for this augmented behavior is open, and can admit anyone who wants to start competing for search, counting, recommendations, or anything else you might want to do with events.

As Alan Perlis said, "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures."

Encrypted Messages

Nostr DMs were a huge mistake, and I want to do it again.

NIP 04 has some obvious problems, most notably metadata leakage. The cryptography also hasn't been audited, and has some potential theoretical attack vectors. I've been working with some other nostr devs on a new cryptographic primitive, which will be audited and hopefully result in a better foundation for future work.

But fixing the cryptography still doesn't fix DMs. Nostr is not really the right architecture for secure private messaging, because ratchets require state. Nostr is designed such that there are no hard dependencies between events, which are replicated across multiple relays. This makes it very hard to manage the kind of state that protocols like SimpleX or Signal require.

Even apart from the difficulty of implementing stateful messaging protocols on nostr, we have two other problems to contend with: metadata leaks and spotty deliverability. Both of these problems come from the reliance of nostr on relays, which generally are not trustworthy or reliable.

Metadata Leakage

NIP 04 is really bad. Anyone can see who you corresponded with, when, how frequently, and how much you said just by looking at the event. Vitor's NIP 24 chat proposal aims to fix this with gift wraps, which use ephemeral keys to obscure the sender, padding to obscure message size, timestamp randomization to obscure timing, and double-wrapping to prevent leakage of the signed message.

This is a big improvement, but there are other forms of metadata that are still visible. To start, the recipient is still visible, along with how many messages they have received and their general size. So you can tell if someone sends lots of messages, or not a lot, but that's about it.

If we don't assume relays are trustworthy though, there are all kinds of implicit metadata that can be used to infer information about the sender, including:

• Client fingerprinting
• Relay selection for publishing and queries
• IP address collection
• First seen timestamp
• Identification of users by AUTH
• Correlation of the event with other messages sent/received during a session

These issues are much harder to solve, because they are part of the process of delivering the message, rather than just constructing it.

Transport and deliverability

The key feature of nostr that makes it work is relays. Relays provide storage redundancy and heterogeneity of purpose with a common interface shared by all instances. This is great for censorship resistance, but everything comes with a tradeoff.

Completeness and consistency are not guaranteed on nostr. This is ok in a social media context, because there's no way to keep up with everything everyone you follow says anyway, and you can rely on interactions and algorithmic tools to catch you up on anything important you missed.

But with private messages, every payload counts. Receiving messages out of order, with a delay, or not at all can severely disrupt conversations between two people, leading to misunderstandings and worse. This is a common experience on nostr; even fiatjaf has expressed skepticism about the viability of private messages on nostr.

This flakiness comes from a combination of low relay quality and poor relay selection on the part of clients. Relays can't deliver messages if they can't keep up with clients, go offline, or drop messages (accidentally or deliberately). Likewise, clients can't find messages if they look for them in the wrong place.

Relays fixes this

Let me just put things into perspective really quick. Twitter DMs are not e2e encrypted, and weren't encrypted at all until earlier this year. Mastodon messages are not encrypted - instance admins can read everything. Facebook has e2ee messages, but only between normal users, not in communities, groups, business chats, or marketplace chats.

So while nostr's architecture is not sufficient to make secure encrypted communication a value proposition of the protocol in itself, there is a lot we can do to improve on the status quo. Because nostr has no admins, the only option for privacy of any kind is end-to-end encryption. Metadata leakage notwithstanding, nostr messages can be "good enough" for some definition of that phrase.

It's important to not just punt on making DMs available in a standard way, because there is an expectation that a complete social media solution will have a way to establish a private communication channel with someone else in the network. Otherwise you're stuck with public correspondence and link sharing in order to hop to another protocol.

Let's just assume that you agree with me at this point, and that until there is a javascript SDK for SimpleX or MLS, we're stuck with nostr DMs of some kind. So how can we reduce metadata leakage and improve deliverability?

Fixing Flakiness

Let's start with deliverability first. NIP 65's inbox/outbox model is great. While it doesn't fix deliverability on its own, if every relay recommended was in fact reliable, and reliably followed by clients, there would be a very clear indication of where a particular user's messages could be found. Neither of these things is really true right now, but I do think clients are moving in this direction. Hopefully we can educate users better on how to pick good relays.

There are some limits to this model of course. For example, if you want to read notes from 1000 pubkeys at once, your client will likely end up connecting to hundreds of relays unless it has a cap, in which case it will miss a lot of notes.

One clean solution to this problem is relay proxies. There are a few implementations, but the one I run at mux.coracle.social is a stateless proxy that uses a wrapper protocol so clients can still select and dispatch to relays. I originally built this to improve mobile data use by deduplicating events on the server, and it does that pretty well, although it currently does it in the most naive way possible.

Multiplextr currently does not qualify as a "real" relay based on my four rules, since it uses a wrapper protocol. However, a smarter implementation would be able to route requests and events effectively by relying on client authentication and filter analysis, without changing the protocol. This would also allow clients to completely offload the complex business of relay selection to its proxy. Clients would also be able to combine multiple proxies just like they combine relays - because proxies are relays!

Proxying Privacy

Another downside of NIP 65 is that it encourages clients to connect indiscriminately to hundreds of different relays based on tag hints, other users' preferences, and relays baked into bech32 identifiers. This is not only bad for battery life and performance, but also allows attackers to easily trick your client into to connecting to their relay so they can start analyzing your traffic.

The best way to defend yourself against nosy relays is to not connect to them at all. And the easiest way to do that is to only connect to relays you control (or trust), and let them do the dirty work for you.

With a proxy, even if you don't control it, you can reduce the number of parties who can snoop on your traffic by orders of magnitude. And since other people use the proxy too, your traffic gets mixed with theirs, making it much harder to analyze. Of course, the downside of using a public proxy is that the proxy sees all messages you send and receive, and the relays you select for each request. Even worse, an untrusted multiplexer can hijack an authenticated session, exfiltrating private data from protected relays.

A proxy you control (or trust, either via social relationships or economic ones) is the best of both worlds. This proxy could also do some smart things for you to improve privacy as well, for example:

• Delayed publishing of gift wrapped messages so relays can't guess as easily when the event was first seen.
• Closing and re-opening connections to reduce the number of messages sent over an AUTH'd session.
• It could block suspicious messages on the fly, for example spam or notes with phishing links.
• If an abundance of options are available for relay selections, it could randomize which relays are used, or avoid using less reputable ones.

So I guess what I'm saying here, is "run a node". But not just any node - in the early days of nostr there were a lot of people spinning up relays because they thought it helped decentralization. Well, as it turns out, just like in bitcoin you have to use your node in order for it to be useful.

Beyond DMs

I'm not done yet, though. What's the point of all this? Why put all this effort into "good enough" DMs when we could spend the effort adopting SimpleX or MLS (said Semisol, who lives rent free in my head)?

Well, because there isn't a single "best" DM option. As I've dug into the topic, I've discovered that a ton of tradeoffs exist between complexity, privacy, and scale. The way I understand it, there are three basic options here:

• One-to-one encryption. This is how both SimpleX and Vitor's group message draft works. It allows for things like ratchets to be added on top, and is fairly simple - but fails to scale because every message has to be encrypted separately for every group member, resulting in n*m messages, where n is the number of group members, and m is the number of messages sent.
• Encryption via shared key. This is how WhatsApp works, and it scales really well because messages don't have to be duplicated. And while it does support forward secrecy, it does not support post-compromise security, so it is significantly less secure.
• MLS has a unique hierarchical approach to managing keys, which improves the complexity of one-to-one encryption so that it's logarithmic rather than exponential. However, MLS is highly stateful and still has scaling limitations.

My takeaway is that there is no perfect solution. Ratchets are the gold standard, but can't really be implemented using nostr relays, since they frequently require servers to have very specific characteristics, like the ability to deliver messages in order.

The fact remains though, that social media needs encrypted messaging of some kind, even if we encourage users to bail to a more complete solution if they want to be truly private. And encryption is useful beyond small group chats - think of Facebook groups, which scale to millions of people and whose users have a much lower expectation of privacy. This is what nostr is good at - social data, whether encrypted or in plaintext.

Belt and Suspenders

For a long time I have waffled between two visions of how private groups ought to work. Should they be encrypted messages sprinkled across the nostr network like confetti? Or should they live on one or two relays in plaintext, protected by AUTH?

Both of these options have issues. Sprinkling encrypted messages across the network almost guarantees deliverability problems, since already many commodity relays block gift wraps. And that's ok! They have no obligation to store your encrypted messages.

Storing group messages in plaintext is also sub-optimal, even if it's on a dedicated relay. Of course we're not going for military-grade secrecy here, but it would be trivially easy through a bug, or even a client that doesn't know how to deal with groups, to re-broadcast events that should stay private. We could do something wacky, like strip the signature from events hosted on a relay, but if we're going to do something non-standard, we might as well do the best we can.

So why not both? Group messages should be encrypted, and they should be stored on a particular set of relays. Again, assuming the relays used are trustworthy, this fixes both privacy and deliverability problems, because it's very clear where messages posted to the group should go.

Promoting relays to trusted status has some nice, unintended benefits too. One common objection to using encrypted content is that it becomes hard to filter. But if a relay is trusted to store a group's messages, it can also be trusted to be admitted as a member to the group. This would allow the relay to decrypt events and filter them to respond to requests by authenticated group members.

Of course, this would violate rule #1 for relays, since they would no longer be supporting standard functionality, but a guy can dream, can't he?

Conclusion

When I started working on nostr I had no idea it would end up being this complex to accomplish what I set out to do. Building a twitter clone was only a first step to an MVP, but I always had my eye on the prize of serving my local church community, which doesn't give a sat about social media per se.

Private groups are important to me not because I'm setting out to support political dissidents or journalists (although I hope they find nostr useful too), but because I want to get my friends off of Facebook, and the only way to do that is to create private marketplaces, calendars, communties, and more.

What it ain't

That fact was brought home to me this week as I worked for the nth time to repair the damage my refactoring did to the public chat feature in Coracle. This was the first feature I added as a response to popular demand and hype over a new NIP being added to nostr, way back in December 2022. Coracle was (I believe) the first general-purpose client to support public chat, which I hoped would bolster its notoriety.

I don't know if it helped Coracle's popularity, but it has cost me several weeks of development time in maintenance and abstraction cost since it was introduced. While it wouldn't be hard to apply the same content filtering work I've added to feeds in Coracle to chat, I'm not sure I want to continue to have my time stolen by sunk cost. And with the increasing frequency with which chat groups are being hijacked by spammers of various kinds, I no longer have the option of leaving chat alone.

So I've decided to remove group chat from Coracle. The implementation is still available, at chat.coracle.social, but unless a wild maintainer appears, it won't be receiving any updates for the foreseeable future. Chat was never a goal of my project, and while it may be one of the more popular features of Coracle, it's one of the least crucial.

This is just one of many illustrative features. In the same release in which I removed chat, I also added a background music player for kind 1808s. This is equally frivolous — but even more fun. It will likely live for a while, and disappear again someday.

Set a course for ???, warp speed

Now that Coracle is basically working™️ I need to take a step back and figure out what I want it to be. Even if I don't know where I'm headed, I'm still moving pretty fast. In order to make the best use of my grant from OpenSats, it's imperative that I at least have a mission statement.

In explaining to Vitor why I was removing chat, I accidentally articulated a pretty good one:

nostr:nevent1qqsfwxqv4qq5ajdx8yaksg2fwcw0enpykxvkl9kf2meqgheaq7f3ycgpz3mhxue69uhhyetvv9ujuerpd46hxtnfdufeeexz

The reason for this emphasis comes from the people I originally set out to build for: my church community. These people are generally not "internet-native". They aren't active on reddit, twitter, discord, twitch, tiktok, or youtube. To the extent they use social media, they post pictures on instagram, and buy and sell goods on facebook marketplace.

The features required to support a facebook analog are fairly obvious, and include event calendars, private groups, and marketplaces. However, there are a few dimensions to the problem that make building such a product more challenging than building a twitter clone.

Product complexity

One of the most consistent pieces of advice I've received about building a facebook clone is: don't. Facebook succeeded because it bundled multiple standalone tools into a single interface, and tied them together with a single social graph.

This is one of the things nostr solves — no longer do you need an everything app, if you can use micro-apps on an everything protocol. When I built zephyr last week, I was again stunned at how powerful building on nostr is. In half a day I put together something that would be impossible with traditional technologies — because if you don't have the ability to produce content on a centralized platform, you don't have the ability to consume it either.

But protocols feel different from platforms, especially for the layman. The other half of facebook's value add is (was) a simple and intuitive user interface, which not only made the different components available, but made them accessible as well. I don't think micro-apps, web app stores, or nostr browsers are complete solutions to the UX problem. Nothing can replace a design tailored to your specific end user.

Privacy and parochialism

Social graph partitioning is an essential part of designing a social network. In twitter-like applications, an an effective strategy might be described as "whatever". In other words, as long as you are able to 1. connect to the people you want to follow and 2. discover new content, you're fine — there's no need to be exhaustive or exclusive.

But tighter-knit communities require exclusivity. And by "community" I should qualify that I do not mean a reddit-style common interest, but a real locus of interdependence — whether intellectual, economic, or familial. You don't self-select into true communities except by way of a long (sometimes very long) probationary period.

But, as we learned from mastodon's failures, people are also members of multiple overlapping communities simultaneously. This means that an application designed to serve a person in multiple roles across multiple communities needs to be able to address those differing social circles according to their permeability. Enforcing community privacy using encryption might be appropriate for very tight groups like families, while other groups might want more lax rules about content sharing or member admission.

On nostr, relays are an excellent primitive for implementing access controls, and may be sufficient for all but the most private communication. I've also experimented with encrypted groups (nostr-in-nostr), but the juice might not be worth the squeeze — the additional complexity of gift wrapping might not be necessary if relays can be trusted not to leak data.

Two different approaches to group-specific relays could be used, depending on the level of trust required for the community: an Uncle Jim model where one or more admins host the infrastructure, or a virtual relay model where the group admin creates a private space on commoditized infrastructure.

The value of decentralization at a small scale

Now, of course if you have admins, you've basically opted back in to siloed, centralized platforms, unless shared identity is a value add. Single sign on could be attractive, but not overwhelmingly so — I think the thing that makes an identity shared across multiple platforms compelling is the ability to cross-post.

Unless content is protected by encryption, this is always possible, and would be very good for end users. Allowing content from outside a group to be pulled into a group allows the group to comment on it using its own idioms and values. It also enables loose coupling between weakly-related communities (for example a church and the town it is located in).

This would work very well for community groups, which aren't usually too picky about driving engagement, but would work less well for publishers or media brands, even if cross-posting of content would be beneficial to their curated communities. Instead of increasing the value of your community, it could be seen (and exploited) as advertising for competing siloes.

Multiple types of engagement

Finally, there are multiple types of relationship that might be emphasized to a greater or lesser degree in one community vs another. Facebook-style "friends" are generally less focused on ideas, and more on maintaining relationships. I wouldn't want to read what my mom posts on social media, but I do want to stay in touch with her. In contrast, twitter-style "follows" are more oriented towards the flow of information itself, rather than relationships. An application directed at handling both types of "community" would need to respect the difference between the two.

Toward something, anything

With all that in mind, here's an attempt at articulating my vision for Coracle, loosely based on the EOS model (no, not that EOS).

While my focus above has been on individuals as members of real-life communities, in practice many of these groups can be defined by a common interest in a given content "publisher", which is a more concrete (and practical) use case. This might be something as big as a newspaper, or as small as a substack or patreon. The content provides something for members of the group to talk about.

It seems to me that groups that are not content-focused operate in much the same way, except that topics are selected by group members, rather than by administrators. So the general strategy would be to build a product that can be used more informally by primarily targeting content publishers.

Core Values

1. Real life > digital life
2. Inorganic advertising is cancer
3. Encourage purposeful engagement
4. Different communities should be distinct, but mutually beneficial

Mission Statement

To enrich the social media experience of individuals in their capacity as members of multiple overlapping real-world intellectual, economic, or relational communities.

Target Market

Small publishers who want to create a place for subscribers to interact with the publisher's (public and paid) content and one another. Media brands who are open to the benefits of cross-posting across siloes.

What does it look like?

• Anyone can create a group hosted on self-hosted or commodity relays, or a mixture of both.
• A mixture of public and encrypted content for a given group
• Cross-posting of public content between groups
• Private content exclusive to group members
• Multiple tiers of group membership
• Configurable access control: admins, moderators, write access, read access
• Invite codes, including referral discounts/rewards
• Integrated calendar events (public and private)
• Integrated marketplaces (publisher merchandise or peer-to-peer)
• Support topical threads (oriented around publisher content) and member microblogging

Goals

• A relay implementation that supports virtual relays with all required access controls
• A group spec that supports granular content permissions and visibility
• An end-user interface that supports browsing multiple groups and non-group content
• An admin interface for group administration
• White-labeled configurable client implementation

Addendum: Ditto

Alex Gleason has been working on Ditto of late, which is first of all a way to reconcile ActivityPub and Nostr using a server implementation that supports both protocols, but which has the interesting side effect of marrying the siloed, exclusive ActivityPub model with nostr identities. This could be a good solution for branded clients and tighter community control, especially since it allows for the use of existing tools from ActivityPub's more mature ecosystem.

I intend to keep a close eye on Ditto as it matures. I hope that even if the two projects have a significant amount of overlap, that the problem space is big enough to allow for variations in execution, especially since Coracle has the ability to be fully nostr-native.

Conclusion

If you're currently a user of Coracle, don't worry too much, I won't be making any drastic changes anytime soon. I do think that the twitter-like, group-independent part of nostr is valuable, at least as a supplement to more cohesive communities. However, in order to achieve such an ambitious project, I'll have to exercise the discipline to remain focused and cut out any fat that accumulates. We'll see how I do.

Data Vending Machines

Pablo followed up here, asking my opinion on custom feeds in general, and his Data Vending Machines proposal in particular. I thought I'd write up my response in long form for posterity.

I like DVMs, although I ignored it for a while because I figured that Pablo was bouncing off the walls as usual (congrats on placing in the AI competition!) and I'd wait till he settled down. I read a bit of his nip this morning and the custom feeds use case caught my eye as an interesting application of the DVM idea.

Algorithm options

I don't know if vending machines are a good solution to custom feeds, but I'm open to being convinced. Below are a few different approaches we could take to the problem:

• Built into clients — closed or open source, not swappable except maybe using NIP 89
• Built into relays — swappable, composable, monetizable, allows for moderation/publishing models, can support "private" algorithms. Also highly consistent, you're not going to get partial data because events weren't where you were looking.
• Published by pubkeys — not parameterizable. We saw how this went with @Sherry​'s recommendations bot. Great idea, but lots of limitations.
• List- or label-based curations — lightweight, not parameterizable, but very clean, could support private use case via wrapping
• DVMs — parameterizable, monetizable, but very heavy, probably not suitable for general purpose custom feeds due to latency
• Custom algorithms published as NostrScript events — swappable, pluggable, very flexible, content-addressable.

I think all those options have their own benefits and trade offs for different use cases. Relay-based feeds are going to be good for low-latency, dynamically-built feeds. Pubkeys are good for publishing-type use cases. NostrScript is good for shareable user-defined feeds (and other things).

I don't think DVMs make sense for "feeds" as we think of them, but could be good for generating more bespoke artifacts, like a "top highlights for this week" email/feed, or something that interleaves nostr events with some external data source.

My favorite of the bunch is lists and/or labels, which can be arbitrarily nested and used by clients/scripts/relays to build more sophisticated feeds (for example, by incorporating distributed moderation or web-of-trust-based recommendations subjective to the target user).

This has actually always been a core part of my vision for nostr and coracle since even before I found nostr. Unlike Amethyst's kind-1984 based content moderation approach, lists and labels can be interpreted using user-defined terminology.

In other words, the definition of "good music" depends on who you ask. Allowing the answer to be generated not by a single pubkey or relay, but by your entire network seems to best embrace the emergent behavior nostr enables. These content recommendations can be weighted as much as desired based on other social data including reactions, replies, mutual follows, how many lists someone appears on, etc.

Of course, this more analysis-heavy approach means that feeds are not defined only by lists and labels, but by an opinionated interpretation layered on top of those, which might best be provided by a relay, DVM, or client.

Do we even want feeds?

While we're on the topic, it's also worth asking if "feeds" are really what we want. The idea is very closely tied to the infinite-scroll pattern common on legacy social media, and the name evokes the process of fattening livestock for the kill. In some ways, I think infinite scroll is with us for better or for worse - pagination buttons at the bottom of a page is no solution, that would just be an inferior version of the same thing.

Perhaps, taking a leaf out of Twitter's book, brevity is the answer. Being forced to fit our thoughts into 280 characters helps us write more clearly and with more interest, what if "feeds" had the same constraints? Having no page bottom incentivises a never-ending list of diffuse non-information.

This is one reason I love email newsletters - at best they are carefully researched, edited, scheduled, and sent to your inbox. The email inbox is a hostile place, missives that land there are likely to suffer a swift demise. If an email is to survive, it must be something the recipient wants to read.

In a way, The Nostr Report is itself a custom feed in this sense, published via a shared pubkey. It would also be possible to write a "client" that links your pubkey and email, and sends you a daily/weekly digest, tailored to your interests and social graph.

Taking this idea a step further and combining it with list and label based feeds, instead of creating lists based on content, you might create a list based on timeframe. Then, in order to generate a digest for that day or week, you would pull all time-based curations from your network, analyze them for common threads, and re-publish as an email, note, or list.

At any rate, I think this is an exciting area of research, and that we've only scratched the surface of what's possible with an open social graph. I'm glad to see Pablo creating an entirely new category of solution with his proposal.

It's a well-known fact that Nostr's NIP 04 DMs leak metadata. This seems like an obvious flaw, and has been pointed out as such many times. After all, if anyone can see who you're messaging and how frequently, what time of the day, how large your messages are, who else is mentioned, and correlate multiple separate conversations with one another, how private are your communications really?

A common retort repeated among those who "get" Nostr (myself included) is "it's not a bug, it's a feature". This hearkens back to the early days of the internet, when internet security was less than an afterthought, and social platforms throve on various permutations of the anonymous confessions-type app. How interesting to be able to flex to your friends about whom you're DMing and how often! Most conversations don't really need to be private anyway, so we might as well gamify them. Nostr is nothing if not fun.

In all seriousness though, metadata leakage is a problem. In one sense, Nostr's DMs are a huge improvement over legacy direct messages (the platform can no longer rat you out to the FBI), but they are also a massive step backward (literally anyone can rat you out to the FBI). I'm completely confident we'll be able to solve this issue for DMs, but solving it for other data types within Nostr might pose a bigger problem.

Social Content

A use case for Nostr I've had on my mind these last few months is web-of-trust reviews and recommendations. The same sybil attack that allows bots to threaten social networks has also been used as a marketing tool for unscrupulous sellers. NPS surveys, purchased reviews, and platform complicity have destroyed the credibility of product reviews online, just like keyword-stuffed content has ruined Google's search results.

Proof-of-work would do nothing to defend against this attack, because the problem is not volume, it's false credibility. The correct tool to employ against false credibility is web-of-trust — verifiable trustworthiness relative to the end user's own social graph.

This is a huge opportunity for Nostr, and one I'm very excited about. Imagine you want to know whether the vibro-recombinant-shake-faker (VRSF) will result in visible abs in under 6 days. Well, it has over 4 thousand 5-star reviews on Amazon, and all the 1-star reviews are riddled with typos and non sequiturs. So it must work, and make you smarter into the deal! Well, sadly no, visible abs are actually a lie sold to you by "big gym".

Now imagine you could find your three friends who fell for this gyp and ask them what they thought — you might just end up with a lower average rating, and you'd certainly have a higher level of certainty that the VRSF is not worth the vibra-foam it's molded from.

This same query could be performed for any product, service, or cultural experience. And you wouldn't be limited to asking for opinions from your entire social graph, it would be easy to curate a list of epicureans to help you choose a restaurant, or trusted bookworms to help you decide what to read next.

Currently, big tech is unable to pull this off, because Facebook won't share its social graph with Google, and Google won't share its business data with Facebook. But if an open database of people and businesses exists on Nostr, anyone can re-combine these silos in new and interesting ways.

Notes and other Spies

So that's the pitch, but let's consider the downsides.

An open social graph coupled with recommendations means that not only can you ask what your friends think about a given product, you can ask:

• What a given person's friends think about a product
• What kind of person likes a given product
• How products and people cluster

That last one in particular is interesting, since it means you could find reasonable answers to some interesting questions:

• Does a given region have fertility problems?
• What are the political leanings of a given group?
• How effective was a particular advertisement with a given group?

This is the kind of social experiment that has historically earned Facebook so much heat. Democratizing this data does not prevent its correlation from being a violation of personal privacy, especially since it will be computationally expensive to do sophisticated analysis on it — and the results of that analysis can be kept private. And to be clear, this is a problem well beyond the combination of social information and public reviews. This is just one example of many similar things that could go wrong with an open database of user behavior.

Not to put too fine a point on it, we are at risk of handing the surveillance panopticon over to our would-be overlords on a silver platter. Just as walled gardens have managed us in the past to sway political opinion or pump the bags of Big X, an open, interoperable content graph will make building a repressive administrative state almost too easy.

Let's not give up just yet

So what can we do about it? I want a ratings system based on my social graph, but not at the expense of our collective privacy. We need to keep this threat in mind as we build out Nostr to address novel use cases. Zero-knowledge proofs might be relevant here, or we might be able to get by with a simple re-configuration of data custody.

In the future users might publish to a small number of relays they trust not to relay their data, similar to @fiatjaf's NIP-29 chat proposal. These relays might then support a more sophisticated query interface so that they can answer questions without revealing too much information. One interesting thing about this approach is that it might push relays towards the PWN model BlueSky uses.

Not all data needs to be treated the same way either, which would give us flexibility when implementing these heuristics. Just as a note might be either broadcast or sent to a single person or group, certain reviews or other activity might only be revealed to people who authenticate themselves in some way.

Like so many other questions with Nostr, this requires our concentrated attention. If all we're doing is building a convenient system for Klaus Schwab to make sure we ate our breakfast bugs, what are we even doing?