Announcing activitypub-testing

A couple months ago, we started working on a conformance testing tool for ActivityPub servers, and now we're sharing an early version publicly for the first time.

I (bengo) blogged about this project for the first time about a week ago, and it implements some of the ideas I described in my more recent blogposts,Testing Ecosystem Wishlist and Use Test Specifications to Improve Testability of ActivityPub Requirements.

In this early alpha release, there are test cases for each of the eleven 'easy to test' requirements I categorized in Which ActivityPub Server Requirements are Easy to Test?.

The ask

1. start trying out the activitypub-testing against real-world activitypub actors and analyzing the results, whether via cli or via our browser-based tool
2. be gentle if something is surprising to you.
   • This is an early release and is full of weakly held opinions and agile design decisions. We expect to iterate on this a lot based on learnings from your feedback, adding more tests, implementing test specifications, etc. The repo has an issue tracker you can add issues to.

How?

See the overview for a tour of the components of the system and the getting started section for a quick guide to the syntax of the CLI interfaces.

Example: Testing an Actor at onepage.pub

onepage.pub is an ActivityPub implementation by https://cosocial.ca/@evan/ that aims to be "A compliant, robust ActivityPub server in one page". Here's how we tested it with activitypub-testing:

1. bengo created an issue activitypub-testing should interoperate with onepage.pub

2. then we did a sample Run 0 with an invocation like

   npx activitypub-testing test actor https://onepage.pub/person/qfU45YyW4aDIG-lN_lHSS
   

3. one test gave a result with outcome failed, which upon closer look didn't seem like a fair determination, so we improved the test to not be so picky

4. we did a Run 1 with the improved version of activitypub-testing and took advantage of a new feature to provide proof of Authorization (required by onepage.pub when handling outbox POST requests)

   activitypub-testing test actor https://onepage.pub/person/qfU45YyW4aDIG-lN_lHSS --input.authorization="Bearer `cat ~/.activitypub-testing/bengo_onepagepub_token`"
   

5. Analysis of Run 1 shows onepage.pub gets 8 passed Assertions which correspond to satisfying 8 conformance requirements.

Request for Comment (RFC)

What will be especially interesting is discussing the results of specific runs against specific actors, e.g. using npx activitypub-testing test actor <your-actor-url>

Comment using ActivityPub

Publish on ActivityPub and link to https://socialweb.coop/activitypub-testing/. Feel free to mention https://mastodon.social/@bengo.

File an Issue

If you'd rather record a comment but without using ActivityPub, the next-best place to publish an issue is into the git repository's issue tracker, e.g. https://codeberg.org/socialweb.coop/activitypub-testing/src/branch/main/issues.