Production Outage/Issue

CPU was slammed in prod

On-Call rotation for production issues

Monitoring systems

Escalation procedures

Procedures for recovery (that anyone in the OC rotation can follow)

CPU lowered for some daemon requests (ListDocs)

SSR reduced

Mitigated/blocked some crawler traffic

Peak traffic was ~12/req/sec

Follow-through!

General Process

Commit msgs should explain "why", not "what"!

Timeouts in the desktop app

Sometimes, occasionally, hard to reproduce

Eric speculates: ListDocuments

Burdi: probably related to non-paginated requests

InteractionSummary expensive in backend

Uses ListCitations internally. Should be indexed material.

Should we build an automated stress test?

Maybe it relates to writing data? Not just "clicking around" the app

Gabo: Sees ListDocuments timeout immediately after starting the app!!

No user-facing bug noticed

Publish document is VERY slow

We have a 5-sec timeout for PushDocuments

@Untitled Contact owning this project for now <3

Private Documents functionality

Phase 1: Caps on private docs - CONSENSUS!

Write+READ caps

Phase 2: Children of private docs?

Somebody to propose a formal def for permission inheritance

Maybe "inherit permissions" is a flag

Anybody with a link can read/write

Revocations?

Related to invite workflow- grant a cap to somebody who doesn't yet have an account (or a known account ID)

Burdi suggesting having a metadata-based inheritance workflow, rather than using path inheritance.

Eric: We could use flat, non-human readable path names for docs. Use redirects for pretty paths?

Is inheritance implemented through path names or relationships between canonicalized document ID. @Untitled Contact thought experimenting. Along with Revocations

Users assume private by default?

"Public by default" was our mindset as a publishing system, but we are trending towards a knowledge system

AI Threats

"Private documents are private until some advanced AI comes along and hacks your data"

What is the budget in engineering hours? (And in tokens?)

Maybe: 2-eng team works on this every Friday. We will seek help from advisors/friendlies

We need to audit+redteam our deps like libp2p+ipfs.

This is Exercise that we need to build strength

Our Frontier AI wranglers, every Friday: @Untitled Contact + @Eric Vicenti

We are not using production infra! Running in VMs only

Starting Stack: Kimi Sub (because its willing to work) and Pi (good harness that is model-agnostic)

More Agenda?

Review where we have broken (or missing) pagination!

Where do we do frontend sorting?

How to learn about indexed DB schema?