git.squeep.com Git - websub-hub/blob - README.md

   1 # Welcome to my WebSub Hub
   2
   3 ## What
   4
   5 [WebSub](https://www.w3.org/TR/websub/) is a protocol for subscribing to content updates from publishers.  The Hub is the central component which manages that relationship.
   6
   7 This Hub implementation was created with personal [self-hostable](https://indieweb.org/WebSub) deployment in mind.  It is content-agnostic, supports multiple database backends, and can scale to multiple nodes for robustness and capacity.
   8
   9 ## Beware
  10
  11 This is currently a Minimum Viable Product release.  Basic functionality is complete, but the administration experience may be challenging.
  12
  13 ## Up And Running
  14
  15 Customize configuration within `config/${NODE_ENV}.js`.  All envs inherit settings from `default.js` if not specified.  Environment is selected using the `NODE_ENV` value, defaulting to `development` if unset.
  16
  17 Database table initialization and schema version migrations are automated.  Configure SQLite with a database file, or point PostgreSQL to a created database.
  18
  19 A user will need to be created in order to view the `/admin` pages; the `bin/authAddUser.js` script will do this.
  20
  21 The bundled logger spews JSON to stdout.
  22
  23 ### Quickstart Example
  24
  25 One way of deploying this server is behind nginx, with the pm2 package to manage the server process, and a local postgres database.  Some details on this are presented here as a rough guide to any parts of this stack which may be unfamiliar.
  26
  27 - Have NodeJS 12-ish available.
  28 - Have PostgreSQL available.
  29 - Clone the server repository.
  30   ```git clone https://git.squeep.com/websub-hub```
  31 - Install the production dependencies.
  32   ```cd websub-hub```
  33   ```NODE_ENV=production npm i```
  34 - Create a ```config/production.js``` configuration file.  See ```config/default.js``` for available settings.
  35   > <pre>
  36   > 'use strict';
  37   > // Minimum required configuration settings
  38   > module.exports = {
  39   >   dingus: {
  40   >     selfBaseUrl: 'https://hub.squeep.com/',
  41   >   },
  42   >   db: {
  43   >     connectionString: 'postgresql://websubhub:mypassword@localhost/websubhub',
  44   >   },
  45   > };
  46   > </pre>
  47 - Prepare PostgreSQL with a user and database, using e.g. ```psql```.
  48   > <pre>
  49   > CREATE ROLE websubhub WITH CREATEDB LOGIN PASSWORD 'mypassword';
  50   > GRANT websubhub TO postgres
  51   > CREATE DATABASE websubhub OWNER=websubhub;
  52   > GRANT ALL PRIVILEGES ON DATABASE websubhub TO websubhub;
  53   > \c websubhub
  54   > CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
  55   > </pre>
  56 - Install the process manager, system-wide.
  57   ```npm i -g pm2```
  58 - Configure the process manager to keep the server logs from growing unbounded.
  59   ```pm2 install pm2-logrotate```
  60   ```pm2 set pm2-logrotate:rotateInterval '0 0 1 * *'``` (rotate monthly)
  61   ```pm2 set pm2-logrotate:compress true```
  62   ```pm2 startup``` (arrange to start process monitor on system boot)
  63 - Launch the server, running one process per CPU, and persist it through reboots.
  64   ```NODE_ENV=production pm2 start server.js --name websubhub -i max```
  65   ```pm2 save```
  66 - Create an administration user.
  67   ```NODE_ENV=production node bin/authUserAdd.js admin```
  68 - Copy the static files to somewhere nginx will serve them from.  This will vary greatly depending on your setup.
  69   ```cp -rp static /home/websubhub/hub.squeep.com/html/static```
  70 - Expose the server through nginx.
  71   > <pre>
  72   > server {
  73   >   listen 443 ssl http2;
  74   >   ssl_certificate /etc/ssl/nginx/server-chain.pem;
  75   >   ssl_certificate_key /etc/ssl/nginx/server.key;
  76   >   server_name hub.squeep.com;
  77   >   root /home/websubhub/hub.squeep.com/html
  78   >   try_files $uri $uri/ @websubhub;
  79   >
  80   >   location @websubhub {
  81   >     proxy_pass http://websubhub$uri;
  82   >     proxy_set_header Host $host;
  83   >     proxy_set_header X-Forwarded-For $remote_addr;
  84   >     proxy_set_header X-Forwarded-Proto $scheme;
  85   >     proxy_http_version 1.1;
  86   >   }
  87   >
  88   >   location = / {
  89   >     proxy_pass http://websubhub$is_args$args;
  90   >     proxy_set_header Host $host;
  91   >     proxy_set_header X-Forwarded-For $remote_addr;
  92   >     proxy_set_header X-Forwarded-Proto $scheme;
  93   >     proxy_http_version 1.1;
  94   >   }
  95   > }
  96   > </pre>
  97   ```nginx -s reload```
  98 - The Hub server should now be available!
  99
 100 ## Frills
 101
 102 A rudimentary tally of a topic's subscribers is available on the `/info?topic=topicUrl` endpoint.  The topicUrl should be URI encoded.  Formats available are SVG badge, JSON, and plain text, selectable by setting e.g. `format=svg` in the query parameters.
 103
 104 ## Architecture
 105
 106 The Hub keeps track of three primary entities:
 107
 108 - Topics: data and metadata for a published content endpoint.  Topics are unique by source URL.
 109 - Subscriptions: the relationship between a client requesting content and the topic providing it.  Subscriptions are unique by topic and client URL.
 110 - Verifications: updates to subscriptions which are pending confirmation.  Verifications are not unique, but only the most recent for any Subscription pairing will be acted upon.
 111
 112 Any tasks in progress (notably: fetching new topic content, distributing that content to subscribers, or confirming pending verifications) are doled out and managed by a cooperative advisory locking mechanism.  The task queue is wrangled in the database within the `*_in_progress` tables.
 113
 114 A Hub node will periodically check for more tasks to perform, executing them up to a set concurrency limit.
 115
 116 ### Quirks
 117
 118 This implementation is built atop an in-house API framework, for Reasons.  It would not be hard to replace such with something more mainstream, but that is not currently a design goal.
 119
 120 ### File Tour
 121
 122 - bin/ - utility scripts
 123 - config/
 124   - default.js - defines all configuration parameters' default values
 125   - index.js - merges an environment's values over defaults
 126   - *.js - environment specific values, edit these as needed
 127 - server.js - launches the application server
 128 - src/
 129   - common.js - utility functions
 130   - communication.js - outgoing requests and associated logic
 131   - db/
 132     - base.js - abstract database class that any engine will implement
 133     - errors.js - database Error types
 134     - index.js - database factory
 135     - schema-version-helper.js - schema migrations aide
 136     - postgres/
 137       - index.js - PostgreSQL implementation
 138       - listener.js - notify/listen connection to support topic content caching
 139       - sql/ - statements and schemas
 140     - sqlite/
 141       - index.js - SQLite implementation
 142       - sql/ - statements and schemas
 143   - enum.js - invariants
 144   - errors.js - local Error types
 145   - link-helper.js - processes Link headers
 146   - logger.js - a very simple logging class
 147   - manager.js - process incoming requests
 148   - service.js - defines incoming endpoints, linking the API server framework to the manager methods
 149   - template/ - HTML content
 150   - worker.js - maintains a pool of tasks in progress, for sending out updates, performing verifications, et cetera
 151 - test/ - unit and coverage tests
 152 - test-e2e/ - support for whole-service testing