allow customizing html footer via config
[websub-hub] / README.md
1 # Welcome to my WebSub Hub
2
3 ## What
4
5 [WebSub](https://www.w3.org/TR/websub/) is a protocol for subscribing to content updates from publishers. The Hub is the central component which manages that relationship.
6
7 This Hub implementation was created with personal [self-hostable](https://indieweb.org/WebSub) deployment in mind. It is content-agnostic, supports multiple database backends, and can scale to multiple nodes for robustness and capacity.
8
9 ## Beware
10
11 This is currently a Minimum Viable Product release. Basic functionality is complete, but the administration experience may be challenging.
12
13 ## Up And Running
14
15 Customize configuration within `config/${NODE_ENV}.js`. All envs inherit settings from `default.js` if not specified. Environment is selected using the `NODE_ENV` value, defaulting to `development` if unset.
16
17 Database table initialization and schema version migrations are automated. Configure SQLite with a database file, or point PostgreSQL to a created database.
18
19 A user will need to be created in order to view the `/admin` pages; the `bin/authAddUser.js` script will do this.
20
21 The bundled logger spews JSON to stdout.
22
23 ### Quickstart Example
24
25 One way of deploying this server is behind nginx, with the pm2 package to manage the server process, and a local postgres database. Some details on this are presented here as a rough guide to any parts of this stack which may be unfamiliar.
26
27 - Have NodeJS 12-ish available.
28 - Have PostgreSQL available.
29 - Clone the server repository.
30 ```git clone https://git.squeep.com/websub-hub```
31 - Install the production dependencies.
32 ```cd websub-hub```
33 ```NODE_ENV=production npm i```
34 - Create a ```config/production.js``` configuration file. See ```config/default.js``` for available settings.
35 > <pre>
36 > 'use strict';
37 > // Minimum required configuration settings
38 > module.exports = {
39 > dingus: {
40 > selfBaseUrl: 'https://hub.squeep.com/',
41 > },
42 > db: {
43 > connectionString: 'postgresql://websubhub:mypassword@localhost/websubhub',
44 > },
45 > };
46 > </pre>
47 - Prepare PostgreSQL with a user and database, using e.g. ```psql```.
48 > <pre>
49 > CREATE ROLE websubhub WITH CREATEDB LOGIN PASSWORD 'mypassword';
50 > GRANT websubhub TO postgres
51 > CREATE DATABASE websubhub OWNER=websubhub;
52 > GRANT ALL PRIVILEGES ON DATABASE websubhub TO websubhub;
53 > \c websubhub
54 > CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
55 > </pre>
56 - Install the process manager, system-wide.
57 ```npm i -g pm2```
58 - Configure the process manager to keep the server logs from growing unbounded.
59 ```pm2 install pm2-logrotate```
60 ```pm2 set pm2-logrotate:rotateInterval '0 0 1 * *'``` (rotate monthly)
61 ```pm2 set pm2-logrotate:compress true```
62 ```pm2 startup``` (arrange to start process monitor on system boot)
63 - Launch the server, running one process per CPU, and persist it through reboots.
64 ```NODE_ENV=production pm2 start server.js --name websubhub -i max```
65 ```pm2 save```
66 - Create an administration user.
67 ```NODE_ENV=production node bin/authUserAdd.js admin```
68 - Copy the static files to somewhere nginx will serve them from. This will vary greatly depending on your setup.
69 ```cp -rp static /home/websubhub/hub.squeep.com/html/static```
70 - Expose the server through nginx.
71 > <pre>
72 > server {
73 > listen 443 ssl http2;
74 > ssl_certificate /etc/ssl/nginx/server-chain.pem;
75 > ssl_certificate_key /etc/ssl/nginx/server.key;
76 > server_name hub.squeep.com;
77 > root /home/websubhub/hub.squeep.com/html
78 > try_files $uri $uri/ @websubhub;
79 >
80 > location @websubhub {
81 > proxy_pass http://websubhub$uri;
82 > proxy_set_header Host $host;
83 > proxy_set_header X-Forwarded-For $remote_addr;
84 > proxy_set_header X-Forwarded-Proto $scheme;
85 > proxy_http_version 1.1;
86 > }
87 >
88 > location = / {
89 > proxy_pass http://websubhub$is_args$args;
90 > proxy_set_header Host $host;
91 > proxy_set_header X-Forwarded-For $remote_addr;
92 > proxy_set_header X-Forwarded-Proto $scheme;
93 > proxy_http_version 1.1;
94 > }
95 > }
96 > </pre>
97 ```nginx -s reload```
98 - The Hub server should now be available!
99
100 ## Frills
101
102 A rudimentary tally of a topic's subscribers is available on the `/info?topic=topicUrl` endpoint. The topicUrl should be URI encoded. Formats available are SVG badge, JSON, and plain text, selectable by setting e.g. `format=svg` in the query parameters.
103
104 ## Architecture
105
106 The Hub keeps track of three primary entities:
107
108 - Topics: data and metadata for a published content endpoint. Topics are unique by source URL.
109 - Subscriptions: the relationship between a client requesting content and the topic providing it. Subscriptions are unique by topic and client URL.
110 - Verifications: updates to subscriptions which are pending confirmation. Verifications are not unique, but only the most recent for any Subscription pairing will be acted upon.
111
112 Any tasks in progress (notably: fetching new topic content, distributing that content to subscribers, or confirming pending verifications) are doled out and managed by a cooperative advisory locking mechanism. The task queue is wrangled in the database within the `*_in_progress` tables.
113
114 A Hub node will periodically check for more tasks to perform, executing them up to a set concurrency limit.
115
116 ### Quirks
117
118 This implementation is built atop an in-house API framework, for Reasons. It would not be hard to replace such with something more mainstream, but that is not currently a design goal.
119
120 ### File Tour
121
122 - bin/ - utility scripts
123 - config/
124 - default.js - defines all configuration parameters' default values
125 - index.js - merges an environment's values over defaults
126 - *.js - environment specific values, edit these as needed
127 - server.js - launches the application server
128 - src/
129 - common.js - utility functions
130 - communication.js - outgoing requests and associated logic
131 - db/
132 - base.js - abstract database class that any engine will implement
133 - errors.js - database Error types
134 - index.js - database factory
135 - schema-version-helper.js - schema migrations aide
136 - postgres/
137 - index.js - PostgreSQL implementation
138 - listener.js - notify/listen connection to support topic content caching
139 - sql/ - statements and schemas
140 - sqlite/
141 - index.js - SQLite implementation
142 - sql/ - statements and schemas
143 - enum.js - invariants
144 - errors.js - local Error types
145 - link-helper.js - processes Link headers
146 - logger.js - a very simple logging class
147 - manager.js - process incoming requests
148 - service.js - defines incoming endpoints, linking the API server framework to the manager methods
149 - template/ - HTML content
150 - worker.js - maintains a pool of tasks in progress, for sending out updates, performing verifications, et cetera
151 - test/ - unit and coverage tests
152 - test-e2e/ - support for whole-service testing