### Changed
- **Breaking:** Pleroma won't start if it detects unapplied migrations
-- **Breaking:** attachments are removed along with statuses. Does not affect duplicate files and attachments without status.
- **Breaking:** Elixir >=1.8 is now required (was >= 1.7)
- **Breaking:** `Pleroma.Plugs.RemoteIp` and `:rate_limiter` enabled by default. Please ensure your reverse proxy forwards the real IP!
- **Breaking:** attachment links (`config :pleroma, :instance, no_attachment_links` and `config :pleroma, Pleroma.Upload, link_name`) disabled by default
### Added
- `:chat_limit` option to limit chat characters.
+- `cleanup_attachments` option to remove attachments along with statuses. Does not affect duplicate files and attachments without status. Enabling this will increase load to database when deleting statuses on larger instances.
- Refreshing poll results for remote polls
- Authentication: Added rate limit for password-authorized actions / login existence checks
- Static Frontend: Add the ability to render user profiles and notices server-side without requiring JS app.
account_field_name_length: 512,
account_field_value_length: 2048,
external_user_synchronization: true,
- extended_nickname_format: true
+ extended_nickname_format: true,
+ cleanup_attachments: false
config :pleroma, :feed,
post_title: %{
"Set to `true` to use extended local nicknames format (allows underscores/dashes)." <>
" This will break federation with older software for theses nicknames."
},
+ %{
+ key: :cleanup_attachments,
+ type: :boolean,
+ description: """
+ "Set to `true` to remove associated attachments when status is removed.
+ This will not affect duplicates and attachments without status.
+ Enabling this will increase load to database when deleting statuses on larger instances.
+ """
+ },
%{
key: :max_pinned_statuses,
type: :integer,
- 404 Not Found `"Not found"`
- On success: 200 OK `{}`
+## `GET /api/pleroma/admin/restart`
+
+### Restarts pleroma application
+
+- Params: none
+- Response:
+ - On failure:
+ - 400 Bad Request `"To use this endpoint you need to enable configuration from database."`
+
+```json
+{}
+```
+
## `GET /api/pleroma/admin/config/migrate_from_db`
### Run mix task pleroma.config migrate_from_db
```
## How it works
-Settings are stored in database and are applied in `runtime` after each change. Most of the settings take effect immediately, except some, which need instance reboot. These settings are needed in `compile time`, that's why settings are duplicated to the file.
-
-File with duplicated settings is located in `config/{env}.exported_from_db.exs` if pleroma is runned from source. For prod env it will be `config/prod.exported_from_db.exs`.
-
-For releases: `/etc/pleroma/prod.exported_from_db.secret.exs` or `PLEROMA_CONFIG_PATH/prod.exported_from_db.exs`.
+Settings are stored in database and are applied in `runtime` after each change. Most of the settings take effect immediately, except some, which need instance reboot.
## How to set it up
You need to migrate your existing settings to the database. This task will migrate only added by user settings.
mix pleroma.config migrate_to_db
```
-Now you can change settings in admin interface. After each save, settings from database are duplicated to the `config/{env}.exported_from_db.exs` file.
+Now you can change settings in admin interface. If `reboot time` settings were changed, pleroma must be rebooted.
<span style="color:red">**ATTENTION**</span>
- all settings inside these keys:
- `:hackney_pools`
- `:chat`
+ - `Oban`
+ - `:rate_limit`
+ - `:markup`
+ - `:streamer`
- partially settings inside these keys:
- `:seconds_valid` in `Pleroma.Captcha`
- `:proxy_remote` in `Pleroma.Upload`
- `:upload_limit` in `:instance`
+ - `:digest` in `:email_notifications`
+ - `:clean_expired_tokens` in `:oauth2`
+ - `:enabled` in `Pleroma.ActivityExpiration`
+ - `:enabled` in `Pleroma.ScheduledActivity`
+ - `:enabled` in `:gopher`
## How to dump settings from database to file
```sql
TRUNCATE TABLE config;
```
-2. Delete `config/{env}.exported_from_db.exs`.
+2. If migrate_from_db task was runned, backup and delete `config/{env}.exported_from_db.exs`.
For `prod` env:
```bash
* `account_field_name_length`: An account field name maximum length (default: `512`).
* `account_field_value_length`: An account field value maximum length (default: `2048`).
* `external_user_synchronization`: Enabling following/followers counters synchronization for external users.
+* `cleanup_attachments`: Remove attachments along with statuses. Does not affect duplicate files and attachments without status. Enabling this will increase load to database when deleting statuses on larger instances.
## Federation
### MRF policies
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Config.Loader do
- @paths ["config/config.exs", "config/#{Mix.env()}.exs"]
-
@reject_keys [
Pleroma.Repo,
Pleroma.Web.Endpoint,
def load_and_merge do
all_paths =
if Pleroma.Config.get(:release),
- do: @paths ++ ["config/releases.exs"],
- else: @paths
+ do: ["config/config.exs", "config/releases.exs"],
+ else: ["config/config.exs"]
all_paths
|> Enum.map(&load(&1))
require Logger
+ @type env() :: :test | :benchmark | :dev | :prod
+
+ @reboot_time_keys [
+ {:pleroma, :hackney_pools},
+ {:pleroma, :chat},
+ {:pleroma, Oban},
+ {:pleroma, :rate_limit},
+ {:pleroma, :markup},
+ {:plerome, :streamer}
+ ]
+
+ @reboot_time_subkeys [
+ {:pleroma, Pleroma.Captcha, [:seconds_valid]},
+ {:pleroma, Pleroma.Upload, [:proxy_remote]},
+ {:pleroma, :instance, [:upload_limit]},
+ {:pleroma, :email_notifications, [:digest]},
+ {:pleroma, :oauth2, [:clean_expired_tokens]},
+ {:pleroma, Pleroma.ActivityExpiration, [:enabled]},
+ {:pleroma, Pleroma.ScheduledActivity, [:enabled]},
+ {:pleroma, :gopher, [:enabled]}
+ ]
+
+ @reject [nil, :prometheus]
+
def start_link(_) do
load_and_update_env()
if Pleroma.Config.get(:env) == :test, do: Ecto.Adapters.SQL.Sandbox.checkin(Repo)
end
@spec load_and_update_env([ConfigDB.t()]) :: :ok | false
- def load_and_update_env(deleted \\ []) do
+ def load_and_update_env(deleted \\ [], restart_pleroma? \\ true) do
with true <- Pleroma.Config.get(:configurable_from_database),
true <- Ecto.Adapters.SQL.table_exists?(Repo, "config"),
started_applications <- Application.started_applications() do
# We need to restart applications for loaded settings take effect
+
in_db = Repo.all(ConfigDB)
with_deleted = in_db ++ deleted
- with_deleted
- |> Enum.map(&merge_and_update(&1))
- |> Enum.uniq()
- # TODO: some problem with prometheus after restart!
- |> Enum.reject(&(&1 in [:pleroma, nil, :prometheus]))
- |> Enum.each(&restart(started_applications, &1))
+ reject_for_restart = if restart_pleroma?, do: @reject, else: [:pleroma | @reject]
+
+ applications =
+ with_deleted
+ |> Enum.map(&merge_and_update(&1))
+ |> Enum.uniq()
+ # TODO: some problem with prometheus after restart!
+ |> Enum.reject(&(&1 in reject_for_restart))
+
+ # to be ensured that pleroma will be restarted last
+ applications =
+ if :pleroma in applications do
+ List.delete(applications, :pleroma) ++ [:pleroma]
+ else
+ applications
+ end
+
+ Enum.each(applications, &restart(started_applications, &1, Pleroma.Config.get(:env)))
:ok
end
group = ConfigDB.from_string(setting.group)
default = Pleroma.Config.Holder.config(group, key)
- merged_value = merge_value(setting, default, group, key)
+ value = ConfigDB.from_binary(setting.value)
+
+ merged_value =
+ if Ecto.get_meta(setting, :state) == :deleted do
+ default
+ else
+ if can_be_merged?(default, value) do
+ ConfigDB.merge_group(group, key, default, value)
+ else
+ value
+ end
+ end
:ok = update_env(group, key, merged_value)
if group != :logger do
- group
+ if group != :pleroma or pleroma_need_restart?(group, key, value) do
+ group
+ end
else
# change logger configuration in runtime, without restart
if Keyword.keyword?(merged_value) and
end
end
- defp merge_value(%{__meta__: %{state: :deleted}}, default, _group, _key), do: default
+ @spec pleroma_need_restart?(atom(), atom(), any()) :: boolean()
+ def pleroma_need_restart?(group, key, value) do
+ group_and_key_need_reboot?(group, key) or group_and_subkey_need_reboot?(group, key, value)
+ end
- defp merge_value(setting, default, group, key) do
- value = ConfigDB.from_binary(setting.value)
+ defp group_and_key_need_reboot?(group, key) do
+ Enum.any?(@reboot_time_keys, fn {g, k} -> g == group and k == key end)
+ end
- if can_be_merged?(default, value) do
- ConfigDB.merge_group(group, key, default, value)
- else
- value
- end
+ defp group_and_subkey_need_reboot?(group, key, value) do
+ Keyword.keyword?(value) and
+ Enum.any?(@reboot_time_subkeys, fn {g, k, subkeys} ->
+ g == group and k == key and
+ Enum.any?(Keyword.keys(value), &(&1 in subkeys))
+ end)
end
defp update_env(group, key, nil), do: Application.delete_env(group, key)
defp update_env(group, key, value), do: Application.put_env(group, key, value)
- defp restart(started_applications, app) do
+ defp restart(_, :pleroma, :test), do: Logger.warn("pleroma restarted")
+
+ defp restart(_, :pleroma, _), do: send(Restarter.Pleroma, :after_boot)
+
+ defp restart(started_applications, app, _) do
with {^app, _, _} <- List.keyfind(started_applications, app, 0),
:ok <- Application.stop(app) do
:ok = Application.start(app)
with {:ok, _obj} = swap_object_with_tombstone(object),
deleted_activity = Activity.delete_all_by_object_ap_id(id),
{:ok, true} <- Cachex.del(:object_cache, "object:#{id}"),
- {:ok, _} <- Cachex.del(:web_resp_cache, URI.parse(id).path),
- {:ok, _} <-
- Pleroma.Workers.AttachmentsCleanupWorker.enqueue("cleanup_attachments", %{
- "object" => object
- }) do
+ {:ok, _} <- Cachex.del(:web_resp_cache, URI.parse(id).path) do
+ with true <- Pleroma.Config.get([:instance, :cleanup_attachments]) do
+ {:ok, _} =
+ Pleroma.Workers.AttachmentsCleanupWorker.enqueue("cleanup_attachments", %{
+ "object" => object
+ })
+ end
+
{:ok, object, deleted_activity}
end
end
Ecto.get_meta(config, :state) == :deleted
end)
- Pleroma.Config.TransferTask.load_and_update_env(deleted)
+ Pleroma.Config.TransferTask.load_and_update_env(deleted, false)
+
+ need_reboot? =
+ Enum.any?(updated, fn config ->
+ group = ConfigDB.from_string(config.group)
+ key = ConfigDB.from_string(config.key)
+ value = ConfigDB.from_binary(config.value)
+ Pleroma.Config.TransferTask.pleroma_need_restart?(group, key, value)
+ end)
- Mix.Tasks.Pleroma.Config.run([
- "migrate_from_db",
- "--env",
- to_string(Pleroma.Config.get(:env))
- ])
+ response = %{configs: updated}
+
+ response =
+ if need_reboot?, do: Map.put(response, :need_reboot, need_reboot?), else: response
conn
|> put_view(ConfigView)
- |> render("index.json", %{configs: updated})
+ |> render("index.json", response)
+ end
+ end
+
+ def restart(conn, _params) do
+ with :ok <- configurable_from_database(conn) do
+ if Pleroma.Config.get(:env) == :test do
+ Logger.warn("pleroma restarted")
+ else
+ send(Restarter.Pleroma, {:restart, 50})
+ end
+
+ json(conn, %{})
end
end
defmodule Pleroma.Web.AdminAPI.ConfigView do
use Pleroma.Web, :view
- def render("index.json", %{configs: configs}) do
- %{
+ def render("index.json", %{configs: configs} = params) do
+ map = %{
configs: render_many(configs, __MODULE__, "show.json", as: :config)
}
+
+ if params[:need_reboot] do
+ Map.put(map, :need_reboot, true)
+ else
+ map
+ end
end
def render("show.json", %{config: config}) do
post("/config", AdminAPIController, :config_update)
get("/config/descriptions", AdminAPIController, :config_descriptions)
get("/config/migrate_from_db", AdminAPIController, :migrate_from_db)
+ get("/restart", AdminAPIController, :restart)
get("/moderation_log", AdminAPIController, :list_log)
elixir: "~> 1.8",
elixirc_paths: elixirc_paths(Mix.env()),
compilers: [:phoenix, :gettext] ++ Mix.compilers(),
- elixirc_options: [warnings_as_errors: true],
+ elixirc_options: [warnings_as_errors: warnings_as_errors(Mix.env())],
xref: [exclude: [:eldap]],
start_permanent: Mix.env() == :prod,
aliases: aliases(),
defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_), do: ["lib"]
+ defp warnings_as_errors(:prod), do: false
+ # Uncomment this if you need testing configurable_from_database logic
+ # defp warnings_as_errors(:dev), do: false
+ defp warnings_as_errors(_), do: true
+
# Specifies OAuth dependencies.
defp oauth_deps do
oauth_strategy_packages =
{:captcha,
git: "https://git.pleroma.social/pleroma/elixir-libraries/elixir-captcha.git",
ref: "e0f16822d578866e186a0974d65ad58cddc1e2ab"},
- {:mox, "~> 0.5", only: :test}
+ {:mox, "~> 0.5", only: :test},
+ {:restarter, path: "./restarter"}
] ++ oauth_deps()
end
--- /dev/null
+defmodule Restarter.Pleroma do
+ use GenServer
+
+ def start_link(_) do
+ GenServer.start_link(__MODULE__, [], name: __MODULE__)
+ end
+
+ def init(_), do: {:ok, %{}}
+
+ def handle_info(:after_boot, %{after_boot: true} = state), do: {:noreply, state}
+
+ def handle_info(:after_boot, state) do
+ restart(:pleroma)
+ {:noreply, Map.put(state, :after_boot, true)}
+ end
+
+ def handle_info({:restart, delay}, state) do
+ Process.sleep(delay)
+ restart(:pleroma)
+ {:noreply, state}
+ end
+
+ defp restart(app) do
+ :ok = Application.ensure_started(app)
+ :ok = Application.stop(app)
+ :ok = Application.start(app)
+ end
+end
--- /dev/null
+defmodule Restarter do
+ use Application
+
+ def start(_, _) do
+ opts = [strategy: :one_for_one, name: Restarter.Supervisor]
+ Supervisor.start_link([Restarter.Pleroma], opts)
+ end
+end
--- /dev/null
+defmodule Restarter.MixProject do
+ use Mix.Project
+
+ def project do
+ [
+ app: :restarter,
+ version: "0.1.0",
+ elixir: "~> 1.8",
+ start_permanent: Mix.env() == :prod,
+ deps: deps()
+ ]
+ end
+
+ def application do
+ [
+ mod: {Restarter, []}
+ ]
+ end
+
+ defp deps, do: []
+end
defmodule Pleroma.Config.TransferTaskTest do
use Pleroma.DataCase
+ import ExUnit.CaptureLog
+
alias Pleroma.Config.TransferTask
alias Pleroma.ConfigDB
Application.put_env(:pleroma, :assets, assets)
end)
end
+
+ describe "pleroma restart" do
+ test "don't restart if no reboot time settings were changed" do
+ emoji = Application.get_env(:pleroma, :emoji)
+ on_exit(fn -> Application.put_env(:pleroma, :emoji, emoji) end)
+
+ ConfigDB.create(%{
+ group: ":pleroma",
+ key: ":emoji",
+ value: [groups: [a: 1, b: 2]]
+ })
+
+ refute String.contains?(
+ capture_log(fn -> TransferTask.start_link([]) end),
+ "pleroma restarted"
+ )
+ end
+
+ test "restart pleroma on reboot time key" do
+ chat = Application.get_env(:pleroma, :chat)
+ on_exit(fn -> Application.put_env(:pleroma, :chat, chat) end)
+
+ ConfigDB.create(%{
+ group: ":pleroma",
+ key: ":chat",
+ value: [enabled: false]
+ })
+
+ assert capture_log(fn -> TransferTask.start_link([]) end) =~ "pleroma restarted"
+ end
+
+ test "restart pleroma on reboot time subkey" do
+ captcha = Application.get_env(:pleroma, Pleroma.Captcha)
+ on_exit(fn -> Application.put_env(:pleroma, Pleroma.Captcha, captcha) end)
+
+ ConfigDB.create(%{
+ group: ":pleroma",
+ key: "Pleroma.Captcha",
+ value: [seconds_valid: 60]
+ })
+
+ assert capture_log(fn -> TransferTask.start_link([]) end) =~ "pleroma restarted"
+ end
+
+ test "don't restart pleroma on reboot time key and subkey if there is false flag" do
+ chat = Application.get_env(:pleroma, :chat)
+ captcha = Application.get_env(:pleroma, Pleroma.Captcha)
+
+ on_exit(fn ->
+ Application.put_env(:pleroma, :chat, chat)
+ Application.put_env(:pleroma, Pleroma.Captcha, captcha)
+ end)
+
+ ConfigDB.create(%{
+ group: ":pleroma",
+ key: ":chat",
+ value: [enabled: false]
+ })
+
+ ConfigDB.create(%{
+ group: ":pleroma",
+ key: "Pleroma.Captcha",
+ value: [seconds_valid: 60]
+ })
+
+ refute String.contains?(
+ capture_log(fn -> TransferTask.load_and_update_env([], false) end),
+ "pleroma restarted"
+ )
+ end
+ end
end
describe "delete attachments" do
clear_config([Pleroma.Upload])
+ test "Disabled via config" do
+ Pleroma.Config.put([Pleroma.Upload, :uploader], Pleroma.Uploaders.Local)
+ Pleroma.Config.put([:instance, :cleanup_attachments], false)
+
+ file = %Plug.Upload{
+ content_type: "image/jpg",
+ path: Path.absname("test/fixtures/image.jpg"),
+ filename: "an_image.jpg"
+ }
+
+ user = insert(:user)
+
+ {:ok, %Object{} = attachment} =
+ Pleroma.Web.ActivityPub.ActivityPub.upload(file, actor: user.ap_id)
+
+ %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} =
+ note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
+
+ uploads_dir = Pleroma.Config.get!([Pleroma.Uploaders.Local, :uploads])
+
+ path = href |> Path.dirname() |> Path.basename()
+
+ assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+
+ Object.delete(note)
+
+ ObanHelpers.perform(all_enqueued(worker: Pleroma.Workers.AttachmentsCleanupWorker))
+
+ assert Object.get_by_id(note.id).data["deleted"]
+ refute Object.get_by_id(attachment.id) == nil
+
+ assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+ end
+
test "in subdirectories" do
Pleroma.Config.put([Pleroma.Upload, :uploader], Pleroma.Uploaders.Local)
+ Pleroma.Config.put([:instance, :cleanup_attachments], true)
file = %Plug.Upload{
content_type: "image/jpg",
ObanHelpers.perform(all_enqueued(worker: Pleroma.Workers.AttachmentsCleanupWorker))
+ assert Object.get_by_id(note.id).data["deleted"]
assert Object.get_by_id(attachment.id) == nil
assert {:ok, []} == File.ls("#{uploads_dir}/#{path}")
test "with dedupe enabled" do
Pleroma.Config.put([Pleroma.Upload, :uploader], Pleroma.Uploaders.Local)
Pleroma.Config.put([Pleroma.Upload, :filters], [Pleroma.Upload.Filter.Dedupe])
+ Pleroma.Config.put([:instance, :cleanup_attachments], true)
uploads_dir = Pleroma.Config.get!([Pleroma.Uploaders.Local, :uploads])
ObanHelpers.perform(all_enqueued(worker: Pleroma.Workers.AttachmentsCleanupWorker))
+ assert Object.get_by_id(note.id).data["deleted"]
assert Object.get_by_id(attachment.id) == nil
assert {:ok, files} = File.ls(uploads_dir)
refute filename in files
test "with objects that have legacy data.url attribute" do
Pleroma.Config.put([Pleroma.Upload, :uploader], Pleroma.Uploaders.Local)
+ Pleroma.Config.put([:instance, :cleanup_attachments], true)
file = %Plug.Upload{
content_type: "image/jpg",
ObanHelpers.perform(all_enqueued(worker: Pleroma.Workers.AttachmentsCleanupWorker))
+ assert Object.get_by_id(note.id).data["deleted"]
assert Object.get_by_id(attachment.id) == nil
assert {:ok, []} == File.ls("#{uploads_dir}/#{path}")
test "With custom base_url" do
Pleroma.Config.put([Pleroma.Upload, :uploader], Pleroma.Uploaders.Local)
Pleroma.Config.put([Pleroma.Upload, :base_url], "https://sub.domain.tld/dir/")
+ Pleroma.Config.put([:instance, :cleanup_attachments], true)
file = %Plug.Upload{
content_type: "image/jpg",
ObanHelpers.perform(all_enqueued(worker: Pleroma.Workers.AttachmentsCleanupWorker))
+ assert Object.get_by_id(note.id).data["deleted"]
assert Object.get_by_id(attachment.id) == nil
assert {:ok, []} == File.ls("#{uploads_dir}/#{path}")
Application.delete_env(:pleroma, Pleroma.Captcha.NotReal)
Application.put_env(:pleroma, :http, http)
Application.put_env(:tesla, :adapter, Tesla.Mock)
- :ok = File.rm("config/test.exported_from_db.secret.exs")
end)
end
assert Application.get_env(:idna, :key5) == {"string", Pleroma.Captcha.NotReal, []}
end
- test "save config setting without key", %{conn: conn} do
+ test "save configs setting without explicit key", %{conn: conn} do
level = Application.get_env(:quack, :level)
meta = Application.get_env(:quack, :meta)
webhook_url = Application.get_env(:quack, :webhook_url)
}
end
+ test "saving config which need pleroma reboot", %{conn: conn} do
+ chat = Pleroma.Config.get(:chat)
+ on_exit(fn -> Pleroma.Config.put(:chat, chat) end)
+
+ conn =
+ post(
+ conn,
+ "/api/pleroma/admin/config",
+ %{
+ configs: [
+ %{group: ":pleroma", key: ":chat", value: [%{"tuple" => [":enabled", true]}]}
+ ]
+ }
+ )
+
+ assert json_response(conn, 200) == %{
+ "configs" => [
+ %{
+ "db" => [":enabled"],
+ "group" => ":pleroma",
+ "key" => ":chat",
+ "value" => [%{"tuple" => [":enabled", true]}]
+ }
+ ],
+ "need_reboot" => true
+ }
+ end
+
test "saving config with nested merge", %{conn: conn} do
config =
insert(:config, key: ":key1", value: :erlang.term_to_binary(key1: 1, key2: [k1: 1, k2: 2]))
end
end
+ describe "GET /api/pleroma/admin/restart" do
+ clear_config(:configurable_from_database) do
+ Pleroma.Config.put(:configurable_from_database, true)
+ end
+
+ test "pleroma restarts", %{conn: conn} do
+ ExUnit.CaptureLog.capture_log(fn ->
+ assert conn |> get("/api/pleroma/admin/restart") |> json_response(200) == %{}
+ end) =~ "pleroma restarted"
+ end
+ end
+
describe "GET /api/pleroma/admin/users/:nickname/statuses" do
setup do
user = insert(:user)
projects / akkoma / commitdiff