# Background Services

Automations and logs workers with BullMQ and Redis


When to run the `logs-service` and `automations-service` workers alongside the APIs, and how work reaches them.

For local startup, see [Local Setup](https://erxes.io/docs/local-setup). For queue inspection, see [Logs & Troubleshooting](https://erxes.io/docs/self-hosting-troubleshooting). For gateway registration mechanics, see [Backend & API Gateway](https://erxes.io/docs/architecture-backend-gateway).

## The two services

| Service | Nx project | Port | Registers in Redis | Queues read |
| --- | --- | --- | --- | --- |
| Logs | `logs-service` (`backend/services/logs`) | `3301` | `erxes-service-logs` | `logs-put_log`, `logs-activity_log`, `logs-segment-{changed,forget,rebuild,reconcile}` |
| Automations | `automations-service` (`backend/services/automations`) | `3302` | `erxes-service-automations` | `automations-trigger`, `automations-action`, `automations-aiAgent` |

Both are plain Express apps exposing `/health` (JSON via `createHealthRoute`) but **no GraphQL subgraph**. They do not appear in `getPlugins()`, so the gateway never waits for them or routes GraphQL to them. Each registers itself manually on startup: the address under `erxes-service-<name>` and `{"dbConnectionString": MONGO_URL}` under `erxesservice:config:<service-name>`, using `LOAD_BALANCER_ADDRESS` when set.

The automations service additionally mounts webhook routes (`webhookRoutes`) and a `/trpc` router serving the `automations.trigger` and `automations.completeDeferredAction` procedures that plugins call back into.

## How work reaches the queues

All queues are BullMQ queues on the shared Redis, named `<service>-<queue>`. Producers call `sendWorkerQueue(service, queue)`; workers call `createMQWorkerWithListeners(service, queue, handler)`: both live in [mq-worker.ts](https://github.com/erxes/erxes/blob/f91e8c6f4714c910a32c6b6e14fe085f6435c89e/backend/erxes-api-shared/src/utils/mq-worker.ts). Notable producers:

- `sendWorkerQueue('logs', 'put_log')`: event/audit log entries written by `logHandler`, which wraps plugin `apiHandlers` and log-enabled code paths. `logHandler` first checks `checkServiceRunning('logs')` (the `erxes-service-logs` key); with no logs service registered, the work still executes but nothing is persisted.
- `sendWorkerQueue('logs', 'activity_log')`: activity feed entries.
- `sendWorkerQueue('logs', 'segment-*')`: segment membership jobs produced through `SEGMENT_QUEUES` in `core-modules/segments`.
- `sendWorkerQueue('automations', 'trigger')`: automation trigger dispatch from `sendAutomationTrigger` in `core-modules/automations`.
- `sendWorkerQueue('gateway', 'update-apollo-router')`: production supergraph recomposition; see [Backend & API Gateway](https://erxes.io/docs/architecture-backend-gateway).

The gateway also creates a `gateway-service-discovery` queue for its Bull Board UI at `/bullmq-board`.

## How plugins register automation and log behavior

Plugins declare background behavior in the `meta` object passed to `startPlugin`: no direct queue access needed:

- `meta.automations`: `startAutomations` writes the config into `erxesservice:config:<plugin>.meta.automations` and mounts typed tRPC producers on the plugin's `/trpc` router (`receiveActions`, `setProperties`, `checkCustomTrigger`, `findObject`, `generateAiContext`, `loadAiKnowledgeDocumentBatch`, `lookupAiTool`). The automations service reads every plugin's meta from Redis (`getPlugin(name).config.meta.automations`) and calls these procedures while executing triggers and actions.
- `meta.afterProcess`: registers `afterMutation`, `afterAuth`, `afterApiRequest`, `afterDocumentUpdated`, and `afterDocumentCreated` producers the same way.
- `meta.logs`: declares log content types so event and activity entries are labeled correctly.
- `meta.segments`: `initSegmentProducers` registers the plugin's segment producers.
- `meta.importExport`: starts an in-process import/export worker on the plugin.

## Data layout

The logs service resolves tenant models per subdomain like an API, then splits storage:

- `activityLogs` and `users` live in the tenant database (same as the app DB).
- `logs` documents go to a separate database named `<tenant-db>_logs` via `db.useDb(...)` in [connectionResolvers.ts](https://github.com/erxes/erxes/blob/f91e8c6f4714c910a32c6b6e14fe085f6435c89e/backend/services/logs/src/connectionResolvers.ts); for `MONGO_URL=.../erxes`, that is `erxes_logs`.

This is also where the automatic undo journal lands: writes through `erxes-api-shared` models are journaled to `<tenant>_logs` so System Logs → Undo can revert updates and deletes. The journal capture always runs; without the logs service, entries are queued but never persisted. `LOG_RETENTION_DAYS` (default `365`) sets the log document TTL.

## Enabling services locally

[scripts/start-api-dev.js](https://github.com/erxes/erxes/blob/f91e8c6f4714c910a32c6b6e14fe085f6435c89e/scripts/start-api-dev.js) (behind `pnpm dev:apis`) reads three variables and runs `nx run-many -t serve -p core-api <plugin>_api <service>-service gateway`:

```dotenv
ENABLED_PLUGINS=sales
ENABLED_PLUGINS_ONLY_API=posclient
ENABLED_SERVICES=automations,logs
```

`ENABLED_SERVICES` entries are mapped to `<name>-service` Nx projects (`automations` → `automations-service`). Use base names with no `_api`/`_ui`/`_service` suffixes, and restart `pnpm dev:apis` after changing the list.

## Production notes

CI publishes `erxes/erxes-next-logs` and `erxes/erxes-next-automations` images. Deploy a container per service you need with the same `MONGO_URL`, `REDIS_*`, `JWT_TOKEN_SECRET`, and `DOMAIN` environment as the APIs. A gateway plus Core API alone does not persist logs or execute automations. `ENABLED_SERVICES` is a development-startup variable; it does not create containers.

Segment worker concurrency is tunable per queue: `SEGMENT_CONCURRENCY_CHANGED` (10), `SEGMENT_CONCURRENCY_FORGET` (5), `SEGMENT_CONCURRENCY_REBUILD` (1), `SEGMENT_CONCURRENCY_RECONCILE` (3).