> For the complete documentation index, see [llms.txt](https://docs.codegiganten.de/plugin-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/push-and-live-sync.md).

# Push and live synchronisation

How to transfer your products by push, which background processing has to run for it, and how live synchronisation brings changes into the search within about two minutes.

With push, your shop sends its products to our search system itself. This is the way to go for development and staging environments that cannot be reached from the outside. This page shows when you need push, how a run works, what has to run on the server and how to get changes to prices and stock into the search quickly.

The page is also meant for agencies and administrators. Technical terms are explained briefly the first time they appear, commands are shown in code blocks.

## When push, when pull?

There are two ways to get your products into the search:

|                              | Pull (feed)                                                      | Push                                                                                 |
| ---------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Who fetches or sends?        | Our search system retrieves your shop's product feed             | Your shop sends the products itself                                                  |
| Shop reachable from outside? | Yes, the feed address must be public and reachable without login | No, not required                                                                     |
| Suitable for                 | Live shops                                                       | Development and staging environments, shops behind password protection or a firewall |
| Timing                       | Once a day at the configured time, manually at any time          | Daily through a scheduled task, manually at any time                                 |
| Live synchronisation         | Not available                                                    | Available                                                                            |
| Background processing        | Only for statistics messages                                     | Mandatory                                                                            |

You choose the way in the setup wizard under indexing type: "Push: The plugin sends the products (for DEV/Stage)". With push, the plugin does not create a product comparison sales channel or a product export. Instead it stores the settings (sales channel, domain, customer group, dynamic product group, language, currency) in the shop itself.

{% hint style="info" %}
How to set up and check a pull feed is described in [Installation and setup](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/installation-and-setup.md). For staging environments with password protection, push is the right way, see [Stage and Live](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/stage-and-live.md).
{% endhint %}

## How a push run works

1. **Register:** The plugin creates the push index in our search system, with sales channel, domain and customer group. This way, configurations with the same name for different channels each get their own index.
2. **Start:** A run begins in the setup wizard, with "Re-import product data" in the "Sources" area, or through the daily scheduled task. The plugin checks credentials and settings and counts the products. If there is no matching product, the run does not start ("No products found"). Then the plugin splits the products into **packages of 50 products each**.
3. **Send packages:** The packages are processed in the background. For each package the plugin loads the products (by sales channel, visibility "Search" or "All", active status, dynamic product group, currency, language and customer group) and sends them. In the "Sources" area you see the progress with percentage, estimated remaining time and throughput (products per second).
4. **Finish:** When all packages are processed, the plugin reports the run as finished. Our search system then completes the search index.
5. **Cancel or restart:** "Cancel" ends a running run after a confirmation ("Really cancel the running indexing?"). Afterwards you start again with "Restart".

Good to know:

* Only **one** push run per installation runs at a time. A second start is rejected.
* A run without any sign of life for 15 minutes counts as dead and may be restarted.
* Products that are visible in the search only (Shopware visibility "search only") are transferred and do not appear on category pages.
* Variants without a category of their own take over the categories of the main product.
* A push is never repeated automatically. You restart a failed run by hand with "Try again" or "Restart".
* Manually started runs are possible at any time, as long as the data source is not locked by InsightSearch and no run is active yet.

## Background processing (worker)

Push does not run directly when you click, but through the **message queue** of Shopware. That is a waiting line into which the shop puts jobs. A **worker** is a permanently running process that works through these jobs. Without a worker, a push run simply stops. The messages about cart, order and payment status for the statistics also go through this queue.

In the "Sources" area the plugin points this out itself: Shopware's background processing must run permanently, either with an active admin worker or with `messenger:consume` as a service.

### Starting a worker

For permanent operation, a worker as a service is the right choice, which Supervisor or systemd restarts after it ends:

```bash
bin/console messenger:consume async low_priority --time-limit=3600 --memory-limit=512M
```

* `async` is the queue the plugin uses. Add `low_priority` as Shopware recommends, so that no messages are left lying around.
* `--time-limit=3600` ends the worker cleanly after one hour, the service restarts it.
* `--memory-limit=512M` limits the memory of the worker. The exact values depend on your server and the Shopware documentation.

The **admin worker** of Shopware only runs while an administrator has the admin open in the browser. It is enough for a test, not for permanent operation.

### Memory limit

Large catalogues need a lot of memory when packages are processed. The plugin protects the worker from crashing hard:

| Situation                                                                          | Reaction                                                             |
| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| The worker uses at least 70 % of the PHP memory limit and less than 64 MB are free | It ends cleanly after the current message, a fresh worker takes over |
| The worker uses 80 % or more and less than 48 MB are free                          | A new package no longer begins, the worker ends first                |
| The memory limit is unlimited (`-1`)                                               | The protection does not apply                                        |

The log then says "\[WorkerMemoryGuard] Worker wird wegen Speicherverbrauch beendet und neu gestartet" (the log text is German). For the restart to work, a service such as Supervisor or systemd must start the worker again. The memory limit refers to PHP on the command line (CLI), not to the limit for web pages.

{% hint style="warning" %}
If a worker aborts hard, for example through a PHP error caused by too little memory, the message stays invisible in the queue. It only returns after one hour. This was the cause of a run that stopped at 96 percent.
{% endhint %}

### Restart after updates

After a plugin update or a configuration change, running workers should load the new code:

```bash
bin/console messenger:stop-workers
```

Supervisor or systemd then start them again.

## Scheduled tasks of the plugin

The plugin uses the **scheduled tasks** of Shopware. These must run permanently, for example by cron or as a service with `scheduled-task:run`. Without them there is no daily push, no monitoring of stuck runs, no connection check, no catching up on orders and no live synchronisation.

| Task                     | Purpose                                                                                            | Interval         |
| ------------------------ | -------------------------------------------------------------------------------------------------- | ---------------- |
| Daily push               | Transfers all products of all push indexes completely. Skips an index whose run is already active. | every 24 hours   |
| Monitoring of stuck runs | Checks runs without progress, requeues packages or completes a run (see below)                     | every 10 minutes |
| Connection check         | Reports a sign of life, compares purchased additional functions and the list of indexes            | hourly           |
| Live synchronisation     | Transfers queued product changes, only if the switch is on                                         | every minute     |
| Catching up on orders    | Fetches orders of the last 7 days that our search system does not know yet                         | every 6 hours    |

The daily push has no adjustable time. It runs every 24 hours from the last run. The setting "Automatic indexing (time)" in the "Sources" area only applies to pull.

You register and start tasks with the Shopware commands described on the page [Commands](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/commands.md).

## When the push is stuck at 96 percent

A run can stop, for example because a worker crashed or none is running. The plugin monitors this and helps itself. The monitoring runs every 10 minutes and counts from the last progress:

| Time without progress                             | What happens                                                                                                                                                                                                                                                                           |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| from about 4 minutes                              | Open packages are requeued automatically, up to 3 times per package and at most 50 packages per pass, without sending or counting twice. The "Sources" area shows: "There has been no progress for a few minutes. Stuck packages are requeued automatically, then indexing continues." |
| 10 minutes                                        | The display warns that no progress has been visible for over 10 minutes                                                                                                                                                                                                                |
| 15 minutes, no more packages can be requeued      | The run counts as failed ("Indexing is stuck (no progress). Please start again.") and can be restarted                                                                                                                                                                                 |
| 1 hour, if no worker is running at all            | The run counts as failed and can be restarted                                                                                                                                                                                                                                          |
| All packages done, completion missing (5 minutes) | The monitoring completes the run itself                                                                                                                                                                                                                                                |

### What you can check

1. Is a worker running? Check the status of Supervisor or systemd, or whether an admin is open in the browser.
2. Is the PHP memory limit of the worker process (CLI) sufficient? For very large catalogues, increase it.
3. Is `scheduled-task:run` running? Without the scheduled tasks, the monitoring does not run.
4. Does the "Sources" area show the note about automatic requeuing?
5. Look into the plugin log under `var/log` (file `cogi_insightsearch-<date>.log`) for entries with "\[PushIndexing]", "\[PushFinalize]" and "\[WorkerMemoryGuard]".
6. Then start the run again with "Restart".

More steps are on the page [Troubleshooting](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/troubleshooting.md).

## Live synchronisation

Without live synchronisation, visitors only see changes to prices, stock or visibility after the next push run, that is, up to a day later. With live synchronisation they are in the search after **about two minutes**.

### Setting

You find the option in the plugin configuration in the card "Live synchronisation":

| Setting                                                    | Effect                                                                         | Default |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------ | ------- |
| "Transfer price, stock and visibility changes immediately" | Changed and deleted products are taken into the search after about two minutes | off     |

{% hint style="warning" %}
Set the switch under "All sales channels". It always applies to all sales channels. A value that you set for a single sales channel is not evaluated.
{% endhint %}

Live synchronisation applies to **push indexes only**. In pull operation nothing happens: there, the next retrieval of the feed brings the changes.

### What is transferred

The plugin notes the products when price, stock, visibility, active status or categories change, when an order changes the stock, or when a product is deleted. The scheduled task runs every minute and sends entries that are at least 20 seconds old. This way, several changes to the same product are combined.

* Products that no longer belong to the scope of the index (invisible, inactive, not in the dynamic product group) are removed from the search.
* If a main product is deleted, its variants disappear with it.
* A product that is deleted and immediately created again with the same ID (for example in an ERP import) stays in the search.

### Pauses and protection

Live synchronisation pauses itself when something is wrong, so that no load builds up:

| Situation                       | Reaction                                                                                                            |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Credentials are rejected        | 1 hour pause                                                                                                        |
| Other error or server error     | 5 minute pause                                                                                                      |
| Too many requests (throttling)  | Wait as requested, at least 30 seconds. If only one index is blocked, the others continue.                          |
| More than 20,000 queued changes | The queued entries are discarded, the next complete push takes over everything (message "Too many changes at once") |

Prerequisites are running scheduled tasks and a running worker.

{% hint style="info" %}
Large imports, for example a rebuild of the Shopware indexes with `dal:refresh:index`, can queue very many changes. Plan such runs deliberately when live synchronisation is active.
{% endhint %}

## Further reading

{% content-ref url="/pages/kRomH0o5aBX5iDGMqpD3" %}
[Stage and live](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/stage-and-live.md)
{% endcontent-ref %}

{% content-ref url="/pages/mU4m1YpoKatMe6lLhFMB" %}
[Commands](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/commands.md)
{% endcontent-ref %}

{% content-ref url="/pages/cPKiMg6nNvV0NE0owXnG" %}
[Troubleshooting](https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/troubleshooting.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.codegiganten.de/plugin-documentation/en/apps/search-intelligent-and-ai-optimized/push-and-live-sync.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
