Current File : //www/varak.net/nextcloud.varak.net/core/doc/admin/ai/overview.html
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Overview — Nextcloud latest Administration Manual latest documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=19f00094" />
<link rel="stylesheet" type="text/css" href="../_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="../_static/dark_mode_css/general.css?v=c0a7eb24" />
<link rel="stylesheet" type="text/css" href="../_static/dark_mode_css/dark.css?v=70edf1c7" />
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=c6e86fd7"></script>
<script src="../_static/doctools.js?v=888ff710"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/clipboard.min.js?v=a7894cd8"></script>
<script src="../_static/copybutton.js?v=f281be69"></script>
<script src="../_static/dark_mode_js/default_light.js?v=c2e647ce"></script>
<script src="../_static/dark_mode_js/theme_switcher.js?v=358d3910"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Nextcloud Assistant" href="app_assistant.html" />
<link rel="prev" title="Artificial Intelligence" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../contents.html">
<img src="../_static/logo-white.png" class="logo" alt="Logo"/>
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../index.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release_notes/index.html">Release notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release_schedule.html">Maintenance and release schedule</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/index.html">Installation and server configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration_server/index.html">Nextcloud configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../apps_management.html">Apps management</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration_user/index.html">User management</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration_files/index.html">File sharing and management</a></li>
<li class="toctree-l1"><a class="reference internal" href="../file_workflows/index.html">Flow</a></li>
<li class="toctree-l1"><a class="reference internal" href="../groupware/index.html">Groupware</a></li>
<li class="toctree-l1"><a class="reference internal" href="../office/index.html">Office</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/index.html">Reference management</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Artificial Intelligence</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">Overview</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#overview-of-ai-features">Overview of AI features</a></li>
<li class="toctree-l3"><a class="reference internal" href="#ethical-ai-rating">Ethical AI Rating</a></li>
<li class="toctree-l3"><a class="reference internal" href="#features-used-by-other-apps">Features used by other apps</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#text-processing">Text processing</a></li>
<li class="toctree-l4"><a class="reference internal" href="#machine-translation">Machine translation</a></li>
<li class="toctree-l4"><a class="reference internal" href="#speech-to-text">Speech-To-Text</a></li>
<li class="toctree-l4"><a class="reference internal" href="#image-generation">Image generation</a></li>
<li class="toctree-l4"><a class="reference internal" href="#context-chat">Context Chat</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#improve-ai-task-pickup-speed">Improve AI task pickup speed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#screen-or-tmux-session">Screen or tmux session</a></li>
<li class="toctree-l4"><a class="reference internal" href="#systemd-service">Systemd service</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#frequently-asked-questions">Frequently Asked Questions</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#why-is-my-prompt-slow">Why is my prompt slow?</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="app_assistant.html">Nextcloud Assistant</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_translate2.html">App: Local Machine translation 2 (translate2)</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_llm2.html">App: Local large language model (llm2)</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_stt_whisper2.html">App: Local Whisper Speech-To-Text (stt_whisper2)</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_recognize.html">App: Recognize</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_context_chat.html">App: Context Chat</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_summary_bot.html">App: Summary Bot (Talk chat summarize bot)</a></li>
<li class="toctree-l2"><a class="reference internal" href="app_api_and_external_apps.html">AppAPI and External Apps</a></li>
<li class="toctree-l2"><a class="reference internal" href="ai_as_a_service.html">AI as a Service</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../webhook_listeners/index.html">Webhook Listeners</a></li>
<li class="toctree-l1"><a class="reference internal" href="../windmill_workflows/index.html">Windmill Workflows</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration_database/index.html">Database configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration_mimetypes/index.html">Mimetypes management</a></li>
<li class="toctree-l1"><a class="reference internal" href="../maintenance/index.html">Maintenance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../issues/index.html">Issues and troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../gdpr/index.html">GDPR-compliance</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../contents.html">Nextcloud latest Administration Manual</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content style-external-links">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../contents.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html">Artificial Intelligence</a></li>
<li class="breadcrumb-item active">Overview</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/nextcloud/documentation/edit/master/admin_manual/ai/overview.rst" class="fa fa-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="overview">
<h1>Overview<a class="headerlink" href="#overview" title="Link to this heading"></a></h1>
<p>We strive to bring Artificial Intelligence features to Nextcloud. This section highlights these features, how they work and where to find them.
All of these features are completely optional. If you want to have them on your server, you need install them via separate Nextcloud Apps.</p>
<section id="overview-of-ai-features">
<h2>Overview of AI features<a class="headerlink" href="#overview-of-ai-features" title="Link to this heading"></a></h2>
<p>Nextcloud uses modularity to separate raw AI functionality from the Graphical User interfaces and apps that make use of said functionality. Each instance can thus make use of various backends that provide the functionality for the same frontends and the same functionality can be implemented by multiple apps using on-premises processing or third-party AI service providers.</p>
<table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head"><p>Feature</p></th>
<th class="head"><p>App</p></th>
<th class="head"><p>Rating</p></th>
<th class="head"><p>Open source</p></th>
<th class="head"><p>Freely available model</p></th>
<th class="head"><p>Freely available training data</p></th>
<th class="head"><p>Privacy: Keeps data on premises</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>Smart inbox</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/mail">Mail</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Image object recognition</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/recognize">Recognize</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td><p>Image face recognition</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/recognize">Recognize</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Video action recognition</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/recognize">Recognize</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td><p>Audio music genre recognition</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/recognize">Recognize</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Suspicious login detection</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/suspicious_login">Suspicious Login</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td><p>Related resources</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/related_resources">Related Resources</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Recommended files</p></td>
<td><p>recommended_files</p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td><p>Text processing using LLMs</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/llm2">llm2 (ExApp)</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes - Llama 3.1 model by Meta</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a></p></td>
<td><p>Red</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via LocalAI)</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - e.g. Llama models by Meta</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Machine translation</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/translate2">Local Machine Translation 2 (ExApp)</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes - MADLAD models by Google</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_deepl">DeepL integration</a></p></td>
<td><p>Red</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-odd"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a></p></td>
<td><p>Red</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via LocalAI)</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Speech to Text</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/stt_whisper2">Local Whisper Speech-To-Text 2 (ExApp)</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - Whisper models by OpenAI</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - Whisper models by OpenAI</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-odd"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_replicate">Replicate integration</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - Whisper models by OpenAI</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-even"><td><p>Image generation</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/text2image_stablediffusion">Local Stable Diffusion</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - StableDiffusion XL model by StabilityAI</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a></p></td>
<td><p>Red</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via LocalAI)</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - StableDiffusion models by StabilityAI</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_replicate">Replicate integration</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes - StableDiffusion models by StabilityAI</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/llm2">Local large language model 2 (ExApp)</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a></p></td>
<td><p>Red</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
<td><p>No</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via LocalAI)</a></p></td>
<td><p>Green</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p>Context Chat</p></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/context_chat">Nextcloud Assistant Context Chat</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-even"><td></td>
<td><p><a class="reference external" href="https://apps.nextcloud.com/apps/context_chat_backend">Nextcloud Assistant Context Chat (Backend)</a></p></td>
<td><p>Yellow</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
</tr>
</tbody>
</table>
</section>
<section id="ethical-ai-rating">
<h2>Ethical AI Rating<a class="headerlink" href="#ethical-ai-rating" title="Link to this heading"></a></h2>
<p>Until Hub 3, we succeeded in offering features without relying on proprietary blobs or third party services. Yet, while there is a large community developing ethical, safe and privacy-respecting technologies, there are many other relevant technologies users might want to use. We want to provide users with these cutting-edge technologies – but also be transparent. For some use cases, ChatGPT might be a reasonable solution, while for more private, professional or sensitive data, it is paramount to have a local, on-prem, open solution. To differentiate these, we developed an Ethical AI Rating.</p>
<dl class="simple">
<dt>The rating has four levels:</dt><dd><ul class="simple">
<li><p>Red</p></li>
<li><p>Orange</p></li>
<li><p>Yellow</p></li>
<li><p>Green</p></li>
</ul>
</dd>
<dt>It is based on points from these factors:</dt><dd><ul class="simple">
<li><p>Is the software (both for inferencing and training) under a free and open source license?</p></li>
<li><p>Is the trained model freely available for self-hosting?</p></li>
<li><p>Is the training data available and free to use?</p></li>
</ul>
</dd>
</dl>
<p>If all of these points are met, we give a Green label. If none are met, it is Red. If 1 condition is met, it is Orange and if 2 conditions are met, Yellow.</p>
</section>
<section id="features-used-by-other-apps">
<h2>Features used by other apps<a class="headerlink" href="#features-used-by-other-apps" title="Link to this heading"></a></h2>
<p>Some of our AI features are realized as generic APIs that any app can use and any app can provide an implementation for by registering a provider. So far, these are
Machine translation, Speech-To-Text, Image generation, Text processing and Context Chat.</p>
<section id="text-processing">
<h3>Text processing<a class="headerlink" href="#text-processing" title="Link to this heading"></a></h3>
<p id="tp-consumer-apps">As you can see in the table above we have multiple apps offering text processing using Large language models.
In downstream apps like Context Chat and assistant, users can use the text processing functionality regardless of which app implements it behind the scenes.</p>
<section id="frontend-apps">
<h4>Frontend apps<a class="headerlink" href="#frontend-apps" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><em>Text</em> for offering an inline graphical UI for the various tasks</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/assistant">Assistant</a> for offering a graphical UI for the various tasks and a smart picker</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/mail">Mail</a> for summarizing mail threads (see <a class="reference internal" href="../groupware/mail.html#mail-thread-summary"><span class="std std-ref">the Nextcloud Mail docs</span></a> for how to enable this)</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/summary_bot">Summary Bot</a> for summarizing chat histories in <a class="reference external" href="https://apps.nextcloud.com/apps/spreed">Talk</a></p></li>
</ul>
</section>
<section id="backend-apps">
<h4>Backend apps<a class="headerlink" href="#backend-apps" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference internal" href="app_llm2.html#ai-app-llm2"><span class="std std-ref">llm2</span></a> - Runs open source AI LLM models on your own server hardware (Customer support available upon request)</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a> - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see <a class="reference internal" href="ai_as_a_service.html#ai-ai-as-a-service"><span class="std std-ref">AI as a Service</span></a>)</p></li>
</ul>
</section>
</section>
<section id="machine-translation">
<h3>Machine translation<a class="headerlink" href="#machine-translation" title="Link to this heading"></a></h3>
<p id="mt-consumer-apps">As you can see in the table above we have multiple apps offering machine translation capabilities. Each app brings its own set of supported languages.
In downstream apps like the Text app, users can use the translation functionality regardless of which app implements it behind the scenes.</p>
<section id="id13">
<h4>Frontend apps<a class="headerlink" href="#id13" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><em>Text</em> for offering the translation menu</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/assistant">Assistant</a> offering a graphical translation UI</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/analytics">Analytics</a> for translating graph labels</p></li>
</ul>
</section>
<section id="id15">
<h4>Backend apps<a class="headerlink" href="#id15" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference internal" href="app_translate2.html#ai-app-translate2"><span class="std std-ref">translate2 (ExApp)</span></a> - Runs open source AI translation models locally on your own server hardware (Customer support available upon request)</p></li>
<li><p><em>integration_deepl</em> - Integrates with the deepl API to provide translation functionality from Deepl.com servers (Only community supported)</p></li>
</ul>
</section>
</section>
<section id="speech-to-text">
<h3>Speech-To-Text<a class="headerlink" href="#speech-to-text" title="Link to this heading"></a></h3>
<p id="stt-consumer-apps">As you can see in the table above we have multiple apps offering Speech-To-Text capabilities. In downstream apps like the Talk app, users can use the transcription functionality regardless of which app implements it behind the scenes.</p>
<section id="id16">
<h4>Frontend apps<a class="headerlink" href="#id16" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/assistant">Assistant</a> offering a graphical translation UI and a smart picker</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/spreed">Talk</a> for transcribing calls (see <a class="reference external" href="https://nextcloud-talk.readthedocs.io/en/latest/settings/#app-configuration">Nextcloud Talk docs</a> for how to enable this)</p></li>
</ul>
</section>
<section id="id19">
<h4>Backend apps<a class="headerlink" href="#id19" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference internal" href="app_stt_whisper2.html#ai-app-stt-whisper2"><span class="std std-ref">stt_whisper2</span></a> - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request)</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a> - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see <a class="reference internal" href="ai_as_a_service.html#ai-ai-as-a-service"><span class="std std-ref">AI as a Service</span></a>)</p></li>
</ul>
</section>
</section>
<section id="image-generation">
<h3>Image generation<a class="headerlink" href="#image-generation" title="Link to this heading"></a></h3>
<p id="t2i-consumer-apps">As you can see in the table above we have multiple apps offering Image generation capabilities. In downstream apps like the Text-to-Image helper app, users can use the image generation functionality regardless of which app implements it behind the scenes.</p>
<section id="id21">
<h4>Frontend apps<a class="headerlink" href="#id21" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/assistant">Assistant</a> for offering a graphical UI and a smart picker</p></li>
</ul>
</section>
<section id="id23">
<h4>Backend apps<a class="headerlink" href="#id23" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p>text2image_stablediffusion (Customer support available upon request)</p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/integration_openai">OpenAI and LocalAI integration (via OpenAI API)</a> - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see <a class="reference internal" href="ai_as_a_service.html#ai-ai-as-a-service"><span class="std std-ref">AI as a Service</span></a>)</p></li>
<li><p><em>integration_replicate</em> - Integrates with the replicate API to provide AI functionality from replicate servers (see <a class="reference internal" href="ai_as_a_service.html#ai-ai-as-a-service"><span class="std std-ref">AI as a Service</span></a>)</p></li>
</ul>
</section>
</section>
<section id="context-chat">
<h3>Context Chat<a class="headerlink" href="#context-chat" title="Link to this heading"></a></h3>
<p>Our Context Chat feature was introduced in Nextcloud Hub 7 (v28). It allows asking questions to the assistant related to your documents in Nextcloud. You will need to install both the context_chat app as well as the context_chat_backend External App. Be prepared that things might break or be a little rough around the edges. We look forward to your feedback!</p>
<section id="id25">
<h4>Frontend apps<a class="headerlink" href="#id25" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/assistant">Assistant</a> for offering a graphical UI for the context chat tasks</p></li>
</ul>
</section>
<section id="id27">
<h4>Backend apps<a class="headerlink" href="#id27" title="Link to this heading"></a></h4>
<ul class="simple">
<li><p><a class="reference internal" href="app_context_chat.html#ai-app-context-chat"><span class="std std-ref">context_chat + context_chat_backend</span></a> - (Customer support available upon request)</p></li>
</ul>
</section>
<section id="provider-apps">
<h4>Provider apps<a class="headerlink" href="#provider-apps" title="Link to this heading"></a></h4>
<p>Apps can integrate their content with Context Chat to make it available for querying using Context Chat. The following apps have implemented this integration so far:</p>
<ul class="simple">
<li><p><em>files</em></p></li>
<li><p><a class="reference external" href="https://apps.nextcloud.com/apps/analytics">Analytics</a></p></li>
</ul>
</section>
</section>
</section>
<section id="improve-ai-task-pickup-speed">
<span id="ai-overview-improve-ai-task-pickup-speed"></span><h2>Improve AI task pickup speed<a class="headerlink" href="#improve-ai-task-pickup-speed" title="Link to this heading"></a></h2>
<p>Most AI tasks will be run as part of the background job system in Nextcloud which only runs jobs every 5 minutes by default.
To pick up scheduled jobs faster you can set up background job workers that process AI tasks as soon as they are scheduled.
If the PHP code or the Nextcloud settings values are changed while a worker is running, those changes won’t be effective inside the runner. For that reason, the worker needs to be restarted regularly. It is done with a timeout of N seconds which means any changes to the settings or the code will be picked up after N seconds (worst case scenario). This timeout does not, in any way, affect the processing or the timeout of the AI tasks.</p>
<section id="screen-or-tmux-session">
<h3>Screen or tmux session<a class="headerlink" href="#screen-or-tmux-session" title="Link to this heading"></a></h3>
<p>Run the following occ command inside a screen or a tmux session, preferably 4 or more times for parallel processing of multiple requests by different or the same user (and as a requirement for some apps like context_chat).
It would be best to run one command per screen session or per tmux window/pane to keep the logs visible and the worker easily restartable.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nb">set</span> <span class="o">-</span><span class="n">e</span><span class="p">;</span> <span class="k">while</span> <span class="n">true</span><span class="p">;</span> <span class="n">do</span> <span class="n">sudo</span> <span class="o">-</span><span class="n">u</span> <span class="n">www</span><span class="o">-</span><span class="n">data</span> <span class="n">occ</span> <span class="n">background</span><span class="o">-</span><span class="n">job</span><span class="p">:</span><span class="n">worker</span> <span class="o">-</span><span class="n">v</span> <span class="o">-</span><span class="n">t</span> <span class="mi">60</span> <span class="s2">"OC\TaskProcessing\SynchronousBackgroundJob"</span><span class="p">;</span> <span class="n">done</span>
</pre></div>
</div>
<p>You may want to adjust the number of workers and the timeout (in seconds) to your needs.
The logs of the worker can be checked by attaching to the screen or tmux session.</p>
</section>
<section id="systemd-service">
<h3>Systemd service<a class="headerlink" href="#systemd-service" title="Link to this heading"></a></h3>
<ol class="arabic simple">
<li><p>Create a systemd service file in <code class="docutils literal notranslate"><span class="pre">/etc/systemd/system/nextcloud-ai-worker@.service</span></code> with the following content:</p></li>
</ol>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="n">Unit</span><span class="p">]</span>
<span class="n">Description</span><span class="o">=</span><span class="n">Nextcloud</span> <span class="n">AI</span> <span class="n">worker</span> <span class="o">%</span><span class="n">i</span>
<span class="n">After</span><span class="o">=</span><span class="n">network</span><span class="o">.</span><span class="n">target</span>
<span class="p">[</span><span class="n">Service</span><span class="p">]</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">opt</span><span class="o">/</span><span class="n">nextcloud</span><span class="o">-</span><span class="n">ai</span><span class="o">-</span><span class="n">worker</span><span class="o">/</span><span class="n">taskprocessing</span><span class="o">.</span><span class="n">sh</span> <span class="o">%</span><span class="n">i</span>
<span class="n">Restart</span><span class="o">=</span><span class="n">always</span>
<span class="p">[</span><span class="n">Install</span><span class="p">]</span>
<span class="n">WantedBy</span><span class="o">=</span><span class="n">multi</span><span class="o">-</span><span class="n">user</span><span class="o">.</span><span class="n">target</span>
</pre></div>
</div>
<ol class="arabic simple" start="2">
<li><p>Create a shell script in <code class="docutils literal notranslate"><span class="pre">/opt/nextcloud-ai-worker/taskprocessing.sh</span></code> with the following content and make sure to make it executable:</p></li>
</ol>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="n">echo</span> <span class="s2">"Starting Nextcloud AI Worker $1"</span>
<span class="n">cd</span> <span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">nextcloud</span>
<span class="n">sudo</span> <span class="o">-</span><span class="n">u</span> <span class="n">www</span><span class="o">-</span><span class="n">data</span> <span class="n">php</span> <span class="n">occ</span> <span class="n">background</span><span class="o">-</span><span class="n">job</span><span class="p">:</span><span class="n">worker</span> <span class="o">-</span><span class="n">t</span> <span class="mi">60</span> <span class="s1">'OC\TaskProcessing\SynchronousBackgroundJob'</span>
</pre></div>
</div>
<p>You may want to adjust the timeout to your needs (in seconds).</p>
<ol class="arabic simple" start="3">
<li><p>Enable and start the service 4 or more times:</p></li>
</ol>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>for i in {1..4}; do systemctl enable --now nextcloud-ai-worker@$i.service; done
</pre></div>
</div>
<p>The status of the workers can be checked with (replace 1 with the worker number):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">systemctl</span> <span class="n">status</span> <span class="n">nextcloud</span><span class="o">-</span><span class="n">ai</span><span class="o">-</span><span class="n">worker</span><span class="o">@</span><span class="mf">1.</span><span class="n">service</span>
</pre></div>
</div>
<p>The list of workers can be checked with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">systemctl</span> <span class="nb">list</span><span class="o">-</span><span class="n">units</span> <span class="o">--</span><span class="nb">type</span><span class="o">=</span><span class="n">service</span> <span class="o">|</span> <span class="n">grep</span> <span class="n">nextcloud</span><span class="o">-</span><span class="n">ai</span><span class="o">-</span><span class="n">worker</span>
</pre></div>
</div>
<p>The complete logs of the workers can be checked with (replace 1 with the worker number):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">journalctl</span> <span class="o">-</span><span class="n">xeu</span> <span class="n">nextcloud</span><span class="o">-</span><span class="n">ai</span><span class="o">-</span><span class="n">worker</span><span class="o">@</span><span class="mf">1.</span><span class="n">service</span> <span class="o">-</span><span class="n">f</span>
</pre></div>
</div>
</section>
</section>
<section id="frequently-asked-questions">
<h2>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Link to this heading"></a></h2>
<section id="why-is-my-prompt-slow">
<h3>Why is my prompt slow?<a class="headerlink" href="#why-is-my-prompt-slow" title="Link to this heading"></a></h3>
<p>Reasons for slow performance from a user perspective can be</p>
<blockquote>
<div><ul class="simple">
<li><p>Using CPU processing instead of GPU (sometimes this limit is imposed by the used app)</p></li>
<li><p>High user demand for the feature: User prompts and AI tasks are usually processed in the order they are received, which can cause delays when a lot of users access these features at the same time.</p></li>
</ul>
</div></blockquote>
</section>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="index.html" class="btn btn-neutral float-left" title="Artificial Intelligence" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="app_assistant.html" class="btn btn-neutral float-right" title="Nextcloud Assistant" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2024 Nextcloud GmbH.</p>
</div>
</footer>
</div>
</div>
</section>
</div>
<div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="Versions">
<span class="rst-current-version" data-toggle="rst-current-version">
<span class="fa fa-book"> Read the Docs</span>
v: latest
<span class="fa fa-caret-down"></span>
</span>
<div class="rst-other-versions">
<dl>
<dt>Versions</dt>
<dd><a href="https://docs.nextcloud.com/server/28/admin_manual">28</a></dd>
<dd><a href="https://docs.nextcloud.com/server/29/admin_manual">29</a></dd>
<dd><a href="https://docs.nextcloud.com/server/stable/admin_manual">stable</a></dd>
<dd><a href="https://docs.nextcloud.com/server/latest/admin_manual">latest</a></dd>
</dl>
<dl>
<dt>Downloads</dt>
</dl>
<dl>
<dt>On Read the Docs</dt>
<dd>
<a href="///projects//?fromdocs=">Project Home</a>
</dd>
<dd>
<a href="///builds//?fromdocs=">Builds</a>
</dd>
</dl>
</div>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Mini Shell