🤖 Self-hosted, community-driven, local OpenAI-compatible API with Keycloak Auth Flak app as frontend. 🏠
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
mudler 4de7f55f2f Make REBUILD=false default behavior 1 year ago
.github feat: fix CUDA images and update go-llama to use full GPU offloading (#618) 2 years ago
.vscode feat: Add more test-cases and remove dev container (#433) 2 years ago
api Add grammar_json to the request parameters to facilitate JSON generation 1 year ago
examples example(slack-qa-bot): Add slack QA bot example (#654) 2 years ago
internal feat: cleanups, small enhancements 1 year ago
models Add docker-compose 2 years ago
pkg Add grammar_json to the request parameters to facilitate JSON generation 1 year ago
prompt-templates docs: enhancements (#133) 2 years ago
tests feat: update go-gpt2 (#359) 2 years ago
.dockerignore Remove .git from .dockerignore 1 year ago
.env Make REBUILD=false default behavior 1 year ago
.gitignore fix: copy metal file from build (#564) 2 years ago
Dockerfile Make REBUILD=false default behavior 1 year ago
Earthfile Rename project to LocalAI (#35) 2 years ago
LICENSE docs: update docs/license(clarification) and point to new website (#415) 2 years ago
Makefile invoke go mod clean before rebuilds 1 year ago
README.md Update README.md 2 years ago
assets.go feat: Update gpt4all, support multiple implementations in runtime (#472) 2 years ago
docker-compose.yaml images: cleanup, drop .dev Dockerfile (#437) 2 years ago
entrypoint.sh Make REBUILD=false default behavior 1 year ago
go.mod fix(deps): update github.com/go-skynet/go-llama.cpp digest to 42ba448 (#698) 2 years ago
go.sum fix(deps): update github.com/go-skynet/go-llama.cpp digest to 42ba448 (#698) 2 years ago
main.go feat: cleanups, small enhancements 1 year ago
renovate.json ci: manually update deps 2 years ago

README.md



LocalAI

tests build container images

Documentation website

LocalAI is a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU.

For a list of the supported model families, please see the model compatibility table.

In a nutshell:

  • Local, OpenAI drop-in alternative REST API. You own your data.
  • NO GPU required. NO Internet access is required either
    • Optional, GPU Acceleration is available in llama.cpp-compatible LLMs. See also the build section.
  • Supports multiple models:
    • 📖 Text generation with GPTs (llama.cpp, gpt4all.cpp, ... and more)
    • 🗣 Text to Audio 🎺🆕
    • 🔈 Audio to Text (Audio transcription with whisper.cpp)
    • 🎨 Image generation with stable diffusion
  • 🏃 Once loaded the first time, it keep models loaded in memory for faster inference
  • Doesn't shell-out, but uses C++ bindings for a faster inference and better performance.

LocalAI was created by Ettore Di Giacinto and is a community-driven project, focused on making the AI accessible to anyone. Any contribution, feedback and PR is welcome!

See the Getting started and examples sections to learn how to use LocalAI. For a list of curated models check out the model gallery.

ChatGPT OSS alternative Image generation
Screenshot from 2023-04-26 23-59-55 b6441997879
Telegram bot Flowise
Screenshot from 2023-06-09 00-36-26 Screenshot from 2023-05-30 18-01-03

Hot topics / Roadmap

News

For latest news, follow also on Twitter @LocalAI_API and @mudler_it

Media, Blogs, Social

Contribute and help

To help the project you can:

  • Hacker news post - help us out by voting if you like this project.

  • If you have technological skills and want to contribute to development, have a look at the open issues. If you are new you can have a look at the good-first-issue and help-wanted labels.

  • If you don't have technological skills you can still help improving documentation or add examples or share your user-stories with our community, any help and contribution is welcome!

Usage

Check out the Getting started section. Here below you will find generic, quick instructions to get ready and use LocalAI.

The easiest way to run LocalAI is by using docker-compose (to build locally, see building LocalAI):


git clone https://github.com/go-skynet/LocalAI

cd LocalAI

# (optional) Checkout a specific LocalAI tag
# git checkout -b build <TAG>

# copy your models to models/
cp your-model.bin models/

# (optional) Edit the .env file to set things like context size and threads
# vim .env

# start with docker-compose
docker-compose up -d --pull always
# or you can build the images with:
# docker-compose up -d --build

# Now API is accessible at localhost:8080
curl http://localhost:8080/v1/models
# {"object":"list","data":[{"id":"your-model.bin","object":"model"}]}

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.bin",            
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

Example: Use GPT4ALL-J model

# Clone LocalAI
git clone https://github.com/go-skynet/LocalAI

cd LocalAI

# (optional) Checkout a specific LocalAI tag
# git checkout -b build <TAG>

# Download gpt4all-j to models/
wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O models/ggml-gpt4all-j

# Use a template from the examples
cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/

# (optional) Edit the .env file to set things like context size and threads
# vim .env

# start with docker-compose
docker-compose up -d --pull always
# or you can build the images with:
# docker-compose up -d --build
# Now API is accessible at localhost:8080
curl http://localhost:8080/v1/models
# {"object":"list","data":[{"id":"ggml-gpt4all-j","object":"model"}]}

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "ggml-gpt4all-j",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

# {"model":"ggml-gpt4all-j","choices":[{"message":{"role":"assistant","content":"I'm doing well, thanks. How about you?"}}]}

Build locally

In order to build the LocalAI container image locally you can use docker:

# build the image
docker build -t localai .
docker run localai

Or you can build the binary with make:

make build

See the build section in our documentation for detailed instructions.

Run LocalAI in Kubernetes

LocalAI can be installed inside Kubernetes with helm. See installation instructions.

Supported API endpoints

See the list of the supported API endpoints and how to configure image generation and audio transcription.

Frequently asked questions

See the FAQ section for a list of common questions.

Projects already using LocalAI to run local models

Feel free to open up a PR to get your project listed!

Star history

LocalAI Star history Chart

License

LocalAI is a community-driven project created by Ettore Di Giacinto.

MIT

Author

Ettore Di Giacinto and others

Acknowledgements

LocalAI couldn't have been built without the help of great software already available from the community. Thank you!

Contributors