From 4275bfc8c03679366ce43c29019b59c0935315b5 Mon Sep 17 00:00:00 2001
From: mudler <mudler@c3os.io>
Date: Mon, 20 Mar 2023 21:30:55 +0100
Subject: [PATCH] Add README

---
 README.md | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 102 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..a319760
--- /dev/null
+++ b/README.md
@@ -0,0 +1,102 @@
+## :camel: llama-cli
+
+
+llama-cli is a straightforward golang CLI interface for [llama.cpp](https://github.com/ggerganov/llama.cpp), providing a simple API and a command line interface that allows text generation using a GPT-based model like llama directly from the terminal.
+
+## Container images
+
+The `llama-cli` [container images](https://quay.io/repository/go-skynet/llama-cli?tab=tags&tag=latest) come preloaded with the [alpaca.cpp](https://github.com/antimatter15/alpaca.cpp) model, enabling you to start making predictions immediately! To begin, run:
+
+```
+docker run -ti --rm quay.io/go-skynet/llama-cli:latest  --instruction "What's an alpaca?" --topk 10000
+```
+
+You will receive a response like the following:
+
+```
+An alpaca is a member of the South American Camelid family, which includes the llama, guanaco and vicuña. It is a domesticated species that originates from the Andes mountain range in South America. Alpacas are used in the textile industry for their fleece, which is much softer than wool. Alpacas are also used for meat, milk, and fiber.
+```
+
+## Basic usage
+
+To use llama-cli, specify a pre-trained GPT-based model, an input text, and an instruction for text generation. llama-cli takes the following arguments:
+
+```
+llama-cli --model <model_path> --instruction <instruction> [--input <input>] [--template <template_path>] [--tokens <num_tokens>] [--threads <num_threads>] [--temperature <temperature>] [--topp <top_p>] [--topk <top_k>]
+```
+
+| Parameter    | Environment Variable | Default Value | Description                            |
+| ------------ | -------------------- | ------------- | -------------------------------------- |
+| template     | TEMPLATE             |               | A file containing a template for output formatting (optional).  |
+| instruction  | INSTRUCTION          |               | Input prompt text or instruction. "-" for STDIN.   |
+| input        | INPUT                | -             | Path to text or "-" for STDIN.                    |
+| model        | MODEL_PATH           |               | The path to the pre-trained GPT-based model.      |
+| tokens       | TOKENS               | 128           | The maximum number of tokens to generate. |
+| threads      | THREADS              | NumCPU()      | The number of threads to use for text generation. |
+| temperature  | TEMPERATURE          | 0.95          | Sampling temperature for model output.  |
+| top_p        | TOP_P                | 0.85          | The cumulative probability for top-p sampling. |
+| top_k        | TOP_K                | 20            | The number of top-k tokens to consider for text generation.  |
+
+
+Here's an example of using `llama-cli`:
+
+```
+llama-cli --model ~/ggml-alpaca-7b-q4.bin --instruction "What's an alpaca?"
+```
+
+This will generate text based on the given model and instruction.
+
+## Advanced usage
+
+`llama-cli` also provides an API for running text generation as a service. You can start the API server using the following command:
+
+```
+llama-cli api --model <model_path> [--address <address>] [--threads <num_threads>]
+```
+
+The API takes takes the following arguments:
+
+| Parameter    | Environment Variable | Default Value | Description                            |
+| ------------ | -------------------- | ------------- | -------------------------------------- |
+| model        | MODEL_PATH           |               | The path to the pre-trained GPT-based model.      |
+| threads      | THREADS              | CPU cores     | The number of threads to use for text generation. |
+| address      | ADDRESS              | :8080         | The address and port to listen on. |
+
+
+And you'll see:
+```
+┌───────────────────────────────────────────────────┐ 
+│                   Fiber v2.42.0                   │ 
+│               http://127.0.0.1:8080               │ 
+│       (bound on host 0.0.0.0 and port 8080)       │ 
+│                                                   │ 
+│ Handlers ............. 1  Processes ........... 1 │ 
+│ Prefork ....... Disabled  PID ................. 1 │ 
+└───────────────────────────────────────────────────┘ 
+```
+
+Once the server is running, you can make requests to it using HTTP. For example, to generate text based on an instruction, you can send a POST request to the `/predict` endpoint with the instruction as the request body:
+
+```
+curl --location --request POST 'http://localhost:8080/predict' --header 'Content-Type: application/json' --data-raw '{
+    "text": "What is an alpaca?",
+    "topP": 0.8,
+    "topK": 50,
+    "temperature": 0.7,
+    "tokens": 100
+}'
+```
+
+Example of starting the API with `docker`:
+
+```bash
+docker run -ti --rm quay.io/go-skynet/llama-cli:latest api
+```
+
+### Kubernetes
+
+You can run the API directly in Kubernetes:
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/go-skynet/llama-cli/master/kubernetes/deployment.yaml
+```
\ No newline at end of file