ChatOpsLLM
To simplify and streamline LLM operations, empowering developers and organizations to harness the full potential of large language models with ease.
Install / Use
/learn @bmd1905/ChatOpsLLMREADME
ChatOpsLLM
ChatOpsLLM: Empowering Chatbots with Effortless DevOps.
Introduction
ChatOpsLLM is a project built with Open WebUI that can be deployed on Google Kubernetes Engine (GKE) for managing and scaling language models. It offers both Terraform and manual deployment methods, and incorporates robust MLOps practices. This includes CI/CD pipelines with Jenkins and Ansible for automation, monitoring with Prometheus and Grafana for performance insights, and centralized logging with the ELK stack for troubleshooting and analysis. Developers can find detailed documentation and instructions on the project's website.
https://github.com/user-attachments/assets/cf84a434-0dae-47b9-a93d-49a37965d968
Features
- Ease of Use: ChatOpsLLM provides an intuitive interface and streamlined workflows that make managing LLMs simple and efficient, regardless of your experience level.
- Scalability & Flexibility: Scale your LLM deployments effortlessly, adapt to evolving needs, and integrate seamlessly with your existing infrastructure.
- Reduced Complexity: Eliminate the hassle of complex configurations and infrastructure management, allowing you to focus on building and deploying powerful LLM applications.
- Enhanced Productivity: Accelerate your LLM development lifecycle, optimize performance, and maximize the impact of your language models.
Target Audience
Developers building and deploying LLM-powered applications. Data scientists and machine learning engineers working with LLMs. DevOps teams responsible for managing LLM infrastructure. Organizations looking to integrate LLMs into their operations.
Table of Contents
- Introduction
- Features
- Target Audience
- Getting Started
- Quick Start
- Using Terraform for Google Kubernetes Engine (GKE)
- Manual Deployment to GKE
- Continuous Integration/Continuous Deployment (CI/CD) with Jenkins and Ansible
- Monitoring with Prometheus and Grafana
- Logging with Filebeat + Logstash + Elasticsearch + Kibana
- Optimize Cluster with Cast AI
- Log and Trace with Langfuse and Supabase
- Contributing
- License
- Citation
- Contact
Getting Started
In case you don't want to spend much time, please run this script and enjoy your coffee:
chmod +x ./cluster.sh
./cluster.sh
Remember to authenticate with GCP before using Terraform:
gcloud auth application-default login
<a name="quick-start"></a>Quick Start
This section provides a very quick start guide to get the application up and running as soon as possible. Please refer to the following sections for more detailed instructions.
<a name="using-terraform-for-google-kubernetes-engine-gke"></a>Using Terraform for Google Kubernetes Engine (GKE)
1. <a name="set-up-the-cluster"></a>Set up the Cluster:
If you're deploying the application to GKE, you can use Terraform to automate the setup of your Kubernetes cluster. Navigate to the iac/terraform directory and initialize Terraform:
cd iac/terraform
terraform init
Plan and Apply Configuration:
Generate an execution plan to verify the resources that Terraform will create or modify, and then apply the configuration to set up the cluster:
terraform plan
terraform apply
2. <a name="retrieve-cluster-information"></a>Retrieve Cluster Information:
To interact with your GKE cluster, you'll need to retrieve its configuration. You can view the current cluster configuration with the following command:
cat ~/.kube/config
https://github.com/user-attachments/assets/3133c2a8-8475-45c6-8900-96c2af8c5ad5
Ensure your kubectl context is set correctly to manage the cluster.
<a name="manual-deployment-to-gke"></a>Manual Deployment to GKE
For a more hands-on deployment process, follow these steps:
1. <a name="deploy-nginx-ingress-controller"></a>Deploy Nginx Ingress Controller:
The Nginx Ingress Controller manages external access to services in your Kubernetes cluster. Create a namespace and install the Ingress Controller using Helm:
kubectl create ns nginx-system
kubens nginx-system
helm upgrade --install nginx-ingress ./deployments/nginx-ingress
Please story the Nginx Ingress Controller's IP address, as you'll need it later.
https://github.com/user-attachments/assets/f329a8ee-cd4d-44e8-bb12-d1ff39dce4b8
2. <a name="configure-api-key-secret"></a>Configure API Key Secret:
Store your environment variables, such as API keys, securely in Kubernetes secrets. Create a namespace for model serving and create a secret from your .env file:
kubectl create ns model-serving
kubens model-serving
kubectl delete secret ChatOpsLLM-env
kubectl create secret generic ChatOpsLLM-env --from-env-file=.env -n model-serving
kubectl describe secret ChatOpsLLM-env -n model-serving
https://github.com/user-attachments/assets/fab6aa93-2f68-4f36-a4d8-4a1d955596f2
3. <a name="grant-permissions"></a>Grant Permissions:
Kubernetes resources often require specific permissions. Apply the necessary roles and bindings:
cd deployments/infrastructure
kubectl apply -f role.yaml
kubectl apply -f rolebinding.yaml
https://github.com/user-attachments/assets/9c1aa6e1-6b8c-4332-ab11-513428ef763b
4. <a name="deploy-caching-service-using-redis"></a>Deploy caching service using Redis:
Now, deploy the semantic caching service using Redis:
cd ./deployments/redis
helm dependency build
helm upgrade --install redis .
https://github.com/user-attachments/assets/ef37626a-9a98-473e-a7e0-effcaa262ad5
5. <a name="deploy-litellm"></a>Deploy LiteLLM:
Deploy the LiteLLM service:
kubens model-serving
helm upgrade --install litellm ./deployments/litellm
https://github.com/user-attachments/assets/0c98fe90-f958-42fc-9fa6-224dcf417e29
6. <a name="deploy-the-open-webui"></a>Deploy the Open WebUI:
Next, Deploy the web UI to your GKE cluster:
cd open-webui
kubectl apply -f ./kubernetes/manifest/base -n model-serving
https://github.com/user-attachments/assets/60ad30e3-e8f8-49a6-ab96-d895fe7986cb
7. <a name="play-around-with-the-application"></a>Play around with the Application:
Open browser and navigate to the URL of your GKE cluster (e.g. http://172.0.0.0 in step 1) and add .nip.io to the end of the URL (e.g. http://172.0.0.0.nip.io). You should see the Open WebUI:
https://github.com/user-attachments/assets/4115a1f0-e513-4c58-a359-1d49683905a8
<a name="continuous-integrationcontinuous-deployment-cicd-with-jenkins-and-ansible"></a>Continuous Integration/Continuous Deployment (CI/CD) with Jenkins and Ansible
For automated CI/CD pipelines, use Jenkins and Ansible as follows:
1. <a name="set-up-jenkins-server"></a>Set up Jenkins Server:
First, create a Service Account and assign it the Compute Admin role. Then create a Json key file for the Service Account and store it in the iac/ansible/secrets directory.
Next create a Google Compute Engine instance named "jenkins-server" running Ubuntu 22.04 with a firewall rule allowing traffic on ports 8081 and 50000.
ansible-playbook iac/ansible/deploy_jenkins/create_compute_instance.yaml
Deploy Jenkins on a server by installing prerequisites, pulling a Docker image, and creating a privileged container with access to the Docker socket and exposed ports 8081 and 50000.
ansible-playbook -i iac/ansible/inventory iac/ansible/deploy_jenkins/deploy_jenkins.yaml
https://github.com/user-attachments/assets/35dae326-aa8f-4779-bf67-2b8d9f71487b
2. <a name="access-jenkins"></a>Access Jenkins:
To access the Jenkins server through SSH, we need to create a public/private key pair. Run the following command to create a key pair:
ssh-keygen
Open Metadata and copy the ssh-keys value.
https://github.com/user-attachments/assets/8fd956be-d2db-4d85-aa7c-f78df160c00c
We need to find the Jenkins server password to be able to access the server. First, access the Jenkins server:
ssh <USERNAME>:<EXTERNAL_IP>
Then run the follo
