[ROSA] ROSA로 turtlesim 작동시키기

https://github.com/nasa-jpl/rosa

GitHub - nasa-jpl/rosa: ROSA 🤖 is an AI Agent designed to interact with ROS1- and ROS2-based robotics systems using natural l

 

무려 나사에서 ROSA라는 프로젝트를 공개했는데

 

ROS Agent라고 LLM를 이용해 ROS로 로봇을 컨트롤 시스템이라고 볼 수 있다. 

LangChain framework로 build했다고 한다. 

ROSA는 nautral language를 사용해서 로봇과 사용자의 interact를 도와주고 로봇 개발을 더욱 접근성이 좋고 효율적으로 만들어준다고 소개해주고 있다. 

 

 

https://www.youtube.com/watch?v=-C43j2HgQdY&t=359s

 

NVIDIA Developer에서 Isaac Sim에서 ROSA extension으로 예시를 보여주고 있지만 아쉽게도 

 

 

아직 정식 런칭되지는 않았다. 

 

그래서 일단 TurtleSim Demo만 해보려고 한다. 

 

 

TurtleSim Demo Guide 링크를 들어가본다. 

 

 

Isaac Sim도 안되어서 살짝 할까말까 짜증났는데, TurtleSim Demo 조차, ROS1만 지원한다고 한다... ㅂㄷㅂㄷ 

 

하지말까 하다가 따라쟁이 대신맨으로써 한번 해본다. 

 

깃클론 이후 

 

./demo.sh

 

를 해야하는데. 

 

그전에 먼저 .env 파일에서 자신의 OPENAI_API_KEY를 입력해준다. 

 

src/turtle_agent/scripts/llm.py를 살펴본다. 

 

 

여기서 llm을 ChatOpenAI를 사용할 수 있게 코드를 고친다. 그리고 AzureChatOpenAI나 ChatOllama도 사용할 수 있게 코드를 고쳤다.

 

#  Copyright (c) 2024. Jet Propulsion Laboratory. All rights reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#  https://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.

import os

import dotenv
from azure.identity import ClientSecretCredential, get_bearer_token_provider
from langchain_openai import AzureChatOpenAI, ChatOpenAI
from langchain_ollama import ChatOllama

def get_llm(streaming: bool = False, llm_type: str ='openai'):
    """A helper function to get the LLM instance."""
    dotenv.load_dotenv(dotenv.find_dotenv())

    if llm_type=='openai':
        # Initialize ChatOpenAI instance
        llm = ChatOpenAI(
            openai_api_key=os.getenv("OPENAI_API_KEY"),
            streaming=streaming,
            temperature=0,
            model_name="gpt-4o-mini",  # 모델명          # gpt-4o  gpt-4-turbo    gpt-4o-mini  o1-preview  o1-mini  gpt-4o-realtime
        )   
    
    elif llm_type=='ollama':
        llm = ChatOllama(
            model="llama3.1:7b",  # or your preferred model
            temperature=0,
            num_ctx=8192,  # adjust based on your model's context window
            # base_url="host.docker.internal:11434",
        )   
    
    else:
        APIM_SUBSCRIPTION_KEY = get_env_variable("APIM_SUBSCRIPTION_KEY")
        default_headers = {}
        if APIM_SUBSCRIPTION_KEY is not None:
            # only set this if the APIM API requires a subscription...
            default_headers["Ocp-Apim-Subscription-Key"] = APIM_SUBSCRIPTION_KEY

        # Set up authority and credentials for Azure authentication
        credential = ClientSecretCredential(
            tenant_id=get_env_variable("AZURE_TENANT_ID"),
            client_id=get_env_variable("AZURE_CLIENT_ID"),
            client_secret=get_env_variable("AZURE_CLIENT_SECRET"),
            authority="https://login.microsoftonline.com",
        )

        token_provider = get_bearer_token_provider(
            credential, "https://cognitiveservices.azure.com/.default"
        )

        llm = AzureChatOpenAI(
            azure_deployment=get_env_variable("DEPLOYMENT_ID"),
            azure_ad_token_provider=token_provider,
            openai_api_type="azure_ad",
            api_version=get_env_variable("API_VERSION"),
            azure_endpoint=get_env_variable("API_ENDPOINT"),
            default_headers=default_headers,
            streaming=streaming,
        )

    return llm


def get_env_variable(var_name: str) -> str:
    """
    Retrieves the value of the specified environment variable.

    Args:
        var_name (str): The name of the environment variable to retrieve.

    Returns:
        str: The value of the environment variable.

    Raises:
        ValueError: If the environment variable is not set.

    This function provides a consistent and safe way to retrieve environment variables.
    By using this function, we ensure that all required environment variables are present
    before proceeding with any operations. If a variable is not set, the function will
    raise a ValueError, making it easier to debug configuration issues.
    """
    value = os.getenv(var_name)
    if value is None:
        raise ValueError(f"Environment variable {var_name} is not set.")
    return value

 

그리고 demo.sh를 하려니깐 또 안된다. demo.sh를 아래 내용으로 고쳐야한다.

 

 

#!/usr/bin/env bash
# Copyright (c) 2024. Jet Propulsion Laboratory. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script launches the ROSA demo in Docker

# Check if Docker is installed
if ! command -v docker &> /dev/null; then
    echo "Error: Docker is not installed. Please install Docker and try again."
    exit 1
fi

# Set default headless mode
HEADLESS=${HEADLESS:-false}
DEVELOPMENT=${DEVELOPMENT:-false}

# Enable X11 forwarding based on OS
case "$(uname)" in
    Linux*)
        echo "Enabling X11 forwarding..."
        export DISPLAY=:0
        xhost +
        ;;
    *)
        echo "Error: Unsupported operating system."
        exit 1
        ;;
esac

# Check if X11 forwarding is working
if ! xset q &> /dev/null; then
    echo "Error: X11 forwarding is not working. Please check your X11 server and try again."
    exit 1
fi

# Build and run the Docker container
CONTAINER_NAME="rosa-turtlesim-demo"
echo "Building the $CONTAINER_NAME Docker image..."
sudo docker build --build-arg DEVELOPMENT=$DEVELOPMENT -t $CONTAINER_NAME -f Dockerfile . || { echo "Error: Docker build failed"; exit 1; }

echo "Running the Docker container..."
sudo docker run -it --rm --name $CONTAINER_NAME \
    -e DISPLAY=$DISPLAY \
    -e HEADLESS=$HEADLESS \
    -e DEVELOPMENT=$DEVELOPMENT \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -v "$PWD/src":/app/src \
    -v "$PWD/tests":/app/tests \
    --network host \
    $CONTAINER_NAME

# Disable X11 forwarding
xhost -

exit 0

 

이제 실행하자.

 

./demo.sh

 

다음으로 streaming mode로 진행하기 위해 

 

start streaming:=true

 

다음으로 사용 예시를 살펴보고 따라해보자.