PaLM: Predicting Actions through LLM

We propose a solution to the Long-Term Action Anticipation (LTA) task utilizing a vision-language model (VLM) and large language models (LLM): chaining a captioning model and an action recognition transformer with a retrieval-augmented LLM. The approach was the winning entry of the Ego4D LTA challenge at CVPRW23’ and was presented as a poster at ECCV24’.

More details can be found on the project page.

As a side project, I hosted VLM and LLM inference servers through API/RPC and onnxRT.