Vectorizing Real-Time Kafka Events for RAG

In the previous post, I set up pgvector and Ollama for embedding and vector search. Now I need to fill the database with data. Not documents or PDFs. Real-time saga events flowing through Kafka.

Every time a saga finishes (success or failure), the system publishes to a notify-ending topic. My AI agent listens to that topic and vectorizes every event. Over time, this builds a searchable history of all saga executions.

The Kafka Consumer

The entry point is a standard Spring Kafka listener:

@KafkaListener ( groupId = "${spring.kafka.consumer.group-id}" , topics = "${spring.kafka.topic.notify-ending}" ) public void onSagaEnded ( String payload ) { var event = parseEvent ( payload ). orElse ( null ); if ( event == null ) { log . warn ( "[OperationsService] Failed to parse event payload" ); return ; }

// Vectorize ALL events for learning String historyText = buildHistoryText ( event ); vectorize ( event , historyText );

// Diagnose only failures if ( event .…