The Ask LukeW feature on my website has been answering people's questions about digital product design for over three years. How well it answers depends on the content available to it. So in addition to improving content retrieval recently, I've also improved content indexing: a key part of making personal AI work.
Before any of my articles, presentations, or files can be used to answer people's questions, they need to be cleaned up and turned into a format that makes semantic search possible. This happens through a content ingestion pipeline, which I had to kick off manually for every piece of new content.
Now the system automatically checks for anything new or changed each hour. Audio, video, PDFs, and images it finds get queued, processed in the background, and added to the corpus. How that happens depends on the types of content.
Audio and video files get transcribed into text spelling out what was said and who said it.…