The best format for storing video/vector mappings depends on balancing efficiency, scalability, and ease of access. For most use cases, a combination of a structured file format (like Protocol Buffers or Parquet) paired with a database system (such as SQL or a vector database) works well. These formats allow you to store metadata (timestamps, video IDs) alongside vector embeddings while supporting fast retrieval and updates. The key is to structure the data to minimize redundancy and enable efficient querying, especially when dealing with large datasets.
For structured file storage, Protocol Buffers (Protobuf) is a strong choice because it efficiently serializes structured data into a compact binary format. For example, you could define a Protobuf message that includes fields like video_id, start_time, end_time, and a repeated float field for the vector. Parquet is another good option, particularly if you’re working with batch processing frameworks like Apache Spark. A Parquet file could organize data into columns such as frame_number, timestamp, and vector_data, leveraging columnar storage for faster queries on specific attributes. Both formats avoid the overhead of JSON (which can become bloated with repeated keys in large datasets) while maintaining schema consistency.
When using a database, consider a hybrid approach. A relational database like PostgreSQL can store metadata in tables with columns for video segments and foreign keys linking to vector embeddings stored as BLOBs or in dedicated vector columns. For applications requiring similarity searches (e.g., finding videos with vectors closest to a query), specialized vector databases like Milvus or Pinecone are optimized for this task. These systems index vectors using algorithms like HNSW (Hierarchical Navigable Small World) to enable fast nearest-neighbor lookups. For example, Pinecone allows you to associate each vector with a JSON payload containing video metadata, making retrieval straightforward. If storage cost is a concern, compressing vectors (e.g., using scalar quantization) or partitioning data by video ID can reduce overhead while maintaining performance.