RQ-113 deep research 生出力 (openai o3-deep-research / 2026-06-26 / $4.24 / 17 分)。突合・採択は Synthesis 参照。

1. Architecture Fit after Residency Removal

With the data residency constraint lifted, we can reconsider three architectural approaches for semantic + full-text search and RAG (Retrieval-Augmented Generation): (A) a BigQuery-centric solution (DIY hybrid search), (B) a Vertex AI Search-managed solution, or (C) a hybrid combination (e.g. BigQuery retrieval feeding Vertex AI for Q&A). Below we evaluate each on retrieval quality, analytics capability, latency, operations burden, and managed integration:

• Approach A – BigQuery Hybrid (DIY):

  • Retrieval Quality: Uses BigQuery’s vector search (IVF) + term search. Quality depends on the chosen embedding and custom ranking logic. It can achieve high recall (the PoC hit 90% recall@10) but may miss advanced features like synonym expansion or spell correction unless explicitly implemented. By default, BigQuery’s search is literal; you would need custom functions or additional preprocessing to handle synonyms or typos (Vertex AI Search provides these out-of-the-box ([docs.cloud.google.com](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/learn/vertex-ai-search#::text=Vertex%20AI%20Search%20provides%20natural,suggest%20out%20of%20the%20box))).
  • Analytics Capability: Excellent – all data lives in BigQuery, enabling powerful SQL querying and analysis. You can directly run ad-hoc queries, joins, and even apply LLM functions (e.g., AI.GENERATE_*) on the corpus for insights. This approach naturally supports analytics beyond search since the raw text and metadata are queryable.
  • Latency: Moderate – BigQuery can perform vector+keyword retrieval in a few hundred milliseconds to a second for ~50k chunks, especially with indexes. However, it may not be as low-latency as a specialized search engine. Each query scans the vector index and text index – efficient for this scale, but slower if data grows large or if complex ranking logic is applied. In practice, the PoC saw latency that was acceptable for interactive use (likely on the order of 1–2s total per query).
  • Ops Burden: High – this is a custom pipeline. You must manage data ingestion (split documents into chunks, generate embeddings, load to BQ), maintain indices (vector and text indexes or an arrangement like Reciprocal Rank Fusion manually). Ongoing updates (new data) require re-embedding and updating the BigQuery tables. Monitoring and tuning (index parameters, query functions) are your responsibility. In exchange, you have full control.
  • Managed Integration: Low – aside from BigQuery itself, which is managed, the search/Q&A logic is DIY. Integration with other managed services (Vertex AI) is manual. For example, if you want grounded answer generation, you’d call a Vertex LLM and perhaps the Vertex Grounding API explicitly. There’s no “one-click” managed UI or chatbot – you’d integrate the retrieval in your custom chat app (as done in the intake-chat-rag-sidecar). This offers flexibility but limited turnkey functionality.

• Approach B – Vertex AI Search (Managed):

  • Retrieval Quality: High – Vertex AI Search is “Google-quality” search for your data (cloud.google.com). It uses LLM-powered semantic indexing and supports hybrid search by default. Expect better recall and relevance: it can understand synonyms, correct spelling, and leverage semantic context beyond exact keywords (docs.cloud.google.com). The engine is tuned on Google’s search expertise, so it’s likely to surface relevant results even for natural language queries or when terminology differs. It also handles unstructured file parsing (PDF, HTML, etc.) automatically (docs.cloud.google.com), which improves retrieval from diverse docs.
  • Analytics Capability: Moderate – Vertex AI Search abstracts away the raw data behind a search index. You can query for relevant documents and get results, but you can’t run arbitrary SQL on the index. For analytical questions (trends, counts, etc.), you’d still need the data in BigQuery or another store. In other words, this approach is excellent for Q&A and search but not a data warehouse. One compromise is to use Vertex AI Search for retrieval and keep the source data in BigQuery for analytical queries, effectively maintaining two copies.
  • Latency: Very Low – Vertex AI Search is designed for real-time search at scale (payshopp.net). It leverages purpose-built infrastructure for fast retrieval (low 10s of ms typical for query latency). Even with complex corpora, it’s optimized to handle large indexes with minimal delay (payshopp.net). The result is likely faster response times than BigQuery for the retrieval step. End-to-end Q&A latency is also optimized since the LLM generation part is integrated (no additional network hops between retrieval and answer generation). This makes interactive chat feel snappy.
  • Ops Burden: Low – This is a fully managed solution. You ingest data once (via console or API), and Vertex AI Search handles chunking, embedding, indexing, and retrieval behind the scenes (cloud.google.com). It continuously manages the index (scaling, optimization) without user intervention. New data can be added via an ingestion pipeline, but you don’t worry about how embeddings or indexes are stored. Maintenance tasks like hardware provisioning, index rebuilds, etc., are abstracted away. The trade-off is less control over the internals, but far less day-to-day upkeep. Deploying a new search app can be done with a few clicks or API calls (cloud.google.com).
  • Managed Integration: High – Vertex AI Search is part of the Vertex AI ecosystem, which means it natively integrates with other managed services. For example, it can serve as the retrieval backend for Vertex AI’s RAG Engine or conversational agents (payshopp.net) (cloud.google.com). It provides an out-of-the-box Q&A interface (including a prefab search UI widget and REST APIs). Features like the Grounded Generation API, Safety filters, and self-learning rerankers are available to use on top. In short, it unlocks options like one-click “Chat with your data” agents, without needing to assemble the plumbing yourself.

• Approach C – Hybrid (BQ + Vertex combo):
  This approach would use BigQuery for certain retrieval or storage tasks and Vertex AI Search for others. Two possible interpretations: (1) BigQuery retrieval + Vertex AI for Q&A – e.g., use existing BQ search to get top N documents, then feed those to a Vertex LLM (or Vertex’s Ranking and Grounding APIs) for answer generation; or (2) Vertex AI Search retrieval + BigQuery for analytics – e.g., keep data in BQ for analysis but also ingest it into Vertex for user-facing search. We’ll consider the trade-offs in general:

  • Retrieval Quality: Potentially high, but complexity is added. If using BQ then Vertex LLM, the retrieval quality depends on BQ (with its limitations on semantic nuance). If using Vertex retrieval and BQ as secondary, Vertex ensures high quality search. In practice, a hybrid might not improve search quality over pure Vertex – it’s likely done for other reasons (cost or continuity). One could also combine results: e.g., use BQ’s BM25 text search and vector search and Vertex’s semantic results together in a fusion. That could marginally boost recall, but at cost of complexity. Since Vertex Search already supports hybrid (keyword + semantic) internally, a double-hybrid is probably unnecessary.
  • Analytics Capability: High – by keeping BigQuery in the loop, you retain the ability to query the raw data. For instance, you might store all chunks in BQ (for analysis and backup) but also index them in Vertex for search. This way, operational decisions or deeper analyses can query the BQ dataset, while end-user queries hit Vertex. The cost is data duplication and pipeline complexity, but it gives the best of both: Vertex for retrieval and BQ for analytics.
  • Latency: Moderate – a hybrid pipeline likely adds latency. For example, if you retrieve via BigQuery then call Vertex rerank or generate, that’s two calls (BQ query, then LLM). If you retrieve via Vertex but still need a BQ lookup, similarly you might introduce extra steps. Direct Vertex AI Search (Approach B) would be faster. However, if latency is not critical or query volume is low, a slight increase is tolerable. Caching could mitigate this if repeat queries occur.
  • Ops Burden: High – you now maintain both systems. All of Approach A and some of Approach B apply. For instance, you might need to run a pipeline to feed Vertex AI Search from BigQuery exports, monitor two indexes, and handle consistency (ensure updates go to both places). Unless there is a compelling benefit, this might be over-engineering for a single-user scenario. We’d only choose a hybrid if we truly need capabilities that one service alone cannot provide. In the new context (no residency requirement), Vertex AI Search alone can likely handle all retrieval needs, so pure approach B is preferable for simplicity.
  • Managed Integration: Medium – you do gain the integrations of Vertex AI Search, but still rely on custom code for the BQ parts. For example, your chat app might call Vertex AI Search for most queries but fall back to a BQ query for some structured lookup or to access data not indexed in Vertex. You’d need to orchestrate these paths in code. It’s doable (especially with conditional tool use in an agent), but again, it adds development overhead.

Conclusion: With residency no longer a concern and emphasis on quality and future expandability, Vertex AI Search (Approach B) emerges as the best-fit core. It offers superior retrieval quality (semantic understanding, synonyms, citations) and minimal ops overhead, while opening the door to advanced Vertex capabilities (managed Q&A, agents, etc.). The BigQuery approach was valuable under strict regional/privacy constraints, but now it appears suboptimal given the available managed alternative. Vertex AI Search’s built-in LLM and grounding features directly address the “grounded citation quality” requirement – something we’d otherwise have to build ourselves (cloud.google.com).

That said, it’s wise to continue leveraging BigQuery for analytics. We can maintain the source of truth (the archive text) in BigQuery for analysis and audit, and feed the same data into Vertex AI Search for actual querying. This dual setup takes a bit more initial work (setting up ingestion), but it ensures we meet both retrieval quality and analytics needs. In summary, RQ-105 excluded Vertex AI Search only due to residency limits; with that barrier removed, the arguments to use Vertex AI Search as the core are strong. The only reasons to not use Vertex now would be cost (addressed in §7) or needing extremely fine-grained control. Given our scale and goals, those are secondary. Vertex AI Search should become the core retrieval backend, with BigQuery supporting as an analytics and data preparation layer.

2. Embedding Model Selection

We now consider the embedding model choices for representing our archive (which is a mix of Japanese and English content). The goal is to improve semantic recall@10, clustering/topic purity, and overall embedding quality for mixed-language data. The candidates are:

• (1) embeddinggemma-300m (768-dim) – Current model in use (Tokyo region in-slot). This is a 308M-parameter model from Google/DeepMind, optimized for on-device and multilingual use (deepmind.google). It yields 768-dimensional embeddings. Pros: Runs in-slot (no external API calls, minimal cost/latency), and strong multilingual performance for its size (trained on 100+ languages) (deepmind.google). In fact, EmbeddingGemma is the highest-ranking open model under 500M params on the MTEB benchmark (deepmind.google) – it’s quality-leading among lightweight models (61 average score on MTEB ([deepmind.google](https://deepmind.google/models/gemma/embeddinggemma/#::text=Image%3A%20Scatter%20plot%20titled%20%27MTEB,similarly%20sized%20models%20like%20%27gte))). It likely handled our Japanese-English data reasonably well. Cons: Being a smaller model, its embedding quality, while good, may not match larger latest-generation models. Complex semantic relationships or cross-language nuance might be missed or lower in accuracy. Also, because it’s run locally, we had a fixed 768-dim output. Newer models with higher dimensions capture more information. Overall, Gemma provided excellent cost-efficiency, but we might be hitting a quality ceiling with it.

• (2) text-embedding-004 (768-dim) – Google Vertex AI API model (GA in Tokyo & US regions). This appears to be the latest generation of Google’s general-purpose text embedding model (possibly an evolution of the “Gecko” series or similar). Pros: It’s available in our regions and likely has up-to-date training. Although specifics aren’t publicly documented in detail, we expect it to be on par with or better than EmbeddingGemma in quality. Being GA, it’s production-ready. It also outputs 768-dim vectors, so it’d be a drop-in replacement dimension-wise. Cons: If it’s primarily monolingual or English-optimized, it might not capture Japanese semantics as well as a dedicated multilingual model. We’d need to verify its multilingual capability. That said, Google’s newer embeddings usually do handle major languages, and the fact it’s available in Tokyo suggests it likely supports Japanese. Cost is low – roughly on the order of $0.03 per million chars embedded (vercel.com), so negligible for our volume. This model could be a safe intermediate choice if we want a step up from Gemma without jumping to very large embeddings.

• (3) text-multilingual-embedding-002 (768-dim) – Google Vertex AI multilingual embedding (GA in US/EU). This model is specialized for cross-lingual and multilingual retrieval, covering 18 languages (including English and Japanese). It achieves about a 56.2% average score on MIRACL, a benchmark for cross-language info retrieval (vercel.com). Pros: Designed for scenarios where a query in one language needs to find content in another. In our archive, while most conversations likely contain both languages intermingled (and ADRs/RQs might be bilingual in content), having strong cross-lingual understanding could be useful. For example, if an ADR is written in English but a question is asked in Japanese phrasing, this model might better align them. Cons: Focusing on cross-lingual performance sometimes comes at a slight cost to monolingual performance. Its overall embedding quality (e.g., semantic clustering, MTEB score) might be a bit lower than same-generation English models. If our usage is mostly within each language (e.g., Japanese Q -> Japanese context, English Q -> English docs), a higher-accuracy monolingual model could outperform it. Also, it’s only available in US/EU, meaning slightly higher latency from Tokyo if used live (though we no longer mind US processing). Cost is similarly low (about $0.03 per 1M tokens). This model is a strong candidate if we observe cross-language retrieval issues with others.

• (4) Gemini text embedding (latest, e.g., gemini-embedding-001 or newer) – Google’s state-of-the-art foundation model for embedding. Gemini embedding models are now available (GA in US/EU), offering 3,072-dim output vectors and trained on multimodal data. Pros: These have top-tier semantic performance – for instance, gemini-embedding-001 scores about 65.4 on MTEB, significantly higher than smaller models (embeddingcost.com). It’s tuned for a wide variety of tasks and should handle multilingual input well (Google has likely ensured Gemini supports major languages; it’s the successor to PaLM). It also supports Matryoshka representations (MRL), meaning we could possibly reduce dimension if needed while retaining performance (deepmind.google). With Gemini, we maximize recall and embedding fidelity – subtle contextual cues or relationships are more likely to be captured. This directly can improve cluster purity (related content will cluster tighter in vector space) and similarity search accuracy. Cons: The vector is large (3072 dims), which has implications on storage and latency. Memory: each embedding is 4× larger than a 768-dim one. If our archive grows to 50k chunks, a single 3072-dim vector (float32) is ~12 KB, so 50k is ~600 MB just for raw vectors. BigQuery can handle this, but indexes will use more RAM and potentially slightly slower ANN search due to higher dimensionality. However, given our scale, this is still manageable – and BigQuery’s vector search can be optimized with IVF indices (which are mostly affected by number of vectors rather than dimension). Another con is cost: Gemini’s embedding API is pricier than smaller models (approx $0.15–$0.20 per 1M tokens input ([embeddingcost.com](https://embeddingcost.com/compare#::text=200M%20tokens%20%20%7C%20High,sensitive))). Even so, embedding our entire dataset (~27 MB docs + conversation text) might cost only a few dollars – trivial in monthly terms. Considering our priority on quality over cost, this is acceptable. Latency for embedding calls might be slightly higher (bigger model), but since ingestion is offline/batch, that’s fine. For queries, Vertex AI Search likely uses a strong embedding internally (maybe Gemini or similar), so if we go that route, we’d implicitly use something of this caliber.

• (5) OpenAI text-embedding-3-large (3072-dim) – OpenAI’s latest embedding model (presumably GPT-4 derived). This model provides 3,072-dim vectors and is likely one of the most powerful on the market in terms of pure semantic accuracy. It reportedly scores around 64.6 on MTEB (embeddingcost.com), comparable to Gemini’s range. Pros: It may excel at nuanced understanding, given OpenAI’s training, and could handle multilingual content reasonably (OpenAI’s models are quite capable in many languages, though perhaps not as tuned on Japanese as Google’s). If we were using an OpenAI-based pipeline for other things, keeping everything in that ecosystem might simplify integration. Cons: It’s an external API outside our GCP environment. This means data leaves our GCP project and goes to OpenAI – we’ve allowed that now (no strict privacy concerns, since it’s our own data), but it’s still a consideration. Integration-wise, we’d need to call OpenAI’s API for every embedding, manage API keys, etc., instead of using GCP’s managed services. Cost is also a factor: OpenAI’s embeddings are not as cheap as Google’s. From the latest pricing, text-embedding-3-large is around $0.13 per 1K tokens ($130 per 1M) (embeddingcost.com) – actually much higher than Google’s $0.15 per 1M for Gemini. (It appears OpenAI 3-large is ~$0.13 per 1K, which is $130 per 1M; by comparison Gemini is $0.15 per 1M – almost three orders of magnitude cheaper, if those figures are correct.) Even if those numbers are off, OpenAI traditionally has charged more for embeddings than Google. Hence, cost could approach tens of dollars for a full re-embedding, versus a few dollars with Google – still not huge, but notable. Also, using OpenAI means capability breadth is limited (we can’t easily plug these vectors into Vertex AI Search, for example – Vertex Search would use its own embedding anyway). Given our new strategy to leverage Vertex AI’s ecosystem, bringing OpenAI in for embeddings might complicate things. And finally, OpenAI’s model might not be fine-tuned for enterprise retrieval tasks in the same way (Google and Cohere emphasize MIRACL, MTEB, etc., whereas OpenAI’s are general). The quality difference between OpenAI 3-large and Google’s Gemini is likely minimal for us – both are top-tier.

Impact of Embedding Dimension Increase: Choosing a model with 3072 dimensions (Gemini or OpenAI) instead of 768 (Gemma or Text-004) will roughly quadruple the size of our embedding index. This means higher memory usage and slightly longer computation for similarity searches (computing a distance in 3072-D takes 4x the operations of 768-D). BigQuery’s vector search index (IVF) will also take more storage. However, with only 50k vectors, this is well within modern capabilities. BigQuery compresses index data and can handle millions of vectors, so 4× more is not a deal-breaker. We might see a small increase in query latency (e.g., an ANN search might take, say, 50ms instead of 30ms) but still very fast compared to network or LLM time. The benefit is richer semantic representation: higher-dim embeddings often separate concerns into different vector subspaces, leading to more accurate similarity calculations in high-dimensional problems. Empirically, going from 768→1536→3072 dims has yielded a few percentage points improvement in retrieval metrics in benchmarks ([embeddingcost.com](https://embeddingcost.com/compare#::text=,100%2B%20languages)) (embeddingcost.com). For example, OpenAI’s 1536-dim to 3072-dim jump improved MTEB score from 62.3 to 64.6 ([embeddingcost.com](https://embeddingcost.com/compare#::text=,100%2B%20languages)). So we might expect a small but non-trivial gain in recall@10 or topic purity by using a larger model. It’s a classic quality vs. efficiency trade-off.

Recommendation: Since information extraction quality is top priority, we should opt for the most powerful embedding available within our practical constraints. That likely means using Google’s Gemini embedding model (assuming it’s production-ready for text) for our new index. It offers state-of-the-art semantic retrieval performance and keeps us within the GCP ecosystem (which aligns with our “capability breadth” goal – e.g., easier integration with Vertex AI Search or Vertex Vector Search). The cost per embedding is so low in absolute terms that we can afford it, and the latency impact is negligible for our use case.

However, we should also be mindful of our data characteristics: predominantly bilingual (JA/EN). Gemini should handle that, but if we encounter issues where a Japanese query isn’t retrieving an English document it should, we could consider a cross-lingual model like text-multilingual-002. Another approach is to use two models: e.g., create two sets of embeddings, one from a multilingual model and one from a high-accuracy model, and fuse results. This might be overkill unless we detect a problem. Most likely, a single high-quality model (Gemini) will suffice, as it’s been trained on multilingual data as well. (EmbeddingGemma’s success on multilingual tasks (deepmind.google) gives confidence that Google’s larger models are also strong in that regard.)

In summary: switch to Gemini 3k-dim embeddings for the new US-based archive. This maximizes semantic recall and future-proofs us (Gemini will integrate well with Vertex AI Search and any upcoming features like auto-embedding fine-tuning). We will regenerate all chunk embeddings with this model (details in migration steps below). The slight increase in index size and query latency is acceptable given our scale. OpenAI’s latest model is a close alternative in quality, but considering integration, cost, and data locality, sticking with GCP’s offerings is the better path. We should monitor recall@10 on a validation set after re-embedding – it wouldn’t surprise us to see recall improve from 0.90 to >0.93–0.95 with the more powerful model, especially on queries that require understanding nuanced context or cross-language references. Cluster purity (e.g., grouping related ADR topics) should also improve, making analytics like topic clustering more meaningful.

3. Vertex AI Search vs. Custom Grounded Q&A

In RQ-105, due to Tokyo residency limits, we devised a custom grounded Q&A pipeline: BigQuery for retrieval, a local Gemini (Tokyo) model for generation, and Vertex APIs (Ranking, Grounding Check) for refinement. Now we can consider using Vertex AI Search’s turnkey Q&A instead. Let’s compare the two approaches:

Decision: Given the above, Vertex AI Search’s turnkey Q&A is the preferred choice now. It significantly reduces complexity while providing high-quality retrieval and grounded answers. The integration of retrieval+generation means we get end-to-end “question to cited answer” functionality with minimal development. This directly addresses our KPI of grounded citation precision – Vertex is designed to output answers with citations and can be audited via its grounding score. In contrast, the custom route, while powerful, would require us to implement and tune many pieces that Vertex Search offers out-of-the-box (and likely with higher quality, benefiting from Google’s internal testing/tuning).

One crucial point: Vertex AI Search was previously excluded solely because of data residency. Now that we only deal with our own non-sensitive data and accept US processing, that concern evaporates. The potential upside of Vertex – such as auto-OCR and parsing of docs, continuous improvements, and alignment with future Vertex features – outweighs the remaining benefits of a custom build. We also regain features like the Vertex Search widget (for easily embedding a search box in our docs site) and the ability to use Vertex Conversational Agents with our data. These were not possible under the old plan.

There may be niche cases where we still do something custom (for example, a specialized analysis query where we generate a table or do something analytical – that might use BigQuery + an LLM via AI.GENERATE_TABLE rather than the search service), but for general Q&A and search, Vertex AI Search should handle it.

In summary: We will “bring Vertex AI Search back to core” as our RAG backbone for Q&A. The custom pipeline from RQ-105 will be deprecated or used only as a fallback. We will ingest our archive into Vertex AI Search (likely as a Search DataStore + Search Application in us-central). Our chat interface (intake-chat) can call the Vertex Search API for retrieve or even directly get answers. If needed, we can still call the Check Grounding API on Vertex’s answer to double-verify correctness, but we might find the answers are already reliable. The Vertex approach scores extremely well on our evaluation axes: info extraction quality (Google relevance + fewer hallucinations), capability breadth (integrates with Agents, future Gemini upgrades, etc.), and ease of ops (dramatically simpler pipeline). The cost is higher than DIY on paper, but still low in absolute terms (see §7), and residency is no longer a blocker. Thus, the balance now tilts heavily in favor of managed Vertex AI Search.

(As an aside, we should document this decision in an ADR format for posterity – essentially overturning the RQ-105 decision of a fully custom RAG in favor of a Vertex-managed solution, due to the changed constraints.)

4. BigQuery Dataset Location Migration Strategy

We need to migrate our existing BigQuery archive dataset from asia-northeast1 (Tokyo) to a US multi-region location, and simultaneously plan for re-building embeddings if the model changes. Below is the recommended strategy and key considerations:

a. Create a New Dataset in US:
In BigQuery, dataset locations are fixed at creation. We will create a new dataset (e.g., archive_us) in a US multi-region (probably “US” which spans multiple US data centers). This will be the target for all our tables (chunks, embeddings, etc.). We’ll also set it up with the same structure (tables, schemas) as the Tokyo dataset. Note that you cannot simply switch an existing dataset’s region (docs.cloud.google.com) – creating a new one is required.

b. One-time Data Copy (Cross-Region):
We have to get the existing data (~38k chunk records + associated embeddings) over to the US dataset. There are a couple of methods:

• Method 1: BigQuery Data Transfer Service – BigQuery now supports cross-region dataset copy in Beta (docs.cloud.google.com). We can use the “Dataset Copy” feature to copy archive (Tokyo) to archive_us (US). This automates the extraction and import behind the scenes. It handles data compression during transfer to minimize cost (docs.cloud.google.com). We should ensure we have appropriate permissions (BigQuery Admin roles) and then initiate a copy. This could be done via the GCP Console UI or via a bq command. Since our dataset is not huge (likely a few hundred MB at most), this will be quick. The transfer service will replicate tables and their data. However, note that some things may not copy: for example, BigQuery indexes (like any SEARCH or VECTOR indexes on the tables) might not be carried over – those might need to be recreated later in the new dataset.

• Method 2: Manual Export/Import – Alternatively, we could manually export tables from Tokyo to GCS (in US region) and then load into the US dataset. For instance, export chunks and chunk_embeddings as CSV or Avro files to the US GCS bucket, then use LOAD DATA into the new dataset. Given BigQuery’s transfer feature, this is probably unnecessary extra work, but it’s an option if we want full control. The export files could also serve as a backup.

Using Method 1 is straightforward: it’s essentially a no-code approach and will preserve table schemas. We should double-check after copy that the row counts match and the data looks correct in archive_us.

c. Dual-Running / Consistency:
During the transition, we might want to keep the old dataset intact until we fully switch over. The copy operation is one-time (but we could re-run it if needed). A safe approach is: perform the copy, verify all data is in US, then update our applications to point to the new dataset. For a time, we can maintain both (the old archive in Tokyo can be left read-only as a fallback). Since going forward we’ll likely recompute embeddings, the old and new data won’t be identical at the vector level, but the chunk text and IDs should initially match. It might be useful to keep the same primary keys/IDs in the new dataset so that references remain consistent.

d. Regenerating Embeddings (if dimension/model changes):
Because we plan to switch embedding models (e.g., from 768-dim Gemma to 3072-dim Gemini), we will recompute the embeddings for all chunks. There’s little value in copying the old chunk_embeddings data as-is, since those vectors are tied to the old model. Instead, we can do the following:

1. Copy or export the chunks table (text data) to the US dataset. This table with chunk IDs, text, metadata (doc id, etc.) should be copied to preserve all chunk references. We’ll use it as the source for new embeddings.
2. In the US dataset, create a new table for embeddings (e.g., chunk_embeddings with a vector column of the new dimension). The schema will differ (3072-length array instead of 768).
3. Run an embedding job over all chunks. This can be done via a Python script or notebook using the Vertex AI API for the chosen model. Because we have BigQuery and Vertex AI in the same region now, we could even explore BigQuery ML’s remote model inference (for example, using the AI.EXPLAIN or similar functions) – but it’s likely simpler to use a Python client to fetch chunks in batches, call the embed model, and write results. Another approach: Vertex AI Search itself, if we use it, will embed on ingestion (so we might only need to ingest the text into Vertex and not store in BQ at all). However, we probably want the embeddings in BQ for our own query/analysis use too.
4. As we generate embeddings for each chunk, insert them into the chunk_embeddings table (or if using BigQuery directly, possibly perform an UPDATE... But better is to rebuild from scratch to avoid any inconsistency).
5. Verify a few random entries to ensure embeddings were generated correctly (e.g., not all zeros, and that similar chunks have similar vectors qualitatively).

Because the embedding dimension changed, any existing BigQuery VECTOR INDEX on the old 768-dim column can’t be reused. We will need to create a new index on the 3072-dim column. In BigQuery, we’d do something like:

CREATE INDEX archive_us.idx_chunks_embedding 
ON archive_us.chunk_embeddings(embedding) 
STORING(chunk_id, text) 
OPTIONS(..., algorithm = 'IVF_FLAT', ...);

The exact syntax might vary, but essentially, re-indexing is needed. This will add some one-time cost (BigQuery will build the IVF structure), but for 50k vectors it should be quick (minutes). If our search was using a combined FULLTEXT index (for keywords) in BigQuery, we’d also recreate that on the chunks text column in the US dataset, if we plan to do hybrid search in BQ.

e. Update References & Testing:
Once data is moved and new embeddings are in place, we need to update any code/configuration that pointed to the old dataset. For example:

• The retrieve_evidence tool in our intake-chat – ensure it queries archive_us.search_hybrid() UDF (or whatever we set up) instead of the Tokyo one.
• Any BigQuery ML or views that referenced bizlp-gas-accounting-dev.archive... should point to archive_us.
• Our Claude Code / Jupyter integrations (via BigQuery proxies) should likewise query the new location. (We might temporarily enable the old and new side by side so we can compare results easily.)

It’s wise to run some search queries on the new dataset and compare with the old outputs to ensure the migration didn’t lose anything. Because we’re also changing embedding model, results will differ – ideally for the better, but we should validate on known queries if the relevant results still appear.

f. Decommission Old Dataset:
After a period of parallel run and confidence building, we can delete or archive the old Tokyo dataset. If there’s any concern we might need it (perhaps for fallback or verification), we could keep it around but not update it. BigQuery storage cost for 38k rows is trivial, so it doesn’t hurt to keep as backup if desired. Or we could export it to GCS (as JSON files) as a historical archive, then drop it. The main goal is to avoid confusion and ensure all new workflows use archive_us.

Additional Considerations:

• Cross-region cost: Copying data from Tokyo to US will incur some egress charges, but Google’s transfer service compresses data in transit (docs.cloud.google.com). Our dataset might be, say, <1 GB, so even uncompressed egress (at $0.12/GB) is negligible (~$0.12). With compression, possibly much less. So cost isn’t a big factor here.
• Time synchronization: If new data is still being added (though in our case, the archive is mostly static or grows slowly), we should pause ingestion during migration to avoid missing anything. Since it’s a one-user archive, we can likely have a quiet period for the transfer.
• Testing search_hybrid TVF: We had a table-valued function archive.search_hybrid(q, topk) in the old dataset implementing hybrid search. We should recreate this in archive_us (adjusting to new table names/fields if needed). If embedding dimension changed, the UDF logic might need minor tweaks (e.g., distance function calls).
• Deploying new embedding model: Ensure the Vertex AI project has access to the chosen embed model (enable the API, check quotas). For example, to use gemini-embedding-001, our project might need to be allow-listed or have the correct permissions since Gemini might still be in preview as of mid-2026. But given it’s GA in the prompt, should be fine.
• Parallel search index: During migration, we can even stand up a parallel Vertex AI Search index on the US data to test. That means we could ingest the US-based data into Vertex Search and try some queries before fully switching our chat to use it. This isn’t directly about BQ, but it’s part of overall migration to the new architecture.

By following these steps, we ensure a smooth transition of data location. The process essentially is extract, copy, re-index, swap. BigQuery’s tooling makes cross-region moves quite straightforward now (docs.cloud.google.com), which reduces risk of human error. The biggest “moving part” is the embedding recomputation, but that’s an opportunity to improve our system.

In summary, our plan:

1. Set up archive_us dataset in BigQuery (US multi-region).
2. Copy base data (chunks table, etc.) from Tokyo to US using Data Transfer (preserving text and IDs) (docs.cloud.google.com).
3. Recompute embeddings using the new model for all chunks, populate the US embeddings table (with new schema).
4. Recreate indexes (vector and text) in archive_us.
5. Update application configs to use archive_us.
6. Test search and chat on new dataset (both via BigQuery and via Vertex Search, if applicable).
7. Deactivate old dataset once satisfied.

This way, we minimize downtime and ensure the new environment is fully ready before cutting over. The old PoC (intake-chat-rag-sidecar) should remain functional during the process (pointing at old data) until we switch it at the end, at which point it will query the new dataset transparently. Because we keep the interface (UDF name, etc.) the same in the new dataset, the chat tool might just need a dataset name change in its BigQuery query string.

One more note: If we want to avoid complexity, we don’t necessarily need BigQuery for embeddings at all if going all-in on Vertex AI Search (since Vertex Search has its own store). But maintaining a BigQuery copy of chunks+embeddings is useful for our analytics and for validation. We’ll likely do both ingestion paths: BigQuery for analysis, Vertex AI Search for serving.

5. Additional Ingest of Docs/ADR/RQ/D1 and Chunking Strategy

We plan to extend the archive with new content sources: roughly 856 Markdown files from our docs site (including internal docs, architecture docs, operations docs), ADR documents (166), Research Questions (RQs) (94), and possibly D1 telemetry data. In Phase 2 of the PoC, these were to be ingested with a source tag. Now under the new approach, we will re-evaluate how to chunk and index this data in the US dataset with the new model.

Content Characteristics:

• Docs/ADR/RQ: These are Markdown text files, varying in length. They often have a hierarchical structure (sections, bullet points, etc.). Each file is a standalone document with a title, and many have subheadings. The content is a mix of prose, lists, maybe tables or code snippets (especially in architecture docs).
• D1 telemetry: Unclear from context, but likely some log or structured data. If it’s text (like event descriptions or error logs), it could be chunked; if it’s numeric or tabular, it might not be suitable for semantic search (perhaps better kept in a database for direct query).

Chunking Strategy:

The goal of chunking is to strike a balance: chunks should be small enough to be specific (so irrelevant content isn’t dragged in) but large enough to contain a complete thought or answer so the LLM has sufficient context. A well-known guideline is that 200–400 tokens per chunk is the sweet spot for many embedding-based retrieval tasks ([ashishsinghal.dev](https://ashishsinghal.dev/insights/rag/chunking-strategies#::text=Rule%20of%20thumb)). Too small (50-token snippets) and the meaning may be fragmented, hurting recall. Too large (1000+ tokens) and the chunk may dilute the relevance score with unrelated info.

Given our content, we propose a hierarchical, content-aware chunking approach:

• Split by Document Structure: Use the Markdown structure to guide chunking. We can split each file at top-level or second-level headings. For example, each ## section (plus maybe the subsequent text until the next ##) becomes a chunk. This respects logical divisions the author made. For ADRs and RQs, which might not be very long or have many headings, one chunk per ADR might suffice if short, or per section if long.
• Ensure Size Limits: If a section is very large (say >400 tokens), further split it. We can use paragraph breaks or subheadings (###) as secondary split points. If even paragraphs are long, we could fall back to a fixed-size chunk (e.g., split every ~300 tokens) within that section. But try to cut at semantic boundaries (sentence ends, paragraph ends) to keep chunks cohesive.
• Avoid Overlap vs. Overlap: We need to decide if chunks should overlap (e.g., sliding window) or be distinct. For our documentation, distinct chunks are fine because each section usually stands on its own. Overlap is more useful in narrative text to capture context that straddles boundaries, but in docs, better to keep them separate and let the retrieval pull multiple chunks if needed.
• Metadata tagging: Each chunk will carry metadata indicating the source document and section. For example, metadata could include: source type (ADR vs RQ vs docs vs conversation vs telemetry), document title or ID (like ADR number, RQ number, file path), and the section heading it came from. This is crucial for both debugging and for generating proper citations in answers.

Impact on Recall: Breaking documents into chunks will improve recall of relevant information. If we left an entire file (which could be several pages) as one chunk, a query might get a low similarity score unless the whole file is on-topic. By chunking, the specific relevant section can be retrieved. The trade-off is that an answer might need multiple chunks to cover the whole context, but our retrieval can return top-3 or top-5 chunks easily. For example, if a doc has two relevant sections for a query, those could both appear in the top results instead of the doc being a single result. This increases recall@N (the chance that at least one of the top N chunks contains the answer). It may slightly reduce precision if too many tiny chunks on the same topic appear, but since we prioritize recall and have ranking to sort relevance, that’s acceptable.

Chunk = 1 file vs smaller: Definitely we should not do 1 file = 1 chunk for anything but very short files. Many docs have multiple distinct sections; a query likely pertains to one section. One-chunk-per-file would mean the embedding is like an average of all topics – relevant only in a broad sense. It would hurt retrieval. So we’ll do at least section-level chunking. Possibly even paragraph-level in long narrative sections.

For example, consider an ADR that has sections: Context, Decision, Consequences. If someone asks “What were the consequences noted in ADR-0123?”, having a chunk that is just the Consequences section is ideal. If the whole ADR is one chunk, the query embedding might not closely match because the ADR chunk also contains context and decision text. By chunking, the Consequences chunk will be a direct hit.

This strategy aligns with known RAG best practices where each chunk = a semantically coherent piece of knowledge (often a paragraph or section).

Chunk Size Tuning: After initial chunking, we should check sizes. We can programmatically count tokens (using a tokenizer) for a few sample chunks. If some chunks exceed let’s say 500 tokens, we might split them further. If some are extremely short (like titles alone), we might decide to merge a short section with the following section to give context, unless it’s really separate. There’s some art to this, but the 200-400 token rule (ashishsinghal.dev) is a good target range.

D1 Telemetry: It’s unclear what format this is. If it’s textual (e.g., incident reports, commit messages, or log lines with descriptions), we can treat each entry or a day’s logs as a chunk. Telemetry data might also be better suited for structured queries than semantic search. However, since Phase 2 included it, perhaps they want to search it too. We could embed e.g. each log message or each event description as a chunk (with metadata like timestamp). This might allow queries like “when did we last deploy version X?” to retrieve a log line. The chunking for telemetry would likely be trivial (each record is already small). We’d definitely tag these with source “telemetry” so we can filter or identify them in results.

Indexing and Source Field: When we ingest these new chunks into BigQuery and Vertex AI Search, we will include a source column (as mentioned). Likely a column source_type with values like “conversation”, “ADR”, “RQ”, “docs”, “telemetry”. Possibly also a source_id (like the file name or ADR number). This allows:

• Filtering: e.g., if we want to restrict a query to only ADRs, we could add a filter.
• Structured citations: we can use source info to format the citation (if answer comes from an ADR chunk, cite it as “ADR-XXX”, if from a doc, maybe cite the doc title).

In BigQuery, we can partition or cluster by source_type if that helps queries (not too important unless dataset grows massive).

Procedure to Ingest New Docs:
We will write a script to iterate over each Markdown file:

• Parse the Markdown (we can use a library or simple regex for headings).
• Split into sections (e.g., using heading markers).
• Generate chunk records: for each section, create a text snippet. Possibly prefix the chunk with the section title for context (if needed, though the model might not need it if we store section title separately).
• Store chunk text along with metadata (doc name, section title, source_type).
• (Optional) For very large sections, do further splits as discussed.
• Push these chunks into the BigQuery chunks table (or a new combined table that includes all sources; we might unify conversation chunks and doc chunks in one table with the source field, which is simplest for unified search).

After that, we generate embeddings for each new chunk using the chosen model and add them to the chunk_embeddings table. If using Vertex AI Search, we also ingest these chunks into the search index with their metadata (Vertex might do the chunking for us if we just feed whole files – but we prefer control over chunking to ensure consistency with our BigQuery approach).

Recall Effects: With these additional chunks, our total corpus size increases (perhaps from ~38k to ~50k or more chunks). Typically, a larger corpus can slightly decrease precision (more candidates to choose from), but because we’re also improving embedding quality, we expect recall@10 to remain high or even improve for questions answerable by these docs. The key point: previously, certain queries about design decisions or architecture could only find answers in conversation transcripts (if at all). Now, with ADRs and docs ingested, those queries will retrieve the actual authoritative source. This improves the quality of answers (and reduces hallucination, since the LLM can quote the actual doc). We should keep an eye on queries that are generic – e.g., “what is our architecture for X?” might return multiple chunks from different docs; we’ll rely on the ranker or LLM to consolidate.

Example: If a user asks a question that was discussed in a conversation (in the archive) and also formally written in an ADR, we might get both sources in results. That’s actually good: the LLM can then choose the more authoritative one (likely the ADR) to cite, or even cite both (“According to ADR-123 and a related discussion on 2023-05-06, ...”). Vertex AI Search’s self-learning might eventually learn to prioritize ADR chunks if they consistently provide direct answers.

Potential Chunking Pitfall – Context windows: One risk of very small chunks is that the model, when seeing them, might miss context that was in an adjacent chunk. Our solution is to ensure important context isn’t orphaned. For example, if a section heading is short and its content is separate, the heading might not mean much alone. In such cases, it’s wise to attach the heading to the first paragraph chunk of that section. We can do that by prefixing chunk text with the section heading or combining a short heading line with the next paragraph when chunking. This way, the chunk has enough context to be meaningful. We’ll do that in our parsing logic.

Performance Consideration: More chunks means the vector index is bigger, which can increase query time slightly, but as discussed, going from 38k to ~50k is not a big jump. It’s still peanuts for BigQuery or Vertex to handle. We might need to raise top_k slightly in retrieval if relevant info is spread across sources, but top 10 is usually fine.

Ingestion Pipeline Parallelization: We should be able to ingest docs/ADRs fairly easily. It’s on the order of ~850 files; if each becomes say 3-5 chunks on average, that’s a few thousand chunks – nothing heavy. We can embed them in batches (maybe 100 chunks per API call if using batch endpoints, or one by one – either is fine given volume).

Quality Assurance: We’ll test a few known queries after ingesting, such as:

• “Where was X decision documented?” expecting an ADR chunk to surface.
• “Summarize how feature Y works” expecting architecture docs chunks.
• Compare the answer from conversation vs docs: ideally, the answer should now use the docs for factual info and perhaps reference the conversation for additional context if needed.

Conclusion: We will ingest all Markdown files (docs, ADRs, RQs) by chunking them into section-sized pieces. This method maximizes recall and maintains context integrity. The source field in each chunk will allow us to trace answers back and potentially filter (for instance, the docs site search might choose to only show chunks from docs and ADR sources, excluding casual conversation chunks unless explicitly asked).

As for D1 telemetry, if it’s textual, we’ll treat each entry or logical group as a chunk (with source “telemetry”). If it’s structured numeric data, we may skip embedding it – instead, that could be left for direct BigQuery analysis or some other approach. Perhaps we can clarify what “D1 telemetry” contains and decide. If it’s something like user behavior logs or performance metrics in text form, we might ingest summary notes or anomalies as text chunks.

Summing up:

• Use semantic chunking of docs: each section = 1 chunk, further split if >~300 tokens.
• Preserve structure via metadata (doc title, section name).
• Maintain an identifier for each chunk linking to the original file and section (for citation URL generation – we can even store the relative URL or anchor).
• The added ~10k chunks will be embedded and indexed just like the conversation chunks.
• Expect improved coverage of topics and higher answer quality for queries related to these official documents.
• Monitor recall by source: e.g., do we retrieve at least one relevant ADR chunk for queries about decisions (that would be a metric to track as a KPI – how often the relevant ADR is retrieved in top 5 for a decision-related question).

By re-evaluating Phase 2 ingestion under the new plan, we affirm that it’s even more beneficial now: Vertex AI Search, for example, has layout parsing and handling for docs (docs.cloud.google.com), which means even if we feed whole PDFs or markdown, it could chunk internally. But we prefer to control chunking to ensure consistency with how we query in BigQuery. The ingestion of docs/ADR will significantly enrich our knowledge base, enabling the assistant (and us via queries) to draw on the explicit written records rather than only chat transcripts.

6. Expansion Potential: Managed Agent, AI Functions, Dashboards

With the data residency restriction lifted, we can now leverage additional managed AI services to get more value from our archive. We discuss a few possibilities – Vertex AI Conversational Agents, BigQuery’s AI.GENERATE_ functions*, and Looker Studio dashboards – and consider their ROI for a 1-person company at the dogfooding stage.

• Vertex AI Conversational Agent (Managed Agent): Now that our data can be used in Vertex’s Agent platform, we could set up a Conversational Agent powered by our knowledge corpus. Vertex AI’s Agent Search (Gemini Enterprise Agent Platform) basically combines an LLM with our Vertex AI Search backend (cloud.google.com). We could configure an agent with our search index as a knowledge connector (grounding source). The agent would be able to have multi-turn dialogues, maintain context, and even use tools (like making calculations or API calls) if we set that up. The benefit of using a managed agent: it provides an off-the-shelf chat interface (or an API) that can handle things like follow-up questions, clarifications, and certain tool integrations without us coding the logic. For example, if the user asks, “Show me the quarterly trend of ADRs,” a well-configured agent might automatically invoke a tool (like BigQuery or Looker) if set up, or at least respond it cannot. Vertex Agents also come with safety and guardrails by default, which could be helpful if we ever expose this assistant more broadly.
  ROI: For a single internal user (us), a managed agent might be somewhat overkill right now. We already have a working chat interface (intake-chat) and we enjoy tinkering with custom prompts. However, if our goal is to dogfood Google’s technology and prepare for a possible offering to others, trying out the managed agent builder is worthwhile. It might accelerate building a polished interface for, say, our docs site where users can “Ask a question” and the agent answers with sources. Setting up an agent with our data can be done relatively quickly (since we have Vertex AI Search index ready) – it’s more about configuration than coding. The cost of running an agent is similar to calling the underlying models (the agent will incur LLM usage costs for its responses, which are manageable for low volume). Given our focus, I’d classify the agent integration as nice-to-have, for experimentation. It’s not critical to our immediate needs since we can achieve similar results with our direct QA pipeline, but it opens doors. For instance, the agent could use conversation memory: it can remember what was said earlier in the chat, whereas our current system is stateless per question (unless we implement memory ourselves). This could improve user experience in multi-turn explorations of the archive. Also, the agent allows orchestrating multiple actions – e.g., it could answer a question and also create a draft ADR if asked (using a generative action), though that might be far-fetched at this stage.

• BigQuery AI.GENERATE_ Functions (e.g., AI.GENERATE_TABLE):* BigQuery’s new AI functions let us perform LLM operations within SQL, using the Gemini models under the hood (docs.cloud.google.com). This is a very intriguing capability for ad-hoc analysis. For instance, we could use AI.GENERATE_TABLE to have the model extract structured data or categorize text from our archive. Some potential uses:

  • Classify each ADR into topics or decision types. E.g., a query could prompt “Read the ADR text and output a table of (ADR_number, decision_category, date)” by having the model extract that info.
  • Analyze sentiment or outcomes of conversations. Possibly, use AI.GENERATE_TABLE to have the model scan conversation chunks and flag if it contains an action item or a decision not captured in an ADR.
  • Summarize log telemetry in a structured way (if we did ingest telemetry texts, we could ask the function to, say, list all incidents and their causes from a log summary).
  • Transcription or parsing tasks: If we had binary data (audio, images) referenced, we could even use AI.GENERATE_TABLE with ObjectRef to transcribe or caption them (since it supports multiple modalities (docs.cloud.google.com) (docs.cloud.google.com), though our primary data is text).

  The advantage of doing this in BigQuery is scalability and integration: we can join AI-generated insights with our existing tables. For example, we could create a view that augments each conversation chunk with an LLM-generated tag like “decision: yes/no”. Then we could query: “list all discussion chunks marked as containing a decision but with no corresponding ADR reference” – this could identify decisions that didn’t get formalized (one of our stated analysis goals). This kind of analysis would be very hard to program manually with regex, but feasible with an LLM’s comprehension, and doing it in BigQuery keeps it within our pipeline (and the results can be output as tables for visualization).

  ROI: For a one-person operation, these AI functions are power tools that can save time in analyzing text. Instead of manually reading through 100 ADRs to categorize them, we can get a first-pass categorization in seconds. It’s a form of accelerated analytics. Since we’re technical and comfortable with SQL, using these functions is within our skillset. The cost is the LLM usage – which in BigQuery’s case is measured in tokens of the model’s “thinking”. For small tasks, it will be low. For example, classifying 38k chunks might cost a few dollars (depending on token count, but BigQuery likely uses reasonably compact prompts for these).

  Another factor: it’s great dogfooding for us to test these functions. If we find good patterns (like using AI.GENERATE_TABLE to extract certain metrics from unstructured text), we can incorporate that into our workflows or even share those patterns in our docs.

  I’d prioritize at least one or two small projects using BigQuery AI functions as a Phase 3 after we stabilize search. For example, generate a table of “ADR decisions by quarter and whether they were reversed later”. This might involve some AI inference (checking if later ADRs or RQs reference earlier ADRs). We can attempt that with an AI function scanning text for references – something that was very hard before.

• Looker Studio Dashboards: With all data in BigQuery (and more data being generated, like usage logs, decisions, etc.), we can consider making some dashboards. Possible dashboards:

  • Archive usage dashboard: number of queries per week, what sources are most often retrieved (conversations vs docs), etc. This would require logging each query and result – we can log interactions from our chat or agent. For now, as a single user, this is low volume, but if we expand to more users, such a dashboard helps show value.
  • Knowledge base evolution dashboard: e.g., count of ADRs over time, topics of ADRs by category (if we categorize them as above), unresolved RQs count, etc. Since we have structured metadata (dates, IDs), this is doable without AI, but we could enrich data via AI as mentioned and then visualize.
  • D1 telemetry dashboard: If telemetry includes performance data or user behavior, a dashboard might be useful. For instance, if D1 is something like daily active user counts or error rates, a chart is more insightful than raw text. We could ingest telemetry into BigQuery (structured) and use Looker Studio to chart trends, anomalies, etc., possibly annotated with events extracted from text (like “major release deployed” from RQs or ADRs as annotations).

  ROI: For internal use, a dashboard can surface insights at a glance that might otherwise be buried. However, creating and maintaining a dashboard does take some effort, especially if the data needs preprocessing. Since it’s just us, one could argue we might as well run SQL queries when we want numbers. But dashboards shine in monitoring scenarios (you leave them up and notice changes). One potential high-ROI dashboard is cost monitoring – tracking our monthly GCP spend vs the $80 budget, broken down by service (BigQuery, Vertex AI, etc.). This could alert us if something is creeping up, without manually checking billing.

  Additionally, a dashboard or two could be a deliverable to demonstrate the system’s value. For example, if we ever present to someone how this archive is used, showing a live dashboard of “Questions answered over time” or “Areas of documentation most queried” is impactful.

  As a one-person company, the question is time: building a robust dashboard might take a day or two. Is that time better spent elsewhere? Because our primary goal is improving decision quality and knowledge usage, I’d say a simple dashboard for tracking KPIs (like the ones in §8) might be worth it. We could whip up a basic Looker Studio linked to a few BigQuery queries fairly quickly. It doesn’t have to be super polished, just functional.

Overall Assessment: The value of these managed layers is that they can amplify the benefits of our archive:

• The Conversational Agent makes the system more interactive and possibly easier to use (especially if we onboard others in the future – they might prefer a friendly agent interface rather than writing SQL or using our dev-oriented chat UI).
• AI analytic functions supercharge our ability to mine the archive for insights, fulfilling the “ad-hoc analysis” part of our use cases with speed. They open possibilities that were not viable manually (or would require writing a separate script with an API).
• Dashboards turn the insights and usage data into an at-a-glance form, which helps ensure we’re meeting our goals (and staying within costs), and can highlight trends (maybe we see certain topics lack documentation if we always ask questions about them – that insight could prompt action).

At this stage (dogfooding, 1 user), the conversational agent and dashboards are nice-to-have, whereas the AI functions are highly useful immediately for analysis. The agent and dashboards likely have a bigger payoff when scaling to more users or when needing to present the info to stakeholders (even if the stakeholder is future-us wearing a different hat).

Given our resource constraints (time more than money), I would:

• Definitely plan to use AI.GENERATE_ functions* in upcoming analysis tasks – low effort, high reward.
• Experiment with a Vertex conversational agent on our data as a prototype (maybe in a dev environment) to evaluate its responses vs our custom system. If it’s comparably good, we might integrate it into our workflow (e.g., use it to answer via a Slack bot or similar). If not, no harm done – we still have our custom approach.
• Create at least one simple Looker Studio report for ourselves: for example, a KPI dashboard that tracks recall (if we log when an answer is found or missed), latency stats, and cost. Also maybe a chart of how many ADRs and docs have been added over time, as a measure of knowledge growth. Keep it minimal initially.

In conclusion: Removing the residency shackle has unlocked these advanced tools, but we should adopt them judiciously. The investment vs. benefit is favorable for the AI analysis functions (which are quick to use ad-hoc). For the managed agent and dashboards, the benefit is a bit more long-term or indirect. They enhance user experience and monitoring respectively, but our current user (us) can manage without them. Still, given our interest in trying new GCP features, it might be worthwhile to allocate some time to set these up. As long as we stay within the $80/mo budget and our own available time, the experimentation value is high – we’ll learn from using them. And if any one of these starts proving obviously useful (say, the agent catches on and we find ourselves preferring it to the custom chat), we can lean into it.

To summarize the value proposition: these managed layers give us capability breadth – exactly one of our pillars. Even if the immediate ROI is moderate, they prepare the groundwork for scaling and demonstrate an integrated, modern solution. For a 1-person operation, it’s about finding the right balance of how many bells and whistles to turn on. We won’t let them distract from the core mission (which is having accurate, accessible knowledge), but we will use them to amplify that mission where reasonable.

7. Cost Structure Re-evaluation

Under the RQ-105 PoC, our monthly cost was extremely low (~$1) by using only in-region free components. Now, by introducing Vertex AI Search, Vertex embedding models, and possibly cross-region data transfer, we expect higher costs – but still within a modest budget. Let’s break down the cost components in the new architecture:

• Vertex AI Search Query Costs: We plan to use Vertex AI Search as the core. There are two pricing tiers we might use:

  • Search Enterprise Edition (with basic generative answers) at $4 per 1,000 queries (cloud.google.com).
  • If we enable Advanced Generative Answers (for follow-up questions, very long answers, etc.), that’s an extra $4 per 1,000 on top (cloud.google.com). So worst-case, fully loaded, $8 per 1k queries. Given one user (even a heavy user might do, say, 20 queries a day = 600/month), we’re talking <$5 per month on Vertex Search queries. Even if we somehow did 5,000 queries (which is 160 per day), cost would be $40. So this is comfortably within budget. Also note: Vertex Search offers 10k free queries per account per month (excluding generative answers) ([cloud.google.com](https://cloud.google.com/generative-ai-app-builder/pricing?hl=pt-br#::text=Teste%20sem%20custo%20financeiro%3A%20voc%C3%AA,Exclui%20respostas%20generativas%20avan%C3%A7adas)), but since we will use generative mode, we can assume those free queries might not apply or only apply to the search portion, not sure. In any case, the cost is low for our usage. Latency or concurrency is not an issue cost-wise since it’s per query, not per second or something.

• Vertex AI Search Storage Costs: This is $5 per GB per month after a 10 GB free allowance (cloud.google.com). Our entire text corpus (38k convo chunks + ~10k doc chunks) is probably on the order of 50 MB or less (each chunk maybe a few hundred bytes of text on average). Even with embedding indexes, it should still be far under 10 GB. So likely $0 here – we’ll be in the free tier for index storage. If not, at most a few bucks (if, say, uncompressed index or metadata somehow is a couple GB, which is unlikely).

• Vertex AI Embedding API Usage: We will incur costs when generating embeddings for our data (unless we rely entirely on Vertex Search to do it internally). Two scenarios:

  • Indexing cost in Vertex AI Search: When we ingest data into Vertex Search, it will internally call an embedding model to vectorize documents. It’s not explicitly billed per embedding; it’s rolled into either storage costs or maybe a one-time indexing cost. Reading pricing docs, it seems only storage and query are billed, not ingestion per se (aside from possibly charging for “Semantic add-on” in the configurable plan (cloud.google.com), but in the standard per-query model, the cost of embedding might be factored indirectly). In the example pricing (cloud.google.com) (cloud.google.com), they considered 100k docs (100 GB) and didn’t add extra cost for ingestion beyond storage.
    • However, if using the Configurable pricing model, there was mention of $1.50/GB per month for embeddings as an add-on (cloud.google.com). In our case, we’ll likely stick to the standard model initially (pay per query).
  • Manual embedding (BigQuery path): If we recompute embeddings via Vertex API for BigQuery usage, we’ll pay per token. For Gemini embedding, the cost is around $0.15 per million input characters (approximately, since $0.15 per 1M tokens and 1 token ~4 chars). Our total corpus might be ~27 MB of text (docs) + conversations maybe 10 MB, total ~37 MB of text. In characters that’s ~37 million characters. At $0.15 per million tokens (roughly 4 million chars), that’s about $1.39 (very rough calculation). Even adding overhead or inefficiencies, it should be well under $5 one-time to embed everything. And we don’t do this every month, just when data updates significantly. If we update, say, 100 new chunks in a month, that cost is pennies.
  • If we were to use OpenAI’s embedding for some reason: it’s pricier, but still on the order of <$10 for initial ingest. So either way, embedding generation is a minor cost relative to our budget.

• Vertex LLM Generation (for Q&A): If using Vertex AI Search’s included generation, the cost is already counted in the per-query fee. We might not separately call Vertex text models outside of that. But suppose we do ad-hoc BigQuery AI.GENERATE_TEXT or similar (which uses Gemini models), or if we use a Vertex model in custom flows:

  • Vertex PaLM 2 text models (like text-bison) cost around $0.003 per 1K input tokens and $0.006 per 1K output (just an example pricing). If we occasionally do that, it’s negligible. Even a 1000-token prompt+response is <$0.01.
  • If we somehow use the Check Grounding API or others heavily: The Check Grounding API returns a score, likely also costing on the order of a fraction of a cent per call (since it’s a small model). We might not need it often if Vertex Search is doing its job; maybe just for evaluation or if we integrate it explicitly.
  • The Ranking API if used is also a small cost per 1000 items ranked, negligible for our top-k sizes.

• BigQuery Costs:

  • Storage: 38k chunks + new docs chunks ~50k rows, with text and some metadata. That’s maybe tens of MB of storage. BigQuery storage is ~$0.02 per GB-month, so effectively $0.00x per month for us.
  • Query: If we continue to use BigQuery for any retrieval (maybe less so now) or analysis, we pay per data scanned. Our queries usually scanning 50k rows with maybe a couple columns – that’s trivial (maybe 50k * a few KB each at most = a few hundred MB worst-case if scanning text, or if using index doesn’t count all data). Each such search might be $0.001 or less. Even 1000 such searches = $1. In the PoC we saw ~$1/month because of the search queries.
  • Using AI.GENERATE_TABLE or similar in BigQuery will incur model inference costs. It uses the remote model pricing (Gemini). For instance, if we generate a table with Gemini 2.5 model, it might cost something like $0.02 per 1000 tokens processed (just ballparking). So if we process 50k chunks (each maybe 100 tokens) for classification, that’s 5 million tokens ~ cost of perhaps $0.1. Very cheap. The BigQuery integration doesn’t have an obvious premium beyond the model cost. BigQuery might also charge query cost for scanning the input text, but again, 50k rows of text is tiny (and if we filter columns, it’s fine).
  • Data Egress: Now that our computing (Vertex, BQ) is in US, and presumably we’ll also run clients in US (maybe our Cloud Run or Cloud Function if any, or just using the console), egress might be minimal. If we from Tokyo manually query data in US, that could incur egress (like downloading large query results). But likely minimal. If our chatbot still runs on a server in Asia and calls Vertex US, that’s egress on responses perhaps. But the responses are small (KBs per query). So pennies maybe.

• Cross-Region Considerations: We do have one cross-region element: our primary user is in Japan, interacting with services in US. Google Cloud traffic from US multi-region to an Asia client might incur network egress costs at ~$0.12/GB. Our usage (a few KB per answer) is negligible in volume. For example, 1MB of data egress (which is a lot of text answers) costs $0.00012. So nothing to worry about. And if we host any interface, we could even host it in a US region to minimize cross-region chat -> service calls (taking the hit perhaps when we view it from Japan, but again, static content is small).

• Monitoring and Hidden Costs: We should keep an eye on:

  • Vertex AI minimums: The pricing we rely on has no monthly minimum beyond usage. Some older services or some tiers might have flat fees, but Vertex AI Search’s standard model is pay-as-you-go.
  • Idle costs: none really, since we’re not planning on running any custom servers 24/7. For example, if we spin up a Cloud Run for an interface, that might have minimal always-on cost, but likely near 0 if infrequent use.
  • One-time costs: like the data transfer for migration (maybe $0.10 or so), index creation (BigQuery indexing has a cost in CPU but that’s part of query costs, not itemized).
  • Looker Studio: usually free to use; it will just run BigQuery queries (so any cost is from those queries). If we make a dashboard that auto-refreshes, we ensure the queries are lightweight or cached to avoid surprise BQ charges.

Now, compare this new cost structure to our budget threshold of $80/month (which was set as a “if above this, it’s not worth it” limit):

• By all estimates, we will likely be far below $80. Even if we generously add up:
  • Vertex Search queries: $5
  • Vertex Search storage: $0
  • Vertex embed (one-time): $2 (we won’t count this monthly)
  • Vertex agent usage: if we experiment, maybe a few dollars (each agent turn might cost similar to a query; if we do 100 turns, negligible).
  • BigQuery analysis queries: $1-2
  • BigQuery AI functions: $1 (maybe more if we do heavy analysis in a given month, but let’s say we do something big that uses 100k tokens in LLM -> maybe $0.50).
  • Misc (Cloud Run or GCS egress small stuff): $1
  • Total: on the order of $10 or less in a typical month.

Even if we have an unusually active month where we or others are hitting the system with thousands of queries, we might approach $20-$30. To hit $80, we’d need something like 10k+ queries with advanced LLM usage or doing a massive batch of AI analysis.

One potential cost increase could be if we incorporate OpenAI API usage, since that’s generally pricier. But in the new plan, we lean on Vertex for almost everything. We might still use Claude or GPT-4 in non-archive contexts (like coding, etc.), but that’s outside this system’s cost accounting. If we did want to, for example, occasionally ask GPT-4 to write a summary of something from the archive, that single call (~$0.03-$0.10) won’t move the needle much.

We should also consider that Gemini might have higher pricing tiers if we used a large version for generation. However, we plan to use the ones integrated into Vertex Search, whose cost is already included in the per-query fee. If we manually called text-bison-001 or similar for generation, it’s still cheap. Even if we had to call text-gpt-4 (not a Vertex model, hypothetical) it would be pricey, but we have no need with Vertex’s capabilities.

Cost of Expansion: If in future, say, we allowed a couple more team members to use this search, the query volume might double or triple, still fine. If we ingested a lot more data (say we add 100k more documents), storage might go into a few GB – maybe $10-$20 month max, still fine. So we have headroom.

Comparing to RQ-105’s situation: There, cost was basically negligible ($1). We are now accepting a higher monthly cost in exchange for quality. But the numbers show it’s still very low. We are nowhere near the $80 threshold. In fact, $80 would likely allow enterprise-scale usage (like millions of queries or tens of GB of data). For our scale, cost is not a major concern; it’s been relegated to secondary priority as decided.

To be concrete, let’s do a monthly cost estimate for expected usage:

• 1000 search/chat queries (with gen) = $4 (Enterprise) + $4 (advanced) = $8.
• Data storage: 0 (under free 10GB).
• BigQuery 50GB data processed (that would be huge for us; we usually process <1GB) = $5. In reality, maybe $0.50.
• Vertex AI analysis functions: Suppose we run 5 analysis jobs of 100k tokens each = 500k tokens = maybe $0.50.
• [Optional] Cloud Run to host a small UI: If using always-on, maybe $5; if using Cloud Run on demand, likely <$1.
• So probably $10-$15 total.

We should still track usage because if, for example, the generative mode usage somehow skyrockets (maybe the LLM chooses to use a lot of tokens per answer, or advanced mode with follow-ups triggers multiple calls per query), costs could creep. But given answers are short-ish and we control the modes, it’s predictable.

Cross-region charges: Since bucket is now US multi and data processing is US, our biggest egress might ironically be if we (in Asia) download a lot of data from GCS or BQ. But interacting with it via GCP console (which endpoints?) might actually not charge egress if we’re doing it within the console. Hard to guess, but any such egress will be minimal. If it becomes an issue (like we notice charges we can’t identify), we might choose to deploy a proxy in US or something.

Cost of alternatives: It might be informative to note that if we had stuck purely to BigQuery for search and a bit of local LLM, we had $1. So now maybe $10. That $9 difference is absolutely worth it for the improved capabilities (which would otherwise require lots of dev time to approximate). Also, consider if we had gone with OpenAI embeddings and GPT-4 for answers:

• Embeddings one-time $3,
• GPT-4 at 600 queries (assuming 1000 tokens each) ~ $18 (since ~$0.03 per query),
• plus some vector DB cost maybe. It might come to similar ~$20, but OpenAI would have privacy and integration issues. So Vertex appears more cost-effective for what we get (plus deeper integration).

Conclusion: The new architecture remains cost-efficient. We forecast monthly costs on the order of tens of dollars at most, well below the $80 threshold. We should plan to monitor costs through GCP’s billing reports or set an alert at, say, $50 just in case. But given usage patterns, it’s unlikely we’ll approach that. Even scaling up usage moderately or adding features won’t blow the budget.

One area to be mindful: if we enable something like continuous indexing or have a bug that re-ingests data repeatedly, that might cause unexpected charges (for example, if we accidentally send the same document to Vertex Search 1000 times due to a script bug, we might incur lots of needless embedding operations or index churn – still likely minor, but good to watch).

We will incorporate cost monitoring as part of our KPIs (see next section) to ensure we never unknowingly exceed our comfort zone. But overall, cost is no longer the limiting factor as it was in the initial PoC – we have plenty of buffer to prioritize quality and capability now.

8. KPI and Retreat Criteria Revisions

With the approach pivoted to emphasize quality and capability (and relaxed cost/residency constraints), we need to establish new Key Performance Indicators (KPIs) and clearly define “retreat” conditions (when to reconsider or roll back the approach). We’ll compare these to the previous KPIs from RQ-105/PR-2849 and note what’s changed.

Proposed KPIs for the new approach:

• Retrieval Effectiveness (Recall@10): This remains a critical metric. We aim for at least 90% recall@10, meaning for any factual question answerable from the archive, the correct supporting chunk is present in the top 10 retrieved. In RQ-105 PoC, we achieved ~0.90 recall@10 with the hybrid BQ approach. With the new embedding model and Vertex search ranking, we hope to maintain or exceed this (maybe 0.92–0.95). We will measure this by a set of test queries (perhaps the same set used in PoC evaluation, plus new ones for docs/ADRs). If recall@10 drops significantly (say below 0.85 consistently), that’s a red flag – maybe the new embedding isn’t working as expected or chunking needs adjustment. High recall is non-negotiable for an effective RAG system, so this is our primary quality KPI.

• Grounded Answer Precision / Grounding Score: We want answers that are correctly supported by sources. A possible KPI is “Precision of grounded answers” – e.g., the percentage of answers where all factual claims are backed by the provided citations. We could use Vertex’s Check Grounding API to compute a score (it gives 0 to 1) (docs.cloud.google.com). We might set a target like “Average CheckGrounding score ≥ 0.8” for answers, or that 95% of answers have no unsupported statements. In RQ-105’s custom pipeline, we attempted to enforce grounding via a final check. Now, with Vertex doing this internally, we’ll rely on its quality. If we notice hallucinations or unsupported claims sneaking in, that’s a KPI failure. Essentially, this KPI measures accuracy/trustworthiness of the answers. It’s a bit qualitative to measure without ground-truth for every question, but we can sample and verify. A related metric: Citation Precision – are the citations actually relevant to the answer (not just randomly retrieved)? That should be high if grounding is good. We can manually review a sample of answers and ensure, say, >90% of citations are directly supporting the content of the answer. Any lower and we need to tweak the system (maybe retrieval or prompt).

• Latency (P95 response time): User experience depends on how fast the system answers. We set a KPI for latency, for example **P95 < 5 seconds** for a full query-response cycle (and P50 perhaps ~2 seconds). In the PoC, BigQuery + local LLM was perhaps around 4-5 seconds typically. Vertex Search should ideally be faster. We should measure over a set of queries the time from query submission to answer received, and ensure the 95th percentile is under our threshold (5s is a reasonable upper bound for interactive chat; faster is better, but 5s is okay). If we see many queries taking >5s or spikes beyond 10s, that’s an issue. Latency could be impacted by network or large outputs, but given our scale it should be fine. This KPI ensures performance is acceptable and influences user satisfaction. If latency goes beyond the threshold consistently, we’d investigate (maybe index issues or choose a simpler model for generation).

• Cost Control: While cost is deprioritized, we still have a monthly budget cap of $80. We will track the monthly accrued cost for this system. A KPI could be “Monthly cost ($) vs $80 budget” – with a target of staying <50% of that in normal months (i.e., <$40 typical, which we expect to do). If we ever exceed, say, $80 in a month without deliberate decision, that triggers a review. Actually, even exceeding ~$50 unexpectedly would prompt investigation. We may set up billing alerts at $50 and $80. This is essentially the retreat cost threshold: if costs consistently approach or exceed $80 and we don’t see proportional value gain, we’d consider scaling back features (maybe disabling advanced mode, etc.). However, given our earlier analysis, this is unlikely for one user – but we keep it as a guardrail. In PR-2849, the $80 was a “kill switch” threshold. That remains: if our experiment somehow balloons in cost (maybe due to far more usage or mistakes), we must re-evaluate architecture or usage.

• Capability Utilization (Feature Adoption Count): One of our goals is capability breadth – leveraging new Vertex features. We can define a KPI to ensure we actually use those features. For example, how many of the following have we successfully integrated: Vertex AI Search, Vertex Agent, Vertex Ranking API, Grounding API, AI Functions in BQ, etc. We don’t need all, but we might set a target like “Integrate at least 3 managed Vertex AI capabilities into our workflow.” Right now, we have Vertex AI Search (1), likely will use AI.GENERATE for analysis (2), possibly a conversational agent (3), and maybe the Ranking or Grounding API indirectly (4). Achieving a number indicates we’ve extended our toolkit as planned. This is a qualitative KPI, but it helps track that we’re taking advantage of the “capability breadth” we prioritized. If we ended up, for instance, not using Vertex Search and falling back to old methods, that would mean we failed to leverage capabilities (which could be a retreat sign – something must have gone wrong to abandon those).

• User Satisfaction / Solution Adoption: Since the “user” is primarily ourselves, this is subjective, but important. Are we (and any future users) actually preferring this new system for getting info? One could measure this by counts of queries or interactions per week (if it’s useful, we’ll use it often). If we find ourselves not using it and instead manually searching docs, that’s a problem. So a possible KPI: usage frequency – e.g., at least N queries per week. Or simpler: track number of resolved questions or tasks using the system vs outside methods. We might set a goal like “use the RAG system for >80% of information queries related to project history/decisions.” This ensures the system is delivering value and being adopted in practice. Low usage would indicate either we don’t trust it or find it cumbersome (hitting maybe quality issues), which would be cause to re-think. So, as a KPI: we’ll monitor how often we use the archive search (via logs), aiming for increased usage as more content is added (because it’s more useful).

• Content Coverage KPI: Now that we’re ingesting more sources (docs, ADRs, etc.), we might want a metric for how completely the relevant knowledge is in the system. For example, % of ADRs ingested vs total ADRs (target 100%), % of new docs ingested within X days (if new docs are created, are we adding them promptly). This is more of an operational KPI: ensures the archive stays up-to-date. We can set a goal that, say, by end of the ingestion phase, 100% of the ~856 docs are indexed. And moving forward, any new internal doc or ADR will be indexed within, say, 1 week of creation. This prevents the system from stagnating or missing recent info. If this KPI fails (e.g., we forget to ingest new ADRs and the system starts giving outdated answers), that undermines trust.

• Maintenance Effort KPI (Ease-of-Ops): One reason for using managed services is to reduce maintenance. We can track the operational overhead in terms of hours per month spent maintaining the system. We’d expect after initial setup, near-zero maintenance (maybe just monitoring). If we find we are spending significant time troubleshooting Vertex AI Search or adjusting indexes, that indicates the simplicity benefit isn’t materializing. So qualitatively, KPI: “Monthly maintenance time < 2 hours” (just as a gauge). This is informal but if it starts taking much more, we should question if the managed route is actually easier.

Now regarding Retreat (Rollback) Criteria: When would we consider reverting or changing course? Based on the KPIs above, triggers for retreat might be:

• Recall@10 or answer quality significantly worsens: If despite using bigger models and Vertex, we see that relevant info is not being retrieved or answers are incorrect more often than before, that’s a fail. For example, if recall dropped under the 0.8 mark, or we encounter frequent missed answers that the old system got, it suggests something’s wrong (maybe the new model doesn’t handle domain jargon or the chunking strategy lost context). In that case, we’d need to adjust (maybe fine-tune, or in worst case, consider going back to a known-good approach).
• Latency or usability issues: If the new system is too slow or unreliable (e.g., Vertex Search downtime or high latency), that could force a retreat to a simpler approach. Unlikely, but we monitor it. A 95th percentile latency > 8-10s or frequent timeouts would be unacceptable.
• Costs exploding beyond budget: If we somehow approach that $80/month regularly and that cost isn’t justified by equivalent value, we’d retrench. However, as we saw, it’s unlikely with planned usage. If it happened, perhaps due to increased user count or some inefficient usage, we might either optimize (like use a cheaper model) or consider alternative architectures (maybe self-hosted vector DB to avoid per-query costs, etc.).
• Managed feature under-delivers: If, for example, Vertex AI Search’s generative answers were low quality or too generic, and we find our custom QA was better, we might consider not using that part. Retreat doesn’t necessarily mean scrapping Vertex entirely, but could mean adjusting scope – e.g., using Vertex for retrieval but doing custom generation again if needed.
• Security/Policy changes: If any new constraint appears (say company policy suddenly disallows US processing or a client’s data enters the mix requiring on-prem), we’d have to retreat from Vertex Search and revert to an in-region solution. This is more external, but worth having as a criterion (since originally residency was the reason).
• User feedback: If others (or future us) find the system’s answers unhelpful or prefer old ways (like manually Googling through docs), that indicates failure in usability. We should gather any feedback (even our own impressions) and if negative, rethink design or training.

Comparing to RQ-105/PR-2849 KPIs: Previously, key focus was: cost minimal, data residency, and basic recall. We achieved cost ~ $1 (target was <$5), recall ~0.9 (target ~0.8 maybe), and residency compliance. Now, cost and residency are relaxed, so our KPIs shift to quality and capability:

• We raise the bar on quality metrics (ensuring high recall and precision of answers).
• We add metrics about advanced usage (capabilities integration, usage frequency).
• Cost becomes a secondary metric just to ensure it’s controlled.
• Residency is removed as a KPI entirely (not relevant now).
• In PR-2849, perhaps there was a KPI like “zero data leaves Tokyo” – that is dropped.

One new KPI domain is benefit realization: Are we actually getting better decisions/insights thanks to this system? That’s hard to quantify but we can attempt proxies. For example: in the next quarter, did we catch any mistakes or save time due to using the archive? We might log “success stories” (qualitative). Or measure reduction in time to find info. If historically, to find an ADR detail took 10 minutes of manual search, now it takes 1 minute via RAG – that time saving is a success metric (not easily measured without logs, but we can anecdotally note it).

For retreat threshold differences: in RQ-105, retreat was basically if cost > $80 or if recall was too low or if technical feasibility failed. We fortunately succeeded so far. Now, retreat is less about cost (unless truly runaway) and more about if the fancy approach fails to deliver better results. A partial retreat scenario might be: if Vertex AI Search doesn’t perform well, we might fall back to BigQuery hybrid for retrieval but still use a better embedding. Or if generative answers have issues, we might revert to our own prompt with more control. These nuanced outcomes are possible. But a full retreat (like scrapping the whole idea of semantic search) is unlikely given how much value we already see.

We can formalize a couple of hard retreat triggers:

• If recall@10 < 0.8 for our evaluated questions AND we cannot fix it quickly, we should consider returning to the prior solution (or another solution) for retrieval.
• If monthly cost > $80 for 2 consecutive months (without an intentional project scale-up) – re-evaluate continuation.
• If avg grounding score < 0.5 or frequent hallucinations – need to change approach (maybe go back to simpler answers or include more references).
• If the integrated system fails in a critical moment (e.g., gave a confidently wrong answer that misled a decision), we pause and inspect whether the process failed – if it’s systemic, maybe the approach isn’t trustworthy enough, prompting alterations.

Finally, we keep a continuous improvement mindset: these KPIs will be monitored and if any fall short, we iterate (choose a different model, adjust chunking, etc.). Retreat (like scrapping Vertex Search) is a last resort if we can’t meet KPIs through tweaks.

To wrap up, the new KPI list vs old:

• Quality: Recall@10 ≥ 0.90 (unchanged target, but now with higher expectation it stays high across more data), Answer grounding high (new focus).
• Performance: Latency P95 ≤ 5s (similar to before maybe, but now explicitly tracked).
• Cost: ≤ $80 (was key before, now just a ceiling; earlier we targeted ~$5, now we accept higher but cap at 80).
• Capabilities: at least 3 new Vertex AI features in use (was 0 before by design; now a measure of breadth).
• Ease-of-use: measured via our usage and satisfaction (qualitative; previously not measured, as user was just us anyway).
• Maintenance: time low (previously the system was also low maintenance by being simple; we assume managed services also low maintenance – if not, that’s a disappointment).

By monitoring these, we can declare success or identify issues early. Ultimately, the key metric of success for this project is: Are we getting better, faster, more reliable insights from our archive than before, without undue cost or effort? All these KPIs feed into that central question. If the answer is yes (which we expect), then the architecture re-evaluation is a success. If not, we’ll know where the shortcomings are (thanks to the KPI breakdown) and can address them or roll back accordingly.